Hits and Misses

The sun shone on Santa Clara in October as the combined Intranets 2002 and KM World conferences took place in the Convention Center. Information Today Inc. took quite a gamble in running the events at such a difficult time for the economy and corporate budgets, and would no doubt like to have seen more delegates and exhibitors. But those who came certainly gained considerable benefit from attending the six tracks of presentations, as well as many workshops and evening events. Indeed, one of the challenges was how to be at several different presentations at the same time! However, there is one thing from the event that I will recall for many years—HITS is an acronym for How Idiots Track Success (in an intranet context). Embarrassingly, I can't recall where I heard it, so if the owner would like to send me an email, I will ensure that due tribute is paid in a subsequent column.

One of the problems of intranet management is that it is difficult to establish reliable metrics for an intranet that enable its effectiveness and progress to be monitored, or for an intranet to be benchmarked against other intranets. One service that does provide some benchmarks comes from CIBA Solutions in Australia (http://www.cibasolutions.com.au) and there may well be others. I myself am working with a group of intranet consultants to develop a checklist approach that while enable an intranet manager to gain some appreciation of the extent to which their intranet(s) match current good practice, and I may have more news on this service in a couple of months.

To go back to HITS, one metric that does not work on its own is page hits. This is a passable metric with Web sites, but a simplistic ranking of intranet pages can be very misleading. Recently, I was using a client's intranet, which contained content on China along with a number of other countries, and quite often a search was carried out just for the word "China." Close to the top of the list of search results was the Web page for an airline that the organization frequently used, which referred to the use of china crockery for the meal service. I could not believe what I was seeing on the results list, and clicked on the page. A later analysis of the logs revealed that this airline page was quite popular, just because everyone clicked on it on the way to looking for other (more relevant!) documents.

Any analysis of hits has to include the context for the page. How was the page found, and what did users do after viewing the page? This is quite laborious even with a good stats package, but this is the only way that you are going to gain any meaningful information. This analysis has to be complemented with a range of other techniques. In my view, the most important metric is the impact that the intranet is having on information handling and decision-making in the organization. The only way to find this out is to conduct a regular survey, perhaps every six months, which includes a mixture of quantitative questions ("How long on average do you use the intranet each day?"), and also qualitative questions which help to identify examples of where the intranet had a significant positive impact on someone's work (publicize widely) or the user failed to find what he or she was looking for (fix the problem).

Tracking search inquiries can also be illuminating. The logs will probably indicate where the other navigational features have failed to be effective, or where information is being sought on a topic that was not initially seen as important in other aspects of the information architecture. Any logs like this need to be seen in the context of the person and department involved so that trends are seen rather than there being a knee-jerk reaction to an individual complaint. Incidentally, do pay especial attention to the searches that resulted in no hits. This means that you should be considering the extent to which your search engine vendor is able to provide useful log information when selecting a search engine.

Other ways of assessing effectiveness include ensuring that the intranet is on the agenda for regular departmental meetings so that suggestions and criticisms can be tabled openly, rather than at the water cooler. Just walking around and talking to users can also shed valuable light on use patterns. The bottom line is that you need to build a suite of metrics, and not rely just on what is easy to measure.

Linked into this discussion about assessing use is the closely related subject of usability. Experts like Jakob Nielsen have been talking up intranet usability for some time now, and conducting a carefully constructed suite of usability tests will reveal much about the way in which the intranet is being used.

All of this information should be widely communicated on the intranet itself, even if all the news is not good. The openness of this tactic will almost certainly gain you valuable suggestions on how to continue to enhance the value of the intranet in meeting organization requirements.