When we stoop-backed, wizened online veterans gather around the pot-bellied stove down in the basement of the Smithsonian Institution, our cackling often turns to tall tales of the early days of online. For, as we ancient warriors of the online wars like to tell it, "We were online before online was online."
Why, you young whippersnappers, I can remember all the way back to when subject headings and subject heading authority lists first became descriptors and thesauri. And I've lived to hear the very same objects renamed metatags and taxonomies. Well, whatever sells, young-un's, whatever sells.
The term "search engine" has its predecessors, too. In decades gone by, we referred to the software that built accessible data compilations as information retrieval software. Today's "search engines" have their own confusions of terminology. The most prominent and high profile use of search engine software in the Internet Age comes from Web services like Google, AltaVista, AlltheWeb, Teoma, and their kin. In fact, when people use the term "search engine," they are, more often than not, referring to these services, which build sophisticated, searchable databases from data gathered by "spider" software programs. These databases are designed to guide Web users to the perfect site to suit any and every purpose. Actually, even among such services, one can see the role of the search engine software extend beyond Web tracking to reach non-Web material, such as Google Group's Usenet archives. Now that many of the major Web search engines have begun selling their software to institutional intranets and to organizations connecting large in-house databases to Web site users, the confusion in terminology could grow.
Terminology aside, today's search engine software often employs techniques that were only a dream to online searchers from the earlier era of Dialog, LexisNexis, Dow Jones (now Factiva), and the lot. The "traditional" search services of that earlier era often still run "legacy" software underneath their spruced up Web offerings. However, Web development continues to challenge the old-time online services. There is no question that top of the line ecommerce sites, like Amazon, have taught generations of Web users to expect a lot of TLC from a good search engine. This can make for a nasty shock when the same user approaches an "i-commerce" site only to confront much less effective and almost user-hostile software. That initial shock blossoms into a hard sell when the user spots the rates the i-commerce site plans to charge for the search experience. Ecommerce sites have a lot to teach the pure content sites in how to keep users happy.
However, some key observers of the ecommerce scene believe that, when it comes to search engines and their applications, everyone has a lot to learn. I spoke with Marshall Harrison, founder and CEO, and Sara Hiner, president, of RelevantView, a software house producing tools to test Web usability. They pointed out that many vendors new to the ecommerce field don't understand the value of fully analyzing the search experience from the user's standpoint. Webmasters often don't analyze or, in some cases, even gather all the information they could on user behavior. Without that information and analysis, improving the search experience becomes very difficult. Many sites offer minimal indexing or categorization, according to Harrison and Hiner. Sites don't integrate free-text specific search terms supplied by the user with the categories or taxonomies developed by the vendor into an easy natural browsing medium.
Taxonomy Without Representation
Larry Harris, founder and chair of EasyAsk has worked with both ecommerce and pure content sites in applying a sophisticated new search engine. And, without even looking at EasyAsk in action, I would bet it works well. Harris tells us that they offer clients a "no benefit, no pay" pricing plan that ties the fees paid EasyAsk to improvements in revenue after adoption of its search engine. With this kind of a "shared risk" pricing structure in place, Harris makes very sure that his search engine has improved a site's usability substantially before he'll let it "go live." Nonetheless, he admitted that, at the end of the day, most clients only use the search engine at 40% of its capacity. Some clients insist on using old taxonomies not suited to all the ways people can refer to a product; other clients have no taxonomies at all.
Webmasters of ecommerce sites have one major advantage over traditional online services. No matter how large their databases—and, in the case of major retailers like Amazon or Buy.com, files can contain millions of records—they have a pretty definite idea of the way the databases should or will be used. They design their databases and apply the search engine features always with the goal of supporting a purchase transaction.
Traditional online services build massive arrays of data, stretching across thousands of sources and dozens of years. But the sheer size of the data arrays and the rigidity in some of the legacy software makes it difficult for them to design effective paths to specific answers. So they often leave the struggle up to the user ("There's a pony in there somewhere."). More and more, i-commerce sites have come to realize that this approach will not work any longer. Some have begun controlled experiments with portions of their data arrays to see how new advanced search engines would work. For example, LexisNexis recently went online with their new application of iPhrase Technologies' OneStep software on their Advertising Red Books and Directory of Corporate Affiliations.
Oddly, the oldest of the traditional pure content players—primary publishers of journals, newspapers, trade press, etc.—have jumped to the future better than the traditional online services by waiting for the Web to go online. ("I'm so far behind, I think I'm first.") On publisher Web sites, users will usually see the familiar empty dialog box with the button "Search" next to it, waiting for searchers to slap any terms into it. Because publisher sites create and control their own data, their designers may have better awareness of how users will approach specific sources than large database aggregators. Unfortunately, though, no matter how large the publisher, users will inevitably want to extend searches beyond a single publisher's product. Some publishers in the sci-tech or scholarly publishing arena have begun to explore add-on alternatives. The Publishers International Linking Association's CrossRef service plans to produce a full-text service merging the collections of many publishers. A user will go to one publisher's Web site, conduct a search, and—if still dissatisfied—click on the CrossRef Search option to expand the search across many publishers' collections.
Shopping for an Education in Good Searching
The best applications of search engines can give users a sense of intimacy joined with a sense of the majesty and magic of online. For example, go into a mega-shopping site like Yahoo! Shopping or Google's Directory for Shopping and put in a name—yours, a beloved relative's, or a dear friend's. Whoosh! Up jumps a number of sites where people sell products using that name: The Ultimate Gift Selector! Did you know there's an outfit in Texas called Quintessential Spirits that makes chocolates filled with bourbon or scotch or even peach schnapps? (All gifts to author humbly accepted.) Last year's Christmas present to some cousins up north came from a small Kentucky distiller that specialized in single-barrel Bourbon with the same name as my cousins. I even got double credit for the gift. The lovely bottle came in a brown velvet wrapping with the name of the distiller embroidered on the outside. The cousins thought I had embroidered their name myself. (We're not as close as we should be. New Year's Resolution # 262.)
Which shopping portal should one choose? Well, look to the add-on features. Google Directory aims at supplying a broad base of information on all kinds of hidden data, shopping sites included. It can probably find more sites or, if it doesn't, it soon will. Yahoo! Shopping aims more at a complete purchase transaction and wants users and listed vendors to subscribe to its ecommerce support (with features like Yahoo! Wallets that simplify the final purchase form). With a sales goal in mind, Yahoo! spends a good deal of effort on categorizing—letting users set price ranges to filter results, and on digging into and linking to each vendor's inventory.
Amazon.com's greatest strength in using their search engine for books and music and videos, lies in its talent for lateral movement to supporting databases. You find an item and get a complete description, including bibliographic details, published reviews, ratings and reviews from "real people," samples of the material (chapters, selected tunes, etc.). At the same time, you will get a list of items other people who purchased this item purchased, a way to reach more from the same author or artist. In the initial listing following a search, sideline entries connect you to lists of recommendations from Amazon users, lists selected based on the presence of your item. Here comes the intimacy factor. It's like striking up a conversation in a store while browsing and finding a really knowledgeable person giving you free advice.
Speaking of Amazon, as we went to press, a price war had broken out between Amazon and its competitors, primarily in the book, music, and software area. Buy.com had challenged Amazon by offering free shipping and handling for anything light—no TV sets or heavy appliances. If that weren't enough, they then promised to undercut Amazon's prices by 10%. Other online book and music vendors had also started cutting prices. Amazon had already cut the minimum order for its own "free" shipping and handling offer from $99 to $49.
Loyal though I am to Amazon, as a consumer, I had to consider my options. What about going to Amazon, using all their wonderful data, making a list of desirable items, and then sidling over to Buy.com with list in hand? Then I realized that Amazon had me in a vice of customer loyalty. The greatest fear of an impulse buyer of books or music is that you've already bought it. I cannot count the times that I have been saved from such folly by Amazon's quick check of its records to confirm a previous purchase (date included). If I buy books or CDs from anyone but Amazon, I lose not only access to their massive inventory databases, but the protection of their database on me as a purchaser.
Though ecommerce sites set the standard in use of search engine features to improve the user experience, the old traditionals may still have a lesson or two left for the Net Newbies. Most important, they would remind ecommerce database handlers to respect data in and of itself. Right now, a major defense against Buy.com lies in the sheer size of Amazon's inventory database, but some years back Amazon began to drop smaller publishers making it less complete than it had been. They may come to regret that decision.
Bottom line: i-commerce sites know that people will pay for information itself—not all the people, not all the time, but enough people, enough of the time. Major ecommerce sites might re-examine their databases to see if they have data that could sell. For example, since that darned Ken Burns documentary series on Jazz played, I have indulged in an orgy of CD purchases that have turned me into my city's foremost library of Jazz Through the Ages. Problem: my inventory is completely out of control. Now Amazon not only knows what I bought, they have the names of the artists, the dates, and the names of all the tunes. They have all the data I would need to construct a personal inventory control system. Why not sell me the data and maybe even a software interface with which to maintain it on my PC and receive periodic updates? This would also give them a chance to monitor when and where I had not chosen them as my vendor. They might even partner with a music software publisher or one of those emerging jukebox services to create inventories.
Traditionals reading this article may be asking, "But what about us? What were those lessons we should be learning from the Net Newbies?" Unfortunately, EContent doesn't have enough paper available for that lengthy a list. (For complete coverage in the what-should-I-do-next department, my Quint's Online column in Information Today is online full-text in most major full-text aggregations.) In the short space left me, let me give you one valuable word of advice: feedback. If you want customers, you must listen to customers. You must seek out their opinions, soliciting them at the end of every search. Even if they don't fill out the forms, it's just common business courtesy to ask. You should recognize customers when they return online and automatically customize the interchange, not require the customer to program your machine themselves to get attentive service. The prices you charge do not allow for a self-service policy. Keep your support desks open 24/7/365. Link data to what people want and expect, (e.g., bibliographic citations connected to full text). Accommodate any payment mechanisms that suit the payee, not just the ones that will not bedew the brows of your accounting staff. (If you need new accountants, there's a garage sale going on at Arthur Andersen.) If the feedback you get from customers tells you that your data did not answer their questions well enough, then find sources that can supply those answers—up to and including individual authors—and solve the problem.
When you charge solution prices, you must provide real solutions and, when it comes to solutions, only the people with the problems can tell you if you've succeeded—or not. To get the feedback, you must attract the customers with compelling content, sure, but to keep those customers, you will need to listen (and respond to) the bad and the good. To keep the customers coming back for more, you need feedback.