EVALUATING ELECTRONIC PRODUCTS AND SERVICES

By Richard W. Boss

Libraries are spending an ever-increasing percentage of their acquisitions budgets-as much as 35 percent--on electronic products and services. Electronic "products" are information resources that are purchased and mounted on in-library computers (standalone PCs or multi-user servers); electronic "services" are information resources mounted on remote servers at a producer or a service bureau. A service bureau that mounts several information resources is usually called an "online reference service," but when it provides access to several different electronic services it is more appropriately called an "information aggregator."

As recently as five years ago, electronic products--especially those on CD-ROM--were more common than electronic services, but with the dramatic reduction in telecommunications costs since the emergence of the Web, electronic services have become far more important for libraries. Most of the electronic products that libraries continue to purchase are ones that were developed for the consumer market, among them games, dictionaries, and encyclopedias. Electronic services include indexes and abstracts, the full-text or images of journals, and a wide variety of reference materials.

Given the amount of money spent on electronic products and services, it is important that libraries be able to determine whether their patrons are utilizing the products and services enough to warrant their costs. The simplest way of doing that is to determine the cost per use-a log on and search against a specific product or service that produced a result that is viewed, downloaded, and/or printed. While that does not answer the question of value to the user, it does make it possible to identify the electronic resources that are too infrequently used to justify the cost.

Determining Use

The number of uses of an electronic product can be determined by mounting "key logger" software on each PC from which access to the product is available, on a network server, or on a Web server or library portal through which access to electronic products is routed. PC Activity Monitor, Starr Pro, Pro Bot SE, and Spy Agent are among the most widely used packages. A summary of the features of each is available at www.keylogger.org.

The number of uses of an electronic service can be determined by using statistics supplied by the online reference service or information aggregator, or by mounting access management software on a network server, Web server, or library portal through which the access is routed. A number of the access management software packages that include use measurement are bundled with blocking/filtering and print control software. Among the most widely used are Comprise’s Internet Manager and Cybrary N’s Internet Control. Libraries that only want to measure use, not restrict it, should consider a more narrowly focused metering product such as EnvisionWare’s Launch Command.

There is an effort underway to develop a single, international extendible Code of Practice that allows the usage of online information services to be measured in a credible, consistent, and compatible way using vendor-generated data. The COUNTER Code of Practice specifies: the data elements to be measured, definitions of these data elements, usage report content, format, frequency and methods of delivery, protocols for combining usage reports from direct use and from use via intermediaries. Release 1 of the COUNTER Code of Practice issued in December of 2002 focuses on the usage of journals and databases. Future releases will extend the scope of COUNTER not only to other content types, but also to more detailed levels of reporting for each content type. It may be another two years before Release 2 becomes available. See www.projectCOUNTER.org for additional information.

Determining Cost

A library must not only determine the number of uses of a product or service, but also the cost. Ideally, a cost study should look at all costs, including not only the cost of the purchase or subscription, but also all equipment, supply, personnel, and overhead costs. Such a comprehensive study can be time-consuming and expensive to undertake and is justified only if there is reason to believe that there will be significant differences in the costs other than product purchase or service subscription. After undertaking a score of studies, the author is convinced that looking only at purchase price or subscription cost and number of uses usually can make an evaluation.

The cost of a product should be annualized by determining the frequency with which the library is likely to purchase new editions and determining the cost for each year of ownership. For example, a CD-ROM that costs $450 and is not going to be replaced for three years would have an annualized cost of $150. In the same way, the cost of a subscription should be annualized. For example, a two-year subscription price of $3,000 would have an annualized cost of $1,500.

Cost Per Use

The cost per use will vary dramatically depending on the type of product or service, the breadth of its appeal, and the pricing policy of the provider. An encyclopedia on CD-ROM will usually cost less than $.01 per use; a specialized database such as Biosis that is accessed through an online reference service may cost as much as $5.00 per use in a small medical library environment. Not only is the encyclopedia less expensive to produce, but also it has much broader appeal.

The source of the electronic information also has an effect on cost per use. An ERIC clearinghouse database may cost less than $.20 per use because it is created using federal funds while most STM (science, technology, and medicine) publishers are for-profit organizations that take substantial mark-ups.

Evaluating Costly Products and Services

As a rule of thumb, a library should look critically at all products and services that cost $.75 or more per use. That does not mean cancellation, however, because the product or service may fill an important need. The evaluation might include interviews of users to determine how valuable they consider the product or service. The findings could lead to any one of several courses of action: price negotiation, seeking an alternate provider, reducing the number of user licenses, sharing the subscription with one or more other institutions, substituting a comparable product that is available at lower cost, or cancellation. A library may also choose to promote the product or service to increase use by mentioning it to patrons, putting up posters or publishing news stories, and/or including a bibliographic record in the patron access catalog.

Recommendations for Optimizing Cost Per Use:

The author has reviewed the evaluation reports he has written over the past five years and has generalized the requirements that he submitted to his clients.. They are:

1. Maintain a comprehensive collection development policy that encompassesinformation in all formats: print, electronic, and audio-visual.

A library should specifically address electronic products and services in its collection development policy. It should develop and maintain electronic resources selection criteria, and provide for special scrutiny of electronic products and services that are republications of printed products and/or that are expensive. An electronic product or service that is a republication of a print product should be purchased only when it is superior to the print version in cost and/or utility.

There are some electronic products that are less expensive than their print counterparts because they are less expensive to produce and sell that way. The best examples are electronic encyclopedias. Some electronic services are less expensive than their print product counterparts because a number of print products have been combined into a single electronic service. The key then is not whether the total value of the "package' is better, but whether the comparative cost of that which is wanted favors the electronic. A product that contains hundreds of periodical backfiles is an example.

A library should avoid acquiring and retaining multiple formats of the same publication. If an electronic product or service is purchased, a library should avoid maintaining a subscription to the print product. The exceptions should be publisher pricing which makes a dual subscription cost effective or there is a risk that subsequent cancellation of the subscription to the electronic product or service will leave the library without a needed backfile.

Utility should be addressed in the collection development policy. Utility has at least four aspects: currency, ease of retrieval, ease of manipulation, and ease of remote access.

While many electronic products are available on a highly current basis, some are published less frequently than their print counterparts. The greater economy of the electronic product may not be worth it if the users are less well served. However, most electronic services are more current than their print counterparts.

Retrieving information from an electronically republished monograph is often no easier or faster than working with the print version, but searching through an electronic bibliography or periodical index is substantially easier, faster, and more reliable than leafing through hundreds or thousands of pages in multiple volumes.

If the information is of a type that the user wants to manipulate, an electronic product or service offers a substantial benefit over print. However, the benefit is realized only if the information is truly "full-text," meaning that it is machine-readable. Digitized images are no more flexible than photocopies. Many of the so-called "full-text" files offered by electronic republishers consist primarily of images, rather than full-text.

Remote access is a great advantage of electronic products and services. A person with the appropriate hardware and software need not come to a library, but can access electronic resources remotely from elsewhere on the campus or from across town. However, there is virtually no area of research in which it is possible to limit one's sources to electronic ones. It will continue to be necessary for library patrons and staff to use print as well as electronic products for years to come.

As much as possible electronic products and services should be charged to departmental or branch allocations, rather than coming out of the general fund (i.e., "off-the-top"). Savvy selectors in libraries that do not charge electronic publications to departmental or branch allocations appear to have a strong preference for electronic products and services over print products that would be charged to them. The may, therefore, not be sensitive to the cost per use.

2. Acquire and install use measurement tools, and determine cost per use.

Every library, even the few that spend less than ten percent of the acquisitions budgets on electronic products and services should measure the use of the electronic products and services that they purchase, and determine the cost per use as described in the foregoing pages. Only in that way can they determine that their money is well spent

Some librarians and library users are uncomfortable with the idea of monitoring use. It is, therefore, important to purchase software that captures needed information, but protects the privacy of individual users.

3. Critically examine all electronic products and services that cost an average of more than $.75 per use.

There is a general rule-of-thumb with regard to the cost of electronic products and services: those costing over $.60 per use are expensive. If electronic products and services are well chosen and well used, the cost should be less than $.60 per use. OCLC, a major information aggregator, has analyzed FirstSearch usage of its scores of databases on several occasions and has determined that most libraries experience average costs of $.55 to $.58 per use--a log-on and search against a specific database. [The lower figure generally is realized when subscription-based pricing is used; the higher when blocks of individual searches are purchased]. Concurrent searches against multiple electronic services cost somewhat more.

The author has attempted to use this range of figures developed by OCLC as a baseline for evaluating the cost effectiveness of other electronic products and services. He has found that the average is quite reliable, but the costs for individual electronic products and services can differ greatly; often from $.01 to $5.00 per use.

The author has chosen $.75 as the figure at which a product or service should be critically reviewed because that reduces the number of products and services requiring attention to a manageable number. Typically, half or more of all products and services cost more than $.50 per use, but only 20-25 percent cost more than $.75 per use. A library can always undertake a second round to look at products that cost between $.50 and .75 per use.

Most libraries will find that electronic products and services in science, technology, and medicine (STM) are likely to be the most costly, followed by titles in business. Titles in liberal arts and the performing arts tend to be in the middle-range, and large packages of journals in the low range. For that reason, some libraries have chosen different figures for different subject areas: $1.00 for STM, $.75 for the liberal and performing arts, and $.50 for packages of journals.

A level of usage which results in a cost of use greater than $.75, or whatever other figure a library chooses, does not necessarily mean that a product or service is a poor choice. Users may be unaware of a product or service, or they may have an incorrect understanding of its coverage. A library that undertook a survey of its patrons determined that over 65 percent were not aware of half of the general electronic products and services in which the library had invested, and an even larger percentage could not identify a single specialized database in their own areas of interest.

4. Seek to reduce cost by undertaking competitive procurements.

There is a compelling argument for seeking to bundle a broad range of electronic products and/or services into a single master contract. Almost all of information aggregators offer master contracts that cover from ten to sixty electronic services. Among the options are not only periodical indexes, but full-text/image files of periodicals. While a number of payment options are available, the most attractive is pricing for a specified number of ports or concurrent users. That permits the use to spread unevenly across several electronic services without the subscribing institution worrying whether one is cost effective and another is not. Even a few dozen uses a year of an electronic service can be justified.

The market is highly competitive. OCLC, EBSCO, and Gale, the three most popular information aggregators, regularly engage in discounting when asked to respond to an RFP, and enter into price negotiation when invited to do so. This is especially the case when a library is a potential major account, one budgeted to spend more than $50,000 per year. Libraries that have less to spend, should consider forming a consortium with other libraries to increase the attractiveness of the account to the information aggregator.

If the procurement is for full-text or images of periodicals, the RFP should include a copy of the library's(ies') continuation list(s). Vendors should be asked to annotate the list(s) so that the library(ies) will be able to quickly determine which of its/their current print subscriptions will be included.

5. Promote use by including electronic products and services in the patron access catalog.

A library should catalog electronic products and services. This effort should not be limited to electronic products and services for which it pays a subscription fee, but also any others that are potentially useful to its patrons. Cataloging these sources and providing the URLs (addresses) will alert those who consult the online patron access catalog to their availability. There is ample precedent for this in libraries, including the filing of cards for the holdings of the Center for Research Libraries by scores of members beginning in the 1970's. A few months after a library cataloged some of the major collections in the Library of Congress' National Digital Library Project, it experienced a dramatic surge in Internet usage, with the papers of Presidents Washington and Jefferson extensively used.

6. Add icons for selected resources.

The icons on the Web-based user interface of a library’s Web server or portal should include not only all popular products and services, but also all electronic services to which the library subscribes. However, it is important to limit the number of icons on a single screen to 20 or fewer. When there is more than that, a library should prompt the user to narrow the scope of the search by broad subject area or other category. For searches of the catalogs of other libraries, it should give that general option, with the names of the other libraries on the next screen.

The major problem is where to place interdisciplinary products and services. The simplest approach is to include them under each category for which they might be useful.

7. Utilize a library portal as the primary access tool to all electronic products and services.

A library portal provides single-search access and patron authentication to a wide variety of electronic resources, both within and outside the library. It is possible to search a library’s patron access catalog, the catalogs of other libraries, online reference services to which the library subscribes, and selected Web sites and bring back a unified search result. Most portals also offer linking to related resources. Every major automated library system vendor now offers portal technology as an optional module. Portal technology can also be purchased directly from companies such as MuseGlobal and WebFeat.