ALA Home  Moving Toward Outcomes Evaluation

Volume24, Number 4, 2002

Moving Toward Outcomes Evaluation

The emergence of a service economy has been accompanied by a greater focus on evaluating the quality of services. Libraries and librarians can benefit from the evaluation techniques that are being developed and use the results to continually improve their collections and services. In order to benefit most from evaluation, it is important to have a positive attitude towards the process. Many of us have negative associations with various forms of evaluation, probably going back many years to our school days. We need to replace any negativity with a positive view of evaluation as a way to improve and not see it as a punishment for things that we have done poorly. As managers we also need to encourage our staff and our entire organization to think of evaluation as a pathway to improvement so that there will be maximum cooperation at the planning and implementation levels. Most of all, we need to "walk the talk" when it comes to being positive about quality improvement.

Over the years, my involvement with evaluation studies has suggested to me that multiple forms of evaluation are still valuable. We will continue to need standards to guide us, as well as input and output measures. Input measures quantify the resources that we have available to provide library service, such as number of staff, size of collection, space and so on. Output measures quantify what we actually produce with the inputs, such as number of reference questions answered, number of loans made and number of hits on the library web site. A recognition of the importance of library users as "customers" has led to the development of customer satisfaction measures for libraries that add yet another dimension to evaluation. Most recently, outcome or impact measures have also started to be explored.

Outcome or impact measures answer the question, "What difference do we make?" When users access the library, they are often doing so because they want some information on a specific topic. In academic and work situations, the right information at the right time can lead to changes in decision-making behavior. In personal information-seeking situations, information can lead to changed behavior as well, but may also be used for recreational reading purposes. In both cases, information seeking is a mechanism that we use for finding and using information resources that will enhance our lives and contribute to life-long learning.

Earlier approaches to evaluation have tended to avoid outcome evaluation, often because we have considered user satisfaction to be as far as we need to go. Increasingly, funders are asking the tougher question of what difference it makes to have a library and the answer is not a simple one. Outcome evaluation can help because it does go farther than traditional approaches. The dilemma is how to implement it. My own experience is that outcomes evaluation can be incorporated into user satisfaction surveys by asking users to indicate ways in which information from the library affected key decision-making areas. In order to ask the right questions, you need to understand the kind of key decisions that are made in an organization. You can do this using focus groups and through observation of the everyday work world of your users. My recent article (Marshall JG. "Determining our worth, communicating our value." Library Journal 125, no.19: 28-30, Nov 15, 2000) describes outcome areas that were explored in medical, corporate and government areas.

A less sophisticated approach is to keep a record of situations in which individual users have told you about how they used information from the library in an important situation -- ask them to send you a brief e-mail or note about it and keep these in a file for strategic placement in your annual report. A more sophisticated approach is to track the impact of the changed behavior through the system. In the medical environment this would mean seeing if patient care was improved, length of stay in hospital shortened, etc. These latter approaches are the most difficult to do. Whatever approach you choose, it is time to think about incorporating some outcome measures into your evaluation thinking and practice. Your library and your users will benefit from this bigger picture thinking and from the focus it brings to the services that matter most.