by Lisa M. Metzer
I recently read Stones into Schools by Greg Mortenson, the sequel to the #1 New York Times Bestseller Three Cups of Tea. This is the story of a man who uses education and literacy to promote peace in remote areas of Pakistan and Afghanistan. Much like librarians, he wants to make a difference by empowering people and communities.
This powerful passage impressed me, and in my opinion it expresses the cardinal pursuit of librarians: "If there is a metric by which I measure the achievements of the Central Asia Institute, it is not the amount of donations we receive each year, or the number of people who have read Three Cups of Tea, or even the number of schools we have built. In fact, it really has nothing to do with math and everything to do with the girls whose lives have been changed through education [emphasis mine]."
Librarians want to change lives through literacy. The number of patrons we serve remains an important metric and validates our work, but more compelling is the evidence of patrons whose lives have been changed as a result of our work.
Outcomes based evaluation (OBE) is a powerful tool in demonstrating the difference we make. I write this article to introduce this evaluation tool and to briefly share my experiences with OBE in my current professional position at the Wells Branch Community Library.
What is outcomes based evaluation? The United Way defines it as "the regular, systematic tracking of the extent to which program participants experience the benefits or changes intended," and the Institute of Museum and Library Services (IMLS) describes it as "a systematic way of assessing the extent to which a program has achieved its intended result." The United Way developed the model in 1995 to increase the effectiveness of their programs, and since then it has reached the library world.
To help libraries communicate the value of their programs, the New York State Library and the IMLS partner to deliver OBE training to member libraries. You can learn more about their training at the New York State Library's Outcome-Based Evaluation website (http://www.nysl.nysed.gov/libdev/obe/index.html). I also highly recommend the United Way's Outcome Measurement Resource Network (http://www.liveunited.org/outcomes/) for additional information.
Although OBE does not replace the necessary tracking of library inputs (i.e. the number of patrons attending our programs), it remains integral to effective evaluation. As Albert Einstein aptly said, "Not everything that can be counted counts and not everything that counts can be counted."
OBE enhances planning and evaluation in three ways. First, it identifies a need. Second, it suggests an appropriate solution (i.e. a program or service). Third, it considers desired outcomes. Outcomes are the direct benefits for program participants, and they demonstrate how lives have changed as a result of attending a program. Examples may include new knowledge, increased skills, changed attitudes, improved conditions, or altered status.
I use OBE principles to evaluate adult programs at the Wells Branch Community Library. The children's librarian and I developed standardized worksheets to increase the effectiveness of our programming. Planning worksheets based on OBE concepts identify needs, solutions, and desired outcomes. Evaluation worksheets illustrate the extent to which programs achieve the desired outcomes.
The following examples illustrate how I recently used the worksheets to develop and evaluate computer workshops at the library. Notice the emphasis on needs, solutions, and desired outcomes. This information represents an excerpt of the worksheet's full content.
Program Planning Worksheet for 2010 Adult Computer Workshops (based on OBE concepts):
Need/s - Patrons need help using the library's free public computers to perform simple, functional tasks:
- Create documents
- Find and apply for jobs
- File for unemployment
- Manage a small business
- Find information
Solution/s - Provide free computer workshops targeting specific computer skills:
- Computer Basics
- Email Basics
- Searching the Internet
- Finding Jobs Online
- Word Basics
- Excel Basics
- PowerPoint Basics
Desired Outcome/s -
- Participants will practice new computer skills with an instructor's guidance.
- Participants will demonstrate increased confidence using computers.
- Participants will sign up for future computer workshops.
- Participants will use new computer skills to perform necessary tasks.
How did we identify the needs?
- Patrons regularly request help using the computers.
- We observe how patrons use our public access computers.
How do the workshops relate to the library's mission?
- They will strengthen library services.
- They will utilize the library's physical space for programs.
During and after the computer workshops, I collected data to ascertain the impact of the programs. To what extent did they achieve the desired outcomes? I asked participants to share something helpful they learned, and I also encouraged them to complete a short survey. The surveys asked the following questions:
- 1. What was the most helpful information you learned today?
- 2. What did you like best about this workshop?
- 3. What is one thing you did not learn today that you wanted to?
- 4. How did you find out about this program?
- 5. Suggestions for future computer workshops?
Upon collecting this data, I analyzed the information to determine if and how the workshops benefited the participants. Furthermore, I used it to improve future computer classes.
Many patrons shared stories. One participant attended a workshop to learn how to find jobs online. The first time she used a computer at a public library, she cried because she didn't know where to begin. Her confidence increased after attending the workshop, and she left with a smile on her face. She even signed up for another workshop to learn more! A second patron received a computer from his grandchildren but didn't know how to use it. He attended a class to learn how to use email, and now he communicates with his grandchildren every week. I value these stories because they reveal our patrons' needs and how we can serve them.
Imagine the future, potential triumphs of patrons attending our computer workshops. Mr. Mortenson writes in Stones to Schools, "In the end, the thing I care most about - the flame that burns at the center of my work, the heat around which I cup my hands - are their stories."
Librarians can use OBE to share their success stories!