Using Project Outcome with Summer Library Program to Track Impact Across Time and Strengthen Championship
Appleton Public Library (APL) first experienced the value of measuring outcomes through their ongoing use of the Impact Survey. Leadership viewed Project Outcome as an opportunity to extend outcome measurement to their Summer Library Program and support program improvement and communications with the library’s Board.
- Library Name: Appleton Public Library
- Library Location: Appleton, Wisconsin
- Library Size: Medium-large (1 building, service area: 116,000)
- Immediate Surveys Used: Summer Reading (Teen/Child, Caregiver, and Adult versions)
Data-Driven Changes Strengthen Summer Library Program, and Evidence of Patron Outcomes Supports Library Championship
APL offers a range of activities for children, teens, and adults as part of their Summer Library Program. In an online component of the program, patrons can earn rewards by completing missions, such as visiting a local nature preserve or the police department. These missions help patrons build connections with community resources and other community members through informal interactions. Project Outcome surveys conducted in 2016 showed that patrons benefited from the program, but found some of the missions confusing and hard to follow. APL has since improved the descriptions and layout of the missions, and they look forward to reviewing 2017 survey results to measure if these changes resulted in more patrons taking part in missions.
APL reports on the impacts of library programs and services to its Board based on the results of Project Outcome surveys and Impact Surveys. Library leaders have found that outcomes resonate with Board members, strengthening how they voice their support for the library. One leader shared, “I think [reporting on outcomes] really transformed conversations with our Board, so that they're much more interested… It lets us tell a more complete story about the library, and what we're offering… so they can understand it more fully… When you talk strictly in numbers, or outputs, it's not something that people can hold on to as clearly. And so when you start talking about outcomes and the impact that a library is having, that's where the true heart of what we're doing is. And our Board feels that, as well… It allows them to… have something more concrete to hold on to, and to talk to people about what the library does in a more complete way than ‘X number of people walk in the door every year’ or ‘X number of books walk out.’” A Board member shared, “The [outcome] data provides an objective story, backing up much more engaging stories from staff about serving the community with objective numbers… [and] includes things the Board may not think to ask for. This adds dimensions to how the Board considers the library's success in serving the community.”
Survey Administration Adjustments Improve Response Rates
Getting patrons to fill out surveys has been a challenge for the library. When APL first started using Project Outcome surveys in 2015 for their Summer Library Program, they made the surveys available only on paper; in 2016, they made them available online. Each set of surveys produced valuable information, but fewer than 20 surveys were completed each year, and APL wanted more responses to better understand how to improve their programs and be more confident in their assessment of impact.
For 2017, APL is using some new strategies to increase online survey completion. For example, when children come in to get their prizes for completing the Summer Library Program, the adult accompanying them receives a slip of paper encouraging them to take the survey. The library has also dedicated a computer near the front of the library to survey-taking and staffed it with a librarian who encourages patrons to complete the survey; and “boosted” a Facebook post about completing the survey that appears on the News Feeds of everyone who “likes” the library’s Facebook page. The preliminary results of this effort have been positive, with over 50 surveys completed so far.
Factors That Supported Successful Use of Project Outcome
APL was an early adopter of the Impact Survey. Their experience with administering the survey laid the groundwork for Project Outcome’s traction in the library, as staff were already familiar with the value of using outcome data to explain their impact. As a library leader explained, “We'd already done the Impact Survey, and so [the staff] understood the importance of moving to outcomes rather than outputs… With our [traditional output] numbers sliding, it tells a much better and much more complete story for us than our disappointing numbers were for several years… I think it would've been different if our numbers were going up. [The Project Outcome surveys] could have been seen as criticism. And instead, with our numbers going down, it was seen as a way for them to tell the story of what the library was actually doing.”
Several APL staff members took part in trainings offered by PLA, which helped build familiarity with Project Outcome and led to their early use of the tools in 2015. APL staff appreciated that Project Outcome offered ready-to-use surveys, which the library did not have the expertise to create in-house.
APL plans to review this year’s Summer Library Program survey results to assess the effectiveness of their efforts to reach and meet the needs of more economically diverse residents. These efforts were part of the library’s participation in POINT (Poverty Outcome and Improvement Network Team), a local collaborative initiative. New Project Outcome survey functionality allowing them to add custom survey questions specific to this effort was a timely addition. A leader shared, “We were able to add questions about the pieces [specific to] the POINT initiative… I'm really glad to have that flexibility. So when we're doing something different, or something where we're trying to target a population, we can start to look at those factors.”
Project Outcome is growing in importance in the library’s work. APL will continue to use the data to help improve their Summer Library Program, strengthen their community impact, and provide their board with a comprehensive picture of their value in the community. APL will begin reporting data from Project Outcome surveys in their budget requests to the City of Appleton in 2018, when they will have three years’ worth of data, as required by the City.
Prepared by ORS Impact, September 2017