Outcomes-Based Evaluation - Not as Easy as It Looksby: Elizabeth A. Breedlove, LSTA Program Coordinator, New Jersey State Library
A two year college library's application, submitted for the New Jersey State Library's Library Services for Persons with Disabilities grant program, was chosen as a pilot project for implementing an outcome-based evaluation (OBE) plan because it provided a situation where positive changes in students' skills, knowledge, behaviors and attitudes were likely to occur as a result of receiving library training and using adaptive technology. An adaptive technology workstation and assistive equipment was provided for community college students with visual impairments and learning disabilities to increase their access to the Internet and library resources. The State Library's LSTA Program Coordinator, who had received training from the Institute of Museums and Library Services (IMLS), developed an OBE model evaluation plan in cooperation with the College Library Director and the Coordinator of Special Services for Students with Disabilities. In addition to the plan, survey instruments were developed to collect students' self-assessments of their knowledge and skills as a result of attending orientation and training sessions and using adaptive technology to complete class assignments.
As a result of the project, students with visual impairments and learning disabilities were expected to achieve the following outcomes:
- Increase their knowledge about the benefits of using the library's adaptive technology workstation and/or assistive equipment.
- Be able to independently make effective use of the library's adaptive technology workstation and/or assistive equipment.
- Increase their ability to successfully complete class assignments.
In developing this adaptive technology grant program, the State Library made the following assumptions that were not validated by the experiences that the majority of the fourteen public and academic libraries had in implementing their projects:
- Adaptive technology was user friendly and librarians receiving basic training could provide customers with enough skill to successfully use the sophisticated software.
- Librarians with no previous experience of serving people with vision or learning disabilities could implement successful projects without a disproportionate amount of intensive staff involvement.
- People with visual impairments or learning disabilities who do not own PCs would come to the library and be able to use the adaptive workstations after receiving brief training from library staff.
Unlike United Way Agencies, who successfully collect and use outcome-based evaluation data, the College Library does not have the same relationship with their students where they can demand information in exchange for services and track their students' positive changes in skills, knowledge, behaviors and attitudes over a long period of time. The one-year LSTA grant was too little time for the College Library staff to learn to use the adaptive technology, learn to train students, learn about OBE and collect the required data.
Both the State Library and the College Library derived some positive outcomes from this pilot project. The State Library learned a great deal by actually testing its OBE plan in a working environment. Another positive result was the College Library Director's decision to develop an assignment involving the use of assistive equipment and library resources for students with disabilities who take her remedial study skills course. This close interaction with her students might provide an opportunity to gain further insights on developing outcomes and on developing realistic targets to measure the success of those outcomes. Two lessons learned:
- It is important to have an accurate and thorough understanding of the needs of the targeted population for the proposed services.
- Implementing survey instruments that have not first been pre-tested to determine their effectiveness causes difficulties.