Volume24, Number 4, 2002

Outcomes-Based Evaluation - Not as Easy as It Looks

by: Elizabeth A. Breedlove, LSTA Program Coordinator, New Jersey State Library

A two year college library's application, submitted for the New Jersey State Library's Library Services for Persons with Disabilities grant program, was chosen as a pilot project for implementing an outcome-based evaluation (OBE) plan because it provided a situation where positive changes in students' skills, knowledge, behaviors and attitudes were likely to occur as a result of receiving library training and using adaptive technology. An adaptive technology workstation and assistive equipment was provided for community college students with visual impairments and learning disabilities to increase their access to the Internet and library resources. The State Library's LSTA Program Coordinator, who had received training from the Institute of Museums and Library Services (IMLS), developed an OBE model evaluation plan in cooperation with the College Library Director and the Coordinator of Special Services for Students with Disabilities. In addition to the plan, survey instruments were developed to collect students' self-assessments of their knowledge and skills as a result of attending orientation and training sessions and using adaptive technology to complete class assignments.

As a result of the project, students with visual impairments and learning disabilities were expected to achieve the following outcomes:

  • Increase their knowledge about the benefits of using the library's adaptive technology workstation and/or assistive equipment.
  • Be able to independently make effective use of the library's adaptive technology workstation and/or assistive equipment.
  • Increase their ability to successfully complete class assignments.
The project failed to meet these outcomes. Only two students attended an orientation session and no students signed up for training sessions. The staff could not identify any regular users of the adaptive technology workstation or assistive equipment, with the exception of one legally blind man who was using the screen magnifier to assist in downloading pornography from Internet sites.

In developing this adaptive technology grant program, the State Library made the following assumptions that were not validated by the experiences that the majority of the fourteen public and academic libraries had in implementing their projects:

  • Adaptive technology was user friendly and librarians receiving basic training could provide customers with enough skill to successfully use the sophisticated software.
  • Librarians with no previous experience of serving people with vision or learning disabilities could implement successful projects without a disproportionate amount of intensive staff involvement.
  • People with visual impairments or learning disabilities who do not own PCs would come to the library and be able to use the adaptive workstations after receiving brief training from library staff.
In the case of the two-year college trying to implement the OBE plan, several factors contributed to the project's lack of success. First, there was a mistaken assumption that there was a need for the College's students with disabilities to do research; when 90% of the self-identified students with vision and learning disabilities were involved in taking remedial, pre-college courses or introductory courses which do not require research or use of library materials. Therefore there was very little motivation for these students to learn how to use the adaptive technology. It was also unrealistic to expect that the project manager, who had no experience with adaptive technology and had not previously trained students with disabilities, would be able to successfully implement the project and collect the OBE data. There was another false assumption that self-identified students with disabilities would be more willing to provide self-assessment information about how they benefited from the library's adaptive technology, since they were already receiving services from the Special Services section. A comment made by one student who refused to fill out a survey brought up the possibility that the survey questions could be perceived as intrusive or an invasion of privacy. The College Library would have been in a better position to get responses to the survey questions if they had been successful in establishing an Advisory Council which included students with disabilities; or in attracting an enthusiastic group of students who used the adaptive technology. Under these circumstances, students involved in these groups may have been more willing to volunteer information for the outcome measurements.

Unlike United Way Agencies, who successfully collect and use outcome-based evaluation data, the College Library does not have the same relationship with their students where they can demand information in exchange for services and track their students' positive changes in skills, knowledge, behaviors and attitudes over a long period of time. The one-year LSTA grant was too little time for the College Library staff to learn to use the adaptive technology, learn to train students, learn about OBE and collect the required data.

Both the State Library and the College Library derived some positive outcomes from this pilot project. The State Library learned a great deal by actually testing its OBE plan in a working environment. Another positive result was the College Library Director's decision to develop an assignment involving the use of assistive equipment and library resources for students with disabilities who take her remedial study skills course. This close interaction with her students might provide an opportunity to gain further insights on developing outcomes and on developing realistic targets to measure the success of those outcomes. Two lessons learned:

  • It is important to have an accurate and thorough understanding of the needs of the targeted population for the proposed services.
  • Implementing survey instruments that have not first been pre-tested to determine their effectiveness causes difficulties.