Home Asking better questions

Asking better questions: Small-scale assessment measures that inform ongoing work

C&RL News, June 2005
Vol. 66, No. 6

by Harvey Varnet and Martha Rice Sanders

The late Dick Palmer, professor of library and information science at Simmons College, loved words. He had served in the navy as an intelligence officer, code-breaking Japanese during World War II. Palmer loved to challenge his classes to define the word knowledge. His favorite definition was “that information which allows us to ask better questions.” Palmer was a pioneer in information science and, though he didn’t know it at the time, he was in front of the assessment movement in higher education. He always asked “What for?” If you didn’t have an answer, he quickly followed that remark with “Why?” The approach was gentle but firm and purposeful.

In small- and medium-sized colleges and universities, undertaking library assessment is definitely a challenge. First, there’s figuring out who has the time or on-staff expertise to do this in a meaningful way; getting staff interested in this topic is not always easy. Then there’s the complexity of linking anything you do in the library to the overall campus assessment program (if indeed one is fully formulated).

In the northeast, NEASC (the New England Association of Schools and Colleges) has been the prime driver in the assessment movement. On the NEASC Web site,1 the Commission on Institutions of Higher Education outlines standards and other policies, including an important “Policy on Institutional Effectiveness.” NEASC has attempted to assist accreditation site visitors by formulating what it calls “pilot site visit questions.”2 While the questions relating to Standard 7, Library and Information Resources, are helpful, they are limited in scope. One area these questions do ask site visitors to consider is the development of information literacy skills—meaning, for most of us, bibliographic instruction or some library and/or institutional attempt to foster such skills. This is an area where many of our colleagues do an excellent job of measuring their effectiveness. Otherwise, what to do?

The culture of evidence
One less intimidating way to look at assessment is to see our efforts as part of the greater culture of evidence. Peggy Maki, a noted authority on assessment and the director of assessment at the American Association for Higher Education, talks about support units and how outcomes assessment, while helpful, may not always fit our operations. However, libraries perform lots of services, with accompanying data/statistics, that can inform our decision making. In Maki’s words, aggregating the data, analyzing it, and then doing something with it—asking better questions—is an important aspect in helping to create the culture of evidence that what we are doing is meaningful and worthwhile.

Small-scale measures
At Providence College, we’ve tried to adopt techniques that tell us something about ourselves. On an annual basis, we look at interlibrary loan (ILL) fill-rate. These data are easy to gather, as they are primarily OCLC-based. The data tell us that we are very efficient at getting requests from students and faculty out to other colleges—in less than one day, typically. The chart below also shows us how, over time, the fill rates for copies (mostly journal articles) and monographs are remarkably consistent. This allows our service staff to tell students and faculty, with some confidence, what they can expect if and when we don’t have an item and it is not available nearby in a sister consortium library.

_{Interlibrary Loan Fill Rates-Borrowing}

When considering budgeting challenges, we occasionally look at book use rates. Our process is to review whether the books we acquired two fiscal years ago circulated or not. This tells us in some measure that we are buying the books that are needed to support the curriculum. With a campus that is not research driven but research interested, we surely acquire books for their long-term value. However, our mission is to provide an undergraduate, liberal arts education, so our purchasing decisions are made with this in mind. One might ask, what is “success” when looking at use rates. One interesting way to look at this is via Richard Trueswell’s3 seminal work years ago that outlined the 80/20 rule—over time, 80 percent of circulation tends to come from 20 percent of a collection. Our fiscal year 2002 acquisitions circulated as indicated below—not bad, considering the compressed time period of analysis and the oft-reported general downward trend in book circulation.

Another small-scale approach we’ve taken is to ask faculty for the bibliographies of papers or presentations that their seminar students take or from major departmental courses. We’ve had limited success at getting these bibliographies, but each set we have looked at has proven useful. Our goal in the review of the bibliographies is to determine if, within the library or our consortium, we can provide 80 percent of the materials the students needed. This approach also deals in reality—we look at actual resources used, versus the students’ speculation that “they don’t have anything I need to do my course work.”

Our analysis of bibliographies in social work benefited both areas. The perception that we had very few journal subscriptions was incorrect (we were able to prove this) and this helped change the tone of our discussions with the department. We reviewed the senior seminar papers for two years and discovered that we were able to supply 62.5 percent of the citations in 2001 and 87 percent in 2002. When the department came up for its professional accreditation, we were an active and welcomed partner in the process, and the library was cited as a program strength.

In music, we were able to review the bibliographies for two courses. In each course, we held more than 90 percent of the books used and 100 percent of the journals cited. This data helped dispel the perception among the music department faculty that the library held none of the titles needed by students for doing their research. In addition, the library purchased one book for the collection that was accessed by a student through ILL.

In chemistry, analysis of student seminar citations revealed that we held or provided access to 77 percent of the journals cited. For books, 46 percent were available in the consortium, though we held none used by students. This lead to an interesting dialogue with the chair of the department. After reviewing what we did not hold, the chair indicated that we ought not to buy the works cited because, though useful for students’ papers, the books were not a particularly good fit for our collection.

Larger-scale measures
Another aspect of operational analysis is benefiting from the overall assessment work that the college undertakes. Similar to many institutions, Providence College administers the Student Satisfaction Inventory (SSI)—in our case, to freshmen and seniors.

The Noel-Levitz4 SSI has two library-related questions: Library staff are helpful and approachable? and Library resources and services are adequate? While these questions may not be ideally stated (for many of us, this is actually four and not two questions), the results give us a look at how our primary clientele generally perceive us. The SSI uses the gap analysis technique for assessment, with the “gap” being the difference between students’ ratings of importance and their ratings of satisfaction, each on a 1 to 7 point scale (7 is best). We have data from three such SSI surveys (1997, 2000, 2003). In 2003, for the staff question, students rated us well, with a gap of 0.26. For the resources and services question, students weren’t quite as satisfied—our gap is 1.02. In both cases our scores are slightly ahead of our national peers. And, in both cases, the gap has decreased from the previous assessment, a trend in the right direction. Our scores for the staff question also rank us as one of the highest-rated service/support areas on campus.

Because the college has an ongoing assessment program, we received institutional support in 2003 to participate in the LibQual+ survey.5 This nationally normed survey was developed at Texas A & M University in cooperation with the Association of Research Libraries. This survey measures four areas: affect of service, access to information, personal control, and library as place.

Our library was built in 1969, and we knew it was badly in need of updating and renovating. What we learned from LibQual+ was that students felt strongly about the library as place, in our case in the negative. The radar chart produced in our results summary is very red—not a good color to see, but a clear message to the library and college administration.

We began a long-term renovation process in the summer of 2003, completing the library’s main floor in the spring of 2004. Our plan is to participate in Libqual+ next year, again surveying sophomores and juniors to determine if their perceptions have changed.

As the library improved physically, it seemed to us that students were coming in to use the library in larger numbers. We keep evening attendance headcounts, so we decided to compare the current (renovation) year to the previous year’s counts. The chart below shows how attendance was up for each week analyzed. The counts for each year were taken during the week before Thanksgiving, in the first two weeks of December, and for all of January.

_{Library Attendance}

We reported this information to the college administration, and they in turn reported it to the Board of Trustees at its late January meeting. There was general recognition that the renovations were well worth the expense involved, and the Board indicated its support for the next phase of renovations.

The future
We plan to continue with a mix of assessment activities. We are now planning to look much more closely at our instruction program. We teach between 50 and 70 classes per year, and we need to know more from the students as to whether the information they receive has utility and value.

We are also interested in some general assessment of the high-profile activity of reference desk service. We’re working with our campus assessment coordinator on a point-of-use survey for reference, hoping that we’ll get useful feedback from users. While we get anecdotal information about reference assistance, this information tends to be at the extremes—either someone’s very pleased or quite the opposite. We’re hopeful that more user-response data will give us a balanced picture of this primary and important service.

We will continue to analyze and review interlibrary loan data and book use rates. We also plan to review course-related bibliographies from those faculty willing to share. This work has proven to be very beneficial and has qualitatively changed the discussions we have regarding library services with our teaching colleagues. In our next review of bibliographies, we’ll attempt to see if there is any correlation between the grade students earn on their papers and their use of refereed, professional sources.

The college will continue to do the SSI survey on a regular basis, and we’ll extract that data to see how students are perceiving us in general. And, finally, we are hopeful for institutional support this coming year to do another LibQual+ survey. The cost for this activity is quite different from all our other assessment efforts, but the return on investment is well worth the effort.

In its “Standards Revision Discussion Paper,” NEASC notes: “Evidence is fundamental to institutional decision-making . . . Increasingly, as will be reflected in the revised Standards, the Commission expects that institutions gather and use evidence as the basis for this evaluation and improvement.”6 At Providence College, we in the library are attempting to take this evidence-based approach seriously. We’re confident our “mixed” approach to analysis is on track. The blend of analytical activities we’ve undertaken has helped to make us a better campus service. This mixed approach is also on a scale such that the evaluation work doesn’t overwhelm any other ongoing activity.

To conclude with more wisdom from Dick Palmer, “Just count—it’s amazing what the numbers tell you about your prejudices.”

Providence College profile
Providence College (www.providence.edu) is a primarily undergraduate, liberal arts, Catholic institution that is committed to fostering academic excellence through the sciences and humanities. The college enrolls approximately 4,200 FTE undergraduate students and 330 FTE graduate students. There are 250 faculty offering degree programs in more than 40 areas.

Notes
1. New England Association of Schools and Colleges Web site: www.neasc.org/cihe/assessment.htm.

2. NEASC Pilot Questions Web site: www.neasc.org/cihe/site_visit_questions.PDF.

3. Those unfamiliar with Dr. Trueswell’s work should review one or more of the following citations: Richard Trueswell, “Some behavioral patterns of library users: The 80/20 rule,” Wilson Library Bulletin, 1969; 43: 458–461. Richard Trueswell, “User circulation satisfaction vs. size of holdings at three academic libraries,” College and Research Libraries (May 1969): 30(3): 204–13. Richard Trueswell, “Determining the optimal number of volumes for a library’s core collection,” Libri (International Library Review) 1976; 19(1): 49–60.

4. The corporate Web site for Noel-Levitz, Inc. can be found at www.noellevitz.com/solutions/retention/satisfaction/index.asp. To review gap analysis use, see “The Measurement of Gaps in Service Quality” by Robert Lewis and David Klein (pages 33 – 38) in The Service Challenge: Integrating for Competitive Advantage (Chicago, American Marketing Association, 1987).

5. The underpinnings of LibQual+ are based on the pioneering “servqual” work of A. Parasuman and his colleagues. See Valarie A. Zeithaml, A. Parasuraman, and Leonard L. Berry, Delivering quality service: Balancing customer perceptions and expectations (New York, Free Press, 1990). For Parasuman et al., the only opinion that matters is the customers —all others are irrelevant. The Libqual+ Web site can be found at www.libqual.org

6. NEASC’s “Standards Revision Discussion Paper” (issued in December 2003) can be found at www.neasc.org/cihe/revisions/summary_of_significant_changes.pdf. The quote from the paper is on page three, second bullet.

Harvey Varnet is library director, e-mail: hvarnet@providence.edu, and Martha Rice Sanders is coordinator of collections services and catalog librarian, e-mail: sandersm@providence.edu, at Providence College in Rhode Island

© 2005 Harvey Varnet and Martha Rice Sanders