Measures, Statistics, and Standards for Assessing Digital Reference Performance

3. 1 Reference Transactions – Volume, Cost, Benefits, and Quality

[see operational definitions, guidelines, and data collection procedures in: Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. McClure, Charles, R. et. al. Information Institute of Syracuse, School of Information Studies, 2002.]

Descriptive Statistics and Measures

  • Number of digital reference questions received
  • Number of digital reference responses
  • Number of digital reference answers
  • Number of questions received digitally but not answered or responded to by completely digital means
  • Total reference activity – questions received
  • Percentage of digital reference questions to total reference questions
  • Digital reference correct answer fill rate
  • Digital reference completion time
  • Number of unanswered digital reference questions
  • Type of digital reference questions received
  • Total number of referrals
  • Saturation rate
  • Sources used per question
  • Repeat users

Log Analysis

  • Number of digital reference sessions
  • Usage of digital reference service by day of the week
  • Usage of digital reference service by time of the day
  • User’s browser
  • User’s platform

User Satisfaction Measures

  • Awareness of service
  • Accessibility of service
  • Expectations for service
  • Other sources user tried
  • Reasons for use
  • Reasons for non use
  • Improvements needed/Additional services that need to be offered
  • Satisfaction with staff service
  • Delivery mode satisfaction
  • Impact of service on users
  • User demographic data

Cost

  • Cost of digital reference service
  • Cost of digital reference service as a percent of total reference budget
  • Cost of digital reference service as a percent of total library or organizational budget

Staff Time Expended

  • Percent of staff time spent overseeing technology
  • Percent of staff time spent assisting users with technology

Other Assessment Options

  • Peer Review
  • Enhanced reference transaction logs
  • Librarian discussion groups

Quality Standards (examples)

  • Courtesy
  • Accuracy
  • Satisfaction
  • Repeat Users
  • Awareness
  • Cost
  • Completion Time
  • Accessibility

This document is part of the Measuring and Assessing Reference Services and Resources: A Guide prepared by the RUSA RSS Evaluation of Reference and User Services Committee.