Evaluation of the New Jersey Digital Highway

Judy Jeng (jjeng@njcu.edu) is Head of Collection Services,New Jersey City University, New Jersey.

The aim of this research is to study the usefulness of the New Jersey Digital Highway (NJDH, www.njdigitalhigh way.org) and its portal structure. The NJDH intends to provide an immersive and user-centered portal for New Jersey history and culture. The research recruited 145 participants and used a Web-based questionnaire that contained three sections: for everyone, for educators, and for curators. The feedback on the usefulness of the NJDH was positive and the portal structure was favorable. The research uncovered several reasons why some collections did not want to or could not participate. The findings also suggested priorities for further development. This study is one of the few on the evaluation of cultural heritage digital library.

The New Jersey Digital Highway (NJDH, www.njdigitalhighway.org) is a digital library for New Jersey history and culture, including collections of New Jersey libraries, museums, archives, and historical societies. The NJDH, funded in part by the 2003 National Leadership Grant of the Institute for Museum and Library Services, is a joint project by New Jersey State Library, the New Jersey Division of Archives and Records Management at Rutgers University Libraries, the New Jersey Historical Society, and the American Labor Museum. As part of the project, the NJDH identifies 686 cultural heritage institutions (public libraries, archives, historical societies, and museums). As of November 2007, there are more than ten thousand objects (pictures, records, and oral histories) in the repository. More are being added daily. The NJDH, at this writing, is still very much a work in process. The principal investigator of this project continues to extend opportunities to more communities to link their sites and scan their images.1

The NJDH provides portals for four different groups of people: everyone, educators, students, and librarians and curators. Its mission is to develop an immersive, user-centered information portal and to support the New Jersey learner through a collaboration among cultural heritage institutions that supports preservation of the past, new access strategies for the future, and active engagement with resources at the local and the global level for shared access and local ownership.

The NJDH uses FEDORA (Flexible Extensible Digital Object Repository Architecture) as a platform to mount participating institutions’ digital objects and metadata. FEDORA is developed jointly by Cornell University and the University of Virginia and is currently supported through an Andrew W. Mellon Foundation grant that is customizable and allows local institutions to have true control over what they digitize and post.2 FEDORA is built on XML with core standards that support flexibility and interoperability such as METS (Metadata Encoding and Transmition Standard, www.loc.gov/standards/mets) and OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting, www.openarchives.org) functions. FEDORA is chosen for the NJDH because it can effectively accommodate and manage a broad array of information sources with the flexibility to integrate with other information repositories.

The NJDH uses a metadata structure based on MODS (Metadata Open Description Schema, www.loc.gov/standards/mods), METS, NISO, and PREMIS (Preservation Metadata, www.loc.gov/standards/premis) metadata standards to support preservation of digital objects, to ensure scalability for projects and interoperability with other systems through OAI-PMH. This hybrid approach enables NJDH collection managers and metadata creators to provide information through multiple presentation standards in a schema easily understood within distinctive cultural heritage organization communities. MODS is used for descriptive metadata, provides and retains standard bibliographic cataloging principles, and is therefore easily mapped to MARC. The NJDH therefore includes a mapping utility that allows the export of records from the NJDH to online catalogs for any organization that wants to make its digital objects accessible within its integrated library system. Additionally, there are four other types of metadata in NJDH: source metadata describes provenance, condition, and conservation of analog source materials such as photographs, books, maps, audio, and video; technical metadata describes born digital images and provides information about the digital master files that will be maintained for long-term preservation and access; rights metadata identifies the rights holder(s) for each information source, identifies the permissions for use including any restrictions, and documents the copyright status of each work; digital provenance metadata provides a digital “audit trail” of any changes to the metadata.3

The use of the NJDH has steadily grown and has some three thousand unique visitors a month averaging eight to ten thousand visits per month.4

Prior cultural heritage digital library evaluations

Literature review indicates that few researchers have investigated the usability or the evaluation of cultural heritage digital libraries.

The MINERVA (Ministerial Network for Valorising Activities) project proposed a number of criteria and principles specifically for usability evaluations of cultural Web applications, including visibility, affordance, natural mapping, constraints, conceptual models, feedback, safety, flexibility, the scope and aim of the site, meaningful organization of the website’s functions, quality of content (for example, consistency, completeness, conciseness, accuracy, objectivity), design of functional layout, consistent use of graphics and multimedia components, as well as provision for navigation tools and search mechanisms.5 In addition, Vaki, Dallas, and Dalla proposed sixteen usability guidelines for cultural applications.6

Garoufallou, Siatri, and Balatsoukas reported their research on the user interface of the VeriaGrid application.7 The VeriaGrid system (www.theveriagrid.org) is a platform based on digital cartography that supports a vector map of the city of Veria organized by layers and linked to multimedia objects such as text, images, photos, and video clips. The researchers were interested in learnability, errors, and satisfaction.

Usefulness as the primary evaluation criterion for the NJDH

The NJDH aims to serve heterogeneous communities and information needs. Like other digital cultural services, it is not easy to address usability issues. Lynch has said that digital libraries of cultural heritage don’t really have natural communities around them and that digital materials find their own unexpected user communities.8 Garoufallou, Siatri, and Balatsoukas said that “different types of users, such as students and scholars or tourists and travelers look at these services from different angles (for example, scholarly or recreational needs). Thus, the provision of accessible and user-friendly systems is important for the wider use and acceptance of these services.”9

The aim of this evaluation was to assess usefulness of the NJDH from the perspectives of general users, educators, and cultural heritage professionals. Usefulness is one of the criteria of usability with a focus on “Did it really help me?” and “Was it worth the effort?” Usefulness differs from usableness in that usableness refers to functions such as “Can I turn it on?” or “Can I invoke that function?” Usefulness can also mean “serving an intended purpose.” In the Technology Acceptance Model (TAM) developed by Davis and his colleagues, perceived usefulness refers to the extent to which an information system will enhance a user’s performance.10

In addition to usefulness and usableness, Jeng has gathered a comprehensive collection of usability criteria such as effectiveness, efficiency, satisfaction, learnability, ease of use, memorability, mistake recovery, and interface effectiveness.11 Usability is a multidimensional construct and has a theoretical root in human–computer interaction.

Although usefulness may be an important evaluation criterion, Thomas and Jeng report that usefulness is an often overlooked criterion of usability.12 Literature review indicates that usefulness has been used as either the primary or one of the criteria in the following evaluations of digital libraries: eLibraryHub, the Digital Work Environment, GROW (Geotechnical, Rock, and Water Engineering, www.grow.arizona.edu), McMaster University Library’s Gateway, the Miguel de Cervantes Virtual Library, Minnesota’s Foundations Project, and the Moving Image Collections.13 This paper reports the evaluation of the NJDH.

Research method

A Web-based online survey was conducted in September. December 2006. The questionnaire was designed, collected, and analyzed using Web-based software called SurveyMonkey. Convenience sampling method was used in this study. Subjects were recruited by posting a link on the NJDH website, by posting announcements on a number of electronic discussion lists for educators and cultural heritage professionals, and by word-of-mouth invitations.

The participants were asked to complete a two-part questionnaire. The first part gathered demographic data such as gender, age, ethnic background, educational background, the county they live in, and how they learned about the NJDH. The second part contained three sections: one for everyone, one for educators, and one for cultural heritage professionals.

The section for everyone contained twenty-six questions, including seven-point Likert scales and open-ended questions with a focus on the digital library’s usefulness, navigation, design, terminology, and user lostness. In addition to this general section, educators were asked to complete another fifteen questions pertaining specifically to the educators’ portal; the cultural heritage professionals had another thirteen questions regarding the librarians and curators’ portal.

A total of 145 individuals participated in the survey, of which 32 were educators (22%) and 28 (20%) were cultural heritage professionals. The participants were mostly white (127 respondents or 89%), mostly female (118 respondents or 81%), and most had a master’s or doctoral degree (114 respondents or 79%). In terms of age distribution, more than half of the participants were over 50 (79 respondents or 55%) (see table 1). Nearly all (136 respondents or 94%) were residents of New Jersey.

Among the educators that participated in this survey who evaluated the educators’ portal, 56% (18 respondents) worked at colleges or universities, 16% (5 respondents) worked at high schools, 13% (4 respondents) worked at elementary or middle schools, and 6% (2 respondents) identified themselves as specialists in museums, libraries, or archives. Roughly a third (10 respondents or 31%) were teachers, 3% (1 respondent) was a teaching assistant, 13% (4 respondents) were school administrators, and 28% (9 respondents) were school library media specialists or librarians (see table 2). In terms of what they teach, 27% (7 respondents) teach New Jersey history, 23% (6 respondents) teach social studies, 12% (3 respondents) teach civics, 8% (2 respondents) teach geography, and 8% (2 respondents) teach popular culture.

As to the survey participants who identified themselves as cultural heritage professionals, 61% (17 respondents) worked at libraries, 11% (3 respondents) worked at museums, 11% (3 respondents) worked at historical societies, and 4% (1 respondent) worked with archives. In terms of their roles at those organizations, 61% (17 respondents) said they were faculty or staff, 18% (5 respondents) were administrators, one was a consultant, one was a librarian, and one was a volunteer (see table 3).

Findings

How do users find out about the NJDH and will they come back?

Figure 1. Where did you hear about NJDH?

Figure 2. Purpose of the most recent visit

Figure 3. Will you use NJDH again?

The survey found that more than half of the respondents (58 participants or 40%) learned about the NJDH from their colleagues or friends, 19 participants (13%) learned through attending conferences, 16 participants (11%) were linked from other websites (see figure 1). The NJDH digital library intends to build rich and “one stop shop” digital collections of New Jersey history and culture. Cultural heritage digital library plays a particularly important role for students of the humanities because the digital library is the humanist’s laboratory, its resources are the scholar’s primary data.14 It is important to enhance users’ awareness of this digital library among New Jerseyans and even promote this cultural heritage digital library to users at global level.

In terms of the purposes of visiting the NJDH, the study found 72 respondents (76%) were just browsing and 23 respondents (24%) were looking for specific information such as a specific county information, history, and family genealogy (see figure 2).

Seventy-two respondents (74%) replied that they will come back to use the NJDH again (see figure 3). Those who said “No” gave reasons such as their doubts on whether the information in the NJDH is reliable and authoritative, the depth and breadth of content in this digital library, and the inconsistency of fonts and font sizes.

Navigation

Navigation has been reported in literature as a common problem in a digital library. Users could accidentally leave the digital library, following the links to other Web-based resources, and were unaware that they were no longer using the digital library. Brinck, Gergle, and Wood report that disorientation is among the biggest frustrations for Web users.15

This survey found the overall response regarding the navigation of the NJDH was very positive (response average 2.54 on a 7-point Likert scale, 1 being easy to navigate and 7 being difficult to navigate). Twentythree participants (25%) marked 1 on the Likert scale, 28 participants (30%) marked 2, and 26 participants (28%) marked 3. These brought the total of the top three points to 83%.

The overall response regarding user lostness was also not a problem (response average 2.42 on a 7-point Likert scale, 1 being not lost at all and 7 being very lost). Only two participants expressed they were very lost and one expressed lost. The reasons that could lead to user lostness include the lack of material in the collections so far, the need for explanation of how relevance is ranked, the home page being text heavy and cluttered, the photos not being legible, the lack of author information in documents, no indication of a trail of how one got there, lengthy URLs, the need for better chosen direct links instead of layered links, and patrons’ unfamiliarity with icons and their functions.

Layout

The rating for the layout of the NJDH was very positive (response average 2.54 on a 7-point Likert scale, 1 being good and 7 being bad). However, the site may improve its appearance in the following areas: there is currently too much text per page (the font is too small and the use of typography, informational hierarchy, and white space must be improved); more important information needs to go at the top of pages; and more colors need to be used.

Terminology

The degree to which users interact with a digital library depends on how well users understand the terminology displayed on the system interface. Literature review has indicated that the inappropriate use of jargon has been a common problem in digital library design. Hartson, Shivakumar, and Perez-Quinones report from their usability inspection of the Networked Computer Science Technical Reference Library (www.ncstrol.org) that problems with wording accounted for 36% of the digital library’s usability problems.16 System designers often assume too much about the extent of user knowledge. The precise use of words in a user interface is one of the utmost important design considerations for usability.

This research found that the overall response regarding terminology and labeling in the NJDH was positive (response average 2.34 on a 7-point Likert scale, 1 being clear and 7 being not clear).

Usefulness

Usefulness was the fundamental research focus of this study. This research investigated whether the NJDH was useful to the general public, educators, and students. The responses were overwhelmingly positive: 73% of the respondents gave 1.3 ratings on the 7-point Likert scale (1 being useful and 7 being not useful)—30% (29 respondents) marked 1, 33% (32 respondents) marked 2, and 12% (12 respondents) marked 3. The average response was 2.63. This was a very positive response.

When it comes to the specific section for educators to evaluate the educator’s portal, the rating was also positive (response average 3.04). Those educators felt that the most useful information was the “how to” information for teaching with digital resources, research genealogy, developing an oral history, and so on. Twelve respondents (44%) indicated they would encourage their students to use the NJDH site for term papers or homework assignments. Thirteen respondents (50%) indicated they would make their own lesson plans using the resources and information from the NJDH.

Regarding the student’s portal, those educators who responded to the survey indicated that, from their perspectives, the most useful information for students was the general information about New Jersey, including a directory of cultural heritage organizations, places to visit, etc.

As for the librarians and curators’ portal, those cultural heritage professionals identified the Librarians and Curators’ Resource Center as the most useful resource in the NJDH, followed by the Digital Highway Collections Roadmap and associated guidelines, calendar, the searching capabilities of New Jersey Cultural Heritage Organizations, and New Jersey information. Sixteen respondents (67%) said they would recommend this digital library to their patrons, two respondents (8%) won’t, and six respondents (25%) were not sure. It is obvious that the NJDH administrators need to work harder in this area to enhance usefulness for cultural heritage professionals and their patrons.

The survey asked all respondents to suggest what themes should be enriched in the NJDH collections. The suggestions were, in this order: New Jersey history, New Jersey state and county documents, New Jersey culture, genealogy, everyday life in New Jersey, New Jersey industry, more immigration resources, education in New Jersey, New Jersey in wartime, and transportation.

Regarding the librarians and curators’ portal, the respondents suggested the contents of this particular portal should be enhanced in the following priority order: (1) more links to other websites with history resources and activities, (2) access to mentors experienced in digitizing and metadata who can provide one-to-one assistance, (3) a discussion list or blog where users can ask questions or share ideas with others, (4) information about training sessions around New Jersey on digitization and metadata, (5) more resources on digital preservation and metadata, (6) educational activities that users can share with their patrons, (7) a tool for users to create their own interactive activities using the NJDH resources, and (8) more information about helping patrons to use the NJDH more effectively.

Portal structure

The NJDH provides four portals for different target users: everyone, educators, students, and librarians and curators. Each portal provides different interface and packages different information for a different type of user. The survey found 80% of the subjects understood the purpose of the four portals (by marking 1 or 2 on the 7-point Likert scale) and only 4 participants (4%) found this type of portal structure confusing. The survey further found 65% of participants felt this kind of portal structure helpful to them.

Why not contributing to the NJDH collections?

The respondents indicated that the barriers for them to contribute collections or resources to the NJDH were, in this order: (1) lack of staff or time, (2) lack of funding, (3) lack of knowledge, and (4) copyright concerns.

Statistical analyses

The study found demographic factors, such as age, gender, ethnic background, and educational level, do not have significant effects on a number of areas: (1) how the participants ranked usefulness of the digital library, (2) usefulness evaluation of the four-portal structure, (3) understanding of terminology, (4) ease of navigation, and (5) lostness.

The study found the correlation between navigation and lostness was statistically significant: r (66) = .83, p < .001. When a user felt the system easy to navigate, the user felt less lost.

The study also found usefulness of the digital library has a statistically significant effect on a user’s return decision. A one-way analysis of variance was conducted. The analysis of variance was significant, F (2, 59) = 20.42, p < .001. The strength of relationship between usefulness ranking and the decision of whether to revisit the digital library, as assessed by n², was strong, with the usefulness factor accounting for 41% of the variance of the return decision. Because the overall F test was significant, follow-up tests were conducted to evaluate pairwise differences among the means. Using the Turkey test, the pairwise comparisons Yes vs. No and Yes vs. Not Sure were significant. The pairwise comparison No vs. Not Sure was not significant.

Conclusions

Usability evaluation is a user-centered evaluation to learn from users’ needs, expectations, and satisfaction. This research studied usefulness, navigation, user lostness, terminology, and layout. The overall response was positive, and the finding was that the NJDH was useful in providing New Jersey history and culture information. Designers of the NJDH learned from the study the priorities of adding various New Jersey themes to the collections and how to make the site easier to use. As a result of the study, lifelong learners are identified as an important target audience.

This research provided insights on why people came to use this particular digital library, their pleasure of using it, how to improve ease-of-use, navigation, website appearance, and the use of terminology and labeling. The front page of the website was redesigned to address the overuse of text on each page. The study also helped to discover what components of the site were more useful and why. Furthermore, it investigated why some museums or collections in New Jersey have not participated in this digital library development project. As a result of the study, more emphasis has been placed on building tools to increase independent collection contribution by museums and archives. The observations of this study may help the development of other academic digital libraries because the barriers found in the study are common obstacles. After eighteen months of the study, the NJDH Governance Planning Committee still uses the evaluation report to address more complex and fundamental changes and the reorganization of the digital library.

The study confirmed that users of this digital library appreciated the idea of providing different portals for different users.

The study did not find demographic factors (age, gender, ethnic background, and educational level) play statistically significant roles in the usefulness rankings of the digital library or portal structure, terminology, ease of use, or user lostness.

The study found there was a strong correlation between ease of navigation and user lostness. Users don’t have feelings of lostness when a system is easy to navigate.

The study also found users will come back to revisit a digital library when they find the site is useful.

Acknowledgments

Judy Jeng and Grace Agnew were the codesigners of the questionnaire for this study. Judy served as the evaluation consultant for the NJDH. Grace Agnew, the Associate University Librarian for Digital Library Systems at Rutgers University, was the principal investigator of the NJDH. The NJDH received funding from Institute of Museum Library Services Grant LG30-03-0269-03.

References

1. Linda Langschied, “History and High-Tech Intersect on the New Jersey Digital Highway,” www.imls.gov/profiles/Nov07.shtm (accessed Aug. 12, 2008).
2. Linda Langschied and Ann Montanaro, “The New Jersey Digital Highway: A Next-Generation Approach to Statewide Digital Library Development,” Microform & Imaging Review 34, no. 4 (2005): 167-73.
3. The New Jersey Digital Highway: Final Report on IMLS Grant #LG30-03-0269-03, www.njdigitalhighway.org/documents/njdh-final_report_www_version.pdf (accessed Aug. 12, 2008).
4. Ibid.
5. MINERVA Working Group 5, Handbook for Quality in Cultural Web Sites Improving Quality for Citizens: Version 1.2—Draft. (2003), www.minervaeurope.org/publications/qualitycriteria1_2draft/qualitypdf1103.pdf (accessed Aug. 12, 2008).
6. Elina Vaki, Costis Dallas, and Christina Dalla, Calimera: Cultural Applications: Local Institutions Mediating Electronic Resources: Deliverable D 18: Usability Guidelines, www.calimera.org/Lists/Resources%20Library/The%20end%20user%20 experience,%20a%20usable%20community%20memory/Usability%20Guidelines.pdf (accessed Aug. 12, 2008).
7. Emmanouel Garoufallou, Rania Siatri, and Panagiotis Balatsoukas, “Virtual Maps.Virtual Worlds: Testing the Usability of a Greek Virtual Cultural Map,” Journal of the American Society for Information Science and Technology 59, no. 4 (2008): 591–601.
8. Clifford Lynch, “Digital Collections, Digital Libraries and the Digitization of Cultural Heritage Information,” First Monday 7, no. 5 (2002), www.firstmonday.org/issues/issue7_5/lynch/ (accessed Aug. 12, 2008).
9. Garoufallou, Siatri, and Balatsoukas, “Virtual Maps. Virtual Worlds,” 591–601.
10. Fred D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly 13, no. 3 (1989): 319–40; Fred D. Davis, Richard P. Bagozzi, and Paul R. Warshaw, “User Acceptance of Computer Technology: A Comparison of Two Theoretical Models,” Management Science 35, no. 8 (1989): 982–1003.
11. Judy Jeng, “Usability of the Digital Library: An Evaluation Model” (PhD diss., Rutgers University, 2006): 10–19; Judy Jeng, “Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability,” Libri: International Journal of Libraries and Information Services 55, no. 2/3 (2005): 96–121; Judy Jeng, “What is Usability in the Context of the Digital Library and How Can It Be Measured?” Information Technology and Libraries 24, no. 2 (2005): 47–56.
12. Rita Leigh Thomas, “Elements of Performance and Satisfaction as Indicators of the Usability of Digital Spatial Interfaces for Information-Seeking: Implications for ISLA” (PhD diss., Univ. of Southern California, 1998); Judy Jeng, “Usability of the Digital Library: An Evaluation Model” (PhD diss., Rutgers University, 2006): 33.
13. Yin-Leng Theng, Mei-Yee Chan, Ai-Ling Khoo, and Raju Buddharaju, “Quantitative and Qualitative Evaluations of the Singapore National Library Board’s Digital Library,” in Design and Usability of Digital Libraries: Case Studies in the Asia Pacific, ed. Yin-Leng Theng and Schubert Foo (Hershey, Pa.: Information Science Publishing, 2005): 334–49.; N. Meyyappan, Schubert Foo, and G. G. Chowdhury, “Design and Evaluation of a Task-Based Digital Library for the Academic Community,” Journal of Documentation 60, no. 4 (2004): 449–75; Janice Lodato, “Creating an Educational Digital Library: GROW a National Civil Engineering Education Resource Library,” (paper presented at the Conference on Human Factors in Computing Systems, Vienna, Austria, Apr. 24–29, 2004), in the ACM Digital Library, http://portal.acm.org/citation.cfm?id=985942&coll=portal&dl =ACM&CFID=32427354&CFTOKEN=28824529 (accessed Aug. 12, 2008); Brian Detlor et al., Fostering Robust Library Portals: An Assessment of the McMaster University Library Gateway (Hamilton, Ont.: Michael G. DeGroote School of Business, McMaster University, 2003); Alvaro Quijano-Solis and Raul Novelo-Pena, “Evaluating a Monolingual Multinational Digital Library by Using Usability: An Exploratory Approach from a Developing Country,” The International Information & Library Review 37, no. 4 (2005): 329–36; Eileen Quam, “Informing and Evaluating a Metadata Initiative: Usability and Metadata Studies in Minnesota’s Foundations Project,” Government Information Quarterly 18, no. 3 (2001): 181–94; Judy Jeng, “Metadata Usefulness Evaluation of the Moving Image Collections” (paper presented at the New Jersey Library Association annual conference, Long Branch, New Jersey, Apr. 23–25, 2007), www.njla.org/conference/2007/presentations/Metadata.pdf (accessed Aug. 12, 2008).
14. Gregory Crane and Clifford Wulfman, “Towards a Cultural Heritage Digital Library,” Proceedings of the 3rd ACM/IEEE-CS Joint Conference on Digital Libraries, in the ACM Digital Library, http://delivery.acm.org/10.1145/830000/827150/p75 -crane.pdf?key1=827150&key2=9784876911&coll=ACM&dl=A CM&CFID=8598346&CFTOKEN=44546164 (accessed Aug. 12, 2008).
15. Tom Brinck, Darren Gergle, and Scott D. Wood, Designing Web Sites that Work: Usability for the Web (San Francisco: Morgan Kaufmann, 2002).
16. H. Rex Hartson, Priy A. Shivakumar, and Manuel A. Perez-Quinones, “Usability Inspection of Digital Libraries: A Case Study,” International Journal on Digital Libraries 4, no. 2 (2004): 108–23.