SLMQ Volume 22, Number 1, Fall 1993

Current Research: A Study of High School Students' Online Catalog Searching Behavior

Shu-Hsien Chen is Library Media Specialist, Flynt Street Middle School, Griffin, Georgia. At the time of this article she had twenty years of experience as a library media specialist, primarily at the middle-school level, and was Assistant Professor of Library and Information Studies at Clark Atlanta University in Atlanta, Georgia. This paper is based upon her dissertation for her Ed.D. in Instructional Technology from the University of Georgia in Athens and reports the segment of her timings that is most closely related to library media specialists' instructional concerns.

How do secondary students use an online catalog? Can they use it to retrieve information? What are some of the errors they make? How, if at all, do they recover? These are some of the questions about online searching to which school library media specialists need answers. Educators must have a fundamental grasp of students' search behaviors and difficulties in order to design effective and efficient instruction for searching in electronic sources. As Sullivan and Seiden stated, "Knowing what problems do occur makes it possible for user educators to begin planning strategies to address the problems."(1) This study investigated the search behavior of high school students using an online catalog. By examining students' actions in this environment in detail, the study attempted to pinpoint both general and specific problems the school library media specialist should address in information-skills instruction.

High school students, like adults, must know how to locate information for decision making regardless of the format in which it is stored.(2) To meet their information needs for completing school assignments and pursuing personal interests, students today must often master an online catalog system to gain easy, direct, and immediate access to library holdings.(3) As more online catalog systems become available for schools(4) and online searching programs are adopted in school library media centers,(5) online searching skills must soon become part of library and information-skills curricula across the nation.

Most research to date, however, has focused on adult users rather than students. A national survey of online catalog use indicated these users' widespread acceptance of and satisfaction with this new technology.(6) The study also revealed that a very small number of teenagers among the total group of survey respondents accepted the online catalog even more readily than the adults.(7)

Despite their generally favorable attitudes toward the online catalog, users have experienced a number of problems while searching for information.(8) Earlier studies investigating the effectiveness of the online catalog found that adult users experienced little success with their search results.(9) A more recent study found that elementary and middle-school students used the online catalog less effectively than the card catalog, with search results at success rates of 10 percent and 65 percent, respectively.(10) Research is just beginning to emerge that addresses the implications of these and similar results both for system design and for student instruction.(11)

Purpose

The purpose of this study was to develop a beginning understanding of the search behaviors of high school students using an online catalog. Four major research questions guided the investigation:

1. When high school students conduct online catalog searches, do they have a higher success rate with an author, a title, or a subject search?.
2. What errors do students make while conducting an online catalog search?
3. What types of errors do students most frequently make in conducting an online catalog search?
4. How do students reformulate a search when experiencing problems or difficulties in the online catalog search?

Research Design

The study was conducted in a Georgia high school where Mac Library Access Program (MacLAP),(12) a Macintosh-based online catalog, had been recently installed. An exploratory study rather than an experiment, the research emphasized observation of students' search behavior and analysis of their interim and final results.

Subjects

Thirty-five subjects were randomly selected from a group of eighty-nine students enwlled in English III, a course geared toward college-bound eleventh graders of average or slightly less than average academic ability. The subjects were twenty-one males and fourteen females, ages sixteen to eighteen.

Subjects were assumed to have minimal experience with the online catalog at their school or elsewhere. At the time of the study, MacLAP had not been fully implemented, and the media specialists had not yet pwvided orientation or instruction on the new system to faculty and students. Although the English teacher had taken the students to the library media center to examine the new online catalog, they had no formal experience with the system prior to the study.

Procedures

Only two terminals were available for the study, and the researcher worked with pairs of subjects so that each student had individual access to a terminal. The researcher conducted two 50-minute sessions-one for instruction and one for the actual searches-for each pair. The instructional session focused on the nature of online searching, how it differs from searching in print resources, how to perform various types of searches, and how to use Boolean operators. During this session students also worked on practice search problems to gain hands-on experience and took a twelve-item written test on online catalog use.

Students returned the day after their instructional sessions for their search sessions, in which each student worked out a set of eight search problems on the online catalog and recorded the search results. The researcher observed the students while they searched, and two video cameras (one for each terminal) recorded each step of each student's efforts.

Instrumentation

The written test that evaluated students' general comprehension of the basics of online catalog use was used to provide contextual information for the study. The primary instruments were the Search Problem sheets (see figure 1), which set out problems for students to solve by identifying twenty-seven items of specific bibliographic information: authors, titles, topics, call numbers, and publication dates of several books cataloged in the local Mac-LAP system. The Search Problem sheets were used to measure students' search success; to determine their search errors; and, with data from the videotapes, to study their search reformulations.

Both the written test and the search problems were formulated by the researcher on the basis of her personal experience as a media specialist and through informal interviews with four teenagers and a colleague. The instruments were reviewed by a panel of six university faculty members and pilot-tested with five students not included in the study.

Videotapes of the students' searches also provided data for analysis. These tapes did not capture sound but recorded the search terms entered by students, the search errors they made, their changes in search tactics, and the final search results displayed on the screen. The researcher transcribed each videotape-that is, made exact notations of the steps and results of each student's search.

Data Analysis

Search success (Research Question 1) was operationally defined as the scores achieved by students on the set of search problems. Each student's total score was determined simply by computing the percentage of the total number of items (n = 27) the student identified correctly:

Success rate = Number of correct items/27 x 100%.

Thus, a student who identified eighteen items correctly received a score of 67 percent. To be correct, an answer must have been 1) selected correctly from the screen display and 2) transcribed accurately on the search problem sheet. Students' success rates on the various types of searches (author, title, and subject) were computed in similar fashion.

The study's most extensive analysis focused on the types of errors students made. These errors were considered to be of two types: "impact errors," which were defined as errors in the final answers on the sheets, and "potential errors," which were defined as errors students made during the search process that they detected and correctedbefore recording their final answers. Research Questions 2 and 3 focused on impact errors, while Research Question 4 focused on potential errors in order to shed light on students' reformulation patterns.

Limitations

Several limitations must be kept in mind when interpreting the results of the study. First, because it is a case study of an individual site rather than a broadly based statistical study, the results are not generalizable to other settings. Second, the short duration of the study (two sessions for each student over a three-week period) and the fact that all the instruction was provided by the researcher also preclude applying the findings to other settings, which are likely to be substantially different. Third, limitations related to the MacLAP system itself suggest caution in applying the findings to other situations: the system was new to the research site, is not widely used within schools, and has a variety of technical flaws not necessarily found in other systems. While the full dissertation describes these flaws and suggests a variety of improvements in the MacLAP system,(13) this column focuses only on the instructional implications of students' errors in order to provide general insights that school library media specialists might adapt to their own situations.

Results

Search Success Rates

Students had the highest success rates with author searches (69 percent), followed by title searches (67 percent) and subject searches (60 percent). Although the differences among these three search types were not statistically significant, some variations in success rates for individual students with different search types were found. Generally, however, the success rates for author and title searches were higher than those for subject searches.

Search Errors

The results indicated that students made five types of errors in their searches: (1) typographical and spelling errors, (2) errors in using the system, (3) errors in generating search terms, (4) errors in using information, and (5) errors in recording search results.

Students made errors in typing (including spelling and spacing) when entering search terms. Several students failed initially to invert the authors' name order in author searches and to delete leading articles in title searches. Some students failed to leave spaces following authors' surnames; a few used two spaces between words. Table 1 shows some examples of students' typographical errors.

Students' problems in using or understanding system commands included errors in 1) obtaining full information, 2) extending searches, 3) employing search approaches, and 4) starting searches. Data from the videotapes indicated that after retrieving briefbibliographic information, several students had difficulty obtaining the full bibliographic information that the system also made available. Students also failed to use such screen information as tracings (subject headings and other added entries) and classification numbers as clues to conduct further searches in related areas or to identify books on similar topics. Two students used every available search approach regardless of the type of search terms they entered: after they had chosen the subject approach, typed in "en-viromental" [sic], and failed to retrieve an item, they tried all other search types-even author and call number approaches-using the very same term. One student repeatedly failed to click "Find" or press "Return" to launch a search. Some other students continued to press "Find" as many as twenty-five times, using an identical search approach and same term even though the message "no item found" was displayed on the screen again and again.

Errors in generating seamh terms occurred most frequently when students worked on problem 4 about the relationship between the United States and China after World War II. As table 2 suggests, students used natural language expressions that were appropriate for them but inappropriate within the controlled vocabulary of the MacLAP system.

Students also made errors using substantive information during their searches, and these mistakes can be grouped into three areas: errors related to the searchproblems, to search information displayed on the screen, and to extracting screen information. In making errors related to the search problems, students chose religion, social science, or history titles when science titles were required; they selected fiction titles for "drug use" and "drunk driving" when the search problems specified nonfiction titles. The most striking example related to the search problems themselves involved a classification number search (problem 5). Here, thirty-four of the thirty-five students in the study were able to obtain a screen that included the correct answer (one of four items listed). Fifty percent of these students, however, failed to select the item that satisfied all the problem elements specified by the search question.

Students made errors in relation to screen information when they searched for citations about the U.S.-China relationship and about drug use. Students generally found answers for these problems, yet some of them apparently did not recognize the answers when they appeared in the middle of a long list. A small number of students retrieved a potential item several times under different terms but left the search question unanswered. In errors related to extracting screen information, several students combined information from one title with information from another title in order to find an answer.

Students also made errors copying information from screen displays to their Problem Sheets after they had retrieved correct information. Students miscopied authors' names or provided incomplete or inaccurate bibliographic information about authors, titles, and call numbers. Table 3 displays some errors of this type.

Table 4 summarizes the numbers and percentages of students' errors. As the table indicates, more students made errors in using substantive information than in any other area. Eighty-five percent of the students (30) made errors of this type, while 48 percent made errors in using the system and 34 percent erred in typing and spelling. Even after fruitful searches, 22 percent of the students miscopied results. The table also indicates that the greatest number of errors made (130) were related to using substantive information, while typographical and spelling errors numbered 70 and errors in using the system numbered 52.

Search Reformulation Patterns

Analysis of the data from the videotapes revealed several reformulation patterns that emerged from students' repeated attempts to improve their search results. Generally, students employed two techniques to reformulate their searches: switching search types (e.g., from author to title) and rephrasing search terms. Within the second technique, reformulation for author and title searches seemed to focus on correcting spelling and spacing. Reformulation also occurred when students reversed the order of authors' names and deleted the initial articles in titles.

Although students made some changes in search terms in author and title searches, modification of terms was more pronounced for subject and key word searches. Three major patterns emerged from students' efforts to rephrase search terms: changing terms from the specific to the general, using similar or synonymous terms after initial searches failed, and adopting different concepts and terms to continue searching. All these strategies had been taught in the initial training sessions, but students applied them without any assistance from the researcher during their searches.

Students tended to pick a string of words directly from a search problem to use as a search term. Sometimes they used major phrases; at other times they tried key words. In either case, students' choices frequently fell outside the system's controlled vocabulary for subject searches. When such searches failed, many students tended to rephrase their search terms from the specific to the general (see table 5).

Some students tried to reformulate subject and key word searches by expressing related concepts in similar terms. Table 6 shows that the initial and modified terms were synonymous or very close in meaning. The third reformulation pattern was the adoption of an entirely different concept to continue the search. Table 7 shows the kinds of choices students made when they followed this pattern after encountering difficulty with an initial search.

Discussion

The nature and types of errors students made suggest that some of their difficulties stemmed from their lack of general information-seeking and language skills. Even if such skills were present, students may have had difficulty transferring them from the traditional to the electronic searching environment. It is important to note that students' difficulties in this study were amplified by a system that required highly specific behaviors for successful searching.

Information-Seeking Skills

Students had few problems identifying descriptive bibliographic information-that is, the author, title, publisher, copyright date, etc.-on an electronic "catalog card." Half the students, however, did not know where to locate additional subject headings or how to use these tracings to design additional searches. This problem was evident in problem 8, in which students were to use a subject heading such as "drunk driving" to search further for a nonfiction title. This finding echoes others in the literature: Lancaster' s review of card catalog studies indicated that most library users have little knowledge of the structure of the card catalog,(14) while Dickson also found that library patrons do not understand the card catalog very well.(15) As McDonald and Searing reported, "What really becomes obvious when you implement an online catalog is that users do not know how to use any catalog, card or online, effectively."(16) Thus, although technology may change the mode of information searching, this study suggests that the old problem of not understanding the basic catalog structure may still remain.

Fifty-seven percent of these students did not understand the difference between a call number and a classification number. Four search problems required call numbers for partial answers, but many students answered with classification numbers or used these two kinds of numbers interchangeably. It seems that students did not truly understand the functions of the Dewey Classification System in relation to locating books on library shelves. Many students seemed unaware that using an identical classification number as a clue is the easiest way to locate books on a similar topic. These findings support Moore's and St. George's contention that "many children did not have enough knowledge of the relationship between catalog cards, books, and shelving systems to put their theoretical knowledge of the Dewey system into practice."(17)

The results of the classification number search (problem 5) in which students retrieved appropriate screens but failed to select the correct answer suggest either that students did not understand the problem or that they did not evaluate their search results carefully before selecting an answer. The latter interpretation coincides with Fenichel's findings: two-thirds of the subjects in her study did not take the opportunity to review their retrieved items.(18) Skills in analyzing search problems and in reviewing search results are clearly related to successful searching.

As noted above, students had difficulty with the mechanical aspects of searching as well as with its conceptual components. Some students entered authors' names without reversing forenames and surnames or typed in the titles of books without eliminating initial articles. Some did not include the comma after reversing a name; others failed to leave a space after the comma. Dickson reported similar findings in a study on college students, in which she found that 18 percent of the unsuccessful author searches were entered with forenames tirst and 16 percent of the failed title searches were conducted under initial articles.(19)

Several students in the study also felt no hesitation about abbreviating a book title, shortening an author's name, or truncating a call number when recording the search results on paper. Students seemed not to understand that such shortcuts can lead to failure to find a book in the stacks-a problem in electronic environments that could, of course, be eliminated by attaching printers to online catalogs.

Language Skills

The study also suggests that some of these average and below-average students probably did not have adequate language skills to search productively. Their language-re-lated problems seemed to focus in three areas: spelling, typing, and usage; reading ability; and vocabulary.

Spelling and typing problems were discussed above, and problems in usage also seemed to contribute to students' errors. For example, in problem 2, the search term "Tunis"'-the author's sum, sine entered with the apostrophe-was found among other search terms, suggesting that students did not realize that the apostrophe was a punctuation mark showing possession rather than part of the author's name.

For some students, retrieving information to the green was far less difficult than reading it-that is, interpreting findings correctly and selecting correct titles from long lists. This problem was most evident in searches for information on drugs and on the relationship between the United States and China after World War II. The videotapes clearly show that some students were able to retrieve appropriate lists two or three times under various search terms but failed to answer the questions correctly. These findings concur with the results of both Fenichel's study (20) and the Matthews et al. national survey that reported users' difficulties in identifying desired items in long display lists.(21)

Students also had difficulty thinking of key words to use as search terms. They often copied partial search questions verbatim and entered long phrases into the computer to launch their searches. For instance, many used "relation between U.S. and China after World War II" or "Problems of drug use by athletes" as search terms. One unusual search term ("drunk driving-nonfiction") was used by one student. The problem of using appropriate search vocabulary is clearly related to language ability: the searcher must grasp the main concepts in a statement, identify potential key words for searches, and generate related search terms if the initialterm fails. When such sophisticated linguistic processing must occur within the constraints of a controlled vocabulary, the demands on users' language skills are greatly heightened.

In sum, the substantive errors of the students in this study can be described by the remarks of Sullivan and Seiden, who noted that "strategic errors may arise from a lack of library knowledge, a lack of system knowledge, or a lack of subject knowledge."(22) In addition to deficiencies in these areas, this study found errors related to students' language abilities as they relate to understanding search problems, interpreting screen information, extracting main concepts, and expressing them in appropriate key words. While the flaws of the particular system used for this study also contributed to students' search problems, it is clear that students' errors related to general issues as well as to system-specific ones.

Implications for Instruction

Instructional attention to these general issues is clearly warranted. Merely emphasizing the mechanical aspects of online technology will probably not serve students very well. As Harter noted, conceptual aspects such as the formulation of search objectives and.the identification of major concepts for successful searching should be emphasized.(23)

This study suggests that a review of the fundamentals of the Dewey Classification System and of the elements of bibliographic information, particularly tracings, is as important as instruction on the mechanics of system operation. Further, developing the concept of key words is also crucial. For productive searches, students must be able to read search problems or statements, extract key concepts, use appropriate terms to express the concepts, and enter them for searching. If a search fails, students must know how to try alternative concepts. For any search, whether online or offline, "The efficacy of the search depends on the individual's ability to match key words to information sources, to see relationships between differing aspects of the topic . . . to monitor the outcome of search strategies, and to regulate them accordingly."(24)

Conclusions

Many, but not all, of the students in this study were able to use the online catalog successfully to locate information. Students were considered successful in online catalog searching despite their average or below-average academic standing (some were repeating eleventh grade), their lack of online searching experience, and the short duration of the instruction given tothem. In spite of the difficulties they experienced, students performed with interest and persistence. Not surprisingly, they found it easier to locate information on an author or a title than on a topical area, as in subject searches.

As students become more accustomed to online searching, they may be expected to search with greater ease and speed. Typing and spacing errors can be overcome with practice. Many errors resulting from inadequate information-seeking and language skills can also be eliminated with instruction. What will constitute an effective and efficient instructional program to teach students online searching skills? This is clearly a topic that must be addressed in further detail.

It seems evident from this single study, however, that such a program must address basic academic skills as well as the new conceptual demands that online catalog technology places upon students. No matter how advanced technology facilitates information seeking in the future, students will still need a familiar assortment of skills to perform successful searches: typing, spelling, usage, reading, interpreting, knowing key word concepts, and understanding the fundamentals of a school's classification system. These basic information and language skills are essential for productive search results.

As online skills become increasingly important in this information-rich society, these skills will become vital to children as well as to adults. As Aversa and Maneall stated, "There is no longer any question that knowing how to seek information electronically will be an essential skill for all individuals."(25)

Research on various systems as used by different age groups is needed to shed light on the special problems and difficulties that students encounter as they employ the new technology to search for information. Only the findings from a variety of research studies can provide a basis for designing effective online skill development programs for younger users.

References

1. Patricia Sullivan and Peggy Seiden, "Educating Online Catalog Users: The Protocol Assessment of Needs," Library Hi-Tech 3 (Issue 10, 1985): 18.
2. Jacqueline C. Mancall, "Training Students to Search Online: Rationale, Process, and Implications," Drexel Library Quarterly 20 (Winter 1984): 64-84.
3. Robert Skapura, "A Primer on Automating the Card Catalog," SLMQ 18 (Winter 1990): 75-78; M. Turnage, "User Education," Texas Library Journal 61 (Winter 1985): 112-13, 128-29.
4. Catherine Murphy, ed. Automating School School Library Catalogs: A Reader (Englewood, Colo.: Libraries Unlimited, 1992).
5. Doris M. Epler, Online Searching Goes to School (Phoenix, Az.: Oryx, 1989).
6. Joseph R. Matthews, Gary S. Lawrence, and Douglas K. Ferguson, eds., Using Online Catalogs: A Nationwide Survey (New York: Neal-Schuman, 1983).
7. Karen Markey, Subject Searching in Library Catalogs: Before and after the Introduction of Online Catalogs (Dublin, Ohio: OCLC, 1984).
8. Christine L. Borgman, "Why Are Online Catalogs Hard to Use? Lessons Learned from Information-Retrieval Studies," Journal of the American Society for Information Science 37 (Nov. 1986): 387-400.
9. Sue Pease and Mary Noel Gouke, "Patterns of Use in an Online Catalog and a Card Catalog," College & Research Libraries 43 (July 1982): 279-91; Carol Watson, Susan Williamson, and Howard D. White, "Resistance to Online Catalogs: A Comparative Study at Bryn Mawr and Swarthmore Colleges," Library Resources and Technical Services 30 (Oct./Dec. 1986): 388-401.
10. Leslie Edmonds, Paula Moore, and Kathleen M. Balcom, "The Effectiveness of an Online Catalog," School Library Journal 36 (Oct. 1990): 28-32.
11. Delia Neuman, "Designing Databases as Tools for Higher-Level Learning," Educational Technology Research and Development (in press); Paul Solomon, "Children's Information Retrieval Behavior: A Case Analysis of an OPAC," Journal of the American Society for Information Science 44 (June 1993): 245-64.
12. MacLAP: User's manual (Cupertino, Calif.: CASPR, 1989).
13. Shu-Hsien Chen, "A Study of Online Catalog Searching Behavior of High School Students" (Ed.D. diss., Univ. of Georgia, 1991).
14. F. M. Lancaster, The Measurement and Evaluation of Library Services (Washington, D.C.: Information Resources Press, 1977).
15. Jean Dickson, "An Analysis of User Errors in Searching an Online Catalog," Cataloging and Classification Quarterly 4 (Spring 1984): 1-38.
16. David R. McDonald and Susan E. Searing, "Bibliographic Instruction and the Development of Online Catalogs." College & Research Libraries 44 (Jan. 1983): 10.
17. Penelope A. Moore and Alison St. George, "Children as Information Seekers: The Cognitive Demands of Books and Library Systems," SLMQ 19 (Spring 1991): 165.
18. Carol H. Fenichel, "Online Searching: Measures that Discriminate among Users with Different Types of Experiences," Journal of the American Society for Information Science 32 (Jan. 1981): 23-32.
19. Dickson, "Analysis of User Errors."
20. Fenichel, "Online Searching."
21. Matthews, Lawrence, and Ferguson, Using Online Catalogs.
22. Sullivan and Seiden, "Educating Online Catalog Users," 18.
23. Stephen P. Harter, Online Information Retrieval: Concepts, Principles, and Techniques (New York: Academic Press, 1986).
24. Moore and St. George, "Children as Information Seekers," 167.
25. Elizabeth S. Aversa and Jacqueline C. Mancall, Management of Online Search Services in Schools (Santa Barbara, Calif.: ABC-CLIO, 1989), xiii.