College and Research Libraries The Effect of CD-ROM Instruction on Search Operator Use Trudi E. Jacobson and Janice G. Newkirk This study reports the results of a survey of 675 CD-ROM users, obtain- ing data from both questionnaires and actual search strategies. One of the authors' primary concerns involved investigating the effect of prior instruction or assistance on search strategy. The authors found a statis- tically significant but weak relationship between prior instruction and level of search skill. Other measures such as user status, department affiliation/major, age, sex, and number of previous CD-ROM uses were compared to search skill. Only user status and department affiliation/ major were significantly correlated to search skill. Additional research is needed on the effectiveness of various types of CD-ROM instruction. D-ROM searching has sub- stantially increased the ser- vice demands in libraries since its advent almost a de- cade ago. Public service academic librar- ians spend a considerable amount of time assisting and instructing patrons in the use of these end-user databases. Yet, for all the time and money devoted to these activities, little research has been done on their effectiveness. The authors based their instruction and assistance approaches on what they discovered helping people with paper resources, adding, of course, infor- mation about the technology. Research indicates that many library patrons do not search very well.l Con- versely, research shows that patrons en- joy using CD-ROM indexes, whether they use them successfully or not. Research also indicates that librarians are assisting and instructing patrons in increasing numbers. (At the University at Albany, State University of New York, CD-ROM questions as a percentage of total refer- ence questions have steadily increased at a rate of two percent per year since 1992.) But does such assistance and instruction affect patrons' search skills? Does the im- mense investment in patron assistance with CD-ROMs matter? These may be two of the most impor- tant questions in academic library public services today, yet remarkably little re- TrudiE. Jacobson is Coordinator of User Education Programs for the University Libraries at the Univer- sity at Albany, State University of New York, and the late Janice G. Newkirk was CD-ROM Coordinator for the University Libraries at the University at Albany. The research reported here was assisted by a grant from the Spencer Foundation. The data presented, statements made, and views expressed are solely the responsibility of the authors. The authors would like to thank Rhonda Allen, a doctoral student at the University at Albany, and John Wuthrich for their assistance and valuable insights on the data analysis and on this article. The authors acknowledge the assistance of Thomas]. Galvin, Director of the Informa- tion Science Doctoral Program in the University at Albany's Nelson A. Rockefeller College of Public Affairs and Policy, in connection with the Spencer Foundation grant that supported this study. 68 search has directly addressed them. At the University at Albany, the authors thought it was time to examine these is- sues. With a grant from the Spencer Foun- dation, the authors set out to study what impact librarian instruction and assis- Although the CD-ROM and end- user literature include evaluations of end-user searching ... there are few studies that assess the impact of instruction on end-user searching. tance were having on CD-ROM search- ing. We hoped to find out whether classes and one-on-one assistance from librarians had a considerable impact on end-user search proficiency. We hypothesized that patrons who had attended searching classes or who had been directly assisted by librarians (or trained library assistants) would be far more sophisticated search- ers. Specifically, we wanted to test whether end-users having training or as- sistance would use more Boolean opera- tors, field searches, and referrals to pre- vious search terms than end-users who had no assistance or instruction. Literature Review Although the CD-ROM and end-user lit- erature includes evaluations of end-user searching, accomplished by either sur- veying patrons on their satisfaction with their searches or actually analyzing the results of their searches, there are few studies that assess the impact of instruc- tion on end-user searching. Domenica M. Barbuto and Elena E. Cevallos found that students generally do not use advanced searching techniques emphasized in the training sessions. 2 Rebecca Bostian and Anne Robbins examined PsycLIT search results from students who had a range of instructional experiences. They con- cluded that the only instruction that caused a significant difference in search capability was a live search demonstra- tion.3 Dorothy F. Davis, who also studied the effectiveness of different types of CD-ROM Instruction 69 PsycLIT CD-ROM instruction, suggests that video instruction is the most effec- tive method. 4 Linda Stewart and Jan Olsen found that students given formal instruction on the CD-ROM version of ERIC did search more effectively.5 Their instruction concentrated on the use of Boolean logic and vocabulary control, and included a search demonstration. Methodology Project Design The current project involved a two-part collection of data. One part involved ad- ministering a survey to all library patrons using SilverPlatter CD-ROMs during se- lected periods. This survey included a questionnaire (available from the authors) to collect basic demographic data, infor- mation on the respondents' experience with computers, and information on what type of prior instruction and/ or assis- tance respondents had in CD-ROM searching. The second part of the data collection involved saving each respon- dent's search strategy and attaching it to the questionnaire. We employed three students to administer the survey dur- ing morning, afternoon, and evening hours, weekends as well as weekdays, from February through May of 1994. These students were available to answer questions that arose as respondents filled out the questionnaire. In all, the students approached more than a thousand pa- trons for the survey. We discarded sev- eral hundred surveys because the patrons indicated that they had received the im- mediate and direct assistance of a librar- ian with the search in question. We con- cluded that these search strategies would represent the librarians' rather than the patrons' efforts. We collected a total of 675 usable surveys. Survey Instrument The questionnaire was divided into four sections: section 1 gathered basic demo- graphic data, and section 2 evaluated the respondents' computer experience. We 70 College & Research Libraries hypothesized that CD-ROM searchers with extensive experience using other computer applications are probably more effective searchers than those who are unfamiliar with computers. Section 3 asked about the purpose of, and satisfac- tion with, the current search. It also asked respondents to indicate how often they had searched using CD-ROMs in the past. We used this section minimally in our analysis. Section 4, the most important section of the questionnaire, identified the types of CD-ROM training and assistance that the respondents had previously re- ceived. Because we were trying to iden- tify the most effective types of assistance and instruction, this last section of the sur- vey contained thirty categories, which in- cluded various types of classroom in- struction, as well as other forms of in- structional assistance, classified by time and location of instruction. The second part of the data collection involved the respondents' search strategy. Though the survey data were subject to the usual problems of self-reported data, we determined the search skill level based on students' actual search strategies. One of the authors coded all the strategies for the number of different Boolean opera- tors, field searches, and referrals to pre- vious search terms used. In the follow- ing discussion, references to "operators" or "Boolean operators" should be under- stood to include the concepts of field searches and referrals to previous search statements. Although we recognize that the use of a variety of search operators is not always necessary for a good or use- ful search, correct use of a variety of op- erators is a clear indication of knowledge of, or skill at, searching and should di- rectly relate to good searching. In addi- tion, these elements are easy to measure objectively. The Survey Population Although we had hoped to collect sur- veys from a representative cross section of the population that uses the Univer- January 1996 sity at Albany's CD-ROMs, the persons participating in the survey were predomi- nantly female (almost 66%) and were overwhelmingly involved in research in the social sciences (over 28% in psychol- ogy, over 21% in education, and over 14% in other social science disciplines) (see table 1). In other words, over 60 percent of those included in the survey were working in one of these three areas. We cannot say for certain how representative our survey TABLEt Summary of Characteristics for Survey Respondents Age 0-17 18-25 26-34 % of respondents 35 or older Missing 0.6 59.3 22.1 17.9 0.1 Sex Female Male % of respondents 66.0 34.0 Major/dept. Agriculture % of respondents 0 .1 Area Studies Biological Scis. Business & Mgt. Communications Comp. & Info . Sci. Education Fine & Applied Arts Foreign Language Health Professions Humanities Library Sciences Mathematics Physical Sciences Psychology Public Affairs Social Sciences Interdisciplinary University Admin. Not Applicable Missing Data 1.9 4.3 2.4 1.0 0.9 21.5 0.4 0.7 1.5 8.3 0.1 1.0 1.0 28.6 4.9 14.5 0.3 0.9 1.2 4.3 CD-ROM Instruction 71 TABLE2 lation: 26.1 percent Cross-tabulation of Training by Skill Level of all searches done on SilverPlatter da- tabases were con- ducted on PsycLIT, and 22 percent were conducted on ERIC. None or Low Incorrect Use Skill Level Moderate Skill Level High Skill Level Ever-trained 56 60 98 73 19.5% 20.9% 34.1% 25.4% Never-trained 116 106 111 55 29.9% 27.3% 28.6% 14.2% As for age distri- bution, over 59 per- cent of the respon- dents were be- is because no previous survey has been done to explore the demography of our CD-ROM users. However, with the excep- tion of the gender breakdown, these fig- ures closely reflect our overall user popu- lation and use of our SilverPlatter data- bases. A majority of our graduate stu- dents (73.7%) are in the social sciences, as are 30.8 percent of undergraduates who have declared majors (almost half of our undergraduates are undecided or not in a program). Overall, 44 percent of the students are in the social sciences. This is Although a low skill level was exhibited by 20.9 percent of those who had received instruction and 27.3 percent of those who had not, a moderate level of skill was used by 34.1 percent of those having received instruction and 28.6 percent of those who had not. less than in our study population, but a large percentage of those who are unde- clared are taking courses in the social sci- ences. The survey population includes a higher percentage of females than does . our campus population, where 47.9 per- cent of undergraduates and 58.5 percent of graduate students are female. This rep- resentation, interestingly, is at odds with research reviewed by Elizabeth Cardman in her article on computer use and gen- . der. 6 Statistics on overall database use for the period of the study (February through May 1994) show that use of ERIC and PsycLIT closely reflects the survey popu- tween eighteen and twenty-five years old, over 22 percent were between twenty-six and thirty-four, and almost 18 percent were over thirty-five. Most of those par- ticipating in the survey were graduate students (53.6%) or undergraduate stu- dents (40.9%). A huge majority (over 96%) indicated that they had some computer experience outside the library. In addi- tion, in keeping with previous surveys, over 78 percent indicated that they thought the results of their search were valuable (with another 18 percent indi- cating that they did not yet know). Al- though we did not keep an exact count, very few of the people approached re- fused to complete the survey. Results The authors coded and analyzed re- sponses to the survey through the use of the statistical package SPSS. We created a measure of the level of success of search- ing (in this case determined by the suc- cessful use of Boolean operators) by col- lapsing all the possible combinations of use of Boolean operators into three dis- tinct categories. The first category-high skill level-included those persons who used three or more Boolean operators in their search; the second category-mod- erate skill level-included all those who used two Boolean operators; and the third category-low skill level-included all those who used one operator. We treated persons who used no operators or who only used incorrect operators separately in the analysis. 72 College & Research Libraries The authors then performed cross- tabulations on all the important mea- sures. We compared "ever-trained" with "never-trained" searchers, and combined all the measures pertaining to any sort of end-user training into one category. The authors treated respondents who indi- cated they had never had any kind of training or class instruction on CD-ROM searching as another category. Although a low skill level was exhibited by 20.9 per- cent of those who had received instruc- tion and 27.3 percent of those who had not, a moderate level of skill was used by 34.1 percent of those having received in- struction and 28.6 percent of those who had not. Furthermore, almost twice as many of those who had prior instruction used the high level of search skill as de- fined in our study (25.4 %), compared to those who received no instruction (14.2 %) (see table 2 and figure 1). To study the relationship between the two variables of class instruction and skill level indicated by the search strat- January 1996 egy, we first examined the various row percentages in the cross-tabulations (the percent at each skill level divided be- tween those who had had instruction and those who had not). We then con- ducted a chi-square test to see if there was a statistically significant relationship be- \ tween the two variables. The chi-square for the measure of class instruction by the measure of search skill-level cross-tabu- lation resulted in a value of 22.4 (df=3, p<.OS), indicating a significant relation- ship. Cramer's V was then used to deter- mine the strength of the relationship. The value from this test "ranges from 0 to + 1; the larger the value of V, the greater is the association in the variables." 7 The mea- sure of the association was 0.182, indicat- ing a weak association. We next consid- ered each type of instruction as an indi- vidual measure and performed cross- tabulations. These cross-tabulations showed no statistically significant rela- tionships between type of training and skill level. FIGURE 1 Relationship between Training and Skill Level 35%~----------------------------------------------, Qi 30% > Q) _J ~ 25% [/) 0 c 20% Q) u I.. Q) a.. 15% ........................................................................... ........................... ......................................................... ························· 10%~----------------.----------.-----------,-----J None or Incorrect Low Moderate High Skill Level 1--- Some Training -+-No Training CD-ROM Instruction 73 TABLE3 User Groups by Skill Level cells did not contain at least five items, a requirement for a statistically valid chi-square analysis, the results were re- vealing (see table 3). Faculty do the high- est percentage of searches using incor- rect or no operators. Staff and graduate students do the most None or Incorrect Use Low Skill Level Moderate Skill Level High Skill Level Faculty 10 . 2 45 .5% 9.0% Graduate 77 77 21.6% 21.6% Undergrad. 77 84 28.3% 30.9% Staff 4 2 26.7% 13.3% Chi-square 25.5 (df=9, p<.05) The authors also performed cross- tabulations that did not involve instruc- tion. We compared user group with skill level, combining the University at Albany and non-University at Albany respon- dents in each category. Even though five 6 27.3% 4 18.2% 121 34.0% 81 22.8% 76 35 27.9% 12.9% searches indicating a 5 4 moderate skill level, 33.3% 26.7% whereas staff mem- bers perform the most searches using a high skill level. Undergraduate students hover at about 30 percent for the three lower skill levels, but drop to 12.9 percent for high-skill level searches (see figure 2). The chi- square for the measure of user group by the measure of skill-level cross-tabulation FIGURE2 User Group Percentages vs. Skill Level 50%~----------------------------------------------, 45% a. 6 40% L.. '-' L.. 35% Ql If) ::::l 30% c :c :!:: 25% 3: c 20% Ql C) ~ 15% 0... 10% 5%~----~-------------.------------~------------~----~ None or Incorrect Low Moderate High Skill Level ---Faculty --+-- Graduate ---*- Undergrad -a- Staff 74 College & Research Libraries January 1996 FIGURE3 Department Percentages vs. Skill Level 55%~-------------------------------------------------~ SO%+-----~--------------------------------------~ ~ 45%+-------~~------------------------------------------~ E ~ 40%+---------~~----------------------------------------~ 0 g. 3 5% +-------------~--------------------------------------~ 0 .~ 30% +---------~:-------::::;;.,....:::..---~~--------; £ ~25%+------------=~~~------------~~------~ ~ 20%+-----------------------~~~~~--------------~----~ 0 L.. &. 1 5% +---------------------------------------"'~~------------~ 10%+---------------------------------------~~----~ 5%~----~-------------.------------~------------.-----~ None or Incorrect Low Moderate High Skill Level ---Science -+- Social/Bus --*- Humanities -a- Area Studies resulted in a value of 25.5 (df=9, p<.OS), indicating a significant relationship. Cramer's V was 0.113, signifying a weak association. We also compared department affilia- tion/ major with skill level. In order to obtain an adequate count in each cat- using no operators or using operators in- correctly, and had the fewest high skill- level searches (see figure 3). Although this might suggest that these users would ben- efit most from instruction, it should be noted that, for the humanities, our only TABLE4 egory, we collapsed departments into four broad areas: sciences, social sciences/busi- ness, humanities, and area studies. This com- bination of depart- ments still resulted in two cells with fewer than five items, so the conclusions should be viewed with caution (see table 4). Searchers affiliated with-depart- ments in the humani- ties and area studies did the most searches Department/Major by Skill Level Science Social Sci./ Busin ess Humanities None or Incorrect Use 13 2 1.7 % 108 2 1.6% 34 53. 1% Area Studies 11 47 .8% Low Skill Level 14 23.3% 123 24 .6% 14 2 1.9% 6 26.1% Chi-square 39.5 (df=9, p<.05) Moderate Skill Level 20 33.3% 163 32.7 % II 17.2% 4 17.4% High Skill Level 13 21.7% 105 2 1.0% 5 7.8% 2 8. 7% SilverPlatter database is the MLA Bibli- ography. Although not all humanities-af- filiated searchers would necessarily have used MLA, those that did were often searching the title of a literary work or an author's name-searches that do not always require operators to be good searches. The chi-square for the measure of department affiliation by the measure of skill-level cross-tabulation resulted in a value of 39.5 (df=9, p<.05), indicating a significant relationship. Cramer's V was 0.247 in the range from 0 to 1. Neither age nor sex showed a relation- ship with search skill. However, when the age categories zero to seventeen and eigh- teen to twenty-five were combined to pro- vide sufficient frequencies in each cell, we did note that there was a larger differen- tiation in the skill levels of trained search- ers in the twenty-six to thirty-four age range. Of those searchers who used no operators or who used them incorrectly, 17.4 percent fell into this age range. Of those who had a low skill level, 23.5 per- cent were between twenty-six and thirty- four. Of those with a moderate skill level, 21 percent were in this age range, and 28.4 percent of those with a high skill level were between twenty-six and thirty-four. The percentages in the zero to twenty-five and the thirty-five-plus age ranges stayed more constant across skill levels. In asking survey respondents how many times they had searched CD-ROMs in CD-ROM Instruction 75 through the most experienced searchers. In addition, whereas 19.7 percent of the most experienced searchers did the best searches, so did 18.8 percent of the new searchers (see table 5). Similarly, experi- ence in online catalog searching, use of interactive learning software, online da- tabase searching, and e-mail experience did not relate to search competency. Conclusions Our hypothesis-that patrons who had attended searching classes or who had been directly assisted by librarians (or trained library assistants) would be far more sophisticated searchers-seems to be correct based on one test, but addi- tional research should be undertaken be- fore we can state with resounding affir- mation that classes help students. Several of the analyses of the results of this study showed no statistically significant rela- tionships between the skill level and other variables. The chi-square test did demon- strate that there is a positive association between class instruction and more skill- ful searching; however, based on our Cramer's V test result, class instruction makes a less pronounced difference than we expected. None of the other individual instruction analyses detected statistically significant associations, in part because insufficient data existed. In retrospect, we realize that the last section of the ques- TABLES the past, we hypoth- esized that those who Prior Use of CD-ROMs by Skill Level were frequent searchers None or Low Moderate High would have the best search strategies. Al- Incorrect Use Skill Level Skill Level Skill Level though 31.3 percent of Never 15 14 10 9 those who had never 31.3% 29.2% 20.8% 18.8% searched before did not 1 time 10 14 15 13 use any operators or 19.2% 26.9% 28 .8% 25.0% used them incorrectly, 2-5 times 40 43 52 24 there was not a clear 25.2% 27.0% 32.7% 15.1 % trend in increased op- 6 or more 107 95 132 82 era tor use among those times 25.7% 22.8% 31.7% 19 .7 % who had never searched 76 College & Research Libraries tionnaire was too complex to net sufficient responses in any specific category to al- low for a useful analysis. If we were to repeat this study, we would only differ- entiate hands-on versus demonstration classes during the current semester or an earlier semester. The first question librarians need to ask after reviewing this research is: Are pub- lic service librarians spending their time wisely teaching and assisting end-users? The answer, we believe, is yes. This study involved only the use of Boolean opera- tors, field searching, and referrals to pre- vious search statements, and not other search skills. It is possible that Boolean operators are the most "foreign" concepts; the least easily grasped of any that librar- ians try to impart.8 Possibly the amount of information presented in instruction sessions overwhelms patrons. When teaching the local setup, the concept of a database, the variety of databases avail- able, and specific searching techniques, January 1996 much is bound to be lost. In addition, if students learn little but become comfort- able with CD-ROM technology or with the library in general, our work is clearly purposeful. Thus, although we do not believe that the results of our research refute the im- portance of CD-ROM instruction and as- sistance, we do believe the results make a persuasive case for more research into the issues we've explored. If there isn't a strong association between what we are teaching and what end users are learn- ing, we need to discover why. We need to know how much end users are motivated to learn search skills (perhaps their high satisfaction level with any search, how- ever poor, prevents them from appreciat- ing the value of instruction) or if the soft- ware itself somehow leads end users away from employing more sophisticated search skills. Most important, we need to discover if our teaching and assistance techniques are inappropriate for the technology. Notes 1. Michael Culbertson, "Analysis of Searches by End-Users of Science and Engineering CD- ROM Databases in an Academic Library," CD-ROM Professional 5 (Mar. 1992): 76-79; Susan K. Charles and Katherine E. Clark, "Enhancing CD-ROM Searches with Online Updates: An Ex- amination of End-User Needs, Strategies, and Problems," College & Research Libraries 51 (July 1990): 321-28; Janet L. Nelson, "An Analysis of Transaction Logs to Evaluate the Educational Needs of End Users," Medical Reference Services Quarterly 11 (winter 1992): 11-21. 2. Domenica M. Barbuto and Elena E. Cevallos, "End-User Searching: Program Review and Future Prospects," RQ 31 (winter 1991): 214-27. 3. Rebecca Bostian and Anne Robbins, "Effective Instruction for Searching CD-ROM Indexes," Laserdisk Professional 3 (Jan. 1990): 14-17. 4. Dorothy F. Davis, "A Comparison of Bibliographic Instruction Methods on CD-ROM Da- tabases," Research Strategies 11 (summer 1993): 156-63. 5. Linda Stewart and Jan Olsen, "Compact Disk Databases: Are They Good for Users?" On- line 12 (May 1988): 48-52. 6. Elizabeth R. Cardman, "The Gender Gap in Computer Use: Implications for Bibliographic Instruction," Research Strategies 8 (summer 1990): 116-28. 7. Peter Hernon, Statistics for Library Decision Making: A Handbook (Norwood, NJ: Ablex, 1989), 104. 8. Rasmussen lists "user-unfriendliness" as one of the problems identified with online systems using Boolean retrieval. Edie M. Rasmussen, "Current Research in Information Science and its Impact Upon Librarianship," Organizing a Research Agenda: Information Studies for the 1990s (Halifax, Nova Scotia: Dalhousie Univ. School of Library and Information Studies, 1990), 34.