College and Research Libraries Online Searching Styles: An Exploratory Study Stephen P. Harter An exploratory study of self-reported behaviors and attitudes toward online searching was con- ducted through a questionnaire survey of Florida searchers. The study tested several hypothe- ses relating specific behaviors and attitudes to experience and institutional setting. Among other conclusions it was found that there are wide individual differences among online search- ers in attitudes as well as behaviors, that online searchers from academic institutions differ in several important ways from their counterparts in other library settings, and that experience appears to be positively related to a flexible, trial-and-error approach to online searching. s online searching has devel- oped into an important refer- ence tool in academic libraries, a substantial research effort has been undertaken into the performance of the online search specialist. This research has produced an impressive body of evi- dence suggesting that many searchers perform in unexpectedly simplistic ways. The results indicate, for example, that "re- markably little attention" is paid by searohers to file differences and thC!-t inap- propriate vocabulary selection and strate- gies often result; that most searchers plan their search strategies logically and follow them consistently but are sometimes too uncritical about them; 1 that several system commands are never used and that most -commands are issued in fewer than half of the searches conducted/ that searching and printing commands account for two- thirds of the commands issued; 3 that the interactive capabilities of the online sys- tems are comparatively little used; 4 and that between 46 percent and 78 percent of searches conducted are not modified at all after the initial formulation is put to the system. 5 These and similar findings have stimu- lated the study of variables affecting search behavior. In a large scale investiga- tion of online search behavior in con- trolled experimental environments, Carol Fenichel examined the effect of experience on performance, concluding that the mod- erately experienced subjects with (ERIC) database experience performed the most brief, cost-effective searches. 6 However, novice searchers performed surprisingly well in comparison to the other experience groups. Fenichel remarked on the striking simplicity of even the experienced sub- jects' searches, and found "enormous variability in searching behavior ... [even] in searches of persons in the same • n7 expenence groups .... In another controlled experimental study, conducted with Medline searchers, Wanger, McDonald, and Berger examined several variables for possible effect on search performance, including type of training (formal, informal), type of organi- zation (academic, health care, and re- search institutions), and National Library of Medicine searching experience. 8 The methodology and findings of this study are too extensive to summarize here. Among these results, however, it was found that type of training had no rela- tionship to any of the measures of search performance studied. Type of organiza- tion was found to affect the time taken to conduct a search but was not related to ei- ther recall or precision. Finally, the Stephen P. Harter is associate professor, School of Library and Infonnation Science, Indiana University, Bloomington, Indiana 47405. 249 250 College & Research Libraries amount of NLM searching experience was found to be related to search preparation time but again n9t to recall or precision. The authors concluded that search perfor- mance varied most significantly with the nature and difficulty of the search request. It was also noted that many searchers failed to browse titles or descriptors of re- trieved citations artd generally did not be- gin to make full use of the interactive fea- tures of the se_arch systems. Surprised by this result, the authors termed this mode of searching "fast batch. " 9 In a study similar in many respects to that of Fenichel, Howard examined train- ing and experience as independent vari- ables affecting search performance and found that type of training was not related to any of seven output variables. The ef- fect of experience on search results was mixed; while experienced searchers were found to achieve the most cost-effective searches, the novice searchers were found to achieve the highest precision. Howard commented that the differences in the val- ues between the experience groups, though statistically significant, were not great. 10 Lowry11 reported achieving im- pressive gains in the initial performance of novice searchers but also reported wide individual variability within the same ex- perience groups. These findings support the notion that individual differences among online searchers may be more important, in the long run, than individual training or expe- rience. Several conjectures might be of- fered in explanation of these differences. Perhaps institutional setting significantly affects behavior, in particular, regarding one's attitude toward cost <,ind subsequent behavior. Some searchers may lack full knowledge of system capabilities or have a naive view of the effectiveness of using a controlled vocabulary. ·It is possible that some searchers do not feel a responsibility to evaluate their work or that system com- mands are too complex for many search- ers to master. These ideas are consistent with the finding of simplistic, perhaps overly simplistic, searches, as reported in previous research. In addition to the hypotheses suggested in the previous paragraph, we might con- July 1984 sider the following: that controlled experi- ments affect search behavior in important ways; that searchers differ significantly in their attitudes toward their craft or differ in the specific problem-solving, concep- tual skills that are required. Finally, we should observe that most of the experi- mental work cited, except for Fenichel' s study, did not give searchers instructions about the level of recall or precision that was desirable, contrary to what one would expect to result from a real-world refer- ence interview. Perhaps we should not be surprised at finding a wide variability in the search performance among groups' of homogeneous searchers without such re- trieval goals. It should be noted that we are not here equating simple searches with bad searches. Indeed, it may be that for many search questions a fast-batch search is more cost-effective than an interactive one. This question has not been seriously explored in the online searching litera- ture, but it is an interesting problem for fu- ture research, particularly because of the conventional assumption made by many writers that effective searches must be in- teractive, heuristic exercises in problem solving. RESEARCH PROBLEM The present study attempted to explore the following questions: 1. Can differences among searchers be determined with respect to attitudes re- garding online searching? For specific search behaviors? How are these attitudes and behaviors related to experience and institutional setting? One might suspect, for example, that academic librarians might differ significantly from many spe- cial librarians about attitudes toward cost, and that searchers may change certain at- titudes toward their craft as they gain in experience. The following attitudes were investi- gated: a. the importance of cost as the princi- pal reason for terminating a search b. responsibility of an analyst to evalu- ate the results of searches c. the nature of online searching as a . .. -~ I f problem-solving rather than an al- gorithmic process d. the importance of the ability to be flexible in an online search e. the value of trial-and-error methods f. the efficacy of searching using a con- trolled vocabulary 2. A second purpose of the study was to examine the frequency of use of certain re- ported search behaviors and to test for re- lationships between these behaviors and experience and institutional setting . Be- haviors inves.tigated include a. use of truncation b. review of subject-related terms on- line c. review of alphabetically related terms online d. free-text searching e. logging off to consider one's next actions without accumulating costs f. review of titles from retrieved records online g. review of descriptors from retrieved records online Specifically, the following hypotheses were tested for each attitude and behav- ior: i. There is no significant difference be- tween academic and special librarians in their attitudes toward online searching or reported online searching behaviors. ii. There is no significant difference be- tween librarians with different levels of experience in their attitudes toward online searching or reported online searching be- haviors . METHODOLOGY Since a primary purpose of the study was to investigate attitudes, a survey ap- proach was considered appropriate for an exploratory study. To the extent that data were gathered concerning searcher behav- iors, one must be cautious in drawing con- clusions from questionnaire results-self- reported behaviors obviously may not coincide with actual behaviors. However, we have noted our suspicion that con- trolled experimental studies may have sig- nificantly affected search behavior in Online Searching Styles 251 some past research. It is at least possible that a carefully prepared questionnaire concerning behaviors has the potential of resulting in data with a higher degree of validity than data resulting from con- trolled experiments. It is also possible that self-reported search behaviors may differ in significant ways from actual behaviors, and that, for a · variety of reasons, questionnaire-generated data distort "real world" search behaviors as much or more than do controlled · experiments. While this possibility was acknowledged, it was assumed that such was not the case. A survey approach to the problem was therefore taken, and a questionnaire was prepared and pretested. A detailed description of the methodol- ogy employed in the study is presented by this author elsewhere, 12 and only a sum- mary is provided here . The sample stud- ied consisted of online searchers in the state of Florida, a group from which find- ings can probably be generalized beyond the confines of the state. Florida searchers were taken to be those persons found in the Directory of Florida Online Searchers, 13 augmented by the list of members of the Florida Online Searchers Group.* A cover letter, questionnaire, and a self- addressed, stamped return envelope, were mailed to 121 subjects on November 18, 1982. Ofthis number, seventy-two us- able questionnaires were returned in time to be included in the data analysis, are- sponse rate of 59.5 percent. The percent- age of practicing Florida online searchers responding to the survey is probably much higher than 59.5 percent, however, since several additional respondents com- mented that they were not presently on- line searchers, and other questionnaires were returned by the employers of search- ers who had left their former positions. The variables of interest-specific atti- tudes and behaviors-are operationally defined by the specific questions appear- ing in the questionnaire (see "Results"). Hypotheses were tested using the chi- square test of statistical significance, uti- lizing A-Stat, a statistical software pack- *Thanks are due to L. Susan Hayes for making this list available . 252 College & Research Libraries July 1984 age running on an Apple II Plus microcomputer. complete range of frequencies was ob- tained, from "in none of my searches" to "in all or nearly all my searches." What is apparently a common behavior for one searcher is never done by another; and this was found to be true for every behav- ior examined. RESULTS Behaviors and Attitudes Table 1 provides the questions asked re- garding the use of system features and a summary of responses. It can be seen that online search specialists vary widely in their reported online search behaviors. For nearly every behavior investigated, a The overriding observation to be made regarding table 1, then, is the great vari- ability among the respondents, especially for the questions addressing free-text TABLE 1 BEHAVIORS OF ONLINE SEARCHERS: OVERALL FREQUENCY OF RESPONSES Number of Respondents 5 54 9 2 2 22 36 10 1 3 1 27 14 19 11 3 12 25 30 1 10 9 22 30 Frequency of logoff. In what proportion of your online searches would you say you go offline (that is, log off) during the search to review system feedback and to decide whether (and how) to revise your initial formulation? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches Frequency of review of alphabetically related terms online. In what p~oportion of your online searches would you say you review alphabetically related terms on1ine one or more times during the course of the search? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches Frequency offree-text searchzng. Consider those online searches that you conduct in data- bases for which there is a controlled vocabulary (thesaurus). In what proportion of your searches carried out in such databases would you say you search in free-text or natural language fields (for example, title or abstract)? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches Searcher type. The behavior that best describes your usual style of online searching is a. I prepare the search formulation in advance. I log on and enter the formulation to the system. I print as many citations (online or offline) as the end user can afford. I log off. b. I prepare an initial formulation in advance, including "what if" plans for broadening or narrowing the search if necessary. I log on and enter the formulations to the system. Based on the size of the resulting final set, I revise (broaden or narrow) my initial strat- egy with the appropriate what-if plans. I enter the revised formulation to the system and print (online or offline) as many citations as the end user can afford. I log off. c. Same as b, except that I will also assess the relevancy of mr intermediate and final sets by browsing a sample of titles . I may need to make severa adjustments to the original strategy, based on my assessment of the relevancy and size of the intermediate sets, before printing the results in an end-user format . d. Same as c, except that I also typically browse the descriptors associated with relevant citations, to identify additional terms to be added to my final search formulation. Frequency of review of titles online. In what proportion of your online searches would you say you review samples of retrieved titles to make preliminary checks for relevance? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches f t Number of Respondents 2 30 17 10 13 15 34 9 9 2 0 9 17 19 27 TABLE 1 (CONTINUED) Online Searching Styles 253 Frequency of review of descriptors online. In what proportion of your online searches would you say you review samples of descriptors from retrieved citations to suggest ways of ex- tending the search? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches . d. in the majority of my searches e. in all or nearly all of my searches Frequency of use ot online thesaurus. In what proportion of your online searches would you say you review subject-related terms online? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches Frequency of use of truncation. In what proportion of your online searches would you say you use truncation? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches searching, review of titles and descriptors online, the use of online thesauri, and truncation. In particular, there appears to be a significant number of searchers that might be called fast batch, characterized by little or no interaction with the system, perhaps not even printing a sample of ti- tles to assess the relevance of intermediate output. Attitudes and behaviors of the fast-batch searcher are exglored in depth by this author elsewhere. 4 Table 2 lists the attitudes investigated TABLE 2 ATTITUDES OF ONLINE SEARCHERS: OVERALL FREQUENCY OF RESPONSES Number of Respondents 3 24 33 3 3 2 7 59 0 5 14 41 5 Attitude toward online searching as an activity. In its overall characteristics, which of the fol- lowing activities would you say online searching is most like? a. looking up several numbers in a telephone oook b. solving a crossword puzzle . c. doing scientific research d. using an automated bank teller to conduct a transaction e. doing an arithmetic computation on a calculator Attitude toward batch searching. A batch-processing information retrieval system is a com- puter system in which the search request is formufated in one step and put to the system in a separate step. The results are then presented to the searcher or requestor, perhaps days later . How would you say an online system compares to a batch-processing system in terms of search effectiveness? a. · batch system is superior b. no important difference c. online system is superior Attitude toward descriptors. Consider those online searches that you conduct in databases that use a controlled vocabulary (thesaurus). What proportion of your searches carried out in such databases would you say have succeeded in the retrieval of all, or nearly all, of the relevant documents in the database? a. in none of my searches b. in a small proportion of my searches c. in perhaps half of my searches d. in the majority of my searches e. in all or nearly all of my searches 254 Number of Respondents 8 26 33 2 12 25 23 8 1 9 28 33 2 22 28 19 4 28 26 11 College & Research Libraries . July 1984 TABLE 2 (CONTINUED) Attitude toward flexibility. Please assess the extent to which you agree or disagree with the following statement: "I dislike dividing a search up into concerts. I keep the ultimate question in mind at all times and juggle ideas as I see the retrieva , often changinp- 'concept' order. I think when you set terms down under .concepts they tend to become set in cement' and you tend to become too rigid in your searching. You have to be ready to adapt at every line of print you receive back. That's the challenge and the fun of it!"* a. agree completely b. mostly a~ee c. mostly disagree d. disagree completely Attitude toward cost. Please assess the extent to which you agree or disagree with the follow- ing statement: Among the several variables that might affect my decisio·n to terminate a search and print the results, the most important factor is cost. a. agree completely b. mostly a~ee c. mostly disagree d. disagree completely Attitude toward evaluation. Please assess the extent to which you agree with the following statement: It isn't part of my job as an online searcher to evaluate the results of a search. The end user has specified his or her search request and it is my responsibility to execute it as requested. By retrieving the citations containing the terms specified, I have fulfilled my responsibility. a. agree completely b. mostly a~ee c. mostly disagree d. disagree completely Attitude toward descriptors. Please assess the extent to which you agree or disagree with the following statement: In databases with a controlled vocabulary, there is no particular point in browsing among retrieved titles because it wastes time and money and because a carefully planned search. using descriptors will always produce acceptable results . a. agree completely b. mostly a~ee c. mostly disagree d. disagree completely Attitude toward trial and error. Please assess the extent to which you agree or disagree with the following statement: · In online searching, simplicity is a virtue. I don't believe in experimenting once I have thought the searcn through and entered the strategy into the system. Trial and error- "fooling around"-online is not only expensive, but it also reflects fuzzy thinking and poor search preparation. a. agree completely b. mostly a~ee c. mostly disagree d. disagree completely *Quoted verba tin{ from Judith Wanger, Dennis McDonald, and Mary C. Berger, Evaluation of the On -Line Process (Bethesda, Md. : Na- tional Library of Medicine, 1980), p .IV- 17. and a summary of responses. In an effort to determine how online searching is re- garded, searchers were asked to select the activity most like it. Some of the options included were deliberately algorithmic rather than heuristic in nature, to deter- mine if some searchers look at their craft as a mechanical, almost deterministic pro- cess. Perhaps surprisingly, several of the respondents apparently do. Looking up numbers in a telephone book, using an au- tomated bank teller, and using a calculator were selected as the best responses by a total of nine searchers. Even solving a crossword puzzle may be viewed as a questionable response, since a crossword j ~ 1 puzzle has one and only one acceptable solution-surely not ever the case in any online search. Nine respondents did not feel that on- line search systems were superior in any way to batch systems, again suggesting an attitude not ·supported by the online ~earching literature, which tends to stress the value of the interactive nature of on- line searching. Judging by the results of the two ques- tions dealing with attitudes toward con- trolled vocabularies, many searchers have a great deal of confidence in their use, con- fidence that is perhaps not justified in the light of numerous research studies show- ing widespread inconsistenc~ among in-: dexers (e.g., Tarr and Borko1 ). Forty-one of sixty-five respondents indicated that the majority of their searches conducted in files using a controlled vocabulary re- sulted in the retrieval of all, or nearly all, the relevant documents in the database. On the other hand, several respondents refused to select an answer for this ques- tion, writing in the margin a comment such as ''How could I possibly know?'' In- deed, one cannot in general know the per- centage of Televant documents retrieved in a search, except insofar as an act of faith supports such belief. Respondents were almost evenly di- vided in their attitudes toward the impor- tance of being flexible when online. Some searchers believe in being extremely flexi- ble, ready to react to every line of print re- ceived. Nearly an equal number tend to rely on careful preplanning as a substitute for reacting to the output received. The importance of the interactive nature of the online process is again minimized for searchers with this latter attitude. Most searchers indicated that they take their responsibilities toward evaluation of results seriously, although ten of seventy- one respondents indicated a tendency to consider a search to be complete if a re- quest is executed exactly as requested by an end user, suggesting a literal interpre- tation of a search request as contrasted with an attempt to satisfy an underlying information need. The possibility that a stated request may not express well what is really needed by a client does not seem Online Searching Styles 255 to be fully appreciated by these respon- dents. Alternatively, perhaps they feel that it is not appropriate for them to go be- yond the literal statement of a search re- quest. Finally, the value of trial-and-error methods, exemplified by interactive, problem-solving search behavior, was viewed in different ways by the respon- dents. Nearly half indicated their view that trial-and-error methods reflected fuzzy thinking and poor search prepara- tion, equating trial-and-error methods with ''fooling around'' online. Again, the potential value of interactive methods in an online search is negated by this atti- tude. Differences between Academic and Special Librarians Of the seventy-two respondents, forty were associated with academic libraries and thirty-two with special libraries, with a few ''free-lance'' search specialists being classified with the special library group. In an effort to determine whether there is any difference between attitudes of aca- demic and special librarians, chi-square tests were performed. Table 3 summarizes these results. Perhaps not surprisingly, there was found to be a highly significant difference between academic and special librarians in their attitude toward cost (p= .006). Academic librarians tend to be much more "cost conscious," tending to consider cost to be the most important fac- tor in deciding when to terminate a search. There was also a significant differ- TABLE 3 RELATIONSHIP BETWEEN ATTITUDES AND TYPE OF LIBRARY Variable Attitude toward online searching as an activity Attitude toward batch searching Attitude toward descriptors Question 1 Question 2 Attitude toward flexibility Attitude toward cost Attitude toward evaluation Attitude toward value of trial-and-error methods Chi-square Probability .7 .02 .07 .35 .05 .006 .7 .25 256 College & Research Libraries ence between academic and special librari- ans in their attitude toward batch- processing offline retrieval systems as compared to online systems. Significantly more academic librarians indicated their belief that batch systems were either supe- rior to, or no different from, online sys- tems in search effectiveness (p= .02). Academic librarians also tended to dis- like the idea of being flexible in one's be- havior at the terminal (p= .05) and tended to have more confidence in controlled vo- cabularies than did special librarians (p= .07) . No other statistically significant attitudes were found. The use of other system features was also analyzed for type of library. Table 4 reports these findings . Again, there were differences found between academic and special librarians. Academic librarians re- ported reviewing subject-related terms and alphabetically related terms online significantly less frequently than did spe- cial librarians, no doubt as a function of their differing attitudes toward the impor- tance of cost. No other differences in use of system features were found. TABLE4 RELATIONSHIP BETWEEN USE OF SYSTEM FEATURES AND TYPE OF LIBRARY System Features Frequency of log-off Frequency of review of alphabetically related terms online Frequency of free-te xt searching Frequency of review of titles on1ine Frequency of review of descriptors oriline Frequency of use of online thesaurus Frequency of use of truncation Effects of Experience Ch i-square Probability .6 .008 .8 .3 .6 .01 .3 It was noted earlier that previous stud- ies have suggested that searching experi- ence is not strongly related to overall search performance. Because of these findings, it was speculated that experi- ence would not prove to be an important factor for either attitudes or behaviors. This hypothesis was at least partially sub- stantiated by the results of the present July 1984 study. For purposes of statistical analysis, respondents were divided into three expe- rience classes : two or fewer years of searching experience-twenty-four re- spondents; three or four years of experience-twenty-three respondents; more than four years of searching experience-twenty-three respondents . Frequency of searching was not consid- ered . The relationship between attitudes and experience is summarized in table 5. Only one attitude was found to be significantly related to years of experience at the .05 level-the willingness of a searcher to dis- play flexibility at the terminal. The more experienced a searcher was, the more he or she tended to dislike dividing a search up into concepts, to become too rigid in search planning. Apparently, as searchers gain in experience, they become more willing and able to "play it by ear"-to adapt to changing conditions and results, to interact with the system. Presumably, this change in attitude may be a function of gains in self-confidence as well as an in- creased appreciation for a need to be inter- active. This result is similar to that found for the question dealing with the value of trial-and-error methods. As searchers gain in experience, they tend to disagree with the questionnaire statement that II trial-and-error methods . . . reflect fuzzy thinking and poor search preparation." Less experienced searchers tended to agree with this statement (p= .08). The relationships between the use of certain system features and experience are TABLE 5 RELATIONSHIP BETWEEN ATTITUDES AND EXPERIENCE Variable Chi-square Probability Attitude toward online searching as an activity Attitude toward batch searching Attitude toward descriptors Question 1 Question 2 Attitude toward flexibility Attitude toward cost Attitude toward evaluation Attitude toward value of trial-and-error methods .7 .5 .2 .9 .02 .25 .65 .08 summarized in table 6. No significant rela- tionships were found. Thus habits of sys- tem use seem to be well established early in a searcher's career, a finding consistent with the earlier research cited above. TABLE6 RELATIONSHIP BETWEEN USE OF SYSTEM FEATURES AND EXPERIENCE System Features Frequency of log-off Frequency of review of alphabetically related terms online Frequency of free-text searching Frequency of review of titles online Frequency of review of descriptors online Frequency of use of truncation Frequency of use of online thesaurus CONCLUSIONS Chi-square Probability .4 .4 .5 .5 .2 .9 .7 This study has confirmed several rela- tionships between certain attitudes and behaviors and the number of. years of ex- perience and the type of library with which the searcher is associated. Online searchers from academic institutions are found to have a more cost-conscious ap- proach to online searching, and to have more faith in the use of controlled vocabu- laries than their counterparts in special li- braries. Experience was found to have no relation to the reported use of system fea- tures investigated. However, experience in online searching does seem to be posi- tively related to a flexible, trial-and-error approach to online searching. We must be cautious in coming to final Online Searching Styles 257 conclusions regarding the findings of this study. The validity of results obtained through a questionnaire approach can be challenged since reported behaviors need have no particular relationship to actual behaviors. However, we have suggested that controlled experimental studies of the online process also may have problems with validity. The results of this study suggest that with experience, perhaps be- cause of increased knowledge and confi- dence in their abilities, searchers tend to soften their attitudes, becoming more flex- ible and more willing to use trial-and-error methods. There also seems to be a changed view toward the utility of search- ing using controlled vocabularies. Per- haps this change reflects a more realistic view of the relative strengths and weak- nesses of free-text and controlled- vocabulary approaches to searching. There are implications in these findings for the education and training of search - analysts, as well as questions for further research. Should different approaches to searching be taught for different institu- tional settings? Is a trial-and-error problem-solving approach to online searching actually superior to fast-batch methods? If so, how can appropriate atti- tudes and skills best be taught to potential analysts? Among its most important results, this research has confirmed that there are ap- parently great individual differences among online searchers in attitudes as well as behaviors. A rich area for future re- search lies in the in-depth exploration of these differences and of their relationship to the effectiveness of retrieval. REFERENCES 1. Betty K. Oldroyd and Charles L. Citroen, "Study of Strategies Used in On-Line Searching," On- line Review 1: 295-310 (Dec. 1977) . 2. Oldrich Standera, "On-Line Retrieval Systems: Some Observations on the User/System Inter- face," in Information Revolution 12:38-40, proceedings of the twelfth ASIS annual meeting (Wash- ington, D.C.: American Society for Information Science, 1975). 3 . Raymond N. Brown and Ashok K. Agrawala, "On the Behavior of Users of the MEDLINE Sys- tem," in Changing Patterns in Information Retrieval, ed. by Carol Fenichel, tenth annual National Information Retrieval Colloquium, May 3-4, 1973 (Washington, D . C.: American Society for Infor- mation Science, 1974), p .36-38. . 4. F. Wilfrid Lancaster, " Evaluation of On-Line Searching in MEDLARS (AIM-TWX) by Biomedical Practitioners,'' University of Illinois Graduate School of Library Science Occasional Paper No. 101 258 College & Research Libraries July 1984 (Urbana, ill.: Univ. of illinois Graduate School of Library Science, 1972). 5. A. S. Polli-t, CANCERLINE Evaluation Project: Final Report (Leeds, England : Univ. of Leeds School of Medicine, Medical Library, 1977). 6. Carol H. Fenichel, "Online Information Retrieval: Identification of Measures that Discriminate Among Users with Different Levels and Types of Experience" (Ph.D. diss., Drexel Univ., 1979), p.xiv . 7. Ibid., p.xvi. 8. Judith Wanger, Dennis McDonald, and Mary C. Berger, Evaluation of the On-Line Process (Be- thesda, Md.: National Library of Medicine, 1980). 9. Ibid., p .IV-29. 10. Helen Howard, "Measures that Discriminate Among Online Searchers with Different Training and Experience," Online Review 6:324 (Aug. 1982). . 11 . Glenn R. Lowry, "Improving the Initial Performance of Novice Online Search Intermediaries," in proceedings of the forty-fifth ASIS annual meeting v .19 (Columbus, Ohio: American Society for Information Science, 1982), p .173-75. 12. Stephen P . Harter, ''The Online Information Specialist: Behaviors, Philosophies, and Attitudes,'' in The Online Age: Assessment/Directions, proceedings of the twelfth ASIS midyear meeting (Lex- ington, Ky.: American Society for Information Science, May 1983), p.201-12 (microfiche). 13. Pat Boody and Maureen Sf>rcoran, Directory of Florida Online Searchers (Gainesville, Fla .: Florida Online, 1982). 14. Harter, "The Online Information Specialist." 15. D . Tarr and H . Borko, "Factors Influencing Inter-Indexer Consistency," in proceedings of the thirty-seventh ASIS annual meeting, v .11 (Washington, D. C., American Society for Information Science, 1974), p .50-55. j