College and Research Libraries G. L. GARDINER The Empirical Study of Reference The first section of this essay reconstructs four empirical studies of reference, which then serve as a basis for the discussions which fol- low. Section II shows that the principal notions on which these studies are based are deficient. It is observed that their deficiencies do not permit reliable results. Section III discusses the conventional concept of reference, through which reference is perceived as per- formed only by reference librarians; it concludes that the empirical interpretation of the conventional concept is fruitless as a framework for the development of the empirical study of reference. Section IV considers reference outside of librarianship. It suggests that reference is a special case of problem solving. On the basis of this consideration, Section V proposes a new framework for the empirical study of ref- erence. SECTION I FOUR EMPIRICAL STUDIES of reference are briefly reconstructed in this section. Of the studies of reference with which the author is familiar, they appear to contain the most insight. They are presented in chronological order. By chance the first two studies were written by librarians; the second two, by non- librarians. The reconstructions serve as a basis for the discussions of the follow- ing sections. Cole understood reference to be the "queries which people bring to the desk of the reference librarian" and the li- brarian's response to the queries in terms of "materials best suited to the needs of the reader."1 Her objective was "to identify reference questions ... and 1 Dorothy Ethlyn Cole, "An Analysis of Adult Ref- erence Work in Libraries" (unpublished master's the- sis, Graduate Library School, University of Chicago, 1943 ), p. 1. Mr. Gardiner is Director of Libraries in Normal State University, Normal, Illinois. 130/ describe . . . the sources of reference in- formation"2 in public, academic, and speaiallibraries. The data of her study were the result of a questionnaire. For each reference, as interpreted under Cole's definition, the questionnaire requested the follow- ing returns: 1. What the patron wanted. 2. Exact statement of the question. 3. Occupation of the patron. 4 . Specific books and other tools used in finding the answer. 5. Other functions performed. 6. Was the question answered satisfactorily. If not, indicate rea- son. 3 Fourteen libraries reacted to the ques- tionnaire. Four of them were public, four were academic, and six were spe- cial libraries . The data were first manip- ulated for readings on reference ques- tions, then for readings on reference sources. 2 Ibid., p. 2. a Ibid., p. 24. The first set of manipulations includ- ed these. The returns of No. 2, above, were classified by subject. That is, each statement of a question was mapped into one of the divisions of the Decimal Classification. The subject of a question was then characterized as a function of its origin, i.e., public, academic, or spe- cial library. The returns of No. 3 were classed into such occupational categories as housewives, unskilled laborers, and skilled laborers. The returns of No. 2 were classified by time. The temporal classification consisted of the following historical periods: -499, 500-1899, 1900- 1940, 1941, 1942- . The subject of a question was characterized as a function of its time. The returns of No. 2 were classified by complexity. This classifica- tion, in outline, included: fact, how-to- do, supporting evidence, and general in- formation categories. The returns of No. 1 were classed by specificity. The re- turns of No. 1, that is, were compared with the respective returns of No. 2. If "what the patron wanted" was the same as "the exact statement of the question," then that question was classed as spe- cific; otherwise it was classed as non- specific. Specificity was then character- ized as a function of both origin and occupation. The second set of manipulations re- quired the following operations. The re- turns of No. 2 were classified by com- plexity. The returns of No. 4 were clas- sified by form, i.e., such as reference, non-reference, periodical, and docu- ment. The form of a source was then characterized as a function of the com- plexity of a question. Unlike Cole, Breed4 did not indicate his concept of reference. Since his study was largely a follow-up of Cole's study, it was entirely unnecessary that he should. Indeed, as the following recon- 4 Paul Francis Breed, "An Analysis of Reference Pro- cedures in a Large University Library" (unpublished master's thesis , Graduate Library School, University of Chicago, 1955). Empirical Study of Reference I 131 struction will show, it is reasonable to assume his entertainment of a concept of reference more similar to Cole's than to any alternative. Breed undertook the study of difficult reference questions in a large university library. To Breed a difficult reference ques- tion was one for which "the ordinary ref- erence sources failed to produce an an- swer."5 Breed purposed "to discover something about the persons asking the questions . . . the kinds of questions asked" and "to compare the extent to which certain kinds of knowledge ... were called upon by the reference li- brarian in the search process."6 Breed's knowledge typology included knowl- edge associated with a general liberal arts background, subject specialization, personal knowledge, knowledge · gained in the process of search, and library school and library experience. These knowledge types wer.e intended to ex- plain each step of the reference librar- ian's search. A reference step was de- fined as the reference librarian's consul- tation of any source or entry in a source. The data of Breed's study was there- sult of a questionnaire. The question- naire provided for the following returns on each reference: 1. A precise statement of the · information wanted . 2. The name and/ or occupation of the person or organization making the request, and the manner in which it was made (telephone, letter, or in person). 3. The time required to answer the question. 4. A step-by-step account of the method employed to deal with the question. 5. Whether the question was answered, par- tially answered, or not answered. 7 The reference staff of Harper Library, University of Chicago, acted upon the questionnaire. After difficult questions were isolated from other questions, the data were manipulated for readings on those questions and for readings on the 5 Ibid ., p. 19. s Ibid., p. 1-2. 1 Ibid. , p. 4. 132 I College & Research Libraries • March 1969 processes by which those questions were answered. For readings on the difficult questions the following manipulations were made. The returns of No. 1, above, were classed by subject, i.e., social, biological, or physical sciences, and the humanities. The returns of No. 2 were classified by origin, campus or non-campus. The questions falling into the former cate- gory, "were examined for deviations from a known subject specialization."8 The returns of No. 1 were classified as being specific or general. Generality was indicated if a question "called for broad treatment of a subject or . . . in- volved a process of selection and evalua- tion of material."9 There were very few general questions; they were not given further consideration in Breed's study. Those questions characterized by spec- ificity were dichotomously classed as bibliographical or factual. Bibliographi- cal questions were first classed by form , i.e. , questions about books, periodicals, learned society publications, or U.S. government documents, then classed by inaccuracies of personal or corporate authorship, title, and date of publication. The inaccuracy of a question was then characterized as a function of its form. Factual questions were classified into numerous categories, often individual categories. The second set of manipulations were undertaken for readings on the librar- ian's search process. The returns of No. 4 were categorized by knowledge, as indicated above. The subject of a ques- tion was then characterized as .a func- tion of the knowledge required to an- swer it. Final steps used in answering questions were isolated. A final step was defined as "the step which resulted in a solution."10 Such steps were not necessarily final · in the sense of last steps. The subject of a question was 8 Ibid ., p. 22. ~ Ib id ., p. 24. 10 Ibid. , p. 52. then characterized as a function of the knowledge utilized by the librarian in taking final steps. To the Hemers, reference consisted of (Call the problems within organizations or among groups of workers which require searches for information."11 Though they condemned .a study of ref- erence «which leads or permits the in- formation seeker ... to place the task of getting the information he needs in the hands of a person or group outside of himself" as "a study of a certain, nar- row type of information requirement,"1 2 they produced such a study. And, though their concept of reference differed from Cole's, the concept of reference which guided their study did not. Their ob- jective was to define "those information requirements which reference librarians and other information specialists are likely to be called upon to meet."13 The data of the Herners' study also resulted from a questionnaire. It re- quested the name of the organization re- ceiving the question and the question itself. Fourteen atomic energy research and reference organizations forwarded to the Herners all questions received from the fall of 1956 through the fall of 1957. The data which the Herners re- ceived were manipulated only for read- ings on the questions. After technical questions, those "in- volving one or more of the natural or engineering sciences,"14 were isolated; technical as well as non-technical ques- tions were separately classified by sub- ject. Non-technical questions were then disregarded. Technical questions, how- ever, were further classified by number of discrete concepts and the logical re- u Saul Herner and Mary Herner, " Determining R e- quirements for Atomic Energy Information from Ref- erence Questions," Preprints of Pap ers for the Inter- national Conference on Scientific Information, Wash- ington, D .C. , Novemb er 16-21 , 1958 (Washington: Na- tional Academy of Sciences-National Research Council, 1958 ), p. 172. 12 Ibid. 1 3 Ibid. 14 Ibid. , p. 173. lationships among the discrete concepts. A discrete concept was defined as a significant concept which could not be subdivided without changing its essen- tial meaning. For example, "the question 'Give me information on engineering in nuclear reactors,' was taken to contain two concepts, 'engineering' and 'nuclear reactors.' "15 Only questions containing two or more· discrete concepts were classed by logical relationships. The logical relationships were defined as log- ical sums, "where the requestor would settle for information about concept A or concept B"; logical products "where the requestor had to have information about concept A and concept B"; and logical differences, "where the requestor was interested in concept A, but not concept B."16 Carlson described reference as ''hu- man searching behavior."17 His study emerged from the problem of determin- ing an optimum search procedure of a very large file. Reference librarians were chosen as subjects for his study for they are "the most trained in search proce- dures."18 Like the Herners, though his concept of reference differed from Cole's, the concept of reference which guided his study did not. The objective of Carlson's study was to describe search procedures used by reference librarians in order: · 1. To illustrate that human search behav- ior can be precisely described . . . . 2. To improve the present search procedures used b.y humans .... 3. To develop new training procedures for librarians. 4. To make it possible to develop computer rou- tines that could assist the human in making complex searches of a file.1 9 Three persons served as subjects. 1s Ibid., p. 174. 16 Ibid., p. 175. 17 G. Carlson, S earch Strategy by R e ference Librarians ( Part 3 of the Final Report on the Organization of Large Files [Sherman Oaks, California: Advance Sys- tems , Hughes Dynamics, 1964] ), p. ii. 18 Ibid., p. 2. 19 Ibid ., p. 1. Empirical Study of Reference I 133 They were the reference librarians in a university medical library. The file searched was the library collection with which the subjects were familiar. "The basic procedure was to present a search problem to a reference librarian and then record in detail his search tech- niques.''20 Protocols, · verbatim records, and descriptions of all that subjects say and do during a study, were obtained from the observations of two persons. The observers recorded everything they could see or hear during the searches. Presumably the subjects were requested to "think aloud" during the searches. Af- ter several protocols were gathered, they were analyzed for consistent search rou- tines. The resultant routines were then tested against another protocol. The process was repeated. The product of these operations was a flow chart. The initial flow chart was compared with and modified by the original and addi- tiona! protocols. SEcriON II In this section it is argued that the major notions on which the recon- structed studies of reference are built are productive of results which are un- reliable. For various, often similar rea- sons, these notions are in need of serious explication. In several cases it is noted that such explication does not seem forthcoming. That the notion, the subject of a ques- tion, warrants discussion is by no means obvious . Cole was unable to correlate the subject of a question with either its origin or its historical period. Indeed, the Herners observed that "the results of a study such as the present one, based on questions from workers in a field other than atomic energy, would pro- duce results quite different from those obtained in the present one . . . conduct- ed at some future time, would also pro- duce results at a variance with the pres- ent ones."21 20 Ibid. , p. 3. 21 H erner and Herner, op. cit., p. 176. 134 I College & Research Libraries • March 1969 To Cole the significance of the subject of a question was its conjectured power to predict subject interests of reference patrons. Its significance to the Herners presumably was quite similar. It is ques- tionable, however, that either Cole's or the Herners' results characterized the subject interests of patrons in the simple one-to-one correspondence supposed. Cole defined the subject of a question by those operations which specify the class number of a book by means of the Decimal Classification. The subject of a question, then, was the subject as in- terpreted under that scheme. The in- terpretation of questions under that scheme, however has several obvious disadvantages. Under interpretation of the Decimal Classification, for example, it is obvious that two or more questions about the same subject are not necessarily mapped into the same class. Though the pair of questions "Who chopped down the cherry tree?" and eeWho was the first president of the U.S.?" both have the same subject, the latter is decimally classifiable under history but the former is classifiable under the social sciences. Or, given the pair of questions "What are the properties of an irregular cube?" and e'What are the basic tenets of Law- sonomy?" the latter is unclassifiable un- til or unless it is known what indeed Lawsonomy is and the former, though it has no subject, is classifiable in the pure sciences. To circumscribe the artificiality of the Decimal and similar classification schemes, consider criteria for a more natural scheme: 22 (1) the subject of a question is independent of its various formulations and ( 2) it is about the ex- tension of its subject term ( s). Though these criteria dispel some of the diffi- culties just observed in connection with 22 For this example credit is due Hilary Putnam, "Formalization of the Concept 'About,'" Philosophy of Science, XXV (April 1958), 125; cf. Nelson Good- man, "About," Mind, LXX (January 1961). the Decimal scheme, they create diffi- culties of their own. By condition 1, question (a) "Is Henry Miller the au- thor of Little Women?" is by subject equivalent to question (b) eels the au- thor of Little Women, Henry Miller?" By condition 2, however, question (a) is about Henry Miller and question (b) is about Louisa May Alcott; therefore, questions (a) and (b) do not have the same subject and criteria 1 .and 2 are contradictory. Indeed, besides the absence of a simple one-to-one correspondence be- tween the subject of a question and its interpretation under a classification scheme, as well as the lack of a formal- ization of our concept eeabout," there is experimental evidence that subject clas- sification is far from the objective proc- ess assumed by either Cole or the Hern- ers.23 Appropriately recast, the subject of a question undoubtedly has some- thing important to tell us of reference behavior. Under its present formula- tions, however, it has almost nothing to tell us. Breed's notion of the difficulty of a question is similar to Cole's notion of the specificity of a question in several respects. Rather than descriptive of the patron, as she supposed, Cole's notion of the specificity of a question was equally descriptive of the librarian. Cole defined specificity as the absence of a difference between the patron's ques- tion and the librarian's interpretation of that question. To exemplify non-spec- ificity Cole wrote of the student who asked for the other author of The Tam- ing of the Shrew and the librarian who gave Henry James as the answer. If the librarian of Cole's example arrived at his answer by comparing the formula- tion of the question and the formulation of the solution at hand, then it is ob- 23 See Mary Cuddy St. Laurent, "A Review of the Literature of Indexer Consistency" (unpublished mas- ter's thesis, Graduate Library School, University of Chicago, 1966). vious that a librarian who held "the Taming of a Shrew," or possibly Turn of the Screw as well, in his cognitive structure, undoubtedly would have an- swered the student of Cole's example in quite another way. Indeed, rather than exemplifying non-specificity, the ques- tion "Who besides Shakespeare wrote The Taming of the Shrew?"24 would have exemplified specificity. Breed observed that, when they were received, difficult reference questions "appeared on the surface to be unex- ceptional questions which the usual ref- erence books would answer."25 For that reason he defined difficult questions as those for which "the ordinary reference sources failed to ·produce an answer."26 Like Cole's notion of specificity, Breed's notion of difficulty is indeterminately characteristic of either the patron or the librarian or, indeed, the environment in which the question is asked. Under Breed's definition the difficulty of a question is an empty notion. Under Breed's definition, that is, a question is difficult only if a question is difficult. Breed's manipulations on the set of difficult questions were productive, then, of .a miscellany of tenuously related re- sults. Let us probe this matter somewhat further. Given Reitman's description of a problem as having an initial state and a terminal state, a problem solution may be described as being a set of operations by which the former is convertible into the latter.27 On the basis of this de- scription it is possible, for example, to express the question "How is a sow's ear changed into a silk purse?" by the fol- lowing rule: Description-contains sublist 1. sublist 1-sublist 2, sublist 3, sublist 4. 24 Cole, op. cit., p. 40. 25 Breed, op. cit., p. 19. 26 Ibid. 27 Walter R. Reitman, Cognition and Thought, An Information-Processing Approach (New York: John Wiley & Sons, 1965), p. 133. Empirical Study of Reference I 135· sublist 2-is, initial state; contains sublist 5. sublist 3-is terminal state; contains sub- list 6. sublist 4-process; input, sublist 5; out- put sublist 6. sublist 5-is, sow's ear. sublist 6-is, silk purse. 2 8 If a person has an element in his cognitive structure which satisfies this rule, then that person has a solution to this question. If, however, that person does not have an element in his cogni- tive structure which satisfies this rule, then he cannot achieve a solution to the question unless he is successful in set- ting up substitutive or subordinate ques- tions to the one at hand. For example, it is possible to achieve a solution to the question given above by replacing the implicit physical constraint by a value constraint such that the value of the sale of a sow's ear will equal the value, or partial value, of the purchase of a silk purse. In any case, short of knowing pre- cisely what elements are held by a given cognitive structure and its environment and, indeed, of knowing what counts as a satisfactory solution to a given ques- tion, the task of explicating the notion "the difficulty of a question" will prove a formidable task. At various points in his study Breed observed deficiencies in his knowledge typology. For example, he observed that liberal arts and personal knowledge "were too subtle to be grasped in a study of this kind,"29 that knowledge gained in the process of search and library spe- cialized knowledge were not readily distinguishable "where final successful steps were concerned,"30 and that "the separation of kinds of knowledge is slightly unrealistic; the reference librar- ian combines knowledge of many differ- 28 Ibid., p. 134. 2o Breed, op. cit., p. 52. ao Ibid. , p. 56-57. 136 I College & Research Libraries • March 1969 ent kinds in his exercise of judgement and his interpretation of the data at hand."3 1 Although almost nothing is known about one's personal knowledge, what is reasonably conjectured when re- lated to Breed's self-criticism, makes that criticism seem largely understated. Newell, Shaw, and Simon, for example, have conjectured that cognitive domains are structured by 1. A control system consisting of a num- ber of memories, which contain symbol- ized information and are interconnected by various ordering relations. . . . 2. A number of primitive information processes, which operate on the information in the memories .... 3. A perfectly definite set of rules for combining these processes .... 32 If this conjecture is correct or even nearly correct, then one's knowledge consists not only of hierarchies of in- formation but also hierarchies of rules for manipulating and modifying the in- formation as well as the rules them- selves. In that case, each sort of one's knowledge would be so intimately bound up with every other sort that to speak of identifying the sort of knowl- edge which led to a specific action would be misleading; to suppose the reality of such identification would be ridiculous. Breed defined a reference step ·as the librarian's consultation of any source or entry in a source. Use of the reference step as the basic unit in the process of answering reference questions, however, has the same obvious disadvantages as the use of the human pulse beat to meas- ure time. Like the human pulse beat, the reference step would differ for dif- ferent people and would differ for the same person at different times, at dif- ferent places, and under different con- ditions of mental and physical health. 31 Ibid., p. 58. 32 Allen Newell, J. C. Shaw, and Herbert A. Simon, "Elements of a Theory of Human Problem Solving," Psychological R eview, LXV (May 1958), 151. Gagne has suggested several variables of individual difference which point up the problems inherent in such notions as the specificity or difficulty of a ques- tion, the knowledge used in answering questions and the reference step as de- fined by Breed.33 One person, for ex- ample, may possess more information and more rules for applying that infor- mation than another; be more facile in the recall of stored information; more able to select and maintain conceptual distinctions in the face of conflicting cues; more fluent than another in formu- lating hypotheses; more steady in the retention of a solution model; more rapid in verifying a solution. Implicit in Gagne's comments in relation to such notions is that variables of individual difference must be given experimental control or that such notions must be given independent study and explica- tion. Simon .and Kotovsky, for example, have proposed a theory which "predicts rather successfully which problems, from a set of letter series completion test items, will be the more difficult for human S's."34 In Section VI Hayes's ex- perimental control over variables of in- dividual difference is observed in a study of problems characterized by se- quences of well-defined steps. Though the Herners interpreted their results on the conceptual structure of reference questions into information re- trieval, intuitively at least, these results would seem to have implications for ref- erence. It is intuitively reasonable to suspect that in a spoken question-an- swering system, the number of signifi- cant concepts per question will not ex- ceed the span of immediate memory, i.e., the number of symbols which a per- son can repeat, orally or manually, in 33 Robert M. Gagne, "Human Problem Solving : In- ternal and External Events," Benjamin Kleinmuntz ( ed. ), Problem Solving: Research, Method, and Theory (New York: John Wiley & Sons, 1965), p. 143-46. 34 Herbert A. Simon and Kenneth Kotovsky, "Human Acquisition of Concepts for Sequential Patterns," Psy- chological Review, LXX (November 1963), 534. correct sequence and without error after a single presentation. For example, as Miller has observed, "a person who can repeat nine binary digits will have a span of about eight decimal digits, sev- en letters of the alphabet, or five mono- syllabic English words."35 Also, it seems reasonable to expect a human encod- ing device to produce logical products more frequently than logical sums and to produce logical sums more frequent- ly than logical differences. Several studies have suggested that humans categorize conjunctively more readily than disjunctively and prefer, in cate- gonzmg, positive over negative in- stances.36 Quite why and under what conditions these results obtain is barely speculated upon as, in Bruner's words, "one eventually begins to wonder wheth- er Nature herself does not abhor dis- junctive concepts."37 The Herners' re- sults on the conceptual structure of ref- erence questions reinforce these suspi- cions. Recall that a discrete concept was de- fined by the Herners as a concept which is both significant to the information de- sired and indivisible with respect to its essential meaning. Essential meanings, however, presuppose criteria, determi- nancy, and uniformity of usage, which common language does not meet. 38 At the present time it can, at best, be said that a person "has a concept if he has a disposition on the basis of which he can make nominal classificatory statements or responses ('this is X that is not X'); it is assumed that the disposition is 35 George A. Miller, "Human Memory and the Stor- age of Information," IRE Transactions on Information The ory, IT-2, 1956 Symposium on Information Theory, Held at Massachusetts Institute of Technology, Cam- bridge, Massachusetts, September 10-12, 1956 (Cam- bridge, Massachusetts: M.I.T. Press, 1956), p. 131. 36 Jerome S. Bruner, Jacqueline J. Goodnow and George A. Austin, A Study of Thinking (New York: John Wiley & Sons, 1956). 37 Ibid., p. 162. 38 Carl G. Hempel, Fundamentals of Concept Forma- tion in Empirical Science, Vol. II, No. 7 of the Inter- national Encyclopedia of Unified Science, ed. Otto Neurath, Rudolph Carnap and Charles Morris ( Chica- go: University of Chicago Press, 1952), p. 9-10. Empirical Study of Reference I 137 learned from a number of instances which vary among themselves; it is as- sumed that the responses also be made to instances other than those contained in the set on which the concept was learned; it is taken that the classificatory response should not be the only possible one."39 Indeed, the essential meaning of little from the essential subject of a question in Cole's study. In Cole's study it was the Decimal Classification which seemed to provide intersubjectively syn- onymous interpretations of subjects. In the Herners' study it undoubtedly was "a special classification scheme for li- brarians in the atomic energy field,"40 a question in the Herners' study differs rather than the questions alone, which provided the results shown in Table 1 on the number of discrete concepts per question. Logical products were defined by the Herners as information about A and B; logical sums, as information about A or B; and logical differences, as informa- tion about A and not B. Under this in- terpretation of the conceptual structure of reference questions the Herners re- ported the results shown in Table 2. What is of interest to reference about these results is that they need not have turned out as they did. Assuming the psychological reality of discrete con- cepts, for the sake of argument, it is pos- sible to interpret the logical relation- ships among the discrete concepts of th,e question, "Who besides Shakespeare wrote The Taming of the Shrew?'' in any one of the following ways: (author and The Taming of the Shrew), ( Henry James and The Taming of the Shrew) or (Henry James and Turn of the Screw), (The Taming of the Shrew and not Shakespeare) and so forth. In- deed, it is entirely clear that such re- 39 John P. Van De Geer and Joseph M. F. Jaspers, " Cognitive Functions," Annual Review of Psychology, XVII (Palo Alto, California: Annual Reviews, 1966 ), 149. 40 Herner and Herner, op. cit., p. 171. 138 I College & Research Libraries • March 1969 TABLE 1 NuMBER OF DISCRETE CoNCEPTS PER QuESTION 41 Number of concepts Number of questions Per cent of questions 1 466 12.1 2 1818 47.2 3 1167 30.3 4 327 8.5 5 73 1.9 6 0 0 TABLE 2 LoGICAL RELATIONSHIPS AMONG DISCRETE CoNCEPTS 41 Logical Logical Logical products sums differences Totals Number of questions (with more than one concept) Per cent of questions (with more than one concept) 3773 45 98.0 1.2 33 0.8 3851 100 suits as those given in Table 2 are largely dependent on the experimenter. For a more general example, from an arbitrary array of four attributes of two values each, as Bruner has observed, «one can define the same subset of in- stances with different types of concepts ... the way in which a person will categorize new instances encountered will depend drastically upon the type of concept he has constructed . . . when one learns to categorize a subset of events in a certain way, one is doing more than learning to recognize new instances. . . . One is also learning a rule that may be applied to new in- stances."42 The major notions around which the results of Carlson's study were devel- oped are what he called regular human behavior and inconsistent human behav- ior. Since what he intended by either .regular or inconsistent human behavior is not at all clear, the following para- graphs attempt to clarify the meanings of these notions and to point up their shortcomings. One of Carlson's "most encouraging findings" was that "human search be- havior is really quite regular ."43 By this .,_Ibid., p. 175. 42 Bruner, Goodnow, and Austin, op. cit., p. 44-45. 43 Carlson, OJJ· cit.~ p. 28. he meant, as he explained two sentences below, that "there is a great deal of common behavior"44 in human search behavior. If regular behavior is that be- havior which is common among search- ers, then it is not unreasonable to ex- pect inconsistent behavior to be that be- havior which is not common among searchers. As Carlson was at pains to point out by numerous examples, how- ever, inconsistent behavior was consist- ently practiced by the subjects of his study. In other words, it was also found to be quite common. Indeed, he ex- plained that inconsistent behavior is that behavior which is detrimental to a suc- cessful search, e.g., ''human searchers are often influenced by the physical en- vironment . . . he scans mostly those documents at a convenient eye level . . . he has a strong tendency to select clean, brightly colored books."45 If, then, inconsistent behavior is that behavior which is both common and detrimental to a successful search, it does not seem unreasonable to surmise that regular be- havior, in contrast, is that behavior which is both common and characteris- tic of successful searches. If these are the meanings which Carlson intended for these notions, then we shall see in 44 Ibid. 45 Ibid., p. 131-32. considering his flow charts of regular behavior and his discussion of inconsist- ent behavior that he stipulated unfor- tunate meanings for both of these terms. If Carlson has indeed How-charted hu- man search behavior, then his flow charts must tend to meet at least one criterion. They must tend to specify a set of op- erations which is sufficient to achieve an answer to a question much as a refer- ence librarian would. However, since a description of human search behavior must account for both that behavior which is successful and that behavior which is not successful in a search, Carl- son's elimination of unsuccessful search behavior from regular behavior, indeed from the entire study, precludes that possibility. One is led to suspect, in fact, that instead of preparing a flow chart of human search behavior Carlson pre- pared a flow chart of information re- trieval. Considering Carlson's Generalized Flow Chart, reproduced on the follow- ing page, first from back to front, it is evident, on inspecting the second level of detail, that his "Exit" intends the printout of a bibliography. Unless Carl- son intends something more general than what is ordinarily intended by ~'bibli­ ography," then one must conclude that either Carlson equated the conclusion of all successful searches with listings as outputs or that Carlson has, indeed, not How-charted human search behavior at all. That is, certainly, successful searches conclude with answers other than bibli- ographical listings, as answers which are simply affirmatives or negatives. Considering this Generalized Flow Chart from front to back, again on the second level of detail, his "Select 2-4 keywords" intends: If terms of input not previously encountered, generate syn- onyms from a general dictionary; other- wise, generate synonyms from memory; then, if synonyms are not available from either a general dictionary or from mem- ory, generate synonyms from a medical Empirical Study of Reference I 139 dictionary. "Enter synonyms into syn- onym list; enter adjectives into search list, and rank."46 Delete common nouns. Finally, place the remaining nouns on the search list and rank them. Surely, however, "Select 2-4 keywords," as just one detailed example from the General- ized Flow Chart, is not representative of successful or unsuccessful human search behavior, not even in Carlson's own terms since he observed that the ref- erence librarians of his study never checked "the accuracy of their spellings, pronunciation, and meaning before the search starts."47 Indeed, if in an act of communication, the decoding process of the listener is an approximate inverse of the encoding process of the speaker, then Carlson's "Select 2-4 keywords" taken inversely will surely not elicit a question from any speaker. The suspi- cion that Carlson's Generalized Flow Chart is a model of information retrieval rather than of human search behavior, as advertised, is more than warranted. One of Carlson's "most significant findings" of the inconsistent sort was that "there is no feedback of the results of their search to the librarian."48 On the basis of this finding Carlson inferred that "the reference librarian has no con- sistent way of improving the quality of his search."49 He concluded that ~'any searches completed for a requestor should only be given out on the strict condition that the librarian be told about the usefulness or inadequacies of the result."50 The inference that there is no feed- back to the reference librarian was based on evidence illustrated by the following example: During the record- ing of protocols one of the librarians ~ Ibid., p. 8. 41 Ibid., p. 29. 48 Ibid., p. 35. 49 Ibid., p. 35-36. 50 Ibid., p. 36. 140 I College & Research Libraries • March 1969 1 ) 2) 3) Select 2-4 key words D etermin e requ estor language res trictions D etermi ne cita ti on age restrictions D e rive general approach 4) Determine next source 5) Scan for match wi th search tenn s and develop synonyms .,._.-~----.... from present source 6) 7) Use crite ria to accept or reject each entry · Any more e ntri es for this t erm match? 8) 9) 10) Are criteria met to t erminat e search of thi s sou~ce? Are criteria met to terminate search? Exit 11) FIG. !.-Generalized Flow Chart of Reference Librarians Search Behavior. G. Carlson, Search Strategy by Reference Librarians, Part 3 of the Final Report on the Organization of Large Files (Sherman Oaks, Calif.: Advance Information Systems, Hughes D ynamics, 1964 ), p. 7. stated, "I've never had a reader along on this kind of search."51 On the basis of this protocol, it seems reasonable to con- jecture that on other kinds of searches the librarian was sometimes accompa- nied by the reader. According to Carl- son's construal of feedback, then, feed- back must have occurred in such cases as these. It seems more reasonable to assume, however, that patron feedback can occur only after the patron has had time to evaluate the work done for him in relation to the work for which he re- quested the librarian's help. In that case, having the reader along would for the most part be irrelevant. But what does Carlson intend by feedback? Surely in a complex question-answer system as the one under discussion, feedback is not the simple or obvious activity which Carlson describes as "He receives a re- quest from a person; makes the search, often without the requestor present; gives the requestor the results; and hears no more."52 In the simplest man- ner feedback should have to be de- scribed as the librarian's response to the patron's output in such a way as to re- duce the difference between it and a prescribed input and the patron's re- sponse to the librarian's output in such a way as to reduce the difference be- tween it and a prescribed input; as well as the librarian's response to his own output in such a way as to reduce the difference between it and a prescribed output and the patron's response to his own output in such a way as to reduce the difference between it and a pre- scribed output. "The picture roughly re- sembles two rings (but rings each of which is composed of a series of on- going and intervening events ) that are placed in planes at an angle to each other and touching each other at com- 51Jbid., p. 35. 52 Ibid. Empirical Study of Reference I 141 mon points in their circumferences."53 To describe feedback in this, the sim- plest of ways, makes it absurd to assert the absence of feedback on the basis of data derived from protocol analysis of the librarian alone. Indeed, to refuse the reader the desired information until he feeds back to the librarian in an obvious manner surely makes this the most ab- surd of Carlson's findings, especially since Carlson's construal of inconsistent behavior implies that it, like regular be- havior, results in a successful search. SECTION III This section of the essay discusses the conventional concept of reference and its relationship to the empirical study of reference. Derived from the preparation for and the practice of reference, the conven- tional concept is made explicit in nu- merous definitions in the literature of li- brarianship. In these definitions it is clearly asserted that only reference li- brarians perform reference. As focal points for our discussion, let us consider several definitions of reference; as our point of departure, Pierce Butler's coun- ter to the conventional concept. To Pierce Butler such definitions of reference as "the assistance given by the librarian to readers in acquainting them with the intricacies of the catalogue, in answering questions, and, in short of do- ing anything and everything ... to fa- cilitate access to the resources of the li- brary,"54 were invalid. To Butler such definitions were invalid in that they tended to define reference in the trivial circle as "what goes on in the reference room" and the reference room as "the 53 Floyd H . Allport, Theories of Perception and the Concept of Structure, A Review and Critical Analysis with an Introduction to a Dynamic-Structural Theory of Behavior (New York: John Wiley & Sons, 1955), p. 526-27. ~>' William B. Child, "Reference Work at the Colum- bia College Library," Library Journal, XVI ( October 1891) , 298. 142 I College & Research Libraries • March 1969 place where reference work is carried on."55 Pierce Butler proposed that refer- ence is "that process by which civilized man is able to obtain specific informa- tion at will by use of books which have been organized into a library."56 Unlike other definitions of reference, Butler suggested that reference is not per- formed solely by librarians and is not performed solely in reference rooms. Butler's counter to the traditional con- cept of reference, however, has made little impression. In her empirical study of reference, Dorothy E. Cole observed no difference between Butler's concept of reference and her own, as we have noted, the "queries which people bring to the desk of the reference librarian" and the ref- erence librarian's response to the queries in terms of "materials best suited to the needs of the re.ader."57 Indeed, Cole felt Butler's to be supportive of her concept of reference. In another empirical study of reference a difference was observed. With unshaken confidence in the tradi- tional concept, however, Lois Fern58 re- jected Butler's co~nter. Fern, that is, first observed that Butler's concept was inconvenient, i.e., she would have had to consider "those hundreds of instances in which the patron engaged in 'refer- ence work' without the librarian's as- sistance"; she then observed that Butler's concept was absurd, i.e., she would have had to entitle her study "Use of Library Catalogs by Reference Librarians En- gaging in Reference W ork."59 As students of reference, librarians either have not perceived Fern's "hun- 55 Pierce Butler, "Survey of the Reference Field," Pierce Butler, ed., The Reference Function of the Li- brary, Papers Presented before 1 th~ Library Institute at the University of Chicago, June 29 to July 10, 1942 ( Chicago: University of Chicago Press, 1943), p. 4. 56 Ibid., p. 11. 57 Cole, op. cit., p. 1. 58 Lois Fern, "Library Catalogs as Reference Sources" (unpublished master's thesis, Graduate Library School, University of Chicago, 1962). 59 Ibid., p. 8. dreds of instances" or perceived them, like Fern, as something different from reference, perhaps something less than reference. They have perceived refer- ence through the conve·ntional concept. They have perceived only what experi- ence in the preparation for and practice of reference has led them to expect. There is evidence that "when such ex- pectations are violated by the environ- ment, the perceiver's behavior can be described as resistance to the recog- nition of the unexpected or incongru- ous."60 In the familiar Bruner and Post- man experiment "On the Perception of Incongruity," subjects were shown by tachtistoscope a series of five playing cards at controlled and gradually in- creasing levels of exposure. Some of the playing cards were normal, i.e., expect- ed, as the five of hearts; others were incongruous, i.e., unexpected, as the red six of clubs. «At each exposure the sub- ject was asked to report everything he saw or thought he saw."61 A stimulus series was completed by three succes- sive correct responses to each of the five cards at various exposure levels from ten up to one thousand milliseconds. A few of the subjects perceived the normal cards erroneously. One subject, for example, mistook the black five of spades for the red five of hearts. The vast majority of subjects, however, per- ceived the incongruous cards erroneous- ly. Subjects identified the red six of spades, in a number of cases, as the six of spades or as the six of hearts. The subjects reported such perceptions with considerable assurance. Without uncer- tainty, that is, they fitted what they presumably saw into a normal form or color pattern that they expected to see. Often subjects compromised on what 60 Jerome S. Bruner and Leo Postman, "On the Per- ception of Incongruity: A Paradigm," Journal of Per- sonality, XVIII (December 1949), 222. et Ibid ., 210. they saw. With the red spade and club cards, for example, subjects reported seeing such colors as black with red lights, lighter than black but blacker than red, olive drab, and so on. A few of the subjects came suddenly to recognize the incongruity of the trick cards. Other subjects found something wrong with them and came to recognize the incon- gruity gradually. "A subject viewing a red spade may start by reporting a red tint which gradually becomes redder on succeeding trials until he finally asserts that the card is .a red spade."62 Indeed, there were several subjects who never managed to unmask the incongruity of certain of the playing cards. Analogically, the Bruner-Postman ex- periment provides a suggestive schema for viewing the librarian's perception of reference; fitting, that is, the incongruity of reality to one of the neat conceptual categories prepared by prior training and experience, training and experience which do not and never need to indi- cate that anyone other than librarians perform reference. Let us consider now some of the implications of the librar- ian's perception of reference. If the student of reference perceives reference as performed only by refer- ence librarians, then in his research he is constrained into conceptually setting reference librarians apart from other human beings. In setting reference li- brarians apart from other persons, it has seemed manifest to the student of ref- erence that what differentiates the ref- erence librarian from other persons is his preparation for and practice of ref- erence librarianship. Under the conven- tional concept of reference, then, the empirical study of reference poses two pertinent questions to the student of reference: How efficient is education for reference librarianship? How successful is the practice of reference librarian- 62 Ibid. , 222. Empirical Study of Reference I 143 ship? In the former case the empirical study of reference has become an in- strument for investigating some aspect of that preparation and, in the process, for suggesting some modification of it; in the latter case, an instrument for in- vestigating and modifying some aspect of that practice. Indeed, the results of the reference studies reconstructed above were interpreted in just that man- ner. Cole, for example, observed that classifying reference questions should aid the library administrator in evaluat- ing "the amount and kind of work done by the reference department" and aid the library training agencies in deter- mining the nature and scope of the ref- erence courses."63 Breed wrote, "The findings concerning the reference proc- ess related to difficult questions have implications for library schools and the material presented in courses on refer- ence work. More attention might well be paid to the analysis of the reference process itself-and problems on the ad- vanced level -might include more exam- ples of the kind of incomplete and in- accurate data with which the reference librarian must so frequently work."64 The Herners said, "It is evident that useful data on the information require- ments of a body of users can be obtained from collecting and analyzing statisti- cal quantities of their reference ques- tions."65 By useful the Herners presum- ably meant helpful in specifying "those information requirements which librar- ians ... are likely to be called upon to meet."u6 Carlson declared that "In many cases, human behavior is inconsistent to the detriment of successful search. . . . These [inconsistencies] could be used as guidelines by librarians to improve their search procedures. Perhaps some of these inconsistencies or omissions could 63 Cole, op. cit., p. 58. 64 Breed, op. cit., p. 65. 65 Herner and Herner, op. cit., p. 176. oo Ibid. , p. 172. 144 I College & Research Libraries • March 1969 be generalized and provide guidelines for the training program of reference librarians."67 This viewpoint, of course, does not at all invalidate the results of the empiri- cal studies of reference. It does, how- ever, make the process by which they are derived circular. It does, therefore, profoundly affect the manner in which they are perceived. Given that the prep- aration or practice of reference is both point of departure and point of con- clusion of the empirical studies of ref- erence, the results of these studies must often seem, then, to be little more than reflective formulations of the very ob- servations which reference librarians en- counter in their education or the very observations which reference librarians make in their work Consider now sev- eral of the results derived from these empirical studies of reference. Cole, again as an initial example: Reference patrons most frequently ask questions in the social sciences, useful arts, and history. Breed: Difficult questions are characteristically specific rather than general and involve bibliographical ref- erences rather than facts. Bibliographi- cal questions are difficult to answer be- cause of omissions or inaccuracies in the patron's citation of author, titles, and dates of publication. The Herners: Pa- trons generally ask non-technical ques- tions .about descriptions of processes or methods of procedure. Carlson: "Human searchers should be very careful to write down their search terms and check the accuracy of their spellings, pronun- ciation, and meaning before the search starts, and whenever they adopt a new search term as the search progresses .... A human searcher would be more con- sistent if he would follow more rigid rules in checking every document or source. . . . The reference librarian has no consistent way of improving the quality of his own search. Any searches 67 Carlson, op. cit., p. 28. completed for a requester should only be given out on the strict condition that the librarian be told about the useful- ness or inadequacies of the results."68 If the results of the empirical studies of reference seem to be little more than reflective formulations of everyday ob- servations, observations for which there .are hundreds of counter-examples ready at hand, then it is hardly surprising that Ennis, for example, has characterized the endeavor which has produced such results as "among the weakest in all li- brary research."69 To the non-librarian, the librarian's ' commitment to the conventional con- cept of reference in his empirical re- search perhaps seems strange. To the librarian, for the non-librarian to con- demn and then use that concept to guide his empirical study of reference must likewise seem strange. Consider, then, two cases: the one illustrated by what psychologists have called function- al fixedness, the other illustrated by what they have called distortion pro- duced by popular opinion. First consider the case for librarians, as Cole and Breed. "The student attends an educational institution on the pre- sumption that the training he receives there will enable him to solve problems in later life more adequately and effi- ciently than would be the case had he not received this training .... It is also possible, however, that the training which the student receives may often, alas perhaps more often than we .are aware of, make the student less able to solve certain new problems."70 The clas- sical description of functional fi xedness is Maier's pendulum problem. 71 68 Ibid., p. 29-36. 69 Philip H. Ennis, "The Study of the Use a nd Users of Recorded Knowledge," Library Quarterly, XXXIV ( October 1964) , 309. 70 Rudolph W. Schulz, "Problem Solving Behavior and Transfer," Harv ard Educational Review, XXX ( Winter 1961), 61. 71 Norman R. F. Maier, "Reasoning in Hum an s: II, the Solution of a Problem and Its Appearance in Con- sciousness," Journal of Comparative Psychology, XI (August 1931), 181-94. Subjects were introduced into a room where two pieces of cord were suspend- ed from the ceiling. The room con- tained, besides the suspended cords, "many objects such as poles, ringstands, clamps, pliers, extension cords, tables and chairs."72 The subjects were told to tie the two pieces of cord together. The cords, however, were too far apart for a single individual to hold both ends simultaneously. Though several solutions to the problem were possible, one seemed especially difficult for the sub- jects to grasp. It was the solution achieved by weighting one of the cords with pliers, setting the cord in motion and catching it after the other cord had been seized. Most of the subjects did not perceive the cord as a potential pen- dulum and the pliers as a potential weight. In general terms the subjects of Maier's study demonstrated an in- ability to displace old, learned concepts with new ones required by a novel task. Similarly, the subjects who are our con- cern in this essay have demonstrated an inability to displace that concept of ref- erence learned in the preparation for its practice and in the practice itself by concepts required for the task, generally novel to librarians, of the study of refer- ence. Consider now the case for non-librar- ians as the Herners and Carlson. Recall that the Herners condemned and that Carlson called the conventional concept of reference into question. Recall that unlike Cole and Breed, who were com- mitted to that concept by prior training and experience, the Herners and Carl- son were not. Recall that though their concepts of reference differed from that of Cole and Breed, the concept of refer- ence which guided their studies did not. The evidence here suggests that "the in- terfering influence of the familiar ex- tends even to situations where the sub- ject is presumably confined to rigid rules 72 lbid ., 18 2. Empirical Study of Reference I 145 of thought that should exclude it."73 In a series of tests on "The Distortion of Syllogistic Reasoning Produced by Personal Convictions," Morgan and Morton showed that subjects tend to reach conclusions which, though logical- ly invalid, are consistent with popular opinion. 74 The subjects of these tests were required to draw conclusions from two premises by the process of immedi- ate inference, as If all men are mortal (major premise) , and If John is a 1nan (minor premise) ; then John is mortal (conclusion). 7 5 The subjects of this study were given two sets of fifteen syllogisms in parallel form. In the first set of syllogisms letter symbols as X, Y, and Z were used as terms of the premises; in the second set, "vital issues which were being currently discussed in the papers and over the air were"76 incorporated as the terms. Given the invalid syllogism, for exam- ple, Some ruthless men deserve a violent death; since one of the most ruthless of men was Heydrich, the Nazi hangman: 1. Heydrich, the Nazi hangman, de- served a violent death. 2. Heydrich, the Nazi hangman, may have deserved a violent death. 3. Heydrich, the Nazi hangman, did not deserve a violent death. 4. Heydrich, the Nazi hangman, may not have deserved a violent death. 5. None of the given conclusions seems to follow logically. 77 in letter symbols, subjects tended to choose· the second conclusion presum- ably under the influence of the particu- 73 Bernard Berelson and Gary A. Steiner, Human Be- havior, An Inventory of Findings (New York: Har- court, Brace & World, 1964), p. 205. 7 4 John J, B. Morgan and James T. Morton, "The Distortion of Syllogistic Reasoning Produced by Per- sonal Convictions," Journal of Social Psychology, XX (August 1944), 39-59. 75 Ibid., 40. 76 Ibid., 45. 77 Ibid., 48. 146 I College & Research Libraries • March 1969 Jar-affirmative atmosphere of this syllo- gism. Given the same syllogism as a pop- ular issue, however, the same subjects tended to shift from the second to the first conclusion, an opinion which was "quite strong at the time that this test was given."78 In general terms the sub- jects of the Morgan- Morton study be- lieved that they were guided in their choice of conclusions by logical thought, though, in fact, when current issues were embodied in the syllogisms, they were being guided by popular opinion. Sim- ilarly, it is not implausible that the oth- er subjects who are our concern in this essay were influenced by a point of view traditional among librarians even though they faced a situation requiring rigid rules of thought which should have eliminated that point of view from con- sideration. Whether it has been librarians or non- librarians who have used the conven- tional concept of reference to guide their empirical studies is largely unim- portant. What is important is this. Under the conventional concept it has been as- sumed in the empirical study of refer- ence that only reference librarians per- form reference, that the librarian's pro- ficiency in the performance of reference is the result of his professional training and. experience, that the goal of the empirical study of reference is the mod- ification of the librarian's education or of his practice, that observations of the librarian's performance are a sufficient basis on which to propose those modi- fications which will improve the librar- ian's proficiency. Since the empirical study of reference under these assump- tions is capable only of augmenting the rules of thumb which underlie the li- brarian's professional training and prac- tice, it is clear under these assumptions 78 Robert B. Downs, assisted by Elizabeth C. Downs, How to Do Library Research (Urbana: The University of Illinois Press, 1966). that no amount of study will permit an explanation of how the performance of reference takes place except in the trivi- al sense that a manual of reliable refer- ence guidelines, .e.g., Downs's How to Do Library R.esearch,78 explains the perform- ance of reference. Though the conven- tional concept of reference serves a use- ful purpose, the basis for an esprit de corps, in the training and molding of reference librarians, it can hardly be surprising that this section of the essay concludes that empirical interpretation of the conventional concept is fruitless as a framework for the development of the empirical study of reference. SECITON IV In a previous section of the essay we observed that the notions basic to four empirical studies of reference were de- ficient. They did not permit results which are reliable. In the last section, we observed that the empirical inter- pretation of the conventional concept of reference is fruitless as a framework for the development of the empirical study of reference. For these reasons, this sec- tion of the essay considers reference out- side of librarianship. It suggests, intui- tively, that reference is a special case of problem solving. First, several definitions of problem- solving behavior are discussed, then sev- eral approaches to the study of problem solving are described, first on the theo- retical, then on the empirical level. The studies of problem solving which are described below were selected from those known to the author as the most suggestive for reference. In the next sec- tion, these studies are briefly compared with the studies of reference previously reconstructed. To Duncker "a problem arises when a living creature has a goal but does not know how this goal is to be reached. Whenever one cannot go from the given situation to the desired situation simply by action, then there has to be recourse to thinking."79 Duncker, as have many psychologists, differentiated between routine and non-routine problem solving. In the terms of Dollard and Miller, that is between problems solved by automatic habits and trains of thought. 80 Surely, however, this distinction, used to differ- entiate illegitimate from legitimate prob- lems, is a fruitless one. What is reflexive to one may well be cognitive to another. To Reitman "a system has a problem when it has or has been given a de- scription of something but does not yet have anything that satisfies the descrip- tion."81 But in emphasizing what he calls "degrees of problematicality,"82 Reitman must tend to ignore a question implicit in Duncker' s definition, why may a given task be reflexive to one but cog- nitive to another person. Indeed, Gagne suggests that "one of the fundamental criteria of problem solving is that a kind of performance which could not be ex- hibited before the 'problem' was solved can be exhibited after the 'problem' is solved."83 To Gagne problem solving is "an inferred change in human capability that results in the acquisition of a gen- eralizable rule which is novel to the in- dividual, which cannot have been es- tablished by direct recall, and which can manifest itself in applicability to the solution of a class of problems."84 The simulation of cognition is one theoretical approach towards an under- standing of problem-solving behavior. A computer is programmed to respond as a human would respond within a speci- 79 Karl Duncker "On Problem Solving," Psychologi- cal Monographs, LVIII ( 1945), 1. 80 John Dollard and Neal E. Miller, Personality and Psychotherapy, An Analysis in Terms of Learning, Thinking, and Culture (New York: McGraw-Hill Book Co., 1950), p. 97-105. 81 Reitman, op. cit., p. 126. s2 Ibid., p. 130. 83 Gagne, op. cit., p. 130. 84 Ibid., p. 132. Empirical Study of Reference I 147 fied task domain. The computer pro- gram, then, is a model or set of hypoth- eses about the internal processes under- lying human performance of a given task. "No physiological or neurological assumptions are made . . . these models conceive of the brain as an information processor."85 A program is run. The run generates predictions of the model. "In order to provide an adequate test, the behavior with which the computer out- put is compared must, of course, be different from the behavior which served as a basis for the model's construction."86 The results of the run are compared with relevant human results. That is the way predictions of the model are tested and, potentially, improved. Presumably a model of artificial intel- ligence may also model human behav- ior. The difference between models of cognitive simulation and artificial intel- ligence is this: "An artificial intelligence researcher interested in programming a computer to play chess would be happy only if his program played good chess, preferably better chess than the best human player. However, the researcher interested in simulating the chess-play- ing behavior of a given individual . . . wants his program to make the same moves as the human player, regardless of whether these moves are good, bad, or indifferent."87 Several of the following paragraphs describe Paige and Simon's test of Bobrow's artificial "Question-An- swering System for · High School Algebra Word Problems"88 as a first approxima- tion towards a model of "Cognitive Proc- 85 Edward A. Feigenbaum, "The Simulation of Ver- bal Learning Behavior," Edward A. Feigenbaum and Julian Feldman ( eds.), Computers and Thought (New York: McGraw-Hill Book Co., 1963 ), p. 297. 86 K. R. Laughery and L. W. Gregg, "Simulation of Human Problem-Solving Behavior," Psychometrika, XXVII (September 1962 ), 279-80. 87 Feigenbaum and Feldman, op. cit., p. 269. 88 Daniel B. Bobrow, "A Question-Answering System for High School Algebra Word Problems," AFIPS Con- ference Proceedings, XXVI, 1964 Fall Joint Computer Conference (Baltimore: Spartan Books, 1964). · 148 I College & Research Libraries • March 1969 esses in Solving Algebra Word Prob- lems."89 The question posed was what cognitive processes are necessary in solv- ing algebra word problems; what proc- esses are sufficient? The model is STUDENT. STUDENT com- municates with human beings over a limited range of natural language. Its task environment is algebra word prob- lems as (A NUMBER IS MULTIPLIED BY 6. THIS PRODUCT IS INCREASED BY 44. THIS RESULT IS 68. FIND THE NUMBER. ) 90 STUDENT reads a problem statement, re- writes it as a set of simpler sentences, transforms these sentences into equa- tions, and attempts to solve the result- ant set of equations. The student model understands a problem statement by "transforming it into an equivalent (in meaning) sequence of simpler kernel sentences. A kernel sentence is one which the listener can understand di- rectly; that is, one for which he knows a transformation into his information store.''91 Paige and Simon compared STUDENT's methods of transforming with human methods of translating algebra word problems into algebraic equations. The humans in this comparison were largely students. The students' processes were established on the basis of protocols; STUDENT's processes, on the basis of Bo- brow's descriptions of his computer pro- gram. STUDENT's transform of the problem statement given above reads: (THE EQUATIONS TO BE SOLVED ARE) (EQUAL X()()()() I (NUMBER) ) (EQUAL (PLUS (TIMES (NUMBER) 6) 44) 68)92 811 J effre~ M. Paige and Herbert A. Simon, "Cognitive Pr?cesses. m Solving Algebra Word Problems," Benja- mm Klemmuntz ( ed.), Problem Solving: Research, Method, and Theory (New York: John Wiley & Sons 1966). ' 90 Bobrow, op. cit., p. 604. 91 Ibid., p. 593. 92 Ibid., p. 604. Some of the students translated the same problem statement in a similar manner, as The unknown is a 'certain number,' which would be x. Multiply x by 6. Write down '6x' please. 'Increased by' means add, so you put a plus 44. 'The result is' -indicate equals-write please-'68.'93 Paige and Simon labeled this method direct translation, "a step-by-step sub- stitution of algebraic symbols and ex- pressions for the English words and phrases of the original problem state- ments."94 One of the sub-processes of translat- ing English prose into algebraic equa- tions, observed in both students and STUDENT, is the labeling of variables. The labels are of two sorts, conventional, i.e., x may be used as the abbreviatory name of some unknown, and relational, i.e., 6x or 6x + 44 = 68 may be used as the relational names between the same un- known and some known quantity. Un- like the students, however, STUDENT "in- troduces relational names only in the special case . . . where it is cued by the occurrence of the word 'this.' "95 A sub-problem of naming, again ob- served in both STUDENT and students is ambiguity of names. Two similar but distinct common language terms, for ex- ample, may refer to the same object, though two distinct but similar common language terms may refer to different objects. To the problem (THE NUMBER OF SOLDIERS THE RUSSIANS HAVE IS ONE HALF OF THE NUMBER OF GUNS THEY HAVE. THE NUMBER OF GUNS THEY HAVE IS 7000. WHAT IS THE NUM- BER OF SOLDIERS THEY HAVE Q. ) 96 a problem in which two different terms "Russian soldiers" and "they," refer t~ the same object, Russian soldiers, sTU- DENT responds: 93 Paige and Simon, op. cit., p. 70. 9• Ibid., p. 82-83. 95 Ibid., p. 78. 96 Bobrow, op. cit., p. 605. THE EQUATIONS WERE INSUFFICIENT TO FIND A SOLUTION (ASSUMING THAT) ( (NUMBER OF SOLDIERS (THEY /PRO) (HAVE/VERB) ) IS EQUAL TO (NUMBER OF SOLDIERS RUSSIANS (HAVE/ VERB)) (THE NUMBER OF SOLDIERS THEY HAVE IS 3500) 97 To the problem The number of quarters a man has is seven times the number of dimes he has. The value of the dimes exceeds the value of the quarters by two dollars and fifty cents. How many has he of each coin?98 a problem in which the expressions num- ber and value are synonymous in cer- tain contexts, a student responded in this fashion: Let's represent-- the fir-- --x as the dimes. Write down x please. The number of quar- ters is-is seven times the dimes. There- fore, the number of quarters is represented by 7x. Put '7x.' 'The value of the dimes exceeds the value of the quarters by two- fifty'; therefore, x . . . Since it really is a money problem we will check this and dis- cover that we really should multiply to make this-ah-to make it equal in value. Since we have 250 pennies we have to multiply the dimes by 10. So that should be lOx and a plus. Since it's quarters it would be 7 times 25 times x, on the other side. That is the correct solution. 99 In the former, the soldier problem, the cue through which STUDENT achieved a sufficient equation was a grammatical cue. The relationship, that is, between "Russian soldiers" and "they" is a gram- matical one. In the latter, the coin prob- lem, the cue was a substantive one. The student, that is, had to "know that the value of a quantity of coins equals the number of coins times the value per coin."100 Given the problem u1 Ibid. 98 Paige and Simon, op. cit., p. 79. 99 Ibid., p. 80-81. 1oo Bobrow, op. cit., p. 604. . Empirical Study of Reference I 149 (IF 1 SPAN EQUALS 9 INCHES, AND 1 FATHOM EQUALS 6 FEET, HOW MANY SPANS EQUAL 1 FATHOM Q. )100 like a student, STUDENT "is able to draw on a body of facts, picking out relevant ones, and use them to obtain a solu- tion"101 from its store of global informa- tion. To that problem STUDENT responds: THE EQUATIONS WERE INSUFFICIENT TO FIND A SOLUTION (USING THE FOLLOWING KNOWN RELA- TIONSHIPS) (EQUAL (TIMES 1 (FEET) ) TIMES 12 (INCHES)) ( 1 FATHOM IS 8 SPANS )102 It became evident, however, that the students were using more powerful proc- esses than those exhibited by STUDENT's use of definitional relationships. In trans- lating the coin problem, one of the stu- dents . . . found incongruity . . . if a man has seven times as many quarters as he has dimes and the value of the dimes exceeds the value of the quarters by two-fifty, the quarters must really not be worth too much, because if he has 7 times as many quarters as he has dimes, the number of- the value of the quarters must exceed the dimes by 7 ° 2.5 - x, or what not .... negative quarters-no such thing exists.1o3 Presumably, STUDENT would not have perceived this incongruity. Indeed, giv- en the problem A car radiator contains exactly one liter of a 90 per cent alcohol-water mixture. What quantity of water will change the liter to an 80 per cent alcohol mixture?l04 which is anomalous in the sense that it cannot be translated solely on the basis of the information given, it became ob- vious that those students who achieved a successful translation of the problem statement did so by creating a physical representation of the problem which en- 101 Ibid., p. 605. 102 Ibid., p. 604. 103 Paige and Simon, op. cit., p. 86. 1CM Ibid., p. 90. 150 I College & Research Libraries • March 1969 abled them to read off certain conserva- tion assumptions as though they were stated as part of the problem. As illus- trated by De Soto, humans translate such problems as "Alice is taller than Mary; Elsie is shorter than Mary: Is Elsie taller than Alice? ... into an