blecic.p65 A Longitudinal Study of the Effects of OPAC Screen Changes 515 A Longitudinal Study of the Effects of OPAC Screen Changes on Searching Behavior and Searcher Success Deborah D. Blecic, Josephine L. Dorsch, Melissa H. Koenig, and Nirmala S. Bangalore A longitudinal study of four sets of OPAC transaction logs was conducted over a four-year period. Analysis of the initial set of transaction logs revealed problems that users experienced while searching the OPAC. Over time, a series of OPAC screen changes were implemented in an attempt to help searchers improve their searching behavior. After each series of changes, a set of transaction logs was analyzed to assess the impact of the changes and to study whether earlier improvements in searching behavior persisted over time. Although the screen changes initially had a positive impact on searching behavior, in some cases the initial improvements in searching success were not sustained over time. se of transaction log analysis is A previous transaction log analysis an established method of data study conducted by the authors of this collection that provides librar- study between 1995 and 1996 at the Uni­ ies a record of users’ actual versity of Illinois at Chicago (UIC) com­ online search experience. Transaction log analysis is an unobtrusive and inexpen­ sive method that gives an unbiased view of how users navigate the catalog. In 1995, Lynn Silipigni Connaway, John M. Budd, and Thomas R. Kochtanek observed: “There is an underlying assumption that by identifying the needs and behaviors of catalog users, user-centered catalogs can be developed.”1 However, this assumption has not been fully explored. Does transac­ tion log analysis and subsequent OPAC screen redesign affect changes that result in more successful use of online catalogs? pared two sets of logs within a six-month period.2 Analysis of the first set of data revealed that users experienced difficulty with searching techniques. The OPAC introductory screens were simplified in an attempt to help users improve search success rates. The second set of data, ana­ lyzed after screen changes had been made, showed statistically significant improvements in searching behavior. Fewer searches resulted in zero postings, were missing the search argument, had the author in incorrect order, or incor­ rectly included an initial article, whereas Deborah D. Blecic is the Bibliographer for the Life and Health Sciences and Assistant Professor at the University of Illinois at Chicago; e-mail: dblecic@uic.edu. Josephine L. Dorsch is Acting Health Sciences Librarian (Peoria) and Associate Professor at the University of Illinois at Chicago; e-mail: jod@uic.edu. Melissa H. Koenig is an Assistant Reference Librarian and Assistant Professor at the University of Illi­ nois at Chicago; e-mail: mkoenig@uic.edu. Nirmala S. Bangalore is an Assistant Catalog Librarian and Clinical Assistant Professor at the University of Illinois at Chicago; e-mail: nbangalo@uic.edu. 515 mailto:nbangalo@uic.edu mailto:mkoenig@uic.edu mailto:jod@uic.edu mailto:dblecic@uic.edu 516 College & Research Libraries November 1999 a greater percentage of searches had cor­ rect syntax. These results were deemed improvements in searching success. The study also established a process and set in place a methodology to review trans­ action logs regularly and to monitor the effectiveness of any subsequent screen changes, a charge the authors carried out initially as part of a Transaction Logs Task Force and later as a research group. Transaction logs were captured and analyzed at the middle of the semester following each set of screen changes. In the present study, the authors com­ bined the two sets of data collected dur­ ing the initial study with two additional sets of data to provide a longitudinal analysis of OPAC transaction logs from 1995 to 1998. The present study was un­ dertaken with four purposes in mind: to determine whether, after the initial screen changes, improvements in user searching behavior were sustained over time; to examine the effect of subsequent screen changes; to continue to review the reasons for failed searches; and to measure the use of keyword searching. Screen changes were implemented after the first three sets of transaction logs in response to docu­ mented user search patterns. In addition, input was solicited from public services staff based on their experiences with ref­ erence transactions and bibliographic in­ struction. The research reported here tracked OPAC transaction log analysis over an extended period of time and mea­ sured the effect of screen changes on user searching behaviors. Literature Review A recent review of the transaction log analysis research literature is provided by Andrew Large and Jamshid Beheshti.3 This overview identified and analyzed studies published since 1990, discussed problems faced by researchers, examined data col­ lection methodologies, and gave research recommendations for bibliographic record enhancement, search capabilities, and in­ terface design. Beth Sandore and Thomas A. Peters previously provided thorough reviews of the history and uses of transac­ tion log analysis.4–5 In their previous study, the authors reviewed the literature to present several uses of transaction log analysis including the improvement of user failure rates,6–8 its use as a manage­ ment tool,9–10 and its use as an aid to im­ proving catalog design.11–13 Keyword searching is another area of discussion in the OPAC transaction log analysis literature. Joan M. Cherry re­ ported that changing a zero postings query to a keyword search was as fruit­ ful as, or more fruitful than, a new search constructed from cross-references pro­ vided by Library of Congress Subject Headings (LCSH).14 A later study by Joy Tillotson found that keyword searching was likely to yield more results than sub­ ject heading searching.15 Rosemary Thorne and Jo Bell Whitlatch determined that users attempting to locate informa­ tion on subjects should select keyword rather than subject heading access as their first access strategy.16 However, results of a recent survey and transaction log analy­ sis by Charles R. Hildreth found that fail­ ure often occurred with keyword search­ ing and that users usually had difficulty understanding Boolean searching.17 There is evidence in the literature that suggests screen changes can make a dif­ ference in the way users search the catalog and the extent of their success. Terry Ballard documented a change in the amount of keyword searching following a change in the position of the keyword search option on the introductory menu.18 Likewise, Jane Scott and colleagues found improvement in the success of both key­ word and subject searching when keyword searching was promoted to the top of all database menus on the OPAC screens.19 The previous study by the authors of the present study reported several areas in which screen changes appeared to in­ fluence users’ search behavior.20 One set of transaction logs in 1995 showed that OPAC users had difficulties with basic searching techniques. The authors de­ http:behavior.20 http:screens.19 http:searching.17 http:strategy.16 http:searching.15 http:LCSH).14 A Longitudinal Study of the Effects of OPAC Screen Changes 517 cided that the following changes to the OPAC introductory screens might allevi­ ate these problems: simplification and clarification of wording on introductory screens; use of the same publication for all search examples; use of a publication indexed by both of the controlled vocabu­ laries used at the libraries (LCSH and Medical Subject Headings); relocation of additional search examples to a second screen to reduce clutter on the first; and placement of the keyword option as the first choice on the list of search options. These screen changes were adopted and implemented in March 1996, and a sec­ ond set of transaction logs, using the same parameters as the first set, was run in April 1996. Data from the analysis of the second set showed statistically significant changes in several areas, including fewer errors in search formulation, a reduction in zero postings, and an increase in the use of keyword searching. Although short-term, limited transac­ tion log analysis studies are numerous, a review of the literature revealed few lon­ gitudinal or follow-up studies. Terry Ballard and Jim Smith’s eighteen-month transaction log analysis studied searcher errors and resulted in the addition of an OPAC search tips screen to aid searchers.21 Kathlin L. Ray and Mary S. Lang repli­ cated Ballard’s study in 1995 and 1996 to produce data comparing the results of transaction log analysis at two similar institutions.22 Stephen E. Wiberley Jr., Robert Allen Daugherty, and James A. Danowski conducted a follow-up inves­ tigation of user persistence in displaying online catalog postings.23 Nirmala S. Ban- galore focused on zero hits in her follow- up study of OPAC users.24 In 1996, Terry Ellen Ferl and Larry Millsap compared data collected from in-library users of the University of California’s catalog to an earlier study they had conducted in 1991 of users who accessed the catalog from remote sites.25 The research reported here represents an effort to add to the field’s knowledge of user behavior as tracked by OPAC transaction log analysis over an extended period of time. Methodology The authors previously documented that introductory screen changes had a posi­ tive impact on the searching behavior and success of OPAC users.26 Based on this encouraging outcome, the authors initi­ ated additional OPAC screen changes in the hope of further improving search be­ haviors and user success. Overall, four sets of transaction logs were analyzed over the course of four years to investi­ gate both the effects of a series of screen changes and whether initial improve­ ments in search success were sustained over time. Transaction logs were captured and analyzed at the middle of the semes­ ter following each set of screen changes. Midterm was chosen as a representative period because by this point students would have become familiar with the OPAC and daily use of the catalog would be sufficient to obtain a good sample. Each set of transaction logs was col­ lected from the eight busiest public ports over a four-day period. Public computers in the libraries were hardwired to the NOTIS system, and the first available port was engaged as users logged on, making it impossible to track use of a particular computer. Hardware changes in all eight libraries of the UIC system resulted in dif­ ferent user populations during the course of the longitudinal analysis. Whereas the first and second sets of transaction logs included users from all libraries (Main, Architecture and Art, Mathematics, Sci­ ence, Library of the Health Sciences at Chicago, Peoria, Rockford, Urbana), the third and fourth sets included only the Main Library and the regional medical sites at Peoria, Rockford, and Urbana. Be­ cause both user sets included undergradu­ ate, graduate, and professional students, the sets were considered comparable. All OPAC searches are recorded by the system and can be retrieved. The NOTIS transaction log provides useful informa­ tion on each transaction, including time of query, database name, catalog view mode selected by the user (brief or long view), search mode, validity of com­ mands issued, number of postings re­ http:users.26 http:sites.25 http:users.24 http:postings.23 http:institutions.22 http:searchers.21 FIGURE 1 Transaction Log Sample Time View Search Posting Resulting Screen Character String o uerv Database Mode* Mode** Received Screen Tvpe Number*** Input bv �ser 8:04:57 UI BR FIN K 0 NEF1K 111 K=BEAUT� PEGENT 8:05:12 UI BR FIN T 0 NEF1 T 111 T=THE PRISON 8:08:39 UI BR FIN T 30 T1I T=B�TE 8:08:41 UI BR DIS # B1R 111 S 1 8:09:18 UI BR FIN S 163 S1G S=MARI�UANA 8:11:37 UI BR MSG SRCH ARG LBC1INT 111 T 8:11:39 UI BR EXP T E1T 113 EXP1T 8:19:19 UI BR FIN S 0 NEF1S 111 S=RICHARD �RIGHT 8:19:33 UI BR FIN K 233 K1I K=RICHARD �RIGHT 8:27:43 UI BR FIN SM 0 NEF1SM 111 SM=PS�COPATH 8:28:41 UI LO INVLD CNTX LO1V 111 TART OVER 8:31:06 UI BR FIN C C1B C=HUM97011028 8:31:15 UI BR INVLD CMND BR1V 111 IN LIBRAR� ASSISTANCE PLEASE * View mode: BR (brief view) in NOnIS provides the user with the following information: author (if main entry), title, name of publisher, date of publication, location, call number, and circulation status. In addition to all the data in the brief view, LO (long view) provides the user all information contained in the bibliographic record with appropriate labels (e.g., added entries for authors have the following label: For other items by author(s) type A=<»). **Search mode (command issued): UICCAn users can query the database by title (t=), author (a=), LC subject (s=), medical subject (sm=), call number (c=), or �eyword (�=). *** In the above log, 111 represents screen 1 of a one-screen display. 518 C ollege & R esearch L ib raries N ovem b er 1999 A Longitudinal Study of the Effects of OPAC Screen Changes 519 FIGURE 2 Time Line of Longitudinal Study Action Date First set of transaction logs run General introductory screen redesigned Second set of transaction logs run Explain screens redesigned Specialized introductory screens redesigned Zero hits screens redesigned for author, title, subject Third set of transaction logs run Second redesign of general introductory screen Zero hits screen redesigned for keyword Fourth set of transaction logs run 10/30/95-11/2/95 3/22/96 4/16/96-4/19/96 5/20/96 5/20/96 5/20/96 10/28/96-10/31/96 7/27/97 1/19/98 4/13/98-4/16/98 turned, screen type, number of postings displayed, and the exact character string as input by the user. Figure 1 illustrates a sample transaction log. All four sets of transaction logs were captured using the same parameters and similar methodology. Each set provided both correct and incorrect OPAC usage. The following items were counted for each set of transaction logs: • number of transactions • number of search statements with correct syntax • number of correct search state­ ments resulting in zero postings • number of correct search state­ ments resulting in one to ninety-nine postings • number of correct search state­ ments resulting in more than one hun­ dred postings • number of explain commands • number of invalid commands • number of invalid context situa­ tions • number of search statements with no search argument • number of keyword searches • number of redirected searches for author, subject, and title. The correct syntax search statements that resulted in zero postings were fur­ ther analyzed. The total number of cor­ rect search statements resulting in zero postings was subdivided into the follow­ ing categories: incorrect spelling, subject wrong or not in catalog, title wrong or not in catalog, initial article included in title search, author wrong or not in catalog, author search done in incorrect order, exact retry of previous search, and unsuc­ cessful keyword search. It was expected that searching behavior would continue to improve as further screen changes were implemented, but this was not the case in every instance. A time line of when screen changes were implemented and when transaction log sets were collected is provided in figure 2. Initial revision of the basic OPAC introduc­ tory screen was followed by an improve­ ment in user searching behavior and an increase in the use of keyword searching from the first set of logs to the second.27 The basic introductory screen is used by the Main Library and the four sites of the Library of the Health Sciences. Based on the positive results, changes were subse­ quently made to the introductory screens used by specialized libraries within the University: the Architecture and Art, Math­ ematics, and Science libraries. Unfortu­ nately, the impact of the specialized screens could not be measured because these librar­ ies subsequently switched to a Web-based version of the OPAC that does not provide transaction logs. During the process of redesigning the introductory screens, online assistance http:second.27 520 College & Research Libraries November 1999 FIGURE 3 Explain Title (before 5/20/96) Explain Title Searching To search by title, type T= followed by as much of the title of which you are certain. Omit initial articles (A, AN, and THE). Examples: T=RED BADGE OF COUR You need not include the entire title. T=MAN FOR ALL SEASONS Omit initial articles. T=RED WHITE AND BLACK Omit all punctuation. Because of variations in titles, you may be presented with *Search Under... or *Search Also Under... cross-reference. You can redirect your search to these alternative forms by typing the line number. Remember: You may begin a new search at any time and from any screen. - - - - - - - - - - Page 1 of 2 - - - - - - - - - - - Hints for Title Searching 1. Omit initial articles, accent marks, and punctuation. 2. If the correct form of a word in a title is not known, try an alternative form of the word. Common variations are: -initials, acronyms, and abbreviations (abbreviated or spelled out); -numbers (in numeric form or spelled out); -hyphenations (one word or hyphenated); -spelling variations; -the word AND (spelled out or as '&'). 3. If you are unsure of the title and a search results in no matches: -Try shortening the title. -Try searching by author, subject, or keyword. (Help is available for each by typing EXP followed by A, S, or K.) For more help with TITLE SEARCHING, ask at the Reference or Information Desk. - - - - - - - - - - Page 2 of 2 - - - - - - - - - - - was improved by placing the “explain” command prominently on the new intro­ ductory screens, thereby promoting its use. This modification was followed by an increased use of the explain command. Explain screens for each search option (keyword, author, title, LCSH, MeSH, and call numbers) were then redesigned. In each case, an attempt was made to elimi­ nate jargon, use fewer words, present a less cluttered look, and give clear ex­ amples. As an example of explain screen changes, figures 3 and 4 give a before­ and-after picture of the explain title com­ mand. Finding no matches to a query can be very frustrating to OPAC users. To help al­ leviate this problem, whenever a UIC OPAC user encounters zero postings, he or she is automatically routed to a screen of helpful hints, a zero postings screen. However, the zero postings screens were crowded and difficult to read. To improve them, the task force made the text clearer, used fewer words, used bullets to highlight and separate each hint, and referred back to the explain commands for each search option. The revised zero postings screens for author, title, and subject searches went into effect in the spring of 1996. See, for example, the revised zero postings screen for title searching, found in figure 5. A third set of transaction logs was analyzed in the fall of 1996 to assess the effects of the new explain and zero postings screens. The modified zero postings screen for keyword searching was not implemented until the winter of 1998, so any effect of that change could not be assessed until the fourth set of logs was analyzed. A Longitudinal Study of the Effects of OPAC Screen Changes 521 The percentage of searches with author in incorrect order unexpectedly rose be­ tween the second and third sets of trans­ action logs, from 0.38 to 0.62 percent. Feedback from public services staff indi­ cated that the author example on the in­ troductory screen (“Preston, Richard”) was causing some confusion because it was not intuitively obvious that “Preston” was the author ’s last name. A new title was chosen for the general introductory screen with an author example of “Benson, Herbert.” Figures 6 and 7 illus­ trate the introductory screens for the Main Library and the Library of the Health Sci­ ences before and after the changes in the FIGURE 4 Explain Title (after 5/20/96) Introduction to Title Searching To search by title, type T= followed by as much of the title as you know. *Entire title not needed. Example: t=red badge of cour *Omit initial article. Example: t=hot zone *Cross-references direct a search to variations in the title. Type the line number to select the suggested title. -Search Under means use instead of. -Search Also Under means use in addition to. Press for more information on Title Searching. - - - - - - - - - - Page 1 of 3 - - - - - - - - - - - Hints for Title Searching *Omit initial articles, accent marks, and punctuation. *Try an alternate form of the word. Common alternatives are listed on the next screen. *If the search results in zero entries: -Shorten the title. -Retry as a keyword search (k=). *If search still results in zero entries, the library may not own the title. Check IUUINET for the holdings of other libraries. Ask for assistance at the Reference or Information Desk. Press for more information on Title Searching. - - - - - - - - - - Page 2 of 3 - - - - - - - - - - - Common Alternate Forms of Words The following are common alternate forms of words that may exist in the catalog. If search results in zero entries, try an alternate form of the word. *initialisms, acronyms, and abbreviations (abbreviated or spelled out); *numbers (in numeric form or spelled out); *hyphenations (one word or hyphenated); *spelling variations; *the word 'and' (spelled out or as '&'). Ask for further assistance at the Reference or Information Desk. You may begin a search from any screen. Press for more information on Title Searching. - - - - - - - - - - Page 3 of 3 - - - - - - - - - - - 522 College & Research Libraries November 1999 FIGURE 5 Zero Postings Screen for Title Searching (after 5/20/96) Search request: T= A HISTORY OF THE FILM UIC University Library Search results: 0 Entries Found No Title Entries Found No Title Entries Found Try the following: *Omit the initial article ("A", "An", "The" in English or any other language). *Check spelling. *Remove all punctuation. *Try keyword search (K=). If still unsuccessful, the library may not own anything with this title. Check ILLINET for the holdings of other libraries. For assistance, ask at the Reference or Information Desk. Type Review to revise search. For more information on Title Searching, type EXP and press . summer of 1997. A fourth set of transac­ tion logs run in the spring of 1998 was used to assess the effect of the newest general introductory screen. Results and Analysis The results of the analysis of the four sets of transaction log sampling are detailed in table 1. The table includes the total num­ ber of transactions, the total number of search statements, and the items that were expected to be influenced by the screen changes. The total number of transactions was a figure provided at the end of each transaction log, which included all patron inputs at the terminal: searches, invalid commands, navigational instructions, screen view changes, and exit commands. The total number of search statements was derived from the number of searches with correct syntax plus those searches that failed because the search arguments were missing. This number was the most accu­ rate indication of the total number of at­ tempted searches that did not require sec­ ond-guessing the patron’s intention based solely on the transaction log record. It is possible that some of the other error mes- FIGURE 6 Initial Revision of Introductory Screen (before 7/27/97) Welcome to UICCAT UICCAT is the computerized catalog of materials held by the UIC Libraries. You may begin a search from any screen. TO SEARCH BY: FOR EXAMPLE TYPE: TO GET HELP TYPE: Keyword k virus and animals exp k Author (last name first) a preston richard exp a Title (skip initial article) t hot zone exp t LC Subject Heading s ebola virus disease exp s Medical Subject Heading sm ebola virus exp sm Call number c rc140.5 exp c For more information on UICCAT, press . Press to exit. A Longitudinal Study of the Effects of OPAC Screen Changes 523 FIGURE 7 Second Revision of Introductory screen (after 7/27/97) Welcome to UICCAT UICCAT is the computerized catalog of materials held by the UIC Libraries. You may begin a search from any screen. TO SEARCH BY: FOR EXAMPLE TYPE: TO GET HELP TYPE: Keyword k health and stress exp k Author (last name first) a benson herbert exp a Title (skip initial article) t wellness book exp t LC Subject Heading s relaxation exp s Medical Subject Heading sm mental healing exp sm Call number c wa590 exp c For more information on UICCAT, press . Press to exit. sages that patrons received, such as “in­ valid command” and “invalid context,” resulted from a failed search attempt, but analysis of these searches proved difficult without the benefit of patron interviews. Data from the first and second set of logs have been reported previously but are included here because this longitudi­ nal analysis requires a retrospective dis­ cussion of all the sets. The previous pa­ per detailed statistically significant changes in the number of times each of the following occurred in the wake of the initial introductory screen changes: • use of the explain command • search statements with correct syn­ tax • search statements missing the search argument • searches resulting in zero postings • title searches that included an ini­ tial article • author searches in incorrect order • use of keyword searching. All of the changes were considered im­ provements in searching behavior. It was expected that searching behavior would continue to improve as further screen changes were implemented, but this was not the case in every instance. To test whether the changes between two sets of data were significant, a z-test for the equal­ ity between two proportions (binomial dis­ tribution) was used and a one-tailed analy­ sis was done. Although changes between transaction log sets one and two will be discussed, z-values will not be repeated here. All of the changes between sets one and two that are presented in table 1 were statistically significant to the .0005 level. Use of the explain command showed a marked increase between sets one and two, from 0.038 to 0.199 percent. Between sets two and three it decreased slightly, but the change was not statistically sig­ nificant (z = 1.45; p>.05). Between sets three and four, use of the explain com­ mand once again increased significantly, to 0.3 percent (z = -3.42, p<.0005). The change between sets two and four also was checked and it too was significant (z = -2.19, p<.025) to a lesser degree. Over­ all, use of the explain command increased over time as additional screen changes were implemented. The increase from set one to set four was almost tenfold. Not all improvements were sustained through the four sets of logs. The percent­ age of search statements with correct syn­ tax increased from 96.45 to 97.68 percent between sets one and two and then to 98.6 percent in set three (z = -2.99, p<.005). Set four showed a significant decrease (z = 6.04, p<.0005) from set three to 96.46 per­ cent, basically the percentage found in set one. 524 College & Research Libraries November 1999 TABLE 1 Transactions Analyzed First Set Second Set Third Set Fourth Set Total number of transactions Total number of explain commands Percentage of transactions that were explain commands (%) Total number of search statements Total number of correct syntax searches Percentage of search statements that were correct syntax searches (%) Total number of search statements missing the search argument Percentage of search statements missing the search argument (%) Total number of correct syntax searches resulting in zero postings Percentage of correct syntax searches resulting in zero postings (%) Total number of correct syntax searches with initial article included Percentage of correct syntax searches with initial article included (%) Total number of correct syntax searches with author in incorrect order Percentage of correct syntax searches with author in incorrect order (%) Total number of keyword searches Percentage of correct syntax searches that were keyword searches (%) 39421 15 0.038 7204 6948 96.45 256 3.55 2435 35.05 100 1.44 71 1.02 924 13.3 20585 41 0.199 4605 4498 97.68 107 2.32 1410 31.35 27 0.6 17 0.38 712 15.83 16178 22 0.136 3438 3390 98.6 48 1.4 1036 30.56 17 0.5 21 0.62 646 19.06 29276 88 0.3 5283 5096 96.46 187 3.54 1665 32.67 43 0.84 29 0.57 1442 28.3 The percentage of search statements that were missing the search argument demonstrated a similar pattern. Improve­ ments were found through the first three sets of logs, decreasing from 3.55 to 2.32 percent between one and two and 2.32 to 1.4 percent between two and three (z = 2.97, p<.005). However, the percentage in­ creased to 3.54 in set four (z = -6.04, p<.0005), once again basically the percent­ age found in set one. The percentage of correct syntax searches resulting in zero postings showed an improvement, decreasing between sets one and two, 35.05 to 31.35 percent. Between sets two and three there was a statistically insignifi­ cant decrease from 31.35 to 30.56 (z = 0.75, p>.10). The increase between sets three and four from 30.56 to 32.67 per­ cent was statistically significant (z = ­ 2.04, p<.025), but the difference be­ tween sets two and four was not (z = ­ 1.38, p>.05). The overall change be­ tween sets one and four was statisti­ cally significant (z = 2.72, p<.005), in­ dicating that although some of the ini­ tial improvements were lost over time, overall there was significant improve­ ment from the initial set of logs to the final set of logs. The percentage of correct syntax searches with the initial article (a, an, or the) included followed a similar pattern. Decreases were seen between sets one and two, 1.44 to 0.6 percent, and between sets two and three, 0.6 to 0.5 percent (not sig­ nificant, z = 0.59, p>.05), and then in set A Longitudinal Study of the Effects of OPAC Screen Changes 525 four the percentage increased to 0.84. Even though the change between sets two and three was not significant, the tests on the changes between sets three and four and sets two and four had different re­ sults due to the 0.1 percentage point dif­ ference. The change between sets two and four is not significant (z = -1.37, p>.05), but the change between sets three and four is (z = -1.83, p<.05). Overall, the dif­ ference between sets one and four is also significant (z = 3.0, p<.005), so that al­ though some improvement was lost with time, there was still significant improve­ ment after screen changes were initiated. The percentage of correct syntax searches with the author’s name in incor­ rect order also showed a significant im­ provement after screen changes were ini­ tiated, but again some of the improve­ ments were lost over time. Before screen changes, 1.02 percent of correct syntax searches had the author in incorrect order. This decreased to 0.38 percent in set two. Between sets two and three, the percent­ age increased from 0.38 to 0.62, not a sig­ nificant increase (z = -1.52, p>.05), and then decreased from 0.62 to 0.57 in set four, also not significant (z = 0.29, p>.05). However, the overall improvement from set one to set four was significant (z = 2.69, p<.005). The percentage of correct syntax searches that were keyword searches showed a statistically significant increase with each successive set of logs. Keyword searching increased from 13.3 percent in set one to 15.83 percent in set two. From set two to three, keyword searching in­ creased to 19.06 percent of searches (z = ­ 3.76, p<.0005). With set four keyword searching increased to 28.3 percent of searches (z = -9.68, p<.0005). Overall, key­ word searching more than doubled from set one to set four. Searches that resulted in zero postings were analyzed in detail, and the results are presented in table 2 and figure 8. It was expected that the screen changes would influence the percentage of searches with the subject wrong or not in catalog, initial article included in title search, author search done in incorrect order, and unsuc­ cessful keyword search. For these, the re­ sults from each set of logs were tested for TABLE 2 Reasons for Zero Postings First Set Second Set Third Set Fourth Set Total % Total % Total % Total % Incorrect spelling 259 10.64 129 9.15 94 9.07 191 11.47 Subject wrong or not in catalog 560 23.00 217 15.39 151 14.58 228 13.69 Title wrong or not in catalog 847 34.78 597 42.34 426 41.12 508 30.51 Initial article included in title search 100 4.11 27 1.91 17 1.64 43 2.58 Author wrong or not in catalog 204 8.38 211 14.96 110 10.62 213 12.79 Author search done in incorrect order 71 2.92 17 1.21 21 2.03 29 1.74 Exact retry of previous search 182 7.47 64 4.54 74 7.14 107 6.43 Unsuccessful keyword search 212 8.71 148 10.5 143 13.8 346 20.78 Total number of zero postings searches 2,435 1,410 1,036 1,665 526 College & Research Libraries November 1999 FIGURE 8 Reasons for Zero Postings 0 5 10 15 20 25 30 35 40 45 Incorrect spelling Subject wrong or not in catalog Title wrong or not in catalog Initial article included in title search Author wrong or not in catalog Author search done in incorrect order Exact retry of previous search Unsuccessful keyword search Reasons P er ce n ta g es 1st set % 2nd set % 3rd set % 4th set % statistical significance. Although the per­ centages of zero postings due to incorrect spelling, title or author wrong or not in catalog, and exact retries of previous searches also changed, the changes could not be attributed to the screen revisions except as artifacts of changes in other cat­ egories. Thus, they were not analyzed for statistical significance. Web interfaces do not require users to enter complicated command strings that use search statements and arguments. The percentage of zero postings due to subject wrong or not in catalog de­ clined in each successive set of transac­ tion logs, from 23 percent in set one to 13.69 percent in set four. The change be­ tween sets one and was statistically sig­ nificant, but not the changes between sets two and three (z = 0.55, p>.05) and sets three and four (z = 0.65, p>.05). The change between sets one and four was highly significant (z = 7.43, p<.0005), as was the overall improvement over time. This might be attributed to the emphasis on keyword searching in the revised screens. Although zero postings from subject searching declined, those due to unsuc­ cessful keyword searches showed a cor­ responding increase, which might be at­ tributed to the overall increase in key­ word searches. Unsuccessful keyword searches accounted for 8.71 percent of zero postings in the first set of logs, 10.5 percent in the second, 13.8 percent in the third, and 20.78 percent in the fourth. All of these increases were statistically sig­ nificant, from set one to set two, from two to three ( z=-2.49, p<.01), and from three to four ( z= -4.58, p<.0005). The percentage of times that a zero postings search was the result of includ­ ing an initial article in a title search showed a steady decrease over the first three sets of logs, followed by an increase in the fourth set. Initial articles in title searches were used in 4.11 percent of zero postings searches in the first set of logs, 1.91 percent in the second, 1.64 percent in the third, and 2.58 percent in the fourth. The change from set one to set two was significant. The change from sets two to three (z = 0.50, p>.05) and from sets three to four (z = -1.62, p>.05) were not statisti­ cally significant. The overall change from set one to four was significant (z = 2.62, A Longitudinal Study of the Effects of OPAC Screen Changes 527 p<.005), suggesting that perhaps the screen changes did help improve search­ ing behavior. The percentage of zero postings attrib­ utable to an author search done in the in­ correct order followed a similar pattern. The decline from set one to set two, 2.92 to 1.21 percent, was statistically signifi­ cant. This was followed by several changes that were not statistically signifi­ cantly different from one another. From set two to set three the percentage changed from 1.21 to 2.03 percent (z = ­ 1.62, p>.05), and from set three to set four the percentage changed from 2.03 to 1.74 percent (z=0.54, p>.05). The overall change from set one to set four was sig­ nificant (z = 2.42, p<.01), indicating that although the numbers fluctuated, there was overall improvement over time. Discussion The results of the first study showed that changes to the introductory screen had a positive effect on user searching behav­ ior and success, reducing some of the more common searching errors.28 Based on these findings, the task force hypoth­ esized that further improvements to screen design would enhance these posi­ tive results. To this end, the task force looked at improving the zero postings screens, explain screens, and the introduc­ tory screens for specialized libraries. Sub­ sequent analysis of transaction logs over four years revealed that the screen modi­ fications appear to have affected search behavior, but that not all effects persisted over time. Increased use of the explain command did not just persist over time but, rather, continued to increase throughout the four sets of logs. Use of the explain command increased almost tenfold between sets one and four (from 0.038 to 0.3 percent of transactions). This increase is important because it indicates that a previously underutilized instructional component of the OPAC is now being utilized. Whether use of the explain command increased the users’ ability to effectively search the cata­ log is difficult to determine solely from transaction log information. Additional research on this aspect could yield fur­ ther improvements to online help for OPACs. The transaction log analysis from the previous study supports the conclusion that screen improvements had a positive effect on how users formulate their search statements. Clarification of examples on the introductory screen seemed to help the searcher ’s ability to construct searches that included both a search argument and a search statement, essential elements of a successful search request. Unfortunately, the observed positive effect was not sus­ tained in subsequent transaction logs. The counts for both the number of correct syn­ tax searches and the number of searches missing the search argument rebounded to figures close to their original numbers. There is no way to know precisely why the earlier improvements decreased over time, but one possible explanation may have to do with the increase in Web-based search interfaces. During the time this lon­ gitudinal study was conducted, the uni­ versity library introduced more library bibliographic databases and catalogs with Web interfaces, including a Web version of the UIC OPAC. Web interfaces do not require users to enter complicated com­ mand strings that use search statements and arguments. Familiarity with this type of system could account for the regression of syntax and search argument errors to nearly the levels found in the first set of logs. The influence of Web interfaces also may have had an impact on the percent­ age of both correct syntax and zero postings searches that were title searches that included the initial article and author searches entered in the incorrect order. Web interfaces often allow users to enter titles with initial articles and author names in any order. However, with both of these search types, the positive changes did not totally regress over time; the dif­ ferences between the first and fourth sets were a statistically significant improve­ ment despite fluctuations in the numbers. This improvement might be attributed to http:errors.28 528 College & Research Libraries November 1999 the OPAC screen changes. Interestingly, the second revision of the introductory screen, which attempted to provide a clearer example of correct author order, did not result in a statistically significant improvement in author order errors from the first screen revision. Although there was some reversal in the decrease of the number of correct syn­ tax searches that resulted in zero postings in later logs, the overall decrease and im­ provement was significant. It is to be ex­ pected that these numbers may fluctuate depending on the searches being con­ ducted. Zero postings searches include searches for material not owned by the library as well as misspellings, an ongo­ ing problem in OPACs. Keyword searching was promoted on both the introductory and zero postings screens. Overall, keyword searches in­ creased from 13.3 to 28.3 percent over the series of transaction log sets. Because this study only looked at the transaction logs, it is impossible to know whether success­ ful keyword searches produced lists of materials relevant to the patron’s infor­ mation needs. Keyword searching did ac­ count for an increasing percentage of un­ successful searches. The percentage of zero postings from unsuccessful keyword searches increased in each successive set of transaction logs, whereas the percent­ age of zero postings due to a subject be­ ing wrong or not in the catalog declined in each successive set of transaction logs. However, the increase in zero postings keyword searches is greater than the de­ crease in zero postings subject searches. Zero postings subject searches fell from a high of 23 percent to a low of 13.69 per­ cent while zero postings keyword searches started at a low of 8.71 percent rising to a high of 20.78 percent. The re­ designed zero postings screens suggest ways to redesign searches to retrieve re­ sults, and one suggested method on all of them is to repeat the search as a key­ word search. Therefore, some unsuccess­ ful author and title searches may also have been retried as keyword searches, leading to the large keyword percentage. Conclusion Longitudinal study of transaction logs can assist librarians in improving OPAC interfaces and in assessing the long-term effects of screen changes. Patterns of user behavior become apparent during such a study. Librarians then can redesign OPAC screens to address certain behaviors that may cause zero postings. However, this study demonstrates that in some cases ini­ tial positive improvements in searching behavior possibly attributable to screen changes can regress over time, even back to the original levels. Although frustrat­ ing, this finding is important to the con­ tinued improvement of OPAC interface design, indicating that initial successes may be temporary and longitudinal analyses are needed to see which screen changes contribute to long-term behav­ ior changes. Why some improvements persist and others regress is ultimately not ascertainable from the transaction logs. The ever-changing population in an aca­ demic library setting may have exerted some influence over this longitudinal study. With a different population exist­ ing from year to year, it was impossible to study the same group of users in each of the four transaction log samples. John Leslie King and Kenneth L. Kraemer re­ viewed computers and communications technology and reported that “common predictions about the effects of using in­ formation technology frequently fail to materialize as expected. The failure of pre­ diction is not a signal that the outcome is negative. Rather it is a sign that the im­ pacts are richer and more complex than anticipated.”29 Even so, librarians should continue to alter catalog design in re­ sponse to what they learn from transac­ tion log analysis because this study shows that OPAC screen redesign may positively influence some search behaviors over the long term. Further research using the data col­ lected will focus on how instructional programs and aids can provide additional benefit to patrons and influence OPAC searching behavior. Analysis of common searching mistakes can assist librarians in A Longitudinal Study of the Effects of OPAC Screen Changes 529 the development of instructional units and tutorials to aid users in correcting un­ successful searching behavior. As libraries move toward Web-based catalog interfaces, lessons learned in this non-Web environment will aid in the de­ sign of appropriate help and introductory screens. Although some of the problems patrons currently experience searching the OPAC should disappear in the Web environment, close monitoring of logs will most likely uncover new difficulties in their place, if Web-based logs permit analysis of transactions. Unfortunately, current Web-based logs do not capture the types of data found in traditional OPAC transaction logs. If librarians wish to con­ tinue to use this unobtrusive and inexpen­ sive method of obtaining unbiased infor­ mation about how users navigate the cata­ log, they will need to work with Web cata­ log vendors to develop logs that provide the types of information needed for analy­ sis of patron search behavior. Notes 1. Lynn Silipigni Connaway, John M. Budd, and Thomas R. Kochtanek, “An Investigation of the Use of an Online Catalog: User Characteristics and Transaction Log Analysis,” Library Re­ sources & Technical Services 39 (Apr. 1995): 142–52. 2. Deborah D. Blecic, Nirmala S. Bangalore, Josephine L. Dorsch, Cynthia L. Henderson, Melissa H. Koenig, and Ann C. Weller, “Using Transaction Log Analysis to Improve OPAC Re­ trieval Results,” College & Research Libraries 59 (Jan. 1998): 39–50. 3. Andrew Large and Jamshid Beheshti, “OPACs: A Research Review,” Library & Information Science Research 19 (1997): 111–33. 4. Beth Sandore, “Applying the Results of Transaction Log Analysis,” Library Hi Tech 11 (1993): 87–97. 5. Thomas A. Peters, “The History and Development of Transaction Log Analysis,” Library Hi Tech 11 (1993): 41–66. 6. ———, “When Smart People Fail: An Analysis of the Transaction Log of an Online Public Access Catalog,” Journal of Academic Librarianship 15 (1989): 267–73. 7. Pauline A. Cochrane and Karen Markey, “Catalog Use Studies since the Introduction of Online Interactive Catalogs: Impact on Design for Subject Access,” in Redesign of Catalogs and Indexes for Improved Online Subject Access: Selected Papers of Pauline A. Cochrane (Phoenix, Ariz: Oryx Pr., 1985), 159–84. 8. Steven A. Zink, “Monitoring User Success through Transaction Log Analysis: The WolfPAC Example,” Reference Services Review 19 (spring 1991): 49–56. 9. Sandore, “Applying the Results of Transaction Log Analysis,” 92. 10. Peters, “When Smart People Fail,” 272. 11. Jane Scott, Jeffrey A. Trimble, and L. Fleming Fallon, “@&#@ This Computer and the Horse It Rode In On: Patron Frustration and Failure at the OPAC,” in Continuity and Transformation: The Promise of Confluence: Proceedings of the ACRL 7th National Conference (Chicago: ACRL, 1995), 247– 56. 12. Christine L. Borgman, “Why Are Online Catalogs Still Hard to Use?” Journal of the Ameri­ can Society for Information Science 47 (July 1996): 493–503. 13. Patricia M. Wallace, “How Do Patrons Search the Online Catalog When No One’s Look­ ing? Transaction Log Analysis and Implications for Bibliographic Instruction and Design,” RQ 33 (winter 1993): 239–52. 14. Joan M. Cherry, “Improving Subject Access in OPACs: An Exploratory Study of Conver­ sion of Users’ Queries,” Journal of Academic Librarianship 18 (May 1992): 95–99. 15. Joy Tillotson, “Is Keyword Searching the Answer?” College & Research Libraries 56 (May 1995): 199–206. 16. Rosemary Thorne and Jo Bell Whitlatch, “Patron Online Catalog Success,” College & Re­ search Libraries 55 (Nov. 1994): 479–497. 17. Charles R. Hildreth, “The Use and Understanding of Keyword Searching in a University Online Catalog,” Information Technology & Libraries 16 (June 1997): 52–62. 18. Terry Ballard, “Comparative Searching Styles of Patrons and Staff,” Library Resources and Technical Services 38 (1994): 293–305. 19. Scott, Trimble, and Fallon, “@&#@ This Computer and the Horse It Rode In On,” 255. 20. Blecic et al., “Using Transaction Log Analysis to Improve OPAC Retrieval Results,” 48. 21. Terry Ballard and Jim Smith, “The Human Interface: An Ongoing Study of OPAC Usage at Adelphi University,” in Advances in Online Public Access Catalogs 1, ed. Marsha Ra (Westport, 530 College & Research Libraries November 1999 Conn.: Meckler, 1992), 58–73. 22. Kathlin L. Ray and Mary S. Lang, “Analyzing Search Styles of Patrons and Staff: A Repli­ cative Study of Two University Libraries,” Library Resources & Technical Services 41 (July 1997): 219–35. 23. Stephen E. Wiberley Jr., Robert Allen Daugherty, and James A. Danowski, “User Persis­ tence in Displaying Online Catalog Postings: LUIS,” Library Resources & Technical Services 39 (July 1995 ): 247–64. 24. Nirmala S. Bangalore, “Mystery Names: Plausible Sounding Names Resulting in Zero Hits in OPACs,” Technical Services Quarterly 14 (1996): 15–31. 25. Terry Ellen Ferl and Larry Millsap, “The Knuckle-Cracker ’s Dilemma: A Transaction Log Study of OPAC Subject Searching,” Information Technology & Libraries 15 (June 1996): 81–98. 26. Blecic et al., “Using Transaction Log Analysis to Improve OPAC Retrieval Results”, 47. 27. Ibid., 45. 28. Ibid., 49. 29. John Leslie King and Kenneth L. Kraemer, “Computer and Communication Technologies: Impacts on the Organization of Enterprise and the Establishment and Maintenance of Civil Soci­ ety,” in Fostering Research on the Economic and Social Impacts of Information Technology (Washing­ ton, D.C.: National Academy Pr., 1998): 188–210.