perkins.p65 A Comparison of Satisfaction Survey Results 369 A Comparison of Web-based and Paper-and-Pencil Library Satisfaction Survey Results Gay Helen Perkins and Haiwang Yuan Although authors have done validation or comparison studies of Web- based and paper-and-pencil surveys for different samples, few have published such studies for library patrons. After publishing its previous Web-based library satisfaction survey, Western Kentucky University Li­ braries developed a similar survey with identical content for library Web and exit patrons to compare these groups’ responses. This article fo­ cuses on the collection of Web and exit survey responses in a two-week period, the transformation of response data for analysis, a comparison of the two samples, and discussion of the potential use of the results. Future research on the Web-based and paper-and-pencil methods them­ selves is indicated. ith the advent of the Internet ity of looking at, or interacting with, pas- and the World Wide Web, re- sive data; convenient time and location searchers have innovated the for subjects; anonymity for researchers use of Internet surveys, in- and subjects; minimization of research­ cluding Web-based surveys, and have cited many considerations for their use. For example, authors such as Michael A. Smith and Brant Leigh as well as Robert N. Davis have suggested several advan­ tages and cautions.1, 2 The advantages of such surveys included accessibility to ei­ ther large sample sizes or specialized sample groups; the ability for subjects to interact with multimedia; the availabil­ ers’ time and resources; automatic trans­ formation of subject responses for data analysis; and the opportunity to answer new research questions about a unique Web culture. However, Internet surveys, including Web-based surveys, also have brought responsibilities such as the need for informed consent, easy withdrawal from the study, data security; generali­ zation of the samples to an entire popu- Gay Helen Perkins is the Business Librarian at Western Kentucky University Libraries; e-mail: Gay.Perkins@wku.edu. Haiwang Yuan is the Website and Virtual Library Coordinator at Western Ken­ tucky University Libraries and Museum; e-mail: Haiwang.Yuan@wku.edu. The authors thank Bob Cobb, Director, Western Kentucky University Institutional Research, and Tuesdi Helbig, Research Associate, Institutional Research, for originating the idea for this paper and for consultation on the research design and data analysis. Thanks also go to WKU Libraries’ Satisfaction Survey Committee members Bryan Carson and Beth Knight for their help with the survey design and other participation; Dr. Michael Binder, Dean of WKU Libraries and Museum, and Dr. Brian Coutts, Head of WKU Department of Library Public Services, for making available the use of tremendous library resources; WKU Libraries’ faculty, staff, and students for help with administering the paper-and-pencil survey; Sarah Davasher for editing and proofing the manuscript; and John Sullivan, who assisted in file transfers. 369 mailto:Haiwang.Yuan@wku.edu mailto:Gay.Perkins@wku.edu 370 College & Research Libraries July 2001 lation, and validation or other compari­ son studies that address the possibility of subjects being more heterogeneous than typical paper-and-pencil recruits. The loss of control over the testing envi­ ronment and multiple data entries by frustrated or mischievous subjects also need to be addressed. Several authors have compared the results of Web-based surveys with those of paper-and-pencil surveys for psycho­ logical research. In general, such studies compared the responses of somewhat similar samples of psychology students and self-selected respondents. Compari­ sons of the two methods have indicated similar responses and/or similar inter­ nal consistency. For example, in 1997, John H. Krantz, Jody Ballard, and Jody Scher compared laboratory responses of psychology students to an experiment on the determinants of female attractiveness with Web responses.3 Web individuals re­ sponded to postings of the experiment on the Hanover College Psychology De­ partment home page or to the American Psychological Society’s online research Web site. Both correlational and regres­ sion analyses suggested validity of the Web-based studies. In a series of three studies in 1998, Karen A. Pasveer and John H. Ellard compared paper-and-pen­ cil responses of two samples of univer­ sity undergraduates with two samples of Web-based responses.4 Web-based re­ spondents were recruited via e-mail from a membership directory of the Interna­ tional Network of Personal Relation­ ships, or the International Society for the Study of Personal Relationships, and from respondents to links with the American Psychological Society and various search engines. The question­ naire was a new self-trust instrument. Both descriptive and psychometric analyses suggested similar results for the two sources of samples; however, there was more variance in the Web-based scale scores. In 1999, Tom Buchanan and John L. Smith found similar psychomet­ ric characteristics for the Self-Monitoring Scale (revised) in a large-scale compari­ son between paper-and-pencil and Web- based responses.5 In a second study later that year, they identified and compared the handles, or screen names, of two groups of high self-monitoring and low self-monitoring Usenet Newsgroups.6 In a related study, newsgroup participants from the high self-monitoring group scored higher on a Web-based survey, the Self-Monitoring Scale (revised), suggest­ ing construct validity on the Web test. Finally, also in 1999, Davis compared pa­ per-and-pencil questionnaire responses from two groups of psychology students and one group of nonpsychology stu­ dents with responses from a Web sample recruited from university flyers that listed a Web site address for filling out a questionnaire.7 The survey was the Ru­ minative Response Scale. Scores of self- focused rumination were somewhat higher for the Web sample, but internal consistency of the instrument was simi­ lar across the two samples. In 1998, Jeffrey M. Stanton compared the paper-and-pencil survey responses of professional employees at sixteen orga­ nizations with responses from Web sur­ veys filled out by individuals from twenty organizations who were con­ tacted by e-mail.8 The survey concerned employee perception of supervisor fair­ ness. There was a small, but significant, difference in missing values, and there were similar internal covariance patterns across the two samples. Comparison of Web-based and Paper-and-Pencil Satisfaction Surveys at Western Kentucky University Libraries Participating in the proliferation of Web- based satisfaction surveys in academic libraries, Western Kentucky University (WKU) Libraries developed and pub­ lished a Web-based library instrument in 2000 to study patron satisfaction with the libraries’ resources and services.9 The advantages of Web-based surveys al­ ready have been cited, not the least of which is ease of administration, data col­ lection, and data analysis.10 However, http:analysis.10 A Comparison of Satisfaction Survey Results 371 generalization to an entire population was a concern, and there was no valida­ tion or other comparison of Web-based and paper-and-pencil surveys in the li­ brary literature. Thus, the WKU Librar­ ies’ Satisfaction Survey Committee de­ veloped a single instrument to compare Web-based and paper-and-pencil library satisfaction survey results, approved by the Western Kentucky University Hu­ man Subjects Review Board, Office of Sponsored Programs. The purposes of this satisfaction survey were as follows: • to compare descriptions of samples of patrons who access WKU Libraries’ home page and patrons who exit the Main Library; • to compare how satisfied these two groups are with WKU Libraries’ elec­ tronic and printed resources as well as the services of library faculty and staff ; • to compare these two groups’ gen­ eral comments about WKU Libraries; • to possibly use the survey data to promote use of Web-based surveys in li­ braries; • to possibly use the survey data to effect improvement. Administration of WKU Libraries' Satisfaction Survey Two groups of individuals were sampled in this study: WKU Libraries’ Web re­ spondents and its Main Library exit re­ spondents. Both groups took the survey and were sampled during the last two weeks of February 2000. The Web respon­ dents accessed the survey from WKU Li­ braries’ home page, which used JavaScript to pop up a window before the page was launched asking users to choose to either take the survey or by­ pass it. A total of 458 survey responses were registered out of 4,554 counts of ac­ cessing WKU Libraries’ home page, a 10.06 percent response rate. The survey is included here as figure 1. Like the first WKU Libraries’ Web- based satisfaction survey, the inexpen­ sive and user-friendly client PC software, Message Parse, was used to abstract the needed data from a designated e-mail ac­ count that had received all the survey re­ sponses sent from a Web form filled out by respondents. Quantitative and quali­ tative data were collected separately in prescribed tabulated ASCII text files. The files then were filtered through MS Ex­ cel to become spreadsheets exportable to package analysis programs such as SPSS or SAS.11 Needless to say, gathering data from paper-and-pencil questionnaires was a nightmare. The paper-and-pencil survey was ad­ ministered during the open hours of the Main Library on Monday, February 14; Thursday, February 17; Sunday, Febru­ ary 20; and Friday, February 25. The four sampling days included a holiday (Valentine’s Day), two weekdays, and a weekend day, a combination that would yield a sampling size close to those that could have been obtained through a ran­ dom sampling. Unlike the Web-based survey, the paper-and-pencil survey re­ quired tremendous library resources. Apart from the cost of printing five hun­ dred copies of professional question­ naires, fifty-six hours were needed just to get the questionnaires filled out at the gate of the Main Library during the four days of the exit survey. With the support of the library administration, faculty, and staff, teams were formed consisting of one WKU Libraries’ Satisfaction Survey Committee member paired with a library faculty or staff member or a student as­ sistant. The team offered a survey to ev­ ery tenth individual exiting the Main Library. During the four days of the sur­ vey, 366 survey responses were collected out of a gate count of 4,831, registering a response rate of 7.58 percent. This per­ centage was lower than 10 percent be­ cause library faculty and staff, library student assistants on duty, repeat pa­ trons, and patrons who did not want to participate were excluded. After data collection, data analyses were initiated, with the following as their goals: 372 College & Research Libraries July 2001 FIGURE 1 WKU Libraries' Satisfaction Survey A Comparison of Satisfaction Survey Results 373 TABLE 1 Academic Status of Paper-and-Pencil and Web-based Survey Respondents Paper-and-Pencil Survey Web-based Survey Academic Status Frequency % Frequency % WKU faculty 8 2.20 34 7.42 WKU staff 6 1.65 23 5.02 Undergraduate students 294 80.77 309 67.47 Graduate students 34 9.34 63 13.76 Other 22 6.04 29 6.33 Total 364 100.00 458 100.00 • to describe the responses of the Web-based library satisfaction surveys; • to describe the responses of the pa­ per-and-pencil exit library satisfaction surveys; • to compare the responses of the Web-based and paper-and-pencil exit li­ brary satisfaction surveys. Needless to say, gathering data from paper-and-pencil questionnaires was a nightmare. Although quantitative data can be machine-scanned at the WKU Of­ fice of the Institutional Research, record­ ing data from the three open-ended ques­ tions—needed to access users’ responses and suggestions for implementing im­ provement of WKU Libraries’ resources and services—required a good deal of la­ bor and time. It took a full-time summer student assistant twenty days to key in the data character by character onto a similar spreadsheet format to be compa­ rable with the Web-based survey. None­ theless, all the time and labor would be worthwhile to compare Web-based and library exit survey re­ sults. = 294) of the paper-and-pencil survey respondents were undergraduate stu­ dents; 14 percent (n = 63) of the Web- based survey respondents and 9 percent (n = 34) of the paper-and-pencil survey respondents were graduate students; 6 percent (n = 29) of the Web-based sur­ vey respondents and 6 percent (n = 22) of the paper-and-pencil survey respon­ dents were in the “Other” classification; 7 percent (n = 34) of the Web-based sur­ vey respondents and 2 percent (n = 8) of the paper-and-pencil survey respondents were WKU faculty members; and 5 per­ cent (n = 23) of the Web-based survey re­ spondents and 2 percent (n = 6) of the paper-and-pencil survey respondents were WKU staff. At the time of taking the survey, 59 percent (n = 270) of the Web-based sur­ vey respondents and, of course, 100 per­ cent (n = 364) of the paper-and-pencil survey respondents were located at the WKU Bowling Green, Kentucky Cam­ pus, South Campus, or Residence Halls. TABLE 2 Comparison of Paper-and-Pencil and Web-basedSurvey Results Survey Mean Responses to Item 4, Frequency of UseResults indicated that of WKU Libraries' Electronic Resourcesboth the Web-based and paper-and-pencil urvey Method X X survey respondents comprised somewhat Paper-and-pencil 3.42 1.15 .34 365 813 4.41** similar groups (table Web-based 3.08 1.10 450 1). Sixty-seven percent Note: The rating scale for this item was: 1 = Once a year or less, (n = 309) of the Web­ 2 = A few times a year, 3 = Monthly, 4 = Weekly, 5 = Daily. based survey respon­ **p < .001 dents and 81 percent (n 374 College & Research Libraries July 2001 were classified based TABLE 3 on the following crite-Comparison of Paper-and-Pencil and Web-based ria: item means in theSurvey Mean Responses to Item 5, Frequency of Use range 3.5–5.0 were con-of WKU Libraries' Printed Resources sidered positive, item means in the range 2.5– urvey Method X X_ _ 3.5 were considered Paper-and-pencil 2.97 .94 .26 365 808 3.68** neutral, and itemWeb-based 2.71 1.08 449 means in the range 1.0– 2.5 were considered Note: The rating scale for this item was: I = Once a year or less, 2 = negative.A few times a year, 3 = Monthly, 4 = Weekly, 5 = Daily. Comparison of the**p < .00I two groups’ mean re­ sponses indicated Nineteen percent (n = 88) of the Web- small, but significant, differences for five based survey respondents were located of the six items; given the large sample at noncampus locations other than Bowl- sizes, significance was not surprising. For ing Green, Kentucky; 16 percent (n = 72) each of these items, the Web-based survey at noncampus locations in Bowling group had a somewhat lower mean than Green, Kentucky; and 6 percent (n = 28) that of the paper-and-pencil survey group. at WKU Extended Campus (Glasgow, On average, Web-based survey respon- Owensboro, Elizabethtown, and others). dents felt that they used WKU Libraries’ electronic resources “monthly” (X = 3.08, A review of this study’s results S.D. = 1.10) and paper-and-pencil survey suggested very minor, but signifi- respondents felt that they used WKU Li­ cant, differences in the item means braries’ electronic resources “monthly” (X for the Web-based and paper-and­ =3.42, S.D.=1.15) (table 2). The mean scores pencil methods. of these respondents differed by .34, a small, but significant, difference. Web- Finally, 67 percent (n = 305) of the Web- based survey respondents felt that they based survey respondents and 69 percent used WKU Libraries’ printed resources of the paper-and-pencil survey respon- “monthly” (X =2.71, S.D.=1.08) and paper- dents had attended a library orientation and-pencil survey respondents felt that session such as a class or workshop. they used WKU Libraries’ printed re­ Thirty-three percent (n = 153) of the Web- sources “monthly” (X =2.97, S.D.=.94) based survey respondents and 31 percent (table 3). The mean scores of these respon­ of the paper-and-pencil survey respon- dents differed by .26, a small, but signifi­ dents had not attended such a class. cant, difference. Web-based survey respon- Tables 2 through 7 summarize the com- TABLE 4 parison of the Web- Comparison of Paper-and-Pencil and Web-based based and paper-and- Survey Mean Responses to Item 7, Helpfulness ofpencil survey respon- WKU Library Personneldents to the six Likert­ scaled items. Because urvey Method X X the choice of response alternatives for each of Paper-and-pencil 4.28 1.01 .35 366 805 4.61** Web-based 3.93 1.14 446six items was distinct, each item was inter- Note: The rating scale for this item was: I = Never, 2 = Occasionally,preted on an individual 3 = Neutral, 4 = Moderately, 5 = Very, 6 = Not applicable. basis and with an indi­ **p < .00I vidual table. Responses http:S.D.=.94 http:S.D.=1.08 http:S.D.=1.15 A Comparison of Satisfaction Survey Results 375 dents thought that WKU Libraries person­ nel were “moderately” helpful (X =3.93, S.D.=1.14) and paper­ and-pencil survey re­ spondents thought that the WKU Libraries per­ sonnel were “moder­ ately” helpful (X =4.28, S.D.=.1.01) (table 4). The mean scores of these re­ spondents differed by .35, a small, but signifi­ cant, difference. Web- based survey respon­ dents felt that the WKU TABLE 5 Comparison of Paper-and-Pencil and Web-based Survey Mean Responses to Item 8, Adequacy of WKU Libraries' Collections in Meeting Information Needs Survey Method X S.D. X_ _ N dF t Paper-and-pencil 3.94 .87 .45 355 797 6.44** Web-based 3.49 1.09 444 Note: The rating scale for this item was: 1 = Not adequate, 2 = Occasionally adequate, 3 = Neutral, 4 = Moderately adequate, 5 = Very adequate. **p < .001 Libraries’ collections were “neutral” (X =3.49, S.D.=1.09) in meeting their informa­ tion needs and paper-and-pencil survey respondents felt that WKU Libraries’ col­ lections were “moderately adequate” (X =3.94, S.D.=0.87) in meeting their informa­ tion needs (table 5). The mean scores of these respondents differed by .45, a small, but significant, difference. Web-based sur­ vey respondents felt that they would rate the WKU Libraries overall as “good” (X =3.61, S.D.=0.89) and paper-and-pencil survey respondents felt that they would rate the WKU Libraries overall as “good” (X =4.09, S.D.=.69) (table 6). The mean scores of the respondents differed by .48, a small, but significant, difference. Comparison of the two groups’ mean responses indicated no significant differ­ ence for one item that concerned TOPCAT 2000, the WKU Libraries’ online catalog. TABLE 6 Web-based survey respondents rated TOPCAT 2000’s ease of use as “easy” (X =3.76, S.D.=.1.09); and paper-and-pencil survey respondents rated TOPCAT 2000’s ease of use as “easy” (X =3.87, S.D.=.1.06) (table 7). The mean scores of these respon­ dents differed by .11, a difference that was not significant. In addition, analysis of variance sta­ tistics compared Web-based and paper­ and-pencil survey variances for each of the six Likert-scaled items. These statis­ tics were computed on each of four samples: only undergraduates, only WKU Bowling Green respondents, only respondents who had attended library orientation sessions, and only respon­ dents who met all three requirements. In general, there were small, but significant, statistical differences for 22/24 of the comparisons between the two survey methods, even when type of respondent was controlled. Further in-Comparison of Paper-and-Pencil and Web-based formation is beyondSurvey Mean Responses to Item 11, Rating of the the scope of this articleWKU Libraries Overall but is available on re­ quest.Survey Method X S.D. X_ _ N dF t Web-based survey Paper-and-pencil 4.09 .69 .48 358 791 8.51** responses and paper­Web-based 3.61 .89 437 and-pencil survey re­ sponses to each of theNote: The rating scale for this item was: 1 = Poor, 2 = Fair, 3 = three open-ended items Average, 4 = Good, 5 = Excellent. also were compared. **p < .001 Each open-ended re­ http:S.D.=.1.06 http:S.D.=.1.09 http:S.D.=.69 http:S.D.=0.89 http:S.D.=0.87 http:S.D.=1.09 http:S.D.=.1.01 http:S.D.=1.14 376 College & Research Libraries July 2001 TABLE 7 Comparison of Paper-and-Pencil and Web-based Survey Mean Responses to Item 6, TOPCAT 2000's Ease of Use Survey Method x S.D. x_ _ N dF t Paper-and-pencil 3.87 1.06 .11 365 814 1.39 Web-based 3.76 1.09 451 Note: The rating scale for this item was: I = Very Hard, 2 = Hard, 3 = Neutral, 4 = Easy, 5 = Very Easy, 6 = Not applicable. in the item means for the Web-based and pa­ per-and-pencil meth­ ods. Somewhat similar differences also were there, even when type of respondent was con­ t rolled. Thus, both groups of respondents comprised somewhat similar patrons who were viewing the li­ sponse was assigned a category, and fre­ quencies of category by method and item number also were reviewed. A few num­ bers of categories differed sizably between the Web-based and the paper-and-pencil survey responses. Differences seemed to relate to the physical premises of the Main Library: 2.84 percent (n = 13) of the Web- based survey responses and 15.03 percent (n = 55) of the paper-and-pencil survey re­ sponses to item ten commented on “com­ puters/computer labs”; and 2.18 percent (n = 10) of the Web-based survey responses and 14.75 percent (n = 54) of the paper­ and-pencil survey responses for this item commented on “study area/quiet/meet­ ing area/seating.” The most interesting difference in Web- based and paper-and-pencil survey open- ended responses concerned the “None/ Blank/Don’t Know/Undecided/Unclear” response. For each of the three open-ended questions, there was a greater percentage of these responses for the Web-based sur­ vey respondents. In other words, a higher percentage of Web-based survey respon­ dents made “no” or related responses than did the paper-and-pencil survey respon­ dents for each of the three open-ended items. This may be attributed to the fact that respondents who used the Web might feel more independent in front of their computers than those who had to face the researchers at the library exit door. Administrative Use of Results A review of this study’s results suggested very minor, but significant, differences brary in like fashion. These results suggest that it is the method, and not the instru­ ment or demographics, that may be re­ lated to the small differences. Given the above results, as well as the previously mentioned ease in adminis­ tration, data collection, and data analy­ sis, the Web-based library satisfaction survey may be used to evaluate attitudes of remote patrons in an existing or vir­ tual library.12 The data require only in­ expensive software and minor labor for analysis. Future research needs to focus on the Web-based versus paper-and-pen­ cil method and possibly additional types of demographics as the cause of minor differences in responses. Summary This study provided a comparison of the survey responses of library Web patrons and library exit patrons. In a two-week period, 458 Web-based survey respon­ dents and 366 paper-and-pencil survey respondents took WKU Libraries’ satis­ faction survey. The response data were transformed and analyzed. Results sug­ gested small, but significant, differences in item means, as well as similar demo­ graphics for the two groups of library pa­ trons. Because the differences may be attributable to the large samples of re­ spondents, they still warrant Web-based surveys as an alternative to the paper­ and-pencil surveys in the library envi­ ronment. Future research could focus on the Web-based and paper-and-pencil sur­ vey methods themselves to further evalu­ ate these minor differences. http:library.12 A Comparison of Satisfaction Survey Results 377 Notes 1. Michael Smith and Brant Leigh, “Virtual Subjects: Using the Internet as an Alternative Source of Subjects and Research Environment,” Behavior Research Methods, Instruments, & Com­ puters 29, no. 4 (1997): 496–505. 2. Robert N. Davis, “Web-based Administration of a Personality Questionnaire: Compari­ son with Traditional Methods,” Behavior Research Methods, Instruments, & Computers 31, no. 4 (1999): 572–77. 3. John H. Krantz, Jody Ballard, and Jody Scher, “Comparing the Results of Laboratory and World Wide Web Samples on the Determinants of Female Attractiveness,” Behavior Re­ search Methods, Instruments, & Computers 29, no. 2 (1997): 264–69. 4. Karen A. Pasveer and John H. Ellard, “The Making of a Personality Inventory: Help from the WWW, Behavior Research Methods, Instruments, & Computers 30, no. 2 (1998): 309–13. 5. Tom Buchanan and John L. Smith, “Using the Internet for Psychological Research: Per­ sonality Testing on the World Wide Web,” British Journal of Psychology 90, part 1 (Feb. 1999): 125–44. 6. ———, “Research on the Internet: Validation of a World Wide Web Mediated Personal­ ity Scale,” Behavior Research, Methods, Instruments, & Computers 31, no. 4 (1999): 565–71. 7. Davis, “Web-based Administration of a Personality Questionnaire.” 8. Jeffrey M. Stanton, “An Empirical Assessment of Data Collection Using the Internet,” Personnel Psychology 51 (1998): 709–25. 9. Gay Helen Perkins and Haiwang Yuan, “Genesis of a Web-based Satisfaction Survey in an Academic Library: The Western Kentucky University Libraries’ Experience,” Library Ad­ ministration & Management 14, no. 3 (summer 2000): 159–66. 10. Davis, “Web-based Administration of a Personality Questionnaire.” 11. Perkins and Yuan, “Genesis of a Web-based Satisfaction Survey in an Academic Library.” 12. Davis, “Web-based Administration of a Personality Questionnaire.”