Evidence Summary

Secret Shopping is an Effective Tool for Identifying Local Patterns in Library User Experience

A Review of:

Boyce, C. M. (2015). Secret shopping as user experience assessment tool. Public Services Quarterly, 11(4), 237-253. doi:10.1080/15228959.2015.1084903

Reviewed by:

Kelley Wadson

Librarian

Bow Valley College

Calgary, Alberta, Canada

Email: kwadson@bowvalleycollege.ca

Received: 27 June 2016 Accepted: 19 Oct. 2016

2016 Wadson. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

Abstract

Objective – To assess library user experience (UX) at two entry-level service desks to determine the need for, and inform the aspects in which to improve, services and staff training.

Design – Observational study using secret shopping.

Setting – A small, private university in Illinois, United States of America.

Subjects – Library employees, comprised primarily of student assistants; and 11 secret shoppers, comprised of 5 faculty members, 4 staff members, and 2 first-year students from the university.

Methods – Recruitment methods for shoppers consisted of the campus electronic mail list, flyers, directed requests, and a $10 gift certificate to the campus bookstore following their participation. Both groups (library employees and secret shoppers) were briefed on the purpose of the study and completed informed consent forms. Shoppers attended face-to-face training sessions in which they selected two questions to ask from a list of options, one for each service desk in the library. Shoppers were not told at which desk to ask their questions. The list of options included informational and research assistance questions; shoppers also had the option of asking a question based on an actual information need. Two service desks were involved: one for circulation and one for research/IT support. Since IT staff and Librarians were excluded from the study, shoppers were directed to tactfully end the transaction if a referral to an expert was made.

Within two weeks of the training session, shoppers made two separate visits to the library at a time convenient to them to ask the question and observe the transaction at each of the two service desks. For each round of secret shopping, shoppers completed an electronic evaluation form afterward on the Qualtrics platform. The evaluation form consisted of yes/no, multiple-choice, and open-ended comment questions with two questions employing skip logic for a total of 29 possible questions. The questions covered the following variables both quantitatively and qualitatively: how well the question was answered, the customer service skills (responsiveness, approachability, and respectfulness) of the library employee, and if applicable, the quality of the referral to other library staff or services.

Main Results – The shoppers evaluated a total of 21 transactions: 11 for the circulation desk and 10 for the research/IT support desk (1 shopper did not evaluate this desk). Eighteen of the transactions were in-person and three were by phone. Eight of the questions asked were based on the participants’ actual information need.

On the variable of satisfaction with the answer received, the research/IT support desk scored higher than the circulation desk. The circulation desk received 7 very satisfactory ratings, 3 satisfied ratings, and 1 neutral rating; whereas the research/IT support desk received 10 very satisfactory ratings and 1 satisfied rating. The lower scores of the circulation desk may be related to the variables of responsiveness and approachability, as library employees on the circulation desk were scored lower in these areas and observed as not paying attention in two interactions and this was not observed at all at the research/IT support desk. However, the study did not collect sufficient data to test this relationship. All shoppers gave positive ratings on whether they were treated respectfully and if the library employee waited for the shopper to state the question fully and with the exception of one transaction, clarify the question if necessary.

Responses to the open-ended comment questions were reviewed by investigators, who found that in five cases the transaction would have been improved by consulting library faculty. On the variables related to customer service, responses were generally positive but in several transactions the library employees failed to appear attentive and ready to help the shopper.

Conclusion – The author found secret shopping was an effective tool for evaluating library UX to identify both positive and negative patterns and better inform responses to areas in need of improvement. The author identified two areas for improvement to the library. First, library employees at the circulation desk require additional training that would encourage them to refer transactions to library faculty where necessary. Second, although evaluation of customer service skills was generally positive, library employees will also receive additional training that will emphasize listening and role-playing scenarios. These areas for improvement will also support the library’s plans to combine research and circulation functions into a single service desk on the entry level and move the IT support desk to the third floor.

Commentary

Secret shopping is a form of participant observation for evaluating customer service that has been widely used in the retail sector (Zorica, Ivanjko & Spiranec, 2014). The author notes that libraries have been employing secret shopping since 1970, but mainly in public libraries. The article cites four studies of academic libraries as of 2013 and seven studies of public libraries. This study thus contributes to the growing discussion and application of secret shopping in academic libraries, which includes more recent publications by Crowe and Bradshaw (2015) and Zorica, Ivanjko, and Spiranec (2014).

Using Glynn’s EBL Critical Appraisal Checklist (2006), the study’s limitations are related most significantly to population and, to a lesser extent, data collection and results. The study aimed to recruit five participants from each of the university’s main stakeholder groups: faculty, staff, and students. The actual total figure was 11 participants, only 2 of them students. The author noted that comments from faculty and staff were richer than those from the students. The study thus exhibits selection bias, whereby the sample was not fully representative of the entire population. For data collection, the author did not state at what point after the transaction participants completed the evaluation form. Two weeks was allotted for both rounds of secret shopping, but it is unclear whether the participant completed the form immediately or waited several hours or possibly longer. There was thus a risk of recall bias, in that the data relied on the shopper’s memory of the event rather than the actual event itself.

Unfortunately, the results have low external validity as the sample size was too small for generalizable conclusions beyond a local context. However, this highlights two strengths of the study. Firstly, the study exhibits strong ecological validity as it was conducted in a natural setting with actual patrons who were encouraged to present a real information need by selecting their own question. Secondly, the author used trends identified through previous research, namely campus surveys, to inform and enhance the precision of the evaluation form.

The implications for practitioners are related to the study’s methodology rather than its results. As UX and service delivery become increasingly important to academic libraries (Bell, 2014), this study adds to the research applying secret shopping as an alternative to what some have claimed is an over-dependence on surveys in LIS research (Halpern, Eaker, Jackson, & Bouquin, 2015). In sum, secret shopping is an effective tool for exposing or verifying local patterns in library UX. For more reliable results, it should be combined with other methods or designed to avoid this study’s limitations by seeking a larger sample size or requiring participants to complete multiple rounds at specified time intervals, for example.

References

Bell, S. J. (2014) Staying true to the core: Designing the future academic library experience. portal: Libraries and the Academy, 14(3), 369–382. doi: 10.1353/pla.2014.0021

Crowe, K., & Bradshaw, A. K. (2016). Taking a page from retail: Secret shopping for academic libraries. Evidence Based Library & Information Practice, 11(1), 40-56. doi:10.18438/B85S6H

Glynn, L. (2006). A critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387-399. doi: 10.1108/07378830610692154

Halpern, R., Eaker, C., Jackson, J., & Bouquin, D. (2015). #DitchTheSurvey: Expanding methodological diversity in LIS Research. In the Library with the Lead Pipe, 1-13. Retrieved from http://www.inthelibrarywiththeleadpipe.org

Zorica, M. B., Ivanjko, T., & Spiranec, S. (2014). Mystery shopping in libraries - Are we ready? Qualitative & Quantitative Methods in Libraries, 2, 433-442. Retrieved from http://www.qqml.net/