Evidence Summary

Undergraduates Have Difficulty Distinguishing Formats Based on Discovery Tool Search Results

A Review of:

Gewirtz, S. R., Novak, M., & Parsons, J. (2014). Evaluating the intersection between WorldCat Local and student research. Journal of Web Librarianship, 8(2), 113-124. http://dx.doi.org/10.1080/19322909.2014.877312

Reviewed by:

Diana K. Wakimoto
Associate Librarian
California State University, East Bay
Hayward, California, United States of America
Email: diana.wakimoto@csueastbay.edu

Received: 7 Jan. 2015 Accepted: 20 Apr. 2015

2015 Wakimoto. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

Abstract

Objective – To evaluate the usability of WorldCat Local for undergraduate students conducting research.

Design – Usability study.

Setting – Two small, liberal arts colleges in central Minnesota.

Subjects – 13 undergraduates (7 females and 6 males).

Methods – To simulate an authentic research process, researchers created a thesis statement and a list of materials students needed to find using WorldCat Local. The students were video recorded and instructed to use the “think aloud” protocol as they worked through the list of materials to find. Researchers analyzed the recordings and evaluated the efficiency of the students’ searching processes using a rubric with scores from 1 to 5.

Main Results – Students were able to find books relevant to their topic, but had difficulty in identifying a book that their college library did not own. Students had more difficulty finding current scholarly journal articles and encyclopedias. Additionally, students had trouble distinguishing different formats in the results list.

Conclusion – The WorldCat Local results interface confused students, especially when they tried to determine the types of materials found (e.g., article, book, etc.). The students showed little understanding of relevance sorting and facets, although they did attempt to use them while searching. Despite the difficulties, the colleges will keep WorldCat Local as their discovery tool while exploring alternative options. The researchers suggest the need for future research to confirm their findings and determine what changes to the discovery tool interface would be most beneficial for the users.

Commentary

As the popularity of discovery services increases, usability studies become important. This study fits into the larger context of understanding how undergraduates use, or are confused by, result interfaces when conducting research.

The study adds to the growing literature on usability testing of discovery tools. The study is valid when evaluated with Glynn’s (2006) critical appraisal checklist. The sample size, though small, fits into the accepted practice of having between 10 and 20 users for usability testing (Emanuel, 2013). The strengths of the study include providing the research assignment given to the participants, acknowledging limitations of the study, and not overgeneralizing the results. The authors noted how they achieved inter-rater reliability using their created Efficiency Score Rubric, although the rubric may prove difficult for others to use given some vagueness in the definitions in the scoring categories. Including the average length of time that students took to complete or abandon the assigned tasks would strengthen the rubric.

A few clarifications in the methodology would have strengthened the study. There is no mention of a pilot study and it is unclear whether the “sample recordings” noted by the authors were considered part of the main study data set or excluded and simply used for norming purposes. There is also no explanation for including only 13 participants when over 300 volunteered to participate and the authors had planned for 16 participants. While the sample size is aligned with other usability studies, there is no justification for the number of participants or note of reaching a saturation point in data collection, as is a standard practice.

The authors make a convincing case for needing more authentic tests of students’ ability to effectively use discovery tools. Reviewing further literature specific to conducting usability studies with WorldCat Local (Boock, Chadwell, & Reese, 2009; Thomas & Buck, 2010), in addition to the literature cited, may have helped in designing methodology and in contextualizing results. Additionally, a more thorough literature review would have allowed the authors to compare their methodology with other WorldCat Local-specific evaluative methodologies, better supporting their stated outcome of designing an evaluative methodology for discovery systems.

While small in scale and scope, this study should give confidence to other librarians designing and executing their own usability testing. As discovery tools evolve and more libraries come to depend on them for a single, streamlined portal to resources, librarians should conduct usability testing and improve these services in ways that allow students to use the tools most effectively. Librarians can use the results to suggest local interface changes and to develop better training and instruction methods. In addition, discovery tool vendors can use usability test results to improve their products.

References

Boock, M., Chadwell, F., & Reese, T. (2009, March 27). WorldCat Local task force report to LAMP. Retrieved from http://hdl.handle.net/1957/11167

Emanuel, J. (2013). Usability testing in libraries: Methods, limitations, and implications. OCLC Systems & Services: International Digital Library Perspectives, 29(4), 204-217. http://dx.doi.org/10.1108/OCLC-02-2013-0009

Glynn, L. (2006). A critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154

Thomas, B., & Buck, S. (2010). OCLC’s WorldCat Local versus III’s WebPAC: Which interface is better at supporting common user tasks? Library Hi Tech, 28(4), 648-671. http://dx.doi.org/10.1108/07378831011096295