Bento Box User Experience Study at Franklin University ARTICLES Bento-Box User Experience Study at Franklin University Marc Jaffy INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2020 https://doi.org/10.6017/ital.v39i1.11581 Marc Jaffy (marc.jaffy@franklin.edu) is Acquisitions Librarian, Franklin University. ABSTRACT This article discusses the benefits of the bento-box method of searching library resources, including a comparison of the method with a tabbed search interface. It then describes a usability study conducted by the Franklin University Library in which 27 students searched for an article, an ebook, and a journal on two websites: one using a bento box and one using the EBSCO Discovery Service (EDS). Screen recordings of the searches were reviewed to see what actions users took while looking for information on each site, as well as how long the searches took. Students also filled out questionnaires to indicate what they thought of each type of search. Overall students found more items on the bento-box site, and indicated a slight preference for the bento-box search over EDS. The bento-box site also provided quicker results than the EDS site. As a result, the Franklin University library decided to implement bento-box searching on its website. INTRODUCTION “One page, one search box, results from as many library-resource types as possible.”1 In 2018, the Franklin University Library redesigned its website to provide users with a more modern interface that more closely matched Franklin University’s website. The library also wanted to improve the site’s usability and make it easier for students to find information. To determine how to best improve the user experience, library staff members held a number of meetings to discuss the new site’s layout and contents. Because “students almost always resort[] to searching via Web site search boxes rather than navigating through the Web site by browsing,” a crucial decision involved what search results the redesigned library website would provide. 2 As a result of these discussions, the Franklin University Library’s initial website redesign included a persistent search bar in the upper left of each page which searched the library’s website, as well as a prominent tabbed search bar on the library’s homepage (see figure 1). The homepage search bar provided a default tab that used EBSCO Discovery Service (EDS) to search the library resources cataloged in EDS (most of the library’s databases and catalog) and a second tab which used EBSCO’s Journal Finder to look for e-journals. Once our new website went live, feedback from patrons demonstrated that the persistent website search bar caused confusion among users who expected it to search the library’s databases rather than the library’s website. We also found the “search journals” tab on the homepage unnecessary. As a result, we removed both the persistent search bar and the journals tab. After these changes, the main search option provided to library users was the EDS search bar on the library’s homepage, although some interior pages of the library’s website provided a search bar related to that page (such as an option to search the catalog on the catalog page). Although EDS searches mainly for articles and books, it “may overlook user needs for other types of library mailto:marc.jaffy@franklin.edu INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 2 resources or services.”3 This is a problem because “library users increasingly perceive the discovery interface as a portal to all of the library’s resources.”4 Due to dissatisfaction with the EDS search, the library decided to look for alternatives. One alternative which “[a] number of libraries have turned to [is] the bento-based approach to discovery and display.”5 Figure 1. Redesigned Franklin University Library website with two search boxes on the homepage. The circled search box in the upper left was initially persistent across the entire site. To determine whether the bento-box search format would serve our users better than EDS, the library designed and conducted a usability study comparing EDS and bento-box searches. By comparing user search behavior and results for each search method, as well as user opinion regarding these different methods of searching library websites, the library hoped to gain a clearer understanding of how its users interact with search boxes on the library’s website and— most importantly—which search method would best serve its users. The remainder of this article sets forth the results of that trial. After explaining what bento-box search is, as well as reasons a library might want to use bento-box search results, it reports on a usability study the Franklin University Library conducted by discussing both the observations of screen recordings demonstrating user search behavior and responses to questionnaires. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 3 BENTO-BOX LIBRARY SEARCH What Is Bento-Box Search? The term “bento box” is based on “Japanese cuisine where different parts of a meal are compartmentalized in aesthetically pleasing ways.”6 Instead of compartmentalizing food, a bento- box search results page compartmentalizes search results from a variety of different resources on a single page. The user sees a single search box which gives “side-by-side results from multiple library search tools, e.g., library catalog, article index, institutional repository, website, LibGu ides, etc.”7 A bento-box search provides results based on searches of individual library resources. This is important because of the difficulty of providing a single search that includes combined results from all resources: “the nature of library content and current technology makes it difficult to create usable ‘blended’ results; catalog materials may crowd out books or vice versa.” 8 Bento-box results avoid this problem by “provid[ing] these resources on equal footing, leveraging the ranking algorithms internal to each resource’s individual search interface.” 9 As a result, the bento box gives libraries “the best cost/benefit way to improve article search relatively quickly with relatively large user benefit.”10 Figure 2, the University of Michigan’s bento-box results page, illustrates how a bento box provides search results from a variety of resources in visually discrete boxes. This is done behind the scenes by using separate searches to query the individual resources, as demonstrated by figure 3, the architecture of Wayne State University’s bento-box search. Benefits of a Bento-Box Search Results Page Wayne State University’s switch to a bento box “resulted in increased access to resources.”11 A bento box can increase access to resources both because it makes library search easier for users and because it provides results in a format that makes it easier for users to find information. Simplified Search When deciding what type of search to provide on the library website, the main consideration involves what users expect when searching. Student experiences with internet search engines have influenced their expectations for library search, which leads them to “approach library search interfaces as if they were Google.”12 What do users like about Google? “One of the main reasons that users are satisfied with Google is its simple user interface.”13 Based on their experience with Google and other search engines, library users “expect easy searching” that provides “one-step immediate access to the full text of library resources.”14 Bento-box results let libraries meet these expectations by presenting users with a simple interface that permits easy searching and “returns results across many library sources.”15 Additionally, bento-box results “can integrate library website results, allowing users to type things like ‘how do I renew a book’ into a single search box and get meaningful results.”16 As a result, adopting a bento-box search results page can permit a library to satisfy user search expectations. The bento-box format will provide users the information they seek whether they are looking for an article, a book, or information about the library. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 4 Figure 2. University of Michigan Library’s bento-box results. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 5 Figure 3. Wayne State University Library bento box architecture, from Cole Hudson and Graham Hukill, “One-to-Many: Building a Single-Search Interface for Disparate Resources,” in K. Varnum (ed.) Exploring Discovery: The Front Door to Your Library’s Licensed and Digitized Content (Chicago: ALA Editions, 2016): 147. Better Presentation of Results Bento-box results can help alleviate user confusion because “format types [are] more evident: Novice users, such as undergraduates, may not have a good understanding of the difference between books, journals, and articles.”17 The bento-box presentation makes it easier for users to find information since “results from different sources are returned to visually discrete boxes” 18 on a single page. This presentation of grouped results benefits library users because “[b]y presenting search results in separate streams, users can more easily navigate to what they need.”19 When the Princeton University Library implemented a bento-box results page (termed All Search), it found that “most users praised the new All Search for its ease of use and also for the ‘bento-box’ approach of grouping results by category. They felt that this clarified the results they were seeing and made it easier for them to pursue different avenues of inquiry.”20 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 6 Comparison with Tabbed Searching One alternative to the bento box is to offer users a tabbed search box which lets users select a specific resource to search by selecting a tab on the search bar. Before the 2018 redesign, the Franklin University Library’s website provided users with a tabbed search box, as shown in figure 4. Our redesigned website reduced the number of tabs from four to two, but we ultimately removed tabbed search from our website because we did not find it effective. Figure 4. Previous Franklin University Library website with tabbed search box. Tabbed search requires users to decide which tab to use. When the Franklin University Library provided a tabbed search box we found that users had difficulty identifying which tab they should use. In addition to causing user confusion over which tab to use for their search, because each search tab only searches a portion of library resources, tabbed searching “misses a wide swath of available information and resources [which] will make that missing information practically invisible.”21 Another problem with tabbed search is that it requires a library to designate one of the tabs as a default search. This can lessen the chances that users will search (or use) resources in the non- default tab(s) because library users “tend to favor the most prominent search option.”22 Lown, Sierra, and Boyer cited a study which found that the default option was used 60.5 percent of the time in a tabbed interface, and reported that on the North Carolina State University website the default tab was used 73.7 percent of the time.23 Tabbed search does not meet library user needs because it is “inconsistent and confusing.”24 When Wayne State University switched from tabbed searching to a bento box “many library resources that were previously hidden from search and discovery on the main library website were, for the first time, exposed to all searches, for all users . . . [which] resulted in increased use and awareness of these resources.”25 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 7 Design Considerations Bento-box results pages are highly customizable. A 2017 review of 38 academic libraries using bento-box search found “much variation in the implementation.”26 A bento-box results page needs to balance providing necessary information with displaying results in a way that is not too overwhelming—or too cluttered. The University of Michigan analyzed usage of its bento-box results page and redesigned it to improve how it presented results to library users by displaying “[f]ewer results . . . in each section—with a more prominent ‘see all’ link—than in the original design.”27 Given user expectations and the challenges previously discussed, libraries must design the results to “maximize the exposure of [their] collection[s] and services, in the most appropriate precedence, while preventing cognitive overload.”28 A cluttered results page, with a lack of distinction between categories, will make it difficult for users to find information and will cause confusion rather than ease it. Another concern with a bento-box results page occurs when some of the result boxes “end up ‘below the fold,’ meaning users will need to scroll down to see them. This creates the same problem as a tabbed search box—users don’t see results from all library sources.”29 Because users are less likely to see below the fold search results, the bento-box results page needs to prioritize category locations so that the more important results are above the fold (which requires the library to determine the relative importance of search result categories). USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY The Trial Design During Franklin University’s fall 2018 and spring 2019 welcome weeks, in addition to providing students with information about the library’s services, staff at the library’s information table asked students to participate in a trial to help determine whether adopting a bento-box results page would benefit our users. Participants were offered a Franklin University coffee mug as an incentive. The trial asked participants to look for information on two different library websites: one using EDS (Franklin University Library) and one using a bento box (Wayne State University Library). We set up two laptops for participants to use and made screen recordings of the participants’ actions during their searches for later viewing and analysis. After they finished the tasks we asked participants to fill out a questionnaire (reproduced in Appendix A), which had three background questions and six questions about their experience/thoughts relating to the tasks. To decide what information to ask participants to look for, we reviewed library websites that used bento boxes to see what categories they searched. We compared those categories to the types of information available on our library’s website. Although we identified a number of possible categories, we decided to limit the trial to three tasks because we did not want to overburden participants. Based on our experience working with students, we decided the three categories we were most interested in investigating: articles, books, and journals. To see how users searched for items in these categories we asked participants to complete the following three tasks on each website: INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 8 1. Find an article available through the library on the topic “criminal justice.” 2. Find the ebook Lean Six Sigma for Leaders by Martin Brenig-Jones and Jo Dowdall. 3. Find the electronic journal Business Today. Participants Thirty-four people participated in the trial. However, not all of the participants completed all of the tasks on each library’s website. We discarded the results from participants who did not attempt at least two tasks on each library’s website. Removing those who did not complete at least two tasks on each site left 27 participants (“adjusted” results). Unless otherwise noted, the data discussed below refers to the adjusted results. The trial was open to students, faculty, and staff. Eleven participants took the trial in fall 2018 and 16 participants took it in spring 2019. Most of the participants were undergraduates (21), with some graduate students (6). No doctoral students, faculty, or staff-only participated. (One staff member participated and completed a questionnaire but did not perform enough tasks for their results to be included.) Results We watched the screen recordings to time how long it took students to complete the three tasks on each library’s website. However, if a student flipped between the sites while searching instead of first completing all three searches on one site we did not time the results. We also observed what students did while searching to gain an understanding of how they searched for information. Overall, students spent less time searching on the site using the bento box (Wayne State University Library) than they did on the Franklin University Library site: • Students spent an average of 2 minutes, 35 seconds to complete the tasks on the Wayne State site compared to 3 minutes, 28 seconds on Franklin’s. • Twelve students finished their searches quicker on Wayne State’s site, while six had quicker results from Franklin’s. How Students Searched for Information The screen recordings showed that students looking for information often went to parts of the library websites which did not contain the content they sought. Frequently, they would search whatever part of the site they were on—even if it did not contain the content they needed. On the Wayne State site, when a student used an interior search bar the bento box provided results from a wide range of library resources. A search on the journals page would give results for journals, books, articles, and more—even if the page they were on did not contain the resource they needed to find. By contrast, any interior search box found on the Franklin University page would only provide results for whatever portion of the library’s resources that search box accessed: a search on the journals page would only provide journal results, a search on the catalog page would only provide catalog results, etc. Student action after the initial search demonstrates the need for interior search boxes which can search the library’s entire site. Twelve of 27 students on each site (although not the same 12 students) followed up a search by using a search bar they found on their results page without returning to the homepage to use the main search bar. Students did this even when the page they were on did not relate to the content they were looking for. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 9 For library users, the division between the content of the library’s website and the content provided by the library “’is not obvious and makes no real sense.’”30 The screen recordings of student behavior when searching for content on library websites demonstrated that students also could not distinguish between different areas within the library’s website. Search for Articles The first task asked students to find an article on criminal justice. Because it was the first task on the list, most students started with this search. Students had more success finding the article on Franklin’s site than on Wayne State’s site (18 to 14). Although only 14 students were credited with finding the article on Wayne State’s site, several students actually reached the bento-box page which included a category for article results. However, they did not realize that they had found the article and selected an ebook or database instead. Search for Ebooks The second task asked students to find the ebook Lean Six Sigma for Leaders by Martin Brenig- Jones and Jo Dowdall. Students had more success finding this book on the bento-box site. Twenty students found the book using the Wayne State Library’s bento-box search, compared to ten who found the book on the Franklin University Library’s site. Many students, in searching for the ebook, typed their search on the results page from the previous search. On the Wayne State University Library’s site, this led to a bento-box results page which included the book. On Franklin University Library’s site, the results were more complex. Between fall 2018 and spring 2019, our EBSCO EDS custom catalog was not renewed, which resulted in the search results for the ebook no longer displaying in EDS. [This non-renewal was not intentional.] As a result, in fall 2018, the ebook that students looked for appeared in EDS search results (on both the Franklin University Library’s main search bar and interior EBSCO search boxes); however, in spring 2019 it did not. To see what effect this had, we looked at the more limited search results from the fall 2018 trial when an EDS search on Franklin University Library’s site would successfully find the book. Even then, students had more success finding the book on the bento-box site: 9 students successfully found the book on Wayne State’s site, compared to 7 on Franklin University’s site. Some students on Franklin University Library’s site tried, unsuccessfully, to identify the proper page of the library’s site to find “books.” Instead of the catalog page, they went to a page labeled “textbooks” which helped students locate course reserves. Because there was no search option for the entire site on that page, those students did not successfully find the ebook. Search for Journals The third task asked students to look for the journal Business Today. As with the ebook search, students had more success finding the journal on the bento-box site: 19 students found the journal on Wayne State Library’s site compared to 10 on Franklin University Library’s site. Students searched for the journal in a similar manner to the way they searched for the ebook. Many just put the journal title in the search bar on the page they were on—if that search bar on Franklin University Library’s site provided access to the journal, they would find it with their INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 10 search. But if they were on a page which did not have a search which included the journal as a result, they would not. The journal search on the Wayne State University Library’s site demonstrated the “below th e fold” problem discussed above because the bento-box result for “journals” was below the fold. As a result, at least one student properly searched for the journal, but did not find it because the result was not visible on the screen and they did not scroll down. Questionnaires We asked students a series of questions about their experience searching for information on the two library websites (see Appendix A for the questionnaire and Appendix B for the results). Slightly more students preferred the Wayne State library’s site (14) to Franklin University’s (12). However, five of the users who preferred Franklin University’s site referenced their familiarity with that site. Four of these users specifically referenced their familiarity with the Franklin University Library site in response to a question asking “[w]hy did you prefer the type of search you picked,” while one referenced their familiarity with Franklin University Library’s site in response to a question asking whether there was “anything else you’d like to tell us.” The questionnaire also asked students why they preferred one site to the other and what they liked and disliked about each site. Preference: Franklin University Library As mentioned above, many of the comments from those who indicated a preference for the Franklin University Library site indicated the preference was due to familiarity: • “Might be because I am a bit used to it, I just found it easier to navigate.” • “Because it’s the one I am familiar with.” • “Because I'm familiar with it.” • “I liked both. Both easy to use. Familiar with Franklin's.” Other comments favored Franklin University Library’s overall website design (as opposed to search): • “The website is cleaner. Easier to use.” • “Easier to navigate.” Some of the comments did indicate a preference for the search technique: • “One search bar to search all types. Seemed to include more in search results.” • “It easy to search and straight forward.” • “Access to research was quicker on the Franklin website. Also, I felt like there was more research material available.” Preference: Wayne State University Library Those who preferred the Wayne State search did so more based on the search technique than did those who preferred Franklin: • “Simple and the search was in one spot.” INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 11 • “One search bar that pulled from the [catalog].” • “Because you can type in exactly what you were looking for and it comes up.” Others appreciated the way search results were displayed: • “Their search system organizes the result by type of information, whereas Franklin's website makes you search for the type of material information before displaying the results.” • “Better layout breaks articles, journals, etc. into separate columns.” • “Wayne had each section (book, e-journal, article) separately which was easier to find.” • “The layout.” Still others just found the Wayne State Library search easier to use: • “It is more visual and easy to find and easy to use.” • “It presented the information in an easy way to find.” • “Easy—all in one.” What Search Results Do Library Users Want? We asked participants to rank which results they would like to see displayed when searching on the library’s site. While most applied a numerical ranking, some just circled items. All questionnaire responses were included when compiling these rankings, including rankings from those who did not complete at least two tasks on each website, because user preferences about what search results they want are valid even if they did not perform the required tasks on each library’s website. We converted numerical rankings so that the first choice received six points, the second choice five points, etc. Of the 34 participants who answered this question, 24 provided rankings and ten circled items without indicating how they ranked those items. Where participants circled items, we converted their responses to a numerical equivalent based on how many answers they circled. If they only circled one, it was treated as the first choice, and given 6 points. If they circled more than one, we combined the numerical value of the answers and each answer received the average value. (For example, if two answers were circled, they were treated as a first and second choice, and each circled answer was given a score of 5.5.) The responses indicate students most wanted library search to provide results for articles and journals, followed by databases and ebooks: • Articles: 144 • Journals: 125 • Databases: 112 • Books/ebooks: 111 • Research guides: 69.5 • Library site: 63.5 Mischo, Norman, and Schlembach reported actual usage of the University of Illinois’ bento-box results page by category between 2015 and 2017. How do the categories our users indicated they INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 12 would like to see on a bento-box results page compare with the actual use of bento-box results at the University of Illinois? At the University of Illinois, 56.1 percent of click-throughs were for articles (Franklin University students’ first choice), while 33.6 percent were for book and online catalog content (our students’ fourth choice).31 Databases, our students’ third choice, were not a listed category, while journals (Franklin University students’ second choice) were only the fifth-most used resource (and the percentage of click-throughs was low, at only 3.6 percent).32 Limitations There were a few issues with the study which should be kept in mind when evaluating the results. Number of Participants Thirty-four individuals participated in the trial. After removing results from participants who failed to complete a sufficient number of tasks, only 27 participants remained. While this number is small, it does provide information on what students think and, more importantly, how they act when searching for various types of information on the library’s website. Examples of library user experience testing based on similar numbers include: • The University of Kansas Library conducted “usability testing of our Primo discovery interface on basic library searching scenarios for undergraduate and graduate students” and reported results from 27 users.33 • The University of Southern Mississippi Library conducted usability testing of 24 users (“six participants from each of the following library user groups: undergraduate students, graduate students, faculty, and library employees”) to evaluate and modify their website.34 • Syracuse University conducted usability testing on “ten students . . . and eighteen library staff members.”35 Familiarity with Franklin University Library Website Student familiarity with the Franklin University Library website affected student opinion. Of 12 students in the adjusted results who preferred Franklin University Library’s site, 5 (41.7 percent) gave an answer indicating that familiarity was a factor in their preference. When considering all the questionnaires (including those from participants who were not included in the adjusted results), 7 out of 17 users (41.2 percent) who preferred Franklin University Library’s site gave an answer referencing familiarity. As a result, opinion may have been skewed in favor of Franklin University, despite Wayne State being slightly favored overall. A good illustration of this problem is the response from a student who the screen recording showed did not even attempt any of the tasks on the Franklin University Library website but indicated a preference for the Franklin University Library site because it’s “easy to use.” CONCLUSION As a result of the user experience study, the Franklin University Library decided that providing bento-box search results would benefit our library’s users. The trial showed that students required less time to conduct their searches using Wayne State’s bento-box search and found more items successfully on Wayne State’s site. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 13 The lack of student distinction between different types of library content, along with the likelihood of their entering a search in whatever search box they see on a page further supports providing bento-box results. Adopting a bento-box results page will permit the library to provide search boxes on interior pages which permit students to search for materials site-wide. The bento box will let students search for content anywhere on the library’s web site without requiring them to first figure out what type of library resource they are looking for and then find the correct section of the library’s website. Additionally, the ebook search issue previously discussed demonstrates the benefits of switching to a bento box. The disappearance of ebook search results from the EDS listing would not have mattered with a bento-box style search because the bento box would have displayed a box for catalog results. Comments from two of the students who preferred the Wayne State library website demonstrate the benefits of a bento-box format. The bento-box search meant that Wayne State’s site is “simple and the search was in one spot.” It also helps students because “[Wayne State’s] search system organizes the result by type of information, whereas Franklin's website makes you search for the type of material information before displaying the results.” INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 14 APPENDIX A: QUESTIONNAIRE About this Study The Franklin University library is studying how users search for, and find, information on library websites. The purpose of this study is to ask library users (and potential library users) to search for information on two different library websites and give their opinion on their search experience. You are asked to be a participant as a member of the Franklin University community who is a library user, or a potential library user. We hope to have between 20 and 50 people participate in this study. If you agree to participate in this study, you will be asked to look for information to find four different resources on two library websites (Franklin University’s website and Wayne State University’s website). You will only be asked to find the information, not to access the information. You will then be asked to fill out a questionnaire providing demographic information and your opinion about library search. As part of this study, your search activity on the websites may be recorded by screen recording software. Your participation in this study is anticipated to take about 15 minutes. There are no known risks to participation in the study. The benefits of participation include helping the library to better serve its users by identifying how users search for information on the library’s website. In return for your participation in this survey, you will receive a Franklin University coffee mug. This study is conducted anonymously—no personally identifiable information will be collected. Your participation in this survey is voluntary. If you decide to participate, you have the right to refuse to answer any of the questions that make you uncomfortable. You also have the right to withdraw from this study at any time, with no repercussions. This research has been reviewed and approved by the Franklin University Institutional Review Board. For questions regarding participants’ rights, you can contact the Institutional Review Board at irb@franklin.edu. INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 15 Please answer the following background questions: 1) What do you do at Franklin University (circle all that apply)? a) Non-degree-seeking student b) Undergraduate student c) Master student d) Doctoral student e) Staff f) Faculty 2) How often do you use the library’s website (circle the best choice)? a) Frequently (every week) b) Occasionally (every month) c) Rarely (less than once a month) d) Never 3) How often do you use the search function on the library’s website? a) Frequently b) Occasionally c) Rarely d) Never INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 16 Please answer the following questions about your experience and preferences when searching for information on library websites: 1) Which library’s search did you prefer: a) Franklin University Library b) Wayne State University Library 2) Why did you prefer the type of search you picked in the answer to question 1? 3) For Franklin University’s search results: a) What did you like? b) What didn’t you like? 4) For Wayne State University’s search results: a) What did you like? b) What didn’t you like? 5) Please rank in order of preference what search results you would want to see displayed when searching on the library’s website: a) Articles related to a topic b) Books / Ebooks c) Databases d) Library website e) Journals f) Research Guides g) Other (please list): 6) Is there anything else you’d like to tell us about your experience looking for information on these library websites? INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 17 APPENDIX B: ADJUSTED QUESTIONNAIRE RESULTS Below are the results from participants who completed at least two tasks on each university library’s website. Where the screen recordings indicated that participants did not complete at least two tasks on each of the websites the questionnaire responses were not recorded. What do you do at Franklin University (circle all that apply)? Undergraduate: 21 Masters: 6 How often do you use the library’s website (circle the best choice)? Occasionally: 9 Frequently: 12 Rarely: 6 How often do you use the search function on the library’s website? Occasionally: 7 Frequently: 13 Rarely: 7 Which library’s search did you prefer: Franklin University: 12 Wayne State University: 14 N/A: 1 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 18 APPENDIX C: SCREEN RECORDING RESULTS Analysis of screen recordings from participants who completed at least two tasks on each university library’s website. For timed results, we did not include the results of students who flipped between library websites while completing the tasks. (An example of flipping between sites occurred when a student found the article on the Franklin University Library site, then looked for it on Wayne State University Library site before looking for the ebook on the Franklin University site.) Time to complete tasks (average): Franklin University Library site: 3:28 Wayne State University Library site: 2:35 Site where student finished search quicker: Franklin University Library site: 6 Wayne State University Library site: 12 Total items found: Franklin University Library site: 38 Wayne State University Library site: 53 Articles found: Franklin University Library site: 18 Wayne State University Library Site: 14 Books found: Franklin University Library site: 10 Wayne State University Library site: 20 Journals Found: Franklin University Library site: 10 Wayne State University Library site: 19 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 19 ENDNOTES 1 Cole Hudson and Graham Hukill, “One-to-Many: Building a Single-Search Interface for Disparate Resources,” in K. Varnum (ed.) Exploring Discovery: The Front Door to Your Library’s Licensed and Digitized Content (Chicago: ALA Editions, 2016): 146. 2 Suzanna Conrad and Nathasha Alvarez, “Conversations with Web Site Users: Using Focus Groups to Open Discussion and Improve User Experience,” Journal of Web Librarianship 10, no. 2 (April 2016): 71, https://doi.org/10.1080/19322909.2016.1161572. 3 Scott Hanrath and Miloche Kottman, “Use and Usability of a Discovery Tool in an Academic Library,” Journal of Web Librarianship 9, no. 1 (January 2015): 4, https://doi.org/10.1080/19322909.2014.983259. 4 Irina Trapido, “Library Discovery Products: Discovering User Expectations through Failure Analysis.” Information Technology and Libraries 35, no. 3 (2016): 22, https://doi.org/10.6017/ital.v35i3.9190. 5 William Mischo, Michael Norman, and Mary Schlembach, “Innovations in Discovery Systems: User Studies and the Bento Approach,” Proceedings of the Charleston Library Conference (2017): 299, https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1991&context=charleston. 6 Hudson and Hukill, “One-to-Many,” 142. 7 Emily Singley, “To Bento Or Not to Bento—Displaying Search Results,” http://emilysingley.net/usablelibraries/to-bento-or-not-to-bento-displaying-search-results/. 8 Jonathan Rochkind, “Article Search Improvement Strategy,” https://bibwild.wordpress.com/2012/10/02/article-search-improvement-strategy/. 9 Hudson and Hukill, “One-to-Many,” 145. 10 Rochkind, “Article Search Improvement Strategy.” 11 Hudson and Hukill, “One-to-Many,” 142. 12 Nancy Turner, “Librarians do it Differently: Comparative Usability Testing with Students and Library Staff,” Journal of Web Librarianship 5, no. 4 (October 2011), https://doi.org/10.1080/19322909.2011.624428; Elena Azadbakht, John Blair, and Lisa Jones, “Everyone's Invited: A Website Usability Study Involving Multiple Library Stakeholders,” Information Technology & Libraries 36, no. 4 (2017), 43, https://doi.org/10.6017/ital.v36i4.9959. 13 Colleen Kenefick and Jennifer A. DeVito, “Google Expectations and Interlibrary Loan: Can We Ever Be Fast Enough?” Journal of Interlibrary Loan, Document Delivery & Electronic Reserves 23, no. 3 (July 2013): 158, https://doi.org/10.1080/1072303X.2013.856365. https://doi.org/10.1080/19322909.2016.1161572 https://doi.org/10.1080/19322909.2014.983259 https://doi.org/10.6017/ital.v35i3.9190 https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1991&context=charleston http://emilysingley.net/usablelibraries/to-bento-or-not-to-bento-displaying-search-results/ https://bibwild.wordpress.com/2012/10/02/article-search-improvement-strategy/ https://doi.org/10.1080/19322909.2011.624428 https://doi.org/10.6017/ital.v36i4.9959 https://doi.org/10.1080/1072303X.2013.856365 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2020 BENTO BOX USER EXPERIENCE STUDY AT FRANKLIN UNIVERSITY | JAFFY 20 14 Kenefick and DeVito, “Google Expectations and Interlibrary Loan,” 157; Carol Diedrichs, “Discovery and Delivery: Making it Work for Users,” Serials Librarian 56, no. 1-4 (January 2009): 81, https://doi.org/10.1080/03615260802679127. 15 Singley, “To Bento or Not to Bento.” 16 Singley. 17 Singley. 18 Hudson and Hukill, “One-to-Many,” 146. 19 Singley, “To Bento or Not to Bento.” 20 Eric Phetteplace and Jeremy Darrington, “A Hybrid Approach to Discovery Services,” Reference & User Services Quarterly 53, no. 4 (2014): 293. 21 Cory Lown, Tito Sierra, and Josh Boyer, “How Users Search the Library from a Single Search Box,” College & Research Libraries 74, no. 3 (2013): 229. 22 Singley, “To Bento or Not to Bento.” 23 Lown, Sierra, and Boyer, “How Users Search the Library from a Single Search Box.” 24 Hudson and Hukill, “One-to-Many,” 150. 25 Hudson and Hukill, 150. 26 Mischo, Norman, and Schlembach, “Innovations in Discovery Systems,” 304. 27 Suzanne Chapman et al., “Manually Classifying User Search Queries on an Academic Library Web Site,” Journal of Web Librarianship 7, no. 4 (October 2013): 419, https://doi.org/10.1080/19322909.2013.842096. 28 Aaron Tay and Feng Yikang, “Implementing a Bento-Style Search in LibGuides V2,” Code4lib Journal no. 29 (July 2015), https://journal.code4lib.org/articles/10709. 29 Singley, “To Bento or Not to Bento.” 30 Chapman et al., “Manually Classifying User Search Queries,” 406. 31 Mischo, Norman, and Schlembach, “Innovations in Discovery Systems.” 32 Mischo, Norman, and Schlembach. 33 Hanrath and Milche Kottman, “Use and Usability of a Discovery Tool,” 5. 34 Azadbakht, Blair, and Jones, “Everyone's Invited,” 34. 35 Turner, “Librarians Do It Differently,” 290. https://doi.org/10.1080/03615260802679127 https://doi.org/10.1080/19322909.2013.842096 https://journal.code4lib.org/articles/10709 ABSTRACT Introduction Bento-box Library Search What Is Bento-Box Search? Benefits of a Bento-Box Search Results Page Simplified Search Better Presentation of Results Comparison with Tabbed Searching Design Considerations User Experience Study at Franklin University The Trial Design Participants Results How Students Searched for Information Search for Articles Search for Ebooks Search for Journals Questionnaires Preference: Franklin University Library Preference: Wayne State University Library What Search Results Do Library Users Want? Limitations Number of Participants Familiarity with Franklin University Library Website Conclusion Appendix A: Questionnaire About this Study Appendix B: Adjusted Questionnaire Results What do you do at Franklin University (circle all that apply)? How often do you use the library’s website (circle the best choice)? How often do you use the search function on the library’s website? Which library’s search did you prefer: Appendix C: Screen Recording Results Time to complete tasks (average): Site where student finished search quicker: Total items found: Articles found: Books found: Journals Found: Endnotes