Usability Study of a Library’s M obile Website: An Example from Portland State University Kimberly D. Pendell and Michael S. Bowman USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 45 ABSTRACT To discover how a newly developed library mobile website performed across a variety of devices, the authors used a hybrid field and laboratory methodology to conduct a usability test of the website. Twelve student participants were recruited and selected according to phone type. Results revealed a wide array of errors attributed to site design, wireless network connections, as well as phone hardware and software. This study provides an example methodology for testing library mobile websites, identifies issues associated with mobile websites, and provides recommendations for improving the user experience. INTRODUCTION Mobile websites are swiftly becoming a new access point for library services and resources. These websites are significantly different from full websites, particularly in terms of the user interface and available mobile-friendly functions. In addition, users interact with a mobile website on a variety of smartphones or other Internet-capable mobile devices, all with differing hardware and software. It is commonly considered a best practice to perform usability tests prior to the launch of a new website in order to assess its user friendliness, yet examples of applying this practice to new library mobile websites are rare. Considering the variability of user experiences in the mobile environment, usability testing of mobile websites is an important step in the development process. This study is an example of how usability testing may be performed on a library mobile website. The results provided us with new insights on the experience of our target users. In the fall of 2010, with the rapid growth of smartphones nationwide especially among college students, Portland State University (PSU) Library decided to develop a mobile library website for its campus community. The library’s lead programmer and a student employee developed a test version of the website. This version of the website included library hours, location information, a local catalog search, library account access for viewing and renewing checked out items, and access to reference services. It also included a “Find a Computer” feature displaying the availability of work stations in the library’s two computer labs. Kimberly D. Pendell (kpendell@pdx.edu) is Social Sciences Librarian, Assistant Professor, and Michael S. Bowman (bowman@pdx.edu) is Interim Assistant University Librarian for Public Services, Associate Professor, Portland State University Library, Portland, Oregon. mailto:kpendell@pdx.edu mailto:bowman@pdx.edu INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 46 The basic architecture and design of the site was modeled on other existing academic library mobile websites that were appealing to the development team. The top-level navigation of the mobile website largely mirrored the full library website, utilizing the same language as the website when possible. The mobile website was built to be compatible with WebKit, the dominant smartphone layout engine. Use of JavaScript on the website was minimized due to the varying levels of support for it on different smartphones, and Flash was avoided entirely. Figure 1. Home Page of Library Mobile Website, Test Version We formed a mobile website team to further evaluate the test website and prepare it for launch. Three out of four team members owned smartphones, either an iPhone 3GS or an iPhone 4. We soon began questioning how the mobile website would work on other types of phones, recognizing that hardware and software differences would likely impact user experience of the mobile website. Performing a formal usability test using a variety of Internet-capable phones quickly became a priority. We decided to conduct a usability test for the new mobile website in order to answer the question: How user-friendly and effective is the new library mobile website on students’ various mobile devices? LITERATURE REVIEW Smartphones, mobile websites, and mobile applications have dominated the technology landscape in the last few years. Smartphone ownership has steadily increased, and a large percentage of USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 47 smartphone owners regularly use their phone to access the Internet. The Pew Research Center reports that 52 percent of Americans aged 18–29 own smartphones, and 81 percent of this population use their smartphone to access the Internet or e-mail on a typical day. Additionally, 42 percent of this population uses a smartphone as their primary online access point.1 The 2010 ECAR Study of Undergraduate Students and Information Technology found that 62.7 percent of undergraduate students own Internet-capable handheld devices, an increase of 11.5 percent from 2009. The 2010 survey also showed that an additional 11.3 percent of students intended to purchase an Internet-capable handheld device within the next year.2 In this environment academic libraries have been scrambling to address the proliferation of student owned mobile devices, thus the number of mobile library websites is growing. The Library Success wiki, which tracks libraries with mobile websites, shows an 66 percent increase in the number of academic libraries in the United States and Canada with mobile websites from August 2010 to August 2011.3 We reviewed articles about mobile websites in the professional library science literature and found that mobile website usability testing is only briefly mentioned. In their summary of current mobile technologies and mobile library website development, Bridges, Rempel, and Griggs state that “user testing should be part of any web application development plan. You can apply the same types of evaluation techniques used in non-mobile applications to ensure a usable interface.”4 In a previous article, the same authors also note that not accounting for other types of mobile users is easy to do but leaves a potentially large audience for a mobile website “out in the cold.”5 More recently, Seeholzer and Salem found the usability aspect of mobile website development to be in need of further research.6 Usability evaluation techniques for a mobile website are similar to those for a full website, but the variety of smartphones and Internet-capable feature phones immediately complicates standard usability testing practices. The mobile device landscape is fraught with variables that can have a significant impact on the user experience of a mobile website. Factors like small screen size, processing power, wireless or data plan connection, and on-screen keyboards or other data entry methods contribute to user experience and impact usability testing. Zhang and Adipat note that, Mobile devices themselves, due to their unique, heterogeneous characteristics and physical constraints, may play a much more influential role in usability testing of mobile applications than desktop computers do in usability testing of desktop applications. Therefore real mobile devices should be used whenever possible.7 One strategy for usability testing on mobile devices is to identify device “families” by similar operating systems or other characteristics, then perform a test of the website. For example, Griggs, Bridges, and Rempel found representative models of device families at a local retailer, where they tested the site on the display phones. The authors also recommend “hallway usability testing,” an impromptu test with a volunteer.8 Zhang and Adipat go on to outline two methodologies for formal mobile application usability testing: field studies and laboratory experiments. The benefit of a mobile usability field study is INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 48 the preservation of the mobile environment in which tasks are normally performed. However, data collection is challenging in field studies, requiring the participant to reliably and consistently self- report data. In contrast, the benefit of a laboratory study is that researchers have more control over the test session and data collection method. Laboratory usability tests lend themselves to screen capture or video recording, allowing researchers more comprehensive data regarding the participant’s performance on predetermined tasks.9 However, Billi and others point out that there is no general agreement in the literature about the significance or usefulness of the difference between laboratory and field testing of mobile applications.10 One compromise between field studies and laboratory experiments is the use of a smartphone emulator: an emulator mimics the smartphone interface on a desktop computer and is recordable via screen capture. However, desktop emulators mask some usability problems that impact smartphones, such as an unstable wireless connection or limited bandwidth.11 In order to record test sessions of users working directly with mobile devices, Jakob Nielsen, the well-known usability expert, briefly mentions the use of a document camera.12 In another usability test of a mobile application, Loizides and Buchanan also used a document camera with recording capabilities to effectively record users working with a mobile device.13 Usability attributes are metrics that help assess the user-friendliness of a website. In their review of empirical mobile usability studies, Coursaris and Kim present the three most commonly used measures in mobile usability testing: Efficiency: degree to which the product is enabling the tasks to be performed in a quick, effective and economical manner or is hindering performance; Effectiveness: accuracy and completeness with which specified users achieved specified goals in particular environment; Satisfaction: the degree to which a product is giving contentment or making the user satisfied.14 The authors present these measures in an overall framework of “contextual usability” constructed with the four variables of user, task, environment, and technology. An important note is the authors’ use of technology rather than focusing solely on the product; this subtle difference acknowledges that the user interacts not only with a product, but also other factors closely associated with the product, such as wireless connectivity.15 A participant proceeding through a predetermined task scenario is helpful in assessing site efficiency and effectiveness by measuring the error rate and time spent on a task. User satisfaction may be gauged by the participant’s expression of satisfaction, confusion, or frustration while performing the tasks. Measurement of user satisfaction may also be supplemented by a post-test survey. Returning to general evaluation techniques, mobile website usability employs the use of task scenarios, post-test surveys, and data analysis methods, similar to full site testing. General guides such as The Handbook of Usability Testing by Rubin and Chisnell and George’s User-Centered Library Websites: Usability Evaluation Methods provide helpful information on designing task scenarios, how to facilitate a test, post-test survey ideas, and methods of analysis.16 Another USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 49 common data collection method in usability testing is the think aloud protocol as it allows researchers to more fully understand the user experience. Participants are instructed to talk about what they are thinking as they use the site; for example, expressing uncertainty of what option to select, frustration with poorly designed data entry fields, or satisfaction with easily understood navigation. Examples of the think aloud protocol can also be found in mobile website usability testing.17 METHOD While effective usability testing normally relies on five to eight participants, we decided a larger number of participants would be needed in order to capture the behavior of the site on a variety of devices. Therefore, we recruited twelve participants to accommodate a balanced variety of smartphone brands and models. Based on average market share, we aimed to test the website on four iPhones, four Android phones, and four other types of smartphones or Internet-capable mobile devices (e.g., BlackBerry, Windows phones). All study participants were university students, the primary target audience of the mobile website. We used three methods to recruit participants: a post to the library’s Facebook page, a news item on the library’s home page, and two dozen flyers posted around campus. Each form of recruitment described an opportunity for students to spend less than thirty minutes helping the library test its new mobile website. Also, participants would receive a $10 coffee shop gift card as an incentive. A project-specific email address served as the initial contact point for students to volunteer. We instructed volunteers to indicate their phone type in their e-mail; this information was used to select and contact the students with the desired variety of mobile devices. If a scheduled participant did not come to the test appointment, another student with the same or similar type of phone was contacted and scheduled. No other demographic data or screening was used to select participants, aside from a minimum age requirement of eighteen years old. We employed a hybrid field and laboratory test protocol, which allowed us to test the mobile website on students’ native devices while in a laboratory setting that we could efficiently manage and schedule. Participants used their own phone for the test without any adjustment to their existing operating preferences, similar to field testing methodology. However, we used a controlled environment in order to facilitate the test session and create recordings for data analysis. A library conference room served as our laboratory, and a document camera with video recording capability was used to record the session. The document camera was placed on an audio/visual cart and the participants chose to either stand or sit while holding their phones under the camera. The document camera recorded the phone screen, the participant’s hands, and the audio of the session. The video feed was available through the room projector as well, which helped us monitor image quality of the recordings. INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 50 Figure 2. Video Still from Test Session Recording The test session consisted of two parts: the completion of five tasks using participants’ phones on our test website recorded under the document camera, and a post-test survey. Participants were read an introduction and instructions from a script in order to decrease variation in test protocol and our influence as the facilitators. We also performed a walk-through of the testing session prior to administering it to ensure the script was clearly worded and easy to understand. We developed our test scenarios and tasks according to five functional objectives for the library mobile website: 1. Participants can find library hours for a given day in the week. 2. Participants can perform a known title search in catalog and check for item status. 3. Participants can use My Account to view checked out books.18 4. Participants can use chat reference. 5. Participants can effectively search for a scholarly article using the mobile version of EBSCOhost Academic Search Complete. Prior to beginning the test, we encouraged participants to use the “think aloud” protocol while performing tasks. We also instructed them to move between tasks however they would naturally in order to capture user behavior when navigating from one part of the site to another. The post-test survey provided us with additional data and user reactions to the site. Users were asked to rate the site’s appearance, ease of use, and how frequently they might use the different website features USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 51 (e.g., renewing a checked out item). The survey was administered directly after the task scenario portion of the test in order to take advantage of the users’ recent experience with the website. We evaluated the test sessions utilizing the measures of efficiency, effectiveness, and satisfaction. In this study, we assessed efficiency as time spent performing the task and effectiveness as success or failure in completing the task. We observed errors and categorized them as either a user error or site error. Each error was also categorized as minor, major, or fatal: minor errors were easily identified and corrected by the user; major errors caused a notable delay, but the user was able to correct and complete the task; fatal errors prevented the user from completing the task. To assess user satisfaction, we took note of user comments as they performed tasks, and we also referred to their ratings and comments on the post-test survey. Before analyzing the test recordings, we normalized our scoring behavior by performing a sample test session with a library staff member unfamiliar with the mobile website. We scored the sample recording separately and then met to discuss, clarify, and agree upon each error category. Each of the twelve test sessions was viewed and scored independently. Once this process was completed, we discussed our scoring of each test session video, combining our data and observations. We analyzed the combined data by looking for both common and unique errors for each usability task across the variety of smartphones tested. To protect participants’ confidentiality, each video file and post-test survey was labeled only with the test number and device type. Prior to beginning the study, all recruitment methods, informed consent, methodology, tasks and post-test survey were approved by Portland State University Human Subjects Research and Review Committee. FINDINGS Our recruitment efforts were successful with even a few same-day responses from the announcement posted on the library’s Facebook page. Some students also indicated that they had seen the recruitment flyers on campus. A total of fifty-two students volunteered to participate; twelve students were successfully contacted, scheduled, and tested. The distribution of the twelve participants and their types of phones is shown in table 1. Number of participants Operating system Phone model 4 Android HTC Droid Incredible 2; Motorola Droid; HTC MyTouch 3G Slide; Motorola Cliq 2 3 iOS iPhone 3GS 2 BlackBerry Blackberry 9630; Blackberry Curve INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 52 1 Windows Phone 7 Windows Phone 7 1 webOS Palm Pixi 1 Other Windows Kin 2 feature phone (a phone with Internet capability, running KinOS) Table 1. Test Participants by Smartphone Operating System and Model Usability Task Scenarios All test participants quickly and successfully completed the first task, finding the library hours for Sunday. The second task was to find a book in the library catalog and report whether the book was available for check out. Nine participants completed this task; the Windows Phone 7 and the two Blackberry phones presented a fatal system error when working with our mobile catalog software, MobileCat. These participants were able to perform a search but were not able to view a full item record, blocking them from seeing the item’s availability and completing the task. This task also revealed one minor error for iPhone users: the iPhone displayed the item’s ten digit ISBN as a phone number, complete with touch-to-call button. Many users took more time than anticipated when asked to search for a book. The video recordings captured participants slowly scrolling through the menu before choosing “Search PSU- only Catalog.” A few participants expressed their hesitation verbally: ● “Maybe not the catalog? I don't know. Yeah I guess that would be the one.” ● “I don't look for books on this site anyway...my lack of knowledge more than anything else.” ● “Search PSU library catalog I'm assuming?” The Blackberry Curve participant did not recognize the catalog option and selected “Databases & Articles” to search for a book. She was guided back to the catalog after her unsuccessful search in EBSCOhost. We observed an additional delay in searching for a book when using the catalog interface. The catalog search included a pull down menu of collections options. The collections menu was included by the site developers because it is present in the full website version of the local catalog. Users tended to explore the menu looking for a selection that would be helpful in performing the task; however, they abandoned the menu, occasionally expressing additional confusion. USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 53 Figure 3. Catalog Search with Additional “Collections” Menu The next task was to log into a library account and view checked out items. All participants were successful with this task, but frequent minor user errors were observed, all misspelling or numerical entry errors. Most participants self-corrected before submitting the login; however, one participant submitted a misspelled user name and promptly received an error message from the site. Participants were also instructed to log out of the account. After clicking “logout” one participant made the observation; “Huh, it goes to the login screen. I assume I'm logged out, though it doesn't say so.” The fourth task scenario involved using the library’s chat reference service via the mobile website. The chat reference service is provided via open source software in cooperation with L-net, the Oregon statewide service. Usability testing demonstrated that the chat reference service did not perform well on a variety of phones. Also, a significant problem arose when participants attempted to access chat reference via the university’s unsecured wireless network. Because the chat reference service is managed by a third-party host, three participants were confronted with a non-mobile friendly authentication screen (see discussion of the local wireless environment below). As this was an unexpected event in testing, participants were given the option to authenticate or abandon the task. All three participants who arrived at this point chose to move ahead with authentication during the test session. INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 54 Once the chat interface was available to participants, other system errors were discovered. Only three out of twelve participants successfully sent and received a chat message. Only one participant (HTC Droid Incredible) experienced an error-free chat transaction. Various problems encountered included: · unresponsive or slow to respond buttons, · text fields unresponsive to data entry, · unusually long page loading time, · non-mobile-friendly error message upon attempting to exit, and · non-mobile-friendly “leave a message” webpage. Another finding from this task is that participants expressed concern regarding communication delays during the chat reference task. If the librarians staffing the chat service are busy with other users, a new incoming user is placed in a queue. After waiting in the chat queue for forty seconds, one participant commented, “Probably if I was on the bus and it took this long, I would leave a message.” Being in a controlled environment, participants looked to the facilitator as a guide for how long to remain in the chat queue, distorting the indication of how long users would wait for a chat reference transaction in the field environment. Figure 4. Chat Reference Queue USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 55 The last task scenario asked participants to use the mobile version of EBSCOhost’s Academic Search Complete. Our test instance of this database generally performed well with Android phones and less well with webOS phones or iPhones. Android participants successfully accessed, searched, and viewed results in the database. iPhone users experienced delays in initiating text entry, three consecutive touches being consistently necessary to activate typing in the search field. Our feature phone participant with a Windows Kin 2 was unable to use EBSCOhost because the phone’s browser, Internet Explorer 6, is not supported by the EBSCOhost mobile website. The Palm Pixi participant also experienced difficulty with very long page loading times, two security certificate notifications (not present on other tests), and our EZproxy authentication page. With all these obstacles, the Palm Pixi participant abandoned the task. Another participant, Blackberry 9630, also abandoned the task due to slow page loading. A secondary objective of our EBSCOhost search task was to observe if participants explored EBSCOhost’s “Search Options” in order to limit results to scholarly articles. Our task scenario asked participants to find a scholarly article on global warming. Only one participant explored the EBSCOhost interface, successfully identified the “Search Options” menu, and limited the results to “scholarly (peer reviewed) articles.” Another participant included the words “peer reviewed” with “global warming” in the search field in an attempt to add the limit. A third expressed the need to limit to scholarly articles but was unable to discover how to do so. Of the remaining seven participants who searched Academic Search Complete for the topic “global warming” none expressed concern or awareness of the scholarly limit in Academic Search Complete. It is unclear whether this was a product of the interface design, users’ lack of knowledge regarding limiting their search to scholarly sources, or if our task scenario was simply too vague. Though participants’ wireless configurations, or lack thereof, was not formally part of the usability test, we quickly discovered that this variable had a significant impact on the user’s experience of the mobile website. In the introductory script and informed consent we recommended to participants that they connect to the university’s wireless network to avoid data charges. However, we did not explicitly instruct users to connect to the secure network. Most participants chose to connect to the unencrypted wireless network and appeared to be unaware of the encrypted network (PSU and PSU Secure respectively). Using the unencrypted network led to authentication requirements at two different points in the test: using the chat service and searching Academic Search Complete. Other users who were unfamiliar with adding a wireless network to their phone used their cellular network connection. These participants were asked to authenticate only when accessing EBSCOhost’s Academic Search Complete (see table 2). Participants expressed surprise at the appearance of an authentication request when performing different tasks, particularly while connected to the on-campus university wireless network. The required data entry in a non-mobile friendly authentication screen, and the added page loading time, created an obstacle for the participant to overcome in order to complete the task. Notably, three participants also explained their naivete on how to find and add a wireless network to their phone. INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 56 Internet connection Library mobile website Chat reference EBSCOhost On campus, unencrypted wireless No authentication required Authentication required Authentication required On campus, encrypted wireless No authentication required No authentication required No authentication required On campus, cellular network No authentication required No authentication required Authentication required Off campus, any mode No authentication required No authentication required Authentication required Table 2. Authentication Requirements Based on Type of Internet Connection and Resource. Post -Test Survey Each participant completed a post-test survey that asked them to rate the mobile website’s appearance and ease of use. The survey also asked participants to rank how frequently they were likely to use specific features of the website such as search for books and ask for help on a rating scale of more than weekly, weekly, monthly, less than monthly, and never. Participants were also invited to add general comments about the website. The mobile website’s overall appearance and ease of use was highly rated by all participants. The straightforward design of the mobile website’s homepage also garnered praise in the comment section of the post-test survey. Comments regarding the site’s design included: “Very simple to navigate,” and “The simple homepage is perfect! Also, I love that the site rotates sideways with my phone.” For many of the features listed on the survey participants selected an almost even distribution across the frequency of use rating scale. However, two features were ranked as having potential for very high use. Nine out of twelve participants said they would search for articles weekly or more than weekly. Eight out of twelve participants said they would use the “Find a Computer” function weekly or more than weekly. Two participants additionally wrote in comments that “Find a Computer” was “very important” and would be used “every day.” At the other end of the scale, our menu option “Directions” was ranked as having a potential frequency of use of never, with the exception of one participant marking less than monthly. DISCUSSION Usability testing of the library’s mobile website provided the team with valuable information, leading us to implement important changes before the site was launched. We quickly decided on a USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 57 few changes, while others involved longer discussion. The collections menu was removed from the catalog search; this menu distracted and confused users with options that were not useful in a general search. “Directions” was moved from a top level navigation element to a clickable link in the site footer. Also, the need for a mobile version of the library’s EZproxy authentication page was clearly documented and has since been created and implemented. However, the team was very pleased with the praise for the overall appearance of the website and its ease of use, especially considering the significant difficulties some participants faced when completing specific tasks. The “Find a Computer” feature of the mobile website was very popular with test participants. The potential popularity among users is perhaps a reflection of overcrowded computer labs across campus and the continued need students have for desktop computing. Unfortunately, “Find a Computer” has been temporarily removed from the site due to changes in computer laboratory tracking software at the campus IT level. We hope to soon again have access to the workstation data for the library’s two computer labs in order to develop a new version of this feature. The hesitation participants displayed when selecting the catalog option in order to search for a book was remarkable for its pervasiveness. It’s possible that the term “catalog” has declined in use to the point of not being recognizable to some users, and it is not used to describe the search on the homepage of the library’s full website. In fact, we had originally planned to name the catalog search option with a more active and descriptive phrase, such as “Find books and more,” which is used on the library’s full website. However, the full library website employs WorldCat Local, allowing users to make consortial and interlibrary loan requests. In contrast, the mobile website catalog reflects only our local holdings and does not support the request functionality. The team decided not to potentially confuse users further regarding the functionality of the different catalogs by giving them the same descriptive title. In the case that WorldCat Local’s beta mobile catalog increases in stability and functionality, we will abandon MobileCat and provide the same request options on the mobile website as on the full website. We discussed removing the chat service option from the “Ask Us” page. During usability testing, it was demonstrated that users would too frequently have poor experiences using this service due to slow page loads on most phones, the unpredictable responsiveness of text entry fields and buttons, and the wait time for a librarian to begin the chat. Also, it could be that waiting in a virtual queue on a mobile device is particularly unappealing because the user is blocked from completing other tasks simultaneously. The library recently implemented a new text reference service, and this service was added to the mobile website. The text reference service is an asynchronous, non-web- based service that is less likely to pose similar usability problems as those found with the chat service. This reflects the difference between applications developed for desktop computing, such as web-based instant messaging, versus a technology that is specifically related to the mobile phone environment, like text messaging. However, tablet device users complicate matters since they might use the full desktop website or the mobile website; for this reason, chat reference is still part of the mobile website. INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 58 Participants’ interest in accessing and searching databases was notable. During the task, many participants expressed positive reactions to the availability of the EBSCOhost database. The post- test survey results demonstrated a strong interest in searching for articles via the mobile website, giving their potential frequency of use as weekly or more than weekly. This evidence supports the previous user focus group results of Seeholzer and Salem.19 Students are interested in accessing research databases on their mobile devices, despite the likely limitations of performing advanced searches and downloading files. Therefore, the team decided to include EBSCOhost’s Academic Search Complete along with eight other mobile-friendly databases in the live version of the website launched after the usability test. Figure 5. Home Page of the Library Mobile Website, Updated USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 59 The new library mobile website was launched in the first week of fall 2011 quarter classes. In the first full week there were 569 visits to the site. Site analytics for the first week also showed that our distribution of smartphone models in usability testing was fairly well matched with the users of the website, though we underestimated the number of iPhone users: 64 percent of visits were from Apple iOS users, 28 percent from Android users, 0.7percent Blackberry users, and the remaining a mix of users with alternative mobile browsers and desktop browsers. Usability testing with participants’ native smartphones and wireless connectivity revealed issues which would have been absent in a laboratory test that employed a mobile device emulator and a stable network connection. The complications introduced by the encrypted and unencrypted campus wireless networks, and cellular network connections, revealed some of the many variables users might experience outside of a controlled setting. Ultimately, the variety of options for connecting to the Internet from a smartphone, in combination with the authentication requirements of licensed library resources, potentially adds obstacles for users. General recommendations for mobile library websites that emerged from our usability test include: · users appreciate simple, streamlined navigation and clearly worded labels; · error message pages and other supplemental pages linked from the mobile website pages should be identified and mobile-friendly versions created; · recognize that how users connect to the mobile website is related to their experience using the site; · anticipate problems with third-party services (which often cannot be solved locally). Additionally, system responses to user actions are important; for example, provide a “you have successfully logged out” message and an indicator that a catalog search is in progress. It is possible that users are even more likely to abandon tasks in a mobile environment than in a desktop environment if they perceive the site to be unresponsive. As test facilitators, we experienced three primary difficulties in keeping the testing sessions consistent. The unexpectedly poor performance of the mobile website on some devices required us to communicate with participants about when a task could be abandoned. For example, after one participant made three unsuccessful attempts at entering text data in the chat service interface, she was directed to move ahead to the next task. Such instances of multiple unsuccessful attempts were considered to be fatal system errors. However, under these circumstances, it is difficult to know whether our test facilitation led participants to spend more or less time than they normally would attempting a task. Secondly, the issue of system authentication led to unexpected variation in testing. Some participants proceeded through these obstacles, while others either opted out or had significant enough technical difficulties that the task was deemed a fatal error. Again, it is unclear how the average user would deal with this situation in the field. Some users INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 60 might leave an activity if an obstacle appears too cumbersome, others might proceed. Finally, participants demonstrated a wide range in their willingness to “think aloud.” In retrospect, as facilitators, we should have provided an example of the method before beginning the test; perhaps doing so would have encouraged the participants to speak more freely. The relatively simple nature of most of the test tasks may have also contributed to this problem as participants seemed reluctant to say something that might be considered too obvious. Another limitation of our study is that the participants were a convenience sample of volunteers selected by phone type. Though our selection was based loosely on market share of different smartphone brands, a preliminary investigation into the mobile device market of our target population would have been helpful to establish what devices would be most important to test. Additional usability testing on more complex library related tasks, such as advanced searching in a database, or downloading and viewing files, is recommended for further research. Also of interest would be a study of user willingness to proceed past obstacles like authentication requirements and non-mobile friendly pages in the field. CONCLUSION We began our study questioning whether or not different smartphone hardware and operating systems would impact the user experience of our library’s new mobile website. Usability testing confirmed that the type of smartphone does have an impact on the user experience, occasionally significantly so. By testing the site on a range of devices, we observed a wide variation of successful and unsuccessful experiences with our mobile website. The wide variety of phones and mobile devices in use makes developing a mobile website that perfectly serves all of them difficult; there is likely to always be a segment of users who experience difficulties with any given mobile website. However, usability testing data and developer awareness of potential problems will generate positive changes to mobile websites and alleviate frustration for many users down the road. REFERENCES AND NOTES 1. Aaron Smith, “35% of American Adults Own a Smartphone: One Quarter of Smartphone Owners Use Their Phone for Most of Their Online Browsing,” Pew Research Center, June 15, 2011, http://pewinternet.org/~/media//Files/Reports/2011/PIP_Smartphones.pdf (accessed Oct. 13, 2011). 2. Shannon D. Smith and Judith B. Caruso, The ECAR Study of Undergraduate Students and Information Technology, 2010, EDUCAUSE, 2010, 41, http://net.educause.edu/ir/library/pdf/ERS1006/RS/ERS1006W.pdf (accessed Sept. 12, 2011); Shannon D. Smith, Gail Salaway, and Judith B. Caruso, The ECAR Study of Undergraduate Students and Information Technology, 2009, EDUCAUSE, 2009, 49, http://www.educause.edu/Resources/TheECARStudyofUndergraduateStu/187215 (accessed Sept. 12, 2011). http://pewinternet.org/~/media/Files/Reports/2011/PIP_Smartphones.pdf http://net.educause.edu/ir/library/pdf/ERS1006/RS/ERS1006W.pdf http://www.educause.edu/Resources/TheECARStudyofUndergraduateStu/187215 USABILITY STUDY OF A LIBRARY’S MOBILE WEBSITE | PENDELL AND BOWMAN 61 3. A comparison count of U.S. and Canadian academic libraries with active mobile websites, wiki page versions, August 2010 (56 listed) and August 2011 (84 listed). Library Success: A Best Practices Wiki, “M-Libraries: Libraries Offering Mobile Interfaces or Applications,” http://libsuccess.org/index.php?title=M-Libraries (accessed Sept. 7, 2011). 4. Laurie M. Bridges, Hannah Gascho Rempel, and Kim Griggs, “Making the Case for a Fully Mobile Library Web Site: From Floor Maps to the Catalog,” Reference Services Review 38, no. 2 (2010): 317, doi:10.1108/00907321011045061. 5. Kim Griggs, Laurie M. Bridges, and Hannah Gascho Rempel, “Library/Mmobile: Tips on Designing and Developing Mobile Web Sites,” Code4Lib Journal no. 8 (2009), under “Content Adaptation Techniques,” http://journal.code4lib.org/articles/2055 (accessed Sept. 7, 2011). 6. Jamie Seeholzer and Joseph A. Salem Jr., “Library on the Go: A Focus Group Study of the Mobile Web and the Academic Library,” College & Research Libraries 72, no. 1 (2011): 19. 7. Dongsong Zhang and Boonlit Adipat, “Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,” International Journal of Human-Computer Interaction 18, no. 3 (2005): 302, doi:10.1207/s15327590ijhc1803_3. 8. Griggs, Bridges, and Rempel, “Library/Mobile.” 9. Zhang and Adipat, “Challenges, Methodologies,” 303–4. 10. Billi et al., “A Unified Methodology for the Evaluation of Accessibility and Usability of Mobile Applications,” Universal Access in the Information Society 9, no. 4 (2010): 340, doi:10.1007/s10209-009-0180-1. 11. Zhang and Adipat, “Challenges, Methodologies,” 302. 12. Jakob Nielsen, “Mobile Usability,” Alertbox, September 26, 2011, www.useit.com/alertbox/mobile-usability.html (accessed Sept. 28, 2011). 13. Fernando Loizides and George Buchanan, “Performing Document Triage on Small Screen Devices. Part 1: Structured Documents,” in IIiX ’10: Proceeding of the Third Symposium on Information Interaction in Context, ed. Nicholas J. Belkin and Diane Kelly (New York: ACM, 2010), 342, doi:10.1145/1840784.1840836. 14. Constantinos K. Coursaris and Dan J. Kim, “A Qualitative Review of Empirical Mobile Usability Studies” (presentation, Twelfth Americas Conference on Information Systems, Acapulco, Mexico, August 4–6, 2006), 4, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.4082&rep=rep1&type=pdf (accessed Sept. 7, 2011) 15. Ibid., 2. http://libsuccess.org/index.php?title=M-Libraries http://journal.code4lib.org/articles/2055 file:///C:/Users/GERRITYR/Desktop/ITAL%2031n2_PROOFREAD/www.useit.com/alertbox/mobile-usability.html http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.4082&rep=rep1&type=pdf INFORMATION TECHNOLOGY & LIBRARIES | JUNE 2012 62 16. Jeffrey Rubin and Dana Chisnell, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd ed. (Indianapolis, IN: Wiley, 2008); Carole A. George, User-Centered Library Web Sites: Usability Evaluation Methods (Cambridge: Chandos, 2008). 17. Ronan Hegarty and Judith Wusteman, “Evaluating EBSCOhost Mobile,” Library Hi Tech 29, no. 2 (2011): 323–25, doi:10.1108/07378831111138198; Robert C. Wu et al., “Usability of a Mobile Electronic Medical Record Prototype: A Verbal Protocol Analysis,” Informatics for Health & Social Care 33, no. 2 (2008): 141–42, doi:10.1080/17538150802127223. 18. In order to protect participants’ confidentiality a dummy library user account was created; the user name and password for the account were provided to the participant at the test session. 19. Seeholzer and Salem, “Library on the Go,” 14.