![[Refereed]](http://www.istl.org/graphics/r2.gif){kind=small}
A Framework for Evaluating Science and Technology Electronic Reference Books: A Comparison of Five Platforms in Chemistry
Chemistry & Chemical Engineering Librarian
Science & Engineering Library
University of Minnesota
Minneapolis, Minnesota
mlaffert@umn.edu
Abstract
This article examines what is desirable in online reference books in science and technology and outlines a framework for evaluating their interfaces. The framework considers factors unique to these subject areas like chemical structures and numerical data. Criteria in three categories, navigability, searchability, and results, were applied to five key chemistry e-resources. Some e-book publishers are taking e-books farther and changing expectations for the rest. The framework offers a guide for holding publishers accountable for developing e-reference books that meet users' needs.
Introduction
The number of classic reference works now available online is growing rapidly, and libraries are making concerted efforts to add them to their collections largely because users prefer to access information this way (Webster 2003). Reference books are a natural fit for the online environment as they are geared toward consultation on an as-needed basis for smaller pieces of information rather than sustained reading. However, just being online is not sufficient to make an online reference source useful. Trustworthy content is as important as it always has been, but the ease of using a resource to find the desired information is equally crucial. Print reference book "interfaces" do not vary as widely as those of e-books. Evaluating reference books in the electronic environment requires different considerations.
What is desirable in a reference e-book? As e-books in general become more sophisticated, expectations should accordingly be higher. In the process of examining some key chemistry reference books in electronic form, I developed criteria for what features are important, taking into consideration some factors that are unique to science and engineering reference books, chemistry in particular. The major goal is to give librarians a framework with which to make collection decisions and hold publishers accountable. This is especially important now given budgetary limitations and the current economic situation.
Literature Review
Reference books are a better fit for electronic access than other types of books such as fiction or textbooks. They are used for quick consultation, not sustained reading, a type of usage that meshes well with how people often read online -- scanning and searching for useful pieces of information rather than reading content in its entirety (Gunter 2005; Liu 2005; Safley 2006). The ability to offer packages of online reference books, which allow users to search many titles at once, adds to their suitability for the electronic environment; it is both more efficient and increases the user's chances of finding useful information in books they might not otherwise have consulted because of their age or subject matter (Blankenship & Leffler 2006; Miller 2005).
We are not yet at a point when reference e-books are the clear first choice of users although they may go to them for purposes of convenience (Robbins et al. 2006). Finding information in an e-book is not necessarily faster than print, and users will stick with print if they find it easier (Webster 2003). On the basis of student feedback on ENGnetBASE, an online collection of engineering handbooks, Barbara Williams, engineering librarian at the University of Arizona, decided to cancel the library's subscription as the students found it easier to locate information using the print version of a handbook than the electronic. In experiments comparing how quickly groups of students could find the same information in the print versus the electronic versions of handbooks, the students using print were faster. The same thing happened when the student groups switched type of book and when Williams repeated the experiment with a group of librarians. The students found the user interface unintuitive, and searches often retrieved far too many hits (Williams 2007). Another librarian's comparison of how quickly students found information in Statistical Abstracts in print versus online had similar findings (Van Epps 2005).
Web site usability has been addressed extensively elsewhere and informs my criteria, particularly with respect to navigation (Holmes 2002; Horton 2006; Johnson 2003; Lazar 2006; Mander & Smith 2002; Menon & Douma 2008). As e-book development has progressed over the last decade, researchers have focused attention on desirable design features (Crestani et al. 2006; Landoni et al. 2000; Sun et al. 2004; Wilson et al. 2002). Considerations for online reference books are not dissimilar from those of other e-books, but there are differences, namely the importance of search. Some work has been done on evaluating resources central for science and technology libraries like e-journals, library databases, and scholarly books in general, but there has been little done to examine the unique needs of science and technology, specifically chemistry, reference books online (Coker 2007; Sabin-Kildiss et al. 2001; Shiao-Feng Su 2005; Vilar & Žumer 2005).
Methods
The focus of this article is reference works with chemistry-related content that do not require any special equipment for access other than a computer with an Internet connection. Article indexes are outside the scope of this article; there is little doubt that users have been thoroughly converted. I also excluded electronic book collections without significant amounts of chemistry content like NetLibrary. Resources covered include both single titles and collections.
I compared the interfaces of CHEMLIBnetBASE; CHEMnetBASE and its contents; Kirk-Othmer Encyclopedia of Chemical Technology; Knovel; and Merck Index. There are differences between these five sources, but all have content devoted to chemical substances, their properties, and other information about them. The five e-books are largely considered well-regarded and essential reference sources for chemistry.
CHEMLIBnetBASE is produced by Chapman and Hall/CRC (see Figure 1). Its focus is chemistry-related reference books although some of the over 500 books in this searchable collection, including some handbooks, might be considered monographs rather than reference (CRC Press 2009).
CHEMnetBASE is another Chapman and Hall/CRC collection of reference books with a very different interface (see Figure 2). It includes the classic Handbook of Chemistry and Physics, or CRC, as chemists commonly refer to it, which is often the first reference source a chemist uses; the Combined Chemical Dictionary (CCD), a source of basic information on a variety of chemical substances that is comprised of the Dictionary of Commonly Cited Compounds, Dictionary of Drugs, Dictionary of Inorganic and Organometallic Compounds, Dictionary of Natural Products, and Dictionary of Organic Compounds; the Polymer Properties Database; and Properties of Organic Compounds (POC), the online equivalent of the Handbook of Data on Organic Compounds or HODOC. Unlike CHEMLIBnetBASE, each resource has a separate interface although the CCD, Polymers, and individual dictionary interfaces are very similar. Except for the CCD, it is not possible to search multiple titles at once. Three additional dictionaries covering carbohydrates, food compounds, and marine natural products have recently been added to CHEMnetBASE but are not currently included in the CCD.
Kirk-Othmer Encyclopedia of Chemical Technology, published by Wiley, is arguably the most important and best known chemistry encyclopedia (see Figure 3). It covers a wide range of industrially important chemical substances as well as relevant basic science (Maizell 1998).
Knovel is an online collection of about 2000 scientific and engineering reference books (see Figure 4) (Knovel 2003). It includes classic chemistry and chemical engineering reference works such as the Lange's Handbook of Chemistry, Perry's Chemical Engineer's Handbook, and Polymer Handbook (Knovel 2008).
The Merck Index is another classic chemistry reference source and, like the CRC, often one of the first with which chemists are acquainted (see Figure 5). The content is produced by Merck, the pharmaceutical company; I examined the CambridgeSoft online version. Merck provides information on names, properties, uses, and more for a wide range of biologically and pharmaceutically relevant chemical substances. It is also available through Dialog, Knovel, Medicines International, and STN International. Access to Merck via Knovel is offered separately from the Knovel package.
My primary interest is the electronic books' interfaces and ease of use rather than the content. My criteria are listed in Table 1. Distinctions between categories can be fuzzy, and there is a bit of overlap.
| Table 1. Criteria for Evaluation | ||
|---|---|---|
Navigability
| Searchability
| Results
|
Results and Discussion
Navigability
Clean, uncluttered design
Simplicity in web design is more highly valued now than in the early days of the web. This is evident in that most resources examined have relatively clean and uncluttered design. The chief exception is CHEMLIBnetBASE. There is almost no white space on the home page to give users' eyes relief, and quite a bit of content is below the fold. All titles are in a single long list on the homepage along with boxes for content categories, newly added titles, and promotions for related Taylor and Francis books and journals. The entry point for the web site is unnecessarily overwhelming. On the positive side, the sidebar content on related books is dynamic and only includes cover images and titles of two books at a time.
Efficient use of space
Web sites consist of fairly small spaces, so making key information visible while keeping less crucial content accessible is especially important (Nielsen 2003). On search pages, allowing users to add search fields to a short list of those most commonly used is one way to maximize space; the CCD, Polymers, and the CRC structure/property search page do this (see Figures 6, 7, and 8). Tabs indicating different areas of a web site are another strategy that both Knovel and Merck use.
The POC makes reasonably good use of its space although a "Browse Index" button for all 18 search boxes seems unnecessary (see Figure 9). A single browsing link with a drop-down menu for the indexes would be preferable. CHEMLIBnetBASE has very little content in the top horizontal navigation bar. Instead many options are crowded into the left vertical navigation bar, a wasted opportunity since some (e.g., "Download Sales and Information Sheet") would be more appropriate elsewhere on the web site.
Minimal clicks to search
Extra mouse clicks mean more time, frustrating to users. Getting to a search box, so crucial to a web site's functionality, is an obvious place to minimize clicks (Nielsen 2001). CHEMnetBASE, POC, Knovel, and Merck all add at least one extra step. The CHEMnetBASE homepage has no search functionality; users access its content via the individual title links. POC requires another click beyond CHEMnetBASE and its own entry page to search. Knovel's opening page keeps changing and currently includes company information and other content in addition to a link to the search page. The Merck entry point is a menu including two search options and information about the resource (e.g., help); this content could easily be placed, and is mostly duplicated, in navigation bars on the search page.
Unambiguous site navigation
Unambiguous web site navigation is a must for a useful interface. As Donald Norman pointed out in Design of Everyday Things, it should be obvious how to operate something (Norman 2002). Navigation through problematic e-books is not uniformly bad; in CHEMLIBnetBASE, Kirk-Othmer, Merck, and POC, a single, albeit important, confusing feature stands out.
Navigating CHEMLIBnetBASE is relatively straightforward. The location and function of the search box and browsing lists are obvious as is getting to a book's text via search; however, browsing is not. Choosing where to click for access to the book is a guessing game. Clicking on the chapter titles in the Table of Contents tab seems sensible as do the book icon or the book's title or image (see Figure 10). However, none of these are links. Though both plain black and not underlined, the text "Read It Online!" is the active link. This non-obviousness certainly violates Norman's principles for good design. In contrast, for books in Knovel, the table of contents consists of underlined grey links that state "Text" beside the chapter or section name that turn blue when moused over. It is obvious where to click because users are accustomed to looking for underlined links.
Kirk-Othmer has two search boxes. The top box searches Kirk-Othmer, and the bottom one searches all of Wiley Interscience which includes journals, databases, reference e-books, and more across all subjects (John Wiley & Sons, Inc. 2008). Depending on the box selected, vastly different results are retrieved. The search box location on the Kirk-Othmer site, while not terrible, is also not ideal. There is not yet a standard search box location on web sites, but eye-tracking studies provide evidence that people frequently have a roughly F-shaped browsing pattern (Nielsen 2004; Nielsen 2006). The search box location on the right side, part way down the page, is not where people's eyes are more likely to scan. Also search boxes are not consistent site-wide as only the Kirk-Othmer search box is on results pages.
Another example of confusing navigation is the nearly identical gray oval buttons to the left of substance records in a Merck list of hits. The top button reads "Mark Record" in order to export or combine sets, and the bottom says "Show Details" to open the record, a more likely user action and thus a better choice for the top button.
The location and appearance of the search button in POC are especially non-obvious. No lines or color changes clearly indicate it is a link or button. A "Browse Index" button, which takes the user to the field's index, is located where search buttons commonly are. Users, accustomed to clicking search buttons to the right of search boxes without much thought, can easily end up in an index unexpectedly.
User-friendly language
An easily navigable web site avoids language potentially confusing to novice users. None of the resources are entirely free of jargon; most refer to Boolean operators and other similar terms in their help sections. Merck had some of the most glaring examples with an "Explanatory Monograph" link in the help menu and options to "Manage" the "Hit List" by choosing to intersect, subtract, or union saved and current hit lists. The CHEMLIBnetBASE advanced search offers "Stemming," "Phonic," and "Natural Language" options, all unlikely to be meaningful to non-expert users.
Help availability
Librarians have long known that users rarely open help, but its availability is still important (Capobianco & Carbonell 2003). The searchable sites all offer some form of help. Contextual help or at least a searchable help section is preferable. No resource has a searchable help section; though Knovel's suggested search strategies under the basic search box and "More Search Tips" in Kirk-Othmer could be considered forms of contextual help (see Figures).
Help should be easy to locate. For CCD, CRC, Kirk-Othmer, Knovel, Merck, and Polymers, the links are clearly labeled and spatially set apart from other links (e.g., at the end of a row or column), but I repeatedly had difficulty locating what seems to be the CHEMLIBnetBASE help section, "How It Works." The wording and the link's placement in the middle of a left navigation bar make it less obvious. Oddly, its help section only describes how to enter search terms and view results whereas the presumably similar ENGnetBASE links to its much larger help section under its search box.
The Merck help section was the most awkward. Clicking "Help" links on the search page opens a menu with four options: search screen guide, user search guide, explanatory monograph, and therapeutic index category terms. The number of choices and language used seem likely to overwhelm the novice. All the links open PDF files, more sensible for content likely to be printed out rather than scanned for relevant information, more typical for help sections and easier with HTML.
Summary
No resource was without navigability flaws (see Table 2 for an overview). POC is the most problematic; the copyright date at the bottom of the opening page, 1998-2005, gives a likely indication as to why. Presumably the site has not been updated as web standards have evolved. Given other recent changes to CHEMnetBASE interfaces, improvements may be forthcoming. Merck is slightly better but still very clunky and awkward to navigate. Given Merck's importance, the Cambridgesoft interface's flaws were especially disappointing; it was much easier to use via Knovel. Knovel is the simplest to use overall, and Kirk-Othmer is the next best.
Table 2: Navigability
Searchability
Browse
Browsing is still relevant and important for users; they may be unsure of what they are seeking, how to describe it, or even how to spell it (Beall 2007). Browsing can lead to serendipitous discoveries of information in unexpected or unfamiliar places, and it can also help locate information described in broader, narrower, different, or otherwise related terms a searcher might miss (Bates 2007; Beall 2007). Bodoff suggests that relevance for browsers and searchers are not necessarily the same, even more reason to include browsing functionality in e-books (Bodoff 2006). All the works except for Merck had browsing options (though it is possible to browse the Human Therapeutic Use index terms, one of 26 search fields).
Basic search
A user's entry point ought to include basic search, i.e., a single box for entering keyword searches. This is what users want and will primarily use. All of the resources except Merck, CCD, POC, and Polymers include a simple search option. They send users directly to potentially overwhelming pages with numerous search fields though CCD and Polymers are less problematic with a short list of commonly used properties to search and the option to add or hide properties. However, nearly every searchable field seems to be on the Merck and POC default search pages including some irrelevant for most users like "Monograph #" (Merck) and "Beilstein Reg. No." (POC). Merck's other search option is the addition of a chemical structure drawing box to their basic search. POC has no other search options nor does it have free text search. There are at least 31 fields in substance records, but only 17 are searchable. Some excluded fields like Health and Safety Information may well be of interest to users.
The resources vary in implicit basic search functionalities. With a two or more word search string, CHEMLIBnetBASE, Merck, and Polymers search for phrases; the CRC and Kirk-Othmer include AND between search terms; and Knovel treats it like a proximity operator, looking for words within five to ten words of each other. Kirk-Othmer automatically searches for truncated forms of words. Knovel will search for singular and plural forms of words, a limited type of truncation.
Advanced search
Not all users are satisfied with basic search alone so advanced search should be available on the web site. Admittedly, advanced search is of more interest to information professionals, but we are also online reference book users. Advanced search need not be and, in fact, ought not be what users encounter first to avoid confusing them (Nielsen 2001).
All the resources had an advanced search option though there is little consistency among them, even those in CHEMnetBASE. Commonly available options are Boolean operators, truncation, and proximity searching. CHEMLIBnetBASE, Kirk-Othmer and Knovel allow the use of Boolean operators AND, OR, and NOT in search fields whereas CCD, POC, and Polymers only permit AND or OR as options in drop-down boxes between fields. Only Merck does not allow the truncation (*) and wildcard (?) characters. CRC and Polymers allow additional truncation (%) and wildcard (_) symbols. CHEMLIBnetBASE has a formal proximity operator option (w/N where N is the maximum number of words that can occur between two search terms), and proximity searching is implicit with phrases not in quotation marks in Knovel.
The ability to search for specific types of information is important for narrowing results and avoiding false or irrelevant hits. Most of the resources are data-centric and thus have many searchable fields, some presented better than others. As mentioned earlier, the default for Merck and POC is almost all fields. CCD, CRC, and Polymers present options less overwhelmingly by allowing users to add or subtract desired fields; Knovel has drop-down boxes for categories of properties. Available fields vary widely. Knovel allows great specificity and the ability to get directly to relevant data with the capacity to search for types of properties, specific properties, and ranges of values for the properties. Kirk-Othmer, much less data-focused, has fewer searchable fields than the others; they include CAS Registry Numbers, compounds, tables, and figures.
Less common is the ability to search a range of values or just molecular formulas, very useful for e-books with a lot of numerical or chemical data. The CHEMnetBASE resources, Knovel, and Merck let users search ranges of values. Search syntax varies widely between resources; they may require hyphens between numbers, spaces, or spaces on either side of a hyphen. Knovel provides a box for each value. Searching molecular formula fields as in Merck and the CHEMnetBASE titles usually requires Hill order (C, then H, then alphabetical), though it may or may not be case sensitive.
Chemical thesauri
Chemical substance searching has unique challenges with so many possible names for chemical compounds plus CAS Registry Numbers, molecular formulas, and chemical structures. Naïve users may conclude that a substance is not included in a book or database because of their choice of name. Automatically searching thesauri is a distinct advantage possible for e-books that is especially warranted by chemical names' numerous synonyms. CCD, Merck, POC, and Polymers have synonym fields in their substance records, their major emphasis. Knovel has synonym tables for many chemistry titles though it did not appear to be comprehensive on the basis of chemical name variations searches.
Chemical structure searching
Chemical structure searching is helpful for compounds with long or many names. It is a nice-to-have feature as it is best for organic compounds since some types of compounds like alloys or polymers cannot be accurately represented as structures (Cooke & Ridley 2004). Chemical structure searching is available for Merck and the CCD, CRC, POC, and the Monomers section of Polymers in CHEMnetBASE, all good candidates for structure drawing with significant organic content. Structure searching is less logical for Knovel since chemistry makes up only a small portion of the whole although it would be nice for heavily organic works included like Merck. Arguments could be made for and against its importance for Kirk-Othmer and CHEMLIBnetBASE, both primarily chemistry resources but far less focused on chemical substance data.
Metasearch
Searching across many books simultaneously is a key advantage possible with e-books. Of the three multi-title resources examined, only CHEMnetBASE lacked this option; it seems to be merely a portal, without any function other than giving a name to a package of databases and e-books for purposes of marketing. CHEMLIBnetBASE is structured differently from CHEMnetBASE; it is possible to search across its contents. If CHEMnetBASE were searchable as Knovel is, users would be more likely to learn about and use content other than the CRC, its most visible and well-known resource.
Summary
It is easier to say which resources are my least favorite ones to search than which is best (see Table 3). As with navigability, POC ranks near the bottom though it allows structure searching. Merck and CHEMLIBnetBASE are also far from ideal. However, all have pros and cons. Technical difficulties searching Polymers detracted from its positives. The most significant negative overall is a frustrating inconsistency in search options across resources; it may be less problematic for most given user propensity for one or two word searches (Spink & Wolfram 2001).
Table 3: Searchability
Results
Clearly presented
Overall results were easy to read and work with, but small changes would benefit several of the resources. It would be nice if CCD, POC, and Polymers showed search terms on the results page. Links in a CHEMLIBnetBASE search results list are black and not underlined, making it unclear that they are clickable without mousing over them, and all 6 fields (e.g., Book or Chapter Title) in a record are links, each going to the same document. Also, there is no color change for visited links, violating a key principle for web site usability (Nielsen 2008). The POC results list is in a difficult to see pale purple font. Items in the CRC results list can have two links which is potentially confusing; one goes to where search terms occur in the PDF version and another to an "interactive table" limited to relevant data.
Search terms highlighted
Highlighted search terms save user time by preventing the need to scan large amounts of text. For e-books with short entries like CCD, Merck, or POC, none of which highlight search results, it may be less crucial, but it is still helpful, somewhat analogous to daytime running lights on cars, not strictly necessary but a benefit to other drivers. Kirk-Othmer and Polymers highlight search results in HTML records. Search terms are not highlighted in the PDF versions of Kirk-Othmer entries. CHEMLIBnetBASE, CRC, and Knovel highlight results in PDFs and go directly to the words' first occurrence in the text, another time-saving feature. CRC and Knovel also pull relevant data into tables for data-intensive content, taking advantage of what electronic texts can do and more ought to do.
Relevancy ranking
CCD, POC, and Merck, comprised chiefly of short entries with basic substance information, arrange results in alphabetical and numerical order. Kirk-Othmer and Knovel rank results by relevancy to search terms by default. CHEMLIBnetBASE and Polymers rank results by number of hits, a rough measure that does not take into account the proximity of words or where in a document they are located. However, it is preferable to the CRC method of chapter/section order which is potentially time-consuming for a frequently occurring search term. It also is more likely that users will miss relevant information given how rarely most look beyond the first few hits. Ranking results by alphabetical or chapter order or number of hits makes it obvious why one result comes before another, likely of little interest to most users. Results ranking methods are not as transparent, but relevancy is usually preferable for getting to information quickly.
Ease of working with results
Sorting by relevancy, subject, or title name is easy in Knovel with the click of a radio button. Kirk-Othmer and CCD also allow sorting; Kirk-Othmer can sort by match percent, date, or article title, and CCD can sort by chemical name or molecular formula. None of the others do. There are many reasons why the ability to sort results in different ways is useful. For example, scanning an alphabetical list is faster for finding a known article or book not highly ranked by relevancy.
For CHEMLIBnetBASE, Kirk-Othmer, Polymers, and POC, downloading data or results amounts to saving or printing HTML or PDF files. Knovel and CRC allow this as well as the data tables mentioned previously. PDF image quality is occasionally poor in older Knovel titles, possibly unavoidable for some originals.
Knovel's interactive graphs, scanned from the text and digitized or generated from equations, set it apart from the other resources. Users can click on points in a graph to determine x and y values and save them to a table, very useful for tasks like finding density at different temperatures, for example. Graphs only make sense for certain types of information, so it is nice-to-have. It capitalizes on what e-books can do and would makes sense for other resources as well.
Another way Knovel takes advantage of the potential of e-books is the ability to pull data into tables and sort or filter it, enabling users to focus on properties of interest or group substances with similar values for a property; the CRC has similar capabilities. Both also allow data in tables to be selected and exported to an Excel spreadsheet. Knovel has some additional options for manipulating data tables like hiding columns. As with plotting data, the ability to filter, sort, and export data does not make sense for every reference e-book but more could benefit from this treatment. CCD allows exporting a list of results to Excel, but it only includes the chemical name and molecular formulas.
Summary
In addition to technical problems searching Polymers, I had repeated problems accessing its substance records. CRC and Kirk-Othmer are doing a lot of things right (see Table 4). Knovel has some of the most innovative options for dealing with data and results. Though both text-heavy, content in CHEMLIBnetBASE and Kirk-Othmer could certainly benefit from similar treatment.
Table 4: Results
Conclusions
What makes a good electronic reference book? The real test of an e-book's value is whether it is used, particularly if there is also print access. It is telling that over half the libraries surveyed by Robbins, et al. held the CRC in both print and electronic formats (Robbins et al. 2006). Its status as a classic and well-used reference book is a factor, but would users still be so tied to the print if the electronic version were as quick and easy to use? Format duplication is not unique to the CRC as most ready reference titles were held in print and electronically by about one-third of the libraries in the Robbins study (Robbins et al. 2006). Some of the reluctance to ditch duplicate print reference sources is motivated by concerns like guaranteed perpetual access to content, but ease of use is crucial, too.
For our users, quickly finding the desired information is their central concern when they consult a reference book, and many factors can affect the speed of this process with e-books. Some stumbling blocks are unpredictable and probably not currently avoidable, namely technical problems unknown in the print world like incompatible browsers or operating systems, requirements for plug-ins, pages that load slowly, or malfunctioning servers. However, problems related to an e-book's design and functionality like cluttered pages, confusing terminology, and non-obvious links also affect user efficiency. Users can evaluate search results more quickly if relevant data has been pulled into tables or search terms have been highlighted. Features unique to Knovel like interactive data and the ability to search for property information at a very granular level across many sources can help make users productive in ways not previously imagined. They are shifting the paradigm for what users can and should expect from online science and technology reference books.
The framework described provides guidance for identifying potential ways a users' work might be helped or hindered by an online reference book. Unlike technical glitches, the criteria addressed are not entirely out of our control. Holding vendors accountable for product quality, evaluating and sharing feedback before we buy or lease, and the willingness to walk away products that do not meet our standards is imperative. E-reference books have improved greatly over the last decade. Although changes can be slow, vendors respond to librarian feedback on their products.
References
Bates, M.J. 2007. What is browsing -- really? A model drawing from behavioural science research. Information Research 12(4):1-17.
Beall, J. 2007. The value of alphabetically-sorted browse displays in information discovery. Library Collections Acquisitions & Technical Services 31(3-4):184-194.
Blankenship, L. & Leffler, J. 2006. Where are the reference books? Colorado Libraries 32(2):10-13.
Bodoff, D. 2006. Relevance for browsing, relevance for searching. Journal of the American Society for Information Science & Technology 57(1):69-86.
Capobianco, A. & Carbonell, N. 2003. Online help for the general public: specific design issues and recommendations. Universal Access in the Information Society 2(3):265-279.
Coker, G. 2007. Usability in emerging e-content environments. Information Services & Use 27(4):179-184.
Cooke, H. & Ridley, D.D. 2004. The challenges with substance databases and structure search engines. Australian Journal of Chemistry 57(5):387-392.
CRC Press. 2009. CHEMLIBnetBASE: Chemistry References Online. [Online]. Available: http://chemlibnetbase.com/ [Accessed May 4, 2009].
Crestani, F., Landoni, M. & Melucci, M. 2006. Appearance and functionality of electronic books. International Journal on Digital Libraries 6:192-209.
Gunter, B. 2005. Electronic books: a survey of users in the UK. Aslib Proceedings 57(6):513-522.
Holmes, M. 2002. Web Usability and Navigation: A Beginner's Guide. Norwood Mass.: Books24x7. [Online]. Available: http://www.books24x7.com/marc.asp?isbn=0072192615 [Accessed May 4, 2009].
Horton, S. 2006. Access by Design: A Guide to Universal Usability for Web Designers. Berkeley, Calif.: New Riders.
John Wiley & Sons, Inc. 2008. Wiley InterScience: Home. [Online]. Available: http://www3.interscience.wiley.com/cgi-bin/home [Accessed May 4, 2009].
Johnson, J. 2003. Web Bloopers: 60 Common Web Design Mistakes and How to Avoid Them. Norwood Mass.: Books24x7. [Online]. Available: http://www.books24x7.com/marc.asp?isbn=1558608400 [Accessed May 4, 2009].
Knovel - Company Overview. 2008. New York: Knovel Corporation. [Online]. Available: {http://why.knovel.com/company/about-knovel.html} [Accessed May 4, 2009].
Landoni, M., Wilson, R. & Gibb, F. 2000. From the Visual book to the WEB book: the importance of design. Electronic Library 18(6):407.
Lazar, J. 2006. Web Usability: A User-centered Design Approach. Boston: Pearson Addison Wesley.
Liu, Z. 2005. Reading behavior in the digital environment: changes in reading behavior over the past ten years. Journal of Documentation 61(6):700-712.
Maizell, R.E. 1998. How to Find Chemical Information. New York: Wiley. pp. 252-253.
Mander, R. & Smith, B. 2002. Web Usability for Dummies. Norwood Mass.: Books24x7. [Online]. Available: http://www.books24x7.com/marc.asp?bookid=13951 [Accessed May 4, 2009].
Menon, S. & Douma, M. 2008. Factors that improve online experiences. Journal of Interaction Recipes 3(1):3/13/09.
Miller, R. 2005. Ebooks worm their way into the reference market. EContent 28(7/8):30-34.
Nielsen, J. 2008. Top 10 Mistakes in Web Design. [Online]. Available: http://www.useit.com/alertbox/9605.html [Accessed May 4, 2009].
Nielsen, J. 2006. F-Shaped Pattern for Reading Web Content. [Online]. Available: http://www.useit.com/alertbox/reading_pattern.html [Accessed May 4, 2009].
Nielsen, J. 2004. The Need for Web Design Standards. [Online]. Available: http://www.useit.com/alertbox/20040913.html [Accessed May 4, 2009].
Nielsen, J. 2003. Homepage Real Estate Allocation. [Online]. Available: http://www.useit.com/alertbox/20030210.html [Accessed May 4, 2009].
Nielsen, J. 2001. Search: Visible and Simple. [Online]. Available: http://www.useit.com/alertbox/20010513.html [Accessed May 4, 2009].
Norman, D.A. 2002. The design of everyday things. New York: Basic Books.
Robbins, S., McCain, C. & Scrivener, L. 2006. The changing format of reference collections: are research libraries favoring electronic access over print? Acquisitions Librarian 18(35):75-95.
Sabin-Kildiss, L., Cool, C. & Xie, H. 2001. Assessing the functionality of web-based versions of traditional search engines. Online (Weston, Conn.) 25(2):18-26.
Safley, E. 2006. Demand for e-books in an academic library. Journal of Library Administration 45(3):445-457.
Shiao-Feng Su. 2005. Desirable search features of web-based scholarly e-book systems. Electronic Library 23:64-71.
Spink, A. & Wolfram, D. 2001. Searching the web: the public and their queries. Journal of the American Society for Information Science & Technology 52(3):226.
Sun, Y., Harper, D.J. & Watt, S.N.K. 2004. Design of an E-Book User Interface and Visualizations to Support Reading for Comprehension. In Sanderson, M., Järvelin, K., Allan, J. and Bruza P., editors. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; Sheffield UK: ACM. p. 510-511.
Van Epps, A.S. 2005. The evolution of electronic reference sources. Library Hi Tech 23(2):287-298.
Vilar, P. & Žumer, M. 2005. Comparison and evaluation of the user interfaces of e-journals. Journal of Documentation 61(2):203-227.
Webster, P. 2003. Implications of expanded library electronic reference collections. Online (Weston, Conn.) 27(5):24-7, 29-30.
Williams, B. 2007. Evaluating engineering reference eBooks. Against the Grain 19(2):52-56.
Wilson, R., Landoni, M. & Gibb, F. 2002. A user-centred approach to e-book design. Electronic Library 20:322-330.
