key: cord-262084-mvgqlufq
authors: Thorp, Andrea W.; Brown, Lance
title: Accessibility of Internet References in Annals of Emergency Medicine: Is It Time to Require Archiving?
date: 2007-08-31
journal: Annals of Emergency Medicine
DOI: 10.1016/j.annemergmed.2006.11.019
sha: 
doc_id: 262084
cord_uid: mvgqlufq

Study objective We seek to evaluate the accessibility of all Internet references appearing in Annals of Emergency Medicine from 2000, 2003, and 2005. Secondary objectives are to determine whether the number of Internet references is increasing and to describe how Internet references are inaccessible. Methods We visually scanned all articles for references made in the printed version of Annals of Emergency Medicine for 2000, 2003, and 2005. We identified the Internet references and grouped them into 11 categories according to the results of entering the uniform resource locator (URL) into the Internet browser. Results We identified 15,745 references published in Annals of Emergency Medicine. The proportion of Internet references increased from 1% of the total references in 2000 to 5.4% in 2005. Internet references were not readily accessible for 40 of 51 Internet references in 2000 (78%; 95% confidence interval [CI] 65% to 88%), 161 of 286 Internet references in 2003 (56%; 95% CI 50% to 62%), and 111 of 249 Internet references in 2005 (45%; 95% CI 39% to 51%). Inaccessibility was most commonly manifested by URLs that no longer link to active Web sites (172 of 312 inaccessible Internet references [55%]; 95% CI 50% to 61%) and URLs that linked to generic home pages where the authors’ referenced material could not be found (115 of 312 inaccessible Internet references [37%]; 95% CI 32% to 42%). Conclusion In Annals of Emergency Medicine, older Internet references appear to be less accessible than newer references. Internet reference archiving is one solution to preserving this information for future readers.

The desire to preserve important writings for future generations has existed for centuries as people have designed ways to store, catalogue, and retrieve information. [1] [2] [3] [4] [5] Recently, the Internet has become a transformative technologic advance for dispersing information to a widespread audience. Because searching, organizing, and sharing information has never been easier, authors of medical and scientific articles have been incorporating Internet references to support material in their publications. 6, 7 Approximately 3% of all references within recently published scientific and medical journals are Internet references. 6,7 In our own experience retrieving Internet references, we found that many of them were no longer active or no longer contained the material originally referenced in the article. Using Internet references within published articles becomes problematic for the readers of scientific and medical journals if the supporting material for factual statements disappears. This problem has been identified in high-impactfactor journals such as Journal of the American Medical Association, Science, and New England Journal of Medicine. [7] [8] [9] Our objective was to evaluate the accessibility of all Internet references appearing in Annals of Emergency Medicine for 2000, 2003, and 2005 . Secondary objectives were to determine whether the number of Internet references is increasing and to describe how Internet references are inaccessible.

All references published in the printed version of Annals of Emergency Medicine from January through December 2000 December , 2003 December , and 2005 were included in our study. We excluded references within advertisements or supplemental publications. We grouped each reference into one of 8 categories: medical/ scientific journals, books and book chapters, government documents, presentations and announcements, newsletters and newspapers, databases and software, pamphlets and package inserts, and Internet references. The Internet reference category comprised our main study group.

Throughout a 2-month period (August to September 2006), we attempted to access all Internet references cited in Annals of Emergency Medicine during our study years. In a post hoc analysis, we classified the Internet references into 11 categories (Table 1) . To better interpret the data, we then organized the 11 categories in 2 main groups. The first group consisted of Internet references that were readily accessible and contained information that the author intended to cite. The second group consisted of Internet references that failed to contain readily accessible information that the author intended to cite.

We entered all the Internet references into a spreadsheet and checked the spelling of each uniform resource locator (URL) against the original published Internet reference at the time of data entry and again if the Internet reference was placed in the "no site found" category. The spreadsheet was linked to an Internet browser (Firefox, version 1.5.0.1; Mozilla Foundation, Mountain View, CA), and double-clicking on each URL directly from the spreadsheet accessed the URLs. These spreadsheets can be found in Appendix E1, available online at http://www.annemergmed.com. We were concerned that the dynamic nature of the Internet would give the "no site found" message or "temporarily unavailable" message erroneously. To avoid misclassification when these messages appeared, we retried each site at least 24 hours after the failed attempt.

We frequently encountered a situation in which the URL would access a generic home page. The authors' intended information would be related to the content on the home page, but the specific topic cited by the author could not be identified by scrolling through the Web site. For example, a URL that links to the New York Times Web site will be active, but the information the authors intended to cite will no longer be on the home page. Because the main purpose of the study was to evaluate the accessibility of referenced information on the Internet, we thought that classifying these URLs as active sites

What is already known on this topic With increasing frequency, scientific articles cite Internet materials in their reference section. The cited sites may not be stable.

What question this study addressed How often are Internet sites cited in Annals of Emergency Medicine, and do they remain accessible?

What this study adds to our knowledge Less than one quarter of Web sites cited in 2000 articles and less than half of Web sites cited in 2005 articles were accessible in 2006.

Unlike references to printed material, references to Internet material are unstable and decay over time. In light of this evidence, journals should consider requiring that all Internet references be made through WebCite or other free, permanent archival services. was inappropriate. In an attempt to locate the authors' intended information, we used the site-specific search engine. Phrases that represented the authors' intended information as published were then entered into the data spreadsheet. We used the copy and paste functions to transfer the text directly from the "intended information" column in the spreadsheet to the site-specific search bar on the Web site. The first 10 results produced by each search were reviewed, and the Internet references were then categorized according to the results of these individual searches.

If the authors' intended information was found, the Internet reference was classified in the "reference found with the site search" category. If the information could not be found, the Internet reference was placed in the "reference not found with the site search" category.

The number of Internet references in each category was totaled and 95% confidence intervals (CIs) were calculated using the cii, Wilson command (Stata, version 9.1; StataCorp, College Station, TX). Because our study contained information available to the public and did not involve human subject intervention, institutional review board review or oversight was not required.

We identified 15,745 references published in Annals of Emergency Medicine in 2000 Medicine in , 2003 Medicine in , and 2005 . References made to the medical/scientific journals and magazines composed 85% of the total references identified during the 3 years. The Internet reference category was the third largest group of references and contributed 4% of the total references ( Table 2) .

The proportion of references that were Internet references increased throughout our 3 study years. Fifty-one of 5,276 total references (1%; 95% CI 0.7% to 1.3%) in 2000, 286 of 5,876 references (4.9%; 95% CI 4.3% to 5.4%) in 2003, and 249 of 4,593 (5.4%; 95% CI 4.8% to 6.1%) in 2005 were Internet references.

When comparing the categories of Internet references, we found that the number of Internet references that no longer accessed the authors' intended information increased over time. Forty of the 51 Internet references in 2000 (78%; 95% CI 65% to 88%), 161 of the 286 Internet references in 2003 (56%; 95% CI 50% to 62%), and 111 of the 249 Internet references in 2005 (45%; 95% CI 39% to 51%) no longer allowed access to the authors' intended information.

To better describe how the Internet references are inaccessible, we compared the 8 categories of Internet references that did not readily link to the authors' intended information. During the 3 years, 172 Internet references (55% of inaccessible Internet references; 95% CI 50% to 61%) could not locate a Web site. One hundred fifteen (37% of inaccessible Internet references; 95% CI 32% to 42%) of the inaccessible Internet references linked to a generic and related Web site, but the authors' intended information could not be identified (Table 3) . Of the 4 Internet references that gave a "temporarily unavailable" message initially (1% of total Internet references; 95% CI 0.3% to 1.7%), none of those Internet references became readily accessible during our study period and, therefore, remained in this category.

We did not have a computerized method for identifying references within Annals of Emergency Medicine, so we manually reviewed each issue. It is conceivable that we overlooked or misclassified references. Some references were in languages other than English, but none of these references had the format of an Internet reference. We are not aware of a validated method for determining whether Internet references contain the authors' intended information. Difficulty in determining whether the 

The Internet references published in Annals of Emergency Medicine appear to become less readily accessible over time. Five years after publication, 78% of the Internet references no longer allowed the reader to readily access the authors' intended information as cited in the published reference section. Previous publications have reported that 4.4% of Internet references become inaccessible after 3 months, 8 13% after 1 year, 7 and 59% at 3 years. 10 Even though previous publications have different definitions of accessibility, the gradual loss of Internet information demonstrated in our study agrees with previous reports.

Of the 586 Internet references identified in our study, 312 were not readily accessible. Of these inaccessible Internet references, 55% could not link to an active Web site, and 37% would link to a generic but related Web site that did not contain the authors' intended information. During our attempts at accessing these Internet references, we often could not decide whether we were viewing the material the authors had viewed when writing the article. A central function of references is to provide support for statements put forward by an author. If a reader questions a fact, he or she can go to the supporting reference and review that information. Traditionally, if a journal article is cited in a reference section, that journal can be accessed and the reader can be confident that the article he or she is reading was the exact article that the author intended to cite. The reader may then participate in a dialogue with the author by challenging his interpretation of the data presented in the reference. If this dialogue is held in a public forum such as in a letter to the editor, the readers may benefit from a deeper insight into the topic at hand. Authors are held to a high degree of accountability for the factual statements they make and reference. If the supporting material in a reference disappears or changes over time, components of a meaningful dialogue and author accountability are damaged.

Internet references composed approximately 3% of the total references appearing in recently published scientific and medical journals. 6,7 Our data demonstrate increased use of Internet references in Annals of Emergency Medicine. From 2000 to 2005, Internet references increased from 1% to more than 5% of the total references. Because of the convenience of the Internet, authors are recognizing the value of the Internet as a resource. If the use of Internet references continues to increase, the problems associated with inaccessible information will increase also.

Several solutions to the problem of inaccessible Internet references have been proposed. Published reports have suggested that the author or publisher keep a digital or hard copy of the information contained within an Internet reference. 8, 9, 11, 12 These solutions are not intuitively appealing because authors may become unavailable and publishers may go out of business. Now that the technology exists to digitally archive information, a better solution available to authors and publishers is the archiving of Internet references. The 2 archiving systems that are available are Internet Archive (available at http://www.archive.org) and WebCite (available at http:// www.webcitation.org). [9] [10] [11] [12] Both of these systems can be used free by authors, readers, journals, and publishers. 9,10 In comparing Internet Archive to WebCite, we feel that WebCite is the better system. Internet Archive is limited in that it can store only static information, thus excluding video, representation of databases, and other dynamic sources. 12,13 In addition, Internet Archive acts as a crawler, so Internet references that have password restrictions or restrictions to "crawlers" may not be accessible. 12 WebCite, on the other hand, takes a "snapshot" of the Web site and immediately provides a "permalink" URL that the author can publish. 10 The permalink URL will immediately take the reader to the snapshot of the Web site. The WebCite's use of the permalink would ensure that the reader was viewing the exact Web site the author had viewed when writing the article. Last, WebCite has formed partnerships with libraries and preservation consortia that have legally agreed to support the service to ensure long-term availability to authors, readers, and publishers. 10 There is no standardized requirement among scientific or medical journals' instruction to authors for archiving Internet references. 14 As of 2003, 99% of high-impact journals do not provide instructions to authors on archiving digital information. 11 Given the increasing inaccessibility of Internet references over time, we concur with previous recommendations that the time to require archiving of Internet references has arrived. 6-9,11,12

Supervising editor: David L. Schriger, MD, MPH Author contributions: LB conceived the project idea. AWT and LB designed the project, participated in data collection, and analyzed the data. AWT drafted the article, and LB contributed substantially to its revision. AWT takes responsibility for the paper as a whole.

Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article, that might create any potential conflict of interest. See the Manuscript Submission Agreement in this issue for examples of specific conflicts covered by this statement. This project was solely supported by internal departmental funds. 

Online is now reserved for ACEP members and print subscribers.

Online is now available for ACEP members and all print subscribers. To activate your individual online subscription, please visit Annals of Emergency Medicine Online by pointing your browser to http://www.annemergmed.com, follow the prompts to activate your online access, and follow the instructions. To activate your account, you will need your ACEP member number or your subscriber account number, which you can find on your mailing label. If you need further assistance to access the online journal, please contact Periodicals Services at 800-654-2452. Personal subscriptions to Annals of Emergency Medicine Online are for individual use only and may not be transferred. Use of Annals of Emergency Medicine Online is subject to agreement to the terms and conditions as indicated online.

Information is also available at ACEP's home page at www.acep.org. 

Manuscript history. Wikipedia: the free encyclopedia

Saint Catherine's Monastery: Mount Sinai. Wikipedia: the free encyclopedia

Wikipedia: the free encyclopedia