Do Download Reports Reliably Measure Journal Usage? Trusting the Fox to Count Your Hens?

Alex Wood-Doughty, Ted Bergstrom, and Douglas G. Steigerwald*

Download rates of academic journals have joined citation counts as commonly used indicators of the value of journal subscriptions. While citations reflect worldwide influence, the value of a journal subscription to a single library is more reliably measured by the rate at which it is downloaded by local users. If reported download rates accurately measure local usage, there is a strong case for using them to compare the cost-effectiveness of journal subscriptions. We examine data for nearly 8,000 journals downloaded at the ten universities in the University of California system during a period of six years. We find that controlling for number of articles, publisher, and year of download, the ratio of downloads to citations differs substantially among academic disciplines. After adding academic disciplines to the control variables, there remain substantial “publisher effects”, with some publishers reporting significantly more downloads than would be predicted by the characteristics of their journals. These cross-publisher differences suggest that the currently available download statistics, which are supplied by publishers, are not sufficiently reliable to allow libraries to make subscription decisions based on price and reported downloads, at least without making an adjustment for publisher effects in download reports.

Introduction

Measures of the influence of academic research are valuable to many decision makers. University librarians use them to make purchasing and renewal decisions.1 Academic departments use them in their hiring, tenure, and salary decisions.2 Funding agencies use them to assess grant applicants. They are also used in determining the public rankings of journals, academic departments, and universities.3

Citation counts have long been the most common measure of research influence. Eugene Garfield’s Institute for Scientific Information introduced the systematic use of citation data with the Science Citation Index in 1964 and Journal Citation Reports (JCR) in 1975.4 The advent of electronic publishing has given rise to a new measure of research influence: download counts.5 For library evaluations, accurate download counts could offer important advantages over citation counts. Only a minority of those who download a journal article will cite it. Citation counts reflect the activities of scholars worldwide. Subscribing libraries can observe the number of downloads from their own institutions, which reflect their own patterns of research interest.

For academic departments and granting agencies, the use of download data in addition to citation records yields an enriched profile of the influence of individual researchers’ work.6 Download data have the advantage of being much more immediate than citation data, a valuable feature for tenure committees or grant review panels tasked with evaluating the work of younger academics.

Several previous articles have explored correlations between citations and recorded downloads.7 Brody, Harnad, and Carr8 examine the extent to which downloads from the physics e-print archive, arXiv.org, predict later citations of an article. McDonald9 explores the ability of prior downloads at the California Institute of Technology (Caltech) to predict article citations by authors from Caltech. Previous studies of download behavior have been limited to a small number of journals within a few specialized disciplines. Our download data include recorded downloads at the ten University of California campuses from nearly 8,000 academic journals in a wide variety of academic disciplines.

The number of downloads from a volume of an academic journal is highly correlated with the number of times it is cited. A simple linear regression of downloads on citations finds that 78 percent of the variation in downloads can be “explained” by variation in citations. Despite this strong correlation, there are important systematic differences between download rates and citation rates. For example, the ratio of downloads to citations in the arts and humanities is significantly higher than in other disciplines while that in the physical sciences is significantly lower.

Given that a library’s own download rates reflect the demands of its users more closely than citation rates, there appears to be a strong case for using download rates rather than citation rates to evaluate journal subscriptions. The case for using download rates depends, however, on the assumption that these rates are accurately measured.

Libraries do not, in general, maintain their own download counts. This information is collected and supplied by publishers in summary form to subscribing libraries. Davis and Price10 suggest that

The number of full-text downloads may be artificially inflated when publishers require users to view HTML versions before accessing PDF versions or when linking mechanisms, such as CrossRef, direct users to the full text rather than the abstract of each article. The publishers, who control the raw data on downloads, have a strong incentive to release statistics that may overstate the number of actual users.

Subscribers are not given access to the publishers’ web server log files from which the reports they receive are compiled; thus, they have no independent way of verifying the artificial inflation of download counts. Publishers are well aware that their download reports will influence librarians’ subscription decisions. Davis and Price quote Sir Crispin Davis, who as CEO of Reid-Elsevier in 2004 testified to the British House of Commons as follows:

The biggest single factor is usage. That is what librarians look at more than anything else and it is what they [use to] determine whether they renew, do not renew and so on. We have usage going up by an average of 75 per cent each year. In other words, the cost per article download is coming down by around 70 per cent each year. That is fantastic value for money in terms of the institution, so I would say that [usage] is the single biggest factor.

Because download statistics are not managed in a transparent way by impartial arbiters, it is reasonable to ask whether publisher-supplied data on downloads can be reliably compared across publishers. The University of California has “Big Deal” subscriptions for all of the journals published by each of the seven publishers treated here (“Big Deal” refers to an agreement to purchase nearly the entire portfolio of journals from a publisher). If the relation between recorded downloads and actual usage is the same across publishers, we would expect that, after controlling for journal characteristics such as citations, number of articles, and academic discipline, the identity of the publisher should have little or no effect on the number of downloads at the University of California. We find, however, strong and statistically significant publisher effects that are persistent under a variety of specifications of variables.

Data

We have obtained download records from the California Digital Library (CDL), which handles subscriptions for all ten campuses of the University of California system. These records represent about 4.25 million downloads from 7,724 journals published by seven publishers during the years 2010–2016.

Reports on the number of downloads from each article are supplied by the publishers, who prepare this data according to guidelines set by COUNTER (Counting Online Usage of Networked Electronic Resources), a nonprofit organization established by libraries, data vendors, and publishers. Most publishers provide journal download reports at COUNTER level JR1, which records the monthly number of downloads to all articles that have ever been published in that journal, but they do not report the year in which the downloaded articles were published. A few publishers offer more detailed reports at COUNTER level JR5. The JR5 reports record the number of downloads in the current year, while specifying the year in which each downloaded article was published. For example, the JR5 data for 2015 reports the number of articles that were published in each year since 2000 and downloaded in 2015. While many publishers include clauses in their contracts that forbid public access to this information, the CDL contracts do not include such restrictive clauses. The seven publishers used in our analysis are those who supplied the CDL with download data at the JR5 level.

The JR5 reports released by publishers include downloads, not only from their subscription journals, but also from the open access journals that they publish. When a library uses download counts to evaluate a “Big Deal” package that allows access to its subscription journals, it would not be appropriate to include downloads from open access journals, since these are accessible whether or not the library subscribes. For this reason, we confine our analysis to journals that require paid subscriptions for access.

Journals are placed into field classifications according to Scimago’s designation, which uses a classification system developed by Elsevier’s Scopus to partition journals into disciplinary categories at three distinct levels of detail.11 At the broadest level of classification, there are five major categories: life sciences, physical sciences, health sciences, social sciences, and arts and humanities.12 At an intermediate level of detail, Scopus specifies 27 “major fields.” At the most detailed level, Scopus assigns each journal to one or more of 334 “minor fields.” Many journals are classified as belonging to more than one category. Where a journal is assigned to k different categories, we treat it as if one kth of its articles are in each of the k categories to which Scopus has assigned it. For example, if Scopus designates a journal as belonging to three “major” fields (such as mathematics, computer science, and economics), we would assign it an indicator value of 1/3 for each of these three fields. Our sample includes four large commercial publishers—Elsevier, Springer, Taylor & Francis, and Wiley—that publish across many disciplines. We also include the Nature Publishing group, which specializes in life and physical sciences, and two professional disciplinary society publishers: the American Chemical Society (ACS) and the Institute of Electrical and Electronics Engineers (IEEE). For each of these publishers, we have four to six years of reports on the annual number of downloads occurring in years from 2011 to 2016, where the downloaded articles were published between 2000 and 2016.

Table 1 shows the distribution of subscription-based journals by broad research field across publishers.13 As the table shows, each of the four large commercial publishers has a significant presence in all five research fields, while the other publishers have more limited scope. The Nature Publishing Group journals are sorted into Nature-branded, the 30 journals under the imprimatur Nature (subject) (such as Nature Astronomy), and Other, the remaining 42 NPG journals that do not include Nature in their title. As table 3 will show, articles in the Nature-branded journals are much more cited and even more frequently downloaded than the other NPG journals.14

The analysis will also require information on citations and the number of articles published, both by journal. The citations measure is obtained from the website SCImago Journal & Country Rank15 that records, for each journal, and for each year, the number of citations to articles published in that journal in the preceding three years. The number of articles used in our calculation is the annual number of “documents” reported by Scimago for each journal.16

Downloads and Citation Patterns by Field and Publisher

Journal articles in the life and health sciences tend to be more frequently cited than those in the physical and social sciences, while journals in arts and humanities are significantly less frequently cited than those in all other disciplinary areas. Differences in downloads per recent article also differ by discipline, but less drastically.

In table 2, recent UC downloads are measured as the number of downloads from a journal in the first three years after publication (the year of publication and the following two years). The reported ratio is constructed by dividing this download count by the number of articles from the journal in the year of publication. The corresponding measure for citations, more commonly known as the impact factor, simply replaces the number of downloads with the number of citations. This table also shows the ratio of these two measures, the size of which depends on the fact that downloads are counted only from UC campuses, while citations are counted worldwide. For each measure, summary statistics are reported.

According to table 2, the ratio of downloads to citations in arts and humanities is significantly higher than for other disciplines. While journals in the arts and humanities tend to have fewer citations per article than other disciplines, the number of downloads per article is nearly as large as that for the physical and social sciences. This suggests that the use of citation rates rather than download rates is likely to undervalue journals in arts and humanities relative to other fields.

Table 3 shows the distribution of the three measures of downloads and citations for each of the seven publishers in our sample. This table shows that the Nature-branded journals have a far higher ratio of downloads to citations than any of the other publisher groups. The ratio for NPG’s other journals is lower than for the Nature-branded journals, but it remains high relative to most other publishers. NPG’s Nature-branded journals have a special feature that at least partially explains their high download-to-citation ratios: typically more than half of the articles appear in a News and Views section. These articles are brief reports on recent research, targeted at nonspecialists. The News and Views reports are often commissioned to prestigious scholars and closely edited by professional staff. Because these articles are generally not the first to report new results, they are not often cited in the specialist literature. However, they are extremely popular and widely read because they are of high quality and easily absorbed by a wide audience. Among the publishers other than NPG, Elsevier has the highest ratio of reported downloads to citations.

Possibly the differences between publishers’ download-to-citation ratios could be explained by differences in the academic disciplines that they cover or by differences in the impact factors of their journals. Table 2 shows that the ratio of downloads to citations differs among academic disciplines and also differs with the impact factor of the journal, while table 1 shows that the publishers in our sample differ significantly in the distribution of academic disciplines that they cover. In the following sections, we apply statistical analysis to explore the extent to which these cross-publisher differences can be explained by observable characteristics of the journals that they publish.

Predicting Downloads from Journal Characteristics

Table 3 describes the relation between downloads per article and just two variables: impact factor and publishers. To account for the simultaneous effects on downloads of a longer list of characteristics, we estimate a function that predicts the number of downloads from a journal as a function of several variables describing that journal. Among the explanatory variables to be considered are the number of articles in a journal, the average number of citations per article (impact factor), the date of download, and the academic discipline to which the journal is devoted. We consider three specifications, which vary by the level of detail for the field classification of the journals: the five broad categories shown in table 1, the 27 major fields defined by Scopus, and the full set of 334 fields defined by Scopus.

Having controlled for a journal’s citations, impact factor, academic discipline, and year of download, we might expect that the identity of the journal’s publisher would have little or no effect on the predicted number of downloads. To determine whether this is the case, we fit an equation that includes all of the above-mentioned variables as well as an indicator variable for the publisher.

The Function to Be Estimated

We estimate an equation defined as follows. Let D_jy represent the number of times in year y that University of California libraries have downloaded articles that were published in journal j in year y and in the three years prior to year y. Let A_jy be the number of articles published in journal j in the three years previous to year y. Let C_jy be the number of times that articles published in journal j in the previous three years were cited in year y. We assign indicator variables for the academic discipline to which a journal is assigned, the year in which downloads are recorded, and the journal’s publisher. (An indicator variable is either 0 or 1 and indicates a characteristic. If a journal is published by Elsevier, then, for all observations corresponding to this journal, the indicator for Elseveir equals 1 and the indicator for all other publishers equals 0.) We then employ maximum likelihood procedures to estimate a function that predicts downloads and takes the form

where F_j, P_j, and Y_y are multiplicative factors corresponding respectively to the journal’s discipline, its publisher, and the year of download for the observed downloads. (Appendix A presents formal details of our estimation procedure.)

We can rewrite Equation 1 to explicitly show separate effects of citations per article (aka impact factor) and of number of articles (size of journal) on the number of downloads. Equation 1 is equivalent to

We estimate the parameters α + β, β and the coefficients Y_y, F_j, and P_j, corresponding to indicator variables for year of download, journal discipline, and journal publisher. For each of the 7,724 journals in the sample, there are between four and six annual observations, corresponding to downloads in different years. We estimate standard errors using cluster- robust methods to account for within-journal correlation.17

Results

We estimate the joint effects of the variables impact factor, number of articles, year of download, journal discipline, and journal publisher by fitting Equation 2. We fit separate estimates in which fields are specified at each of Scopus’s three levels of detail. We also consider a specification in which separate equations are fit for each of the five broad disciplinary areas, thus allowing the elasticities of downloads with respect to impact factor and to number of articles to differ between broad disciplines. Our discussion reports the effects of each group of explanatory variables, while controlling for the effects of all of the other variables.

The Effects of Impact Factor and Number of Articles

Table 4 shows the estimated elasticities of the number of downloads to the impact factor and number of articles. The estimates in the first column are best estimates when the elasticities are constrained to be the same for all categories. (These estimates are nearly the same for field specifications at all three levels of detail.) The remaining columns show separate estimates when the elasticities are allowed to differ among categories.

The coefficient β captures the responsiveness, in percentage terms, of downloads with respect to impact factor, holding constant the number of articles. Because the impact factor is the ratio of the number of citations to the number of articles, a 1 percent increase in the impact factor, holding articles constant, is equivalent to a 1 percent increase in citations. Thus we can also interpret β as an estimate of the elasticity of downloads with respect to citations. The coefficient 1.11 in the first column indicates that, for a given journal, a 1 percent increase in the number of citations, holding the number of articles fixed, would result in slightly more than a 1 percent increase in downloads.

The coefficient α + β measures the elasticity of downloads with respect to the number of articles, holding impact factor constant. The coefficient of 0.91 for articles in the first column indicates that holding constant a journal’s impact factor, a 1 percent increase in the number of articles would result in slightly less than a 1 percent increase in the number of downloads.

The estimated elasticity of downloads with respect to the number of articles is close to 1 for all five disciplinary categories. For arts and humanities and for social sciences, the elasticity of downloads with respect to impact factor is significantly less than 1. (The statistical significance stems from the result that the estimated coefficient is more than 2 standard errors below 1.) For the health sciences and physical sciences, this elasticity is close to 1, and for the life sciences it is significantly greater than 1.

The Effect of Download Year

Table 5 shows the coefficients of year-of-download from the estimating equations for each of the four broad disciplinary categories. The rows for each download year report the multiplicative factor for that year. The year 2014 is selected as the base year because this is the first year for which we have data for all seven publishers.18

To estimate the average annual growth rate, we fit a linear time trend to annual data, controlling for the number of citations, number of articles, and publisher effects. This growth rate was about 3 percent in the health sciences and roughly 2 percent in the life sciences and social sciences. This rate was not significantly different from zero in the arts and humanities and in the physical sciences.

The Effect of Academic Disciplines

Table 6 compares the download rates across broadly defined disciplines. To facilitate comparison across disciplines, we express these rates relative to those for social sciences. The second column shows a simple ratio of the numbers of downloads to citations without controlling for other variables. The column Category Coefficient reports the coefficient in our fitted equation or an indicator variable for a journal’s disciplinary category. This measures the effect of discipline when controlling for our other variables: impact factor, number of articles, year of download, and publisher. This table shows that, controlling for these other factors, articles in arts and humanities are more than twice as likely to be downloaded as those in the social sciences, while articles in the physical sciences are less than half as likely to be downloaded.

A possible explanation for the relatively low download rates for articles in the physical sciences is that, in many of the physical science fields, a large proportion of published articles also appear on the freely available source arXiv. For example, roughly two-thirds of articles published in astronomy and astrophysics and in nuclear and particle physics, and roughly one-third of articles published in mathematics and in general physics, are available in arXiv.19 A study by Davis and Fromerth20 concluded that arXiv-deposited articles in mathematics received more than 20 percent fewer downloads from the publisher’s website.

Download Rates by Major Field

Table 7 shows the results of fitting download rates to a more finely drawn division of disciplines consisting of the 27 “major fields” assigned to journals by Scopus. The second column compares a simple ratio of downloads to citations in each major field relative to that ratio for all articles in the social sciences. The third column shows the coefficient of each discipline when we fit an equation accounting for impact factor, number of articles, year of download, and publisher. (Table 12, which is found in the appendix, reports the coefficients of 334 “minor fields” as classified by Scopus.) This table shows substantial differences in download behavior between major fields, even within the same broad discipline.

The Effect of Journal Publisher

Table 8 shows the effect of an indicator variable for each publisher on reported download rates in an estimated equation that controls for each journal’s number of articles, impact factor, major field, and the year in which downloads occurred. To facilitate comparison among publishers, these coefficients are expressed as their ratio to the publisher effect of Elsevier. Table 8 presents three alternative specifications, which differ in fineness of detail by which fields are distinguished. This table demonstrates that, after controlling for discipline, impact factor, and number of articles, there remain dramatic publisher effects. These effects are little changed by changes in the granularity with which fields are defined.

To explore the robustness of our estimated publisher effects to alternative specifications, we estimated publisher effects based on a model in which we fit separate equations for each of the five broad disciplinary categories. This specification allows the effects of impact factor, number of articles, and year of download to differ across broad categories. These results are shown in table 9.

Tables 8 and 9 show that the strongest publisher effect by far is for Nature-branded journals. This effect probably is due to the fact that about half of the articles in Nature- branded journals are commissioned summaries of recent research called News and Views, which are written by prominent scholars and intended for nonspecialists. These papers do not present original research that is likely to be cited, but they are frequently downloaded and read by scientists who wish to learn about research that is not directly related to their own work.

The remaining publisher effects fall roughly into two groups. Journals published by Elsevier, American Chemical Society, and by the Nature Publishing Group without the Nature brand consistently show higher publisher effects than those published by IEEE, Springer, Wiley, and Taylor & Francis. These coefficients indicate that Elsevier reports more than twice as many downloads as Taylor & Francis from journals that are in the same discipline and have similar impact factors and numbers of articles. Elsevier reports about 50 percent more downloads than Springer and about 40 percent more than Wiley from journals of similar quality and disciplinary specialization.

Double-counting and the Ratio of PDF to Total Downloads

Davis and Price21 and Li and Wilson22 have suggested that differences in publisher platforms are likely to result in large differences in the number of downloads recorded in a single usage. Some platforms may make it more likely that a user who wants to read an article will download both a PDF copy and an HTML copy, thus counting two downloads for a single usage. In a study of records of downloads from about 800 journals at the Cornell University library in 2004, Davis and Price found wide divergence in the ratio of PDF to HTML downloads among the six publishers they studied.

To explore this hypothesis, we performed a similar exercise with our sample of nearly 8,000 journals from seven publishers at the ten University of California campuses.23 The first column of table 10 displays the ratio of total downloads to PDF downloads for each publisher. The second and third columns provide comparisons of all publishers with Elsevier. In the second column, the ratio of the first column entries appears, thus 0.46 for ACS indicates that the total/PDF ratio for ACS is only 46 percent of the total/PDF ratio for Elsevier. The third column repeats results presented earlier (column 2 of table 8), which allows comparison of the download ratios with the publisher effects reported above.

This table shows that the journals published by Nature and by Elsevier, which have the largest publisher effect, also have much higher ratios of total to PDF downloads than the journals published by Springer, IEEE, Wiley, and Taylor & Francis. This tends to confirm the view of Davis and Price, Wilson and Li, and Wiersma24 that the extent to which download statistics double-counts downloads varies widely among publishers. (The correlation we find is not universal, however. The American Chemical Society’s publisher effect is nearly as large as Elsevier’s, yet its ratio of PDF to total downloads is closer to that of the group of publishers with low estimated publisher effect.)

Because libraries frequently use download statistics to evaluate journal subscriptions, publishers have an incentive to induce users to download the same article multiple times. Some publishers seem to have been more successful in this endeavor than others. The links that appear if one begins a search at the journal’s table of contents appear to be quite similar among the seven publishers in our study.25 The platform that one encounters when accessing an article through a search engine or through Crossref seems much more variable. For some journals, the first link that the search engine points to will open an HTML copy immediately, while offering the option to also download a PDF. For other journals, it opens a page that offers an option to download a PDF before it opens an HTML. Sometimes the first link will take one directly to a PDF file.

The statistics that publishers release to libraries appear in a summary form that conceals much of the information that would be necessary for libraries to estimate the usage that a subscription pays for. When a user is shown an HTML version of a paper and file and then downloads the PDF version, this is counted as two downloads. If the same user downloads the same paper a few hours later, this is counted as an additional download.26 The JR1 and JR5 download reports are compiled from log files that record the exact time of each download, the IP address of the user, and, in some cases, whether the download is an HTML or a PDF download. As Bergstrom, Uhrig, and Antelman27 demonstrate, these log files can be used to estimate the extent of double-counting by publishers.

Conclusion

This paper originated as an exploration of the relation between journal downloads and journal citations. Our study indicates that there is substantial correlation between citations and reported downloads, with an R2 of about .78 in a simple regression. It also shows that the ratio of downloads to citations differs sharply among disciplines and that this ratio tends to be higher for journals with higher impact factors. This suggests that, if download reports accurately measure usage, there is a compelling case that libraries should use download data in addition to or perhaps instead of citation data in deciding how to allocate their subscription expenditures among journals.

Our study uncovered a disconcerting dependence of reported journal downloads on the identity of the journal’s publisher. This dependence persists when we control for academic discipline, impact factor, number of articles, and year of download. When we fit an estimating function that controls for these variables, the numbers of recorded downloads from journals published by Elsevier, the American Chemical Society, and Nature Publishing Group are roughly twice as high as those for journals published by Springer, Wiley, Taylor & Francis, and IEEE.

Large differences in the ratio of reported PDF downloads to reported total downloads provide circumstantial evidence that a) actual usage is exaggerated because users who download both a PDF copy and one or more additional HTML copies are counted as making multiple downloads; and b) this exaggeration differs substantially among publishers.

If the amount of double-counting was relatively constant across disciplines and across publishers, then reported downloads would remain useful for comparing the relative cost-effectiveness of competing journals. But our estimates suggest that this is not the case. Differences among publishers’ ratios of reported downloads to actual usage would mean that download statistics cannot be used to compare the value of similar journals published by different publishers, at least without an adjustment factor to account for publisher effect.

If we use the publisher fixed effects reported in the third column of table 10 to estimate the amount of double-counting by each publisher, then to compare relative numbers of downloads across publishers we must deflate the numbers reported by publishers with large publisher effects. Table 11 reports these deflation factors. Because Taylor & Francis has the smallest publisher effect, their reported downloads are not deflated. For Elsevier, which has a large publisher effect, deflated downloads are only 43 percent of reported downloads.

Our study suggests that the organization COUNTER has not achieved the objective stated on their website:28

“COUNTER provides the Code of Practice that enables publishers and vendors to report usage of their electronic resources in a consistent way. This enables libraries to compare data received from different publishers and vendors.”

Our results strongly indicate that the COUNTER data currently available to university libraries does not enable libraries to reliably compare the value of journals received from different publishers. This is unfortunate because accurate reports of downloads would be a better measure of local usage and hence of the value of a subscription than are citation counts.

We suggest that, if librarians wish to use download statistics to compare the cost- effectiveness of journals offered by different publishers, they should consider adjusting the reported download statistics for each publisher to account for the platform effects shown in table 11.

Download data is currently collected by publishers and reported to subscribing libraries in summary form, often subject to a confidentiality clause that prevents them from sharing download information with researchers or with other libraries. We suggest that, when negotiating contracts with publishers, libraries insist on the right to share this information with researchers and with other libraries.

In the long run, if download statistics are to be a credible and reliable tool for estimating usage, it seems that it would be advisable for libraries to develop a uniform interface for downloading articles from all publishers and to maintain their own records of journal downloads, which they would share as public information.

APPENDIX A. Statistical Methods

The number of downloads is a count variable taking nonnegative integer values. Because count data are not continuous, the traditional approach of specifying the conditional mean of the variable of interest together with a normal error is not always the best approach. For the problem at hand, D_j,y has many small integer values, a large number of zeros, and a small number of very large counts (the source of the positive skewness in the downloads distribution), all of which suggest the normal distribution is not appropriate. One common alternative is to convert the integer values to noninteger values (by using the log of the variable of interest) that are then well approximated by a normal distribution. Such an approach is not appealing here, because the log is not defined for the many observations that equal zero.

Instead, we model the distribution of downloads, conditional on the covariates x_j,y, as a Poisson random variable with distribution defined by

where µ_j,y depends on x_j,y. The Poisson approximation to the distribution of downloads is unlikely to work well for noninteger random variables, in particular for the ratio of downloads to citations.

The key is to specify the relationship between µ_j,y and the covariates, for which a natural specification would be µ_j,y = . One feature of the Poisson distribution is that E[D_j,y|x_j,y] = µ_j,y, hence µ_j,y > 0 because downloads are restricted to be nonnegative. Unfortunately, the linear specification does not satisfy the restriction µ_j,y > 0 for all values of , so the common specification is µ_j,y = exp. Thus

The parameters are estimated via quasi-maximum likelihood. The density for an individual observation is

If we let the full set of observations be denoted , the log likelihood is

with first-order conditions

where βˆ is the maximum likelihood estimator of β. Although (7) does not have a closed-form solution, L is a concave function of β and standard numeric optimization methods can be employed.

Under the Poisson distribution the mean equals the variance, a restriction that is unrealistic for downloads. Yet βˆ remains consistent for β even if this restriction is violated, as long as the conditional mean is correctly specified in (4). More care needs to be taken in estimating the standard error of βˆ. To produce consistent estimators of the standard errors we use the robust variance estimator where .

APPENDIX B. Discipline Effects by Minor Field

The coefficients in table 12 show the coefficients of an indicator for each minor field on our fitted estimate of the annual number of downloads. These are normalized to be expressed as ratios to the coefficient of social science. For example, the coefficient 0.61 for Accounting means that controlling for impact factor, number of articles, year of download, and publisher, accounting journals are downloaded about 61% as often as the average journal in social science.

Notes

1. David Coughlin, Mark Cambell, and Bernard Jansen, “Measuring the Value of Library Content Collections,” Proceedings of the American Society for Information Science and Technology 50, no. 1 (2013): 1–13; John Gallagher, Kathleen Bauer, and Daniel Dollar, “Evidence-based Librarianship: Utilizing Data from All Available Sources to Make Judicious Print Cancellation Decisions,” Library Collections, Acquisitions, and Technical Services 29, no. 2 (2005): 169–79.

2. John Gibson, David Anderson, and John Tressler, “Which Journal Rankings Best Explain Academic Salaries? Evidence from the University of California,” Economic Inquiry 52, no. 4 (2014): 1322–40; Glenn Ellison, “How Does the Market Use Citation Data? The Hirsch Index in Economics,” American Economic Journal: Applied Economics 5, no. 3 (2013): 63–90.

3. Ellen Hazelkorn, Rankings and the Reshaping of Higher Education (London, UK: Palgrave Macmillan, 2015).

4. A brief history of the science citation index and the impact factor appears in Eugene Garfield, “The Evolution of the Science Citation Index,” International Microbiology 10 (2007): 65–69.

5. A broad-ranging summary and history of the application of download information and other direct measures of journal usage is presented in Michael Kurtz and Johan Bollen, “Usage Bibliometrics,” Annual Review of Information Science and Technology 44, no. 1 (2010): 1–64.

6. Michael Kurtz, Gunther Eichhorn, Alberto Accomazzi, and Stephen Murray, “The Effect of Use and Access on Citations,” Information Processing and Management 41, no. 6 (2005): 1395–1402; Michael Kurtz and Edwin Henneken, “Measuring Metrics: A 40-year Longitudinal Cross-validation of Downloads and Peer Review in Astrophysics,” Journal of the Association for Information Science and Technology 68, no. 3 (2017): 695–708.

7. Henk Moed, “Statistical Relationships between Downloads and Citations at the Level of Individual Documents within a Single Journal,” Journal of the American Society for Information Science and Technology 58, no. 1 (2005): 1088–97; Joanna Duy and Liwen Vaughan, “Can Electronic Journal Usage Data Replace Citation Data as a Measure of Journal Use? An Empirical Examination,” Journal of Academic Librarianship 32, no. 5 (2006): 512-17; Jin-kun Wan, Ping-huan Hua, Ronald Rousseau, and Xiu-kun Sun, “The Journal Download Immediacy Index (DII): Experiences Using a Chinese Full-text Database,” Scientometrics 82, no. 3 (2010): 555–66; Juan Gorraiz, Christian Gumpenberger, and Christian Schloegl, “Usage versus Citation Behaviours in Four Subject Areas,” Scientometrics 101, no. 2 (2014): 1077–95; Daniel Coughlin, Mark Campbell, and Bernard Jansen, “Modeling Journal Bibliometrics to Predict Downloads and Inform Purchase Decisions at University Research Libraries,” Proceedings of the American Society for Information Science and Technology 50, no. 1 (2013): 1–13; Henk Moed and Gali Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals,” Journal of the Association for Information Science and Technology 67, no. 2 (2016): 412–31; Liwen Vaughan, Juan Tang, and Rongbin Yang, “Investigating Disciplinary Differences in the Relationships between Citations and Downloads,” Scientometrics 111, no. 3 (2017): 1533–45.

8. Tim Brody, Stevan Harnad, and Leslie Carr, “Earlier Web Usage Statistics as Predictors of Later Citation Impact,” Journal of the American Society for Information Science and Technology 57, no. 8 (2006): 1060–72.

9. John McDonald, “Understanding Journal Usage: A Statistical Analysis of Citation and Use,” Journal of the American Society for Information Science and Technology 58, no. 1 (2007): 39–50.

10. Philip Davis and Jason Price, “eJournal Interface Can Influence Usage Statistics: Implications for Libraries, Publishers, and Project COUNTER,” Journal of the American Society for Information Science and Technology 57, no. 9 (2006): 1243–48.

11. “Appendix D: Field Classification Systems” (2018), available online at https://www.csg.org/programs/ knowledgeeconomy/images/appendixd.pdf [accessed 1 February 2019].

12. Scopus treats “Arts and Humanities” as one of the subject areas within Social Sciences. We have chosen to define Arts and Humanities as a separate broad subject area from Social Sciences. Scopus also has a category called Multidisciplinary. In our sample, the Multidisciplinary category includes only 15 minor journals. Therefore, this category is omitted from most of our discussion.

13. Scopus also includes a category titled “Multidisciplinary.” Because our sample includes only about 15 minor journals classified as Multidisciplinary, we do not include this category in table 1.

14. For NPG we exclude the journal Nature due to its broader, general interest readership.

15. “SJR: Journal and Country Rank” (2018), available online at https://scimagojr.com/journalrank.php [accessed 1 February 2019].

16. Scimago reports both the number of “documents” and the number of “citable documents.” “Citable documents” refers to regular articles, while “documents” also includes book reviews, letters to the editor, and opinion pieces. The “number of articles” used by Web of Science in calculating its impact factor is essentially the same as SCImago’s citable documents. The number of citations reported by SCImago (and also by Web of Science) includes citations to all documents, not only “citable documents.” Elsevier’s Scopus calculates an impact factor called CiteScore that is based on SCImago’s reported number of documents rather than citable documents. See “Journal Metrics-FAQ” (2017), available online at https://journalmetrics.scopus.com/index.php/Faqs [accessed 1 February 2019].

17. When these results are compared with robust standard errors that only account for heteroskedasticity, we find that the cluster-robust standard errors are about twice the estimates found without accounting for within-journal correlation.

18. Our data for the year 2011 included only the publishers Springer and Taylor & Francis. For 2012 we have data from Springer, Taylor & Francis, and Elsevier. For 2013 we have data from all publishers except Wiley.

19. Vincent Lariviere, Cassidy Sugimoto, Benoit Macaouso, Stasa Milojevic, Blaise Cronin, and Mike Theiwall, “ArXiv E-prints and the Journal of Record: An Analysis of Roles and Relationships,” Journal of the American Society for Information Science and Technology 65, no. 2 (2014): 1157–69.

20. Philip Davis and Michael Fromerth, “Does ArXiv Lead to Higher Citations and Reduced Publisher Downloads for Mathematics Articles?” Scientometrics 71, no. 2 (2007): 203–15.

21. Philip M. Davis and Jason S. Price, “eJournal Interface Can Influence Usage Statistics: Implications for Libraries, Publishers, and ProjectCounter,” Journal of the American Society for Information Science and Technology 57, no. 9 (July 2006): 1243–48.

22. Chan Li and Jacqueline Wilson, “Inflated Journal Value Rankings: Pitfalls You Should Know about HTML and PDF Usage” (paper presented at the American Library Association Annual Conference in San Francisco, CA, June 25-30, 2015).

23. The JR5 data that we use does not separately report HTML and PDF downloads, but the JR1 data for recent years does report separate numbers of HTML and PDF downloads. For each of the seven publishers, we have JR1 data with separate reports for PDF and HTML downloads for only some of the years that our JR5 data covers. To estimate ratios of PDF to HTML downloads for our sample, for each publisher, we use the ratio of total PDF downloads to total HTML downloads in the years for which we have JR1 data.

24. Gabriella Wiersma, “Report of the ALCTS CMS Collection Evaluation and Assessment Interest Group Meeting, American Library Association Conference, San Francisco, June 2015,” Technical Services Quarterly 33, no. 2 (2016): 183–92.

25. Typically one sees the article title along with a link for downloading a PDF copy and a link for viewing the abstract. If one clicks the article title, the article is opened as an HTML file and an option to also open it as a PDF appears.

26. According to Counter Project Release 4 (2017), available online at https://www.projectcounter.org/about [accessed 1 February 2019], the data presented in the Counter reports screens for double-clicking by impatient users in the following way. If a user clicks the link to an HTML copy twice within 10 seconds, or a PDF copy twice within 30 seconds, the two clicks count as only one access. It appears, however, that if one clicks a link to an HTML file and also a PDF file, within a short interval, both are counted.

27. Working paper, by Ted Bergstrom, Richard Uhrig, and Kristin Antelman, “Looking under the Counter for Overcounted Downloads,” available online at https://escholarship.org/uc/item/0vf2k2p0 [accessed 1 February 2019].

28. “About Counter” (2017), available online at https://www.projectcounter.org/about [accessed 1 February 2019].