Research Article
Informing Evidence Based Decisions: Usage Statistics
for Online Journal Databases
Alexei Botchkarev
Senior Information
Management Advisor, Ministry of Health and Long-Term Care
Adjunct Professor, Ryerson
University
Toronto, Ontario, Canada
Email: alexei.botchkarev@ontario.ca
Received: 9 July 2016 Accepted:
20 Feb. 2017
2017 Botchkarev. This
is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.
Abstract
Objective – The primary objective was to examine online journal database usage
statistics for a provincial ministry of health in the context of evidence based
decision-making. In addition, the study
highlights implementation of the Journal Access Centre (JAC) that is housed and
powered by the Ontario Ministry of Health and Long-Term Care (MOHLTC) to inform
health systems policy-making.
Methods – This was a prospective case
study using descriptive analysis of the JAC usage statistics of journal
articles from January 2009 to September 2013.
Results – JAC enables
ministry employees to access approximately 12,000 journals with full-text
articles. JAC usage statistics for the 2011-2012 calendar years demonstrate a
steady level of activity in terms of searches, with monthly averages of 5,129.
In 2009-2013, a total of 4,759 journal titles were accessed including 1,675
journals with full-text. Usage statistics demonstrate that the actual
consumption was over 12,790 full-text downloaded articles or approximately
2,700 articles annually.
Conclusion – JAC’s steady
level of activities, revealed by the study, reflects continuous demand for JAC services and products. It testifies that
access to online journal databases has become part of routine government
knowledge management processes. MOHLTC’s broad area of responsibilities with
dynamically changing priorities translates into the diverse information needs
of its employees and a large set of required journals. Usage statistics
indicate that MOHLTC information needs cannot be mapped to a reasonably compact
set of “core” journals with a subsequent subscription to those.
Introduction
Use
of research results in medical evidence based decision-making and healthcare
evidence-informed policy advice has been recognized as essential to improve
health outcomes (Field, Gauld & Lawrence, 2012; Lomas & Brown, 2009;
Sosnowy, Weiss, Maylahn, Pirani & Katagiri, 2013). Despite agreement on the
importance of the issue and general consensus on the approaches, implementation
of evidence based decision-making processes leaves much room for improvement.
Several barriers have been identified by researchers, including: lack of access
and limited awareness of research results (Wallace, Nwosu and Clarke, 2012);
lack of practical use of systematic reviews (Wallace et al., 2012); lack of
organizational culture or supports (e.g. behaviour of supervisors, front-line
staff and other professionals in the organization) (Rapp et al., 2010); lack of
time (Solomons & Spross, 2011); ambiguous and conflicting research
(Madhavji, Araujo, Kim & Buschang, 2011; Ubbink et al., 2011) or research having methodological inadequacies
(O'Connor & Pettigrew, 2009); lack of skills, training or tools to
acquire, assess, synthesize, disseminate and apply research evidence to inform
policy related to health systems (Ubbink, Guyatt & Vermeulen, 2013); lack
of applicability/relevance of research (Humphries, Stafinski, Mumtaz &
Menon, 2014); lack of standard knowledge translation
strategies and processes effective in multiple contexts (Humphries et al.,
2014); lack of timely research outputs (Oliver, Innvar, Lorenc, Woodman &
Thomas, 2014; van der Arend, 2014); and lack of
interaction and collaboration between researchers and policymakers (Oliver et
al., 2014; Wooding, Hanney, Pollitt, Grant & Buxton, 2014).
A
significant challenge for health system practitioners (both in a clinical
setting and in public service) in implementing research evidence is inadequate
access to information, which results in doctors or analysts being unaware of
the research (Brownson et al., 2014; Oliver
et al., 2014; Ubbink et al., 2013; Ubbink et al., 2011; Wallace et al., 2012).
Various types of information are required for producing high-quality evidence
based policy advice, including journals, books, research reports,
professional/trade magazines, etc. Academic journals and professional magazines
are the largest component of the potentially applicable information. Arguably,
almost all new research is published in journals. That makes access to journals
a key pre-requisite for evidence based policy advice.
In Canada, there are thirteen provinces and
territories responsible for implementing evidence based health policies and
services for the benefit of their populations. The Ministry
of Health and Long-Term Care (MOHLTC) of Ontario, is one of the provincial ministries.
MOHLTC “is working to establish a
patient-focused, results-driven, integrated and sustainable publicly funded
health system” (MOHLTC, 2017a). MOHLTC’s mandate includes (but is not limited
to): “establishing overall strategic direction and provincial priorities for
the health system; developing legislation, regulations, standards, policies,
and directives to support those strategic directions; monitoring and reporting
on the performance of the health system and the health of Ontarians; planning
for and establishing funding models and levels of funding for the health care
system;” etc. (MOHLTC, 2017a). From an organizational point of view, MOHLTC is
a large and complex public corporation (MOHLTC, 2017b). The ministry has
several locations in Toronto, Ontario and local units in major provincial
cities. Among the types of documentation produced by the ministry are: draft
legislations and regulations, briefing notes, reports, program reviews and
evaluations, recommendations, guidelines, etc. (MOHLTC, 2015).
The ministry, within the framework of the
government of Ontario, makes decisions or provides advice on a wide spectrum of
issues by setting agendas, implementing and evaluating solutions and promoting
improvements in the health system. The ministry oversees a health budget of
more than C$50 billion per year. The following may serve as examples of types
of evidence based decisions made by the ministry on an on-going basis:
budget-related planning and control; allocation of public resources to the
local health integrated networks; performance management of the health system
and its elements; economic evaluation and health technology assessment of new
medical interventions, drugs and systems; health system capacity planning, etc.
Specific examples of the ministry’s evidence based decision-making can be found
in Lomas and Brown (2009) and Khan et al. (2014). In 2015, the government of Ontario established a Centre of
Excellence for Evidence-Based Decision Making to enhance the capability of all
ministries in making informed decisions.
MOHLTC takes specific measures to encourage
evidence based policy-making to improve the healthcare system. These measures
include, in particular, development and implementation of the policies and
procedures of using research evidence, providing financial support to
universities in generating new evidence and conducting knowledge transfer, and
building and operating information systems to facilitate access to online
journals, e.g. the Journal Access Centre (JAC).
JAC is made available to all MOHLTC
employees – several thousand people. Obviously, actual use varies across the
departments. For example, research, analytics, policy development, program
evaluation and economic assessment units are known to be regular users of JAC.
The departments with primarily operational functions, such as financial
management, corporate supply chain, and information technology, use JAC with
less regularity. Informal surveys of staff conducted in 2011-2012 show that
several hundred employees use JAC weekly or daily and consider their JAC skills
as advanced.
Prior
to JAC, access to journals was organized through subscriptions by individual
ministry departments. Subscriptions were not coordinated. Overall, employees
had access to several dozen printed journals.
Objectives
The primary
objective of the study was to examine online journal databases usage statistics for a provincial
ministry of health in the context of evidence based decision-making. In addition,
the study highlights implementation of the Journal Access Centre that is housed
and powered by the Ontario Ministry of Health and Long-term Care to inform
health systems’ policy making.
The
research questions guiding this study were:
- What are
the usage statistics of searches that reflect user demand for online
journals?
- What are
the usage statistics of downloaded full-text articles that characterize
the desired output of the journal access solution and reflect consumption
of information?
- What are
the usage statistics of the journal titles accessed by JAC users?
- Can
information needs of MOHLTC users be satisfied by subscription to a core
set of “most important” journals?
Intent
The study has been undertaken with intent to use
the results and recommendations of the research to:
- inform
ministry program evaluation and performance management processes;
- provide
input into economic cost-effectiveness models to optimize JAC journal
acquisition;
- inform
ministry senior management decisions on JAC funding.
Management
decisions that were taken based on the recommendations of this study are not
discussed.
Out
of scope
This study is the first step in
assessing JAC’s outputs. To assess value to the users and organization, the
outcomes of higher-level processes of analyzing retrieved information and
making evidence based decisions need to be examined. Research questions at
these phases would include: What are users doing with the articles they got
from JAC?; Are retrieved articles pertinent and of high quality?; How do users
analyze information?; How do they integrate evidence?; What is the process of
making evidence based decisions in the organization? etc. All these research
questions, although related to the topic of this study, are out of scope of this
research.
The
study examines usage statistics at the institutional level. Analyses at lower
levels, e.g. departmental or individual, are out of scope. Similarly, usage
statistics are examined at the journal title level; article-level
considerations are out of scope.
Methods
This
study had a prospective case study design with descriptive analysis of JAC
usage statistics for two datasets. Dataset 1 contains JAC user statistics on the numbers of searches and
downloaded full-text articles for the period from January 1st 2011
to December 31st 2012 (i.e. two full years of historical data). The
following considerations were taken into account while selecting the period of
historical data. Firstly, trends in current research practice. Similar or
shorter periods of data collection to analyze usage patterns are quite common
in academic studies, e.g.: a recent CIBER research report found that two years
of journal usage data is sufficient to provide insight into a journal’s usage
patterns (CIBER Research Limited, 2011), two-year periods were used in the
studies of Wical and Vandenbark (2015), and Chew, Stemper, Lilyard and
Schoenborn (2013). Secondly, availability of data. A reasonable effort has been
made to collect all pertinent available COUNTER-compliant data (COUNTER, n.d.).
Thirdly, alignment of the data collection period with the objective of the
study and research questions. This was the first study to examine online
journal access for a Canadian ministry of health. The focus was primarily on
capturing and interpreting the current state of usage – not on analyzing
changes in trends (as there was no prior knowledge), which would have required
multiple years of observation periods. Analysis of the usage data allowed the
objective of the study to be achieved.
It has been acknowledged that
information needs and hence usage patterns of the online resources at a
ministry of health are different from those in public health organizations
(e.g. Barr-Walker, 2017; Ford & Korjonen,
2012), health care providers (e.g., Younger,
2010) or of academic researchers (e.g., Haglund & Olsson, 2008; Niu
et al. 2010). Information needs of the policy makers are commonly characterized
by: firstly, a broad spectrum of the subject field which spreads beyond health
care and includes economics, education, housing, etc. (e.g., Brownson, Royer,
Ewing & McBride, 2006); secondly, a wide range of information sources which
are not limited to academic journals and include grey literature and
professional magazines with inputs on politics, values and opinions (e.g.,
Ritter 2009); thirdly, a preference towards simple, uncomplicated information,
for example evidence summaries (e.g., Petkovic et al., 2016; Ritter 2009; Tricco et al., 2016); and finally, a
dynamically changing environment and necessity to make rapid decisions which
require availability of tools with timely access to information (e.g., Oliver
et al., 2014), making systems like JAC indispensable.
A
descriptive analysis of the usage of journal articles was conducted through the
JAC access tool from January 2009 to September 2013 (Dataset 2). Journal usage
statistics for MOHLTC users were downloaded from the EBSCOhost administrative
reporting site (EBSCOhost, n.d.). Journal usage is characterized by the
following indicators: i) number of searches, ii) number of full-text articles
accessed in PDF or HTML format, iii) number of abstracts accessed, and iv)
number of turnaways (access denied). These indicators were selected based on
the recommendations of the internationally recognized standard: COUNTER-2008,
Counting Online Usage of NeTworked Electronic Resources (COUNTER, n.d.).
Definitions of the indicators and related terms are available from COUNTER (n.d.).
Numbers of searches characterize overall intensity of use of the JAC and demand
for this service. Number of full-text articles characterizes the desired output
of the solution and can be linked to the consumption
of information provided by the service.
It
should be noted that at the time of data collection, EBSCO adhered to COUNTER
Release 3. Since then, the COUNTER project has published an updated version of
its Code of Practice – Release 4. Certain terminology and types of usage
reports have been changed. There is no direct correspondence between all types
of reports in Release 3 and Release 4. With some approximation, we can indicate
that the data for performance measures used in the paper can be found in the
following current COUNTER reports: the number of searches – in the Database
Report 1 (DB1); number of abstracts – in the Record Views of DB1; number of
full text downloads – in the Journal Report 1 (JR1); and number of turnaways
(access denied) – in Database Report 2 (DR2).
Journal Access Centre
Implementation
To support
evidence based decisions, MOHLTC of Ontario, Canada, built the Journal Access Centre
(JAC). JAC - an online access tool supported by journal content selection,
acquisition and consultation services – has been in operation with the MOHLTC
since 2008, making the ministry one of the Canadian healthcare pioneers of
online access to academic journals. It was conceived and developed to
facilitate online access to journals and serve as an enabling factor for
enhanced evidence-informed policymaking. JAC’s logic model is presented in
Figure 1.
Figure 1
JAC
logic model.
A
systems view of JAC is presented in Figure 2. MOHLTC acquire access to journals
based on the annual subscriptions procured from various vendors including
journal aggregators and individual publishers. The content is discussed and
suggested for acquisition by the JAC Content Selection Advisory Network – a
permanent working group with representation from each of the ministry’s
divisions.
Mostly,
access to content is acquired by journal databases, which represent collections
of journal titles focused on a certain subject area. Examples could be such
well-known databases as MEDLINE, CINAHL from EBSCO (EBSCO Information Services,
n.d.) or Academic OneFile, Academic OneFile from GALE CENGAGE Learning (Gale
Cengage Learning, n.d.). Commonly, each database contains from a few hundred to
several thousand journals. Some journals and databases are acquired
individually, e.g., The Cochrane Library, Longwoods, etc. The total ministry
subscription covers over 17,000 journal titles with over 9.0 million articles.
These numbers include archives of prior years. Journals cover such topics as
health, medicine, social science, business, policy, economics, finance,
management, risk management, etc.
Usually,
a database contains journals with different levels of access. Some journals are
provided with complete full-text article coverage, others are available only at
an abstract or indexing level. Some latest full-text journals have embargoes –
delays in access from six months to three years. Most databases contain a mix
of academic journals and professional non-peer-reviewed magazines. The types of
content of several databases are illustrated in Table 1. The prime purpose of
JAC is to provide access to the full-text articles because abstracts and
bibliographic data for most journals are available on the Internet free of
charge. Hence, the most valuable segment of a database constitutes full-text,
current, non-embargoed journals. Despite the large number of journals and
articles in the JAC repositories, occasionally a need arises for an article
that is not available in full-text. In these cases, the Article on Demand
Service manned by JAC’s support staff orders materials and sends them to the
JAC users.
The
technological backbone of the solution is a cloud-based application, search
engine, which allows MOHLTC users to access journal databases offered by EBSCO
through the ministry’s intranet. The service is provided through the EBSCO
Integrated Search function. End users need only a web browser to access online
journals. The EBSCO search engine provides integrated coverage of the databases
both owned by EBSCO and bought from different providers – so end users can
conduct a one-click search through all subscribed content. In addition, the
EBSCO search is integrated with Google Scholar search, i.e. when a ministry
user is conducting a search in Google Scholar he/she gets reminders if an article,
presented in Google Scholar search results, is available in the ministry’s
EBSCO subscriptions, and can click on the link to be immediately transferred to
a full-text article in EBSCO repositories. The service also provides automatic
e-mail notifications of new content at an article level, which may be very
specific to meet individual information needs. The service is available 24x7
with short periods of maintenance scheduled during weekends. The service has
proved to be highly reliable: just one three-hour incident of service
disruption has been observed in more than four years.
Figure
2
JAC
structure.
Table
1
Types
of Database Contenta
|
Database |
MEDLINE |
CINAHL |
Business
Source |
Health
Business |
Health
Policy Reference Center |
OmniFile |
AgeLine |
TOTAL |
|
Total
Number of Journals in the Database |
2,184 |
5,453 |
5,023 |
714 |
472 |
3,125 |
209 |
17,180 |
|
Abstracts Only |
0 |
3,825 |
1,191 |
64 |
37 |
0 |
209 |
5,326 |
|
Full-Text |
2,184 |
1,628 |
3,832 |
650 |
435 |
3,125 |
0 |
11,854 |
|
Stopped Full-Text |
357 |
537 |
1,139 |
414 |
149 |
982 |
0 |
3,578 |
|
Current Full-Text |
1,827 |
1,091 |
2,693 |
236 |
286 |
2,143 |
0 |
8,276 |
|
Embargoed Current Full-Text |
1,183 |
277 |
638 |
67 |
81 |
274 |
0 |
2,520 |
|
Non-Embargoed Current Full-Text |
644 |
814 |
2,055 |
169 |
205 |
1,869 |
0 |
5,756 |
|
Non-Embargoed Current Full-Text Peer-Reviewed |
608 |
587 |
943 |
140 |
143 |
1,184 |
0 |
3,605 |
|
Non-Embargoed Current Full-Text Magazines, Trade Publications |
36 |
227 |
1,112 |
29 |
62 |
685 |
0 |
2,151 |
aSome databases have overlapping
content (journal titles). The number of unique journal titles in the system is
less than shown in the table.
Figure
3
Number
of searches (monthly average for 2011-2012).
Figure
4
Number
of downloaded full-text articles (quarterly in 2011and 2012).
Usage Statistics
Results
This
section presents the results for the number of searches from a report covering
two full calendar years 2011-2012. Figures 3 and 4 show monthly average number
of searches, and quarterly number
of downloaded full-text articles respectively. Reported number of turnaways
(access denied) is zero.
A
report on the number of full-text articles and abstracts accessed by journal
title for approximately 5000 journals is presented in Additional file
(Appendix). This report covers the period from January 1, 2009 through
September 30, 2013. A list of the most frequently used journals (top-50) is
presented in Table 2.
Table 2
List of the Most
Frequently Used Journalsa
|
ISSN |
Title |
Full Text |
Abstract |
Category |
|
00178012 |
Harvard
Business Review |
2208 |
734 |
NPR |
|
00900036 |
American
Journal of Public Health |
303 |
129 |
PR |
|
00284793 |
New
England Journal of Medicine |
270 |
189 |
PR, E3 |
|
08203946 |
Canadian
Medical Association Journal |
183 |
177 |
PR |
|
00084263 |
Canadian
Journal of Public Health |
129 |
113 |
PR |
|
15445208 |
Health
Affairs |
123 |
139 |
PR |
|
00028614 |
Journal
of the American Geriatrics Society |
118 |
74 |
PR, E12 |
|
03092402 |
Journal
of Advanced Nursing |
118 |
40 |
PR, E12 |
|
01628968 |
Inc. |
114 |
5 |
NPR |
|
13558196 |
Journal
of Health Services Research and Policy |
112 |
64 |
PR, E12 |
|
1357633X |
Journal
of Telemedicine and Telecare |
94 |
9 |
PR |
|
07067437 |
Canadian
Journal of Psychiatry |
89 |
59 |
PR |
|
01607480 |
Modern
Healthcare |
86 |
47 |
NPR |
|
09269630 |
Studies
in Health Technology and Informatics |
69 |
49 |
PR, E12 |
|
10688838 |
HandHN:
Hospitals and Health Networks |
69 |
25 |
NPR |
|
00034819 |
Annals
of Internal Medicine |
68 |
44 |
PR |
|
11707690 |
PharmacoEconomics |
67 |
31 |
PR, E6 |
|
01406736 |
Lancet |
66 |
154 |
PR |
|
0887378X |
Milbank
Quarterly |
64 |
40 |
PR, E12 |
|
17561833 |
BMJ:
British Medical Journal |
63 |
115 |
PR, E3 |
|
10966218 |
Journal
of Palliative Medicine |
62 |
29 |
PR, E12 |
|
14726963 |
BMC
Health Services Research |
58 |
50 |
PR |
|
13652702 |
Journal
of Clinical Nursing |
55 |
23 |
PR, E12 |
|
13869620 |
Health
Care Management Science |
54 |
33 |
PR, E12 |
|
09660410 |
Health
& Social Care in the Community |
53 |
21 |
PR, E12 |
|
13561294 |
Journal
of Evaluation in Clinical Practice |
53 |
19 |
PR, E12 |
|
01958631 |
Health
Care Financing Review |
53 |
15 |
PR |
|
07350732 |
Healthcare
Financial Management |
51 |
40 |
PR |
|
09652140 |
Addiction |
51 |
13 |
PR, E12 |
|
00197939 |
Industrial
and Labor Relations Review |
50 |
21 |
PR |
|
14712458 |
BMC
Public Health |
48 |
29 |
PR |
|
10792082 |
American
Journal of Health-System Pharmacy |
48 |
18 |
PR |
|
1477030X |
Palliative
Medicine |
48 |
14 |
PR |
|
07461739 |
Nursing
Economics |
46 |
20 |
PR |
|
00333107 |
Psychology
Today |
46 |
8 |
NPR |
|
03190781 |
Toronto
Star (Toronto,
Ontario) |
43 |
32 |
NPR |
|
00130613 |
Economist |
42 |
24 |
NPR |
|
1095158X |
Psychiatric
Rehabilitation Journal |
42 |
24 |
PR, E12 |
|
87569728 |
Project
Management Journal |
41 |
39 |
PR, E12 |
|
03616878 |
Journal
of Health Politics, Policy and Law |
40 |
40 |
PR, E12 |
|
14756773 |
Health
Services Research |
40 |
34 |
PR |
|
00413674 |
Trustee |
40 |
31 |
NPR |
|
00296570 |
Nursing
Standard |
40 |
25 |
PR |
|
10903127 |
Prehospital
Emergency Care |
38 |
16 |
PR, E18 |
|
10786767 |
Journal
of Health Care Finance |
37 |
19 |
PR |
|
08982759 |
Physician
Executive |
37 |
14 |
PR |
|
00048674 |
Australian
and New Zealand Journal of Psychiatry |
36 |
16 |
PR |
|
08835381 |
Healthcare
executive |
35 |
24 |
NPR |
|
15414469 |
International
Journal of Health Services |
34 |
44 |
PR, E6 |
|
08949867 |
Journal
of Traumatic Stress (Wiley) |
34 |
23 |
PR, E12 |
aAbbreviations in Table 2: PR – peer-reviewed, NPR –
non-peer-reviewed, E – embargoed with indication of the delay in months.
Figure
5
Distribution
of the number of journals falling into journal groups with similar numbers of
full-text downloads.
Based on the usage statistics, a histogram
was developed. Figure 5 shows distribution of the number of journals falling into
journal groups with similar numbers of full-text downloads. The horizontal axis is non-linear for better
visual presentation.
Discussion
Figures 3 and 4 are interrelated and are
built in a COUNTER-compliant way (searches, and downloads).
Figure 3 shows the number of
searches per month. During 2011-2012, there were over 123,000 searches with monthly averages
of 5,129. Smaller increments of time have been
used to reveal usage patterns on a relatively short data period of Dataset 1.
It should be noted that this was the first effort to collect data and analyze
JAC usage. There was no prior information about the usage: whether the system
had been used only from time to time (e.g. to support decision-making for an
important health care policy or regulation) or whether it had been used at all.
It should be acknowledged that the
charts are not exposing unexpected trends. Figures 3 and 4 can be interpreted
along the following lines. The data provides evidence that December is a
slowdown period in JAC use (minimum number of searches). This is quite
understandable due to a number of statutory holidays and traditional vacation
time. In January and February users increase JAC activity (usage numbers show a
steep rise) to make up for a slow year-end and to retrieve information that
they need in planning for the next fiscal period (April 1 is the starting date
for the government of Ontario fiscal year). Usage during all other months of
the year does not display large variations. JAC usage demonstrates a steady
level of activity most of the year, i.e. it has become part of routine
government knowledge-management processes.
The
number of full-text articles, either downloaded in PDF format or viewed on the
computer screen in HTML, characterizes the desired output of the system. Figure
4 demonstrates the actual consumption of information in JAC databases: in
2011-2012, over 4,600 full-text articles were accessed. During the same period
of time more than 5,800 abstracts were accessed. This indicator is secondary,
keeping in mind that the main purpose of JAC is to provide access to full-text
articles. However, the fact that the user accessed the abstract could be seen
as an evidence of interest that the user had in the article but full-text may
not have been available. If a journal, not available in full-text, has
experienced multiple abstract viewing, it testifies that this journal should be
considered for subscription in full-text version.
It
was noted that some journals were accessed at abstract level extensively, but
had zero full-text downloads. That may demonstrate that these journals publish
pertinent articles but are not available with full-text. The ten journals that
were most frequently accessed in this way (from 117 to 30 times) were: JAMA – Journal of the American Medical
Association, Healthcare Quarterly, Medical Care, Healthcare Papers, Health
Policy, Vaccine, Journal of Palliative Care, Diabetes Educator. MOHLTC may
consider exploring subscription to these journals with full-text taking into
account cost-efficiency (Botchkarev, 2013).
Usage
statistics as numbers of full-text articles and abstracts of individual
journals for the period from 2009 to 2013 are presented in Additional file
(Appendix). Usage statistics for the top-50 most frequently used journals are
shown in Table 2. This data shows that MOHLTC users accessed 12,790 full-text
articles and 14,517 abstracts. A total of 4,759 journal titles were accessed
including 1,675 journals with full-text. The Harvard Business Review is by far the most frequently used journal
– it was used seven times more than the second-ranked journal: the American Journal of Public Health.
Table
2 demonstrates that the most frequently used journals include both
peer-reviewed (82%) and non-peer-reviewed (18%) periodicals. Approximately 50%
of the most frequently used peer-reviewed journals have embargoes from 3 to 18
months, predominantly 12 months.
The
histogram presented in Figure 5 indicates spread of usage across multiple
journal titles. Usage does not demonstrate a “core” set of journals There are
only 10 journals with over 100 downloads each. The total number of downloads
from these journals is 3,678. Similarly, there are only 30 journals that were
accessed in full-text format 50 times or more. These journals contributed only 38%
(4,953) of the accessed articles. Of the accessed journals, 54% (908 titles)
were accessed with full-text only once or twice. This group contributed 1,192
articles (9%). This pattern of usage can be attributed to the following
factors. First, there is an increasing amount of research being conducted that
triggers a persistently growing number of publication outlets. Second, MOHLTC
has a very broad area of responsibilities with dynamically changing priorities,
which translates into the diverse information needs of its employees. A
practical conclusion from the usage analysis is that MOHLTC information needs
cannot be mapped to a reasonably compact set of “core” journals with a
subsequent subscription to those. In this case, subscription economics necessitate
the use of journal aggregators, e.g. EBSCO, Gale, etc., as the main source of
journal access acquisition (Botchkarev, 2013).
Certain
JAC user feedback is notable. This information has been collected in
non-structured conversations with clients and is not supported by quantitative
assessments. Despite availability and actual use of thousands of journals,
there is a need to (i) expand access to
more peer-reviewed journals, (ii) expand access to more journals with full-text
articles, (iii) explore opportunities to reduce embargoes. Some users
expect immediate online access to the full-text articles of interest,
requesting no abstract-only, no delays/embargoes content. If these expectations
are not met, user satisfaction might decline rapidly.
The focus of this empirical study is
on usage statistics in terms of number of downloads, etc., which represent the
outputs of the JAC solution. There is no implication or judgement about the
value that JAC collections provide to users. This study is the first step in assessing
value to the users and organization. It needs to be emphasized that there is no
rationale to judge the value of JAC collections through the rate of downloads
per user (high or low – good or bad) because there is nothing to compare to
either in MOHLTC or in other organizations (e.g. commonly accepted benchmarks
do not exist). To assess value, we would need to consider the outcomes of
higher-level processes.
The findings reported in this paper can be used
in various government organizations to implement journal subscriptions
strategies that will better meet ministry employees’ information needs and
contribute to cost-efficiency of operations.
Study
Assumptions and Limitations
To
the best of our knowledge, this is the first study to address usage statistics
for online journal databases in a Canadian ministry of health. However, this
study has certain limitations that should be made explicit.
Use
of JAC implies that its collections are of interest for the MOHLTC employees.
However, numbers of article downloads may not be equal to actual use or
satisfaction – users may download an article and find it worthless for their
task or they may be unhappy because they did not find specifics they needed.
JAC
statistics used in this study have been downloaded from the EBSCO reporting
site. This data is based on the automatic logs and is believed to be very
accurate. However, the following should be noted.
Firstly,
JAC is not an exclusive channel of information for MOHLTC users. Some users
have access to online journals at local universities through their alumni
connections. Others have access to journal repositories based on their
memberships in professional associations. Certain departments used to have
subscriptions to publications in their specific narrow fields. As a result,
documented in the study number of downloads represents a lower border of actual
downloads.
Secondly,
JAC is using EBSCO integrated search services that allow access to the
databases that are owned by EBSCO, and those of the third parties external to
EBSCO. Search results presented to the user include both internal and external
documents. When a person clicks on the link to an external database, he/she is
transferred to the document in the external database. As soon as a person moves
to an external database, EBSCO usually does not have information about what is
happening there, and so cannot include activity in the report. That pertains
especially to full-text documents. As a result, JAC statistics may be missing
data on the use of full-text documents in external databases. An example of
this situation could be statistics on the use of the Cochrane database shown in
the Additional file (Appendix). JAC has a direct subscription to the Cochrane
database with full-text documents, which makes it external to EBSCO. The usage
statistics indicate zero downloads of Cochrane full-text documents. However,
this is misleading as it has been verified through conversations with JAC users
that the Cochrane database actually has been used.
Thirdly,
in some cases, EBSCO usage reports are not perfect. There were a certain number
of duplications of journal titles in the initial version of Additional file
(Appendix). For example, (i) some titles were duplicated because of using
different online and print ISSNs (e.g., Academic
Emergency Medicine: Official Journal Of The Society For Academic Emergency
Medicine: Usage was attributed to one title. Another was deleted); (ii)
some titles were duplicated because in one case the title had an ISSN and in
another the ISSN was blank (e.g., The
Academy of Management Executive:
Usage was attributed to one title. Another was deleted); (iii) some titles were
duplicated because of spelling mistakes (e.g., American Journal of PublicHealth vs American Journal of Public Health); (iv) different title
abbreviations were used – most likely in different databases – (e.g., BMC HEALTH SERVICES RESEARCH vs BMC Health Serv Res); (v) Use of &
instead of AND; (vi) using titles with or without the definite article; and
(vi) just typos.
Fourthly,
it was shown in annual customer surveys (not reported in the current paper),
that JAC users were not satisfied when search results contained a large number
of articles with abstracts only. It took additional time to look through
several pages of search results to find articles with full text. In March 2012,
according to the recommendation of the JAC Content Selection Advisory Network,
the JAC default search was configured to present full-text articles only. If a
researcher was willing to analyze additional abstracts-only articles, he/she
could adjust the search configuration. That may have decreased the number of
retrieved abstracts from the second half of 2012.
Conclusions
This
is the first paper, to our knowledge, to examine usage statistics for online
journal databases in a ministry of health.
The
Ministry of Health and Long-Term Care built and has maintained the Journal
Access Centre – an online access tool supported by journal content selection,
acquisition and consultation services. As a key prerequisite for
evidence based policy-making, JAC enables access to approximately
12,000 journals with full-text articles.
JAC usage statistics for the 2011 – 2012
calendar years demonstrate a steady level of activities in terms of searches with a monthly average of 5,129. JAC’s steady level of activities, revealed by the
study, reflects a continuous demand for the JAC services
and products. It testifies that access
to online journal databases has become part of routine government
knowledge-management processes.
The
number of downloaded full-text articles characterizes the desired output of the
solution. JAC usage statistics demonstrate that the actual consumption in
2009-2013 was over 12,790 full-text articles or approximately 2,700 articles
annually.
Usage
statistics helped to identify a list of journals that were accessed at abstract
level extensively (117 to 30 times), but had zero full-text downloads. That led
to a practical recommendation to consider
subscription to these journals with full-text, taking into account
cost-efficiency.
JAC
usage statistics for the period 2009-2013 provide evidence that a total of
4,759 journal titles were accessed including 1,675 journals with full-text. MOHLTC’s broad area of responsibilities with dynamically
changing priorities translates into the diverse information needs of its
employees and a large set of required journals. Usage statistics indicate that
MOHLTC information needs cannot be mapped to a reasonably compact set of “core”
journals with a subsequent subscription to those. In this case,
subscription economics necessitate the use of journal aggregators (e.g. EBSCO,
Gale, etc.) as the main source of journal access acquisition.
Future efforts could be focused on studying: (i)
usage statistics complemented with data beyond EBSCO reports and covering all
sources of online academic journals available in JAC; (ii) usage statistics by
individual EBSCO databases, e.g. MEDLINE, CINAHL, etc., and category of
journals, e.g. peer-reviewed, non-peer-reviewed, embargoed, etc.; iii) in-depth
usage of information sources and patterns of behaviour at the level of
individual article as compared to the journal title level in this paper; (iv)
JAC users’ information needs and preferences; (v) JAC’s usability through the
customer satisfaction survey; and (vi) development
and examination of a model of a value creation chain which would integrate JAC
outputs with outcomes of higher-level processes of analyzing retrieved
information and making evidence based decisions.
Acknowledgements
The views, opinions and conclusions expressed in this
document are those of the author alone and do not necessarily represent the
views of the Ontario Ministry of Health and Long-Term Care or any of its
individual departments.
The author is grateful to Dr. Grégory Léon,
Université Laval, Dr. Mathieu Ouimet, Université Laval and Dr. John N Lavis,
McMaster University, for helpful
comments on earlier drafts of the paper that had a distinct scope and
objectives. However, mentioned scholars are not responsible for the content and
conclusions of the published paper.
References
Barr-Walker, J. (2017). Evidence-based
information needs of public health workers: A systematized review. Journal of the Medical Library Association: JMLA, 105(1), 69-79. http://dx.doi.org/10.5195/jmla.2017.109
Botchkarev, A.
(2013). Mining evidence in the online academic journal databases: Access and
performance in public sector. E-prints in
Library and Information Science (E-LIS). Retrieved from http://eprints.rclis.org/19286
Brownson, R. C., Reis, R. S., Allen, P., Duggan, K., Fields, R.,
Stamatakis, K. A., & Erwin, P. C. (2014). Understanding administrative
evidence-based practices. American Journal of Preventive Medicine, 46(1),
49-57. http://dx.doi.org/10.1016/j.amepre.2013.08.013
Brownson, R. C., Royer, C., Ewing, R.,
& McBride, T. D. (2006). Researchers and policymakers: Travelers in
parallel universes. American
Journal of Preventive Medicine, 30(2),
164-172. http://dx.doi.org/10.1016/j.amepre.2005.10.004
Chew, K., Stemper, J., Lilyard, C. and Schoenborn, M.
(2013). User-defined valued metrics for electronic journals. In S. Hiller, M.
Kyrillidou, A. Pappalardo, J. Self & A. Yeager (Eds.). Proceedings of the 2012 Library Assessment Conference, Charlottesville, VA, Oct 29-31.
Retrieved from http://conservancy.umn.edu/bitstream/handle/11299/144461/1/UserDefinedValueBasedMetricsForElectronicJournalsPDF.pdf
CIBER Research Limited. (2011). The journal usage factor: Exploratory data analysis: Stage 2 Final
Report, 27 May 2011. Retrieved from http://ciber-research.eu/download/20110527-The_Journal_Usage_Factor.pdf
COUNTER (n.d.). Counting online
usage of networked electronic resources. Retrieved from http://www.projectcounter.org
EBSCO Information Services. (n.d.). Retrieved from https://www.ebsco.com
EBSCOhost. (n.d.). Retrieved from http://eadmin.ebscohost.com/EAdmin/Reports/SelectReportsForm.aspx
Field, P., Gauld, R., & Lawrence, M. (2012). Evidence-informed
health policy - the crucial role of advocacy. International Journal of
Clinical Practice, 66(4), 337-341. http://dx.doi.org/10.1111/j.1742-1241.2012.02883.x
Ford, J., & Korjonen, H. (2012).
Information needs of public health practitioners: A review of the literature. Health Information & Libraries Journal, 29(4), 260-273. http://dx.doi.org/ 10.1111/hir.12001
Gale Cengage Learning. (n.d.). Retrieved from http://www.gale.cengage.com
Haglund, L., &
Olsson, P. (2008). The impact on university libraries of changes in information
behavior among academic researchers: a multiple case study. The Journal of Academic Librarianship, 34(1), 52-59. http://dx.doi.org/10.1016/j.acalib.2007.11.010
Humphries, S., Stafinski, T., Mumtaz, Z., & Menon, D. (2014). Barriers
and facilitators to evidence-use in program management: A systematic review of
the literature. BMC Health Services Research, 14, 171. http://dx.doi.org/10.1186/1472-6963-14-171
Khan, S., Moore, J. E., Gomes, T.,
Camacho, X., Tran, J., McAuley, G., Juurlink, D.N., Paterson, M., Laupacis, A.,
& Mamdani, M. M. (2014). The Ontario drug policy research network: Bridging
the gap between research and drug policy. Health policy, 117(3),
392-398. http://dx.doi.org/10.1016/j.healthpol.2014.06.007
Lomas, J., & Brown, A. D. (2009). Research and advice giving: A
functional view of evidence-informed policy advice in a Canadian ministry of
health. Milbank Quarterly, 87(4), 903-926. http://dx.doi.org/10.1111/j.1468-0009.2009.00583.x
Madhavji, A., Araujo, E. A., Kim, K. B., & Buschang, P. H. (2011).
Attitudes, awareness, and barriers toward evidence-based practice in
orthodontics. American Journal of Orthodontics and Dentofacial Orthopedics,
140(3), 309-316. http://dx.doi.org/:10.1016/j.ajodo.2010.05.023
MOHLTC. (2015). Publications. Retrieved from http://www.health.gov.on.ca/en/public/publications/default.aspx
MOHLTC. (2017a). About
the Ministry. Retrieved from http://www.health.gov.on.ca/en/common/ministry/default.aspx
MOHLTC. (2017b). Organization chart. Retrieved from http://www.health.gov.on.ca/en/common/ministry/orgchart.pdf
Niu, X., Hemminger, B. M., Lown, C.,
Adams, S., Brown, C., Level, A., McLure, M., Powers, A., Tennant, M.R., & Cataldo, T. (2010). National study
of information seeking behavior of academic researchers in the United States. Journal of the American Society for Information
Science and Technology, 61(5),
869-890. http://dx.doi.org/
10.1002/asi.21307
O'Connor, S., & Pettigrew, C.M. (2009). The barriers perceived to
prevent the successful implementation of evidence-based practice by speech and
language therapists. International Journal of Language & Communication
Disorders, 44(6), 1018-1035. http://dx.doi.org/10.1080/13682820802585967
Oliver, K., Innvar, S., Lorenc, T., Woodman, J., & Thomas, J.
(2014). A systematic review of barriers to and facilitators of the use of
evidence by policymakers. BMC Health Services Research, 14, 2. http://dx.doi.org/10.1186/1472-6963-14-2
Petkovic, J., Welch, V., Jacob, M. H.,
Yoganathan, M., Ayala, A. P., Cunningham, H., & Tugwell, P. (2016). The
effectiveness of evidence summaries on health policymakers and health system
managers use of evidence from systematic reviews: A systematic review. Implementation Science, 11, 162. http://dx.doi.org/10.1186/s13012-016-0530-3
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher,
D., Callaghan J., & Holter, M. (2010). Barriers to evidence-based practice
implementation: Results of a qualitative study. Community Mental Health
Journal, 46(2), 112-118. http://dx.doi.org/10.1007/s10597-009-9238-z
Ritter, A. (2009). How do drug policy
makers access research evidence? International
Journal of Drug Policy, 20(1),
70-75. http://dx.doi.org/10.1016/j.drugpo.2007.11.017
Solomons, N. M., & Spross, J. A. (2011). Evidence-based practice
barriers and facilitators from a continuous quality improvement perspective: An
integrative review. Journal of Nursing Management, 19(1), 109-120. http://dx.doi.org/10.1111/j.1365-2834.2010.01144.x
Sosnowy, C. D., Weiss, L. J., Maylahn, C. M., Pirani, S. J., &
Katagiri, N. J. (2013). Factors affecting evidence-based decision making in
local health departments. American Journal of Preventive Medicine, 45(6),
763-768. http://dx.doi.org/10.1016/j.amepre.2013.08.004
Tricco, A. C., Cardoso,
R., Thomas, S. M., Motiwala, S., Sullivan, S., Kealey, M. R., Hemmelgarn, B.,
Ouimet, M., Hillmer, M.P., Perrier, L., Shepperd, S., & Straus, S.E.
(2016). Barriers and facilitators to uptake of systematic reviews by policy makers
and health care managers: A scoping review. Implementation Science, 11, 4. http://dx.doi.org/10.1186/s13012-016-0370-1
Ubbink, D. T., Guyatt, G. H., & Vermeulen, H. (2013). Framework of
policy recommendations for implementation of evidence-based practice: A
systematic scoping review. BMJ Open, 3(1). http://dx.doi.org/10.1136/bmjopen-2012-001881
Ubbink, D. T., Vermeulen, H., Knops, A. M., Legemate, D. A., Oude
Rengerink, K., Heineman, M. J., Roos Y.B., Fijnvandraat C.J., Heymans H.S.,
Simons R., & Levi, M. (2011). Implementation of evidence-based practice:
outside the box, throughout the hospital. Netherlands Journal of Medicine,
69(2), 87-94.
van der Arend, J. (2014). Bridging the research/policy gap: Policy
officials' perspectives on the barriers and facilitators to effective links
between academic and policy worlds. Policy Studies, 35(6),
611-630. http://dx.doi.org/doi:10.1080/01442872.2014.971731
Wallace, J., Nwosu, B., & Clarke, M. (2012). Barriers to the uptake
of evidence from systematic reviews and meta-analyses: A systematic review of
decision makers’ perceptions. BMJ Open, 2(5). http://dx.doi.org/10.1136/bmjopen-2012-001220
Wical, S.H. and Vandenbark, R.T. (2015).
Notes on operations: Combining citation studies and usage statistics to build a
stronger collection. Library Resources & Technical Services, 59(1),
pp.33-42. Retrieved from https://journals.ala.org/lrts/article/view/2755/2733
Wooding, S., Hanney, S. R., Pollitt, A., Grant, J., & Buxton, M. J.
(2014). Understanding factors associated with the translation of cardiovascular
research: A multinational case study approach. Implementation Science,
9, 47. http://dx.doi.org/10.1186/1748-5908-9-47
Younger, P. (2010). Internet‐based information‐seeking
behaviour amongst doctors and nurses: A short review of the literature. Health Information & Libraries Journal, 27(1), 2-10. http://dx.doi.org/10.1111/j.1471-1842.2010.00883.x