Introduction
Systematic reviews aim
“to systematically search for, appraise and synthesize research evidence”
(Grant & Booth, 2009, p. 95). Unlike traditional literature reviews, the
systematic review aims to minimize bias by following a systematic and
transparent approach to defining the question, searching the literature,
extracting relevant data, assessing the quality of the literature, and
synthesizing and drawing conclusions on the state of the evidence base (Boland, Cherry, &
Dickson, 2013; Higgins et
al., 2011; Centre for Reviews and Dissemination [CRD], 2009).
Systematic reviews in
LIS, as in other professional domains, are becoming increasingly popular.
Paralleling the evidence based medicine paradigm, systematic reviews are
considered high levels of evidence within the LIS domain (Eldredge,
2000) and incorporate into their methodology one of the core skills of LIS
professionals—literature searching. The number of systematic reviews in the LIS
field is growing (Koufogiannakis, 2012), and journal editors are actively
encouraging librarians to undertake them (Sampson, 2014). However, LIS research
methods are diverse (Brettle, 2009), and they are not
easy to work with in the context of a systematic
review (Koufogiannakis, 2012). LIS systematic reviewers, therefore, may be
faced with the prospect of including and assessing the quality of a diverse
evidence base.
This study aims to
provide an overview of approaches to quality assessment used by LIS systematic
reviewers. The paper models and presents a collation of quality assessment
tools. The results will be particularly useful for those undertaking LIS
systematic reviews and evidence summaries.
Literature Review
In defining quality as
a “multidimensional
concept which could relate to the design, conduct, and analysis of a trial, its
clinical relevance, or quality of reporting,” Jüni, Altman, & Egger (2001, p. 42) identify three approaches to quality
assessment: internal validity, external validity, and reporting quality.
Internal validity refers to the robustness of the methods and the extent to
which bias (i.e. systematic errors in studies that may lead to an
overestimation or underestimation of the true result) is minimized and the
results of the study can be considered reliable. External validity is the
extent to which the study results can be considered generalizable to other
settings or populations, and reporting quality assesses the completeness of the
reporting.
Quality assessment is
an integral part of the systematic review process (Boland, Cherry, & Dickson, 2013). Details on how reviewers intend to assess the
quality of the included studies should be outlined a priori in the review
methods section (Moher et al., 2009). In addition, the results of the quality
assessment of the included studies should be presented and consideration given not
only to the quality of the individual studies but also to how the overall
quality of the included studies impacts on the validity of the review findings
(Shea et al., 2007). Given the subjective nature
involved in quality assessment, it is imperative that more than one reviewer
undertakes this process.
A multitude of tools
are available to help reviewers assess the quality of the included studies.
Tools for assessing the quality of research fall into three categories: scales,
checklists, and domains (see box 1; West et al., 2002). Despite a lack of
consensus about which quality assessment tools are best, there is some
agreement that scales are not the most appropriate tool. The shortcomings of
scales are that different scales can produce different results (Jüni et al., 2001; Booth, 2007) and that users may assume
each criteria on a scale is equally weighted. In the
case of double-blinding in a randomised controlled trial, for example, such an
assumption may not be appropriate (Schulz, Chalmers, Hayes, & Altman,
1995).
Quality assessment of
studies is essential to maintaining the integrity of the systematic review
since the validity of the review conclusions rely upon the quality of the
included studies (Jüni et al., 2001; Deeks et al., 2003;
Sampson, 2014). Empirical research in the health and medical field found that
poor methodological quality of trials can overestimate or underestimate the
true result (Schulz et al., 1995). Combining such studies in a systematic
review, therefore, can only serve to emphasize such biases even further.
There is limited
empirical evidence of how quality assessment is incorporated in LIS systematic
reviews. A recent editorial undertook a brief examination of systematic reviews
published in the Journal of the Medical
Library Association (Sampson, 2014). Whilst the number of systematic
reviews identified was small (four), only two of them assessed the quality of
the literature of the included studies. With the growing popularity of
systematic reviews within the LIS field, it is timely to consider how and to
what extent LIS reviewers incorporate assessments of quality in LIS systematic
reviews.
Box 1
Categories of Quality
Assessment Toolsa
|
Checklist: “Instruments that contain a number of
quality items, none of which is scored numerically.”
Scale: “Instruments that contain several quality
items that are scored numerically to provide a quantitative estimate of
overall study quality.”
Domain (or component): “Individual aspect of study
methodology (e.g. randomisation, blinding,
follow-up) that has a potential relation to bias in estimation of effect.”
|
aWest et al., 2002, p. 33
Aims
To provide an overview
of approaches to quality assessment in published LIS systematic reviews. In
particular, to:
- assess whether and how LIS reviewers
report on quality assessment a priori in systematic reviews,
- model the different quality assessment
aids used by LIS reviewers, and
- explore if and how LIS reviewers report
on and incorporate the quality of included studies into the systematic
review analysis and conclusions.
Methods
The authors undertook
a methodological study of published LIS systematic reviews. A cohort of
published systematic reviews of LIS-related research was identified from the literature
in November 2014, using an existing wiki that claims to document “all the known
systematic reviews in library and information studies” (Koufogiannakis, Brettle, Booth, Kloda, &
Urquhart, 2015).
Studies in the list
were included if they were reported as a systematic review in the title,
abstract, or methods section. Meta-analyses that did not incorporate a
systematic review and studies in which the systematic review was not a main
objective were excluded. Two reviewers independently assessed the studies
against the inclusion criteria. In the event of disagreement, consensus was
reached via discussion.
Data were extracted on
the type of synthesis, whether quality assessment was planned and undertaken,
the number of reviewers involved in assessing quality, the types of tools or
criteria used to assess the quality of the included studies, how quality
assessment was presented in the systematic review, and whether the quality of
the included studies was considered in the analysis and conclusions of the
review.
To determine the
quality of the reporting and incorporation of quality assessment in LIS
systematic reviews, two reviewers independently assessed each included study
against criteria relating to quality assessment in the PRISMA reporting guidelines
for systematic reviews and meta-analyses (Moher et al., 2009) and the AMSTAR
tool (Shea et al., 2007) (see box 2). The PRISMA guidance (Moher et al., 2009)
assesses the quality of the reporting in systematic reviews. Within the 27-item
checklist, four items relate to the reporting of quality assessment within
systematic reviews. AMSTAR (Shea et al., 2007) is an
11-item checklist that aims to assess the methodological quality of systematic
reviews. Four of the items relate to quality assessment.
Box 2
PRISMA and AMSTAR
Quality Assessment Criteria
|
PRISMAa
Item #12: Risk of bias in individual studies
Describe
methods used for assessing risk of bias of individual studies (including
specification of whether this was done at the study level or outcome level)
and how this information is to be used in any data synthesis
Item #15: Risk of bias across studies
Specify any
assessment of risk of bias that may affect the cumulative evidence (e.g.,
publication bias, selective reporting within studies)
Item #19: Risk of bias within studies
Present data
on risk of bias of each study and, if available, any outcome level assessment
Item #22: Risk of bias across studies
Present
results of any assessment of risk of bias across studies
AMSTARb
Item #7:
Was the scientific quality of the included studies
assessed and documented?
Item #8:
Was the scientific quality of the included studies
used appropriately in formulating conclusions?
Item #10:
Was the likelihood of publication bias assessed?
|
aMoher et al., 2009
bShea et al., 2007
Studies were classed
as adequate if they were explicit in meeting the PRISMA or AMSTAR criteria, inadequate
if they partially met the criteria, or unclear if the study did not report on the
item. A narrative synthesis of the results is presented.
Results
A total of 40 studies
reported on the wiki met the inclusion criteria. Of the 10 studies excluded,
four were meta-analyses (Aabø, 2009; Ankem, 2006b;
Salang, 1996; Saxton, 1997), five were not reported as systematic reviews (Haug, 1997; Julien, Leide, & Bouthillier, 2008; Mairs, McNeil,
McLeod, Prorok, & Stolee,
2013; Ward, Stevens, Brentnall, & Briddon, 2008; Williams, Nicholas, & Rowlands, 2010), and in one study (Sampson, McGowan, Lefebvre,
Moher, & Grimshaw, 2008), the systematic review was not the objective of
the study.
Table 1 outlines the
characteristics of the 40 systematic reviews included in the analysis. More
than half were published in the health LIS field. The number of studies
included in the systematic reviews ranged from 3 to 333. Nine systematic
reviews were undertaken by only one author, 15 reported two authors, and 16
reported more three or more authors. Five systematic reviews reported including
only one type of study design, 22 included more than one type of study design,
and 13 did not report on study designs.
Reporting of Quality
Assessment Methods in LIS Systematic Reviews
Table 2 outlines the
approach to quality assessment undertaken in the included studies. Fourteen of
the 40 failed to state they would undertake quality assessment in the methods.
Of these 14, 10 were narrative reviews, three were qualitative, and one a
meta-analysis with narrative review. Of the 14 systematic reviews that did not
report on quality assessment in the methods, six did report on the quality of
the included studies in the results or discussion (Ankem,
2006a; Fanner & Urquhart, 2008; Genero,
Fernandez-Saez, Nelson, Poels,
& Piattini, 2011; Kelly & Sugimoto, 2013;
Matteson, Salamon, & Brewster, 2011; Wagner &
Byrd, 2004).
Less than half of
those that did undertake quality assessment (11 of 26) actually defined what
they meant by quality. Eight studies defined quality as an assessment of the
methodological quality (Koufogiannakis and Wiebe, 2006; Zhang, Watson, & Banfield, 2007; Joshi & Trout, 2014; Golder & Loke,
2009, 2010; Perrier et al., 2014; Gagnon et al., 2010; Divall,
Camosso-Stefinovic, & Baker., 2013), and three studies (Perrier et al, 2014;
Gagnon et al, 2010; Divall et al., 2013) specified
quality as the risk of bias. Two studies defined quality assessment as
assessing the quality of the study design, or level of evidence (Manning Fiegen, 2010; Ndabarora, Chipps,
& Uys, 2014). One study assessed the quality of the
reporting (Crumley,
Wiebe, Cramer,
Klassen, & Hartling, 2005).
Fourteen studies
reported the number of people involved in undertaking quality assessment. The
number of authors undertaking quality assessment in a systematic review ranged
from 1 to 8. Three studies (Joshi & Trout, 2014; Manning Fiegen, 2010; Perrier et al., 2014) reported an assessment
of the inter-rater agreement of quality assessment between reviewers. Nine studies reported using quality assessment
as part of their inclusion and exclusion criteria.
Quality Assessment
Tools
Table 3 outlines the
aids used by LIS systematic reviewers. LIS reviewers referenced four types of
aids to quality assessment: 14 published tools designed specifically to assess
quality, 3 research design books, 11 journal articles, and 1 web resource.
Twenty-nine unique aids were referenced.
Twenty-one studies
that reported on quality assessment in the methods section referenced the tools
or aids used: 14 studies reported using one aid, four studies reported using
two aids, and three studies used three or more aids (see table 3). Seven
studies modified the aids (see table 2), but in four of these studies
(Beverley, Bath, & Booth, 2004; Brettle,
2007; Brettle et al, 2011; Gray, Sutton, & Treadway, 2012) what modifications were made is
unclear. Four studies reported using bespoke (i.e. custom-made) criteria to
assess the quality of the included studies.
Of the nine studies
that referenced aids other than published tools, six reported the criteria on
which their quality assessment would be undertaken (Brettle
et al., 2011; Gagnon et al., 2010; Golder & Loke,
2009; Crumley et al., 2005; Sommestad,
Hallberg, Lundholm, & Bengtsson, 2014; Weightman
& Williamson, 2005). Three studies (Beverley et al., 2004; Booth, Carroll, Papaioannou, Sutton, & Wong, 2009 [for survey tool]); Gray et al., 2012)
did not report on the criteria used to assess the quality of their included
studies.
Table 1
Characteristics of
Included Studies
|
Characteristics
|
Number of reviews
|
|
Journal
Health Information and Libraries Journal
Journal of the Medical Library Association
Information Research
Journal of Academic Librarianship
Journal of Documentation
Reference Services Review
BMC Medical Research Methodology
Evidence Based Library and Information
Practice
Health and Social Care in the Community
Health Informatics Journal
Information Development
Information Management and Computer Security
International Journal of Medical Informatics
Journal of Information Science
Journal of the American Medical Informatics
Society
Journal of the American Society for
Information Science and Technology
Journal of Database Management
Library and Information Science Research
LIBRES: Library and Information Science
Research Electronic Journal
Mousaion
Reference and User Services Quarterly
|
14
4
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
|
|
Date published
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
|
2
3
3
4
4
3
2
6
3
2
5
4
|
|
Number of systematic review authors
1
2
3
4
5
8
10
13
|
9
15
8
2
3
1
1
1
|
|
Type of systematic review (synthesis)
Narrative
Qualitative (e.g. meta-synthesis,
meta-ethnography, framework)
Meta-analysis with narrative
|
30
7
3
|
Of the 14 published tools
specifically designed to assess the quality of research, nine were checklists,
three were scales and one was domain-based. No single tool was preferred over
others, with the Glynn (2006), CRiSTAL checklist
(Booth, 2000; Booth & Brice, 2004), HCPRDU Evaluation Tools (Long et al.,
2002a, 2002b, 2002c) and the Cochrane Effective Practice and Organisation of Care (2010) each being cited by three
studies. Only one study (Catalano, 2013) reported using a validated tool.
Incorporation of
Quality Assessment in Systematic Reviews
All studies
incorporating quality assessment presented a narrative synthesis of their
results. Twelve studies (Beverley et al., 2004; Booth et al., 2009; Brettle et al., 2011, Brettle,
2007; Catalano, 2013; Divall et al., 2013; Gagnon et
al., 2010; Golder & Loke, 2009; Ndabarora et al., 2014; Perrier et al., 2014; Sommestad et al., 2014; Weightman
& Williamson, 2005) also tabulated their quality assessment. Table 4
outlines the appraisal of the included studies according to the incorporation
of quality assessment criteria in the PRISMA reporting guidelines and AMSTAR
quality assessment tool.
PRISMA Assessment
Of the 26 studies
reporting a planned quality assessment in the methods, all but three did not
report on how the data would be used in the synthesis, thereby failing to meet the first PRISMA quality criteria (item
#12): “Describe methods used
for assessing risk of bias of individual studies (including specification of
whether this was done at the study level or outcome level) and how this information is to be used in any
data synthesis [emphasis added]” (Moher et al., 2009, p. W-67). Only one study (Sommestad
et al., 2014) reported an assessment of publication bias a priori in the methods (PRISMA item #15).
When reporting the
results of the quality assessment, only five studies adequately “presented data
on the risk of bias of each study and, if available, any outcome level
assessment” (PRISMA item #19; Moher et al., 2009, p. 266). Twenty-one studies were classed as inadequate. Of
these, 18 studies (Bergman & Holden, 2010; Beverley et al., 2004; Booth et
al., 2009; Brennan, Mattick, & Ellis, 2011; Brettle, 2003, 2007; Brown, 2008; Catalano, 2013; Crumley et al., 2005; Golder & Loke,
2010; Kelly & Sugioto, 2013; Koufogiannakis &
Wiebe, 2006; Manning Fiegen, 2010; Ndabarora et al., 2014; Perrier et al., 2014; Sommestad et al., 2014; Weightman
& Williamson, 2005; Winning & Beverley, 2003) reported summary data
only. Thus, the systematic reviews were either unclear about which of the
included studies met each of the criteria assessed, or the systematic reviews
were unclear about what criteria was used to assess the included studies. Two
studies reported only criteria relating to the validity and reliability of the
outcome tool (Ankem, 2005; Ankem,
2006a) while one study failed to report on all studies in the quality
assessment (Gray et al., 2012). The remaining 14
studies were assessed as unclear as they did not report on quality assessment
of the included studies in the results section of their review.
Four studies
“presented results of any assessment of risk of bias across studies” (PRISMA
item #22; Moher et al., 2009, p. W-67). Two studies (Brennan et al., 2011;
Perrier et al., 2014) assessed selective reporting; the other two (Divall et al., 2013; Sommestad et
al., 2014) assessed publication bias. Three of the studies (Brennan et al.,
2011; Divall et al., 2013; Perrier et al., 2014)
presented a descriptive analysis while Sommestad et
al. (2014) presented an analytical analysis using a funnel plot.
Table 2
Quality Assessment
(QA) in LIS Systematic Reviews
|
Study
|
QA reported in methods
|
Authors defined QA
|
No. of authors undertaking QA
|
Number of QA tools used
|
Model of QA (tools only)
|
Published, modified, or bespoke
|
QA reported as an
inclusion criteria
|
|
Ankem (2005)
|
✓
|
NR
|
1
|
NR
|
NR
|
NR
|
✓
|
|
Ankem (2006a)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Bergman & Holden
(2010)
|
✓
|
NR
|
NR
|
1
|
Checklist
|
Published
|
NR
|
|
Beverley et al.
(2004)
|
✓
|
NR
|
NR
|
3
|
Checklist, Scale
|
1 Published
2 Modified (unclear)
|
NR
|
|
Booth et al. (2009)
|
✓
|
✓
|
NR
|
3
|
Checklist
|
Published
|
NR
|
|
Brennan et al.
(2011)
|
✓
|
NR
|
NR
|
1
|
Domain
|
Published
|
NR
|
|
Brettle (2003)
|
✓
|
NR
|
1
|
1
|
Checklist
|
Published
|
NR
|
|
Brettle et al. (2011)
|
✓
|
NR
|
8a
|
2
|
Checklist
|
Modified (unclear)
|
NR
|
|
Brettle (2007)
|
✓
|
NR
|
1
|
2
|
Checklist
|
Modified (unclear)
|
NR
|
|
Brown (2008)
|
✓
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Burda & Teuteberg
(2013)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
✓
|
|
Catalano (2013)
|
✓
|
NR
|
1b
|
1
|
Checklist
|
Published
|
NR
|
|
Childs et al. (2005)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Cooper and Crum
(2013)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Crumley et al. (2005)
|
✓
|
✓
|
2
|
1
|
Unclear
|
Bespoke (other
journal article)
|
NR
|
|
Divall et al. (2013)
|
✓
|
✓
|
NR
|
1
|
Domain
|
Published
|
✓
|
|
Du Preez (2007)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Duggan & Banwell (2004)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Fanner &
Urquhart (2008)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Gagnon et al. (2010)
|
✓
|
✓
|
2
|
NR
|
Unclear
|
Bespoke (other
journal articles)
|
✓
|
|
Genero et al. (2011)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Golder & Loke (2010)
|
✓
|
✓
|
NR
|
1
|
Unclear
|
Bespoke (own
criteria)
|
NR
|
|
Golder & Loke (2009)
|
✓
|
✓
|
NR
|
2
|
Unclear
|
Modified (journal
articles, web resource)
|
NR
|
|
Grant (2007)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Gray et al. (2012)
|
✓
|
NR
|
3c
|
3/4d
|
Unclear
|
Modified (journal
articles)
|
NR
|
|
Joshi & Trout
(2014)
|
✓
|
✓
|
2
|
NR
|
NR
|
NR
|
NR
|
|
Kelly & Sugimoto
(2013)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Koufogiannakis and
Wiebe (2006)
|
✓
|
✓
|
1
|
1
|
Unclear
|
Published (other
journal article)
|
NR
|
|
Manning Fiegen (2010)
|
✓
|
✓
|
6a
|
1
|
Checklist
|
Published
|
NR
|
|
Matteson et al.
(2011)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Ndabarora et al. (2014)
|
✓
|
✓
|
NR
|
NR
|
NR
|
NR
|
✓
|
|
Perrier et al.
(2014)
|
✓
|
✓
|
2
|
2
|
Scale
|
Published
|
✓
|
|
Phelps & Campbell
(2013)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Rankin et al. (2008)
|
✓
|
NR
|
NR
|
1
|
Checklist
|
Published
|
NR
|
|
Sommestad et al. (2014)
|
✓
|
NR
|
NR
|
1
|
Unclear
|
Modified (other
journal article)
|
✓
|
|
Urquhart &
Yeoman (2010)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
|
Wagner & Byrd
(2004)
|
NR
|
NR
|
NR
|
NR
|
NR
|
NR
|
✓
|
|
Weightman & Williamson (2005)
|
✓
|
NR
|
2
|
1
|
Unclear
|
Bespoke (books)
|
✓
|
|
Winning &
Beverley (2003)
|
✓
|
NR
|
NR
|
1
|
Checklist
|
Published
|
NR
|
|
Zhang et al. (2007)
|
✓
|
✓
|
2
|
1
|
Scale
|
Modified
|
NR
|
Note: NR = not
reported. Bespoke = custom-made.
aTwo reviewers appraised each paper
bOne study that was appraised by two reviewers
cThree reviewers appraised three included studies
collectively and then they appraised the rest individually. Two reviewers
checked all appraisals for accuracy.
dAuthors report using three tools but they reference
four.
Table 3
Bibliography of
Quality Assessment Tools and Resources Used in LIS Systematic Reviews
|
Quality assessment tools
|
Number of studies
(and the studies) using the tool
|
|
Checklists
CRiSTAL
Booth, A. (2000).
Research. Health Information &
Libraries Journal, 17(4), 232-235.
Booth, A., &
Brice, A. (2004). Appraising the evidence. In Booth & Brice (Eds.), Evidence-based practice for information
professionals A handbook. London, UK: Facet Publishing.
Glynn, L. (2006). A
critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387-99.
HCPRDU Evaluation Tools
Long, A. F., Godfrey, M., Randall, T., Brettle,
A., & Grant, M. J. (2002a). HCPRDU evaluation tool for qualitative
studies. Leeds: University of Leeds, Nuffield Institute for Health.
Long, A. F., Godfrey, M., Randall, T., Brettle,
A., & Grant, M. J. (2002b). HCPRDU evaluation tool for quantitative
studies. Leeds: University of Leeds, Nuffield Institute for Health.
Long, A. F., Godfrey, M., Randall, T., Brettle,
A., & Grant, M. J. (2002c). HCPRDU evaluation tool for mixed methods
studies. Leeds: University of Leeds, Nuffield Institute for Health.
Koufogianniakis, D., Booth, A., & Brettle,
A. (2006). ReLIANT: Readers guide to the literature
on interventions addressing the need for education and training. Library and Information Research, 30,
44-51.
Kmet L. M., Lee R. C., & Cook L. S. (2004). Standard quality assessment criteria for evaluating primary research
papers from a variety of fields. Edmonton: Alberta Heritage Foundation
for Medical Research (AHFMR). HTA Initiative #13.
Atkins, C., &
Sampson, J. (2002). Critical appraisal guidelines for single case study
research. Proceedings of the Xth European Conference on Information Systems (ECIS),
Gdansk, Poland, 6-8 June 2002.
Morrison, J. M., Sullivan, F., Murray, E. & Jolly, B. (1999).
Evidence-based education: development of an instrument to critically appraise
reports of educational interventions.
Medical Education, 33,
890-893.
|
3
(Beverley et al, 2004; Rankin et al., 2008;
Winning & Beverley, 2003)
3
(Bergman et al., 2010; Catalano, 2013;
Manning Fiegen, 2010)
3
(Brettle, 2003a,
2007; Brettle et al., 2011)
1
(Brettle, 2007)
1
(Booth et al., 2009)
1
(Booth et al., 2009)
1
(Koufogiannakis & Wiebe, 2006)
|
|
Scales
Downs, S. H. &
Black, N. (1998). The feasibility of creating a checklist for the assessment
of the methodological quality both of randomised and non-randomised studies of
health care -interventions, Journal of
Epidemiology and Community Health, 52(6),
377-384.
Nelson
E.A. (1999). Critical appraisal 8: Questions for surveys. Nursing Times
Learning Curve, 3(8), 5-7.
Wells G., Shea B. J., O’Connell, D., Peterson, D., Welch, V., Losos, M., & Tugwell, P. (n.d.). The Newcastle-Ottawa Scale (NOS) for assessing the
quality of nonrandomised studies in meta-analyses. Available at
http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
|
1
(Zhang et al., 2007)
2
(Beverley et al., 2004)
1
(Perrier et al.,
2014)
|
|
Domain-based
Cochrane
Effective Practice and Organisation of Care (EPOC) Group. (2010). Draft EPOC
methods paper: Including interrupted time series (ITS) designs in a EPOC review.
|
3
(Brennan
et al, 2011; Divall et
al., 2013; Perrier et al., 2014)
|
|
References to other publications
|
Study referencing
the publication
|
|
Books
Burton, D. (Ed.)
(2000). Research training for social
scientists. London: Sage Publications.
de Vaus, D. A.
(1991). Surveys in social research, 3rd
edition. London: Allen & Unwin.
Gomm, R., Needham, G., & Bullman,
A. (2000). Evaluating research in
health and social care. London: Sage Publications.
Journal articles
Boynton, P. M.
(2004). Hands on guide to questionnaire research: selecting, designing and
developing your questionnaire. British
Medical Journal, 328, 1312.
Jamtvedt, G., Young, J. M., Kristoffersen, D. T., O’Brien, M.,A.,
& Oxman, A. D. (2006). Audit and feedback:
effects on professional practice and health care outcomes. Cochrane Database Systematic Reviews.
http://dx.doi.org/10.1002/14651858.CD000259.pub2
Lijmer J. C., Mol, B. W., Heisterkamp, S., Bonsel, G. J., Prins, M. H.,
van der Meulen J. H., Bossuyt,
P. M. (1999). Empirical evidence of design-related bias
in studies of diagnostic tests. Journal of the American Medical
Association, 282, 1061-1066.
Malhotra, M. K.,
& Grover, V. (1998). An assessment of survey research in POM: from
constructs to theory. Journal of
Operations Management, 16(4), 407-425.
McDonald, S., Crumley, E., Eisinga, A., &
Villanueva, E. (2007). Search strategies to identify reports of randomized
trials in MEDLINE (protocol). Cochrane
Database of Systematic Reviews. http://dx.doi.org/10.1002/14651858.MR000018.pub2
Mi, M., & Gilbert, C., M. (2007). Needs
assessment: Prerequisite for service excellence. Journal of Hospital Librarianship, 7, 31-52.
Polgar, S., & Thomas, S. A. (1995). Critical
evaluation of published research. In Introduction to research in the health sciences, 3rd
edition (343–355). Melbourne: Churchill Livingstone.
Robinson, L., & Bawden, D. (2007). Evaluation of outreach services for
primary care and mental health; assessing the impact. Health Information and Libraries Journal, 24, 57-66.
Urquhart, C. J.,
& Hepworth, J. B. (1996). Comparing and using assessment of the value of
information to clinical decision-making. Bulletin
of the Medical Library Association,
84, 482-489.
Weightman, A. &
Williamson, J. (2005). The value and impact of information provided through
library services for patient care: a systematic review. Health Information and Libraries Journal, 22, 4-25.
Yeoman, A. J.,
Cooper, J. M., Urquhart, C. J., & Tyler, A. (2003). The
management of health library outreach services: evaluation and reflection on
lessons learned on the VIVOS project. Journal
of the Medical Library Association, 91,
426-33.
Web resource
InterTasc Information
Specialists’ Sub-Group Search Filter Resource. (2008). ISSG search filter resource. Retrieved from
https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home
|
(Weightman
& Williamson, 2005)
(Weightman
& Williamson, 2005)
(Beverley et al.,
2004)
(Booth et al., 2009)
(Gagnon et al., 2010)
(Crumley
et al., 2004)
(Sommestad
et al., 2014)
(Golder & Loke, 2009)
(Gray
et al., 2012)
(Gray
et al., 2012)
(Gray
et al., 2012)
(Gray
et al., 2012)
(Brettle
et al., 2011)
(Gray
et al., 2012)
(Golder & Loke, 2009)
|
aBrettle 2003 was based on earlier versions of the
HCPRDU Evaluation Tools
Table 4
PRISMA and AMSTAR
Assessment of Quality Criteria
|
STUDY
|
PRISMA
|
AMSTAR
|
|
|
PRISMA #12a
|
PRISMA #15b
|
PRISMA #19c
|
PRISMA #22d
|
AMSTAR #7e
|
AMSTAR #8f
|
AMSTAR # 10g
|
|
Ankem (2005)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Ankem (2006a)
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Unclear
|
|
Bergman & Holden
(2010)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Beverley et al.
(2004)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Adequate
|
Inadequate
|
Unclear
|
|
Booth et al. (2009)
|
Adequate
|
Unclear
|
Inadequate
|
Unclear
|
Adequate
|
Unclear
|
Unclear
|
|
Brennan et al.
(2011)
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Inadequate
|
Adequate
|
Unclear
|
|
Brettle (2003)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Unclear
|
|
Brettle et al. (2011)
|
Inadequate
|
Unclear
|
Adequate
|
Unclear
|
Adequate
|
Adequate
|
Unclear
|
|
Brettle (2007)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Unclear
|
|
Brown (2008)
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Unclear
|
|
Burda & Teuteberg
(2013)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Catalano (2013)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Adequate
|
Inadequate
|
Unclear
|
|
Childs et al. (2005)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Cooper and Crum
(2013)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Crumley et al. (2005)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Divall et al. (2013)
|
Inadequate
|
Unclear
|
Adequate
|
Adequate
|
Adequate
|
Inadequate
|
Unclear
|
|
Du Preez (2007)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
|
Duggan & Banwell (2004)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Fanner &
Urquhart (2008)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Gagnon et al (2010)
|
Inadequate
|
Unclear
|
Adequate
|
Unclear
|
Adequate
|
Adequate
|
Unclear
|
|
Genero et al. (2011)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Golder & Loke (2010)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Golder & Loke (2009)
|
Inadequate
|
Unclear
|
Adequate
|
Unclear
|
Adequate
|
Unclear
|
Unclear
|
|
Grant (2007)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Gray et al. (2012)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Unclear
|
|
Joshi & Trout
(2014)
|
Inadequate
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
|
Kelly & Sugimoto
(2013)
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Koufogiannakis and
Wiebe (2006)
|
Adequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Unclear
|
|
Manning Fiegen (2010)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Unclear
|
|
Matteson et al.
(2011)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Unclear
|
Unclear
|
|
Ndabarora et al. (2014)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Perrier et al.
(2014)
|
Inadequate
|
Unclear
|
Inadequate
|
Adequate
|
Unclear
|
Inadequate
|
Unclear
|
|
Phelps &
Campbell (2013)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Rankin et al.,
(2008)
|
Inadequate
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Sommestad et al. (2014)
|
Adequate
|
Adequate
|
Inadequate
|
Adequate
|
Adequate
|
Adequate
|
Adequate
|
|
Urquhart &
Yeoman (2010)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
|
Wagner & Byrd
(2004)
|
Unclear
|
Unclear
|
Unclear
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Weightman & Williamson (2005)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Adequate
|
Inadequate
|
Unclear
|
|
Winning & Beverley
(2003)
|
Inadequate
|
Unclear
|
Inadequate
|
Unclear
|
Inadequate
|
Inadequate
|
Unclear
|
|
Zhang et al. (2007)
|
Inadequate
|
Unclear
|
Adequate
|
Unclear
|
Adequate
|
Adequate
|
Unclear
|
aRisk of bias in individual studies; Describe methods used
for assessing risk of bias of individual studies (including specification of
whether this was done at the study level or outcome level) and how this
information is to be used in any data synthesis
bRisk of bias across studies; Specify any assessment of
risk of bias that may affect the cumulative evidence (e.g., publication bias,
selective reporting within studies)
cRisk of bias within studies; Present data on risk of bias
of each study and, if available, any outcome level assessment
dRisk of bias across studies; Present results of any
assessment of risk of bias across studies
eWas the scientific quality of the included studies
assessed and documented?
fWas the scientific quality of the included studies used
appropriately in formulating conclusions?
gWas the likelihood of publication bias assessed?
AMSTAR Assessment
Only one quarter (10
of 40) of the systematic reviews included in this analysis adequately assessed
and documented the scientific quality of the included studies (AMSTAR item #7; Shea et al., 2007). Twenty studies were assessed as
inadequate because, although quality assessment was documented, 14 studies (Ankem, 2005; Bergman & Holden, 2010; Brennan et al.,
2011; Brettle, 2003, 2007; Brown, 2008; Crumley et al., 2005; Golder & Loke,
2010; Gray et al., 2012; Koufogiannakis and Wiebe 2006; Manning Fiegen, 2010; Ndabarora et al.,
2014; Rankin 2008; Winning & Beverley, 2003) failed to report “some kind of result for each study”
(AMSTAR criteria #7; Shea et al., 2007, p. 5) and six
studies (Ankem, 2006a; Fanner & Urquhart, 2008; Genero, 2011;, Kelly & Sugimoto, 2013; Matteson et al.,
2011; Wagner & Byrd, 2004) did not report their quality assessment methods
a priori. In one study (Perrier et
al., 2014), determining whether quality assessment was documented and assessed
in accordance with the AMSTAR item #7 was not possible because the link to the
online supplementary table detailing the quality assessment was unavailable.
The remaining eight studies failed to report on quality assessment at all;
therefore, whether they met AMSTAR item #7 was unclear.
In assessing the
included studies against AMSTAR item #8, which reads
Was the scientific quality of the included studies used
appropriately in formulating conclusions? The results of the methodological
rigor and scientific quality should be considered in the analysis and the
conclusions of the review, and explicitly stated in formulating
recommendations. (Shea et al., 2007, p. 5).
Studies were classed
as adequate if they incorporated how the quality of the included studies
impacted on the validity of the systematic review findings in both the analysis
and conclusion and also considered quality issues in their recommendations. The
studies were classed as inadequate if they addressed the quality of the
included studies in only one of these sections. They were classed as unclear if
the studies did not report the quality of the included studies anywhere.
Using the above
criteria, only 9 of the 40 included systematic reviews adequately incorporated
quality assessment in the analysis, conclusions, and recommendations. Just one
study (Sommestad et al., 2014) met the final AMSTAR
quality criteria—assessing the likelihood of publication bias (item #10)—by
providing a funnel plot.
Five studies (Du Preez, 2007; Fanner & Urquhart,
2008; Genero et al., 2011; Kelly & Sugimoto,
2013; Wagner & Byrd, 2004) incorporated some discussion of the quality of
the included studies without explicitly reporting that quality assessment would
be undertaken in the review methods.
Discussion
The results section
demonstrates great variation on the breadth, depth, and transparency of the
quality assessment process in LIS systematic reviews. Nearly one third of the
LIS systematic reviews included in this study did not report on quality
assessment in the methods. Less than one quarter adequately incorporated quality assessment
in the analysis, conclusions, and recommendations. Quality assessment is an essential part of the
systematic review process (Moher et al., 2009; Higgins et al., 2011; CRD,
2009). Without it, a systematic review loses one of the advantages it has over
traditional literature reviews and is in danger of conforming to the old adage
of “garbage in, garbage out” (Yuan and Hunt, 2009), where ignoring the impact
of methodological quality may result in misleading conclusions (Verhagen, de Vet, de Bie, Boers, &
van den Brandt, 2001; Mallen, Peat, & Croft, 2006).
In particular, a lack
of consistency in the understanding and application of the systematic review
terminology appears to exist not only between LIS authors but also across
studies published in the same journal. For example, the majority (14) of LIS
systematic reviews were published in Health
Information and Libraries Journal. Of these, four reported only one author
(Brettle, 2003, 2007; Brown, 2008, Grant, 2007), and
three did not assess the quality of the included studies (Childs, Blenkinsopp, Hall, A.
& Walton, 2005, Fanner &
Urquhart, 2008; Grant, 2007).
The question is, does
it matter if authors do not consider quality assessment in the analysis of a
systematic review? Although no empirical evidence within the LIS domain
suggests that the quality of the studies impacts on the validity of findings in
LIS-related systematic reviews, there is evidence that the quality of the
included studies can yield differences in review results (Voss & Rehfuess, 2013). Although guidance on the reporting of
qualitative synthesis includes four items on the appraisal of the included
studies (Tong, Flemming, McInnes, Oliver,
& Craig, 2012), the debate on
whether to undertake quality assessment in qualitative systematic reviews is
ongoing with insufficient evidence to support the inclusion or exclusion of
quality assessment (Noyes et al., 2015).
Only nine of the 26
systematic reviews that undertook some form of quality assessment incorporated
considerations of how the quality of the included studies impacted on the
validity of the review findings in the analysis, conclusion, and
recommendations. Ignoring the extent to which the quality of the included
studies may impact on the validity of
the review findings, undertaking quality assessment in isolation makes the act
of quality assessment within the systematic review a rather futile exercise (de
Craen, van Vliet, & Helmerhorst, 2005). The fact that LIS systematic
reviewers fail to incorporate how the quality of the included studies impacts
on the overall review findings is not surprising given that similar studies in
the field of health and medicine have shown only slightly better results (Katikireddi, Egan, & Petticrew, 2015; de Craen et
al., 2005; Moher et al., 1999; Hayden, Côté, & Bombardier, 2006). The findings of this study agree with
Katikireddi et al. when they state that systematic
review conclusions “are frequently uninformed by the critical appraisal
process, even when conducted” (2015, p. 189).
Conversely, a number
of systematic reviews (Du Preez, 2007; Fanner & Urquhart,
2008; Genero et al., 2011; Kelly & Sugimoto,
2013; Wagner & Byrd, 2004) raised the issue of the quality of the included studies in their
discussion; however, their comments may not be valid since it was unclear how
the quality of the studies was assessed. Similarly, four studies (Brennan et al, 2011; Divall et al., 2013; Perrier et al., 2014; Sommestad et al., 2014) reported on publication or selection bias, but only
one outlined their methods a priori (Sommestad et al., 2014).
De Craen
et al. (2005) put forward a number of theories as to why systematic reviewers
may not incorporate quality assessment into the analysis. Firstly, reviewers
may not know that quality assessment should be considered in the analysis, or
secondly, they simply may not know how
to incorporate the quality assessment into the analysis. Conversely, it may be
that the reviewers’ focus is more on the tools used to assess quality, many of
which are designed to assess the quality of individual
studies, rather than across a group of
studies. This raises important questions over the nature of the guidance used
by LIS reviewers when undertaking a systematic review. A quick look at the
guidance referred to in the systematic reviews in this study reveals that LIS
reviewers follow a range of guidance when undertaking a systematic review, from
the more formal (e.g., Higgins & Green, 2011; CRD, 2009) to single journal
articles providing a rather short, introductory overview of the systematic
review.
While there are numerous
texts explaining how to conduct a systematic review, they are largely written
from the perspective of the healthcare professional rather than the LIS
professional (e.g. Booth, Papaioannou, & Sutton, 2012; CRD, 2009; Higgins & Green, 2011).
Currently there is no comprehensive guidance with a focus on the different
approaches to evidence synthesis written purely from a LIS perspective with
relevant guided examples of how to undertake and incorporate quality assessment
in the analysis. The findings of this study appear to demonstrate a need for
such a resource or series of guides. However, even when comprehensive guidance
is available, such as in the healthcare domain, the findings of previous
methodology studies examining the incorporation of quality assessment in
systematic reviews (Hayden et al., 2006; Katikireddi
et al., 2015) seem to suggest that reviewers still fail to address how the
quality of included studies impacts on the validity of the review findings.
De Craen
et al. (2005) also suggest that reviewers may see the incorporation of quality
assessment in the analysis as a “cumbersome procedure” which might “further
complicate the interpretation of its results” (p. 312). It is certainly the
case that the heterogeneous nature of the LIS evidence base requires LIS
reviewers to consider the quality of studies across diverse research designs.
This adds another level of complexity to the quality assessment process since
different biases may arise according to the type of research design, which
makes comparisons across studies more difficult. Furthermore, quality
assessment is something that is out of the comfort zone of many librarians (Maden-Jenkins, 2011).
Critical to the understanding of how quality impacts on
the review findings is the reviewers’ definition of quality. Four definitions
of quality were identified in LIS systematic reviews: reporting quality, study
design, methodological quality (internal and external validity), and risk of
bias (internal validity). While an assessment of bias in research does rely on
the quality of the reporting, assessing the quality of the reporting can become
more of a descriptive exercise in recording whether or not methods were
reported, rather than assessing whether the methods were adequately conducted
in order to reduce bias. Similarly, basing quality assessment on study design
may lead reviewers to base quality on the level of evidence rather than the
process used to conduct the study, which ignores the possibility that high
levels of evidence, such as systematic reviews or randomized controlled trials,
may have been poorly conducted and therefore susceptible to bias.
Part of this problem may be that quality assessment
tools that purport to assess methodological quality are, on further
examination, actually assessing the reporting quality. The JADAD tool (Jadad et al., 1996) is a prime example of this where
reviewers are asked to assess whether the study was described as a double-blinded randomized controlled trial. Even the
criteria used in AMSTAR to critique the approach to quality assessment in
systematic reviews goes no further than to address whether or not the methods
were reported a priori. Reviewers,
therefore, should critique their own approach to quality assessment to ensure
that the criteria or tool they select for quality assessment is appropriate and
fit for purpose.
For those systematic
reviews in this study that do report on quality assessment in the methods,
there is need for greater transparency in the reporting process. This can be a
fairly simple process of tabulating the quality assessment in tables or
figures, such as in Cochrane reviews. Reporting on the quality assessment items
for each study allows the reader to see exactly on what criteria (methodology,
reporting, etc.) judgments of quality were made, while at the same time making
it easier for reviewers to judge the overall quality of the evidence base.
Identifying the type
of tool and resources LIS reviewers used to assess the quality of the evidence
was not straightforward. The aids identified went beyond the use of tools
developed specifically for quality assessment. The large number of different
quality assessment tools identified reflects not only the disparate nature of
the LIS evidence base (Brettle, 2009), but also a
lack of consensus around criteria on which to assess the quality of LIS
research. Given the diverse nature of the LIS evidence base and the multiple
study designs often incorporated into LIS reviews (see table 1), quality assessment
tools with a more generic focus on qualitative, quantitative, or mixed methods
focus rather than a study design focus (e.g. randomized controlled trial) may
help reviewers compare and contrast the quality of the included studies more
easily. LIS reviewers may wish to look at how reviews incorporating a wide
variety of study designs approach quality assessment (e.g. The Campbell
Collaboration).
Due to the broad
nature of some of the AMSTAR and PRISMA criteria, it was sometimes difficult to
interpret the criteria and make a clear judgment on some of the quality items
assessed. For example, AMSTAR item #8 asks “Was the scientific quality of the included
studies used appropriately in formulating conclusions?” (Shea
et al., 2007, p. 5). The accompanying notes suggest that “The results of the
methodological rigor and scientific quality should be considered in the
analysis and the conclusions of the review, and explicitly stated in
formulating recommendations” (Shea et al., 2007, p.
5). For example, some studies reported on the impact of quality assessment on
the review findings in the analysis but not the conclusions, while others
reported recommendations for improving the quality of future research but
failed to assess the impact the quality of the included studies had on the
review findings. The criteria also lacked transparency in assessing whether the
tools and approaches to quality assessment were appropriate.
For those undertaking
LIS systematic reviews, consideration therefore should be given to the PRISMA
and AMSTAR criteria (box 2) for incorporating considerations of quality
assessment in systematic reviews, specifically how the quality of the included
studies may impact on the validity of the overall review findings. In addition,
reviewers should ensure that whatever criteria or tool they use for quality
assessment is fit for purpose. In other words, reviewers should critique their
chosen set of criteria or tool to ensure it reflects the purpose of the quality
assessment (e.g. methodological quality versus reporting quality). Given that
tools aiming to assess methodological quality often, on further examination,
are found to actually assess reporting quality, further research on the
appropriateness of tools and criteria selected to quality assessment in LIS
reviews is warranted. Further research should also examine what criteria are
necessary to adequately assess the quality of studies included in LIS
systematic reviews. Above all, there is a need for tailored LIS systematic
review guidance with accompanying exemplar case studies of LIS systematic
reviews.
Strengths and Limitations
One reviewer of this
study extracted data from all included studies. One of the reviewers (MM) also
co-authored one of the included studies (Brettle et
al., 2011); therefore, a second reviewer (EK) checked the data extraction for
accuracy. While we used an existing resource that listed published LIS
systematic reviews, it is possible that other published LIS systematic reviews
were not listed on the wiki. We included only studies that reported themselves
as being a systematic review. Other studies may have followed systematic review
principles but were not explicit in labelling themselves as such. No attempt
was made to contact the authors of the included studies for further
clarification. This study did not seek to critique the reviewers
choice of quality assessment tool but rather to identify the tools used and the
approach for incorporating considerations of quality assessment in systematic
reviews. Finally, perhaps the major limitation in the way this study was
conducted is that 18 of the included LIS studies were published before the
PRISMA guidelines (Moher et al., 2009) were available, and 11 were published
before AMSTAR tool (Shea et al., 2007) was available.
However, even studies published after these dates show only a very small
improvement in meeting the criteria (see table 4) and there is still a long way
to go in improving quality assessment methods in LIS systematic reviews.
Conclusions
Although quality
assessment of included studies is an integral part of the systematic review
methodology, the extent to which it is documented and undertaken in LIS
systematic reviews varies widely. The results of this study demonstrate a need
for greater clarity, definition, and understanding of the methodology and
concept of quality in the systematic review process, not only by LIS reviewers
but also by editors of journals who accept such studies for publication. Due to
the diverse nature of the LIS evidence base, work still needs to be done to
identify the best tools and approaches for incorporating considerations of
quality in LIS systematic reviews. What is clear from this analysis is that LIS
reviewers need to improve the robustness and transparency with which they
undertake and report quality assessment in systematic reviews. Above all, LIS
reviewers need to be explicit in coming to a conclusion on how the quality of
the included studies may impact on their review findings. In considering this,
LIS reviewers can therefore increase the validity of their systematic review.
Disclaimer: The views expressed are those of the author(s)
and not necessarily those of the NHS, the NIHR or the Department of Health.
References
Aabø, S. (2009). Libraries
and return on investment (ROI): A meta-analysis. New Library World, 110(7/8), 311-324.
http://dx.doi.org/10.1108/03074800910975142
Ankem, K. (2005). Types of
information needs among cancer patients: A systematic review. LIBRES: Library and Information Science
Research Electronic Journal, 15(2). Retrieved from
http://libres-ejournal.info/
Ankem, K. (2006a). Use of
information sources by cancer patients: Results of a systematic review of the
research literature. Information Research,
11(3). Retrieved from
https://dialnet.unirioja.es/servlet/revista?codigo=6956
Ankem, K. (2006b). Factors
influencing information needs among cancer patients: A meta-analysis. Library & Information Science Research,
28(1), 7-23. http://dx.doi.org/10.1016/j.lisr.2005.11.003
Bergman, E. M. L., & Holden, I. I. (2010).
User satisfaction with electronic reference: A systematic review. Reference Services Review, 38(3),
493-509. http://dx.doi.org/10.1108/00907321011084789
Beverley, C. A., Bath, P. A., & Booth, A.
(2004). Health information needs of visually impaired people: A systematic
review of the literature. Health &
Social Care in the Community, 12(1), 1-24. http://dx.doi.org/10.1111/j.1365-2524.2004.00460.x
Boland, A., Cherry, M.G., & Dickson, R.
(2013). Doing a systematic review. A
student’s guide. London: SAGE Publications Ltd.
Booth, A. (2000). Research. Health Information & Libraries Journal,
17(4), 232-235. http://dx.doi.org/10.1111/j.1471-1842.2000.00295.x
Booth, A. (2007). Who will appraise the
appraisers? —The paper, the instrument and the user. Health Information and Libraries Journal, 24(1), 72-76.
http://dx.doi.org/10.1111/j.1471-1842.2007.00703.x
Booth, A., & Brice, A. (2004). Appraising
the evidence. In Booth & Brice (Eds.), Evidence-based
practice for information professionals: A handbook (pp. 96-110). London,
UK: Facet Publishing.
Booth, A., Carroll, C., Papaioannou,
D., Sutton, A., & Wong, R. (2009). Applying findings from a systematic
review of workplace-based e-learning: Implications for health information
professionals. Health Information and
Libraries Journal, 26(1), 4-21.
http://dx.doi.org/10.1111/j.1471-1842.2008.00834.x
Booth, A., Papaioannou,
D., & Sutton, A. (2012). Systematic
approaches to a successful literature review. London: SAGE Publications
Ltd.
Brennan, N., Mattick,
K., & Ellis, T. (2011). The map of medicine: A review of evidence for its
impact on healthcare. Health Information
and Libraries Journal, 28(2), 93-100.
http://dx.doi.org/10.1111/j.1471-1842.2011.00940.x
Brettle, A. (2003).
Information skills training: A systematic review of the literature. Health Information and Libraries Journal, 20(Suppl. 1), 3-9.
http://dx.doi.org/10.1046/j.1365-2532.20.s1.3.x
Brettle, A. (2007).
Evaluating information skills training in health libraries: A systematic
review. Health Information and Libraries
Journal, 24(Suppl. 1), 18-37.
Brettle, A. (2009) Systematic
reviews and evidence based library and information practice. Evidence Based Library and Information
Practice, 4(1), 43-50. Retrieved from
https://ejournals.library.ualberta.ca/index.php/EBLIP/index
Brettle, A., Maden-Jenkins, M., Anderson, L., McNally, R., Pratchett,
T., Tancock, J., Thornton, D., & Webb, A. (2011).
Evaluating clinical librarian services: A systematic review. Health Information and Libraries Journal, 28(1),
3-22. http://dx.doi.org/10.1111/j.1471-1842.2010.00925.x
Brown, C. (2008). The information trail of the
'Freshman 15'—A systematic review of a health myth within the research and
popular literature. Health Information
and Libraries Journal, 25(1), 1-12.
http://dx.doi.org/10.1111/j.1471-1842.2007.00762.x
Burda, D., & Teuteberg, F. (2013). Sustaining accessibility of
information through digital preservation: A literature review. Journal of Information Science, 39(4),
442-458. http://dx.doi.org/10.1177/0165551513480107
Catalano, A. J. (2013). Patterns of graduate
students' information seeking behavior: A meta
synthesis of the literature. Journal of
Documentation, 69(2). http://dx.doi.org/10.1108/00220411311300066
Centre for Reviews and Dissemination (CRD).
(2009). Systematic reviews: CRD's
guidance for undertaking reviews in health care. York, UK: Centre for
Reviews and Dissemination, University of York.
Childs, S., Blenkinsopp, E., Hall, A., & Walton,
G. (2005). Effective e-learning for health professionals and students—barriers
and their solutions. A systematic review of the literature—Findings from the HeXL project. Health
Information and Libraries Journal, 22(Suppl. 2), 20-32. http://dx.doi.org/10.1111/j.1470-3327.2005.00614.x
Cooper, I. D., & Crum, J. A. (2013). New
activities and changing roles of health sciences librarians: A systematic
review, 1990-2012. Journal of the Medical
Library Association, 101(4), 268-277. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/journals/93/
Crumley, E.T., Wiebe, N.,
Cramer, K., Klassen, T.P., & Hartling, L. (2005). Which resources should be
used to identify RCT/CCTs for systematic reviews: A systematic review. BMC Medical Research Methodology, 5:24.
http://dx.doi.org/10.1186/1471-2288-5-24
de Craen, A. J., van
Vliet, H. A., & Helmerhorst, F. M. (2005). An
analysis of systematic reviews indicated low incorporation of results from
clinical trial quality assessment. Journal
of Clinical Epidemiology, 58(3), 311-313.
http://dx.doi.org/10.1016/j.jclinepi.2004.07.002
Deeks, J. J., Dinnes, J., D’Amico, R., Sowden, A. J., Sakarovitch,
C., Song, F., Petticrew, M., & Altman, D. G.. (2003). Evaluating non-randomised controlled
intervention studies. Health Technology
Assessment, 7, 1-173. Retrieved from
http://researchonline.lshtm.ac.uk/id/eprint/8742
Divall, P., Camosso-Stefinovic, J., & Baker, R. (2013). The use of
personal digital assistants in clinical decision making by health care
professionals: A systematic review. Health
Informatics Journal, 19(1),
16-28. http://dx.doi.org/10.1177/1460458212446761
Du Preez, M. (2007).
Information needs and information-seeking behaviour of engineers: A systematic
review. Mousaion, 25(2), 72-94. Retrieved from http://www.unisa.ac.za/default.asp?Cmd=ViewContent&ContentID=2008
Duggan, F., & Banwell,
L. (2004). Constructing a model of effective information dissemination in a
crisis. Information Research, 9(3). Retrieved from
http://www.informationr.net/ir/
Eldredge, J. D. (2000). Evidence-based librarianship:
an overview. Bulletin of the Medical
Library Association, 88(4), 289-302. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/journals/72/
Fanner, D., & Urquhart, C. (2008). Bibliotherapy for mental health service users Part 1: A
systematic review. Health Information and
Libraries Journal, 25(4), 237-252. http://dx.doi.org/10.1111/j.1471-1842.2008.00821.x
Gagnon, M.P., Pluye,
P., Desmartis, M., Car, J., Pagliari,
C., Labrecque, M., Fremont, P., Gagnon, J., Njoya, M., & Legare, F. (2010). A systematic review of interventions promoting
clinical information retrieval technology (CIRT) adoption by healthcare
professionals. International Journal of
Medical Informatics, 79(10),
669-680. http://dx.doi.org/10.1016/j.ijmedinf.2010.07.004
Genero, M., Fernandez-Saez,
A. M., Nelson, H. J., Poels, G., & Piattini, M. (2011). Research review: A systematic
literature review on the quality of UML models. Journal of Database Management, 22(3), 46-70. http://dx.doi.org/10.4018/jdm.2011070103
Golder, S., & Loke,
Y. K. (2009). Search strategies to identify information on adverse effects: a
systematic review. Journal of the Medical
Library Association, 97(2), 84-92. http://dx.doi.org/10.3163/1536-5050.97.2.004
Golder, S., & Loke,
Y. K. (2010). Sources of information on adverse effects: A systematic review. Health Information and Libraries Journal, 27(3),
176-190. http://dx.doi.org/10.1111/j.1471-1842.2010.00901.x
Grant, M. J. (2007). The role of reflection in
the library and information sector: A systematic review. Health Information and Libraries Journal, 24(3), 155-166. http://dx.doi.org/10.1111/j.1471-1842.2007.00731.x
Grant, M., & Booth, A. (2009). A typology of
reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Review, 26(2),
91-108. http://dx.doi.org/10.1111/j.1471-1842.2009.00848.x
Gray, H., Sutton, G.,
& Treadway, V. (2012). Do quality improvement systems improve health
library services? A systematic review. Health
Information and Libraries Journal, 29(3), 180-196.
http://dx.doi.org/10.1111/j.1471-1842.2012.00996.x
Haug, J. D. (1997).
Physicians' preferences for information sources: a meta-analytic study, Bulletin of the Medical Library Association,
85(3), 223–32. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/journals/72/
Hayden, J. A., Côté, P. & Bombardier, C.
(2006). Evaluation of the
quality of prognostic studies in systematic reviews. Annals of Internal Medicine, 144(6), 427-437.
http://dx.doi.org/10.7326/0003-4819-144-6-200603210-00010
Higgins J. P. T., & Green, S. (Eds.).
(2011). Cochrane handbook for systematic reviews of interventions, version
5.3.0. The Cochrane Collaboration, 2011. Available from
http://handbook.cochrane.org/
Jadad A. R., Moore R. A.,
Carroll D., Jenkinson, C., Reynolds, D. J., Gavaghan,
D. J., & McQuay, H. J. (1996). Assessing the quality of reports of
randomized clinical trials: is blinding necessary? Controlled Clinical Trials, 17(1), 1-12.
http://dx.doi.org/10.1016/0197-2456(95)00134-4
Joshi, A, & Trout, K. (2014). The role of
health information kiosks in diverse settings: A systematic review. Health Information and Libraries Journal, 31(4), 254-273.
http://dx.doi.org/10.1111/hir.12081
Julien, C.-A., Leide,
J. E., & Bouthillier, F. (2008). Controlled user
evaluations of information visualization interfaces for text retrieval:
Literature review and meta-analysis. Journal
of the American Society for Information Science and Technology, 59(6),
1012-1024. http://dx.doi.org/10.1002/asi.20786
Jüni, P. Altman, D. G.,
& Egger, M. (2001). Assessing the quality of controlled clinical trials. British Medical Journal, 323(7303), 42-46. http://dx.doi.org/10.1136/bmj.323.7303.42
Katikireddi, S. V., Egan, M.,
& Petticrew, M. (2015). How do systematic reviews
incorporate risk of bias assessments into the synthesis of evidence? A
methodological study. Journal of
Epidemiology and Community Health 69(2), 189-195.
http://dx.doi.org/10.1136/jech-2014-204711
Kelly, D., & Sugimoto, C. R. (2013). A
systematic review of interactive information retrieval evaluation studies,
1967-2006. Journal of the American
Society for Information Science and Technology, 64(4), 745-770. http://dx.doi.org/10.1002/asi.22799
Koufogiannakis, D. (2012). The state of
systematic reviews in library and information Studies. Evidence Based Library and Information Practice, 7(2), 91-95.
Retrieved from https://ejournals.library.ualberta.ca/index.php/EBLIP/index
Koufogiannakis, D., Brettle,
A., Booth, A., Kloda, L., & Urquhart, C. (2015).
LIS systematic reviews. Retrieved from
http://lis-systematic-reviews.wikispaces.com/Welcome
Koufogiannakis, D., & Wiebe, N. (2006).
Effective methods for teaching information literacy skills to undergraduate
students: A systematic review and meta-analysis. Evidence Based Library and Information Practice, 1(3), 3-43. Retrieved from
https://ejournals.library.ualberta.ca/index.php/EBLIP/index
Long, A. F., Godfrey, M., Randall,
T., Brettle, A., & Grant, M.
J. (2002a). HCPRDU evaluation tool for qualitative studies.
Leeds: University of Leeds, Nuffield Institute for Health. Retrieved from
http://usir.salford.ac.uk/12970/
Long, A. F., Godfrey, M., Randall,
T., Brettle, A., & Grant, M.
J. (2002b). HCPRDU evaluation tool for quantitative studies.
Leeds: University of Leeds, Nuffield Institute for Health. Retrieved from
http://usir.salford.ac.uk/12969/
Long, A. F., Godfrey, M., Randall,
T., Brettle, A., & Grant, M.
J. (2002c). HCPRDU evaluation tool for mixed methods studies. Leeds: University of Leeds, Nuffield
Institute for Health. Retrieved from http://usir.salford.ac.uk/13070/
Maden-Jenkins, M. (2011).
Healthcare librarians and the delivery of critical appraisal training. Health Information and Libraries Journal, 28(1), 33-40.
http://dx.doi.org/10.1111/j.1471-1842.2010.00899.x
Manning Fiegen, A.
(2010). Systematic review of research methods: The case of business
instruction. Reference Services Review, 38(3), 385-397.
http://dx.doi.org/10.1108/00907321011070883
Mairs, K., McNeil, H., McLeod, J., Prorok, J. C., & Stolee, P.
(2013). Online strategies to facilitate health-related knowledge transfer: A
systematic search and review. Health
Information and Libraries Journal, 30(4), 261-277.
http://dx.doi.org/10.1111/hir.12048
Mallen, C., Peat, G., & Croft, P.
(2006). Quality assessment of observational studies is not commonplace in
systematic reviews. Journal of Clinical
Epidemiology, 59, 765-769. http://dx.doi.org/10.1016/j.jclinepi.2005.12.010
Matteson, M. L., Salamon, J., & Brewster, L. (2011). A systematic review
of research on live chat service. Reference
& User Services Quarterly, 51(2),
172-190. Retrieved from http://www.jstor.org/stable/refuseserq.51.2.172
Moher, D., Cook,
D. J., Jaded, A. R., Tugwell, P., Moher, M., Jones,
A., Pham, B., & Klassen, T. P. (1999). Assessing the quality of reports of
randomised controlled trials: implications for the conduct of meta-analyses. Health Technology Assessment, 3(12),
1-98. http://dx.doi.org/10.3310/hta3120
Moher, D., Liberati, A., Tetzlaff, J.,
Altman, D. G., & The PRISMA Group. (2009). Preferred reporting
items for systematic reviews and meta-analyses: The PRISMA statement. Annals
of Internal Medicine, 151(4), W65-94.
http://dx.doi.org/10.7326/0003-4819-151-4-200908180-00135
Ndabarora, E., Chipps, J. A., & Uys, L.
(2014). Systematic review of health data quality management and best practices
at community and district levels in LMIC. Information
Development, 30(2), 103-120.
http://dx.doi.org/10.1177/0266666913477430
Noyes, J., Hannes, K., Booth, A., Harris, J.,
Harden, A., Popay, J., Pearson, A., Cargo, M., &
Pantoja, T., on behalf of the Cochrane Qualitative and Implementation Methods
Group. Qualitative research and Cochrane
reviews. In J. P. T. Higgins & S. Green (Eds.), Cochrane handbook for
systematic reviews of Interventions Version 5.3.0 (updated October 2015). The
Cochrane Collaboration, 2015. Available from
http://qim.cochrane.org/supplemental-handbook-guidance
Perrier, L., Farrell, A., Ayala, A. P.,
Lightfoot, D., Kenny, T., Aaronson, E., Allee, N.,
Brigham, T., Connor, E., Constantinescu, T., Mullenbach,
J., Epstein, H. A. B., & Weiss, A. (2014). Effects of librarian-provided
services in healthcare settings: A systematic review. Journal of the American Medical Informatics Association, 21(6), 1118-1124.
http://dx.doi.org/10.1136/amiajnl-2014-002825
Phelps, S. F., & Campbell, N. (2012).
Commitment and trust in librarian-faculty relationships: A systematic review of
the literature. Journal of Academic
Librarianship, 38(1), 13-19. http://dx.doi.org/10.1016/j.acalib.2011.11.003
Rankin, J. A., Grefsheim,
S. A., & Canto, C. C. (2008). The emerging informationist
speciality: A systematic review of the literature. Journal of the Medical Library Association, 96(3), 194-206. http://dx.doi.org/10.3163/1536-5050.96.3.005
Salang, M. M. C. (1996). A meta-analysis of
studies on user information needs and their relationship to information
retrieval. Journal of Philippine
Librarianship, 18(2), 36-56.
Sampson, M., McGowan, J., Lefebvre, C., Moher,
D., & Grimshaw, J. (2008). PRESS: Peer review of electronic search strategies. Ottawa:
Canadian Agency for Drugs and Technologies in Health. Retrieved from
https://www.cadth.ca/media/pdf/477_PRESS-Peer-Review-Electronic-Search-Strategies_tr_e.pdf
Sampson, M. (2014). Welcoming systematic
reviews to the Journal of the Medical Library Association. Journal of the Medical Library Association, 102(3):143-145.
http://dx.doi.org/10.3163/1536-5050.102.3.001
Saxton, M. L. (1997). Reference service
evaluation and meta-analysis: Findings and methodological issues. The Library Quarterly, 67(3), 267-289. Retrieved from
http://www.jstor.org/stable/40039723
Schulz, K. F., Chalmers, I., Hayes, R. J.,
& Altman, D. G. (1995). Empirical evidence of bias. Dimensions of
methodological quality associated with estimates of treatment effects in
controlled trials. Journal of the
American Medical Association, 273(5),
408-412. http://dx.doi.org/10.1001/jama.1995.03520290060030
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter,
L. M. (2007). Development of
AMSTAR: a measurement tool to assess the methodological quality of systematic
reviews. BMC Medical Research
Methodology, 7:10. http://dx.doi.org/10.1186/1471-2288-7-10
Sommestad, T., Hallberg, J., Lundholm, K., & Bengtsson, J.
(2014). Variables influencing information security policy compliance: A
systematic review of quantitative studies. Information
Management & Computer Security,
22(1), 42-75. http://dx.doi.org/10.1108/IMCS-08-2012-0045
Tong, A., Flemming,
K., McInnes, E., Oliver, S., & Craig, J. (2012).
Enhancing transparency in reporting the synthesis of qualitative research:
ENTREQ. BMC Medical Research Methodology,
12:181. http://dx.doi.org/10.1186/1471-2288-12-181
Urquhart, C., & Yeoman, A. (2010).
Information behaviour of women: Theoretical perspectives on gender. Journal of Documentation, 66(1),
113-139. http://dx.doi.org/10.1108/00220411011016399
Verhagen, A. P., de Vet,
H. C., de Bie, R. A., Boers, M., & van den Brandt, P. M. (2001). The art of quality assessment of RCTs included in systematic
reviews. Journal of Clinical Epidemiology, 54(7), 651-654. http://dx.doi.org/10.1016/S0895-4356(00)00360-7
Voss, P. H.,
& Rehfuess. E. A. (2013). Quality appraisal in
systematic reviews of public health interventions: an empirical study on the
impact of choice of tool on meta-analysis. Journal
of Epidemiology and Community Health, 67(1), 98-104. http://dx.doi.org/10.1136/jech-2011-200940
Wagner, K. C., & Byrd, G. D. (2004).
Evaluating the effectiveness of clinical medical librarian programs: A
systematic review of the literature. Journal
of the Medical Library Association, 92(1), 14-33. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/journals/93/
Ward, R., Stevens, C., Brentnall,
P., & Briddon, J. (2008). The attitudes of health
care staff to information technology: A comprehensive review of the research
literature. Health Information and
Libraries Journal, 25(2), 81-97. http://dx.doi.org/10.1111/j.1471-1842.2008.00777.x
Weightman, A. L., & Williamson,
J. (2005). The value and impact of information provided through library
services for patient care: A systematic review. Health Information and Libraries Journal, 22(1), 4-25. http://dx.doi.org/10.1111/j.1471-1842.2005.00549.x
West, S., King, V., Carey, T. S., Lohr, K. N., McKoy, N., Sutton,
S. F., & Lux, L. (2002). Systems to
rate the strength of scientific evidence, Evidence report/technology assessment
no. 47. (AHRQ Publication No. 02-E016). Rockville, MD: Agency for
Healthcare Research and Quality. Retrieved from
http://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf
Williams, P., Nicholas D., & Rowlands, I.
(2010). The attitudes and behaviours of illegal downloaders. Aslib Proceedings, 62(3), 283-301. http://dx.doi.org/10.1108/00012531011046916
Winning, M. A., & Beverley, C. A. (2003).
Clinical librarianship: A systematic review of the literature. Health Information and Libraries Journal, 20(Suppl.
1), 10-21. http://dx.doi.org/10.1046/j.1365-2532.20.s1.2.x
Yuan, Y., & Hunt, R. (2009). Systematic
reviews: The good, the bad, and the ugly. American
Journal of Gastroenterology, 104(5), 1086-1092. Retrieved from
http://www.nature.com/ajg/index.html
Zhang, L., Watson, E. M. & Banfield, L. (2007). The efficacy of computer-assisted
instruction versus face-to-face instruction in academic libraries: a systematic
review. The Journal of Academic
Librarianship, 33(4), 478-484. http://dx.doi.org/10.1016/j.acalib.2007.03.00
2016 Maden and Kotas. This is an Open
Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.