Article

 

The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance

 

Paul Levay

Information Specialist

National Institute for Health and Care Excellence

Manchester, United Kingdom

Email: paul.levay@nice.org.uk

 

Michael Raynor

Information Specialist

National Institute for Health and Care Excellence

Manchester, United Kingdom

Email: michael.raynor@nice.org.uk

 

Daniel Tuvey

Information Specialist

National Institute for Health and Care Excellence

London, United Kingdom

Email: daniel.tuvey@nice.org.uk

 

Received: 22 Oct. 2014    Accepted: 8 Feb. 2015     

 

 

cc-ca_logo_xl 2015 Levay, Raynor, and Tuvey. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

 

Abstract

 

Objective – To make recommendations for the National Institute for Health and Care Excellence (NICE) on the factors to consider when choosing databases and search techniques when producing systematic reviews to support public health guidance development.

 

Methods – Retrospective analysis of how the publications included in systematic reviews commissioned by NICE on obesity, spatial planning, and tuberculosis were retrieved. The

 

included publications were checked to see if they were found from searching MEDLINE, another database or through other search techniques.

 

Results – MEDLINE contributed 24.2% of the publications included in the obesity review, none of the publications in the spatial planning review and 72% of those in the tuberculosis review. Other databases accounted for 9.1% of included publications in obesity, 20% in spatial planning and 4% in tuberculosis. Non-database methods provided 42.4% of the included publications in the obesity review, compared to 5% in the spatial planning review and 24% in the tuberculosis review. It was not possible to establish retrospectively how 24.2% of the publications in the obesity review and 75% in the spatial planning review were found.

 

Conclusions – Topic-specific databases and non-database search techniques were useful for tailoring the resources to the review questions. The value of MEDLINE in these reviews was affected by the degree of overlap with clinical topics, the domain of public health, and the need to find grey literature. The factors that NICE considers when planning a systematic search are the multidisciplinary nature of public health and the different types of evidence required.

Introduction

 

A systematic review is designed to reduce bias, synthesize the available evidence, and answer a specific research question (Higgins & Green, 2011). It is essential that a systematic review begins with a systematic search, in order that the evidence to be appraised is gathered in an unbiased, transparent, and robust manner. The purpose of this study was to explore the challenges that the National Institute for Health and Care Excellence (NICE) has experienced when applying the principles of systematic searching to public health topics. The purpose of public health is “promoting and protecting health and well-being, preventing ill health and prolonging life through the organised efforts of society” (Faculty of Public Health, 2010). The definition encapsulates three key domains in public health: health improvement, improving services, and health protection.

 

Methods for NICE Public Health Guidance

 

NICE is an independent public body responsible for producing public health guidance to promote good health and prevent ill health in England. NICE has produced guidance across all three domains of public health, including behaviour change interventions (health improvement), smoking cessation services (improving services), and preventing infectious diseases (health protection). The guidance is developed following rigorous and transparent methods, as set out in a manual that incorporates public health, social care, safe staffing, medicines practice, and clinical guidelines (NICE, 2014). Once the government has referred a topic to NICE, the process starts with a scope document that outlines the key questions and the populations, settings, and interventions that will, and will not, be covered by the guidance. The next stage is for the evidence to be gathered and appraised in a systematic review. Evidence Statements are prepared to demonstrate the quantity, strength, appropriateness, and applicability of the findings. The evidence is then presented to an independent committee of experts who use it to inform a series of recommendations. The draft recommendations are subject to public consultation with stakeholders before final publication and implementation.

 

Searching is an integral part of the process as NICE is explicit that its recommendations are made “using the best available evidence” (NICE, 2014, p. 5). The chapter on identifying the evidence (NICE, 2014, chapter 5) describes the main stages in the process, which starts with a series of exploratory searches to support writing of the scope. Once the scope has been finalized, a protocol is developed setting out the parameters of the search and a draft strategy for the principal database. The search strategy is subjected to a rigorous quality assurance check involving information specialists and the technical team. The agreed search strategy is run across the sources and the results are gathered in Reference Manager or similar bibliographic management software. The manual has clear requirements on recording searches to ensure that the methods are transparent to the committee, stakeholders, and the wider public (NICE, 2014, section 5.9).

 

At the time of this study, all of the systematic reviews were written independently of NICE by external review teams who were commissioned to run the searches, extract the data, and write the report. The review teams commissioned by NICE have included universities, research institutes, and business consultancies. A NICE team including information specialists, technical analysts, and health economists is responsible for quality assuring the review team’s protocols, searches and reports.

 

It is important for NICE to develop its own methods as the purpose is to produce guidance for England and not just to write a systematic review for academic purposes. NICE has an ongoing commitment to keeping its methods up to date and the manuals have undergone several revisions to ensure they follow current best practice. The periodic process of updating the manuals draws on research conducted by NICE itself, published papers, and the recommendations from other relevant organisations, such as the Campbell Collaboration and the Evidence for Policy & Practice Information and Co-ordinating (EPPI) Centre. The methods are aligned to, and draw on, the recommendations made by other organizations. For example the current manual refers to the Cochrane Collaboration, the InterTASC Information Specialists' Sub-Group, the Canadian Agency for Drugs and Technologies in Health (CADTH), and the Centre for Reviews and Dissemination (NICE, 2014, sections 5.3, 5.4, 5.7, 5.11).

 

NICE has been considering for several years how systematic review methods originally developed in the medical field can be used to review evidence in public health (Kelly et al., 2010). The current study extends a previous case study on cardiovascular disease (Bayliss, Davenport, & Pennant, 2014) to a wider set of topics.

 

Literature Review

 

All systematic searches need to target the right sources of information to find the right types of evidence. Authors of Cochrane systematic reviews are advised that using MEDLINE alone does not constitute a systematic search, as it may retrieve biased or unrepresentative sets of results (Lefebvre, Manheimer, & Glanville, 2011, section 6.1.1.2). The question for the searcher is how to balance the need to use a range of resources beyond MEDLINE with the need to complete the review on time and to budget. There are several issues that make these challenges more pronounced in public health searching, as opposed to, say, finding evidence on the efficacy of a pharmaceutical product (Alpi, 2005). These issues are explored further in relation to two main themes, showing that public health reviews require a variety of databases covering a multidisciplinary evidence base, and a range of search techniques to locate different types of evidence.

 

Multidisciplinary Evidence Base

 

Medical sources should not be overlooked when starting to review a public health topic, even if evidence from other disciplines will need to be considered. This means that MEDLINE can be an important source of evidence for public health. A case study on multi-factor interventions to prevent cardiovascular disease at a population level (Bayliss et al., 2014) found that MEDLINE contributed 91% of the programmes. Most of the programmes were identified through MEDLINE, although it was necessary to search for clinical trials (Cochrane Central Register of Controlled Trials, CENTRAL), the social sciences (Applied Social Sciences Index and Abstracts, ASSIA), and psychology (PsycINFO) to retrieve all of the evidence. The value of MEDLINE was boosted when efforts were made to improve its public health coverage, such as adding 10 main headings to Medical Subject Headings (MeSH) in 2003 (Whitener, Van Horne, & Gauthier, 2005). It is also feasible to draw on the health services research indexed on MEDLINE to inform the service delivery elements of public health guidance, even though this can be challenging (Wilczynski, Haynes, Lavis, Ramkissoosingh, & Arnold-Oatley, 2004).

 

It is instructive to contrast public health topics with clinical topics. A study of clinical guidelines found that MEDLINE accounted for 94.72% of references in head and neck cancer, 88.94% in hepatitis C, 88.18% in bronchiolitis, 82.68% in cervical cancer, and 78.02% in autism (Kelly, 2008). MEDLINE made a similarly high contribution to a systematic review of diabetes epidemiology, accounting for 94% of references (Royle, Bain, & Waugh, 2005). Combining MEDLINE with several other medical databases achieved 90% recall in breast cancer (Lemeshow, Blum, Berlin, Stoto, & Colditz, 2005), 100% in nursing studies (Subirana, Sola, Garcia, Gich, & Urrutia, 2005), and 97% in orthopaedics (Slobogean, Verma, Giustini, Slobogean, & Mulpuri, 2009). Booth (2010) has suggested that MEDLINE consistently delivers up to 80% of the relevant references in health technology assessments. The importance of MEDLINE is clear in these reviews.

 

By contrast, it is not possible to limit searches for public health topics to a defined and regular set of sources in the way that might be possible for clinical topics. It would be unfair to characterize clinical topics as requiring the same resources every time a review is conducted (Beyer & Wright, 2013; Crumley, Wiebe, Cramer, Klassen, & Hartling, 2005). The difference with literature searches in public health is the breadth of potential sources that might need to be captured, as it draws its evidence from a number of disciplines, including psychology, education, sociology, housing, transport, and architecture (NICE, 2012b). Alpi (2005) has shown that a wider range of databases is needed to ensure each of these disciplines is covered adequately. NICE public health guidance has covered topics as diverse as domestic violence (requiring access to criminal justice sources), promoting cycling schemes (transport), body mass index thresholds (epidemiology), looked after children (social care), workplace health (business), and behaviour change (psychology).

 

The issue for the searcher is that the evidence from other disciplines is likely to be indexed on specialized databases, which may increase the time and costs required for the review (Alpi, 2005). Specialized databases were valuable to a review on exercise therapy, where they were used alongside MEDLINE and other techniques (Stevinson & Lawlor, 2004). A review of interventions on encouraging walking and cycling concluded that 4 of the 69 included publications came from “first line health databases”, compared to 8 from social science sources and 33 from a topic-specific transport database (Ogilvie, Hamilton, Egan, & Petticrew, 2005). Further evidence on the value of searching beyond MEDLINE is available on a number of topics relevant to public health, including: occupational health (Rollin, Darmoni, Caillard, & Gehanno, 2010), social welfare (Taylor, Wylie, Dempster, & Donnelly, 2007), maternal health (Betran, Say, Gulmezoglu, Allen, & Hampson, 2005), mental health (Lohonen, Isohanni, Nieminen, & Miettunen, 2010), mental illness (Brettle & Long, 2001), psychiatry (Mcdonald, Taylor, & Adams, 1999), and injury prevention (Lawrence, 2008).

 

Range of Search Techniques

 

It would not be appropriate to focus a public health review on retrieving evidence from randomized controlled trials (Kelly et al., 2010). It would be difficult to blind patients to some public health interventions, such as exercise therapy. Interventions can be organized at a population level (such as taxation) and some trials might be unethical in areas where interventions are known to be beneficial (e.g., smoking cessation). This means that NICE public health guidance does not just assess whether an intervention works, it also addresses “when, why, how, and for whom an approach does (and does not) work” (NICE, 2012b, p. 17). These types of questions are often answered by grey literature, such as reports, case studies, theses, surveys, audits, and other guidelines, which can be difficult to locate and retrieve (Benzies, Premji, Hayden, & Serrett et al., 2006; Turner, Liddy, Bradley, & Wheatley, 2005). Grey literature has previously been found to be a “productive way” of identifying further studies for a different NICE programme that appraises drugs and other health technologies (Royle & Waugh, 2003). The issue in public health is that this type of contextual evidence is key to understanding the interventions and whether it would be appropriate for NICE to recommend them, rather than it just being useful supplementary information to what can be retrieved from databases.

 

The emphasis on grey literature means that non-database searching methods, such as citation searching, website searching, pearl growing, and contacting experts are an essential component of comprehensive searches in public health (Papaioannou, Sutton, Carroll, Booth & Wong, 2010). A search that is too focused on trawling a standard list of databases, covering only peer-reviewed journal articles, is unlikely to leave sufficient time to use additional methods for locating grey literature (Booth, 2010).

 

A comparison of 12 Cochrane systematic reviews in public health found that 9 of them cited evidence that could not be retrieved in 5 large health databases and had benefited from supplementary methods (Morgan, Bauschmann, & Weightman, 2011). This particular study helps to confirm that the recommendation in the NICE manual to search for grey literature is consistent with the findings of the Cochrane Public Health Group. Stansfield, Brunton and Rees (2013) examined qualitative reviews on transport, motherhood, and obesity, finding that around a third of the 229 studies would have been missed if only databases had been used, no matter how many were searched. Similar figures were found in: a review deriving 21% of the included publications from citation searching, websites, and hand-searching (Stansfield, Kavanagh, Rees, Gomersall, & Thomas, 2012); a search on childhood obesity where 13% of reviews came from websites, library catalogues, and bibliographies (Woodman et al., 2010); and a review on the built environment where a quarter of studies required specialized sources and grey literature (Weaver et al., 2002).

 

The aim of Bayliss et al. (2014) was to examine the suitability of different databases for searching on a public health topic and they did not consider other search techniques. The purpose of the current study was to examine a range of search techniques across a larger sample of subjects.

 

Aims

 

The aim of the study was to make recommendations on the factors to consider when choosing databases and search techniques when producing systematic reviews to support public health guidance development at NICE.

 

The objectives of the study were to:

 

·         Assess the value of bibliographic databases (particularly in relation to MEDLINE) for identifying the evidence for the guidance in an appropriate sample

·         Evaluate the contribution of different search techniques for identifying the evidence for the guidance in an appropriate sample

·         Make recommendations on the factors to consider when planning the systematic searches to support public health guidance development.

 

Methods

 

Personnel

 

Three information specialists, each with at least two years’ experience of literature searching to support NICE public health guidance development, carried out the sampling and analysis between February and October 2011. The information specialists involved in this study had not actually conducted the searches, although they had been involved in the procurement process to select the external review teams and had quality assured the protocols and strategies.

 

Sampling of Guidance

 

Selection was carried out in February 2011 and guidance was eligible for inclusion if:

 

·         The search was conducted in accordance with the then current edition of the NICE methods manual (NICE, 2009)

·         The review was completed and had been presented to an expert committee meeting on or before 1 February 2011.

 

The resources were available to analyze three reviews for this study. Maximum variation sampling was chosen so that a full range of issues would be experienced with the three reviews. It was felt that analyzing three reviews from the same review team, taking the same approach or in the same domain of public health would limit the lessons learnt in this study. The value of choosing these three was to help in assessing the feasibility of scaling up the study to a larger sample of reviews.

 

Seven guidance topics were eligible for inclusion. NICE often commissions several systematic reviews on each topic and a total of 17 had been completed by the cut-off date for the 7 topics. The three purposively chosen reviews covered obesity (Garside, Pearson, Hunt, Moxham, & Anderson, 2010), spatial planning (Gray et al., 2010), and tuberculosis (O’Mara et al., 2010). Table 1 shows that they cover at least two different public health domains, both clinical and non-clinical topics, a range of search approaches, and were conducted by different types of review team.

 

 

Table 1

Attributes of Reviews Selected for Study

Topic title

Type of review team

Public health domain

Clinical/non-clinical

Search approach

Obesity: working with local communities

University

Health improvement

Non-clinical

Iterative

Spatial planning for health: local authorities and primary care trusts

University

Health improvement

Non-clinical

Comprehensive

Identifying and managing tuberculosis among hard-to-reach groups

Business consultancy

Health protection

Clinical

Comprehensive

 

 

Data Analysis

 

The three reviews were analyzed to determine how the publications included in them had been retrieved. The unit of assessment in this study is the “included publication”, which refers to the publications cited in the Evidence Statements contained in the reviews. An Evidence Statement is an aggregated summary of all the relevant studies, regardless of their findings, reflecting the balance of the evidence, its strength, applicability, and any gaps (NICE, 2014, p. 107). This means that the analysis was not concerned with citations contributing to the background context (such as epidemiology) or the methods.

 

This study obtained the list of sources used from the original search protocols. The included publications were extracted from the Evidence Statements for each of the three reviews identified in table 1 and the numbers were checked against the literature flow diagram in the systematic review. Review teams usually supply NICE with a database of search results in Reference Manager or similar bibliographic software as part of their contract to produce the systematic review. These databases should show how each publication has been located, whether it was included in the Evidence Statements, or why it was excluded. The Reference Manager files were checked to determine the format of each included publication, the method used to locate it, the database on which it was found, whether it was ordered in full text, and any annotations on the inclusion decision. Any queries were referred to the review teams who had conducted the searches.

 

Results

 

Table 2 shows that 39.4% (13 of 33) of the included publications in the obesity review were journal articles, compared to 80% (16 of 20) in the spatial planning review, and 76% (19 of 25) in the tuberculosis review. The obesity review, with 42.4%, had the highest proportion of included publications classified as grey literature, compared with 20% in the spatial planning review, and 24% in the tuberculosis review.

 

Table 3 shows that the obesity review had 33 included publications and 33.3% of these were from databases and 42.4% were from a variety of non-database techniques (24.2% from checking the reference lists of relevant publications, 15.2% from the call for evidence on the NICE website, and 3% from website searching). The search records did not describe how eight (24.2%) of the publications in the obesity review had been located. Database searches retrieved 76% of the 25 included publications in the tuberculosis review, while non-database methods retrieved 24% (16% from contact with experts and 8% from the National Research Register website).

 

 

Table 2

Format of Included Publications

Format

Obesity

Spatial planning

Tuberculosis

Book or book chapter

6 (18.2%)

0 (0)

0 (0)

Grey literature

14 (42.4%)

4 (20.0%)

6 (24.0%)

Journal article

13 (39.4%)

16 (80.0%)

19 (76.0%)

Total

33 (100%)

20 (100%)

25 (100%)

 

 

Table 3

How the Included Publications Were Found

Search method

Obesity

Spatial planning

Tuberculosis

Call for evidence

5 (15.2%)

1 (5.0%)

0 (0)

Contact with experts

0 (0)

0 (0)

4 (16.0%)

Database (Core)*

11 (33.3%)

0 (0)

19 (76.0%)

Database (Topic specific)

0 (0)

4 (20.0%)

0 (0)

National Research Register

0 (0)

-

2 (8.0%)

Reference list harvesting

8 (24.2%)

-

0 (0)

Website

1 (3.0%)

-

0 (0)

Unknown

8 (24.2%)

15 (75.0%)

0 (0)

Total

33 (100%)

20 (100%)

25 (100%)

* Core database is defined as those listed in the second edition of the NICE Methods Manual (NICE, 2009, appendix A).

 

 

Table 3 shows that it was only possible to determine how 5 of the 20 publications included in the spatial planning review had been retrieved, with 4 from database searching and 1 from the call for evidence that was posted on the NICE website at time the searches were being conducted. The sources were not recorded in Reference Manager and so the original 11 files downloaded from the bibliographic databases were obtained from the review team. Table 5 shows that three of the included publications were found in the Planex file and one in the Urbadoc file; these are specialist databases on town and country planning. None of the included publications were found in the MEDLINE file. It is unlikely that the other 15 publications were found through database searching if they were not contained in these 11 files. It is impossible to replicate the non-database methods to see how the 15 were found; for example, citation searching would now retrieve a different set of results.

 

Tables 4-6 show the contribution of each database in terms of the publications found through database searching and the total number of included publications. It was felt that only reporting how the journal articles had been found would hide the importance of grey literature and unfairly boost the contribution of MEDLINE. Table 4 shows that MEDLINE contributed 72.7% of the publications found using databases and 24.2% of the total number of included publications in the obesity review. On the other hand, as table 6 shows, MEDLINE contributed 94.7% of the publications derived from databases and 72% of the total number in the tuberculosis review. MEDLINE did not contribute any publications to the spatial planning review, as shown in table 5. The purpose of this study was to determine the value of databases in relation to MEDLINE and the retrieval of the same publication from multiple sources was not considered. The resources were not available, and it would require another study, to determine how many of the databases index the included publications, whether the search strategies retrieved them, and the degree of database overlap.

 

 

Table 4

Databases Used To Locate Included Publications in the Obesity Review

Database

No. of publications

% of included publications found by database searching

% of all included publications

ASSIA*

1

9.1%

3.0%

Bibliomap

0

0

0

Cumulative Index to Nursing and Allied Health (CINAHL)*

0

0

0

Database of Promoting Health Effectiveness Reviews (DoPHER)

0

0

0

Health Management Information Consortium (HMIC)*

2

18.2%

6.1%

Intute

0

0

0

MEDLINE*

8

72.7%

24.2%

Obesity and Sedentary Studies Database

0

0

0

Social Science Citation Index

0

0

0

Trials Register of Promoting Health Interventions (TRoPHI)

0

0

0

Total

11

100%

33.3%

 

 

Discussion

 

Multidisciplinary Evidence Base

 

Tables 4-6 show that six different databases contributed unique publications to the three reviews (ASSIA, CINAHL, HMIC, MEDLINE, Planex and Urbadoc). These databases demonstrate the multidisciplinary nature of the resources required, with coverage of the social sciences, nursing, health management, and urban planning, as well as MEDLINE. The manual used at the time of the three reviews (NICE, 2009) included a core list of databases that was to be considered for all topics. The findings from this study suggest that enforcing a standard list of sources on all searches might be of limited value, as some topics will draw heavily on MEDLINE and others will need topic-specific sources. It might be legitimate to prioritize topic-specific resources in some reviews, for example the TRANSPORT database might be more valuable than Embase in a review of interventions to increase the uptake of cycling. The NICE manual now encourages searchers to choose sources “depending on the subject of the review question” (NICE, 2014, p. 78). Choosing the right databases for the review question suggests that it would be useful to conduct scoping searches early in the process to map out where the evidence lies and to identify any major gaps.

 

The cardiovascular case study (Bayliss et al., 2014) suggested that, as some public health topics overlap with clinical issues more than others, the “medical public health topics” might rely on MEDLINE more than “non-medical” ones. The results of this study seem to confirm this finding, as MEDLINE contributed 24.2% to the obesity review, none to the spatial planning review, and 72% to the tuberculosis review. The tuberculosis review was the only topic approaching the benchmark that “MEDLINE consistently delivers an average of 80% of included papers for systematic reviews” (Booth, 2010, p. 432). It would be difficult to classify consistently public health topics as either “medical” or “non-medical” and so it would be worth exploring whether the domains of public health are a more fruitful framework for analysis.

 

The tuberculosis review dealt with an infectious disease, which tends towards the “medical” end of the public health scale but it is also a health protection topic. The tuberculosis review suggests health protection might be the domain of public health most likely to overlap with clinical topics and benefit from MEDLINE. This could be tested against the publications included in NICE guidance on other health protection topics, such as HIV testing, hepatitis testing, and controlling hospital-acquired infections.

 

 

Table 5

Databases Used To Locate Included Publications in the Spatial Planning Review

Database

No. of publications

% of included publications found by database searching

% of all included publications

CAB Abstracts

0

0

0

Embase*

0

0

0

GEOBASE

0

0

0

HMIC*

0

0

0

International Construction Database (ICONDA)

0

0

0

MEDLINE*

0

0

0

Planex

3

75.0%

15.0%

PscyINFO*

0

0

0

Social Science Citation Index*

0

0

0

Transport Research Information Systems (TRIS)

0

0

0

Urbadoc

1

25.0%

5.0%

Total

4

100%

20.0%

* Core databases

 

 

Table 6

Databases Used To Locate Included Publications in the Tuberculosis Review

Database

No. of publications

% of included publications found by database searching

% of all included publications

ASSIA*

0

0

0

British Library Direct

0

0

0

British Nursing Index*

0

0

0

CINAHL*

1

5.3%

4.0%

Cochrane Database of Systematic Reviews (CDSR)*

0

0

0

Community Abstracts

0

0

0

Current Contents

0

0

0

Embase*

0

0

0

Education Resources Information Center (ERIC)

0

0

0

HMIC*

0

0

0

MEDLINE*

18

94.7%

72.0%

PsycINFO*

0

0

0

Social Policy and Practice*

0

0

0

Sociological Abstracts

0

0

0

Social Services Abstracts

0

0

0

Web of Science

0

0

0

Total

19

100%

76.0%

* Core databases

 

 

 

 

 

The second highest ranked review, in terms of MEDLINE contribution, was obesity, which again reflects the type of topic being reviewed. Obesity is itself a topic likely to be indexed in considerable depth on a medical database such as MEDLINE. The review was not, however, concerned with clinically treating obesity and it examined the value of local authorities adopting a “whole-systems approach”. It is difficult to define this topic as either “medical” or “non-medical” and it seems to fit more comfortably in the health improvement domain of public health. The public-health nature of this topic is reflected in the contribution of other databases, with two included publications coming from HMIC and one from ASSIA (table 4). The lack of specific evidence on the topic meant that the obesity angle was dropped from some searches and they were broadened out to explore how the concept of the whole-systems approach had been applied in other domains, such as ecology. This may go some way to explaining why one study was found on a social science source (ASSIA), while databases that might be expected to be useful on the topic of obesity, such as CINAHL or Embase, did not yield any included publications.

 

Range of Search Techniques

 

The results from these three reviews highlight the need to leave sufficient time to use a range of techniques to search for grey literature. Table 2 shows that in the obesity review, 39.4% of the publications were journal articles, 42.4% were grey literature, and a further 18.2% were books or book chapters. The conclusions of this review would have been affected if the search had been restricted to the large medical databases that only index journal articles, such as MEDLINE and Embase. This point is emphasized by the finding that a number of databases did not contribute any included publications to the three reviews and there might have been scope to focus the searches further. Tables 4-6 show that no included publications were retrieved by 7 of the 10 databases used in the obesity review, 9 of the 11 in the spatial planning review, and 14 of the 16 in the tuberculosis review. It was not possible to know in advance which databases would not find any unique publications but this does suggest that a targeted approach may be as productive as searching a long list of sources.

 

The obesity searches progressed iteratively to explore the available literature one step at a time. The concept of a “whole-systems approach” had not been well developed, it did not have an agreed definition, and there was little consensus on how the terminology could be applied. A comprehensive search could not be planned at the outset of the project if the definition of the main concept was contested. The definition emerged from reviewing some of the evidence and this, in turn, led the search in new directions. This approach to searching builds on the “berrypicking model” defined by Bates (1989), which shows how it is legitimate to build a series of results at each stage of an “ever-modifying search” instead of expecting to find a “single final retrieved set”.

 

Table 4 shows that the MEDLINE searches contributed 24.2% of the 33 publications included in the obesity review but this overall number masks the iterative nature of the searching. MEDLINE searches were undertaken at several phases of the process and not as one large search strategy. The first phase involved scoping out the topic and MEDLINE was used alongside website searching and contact with experts. The relevant publications identified during the first phase formed a cluster of documents worthy of further investigation during the second phase. The first search identified several projects that had adopted a relevant approach to obesity and the second phase involved trying to find more evidence on these named interventions. Three of the eight publications identified from MEDLINE were contained in the phase 1 searches and five were only found when searching MEDLINE in phase 2 for the project names found in phase 1. Reference harvesting was an important method, contributing eight publications to the final review (table 3). However, the reference harvesting could only take place in phase 2 using relevant publications found by MEDLINE in phase 1. The numbers for each search technique in tables 3 and 4 mask the complexity of the obesity search, where the iterative approach meant the methods were dependent on each other and they should not be treated in isolation.

 

Table 6 shows that MEDLINE contributed 94.7% of the included publications found through databases in the tuberculosis review and 72% of the total number. This suggests that the type of evidence required for the review could influence the usefulness of the search techniques. The review required qualitative evidence on the barriers and facilitators to increasing the uptake of tuberculosis testing. This qualitative evidence might be published in reports, audits, surveys, and other types of grey literature which would not be indexed on MEDLINE. A separate study has shown that designing a search filter on the hard-to-reach population of interest reduced the volume of MEDLINE hits to be screened by 64% and this was crucial to freeing time for more productive activities for exploring the grey literature (Cooper, Levay, Lorenc, & Craig, 2014). MEDLINE was important, as would be expected in the health protection domain, but this was balanced by the need to find qualitative evidence. The results from one review cannot be extrapolated across public health but it does suggest that targeting resources on grey literature could be particularly beneficial when searching for qualitative evidence.

 

Limitations and Further Research

 

The tuberculosis guidance (NICE, 2012a) and the obesity guidance (NICE, 2012c) have been published since this study was completed, although the spatial planning guidance-development process was subsequently suspended. This change in status did not affect the literature searching that had already taken place, which was fully compliant with NICE methods, and so spatial planning was still considered eligible for this study.

 

This study has built on Bayliss et al. (2014) by extending it to a wider range of topics, although the small sample of reviews still limits the generalizability of the results. The data required had not been collected at the time of the searches and it was difficult to analyze the three reviews retrospectively. This study does, however, provide a framework for the analysis of a larger sample of reviews and highlights the data requirements to carry out such research successfully. The main practical issues encountered related to the removal of duplicate records, resource constraints, and recording of iterative searches.

 

The study was difficult to perform retrospectively because of the way that duplicate records had been processed, as Bayliss et al. (2014) also experienced. The review teams provide NICE with a Reference Manager file with the duplicates removed and so there was no way of knowing if each publication had actually been found on several databases. It would only be possible to know how many times a publication had been retrieved by re-running the exact search across all of the databases used by the review team. It would not be reasonable for NICE to expect the review team to undertake the additional work required to record the source of the duplicates within current budgets for guidance production. There would be a further difficulty, given that the contractual relationship with NICE has ended and any queries or requests for data would depend on the good will of the review team.

 

The duplicates issue did not affect the results in this study but it did show that it would be difficult to conduct a larger retrospective study. The risks were mitigated in this study by checking the original files downloaded from MEDLINE for the included publications, rather than just relying on the review team’s annotations in Reference Manager. This established how many publications MEDLINE retrieved, although there was not sufficient time to test how many were unique to this source or the overlap with other databases. The obesity review was not affected in the same way, given the iterative nature of the evidence gathering described above. It was time consuming to analyze the spatial planning review as the 11 files containing the results from each database had to be checked 20 times: once for each of the included publications.

 

Table 3 shows that it was not possible to establish retrospectively how 75% of the included publications in the spatial planning review and 24% in the obesity review were found. It is unlikely that these publications were found by MEDLINE, given the checking that has been conducted, but it does mean that the non-database search techniques could not be fully assessed. Previous studies have acknowledged that non-database sources can be difficult to record in a standardized format (Rader, Mann, Stansfield, Cooper, & Sampson, 2014). One method to improve recording is to provide a narrative description of how the search progressed to explain the decisions made during each iteration, as was found in a different NICE guidance programme (Craven & Levay, 2011). The narrative is a transparent record of the decision-making process during the search but it is not replicable in the way that a database strategy can be re-run exactly as before.

 

The corollary of tailoring sources to the review question is that the searcher must ensure that maximum benefit is derived from each one. It was beyond the scope of the current study to investigate whether MEDLINE could have been searched more effectively in the three reviews but it is an area worthy of further investigation. The obesity review did not aim to search MEDLINE comprehensively as an iterative approach was adopted. The spatial planning review balanced a specific search on MEDLINE with searches on topic-specific databases, which is consistent with the finding that “undertaking very sensitive searches across a number of databases may not be an effective use of resources” (Bayliss et al., 2014, p. 310). The one publication retrieved from CINAHL in the tuberculosis review (Kelly-Rossini, Perlman, & Mason, 1996) was actually indexed on MEDLINE at the time of searching, although it was felt that retrieving 94.7% of the available publications indicated the search strategy was of sufficient quality. A further study would be required to measure how the quality of the MEDLINE strategy affects the need to search other resources.

 

Conclusions

 

The practical issues experienced in this study suggest it would be difficult to analyze retrospectively a larger sample of the public health systematic reviews commissioned by NICE. It would be worth collecting the necessary data at the outset of guidance development to facilitate a larger study. The findings do suggest several issues that are worth considering when planning future public health guidance development searches. A range of databases was required, confirming the conclusion that public health questions cannot be answered with a “one size fits all solution” (Bayliss et al., 2014, p. 304). Grey literature contributed 42% of included publications in the obesity review, 20% in the spatial planning review, and 24% in the tuberculosis review, showing that sufficient time should be allowed to use additional search techniques to find this type of evidence. Scoping searches will take on greater importance in the guidance development process, as they are useful for choosing sources, identifying topic-specific databases, and locating publications that will be useful in the later phases (such as in citation searching).

 

Further work is required to determine how far the domain of public health influences the choice of sources. The tuberculosis review suggests that health protection, where there is overlap with clinical issues, may benefit more from MEDLINE than health improvement or improving services. This could be a more useful framework of analysis than attempting to make a binary choice between “medical” and “non-medical” reviews.

 

The findings from this study have been incorporated into the searching chapter in the current NICE methods manual (NICE, 2014). The manual no longer recommends a core list of databases that should be considered for all review questions. The manual recommends a list of databases to help searchers without making any of them mandatory (NICE, 2014, Appendix G). The manual also suggests that non-database methods should be used when it is “reasonably likely” they will be a productive source of evidence (NICE, 2014, section 5.4). The manual emphasises the need to tailor the search approach to the topic of the research question and the type of evidence required to answer it. The factors that NICE considers when planning which databases and search techniques to use in a systematic search are the multidisciplinary nature of public health and the different types of evidence required.

 

Recommendations

 

The following recommendations are particularly pertinent to NICE and the review teams it commissions, although they are applicable to anyone planning a systematic review on a public-health topic.

 

1.       Search results should be tagged with the names of all sources where the record was found before removing duplicates.

2.       Searches should be fully recorded and accompanied by a narrative outlining the key decisions, especially when an iterative approach is being used.

3.       The list of sources to be searched should be tailored to the review question.

4.       The appropriateness of databases for a topic should be tested at the start of a project.

5.       The value of topic-specific databases relevant to the review question should be explored early in a project.

6.       Scoping searches can be a useful method for informing the later phases of evidence gathering.

7.       The efficiency and appropriateness of non-database search methods should be tested at the beginning of a project.

8.       Research teams and those commissioning reviews should ensure adequate priority is given to using non-database methods.

9.       Further research is required to compare the contribution of MEDLINE across the three domains of public health.

10.    The feasibility of a prospective study of this type should be investigated.

 

References

 

Alpi, K. (2005). Expert searching in public health. Journal of the Medical Library Association, 93(1), 97-103. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC545128/

 

Bates, M. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Information Review, 13(5), 407-424. http://dx.doi.org/10.1108/eb024320

 

Bayliss, S. E., Davenport, C. F., & Pennant, M. E. (2014). Where and how to search for information on the effectiveness of public health interventions - a case study for prevention of cardiovascular disease. Health Information & Libraries Journal, 41(4), 303-313. http://dx.doi.org/10.1111/hir.12075

 

Benzies, K., Premji, S., Hayden, K. A., & Serrett, K. (2006). State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews on Evidence-Based Nursing, 3(2), 55-61. http://dx.doi.org/10.1111/j.1741-6787.2006.00051.x

 

Betran, A., Say, L., Gulmezoglu, A. M., Allen, T., & Hampson, L. (2005). Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Medical Research Methodology, 5(6). http://dx.doi.org/10.1186/1471-2288-5-6

 

Beyer, F., & Wright, K. (2013). Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Information & Libraries Journal, 30(1), 49-58. http://dx.doi.org/10.1111/hir.12009

 

Booth, A. (2010). How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care, 26(4), 431-435. http://dx.doi.org/10.1017/s0266462310000966

 

Brettle, A. J., & Long, A. F. (2001). Comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. Bulletin of the Medical Library Association, 89(4), 353-362. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC57964/

 

Cooper, C., Levay, P., Lorenc T., & Craig G. (2014). A population search filter for hard-to-reach populations increased search efficiency for a systematic review. Journal of Clinical Epidemiology, 67(5), 554-559. http://dx.doi.org/10.1016/j.jclinepi.2013.12.006

 

Craven, J., & Levay, P. (2011). Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of NICE Interventional Procedures Guidance. Evidence Based Library and Information Practice, 6(4). Retrieved from http://ejournals.library.ualberta.ca/index.php/EBLIP/article/view/11594

 

Crumley, E., Wiebe, N., Cramer, K., Klassen, T., & Hartling, L. (2005). Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Medical Research Methodology, 5(24). http://dx.doi.org/10.1186/1471-2288-5-24

 

Faculty of Public Health. (2010). What is public health? Retrieved from http://www.fph.org.uk/what_is_public_health

 

Garside, R., Pearson, M., Hunt, H., Moxham, T., & Anderson, R. (2010). Identifying the key elements and interactions of a whole system approach to obesity prevention. Exeter: Peninsula Technology Assessment Group. Retrieved from: http://www.nice.org.uk/guidance/ph42/resources/review-1-identifying-the-key-elements-and-interactions-of-a-whole-system-approach-to-obesity-prevention2

 

Gray, S., Barton, H., Mytton, J., Lease, H., Carmichael, L., & Joynt, J. (2010). The effectiveness and cost effectiveness of health appraisal processes currently in use to address health and wellbeing during plan appraisal. Bristol: University of the West of England.

 

Higgins, J. P. T., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions (version 5.1.0). Oxford: The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org/

 

Kelly, J. (2008, October). The value of searching multiple databases for guideline development. Poster presented at the 16thCochrane Colloquium, Freiburg, Germany.

 

Kelly, M., Morgan, A., Ellis, S., Younger, T., Huntley, J., & Swann, C. (2010). Evidence based public health: a review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England. Social Science & Medicine, 71(6), 1056-1062. http://dx.doi.org/10.1016/j.socscimed.2010.06.032

 

Kelly-Rossini, L., Perlman, D., & Mason, D. (1996). The experience of respiratory isolation for HIV-infected persons with tuberculosis. Journal of the Association of Nurses in AIDS Care, 7(1), 29-36. http://dx.doi.org/10.1016/S1055-3290(96)80035-5

 

Lawrence, D. W. (2008). What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention, 14, 401-404. http://dx.doi.org/10.1136/ip.2008.019430

 

Lefebvre C., Manheimer E., & Glanville J. (2011) Chapter 6: Searching for studies. In: Cochrane Handbook for Systematic Reviews of Interventions (version 5.1.0), eds. JPT Higgins, S Green. Oxford: Cochrane Collaboration. Retrieved from: http://handbook.cochrane.org/

 

Lemeshow, A., Blum, R., Berlin, J., Stoto, M., & Colditz, G. (2005). Searching one or two databases was insufficient for meta-analysis of observational studies. Journal of Clinical Epidemiology, 58(9), 867-873. http://dx.doi.org/10.1016/j.jclinepi.2005.03.004

 

Lohonen, J., Isohanni, M., Nieminen, P., & Miettunen, J. (2010). Coverage of the bibliographic databases in mental health research. Nordic Journal of Psychiatry, 64(3), 181-188. http://dx.doi.org/10.3109/08039480903337378

 

Mcdonald, S., Taylor, L., & Adams, C. (1999). Searching the right database. A comparison of four databases for psychiatry journals. Health Libraries Review, 16(3), 151-156. Retrieved from http://www.ingentaconnect.com/content/bsc/hlr/1999/00000016/00000003/art00222

 

Morgan, H., Bauschmann, M., & Weightman, A. (2011, October). Finding public health evidence: how many stones need to be turned? Poster presented at the 19th Cochrane Colloquium, Madrid, Spain. Retrieved from http://www.cochrane.org/sites/default/files/uploads/Abstracts_Book_DEF.pdf

 

NICE. (2009). Methods for the development of NICE public health guidance (2nd edition). London: NICE.

 

NICE. (2012a). Identifying and managing tuberculosis among hard-to-reach groups (PH37). London: NICE. Retrieved from http://www.nice.org.uk/guidance/PH37

 

NICE. (2012b). Methods for the development of NICE public health guidance (3rd edition). London: NICE. Retrieved from http://publications.nice.org.uk/methods-for-the-development-of-nice-public-health-guidance-third-edition-pmg4

 

NICE. (2012c). Obesity: working with local communities (PH42). London: NICE. Retrieved from http://www.nice.org.uk/guidance/PH42

 

NICE. (2014). Developing NICE guidelines: the manual. London: NICE. Retrieved from: http://www.nice.org.uk/article/pmg20/ 

 

Ogilvie, D., Hamilton, V., Egan, M., & Petticrew, M. (2005). Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? Journal of Epidemiology and Community Health, 59, 804-808. http://dx.doi.org/10.1136/jech.2005.034181

 

O’Mara, A., Marrero-Guillamon, I., Jamal, F., Lehmann, A., Cooper, C., & Lorenc, T. (2010). Tuberculosis evidence review 1. Review of barriers and facilitators. London: Matrix Knowledge Group. Retrieved from http://www.nice.org.uk/guidance/ph37/resources/review-1-review-of-barriers-and-facilitators2

 

Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114-122. http://dx.doi.org/10.1111/j.1471-1842.2009.00863.x

 

Rader, T., Mann, M., Stansfield, C., Cooper, C., & Sampson, M. (2014). Methods for documenting systematic review searches: a discussion of common issues. Research Synthesis Methods, 5(2), 98-115. http://dx.doi.org/10.1002/jrsm.1097

 

Rollin, L., Darmoni, S., Caillard, J. F., & Gehanno, J. F. (2010). Searching for high-quality articles about intervention studies in occupational health: what is really missed when using only the Medline database? Scandinavian Journal of Work, Environment & Health, 36(6), 484-487. http://dx.doi.org/10.5271/sjweh.3082

 

Royle, P., & Waugh, N. (2003) Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technology Assessment, 7(34). Retrieved from http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0019/65125/FullReport-hta7340.pdf

 

Royle, P., Bain, L., & Waugh, N. (2005). Systematic reviews of epidemiology in diabetes: finding the evidence. BMC Medical Research Methodology, 5(2). http://dx.doi.org/10.1186/1471-2288-5-2

 

Slobogean, G., Verma, A., Giustini, D., Slobogean, B., & Mulpuri, K. (2009). MEDLINE, EMBASE, and Cochrane index most primary studies but not abstracts included in orthopedic meta-analyses. Journal of Clinical Epidemiology, 62(12), 1261-1267. http://dx.doi.org/10.1016/j.jclinepi.2009.01.013

 

Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people's views: a case study in public health. BMC Medical Research Methodology, 12(55). http://dx.doi.org/10.1186/1471-2288-12-55

 

Stansfield, C., Brunton, G., & Rees, R. (2013). Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health. Research Synthesis Methods, 5(2), 142-151. http://dx.doi.org/10.1002/jrsm.1100

 

Stevinson, C., & Lawlor, D. A. (2004). Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine, 12(4), 228-232. http://dx.doi.org/10.1016/j.ctim.2004.09.003

 

Subirana, M., Sola, I., Garcia, J., Gich, I., & Urrutia, G. (2005). A nursing qualitative systematic review required MEDLINE and CINAHL for study identification. Journal of Clinical Epidemiology, 58(1), 20-25. http://dx.doi.org/10.1016/j.jclinepi.2004.06.001

 

Taylor, B., Wylie, E., Dempster, M., & Donnelly, M. (2007). Systematically retrieving research: a case study evaluating seven databases. Research on Social Work Practice, 17(6), 697-706. http://dx.doi.org/10.1177/1049731507304402

 

Turner, A., Liddy, E., Bradley, J., & Wheatley, J. (2005). Modeling public health interventions for improved access to the gray literature. Journal of the Medical Library Association, 93(4), 487-494. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1250325/

 

Weaver, N., Williams, J. L., Weightman, A. L., Kitcher, H. N., Temple, J. M., Jones, P., & Palmer, S. (2002). Taking STOX: developing a cross disciplinary methodology for systematic reviews of  research on the built environment and the health of the public. Journal of Epidemiology and Community Health, 56, 48-55. http://dx.doi.org/10.1136/jech.56.1.48

 

Whitener, B. L., Van Horne, V. V., & Gauthier, A. K. (2005). Health services research tools for public health professionals. American Journal of Public Health, 95(2), 204-207. http://dx.doi.org/10.2105/AJPH.2003.035030

 

Wilczynski, N. L., Haynes, R. B., Lavis, J. N., Ramkissoosingh, R., & Arnold-Oatley, A.E. (2004) Optimal search strategies for detecting health services research in MEDLINE. Canadian Medical Association Journal, 171(10), 1179-1185. http://dx.doi.org/10.1503/cmaj.1040512

 

Woodman, J., Harden, A., Thomas, J., Brunton, J., Kavanagh, J., & Stansfield, C. (2010). Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. Journal of the Medical Library Association, 98(2), 140-146. http://dx.doi.org/10.3163/1536-5050.98.2.006