Review Article

Librarian Co-Authored Systematic Reviews are Associated with Lower Risk of Bias Compared to Systematic Reviews with Acknowledgement of Librarians or No Participation by Librarians

Mikaela Aamodt

Senior Librarian

Medical Library, University of Oslo

Oslo, Norway

Email: mikaela.aamodt@ub.uio.com

Hugo Huurdeman

Postdoctoral Researcher

University of Amsterdam

Amsterdam, The Netherlands

Email: H.C.Huurdeman@uva.nl

Hilde Strømme

Adviser

Medical Library, University of Oslo

Oslo, Norway

Email: hilde.stromme@ub.uio.no

Received: 2 July 2019 Accepted: 22 July 2019

2019 Aamodt, Huurdeman, and Strømme. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

DOI: 10.18438/eblip29601

Abstract

Objective - To explore the prevalence of systematic reviews (SRs) and librarians’ involvement in them, and to investigate whether librarian co-authorship of SRs was associated with lower risk of bias.

Methods - SRs by researchers at University of Oslo or Oslo University Hospital were counted and categorized by extent of librarian involvement and assessed for risk of bias using the tool Risk of Bias in Systematic Reviews (ROBIS).

Results - Of 2,737 identified reviews, 324 (11.84%) were SRs as defined by the review authors. Of the 324 SRs, 4 (1.23%) had librarian co-authors, in 85 (26.23%) librarians were acknowledged or mentioned in the methods section. In the remaining 235 SRs (72.53%), there was no clear evidence that a librarian had been involved. Librarian co-authored SRs were associated with lower risk of bias compared to SRs with acknowledgement or no participation by librarians.

Conclusion - SRs constitute a small portion of published reviews. Librarians rarely co-author SRs and are only acknowledged or mentioned in a quarter of our sample. The quality and documentation of literature searches in SRs remains a challenge. To minimise the risk of bias in SRs, librarians should advocate for co-authorship.

Background

Over the past decades, systematic reviews (SRs) have increased as a research methodology in the health sciences, and they have a great influence on healthcare. In fact, evidence suggest that SRs are the most commonly cited study type in clinical research (Patsopoulos, Analatos, & Ioannidis, 2005). The foundation of every high-quality systematic review is a carefully designed and well-conducted literature search to avoid missing key studies and minimize bias. In addition to a comprehensive literature search, researchers should aim for full transparency when reporting how studies were identified and what was done to minimize bias, so that readers are able to determine the validity of the results (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009) and to prevent suboptimal SRs (Ioannidis, 2016). Librarians and information specialists (hereafter referred to as librarians) have a significant role to play in expert searching and documentation of search strategies, in the systematic review process. The importance of including a librarian when conducting the literature search has been widely recognized by prominent organisations such as the Cochrane Collaboration, the Campbell Collaboration, and the Joanna Briggs Institute (Aromataris & Munn, 2017; Higgins, Churchill, Chandler, & Cumpston, 2017; The Campbell Collaboration, 2019). Previous studies have shown that systematic review search quality and quality of reporting is a challenge (Golder, Loke, & McIntosh, 2008; Koffel & Rethlefsen, 2016; Toews, 2017), even when librarians are consulted (Meert, Torabi, & Costella, 2016; Rethlefsen, Farrell, Osterhaus Trzasko, & Brigham, 2015). Meert et al. (2016) and Rethlefsen et al. (2015) found that having a librarian as a review team member or co-author improved the quality of searches and reporting. In preparation for developing a formal systematic review service, Ross-White (2016) explored librarians’ involvement in systematic reviews at Queen’s University, Canada. They found that in 231 systematic reviews, librarian co-authorship was granted in 31 and librarians were acknowledged in 36, however they did not investigate whether or not librarian involvement had an impact on the quality of searches and reporting.

Rethlefsen et al. (2015) examined the librarians’ impact on quality of reporting in SRs from high-impact general internal medicine journals and Meert et al. (2016) looked at SRs in the twenty highest impact factor paediatric journals. Our aim is to investigate whether the same tendency is present in SRs published by researchers at the two institutions our library serves: The University of Oslo and Oslo University Hospital. In addition, we explore the prevalence of systematic reviews compared with other types of reviews published by researchers affiliated with these institutions. The Medical Library at the University of Oslo is about to establish a service for researchers embarking on a systematic review. Using published research from our own researchers will give us a solid knowledge base when discussing and deciding how our library’s contribution to SRs is best invested.

In this study, we explore the prevalence of systematic reviews and librarians’ involvement in them, and further, investigate whether librarian co-authored systematic reviews are associated with lower risk of bias compared to SRs with acknowledgement of librarians or no participation by librarians.

Methods

Publication of systematic reviews have increased rapidly since the beginning of the 1990s (Ioannidis, 2016). This increase was also observed in our institutions. Before 2006, our library did not receive many requests from researchers for help with literature searches for SRs. This analysis therefore includes systematic reviews published between 2007 and mid-2018. To collect systematic reviews, we conducted a search in Scopus on 28 June 2018 (Appendix A). We searched for Oslo in the tab Affiliations, where we found 15 University of Oslo or Oslo University Hospital affiliations to include in our analysis. Further, the search was limited to document type review and source type journal. With this search strategy, we could also get an understanding of the types of reviews our researchers were undertaking. A research assistant manually divided the Scopus search results into review categories (systematic and non-systematic) based on how the review authors named the review. No assessment of the method was done to evaluate if the systematic review was conducted in accordance with a specific handbook. We included systematic reviews that had at least one author affiliated with the University of Oslo or Oslo University Hospital. The following publication types were excluded: method articles (systematic review as topic), comments, letters, editorials, guidelines, book reviews, and errata.

To analyse the librarians’ involvement, two investigators reviewed the articles in full-text. We read the methods section, the acknowledgements section, and the author affiliation section in all the systematic reviews. Additionally, we searched for librar or specialist or thank in the PDFs in case a librarian or information specialist was mentioned elsewhere in the document. We also examined search strategies when available.

The publications were divided into three categories: 1) librarian co-author; 2) librarian mentioned in the methods section or acknowledged; 3) librarian not mentioned. We then assessed the quality of the literature searches in a selection of 12 publications, four from each category using the Risk of Bias in Systematic Reviews (ROBIS) tool (Whiting et al., 2016). The rationale behind choosing four publications from each category was that only four publications had a librarian co-author, and we wanted an equal sample from each of the categories. Based on the publication years of these four reports, we used a random number generator (Haahr, 2019) to randomly select four publications with the same publication year from each of the other two categories: librarian mentioned or acknowledged, and librarian not mentioned^{^[1]}. Domain 2 of ROBIS “Identification and selection of studies”, questions 2.1-2.4 aims to assess whether any primary studies that would have met the inclusion criteria were missed (Whiting et al., 2016, Appendix A). The ROBIS (Whiting et al., 2016, Appendix A) questions concerning the search strategies are:

Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
Were methods additional to database searching used to identify relevant reports?
Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
Were restrictions based on date, publication format, or language appropriate?

There is also a question about efforts to minimise error in selection of studies, but we did not apply this question in our assessment. The ROBIS assessment was based on the information available in the article or as supplementary material.

Results

We identified a total of 2,737 reviews published by an author affiliated with the University of Oslo or Oslo University Hospital. The reviews were categorized manually by a research assistant. Of the 2,737 reviews, 324 (11.84%) were systematic reviews as defined by the review authors. Figure 1 reports the number of published SRs per year from 2007 to mid-2018. The figure shows that SRs have increased every year, except in 2010, with a sharp increase from 2014, and the plateau has not yet been reached.

Figure 1

Systematic reviews per year with author affiliated to University of Oslo or Oslo University Hospital. Note that 2018 only contains data up to June 28.

Table 1

Number and Percent of Librarians’ Involvement

Librarians’ involvement	Co-authors	Acknowledged or mentioned in methods section	No evidence of librarian involvement	Total number of SRs
Number & percent	4 (1.23 %)	85 (26.23%)	235 (72.53%)	324 (100%)

The 324 SRs were analysed to uncover to what extent researchers collaborated with librarians. The results can be seen in Table 1. The involvement is distributed in the 324 SRs as follows; four SRs (1.23%) had librarian co-authors, in 85 SRs (26.23%) librarians were acknowledged or mentioned in the methods section. In the remaining 235 SRs (72.53%), there was no clear evidence that a librarian had been involved.

In our third analysis, we wanted to explore whether librarian authorship was associated with a lower risk of bias compared to systematic reviews with no librarian involved or systematic reviews where a librarian was mentioned or acknowledged but not accredited with co-authorship. We used the ROBIS tool (Whiting et al., 2016) to assess risk of bias in four publications in each of the three categories.

First, we present an overall assessment of risk of bias. As can be seen in Table 2, five of the twelve publications were judged to have an unclear risk of bias. The main reason was that the complete search strategy was not published. We contacted authors of the publications which lacked search strategies (Aas et al., 2014; Krølner et al., 2011; Manja, Saugstad, & Lakshminrusimha, 2017; Skarphedinsson et al., 2015; Sugrue, Englund, Solbrekke, & Fossland, 2018). Manja et al. (2017) sent us a description of the search but not the complete search strategy. Skarphedinsson et al. (2015) sent us the complete search strategy. The other three (Aas et al., 2014; Krølner et al., 2011; Sugrue et al., 2018), did not respond. Since we were not able to obtain all search strategies we decided to use only the published material in our ROBIS assessment. In the discussion we elaborate on the findings.

Table 2

Summary of Risk of Bias Assessments

Librarian co-author
Aune, Røislien, Mathisen, Thelle, and Otterstad (2011) “The ‘Smoker's Paradox’ in Patients with Acute Coronary Syndrome: A Systematic Review”	Ruddox et al. (2013) “Is 3D Echocardiography Superior to 2D Echocardiography in General Practice?: A Systematic Review of Studies Published Between 2007 and 2012”	Skarphedinsson et al. (2015) “Standard Individual Cognitive Behaviour Therapy for Paediatric Obsessive–Compulsive Disorder: A Systematic Review of Effect Estimates Across Comparisons”	Borgeraas, Johnson, Skattebu, Hertel, and Hjelmesæth (2018) “Effects of Probiotics on Body Weight, Body Mass Index, Fat Mass and Fat Percentage in Subjects with Overweight or Obesity: A Systematic Review and Meta‐Analysis of Randomized Controlled Trials”
Low risk of bias	Low risk of bias	Unclear risk of bias	Low risk of bias
Librarian mentioned in methods section or acknowledged
Burger, Kornør, Klemp, Lauvrak, and Kristiansen (2011) “HPV mRNA Tests for the Detection of Cervical Intraepithelial Neoplasia: A Systematic Review”	Roe, Soberg, Bautz-Holter, and Ostensjo (2013) “A Systematic Review of Measures of Shoulder Pain and Functioning using the International Classification of Functioning, Disability and Health (ICF)”	Aamotsmo and Bugge (2014) “Balance Artistry: The Healthy Parent’s Role in the Family When the Other Parent is in the Palliative Phase of Cancer — Challenges and Coping in Parenting Young Children”	Sugrue et al. (2018) “Trends in the Practices of Academic Developers: Trajectories of Higher Education?”
High risk of bias	High risk of bias	High risk of bias	Unclear risk of bias
Librarian not mentioned
Krølner et al. (2011). “Determinants of Fruit and Vegetable Consumption Among Children and Adolescents: A Review of the Literature. Part II: Qualitative Studies”	Robsahm et al. (2013) “Body Mass Index, Physical Activity, and Colorectal Cancer by Anatomical Subsites: A Systematic Review and Meta-Analysis of Cohort Studies”	Aas et al. (2014) “A Systematic Review of Cognitive Function in First-Episode Psychosis, Including a Discussion on Childhood Trauma, Stress, and Inflammation”	Manja et al. (2017) “Oxygen Saturation Targets in Preterm Infants and Outcomes at 18-24 Months: A Systematic Review”
Unclear risk of bias	High risk of bias	Unclear risk of bias	Unclear risk of bias

Here we present a more thorough presentation of the reasoning behind the rating. The full assessment of each publication is presented in Appendix B.

ROBIS question 2.1. Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?

According to the ROBIS guidance on how to rate each question, it is anticipated that a minimum of searches in MEDLINE and Embase should be conducted. In addition, ROBIS requires electronic searches for unpublished studies. Searches of material published as conference reports should be considered along with a search of research registries (e.g. ClinicalTrials.gov). Conference reports can have preliminary results and may therefore be considered as a source for unpublished reports, but we considered this to be insufficient as a source for unpublished studies. We answered this question with “Yes” when electronic searches of at least MEDLINE and Embase, or other databases that include conference reports and unpublished material, such as trials registries, were searched. Searches that included MEDLINE and other databases with conference reports, but no other unpublished sources, were judged as “Probably Yes”. A summary of the ratings for this question is presented in Table 3.

Table 3

Summary of Ratings for ROBIS Question 2.1. Did the Search Include an Appropriate Range of Databases/Electronic Sources for Published and Unpublished Reports?

Librarian co-author
Aune et al. (2011)	Ruddox et al. (2013)	Skarphedinsson et al. (2015)	Borgeraas et al. (2018)
Probably Yes	Probably Yes	Yes	Yes
Librarian mentioned in methods section or acknowledged
Burger et al. (2011)	Roe et al. (2013)	Aamotsmo and Bugge (2014)	Sugrue et al. (2018)
Probably Yes	Probably Yes	Probably Yes	Probably Yes
Librarian not mentioned
Krølner et al. (2011)	Robsahm et al. (2013)	Aas et al. (2014)	Manja et al. (2017)
Probably Yes	No	Probably No	No Information

Librarian co-author

All authors searched MEDLINE and Embase, which satisfies the minimum requirement set by ROBIS. We consider Embase to be a satisfactory source for conference reports. However, only Skarphedinsson et al. (2015) and Borgeraas et al. (2018) searched for unpublished reports. See Table 4 for a full list of databases searched.

Table 4

Databases Searched in Publications with a Librarian Co-author

Aune et al. (2011)

Ruddox et al. (2013)

Skarphedinsson et al. (2015)

Borgeraas et al. (2018)

· MEDLINE

· Embase

· CENTRAL

· MEDLINE

· Embase

· PubMed (ahead of print)

· MEDLINE

· Embase

· PsycINFO

· AMED

· CENTRAL

· PubMed

· LILACS

· ClinicalTrials.gov

· WHO International Clinical Trials

· ISRCTN Register

· OpenGrey

· MEDLINE

· Embase

· CENTRAL

· ClinicalTrials.gov

Librarian mentioned in methods section or acknowledged

Burger et al. (2011) and Roe et al. (2013) searched the minimum requirement of MEDLINE and Embase. Aamotsmo and Bugge (2014) did not search Embase, but our judgement is that since this is a SR of qualitative studies, the databases chosen are more relevant than Embase. Sugrue et al. (2018) did not search MEDLINE or Embase, however, considering that the topic of the systematic review was higher education, we judge the databases searched to be more relevant. All authors searched at least one database with conference abstracts (e.g. Embase, CINAHL, PsycINFO). Burger et al. (2011), Roe et al. (2013), and Aamotsmo and Bugge (2014) did not search for unpublished reports. We cannot determine if the EBSCO search performed by Sugrue et al. (2018) included databases of unpublished reports. See Table 5 for a full list of databases searched.

Table 5

Databases Searched in Publications where Librarian was Mentioned in Methods Section or Acknowledged

Burger et al. (2011)

Roe et al. (2013)

Aamotsmo and Bugge (2014)

Sugrue et al. (2018)

· MEDLINE

· Embase

· Cochrane Library

· MEDLINE

· Embase

· PEDro

· CINAHL

· CENTRAL

· MEDLINE

· PsycINFO

· CINAHL

· Australian Education Index

· ERIC

· PsycINFO

· EBSCO (database(s) not specified)

Librarian not mentioned

Krølner et al. (2011) searched a wide range of databases including the minimum requirement of MEDLINE and Embase. Several of the databases they searched include conference abstracts. However, they did not do any electronic searches for unpublished studies. Aas et al. (2014) searched PubMed and PsycINFO. These are relevant to the subject, but based on the ROBIS criteria they should also have searched Embase. They did no electronic searches for unpublished studies. Robsahm et al. (2013) only searched PubMed. The article by Manja et al. (2017) does not have a description of the literature search or search strategy, therefore cannot be assessed. See Table 6 for a full list of databases searched.

Table 6

Databases Searched in Publications where Librarian was not Mentioned

Krølner et al. (2011)

Robsahm et al. (2013)

Aas et al. (2014)

Manja et al. (2017)

· MEDLINE

· Embase

· Anthropology Plus

· CINAHL

· CSA illumine (including ERIC)

· Econlit

· Sociological abstracts

· Social Services abstracts

· Worldwide Political Science abstracts

· International Bibliography of the Social sciences

· PsycINFO

· Web of Science

· PubMed

· PsycINFO

· No information of databases searched

ROBIS question 2.2. Were methods additional to database searching used to identify relevant reports?

According to ROBIS, the authors should use methods additional to electronic searches to identify relevant reports such as reference checking or contacting experts. A summary of the ratings for this question is presented in Table 7.

Table 7

Summary of Ratings for ROBIS Question 2.2. Were Methods Additional to Database Searching Used to Identify Relevant Reports?

Librarian co-author
Aune et al. (2011)	Ruddox et al. (2013)	Skarphedinsson et al. (2015)	Borgeraas et al. (2018)
Yes	Yes	Yes	Yes
Librarian mentioned in methods section or acknowledged
Burger et al. (2011)	Roe et al. (2013)	Aamotsmo and Bugge (2014)	Sugrue (2018)
No Information	No Information	Yes	No Information
Librarian not mentioned
Krølner et al. (2011)	Robsahm et al. (2013)	Aas et al. (2014)	Manja at al. (2017)
No Information	Yes	No Information	No Information

Librarian co-author

In all of the four publications co-authored by librarians, Aune et al. (2011), Ruddox et al. (2013), Skarphedinsson et al. (2015) and Borgeraas et al. (2018), reference lists were checked for additional studies.

Librarian mentioned in methods section or acknowledged

Aamotsmo and Bugge (2014) checked reference lists. The other three, Burger et al. (2011), Roe et al. (2013) and Sugrue et al. (2018), mentioned no other methods to identify relevant reports.

Librarian not mentioned

Robsahm et al. (2013) checked reference lists, while Krølner et al. (2011), Aas et al. (2014) and Manja et al. (2017) mentioned no other methods to identify relevant reports.

ROBIS question 2.3. Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?

According to the ROBIS guidance, a full search strategy that can be replicated is needed to fully assess this question. A summary of the ratings for this question is presented in Table 8.

Table 8

Summary of Ratings for ROBIS Question 2.3. Were the Terms and Structure of the Search Strategy Likely to Retrieve as Many Eligible Studies as Possible?

Librarian co-author
Aune et al. (2011)	Ruddox et al. (2013)	Skarphedinsson et al. (2015)	Borgeraas et al. (2018)
Yes	Yes	No Information	Yes
Librarian mentioned in methods section or acknowledged
Burger et al. (2011)	Roe et al. (2013)	Aamotsmo and Bugge (2014)	Sugrue et al. (2018)
Yes	Yes	No Information	No Information
Librarian not mentioned
Krølner et al. (2011)	Robsahm et al. (2013)	Aas et al. (2014)	Manja et al. (2017)
No Information	Probably No	Probably No	No Information

Librarian co-author

Three of the publications, Aune et al. (2011), Ruddox et al. (2013) and Borgeraas et al. (2018), used a combination of controlled terms and text words appropriately. Skarphedinsson et al. (2015) states that they searched for “an extensive list of synonyms for OCD, CBT, children and adolescents”, but since the full strategy is not attached to the article, we were not able to judge the comprehensiveness of the search. We received the search strategies from one of the librarian co-authors, and if the search strategy in Skarphedinsson et al. (2015) had been published, the answer to this question would have been “Yes”, and the overall judgement would have been low risk of bias.

Librarian mentioned in methods section or acknowledged

Two of the publications, Burger et al. (2011) and Roe et al. (2013), used a combination of controlled terms and text words appropriately. Aamotsmo and Bugge (2014) and Sugrue et al. (2018) list relevant search terms but lack full search strategy and therefore cannot be assessed properly.

Librarian not mentioned

Since Manja et al. (2017) does not describe or publish their search strategy, it cannot be assessed. Aas et al. (2014) presents relevant search terms but not the full search strategy to be assessed. Krølner et al. (2011) describes text words used in all databases. Our judgement is that these terms are likely to retrieve many relevant studies, however since the full search strategy is not available we cannot assess it fully. Robsahm et al. (2013) presents the search strategy from PubMed. It contains only text words and it is difficult to know whether or not these were mapped to MeSH. In addition to terms for the topic, they have searched for terms to identify study designs, but in our opinion they should have searched for variations of these. They also restricted the search with the PubMed humans filter, which is based on MeSH and thus excluded all articles not indexed with MeSH.

ROBIS question 2.4. Were restrictions based on date, publication format, or language appropriate?

ROBIS guidance states that no restrictions should be made to the search strategy in order to answer this question with “Yes”, and that information on all three components is necessary to fully judge this question. None of the publications made any restrictions to publication format, but some were made based on language and date. We judged language restrictions to “Probably No” instead of “No” if they included more languages than English (e.g. Norwegian, Danish, Swedish, and German). We judged unjustified date restrictions as “No”. A summary of the ratings for this question is presented in Table 9.

Table 9

Summary of Ratings for ROBIS Question 2.4. Were Restrictions Based on Date, Publication Format, or Language Appropriate?

Librarian co-author
Aune et al. (2011)	Ruddox et al. (2013)	Skarphedinsson et al. (2015)	Borgeraas et al. (2018)
Probably No	Probably No	Yes	Yes
Librarian mentioned in methods section or acknowledged
Burger et al. (2011)	Roe et al. (2013)	Aamotsmo and Bugge (2014)	Sugrue et al. (2018)
No	No	No	No Information
Librarian not mentioned
Krølner et al. (2011)	Robsahm et al. (2013)	Aas et al. (2014)	Manja et al. (2017)
Yes	No	No Information	No Information

Librarian co-author

Borgeraas et al. (2018) did not make any limitations to the search. Two of the publications, Aune et al. (2011) and Ruddox et al. (2013), restricted the search to English, German, Norwegian, Danish, and Swedish language. Ruddox et al. (2013) also restricted their search to identify studies between 2007-2012, however, this was justified in the introduction. Skarphedinsson et al. (2015) state that they have not done any language restrictions.

Librarian mentioned in methods section or acknowledged

Sugrue et al. (2018) lacks search strategy and does not describe any restrictions in the methods section, hence there is not enough information to assess this question. In three of the publications, Aamotsmo and Bugge (2014), Burger et al. (2011), and Roe et al. (2013), restrictions were made to publication date, none of which were justified in the article. Roe et al. (2013) also restricted the search to English, Norwegian, Danish, and Swedish language. Burger et al. (2011) had no actual restrictions on language in the search strategy even though they state in the methods chapter that publications was restricted to English and Scandinavian languages.

Librarian not mentioned

Robsahm et al. (2013) made restrictions to English language. One of the publications, Krølner et al. (2011), state that they made no restrictions to either publication year or to language. Aas et al. (2014) and Manja et al. (2017) do not mention any restrictions.

Discussion

The popularity of SRs has grown immensely in the past decades (Ioannidis, 2016). Many research funders now request a systematic review of previous research as a part of grant applications, and PhD candidates at the University of Oslo are now allowed – and encouraged to – include one SR in their thesis which contributes to the increase. Our first analysis shows that although the majority of reviews by researchers at the University of Oslo and Oslo University Hospital are non-systematic, the number of SRs are growing and continue to do so.

Our second analysis shows that librarians’ involvement in SRs at the two institutions seems to be rather low. There could be cases where the review authors did get help from a librarian, but failed to mention this. In other cases, they have not involved a librarian at all. There could be several reasons for this, one being that they are not aware of the services the library offers. Some researchers are experienced searchers, and do the searching and documentation themselves. However, as our third analysis shows, the risk of bias is higher when a librarian is not a co-author. Compared to the results from Ross-White (2016), librarian involvement was much lower in our sample. Of the 31 librarian co-authored SRs in the Ross-White sample, as many as 19 were co-authored by a nursing liaison librarian, showing that they have succeeded in advocating for co-authorship in the School of Nursing. Based on this, we see that there is room for a substantial improvement in our collaboration with systematic review authors.

Due to our small sample size, we are not able to draw strong conclusions, but we can observe that overall there are some positive differences when a librarian co-authors the article. In the analysed publications, search strategies are more comprehensive and in particular the documentation is better when a librarian is co-author.

In the publications where a librarian was only mentioned or acknowledged, varying comprehensiveness of search strategies and some lack of documentation resulted in risk of bias from unclear to high risk of bias. The high risk of bias judgement of three publications was mainly due to unjustified date restrictions which may have resulted in relevant studies being missed.

When a librarian was not mentioned in the review, the reporting of search strategies was far less complete. Three of the publications did not report the actual search strategy, and the one that did, only searched PubMed. Manja et al. (2017) did send us a description of the search method, but were unable to provide the actual search strategy.

An explanation as to the lack of published search strategies may be that authors or publishers are not aware of the importance of the search strategy for assessment of the validity of the report and therefore fail to include it in the final article or appendices.

The results from this study are in accordance with the results from Rethlefsen et al. (2015) and Meert et al. (2016) which showed that librarian and information specialist co-authored SRs have better reported search strategies and search documentation than SRs with acknowledgement of librarians or no participation by librarians or information specialists. In order to assess the internal validity of the SR, the search strategies must be documented and reported so that they can be reproduced (Aromataris & Munn, 2017; Higgins et al., 2017; Moher et al., 2009; The Campbell Collaboration, 2019; Whiting et al., 2016). In studies where librarians only perform the search and advise the authors, but are not co-authors, the reproducibility of searches is not ensured. These results are important when planning SR services with regards to database selection, avoidance of restrictions, and documentation and publication of search strategies.

A recent study by Cooper, Booth, Varley-Campbell, Britten, and Garside (2018) shows that information specialists, librarians, or trial search coordinators are pointed out as appropriate researchers in no less than six guidance documents. In the Joanna Briggs Systematic Reviews manual, it is stated that “Dependent upon the type of review being conducted review teams should ideally consist of members with […] an information scientist or research librarian with specialised skills to develop and implement a comprehensive search strategy” (as cited in Aromataris & Munn, 2017). Considering the results from our study and the studies by Rethlefsen et al. (2015) and Meert et al. (2016), we support this recommendation. Medical research has been criticized for being wasteful (Chalmers & Glasziou, 2009), and librarians providing research support should ensure that robust research is conducted in their institutions, with respect to searching and documentation of searches.

We acknowledge some limitations in this study. The main limitation being the small sample size of the second and third analyses. Our search for SRs to include was limited to publications between 2007 and mid-2018 representing a decade of reviews. However, there is a possibility that researchers from our institutions may have published SRs with librarians as co-authors prior to 2007, which could have given us a larger sample size. To be able to generalize on this matter, in the future, the same analysis with a larger sample size, not restricted to researchers from two institutions, could be performed. In this study we aimed to examine to what extent researchers have collaborated with a librarian in the SR process. That is, on the basis of what is stated in the publication. Some researchers might have had help from a librarian but for various reasons this has not been mentioned.

Conclusion

The number of SRs are increasing, but narrative and other types of reviews remain the core of reviews by researchers from University of Oslo and Oslo University Hospital.

Librarians are rarely involved as co-authors of SRs at the two institutions in our sample. In a quarter of the analysed SRs, a librarian was acknowledged or mentioned in the methods section. In the majority of the SRs, there were no clear evidence that a librarian had been in involved.

Due to the small sample size, we cannot draw strong conclusions about the risk of bias in SRs with or without librarian involvement. However, we observed that co-authored SRs have more comprehensive search strategies and better documentation, and have a lower risk of bias compared to SRs with acknowledgement or no participation by librarians.

Implications for practice

As can be observed from this and previous studies, librarian co-authorship improves the quality of searches and reporting in SRs. This implies that librarians should advocate for co-authorship.

The strengths of the librarian co-author lie in translating the objectives into a searchable question, selecting relevant databases, identifying subject headings and text words, planning search strategies and translating them to various databases, executing and documenting the searches and ensuring that search strategies are published with the article. Furthermore, the librarian should describe the search in the methods section of the systematic review.

Librarians should also aim to spread knowledge about our competence, and the positive implications of librarian co-authorship in SRs in medical and health journals and at conferences attended by SR authors (e.g. Cochrane and Campbell Colloquia, HTAi annual meetings).

Library associations such as the Medical Library Association and the European Association for Health Information and Libraries can encourage librarians to aim for co-authorship. They can also liaise with journal editors in order to ensure that search strategies are peer reviewed and published with the article.

Acknowledgements

The authors thank and acknowledge Marte Ødegaard for her valuable contribution with the search strategy, Ole Martin Hagtvedt Holte for valuable assistance with categorising the articles into different types of reviews, Marte J. Søyland for valuable help with data management, and Therese Skagen for valuable contributions in the writing process. Thanks to Veena Manja and Sølvi Biedilæ for additional information about unpublished search strategies in their publications, and to Vanessa Fleming for proof reading.

Funding

This article is produced by the project “Fagbibliotek og systematiske oversikter” supported by the National Library of Norway.

References

Aamotsmo, T., & Bugge, K. E. (2014). Balance artistry: The healthy parent's role in the family when the other parent is in the palliative phase of cancer — Challenges and coping in parenting young children. Palliative & Supportive Care, 12(4), 317-329. https://doi.org/10.1017/S1478951513000953

Aas, M., Dazzan, P., Mondelli, V., Melle, I., Murray, R. M., & Pariante, C. M. (2014). A systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation. Frontiers in Psychiatry, 4, 1-13. https://doi.org/10.3389/fpsyt.2013.00182

Aromataris, E., & Munn, Z. (2017). JBI Systematic Reviews. Retrieved from https://reviewersmanual.joannabriggs.org/

Aune, E., Røislien, J., Mathisen, M., Thelle, D. S., & Otterstad, J. E. (2011). The "smoker's paradox" in patients with acute coronary syndrome: A systematic review. BMC Medicine, 9, 1-11. https://doi.org/10.1186/1741-7015-9-97

Borgeraas, H., Johnson, L. K., Skattebu, J., Hertel, J. K., & Hjelmesæth, J. (2018). Effects of probiotics on body weight, body mass index, fat mass and fat percentage in subjects with overweight or obesity: A systematic review and meta-analysis of randomized controlled trials. Obesity Reviews, 19(2), 219-232. https://doi.org/10.1111/obr.12626

Burger, E. A., Kornør, H., Klemp, M., Lauvrak, V., & Kristiansen, I. S. (2011). HPV mRNA tests for the detection of cervical intraepithelial neoplasia: A systematic review. Gynecologic Oncology, 120(3), 430-438. https://doi.org/10.1016/j.ygyno.2010.11.013

Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374(9683), 86-89.

https://doi.org/10.1016/s0140-6736(09)60329-9

Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018). Defining the process to literature searching in systematic reviews: A literature review of guidance and supporting studies. BMC Medical Research Methodology, 18, 1-14. https://doi.org/10.1186/s12874-018-0545-3

Golder, S., Loke, Y., & McIntosh, H. M. (2008). Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology, 61(5), 440-448. https://doi.org/10.1016/j.jclinepi.2007.06.005

Haahr, M. (2019). RANDOM.ORG: True Random Number Service. Retrieved from https://www.random.org

Higgins, JPT, Churchill R, Chandler J, Cumpston MS (editors), Cochrane handbook for systematic reviews of interventions version 5.2.0 (updated June 2017). Cochrane, 2017. Available from training.cochrane.org/handbook

Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly, 94(3), 485-514. https://doi.org/10.1111/1468-0009.12210

Koffel, J. B., & Rethlefsen, M. L. (2016). Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: A cross-sectional study. PLoS One, 11(9), 1-16. https://doi.org/10.1371/journal.pone.0163309

Krølner, R., Rasmussen, M., Brug, J., Klepp, K. I., Wind, M., & Due, P. (2011). Determinants of fruit and vegetable consumption among children and adolescents: A review of the literature. Part II: Qualitative studies. International Journal of Behavioral Nutrition and Physical Activity, 8, 1-38. https://doi.org/10.1186/1479-5868-8-112

Manja, V., Saugstad, O. D., & Lakshminrusimha, S. (2017). Oxygen saturation targets in preterm infants and outcomes at 18-24 Months: A systematic review. Pediatrics, 139(1). https://doi.org/10.1542/peds.2016-1609

Meert, D., Torabi, N., & Costella, J. (2016). Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association, 104(4), 267-277.

https://doi.org/10.3163/1536-5050.104.4.004

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., Altman, D., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

Patsopoulos, N. A., Analatos, A. A., & Ioannidis, J. P. (2005). Relative citation impact of various study designs in the health sciences. JAMA, 293(19), 2362-2366. https://doi.org/10.1001/jama.293.19.2362

Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology, 68(6), 617-626. https://doi.org/10.1016/j.jclinepi.2014.11.025

Robsahm, T. E., Aagnes, B., Hjartåker, A., Langseth, H., Bray, F. I., & Larsen, I. K. (2013). Body mass index, physical activity, and colorectal cancer by anatomical subsites: A systematic review and meta-analysis of cohort studies. European Journal of Cancer Prevention, 22(6), 492-505. https://doi.org/10.1097/CEJ.0b013e328360f434

Roe, Y., Soberg, H. L., Bautz-Holter, E., & Ostensjo, S. (2013). A systematic review of measures of shoulder pain and functioning using the International Classification of Functioning, Disability and Health (ICF). BMC Musculoskeletal Disorders, 14, 1-12. https://doi.org/10.1186/1471-2474-14-73

Ross-White, A. (2016). Librarian involvement in systematic reviews at Queen's University: An environmental scan. Journal of the Canadian Health Libraries Association / Journal De L’Association Des Bibliothèques de La Santé Du Canada, 37(2), 39-43.

https://doi.org/10.5596/c16-016

Ruddox, V., Mathisen, M., Bækkevar, M., Aune, E., Edvardsen, T., & Otterstad, J. E. (2013). Is 3D echocardiography superior to 2D echocardiography in general practice? A systematic review of studies published between 2007 and 2012. International Journal of Cardiology, 168(2), 1306-1315. https://doi.org/10.1016/j.ijcard.2012.12.002

Skarphedinsson, G., Hanssen-Bauer, K., Kornør, H., Heiervang, E. R., Landrø, N. I., Axelsdottir, B., . . . Ivarsson, T. (2015). Standard individual cognitive behaviour therapy for paediatric obsessive-compulsive disorder: A systematic review of effect estimates across comparisons. Nordic Journal of Psychiatry, 69(2), 81-92. https://doi.org/10.3109/08039488.2014.941395

Sugrue, C., Englund, T., Solbrekke, T. D., & Fossland, T. (2018). Trends in the practices of academic developers: Trajectories of higher education? Studies in Higher Education, 43(12), 2336-2353. https://doi.org/10.1080/03075079.2017.1326026

The Campbell Collaboration. (2019). Campbell Systematic Reviews: Policies and guidelines. https://doi.org/10.4073/cpg.2016.1

Toews, L. C. (2017). Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association, 105(3), 233-239. https://doi.org/10.5195/jmla.2017.246

Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., . . . ROBIS Group (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225-234. https://doi.org/10.1016/j.jclinepi.2015.06.005

Appendix A

Search Strategy SCOPUS

Date searched: 28.06.2018
Number of retrieved references: 2,733
Limits: Source type: Journal articles, Year: 2007-2018

Search strategy:

AF-ID("Universitetet i Oslo" 60010348) OR AF-ID("Rikshospitalet-Radiumhospitalet HF" 60026108) OR AF-ID("Oslo University Hospital" 60105037) OR AF-ID("Ulleval University Hospital" 60068729) OR AF-ID("Aker University Hospital" 60091868) OR AF-ID("University of Oslo Faculty of Medicine" 60003938) OR AF-ID("University of Oslo Institute for Surgical Research" 60069106) OR AF-ID("Mineralogisk-Geologisk Museum" 60071308) OR AF-ID("Research Institute of Internal Medicine" 60069109) OR AF-ID("Ragnar Frisch Centre for Economic Research" 60009748) OR AF-ID("Paleontologisk Museum" 60071309) OR AF-ID("Olafiaklinikken" 60080266) OR AF-ID("University of Oslo Faculty of Humanities" 60080171) OR AF-ID("Oslo Heart Center" 60069110) OR AF-ID("Universitet i Oslo Johan Throne Holst's Institutt for Ernaeringsforsking" 60102103) AND ((TITLE(review) OR ABS(review))) AND ( LIMIT-TO ( PUBYEAR,2018 ) OR LIMIT-TO ( PUBYEAR,2017 ) OR LIMIT-TO ( PUBYEAR,2016 ) OR LIMIT-TO ( PUBYEAR,2015 ) OR LIMIT-TO ( PUBYEAR,2014 ) OR LIMIT-TO ( PUBYEAR,2013 ) OR LIMIT-TO ( PUBYEAR,2012 ) OR LIMIT-TO ( PUBYEAR,2011 ) OR LIMIT-TO ( PUBYEAR,2010 ) OR LIMIT-TO ( PUBYEAR,2009 ) OR LIMIT-TO ( PUBYEAR,2008 ) OR LIMIT-TO ( PUBYEAR,2007 ) ) AND ( LIMIT-TO ( SRCTYPE,"j" ) )

Appendix B

Publication Assessments

SRs with Librarian as Co-author

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Aune et al. 2011 The "smoker's paradox" in patients with acute coronary syndrome: a systematic review.	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	MEDLINE, Embase, CENTRAL. Did not search ClinicalTrials.gov or any other trials registries.
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists.
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Yes	Combination of controlled terms and text words.
	2.4 Were restrictions based on date, publication format, or language appropriate?	Probably No	Language restrictions, but not only English (Danish or English or German or Norwegian or Swedish).
	Concerns regarding methods used to identify and/or select studies:	LOW	Only one potential risk of bias, language restriction, was identified. Our judgment is that this does not exclude too many relevant articles, and since all the other questions were answered Yes or Probably Yes we concluded with low risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Ruddox et al. 2013 Is 3D echocardiography superior to 2D echocardiography in general practice?: A systematic review of studies published between 2007 and 2012	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	MEDLINE, Embase, PubMed (ahead of print). No searches for unpublished reports.
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists.
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Yes	Combination of controlled terms and text words.
	2.4 Were restrictions based on date, publication format, or language appropriate?	Probably No	Language restrictions, but not only English (Danish or English or German or Norwegian or Swedish). Date restriction 2007-2011 was justified.
	Concerns regarding methods used to identify and/or select studies:	LOW	Only one potential risk of bias, language restriction, was identified. Our judgment is that this does not exclude too many relevant articles, and since all the other questions were answered Yes or Probably Yes we concluded with low risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Skarphedinsson et al. 2014 Standard individual cognitive behaviour therapy for paediatric obsessive–compulsive disorder: A systematic review of effect estimates across comparisons	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Yes	MEDLINE, Embase, PsycINFO, AMED, CENTRAL, PubMed, LILACS, ClinicalTrials.gov, WHO International Clinical Trials, ISRCTN Register, OpenGrey.
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists.
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	No Information	Search strategy not attached to article. Was later retrieved from one of the authors.
	2.4 Were restrictions based on date, publication format, or language appropriate?	Yes	None made.
	Concerns regarding methods used to identify and/or select studies:	UNCLEAR	There is insufficient information reported to make a judgement on risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		Comments
Borgeraas et al. 2017 Effects of probiotics on body weight, body mass index, fat mass and fat percentage in subjects with overweight or obesity: a systematic review and meta‐analysis of randomized controlled trials	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Yes	MEDLINE, Embase, CENTRAL, ClinicalTrials.gov
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Yes	Combination of controlled terms and text words. Appropriate use of filters.
	2.4 Were restrictions based on date, publication format, or language appropriate?	Yes	None made.
	Concerns regarding methods used to identify and/or select studies:	LOW	No potential areas of bias identified

SRs with Librarian Lentioned in Method Section or Acknowledged

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Burger et al 2011 HPV mRNA tests for the detection of cervical intraepithelial neoplasia: A systematic review	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	MEDLINE, Embase, Cochrane Library. Did not search for unpublished reports.
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Yes	Combination of controlled terms and text words. Appropriate use of filters.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No	No actual restrictions on language in the search strategy even though the methods chapter says that it was restricted to English and Scandinavian. Date restriction from 1996 not explained.
	Concerns regarding methods used to identify and/or select studies:	HIGH	This was in between low and high risk of bias. We still judged it as high because of the unjustified date restriction.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES
Roe et al 2013 A systematic review of measures of shoulder pain and functioning using the International classification of functioning, disability and health (ICF)	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	MEDLINE, Embase, PEDro, CINAHL, CENTRAL. No electronic searches for unpublished reports.
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Yes	Combination of controlled terms and text words.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No	Limitation to only five years 2005-2010, is not justified. Restricted to English and Scandinavian languages.
	Concerns regarding methods used to identify and/or select studies:	HIGH	Judged to be of high risk of bias because of the strict date restriction. The tool in question was published in 2001.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES
Aamotsmo et al 2014 Balance artistry: The healthy parent’s role in the family when the other parent is in the palliative phase of cancer — Challenges and coping in parenting young children	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	MEDLINE, PsycINFO, CINAHL. No sources for unpublished studies mentioned.
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists.
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	No information	Request for complete search strategy sent to Aamotsmo 1/2/19 No complete search strategy available. Terms listed but no information about subject headings/text words. Only 182 hits.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No	Date restriction 1989-2009, is not justified. We have reason to believe that the topic was also relevant earlier than 1989.
	Concerns regarding methods used to identify and/or select studies:	HIGH	Because of the date restriction and low number of returned hits, we consider methods used to identify studies to be of high risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Sugrue et al 2017 Trends in the practices of academic developers: trajectories of higher education?	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	Australian Education Index, ERIC, PsycINFO. Also says that EBSCO was searched separately without specifying which EBSCO database.
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	No Information	Request for complete search strategy sent to Sugrue 1/2/19.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No Information
	Concerns regarding methods used to identify and/or select studies:	UNCLEAR	There is insufficient information reported to make a judgement on risk of bias.

SRs with Librarian not Mentioned

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Krølner et al 2011 Determinants of fruit and vegetable consumption among children and adolescents: a review of the literature. Part II: qualitative studies	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably Yes	Anthropology Plus, CINAHL, CSA illumine (including ERIC (1966 onwards), Econlit (1969 onwards), Sociological abstracts (1952 onwards), Social Services abstracts (1979 onwards), Worldwide political Science abstracts (1975 and onwards), Embase, International Bibliography of the Social sciences, Medline, PsycINFO, Web of Science. No sources for unpublished studies searched.
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	No Information	Full search strategy not attached, but well reported in the article. Only text words, no subject terms. Restriction to study design was justified.
	2.4 Were restrictions based on date, publication format, or language appropriate?	Yes	No restrictions to the search mentioned.
	Concerns regarding methods used to identify and/or select studies:	UNCLEAR	There is insufficient information reported to make a judgement on risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Robsahm et al 2013 Body mass index, physical activity, and colorectal cancer by anatomical subsites: A systematic review and meta-analysis of cohort studies	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	No	Only PubMed.
	2.2 Were methods additional to database searching used to identify relevant reports?	Yes	Reference lists.
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Probably No	Text words only (probably mapped to MeSH terms). Filter for study design is too restricted, and MeSH filter for humans- excluding not indexed publications.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No	English only.
	Concerns regarding methods used to identify and/or select studies:	HIGH	Because only one database was searched, too restricted filters, and language restriction we consider methods used to identify studies to be of high risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Aas et al 2014 A systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	Probably No	PubMed, PsycINFO. No searches for unpublished reports.
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Probably No	Request for complete search strategy sent to Aas 1/2/19.
	2.4 Were restrictions based on date, publication format, or language appropriate?	No Information	Mention no restrictions.
	Concerns regarding methods used to identify and/or select studies:	UNCLEAR	There is insufficient information reported to make a judgement on risk of bias.

REFERENCE	ROBIS DOMAIN 2: IDENTIFICATION AND SELECTION OF STUDIES		COMMENTS
Manja et al 2017 Oxygen saturation targets in preterm infants and outcomes at 18-24 months: A systematic review	2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?	No Information
	2.2 Were methods additional to database searching used to identify relevant reports?	No Information
	2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	No Information	Request for complete search strategy sent to Saugstad 1/2/19 (initial request sent to corresponding author was not deliverable).
	2.4 Were restrictions based on date, publication format, or language appropriate?	No Information
	Concerns regarding methods used to identify and/or select studies:	UNCLEAR	There is insufficient information reported to make a judgement on risk of bias.

[1] We used the online publication year, which in three cases (Borgeraas, Johnson, Skattebu, Hertel, & Hjelmesæth, 2018; Skarphedinsson et al., 2015; Sugrue, Englund, Solbrekke, & Fossland, 2018) differ from the downloaded citation.