key: cord-0009154-v2xaqo0w
authors: Tugwell, Peter; Knottnerus, J. André
title: Can we measure ‘Sensationalisim’ and ‘Spin’?
date: 2016-06-18
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2016.06.001
sha: 76f6ff5bb2329dfd14355265b01d3615d225fb9f
doc_id: 9154
cord_uid: v2xaqo0w

nan

Can we measure 'Sensationalisim' and 'Spin'? Two papers address these terms, more often seen in newsstands but unusual in a clinical epidemiology journal, of: 'sensationalism' and 'spin'. Hoffman et al report on the development and validation of a method for automatically quantifying sensationalism and comparing this to relevance and scientific quality of news reports. The authors conclude that automated methods can evaluate news records faster, cheaper, and possibly better than humans. From a sample of 163,433 news records mentioning the Severe Acute Respiratory Syndrome (SARS) and H1N1 pandemics, the computer model agreed with human assessment of sensationalism for 73% of the reports, and accurately identified relevance in 86% of reports and the quality in 65%. A second article focusses on 'Spin' e perhaps a somewhat pejorative new term for the well known tendency of authors to ''overstate efficacy and/or understate harm''. In this issue, Yavchitz et al report on an intriguing study to classify 'spin' in systematic reviews. They applied a Delphi and Q sort method to establish 39 classification categories [28 in the main text and 210 in the abstract] that fall in one of the following 3 categories; (1) misleading reporting, (2) misleading interpretation, and (3) inappropriate extrapolation. The item judged most egregious was ''the conclusion formulates recommendations for clinical practice not supported by the findings''. They offer this classification of spin to authors, editors and reviewers; this might be used as a question on peer-review forms to reviewers.

The GRADE community and their publications are redefining /updating many of the clinimetric concepts and terminology of the discipline of clinical epidemiology that Feinstein [1] and Sackett [2] established in the 1960-70's. In this issue in a Variance and Dissent pair of articles Antilla et al and Schunemann debate whether the term 'conclusiveness' should be used instead of the term 'imprecision' when assessing the overall 'certainty' or 'quality' of the evidence around an outcome. Schunemann agrees that imprecision as used in GRADE does include both statistical aspects but also clinical judgement in assessing the delta of interest. Schunemann argues that clinical judgement is indeed needed for assessing imprecision both for individual outcomes and even more so when assessing imprecision across the different benefits and harms of a specific PICO question being addressed in a Guideline. Schunemann also discusses the challenges of estimating the delta/effect size [change of interest] needed to calculate the Optimal Information Size; he offers an alternative when data exists by estimating a 'Review Information Size' [RIS] that uses existing synthesized evidence to provide what is termed a 'plausible estimate' of the delta rather than a guestimated 'worthwhile' effect which is challenging to determine, in particular for single outcomes. We look forward to the views of our readers on which of the terms 'imprecision' or 'conclusiveness' are preferred by clinical epidemiologists, and consensus on how each is defined and applied. This needs to be differentiated from the separate debate within the Cochrane Collaboration as to when to use final 'conclusions' vs 'recommendations'; currently the guidance is to use the term 'conclusion' on the basis of the data -and to avoid the word 'recommendation' since the latter needs to be within the local context and available resources.

Worthington et al in reviewing dental research, provide further documentation of the component of research waste [3, 4] , resulting from the failure of investigators to publish full articles after publishing abstracts. Their systematic review of 16 studies involving 10,365 abstracts presented at 52 conferences showed that less than one third of abstracts presented at dental conferences were published in full more than four years after conference presentation. They label this as 'unethical misconduct'. We need incentives or penalties to address this such as including a commitment from authors when submitting abstracts to meetings.

A major debate in clinical epidemiology research is whether non randomised studies can be trusted; one way to assess this is to compare the effect sizes and confidence intervals between different designs addressing the same research question. Brignardello-Petersen et al report a scoping review of this in which they conducted a careful systematic search for studies published in English between 1990 and February 10, 2015, of dichotomous outcomes in which authors compared the estimates of effects from different study designs. They found 19 studies comparing non randomised studies to RCTs. They propose a list and classification (statistically-related or clinically-related) of the criteria that could be appropriately used to compare two treatment effect estimates. Consensus on this would make an important contribution to this contentious area.

Two articles report on the 'good and bad' in record linkage. The 'good' [positive] first-Millett et al using the example of community acquired pneumonia show how important that there be linked records for conditions that can be treated both in the community and in hospital. There was up to 30% underreporting when only one of these sources was used and demonstrated that the data linkage was feasible and accurate. In contrast, Moore et al demonstrate how data linkage can go wrong. They undertook both a deterministic and probabilistic linkage within a statewide administrative hospital registry compared to two cohorts of, HIV-negative (HIV-ve) and HIV-positive (HIVþve) gay and bisexual men (GBM) recruited to formal studies. They found serious underestimation of hospitalisations with the probabilistic linkage and call for insisting on demonstration of adequate sensitivity and specificity of the data linkage method before being used.

Clinical epidemiologists can be divided into those that base their clinical epidemiology research on theories [5] and others who do not [6] , preferring to be atheoretical, taking a more pragmatic approach see the JCE Variance and Dissent. In this issue, Noyes et al report that they found over 100 social theories either applied or applicable to systematic reviews on complex interventions with increasing numbers over the last 5 years. They argue that guidance is missing but urgently needed in systematic review and methods manuals to aid reviewers in assessing and using social theories in complex intervention reviews. They develop and describe a classification for grand [eg feminism], mid-range[e.g. Theory of planned behaviour] and low-level [e.g. frameworks and checklists]. They present two new searchable author resources: 'Theory in Reviews' Wiki and Mendeley Theory in Reviews Inventory.

Core Outcome Sets [COS] have been the focus of previous articles in JCE [7] e COMET data base [http://www.cometinitiative.org/studies/search/] lists an increasing number of core outcome sets recommended for use [and inclusion in registered protocols] in all trials and systematic reviews addressing similar classes of PICO questions in different clinical specialties. These are important for two main reasons-firstly that with appropriate patient input, of ensuring inclusion of outcomes important to the patient; and secondly for preventing selective outcome reporting of trials. In this issue Sautenet et al report that with renoprotection trials an excess of poorly validated different outcome instruments across 20 outcome domains were found. Most studies only reported on 2 or 3 of these 20 outcome domains; only 5 (8%] of these 66 systematic reviews reported on patient reported outcomes. This is similar to the work of Tong et al in hemodialysis [7] and transplantation and pediatric renal conditions [www.songinitiative.org].

Gaming the system: The finding in the article Dal Re et al in this issue that a quarter of trials in top medical journals are being registered retrospectively is cause for concern. The International Committee of Medical Journal Editors [9] requires, and recommends that all medical journal editors require, registration of clinical trials in a public trials registry at or before the time of first patient enrollment as a condition of consideration for publication [8] . We support the authors recommendation that when publishing results of a retrospectively registered trial, journals should notify readers and clarify why an exception was made to the policy and whether the trial may have been subject to selective outcome reporting.

Comprehensive replicable literature searching provides the evidence base that is absolutely essential for quality systematic reviews and health technology assessment reports. The PRESS Statement Update of the previous 2008 version, provides an easy to use protocol for peer review by an independent librarian scientist. Chen et al report on the dangers of evaluating biomarkers for early detection of conditions by inappropriate sampling of cases and controls in clinical settings. They demonstrate this in a study of 92 candidate cancer protein markers was measured in 35 clinically identified colorectal cancer patients and 35 colorectal cancer patients identified at screening; for each case group, 38 controls were selected without colorectal neoplasms at screening colonoscopy. The majority of the biomarkers derived from the clinical sample turned out to be false positives.

The GRADE system for developing clinical guidelines continues to be adopted by more medical groups and associations. However panels need training for them to make informed contributions to the guidelines panels. Kumar et al report on the partial success of training new guideline panel members in the GRADE process. A panel was established as part of the clinical practice guideline development process of American Association of Blood Banking (AABB) for the use of prophylactic vs. therapeutic platelet transfusion in patients with thrombocytopenia A systematic review with the evidence to recommendation frameworks containing information on four GRADE domains (confidence in estimates, balance of benefits and harms, use of resources, and patients' values and preferences) for 10 key questions re platelet transfusions, was mailed to members of the panel in advance to face-to-face meeting scheduled to formalize the practice guideline with two one-hour lectures [one webinar and one in person]. The inter-rater agreement was good for a] quality of evidence; b] making a recommendation for/against healthcare intervention; it was only fair for fair for c]balance of benefit and harms d] use of resources d]or the GRADE domain of patients' values and preferences d] strong vs weak recommendation. The authors recommend better operationalization and improvement of instruction on the above domains that did not achieve good agreement.

Is there an effect of prestige on article citations? Tanner Smith et al studied the relationships between study quality, author prestige, journal impact factors, and citation rates of trials using a sample of 128 briefalcohol-intervention trials citations from ISI Web of Knowledge and Google Scholar; Articles were more often cited when first authors had higher h-indices, even after adjusting for a range of study quality proxies. The associations between author prestige and study citation rates were mediated by journal impact factor, even after adjusting for a range of study quality. They suggest that bibliometric analyses predicting article citation rates should consider the role of author prestige and journal impact factors, in addition to study quality.

Clinical epidemiology. I. The populational experiments of nature and of man in human illness

Clinical epidemiology

Reporting of trials presented in conference abstracts needs to beimproved

To systematically review reports that queried abstract authors about reasons for not subsequently publishing abstract results as full-length articles

Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings

Variance and Dissent. The OFF theory of research utilization

Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0

on behalf of the SONG-HD Investigators. Establishing core outcome domains in hemodialysis: report of the Standardised Outcomes in Nephrology -Hemodialysis (SONG-HD) consensus workshops

Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals

Peter Tugwell J. Andr e Knottnerus E-mail address: Laura.Tugwell@uottawa.ca (P. Tugwell)