Rapid Response Systems: how to interpret levels of evidence

Marcello Difonzo 1

1 Anesthetist Intensivist, Adjunct Professor of Anesthesiology, Nursing Degree Course, University of Bari “Aldo Moro”, Intensive Care Unit, Ospedale Di Venere, Bari, ASL Bari

Abstract

BACKGROUND AND AIM: The Rapid Response System (RRS) has been introduced to prevent cardiac arrest, unplanned admissions to the intensive care unit, and death in hospitalized patients. Despite the constant and widespread presence of this system in worldwide hospitals, it remains debated whether its use improves patient outcomes. The aim of this narrative review is to describe the available evidence supporting the effectiveness of RRSs and to discuss the controversies on the lack of level 1 evidence studies.

METHODS: The literature search covers the period from 1 January 2000 to 31 March 2016.

RESULTS: Studies with different research designs, observational, quasi-experimental with non-randomized control group and experimental, and aggregate data of meta-analyses indicate a statistically significant reduction of in-hospital cardiac arrests and hospital mortality associated with the deployment of RRSs.

CONCLUSIONS: A RRS is a complex intervention in a complex system, such as a hospital. This complexity does not allow considering experimental trials only as the most appropriate methodology to answer at research objectives. Furthermore, the benefits of a RRS depend greatly on its proper use. Accumulating evidence suggests the importance to investigate barriers and facilitators that can affect the integration, within a hospital, of this complex intervention.

Keywords: Rapid Response Systems; In-hospital Cardiac Arrests; Hospital Mortality; Levels Of Evidence; Complex Interventions

Rapid Response Systems: come interpretare i livelli di evidenza

CMI 2017; 11(2): 71-88

https://doi.org/10.7175/cmi.v11i2.1271

Clinical management

Corresponding author

Marcello Difonzo, MD

Adjunct Professor of Anesthesiology, Nursing Degree Course, University of Bari “Aldo Moro”

mardif@libero.it

Disclosure

The author declares he has no competing financial interests concerning the topics of this article

Background

Since the early Nineties, landmark studies reported the occurrence of unexpected adverse events in hospitalized patients, which were preventable in most cases [1,2]. An adverse event was defined as an injury caused by medical management rather than the underlying disease and leading to prolonged hospitalization, disability, or death [1,2].

From then on, the researchers began to evaluate alterations of clinical signs preceding serious adverse events (SAEs) and the treatment before admission to an intensive care unit (ICU) [3-12]. A high percentage of cardiac arrests (84%) was characterized by a deterioration of vital signs in the previous 8 hours [3]. Recently, a Japanese study [4] found similar results with approximately 60% of the patients with abnormal vital signs before a cardiac arrest. Over than 60% of the patients transferred to an ICU have potentially life-threatening abnormalities in the 8 hours before their admission [9]. Furthermore, the poor management of the acute deterioration can lead to potentially avoidable deaths [10]. The most common changes in vital signs included tachypnea, tachycardia, hypotension, reduced oxygen saturation, and sudden change in the level of consciousness [7-9].

Nowadays, the role of acute hospitals is changing rapidly and the number of critically ill patients with potentially reversible conditions is increasing [13]. Thus, it happens that not all serious diseases occur within an ICU [14]. Often, in general wards, it is necessary to treat a growing number of patients in acute clinical conditions. In these situations, frequently, the treatment is provided by doctors and nurses without sufficient experience in the management of a patient with unstable vital functions. The scenario makes quite logical to provide a treatment by skilled clinicians in critical care that can be made anywhere inside hospital, within minutes [15].

Rapid Response System concept

In June 2005, the first International Conference on Medical Emergency Teams (ICMET) was held. Consensus findings were published [16] and the concept of the Rapid Response System (RRS) was introduced as a clinical and organizational model for the management of in-hospital emergencies. This model provides for an early identification of a patient at risk of deterioration in general wards and a rapid response by a team of critical care experts. The purpose is to prevent the progression to irreversible conditions and SAEs with the aim of reducing cardiac arrest, unplanned admissions to the ICU, and death [16].

The implementation of a RRS is recommended by several organizations centered on patient safety:

• the Institute for Healthcare Improvement (IHI) [17,18], in the USA (2004);
• the Australian Commission in Safety and Quality Healthcare [19], in Australia (2006);
• the Critical Care Plane [20], in Ontario, Canada (2006); and
• the National Institute for Health and Clinical Excellence (NICE) [21], in the United Kingdom (2007).

A RRS is based on 4 essential components [16] (Figure 1):

1. the afferent limb includes physicians and nurses of general hospital wards, who have the task to identify the clinical deterioration of a patient and to activate the response;
2. the efferent limb is the emergency team that can be nurse- or physician-led and can include a respiratory therapist;
3. the administrative limb oversees all system components, allows the working of the team and provides necessary resources; and
4. the quality improvement limb analyzes events data, provides feedback on the team function, monitors quality indicators like the staff satisfaction, and collects data on outcome measures.

In agreement to the countries and the team composition, RRSs are called in different ways: Medical Emergency Team (MET) in Australia, Rapid Response Team (RRT) in the United States, Critical Care Outreach Service (CCOS) in the United Kingdom, and Critical Care Response Team (CCRT) in Canada [16,22] (Box 1).

In late years, the RRS concept has spread in numerous countries. Despite the constant and widespread presence of this system in worldwide hospitals, it remains debated whether its use improves patient outcomes. Several non-randomized trials show the effectiveness of a RRS, but it is argued that levels of evidence are weak in relation to typical evidence-based medicine criteria. Experimental studies are insufficient and this is considered as a lack of rigorous evidence. The aim of this narrative review is to describe the available evidence supporting the effectiveness of RRSs and to discuss the controversies on the lack of level 1 evidence studies.

Methods

The literature search covers the period from 1 January 2000 to 31 March 2016. Studies published in English were identified by a computerized database search applied to the Cumulative Index to Nursing and Allied Health Literature (CINAHL), PubMed, and Google Scholar. The terms used were: rapid response systems, in-hospital cardiac arrests, hospital mortality, levels of evidence, complex interventions, alone and in combination. References cited in key publications were reviewed. Relevant publications were selected in respect to the designs of the studies and clinical outcomes, cardiac arrest, unplanned admissions to ICU, and death, utilized to evaluate the effects of a RRS.

Results

Non-randomized and observational studies

From the early 2000s, a lot of quasi-experimental clinical trials with non-randomized control groups and observational studies evaluated outcomes of patients after the deployment of a RRS. These studies were performed in the adult and pediatric population, mainly in Australia, the USA, Canada, and the United Kingdom and, in recent years, in the rest of Europe (Table I).

In 2000, a multicenter concurrent controlled study, probably the first on a RRS, was published. Bristow and colleagues [34] compared the first Australian hospital with a MET and 2 hospitals with a Cardiac Arrest Team (CAT). The rate of unanticipated admission to the ICU or high-dependency unit (HDU) was lower in MET hospital. However, in all 3 hospitals, the rate of cardiac arrests and total mortality did not differ significantly. The lack of sensitivity of calling criteria, the irreversible pathophysiological processes of patients, despite the alert of the emergency team, and the underutilization of the MET system can explain these results.

Several monocentric and multicentric studies, with a historical control, compared outcomes before and after the implementation of a RRS. In 2002, one of the first studies with a “before and after” design in a single center by Buist and colleagues [35] reported a reduction of cardiac arrests and mortality. There was a 50% incidence reduction of unexpected cardiac arrests, after adjustment for casemix (odds ratio—OR = 0.50; 95% confidence interval—CI: 0.35-0.73), and a statistically non-significant reduction of related mortality (OR = 0.87; 95% CI: 0.76-1.01) after the implementation of the MET system.

Jones and colleagues performed 3 studies in a single center reporting the long-term (4 years) effects of the introduction of the MET on cardiac arrests and total mortality. In 2005, they highlighted a 53% reduction in the incidence of cardiac arrests, from 4.06 to 1.9 per 1000 admissions, through the RRS introduction (OR = 0.47; 95% CI: 0.35-0.62; p < 0.0001) [36]. The Authors described a “dose effect” of the MET that suggested an inverse association between the team utilization and the risk of cardiac arrests; every 17 calls, a single cardiac arrest can be avoided. In 2007, the researchers [37] compared patient mortality after admission for major surgery, during a control period and an intervention period. At the follow-up time of 1500 days (4.1 years), the overall survival was significantly better in the MET group (71.6% vs. 65.8%; p = 0.001). Findings indicated a 23% reduction of 1500-day mortality (OR = 0.77; 95% CI: 0.64–0.91; p < 0.003). In the last paper, the investigators [38] reported, in the 4 years following the introduction of a RRS, a mortality reduction in post-surgery patients and a mortality increase in medical patients. This difference may be related to the extension of the disease complexity and the different ratio of MET calls between medical and surgical patients.

In 2011, a systematic review [39] included several “before and after” design studies with a historical control group in the adult population published between 2000 and 2008 in Australia, the USA, the United Kingdom, and Sweden. Aggregate data on RRSs demonstrated a significant reduction in cardio-respiratory arrest (OR = 0.625; 95% CI: 0.502-0.777) and hospital mortality (OR = 0.886; 95% CI: 0.711-0.994) (9 and 10 studies, respectively). Approximately 37.5% and 11.4% risk reduction in cardio-respiratory arrest and hospital mortality was found, respectively. Some of these studies (3 for cardio-respiratory arrest and 6 for hospital mortality) did not confirm the effects (a wide confidence interval with a non-significant risk reduction).

Other studies with a “before and after” design in the adult and pediatric population included more hospitals. In 2008, Baxter and colleagues [40] demonstrated, in 2 community hospitals, a reduction in cardiac arrests (2.53 ± 0.8 vs. 1.3 ± 0.4 per 1000 admissions; p < 0.001) after the MET introduction and a non-significant reduction in overall hospital mortality (3.57% vs. 3.55%, pre-MET and post-MET, respectively). In 2015, the Cost and Outcomes analysis of Medical Emergency Teams (COMET) study [41] assessed the nationwide introduction of RRSs in the Netherlands. The pragmatic study was multicenter involving 12 hospitals for adults. The composite endpoint of cardiopulmonary arrest, unplanned ICU admission, or death per 1000 admissions was significantly reduced after the introduction of a RRS (adjusted OR = 0.847; 95% CI: 0.725-0.989; p = 0.036). Cardiopulmonary arrests and in-hospital mortality were significantly reduced (OR = 0.607; 95% CI: 0.393-0.937; p = 0.018 and OR = 0.802; 95% CI: 0.644-1.0; p = 0.05, respectively), whereas unplanned ICU admissions showed a non-significant reduction (OR = 0.878; 95% CI: 0.755-1.021; p = 0.092). Kotsakis and colleagues [42] reported a multicenter study in 4 academic pediatric hospitals in Ontario, Canada, and found neutral effects. The introduction of a pediatric rapid response team was not associated with the reduction of cardiopulmonary arrest (1.9 vs. 1.8 per 1000 admissions; p = 0.68) and mortality after urgent admissions in the pediatric ICU (1.3 vs. 1.1 per 1000 admissions; p = 0.25).

In 2014, Salvatierra and colleagues [43] presented an observational cohort study involving nearly half a million adult patients in 10 level III hospitals, in Washington State, in the United States. Cumulative retrospective data demonstrated a 24% relative risk reduction of mortality in the post-RRT period (relative risk—RR = 0.76; 95% CI: 0.72-0.80; p < 0.001), in 6 out of 10 hospitals.

There were 2 studies with an interrupted time series design. Howell and colleagues [44], in an adult teaching hospital in the USA, found an 80% reduction (95% CI: 63%-89%; p < 0.0001) in the adjusted odds of unexpected death during the RRS intervention period (OR = 0.20; 95% CI: 0.11-0.37; p < 0.0001). However, overall hospital mortality showed a non-significant reduction (OR = 0.91; 95% CI: 0.82-1.02; p = 0.09). Hanson and colleagues [45] included a pediatric teaching hospital in the United States. The introduction of a pediatric RRS was associated with a significant reduction of cardiac arrests and duration of clinical instability before the evaluation by the pediatric team (median duration from 9 h 55 min to 4 h 15 min post-intervention; p = 0.028).

A recent paper [46], in 2016, summarized data of 25 non-randomized trials that demonstrated the benefits of a RRS on hospital mortality among adult and pediatric inpatients. The review included 22 “before and after” design studies, 16 in the adult population (1,481,115 patients), risk ratio = 0.88 (95% CI: 0.81-0.95) and 6 in the pediatric population (453,412 patients), risk ratio = 0.80 (95% CI: 0.63-1.00). In 1 study, there was a parallel control cohort design (50,942 adult patients), risk ratio = 0.81 (95% CI: 0.69-0.94). There were 2 interrupted time series studies, the first included 156,541 adult patients, risk ratio = 0.94 (95% CI: 0.87-1.00) and the second included 16,047 pediatric patients, risk ratio = 0.76 (95% CI: 0.53-1.09).

Quasi-experimental and observational cohort studies showed, in majority, positive effects, with a reduction of cardiac arrests and mortality, while lot less showed neutral or negative effects. Studies were well-designed, adjusted for the potential biases (confounding factors, casemix severity, temporal trends) with increased internal validity. Moreover, these studies were performed in various countries and in different hospital settings for size, number, and type, as teaching and non-teaching hospitals. These features increase the generalizability of the results.

Randomized controlled trials

Currently, there are 2 randomized controlled pragmatic trials on RRSs (Table II).

In 2004, Priestley and colleagues [47] analyzed mortality and hospital length of stay (LOS) after the introduction of a Critical Care Outreach Team. The study had a stepped wedge cluster randomized design; the intervention was introduced in all departments with general wards involved sequentially. They included patients of 16 general wards in a single hospital in England. The study found a 48% reduction of hospital mortality (OR = 0.52; 95% CI: 0.32-0.85) and suggested that the LOS was increased.

The MERIT (Medical Early Response, Intervention and Therapy) study [48], published in 2005, was a cluster randomized controlled trial (RCT), carried out in Australia. It involved 23 hospitals, which were categorized in the intervention group (12 hospitals with a MET system implementation) and in the control group (11 hospitals without a MET). This research included 125,132 patients and went on for 12 months: baseline period (2 months), introduction and implementation period (4 months), and intervention period (6 months). In the control hospitals, the CAT continued to operate. In the intervention hospitals, educational paths for doctors and nurses were provided concerning the activation criteria, the identification of patients at risk, the need for a quick call, and the call mode of the MET system. The primary endpoint, a composite index of cardiac arrests without a pre-existing not-for-resuscitation (NFR) order, unplanned ICU admissions, and unexpected deaths without a pre-existing NFR order, in general wards, had a similar incidence in the control hospitals and in the intervention hospitals (5.86 vs. 5.31 per 1000 admissions; p = 0.640). Also the secondary endpoint, the 3 events separately, had a similar incidence: cardiac arrests (1.64 vs. 1.31; p = 0.736), unplanned ICU admissions (4.68 vs. 4.19; p = 0.599), and unexpected deaths (1.18 vs. 1.06; p = 0.752). The introduction of a MET was associated with a greater number of calls of the emergency team, MET or CAT (3.1 vs. 8.7 per 1000 admissions; p = 0.0001). There was a reduction in the rate of cardiac arrests (p = 0.003) and unexpected deaths (p = 0.01) from baseline in both the control hospitals and the intervention hospitals.

The results of the MERIT study, with an intention-to-treat analysis, showed limitations in both the conclusiveness and the generalizability [49]. However, despite the criticism, the results of the Australian trial were substantially correct [50] and several reasons could explain these inconclusive findings (Box 2).

MERIT study investigators

In subsequent years, the researchers of the MERIT trial produced several studies, both secondary analyses and a survey in hospitals involved in the primary study [56-64] (Table III).

Generally, the evidence of post-hoc analyses is weak, with an increased risk of false positive results (type I error). However, MERIT study investigators analyzed different hypotheses from other perspectives. For example, Chen and colleagues [60] evaluated data with an as-treated analysis rather than with an intention-to-treat analysis as in the primary trial [30,48]. The findings demonstrated a significant reduction in unexpected and overall cardiac arrests and in unexpected deaths with the early intervention of teams, both MET and CAT. This relationship between the timely emergency calls and outcomes improvement suggests that the effectiveness of a RRS depends more on its implementation than from the research design.

Population-based studies

In 2014, Chen and colleagues [65] analyzed data of 9,221,138 patients of 82 public acute hospitals, in New South Wales, in Australia (Table IV).

During the study period, from January 1st 2002 to December 1st 2009, the number of hospitals with a RRS increased from 26 (31.7%) in 2002 to 61 (74.4%) in 2009. There was a 52% decrease in in-hospital cardiopulmonary arrest (IHCA) rate, a 55% decrease in IHCA-related mortality rate, a 23% decrease in hospital mortality rate, and a 15% increase in survival to discharge after an IHCA.

In the same year, a further study [66] from the same database compared 3 teaching hospitals (1,088,491 patients) without a RRS and 1 teaching hospital with a mature RRS (479,194 patients) in Sidney. Within the hospital with a RRS since 1990, there was a decrease of more than 50% in IHCA rate, a 40% decrease in IHCA-related mortality, and a 6% decrease in overall hospital mortality. A RRS was introduced in 2009 in 2 hospitals and in January 2010 in the third. In the first year with a RRS, there was a 22% decrease in IHCA rate, a 22% decrease in IHCA-related mortality, and an 11% decrease in overall hospital mortality.

This study emphasizes some debated aspects. Cardiac arrest and hospital mortality decreased, during the study, in non-RRS hospitals; however, this decline was greater within 12 months following the introduction of RRSs [67]. Besides, the involvement of more hospitals reduced limits of “before and after” studies in a single center [67]. Lastly, the evaluation of overall hospital mortality, more than IHCA or IHCA-related mortality, was a response to previous criticism; that is, rapid response teams increase the frequency of documentation of NFR orders, increase the transfer of patients to the ICU, introduce a bias, and overestimate the effects of a RRS [67].

Meta-analyses

A recent meta-analysis [68] analyzed 29 studies published after 2000 and until 2013: 20 studies included data on cardiac arrests outside an ICU and 20 on hospital mortality. The implementation of RRS teams (MET, RRT, or CCOS) was associated with a decrease in cardiopulmonary arrests in both the adult population (RR = 0.65; 95% CI: 0.61-0.70; p < 0.00001) and the pediatric population (RR = 0.64; 95% CI: 0.55-0.74; p < 0.00001). Similarly, hospital mortality decreased in adult (RR = 0.87; 95% CI: 0.81-0.95; p = 0.0002) and pediatric patients (RR = 0.82; 95% CI: 0.76-0.89; p = 0.03).

In 2016, the meta-analysis by Solomon and colleagues [69] included studies until 2014, which analyzed the impact of a RRT and/or a MET on the adult population. Among the 20 studies reporting data on cardiac arrests, 12 demonstrated positive effects and 8 no difference. Among the 20 studies reporting data on hospital mortality, 9 demonstrated positive effects, 10 no difference and 1 study favored the RRS for surgical patients and usual care for medical patients. Aggregate data indicated a significant reduction of non-ICU cardiac arrests (RR = 0.62; 95% CI: 0.55-0.69; p < 0.00001) and hospital mortality (RR = 0.88; 95% CI: 0.83-0.93; p < 0.00001).

Outcome measures

Various measures of outcome can assess the clinical effects of a RRS. In the published literature, the most widely used outcomes are cardiac arrests, unplanned ICU admissions from general wards, and hospital mortality.

There are 2 main reasons that explain differences in reporting data from different studies. Firstly, NFR orders increase with a RRS [58]. For example, Kenward and colleagues [70], in 2004, reported a 22% (28/130) of patients who died for a NFR order after the introduction of a MET. Years later, Jones and colleagues [71] reported about 31% of emergency team calls in patients with a limitation of a medical therapy. Therefore, a RRS becomes a surrogate way of managing the dying in acute hospitals [72], while is it necessary to achieve better methods of identifying those patients who could benefit from ICU interventions [73]. Secondly, several treated inpatients have lots comorbidities and die despite treatments received by a RRS.

The rate of cardiac arrests can be reported in different ways. The MERIT study [48] evaluated the rate of hospital cardiac arrests in general wards, without a pre-existing NFR order. In 2010, a meta-analysis [74] described a reduction in cardiac arrests without reduction in mortality in the adult population. The reason was the use of the rate of cardiac arrests outside the intensive care instead of the rate of hospital cardiac arrests, which introduces a bias by excluding deteriorating patients admitted to an ICU. In addition, emergency teams increase NFR orders, with a decrease rate of cardiopulmonary arrests without a decrease of hospital mortality [74]. Otherwise, a population-based study [65] reported the rate of IHCA (number of IHCAs divided by the total number of admissions) and IHCA-related mortality (number of deaths among those patients who suffered an IHCA divided by the total number of admissions).

Hospital mortality represents the most important outcome measure for RRSs [75], but its reduction depends from what we want to measure. The total hospital mortality considers unexpected plus expected death, that is death after a NFR order. For example, the COMET pragmatic study [41] demonstrated a significant reduction of the primary composite endpoint of cardiopulmonary arrest, unplanned ICU admission, or death after the introduction of a RRS. The investigators, in a subsequent post-hoc study [76], evaluated the death without limitation of a medical treatment, or “unexpected death”, instead of the incidence of overall mortality. The unadjusted OR for unexpected death was 0.557 (95% CI: 0.40-0.78), instead, in the primary study, the unadjusted OR for all-cause mortality was 0.865 (95% CI: 0.77-0.98). Another relevant topic is the baseline mortality rate. In 2012, Simmes and colleagues [77] reported a baseline mortality rate, without NFR orders, of 3.6 per 1000 admissions (5/1376); after the introduction of a RRS, there was a non-significant mortality decrease of 50% (1.7 per 1000 admissions, 4/2410; OR = 0.42; 95% CI: 0.11-1.59). The low incidence induced a statistically non-significant result, unlike from several studies in which the baseline incidence was 10 or more per 1000 admissions [77].

The reduction of unplanned ICU admissions is an expression of the recognition and early treatment of a patient at risk in general wards [16]. However, if unanticipated ICU admissions increase but mortality and cardiac arrest rates decrease, it is inappropriate to consider the outcome a failure [39]. In this case, the avoidance of unplanned ICU admissions is not an appropriate outcome for RRSs [78]. A Dutch study [77] reported an increase in unplanned ICU admissions, from 34/1376 (2.47%) to 100/2410 (4.15%) (OR = 1.66; 95% CI: 1.07-2.55) within 2 years from the introduction of a RRS. At the same time, cardiac arrests decreased from 4/1376 (0.29%) to 3/2410 (0.12%) (OR = 0.38; 95% CI: 0.09-1.73). This situation suggests that a greater number of instable patients, assisted in general wards, was transferred to the ICU.

Discussion

In 2008, Price and colleagues [79] emphasized that positive effects of the METs came from several studies with a “before and after” methodology, in majority realized in single hospitals. Therefore, the studies could show a positive effect of an intervention even if it cannot be present. In agreement with evidence-based medicine perspective, the level of evidence was weak and most of these studies could be considered at best as level 2 evidence [79]. The MERIT trial [48] had a strong level of evidence, but it showed no improvement in outcomes. Moreover, controversies remain on the results of the studies and on the relationship between RRSs and clinical outcomes [75].

Is there a role for evidence-based medicine in the RRS evaluation?

Evidence-based medicine (EBM) is an approach to the patient treatment based on the best scientific evidence to adopt clinical decisions. A fundamental principle of EBM is the recognition of a hierarchy of evidence [80]. The scientific method tends to minimize the risk of random and systematic errors. Therefore, studies less exposed to the risk of bias are at the highest level in the hierarchy of evidence and those most exposed are at the lowest level. Another principle recognizes that evidence alone does not say what to do, but decisions must include the value and personal and clinical context of the patient [81]. Thus, interpreting the levels of hierarchy, it is important not to always consider the level 1 evidence as the best or most appropriate choice for the research question [81].

The first work on levels of evidence appeared in 1979, published by Canadian Task Force on the Periodic Health Examination [82] (Table V). This system was based principally on the design of the studies to grade the quality of evidence, with RCTs at the top of the hierarchy.

In later years, other systems to classify the levels of evidence were described. The GRADE (Grading of Recommendations Assessment, Development and Evaluation) system [83], published in 2004, included 2 categories on the strength of recommendations (strong, weak) and 4 levels of evidence on the quality (high, moderate, low, and very low). The study design is not the only factor to appraise the quality of evidence [84]. A classification upwards (strong recommendation with high-quality evidence) [85] may be justified for observational studies when there are factors that increase the quality of evidence (very large magnitude of the treatment effect, evidence of a dose-response relation, no plausible confounders) [84,85].

Already in 2004, Pronovost and colleagues [86] observed that RCTs were sometimes difficult to perform in critically ill patients, and results may not be generalizable. Therefore, observational studies may supplement clinical trials to inform clinical practice. Quasi-experimental designs are useful where there are political, practical, or ethical barriers to conducting genuine randomized studies [87]. Concato and colleagues [88] showed that well designed observational studies, with either a cohort or a case-control design, do not systematically overestimate the magnitude of treatment effects compared with RCTs on the same topic. Population-based observational studies have a limited internal validity, compared with RCTs, because it may be difficult to separate the effect of a new treatment from the other confounding factors. However, these studies have a good external validity and can provide evidence on effectiveness of a treatment [89]. In 2006, Bruckel [90] argued about the inappropriate application of some principles of EBM. A RRS requires an alteration of organizational design of a hospital, then applying EBM in these cases may misrepresent the situation.

Complex interventions and research

In 2015, Hawe observed: “Complexity, resulting from interactions among many component parts, is a property of both the intervention and the context (or system) into which it is placed” [91]. Complex systems include primary care, hospitals, and schools. The intervention in these settings may be simple or complicated [92]. In health care, interventions involve care providers and patients. In simple interventions, the outcome is an effect of the interaction between a care provider and a patient, and the intervention [93]. Complex interventions can involve interactions between patients and interactions between providers. Components of the intervention itself can also interact and affect the outcome [93]. In 2011, Chen underlined that a complex system intervention in health care often requires changes in the structure, culture, and organizational behavior of an institute, as well as changes in individual practices, all aimed at improving the quality of care [94]. A RRS has the characteristics of a complex intervention in a complex system, such as a hospital.

Data from a systematic review [95] on system-wide interventions in hospital indicated that improved outcomes could be observed when results were measured at least 2 years after the intervention. Single interventions, such as a new drug or procedure conducted at an individual patient level, are appropriately evaluated by RCTs [54]. Complex interventions, such new systems to improve outcomes of patients, are inadequate to be tested by studies with randomized controlled design, conventional and cluster [54]. Moreover, in health services, complex interventions, with a contemporaneous control, often provide null results: it is “the rising tide phenomenon”. The evaluation of the intervention can be done in a setting where the entire system is improving, thus producing a temporal trend, a “rising tide causing all vessels to rise”. Therefore, control sites improve, the difference between intervention and control sites decreases, and the intervention has no effect [96].

In complex interventions, the study should use the most robust possible design to minimize bias and maximize generalizability [87]. However, performing further RCTs on RRSs will be difficult. The reasons are the widespread diffusion of this emergency model, which makes it difficult to find a control group, the heterogeneity regarding the standard treatments, the patient groups, the wards staff and the team composition, and the complexity of the intervention. Moreover, cluster randomization requires the recruitment of large numbers of clusters [87]. Winters and colleagues [97], regarding barriers and facilitators of the implementation of a RRS, found that the acceptance and leadership of the RRS, the rate of calling, and the trigger mechanisms were some factors that could improve the implementation process.

Evaluating the type of intervention

The intervention introduced by a RRS, in a hospital setting, may be influenced by various processes associated with the required changes. “Silo effect” indicates a lack of communication and common goals between departments in an organization [98]. The silo effect, like an agricultural silo prevents mixing of different grains, limits the interactions among members of different branches of company and reduces productivity [98]. In acute care hospitals, silos or vertical structures, such as wards, units and departments, are well developed. Treatments are managed in different sites and by specialized teams [24] and the system is centered on treatment sites and on individual doctors, rather than built around needs of patients [99]. The system may fail at the intersection between silos for patients with complications of the original illness, which are outside the expertise of clinicians who treat them [100]. A RRS is one of the first multiprofessional interventions that challenges this traditional medicine approach [24]. However, a RRS suffers from the lack of integration in hospitals. A silo-based mentality, in acute hospitals, has existed for more centuries and this situation makes it difficult to implement whole-of-hospital systems [101].

A RRS is based on the early identification of the clinical deterioration of a patient and on a rapid response. Nevertheless, the identification of a patient at risk depends on the warning criteria and their measurement. Cretikos and colleagues [56] found, for the ability of the MET activation criteria, a sensitivity of 49.1% (44.4-53.8%) and a specificity of 93.7% (91.2-95.6%). The objective activation criteria alone, with low sensitivity, do not allow identifying all patients at risk. The subjective criteria, with the clinical observation by nurses, allow the identification of a greater number of patients at risk. A comparison between objective and subjective criteria of the MET calls detected that the “worried” criterion was the most frequent reason for MET activation [102]. Douw and colleagues [103] suggested that signs underlying the worry or concern of the nurses, present before changes in vital signs, were potential early indicators of deterioration.

A RRS changes traditional hierarchies of a hospital. In general wards, the MET can be called by doctors, nurses, or health personnel. This situation modifies traditional hierarchies based, firstly, on the involvement of the ward doctors. For example, some factors that negatively affect MET activation by the nurses are the discouragement by the doctors, the fear of being subjected to criticism, and the adherence to models that lead to contact the ward doctors before activating the MET system [104,105]. An Italian multicenter survey [106] involved doctors and nurses, in a group of 10 hospitals; findings of the study showed that, for the nurses, the ward doctors were the main obstacle to MET activation.

Study limitations

This study has some limitations, despite the respect of the method. Indeed, in a narrative review, the identification and filtration of relevant articles are conducted subjectively and may be incomplete. This situation exposes the findings to a high potential level of bias. Nevertheless, the present work provides an accurate analysis and overview of the papers discussed.

Conclusions

A RRS is a complex intervention that works as an integrated set of 4 components, which must interact with each other and with patients. This complexity does not allow considering experimental trials only as the most appropriate methodology to answer at research objectives. Numerous well-designed studies provide high-quality and strong evidence of clinical outcome improvement after the deployment of RRSs. Therefore, despite the controversies, this model for in-hospital emergencies has been worldwide introduced to provide an early response to clinically deteriorating patients. Furthermore, the benefits of a RRS depend greatly on its proper use. Accumulating evidence suggests the importance to investigate barriers and facilitators that can affect the integration, within a hospital, of this complex intervention.

References

1. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991; 324: 370-6; http://dx.doi.org/10.1056/NEJM199102073240604

2. Leape LL, Brennan TA, Laird NM, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991; 324: 377-84; http://dx.doi.org/10.1056/NEJM199102073240605

3. Schein RM, Hazday N, Pena M, et al. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest 1990; 98: 1388-92; http://dx.doi.org/10.1378/chest.98.6.1388

4. Fujiwara S, Koike T, Moriyasu T, et al; and IHCA study group. A retrospective study of in-hospital cardiac arrest. Acute Med Surg 2016; 3: 320-5; http://dx.doi.org/10.1002/ams2.193

5. Franklin C, Mathew J. Developing strategies to prevent in hospital cardiac arrest. Analyzing responses of physicians and nurses in the hours before the event. Crit Care Med 1994; 22: 244-7; http://dx.doi.org/10.1097/00003246-199402000-00014

6. Buist M, Jarmolowski E, Burton PR, et al. Recognising clinical instability in hospital patients before cardiac arrest or unplanned admission to intensive care. A pilot study in a tertiary-care hospital. Med J Aust 1999; 171: 22-5

7. Hillman K, Bristow PJ, Chey T, et al. Antecedents to hospital deaths. Intern Med J 2001; 31: 343-8

8. Kause J, Smith G, Prytherch D, et al. A comparison of antecedents to cardiac arrests, deaths, and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom-the ACADEMIA study. Resuscitation 2004; 62: 275-82; http://dx.doi.org/10.1016/j.resuscitation.2004.05.016

9. Hillman K, Bristow PJ, Chey T, et al. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med 2002; 28: 1629-34; http://dx.doi.org/10.1007/s00134-002-1496-y

10. McGloin H, Adam SK, Singer M. Unexpected deaths and referrals to intensive care of patients on general wards. Are some cases potentially avoidable? J R Coll Physicians Lond 1999; 33: 255-9

11. McQuillan P, Pilkington S, Allan A, et al. Confidential inquiry into quality of care before admission to intensive care. BMJ 1998; 316: 1853-8; http://dx.doi.org/10.1136/bmj.316.7148.1853

12. Massey D, Aitken LM, Choaboyer W. What factors influence suboptimal ward care in the acutely ill ward patient? Intensive Crit Care Nurs 2009; 25: 169-80; http://dx.doi.org/10.1111/j.1365-2702.2010.03394.x

13. Hillman K. Critical care without walls. Curr Opin Crit Care 2002; 8: 594-9; http://dx.doi.org/10.1097/00075198-200212000-00019

14. Rosengart MR, Pinsky MR. Origins of the critically ill: the impetus for critical care medicine. In: Scales DC, Rubenfeld GD (eds.). The organization of critical care. An evidence-based approach to improving quality. Respiratory medicine 18. Springer Science Business Media New York, 2014, pp. 9-24. Available at http://link.springer.com/chapter/10.1007/978-1-4939-0811-0_2 (last accessed March 2017)

15. Tee A, Calzavacca P, Licari E, et al. Bench-to-bedside review: The MET syndrome-the challenges of researching and adopting medical emergency teams. Crit Care 2008; 12: 205; http://dx.doi.org/10.1186/cc6199

16. DeVita MA, Bellomo R, Hillman K, et al. Findings of the first consensus conference on medical emergency teams. Crit Care Med 2006; 34: 2463-78; http://dx.doi.org/10.1097/01.CCM.0000235743.38172.6E

17. Institute for Healthcare Improvement. 5 million lives campaign. Available at http://www.ihi.org/about/documents/5millionlivescampaigncasestatement.pdf (last accessed March 2017)

18. Berwick DM, Calkins DR, McCannon CJ, et al. The 100,000 lives campaign: setting a goal and a deadline for improving health care quality. JAMA 2006; 295: 324-7; http://dx.doi.org/10.1001/jama.295.3.324

19. Australian Government Department of Human Services. Safer systems–saving lives, implementing a Rapid Response System 2006. https://www.humanservices.gov.au/ (last accessed April 2017)

20. Implementation of Critical Care Response Teams (CCRTs) in Ontario hospitals-Year one. Prepared by the critical care secretariat, Ontario Ministry of Health and Long-Term Care. January 2007. Available at http://docplayer.net/26190034-Implementation-of-critical-care-response-teams-ccrts-in-ontario-hospitals-year-one.html (last accessed March 2017)

21. National Institute for Health and Clinical Excellence, NICE. Acutely ill patients in hospital: recognition of and response to acute illness in adults in hospital. July, 2007. Available at https://www.nice.org.uk/guidance/cg50 (last accessed March 2017)

22. Al-Qahtani S, Al-Dorzim HM. Rapid response systems in acute hospital care. Ann Thorac Med 2010; 5: 1-4; http://dx.doi.org/10.4103/1817-1737.58952

23. Lee A, Bishop G, Hillman KM, et al. The medical emergency team. Anaesth Intensive Care 1995; 23: 183-6

24. DeVita MA, Hillman K, Smith GB. Resuscitation and rapid response systems. Resuscitation 2014; 85: 1-2; http://dx.doi.org/10.1016/j.resuscitation.2013.11.005

25. England K, Bion JF. Introduction of medical emergency teams in Australia and New Zealand: a multicentre study. Crit Care 2008; 12: 151; http://dx.doi.org/10.1186/cc6902

26. Hillman K. Rapid response systems. Indian J Crit Care Med 2008; 12: 77-81; http://dx.doi.org/10.4103/0972-5229.42561

27. Hourihan F, Bishop G, Hillman KM. The medical emergency team: a new strategy to identify and intervene in high risk patients. Clin Intensive Care 1995; 6: 269-72; http://dx.doi.org/10.3109/tcic.6.6.269.272

28. Hillman K, Parr M, Flabouris A, et al. Redefining in-hospital resuscitation: the concept of the medical emergency team. Resuscitation 2001; 48: 105-10

29. Pringle R, Hanson C, Falk NL. Literature review on Rapid Response Systems (RRS). March 2011. Available from: http://docplayer.net/38769742-Literature-review-on-rapid-response-systems-rrs-march-2011.html (last accessed March 2017)

30. Jones DA, DeVita MA, Bellomo R. Rapid-Response Teams. N Engl J Med 2011; 365: 139-46; http://dx.doi.org/10.1056/NEJMra0910926

31. Goldhill DR, McNarry A. Intensive care outreach services. Curr Anaesth Crit Care 2002; 13: 356-61; http://dx.doi.org/10.1054/cacc.2003.0433

32. McDonnell A, Esmonde L, Morgan R, et al. The provision of critical care outreach services in England: findings from a national survey. J Crit Care 2007; 22: 212-8; http://dx.doi.org/10.1016/j.jcrc.2007.02.002

33. Critical Care Services Ontario. Critical care plan 2015-2018. June 2015. Available from: https://www.criticalcareontario.ca/EN/AboutUs/Documents/CCSO_OCCP_2015-2018_Web%20FINAL%20Sep_8_15.pdf (last accessed March 2017)

34. Bristow PJ, Hillman K.M, Chey T, et al. Rates of in-hospital arrests, deaths and intensive care admissions: the effect of a medical emergency team. Med J Aust 2000; 173: 236-40

35. Buist MD, Moore GE, Bernard SA, et al. Effects of a medical emergency team on reduction of incidence of and mortality from unexpected cardiac arrests in hospital: preliminary study. BMJ 2002; 324: 387-90

36. Jones D, Bellomo R, Bates S, et al. Long term effect of a medical emergency team on cardiac arrests in a teaching hospital. Crit Care Med 2005; 9: R808-R815; http://dx.doi.org/10.1186/cc3906

37. Jones D, Egi M, Bellomo R, et al. Effect of the medical emergency team on long-term mortality following major surgery. Crit Care 2007; 11: R12; http://dx.doi.org/10.1186/cc5673

38. Jones D, Opdam H, Egi M, et al. Long-term effect of a Medical Emergency Team on mortality in a teaching hospital. Resuscitation 2007; 74: 235-41; http://dx.doi.org/10.1016/j.resuscitation.2006.12.007

39. Winters BD, Pham JC. Rapid Response Systems: a review of the evidence. In: DeVita MA, Hillman K, Bellomo R (eds.). Textbook of Rapid Response Systems. Concept and implementation. Berlin: Springer Science+Business Media, LLC. 2011, pp. 65-78; http://dx.doi.org/10.1007/978-0-387-92853-1_7

40. Baxter AD, Cardinal P, Hooper J, et al. Medical emergency teams at The Ottawa Hospital: the first two years. Can J Anaesth 2008; 55: 223-31; http://dx.doi.org/10.1007/BF03021506

41. Ludikhuize J, Brunsveld-Reinders AH, Dijkgraaf MG, et al. Outcomes associated with the nationwide introduction of Rapid Response Systems in the Netherlands. Crit Care Med 2015; 43: 2544-51; http://dx.doi.org/10.1097/CCM.0000000000001272

42. Kotsakis A, Lobos AT, Parshuram C, et al. Implementation of a multicenter rapid response system in pediatric academic hospitals is effective. Pediatrics 2011; 128: 72-8; http://dx.doi.org/10.1542/peds.2010-0756

43. Salvatierra G, Bindler RC, Corbett C, et al. Rapid response team implementation and in-hospital mortality. Crit Care Med 2014; 42: 2001-6; http://dx.doi.org/10.1097/CCM.0000000000000347

44. Howell MD, Ngo L, Folcarelli P, et al. Sustained effectiveness of a primary-team-based rapid response system. Crit Care Med 2012; 40: 2562-8; http://dx.doi.org/10.1097/CCM.0b013e318259007b

45. Hanson CC, Randolph GD, Erickson JA, et al. A reduction in cardiac arrests and duration of clinical instability after implementation of a paediatric rapid response system. Qual Saf Health Care 2009; 18: 500-4; http://dx.doi.org/10.1136/qshc.2007.026054

46. Maharaj R, Stelfox HT. Rapid response teams improve outcomes: no. Intensive Care Med 2016; 42: 596-8; http://dx.doi.org/10.1007/s00134-016-4246-2

47. Priestley G, Watson W, Rashidian A, et al. Introducing Critical Care Outreach: a ward randomised trial of phased introduction in a general hospital. Intensive Care Med 2004; 30: 1398-404; http://dx.doi.org/10.1007/s00134-004-2268-7

48. Hillman K, Chen J, Cretikos M, et al; MERIT study investigators. Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial. Lancet 2005; 365: 2091-7; http://dx.doi.org/10.1016/S0140-6736(05)66733-5

49. The “MERIT” trial of Medical Emergency Teams in Australia: an analysis of findings and implications for the 100,000 lives campaign. Available at https://tools.patientsafetyinstitute.ca/Communities/rrt/Shared%20Documents/General/merit[1].comments.pdf (last accessed March 2017)

50. Cooper DJ, Buist MD. Vitalness of vital signs, and medical emergency teams. Med J Aust 2008; 188: 630-1

51. Jones D, Bellomo R, DeVita MA. Effectiveness of the Medical Emergency Team: the importance of dose. Crit Care 2009; 13: 313; http://dx.doi.org/10.1186/cc7996

52. Cooper N, Forrest K, Cramp P. Essential guide to acute care. Oxford: Blackwell Publishing Ltd, BMJ Books, 2nd Edition, July 2006, pp. 1-13

53. Gale EAM. The Hawthorne studies-a fable for our times? QJM 2004; 97: 439-49;

54. Hillman K, Chen J, May E. Complex intensive care unit interventions. Crit Care Med 2009; 37 (1 Suppl): S102-S106; http://dx.doi.org/10.1097/CCM.0b013e3181920f18

55. Buist M, Harrison J, Abaloz E, et al. Six year audit of cardiac arrests and medical emergency team calls in an Australian outer metropolitan teaching hospital. BMJ 2007; 335: 1210-2; http://doi.org/10.1136/bmj.39385.534236.47

56. Cretikos M, Chen J, Hillman K, et al. The objective medical emergency team activation criteria: a case-control study. Resuscitation 2007; 73: 62-72; http://dx.doi.org/10.1016/j.resuscitation.2006.08.020

57. Cretikos MA, Chen J, Hillman KM, et al. The effectiveness of implementation of the medical emergency team (MET) system and factors associated with use during the MERIT study. Crit Care Resusc 2007; 9: 205-12

58. Chen J, Flabouris A, Bellomo R, et al.; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group. The Medical Emergency Team System and not-for-resuscitation orders: results from the MERIT study. Resuscitation 2008; 79: 391-7; http://dx.doi.org/10.1016/j.resuscitation.2008.07.021

59. Chen J, Hillman K, Bellomo R, et al.; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group. The impact of introducing medical emergency team system on the documentations of vital signs. Resuscitation 2009; 80: 35-43; http://dx.doi.org/10.1016/j.resuscitation.2008.10.009

60. Chen J, Bellomo R, Flabouris A, et al; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group. The relationship between early emergency team calls and serious adverse events. Crit Care Med 2009; 37: 148-53; http://dx.doi.org/10.1097/CCM.0b013e3181928ce3

61. Chen J, Flabouris A, Bellomo R, et al.; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group. Baseline hospital performance and the impact of medical emergency teams: modelling vs. conventional subgroup analysis. Trials 2009; 10: 117; http://dx.doi.org/10.1186/1745-6215-10-117

62. Flabouris A, Chen J, Hillman K, et al.; MERIT Study Investigators for the Simpson Centre and the ANZICs Clinical Trials Group. Timing and interventions of emergency teams during the MERIT study. Resuscitation 2010; 81: 25-30; http://dx.doi.org/10.1016/j.resuscitation.2009.09.025

63. Chen J, Bellomo R, Hillman K, et al; MERIT Study Investigators for the Simpson Centre and the ANZICS Clinical Trials Group. Triggers for emergency team activation: a multicenter assessment. J Crit Care 2010; 25: 359.e-7; http://dx.doi.org/10.1016/j.jcrc.2009.12.011

64. Chen J, Bellomo R, Flabouris A, et al. Delayed emergency team calls and associated hospital mortality: a multicenter study. Crit Care Med 2015; 43: 2059-65; http://dx.doi.org/10.1097/CCM.0000000000001192

65. Chen J, Ou L, Hillman KM, et al. Cardiopulmonary arrest and mortality trends, and their association with rapid response system expansion. Med J Aust 2014; 201: 167-70; https://doi.org/10.5694/mja14.00019

66. Chen J, Ou L, Hillman KM, et al. The impact of implementing a rapid response system: a comparison of cardiopulmonary arrests and mortality among four teaching hospitals in Australia. Resuscitation 2014; 85: 1275-81; http://dx.doi.org/10.1016/j.resuscitation.2014.06.003

67. Buist M, Mahoney A. In search of the “Holy Grail”: will we ever prove the efficacy of Rapid Response Systems (RRS)? Resuscitation 2014; 85: 1129-30; http://dx.doi.org/10.1016/j.resuscitation.2014.07.007

68. Maharaj R, Raffaele I, Wendon J. Rapid response systems: a systematic review and meta-analysis. Crit Care 2015; 19: 254; http://dx.doi.org/10.1186/s13054-015-0973-y

69. Solomon RS, Corwin GS, Barclay DC, et al. Effectiveness of rapid response teams on rates of in-hospital cardiopulmonary arrest and mortality: A systematic review and meta-analysis. J Hosp Med 2016; 11: 438-45; http://dx.doi.org/10.1002/jhm.2554

70. Kenward G, Castle N, Hodgetts T, et al. Evaluation of a medical emergency team one year after implementation. Resuscitation 2004; 61: 257-63; http://dx.doi.org/10.1016/j.resuscitation.2004.01.021

71. Jones DA, Bagshaw SM, Barrett J, et al. The role of the medical emergency team in end-of-life care: a multicenter, prospective, observational study. Crit Care Med 2012; 40: 98-103; http://dx.doi.org/10.1097/CCM.0b013e31822e9d50

72. Hillman K. Dying safely. Int J Qual Health Care 2010; 22: 339-40; http://dx.doi.org/10.1093/intqhc/mzq045

73. Rozen T, Butt W. Rapid response teams: how are they best used? Crit Care 2016; 20: 253; http://dx.doi.org/10.1186/s13054-016-1425-z

74. Chan PS, Jain R, Nallmothu BK, et al. Rapid Response Teams: a systematic review and meta-analysis. Arch Intern Med 2010; 170: 18-26; http://dx.doi.org/10.1001/archinternmed.2009.424

75. Sandroni C, D’Arrigo S, Antonelli M. Rapid response systems: are they really effective? Crit Care 2015; 19: 104; http://dx.doi.org/10.1186/s13054-015-0807-y

76. Brunsveld-Reinders AH, Ludikhuize J, Dijkgraaf MG, et al. Unexpected versus all-cause mortality as the endpoint for investigating the effects of a Rapid Response System in hospitalized patients. Crit Care 2016; 20: 168; http://dx.doi.org/10.1186/s13054-016-1339-9

77. Simmes FM, Schoonhoven L, Mintjes J, et al. Incidence of cardiac arrests and unexpected deaths in surgical patients before and after implementation of a rapid response system. Ann Intensive Care 2012; 2: 20; http://dx.doi.org/10.1186/2110-5820-2-20

78. DeVita MA, Winters B. It’s not “do” but “why do” rapid response systems work? Crit Care Med 2014; 42: 2133-4; http://dx.doi.org/10.1097/CCM.0000000000000421

79. Price RJ, Cuthbertson BH. Should hospital have a medical emergency team? In: Kuhlen R, Moreno R, Ranieri M, et al (Eds.). Controversies in Intensive Care Medicine. Berlin: MWV Medizinisch Wissenschaftliche Verlagsgesellschaft, 2008, pp. 363-71

80. Elamin MB, Montori VM. The hierarchy of evidence: from unsystematic clinical observations to systematic reviews. Chapter 2. In: Burneo JG, Demaerschalk BM, Jenkins ME (eds.). Neurology: An Evidence-Based Approach. Berlin: Springer Science+Business Media, LLC 2012, pp. 112-4; http://dx.doi.org/10.1007/978-0-387-88555-1_2

81. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in Evidence-Based Medicine. Plast Reconstr Surg 2011; 128: 305-10; http://dx.doi.org/10.1097/PRS.0b013e318219c171

82. Canadian Task Force on the Periodic Health Examination. The periodic health examination. Can Med Assoc J 1979; 121: 1193-254

83. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004; 328: 1490; http://dx.doi.org/.10.1136/bmj.328.7454.1490

84. Brozek JL, Akl EA, Alonso-Coello P, et al.; GRADE Working Group. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy 2009; 64: 669-77; http://dx.doi.org/10.1111/j.1398-9995.2009.01973.x

85. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336: 924-6; http://dx.doi.org/10.1136/bmj.39489.470347.AD

86. Pronovost P, Angus D. Evidence-Based Critical Care Medicine [monograph on the Internet]. Toronto: Centre for Evidence-Based Medicine. 2004. http://ktclearinghouse.ca/cebm/syllabi/crit/print/whole.htm (last accessed April 2017)

87. Eccles M, Grimshaw J, Campbell M, et al. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003; 12: 47-52; http://dx.doi.org/10.1136/qhc.12.1.47

88. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: 1887-92; http://dx.doi.org/10.1056/NEJM200006223422507

89. Booth CM, Tannock IF. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br J Cancer 2014; 110: 551-5; http://dx.doi.org/10.1038/bjc.2013.725

90. Bruckel J. Evidence-based medicine and rapid response team implementation. Mcgill J Med 2006; 9: 5-7

91. Hawe P. Lessons from complex interventions to improve health. Annu Rev Public Health 2015; 36: 307-23; http://dx.doi.org/10.1146/annurev-publhealth-031912-114421

92. Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ 2008; 336: 1281-3; http://dx.doi.org/10.1136/bmj.39569.510521

93. Husereau D, Jacobs P, Manns B, et al; on behalf of the IHE-IHSPR Complex Interventions Working Group. Economic evaluation of complex health system interventions: a discussion paper. Edmonton AB: Institute of Health Economics; 2014 July. Available at http://www.ihe.ca/publications/economic-evaluation-of-complex-health-system-interventions-a-discussion-paper (last accessed March 2017)

94. Chen J. Evaluating effectiveness of complex system interventions. In: DeVita MA Hillman K, Bellomo R. Textbook of Rapid Response Systems. Concept and implementation. Berlin: Springer, 2011, pp. 371-80; http://dx.doi.org/10.1007/978-0-387-92853-1_33

95. Clay-Williams R, Nosrati H, Cunningham FC, et al. Do large-scale hospital- and system-wide interventions improve patient outcomes: a systematic review. BMC Health Serv Res 2014; 14: 369; http://dx.doi.org/10.1186/1472-6963-14-369

96. Chen Y-F, Hemming K, Stevens AJ, et al. Secular trends and evaluation of complex interventions: the rising tide phenomenon. BMJ Qual Saf 2016; 25: 303-10; http://dx.doi.org/10.1136/bmjqs-2015-004372

97. Winters BD, Weaver SJ, Pfoh ER, et al. Rapid-Response Systems as a patient safety strategy: a systematic review. Ann Intern Med 2013; 158 (5_Part_2): 417-25; http://dx.doi.org/10.7326/0003-4819-158-5-201303051-00009

98. Vatanpour H, Khorramnia A, Forutan N. Silo effect a prominence factor to decrease efficiency of pharmaceutical industry. Iran J Pharm Res 2013 Winter; 12 (Suppl): 207-16

99. Hillman K, Chen J, Aneman A. Continuum of hospital care: the role of intensive care. Curr Opin Crit Care 2010; 16: 505-9; http://dx.doi.org/10.1097/MCC.0b013e32833e11dc

100. Nosrati H, Clay-Williams R, Cunningham F, et al. The role of organisational and cultural factors in the implementation of system wide interventions in acute hospitals to improve patient outcomes: protocol for a systematic literature review. BMJ Open 2013; 3: e002268

101. Hillman K, Braithwaite J, Chen J. Healthcare systems and their (lack of) integration. In: DeVita MA Hillman K, Bellomo R. Textbook of Rapid Response Systems. Concept and implementation. Berlin: Springer Science+Business Media, LLC. 2011, pp. 79-86; http://dx.doi.org/10.1007/978-0-387-92853-1_8

102. Santiano N, Young L, Hillman K, et al. Analysis of medical emergency team calls comparing subjective to “objective” call criteria. Resuscitation 2009; 80: 44-9; http://dx.doi.org/10.1016/j.resuscitation.2008.08.010

103. Douw G, Schoonhoven L, Holwerda T, et al. Nurses’ worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review. Crit Care 2015; 19: 230; http://dx.doi.org/10.1186/s13054-015-0950-5

104. Bagshaw SM, Mondor EE, Scouten C, et al. Survey of nurses’ beliefs about the medical emergency team system in a canadian tertiary hospital. Am J Crit Care 2010; 19: 74-83; http:// dx.doi.org/10.4037/ajcc2009532

105. Pusateri ME, Prior MM, Kiely SC. The role of the non-ICU staff nurse on a medical emergency team: perceptions and understanding. Am J Nurs 2011; 111: 22-9; http://dx.doi.org/10.1097/01. NAJ.0000398045.00299.64

106. Radeschi G, Urso F, Campagna S, et al. Factors affecting attitudes and barriers to a medical emergency team among nurses and medical doctors: a multicenter survey. Resuscitation 2015; 88: 92-8; http://dx.doi.org/10.1016/j.resuscitation.2014.12.027