key: cord-0722631-15lig0d4 authors: McInerney, Ciarán D.; Scott, Beverly C.; Johnson, Owen A. title: Are Regulations Safe? Reflections From Developing a Digital Cancer Decision-Support Tool date: 2021-04-02 journal: JCO Clin Cancer Inform DOI: 10.1200/cci.20.00148 sha: 8bcc23feff7595b242acaf8c47c5d9f336f2c8e2 doc_id: 722631 cord_uid: 15lig0d4 Informatics solutions to early diagnosis of cancer in primary care are increasingly prevalent, but it is not clear whether existing and planned standards and regulations sufficiently address patients' safety nor whether these standards are fit for purpose. We use a patient safety perspective to reflect on the development of a computerized cancer risk assessment tool embedded within a UK primary care electronic health record system. METHODS: We developed a computerized version of the CAncer Prevention in ExetER studies risk assessment tool, in compliance with the European Union's Medical Device Regulations. The process of building this tool afforded an opportunity to reflect on clinical concerns and whether current regulations for medical devices are fit for purpose. We identified concerns for patient safety and developed nine practical recommendations to mitigate these concerns. RESULTS: We noted that medical device regulations (1) were initially created for hardware devices rather than software, (2) offer one-shot approval rather than supporting iterative innovation and learning, (3) are biased toward loss-transfer approaches that attempt to manage the fallout of harm instead of mitigating hazards becoming harmful, and (4) are biased toward known hazards, despite unknown hazards being an expected consequence of health care as a complex adaptive system. Our nine recommendations focus on embedding less-reductionist and stronger system perspectives into regulations and standards. CONCLUSION: Our intention is to share our experience to support research-led collaborative development of health informatics solutions in cancer. We argue that regulations in the European Union do not sufficiently address the complexity of healthcare information systems with consequences for patient safety. Future standards and regulations should continue to follow a system-based approach to risk, safety, and accident avoidance. Clinical decision-support tools in primary care can help clinicians make an early diagnosis of cancer, leading to better outcomes for the patient and a more effective use of limited healthcare resources. 1 Informatics solutions are increasingly being adopted, but it is not clear whether existing and planned standards and regulations sufficiently address patient safety nor whether these standards are fit for purpose when embedded within complex and evolving health information systems. The patient safety consequences of rapid development and implementation of such computerized tools are not yet known. Regulatory standards have been developed for clinical decisionsupport tools in the United States 2 and Europe, 3 but it is not clear whether these sufficiently address patient safety. This paper presents a case study in clinical cancer informatics to reflect on whether current standards are fit for purpose. Survival rates are better for early-stage diagnoses across many cancers, and early diagnosis is recognized as a key determinant of improved outcomes. [4] [5] [6] [7] Early diagnosis in primary care can be challenging where multiple vague symptoms might be reported. 8 In the United Kingdom, primary care General Practitioners (GPs) sometimes delay referral for cancer investigation 9 and some GPs require high levels of suspicion before referring. 10 In an effort to minimize delay, 11 clinical decision-support tools and systems can assist GPs to expedite referrals and contribute to earlier diagnosis by recommending an appropriate diagnostic pathway. 1 There are many kinds of decision support with mixed evidence of effectiveness. [12] [13] [14] Symptom checker applications can be used to prompt patients 15 who might otherwise delay a referral because they have trivialized symptoms. 16 Symptoms lacking objective measurement-such as coughing and fatigue-are Author affiliations and support information (if applicable) appear at the end of this article. not always considered with due weight because they are often common and nonspecific, despite being strongly indicative for some conditions. 10 Examples in the UK National Health Service (NHS) include QRisk3, 17 QCancer, 18 and Cancer risk assessment tools (RATs), 19 for which there is evidence of acceptability and positive effect on referral rates. 20 Computerized decision support offers benefits over paperbased alternatives through automation, consideration of all data in the patient's electronic record, computing suggestions faster than by hand, and providing the potential for prompts at optimal points in the clinical pathway. 21, 22 Computerized decision support is often compared against paper-based counterparts on the assumption that the paper versions are a suitable gold standard. On the contrary, a metaregression analysis of the effect of clinical decision support on clinical performances of interest suggested that computerized decision support was associated with greater improvements compared with their nondigital counterparts. 23 As the COVID-19 pandemic accelerates the drive to digital healthcare systems, the persistence of paper-based methods as the primary modality (as opposed to as contingency) makes integration difficult and compromises health system performance and thus the safety of patients. Regardless of its advantages, clinical decision-support software is recognized as safety critical, which means that errors in use can cause significant harm. 24 However, as noted in Miller's 25 2009 historical review, the history of computer-aided, diagnostic, decision support makes little to no mention of a safety perspective. Decision-support software can be inappropriately ad hoc in its development, 24 which has prompted regulation and accreditation in different parts of the world including Europe, with, for example, the Medical Device Directive, 26 Conformité Européenne (CE) marking, 3 and the Medical Device Regulation (MDR). 27 An introductory guide can be found in Medicines and Healthcare products Regulatory Agency: An Introductory Guide to the Medical Device Regulation (MDR) and the In Vitro Diagnostic Medical Device Regulation (IVDR). 28 Although progress has been made in the creation of methods for software development, like Discipline Agile Delivery, 29,30 the focus has been on the technical performance given initial requirement specifications that do not always include safety. This is despite calls from as far back in the 1980s for an acceptable safety level to be designed into software systems before actual production or operation. 31 Reviews and summaries of some regulations have been conducted, 32 but the safety implications of clinical decision tools like symptom checkers are still underappreciated. 33 In the European Union, the Medical Device Directive specifies classifications for different levels of regulation, essential requirements for each level, and conformity routes for medical devices. 26 It was due for a replacement with the MDR in 2020, but this deadline was extended to 2021 because of the COVID-19 pandemic. 34 The intended improvements of the MDR included postmarket surveillance, registration databases of devices and independent accreditors, unique device identification, novel device classifications (notably for software), and additional evidence requirements. Notably, the MDR regulates medical devices but does not specify standards. 35 No single standard sufficiently covers the scope of clinical decision-support systems. Helpfully, Chadwick et al 36 summarize the relevant standards in their review and introduction to IEC 61508. 37 The IEC 61508 is a standard suggested by the International Electrotechnical Commission (IEC) that describes methods for designing, deploying, and maintaining automatic protection systems. Chadwick et al 36 argue that IEC 61508 is the most appropriate standard to reference when considering any medical device software because of its safety and systems perspective. To present a patient safety and system-based perspective on current and proposed regulations for software as a medical device, via a case study of a decision-support tool for cancer risk assessment in primary care. Knowledge Generated Current and proposed standards and regulations do not recognize health care and healthcare technologies as the complex adaptive systems that they are. Patient safety is threatened by not acknowledging emergent consequences, favoring oneshot approval processes, and bias toward loss-transfer approaches to risk management. Relevance Our nine recommendations provide practical and theoretical guidance to clinicians, decision makers in healthcare organizations, policy makers, and developers and regulators of health information technologies. Patient safety can be cultivated and promoted if all those involved in healthcare systems acknowledge and act on their systemic influences and capabilities. In the United Kingdom, where this research was conducted, specific standards include DCB0129 38 for health system suppliers and DCB0160 39 Although these standards are available, they are not compulsory. In the European Union's MDR, CE marking is the requirement for devices that go into service, whether or not they go to market (Sec. 2 Art.52(2) 27 ). CE marking is a declaration by a manufacturer, indicating that they take responsibility for the conformity of the product with the essential requirements of the relevant European health, safety, and environmental protection legislation. 26 As noted by Altenstetter, 35 it "serves as a kind of market authorization but should not be confused with premarket approval of individual products, or with the strict product testing regime operative in the pharmaceutical sector." Shortcomings of CE marking have been explained elsewhere noting, for example, that the product can be tested for safety without being tested for effectiveness. 40 The following safetyspecific criticisms of CE marking are worth noting (for more details, see refs. 35,41): • Manufacturers do not need to provide instructions for safe use if they believe that the device can be used safely without instructions (Annex I Chapter III § 23.1.d 27 ) . This places the responsibility of safety assessment, risk assessment, and risk control in the hands of the manufacturer who might not have sufficient knowledge or capacity to do so and who has competing financial incentives; 42 • The classification rules should be based on the intended use of the device (Annex VIII Chapter II ¶ 3.1 27 ), which does not acknowledge harm that might arise from misuse-one of the three suggested domains of health information-technology safety; 43 • The classification rules should be applied separately to devices that are intended to be used in combination (Annex VIII Chapter II ¶ 3.2 27 ), which does not consider the unknown behaviors that can emerge when healthcare information technologies combine to form health information systems. 44, 45 The greatest concern is that these safety insufficiencies relate to all classifications, including class I, which has the least oversight and requirements (Annex VIII Chapter III 27 ). Manufacturers can accredit their own devices as class I, needing only to make available for possible inspection a technical file detailing a self-determined assessment of conformity to standards. There is a danger that manufacturers of class I devices do not consider the moreadvanced standards discussed previously and put products to market that are not sufficiently safe for patients, albeit they conform to regulations. Any self-certification or other less-stringent route to approval runs the risk of misuse or abuse at the cost of patient safety. For example, the review by Zuckerman et al 46 on device recalls in the United States showed 71% of recalls were for devices approved via the route that did not require clinical trials or manufacturer inspections of safety or efficacy (see ref. 47 for discussion of differences between European Union and US systems, at the time). A further complication is when software is developed for exclusive use within particular health service organizations, which is associated with its own lesser regulation in the United Kingdom. 48 Finally, decision-support software can sit within or on top of an electronic healthcare record with the best intentions of integration in mind. In such cases, the boundaries of legal responsibility and of what constitutes new or existing software are unclear. Examples of software include STarT BACK, a back pain RAT integrated into an electronic health record system, 49 and BMJ Informatica's implementation of Hamilton's 19 Cancer RAT, which interfaces with the electronic health record system. 13 In a previous paper in this journal, we described the development of a clinical decision-support tool with a focus on usability and clinical utility. 30 In this paper, we take a patient safety perspective to describe a case study of the development of a computerized, clinical decision-support system for cancer risk assessment embedded within an electronic health record system. As a class I medical device, the tool requires minimal consideration for patient safety despite influencing clinical decisions that, while not therapeutic nor diagnostic, nevertheless affect patient care and their journey through the health system. We argue that current European regulations are insufficient for facilitating safe development of healthcare information technology. Our intention is to share our experience to support research-led collaborative development of health information technology. The risk assessment system we implemented was a computerization of Hamilton's cancer RATs from the CAncer Prevention in ExetER (CAPER) studies ( Fig 1A) . There is strong case for computerization of Hamilton's cancer RATs because these studies identified features associated with subsequent cancer diagnosis that was clinically coded within patients' electronic health records. 19 The computerization also facilitates distribution of the tool through the UK primary care software systems, which can normalize and automate symptom detection. 51 The Macmillan Cancer Support implementations covered lung cancer and colorectal cancer, whereas our work included bladder, 52 colorectal, 53 kidney, 54 lung, 55 esophagogastric, 56 and ovarian cancers. 57 We did not implement the audit-table function available in the Macmillan Cancer Support implementations but did implement in-consultation prompts and an interactive symptom checker. The inconsultation prompt is an automated function that computed and presented a patient's CAPER RAT score for each cancer when the patient's record was retrieved at the start of the consultation. The symptom checker was a clinicianselected form that structures a patient discussion on other potentially relevant symptoms to compute a CAPER RAT score. We began by assessing the feasibility of creating the electronic-RAT within TPP SystmOne Demo version. SystmOne offers functionality to users for triggering automated protocols, developing e-forms to collect data, and defining bespoke clinical reports. It proved to be feasible to develop a prototype system using automated protocols to calculate an RAT score by processing bespoke clinical reports that queried a patient's records for symptoms ( Fig 1B) . It was also feasible to build a Symptom Checker e-form for focusing consultations. Such a system would be considered a class 1 medical device as per the MDR because it is a noninvasive device (Annex VIII Chapter III ¶ 4.1 Rule 1) that, although an active device (Art. 2 [4] ), is not intended to take decisions with diagnosis or therapeutic purposes (Annex VIII Chapter III ¶ 6.3 Rule 11). The requirements for a class 1 device are shown in Table 1 . 27 In the interests of maintaining a patient safety perspective, we will focus discussions on the General Safety and Performance Requirements and Benefit-Risk Analysis and Risk Management as described in Annex 1 and 2 of the MDR. Much of the requirements can be summarized within our software development lifecycle (Fig 2) . One-shot deal versus gradual reciprocity. Our iterative development approach prompted reflection on the safety standards and regulations. We noted that current regulations offer a one-shot deal wherein manufacturers are judged on a finished product produced by following selfdetermined standards. If the device fails review, the manufacture must restart product development. This is costly for manufacturers, which likely both discourages innovation and hinders learning. Gradual and reciprocal development and safety review might be preferable to encourage manufacturer engagement and integration of safety into device development. Hazard of assuming benignity. Manufacturers' selfclassification of devices' safety class is fundamental to the MDR. The safety classification rules in the MDR diligently classify devices that are expected to be potentially harmful and classify other devices with lesser concern. The implicit bias toward known hazards over unknown ones falls prey to the potential for emergent harm that, by definition, cannot be predicted (or are at least difficult to predict precisely). As a complex system, healthcare provision and regulation operate with a prediction horizon. 58 Far from ignoring or downplaying the events beyond this horizon, regulators should endeavor to prepare for such events as best as they can, similar to how they currently request manufacturers to foresee and prepare for consequences of a device's use. The extreme alternative to Rule 13 (ie, all other active devices are classified as class I) is to say that all devices that are not classified with previous rules must be subject to review by the Notified Body. From the latter perspective, the unknown is approached with caution rather than assumed benignity. Bias toward loss transfer. We also note that the regulations and standards have an unbalanced approach to loss assessment, loss control, and loss transfer. As noted by Chadwick et al, 36 For example, CE marking does not evidence the safety of a device so much as it evidences that a legal entity (the manufacturer) takes responsibility for adherence to safety regulations. Although the intention might be to encourage the legal entity to minimize the risk of harm potentially caused by the product, a simpler solution to adherence can be reached with an insurance model for handling risk, which is part of a loss-transfer approach. Such an approach attempts to manage the fallout of harm instead of upstream mitigation of hazards becoming harm. To illustrate the incoherence between safety and losstransfer approaches, consider insurance premiums for drivers. Premiums are an attempt to cover the costs of road traffic accidents but do not attempt to influence the risk of accidents, which is the product of likelihood and the magnitude of harm. 61 On the other hand, speed limits attempt to decrease the likelihood of accidents occurring and have been shown to influence the risk of road traffic accidents. 62 Thus, speed limits facilitate safety by addressing the harm-generating process further back in the causal chain, which is the intention behind the risk management planning requirements of CE marking. The loss-transfer approach might be used when the likelihood of harm is low, but the magnitude is high, and when the use of resources is valued using short-and mediumterm perspectives. 63 This tactic is likely due to a misunderstanding of probability that interprets a low-probability event as the one that will not happen until the distant future, as opposed to the one that can happen at any time. 64, 65 When combined with humans' tendency for temporal discounting, 66 the choice is made to endure the harm of low likelihood-high magnitude events in the future rather than mitigating them at present. Commercial entities with sufficient collateral can absorb the consequences of safety risks by playing the odds while a product earns in the market. On the contrary, it is more difficult to get products to market with patient safety insights from patient caregivers and safety researchers because existing regulations are not primarily designed for clinical and academic institutions to lead on development. The presence of a commercial bias was evidenced by a 2020 study showing that the public discourse around a regulatory framework for software as a medical device proposed by US Food and Drug Administration lacked scientific support and commonly involved undisclosed financial ties with industry. 67 Despite these criticisms, loss-transfer approaches are rational choices when it is more difficult to predict the behavior of a complex system than it is to manage its FIG 2. Software development lifecycle with five stages (S1-S5), each involving tasks (T1-T10) that produce outputs (O1-O20). Red nodes indicate tasks not undertaken as part of this exploratory development study. McInerney, Scott, and Johnson undesirable consequences. A dynamic approach to risk management that acknowledges a prediction horizon, multiple levels, dependencies, and adaptations is thus needed. 68 The loss-transfer approach has been useful for handling harm but ultimately should only be considered a stopgap while we improve our understanding of harmgenerating processes in the healthcare systems. Our experience of developing a computerized clinical decision-support system provided insight into the potential threats to patient safety inherent in the regulation of medical devices. Table 2 presents nine recommendations to address our identified concerns and the expected improvements for patient safety. Beginning with perhaps the most practicable suggestion, we recommend an explicit rather than implicit definition of class 1 devices, which could be managed by notified bodies [recommendation 1]. As noted in earlier reflections, classification of class 1 devices is implied when not covered by predetermined hazards. An alternative is for unclassified devices to be subject to review by the Notified Body. A second practicable suggestion is for explicit harmonized regulation of devices developed in-house [recommendation 2]. There is intentional lack of regulatory requirements for medical devices developed in-house that assumes regulations should not apply to devices "used only within health institutions…that support the healthcare system and/or address patient needs…since the aims of this Regulation would still be met in a proportionate manner." 27(p 5 ) Although we do not doubt healthcare institution's commitment to patient safety, there is concern about competing incentives, less-stringent national regulations, 48 and lack of knowledge and experience evaluating safety of medical devices. It is already accepted by the clinical academic community that the regulation of medical devices is unfit to protect patients against harm, with thwarted calls for medical devices to be regulated like pharmaceuticals. 69 At first glance, it might seem to be preferable to adopt the more stringent regulatory frameworks used by the European Medicines Agency. These frameworks take a strong Safety-1 approach, which is more concerned with minimizing false negatives than it is about promoting true positives, 70 in other words, stopping unsafe and ineffective devices getting to market even at the expense of hindering access to safe and effective devices. But the rapid innovation of medical devices does not lend itself well to such prolonged evaluations. The Safety-2 paradigm of Hollnagel et al, 70 however, focuses on promoting Manageably increased sensitivity to safety concerns during evaluation. Risk-sharing approach Responsibility for risk mitigation is solely on the manufacturer despite the fact that the safe development and use of devices involve multiple stakeholders. Also, loss-transfer approaches like insurance inevitably discourage innovators with less financial collateral. Broad and less-biased concept of risk. 6 Realign standards and regulations Current regulations are decoupled from standards, which permits gaming to expedite product to market. Constraints on perverse actions. Systems approach to conceptualizing risk Current regulation is biased toward handling known harms despite the complex nature of healthcare, meaning that some harms are emergent. Also, risk classification is separate for components intended to be used together, which ignores the potential for emergent harms from interactions. Increased sensitivity to emergent threats to patient safety. 8 Systems approach to patient safety Current regulation does not conceptualize health care as a complex system so the underlying conception of patient safety is inaccurate. Better understanding of what the structure of patient safety might be. 9 Systems model of accidents Current regulation does not conceptualize health care as a complex system so the underlying conception of causation is inaccurate. Better understanding of what the mechanism of harm generation might be. Abbreviation: IDEAL, Idea-Development-Exploration-Assessment-Long. Are Regulations Safe? Reflections From Developing a Cancer Tool processes that lead to safe performance despite the presence of hazard, by dynamically reflecting and adjusting performance. 71 Future regulations could benefit from adopting a Safety-2 approach to their design [recommendation 3]. Computerized decision-support systems are complex interventions and should be evaluated as such. 22 Nieuwenhuijse et al 72 suggest a regulatory approach reminiscent of Safety-2 by requiring controlled and evidence-based introductions of device innovations to safely handle upgrades, in their case, to orthopedic medical implants. This is one approach to regulate incremental innovation in medical devices, of which the development of software-within-software could be considered an example. 73 On a similar vein, the Idea-Development-Exploration-Assessment-Long-term framework champions gradual approval of medical devices rather than the oneshot approval of CE marking, which would allow graded, responsible, but earlier patient access. 74, 75 In the United States, the Software Pre-Cert Pilot Program focuses on the digital health technology developer rather than the product to support streamlined premarket review and learning from use in the market. 76 Such frameworks address the concern that the increased administrative burden of more stringent regulations might delay products that are imperfect but practically useful [recommendation 4]. 77 With respect to standards, we encourage risk sharing approaches, as promoted by ISO 31000 [recommendation 5]. 63 Such approaches distribute loss assessment, loss control, and loss transfer over all stakeholders at the cost of more complicated relationships between producers, providers, and users. This increased complicatedness of relationships will require resources, but such expense should be seen as an investment in improved system performance, rather than an inconvenience. If not risk sharing, at least an alternative to predominantly loss-transfer approaches would be an increase in the focus of loss-control approaches like risk mitigation. 78 These approaches are the second of three elements of a thorough risk management strategy for patient safety. 79 Although design control and risk management are explicitly mentioned in standards like ISO 13485, risk control is not explicitly mandated. Realignment of law and technical standards in the European Union might be required to facilitate this [recommendation 6]. 35 Of course, loss-control approaches are insufficient on their own. Health care is inherently risky and must approach safety by concurrently avoiding, managing, and embracing risk, depending on which of its range of services it is providing. 80 Although regulation contributes to the avoidance and management of risk, it is not well-placed to help with embracing risk, which requires adaptive processes within the healthcare systems. 80 The proposed solution is for actors in such an environment or system to rely on personal (rather than system) judgment, adaptability, and resilience. On the theme of complex systems, ISO 14971 recommends proactive identification and mitigation of hazards, which assumes at least an approximately deterministic system in which hazards and harms can be foreseen. Health care, however, is a complex adaptive system whose behavior can be emergent, nonlinear, and intractable to predict at arbitrary horizons. 81 Criticisms of the European Union's MDD 41 have partly been addressed in the impending MDR. It has become increasingly apparent, however, that existing regulation of medical devices is insufficient for the digital age 85 and there are difficulties inherent in reaching global coherence. 86 In this article, we argue that regulations in the European Union do not sufficiently address the complexity of healthcare information systems with consequences for patients' safety. Advocates for digital health care tout its speed, coverage, and capacity but perhaps without considering its own suite of challenges. Future development of regulations should make it easier for clinical and academic institutions to produce healthcare information technology so that they contribute their patient care and safety science insights. Finally, future development of standards should continue to follow a systembased view to risk of healthcare information technology The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I =Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs. org/cci/author-center. Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments). We would like to thank Macmillan Cancer Support and acknowledge Prof Willie Hamilton, Dr Raff Calitri, and the ERICA team for their contribution. The role of primary care in early detection and follow-up of cancer Software as a Medical Device (SAMD): Clinical Evaluation Guidance for Industry and Food and Drug Administration Staff. FDA Guid The European Parliment and The Council of the European Union: Council Directive 93/68/EEC of 22 Prostate cancer survival in the United States by race and stage Cancer survival by stage at diagnosis in Kuwait: A population-based study Worldwide comparison of ovarian cancer survival: Histological group and stage at diagnosis (CONCORD-2) Jemal A: Cancer statistics Improving early diagnosis of symptomatic cancer Impact of investigations in general practice on timeliness of referral for patients subsequently diagnosed with cancer: Analysis of national primary care audit data Facilitating early diagnosis of lung cancer amongst primary care patients: The views of GPs The Andersen model of total patient delay: A systematic review of its application in cancer diagnosis Effect of interventions incorporating personalised cancer risk information on intentions and behaviour: A systematic review and meta-analysis of randomised controlled trials Risk prediction tools for cancer in primary care Decision aids for people facing health treatment or screening decisions Digital process innovation for patient centred cancer symptom management Comparison of symptoms and presentation of women with benign, low malignant potential and invasive ovarian tumors Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: Prospective cohort study Identifying patients with suspected lung cancer in primary care: Derivation and validation of an algorithm The CAPER studies: Five case-control studies aimed at identifying and quantifying the risk of cancer in symptomatic primary care patients Evaluation of risk assessment tools for suspected cancer in general practice: A cohort study Viewpoint: Controversies surrounding use of order sets for clinical decision support in computerized provider order entry Effects of computerised decision support systems on nursing performance and patient outcomes: A systematic review Information in practice systematic review of trials to identify features critical to success Why should you trust a decision support system? Toward a quality and safety culture for knowledge based systems Computer-assisted diagnostic decision support: History, challenges, and possible paths forward The European Parliament and The Council of the European Union: Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/ EEC and 93/42/E. Off J Eur Union Medicines and Healthcare products Regulatory Agency: An Introductory Guide to the Medical Device Regulation (MDR) and the In Vitro Diagnostic Medical Device Regulation (IVDR) Lines M: Introduction to Disciplined Agile Delivery Multidisciplinary software design for the routine monitoring and assessment of pain in palliative care services: The development of PainCheck Software safety: Why, what, and how Software-as-a-medical device: Demystifying connected health regulations Safety of patient-facing digital symptom checkers The European Parliament and The Council of the European Union: Regulation (EU) 2020/561 of the European Parliament and of the council of 23rd April 2020 amending Regulation (EU) 2017/745 on medical devices, as regards the dates of application of criterion of its provisions EU and member state medical devices regulation Functional safety of health information technology IEC: Functional Safety of Electrical/electronic/programmable Electronic Safety-Related Systems Part 1: General Requirements DCB0129: Clinical Risk Management: Its Application in the Deployment and Use of Health IT Systems-v4 DCB0160: Clinical Risk Management: Its Application in the Deployment and Use of Health IT Systems-v3 Europeans are left to their own devices Offline: The scandal of device regulation in the UK On the complex definition of risk: A systems-based approach Measuring and improving patient safety through health information technology: The health IT safety framework Complex adaptive systems theory in information systems research-A systematic literature review Health information systems: Failure, success and improvisation Medical device recalls and the FDA approval process Regulation of medical devices in the United States and European Union MHRA: Guidance: In-House Manufacture of Medical Devices The implementation of STarT BACK and an OA e-template: Utilising existing GP electronic clinical systems to manage patients with low back pain or osteoarthritis during a routine consultation Embedding electronic decision-support tools for suspected cancer in primary care: A qualitative study of GPs' experiences The Phoenix Partnership: SystmOne Clinical features of bladder cancer in primary care Clinical features of colorectal cancer before diagnosis: A population-based, case-control study Clinical features of kidney cancer in primary care: A case-control study using primary care records What are the clinical features of lung cancer before the diagnosis is made? A population based case-control study The risk of oesophago-gastric cancer in symptomatic patients in primary care: A large case-control study using electronic records Risk of ovarian cancer in women with symptoms in primary care: Population based case-control study Health care organizations as complex adaptive systems Operating at the sharp end: The complexity of human error Approaches to defining risk Effective interventions for unintentional injuries: A systematic review and mortality impact assessment among the poorest billion Fundamentals of Risk Management: Understanding, Evaluating and Implementing Effective Risk Management People's intuitions about randomness and probability: An empirical study The extent of patients' understanding of the risk of treatments Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making Industry ties and evidence in public comments on the FDA framework for modifications to artificial intelligence/machine learning-based medical devices: A cross sectional study Risk management in a dynamic society: A modelling problem Why aren't medical devices regulated like drugs? From Safety-I to Safety-II: A White Paper Learning from incidence in health care: Critique from a safety-II perspective Appraisal of evidence base for introduction of new implants in hip and knee replacement: A systematic review of five widely used device technologies Regulating incremental innovation in medical devices IDEAL-D: A rational framework for evaluating and achieve this goal No surgical innovation without evaluation: Evolution and further development of the IDEAL framework and recommendations US Food and Drug Administration: Digital Health Innovation Action Plan PP31 medical device Regulations: What is new? Fundamentals of Risk Management: Understanding, Evaluating and Implementing Effective Risk Management Setting up a risk management department Approaches to safety: One size does not fit all Sea gulls, butterflies, and grasshoppers: A brief history of the butterfly effect in nonlinear dynamics Systems approach and systems engineering applied to health care: Improving patient safety and health care delivery Systemic accident analysis: Examining the gap between research and practice A new accident model for engineering safer systems OP143 assessment of mHealth apps: Is current regulation policy adequate? Regulation of medical devices outside the European Union