key: cord-299254-kqpnwkg5 authors: Sun, Yingcheng; Guo, Fei; Kaffashi, Farhad; Jacono, Frank J.; DeGeorgia, Michael; Loparo, Kenneth A. title: INSMA: An integrated system for multimodal data acquisition and analysis in the intensive care unit date: 2020-04-28 journal: J Biomed Inform DOI: 10.1016/j.jbi.2020.103434 sha: doc_id: 299254 cord_uid: kqpnwkg5 Modern intensive care units (ICU) are equipped with a variety of different medical devices to monitor the physiological status of patients. These devices can generate large amounts of multimodal data daily that include physiological waveform signals (arterial blood pressure, electrocardiogram, respiration), patient alarm messages, numeric vitals data, etc. In order to provide opportunities for increasingly improved patient care, it is necessary to develop an effective data acquisition and analysis system that can assist clinicians and provide decision support at the patient bedside. Previous research has discussed various data collection methods, but a comprehensive solution for bedside data acquisition to analysis has not been achieved. In this paper, we proposed a multimodal data acquisition and analysis system called INSMA, with the ability to acquire, store, process, and visualize multiple types of data from the Philips IntelliVue patient monitor. We also discuss how the acquired data can be used for patient state tracking. INSMA is being tested in the ICU at University Hospitals Cleveland Medical Center. Each year, more than four million acutely ill patients are admitted to intensive care units (ICUs) in the U.S. alone; approximately 500,000 of them do not survive [1] [2] [3] . In extreme situations, like the current COVID-19 pandemic, ICUs are essential for treating critically ill coronavirus patients. Given the high stakes involved, timely and effective care is paramount, and this requires continuous patient surveillance using sophisticated monitoring equipment. As a result, ICUs are complex, data-intensive environments and dozens of systemic parameters are monitored, including heart rate, respiration, arterial blood pressure, oxygen saturation, temperature, end tidal CO2 concentration, etc. Enormous volumes of multimodal physiological data are generated including physiological waveform signals, patient monitoring alarm messages, and numerics and if acquired, synchronized and analyzed, this data can been effectively used to support clinical decision-making at the bedside [10, 18] . Clinical personnel rely on information from these signals provided on the patient monitor display for visual assessment or as numerics in the EMR to understand the current state of the patient, and how it is changing over time. Continuous digital monitoring is intended to allow clinicians to dynamically track changes in patient state more closely than would be possible with more sporadic measurements [4] . The hope has been that this would allow for more accurate diagnosis, earlier anticipation of deterioration, and a clearer understanding of the impact of administered treatments, improving quality of care and lowering costs [5] . Even when data can be viewed in real-time, standard approaches provide little insight into a patient's actual pathophysiologic state. Understanding the dynamics of critical illness requires precisely time-stamped physiologic data (sampled frequently enough to accurately recreate the detail of physiologic waveforms) integrated with clinical context, but this will produce an overwhelming amount of data-far too much to be routinely reviewed manually. It is thus necessary to develop a data acquisition system that facilitates the access and review of historical data for medical personnel. The acquired data needs to be synchronized across disparate devices, archived, analyzed and presented to clinical personnel in a manner that supports clinical decision-making at the patient bedside. Previous work includes, for example: Matam and Duncan adopt the real-time data recording system used for F1 race cars to acquire and analyze data from bedside monitors in the pediatric intensive care unit [6] . This system supports the storage and review of electrocardiogram (ECG) data retrospectively. Raimond et al. developed a platform called "WaveformECG" that provides interactive analysis, visualization and annotation of ECG signals [7] . Alexander et al. developed an alarm data collection framework to acquire all alarms generated from Philips IntelliVue MP70 patient monitors installed in each ICU room with the objective of reducing false alarms by leveraging annotations provided by clinicians [4] . Hyung and Chul introduced a physiological data acquisition and visualization program "Vital Recorder" with a user-friendly interface similar to that of a video-editing program for anesthesia information management, where Physiological data can be manipulated like editing video clips [8] . Much of the previous work has focused on the acquisition or visualization of certain physiological data, a complete general purpose solution for data collection, analysis and visualization of multimodal ICU data is currently unavailable. ICU clinical personnel need the ability to effectively deal with different data sources on a patient or departmental level, and need advanced analytic methods that transform this data to actionable and clinically meaningful outcomes for each patent. We have been working on building the Integrated Medical Environment (tIME) [10] to address this critical opportunity and in this paper, we discuss an integrated system (INSMA) that supports multimodal data acquisition, parsing, real-time data analysis and visualization in the ICU. In the current implementation, INSMA acquires data from the Philips IntelliVue patient monitor, and has the ability to store and review the multimodal data acquired either in real-time or on request. The system has been tested in the ICU at University Hospitals Cleveland Medical Center, for multimodal data analysis and patient state tracking. The remainder of the paper is organized as follows. In Section 2, we discuss some related work. In Section 3, we present the INSMA framework, and introduce the details of the data acquisition, parsing and visualization modules. In Section 4, we discuss the applications of our proposed system. We conclude our work and suggest possible future work in Section 5. ICUs provide treatment to patients with the most severe and life-threatening illnesses and injuries. It requires uninterrupted attentiveness and medical care from various clinical specialists and medical equipment to sustain life and help nurture the patients back to health. Effective and reliable patient monitoring and data analysis are of ultimate importance in the ICU to ensure early diagnosis, timely and informed therapeutic decisions, effective institution of treatment and follow-up [30, 32] . Several clinical information systems have been developed from both industry and academia to meet the demanding needs of the ICU. General Electric (GE) Co.'s Centricity Critical Care system introduced in 2003 creates actionable insight across the healthcare system and the care pathway in intensive care units, enabling enhanced clinical quality and operational efficiency. The system collects data from monitors and ventilators and displays it in spreadsheets reminiscent of the typical ICU chart. Data are collected from medical devices through device interfaces that connect with GE's Unity Interface Device network [10] . The Datex-Ohmeda S/5™ Collect program proposed by GE Healthcare can obtain high-resolution data from the Datex-Ohmeda S/5™ series monitors [13] . The program was developed for Windows XP and is not compatible with current Windows operating systems, and the manufacturer does not intend to update it. Philips offers data management solutions that link the Philips ALS monitor/defibrillator and AED and allow quality assurance officers using a direct connection that downloads and forwards every event automatically. Quality assurance officers can then retrieve and review an event summary with confidence [11] . Often, the commercial off-the-shelf products do not support the acquisition, archiving, or annotations of high-resolution physiologic data with bedside observations for clinical applications. Systems have also been developed in academic settings primarily to support clinical research. Tsui et al. developed a system to acquire, model, and predict ICP in the ICU using wavelet analysis for feature extraction [12] . Goldstein et al. proposed and developed a physiologic data acquisition system that could capture and archive parametric data, but the annotation of important clinical events such as changes in a patient's condition or timing of drug administration, was limited [22] . Kool et al. reported that they collected numerical data at five-second intervals from the Datex Ohmeda S/5 TM monitoring system using their own information management system [14] . Liu et al. [15] also reported the collection of vital signal data from 32 surgical patients, from Philips IntelliVue MP series monitors, using a self-developed program that was not disclosed. Lee and Jung developed an anaesthesia information management system (AIMS) for the acquisition of high-quality vital signal data (Vital Recorder) to support research [8] . Physiological data of surgical patients were collected from 10 operating rooms by Vital Recorder through the patient monitor, anaesthesia machine and bispectral index monitor. Winslow et al. proposed a platform called WaveformECG for visualizing, annotating, and analyzing ECG data [7] . As discussed in the first section, these systems only focus on acquiring and analyzing one or two types of physiological data, and that is not sufficient for ICU applications. Matam and Duncan used real-time data recording software, ATLAS from McLaren Electronics Systems that continuously monitor and analyze data from F1 racing cars to implement a similar real-time data recording platform system adapted with real time analytics to suit the requirements of the intensive care environment [6] . The parameter data recorded by Philips MP30 bedside monitors can be transferred to the server in real-time. However, such a third-party data acquisition tool is not flexible enough to customize the functions according to the clinician's requirements, and the compatibility of the data format is another issue. To address the issues described above, our research proposed the INSMA with the aim of obtaining clinical physiological data including electroencephalography (EEG), electrocardiography (ECG), photoplethysmogram (PPG), peripheral capillary oxygen saturation (SpO2), blood pressure (BP) and other signals to be acquired and stored for data sharing, mining, analysis and visualization. The primary data source in our first implantation comes from the IntelliVue MP (Philips, Germany) series of monitors. INSMA contains three independent but data related modules: data acquisition module, parsing module and visualization module. Figure 1 shows the INSMA architecture and its data flow. The Data Acquisition Module establishes the connection with the patient monitor and requests the physiological measurements that are acquired. The raw multimodal data obtained from the monitor includes physiological waveforms, alarm messages and numeric (vitals) data. The specific types of data to be acquired can be chosen by the users according to their needs through the "Data Type Selector" and "Physiological Signal Selector". The data transport rate can also be set by the "Serial Port Selector". Once the raw data has been acquired from the monitor, the Data Parsing Module will process, parse and transform the data into a time-series using the physiological identifiers or codes provided by the monitor. The Data Visualization Module will display the graphs for the parsed time-series results. It can plot both real-time signals and historical data given a time range. All three modules are developed using MFC and C/C++, so that they all run in the same operating environment and use compatible data formats, and therefore provide a complete solution for data acquisition, parsing and visualization. The details of these modules are discussed in next sections. The bedside patient monitor is the most common long-term monitoring medical device used in an ICU. It is used to continuously monitor the physiological parameters of an patient through specially designed sensors, signal acquisition modules, and invasive or noninvasive interfaces: cardiac activity including ECG and heart rate, circulation including blood pressure & cardiac output indices, SPO2, respiratory function including respiration rate, oxygenation, capnography, and Brain through EEG waveforms and derived indicators, temperature and metabolic rate, etc. The Philips IntelliVue MP70 is a bedside patient monitoring device that displays various physiological waves (e.g. ECG and Blood Pressure) and provides important functions such as displaying numeric vitals data (e.g. heartrate, oxygen saturation) and performing alarm functions based on minimum and maximum limits set by the clinical staff in the monitor. A variety of sensors and associated clinical measurement modules can be connected to the monitor, and these modules are generally interchangeable with other monitors provided by Philips [17] . One of these modules is the Philips VueLink module that provides an interface to more than 100 third-party specialty measurement devices like Baudrate Protocol is a connection-oriented, message-based request/respond protocol, based on an object-oriented model concept. All information is stored as attributes within a set of defined object types. The following objects are defined in the protocol: Medical Device System (MDS), Alert Monitor, Numeric, and Patient Demographics. In order for a client application to access the attributes of instantiated objects, it first has to poll the MDS object. Then, the client gets the information of the instantiated object via queries that return the attribute values of these objects. After building the association, the following data can be accessed from the IntelliVue monitor: all measurement numerics and alarm data (real-time update rates up to 1024 ms), wave data, and patient demographic data entered by the user in the IntelliVue monitor. The Data Acquisition Module collects and stores real-time data from patient monitors in intensive care units for further data analytics that supports clinical decision-making. We developed an interface using MFC, to make the data acquisition process easy to be controlled by users. Figure 2 shows the interface with function areas 1-5. The data acquisition and preprocessing tools can perform high-resolution recording and processing tasks, such as simultaneously recording of 1-3 ECG (at 500 samples/s) channels, and additionally up to 8 non-ECG (at 125 or 62.5 samples/s) waves, along with other signal types such as all available numeric values and alert messages. After the program is started, a text file including the monitoring results of the selected data type and signals will be generated for further analysis. The file is named by the date time and patient's demographic information. The Data Parsing Module runs synchronously with the Data Acquisition Module to continuously parse real-time data being streamed from the monitor to increase the efficiency of the data collection and archiving process and to also When parsing the data, we first identify each frame by locating its BOF (0xC0) and EOF (0xC1), and get the type and length of the message from Hdr. Next, we interpret the time stamp from the User Data. In the Data Export Protocol defined by Philips IntelliVue monitor, the time stamps contain two types of data: absolute time and relative time. For the waveform signals, the IntelliVue patient monitor supports the wave types ( Table I) that are defined by sample period, sample size, array size, update period and bandwidth requirement. The Data Visualization Module can display the multimodal data including the wave signals and numeric data of patients. Figure 4 illustrates the main interface. The timestamp is displayed above the chart. The selector (3) lists all the numeric data types obtained from the parsed results. When a numeric data type is chosen, its value will be displayed in the panel (4) in real-time. The list in (3) will update automatically when the program finds "new" data type in the parsed results of that numeric data. The numeric data is displayed as markers (dots) instead of a continuous curve, and each dot represents one data value at that time point. Different types of waveforms or numeric values are displayed in different colors, so it is easier for users to distinguish them. All the control commands are displayed in panel (5) in a log style. The patient's ID, wave and numeric data type list can be set in the "Option" menu. In addition to providing the patient's physiological status in a real-time mode, the Data Visualization Module can also display waveforms from archived data. Users first need to choose the patient and types of waves that are to be visualized, and then set the time range in the "Plot Setting" window, as shown in figure 5. In order to evaluate the performance of INSMA, the system was deployed in the Neurological Surgery ICU at University Hospitals Cleveland Medical Center using a DELL minicomputer with an Intel(R) Dual Core Celeron processor, 2 GB of RAM, and 240 GB hard disk storage. Results to date indicate that it is reliable for collecting data from the patient monitor. The waveform signals (e.g. ECG, respiration, PLETH/CO2), numeric signals and alarm event signals when streamed continuously from the monitor over a 24-hour period generate an approximately 400MB data file for each patient. The parsed results will be larger in size because the absolute time stamp information is added to each sample point in the file. how the patient is progressing by observing the ART and ICP waveforms, and how they are temporally correlated. We also collected alarm messages and numeric measurements in the parsed results. The analysis of ICU data from patients in the clinical setting is generally limited to visualizing waveform and numeric data and computing simple values such as average heartrate, average respiratory rate, average blood oxygen saturation, etc. In INSMA, each waveform can be analyzed independently or in conjunction with other waveforms to extract more information, as shown in Fig. 7 . It is possible to zoom in to provide additional waveform details for visual inspection or apply different analytical analysis techniques to single or multiple waveform signals to better understand the status of the patient and support clinical decision-making. It has been well established that feature extraction for quantifying the complexity and/or variability in physiological time-series data can provide important information related to health and disease [29] . Specifically, even though temporal patterns of variability can be leveraged as powerful diagnostic and/or prognostic indicators, the current use of beat-tobeat and cycle-to-cycle variability dynamics at the bedside is hampered by: (1) lack of high-resolution real-time multimodal clinical data, (2) non-trivial interpretation and integration of these variability metrics into clinical workflows, and (3) lack of a unified framework for classifying variability dynamics into meaningful clinical categories. Algorithms that quantify variability dynamics over multiple temporal scales, such as multiscale entropy (MSE) and multifractal detrended fluctuation analysis (MFDFA) have shown a lot of promise as diagnostic tools in clinical research settings, but the difficulty in interpreting these measures by non-specialists prevents their routine implementation and use in the ICU. The acquired data from patient monitors can be used to develop novel and generalizable methods for quantifying and tracking patient state in real-time [20] . We are developing a patient state tracking system based on the analysis of physiologic time-series dynamics as shown in Fig. 8 . In STAGE II, the data is analyzed using the beat-to-beat or cycle-to-cycle time-series data that is of interest. A new dataset is analyzed (Step 4) with the same algorithm used in Step 2, the ANN classifies the output of the algorithm in Step 5, and the result of the ANN classification is then mapped into the physiological phase space in Step 6. This methodology reduces the dimensionality of multiscale variability dynamics in a clinically relevant manner, thereby facilitating the development of clinician-centric visualization tools that can be implemented in a bedside display, and easily integrated in the ICU workflow as a generalized early warning system for clinical decompensation in ICU patients [19] . Any algorithm that quantifies multiscale variability dynamics [16] [23] can be used to process the waveform data in order to classify the information extracted from the raw data in an intuitive and physiologically relevant manner [24] [25] , and thus to facilitate the incorporation of subtle and dynamic fluctuations in physiological waveform data. By assessing the current status of a patient in the ICU, the system will provide a wealth of information on future trajectories for extracting related clinical information [26] [27] [28] . The amount of data that is available for clinicians to use in support of real-time patient care at the bedside is growing rapidly as a result of advances in medical monitoring and imaging technology. Advances in informatics, whether through data acquisition, physiologic alarm detection, or signal analysis and visualization for decision support have the potential to markedly improve patient treatment in ICUs. Clinical monitors have the ability to collect and visualize important numerics or waveforms, but more work is needed to interface to the monitors and acquire and synchronize multimodal physiological data across a diverse set of clinical devices. Patient monitors offer the opportunity to acquire a number of different physiological signals in a single device, but in certain cases there are other monitors and devices whose data is critical to patient care, but do not interface to the patient monitor. The tIME framework that we are developing is directly addressing this unmet clinical need. An integrated solution for multimodal data acquisition, parsing and visualization in the ICU (INSMA) presented in this paper is an important first step in achieving this overall vision [20] . Particularly in the Neuro-Intensive Care Unit, there are a variety of different devices that provide valuable information for patient care that do not interface directly to a patient monitor including EEG signal data, real-time tissue blood flow (perfusion) data, and advanced hemodynamic data monitoring (e.g. continuous cardiac output) that are cornerstones in the management of critically ill patients. There are options, for example, with Nihon-Kohden EEG acquisition systems to collect patient vitals (similar to a bedside patient monitor) as well as interface to specialized devices such as for hemodynamic monitoring. Simultaneous acquisition of data from Philips patient monitors and Nihon-Kohden EEG systems in the ICU was done to augment data provided in the MIMIC study [21] . The objective of data acquisition was to stream real-time data from both monitors for archiving in a single biorepository. This provides valuable data for research, but the intent of tIME is to stream data for real-time patient care at the bedside. INSMA is an important first step, and we have also developed data techniques for synchronizing data acquisition from a variety of different ICU devices as a core technology for future implementations of tIME in the ICU. We have also demonstrated that patient data acquired from the patient monitor can be used for patient state tracking. The prototype system we developed was optimized to identify the type of dynamics observed in cardiac (ECG or blood pressure) beat-to-beat time-series data collected from ICU patients. The prototype system has been tested using ICU patient data from ECG to understand how variability in the heartbeat time-series can be used to dynamically track patient state [19] . In the current development of INSMA and tIME, we have implemented the INSMA software on Lenovo ThinkCentre M600 computers, and currently have one system connected to the Philips patient monitor in the Neurosurgery ICU at University Hospitals Cleveland Medical Center under the direction of Dr. DeGeorgia to continuously collect patient data. We are completing the development of 21 additional INSMA units that will be connected to each of the Neurosurgery ICU beds. The INSMA system operates in background and once setup for data collection at the bedside and does not require attention from any clinical ICU personnel. INSMA allows the client to send messages to request for patient demographic information, and we implemented a patient demographic request function that monitors any modification of patient information. New patient demographic information will be entered when patients admitted into the ICU are connected to the monitor, and this information will be requested and stored in the raw data file. When the patient is discharged and a new patient is admitted and connected to the monitor, the parsing algorithm will capture the change of patient demographic information (e.g. First Name, Last Name, Age, Weight) and a new patient archive corresponding to the new patient identifier will be created, as shown in Fig. 9 . Each unit is equipped with a wireless communication link that supports remotely monitoring the INSMA units only from within the hospital firewall to protect the privacy and security of the data, and then also moves the data to a permanent secure data storage unit. Example data from patient monitoring is shown in Fig. 10 . The future of critical care will require "information management", that includes the real-time collection, integration, and interpretation of various types of physiological data from multiple sources. The possible research work will focus on (1) the integration and analysis of massive heterogenous medical data to provide scientific decision-making with machine learning methods [9] [31], and (2) the acquisition and processing of vast amount of multi-channel high-density and real-time streaming data using multivariate and nonlinear time series analysis methods to facilitate rapid diagnosis and treatment [32] . Patient care in the ICU can be significantly improved through the application of complex system analysis and information management methods. Daily cost of an intensive care unit day: the contribution of mechanical ventilation Evaluation of acute physiology and chronic health evaluation III predictions of hospital mortality in an independent database Estimating Lives and Dollars Saved from Universal Adoption of the Leapfrog Safety and Quality Standards Clinician-in-the-Loop Annotation of ICU Bedside Alarm Data Big data analytics in healthcare: promise and potential Technical challenges related to implementation of a formula one real time data acquisition and analysis system in a paediatric intensive care unit WaveformECG: A Platform for Visualizing, Annotating, and Analyzing ECG Data Vital Recorder-a free research tool for automatic recording of high-resolution time-synchronised physiological data from multiple anaesthesia devices Intensive care window: A multi-modal monitoring tool for intensive care research and practice Information technology in critical care: review of monitoring and data acquisition systems for patient care and research Streamlining Data Management Workflow Acquiring, modeling, and predicting intracranial pressure in the intensive care unit Entropy Module and the Bispectral Index® monitor during propofol-remifentanil anesthesia Artifacts in research data obtained from an anesthesia information and management system University of Queensland vital signs dataset: Development of an accessible repository of anesthesia patient monitoring data for research Context aware image annotation in active learning Intensive care window: A multi-modal monitoring tool for intensive care research and practice Data Acquisition and Complex Systems Analysis in Critical Care: Developing the Intensive Care Unit of the Future A Framework for Patient State Tracking by Classifying Multiscalar Wavform Features Neurocritical Care Inforamtics: Translating Raw Data into Bedside Action System for Collecting Biosignal Data from Multiple Patient Monitoring Systems Physiologic data acquisition system and database for the study of disease dynamics in the intensive care unit Learning-based adaptation framework for elastic software systems Context aware image annotation in active learning with batch mode Eliminating Search Intent Bias in Learning to Rank Information extraction from free text in clinical trials with knowledge-based distant supervision Knowledge-guided text structuring in clinical trials Complex query recognition based on dynamic learning mechanism A common gene expression signature analysis method for multiple types of cancer Deep learning for heterogeneous medical data analysis. World Wide Web Opinion spam detection based on heterogeneous information network Raim: Recurrent attentive and intensive model of multimodal patient monitoring data  Integrated system for ICU multimodal data acquisition, analysis and visualization Yingcheng Sun: Conceptualization, Methodology, Software, Visualization, Writing -Original Draft. Fei Guo: Conceptualization, Software, Methodology, Visualization. Farhad Kaffashi: Conceptualization, Methodology, Formal analysis. Frank J. Jacono: Resources, Data Curation, Validation. Michael DeGeorgia: Resources, Data Curation, Validation. Kenneth A. Loparo: Conceptualization, Funding acquisition, Investigation, Methodology This work was supported in part by AHRQ Grant 1R18HS022860-01A1 (PI: Dr. Leo Kobayashi). ☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: