key: cord-256862-038rrdtp authors: Oland, Gabriel; Garner, Omai; Maurice, Annabelle de St title: Prospective Clinical Validation of 3D Printed Nasopharyngeal Swabs for Diagnosis of COVID-19 date: 2020-10-22 journal: Diagn Microbiol Infect Dis DOI: 10.1016/j.diagmicrobio.2020.115257 sha: doc_id: 256862 cord_uid: 038rrdtp COVID-19 greatly disrupted the global supply chain of nasopharyngeal swabs, and thus new products have come to market with little data to support their use. In this prospective study, two new 3D printed nasopharyngeal swab designs were evaluated against the standard, flocked nasopharyngeal swab for the diagnosis of COVID-19. 70 adult patients (37 COVID-positive and 33 COVID-negative) underwent consecutive diagnostic RT-PCR testing, with a flocked swab followed by one or two 3D printed swabs. The “Lattice Swab” (manufacturer Resolution Medical) demonstrated 93.3% sensitivity (95% CI, 77.9-99.2%) and 96.8% specificity (83.3-99.9%), yielding κ = 0.90 (0.85-0.96). The “Origin KXG” (manufacturer Origin Laboratories) demonstrated 83.9% sensitivity (66.3-94.6%) and 100% specificity (88.8-100.0%), yielding κ = 0.84 (0.77-0.91). Both 3D printed nasopharyngeal swab results have high concordance with the control swab results. The decision to use 3D printed nasopharyngeal swabs during the COVID-19 pandemic should be strongly considered by clinical and research laboratories. COVID-19, the clinical disease caused by the pathogen SARS-CoV-2, has presented unique challenges to the global community. Among these challenges is the massive need for populationwide diagnostic testing to contain the COVID-19 pandemic. In the United States, various expert groups have estimated a national need for anywhere from 500 thousand to 30 million targeted tests on a daily basis to mitigate and to suppress viral transmission (1) (2) (3) (4) . This immense need combined with traditional swab manufacturing supply chain interruptions (5) presented an Diagnostic Microbiology and Infectious Disease -Manuscript Submission -3 -opportunity for the additive manufacturing industry (also called the 3D printing industry) to fill the need for large-scale production of nasopharyngeal (NP) swabs. In the span of days-to-weeks, numerous 3D printing companies went through an iterative design process (6, 7) to meet the requirements of the NP swab: 1) to collect an adequate patient specimen from the nasopharyngeal mucosa, and 2) to elute the sample into a reverse transcription polymerase chain reaction (RT-PCR) assay without affecting the quality of results. Quality design controls and early clinical use have demonstrated similar safety profiles to NP swabs currently on the market (8) . 3D printed NP swabs are classified as Class I, 510(k) Exempt in vitro diagnostic medical device by the U.S. Food and Drug Administration (FDA) (9) . This designation implies that manufacturers need to follow Good Manufacturing Practices and have a Quality Management System, but it does not dictate the clinical performance standards for the swabs. The clinical validation process is at the discretion of the individual diagnostic laboratory to determine whether a particular NP swab should be considered for diagnostic use for COVID- 19 . The rapid emergence of numerous 3D printed swabs options also implies a dearth of widelypublished clinical efficacy data to recommend for or against their use. Given how important clarity is in the diagnosis of COVID-19, this study sought to determine the clinical performance of 3D printed nasopharyngeal swabs compared to FDA-approved nasopharyngeal swabs already on the market. Many swab designs were considered but two swab types were ultimately chosen for this study based on swab tip and shaft design, material, comfort during use, and ability of the company to manufacture large quantities: the Generation 1 "Lattice Swab" by Resolution Medical (10) , and the "Origin KXG" by Origin Laboratories (11), as shown in Figure 1 . A minimum sample size of 30 COVID-positive and 30 COVID-negative patients for each NP swab type was chosen based on the FDA's recommendation for clinical evaluation of new diagnostics eligible for emergency use authorization (EUA). Inclusion criteria for participants were: 18 years of age and older; an outpatient scheduled for testing in the preprocedural or preoperative setting to confirm COVID-negative status; a hospitalized inpatient having already received a COVID-positive test result in the prior 24 hours. Ambulatory outpatients underwent simultaneous swabbing with the control and experimental swabs. Known COVID-positive Diagnostic Microbiology and Infectious Disease -Manuscript Submission -5 -inpatients underwent serial swabbing with the experimental swab(s) within the 24-hour timeframe of their initial control swab.. Patients assessed for the study were excluded if they had conflicting prior COVID diagnostic results, if they were physiologically unstable for swabbing, had recent facial trauma or nasal surgery, had a platelet count <50,000 cells/μl or absolute neutrophil count <500 cells/μl. All patients underwent a verbal consent process approved by the Institutional Review Board at the University of California, Los Angeles. Sample collection was performed by trained nurses to ensure a high-quality sample. The swab was inserted into either the right or left naris based on nurse and patient preference until the nasopharynx was reached. The swab was then gently rotated for several seconds and withdrawn. This process was used for both experimental and control swabs. With each subsequent swab, the choice of naris was again at the discretion of the nurse and patient. Collections that were felt to be likely inadequate samples based on the nurses' prior experience were immediately discarded. Control samples were collected with the Universal Viral Transport Kit by Becton, Dickinson & Company, which consists of a flocked swab and 3mL Universal Viral Transport medium (12) . Experimental swab samples were collected consecutively to the control and transported in sterile polypropylene tubes with 3mL saline (0.9% sodium chloride in water) as transport media. After collection, the samples were promptly delivered to the laboratory and if not immediately processed via RT-PCR, were stored for less than 48 hours at 4C until processing. The RT-PCR RT-PCR results are reported categorically as COVID-positive or COVID-negative. To analyze performance differences between swab type, sensitivity and specificity with 95% confidence intervals (CI) for each experimental swab type was determined in relation to the control swab. Categorical concordance was calculated using Cohen's kappa (), which is a measure of nonrandom agreement between two different tests, with a k > 0.8 implying excellent agreement. All statistical calculations were performed using SAS version 9.4 (SAS Institute, Cary, NC). Since the Simplexa assay detects both the S gene and ORF1ab gene, these data were analyzed separately. For quantitative Ct value comparison in the COVID-positive patients, a Wilcoxon Signed-Rank test was performed for paired, dependent, nonparametric data between each experimental swab and control, as well as between the two different experimental swabs. In order to preserve data integrity and reduce bias that would result from excluding "non-detected" gene fragments, these results were instead imputed as the value 40, the maximum Ct for the RT- The COVID-negative patient samples were obtained predominantly in the outpatient "drivethrough testing" setting (32 of 33 patients), thereby undergoing the control and experimental swabbing simultaneously. Patient recruitment demographics are represented in Table 1 Performance data of the experimental swabs can be seen in Figure 2 (raw PCR data is included in Supplemental Table S1 ). Table 2 and visualized in Figure 3 , which overall shows dense clustering around the 1:1 line. Therefore, clinicians need to select the most promising designs prior to use in a clinical setting, as was done in this study. The FDA has issued EUAs to COVID-19 molecular diagnostic platforms that show promising early results, with a minimum data requirement of 30 COVID-positive and 30 COVID-negative patients (15) . This small sample size reflects a compromise between demonstrating early convincing data to allow for widespread diagnostic use and getting new technologies to market urgently during a crisisassuming there is careful ongoing monitoring and re-evaluation of (13) . The use of saline media was unavoidable due to shortages in supplies during the pandemic. It is clear that 3D printed swabs have an important role to play in responding to the COVID-19 pandemic, yet certain limitations of this study need to be understood to make its findings actionable. First, there is inherent error in treating any currently-approved NP swab as the provisional "reference standard" for comparison (16) (17) (18) . For example, if two identical swabs were collected simultaneously from a single patient, those two swabs still may not show perfect concordance due to inherent variation in sample collection. Realizing that the comparator reference standard test is imperfect changes the meaning of test sensitivity and specificity, particularly for patients near the limit of detection with a high Ct value. Second, the clinical sensitivity of NP swabs for diagnosing COVID-19 varies widely in the literature, with some estimates as low as 70% (17). This is more likely to be related to error in sample collection and handling than to misdiagnosis by RT-PCR. NP swabbing is a medical procedure that requires a detailed understanding of nasal anatomy, and collecting a blind NP sample can be both uncomfortable for the patient and difficult for the clinician. As with any procedure, there is a significant learning curve to overcome prior to achieving proficiency. Disease-specific sources of uncertainty include the timing of testing during disease course and the anatomic location of sampling. Collection-specific sources of uncertainty include significant time elapsing between sample collection and testing, improper storage temperatures, and inappropriate handling during sample collection, transport, and processing. Many variables affect the overall performance of a test and over-reliance on any one single result can lead to misdiagnosis, particularly when it does not complement the patient's history and clinical presentation. It is the authors' opinion that when clinical diagnosis is in doubt, especially when pretest probability is high but the test returns negative (17), confirmatory testing should likely be repeated as RT-PCR analytical sensitivity is excellent but NP swab clinical sensitivity is more variable. Third, the sensitivity and specificity values reported in this study are derived from different patient populations. The sensitivity is derived from a cohort of known COVID-positive patients by laboratory confirmation. The specificity is derived from a cohort of COVID-negative patients that were anticipated to have a low pretest probability of having COVID-19. Some were sent for testing by their primary care provider for suspicious symptoms, while others were sent by their surgeon or proceduralist without symptoms in order to verify COVID-negative status prior to a planned procedure. This spectrum bias (19) was unavoidable, due in part to the urgent need to enroll patients to understand the effectiveness of the swabs during the pandemic, but also because the local incidence of COVID-positivity was <5% for patients tested during this time What Testing Capacity Do We Need?, on Kaiser Family Foundation Pandemics Explained, on Harvard Global Health Institute Roadmap to Responsibly Reopen America Roadmap to Pandemic Resilience Diagnostic Microbiology and Infectious Disease -Manuscript Submission The Latest Obstacle to Getting Tested? A Shortage of Swabs and Face Masks. The New York Times Open Development and Clinical Validation Of Multiple 3D-Printed Nasopharyngeal Collection Swabs: Rapid Resolution of a Critical COVID-19 Testing Bottleneck Printed Latticed Nasopharyngeal Swab for COVID-19 Testing Made Using Digital Light Synthesis Adverse effects of nasopharyngeal swabs: Three-dimensional printed versus commercial swabs Food and Drug Administration Product Classification Lattice Swab, on U.S. Food and Drug Administration Establishment Universal Viral Transport Kits, on U.S. Food and Drug Administration Establishment Registration and Device Listing Vitro Diagnostics EUAs, on United States Food and Drug Administration 3D Printing in FDA's Rapid Response to COVID-19, on United States Food and Drug Administration emergencysituations-medical-devices/emergency-use-authorizations-medical-devices. Accessed 16. Deeks JJ, Altman DG. 1999. Sensitivity and specificity and their confidence intervals cannot exceed 100% Diagnostic Microbiology and Infectious Disease -Manuscript Submission 17 False Negative Tests for SARS Infection -Challenges and Implications Methods Guide for Medical Test Reviews Statistical Methods in Diagnostic Medicine Virological assessment of hospitalized patients with COVID-2019 Predicting infectious SARS-CoV-2 from diagnostic samples The authors would like to thank the following individuals for ongoing consultation and The authors have no conflicts of interest to disclose.