key: cord-0000781-z5p3a64x
authors: Bhat, Niranjan; O'Brien, Katherine L.; Karron, Ruth A.; Driscoll, Amanda J.; Murdoch, David R.
title: Use and Evaluation of Molecular Diagnostics for Pneumonia Etiology Studies
date: 2012-03-08
journal: Clinical Infectious Diseases
DOI: 10.1093/cid/cir1060
sha: d83f751c542a1f85ba6c4543824c2b261329521e
doc_id: 781
cord_uid: z5p3a64x

Comprehensive microbiological testing will be a core function of the Pneumonia Etiology Research for Child Health (PERCH) project. The development stage of PERCH provided the time and resources necessary for us to conduct a comprehensive review of the current state of respiratory diagnostics. These efforts allowed us to articulate the unique requirements of PERCH, establish that molecular methods would be central to our testing strategy, and focus on a short list of candidate platforms. This process also highlighted critical challenges in the general design and interpretation of diagnostic evaluation studies, particularly in the field of respiratory infections. Although our final molecular diagnostic platform was ultimately selected on the basis of operational and strategic considerations determined by the specific context of PERCH, our review highlighted several conceptual and practical challenges in respiratory diagnostics that have broader relevance for the performance and interpretation of pneumonia research studies.

Comprehensive microbiological testing will be a core function of the Pneumonia Etiology Research for Child Health (PERCH) project. The development stage of PERCH provided the time and resources necessary for us to conduct a comprehensive review of the current state of respiratory diagnostics. These efforts allowed us to articulate the unique requirements of PERCH, establish that molecular methods would be central to our testing strategy, and focus on a short list of candidate platforms. This process also highlighted critical challenges in the general design and interpretation of diagnostic evaluation studies, particularly in the field of respiratory infections. Although our final molecular diagnostic platform was ultimately selected on the basis of operational and strategic considerations determined by the specific context of PERCH, our review highlighted several conceptual and practical challenges in respiratory diagnostics that have broader relevance for the performance and interpretation of pneumonia research studies.

The development of a comprehensive microbiological testing strategy has been a core principle in the conception and design of the Pneumonia Etiology Research for Child Health (PERCH) project [1] . In formulating the most effective approach for respiratory diagnosis, we determined that a multiplex molecular diagnostic platform would be an essential component in our approach. Many of the technical and operational considerations encountered through this process proved relevant to the overall design of the project. We describe here the theoretical and practical challenges encountered in the evaluation and selection of a molecular platform for the diagnosis of pneumonia.

As described elsewhere in this issue [2, 3] , microbiological evidence of infection must be considered in the context of several fundamental difficulties found in respiratory diagnostics, including the frequent lack of access to the site of infection, the insensitivity of available tests, insufficient assay validation, and complexities in determining whether a detected pathogen has a causal role in the illness. The specific research-related demands of PERCH added to these constraints, requiring that our diagnostic strategy must exclude any prior assumptions regarding the likely importance of specific pathogens; must include a full range of respiratory tract specimens, including upper respiratory swab or aspirate, induced sputum, lung aspirate, bronchoalveolar lavage, and pleural fluid; must be comprehensive, yet realistic; must appropriately balance the demands of accuracy and efficiency; must account for both clinical and research ethical issues; and must be feasible for use and support at all participating field sites.

To begin the selection process, the PERCH investigators conducted an extensive review of the microbiologic diagnosis of respiratory infections. Using published and unpublished data, as well as user and developer experiences, our team prepared a strategic summary of the available technologies that could detect pathogens from respiratory tract specimens. We evaluated each major assay category, including traditional bacteriology and viral culture, direct antigen and immunofluorescent antibody detection, and nucleic detection acid tests. It was evident that molecular diagnostics should be among the mix of diagnostic tools required to meet the needs of PERCH.

Nucleic acid detection tests (NADTs) have a number of advantages over other diagnostic platforms for the evaluation of respiratory specimens [4] . They demonstrate superior sensitivity in detecting organisms that are fastidious, less viable, or present in only small amounts [5] . Molecular diagnostics can also be quickly adapted to detect evolving or emerging pathogens and are amenable to efficiencies of scale such as automation. They also allow the simultaneous detection of multiple targets (multiplexing), which in turn allows for testing by clinical syndrome and the detection of co-infections. NADT methods present less of a safety hazard for laboratory personnel compared with culture, typically require less time compared with bacterial culture, and require less technical capacity compared with viral culture. Given these advantages, NADTs have been extensively evaluated in the detection of several viruses and bacteria of the respiratory tract and have become the diagnostic tool of choice for many agents that are difficult to isolate [4] .

Molecular diagnostic platforms are not without their disadvantages. Cost and complexity remain significant barriers to adoption in many laboratories, and NADTs often risk problems of laboratory contamination with amplified products, particularly if the assay procedure requires opening of the reaction tube prior to the target detection step [6] . Measures to limit contamination often require additional laboratory space that may not be available in resource-constrained settings. Nevertheless, NADT methods represent one of the more productive areas of diagnostics research, promising future improvements in automation and speed, smaller devices, improved cost efficiencies, and better detection of emerging pathogens [7] .

The focus on molecular methods for respiratory pathogen detection yielded a large variety of potential technologies for consideration in PERCH (Table 1) . Polymerase chain reaction (PCR) technology is more common at research sites worldwide, can be adapted to various platforms, and easily allows for multiplex amplification. Multiplexing, in which several targets are assayed for simultaneously, is commonly employed in PCR-based assays and offers significant advantages over single-pathogen assays in terms of efficiency and pathogen coverage. Still, developers must overcome considerable complexities in harmonizing the reaction requirements of each individual target and limiting potential competition among the analytes. These factors may result in a measurable decrease in sensitivity compared with single-plex assays. Several techniques have been developed to address such factors, such as alterations in cycling protocols [8] , nested primer combinations [9, 10] , complex primer structures and concentrations [11, 12] , and the use of nontraditional nucleotides [13] [14] [15] . Other NADT technologies, such as nucleic acid sequence-based amplification and loop-mediated isothermal amplification, have been used for the detection of respiratory pathogens, but experience with multiplexing is limited [16] [17] [18] [19] .

Technologies for target detection take on an even larger variety of formats. Older methods include agarose gel electrophoresis, reverse-transcription PCR enzyme hybridization assay [20] [21] [22] and enzyme-linked oligonucleotide capture [18] . More recently, solid-and liquid-phase array platforms have become more useful for the detection of multiple targets. Solid-phase arrays use a variety of formats, typically embedding target-specific oligonucleotides onto a glass or silicon microchip [23] [24] [25] [26] [27] [28] [29] [30] [31] to detect anywhere between dozens to hundreds of thousands of amplified sequences. Several respiratory diagnostic systems have been based on a liquid-phase technology using polystyrene microbeads (Luminex) [9, 10, [13] [14] [15] 32] or mass spectroscopy [33, 34] for amplicon discrimination. Although these approaches have greatly expanded the versatility and sensitivity of multiplex PCR, their complexity, specialized equipment, and high start-up costs have limited their widespread adoption to date. Moreover, these platforms typically require separate steps for amplification and detection, increasing both the workload and the risk of operator error or amplicon contamination.

Real-time PCR assays address these issues by combining amplification and detection in one reaction tube, thus facilitating automation and reducing contamination. In addition, this technique allows for the quantification of pathogens and the assessment of replication efficiency. As with conventional PCR, multiplex real-time assays are subject to competition and inhibition among primers [5, 35] . Real-time assays are also restricted in the number of reaction products that can be detected in parallel [35] , although this problem can be partially circumvented using arrays of uniplex real-time reactions at very small volumes [36] . Successful in-house realtime assays directed against respiratory pathogens have been developed using uniplex [37] and multiplex [38, 39] approaches, but data on the performance of commercialized versions are not readily available.

Much of the effort in NADT development for respiratory diagnostics has focused on the detection of viruses, given the advantages of these techniques over conventional methods in terms of speed, sensitivity, and versatility for detecting this class of pathogens. Multiplex approaches for viral detection have become more common as technologies have improved (reviewed by [5, 7, 40, 41] ). In addition, NADTs have now become the gold standard for the detection of Mycoplasma pneumoniae [42] and Chlamydophila pneumoniae [43] and a useful addition to antigen testing for Legionella species [44] .

Multiplex assays for the detection of more traditional bacterial pathogens have not been studied as frequently in respiratory specimens, primarily because culture techniques are usually adequate for clinical practice. Moreover, molecular methods provide no additional advantage over culture in differentiating infection from colonization of the upper respiratory tract. Nevertheless, multiplex NADTs for bacteria such as S. pneumoniae, Haemophilus influenzae, and Streptococcus pyogenes have been evaluated in respiratory specimens [45] and will likely be incorporated into larger multiplexing assays.

Having conducted our survey of the field, we narrowed our list of candidate molecular diagnostic platforms even further on the basis of the unique needs of our research study. As with clinical laboratories, we closely examined factors such as cost, feasibility, quality assurance, capital investment, platform versatility, and future utility. In contrast with clinical laboratories, we considered issues such as the rapid return of results or regulatory approval for use in patient care to be less crucial to our objectives. Moreover, our approach emphasized comprehensive pathogen detection, rather than focusing primarily on pathogens relevant for clinical management or infection control. Finally, our selected platform would be deployed in low-resource settings, where requirements for a reliable or continuous power supply, adequate access to reagents, sensitivity to extreme environmental conditions, and access to technical support would be highly relevant.

As our appraisals progressed, we encountered many challenges in interpretation that are common to the field of diagnostics evaluation. Most basic among these was confusion regarding the usage of the terms ''sensitivity'' and ''specificity,'' and the evaluations needed to measure these parameters [46] . The distinction between ''analytic'' performance characteristics, as opposed to ''diagnostic'' or ''clinical'' performance characteristics, is essential for properly assessing the validation of any assay, but it is particularly true in the field of molecular diagnostics. For NADTs, analytic sensitivity refers to the lowest concentration of target that can be detected, whereas analytic specificity measures the ability of the test to exclude undesired targets despite similar genetic sequences. In contrast, diagnostic or clinical sensitivity of a nucleic acid detection test refers to the appropriate identification of all patients carrying the agent, and diagnostic or clinical specificity describes the assay's ability to exclude uninfected patients. Clinical performance characteristics are subject to a number of factors, including the patient's disease status, variations in the concentration of the target throughout the course of illness, inhibition by other substances present in the specimen, sample quality, sampling variability, and specimen degradation. Generally, assays should be tested against a reference or gold standard. For tests of microbial detection, the reference standard typically is culture, but molecular diagnostics are often much more sensitive in detecting nucleic acid than is culture for viable organisms, leading to difficulties in interpreting the clinical relevance of false-positive results. The challenges of assessing diagnostic tests have been increasingly recognized in recent years. For instance, the Standards for Reporting of Diagnostic Accuracy initiative [47, 48] offers guidelines on the reporting of diagnostic studies, whereas the Quality Assessment tool for Diagnostic Accuracy Studies provides corresponding guidance on their evaluation [49] .

Nevertheless, respiratory diagnostics are particularly limited by the inability to determine whether the detection of a particular pathogen in a symptomatic patient indicates that it is causative of the illness or results from contamination, colonization, or prolonged shedding from a prior unrelated infection, particularly when testing specimens from the upper respiratory tract. This issue is not typically addressed in diagnostic evaluation studies, but it has become more relevant as molecular diagnostics have expanded the lower limits of pathogen detection by several orders of magnitude. Attempts to answer this question have suggested an additional category of test performance, the ''epidemiological'' specificity of a test, to describe the ability of an assay to assign true etiologic status to a pathogen for a specific illness. Ultimately, determination of the epidemiologic specificity of a respiratory diagnostic would require the interpretation of microbiologic results in conjunction with all other clinical and laboratory data, perhaps in the form of a predictive model. Such analyses are uncommon but will be a main focus of the PERCH study.

Respiratory diagnostics are further complicated by the absence of a perfect gold standard. Culture is difficult or insensitive for some pathogens and unavailable for others (eg, human metapneumovirus, parainfluenzavirus type 4, rhinovirus group C, or Pneumocystis jiroveci). Serologic tests are often not available and usually require paired serum specimens for accurate results. Statistical methods to adjust for such alloyed gold standards, such as discrepant analysis, have been frequently employed, but they can be susceptible to significant bias [50] . Comparative evaluations of respiratory diagnostic assays must also take into account variations in which panel of pathogens is selected, which genetic sequences are targeted, what specimen sources are used [3, 51] , and even what methods are used for nucleic acid extraction [52] . The US Food and Drug Administration has recently published industry guidance that may encourage additional work in this area [53] .

As the PERCH evaluation progressed, the concepts derived from our deliberations were distilled into a list of desirable and essential attributes summarizing our strategy for evaluation ( Table 2 ). This list addressed issues such as assay performance (range of targets, acceptable specimen sources, sensitivity, and specificity), operational concerns (space requirements, assay throughput, quality assurance programs, maintenance requirements, and reagent availability), and strategic issues (capacity for automation, versatility and future utility, start-up and maintenance costs, and developer engagement). For additional input, we presented our summary to the Pneumonia Methods Working Group, an expert committee formed to advise PERCH. Ultimately, this outline of key qualities and data allowed us to articulate our thoughts and communicate our strategy more effectively to collaborators, advisors, and assay developers.

We applied our list of attributes to more than a dozen candidate diagnostic systems that met our initial criteria, and developed a short list of candidate platforms. We then tested these final assays in our PERCH-affiliated laboratories, using a standardized set of mock specimens. This process allowed us to engage with the assay manufacturers and their academic partners, directly compare the performance characteristics of Nucleic acid extraction procedure included in overall process (and automated)

Ability to process a variety of respiratory tract specimens Small specimen volume requirements Specimen collection requirements well-characterized and suitable for field studies Readily available reagents with long expiry dates and room-temperature storage requirements the platforms, and gain essential information that could only be acquired through hands-on experience, such as capabilities for technology transfer, ease of use, and workflow. Details of this evaluation will be the subject of a separate article.

By including a phase for protocol development, the PERCH investigators were able to perform an extensive literature review of respiratory diagnostics, clearly outline the major theoretical and practical concerns, and engage a group of experts for critical input. Through this process, we confirmed the suitability of molecular diagnostics for our needs and identified critical information gaps. Our evaluation highlighted numerous advantages of this technology, including excellent sensitivity and adaptability for a full range of respiratory pathogens and specimen sources, as well as clear capabilities for multiplexing and automation. We nevertheless realized that our conclusions represent but a snapshot in time, and the field of molecular diagnostics is rapidly evolving, with constant improvements in accuracy, speed, automation, and cost. Yet it can also be expected that methods for evaluating respiratory diagnostics will continue to evolve in parallel, providing new answers to the practical and conceptual challenges that shaped the development of a diagnostic testing strategy for PERCH.

PERCH: a 21st century childhood pneumonia etiology study

Laboratory methods for determining pneumonia etiology in children

Specimen collection for the diagnosis of pediatric pneumonia

Emerging advances in rapid diagnostics of respiratory infections

Nucleic acid amplification tests for detection of respiratory viruses

In-house nucleic acid amplification assays in research: how much quality control is needed before one can rely upon the results?

Molecular diagnosis of respiratory viruses

A touchdown nucleic acid amplification protocol as an alternative to culture backup for immunofluorescence in the routine diagnosis of acute viral respiratory tract infections

Evidence from multiplex molecular assays for complex multipathogen interactions in acute respiratory infections

Simultaneous detection and high-throughput identification of a panel of RNA viruses causing respiratory tract infections

Rapid detection and identification of 12 respiratory viruses using a dual priming oligonucleotide system-based multiplex PCR assay

Comparison of the seeplex reverse transcription PCR assay with the R-mix viral culture and immunofluorescence techniques for detection of eight respiratory viruses

High-throughput, sensitive, and accurate multiplex PCR-microsphere flow cytometry system for large-scale comprehensive detection of respiratory viruses

Evaluation of a multiplexed PCR assay for detection of respiratory viral pathogens in a public health laboratory setting

MultiCode-PLx system for multiplexed detection of seventeen respiratory viruses

Loop-mediated isothermal amplification (LAMP): a rapid, accurate, and cost-effective diagnostic method for infectious diseases

Rapid real-time nucleic acid sequence-based amplification-molecular beacon platform to detect fungal and bacterial bloodstream infections

Development of multiplex nucleic acid sequence-based amplification for detection of human respiratory tract viruses

Predicting viruses accurately by a multiplex microfluidic loop-mediated isothermal amplification chip

Rapid diagnosis of human parainfluenza virus type 1 infection by quantitative reverse transcription-PCR-enzyme hybridization assay

Evaluation of the hexaplex assay for detection of respiratory viruses in children

Detection of 11 common viral and bacterial pathogens causing community-acquired pneumonia or sepsis in asymptomatic patients by using a multiplex reverse transcription-PCR assay with manual (enzyme hybridization) or automated (electronic microarray) detection

Use of the DNA flow-thru chip, a three-dimensional biochip, for typing and subtyping of influenza viruses

Oligonucleotide array for simultaneous detection

Molecular Diagnostics for Pneumonia Studies d CID 2012:54 (Suppl 2) d S157 of respiratory viruses using a reverse-line blot hybridization assay

Identifying influenza viruses with resequencing microarrays

Panmicrobial oligonucleotide array for diagnosis of infectious diseases

Pan-viral screening of respiratory tract infections in adults with and without asthma reveals unexpected human coronavirus and human rhinovirus diversity

Detection of respiratory viruses and subtype identification of influenza A viruses by Green-eChipResp oligonucleotide microarray

Using a resequencing microarray as a multiple respiratory pathogen detection assay

Utility of DNA microarrays for detection of viruses in acute respiratory tract infections in children

Comparison of automated microarray detection with real-time PCR assays for detection of respiratory viruses in specimens obtained from children

Development of a respiratory virus panel test for detection of twenty human respiratory viruses by use of multiplex PCR and a fluid microbead-based assay

Diagnostic system for rapid and sensitive differential detection of pathogens

Ibis T5000: a universal biosensor approach for microbiology

Using multiplex real time PCR in order to streamline a routine diagnostic service

Application of TaqMan low-density arrays for simultaneous detection of multiple respiratory pathogens

Comparison of real-time PCR assays with fluorescent-antibody assays for diagnosis of respiratory virus infections in children

Viral etiology of severe pneumonia among Kenyan infants and children

Prospective evaluation of a novel multiplex real-time PCR assay for detection of fifteen respiratory pathogens-duration of symptoms significantly affects detection rate

Detection of respiratory viruses by molecular methods

Utilization of nucleic acid amplification assays for the detection of respiratory viruses

Acute respiratory infection due to Mycoplasma pneumoniae: current status of diagnostic methods

Acute respiratory infection due to Chlamydia pneumoniae: current status of diagnostic methods

Diagnosis of Legionella infection

Development and evaluation of a novel multiplex PCR technology for molecular differential detection of bacterial respiratory disease pathogens

Sensitivity'' and ''specificity'' reconsidered: the meaning of these terms in analytical and diagnostic settings

Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative

The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration

The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews

Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues

Optimal sampling sites and methods for detection of pathogens possibly causing community-acquired lower respiratory tract infections

Comparison of commercial systems for extraction of nucleic acids from DNA/RNA respiratory pathogens

Center for Devices and Radiological Health. Guidance for industry and FDA staff: class II special controls guidance document: respiratory viral panel multiplex nucleic acid assay

Evaluation of commercial ResPlex II v2.0, MultiCode-PLx, and xTAG respiratory viral panels for the diagnosis of respiratory viral infections in adults

Acknowledgments. We thank Bhagvanji Thumar and Trevor Anderson for their technical support and advice.Disclaimer. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or the National Institutes of Health.Financial support.