Original Research

A global proficiency testing programme for Xpert® MTB/RIF using dried tube specimens, 2013–2015

Katherine Klein, Kyle DeGruy, Zilma Rey, Patricia Hall, Andrea Kim, Steve Gutreuter, Heather Alexander

Received: 14 Jan. 2020; Accepted: 12 Aug. 2020; Published: 27 Nov. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: Proficiency testing (PT) is an important quality assurance measure toward ensuring accurate and reliable diagnostic test results from clinical and public health laboratories. Despite the rapid expansion of the Xpert® MTB/RIF assay for the detection of tuberculosis in resource-limited settings (RLS), low-cost PT materials for Xpert MTB/RIF external quality assessment (EQA) are not widely available.

Objective: We sought to determine whether a dried tube specimen (DTS)-based PT programme would be a feasible option to support Xpert MTB/RIF EQA in RLS.

Methods: Between 2013 and 2015, the United States Centers for Disease Control and Prevention developed and conducted a voluntary EQA programme using DTS-based PT material. Eight rounds of PT, each comprising five DTS samples, were provided to enrolled testing sites. After each round, participant results were compared to expected results, scored as satisfactory or unsatisfactory, and sites were provided with performance reports.

Results: Programme enrolment increased from 102 testing sites in seven countries to 441 testing sites in 14 countries over the course of three years. In each PT round, approximately 90% of participating sites demonstrated satisfactory performance. In seven of the 14 enrolled countries, the proportion of sites with a satisfactory score increased between the first round of participation and the most recent round of participation.

Conclusion: This programme demonstrated that it is possible to implement an Xpert MTB/RIF PT programme for RLS using DTS, that substantial demand for Xpert MTB/RIF PT material exists in RLS, and that country performance can improve in a DTS-based PT programme.

Keywords: external quality assessment; Xpert® MTB/RIF; proficiency testing; dried tube specimens; resource-limited settings.

Introduction

The World Health Organization recommends the Xpert^® MTB/RIF (Mycobacterium tuberculosis/rifampicin) assay (Cepheid, Sunnyvale, California, United States) as one of the initial diagnostic tests for all individuals evaluated for tuberculosis.^1,2 The Xpert MTB/RIF assay is an integrated sputum processing and real-time polymerase chain reaction system that rapidly and simultaneously detects M. tuberculosis and the presence of mutations conferring resistance to rifampicin.³ The roll-out of the Xpert MTB/RIF assay to resource-limited settings (RLS) with a high tuberculosis burden has been remarkable. By the end of 2016, 6659 GeneXpert instruments had been purchased at concessional prices for use at multiple levels of the tiered laboratory network.⁴

External quality assessment is a critical component of a laboratory quality management system. Proficiency testing (PT) is one means of conducting external quality assessment for tests in clinical laboratories. Proficiency testing provides objective evidence of accurate testing and may identify areas in the diagnostic process, including pre-analytic, analytic, and post-analytic phases, where quality improvement is needed.^5,6 The rapid scale-up of Xpert MTB/RIF testing in RLS has not coincided with the same investment in quality assurance activities for the assay. Few PT providers for the Xpert MTB/RIF assay offer materials to sites in RLS, leaving a substantial gap in external quality assessment implementation that could otherwise help to confirm site competency in assay administration and accuracy of test results for limited-resource countries.^7,8,9

The United States Centers for Disease Control and Prevention (CDC) developed a PT panel for Xpert MTB/RIF, using a dried tube specimen (DTS)-based method.¹⁰ The DTS technique has successfully been used to develop PT panels for HIV, malaria and syphilis diagnostic assays in RLS.^11,12,13 Our goal in this evaluation was to assess the performance and utility of CDC-prepared DTS panels in an Xpert MTB/RIF PT programme.

Methods

Ethical considerations

No patient specimen collection was required and no patient identifiers were recorded on quality assurance tools. Participation in the Xpert MTB/RIF quality assurance and PT programme was voluntary and free of charge. No incentives were provided. This study was approved by the CDC Center for Global Health Division of Human Research Protection (CGH HSR tracking number 2014-082).

Dried tube specimen production

The method for producing DTS was described previously by DeGruy et al.¹⁰ Briefly, DTS were produced by chemically inactivating liquid cultures of well-characterised mycobacterial strains using equipment and supplies commonly found in laboratories conducting tuberculosis liquid culture.¹⁴ Mycobacteria were grown in BACTEC® MGIT 7 mL tubes (Becton, Dickinson and Company, Sparks, Maryland, United States) and inactivated using a 2:1 ratio of Xpert MTB/RIF sample reagent (Cepheid, Sunnyvale, California, United States) to MGIT culture incubated for 2 hours with intermittent vortexing. Inactivated cultures were washed, concentrated, resuspended, homogenised and diluted, before being dispensed into cryovials and air-dried in a Class II biological safety cabinet within a Biosafety Level III tuberculosis containment laboratory. Inactivation verification was performed by inoculating 0.5 mL of undiluted stock solution into MGIT tubes supplemented with polymyxin B, amphotericin B, nalidixic acid, trimethoprim, and azlocillin (PANTA) (Becton, Dickinson and Company, Sparks, Maryland, United States) and incubating for a total of 84 days over two incubation cycles in the BACTEC^® MGIT 960^® instrument (Becton, Dickinson and Company, Sparks, Maryland, United States) to confirm a lack of growth. Dried tube specimens were validated prior to distribution by testing 10% of DTS produced with the Xpert MTB/RIF assay (Cepheid, Sunnyvale, California, United States), as described previously.¹⁰ To test DTS, the DTS were rehydrated with 2.5 mL of sample reagent, shaken vigorously 20 times and incubated at room temperature for 15 minutes with additional shaking repeated after 10 min. The Xpert MTB/RIF cartridges were inoculated with approximately 2.0 mL of rehydrated samples using the transfer pipette provided with the kit, and tested immediately on a GeneXpert IV or GeneXpert VIII using GeneXpert DX software version 4.0 (Cepheid, Sunnyvale, California, United States) according to manufacturer instructions.¹⁵

Proficiency testing programme

In 2013, select countries in sub-Saharan Africa, Southeast Asia, and the Caribbean, supported by the United States President’s Emergency Plan for AIDS Relief were invited to participate in CDC-Atlanta’s pilot DTS Xpert MTB/RIF PT programme at no cost. Seven countries elected to participate in the pilot, designated a PT country coordinator, and enrolled testing sites that routinely conducted Xpert MTB/RIF testing on patient specimens. These countries continued to enrol additional sites throughout 2013, and four additional countries enrolled in subsequent 2013 PT rounds. All countries remained enrolled throughout 2014. In 2015, enrolment was extended to additional sites in all participating countries plus three new countries in Africa.

The PT kits included a panel of five DTS samples, five transfer pipettes, testing instructions, and a paper form for recording results. Four PT rounds were distributed quarterly in 2013 (2013-A, 2013-B, 2013-C and 2013-D). Only one PT round was distributed at the end of 2014 (2014-A) to allow for evaluation of 2013 data. Three PT rounds were distributed in 2015 (2015-A, 2015-B and 2015-C). Country coordinators were responsible for distributing the panels to enrolled testing sites, collating results (including transferring data from paper forms to a template Microsoft Excel file (Microsoft Corp, Redmond, Washington, United States) and submitting data to CDC-Atlanta via email within nine weeks of panel receipt. A sufficient number of PT kits were sent to enrolled countries to allow for distribution of one PT kit to each enrolled site; enrolled sites that did not submit results were categorised as enrolled but not participating in a given PT round.

Participant data were analysed by CDC-Atlanta using Microsoft Excel. A report was generated for each participating site; this compared the site’s reported results to expected results (CDC-Atlanta validation results) and included the consensus results from all the participating testing sites in all the ountries. Consensus results detailed the number of testing sites detecting and not detecting M. tuberculosis complex, detecting and not detecting rifampicin resistance, reporting indeterminate rifampicin resistance, reporting uninterpretable tests (error, invalid, or no result), and not reporting M. tuberculosis complex and/or rifampicin results for each sample in the panel. To establish a consensus result for each PT sample in the round, we required at least 80% of participant results for that sample to match the expected result. In the 2013 pilot phase, participating sites were provided with qualitative reports indicating overall concordance of submitted results with the expected results. Beginning in 2014, participating sites were assigned a quantitative score for each PT round. Scores were calculated by assigning a value of 20 points to each of the five DTS, for a total of 100 possible points. If the site’s qualitative M. tuberculosis complex and rifampicin resistance detection results matched the expected results for a given sample, 20 points were awarded. If either the M. tuberculosis complex detection result or the rifampicin resistance detection result did not match the expected results, zero points were awarded. If the M. tuberculosis complex detection result matched the expected result but the site’s rifampicin resistance detection result was indeterminate, 10 points were awarded. If a site reported an uninterpretable test, five points were awarded. If a site did not report a result, zero points were awarded. A total score of greater than, or equal to, 80 points was considered satisfactory for the PT round. For this evaluation, 2013 results were retrospectively scored to compare performance across all years of the programme.

To further investigate concordance and discordance of site results with expected results, additional analyses were conducted in SAS version 9.3 (SAS Institute, Cary, North Carolina, United States). Results were categorised as concordant if both the qualitative M. tuberculosis complex and rifampicin results matched the expected results. If discordant, results were categorised according to the type of discordance (false-positive M. tuberculosis complex detection, unsuccessful run, etc.). Tests were considered unsuccessful when the test was started, but a definitive result was not obtained due to an error, invalid result, power failure during testing, or similar problem. Tests were also considered unsuccessful if the site chose not to test the sample, was unable to test the sample due to unavailability of Xpert MTB/RIF kits, or failed to report either the M. tuberculosis complex or rifampicin detection result. Frequencies for results in each category were calculated by PT sample and in aggregate for all samples. The mycobacterial strain used to produce the PT sample and the mean cycle threshold (Ct) of probe A during validation of the PT sample were included in the analysis to look for trends between these variables and the amount and type of discordance observed. In line with previous studies, probe A was selected for most analyses as it was the first probe to reach the detection threshold.⁷

We conducted additional statistical analyses to test for an association between the mean Ct of probe A during validation and several types of discordance including false-negative M. tuberculosis complex detection, false-positive rifampicin resistance, and indeterminate rifampicin resistance. Logistic regression of the type of discordance on mean Ct of probe A was conducted using R glm (R Foundation for Statistical Computing, Vienna, Austria).¹⁶

Results

Enrolment in the DTS-based Xpert MTB/RIF PT programme increased steadily from round 2013-A to round 2015-C, with 102 sites in seven countries enrolled in round 2013-A and 441 sites in 14 countries enrolled in round 2015-C (Figure 1). Participation by enrolled sites increased from 68 sites in six countries in round 2013-A to 342 sites in 13 countries in round 2015-C. The participation among sites ranged from a minimum of 66.7% in round 2013-A to a maximum of 83.8% in round 2015-B.

FIGURE 1: Number of sites enrolled, participating, and achieving a passing score in dried tube specimen-based Xpert® MTB/RIF proficiency testing programme, 2013–2015.

The proportion of participating sites with a satisfactory score ranged from a minimum of 88.1% in round 2015-B to a maximum of 93.1% in round 2014-A (Table 1). Individual countries had as few as 50% and as many as 100% of their participating sites achieve satisfactory scores. In seven of the 14 enrolled countries, the proportion of sites with a satisfactory score increased between the first round of participation and the most recent round of participation. Of the remaining seven countries, the proportion of sites with a satisfactory score remained constant in three countries but decreased in four countries (Table 1).

All 40 PT samples met the criteria for establishing a consensus result except 2015-B-1. 2015-B-1 was below 80% concordance with expected results, primarily due to a high rate of false-negative M. tuberculosis complex detection results (19.9%) (Table 2). This sample is further discussed in the Limitations section. For 29 of the 39 PT samples with a consensus result, over 90% of DTS results received were concordant with the expected result. Five of the 10 PT samples with < 90% concordance had false-negative M. tuberculosis complex detection as the most frequently observed cause for discordance, four of the 10 samples had unsuccessful tests (errors, invalids, and other causes) as the most frequently observed cause for discordance, while one sample had an equal number of unsuccessful tests and indeterminate rifampicin results. Error rates ranged from 0% to 5.9% across PT samples.

There were 765 (9.4%) discordant results out of 8150 DTS results received throughout the eight PT rounds. Of those, 350 (46%) discordant results were unsuccessful tests (Figure 2). False-negative M. tuberculosis complex results were the second-largest contributor to discordant results 224 (29%), followed by indeterminate rifampicin results 79 (10%). The remaining discordant results consisted of: false-negative rifampicin results 45 (6%), false-positive rifampicin results 34 (4%), and false-positive M. tuberculosis complex results 33 (5%).

FIGURE 2: Distribution of 765 total discordant results in Xpert® MTB/RIF proficiency testing programme among participating countries, 2013–2015.

Logistic regression of the proportion of false-negative M. tuberculosis complex detection results on the mean Ct of probe A during validation yielded an odds ratio of 1.42 (95% confidence interval [CI] 1.32–1.52). However, sample 2015-B-1’s unusually high mean probe A Ct, combined with its unusually high proportion of false-negative M. tuberculosis complex detection results, may have unduly influenced this estimate. When this sample was removed from the calculation, the odds ratio decreased to 1.14 (95% CI 1.04–1.26). Logistic regression of the proportion of indeterminate rifampicin results and the proportion of false-positive rifampicin resistance results on the mean Ct of probe A during validation yielded odds ratios of 1.14 (95% CI 1.02–1.25) for indeterminates and 0.96 (95% CI 0.79–1.16) for false positives.

Discussion

We have shown that it is feasible to implement an Xpert MTB/RIF PT programme for RLS using DTS. Over the eight rounds of PT panels provided across a period of three years, the number of countries participating in the programme doubled and the number of participating sites increased five-fold. In each PT round, at least two-thirds of enrolled sites participated in PT panel testing and reported test results. Lower participation rates were seen in rounds during which certain countries or their enrolled sites were subsequently unable to participate due to logistical challenges, such as a lack of funding for in-country panel distribution, lack of staff, PT kits being lost in the mail between the country coordinator and the enrolled testing site, non-functioning Xpert instruments or computers, and stockouts (inability to procure test cartridges). Further work is needed to investigate and address the reasons for non-participation in proficiency testing as part of a comprehensive continuous quality improvement package.

Most participating sites performed well in the DTS-based Xpert MTB/RIF PT programme. Approximately 90% of participating sites in each PT round demonstrated satisfactory performance, and over 90% of returned results during the study period were concordant with expected results. The most common reason for a result that was not concordant with the expected result was an unsuccessful test. Some of the reported causes of unsuccessful tests, such as power failures and stockouts of testing kits, are recurring challenges in RLS.¹⁷

The use of a similar sample type, dried culture spots, for Xpert MTB/RIF external quality assessment in South Africa has been described.^7,8 We observed similar error rates to those reported by Scott et al. and Gous et al.; however, we observed a greater proportion of discordant results.^7,8 The average Ct of probe A obtained from dried culture spots was lower than that of many DTS PT samples. Lower Cts correlate with greater amounts of M. tuberculosis complex DNA present in the sample, indicating that a higher concentration of M. tuberculosis complex DNA was present in the dried culture spots. This may account for the smaller proportion of false-negative M. tuberculosis complex detection results from dried culture spots. We observed that false-negative M. tuberculosis complex detection results were often more common among DTS samples with higher probe A Cts during validation. Our analyses using logistic regression confirmed a statistically significant positive association between higher probe A Cts and false-negative M. tuberculosis complex detection. For DTS samples produced at CDC during the study period, an increase of 1 cycle in the mean probe A Ct resulted in an estimated 42% increase in the proportion of false-negative M. tuberculosis complex detection results (the estimate decreased to 14% when sample 2015-B-1 was excluded). Similarly, we found a statistically significant positive association between higher probe A Cts and indeterminate rifampicin resistance results, with an estimated 14% increase in the proportion of indeterminate rifampicin resistance results for every 1 cycle increase in a sample’s mean probe A Ct. No association was found between mean probe A Ct and false-positive rifampicin resistance, and we did not observe any trends between the strain used to produce DTS and the proportion and types of discordance during the study period. Based on these results, we are investigating modifications to the DTS preparation method that will increase the concentration of M. tuberculosis complex DNA present in DTS. However, it is not possible to rule out the presence of or determine the frequency of other site-specific factors, such as adherence to standard operating procedures, transcription errors, instrument maintenance and calibration, and DTS and test cartridge storage and shipping conditions that may influence the accuracy and reliability of Xpert MTB/RIF test results.

The large increase in enrolment we observed over the course of three years may have been influenced by several factors. As the World Health Organization endorsed Xpert MTB/RIF in 2010 and funds became available for the purchase of instruments, the number of laboratories in RLS utilising Xpert MTB/RIF increased dramatically.⁴ The World Health Organization’s expanded recommendations for use of Xpert MTB/RIF in 2013 also encouraged rapid adoption of the technology.¹ Over the same time period, the number of laboratories in RLS electing to pursue external accreditation also increased, facilitated by such programmes as Strengthening Laboratory Management Toward Accreditation, and Stepwise Laboratory Improvement Program Towards Accreditation.^18,19 Thus, an increasing number of laboratories in RLS are in need of Xpert MTB/RIF PT material to demonstrate proficiency in line with recommendations and requirements for continuous quality improvement and accreditation programmes.

Limitations

Several challenges arose during operation of the DTS-based Xpert MTB/RIF PT programme. The participant consensus for sample 2015-B-1 was below 80% concordance with expected results, mainly due to false-negative M. tuberculosis complex results (failure to detect M. tuberculosis complex when the expected result was ‘MTB detected’). Although this sample was produced using the same method as other DTS samples, the probe A Cts during validation were on average higher than those of other M. tuberculosis complex samples produced. This indicates 2015-B-1 contained fewer inactivated organisms and thus less DNA than other samples, potentially contributing to the higher proportion of false-negative M. tuberculosis complex results received.²⁰ To investigate the low concordance of 2015-B-1, three remaining aliquots were tested and confirmed the lower than average semi-quantitative result. Due to the low participant consensus, the decision was made to award all sites that submitted results for 2015-B-1 full credit (20 points). Based on this experience, we have implemented additional DTS validation criteria to ensure DTS contain enough inactivated organism to perform reliably. Beginning in 2016, only DTS samples with an average Ct value of less than 23 for the first probe detected are included in the distributed PT panels. The mean Ct value of 23 was selected based on the findings of previous investigators, who observed good concordance with mean Ct values up to 23.⁷ In addition, recording and reporting of results was also a recurring challenge. At times, no explanation was provided for missing or incomplete results, and it was not possible to determine whether a site attempted to test the sample or if the test was unsuccessful and the result not reported. More clearly defined reporting language and procedures were added to report forms in future PT rounds to reduce the reliance of free text data entry and improve our ability to provide assistance on determining the root causes of discordant results and unsuccessful tests.

Recommendations and next steps

The steady increase in enrolment and number of sites participating in this DTS-based Xpert MTB/RIF PT programme demonstrates that there is a substantial demand for routine PT material for the Xpert MTB/RIF assay. The ease of large-scale batch preparation of DTS using readily available supplies and equipment, and the successful establishment of country coordinator roles for in-country management of PT programme operations, indicate that a DTS-based Xpert MTB/RIF PT programme could be a feasible option for countries wanting to manage their own Xpert MTB/RIF PT programme. Future work will focus on piloting the transfer of the DTS-based PT package to national and regional programmes as part of a comprehensive continuous quality improvement package, as well as developing a web-based PT data entry and reporting system.

Conclusion

A continuous quality improvement package for Xpert MTB/RIF in RLS that includes routine PT material is an important contributor to ensuring provision of accurate results. A DTS-based PT programme for Xpert MTB/RIF is a useful tool for monitoring and improving Xpert MTB/RIF performance. The simplicity of producing DTS and use of common mycobacteriology laboratory supplies make DTS a feasible way for countries to provide Xpert MTB/RIF PT material to their testing sites.

Acknowledgements

We thank Bharat Parekh for his inspiration and guidance, and all testing sites for their willingness to participate in the proficiency testing program.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

K.D. and H.A. conceived and designed the experiments, which were performed by K.D. and K.K. Also, K.K., K.D. and S.G. analysed the data. K.K., K.D., Z.R., P.H. and A.K. wrote the article and all authors reviewed and approved the manuscript.

This research has been supported by the President’s Emergency Plan for AIDS Relief through the United States Centers for Disease Control and Prevention, Atlanta Georgia.

Data access level is public without identifying information of proficiency testers, testing sites and country.

The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the United States Centers for Disease Control and Prevention. References in this manuscript to any specific commercial products, process, service, manufacturer, or company do not constitute its endorsement or recommendation by the United States government.

References