key: cord-267860-mc0xa5om
authors: Lam, Simon C.; Lui, Andrew K.F.; Lee, Linda Y.K.; Lee, Joseph K.L.; Wong, K.F.; Lee, Cathy N.Y.
title: Evaluation of the user seal check on gross leakage detection of 3 different designs of N95 filtering facepiece respirators
date: 2016-05-01
journal: Am J Infect Control
DOI: 10.1016/j.ajic.2015.12.013
sha: 
doc_id: 267860
cord_uid: mc0xa5om

BACKGROUND: The use of N95 respirators prevents spread of respiratory infectious agents, but leakage hampers its protection. Manufacturers recommend a user seal check to identify on-site gross leakage. However, no empirical evidence is provided. Therefore, this study aims to examine validity of a user seal check on gross leakage detection in commonly used types of N95 respirators. METHODS: A convenience sample of 638 nursing students was recruited. On the wearing of 3 different designs of N95 respirators, namely 3M-1860s, 3M-1862, and Kimberly-Clark 46827, the standardized user seal check procedure was carried out to identify gross leakage. Repeated testing of leakage was followed by the use of a quantitative fit testing (QNFT) device in performing normal breathing and deep breathing exercises. Sensitivity, specificity, predictive values, and likelihood ratios were calculated accordingly. RESULTS: As indicated by QNFT, prevalence of actual gross leakage was 31.0%-39.2% with the 3M respirators and 65.4%-65.8% with the Kimberly-Clark respirator. Sensitivity and specificity of the user seal check for identifying actual gross leakage were approximately 27.7% and 75.5% for 3M-1860s, 22.1% and 80.5% for 3M-1862, and 26.9% and 80.2% for Kimberly-Clark 46827, respectively. Likelihood ratios were close to 1 (range, 0.89-1.51) for all types of respirators. CONCLUSIONS: The results did not support user seal checks in detecting any actual gross leakage in the donning of N95 respirators. However, such a check might alert health care workers that donning a tight-fitting respirator should be performed carefully.

The unremitting worldwide outbreaks of different infectious respiratory diseases, such as severe acute respiratory syndrome, multidrug-resistant tuberculosis, avian influenza A (H5N1, H5N2, H7N2, and H7N3), and human swine influenza (H1N1), 1-6 have caused increased awareness of occupational protection among health care workers. Therefore, use of N95 filtering facepiece respirators (also known as N95 respirators) to prevent spread of droplets transmitted and potential airborne infectious diseases is recommended internationally through announcements by the World Health Organization (WHO) and U.S. Centers for Disease Control and Prevention (CDC). 7, 8 Regardless of the shapes or brands of such respirators, they are generally a tight-fitting half facepiece type, and their reliability is simply dependent on fit to the wearer. 9 According to a laboratory performance evaluation conducted by the CDC, the average penetration by ambient aerosol was found to be 33% in ill-fitting respirators compared with 4% in well-fitting respirators. 9 It is believed that the gap existing between the respirator and the wearer's face contributes to such penetration, which is often regarded as leakage. To achieve creditable occupational protection, most well-known authorities, such as the National Institute for Occupational Safety and Health, CDC, and WHO, made fit testing compulsory for wearers prior to use of an N95 respirator. 7, 8 In Hong Kong, fit testing should be a mandatory measure for frontline health care staff working in public and private hospitals.

Quantitative fit testing (QNFT) is a recognized method to determine whether a tight-fitting respirator fits a wearer. This method adopts an electronic device to measure the ratio of particular air particles inside and outside the breathing zone (when donned with a respirator), and the ratio reflects the degree of leakage. 10 To make it simple, QNFT is "an assessment of the adequacy of respirator fit by numerically measuring the amount of leakage into the respirator." 10 To assess any possible leakage, most of the preset fit testing systems require the wearer with a donned N95 respirator to perform a series of exercises, including a static portion without body movement (ie, normal and deep breathing) and a dynamic portion with both normal breathing and designated movements (ie, side-to-side head movement, up and down head movement, talking or reading a standard set of passages, grimacing, bending over). These exercises simulate the common working activities in the clinical environment; hence, the results of QNFT can conservatively reflect any possible leakage. The characteristics of objective measurement and an automatic process increase the significance of QNFT, which now serves as the gold standard in worldwide guidelines and research literature. [10] [11] [12] [13] [14] [15] [16] [17] Although QNFT warrants reliability of N95 respirator usage, any significant change in facial morphology, body weight, or donning method may contribute to on-site leakage. 10, 18, 19 Therefore, even if a given respirator is considered fit by the recognized fit testing, a user seal check is still suggested in order to check the appropriateness of every donning. 10, [20] [21] [22] A user seal check is a self-examination procedure for wearers of N95 respirators to identify on-site gross leakage through repeated visual checks on obvious gaps and positive and negative pressure checks on the seal. 10, 15, [20] [21] [22] N95 respirator manufacturers and some authorities recommend that this practice should be routinely carried out by frontline health care workers. 10, 20, 21 Previous experimental studies on U.S. subjects suggested that the user seal check improved the donning of N95 respirators. 23, 24 Although the scale of these studies was not large enough (N = 11 and N = 64), 23,24 the rigor of the experimental design and the use of repeated measurements increased the credibility of the results. Some guidelines suggest that no further fit testing is needed for a given respirator if subjective leakage is detected by a user seal check. This check may substitute for fit testing if fit testing is not available because of logistic difficulties or failure of the fit testing system. Several recent studies, nevertheless, have consistently rejected this suggestion of substitution. 12, 14, 15, 25 In Hong Kong, a retrospective study demonstrated that the user seal check failed in determining the fit of N95 respirators because its false-positive (19%-31%) and false-negative (24%-40%) rates were too high among 84 Chinese nursing staff. 12 Lam et al further supported the previously mentioned claim through 2 prospective studies on Chinese nursing students (N = 204 and N = 349, respectively) by presenting the sensitivity (15%-23%), specificity (89%-90%), positive (46%-63%) and negative (60%-67%) predictive values, and Kappa values (-0.031 to -0.047, P > .05) of the user seal check. 14, 15 In a Canadian study, similar results and conclusions were also reported on research involving 784 health care workers (false positive rate: 25%-30%). 25 The congruent results indicate that the user seal check cannot replace the fit testing. It is believed that the user seal check, which does not involve any dynamic body movement, is unlikely to mirror the fit testing results because the latter assessment is performed when the wearer performs 8 sequential exercises involving a series of head and body movement.

However, the user seal check may still be able to identify onsite gross leakage and give some information on the gross leakage on normal breathing or deep breathing without head and body movement. Given its immense implication on occupational protection, its validity has not yet been rigorously studied. Therefore, the research question was as follows: Can the result of the user seal check reflect the actual gross leakage under the conditions of normal and deep breathing? This study, hence, aimed to examine the sensitivity, specificity, predictive values, and likelihood ratios of the user seal check on actual gross leakage detection during normal breath-ing or deep breathing without head and body movement in 3 common respirator models of different designs.

This study used a descriptive, prospective, and cross-sectional research design. From September 2010-December 2011, a convenience sample of 647 Chinese students who studied in different nursing programs (ie, year 1 of bachelor's or higher diploma program) in a local university was invited to participate by internal e-mails and several announcements.

Data collection consisted of 3 phases (ie, registration, training session, fit testing session) (Fig 1) . The demographic data of participants (sex, body height, and weight), the results of the user seal check, and the results of actual gross leakage detection through the fit testing device were recorded in a data sheet. During the registration, all participants were required to sign the consent form and prepare themselves in the same manner and appearance of clinical practicum (eg, pinning up long hair, shaving). In the training session, a 30-minute training, including video, demonstration, and practice on standardized N95 respirator donning technique and the user seal check method, was introduced by trained registered nurses. Through redemonstration, the donning techniques and user seal check method of each participant were assessed by these nurses prior to moving on to the next session. Apart from the time used in registration and the training session (various times among the participants), it took another 15 minutes to complete the remaining process of data collection, namely the user seal check and QNFT on the exercises of normal and deep breathing for the given 3 types of N95 respirators, where the sequence of testing remained unchanged for all participants. To control the environmental factors, such as the concentration of suspended particles and dusts, which may affect the result of fit testing, all of the data were collected in an assigned air-conditioned room with an area of 10 m 2 , temperature at approximately 23°C, and humidity at approximately 75%. 15 

To perform a user seal check, the wearer subjectively assessed and adjusted the position and tightness of a given N95 respirator through a visual check and positive and negative pressure checks. Details on the steps and methods for the user seal check can be found in previous studies. 10, 15, 22 A positive result is indicative of subjective gross leakage. 14,15

The PortaCount Pro+ Respirator Fit Tester 8038 (TSI, St Paul, MN) was adopted to measure the actual gross leakage. The details, including technologic information and protocol setting of this system, were introduced elsewhere. 10,15,18 Currently, this system is widely adopted in public and private hospitals in Hong Kong and is used as a local quality control standard by respirator manufacturers. Figure 2 shows the fit tester system, tubing connection, and respirator. All of the participants were only required to perform the static portion out of the 8 specified exercises (ie, normal breathing, deep breathing). In this portion, the participants should remain still in a normal standing position and breathe as usual for 60 seconds before taking long deep breaths as if working hard for another 60 seconds. The research nurses monitored and assessed chest movement by visual inspection to estimate adequacy of the depth (Fig 2) . These 2 exercises gave particular individual fit factors (FFs; range, 0-200). Each FF is the ratio of a challenge agent (ambient particles) concentration outside the respirator to the concentration of a challenge agent that leaks into the inside of the respirator. A FF <100 under the normal breathing and deep breathing exercises is defined as actual gross leakage. 10, 16 The higher the FF, the lesser amount of leakage. The PortaCount Pro+ Respirator Fit Tester 8038 went through a daily check procedure to warrant the sufficiency of ambient particles and performance of the system. 15 

The cup-shaped 3M-1860s (3M, Minneapolis, MN) (3M-A), 3-panel designed 3M-1862 (3M) (3M-B), and pouch-type Kimberly-Clark 46827 (Kimberly-Clark, Neenah, WI) (KC-C) N95 respirators were selected. The selection was based on 3 reasons. First, these 3 models are typically and widely used in local clinical settings. Second, previous studies demonstrated that the prevalence of the fittesting failure rate was approximately 40% for the 3M models. 11, 12, [14] [15] [16] It is estimated that the prevalence of actual gross leakage would be lower than that. According to our previous experience on QNFT, the obtained FFs of normal and deep breathing were generally higher than that of the other exercises. Extreme prevalence rates, such as <20% or >80%, greatly deteriorated the accuracy of both positive and negative predictive values. 26, 27 The prevalence rate of actual gross leakage among the 3 different designs of respirators should be within the optimal range for calculation of the predictive values. Finally, it is unrealistic and unnecessary to include all types of N95 respirators for fit testing. In general, most of them were designed under these 3 categories. This study used a representative respirator from each category; hence, the results could provide a better evaluation on the validity of the user seal check.

Ethical approval was sought from the President's Advisory Committee on Research and Development, The Open University of Hong Kong. An invitation letter was prepared. Information about the purposes of the study, right to confidentiality, right to withdrawal, and duration of fit testing and a consent statement were provided. Participants' written consent was obtained prior to data collection.

Descriptive statistics were used to present the participants' demographic variables and the results of the user seal check (ie, positive, negative) and actual gross leakage (ie, pass, fail). Independent sample t tests were undertaken to test for the difference between participants in the 2 groups (positive and negative user seal checks) with regard to their results of FF. The significance level was set at P < .05. The results of the user seal check compared with the gold standard QNFT on actual gross leakage through cross tabulation were used to compute the following diagnostic parameters: sensitivity, specificity, positive and negative predictive values, and likelihood ratios (refer to the "NOTE" in Table 4 for the respective formula).

The sensitivity (ability of the user seal check to correctly identify a case with gross leakage) and specificity (ability of the user seal check to correctly identify a case without gross leakage) were calculated from the measurements. According to the evaluation of the performance characteristics of diagnostic tests in the medical literature, a combination of high sensitivity and specificity (>80%) [28] [29] [30] is equally important and is an indication of the characteristics of the user seal check itself (ie, test's ability). 31 Because the user seal check is applied in clinical practice, additional performance evaluations, positive and negative predictive values, 32 are necessary to help interpret the results. A value ≥80% is considered to be satisfactory for both predictive values. 27 Another method for describing the screening accuracy of the user seal check is the likelihood ratios. The ratios have an advantage over the aforementioned sensitivity, specificity, and predictive values because they are independent of the prevalence of actual gross leakage and hence can be applied across settings and populations. 31 According to the recommendation of using probabilistic reasoning, 31, 32 the user seal check is moderately good at ruling in leakage if the positive likelihood ratio is >2. Conversely, such a check is moderately good at ruling out leakage when the negative likelihood ratio is <0.5. Values close to 1.0 represent that the user seal check is useless in predicting the presence or absence of actual gross leakage.

A total of 638 nursing students participated in the study. For those who did not participate or were excluded, the reasons included 6 who were physically unfit (eg, asthmatic attack, influenza), 2 who were absent (eg, withdrew from the program), and 1 who had unshaven bushy facial hair. The participants ranged from 18-30 years of age, and 25.5% of them were men (n = 163). Their mean height was 163.1 ± 7.69 cm, and their weight was 56.2 ± 11.14 kg. As far as the FFs between a group of positive and negative user seal checks were concerned, generally the participants with negative user seal checks obtained an observable higher score in the 3 types of respirators compared with those with a positive check. However, only significant differences were found regarding the use of the KC-C respirator (t = 2.01-2.75, P = .006-.045) ( Table 1) .

The results of the user seal check compared with that of actual gross leakage performed by QNFT are presented in Tables 2 and 3 . Among the participants, 25.7% (n = 164), 20.4% (n = 130), and 24.5% (n = 156) reported positive user seal checks regarding the 3M-A, 3M-B, and KC-C respirators, respectively.

However, the prevalence of actual gross leakage identified by QNFT in normal breathing was 34.3%-39.2% in both of the 3M res- pirator models and 65.8% in the KC-C model. In deep breathing, the prevalence was similar, 31.0%-35.0% in both of the 3M respirator models and 65.4% in the KC-C model. Testing on the 3 different respirators in the 2 breathing conditions, the sensitivity and specificity of the user seal check for identifying a case with actual gross leakage ranged from 21.5%-28.0% and 75.2%-81.7%, respectively (Table 4 ).

Extreme prevalence rates caused deviation of positive predictive values and negative predictive values. 27 According to the current results on prevalence rates of actual gross leakage (ie, between 31.0% and 65.8%), further evaluation on the characteristics of the test's performance of positive and negative predictive values was regarded as appropriate.

Regarding the test of the 3M respirators, the positive predictive values of a positive user seal check for estimating the probability of actual gross leakage ranged from 34.6%-42.7%, whereas the negative predictive values ranged from 62.0%-69.9%. In contrast, the test of the KC-C respirator showed different patterns, which were of relatively high positive predictive values (69.9%-74.4%) and low negative predictive values (36.1%-36.9%).

Finally, both the positive and negative likelihood ratios indicating the post-test probability of the user seal check were close to 1.0 (positive likelihood ratio range, 1.08-1.51; negative likelihood ratio range, 0.89-0.98). Table 4 presents the detailed results.

Concerning the 3M respirators, the observed differences of the FFs between a group of participants with positive and negative user seal checks were minimal and these differences were not statistically significant at all (123.8-134.5 vs 127.2-141.9, respectively). Although a significant difference was found for use of the KC-C respirator, the mean score of the FF of a group of negative user seal checks (no subjective gross leakage) was still <100 (a detection of actual gross leakage), which implies that KC-C respirator is difficult to fit Chinese participants. Concerning donning with 3M respirators, the prevalence of actual gross leakage in this study (31%-39%) was slightly lower than that of the failure rate of fit testing in previous studies (35%-43%). 12, 14, 15 This was not surprising because the fit testing examines the degree of leakage during a series of exercises, whereas the actual gross leakage is computed only based on the measured FF on the static portion. However, the actual gross leakage that was found in the KC-C respirators was still frequent (up to 65%). It may imply that a higher failure rate on fit testing of this model was expected among the Chinese population. This warrants future empirical testing.

The positive user seal checks ranged from 20%-25% in the current study, which is comparable with that of previous studies (10%-29%). 12, 14, 15 In some occasions, participants who felt the gross leakage of a given respirator (assessed by the user seal check) passed the fit testing in normal and deep breathing (ie, false-positive rate: 19.3%-24.8%). In contrast, more frequently, participants subjectively expressed the good fit of a given respirator, but the actual gross leakage was still detected by QNFT in normal or deep breathing mode (ie, falsenegative rate: 72.0%-78.5%). Similar observations were consistently reported in the literature, 12, 15 which reinforced that the leakage between the face and respirator is unlikely identified by human sense.

The literature indicated the sensitivity and specificity of the user seal check in determining the fit of N95 respirators were 15%-23% and 89%-90%, respectively. 14, 15 Such results suggested that the user seal check cannot replace the fit testing because the fit testing simulated a series of head and body movement on leakage detection. The current study hypothesizes that the user seal check may contribute to the detection of gross leakage in normal and deep breathing, which is important information during on-site donning. 25, 26 However, based on the unacceptable sensitivity (21.5%-28.0%) and specificity (75.2%-81.7%) in the current results, the hypothesis that the user seal check is able to detect actual gross leakage in normal and deep breathing is also rejected. Interestingly, the sensitivity and specificity of the user seal check in determining the fit of N95 respirators and in detecting gross leakage are fairly comparable. Such a phenomenon may imply that leakage in normal and deep breathing shall predict the result of fit testing. However, further empirical testing is warranted to work out this possibility.

To illustrate the clinical implication of the current results of predictive values and likelihood ratios, by using an example of donning the 3M-A respirator, an interpretative summary of the validity and test performance of the user seal check for identifying actual gross leakage is presented as follows.

The prevalence of the actual gross leakage was approximately 37% (39.2-35.0%, as indicated in Table 4 ) when donning the given respirator, which was interpreted as pretest probability. 32, 33 Before conducting any kind of testing, a randomly selected nurse wearing the 3M-A respirator would have a 37% chance of having actual gross leakage.

Predictive values vary according to the prevalence of the actual gross leakage. High prevalence tends to have higher positive predictive value, whereas low prevalence tends to have higher negative predictive value. [26] [27] [28] The current prevalence of actual gross leakage was approximately 37% as mentioned, which was satisfactory in further calculating post-test probability.

This nurse then performs a routine user seal check to ensure the absence of subjective gross leakage. Likelihood ratios help to calculate post-test probability of actual gross leakage. The current results indicated that positive and negative likelihood ratios were 1.13 and 0.96, respectively. Therefore, with these ratios, the chance of the nurse with a positive user seal check having actual gross leakage is 41.8% (37% × 1.13), whereas a negative user seal check reduces the chance of the nurse having such leakage from 37% to 35.5% (37% × 0.96). Figure 3 illustrates such probabilities through the nomogram. Based on this example, the practice of the user seal check provides limited information in predicting the actual gross leakage when donning the given respirator.

Several limitations deserve discussing. One is that only 2 brands of respirators (ie, 3M and Kimberly-Clark) were used for gross leakage detection through QNFT. Although our aim was not to investigate the prevalence of gross leakage of all different models of N95 respirators, it was possible that different results might be obtained with different respirators. Nevertheless, we believe that the results supported the unacceptably low sensitivity and positive predictive value and futile likelihood ratios of the user seal check in identifying gross leakage of respirators. Apart from this, participants' characteristics might affect the passing rate of fit testing. First, most participants were novice users, except that some worked in clinical settings as health care workers. Previous experience and knowledge of donning an N95 respirator were insufficient, which may influence the passing rate of QNFT on gross leakage detection. This is different from a previous study, where Viscusi et al recruited subjects who were required to pass a standard QNFT. Therefore, the current results may underestimate the passing rate of QNFT. 24 Second, Asian participants' weight (reported here) and facial anthropometries (eg, face length, face width; not reported here) were significantly different from that of non-Asian people, which hence affects the passing rate of QNFT. Such differences might reduce the generalizability of the results but increase the specificity of that to Asian populations. Concerning environmental factors, the average monthly humidity in Hong Kong (subtropical climate) ranged from 67%-86% in 2014 34 ; yearly humidity computed from 2000-2014 was 78.1%. Most hospitals are only equipped with central air conditioning systems, and indoor humidity of wards may vary from 65%-75%. Relatively high humidity might underestimate the positive result of the user seal check in the current study because participants rely on subjective comparison between inward and ambient air to detect the leakage. Unlike well-controlled internal hospital settings in other regions, these environmental differences may limit the current results in that they are less relevant to other settings but are highly situation-specific results for many hospitals located in subtropical climate regions. Further studies are recommended to replicate the works from Myers et al 23 and Viscusi et al, 24 which examined the effectiveness of the user seal check on improving N95 respirator donning among Asian wearers. Another study may investigate how the change of body weight and facial anthropometries of Asian health care workers contributes to leakage of N95 respirators.

It is difficult to cite any evidence on the value of the user seal check on determining the fit of N95 respirators or even detecting any actual gross leakage during normal and deep breathing. However, the practice of the user seal check might contribute to enhancing the donning procedure of a respirator. Although the leakage is difficult to identify by subjective human sense, this check draws our attention to the issue that the tight-fitting respirator should be worn carefully.

Effectiveness of precautions against droplets and contact in prevention of nosocomial transmission of severe acute respiratory syndrome (SARS)

H5N1 outbreaks and Enzootic influence

Influenza pandemics of the 20th Century

Face mask use and control of respiratory virus transmission in households

Multidrugresistant tuberculosis outbreak among US-bound Hmong refugees

Worldwide emergence of extensively drug-resistant tuberculosis

Epidemic and pandemic alert and response (EPR): infection prevention and control of epidemic-and pandemic-prone acute respiratory disease in health care: WHO interim guidelines

Guidelines for preventing health-care-associated pneumonia, 2003: recommendations of CDC and the healthcare infection control practices advisory committee

Laboratory performance evaluation of N95 filtering facepiece respirators 1996

Regulations (Standards-29 CFR), Occupational Safety and Health Standards

The fitness of N95 respirators among undergraduate Chinese nursing students in Hong Kong

Predictive value of the user seal check in determining half-face respirator fit

Protecting healthcare staff from severe acute respirator syndrome: filtration capacity of multiple surgical masks

Respiratory protection by respirators: the predictive value of user-seal-check for the fit determination in healthcare settings

Sensitivity and specificity of the user-sealcheck in determining the fit of N95 respirators

Testing of the sensitivity and specificity of the user-seal-check procedure on "gross leakage" of N95 respirators

Racial differences in respirator fit testing: a pilot study of whether American fit panels are representative of Chinese faces

Portacount Pro 8030 respirator fit testers: operation and service manual

Respirator donning in post-hurricane New Orleans

3M Occupational Health and Environmental Safety Division

Qualitative fit testing instructions (KCPI-0356)

Use of personal protective equipment for respiratory protection

Effectiveness of fit check methods on half mask respirators

Evaluation of the benefit of the user seal check on N95 filtering facepiece respirator fit

Health care workers and respiratory protection: is the user seal check a surrogate for respirator fit-testing?

Understanding diagnostic tests 1: sensitivity, specificity and predictive values

How to report statistics in medicine

The sensitivity, specificity, and predictive value of traditional clinical evaluation of peripheral arterial disease: results from noninvasive testing in a defined population

Systematic reviews of evaluation of diagnostic and screening tests

Evaluation of sensitivity and specificity of rapid influenza diagnostic tests for Novel Swine-Origin Influenza A (H1N1) virus

Foundations of clinical research: applications to practice

Diagnosis in general practice: using probabilistic reasoning

Evidence-based medicine: how to practice and teach EBM

The Year's Weather-2014

We thank all participants for their contribution to this study; Jojo Y.Y. Kwok, Billy O.Y. Pang and Rebecca C.M. Tsang for supervision on data collection; Ka-yan Chan, Suen-fuk Fan, Nga-yi Lau, Sinting Tai, Tsz-kwan Yuen, Wai-hung Chung, and Wing-on Lui for assistance on data collection and data input; Andy C.Y. Chong for provision of statistical support; and our division for support on specific equipment, consumables, and room usage.