key: cord-0659680-6oikp7wt
authors: Fu, Yanwei; Zhao, Lei; Zheng, Haojie; Sun, Qiang; Yang, Li; Li, Hong; Xie, Jiao; Xue, Xiangyang; Li, Feng; Li, Yuan; Wang, Wei; Pei, Yantao; Wang, Jianmin; Wu, Xiuqi; Zheng, Yanhua; Gu1, Hongxia Tian Mengwei
title: Rapid COVID-19 Risk Screening by Eye-region Manifestations
date: 2021-06-12
journal: nan
DOI: nan
sha: a7df45ec55d34c22ca4095d4009bfdd3276788c3
doc_id: 659680
cord_uid: 6oikp7wt

It is still nontrivial to develop a new fast COVID-19 screening method with the easier access and lower cost, due to the technical and cost limitations of the current testing methods in the medical resource-poor districts. On the other hand, there are more and more ocular manifestations that have been reported in the COVID-19 patients as growing clinical evidence[1]. This inspired this project. We have conducted the joint clinical research since January 2021 at the ShiJiaZhuang City, Heibei province, China, which approved by the ethics committee of The fifth hospital of ShiJiaZhuang of Hebei Medical University. We undertake several blind tests of COVID-19 patients by Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China. Meantime as an important part of the ongoing globally COVID-19 eye test program by AIMOMICS since February 2020, we propose a new fast screening method of analyzing the eye-region images, captured by common CCD and CMOS cameras. This could reliably make a rapid risk screening of COVID-19 with the sustainable stable high performance in different countries and races. Our model for COVID-19 rapid prescreening have the merits of the lower cost, fully self-performed, non-invasive, importantly real-time, and thus enables the continuous health surveillance. We further implement it as the open accessible APIs, and provide public service to the world. Our pilot experiments show that our model is ready to be usable to all kinds of surveillance scenarios, such as infrared temperature measurement device at airports and stations, or directly pushing to the target people groups smartphones as a packaged application.

In December 2019, novel coronavirus disease 2019 (COVID-19) broke out globally. The pathogenic virus was named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). As of June 2, 2021, there are about 170 million confirmed cases and 3.7 million deaths worldwide, and the World Health Organization (WHO) declared COVID-19 a public health emergency of international concern [2, 3] . SARS-CoV-2 belongs to the family of Coronaviridae, which is a large family of positive-stranded single-stranded ribonucleic acids with envelopes. Previous studies confirmed that there are six coronaviruses can infect humans, including human coronavirus 229E (HCoV-229E), HCoV-OC43, HCoV-NL63, HCoV-HKU1, Middle East respiratory syndrome coronavirus (MERS-CoV), SARS-CoV [4] , SARS-CoV-2 is the seventh member of the coronavirus family that can infect humans [5] . When infected with SARS-CoV-2, the most common symptoms are fever, dry cough, and fatigue, the less common symptoms are expectoration, diarrhea, headache, hemoptysis, chest pain, anorexia, myalgia, chills, nausea and vomiting, and so on [6, 7] . In addition, in small cohort studies and case reports dysosmia and taste disorders like anosmia, phantosmia, parosmia ageusia, dysgeusia also have been reported [8] [9] [10] . But it is impossible to make an accurate diagnosis of COVID-19 according to the patient's clinical symptoms. At present, imaging, nucleic acid, and serum antibody are the most common methods used for diagnostic [11] , but limited to laboratory or hospital environments, demanding expert-level operations. Thus, patients must be tested in the hospital, and take some waiting time. All these points, greatly limit these state-of-the-art detection methods deployed at large-scale cities, and enabling real-time patient tracking. With the further development of the research, many studies have reported that COVID-19 patients can be combined with different ocular manifestations, such as hyperemia, increased secretion, epiphora, edema, follicular conjunctivitis, scleritis, photophobia, foreign body sensation, itchiness, and so on [11] .

Recently, deep learning based classification networks have been widely used to support the disease diagnosis and management [16, 17] . Inspired by these points, we propose a rapid COVID-19 risk screening and diagnosis model with the deep learning method based on eye-region images, captured by normal CCD or CMOS camera and cellphone. We propose the classification model with the corresponding eye images as the input to identify the COVID-19 as a binary classification task. We trained the models by the data from ShiJianZhuang and Shanghai. The performance was measured at the Area Under receiver-operating-characteristic Curve (AUC), sensitivity, specificity, and accuracy, and F1.

After more than three months of study and clinical trials, we found that the confirmed cases of COVID-19 present the consistent eye pathological symbols, including the asymptomatic. This study included 20 subjects in the pretest dataset (train and validation) and 212 subjects in the blind test dataset. The subject-level performance of COVID-19 prescreening model in the combined study have achieved an AUC of 0.940(95% CI, 0.888-0.992), 0.990 (95% CI, 0.972-1.000) and 0.975 (95% CI, 0.940-1.000) with respect to the first, second and third blind tests.

As part of the ongoing globally COVID-19 eye test program by AIMOMICS since February 2020, collaborated research worldwide and across racial boundaries has been set up, to call for a more massive database and further refinement to improve the model performance. In the past and ongoing globally registered clinical trials, we show that this model can successfully classify COVID-19 patients from healthy persons, pulmonary patients except for COVID-19 (e.g., pulmonary fungal infection, bronchopneumonia, chronic obstructive pulmonary disease, and lung cancer), liver patients, diabetes patients and ocular patients. The experimental results reveal that patients with COVID-19 have different ocular features from others, which can be used to differentiate them from the public.

The convenient method of eye-region image diagnosis can help the disease control researchers to fully understand the prevalence and pathogenicity of the virus in different ages, time, region, climate, environment, occupation, and population with basic diseases, and guide effective prevention and control measures against COVID-19. Meantime, our model for COVID-19 rapid prescreening have the merits of the lower cost, fully selfperformed, non-invasive, real-time results, and thus enables the continuous surveillance. We further implement it as the open accessible APIs, and provide public service to the world. Essentially, our model is also open to other form of embedding for fast screening of COVID-19, such as all kind of infrared temperature measurement device at airports and stations, or directly push to the target people groups' smartphones as a packaged application. We believe a system implementing such an algorithm should assist the large-scale rapid large-scale screening and real-time follow-up, that can be inspiring and helpful for encouraging more researches on this topic.

COVID-19, eye-region image, symptom classification, deep-learning, rapid screening,

The data for the study comes from the following three resources,

• COVID-19 patients: all COVID-19 cases were acquired from January to Marth, 2021 by the Fifth Hospital of Shijiazhuang, Hebei Medical University, Shijiazhuang, China. All patients were diagnosed according to the diagnostic criteria of the National Health Commission of China and confirmed by RT-PCR detection of viral nucleic acids. the photos as well as the patient's information are collected. Meanwhile, we obtained the epidemiological, medical history, clinical characteristics, laboratory tests and treatment history of the enrolled patients from electronic medical records and nursing records. Especially, all patients were given chest X-rays or CT. Some of the data needs to be supplemented and confirmed, and we obtain the data through direct communication with the doctor at the bedside, the details of demographics, basic characteristics, clinical characteristics and outcomes of the collected COVID-19 patients are summaries in Table. All patients were taken images according to our datacollection guidelines by two smartphones.

• The rehabilitation patients (nucleic acid changed from positive to negative): all rehabilitation cases were acquired from February to Marth, 2021 by the Fifth Hospital of Shijiazhuang, Hebei Medical University, Shijiazhuang, China. These rehabilitation patients have clear examination reports. we balanced the distribution of age and gender. All cases were taken images according to our datacollection guidelines by four smartphones.

• The liver patients and healthy peoples: all cases as control group were obtained from Marth to April 2021 by Medical Examination Center,Hubei Provincial Hospital of Traditional Chinese Medicine,

Wuhan. These liver patients and healthy peoples have clear examination reports. we balanced the distribution of age and gender. All volunteers were taken images according to our data-collection guidelines by three smartphones.

• The data included 1194 photographs (pre-test 101, blind test 1093) from 232 participants (pre-test 20, test 212). We randomly merge the pre-test dataset into the previous work for model updating. The blind test is divided into 3 tests.

• In the pre-test dataset, 20 COVID-19 patients were included, the blind test dataset comprises 66 COVID-19 patients and 166 control group participants.

The demographic and clinical characteristics of COVID-19 patients were shown in Table 1 . In the study there are 2 groups: the COVID-19 group and the control group. The control group comprises of three subgroups: RH (rehabilitation patients -nucleic acid changed from positive to negative), LV (liver patients) and HL (health volunteers). The CV (COVID-19 patients) were confirmed by the RT-PCR detection for viral nucleic acids. According to the eighth version guideline published by the National Health Commission of China, asymptomatic/mild COVID-19 carriers refer to individuals with viral nucleic acid test positive, and no clinical symptoms and no signs of pneumonia in chest imaging. The patients with RH (rehabilitation patients -nucleic acid changed from positive to negative). The patients with LV (liver patients) were diagnosed as hepatopathy. HL (the health volunteers) were collected from individuals who had taken physical examination, no obviously abnormal results were demonstrated and no contact history with COVID-19 patients. All the subjects were tested for the COVID-19, and no participants showed viral nucleic acids positive during the following days, except for COVID-19 patients. No death events were observed in this study.

For each participant, 3-10 photographs of the eye-region were captured by using the common CCD and CMOS cameras, assisted by doctors or healthcare workers. The same shooting plain mode and parameters were used, and avoid shooting filters. The photos were captured in a good lighting condition, and not in dark or red background. The image resolution is at least 1900x500 96dpi. The average time for taking a set of 5 eye photos is around 1 minute. In the re-examination step, all the data that cannot reveal the details of the eyes are discarded.

The study was conducted in accordance with the principles of the Declaration of Helsinki. All participants were provided with the written informed consent at the time of recruitment. And this study was approved by the ethics committee of the fifth hospital of ShiJiaZhuang of Hebei Medical University (approval No.: 2021005), and was approved by ClinicalTrials.gov PRS (ClinicalTrials.gov ID: NCT04907981).

First, we observed the pictures with an unaided eye, and found that among the 212 patients who were included(20 patients with pre-test were not counted), there were 21 patients (9.9%) combined with ocular manifestations. In the COVID-19 group, 17 patients (36.9%) showed ocular manifestations (Figure 1 ), including conjunctival congestion(N=13), increased secretion(N=4), conjunctival hemorrhage(N=1), and ptosis(N=1). While only 4 cases (9.1%) showed conjunctival congestion(N=4) and ptosis(N=2) in the rehabilitation patients. At the same time, no ocular manifestation was observed in liver disease and healthy control group.

Then, in order to evaluate the study, we develop a model from the previous work and the Hubei pre-test data, using eye-region photos on the dataset. The blind test group includes COVID-19 positive patients, and control group as rehabilitation patients (nucleic acid changed from positive to negative), liver patients, healthy people. As shown in Figure 2 , the experiment is divided into 3 blind tests. During the blind test on each group, the data from the previous blind test is used as extra training data. As for the COVID-19 category, the average sensitivity target is no less than 80%, which could demonstrate that the efficacy of our method to distinguish COVID-19 patients.

In the first blind test, the pre-test data is combined to improve the model from previous-work. During the second and third blind, the previous blind test data is used as extra data for training and validation. 

The pipeline of our proposed method is illustrated in Figure 3 , which is composed of two steps, the imagelevel classification, and the subject-level classification. The classification network is built by the deep learning neuron networks to study the characteristics of the cropped eye regions in a latent space. Specifically, we define the subject as a set of photos taken by the same people. The classification result of the same person in might be different due to the noise in data and a proper vote strategy can enhance the robustness of the model.

The results of different images of the same subject are merged by a special vote strategy. Finally, the subjectlevel risk assessment of COVID-19 is generated. To measure the performance of the binary classification network, we define a set of metrics, such as the area under the receiver-operating-characteristic curve (AUC), sensitivity, specificity, and accuracy. Bootstrapping strategy with 1000 replicates was used to estimate 95% confidence intervals of the metrics, with an imagelevel resampling unit. Specifically, the receiver operating characteristic curves (ROCs) were used to illustrate the performance in screening COVID-19 disease.

The classification network has achieved an AUC of 0.940(95% CI, 0.888-0.992), 0.990(95% CI, 0.972-1.000) and 0.975(95% CI, 0.940-1.000) for the first, second, and third blind tests on the subject-level classification. The metrics including AUC, sensitivity, specificity, accuracy, and F1 are shown in Table 2 . The image-level, max-voting ROC is plotted in Figure 4 .

Setting The overall classification system performed well, misjudgment also appeared. In the blind test, the true positive, false positive, false negative and true negative results are summarized in the confusing matrix on both the subject-level and image-level, and the details were shown in Table 3 . To investigate the interpretability of the model, we conduct a visual analysis of the key areas of the model's attention in the classification process. The key areas of the model's attention were converted into heat maps based on gradients and activation maps by GradCAM.

For case study heatmap in Figure 5 , we found that the attention of heatmaps of Non-COVID-19 and COVID-19 mainly covers the iris, upper and lower eyelid, the inner and outer eye corner. 

COVID-19 is an infectious disease caused by SARS-CoV-2. The patients usually have fever, cough, fatigue, shortness of breath, and loss of smell and taste. Nowadays, some researchers also find that Olfactory and Gustatory Dysfunction is one of the common symptoms of the COVID-19. Meantime, more ocular manifestations have been reported in the COVID-19 patients as growing clinical evidence [18] [19] [20] . Accordingly, abnormally high body temperature has been the initial and most widely used screening method for Various publications showed the ocular involvement with Coronavirus family [21] [22] [23] . In our study, the Ocular symptoms of COVID-19 patients accounted for about 36%, the main manifestation is conjunctivitis, which was consistent with the results of previous studies [15, 24] . SARS-CoV-2 infects the body through the binding of angiotensin converting enzyme 2 (ACE2) receptor [25, 26] , it is widely expressed in a variety of cells, primarily in the lungs, kidneys, heart, gastrointestinal tract, and liver [25, 26] . Recently, it has also been reported that ACE2 expressed in conjunctiva [28] . Its expression in conjunctiva may also be related to the occurrence of COVID-19

conjunctivitis.It is worth noting that some of the recovered patients still have ocular manifestations, and there are also some patients are positive in the model test. Studies have shown that the ocular performance of rehabilitated patients is accompanied by a significant increase in the level of inflammatory factors [29, 30] , which indicate the eye performance is not caused by the direct invasion and destruction of SARS-CoV-2, but the virus causes a wide range of pro-inflammatory cytokines and chemokine responses in conjunctival and limbal epithelial cells [29, 30] , resulting in a proliferation of cytokines, resulting in apoptosis of peripheral corneal epithelial cells and reactive inflammatory injury. Therefore, we believe that longer and more frequent follow-up is necessary for the recovered patients. We are conducting the large-scale experiments to further validate the effectiveness and efficacy of our algorithm as the tool of the new screening method. On the other hand, due to the privacy policy and the difficulty of data collection in different countries by racial diversity, we have furthered investigate and extended the capability of our algorithm in few-shot learning settings. We have achieved sustainable stable high performance in a series of registered clinical trials at different counties, thanks to our previous works [32, 33] from the computer vision and machine learning communities.

We believe that this study can be inspiring and helpful for encouraging more researches in this direction, and provide effective and rapid assist for clinical risk screening, especially during outbreaks. To conduct further research, we are building a globally COVID-19 eye test free API and Demo links platform which implements our algorithms and helps fast screening COVID-19. Furthermore, this platform could gradually open more than 300 diseases eye tests and joint study programs' invitation, such as virus influenza, diabetes, hepatopathy, etc. through the open accessible APIs or Demo links, the test algorithm could be easily deployed or embedded in the HD camera and any detect accessories, combined into a multi-modal approach including vision and other sensors, continually monitoring the particular diseases control districts were around the public access, transportation hub, population centre, quarantine house, making everyone health care more accessible with lower cost.

So we are looking forward to more scientists, engineers and application developers join us to explore the eye testing and collaborative work together on the critical challenges, ensuring "equality、altruism and ethics " are the highest principle.For more study information and the access license for joining our the globally COVID-19 eye test program, please contact Dr. Yanwei Fu yanweifu@fudan.edu.cn.

1) For the above sampling, the same model of mobile phone or shooting equipment must be used to prevent the sampling data domain from being interfered with by the equipment.

2) If the conditions are not available, it is necessary to use the same model of mobile phone to collect all the sampling images at the same time, to maximize the elimination of device data domain interference.

3) The same shooting parameters must be used when shooting, and the beauty, soft light, and other shooting filters must not be used.

4) The image resolution of the eyes is at least: 1900x500 96dpi

5) The shooting environment should be well-lit and bright. It should not shoot in front of dark and red backgrounds. A white background is best.

A New Screening Method for COVID-19 based on Ocular Feature Recognition by Machine Learning Tools

A Novel Coronavirus from Patients with Pneumonia in China

Going global -Travel and the 2019 novel coronavirus

Hosts and Sources of Endemic Human Coronaviruses

The genetic sequence, origin, and diagnosis of SARS-CoV-2

Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

Virological assessment of hospitalized patients with COVID-2019

The neuroinvasive potential of SARS-CoV2 may play a role in the respiratory failure of COVID-19 patients

Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention

Diagnosing COVID-19: The Disease and Tools for Detection

Ocular Symptoms among Nonhospitalized Patients Who Underwent COVID-19

Ocular manifestations of a hospitalised patient with confirmed 2019 novel coronavirus disease

Episcleritis as an ocular manifestation in a patient with COVID-19

Characteristics of Ocular Findings of Patients With Coronavirus Disease 2019 (COVID-19) in Hubei Province

Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning

COVID-19 Artificial Intelligence Diagnosis using only Cough Recordings

Ocular manifestations and clinical characteristics of 535 cases of COVID-19 in Wuhan, China: a cross-sectional study

Ophthalmic and Neuro-ophthalmic Manifestations of Coronavirus Disease

Update and Recommendations for Ocular Manifestations of COVID-19 in Adults and Children: A Narrative Review

Evaluation of coronavirus in tears and conjunctival secretions of patients with SARS-CoV-2 infection

2019-nCoV transmission through the ocular surface must not be ignored

Revisiting the dangers of the coronavirus in the ophthalmology practice

Conjunctivitis can be the only presenting sign and symptom of COVID-19

Structural basis for the recognition of the 2019-nCoV by human ACE2

Structure of the SARS-CoV-2 spike receptor-binding domain bound to the ACE2 receptor

ACE2 receptor polymorphism: Susceptibility to SARS-CoV-2, hypertension, multi-organ failure, and COVID-19 disease outcome

Mechanism of the action between the SARS-CoV S240 protein and the ACE2 receptor in eyes

Relapsing viral keratoconjunctivitis in COVID-19: a case report

Late manifestation of follicular conjunctivitis in ventilated patient following COVID-19 positive severe pneumonia

COVID-19 and Eye: A Review of Ophthalmic Manifestations of COVID-19

How to trust unlabeled data Instance Credibility Inference for Few-Shot Learning

Transductive multi-view zero-shot learning

Data-collection guidelines

For scientific research, we need some eye region data. We promise that the data will not be used for commercial, only for scientific research. The specific requirements are as follows

When taking photos, please do not use beauty camera mode, or post-production beauty filters, we need the original images

a) The subject or legal representative has signed an informed consent form b) Age ≥18 and ≤75 years old (under 18 years old, the guardian shall sign the informed consent form) c) Meet the diagnostic criteria for COVID-19 infection

All participants were provided with written informed consent at the time of recruitment. And this study was approved by the ethics committee of The Fifth Hospital of ShiJiaZhuang of Hebei Medical University. The study was in accordance with the Declaration of Helsinki 1964 and its successive amendments. 

All other authors declare no competing interests.

We would like to thank Dr. DaiErHei and Dr. Yu Liu for their kind assistance with this research project.