key: cord-0913216-wwbscuwd
authors: Ara, Jinat; Sik-Lanyi, Cecilia
title: Investigation of COVID-19 Vaccine Information Websites across Europe and Asia Using Automated Accessibility Protocols
date: 2022-03-01
journal: Int J Environ Res Public Health
DOI: 10.3390/ijerph19052867
sha: 0044487c2f3005c7e746c4ac875d1938e9a33b6e
doc_id: 913216
cord_uid: wwbscuwd

Websites content accessibility guidelines (WCAG) ensure that websites should be perceivable, understandable, navigable, and interactive. During the SARS-CoV-2 pandemic, the importance of accessible websites and online content grew throughout the world. Therefore, in this study, we examined COVID-19-related official government websites. This research covered 21 government websites, with 13 websites from European countries and 8 websites from Asian countries, to evaluate their accessibility following WCAG 2.0 and WCAG 2.1 guidelines. The overall goal of this study was to identify the frequent accessibility problems that might help the website owners to identify the shortcomings of their websites. The target websites were evaluated in two steps: in step-1, evaluation was performed through four automatic web accessibility testing tools such as Mauve++, Nibbler, WAVE, and WEB accessibility tools; in step-2, evaluation went through human observation, such as system usability testing and expert testing. The automatic evaluation results showed that few of the websites were accessible; a significant number of websites were not accessible for people with disabilities. In addition, system usability testing found some complexity in website organization, short explanations, and outdated information. The expert testing suggested improving the color of the websites, organization of links, buttons, and font size. This study might be helpful for associated authorities to improve the quality of the websites in the future.

The world wide web (WWW) was designed in 1989 to make information accessible to people [1] . In recent days, a large amount of data has grown faster on the cloud, and people are engaging online rapidly [2] [3] [4] . Therefore, the accessibility of online information is a crucial aspect for technologists and researchers. Nowadays, several past studies revealed that a large amount of online content such as websites are not accessible for people with disabilities [3, 5] . Generally, the website is a digital platform for sharing a variety of information (healthcare, education, e-commerce, etc.) with the people. However, sometimes, these digital resources are not properly organized (such as the presence of broken links, out-of-date content, dilated images, etc.). From the perspective of digital services, the digital resources should be organized to make them accessible and barrier free to the people with disabilities [6] . Thus, to ensure accessible digital services, it is important to ensure that people with disabilities are considered. Their needs should be incorporated in the design and implementation of the web application.

Generally, web accessibility suggests that a website should have an advanced design and the latest technology for development so that people with disabilities can perceive, understand, navigate, and interact with the web more efficiently [7] . The W3C Web Accessibility Initiative (WAI) is an organization that introduces accessibility guidelines known as web content accessibility guidelines (WCAG) [8, 9] . It provides WCAG 1.0, WCAG 2.0, WCAG 2.1, WCAG 2.2 and WCAG 3.0 standards for three conformance levels A, AA, and AAA. The WCAG aims to ensure accessible websites or content for people with disabilities [10] [11] [12] [13] [14] .

According to the World Bank statistics, one billion people (15% of the world population) across the world have some disabilities such as visual problems (vision impairment or vision loss), auditory problems (hearing disability), physical disabilities (disorder in a body part), speech problems (disorder speech production), cognitive or neurological problems (disorder in cognitive functions) and others [15] . According to the report of the World Health Organization (WHO), among the whole population of the world, 3.2% of people have a visual impairment, 6% of people have auditory or hearing difficulties, 2.6% of people have a neurological difficulty, and 1% of people have physical disabilities or require a wheelchair [16] . Based on the statistics of WHO, Figure 1 shows that, for every 100 people, 44 people have auditory, 23 people have visual, 19 people have cognitive, and 7 people have speech and physical issues throughout the world. According to EU statistics, in Europe, one in six people have some disabilities who are actively involved online to search for information [17] . The United Nations reported that, in the Asian region, more than 10% of the people who have disabilities are actively involved in internet platforms for information [18] . In 2019, the whole world experienced the SARS-CoV-2 virus. It was first detected in Wuhan, China, and spread rapidly throughout the world [19, 20] . To control this virus, governments of different countries have started a vaccination program from 14 December 2020. To share the updated COVID-19 vaccine information and to facilitate the vaccination process, the governments of several countries have created their websites. These web-based portals aim to share COVID-19 vaccine information with the people. It helps people understand the vaccines and the importance of vaccination to fight against COVID-19. These government websites are considered to be official sources of COVID-19 vaccine information. Generally, all governments have individual regulations and publish all the information on their official websites, including the prioritized vaccine for senior people/government workers/social workers and others. In some countries, they have age limitations; some require vaccine registration or take online appointments, etc. Therefore, COVID-19 vaccine information seekers are relying on these official portals. In the context of digital accessibility and disability, statistics show that a significant number of people with a disability have accessed the online portal frequently [16] . Thus, accessibility is more important for COVID-19 vaccine information portals because restrictions in COVID-19 vaccine information access might reduce the social support of people with disabilities. Considering these issues, it is important to understand the accessibility of the official government-operated COVID-19 vaccine information websites and to implement inclusive design and development practices to make digital COVID-19 vaccine information portals barrier-free for people with special needs. These deliberations emphasized investigating and understanding the accessibility of official governmentCOVID-19 vaccine information websites to make the COVID-19 vaccine information available to all.

Few researchers from different countries (US, UK) have performed research on the accessibility of COVID-19 vaccination/registration portals [21] . They suggested that a significant number of official and private portals have accessibility issues. The majority of the countries have no strict rules or monitoring system to control these inaccessible websites that create massive barriers for the people who have disabilities. Therefore, researchers emphasized conducting comprehensive research on government and private COVID-19 vaccine information websites to validate their accessibility. Several organizations have showed their impressive effort and created their own websites for sharing information about the COVID-19 vaccine. This research is limited to the investigation of governmentoperated official COVID-19 vaccine information websites, as they tend to be more reliable, effective and accepted by the community.

By the last few years, researchers developed several online web accessibility testing protocols or tools that help the online content authority to understand and assess the accessibility of their online platform [22, 23] . Among numerous web accessibility testing protocols: AChecker, Mauve++, The Accessibility Management Platform (AMP), TAW, WAVE, HTML Validator, and Nibble are more popular and have been used by previous researchers [19] . All the developed accessibility evaluation tools ensure fast and accurate accessibility scores, are cost-effective, and produce reliable evaluation results. Thus, identifying the appropriate and effective tools is challenging and a critical aspect. Therefore, our prime objective of this research is to analyze government COVID-19 vaccine information websites using multiple web accessibility testing protocols. In this research, we employed multiple web accessibility tools, as it is easier to compare and validate the result more effectively. Therefore, this study considers four popular open-source web accessibility testing tools, Mauve++, Nibbler, WAVE, and Web, to evaluate 21 official governmentCOVID-19 vaccine information websites from different parts of Europe and Asia.

Furthermore, we validated the targeted websites through questionnaire-based human observations, including system usability testing and expert testing. Based on our findings, we offer some suggestions for the authorities involved with the COVID-19 information websites and web developers to improve the accessibility of their websites. This research might help to improve the website's accessibility for people with disabilities and make the COVID-19 vaccine information freely accessible without any barriers. Throughout the evaluation, we tried to answer four research questions:

Research Question 1 (RQ1). Is there any difference in the countries' websites accessibility between the two continents in their average number of errors? What about the error if we focus on the website of each country? Hypothesis 1 (H1). There is no difference between European and Asian countries' websites according to their average number of errors. There is a difference if we focus on the website of each country.

Research Question 2 (RQ2). Is there any difference between European and Asian countries websites based on human judgment results?

There is no significant difference between websites from the Europe and Asia considering questionnaire based human judgment results.

. Is there any difference in the result of automatic testing and human judgment result?

Generally, there are several techniques or methods to evaluate web accessibility. Four web accessibility evaluation techniques are more popular and effective: automatic evaluation tools [24] , manual evaluation or expert evaluation [25] , user evaluation (those who have disabilities) [26] , and hybrid evaluation (both automatic and manual judgment). The automatic evaluation technique allows the tester to evaluate the website's accessibility by running the evaluation tools into the browser. It also refers to software which determine the quality of the websites based on predefined web accessibility standard guidelines [27] . Manual evaluation is performed using human experts to evaluate websites accessibility. Usually, manual evaluation or observation helps to identify the limitation of the automated accessibility testing tools. Sometimes, automated tools cannot determine all the accessibility issues according to the accessibility guidelines and, thus, require human judgments. Website accessibility testing with users having disabilities refers to user testing or testing assisted by aid users. In this aspect, users with disabilities can interconnect with website testing directly. The users with disabilities are invited and asked to evaluate the web pages at their discretion. Sometimes this testing procedure is more reliable than other manual or automatic testing processes. Furthermore, hybrid website accessibility testing is the process of performing automated and human evaluation at the same time. Sometimes, automated, human evaluation, and user expert testing are also referred to as hybrid testing. It helps to compare and improve the multiple testing results, simultaneously.

To the result of the background study, the majority of the previous studies proposed web accessibility evaluation techniques using several existing automated tools. AlMeraj et al. [5] present the evaluation result for higher education websites of Kuwait using three automated website accessibility testing tools, such as AChecker [28] , total validator [29] , and WAVE [30] . It shows that, of 48 websites, only 11 websites were found accessible through the evaluation tools. Ali [19] investigated 87 educational websites of the UAE through automatic testing tools: A-CHECKER, CYNTHIA SAYS [31] , and TAW [32] . Results indicated that only 5 websites were accessible through the automated evaluation tools. Zare et al. analyzed 51 medical science websites of Iran for identifying the accessibility of these websites [33] . They suggested that, using the 'WEB accessibility' automated testing protocol, none of the websites were effective and suggested continuing the testing with other existing automatic tools. Król et al. used WAVE to analyze 91 hospital websites of Poland and found that none of the websites passed accessibility testing using this particular testing tool [34] . Alsaeedi et al. [35] and Kuppusamy et al. [36] investigated 6 Saudi Arabian university websites and 25 Indian higher educational institution websites. They used WAVE, SiteImprove [37] , and AChecker and found that most websites have low accessibility scores, and few of them passed the accessibility testing. Alshamari et al. [38] investigated 3 E-commerce websites of Saudi Arabia through 4 accessibility testing tools: Achecker, TAW, MAUVE [39] and FAE [40] . They found that navigation, readability, input assistance and timing are the common accessibility problems. However, the majority of the existing literature investigated private websites. In 2017, Ismail et al. investigated 33 general administrative departments websites of India using some automated tools [41] . They employed AChecker, Cynthia Says, Tenon [42] , WAVE, Mauve, and Hera [43], free web testing protocols to perform the analysis. They found that most of the government websites had poor accessibility, with several errors.

After reviewing the most recent studies, we found some past studies regarding education, healthcare, and E-commerce websites [44] . Due to the COVID-19 pandemic, a significant number of people are engaging in the COVID-19 portal for information. Nowadays, COVID-19 vaccine-related websites are more popular among the people of the different countries of the world. Therefore, evaluating COVID-19 vaccine government websites is crucial. Focusing on this issue, Alismail et al. [21] evaluated 54 COVID-19 vaccination registration websites of the US using WAVE, AChecker, and SortSite [45] evaluation tools. They found that only nine websites were accessible, where two came from WAVE and SortSite and five came from AChecker automated tools.

However, to the best of the author's knowledge, none of the studies performed the evaluation process on government COVID-19 vaccine-related websites, even though the importance of ensuring accessibility is beyond question. The majority of the existing approaches only focused on the automated tools, yet, other testing aspects are also significant and can improve the evaluation result. Therefore, in this study, we evaluated COVID-19 government vaccine websites through the hybrid technique. We implemented both automated tools (Mauve, Nibbler, WAVE, and WEB accessibility) and human observation through questionnaires. Additionally, this analysis will help to understand the effectiveness of the automated tools and human validation.

This study presents the investigation of SARS-CoV-2 related government websites to understand their accessibility for people with disabilities. The government websites provide valuable information such as safety guidelines, COVID-19 symptoms, the total number of active cases, the vaccine registration process, etc. From this perspective, web technologists and previous researchers proposed several works for web accessibility and directed for future research. Therefore, in this work, we employed automatic web accessibility testing tools to investigate the most common accessibility problems of the COVID-19 vaccine information-related websites and to identify the accessibility of these sites. Figure 2 shows the workflow diagram of the investigation process of the COVID-19 vaccine information websites through automated tools. 

In this study, we considered websites from two groups of the country: group-1: Twelve European countries (Estonia, France, Greece, Ireland, Latvia, Luxembourg, Netherlands, Iceland, Austria, Switzerland, Croatia, Germany), and group-2: Eight Asian countries (India, the Philippine, Jordan, UAE, Kyrgyzstan, Maldives, Hong Kong, and Qatar). Initially, 63 websites have been found for both governmental and private institutions (Figure 3 , step-2). The prime goal of this study was to investigate the official government websites only. Therefore, we eliminate all the private organization websites from the investigation. After eliminating the private websites, a total of 21 websites were found as official government websites and were considered for our evaluation, as shown in Figure 3 . The considered websites are shown in Table 1 with their country, region and page URL. Multiguideline Accessibility Usability Validation Environment (MAUVE) is an environment for web content accessibility evaluation according to WCAG standards [39] . The researchers of HIIS Lab developed and provided it for the community as an open access testing environment. Generally, it works for usability criteria identification to improve the website navigation for physically disabled people, specifically focusing on the end-user demands. It helps to determine website accessibility through WCAG 1.0, WCAG 2.0, and WCAG 2.1 standards. It focuses on three conformance levels: A, AA, and AAA to test websites. Conformance levels represent a set of requirements that must be fulfilled to make content available online. According to the WCAG standards, there are three conformance levels; every conformance level has different guidelines or requirements. Therefore, before publishing the content online, authorities should focus on the conformance requirements and should satisfy all the conformance levels (A, AA, AAA) to make it accessible. MAUVE++ automatic accessibility testing tool checks against all three-conformance level requirements during the evaluation process of websites. MAUVE allows running the software in a browser plugin, desktop, smartphone, iPad, tablet, etc. It is possible to download the result in JSON format. It generates the report through the total number of successful checkpoints, warning checkpoints, and erroneous checkpoints. The overall accessibility percentage generated is based on the ratio of the number of checkpoints passed and the number of checkpoints tested. Generally, it provides the error based on HTML and CSS rules violation according to the WCAG guidelines. It can be tested on both single-page and dynamic page websites simultaneously without any interruption. Figure 4 shows the web accessibility result of the European vaccination information website through the Mauve++ testing environment. 

Nibbler is a free website accessibility testing tool that accesses the website URL and generates the accessibility result [46] . In a word, Nibbler is a 'bot' or 'automated computer program that works by looking at the domain name to find the linked web pages. It calculates the overall accessibility score on a 10-points scale based on four key prototypes (accessibility, experience, marketing, and technology). Each prototype analyzes several aspects such as code quality, headings, internal links, mobiles, page titles, URL format, amount of content, Facebook page, freshness, images, popularity, printability, server behavior, Twitter, analytics, domain age, incoming links, meta tags, and social interest to determine the overall accessibility score. Figure 5 shows the screenshot of the Nibbler tool for the European Union vaccination information website. 

WEB accessibility is another popular website accessibility testing tool that highlights the graphical view of the original website [47] . It is free, open-source software and is publicly available for the entire browser (Chrome, Firefox, etc.) as a testing medium. It works by calculating accessibility percentage through compliance score, number of identified violations, number of automated tests, and suggests the additional test that should be carried out. Additionally, it provides details about identified violations with their severity level. Figure 8 shows the result of a vaccination information website for Estonia by WEB accessibility tool with a 95% compliance score. 

According to the past research [3, 5, 21] , automatic web accessibility testing tools are effective. Though automatic tools have high effective measurement, still, several issues are unrevealed by these tools. Thus, we perform human observation through two testing systems: system usability testing and expert testing to validate the accessibility issues.

Through System Usability Scale (SUS), we can understand the effectiveness of the websites for a particular group of people [48] . To do the system usability testing, we invite few foreign volunteers to measure the usability of the websites and provide their feedback according to ten questions. We set the system usability measuring scale from Strongly Disagree (1) to Strongly Agree (5) to provide the usability feedback. Table 2 shows the example of questionnaire-based system usability testing format that have used in this research. The detailed result has shown in Section 4. All the volunteers who participated in this testing were university third-year (of fouryear studies) bachelor students from the computer science and engineering department of Jahangirnagar University, Bangladesh. Every student had completed the 'User Interface Design' course. Thus, they were capable of observing the websites according to the questionnaire. The number of students was 10, including 4 female and 6 male students between 19 and 22 years of age. To investigate 21 websites, we arranged online participation and shared questionnaires and website information with the students. To distribute the work, we classified 10 students into two groups: group-1 belonged to 6 students, and group-2 belonged to 4 students. Group-1 was responsible for evaluating European websites, and group-2 was responsible for the evaluation of Asian websites. In group-1, we asked 5 students to evaluate 2 websites each, and 1 student was assigned to investigate 3 websites. Besides, in group-2, 4 students were responsible for testing 2 websites each. The calculated average investigation time was 15 min/website. The investigation process in detail is shown in Table 3 . 

Expert testing is the process of human testing that identifies the existing problems of a particular website and makes recommendations to upgrade the inaccessible functions. To do the expert testing, we set twelve questions with a yes/no answer option and sent them to the foreign experts to provide their responses according to their observation. For the expert testing, four experts had been invited in this research to validate the selected websites. The occupation of the experts includes an associate professor, an assistant professor, a senior lecturer, and a Ph.D. student from the department of electrical engineering and information systems, University of Pannonia, Hungary, and department of computer science and engineering, Jahangirnagar University, Bangladesh. All the experts validated 21 websites and provided their findings. Every expert had least 5 years of research experience in web accessibility, multimedia design, human-computer interaction, and web informatics fields. Table 4 shows the example of questionnaire based expert testing format that have used in this research. The detailed result is shown in Section 4. 

This section describes the test results in detail based on the investigation we have performed in the period of June-August 2021. During this period, people all over the world were seeking vaccine information more actively. Therefore, identifying the accessibility issues of the websites significantly increased during that time.

To identify the accessibility of the selected COVID-19 government vaccination websites, here, we employed a Mauve test based on WCAG 2.0 and WCAG 2.1 standards according to the POUR principles (P-Perceivable, O-Operable, U-Understandable, and R-Robust). Based on the error rate, the overall performance is calculated for conformance level AAA, as shown in Table 5 . To evaluate the performance or website quality according to accessibility testing results, we classified the website accessibility analysis result into three categories: High, Medium, and Low. The error rate (%) 2qs calculated from the total number of errors found during the test. Generally, fewer errors represent high-performance websites; a higher number of errors represents the low performance of the websites. Therefore, to evaluate the error rate (%), a significance level was taken into consideration. From our empirical observation and researcher opinion, we considered the significance level α = 0.05 and α = 0.09 to evaluate the website performance. Those websites were classified under α <= 0.05, considered as 'High' quality websites. If websites were classified under α > 0.05 to <=0.09, then we considered them to be 'Medium' quality websites. Furthermore, under α > 0.09 significance level, websites were considered to be 'Low' quality websites. Table 4 shows that, according to WCAG 2.0 guidelines, among 21 government websites from different countries (European and non-European/Asian), only 5 websites were classified under significance level (α ≤ 0.05), where four were from Europe, and one was from Asia. These five websites were High-quality websites according to WCAG 2.0 guidelines. After evaluating the results under α > 0.05 to ≤ 0.09 significance level, eight websites were Medium-quality websites, from the European and non-European countries (six from Europe, and two from Asia). Seven websites were Low-quality websites, two from Europe and five from Asia. For WCAG 2.1 guidelines, only two websites were retrieved under the significance level α and marked as High-performance websites that belong to the Europe and Asia region. However, five medium-quality websites were found, three from Europe and two from Asia; 13 Low-quality websites were from Europe (8) and Asian countries (5) .

Additionally, through the MAUVE testing tool, only 4% of the websites (1 out of 21) passed the MAUVE test without any error according to WCAG 2.0 and WCAG 2.1 guidelines. For WCAG 2.0 standards, the remaining website failed, with an average of seven errors. Moreover, for WCAG 2.1 standards, all the websites failed, with an average error of nine. However, for the WCAG 2.1 guideline, the average error of Europe and Asian countries was comparatively higher than the WCAG 2.0 guideline. In both WCAG 2.0 and WCAG 2.1 guidelines, the number of unavailable websites was 1 (WID-12).

Tables A1 and A2 show the accessibility errors of our targeted websites in detail for WCAG 2.0 and WCAG 2.1 standards. The related principles and the corresponding success criteria are shown in the tables. Generally, the MAUVE testing tool generates errors by following the WCAG criteria. Among several violations detected by the Mauve++, size and position, Internal links, CSS, ARIA landmarks, and alignment were the most frequent violations for the majority of the websites. This means that websites were larger than usual, positions were not appropriate, and internal links were not working perfectly, or were not distinguishable from the text. In addition, ARIA landmarks describe various issues, including inappropriate names, heading, etc. These issues cause difficulties for disabled users, specifically, those who use screen readers, braille, or text reader. Furthermore, Figure 9 summarizes all the accessibility errors detected by Mauve++. From this figure, it is clear that, for 21 websites, issues associated with size and position arose 2166 times. Issues with size and position implied that many websites were not focusing on their websites' length. It is the prime issue for lack of support in several systems. Surprisingly, problems associated with internal links arose 1113 times, which meant that internal links might be inappropriate, unavailable or damaged. Some other issues with ARIA landmarks appeared 506 times, alignment appeared 303 times, and link text issues arose 224 times. ARIA landmarks allow assistive technology to understand and navigate the page or content semantically. Alignment issues refer to the fact that the text or content of the website is not aligned correctly. It causes difficulties for blind people who use assistive technology. Besides, the link text issues appear for missing text in the available link. 

Nibbler is an automatic web accessibility testing tool used to evaluate the website's accessibility. Table 6 shows the Nibbler test result of the selected government websites. We evaluated the pages according to their overall score. The evaluation fell into three assessments categories: High, Medium and Low. Websites with an overall score between 10 and 9.50 are considered to be high-quality websites. If the pages had a score between 9.49 to 9.0, they were considered to be Medium-quality websites. Pages with an accessibility score between 8.99 to 0 were considered to be Low-quality websites. One website achieved a 9.70 accessibility score (WID-11.0), indicating a High-quality website. Three websites were determined to be Medium-quality websites; seventeen websites were low accessibility websites. During testing, the Nibbler tool was able to generate the test report for all the websites. Table A3 , shows the summary of accessibility errors obtained from the Nibbler tool for 21 targeted websites. Similar to Mauve++ and WAVE tools, Nibbler also provides some suggestions for associated errors. For example, on the website of Estonia, errors with incoming and internal links appeared four times, issues with popularity; amount of content appeared two times each. On the websites of France, issues with popularity, server behavior, meta tags, and amount of content appeared two times each. Problems with internal links appeared four times. These issues occur for inappropriate link placements, insufficient information, irrelevant tags, limited server ability, etc. However, the conducted experiment showed that these types of errors were common in all the website homepages. Moreover, Table 8 concludes that each country's websites had different issues that cause several problems associated with accessibility. Figure 10 shows the number of errors found by the Nibbler tool. Issues with internal links appeared 63 times, which meant that internal links did not work correctly or had some errors. Then, issues with the amount of content arose 40 times. This refers to the amount of content in the websites that was not sufficient or that was not updated. Few errors with popularity issues appeared 36 times; this indicates that the websites were not accessible for certain people. Some issues with heading, server behavior, meta tags, and URL format detected by the Nibbler tool appeared 33, 28, 24, and 15 times, respectively. Some issues with Twitter, mobile, printability, incoming links, analytics, freshness, and page title were also found, but these issues were not frequent. 

To evaluate the website accessibility and quality, we employed an automatic web accessibility testing tool named 'WAVE'. WAVE evaluates the websites based on six categories, features, errors, alerts, contrast errors, structural elements, and ARIA. After experimenting, the analysis result revealed that (Table 7) , among 21 websites, only 3 websites (WID-1.0, WID-5.0, WID-19.0) passed the accessibility test. The medium performance website had the lowest number of errors, two (WID-6.0, WID-8.0 and WID-20.0). Among 21 websites, 6 websites had fewer than the average number of errors, six. Four websites had six to eleven errors; five websites (WID-14.0, WID-13.0, WID-11.0, WID-18.0, and WID-15.0) had a total of 25, 29, 31, 47, 59 errors, respectively. Considering the contrast error result, sometimes significant contrast errors might affect content viewing and reading. The majority of the websites (71%) had contrast errors; the maximum contrast error was 100, found for WID-14.0. Some websites were free from contrast errors, which were around 28%. Table A4 reports the accessibility errors detected by the WAVE for our targeted websites. It should be highlighted that the Mauve++ test detected more errors than WAVE. However, the WAVE testing tool helped to find some specific issues. For example, WAVE provides several issues associated with links, texts, images, and languages. For instance, broken skip links, empty links, missing alternative text, linked image missing, and language missing. These error details help to understand the exact problem: 'empty link' meant that the link had no information; 'linked image missing' meant that the image was missing in the given link. Figure 11 shows the most frequent violations or errors found by the WAVE. The maximum number of the errors found for linked image missing was 60; empty link errors was 44. Linked image missing meant that the image was not present in the link; empty link error meant that the webpage did not contain text. Empty button, missing alternative text, and language missing accounted for 12, 13, and 8 errors in total, respectively. An empty button means that the button is empty or has no text value. Missing alternative text means the alternative text is not present, and language missing refers to some language barriers present, such as words, sentences, etc. Empty heading, missing form label, broken skip link, and broken ARIA reference appeared with six and five errors, respectively. Empty heading refers to a missing heading. Missing a form label meant that the form control did not have a corresponding label. A few errors appeared for linked image missing alternative text (3) and missing or uninformative page title (1). This error might cause some inaccessibility to the website user. Therefore, web developers should ensure that all the content is accessible to users and should follow WCAG guidelines. 

The evaluation results of the WEB accessibility validation tool are presented in Table 8 with both compliance score and number of violations. Previous studies have considered the compliance score for evaluating the websites. Generally, a compliance score does not always depend on a specific attribute. In some cases, websites have a high compliance score, but also have several instances of violation; sometimes, the scenario is reversed. Therefore, this research did not consider compliance scores. To evaluate the COVID-19 associated government websites, we considered the violations/errors only. The highest number of errors was for WID-7.0, with an average of 104. The lowest number of violations was for WID-4.0, WID-5.0, WID-11.0, WID-20.0, with an average violation of 1. Among 21 websites, only 2 websites (WID-2.0 and WID-12.0) passed the accessibility testing with 0 violations. The majority of the websites recorded a significant number of violations, ranging between 2 and 104.

The accessibility errors determined by WEB accessibility for the official government websites from different parts of Europe and Asia, are shown in Table A5 . For example, on the website of Estonia, 15 images did not have an alternative text. On the website of Greece, four links were not meaningful with the context. There were several other issues associated with the context, headings, and labels.

The WEB accessibility tool retrieved several violations during the testing of the COVID-19 vaccine websites. Figure 12 summarizes the entire violations of the websites. Violation with 'providing alternative text for images' appeared 172 times on 28 websites. 'Ensure link text is meaningful within context' appeared 44 times, and 'Ensure links do not directly target images' appeared 35 times. In addition, violations associated with the error 'Ensure ARIA regions, landmarks, and HTML sections are identifiable' and 'Avoid placeholder values to label or explain input' appeared between 21 to 25 times. 

WID-1.0 84% 17 F WID-2.0 92% 0 P WID-3.0 86% 6 F WID-4.0 87% 1 F WID-4.1 81% 5 F WID-5.0 88% 1 F WID-6.0 83% 5 F WID-7.0 83% 104 F WID-8.0 85% 3 F WID-9.0 86% 9 F WID-10.0 86% 18 F WID-11.0 90% 1 F WID-12.0 92% 0 P WID-13.0 75% 8 F WID-14.0 78% 26 F WID-15.0 80% 23 F WID-16.0 77% 11 F WID-17.0 78% 24 F WID-18.0 76% 66 F WID-19.0 85% 2 F WID-20.0 90% 1 F

To calculate the SUS score [38] of the targeted websites from the questionnaire-based user feedback, first, we calculated the score of the odd number of questions, Qscore(Odd) = Xn − 1; where n = 1, 3, 5, 7, 9 and then we computed the score of the even number of questions, Qscore(Even) = 5 − Xn; where n = 2, 4, 6, 8, 10. Then, we calculated the SUS score through Equation (1).

SUS score = ∑ n=1,3,5,7,9 (Xn − 1) * 2.5 + ∑ n=2,4,6,8,10 (5 − Xn) * 2.5 (1) To determine the quality of the websites, we set six quality parameters based on the SUS score, shown in Table 10 . Based on the quality parameter, we evaluated the websites. Table 9 shows the SUS score of each website with evaluation remarks. After evaluating the websites through the SUS score, only 14% of websites were classified as 'Excellent' websites (WID-12, WID-15, and WID-19); 9% of websites (WID-2.0, WID-3) were found to be the 'Best' quality websites. Furthermore, only 28% of websites (WID-1, WID-5, WID-6, WID-7, WID-14, WID-20) were 'Good' quality websites. These websites were accessible to people who have some disabilities. However, the ratio of these accessible websites' was not sufficient to ensure that all the websites were accessible. Besides, the rest of the websites had no adequate functionality or accessibility; website operators must take websites' accessibility into consideration. For instance, fair, poor, and worst websites' ratio was 19%, 19%, and 9%, respectively. This result suggests that a significant number of the website's accessibility issues should be under consideration for further improvement.

The expert investigation results are displayed in detail in Table 11 . They show that the majority of the websites were not accessible for people who had vision problems or moving disabilities. On average, 70% of websites were not accessible for people who had a low vision concern. Some websites had keyboarded/TAB key accessibility, but not for all the buttons or sub menus, making the website inaccessible. Few of the websites had a color deficiency; it was hard to separate the link and buttons. Surprisingly, none of the websites had manual font size function, which makes the websites difficult for people who have low vision concerns. Additionally, only a few websites were available in English; a few were not responsive. With the automatic tools, we were unable to detect these issues that sometimes fail to produce the appropriate research output. Thus, human observation is effective to identify the unreported problems of automatic tools. 

Ensuring adequate access to online content is a primary objective in the field of information technology. Sik Lanyi et al. have also added that, so far, the majority of websites are not accessible for people with disabilities [3] . Therefore, in this study, we tested 21 official government COVID-19 vaccine information portals from various countries in Europe and Asia. This study aimed to understand the current insights of web accessibility and to identify the frequent violations. In this global pandemic, official government websites were valuable, reliable, and trusted sources for COVID-19 vaccine information for people, including those with disabilities. Thus, compliance checking of these COVID-19 vaccine information portals against the WCAG standards is crucial to make the web accessible and understandable for both people with special needs and non-disabled users. Therefore, we used four automatic accessibility checking tools and human observation techniques to perform the testing process. The automatic testing result revealed that the majority of the websites had accessibility issues. Additionally, human observation suggested some improvements to make the websites accessible.

According to our empirical observation (shown in Table 12 ), Asian countries' websites had a higher number of errors on average compared to websites operated by European countries. From our observation, the cause for higher numbers of errors in Asian countries is because most of the Asian universities lack educational materials for digital accessibility. Therefore, graduate students were unaware of accessible development, its importance, and related consequences. In contrast, European countries have courses regarding digital accessibility that help the student to understand accessibility and implement it at the very beginning of their carrier. Therefore, we rejected the first part of the H1, RQ1. Furthermore, Figure 13 shows the ratio of errors for each country for four automatic testing tools. Through Mauve testing, for 2.0 standards, the maximum number of errors was 14, found on the website of the Philippines. For 2.1 standards, the maximum number of errors was 17, found on the website of Croatia. In the WAVE testing tool, the highest number of errors was 69, found for Jordan. Following Jordan, Maldives had 47 errors, which is the second-highest number of errors. If we look at the results of Nibbler, the maximum error rate was 26, found on the website of India. We found 66 errors on the Maldives website, which is the highest among others (WEB accessibility). Therefore, it is obvious that, in the accessibility results, there were some differences across the regions, continents or countries; we accepted the last part of H1, RQ1. This result also revealed that web developers from Asia should be more aware of accessibility criteria. Table 13 shows that the average SUS score of European and Asian countries' websites were 66.92 and 56.25, respectively. Furthermore, based on the expert questionnaires (shown in Table 4 ), the total average score of European and Asian countries was 0.68 and 0.71, respectively, as shown in Table 14 . Although the average number of SUS score, and the total number of errors, were higher on the Asian websites, no significant difference was found. Additionally, there was no significant difference in the score of each question. Therefore, we accepted the H2, RQ2.

According to the automatic evaluation results, few websites were accessible. However, after going through questionnaire-based human judgment (system usability and expert testing), we found some inaccessible functionalities in these websites. Regarding the SUS testing, we could determine the quality of each website (Table 10) , which is not possible to determine through any automatic testing tool. Through the automatic tool, it was only possible to identify whether the website had passed or failed the accessibility testing. In most cases, the existing automatic tools had limitations for incorporating all the WCAG guidelines. Therefore, a few accessibility issues were not possible to detect by the automated tools that directed future research. However, regarding the expert testing, neither of the websites had manual font resize functionality, which could be a problem for people with poor vision. Some websites had no accessibility option when using the TAB key. An excellent example could be WID-18, having no movement function with the TAB key. This website was not accessible for people who have moving disabilities. Some websites were not in English, and there were no translation options, e.g., WID-9 had no English version or translation options. Therefore, sometimes human judgment helps to find unexplored issues that are not possible through the automatic tools. Thus, there were a few differences between automatic and human judgment results; we accepted H3, RQ3. Furthermore, Table 15 shows the error types, associated principles, and their number of occurrences. Among the four principles, the highest number of errors was for the first principle. Therefore, we accepted H4, RQ4. All the websites failed to add 'size and position', 'internal links', 'linked image missing', and 'alternative text' for images. For instance, the website of India had 1219 issues with size and position and 594 issues with internal links. On the website of the Maldives, 35 links had no images or the images were missing. The website of the Netherlands had 103 images without alternative text, which is higher than other websites (a result of WEB accessibility). Additionally, all the web pages had accessibility issues, such as no added text for links to describe the target links. A proper description of the links could help the people who are using a screen-reader or text browser. The webpage of Germany had 15 empty links, which is higher than any other webpages.

Based on our measurements, we have found that more testing tools need to check the accessibility of vaccination-related websites and even need to be supplemented with expert testing. So, our thesis group is:

Thesis group 1 (T1). There are differences between European and Asian counties' websites accessibility according to their average number of errors. Moreover, there is a difference when focusing on the website of each country.

There is no significant difference between websites from the European and Asian, considering questioner based human judgment results.

There is a significant difference in the result of automatic testing and questioner based human judgment.

Thesis group 4 (T4). The first principle of WCAG 2.0 and WCAG 2.1 is the most frequent violation principle, considering four automatic testing tools. 

Past research revealed that individuals with disabilities have different accessibility issues. Thus, the current internet platform or web-based portals were not completely accessible for people with disabilities, in some cases, not even for non-disabled users [23] . Therefore, this research performed evaluation on the COVID-19 vaccine information websites to investigate their accessibility status. The consequences of the inaccessibility of the COVID-19 vaccine information portals might hamper both social and economic life. For instance, COVID-19 brings restrictions on the border. Without proper vaccination, it's impossible to get treatment, travel, etc. Some countries' vaccination regulations and acceptance were different from others. Thus, without proper vaccine information, it was not possible to serve consistent services for education, business, etc. To access the vaccine information, people frequently used the official COVID-19 vaccine information portals. Therefore, investigation and identification of accessibility issues of COVID-19 vaccine information portals is an open issue for the web researcher.

The automatic testing and human observation results revealed that a significant number of the COVID-19 information websites were not free from accessibility issues. The prime reason is that a majority of the websites did not follow the compliance standards of web content accessibility guidelines. It also indicated that countries were did not consider accessibility issues during the designing and development phase. Frequently, web developers ignore accessibility guidelines for website development. Therefore, our present research emphasized that people with disabilities should be taken into consideration during website development, as they take part in the development of the economy. After evaluating 21 websites from different parts of Europe and Asia, several accessibility problems were identified in this study through four automatic tools and human judgment. These indicated that all the accessibility issues could be easily fixed. Based on the accessibility testing report generated by Mauve++, Nibbler, WAVE, and WEB accessibility, we encourage designers and developers to follow the WCAG standards issued by the W3C. Some of the frequent accessibility violation guidelines are addressed during the testing, which requires additional attention by web developers during development. Guideline 1.4. Distinguishable: This guideline is under the perceivable principle and was the most frequently violated guideline reported by Mauve++ during the COVID-19 vaccine information websites investigation. The Mauve++ automatic accessibility testing tool generated this report and revealed that 135 issues arose under this guideline during the website's investigation. The report using the WCAG 2.0 and WCAG 2.1 version guidelines indicated that the majority of the websites had no clear visual representation or had issues with color, audio quality, text size, contrast ratio, images of texts, and non-text contrast. More attention is needed to improve the visual representation and audio quality to allow consistent services for screen reading application users and to allow modification based on individual needs. Being unable to resolve these issues causes a barrier for people with visual impairment in accessing the information properly.

Guideline 3.2. Predictable: The predictable guideline is under the understandable principle at conformance levels A, AA, and AAA. It was the most violated guideline detected by Nibble automatic testing tool. Nibbler identified 29 issues from this particular guideline, which was the maximum in investigated COVID-19 vaccine information web-sites. Following this guideline, it is possible to design proper user interface components and to ensure equal service for people with disabilities and non-disabled users.

Guideline 2.4. Navigable: Navigability is an important guideline for accessibility under the operable principle at conformance levels A, AA, and AAA. It was the most frequently violated guideline, with 30 issues identified by WAVE during the investigation of COVID-19 vaccine information websites. This guideline aims to ensure consistent navigation to find content and to determine its exact location on the website. The report raised issues with the page title, heading, labels, and link problems. These issues indicated that a majority of the websites lack concern about accessibility guidelines, which would ensure accessible content to people with disabilities. Improper title and heading labels cause inaccessibility and impede the identification of the exact information. These issues also cause a barrier to navigating the websites properly and increase dissatisfaction for people with special needs or who require assistive technology. Guideline 1.1. Text alternatives: This guideline is under the perceivable principle and the frequently violated guideline at conformance level A. The WEB accessibility testing tool detected 17 issues associated with this guideline on 21 official government COVID-19 vaccine information portals. These findings indicate that the majority of the websites had issues with incorporating text alternatives for non-text content. By ensuring text alternatives for non-text content, it is possible to help people with visual challenges. For instance, without text alternatives of images, people with visual impairment would not be able to understand the content of the image. Therefore, it helps visually impaired people and users who use braille or other assistive technology for accessing information. It also helps people with cognitive disabilities who have difficulty understanding the images and who need simpler language.

Several issues are responsible for these accessibility errors. Generally, due to costeffectiveness, developers ignore accessibility testing (automatic, human testing) before publishing the website for the user. Frequently, they use redundant templates instead of considering updated development tools. In addition, most of the developers prefer to choose the most common development tools (HTML, PHP) and do not consider advanced tools like JavaScript, React.js, etc. Lack of monitoring systems also increases the number of inaccessible websites.

Moreover, the findings suggest that the evaluated websites had limited focus on the usage of color, contrast, audio component, text size, resizing, spacing, text alternatives, image alternatives, page title, heading, levels, link purpose, user interface, and visual components. Although these issues were frequent, some additional issues also needed to be considered, such as adaptability, readability, and compatibility. Similar to [21] , we want to emphasize that, with automated accessibility testing, user testing is highly beneficial, as it allows for involving people with special needs. They can provide the actual feedback regarding their capabilities. Additionally, assistive technology should be incorporated during testing.

From our observations, the investigation results imply that authorities should pay more attention to accessibility issues during the design and development phase. More accessibility evaluation processes should be carried out in the future, as websites are updated constantly. Therefore, frequent evaluation will help to identify and solve the emerging issues. In addition, further training should be organized for web designers, web developers, and other associated authorities to increase the awareness and importance of accessibility to make internet platforms accessible.

This section presents some suggestions to improve the accessibility of web pages. All the suggestions are regarding automatic web accessibility testing, system usability, and expert testing results. To develop accessible web platforms, the following suggestions should be incorporated during the development phase.

Image Alternatives: Image alternatives is the process of representing text information through non-textual content (images, videos, animations, etc.). By ensuring alternative representation techniques, it would be barrier-free for users with disabilities.

Text Alternatives: Text alternatives is the process of representing the non-textual (images, videos, animations, etc.) information through some textual information or description. Lack of a text alternative's function makes the websites difficult to understand for users with disabilities, especially for those who use screen readers to consume the information.

Text-size and Color Contrast: Color contrast is one of the most important accessibility criteria for making websites or online content accessible to people with disabilities, as different people have their demands. Several websites use text-size and color contrast adjustment functionalities, but the majority of the websites have no focus on this specific requirement. Thus, website administrators and developers should use these functions.

Multilingual accessibility: For web page development, each web page should have its native language version and the English translated version. English translation helps people speaking different languages to access the website. During translation from one language to another, precise translation must be of great emphasis to reduce the barrier of accessibility.

Design Accessibility: Websites have various statistical and graphical representations; an unorganized design structure could result in an inaccessible website. Graphical representation and website organization should be clear and easy to understand.

Proper title and subtitle: Documents or pages should have an appropriate title and subtitle (if required based on demand) to navigate the page appropriately for users who rely on a screen reader.

Links or hyperlinks placement: In the document or content, links or hyperlinks should be placed correctly so that people with disabilities can easily understand them. Links should be placed with proper attributes. Thus, people could understand and distinguish between links and the main content.

Content availability and Readability: Appropriate and updated content should be on the website to attract the target audience. Additionally, the readability of the content should be ensured.

Redundant templates: Sometimes, web developers consider the same template for multiple website development. This introduces barriers and increases the chances of frequently misusing several functions. Therefore, using the same template may prove to be a poor decision; developers should adapt themselves with new templates and updated technology.

ARIA (Accessible Rich Internet Applications) attributes: Around 50% of the websites did not use the ARIA function. To increase accessibility, web developers should use ARIA attributes in the HTML script.

This paper presents an accessibility analysis of official government COVID-19 vaccine information websites for different countries of Europe and Asia. To assess the accessibility of the target websites, we employed four open access automatic web accessibility tools and human observation techniques. The evaluation results and the human judgment indicated that several accessibility issues existed in those targeted websites. Through both automatic evaluation and human observation, we identified the most frequent problems of the websites. We compared European and Asian countries' results and found few differences in their accessibility. Based on the evaluation, this paper added some suggestions to improve the accessibility of these targeted websites. Through the findings of this paper, we wish to emphasize that, if web developers provide their services or develop the websites without following accessibility guidelines, our research on its own will not be enough to provide an accessible solution. Another remark from the evaluation is that most websites were not tested by people with visual challenges. Besides, due to cost efficiency, developers often reuse the design/development template instead of using an updated version. Therefore, we think that the findings of this study might help web developers and web designers to understand the drawbacks of their developments. In total, we considered only 21 websites from 13 countries. Based on the results of this research, we will incorporate websites from other countries from these two continents and other automated and human testing systems to generate a more systematic result. We will consider all the economic and social issues in our future research (population, GDP, etc.) to perform a depth analysis of these two continents' results regarding web accessibility. Other countries from other continents should be considered and included in similar research in the future. Additionally, this research and the experiments were performed based on the WCAG 2.0 and WCAG 2.1 guidelines. In the future, we will continue our research based on WCAG 2.2 and WCAG 3.0 guidelines. Acknowledgments: We would like to thank 'John Dermott Alexander Withers' for his support in language correction.

The authors declare no conflict of interest.

The following abbreviations are used in this manuscript: 

Development history of the world wide web

Improving accessibility of CMS-based websites using automated methods

Accessibility Testing of European Health-Related Websites. Arab

A practitioner's approach to assess the wcag 2.0 website accessibility challenges

Evaluating the accessibility of higher education institution websites in the State of Kuwait: Empirical evidence

A National Accessibility Audit of Primary Health Care Facilities in Brazil-Are People with Disabilities Being Denied Their Right to Health?

Website Usability and Content Accessibility of the Top USA Universities

Tools for web accessibility evaluation

Web Accessibility Initiative (WAI)

Comparing six free accessibility evaluation tools

Website Accessibility Evaluation: Case Study in Portugal

Accessible Websites for Everyone-A Case of UAE Universities Websites

Real-time, interactive website for US-county-level COVID-19 event risk assessment

Accessibility Evaluation of COVID-19 Vaccine Registration Websites across the United States

WAccess-A Web Accessibility Tool based on the latest WCAG 2.2 guidelines. arXiv 2021

WADcher: A Unified Web Accessibility Assessment Framework

A holistic decision support environment for web accessibility

Monitoring Systems for Checking Websites on

Students with disabilities as partners: A case study on user testing an accessibility website

Towards automated web accessibility evaluation: A comparative study

Academic Medical Libraries and Accessibility Challenges: The Conformance of the Websites with the WCAG2 1

The Quality of Infectious Disease Hospital Websites in Poland in Light of the COVID-19 Pandemic

Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: Proposed frameworks

Evaluating web accessibility of educational institutions websites using a variable magnitude approach

Accessibility evaluation of Arabic e-commerce web sites using automated tools

Multi-tool accessibility assessment of government department websites: A case-study with JKGAD

A systematic literature review to determine the web accessibility issues in Saudi Arabian university and government websites for disable people