Community paper

Including people with psychosocial disability in statistics: Self-report measures for surveys

Marguerite Schneider, Elena De Palma

Received: 15 Feb. 2025; Accepted: 30 Aug. 2025; Published: 30 Nov. 2025

Copyright: © 2025. The Authors. Licensee: AOSIS.
This work is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license (https://creativecommons.org/licenses/by/4.0/).

A tribute to Prof. Leslie Swartz

A person who makes as significant an impact on a field of work as Prof. Leslie Swartz has in disability studies, does so because they are able to see the breadth of a field (or forest) and not focus on only the trees. What Prof. Swartz understood from the start is the importance of measurement in disability and the need for questions that measure what we think they are measuring and hence give us valid and reliable data that can be used for purposes of monitoring inclusion of disability. Through this understanding, Prof. Swartz has provided me (M.S.) with significant support giving me the confidence to pursue it as a relevant area of work. This support was in particular evident when he reviewed my doctoral work as an informal reviewer and helped me clarify the context of what I was writing about in the field of disability measurement. He has provided ongoing opportunities for sharing my knowledge, networking with others and recognition both within and beyond my area of expertise in disability measurement. It has helped me, in turn, contribute as much as I can to the field of disability studies. So, while Prof. Swartz has not directly contributed to this work on disability measurement, he has, in many ways, contributed to it happening.

In writing this paper, I (M.S.) am joined by my Washington Group on Disability Statistics colleague, (E.D.P), my partner in conducting this work.

Introduction

Background

Over the last two decades, there has been significant improvements in the quantity and quality of data on disability globally. The ratification of the United Nations Convention on the Rights of Persons with Disabilities (UNCRPD) (UN 2007) by 192 countries, as of 2025, has reinforced the importance of monitoring the equalisation of opportunities and reduction in inequalities between people with and without disabilities. This has exponentially increased the demand for disability data. More recently, people with psychosocial disability (PSD) have raised their voice in relation to their rights resulting in a growing request for measures that allow for monitoring of their rights. This article addresses how such measures are being developed using the Washington Group on Disability Statistics (WG) methodology and presents some of the early indications on the performance of these measures. This ongoing work has been presented at the WG annual meetings by the two authors. Some of the material will be the same as presented at these meetings, the proceedings of which can be found at https://www.washingtongroup-disability.com/about/past-wg-annual-meetings/

Having a mental health condition (MHC) is generally associated with stigma and negative attitudes leading to discrimination and negative treatment of people with PSD (Crabb et al. 2012; Drake et al. 2011; Eaton et al. 2021). People with MHCs face discrimination even in health care settings, receiving poor treatment by non-psychiatric professionals, and therefore limited access to general health care (De Hert et al. 2011; Isiko, Singhani & Urkmez 2024; McIntyre et al. 2024). Their exclusion is associated with low visibility and a lack of effort to meet their needs that can promote their inclusion. Monitoring trends in outcomes in the areas of social, educational, economic and political life of people with PSD requires accurate data collected using valid measures that are comparable across countries. Such data collected and reported on at regular intervals on national and global platforms, such as the Sustainable Development Goals (SDGs), is one way to increase visibility of people with PSD and ensure visible monitoring of their inclusion.

The Washington Group on Disability Statistics

Two major issues in disability statistics identified in the 1990s and early 2000s were the high variation in disability prevalence across countries and a lack of uniformity in the measurement of disability (United Nations Statistical Division [UNSD] 2001). The work conducted by the WG (see http://www.washingtongroup-disability.com/) since 2002 has greatly improved international data comparability with a change of focus from ‘disability’ to ‘difficulties in functioning’ that people may have.

The WG focuses on basic activities or actions as defined by the World Health Organization’s (WHO) International Classification of Functioning, Disability and Health (ICF) (WHO 2001) in order to ensure cross-nationally comparable data. The questions measure functional difficulties on basic actions that are appropriate to the reference age and not influenced by culture and contextual factors (Madans & Loeb 2013; Madans, Loeb & Altman 2011). This approach to disability measurement was recognised as appropriate by WHO and the World Bank in the World Report on Disability (WHO 2011).

The WG is an international city group established by the UNSD to address key priorities and recommendations that arose from the UN International Seminar on the Measurement of Disability in June 2001 (UNSD 2001). It is a voluntary working group made up primarily of representatives of over 130 National Statistical Offices and including international, non-governmental and disability organisations.

Since 2002, the WG has developed, tested and adopted several question sets, in line with the ICF (WHO 2001) for the collection of internationally comparable disability statistics. Disability data are collected for different purposes and each purpose requires a different data collection approach. The WG chose to develop measures that would allow an assessment of equalisation of opportunities for people with and without disability (Madans & Loeb 2013). Participation restrictions are a key factor in limiting a person with disability in accessing equal opportunities compared to their non-disabled peers. The aim of the measures is to ensure that people who are at risk of experiencing participation restrictions (i.e., respond as having difficulties on the measures) are counted in as ‘with disability’ in national disability statistics. The risk of experiencing participation restrictions is associated with reporting difficulties in the functional domains, and appropriate accommodations are not made. The UNSD has recommended the WG measures, specifically the Short Set (WG SS), as the instrument of choice for measuring disability for disaggregation by, and reporting on, disability for the SDGs (United Nations 2017). In addition, a continuum of functioning is reflected in the response options used: ‘no difficulty’, ‘some difficulty’, ‘a lot of difficulty’ or ‘cannot do at all’. This allows for different cut-off points to be applied for different purposes. The UN recommends that the cut-off for ‘with disability’ is reporting ‘a lot of difficulty’ or ‘cannot do at all’ on at least one question on the WG SS.

A notable feature of the WG approach to measurement and of the WHO’s ICF is the focus on functioning profiles and not diagnosis. Functioning is neutral as to diagnosis as the same functioning profile can arise from different medical diagnoses, and the same diagnosis can result in different profiles. Functioning profiles can also shift the focus to include not only people’s deficits or difficulties, but to also record their abilities or strengths. A diagnosis does not provide information on a person’s abilities and difficulties. There is a continuum from a full-blown MHC that is clinically diagnosable through to full mental wellbeing with many people experiencing mild or transient mental health symptoms that are not severe enough to warrant a diagnosis. A functioning profile approach reflects this as people without a MHC do report mild difficulties in domains of functioning associated with PSD but do not have a clinically diagnosable condition.

Measuring disability in self-report surveys

The measurement of disability in self-report surveys has become a well-established practice with a growing number of countries using the WG SS and WG Extended Set (WG ES) of questions for censuses and surveys, respectively. The domains of functioning measured with the WG SS include vision, hearing, mobility, cognition, self-care and communication. No measure of psychosocial functioning was included as it was not deemed feasible to develop a suitable question that encompasses the multiple aspects such as thoughts, feelings and behaviours that can be universally understood. To address this gap, questions on affect, specifically depression and anxiety, were included in the WG ES on Functioning tool developed for use in surveys where more questions can be included. The questions ask about feelings of anxiety and depression, which are common MHCs. The rationale is that people experiencing intense and frequent feelings of depression and anxiety are at higher risk of experiencing participation restrictions. The affect questions together with the WG SS may already ‘count in’ a number of people with PSD who report difficulties in communication, mobility, cognition, self-care and feelings of anxiety and depression; but many people with PSD would not report such difficulties and remain excluded from the statistics and, hence, the monitoring of their inclusion. Hence, further questions that address specific aspects of psychosocial functioning are required.

To allow for disaggregation of statistics by disability status for all people at risk of participation restrictions, people with PSD must be effectively included in statistics identifying the group at risk – that is ‘with disability’. Such inclusion will allow further comparative analysis of the living conditions, education and employment status of people with psychosocial disabilities relative to people with physical or sensory disability and non-disabled people (without disability). The WG’s Psychosocial Disability and Mental Health Workgroup has been working to ensure that relevant measures are developed to improve inclusion of people with psychosocial disability, who are 18 years and older, in data collection. The age cut-off is appropriate as there are questions addressing psychosocial functioning between the ages of 5 years and 17 years in the WG and UNICEF Child Functioning Module (Crialesi, De Palma & Battisti 2016; Loeb et al. 2018).

Aims

This article aims to, firstly, highlight the importance of ensuring inclusion of people with PSD in national and global statistics to increase their visibility and ensure that there is monitoring of their level of inclusion. Secondly, the article describes the process of developing the psychosocial functioning questions to show the feasibility of ensuring the inclusion of people with PSD. Measures should be valid, accurate and transparent, as well as internationally comparable.

The four cognitive interviewing studies, their methodologies and results are not presented in detail here. The focus of this article is on the process of developing a question set. Where relevant, some emerging (but not confirmed) trends are provided to explain the process and show what type of results can be obtained.

Research methods: The questions’ development and testing process

The first step in developing measures of psychosocial functioning is to identify the most common functional difficulties (or activity limitations as defined in the ICF) that characterise this type of disability. The second step is to identify questions that measure these functional difficulties, followed by a cycle of testing–adaptation–testing of these questions to establish whether the questions perform as intended in population-based data collection platforms. The question testing methodology used by the WG in developing all question sets includes a series of cognitive interviewing tests and field testing. The cognitive interviewing testing is done in as many languages (with translations) and contexts as possible with ongoing revision of the questions as more data are collected on how they are performing. Once questions have been well-tested using cognitive interviewing methodology, a series of field tests are conducted where the questions are asked in a standard self-report survey mode and the data are analysed to confirm if the questions perform adequately. Finally, the questions are adopted and circulated for use by statistical offices globally.

The question development and testing process included the following steps: (1) a review of the literature to determine the most common activity or functional limitations associated with psychosocial disability and which are already covered by the existing WG questions; (2) developing or identifying existing questions measuring the outstanding domains of functioning and adapting them to the WG format; (3) conducting a series of cognitive interviewing studies followed by field tests (not discussed in this article) with iterative revisions and testing of the questions; and (4) adoption of the questions. Each step is described next.

Identifying the most common functional difficulties associated with psychosocial disability

In reviewing the literature, the focus was on identifying the activity limitations commonly associated with mental health conditions that cut across as many different mental health conditions as possible. The disorders considered included anxiety, depression, post-traumatic stress disorder, schizophrenia, schizoaffective disorder, bipolar disorder and major depressive disorder.

In searching the databases, the following search terms were used: ((‘mental disorder’ OR ‘Psychiatric Illness’) AND (disability OR ‘activity limitation’)) with a date limit of 1999–2018. The databases searched included PubMed Central (PMC) and Web of Science Core collection. The aim was not to be exhaustive but rather to reach saturation where no additional domains of psychosocial difficulties were identified with new papers.

A range of studies were included which (1) identified clear domains associated with psychosocial disability (and not comorbidities) in persons with a MHC from age 18 years and older; (2) represented as wide a range of global regions as possible; and (3) were in a language that the authors could read. Studies were excluded if they (1) described the functional status of people with dementia, intellectual disability, alcohol and drug abuse or neurological disorders; (2) were about validation of existing instruments; and (3) had limited or no description of the functional limitations.

The literature review identified the following domains of basic activities as being common areas of difficulty for people with psychosocial disability:

• Self-care: for example, washing self, carrying out daily routine of self-care.
• Communication: for example, verbal fluency.
• Memory and thinking: for example, executive function, attention difficulties, verbal and visual memory.
• Mobility: for example, difficulty walking several blocks, climbing one flight of stairs, transferring oneself.
• Social interaction and relationships: for example, difficulty in forming and maintaining relationships, poor communication with family members and neighbours.
• Controlling behaviour and/or emotions: for example, getting upset, conflict with others, misinterpretations, getting violent.

Of these, the domains of (1) social interaction and relationships; and (2) controlling behaviour and/or emotions were the only ones not included in the WG SS or WG ES. Hence, they were selected as the domains to focus on in identifying suitable questions. A total of seven questions were identified and adapted to fit the format of WG questions – ‘do you have difficulty … ?’ and using the four response options of ‘no difficulty’, ‘some difficulty’, ‘a lot of difficulty’ or ‘cannot do at all’.

Cognitive interviewing methodology

The cognitive interviewing methodology is used to assess how well survey questions perform when administered to respondents; that is, whether respondents understand the questions according to the intended meaning when developed, and can provide accurate answers based on that intent (Miller et al. 2014). This method is about question performance testing. Miller et al. (2014) describe this method as useful to evaluate the cross-cultural equivalence of questions and detect any errors that occurred during the translation process from the original language.

The method is qualitative using in-depth interviews during which the respondent is asked to either ‘think aloud’ (respondent explains what he or she is thinking while answering the question) or respond to ‘probes’ (respondent answers a series of probing questions posed by the interviewer after answering to the question being tested). Purposive sampling is used, ensuring a good representation of the target population.

The method is a form of construct validation as it aims to see if the construct being measured is being interpreted as intended by researchers. If it is interpreted ‘in-scope’, the question is deemed to be performing in a valid manner. If respondents give ‘out-of-scope’ interpretations, the question is deemed faulty and revisions are made before testing, or the question is not used further.

The objectives of cognitive interviewing testing studies (or question performance testing) are:

• To assess participants’ interpretation of the questions.
• To identify potential response problems that could impact data quality.
• To evaluate the cross-cultural equivalence of the questions.
• To conclude on the performance of the selected questions and their suitability in ‘counting in’ people with psychosocial disability as having disability.

Developing and testing the psychosocial functioning measures

When conducting cognitive testing studies, the sample is selected in a purposive manner to ensure a representation of different ages and educational background, inclusion of both men and women, and an equal mix of people diagnosed with MHC and those without. The sample size required is between 20 respondents and 40 respondents. All respondents sign consent forms prior to the start of interview process. Interviews are conducted primarily in a face-to-face mode, but virtual mode is possible depending on the circumstances. All respondents with an MHC must be in a phase of remission with at most mild psychological symptoms to be able to provide coherent narratives for the analysis.

The cognitive interviewing studies conducted to test the psychosocial functioning questions are structured as follows:

• The respondent is asked the question and gives their response.
• The interviewer asks the respondent to explain how they arrived at the answer and to provide examples to support their explanation. General (e.g., tell me more) and question-specific probes (e.g., what does ‘getting along with’ mean to you) are used to elicit as detailed narratives as possible.
• The interview is either recorded or detailed notes are taken with as many verbatim quotes as possible.
• The process is repeated for each question.

As of end of 2024, four cognitive interviewing studies have been completed in: South Africa (English), Costa Rica (Spanish), Hungary (Hungarian) and Kenya (English and Kiswahili). A total of 106 respondents participated, of which 62 had a diagnosed MHC and 44 did not.

Analysis

The analysis conducted includes identifying (1) the interpretations elicited; (2) any difficulties respondents had in answering the question or using the response options; and (3) determining profiles of responses for people with psychosocial disability compared to those for people without psychosocial disability.

In- and out-of-scope interpretations

The narratives are summarised and a thematic analysis was conducted to identify the interpretations by the respondents. The interpretations are categorised into ‘in-scope’ or ‘out-of-scope’ ones. After the initial cognitive interviewing study, some key themes were identified and used in subsequent studies. Additional emerging themes from the subsequent studies were added to the existing themes. We compared the interpretations provided by people with a MHC to those of people without such conditions.

Difficulties with the questions

Interviewers answered three questions for each question being tested. These were (1) if the respondent requested repetition of the question or response options; (2) if they had difficulty using the response options; or (3) changed or qualified their responses after providing their narrative. The number of respondents experiencing each of these difficulties were tabulated for each question.

Profile of responses

The responses to the psychosocial functioning questions were tabulated for all the tested questions and compared across the two groups – those with and without a MHC – to determine if there were clear differences in the response profiles. This also showed how many respondents were identified correctly by one or more of the seven questions using the cut-off described earlier. Responses of ‘a lot of difficulty’ or ‘cannot do at all’ on one or more of the seven questions identified the respondent as ‘with disability’. Responses of ‘no difficulty’ or ‘some difficulty’ identified respondents as ‘without disability’.

Results

The question set

Seven questions (see Box 1) were identified from a number of existing question sets. The WHO Disability Assessment Schedule (WHODAS) provided the majority of questions (Üstün 2010). These were adapted to fit the WG format using the four response options shown precedingly. This initial set includes some overlapping questions to allow testing of different wordings of questions, before selecting a minimum set of questions to be used.

Interpretations of the questions

The majority of interpretations by respondents were in-scope. Occasional out-of-scope interpretations were noted across all four countries, while some misinterpretation such as narrow or broader interpretations were noted. The out-of-scope interpretations will be tested in future cognitive interviewing studies to determine if they are idiosyncratic or real trends that need to be addressed in the wording of questions.

The questions about ‘making new friends’, ‘maintaining friendships’ and ‘controlling behaviour’ or ‘emotions’ highlighted some difficulty on the respondents’ part. For example, a couple of respondents in answering about ‘forming relationships’ considered only romantic relationships or had trouble in deciding who should they consider as friends, or considered ‘controlling behaviour’ in relation to uncontrollable behaviour (e.g., nervous tics). Table 1 presents some of the interpretations identified for each question. In general, there was strong overlap in interpretations across the four countries.

In general, respondents with a MHC showed the following:

• Difficulties because of anxiety, how people react to them and how they react to others.
• Struggled with reflecting on the questions and their responses.
• Had more negative narratives than respondents without a MHC (expressed many difficulties).
• Reported particular difficulty with controlling emotions and behaviour.
• Reported increased difficulties when experiencing severe symptoms of their MHC.

When respondents with a MHC were asked if the questions were sensitive or intrusive, the overriding response was that the questions were sensitive but not intrusive. Some respondents found the questions on emotions and behaviour interesting and useful to reflect on. Overall, the respondents felt that these questions were important topics to ask about.

Difficulties with questions

A small number of respondents requested for the questions to be repeated, or for the answer categories to be repeated. Four respondents with mild intellectual disability requested repetition for more than one question. There were only a few instances where respondents had difficulty in selecting the answer categories. Few clarifications were requested.

Identification of people with and without mental health conditions

Of the 62 respondents with a MHC condition, 32 were identified by only one question, followed by 8 respondents identified by two questions. None of the respondents with a MHC reported ‘a lot of difficulty’ or ‘cannot do’ on all psychosocial questions. The questions on ‘controlling emotions’, ‘dealing with people you do not know’ and ‘getting along with people close to you’ were the most likely to identify people with a MHC as being ‘with disability’. In contrast, only 6 out of 44 respondents without a MHC were identified as ‘with disability’ by one or more questions. Only one reported ‘a lot of difficulty’ or ‘cannot do’ on three questions.

This also reflects that people with a MHC were more likely to report more severe difficulties than those without (i.e., reporting ‘a lot of difficulty’ or ‘cannot do at all’), although both groups reported difficulties. This possibly reflects the continuum of mental health in any population.

Revisions of questions for subsequent testing

Part of the testing process is to review the questions iteratively and revise according to the findings from each testing. Examples of revisions are provided where clear or consistent problems identified with the wording or structure of the questions led to revisions of these questions.

There was a consistent link made between controlling emotions and controlling behaviour in all four studies by respondents with and without a MHC. The decision was made to merge these two questions into one resulting in the following question which will be used in the next rounds of testing: Do you have difficulty in controlling your emotions or behaviour when you are around other people?

In one country in particular, respondents struggled to interpret the notion of ‘new’ friends. The decision was made to revise the question for the next round of testing and potentially take it out completely after that testing: Do you have difficulty making friends?

The phrase ‘other people’ in the question on ‘forming relationships with other people’ raised issues of interpretation as respondents were not sure who ‘other people’ was referring to, especially as preceding questions asked about ‘people you do not know’. The decision was made to delete ‘with other people’ for the next round of testing: Do you have difficulty forming relationships?

Discussion

This article aimed to highlight the importance of monitoring the level of inclusion of people with PSD, and to increase their visibility in national and global statistics, for example in disaggregated data from development programmes or quality of life surveys. The good performance of the seven questions so far shows promise for achieving this statistical inclusion and visibility.

The article further aimed to show the feasibility of developing measures that are valid, accurate, transparent and internationally comparable. This was demonstrated by describing a relevant methodology that produced results showing similar and good performance of the seven questions across four countries. The results to date suggest that the seven questions meet the criteria of construct validity, accuracy, transparency and international comparability. Further cognitive interviewing and field testing are required to confirm these findings and/or further refine the questions and select a minimum set.

Disability represents a multidimensional and complex construct to measure in self-report surveys. It needs to be disentangled to measure each component correctly. The WG measurement approach focuses on difficulty in doing basic actions to identify people at risk of participation restriction in an unaccommodated environment. Once this at-risk group is identified, the data can be compared between those with and without disability across a range of variables, such as employment, school attendance and social inclusion, to measure levels of participation and monitor progress in reducing disparities.

Addressing psychosocial disability in population surveys is complex, not only because of stigma but also because psychosocial functioning encompasses a number of concepts, including thoughts, feelings and behaviours. The seven questions can address these different aspects by including questions on social interactions and on controlling emotions and behaviour. In addition, the methodology applied in developing the seven questions ensures that transparency is provided in how the questions are tested and adapted based on each round of testing.

The findings of these four initial cognitive tests show that there are suitable questions that target specific difficulties experienced by people with psychosocial disability. Indeed, response profiles and types of interpretations across the respondents with and without a MHC show that there are clear differences between these two groups, with higher levels of difficulty reported by people with a MHC across all four countries.

The willingness and interest of most respondents, both with and without a MHC, highlight the importance of addressing the experiences of people with MHCs. Furthermore, the narratives of people with a MHC highlighted the impact of their condition on social interactions and relationships, and how the reaction of other people to their difficulties adds a further burden. Thus, the testing process opens a window on the interaction of a person’s impairment with negative attitudes towards MHCs resulting in participation restrictions, such as social exclusion and isolation, and limited access to employment.

As the UNCRPD makes clear, preventing human rights violations and promoting a rights-based society for people with psychosocial disabilities need accurate data to illuminate their experiences of discrimination and to monitor improvements. Given the high level of stigma associated with psychosocial disability, it is even more urgent to ensure that these are made more visible as reflected in the words of one respondent with a mental condition: ‘It’s like a page that no one wants to open’. Well-performing questions allow that page to be opened.

Progress has been made in the development of a set of questions for population surveys; however, defining the minimum set of questions requires further cognitive tests, especially in geographical areas not yet covered, to ensure that the questions are not influenced by cultural and linguistic factors and are interpreted consistently across countries. Besides reviewing the wording of the questions being tested, based on the cognitive interviewing test results, a further step is to select a minimum set of three or four questions that can be adopted for use in population-based surveys to measure psychosocial functioning. To complete the question validation process, several field tests must be carried out to verify the performance of the set in a standard survey mode in field and to define the most appropriate cut-off for international data comparison. In addition, documents to support the implementation of the survey tool must be produced, such as detailed guidelines on how to use the questions, syntax for the analysis and translation guidelines. The final step is adoption of the question set by the WG.

Conclusion

This article briefly presents the work carried out, so far, by the WG in developing measures to be used in population surveys to include people with psychosocial disabilities into the official disability statistics.

The UNCRPD (UN 2006) explicitly includes, among persons with disabilities, those who have long-term mental impairments ‘which in interaction with barriers of various kinds may hinder their full and effective participation in society on an equal basis with others’ (Art.1). The Convention also requires State Parties to collect ‘appropriate information, including statistical data and the results of research, to enable the formulation and implementation of policies for the purpose of giving effect to the present Convention’ (Art. 31). The psychosocial functioning measures being developed allow statistical offices and other agencies to collect and disseminate internationally comparable data on psychosocial disability.

Ensuring that this work is disseminated and understood by a community beyond statisticians and survey methodologists is greatly facilitated by people with a broad vision who are not directly involved in the work but who understand its significance and promote it. Prof. Leslie Swartz is such a person.

Acknowledgements

This article is based on research originally presented at the 20th Washington Group on Disability Statistics Annual meeting, held virtually on 22–24 September 2020. The content has since been expanded and revised for journal publication. This republication is done with permission from the conference organisers.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

M.S. and E.D.P. contributed equally in the conceptualisation, implementation, analysis and writing of the article.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

The authors declare that all data that support this research article and findings are available in the article and its references.

The views and opinions expressed in this article are those of the authors and are the product of professional research. The article does not necessarily reflect the official policy or position of any affiliated institution, funder, agency or the publisher. The authors are responsible for this article’s results, findings and content.

References