Depression is a global concern as with an estimated 300 million individuals worldwide experiencing depression. In South Africa, the prevalence rate of depression is estimated at 9.7% of the population. With the increase in mobile internet usage in South Africa, an online depression screening tool could provide opportunities for the screening of depression symptoms aiding access to mental health interventions. This project identified an open access tool for screening depression, the Center for Epidemiologic Studies Depression Scale – Revised (CESD-R), and adapted it for online use by the adult South African population. This study followed on from the adaptation phase on the CESD-R and aimed to determine the content validity of the adapted CESD-R for online use in South Africa using the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) methodology. The study followed a two-phased design. Study one utilised a qualitative approach, where 50 experts commented on the content validity of the tool. The results were used to further adapt the tool which resulted in a 20-item depression screening tool. Study two followed a quantitative design in order to establish the content validity in terms of determining the Content Validity Ratios, Item-Content Validity Index as well as the Kappa Statistic of the 20 items. Based on these statistics, 19 of the 20 items were retained. Overall, the adapted online depression screening tool displays good content validity and holds potential as a screening tool where access to mental health may be limited.

Keywords: CESD-R; content validity; online depression screening tool; mental health; South Africa.

Introduction and literature review

According to the World Health Organization’s (WHO, 2017) Global Health estimates for 2017, an estimated 300 million individuals have depression globally, accounting for 4.4% of the world’s population (WHO, 2017). Nine per cent of individuals on the African continent suffer from depression (WHO, 2017). In South Africa, a prevalence rate of 9.7% was attributed to major depression (Tomlinson et al., 2009). The association between mental health and non-communicable diseases (NCD) has been highlighted by the WHO world mental health surveys (Leentjens, 2010). Depression is one of the mental health illnesses that have been found to be comorbid with NCD, such as cancer and diabetes, and respiratory and cardiovascular diseases, therefore interventions for depression are vital in controlling non-communicable diseases (Caruso et al., 2017; Leentjens, 2010; Stein et al., 2019).

The treatment of depression in South Africa is often met with many obstacles, such as stigmas, lack of mental health facilities, lack of depression screening, and under-resourced hospital settings. In 2017, mental health facilities in South Africa were limited with 4.33 beds per 100 000 population in general hospitals, and 16.56 beds per 100 000 in mental health hospitals (WHO, 2017). In 2019, South Africa reported having 0.31 psychiatrists and 0.97 psychologists in the public sector per 100 000 population (Docrat, Besada, Cleary, Daviaud, & Lund, 2019). As a result of the lack of or limited mental health resources in South Africa, mental illnesses, such as depression, are often under-diagnosed and under-treated (Nglazi et al., 2016).

In an attempt to overcome the barriers to mental health access and care, researchers propose the use of digital technologies such as the Internet, portable electronic devices and mobile applications (Aguilera, 2015; Cortelyou-Ward, Rotarius, & Honrado, 2018; Lal & Adair, 2014; Patel et al., 2018). Online mental health screening provides easy and wide access; it is economicaland time-efficient, and allows for early detection of people at risk of depression (Austin, Carlbring, Richards, & Andersson, 2006; Buchanan, 2003; Donker, Van Straten, & Cuijpers, 2010; Lal & Adair, 2014; Patel et al., 2018). In South Africa, 58.7% of individuals have mobile internet access and 63.0% of households have at least one member who has access to internet either at work, at home, a place of study or through an internet café (Statistics South Africa, 2019). The potential benefits of online mental health screening together with the access to benefits of online mental health screening suggest that online mental health resources could provide access to mental health services where they are limited and often inaccessible.

This study focussed on adapting a depression screening tool for online usage. As part of the study, a systematic review was conducted to identify any existing, psychometrically sound, online depression screening tools for the general public of South Africa. Results indicated that the Beck Depression Inventory-II (BDI-II), the Center for Epidemiology Studies Depression Scale (CESD) and the Patient Health Questionnaire (PHQ-9) were the most commonly utilised online depression screening tools, but only one depression screening tool was specifically designed for use by the general public. There were no screening tools specifically designed for the diverse groups in South Africa (Hassem & Laher, 2019).

Based on the results of the systematic review, it was decided to adapt the CESD. The revised version of the CESD (CESD-R) is the most recent version of the CESD which reflects the depression symptoms in accordance with the Diagnostic and Statistical Manual of Mental Disorders - Fourth Edition (DSM-IV) (Eaton, Smith, Ybarra, Muntaner, & Tien, 2004). The CESD-R website (The Center for Epidemiologic Studies Depression Scale Revised, n.d.) confirms that the symptoms as assessed are in accordance with symptoms of Major Depressive Episode (MDE) in the DSM-5. In addition, the CESD-R is the most recently developed depression screening tool of the three commonly utilised depression screening tools (BDI-II, CESD and PHQ-9), and is available as an open access resource (The Center for Epidemiologic Studies Depression Scale Revised, n.d.).

The adaptation of the CESD-R was grounded in the biopsychosocial-spiritual (BPSS) model. The model recognises biological, psychological, social and spiritual as distinct dimensions which cannot be separated from the whole as the components are intertwined allowing for multifactorial understandings of mental illness aetiology (Van Rensburg, Poggenpoel, Myburgh, & Szabo, 2015; Sulmasy, 2002). The BPSS is particularly salient to African contexts where mental health is said to be located in the relationship between the ancestors or spirits and human beings (Meyer, Moore, & Viljoen, 2003; Mufamadi & Sodi, 2010).

While spirituality and culture are relevant across the diverse South African population, screening tools for depression have not been adapted to account for these unique cultural and spiritual presentations of depression. The South Africa population is made up of individuals from various ethnic and religious groups as well as being a multilingual country with 11 official languages. Defining or translating the term ‘depression’ in African cultures is complex because it is not recognised within traditional African practices (Ellis, 2003; Patel, 2001; Stafford, Pedersen, Van Staden, & Jäger, 2008). For example, in the isiZulu language, the following terms are used to approximate depression: Dangala (worn out of body and mind or dejected), Khathele/ukukhathala (sense of worry and also conveys peace) and Ukukhathazeha (conveys grief, worry, hurt, sadness as well as heartache). Ukukhathazeha is also used in the isiXhosa language and has the same meaning as in the isiZulu language (Ellis, 2003).

The distress idiom called ‘thinking too much’ is also often associated with depression or as a symptom of depression (Kaiser et al., 2015). In a study conducted in a small Khwe community in Kimberley, South Africa, ‘thinking too much’ was associated with negative behavioural, emotional, social and somatic complaints (Hertog, De Jong, Van der Ham, Hinton, & Reis, 2016). ‘Thinking too much’ is viewed by traditional healers as well as a sample of women living with human immunodeficiency virus (HIV) as a symptom of depression (Andersen, Kagee, O’Cleirigh, Safren, & Joska, 2015; Ellis, 2003).

When reporting depression symptoms, studies have shown that individuals in African cultures tend to report more somatic than cognitive symptoms (Andersen et al., 2015; Mosotho, Louw, Calitz, & Esterhuyse, 2008). This emphasis on somatic symptoms often compromises the diagnosis of depression in many primary care settings (Mosotho et al., 2008). In studies conducted in South Africa, women tended to have a higher prevalence rate of depression, but this could be attributed to gendered cultural beliefs. In the Sesotho culture, for example, men are viewed as Monna ke nlu ha alle, which implies that men should not display emotions of grief, sadness and depression (Mosotho et al., 2008; Tomlinson et al., 2009).

In addition to cultural factors, English language proficiency needs to be considered. According to the General Household Survey (Statistics South Africa, 2018), English has been ranked as the sixth most common language spoken in South African homes (8.1%), and is ranked as the second most common language spoken outside South African homes (16.6%; spoken by 8.1% of individuals). Given that most South Africans are not English first language speakers, there is a need for a depression screening tool free of psychological jargon. Hence, this was taken into account when adapting the CESD-R for South Africa.

The unique context of the online psychological assessment screening environment, where the typical face-to-face interaction is removed, necessitates for a number of ethical issues to be considered. Draft ethical guidelines recommended by Hassem and Laher (2018) were utilised to ensure that the online screening tool was adapted accordingly. For example, if one is to include items of suicide ideation, designated resources should be available to contact individuals who are at risk of self-harm (Hassem & Laher, 2018). Unfortunately, these resources are limited in South Africa, and in order for these resources to be effective, various stakeholders would need to be involved in ensuring the effective use of such services. Ethically, the risk is too high for an individual completing an online screening (not diagnostic) tool to include an item assessing suicidality. Hence, this was excluded.

The aim of this study was to determine the content validity of the adapted CESD-R for online depression screening for diverse groups in South Africa. Content validity refers to the extent to which items on a scale accurately measure a construct based on the conceptual definition of the construct (Grahn & Gard, 2008; Lenz, 2010). Based on the literature, establishing content validity is typically performed qualitatively with little consensus as to which method should be followed. The Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) were developed in part to create more rigorous, standardised guidelines for establishing content validity (Terwee et al., 2018). The COSMIN guidelines for content validation of a Patient-Reported Outcome Measure (PROM) informed the design of this study (Terwee et al., 2018). According to the COSMIN guidelines, for a tool to display content validity, three broad domains need to be assessed. They are: (1) relevance (items are relevant for the measured construct as well as the context or specific population), (2) comprehensiveness (all aspects of the construct being measured are included) and (3) comprehensibility (all items can be easily understood by the target population) (Terwee et al., 2018). Thus, this study aimed to: (1) determine if the adapted online depression screening tool measures depression; (2) determine if the items of the tool are easily understood and are culturally appropriate for South Africans and (3) determine whether the instructions, response format and instant feedback provided are appropriate.

All participants had the right to decline participation. Participation was anonymous and completely confidential as no identifying information was requested. All participants had the right to stop participation at any point and also had the right not to answer any of the questions. There were no potential risks or benefits to participation in the study.

Methods

Two studies were carried out in order to determine the content validation of the online depression screening tool. Study one followed a qualitative research design, whereas study two followed a quantitative research design. In the following sections two, the studies are discussed in detail.

Study one

Test adaptation

The initial step of study one involved the adaptation of the CESD-R items. This step followed the first three stages of the test development guidelines proposed by Foxcroft (2018) and the International Test Commission Guidelines for Translating and Adapting Tests (ITC, 2017). The adaptation of the CESD-R items during studies one and two is highlighted (Table 1). For study one, CESD-R items were either rephrased or removed from the tool. Three items were removed from the CESD-R, which included the two items which assessed suicide ideation. Guided by the South African literature regarding depression (Anderson et al., 2015; Hertog et al., 2016; Kaiser et al., 2015; Mosotho et al., 2008), three items were added to the tool (see Table 2). The online adapted CESD-R for study one consisted of 20 items, which were free of psychological jargon. The response format (Not at all, 1 to 2 days, 3 to 4 days, 5 to 7 days and nearly every day for 14 days) and time period (2 weeks) that symptoms are experienced were not changed from the CESD-R.

Participants

Individuals were invited to participate in the study based on their expertise, the field of depression screening and depression. A non-probability, purposive sample of 50 mental healthcare personnel participated in the study (Patton, 1990). The participant demographics for study one are highlighted (Table 2). The majority of the sample was psychiatrists (n = 15), followed by psychologists (research, clinical and counselling psychologists) (n = 14) and Psychology Honours¹ students enrolled for a psychological assessment module (n = 13). Seven of the participants described their occupations as ‘other with nurses, religious leaders, a paediatrician and a campus coordinator being specified. The majority of the participants stated that they had been practising in their respective fields for more than 10 years (n = 19), followed by 14 participants practising for less than 10 years. Women made up the majority of the participants (n = 43). Half of the participants identified as white people (n = 25). Most participants identified as being Christian (n = 24), followed by Muslim and no religious affiliation (n = 13 and n = 8, respectively). Thirty-six participants identified English as being their home language, while Afrikaans was spoken by eight individuals and an African language (isiZulu, Ndebele, Sepedi, Setswana & Swati) was spoken by six of the participants as a home language (see Table 3).

Just over half of the sample (n = 26) diagnosed depression² in their capacity as a psychologist or psychiatrist, with the majority diagnosing depression at least weekly (n = 17). Interesting to note was that, out of the 26 individuals who stated that they diagnose depression, only 16 of these individuals had utilised a depression screening tool (see Table 2). The most cited screening tool utilised was the BDI-II, followed by the PHQ and the Hamilton-D (HAM-D).

Instruments

Participants were required to complete a brief demographic questionnaire requesting information on occupation, number of years practising, gender, population group, religious affiliation, home language, frequency in diagnosing depression as well as previous experience using a depression screening tool. Once the demographic questions were completed, participants were presented with a page detailing the content validation instructions. Participants were required to only read the instructions, items, scoring as well as feedback of the adapted CESD-R, and not to complete the tool. This was followed by 10 (yes or no response format) questions which were informed by the COSMIN evaluation of content validity of a PROM. Participants were given the choice to provide additional comments in relation to the 10 questions. Given the unique nature of the tool where instant feedback would be provided to the individual, two specific questions assessing the appropriateness of the scoring and feedback of the tool were included.

Procedure

Once items were finalised, an email was circulated to various South African healthcare professionals listed above (see Table 2) inviting them to participate in the study. Experts were identified based on their clinical and psychometric experience in the field of mental health and depression. The email described the nature of participation and contained a web link to the actual tool and content validation questions. Once participants clicked on the link (generated from SurveyMonkey), a participant information sheet appeared detailing the nature of the study. The survey was anonymous and, on average, took between 10 and 15 min to complete. Data collection took place electronically between July and October 2019 through an online survey tool.

Data analysis

Data were downloaded from the online tool and coded for analysis. Demographic variables and yes or no response options were analysed using frequencies on Statistical Package for the Social Sciences (SPSS) Version 25 (IBM Corp, 2017), and qualitative data were analysed using thematic analysis as specified by Braun and Clarke (2006).

Results of study one

Results are discussed in terms of the three broad themes recommended in the COSMIN guidelines that are relevance and comprehensiveness, comprehensibility and scoring and feedback.

Relevance and comprehensiveness

Because of the open-ended nature of the questions, various themes regarding the relevance of the tool emerged, namely, depression according to the DSM criteria, appropriateness of the items, time period, response format as well as the cultural applicability of the items.

Depression according to the DSM criteria

The majority of the participants indicated that the tool does appear to measure depression (n = 46) (see Table 3), which is echoed through the following comment made by participant 40: ‘It definitely does measure depression according to the DSM 5 framework’. Two participants noted that the items provided on the tool measured depression according to the DSM-5 criteria for depression, while three participants stated that not all the DSM symptoms of depression were measured by the tool. One participant specifically commented that the DSM criteria do not include local idioms of distress or depression, while two participants indicated that suicide ideation had not been included in the tool. It was noted by one of the participant that the tool measured depression similar to the K10 (Kessler Psychological Distress Scale) and CESD (original version of the online adapted CESD-R). One participant specifically cautioned against using the tool as a diagnostic measure.

Appropriateness of items

As evident in Table 3, the majority (n = 31) of the participants felt that the items were appropriate for the South African context. In order to accommodate the African view of the self, participants were asked if items need to be rephrased to ‘My family and friends’. Results indicate that the majority (n = 24) of the participants suggested that items do not need to be rephrased (see Table 3). Item appropriateness can be divided into three sub-themes, namely, items to be added, items to be removed and items to be rephrased or re-considered (see Table 3).

Items to be added: Five participants noted that weight gain and increased appetite are symptoms of depression that are commonly overlooked. Therefore, participants recommended that these two items should be added to the tool. Participant 9 expressed this as follows:

Four participants felt that an item targeting indecisiveness should be added as it is a common cognitive symptom of depressed patients. In addition, suicide ideation items were recommended to be included by five participants as they are symptom criteria for depression, according to the DSM.

Items to be removed: Two participants suggested removing the items: ‘I feel like I am moving too slow’ and ‘I have the need to play with my fingers or move around for no reason’ (psychomotor agitation) (see Table 1), as this is less common in depressed adults. One participant felt item 18 was unclear and another stated that item 17 was unclear, but they did not suggest removing these items. Two participants suggested that there were too many sleep items and some should be removed or replaced with another item. Item 20 (‘I feel bewitched almost all of the time’) was viewed as being the most inappropriate item (n = 11), implying that the item should be removed.

Items that need to be reconsidered or rephrased: Participants felt that items needed to be rephrased so that the tenses would be consistent throughout the tool. A suggestion was made that the items need to be edited for grammatical accuracy.

Cultural applicability of the tool

This theme is closely related to the theme of item appropriateness as the item, ‘I feel bewitched almost all of the time’, was the most commonly cited item that was inappropriate. This was because the term ‘bewitched’ is not culturally appropriate and the term was not clearly understood. Participants suggested that the item questions the cultural aetiology of depression in a tool which needs to relate to the experience of depression. In addition, participants suggested that the term ‘bewitched’ can have both negative and positive connotations. Participants suggested that if a culturally fair item is to be added, it should have follow-up questions.

Time period

With regards to the time frame of ‘two weeks’, three participants noted that the time frame needed to be reconsidered as the symptoms experienced could be a result of life changes or a traumatic event, as indicated by Participant 1:

In addition to the change in the time frame of the symptoms experienced, the attribution of symptoms to life changes was also found in the narratives of four other participants. These participants highlighted that symptoms could be attributed to various medical conditions (chronic conditions or stomach bugs) as well as the lifestyle the individual leads (symptoms of sleep attributed to exhaustion). Therefore, these suggestions showed the need for a statement ruling out medical conditions, lifestyle, as well as life changes causing the symptoms experienced.

Response format

Although no specific questions targeted the response format, six participants made comments which related to it. One participant specifically suggested that it might not be understood by the general public, while four participants suggested that the response format did not match well with the items, recommending that either the items be rephrased or the response format be changed for specific items. One participant felt that placing a time frame in the response format was inappropriate.

Lastly, two participants suggested that the responses can yield many false positives or negatives or be easily faked. The use of reverse scoring as well as a control question was suggested in order to prevent faking.

Comprehensibility

Comprehensibility of the tool is discussed in terms of appropriateness of the instructions, effectiveness of the language used, length of the tool and item order.

Appropriateness of instructions provided

As evidenced in Table 3, a majority of the participants (n = 47) felt that the instructions provided were appropriate. Five participants recommended specific changes to the instructions, while one participant suggested that the last statement in the instructions (‘Ask a friend or family member to assist you’) could deter individuals from taking the test and would need further consideration. Two of the participants suggested grammatical changes be made to the instructions provided. One participant suggested that a sentence encouraging individuals to complete all items be added.

Effectiveness of the language used

A majority of the participants (n = 49) felt that the tool will be easily understood by diverse groups in South Africa (see Table 3), as echoed by the following participant phrases ‘simple English’, ‘does not contain any difficult words’, ‘very simplistic words’ and ‘simply put and easy to understand’. Three participants made reference to the general public of South Africa not being English first language speakers and said it was imperative to ensure that all South Africans understand the items. These three participants suggested that the tool be translated into African languages, with one participant highlighting caution when doing translations.

Length of the tool and item order

All participants who commented on the length of the tool felt that it was appropriate for an individual who is depressed. This sentiment is represented by the following comment by participant 9: ‘The short length of the questionnaire is a strength of the tool as someone with moderate to severe depression will struggle to complete tasks’.

Only two participants mentioned that the order of the items needs to be reconsidered. Participants suggested that items measuring similar symptoms do not appear together and to prioritise symptoms specific to the South African context in the ordering of items.

Scoring and feedback provided

Although scoring and feedback does not form part of the COSMIN criteria for content validity, it is a vital point to discuss as the end-users of the tool will ultimately receive immediate feedback in a non-typical, face-to-face interaction. Table 3 shows that it is evident that the majority of participants felt that the scoring and feedback provided are appropriate (n = 41 and n = 35, respectively).

Only five participants commented on the scoring of the tool, with three participants suggesting that the scoring was not transparent as they did not know the score that was attributed to each response and the cut-offs were not mentioned upfront. Comments made regarding the feedback given to participants all emphasised the need to highlight the urgency of seeking help when individuals receive a high score on the tool. In addition, participants stated that the feedback was calmly stated so that it should not further perpetuate depression symptoms. Participants indicated that individuals who do not meet the criteria for depression should also be offered contact details if they feel the need to seek help. Lastly, participants felt that ‘consulting with your family practitioner’ should be the first point of contact that needs to be included.

Study two

Participants

A non-probability, purposive sample of 21 mental healthcare personnel participated in the study (Patton, 1990), with a majority of the sample represented by psychologists (n = 16). A greater part of the participants stated that they had been practising in their respective fields between 10 and 20 years (n = 10), with only one participant practising for under 10 years. A majority of the sample were females (n = 17). Just over half of participants identified as belonging to the White population group (n = 11) and being Christian (n = 11). English was noted as the dominant home language used by participants (n = 16) (see Table 2).

Only eight participants had not previously diagnosed depression, with the majority of participants (n = 6) diagnosing depression weekly. Lastly, a larger part of the participants (n = 15) had previously utilised a depression screening tool. The most commonly cited screening tool used was the BDI-II followed by the HAM-D.

Procedure

The first step of this study involved item refinement based on the results of study one. Weight, eating patterns and sleep items were represented by one item each which accounted for either side of the scale (weight gain or loss, increased or decreased appetite and less or more sleep) (see Table 1). The item regarding bewitchment was removed from the scale based on the results of study one. Items representing self-blame, loneliness and guilt were added. Therefore, a total of 20 items were included in study two. Given the occurrence of violence and traumatic events in South Africa as well as comments made in study one regarding the time period, this was changed to a 2-month period of symptoms experienced as this will rule out symptoms experienced because of these events which could result in an overdiagnosis of depression. Lastly, based on the feedback received regarding the response format in study one, the response format was changed to: ‘Not at all’, ‘Some of the time’, ‘Most of the time’ and ‘All of the time’.

Data collection and instruments

Participants were required to complete the same brief demographic questionnaire as per study one. Once the demographic questions were completed, participants were presented with a page detailing the content validation instructions. A 3-point Likert scale was used to assess the relevance of each of the 20 items included in the screening tool (1 = item is not essential; 2 = item is useful but not necessary and 3 = item is essential for diagnosing depression). Participants were also asked to rate the response format used as being ‘relevant’, ‘somewhat relevant’ or ‘not relevant’. Lastly, participants were asked for any additional feedback or input on the tool. The questionnaire took, on average, 10 min to complete. Data collection took place electronically between October and December 2019 through an online survey tool.

Data analysiss

Data were downloaded from the online tool and coded for analysis. Demographic variables were analysed using frequencies. There are various statistical calculations for content validity, which either provide a consensus estimate or a consistency estimate. Most commonly used in the field of psychology is the Content Validation Ratio (CVR) proposed by Lawshe (1975). There is debate in the literature as to which CVR value to utilise thus, at a 5% level of significance with 21 experts, a CVR value lower than 0.359 or 0.429 would exclude an item (Ayre & Scally, 2014; Wilson, Pan, & Schumsky, 2012).

Polit et al. (2007) recommended the use of the Item-Content Validation Index (I-CVI), Scale-Content Validation Index (S-CVI) as well as the Kappa statistic. Therefore, this study used the CVR, I-CVI, S-CVI (average) as well as the Kappa statistic to investigate content validity (Polit et al., 2007). With regards to the I-CVI values, any values under 0.70 should be eliminated, values between 0.70 and 0.79 required some revision and any value of at least 0.78 is considered as appropriate. The Kappa statistic values between 0.40 and 0.59 were considered fair, 0.60 and 0.74 were good and 0.75 and higher were considered excellent (Polit et al., 2007). For the development of a new tool, the average S-CVI of 80% or above is considered acceptable (Davis, 1992).

In order to compute the I-CVI and Kappa values, response options 2 (‘item is useful but not necessary’) and 3 (‘item is essential for diagnosing depression’) were combined and represented as a relevant item and the response option 1 (‘item is not essential’) represented a non-relevant item.

Results of study two

When looking at items rated as essential by the experts, only three items (3, 8 and 9) were rated by all the 21 experts as being essential. Fifteen items received an essential rating by the majority of the experts (n > 10) (see Table 4). With regards to experts rating items as relevant, 12 items received a relevant rating by all experts. Item 11 (‘I have been feeling happy’) received the lowest number of experts rating the item as relevant (n = 13).

The CVR calculations had a very wide range from −0.524 to 1. Both I-CVI and Kappa ranged from 0.62 to 1. The lowest CVR value was obtained on item 6, while the lowest I-CVI and Kappa values were obtained for item 11. It is evident that all 20 items included in the tool were considered as good items with the exception of item 11. Item 11 did not meet two out of the three content validity criteria (CVR and I-CVI). The Kappa value was 0.62, which indicated the item was good.

Using the guidelines in the literature (Ayre & Scally, 2014; Polit et al., 2007; Wilson et al., 2012), 10 items (3, 8, 9, 14, 19, 4, 17, 5, 20 and 21) met all the content validity criteria (CVR, I-CVI and Kappa). Items 3, 8, 9 and 14 obtained absolute scores on all three criteria (see Table 4). Ten items did not meet the CVR criteria for inclusion (16, 18, 21, 7, 15, 10, 2, 1, 11 and 6), but nine of these items met the I-CVI and Kappa criteria for an excellent item. Three (16, 18 and 21) out of the 11 items that did not meet the CVR criteria received absolute scores on the I-CVI and Kappa scores.

The average S-CVI score for the tool was 0.94. The majority of participants (n = 18) felt that the adapted time period and response format were relevant.

Discussion

Two studies were used to determine the content validity of an adapted online depression screening tool. For study one, 50 mental health experts unanimously agreed that the tool was valid in terms of the instructions, items, scoring and feedback provided. However, various suggestions were made to improve the quality of the tool. For study two, the recommendations suggested during study one were undertaken and the tool was revised.

In terms of the COSMIN criteria for content validity, the adaption of the depression screening tool for online usage by diverse groups in South Africa shows good relevance.

A follow-up content validation was conducted with 21 experts in the field and three independent researchers. Nineteen of the experts in study two (final tool) felt that all the items on the tool were relevant for screening of depression, with the exception of item 11 (‘I have been feeling happy’). Only 13 experts felt item 11 was relevant. This could be attributed to the item being reversed scored. Therefore, it was decided that this item would be excluded from the final tool. The average S-CVI further highlights the content validity of the overall tool by achieving an average S-CVI score of higher than 80% (Davis, 1992).

Four South African idioms of distress and depression were added to the tool, namely, ‘I have been experiencing more body aches and pains (e.g. headaches, neck pain or back pain)’, ‘I have been thinking too much’, ‘I have been feeling alone’ and ‘I have not felt like myself’. Three of these items did not meet the CVR criteria; however, both the I-CVI and Kappa criteria were met. These items received a low CVR as a result of few experts rating the items as essential in depression screening;, however, many experts indicated that the time was useful in depression screening, hence the items were retained.

The nine items which were rephrased from the CESD-R and which represented the DSM 5 depression criteria received absolute I-CVI and Kappa scores, and met the CVR criteria. Items 7 and 15 did not appear on the original CESD-R, but were recommended by experts in study one to be included in the screening tool. These items did not meet the CVR criterion, but met the I-CVI and Kappa criteria. It should be noted that these two items assessed the symptom criteria for depression as highlighted in the DSM 5 (focus and indecisiveness).

This study demonstrated that a 19-item adapted online depression screening tool displays relevance in terms of the construct being measured and the appropriateness of the target population, and context for which it is intended. There are limitations, in that content validation is a subjective view of experts and therefore could result in bias when items are rated; however, the two studies with various experts and both qualitative and quantitative indicators reduced this risk. It is acknowledged that the pool of experts is small and does not represent the various language groups present in South Africa. Going forward, it would be necessary to establish the content validity in terms of comprehensiveness and comprehensibility ratings as described by COSMIN. These ratings are dependent on the target population views and pilot testing of the tool and not the development stage of the tool. In addition, the construct and criterion validity as well as reliability of the tool would need to be assessed.

Conclusion

This study highlighted that the adapted online depression screening tool designed for diverse groups in South Africa shows good relevance in terms of content validity. In addition, the tool was phrased using simple language free from psychological jargon, which is hoped would encourage better understanding by second language English speakers. Screening individuals for depression allows for early detection of their depression risk and has much to offer for the under-resourced South African mental healthcare landscape. The online screening of depression allows for early detection and self-help options for depression. Further, it empowers individuals to discuss their symptoms from a better knowledge base with a doctor or other healthcare professional allowing for intervention much sooner than would have been the case if the individual had no support or no means of checking their symptoms and of accessing information on depression. The tool, therefore, has the potential to be incorporated as a screening tool for depression across standard platforms in university counselling centres, primary health care intake forms or even on web platforms such as the South African Depression and Anxiety Support Group (SADAG).

Acknowledgements

The author declares that she has no financial or personal relationships that may have inappropriately influenced her in writing this article.

T.H. is the sole author and was responsible for the conceptualisation, data collection and analysis as well as write up for the article.

The study was approved by the Human Research Ethics Committee – Medical (HRECM) of the University of the Witwatersrand, reference number: M180402.

This work is based on the research supported wholly or in part by the National Research Foundation of South Africa (grant number: 112948).

Data sharing is not possible, due to the nature of the data and related ethical principles.

The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of any affiliated agency of the author, and the publisher.

References

Aguilera, A. (2015). Digital technology and mental health interventions: Opportunities and challenges. Arbor, 191(771), a210–a210. https://doi.org/10.3989/arbor.2015.771n1012

Andersen, L., Kagee, A., O’Cleirigh, C., Safren, S., & Joska, J. (2015). Understanding the experience and manifestation of depression in people living with HIV/AIDS in South Africa. AIDS Care, 27(1), 59–62. https://doi.org/10.1080/09540121.2014.951306

Austin, D.W., Carlbring, P., Richards, J.C., & Andersson, G. (2006). Internet administration of three commonly used questionnaires in panic research: Equivalence to paper administration in Australian and Swedish samples of people with panic disorder. International Journal of Testing, 6(1), 25–39. https://doi.org/10.1207/s15327574ijt0601_2

Ayre, C., & Scally, A.J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Buchanan, T. (2003). Internet-based Questionnaire Assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy, 32(3), 100–109. https://doi.org/10.1080/16506070310000957

Caruso, R., Nanni, M.G., Riba, M., Sabato, S., Mitchell, A.J., Croce, E., & Grassi, L. (2017). Depressive spectrum disorders in cancer: Prevalence, risk factors and screening for depression: A critical review. Acta Oncologica, 56(2), 146–155. https://doi.org/10.1080/0284186X.2016.1266090

Cortelyou-Ward, K., Rotarius, T., & Honrado, J.C. (2018). Using technology to improve access to mental health services. The Health Care Manager, 37(2), 101–108. https://doi.org/10.1097/HCM.0000000000000211

Davis, L.L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4

Docrat, S., Besada, D., Cleary, S., Daviaud, E., & Lund, C. (2019). Mental health system costs, resources and constraints in South Africa: A national survey. Health Policy And Planning, 34(9), 706–719. https://doi.org/10.1093/heapol/czz085

Donker, T., Van Straten, A., & Cuijpers, P. (2010). Internet-based mental health screening. In J. Bennett-Levy, D. Richards, P. Farrand, H. Christensen, K. Griffiths, D. Kavanagh, … C. Williams (Eds.), Oxford guide to low intensity CBT interventions (pp. 241–245). Oxford University Press.

Eaton, W.W., Smith, C., Ybarra, M., Muntaner, C., & Tien, A. (2004). Center for Epidemiologic Studies Depression Scale: Review and revision (CESD and CESD-R). In M.E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment: Instruments for adults (pp. 363–377). Lawrence Erlbaum Associates Publishers.

Ellis, C.G. (2003). Cross-cultural aspects of depression in general practice. South African Medical Journal, 93(5), 342.

Foxcroft, C. (2018). Developing a psychological measure. In C. Foxcroft & G. Roodt (Eds.), Introduction to psychological assessment in the South African context (5th ed.). Cape Town, South Africa: Oxford University Press.

Grahn, B., & Gard, G. (2008). Content and concurrent validity of the motivation for change questionnaire. Journal of Occupational Rehabilitation, 18(1), 68–78. https://doi.org/10.1007/s10926-008-9122-7

Hassem, T., & Laher, S. (2018, August21–24). Ethics of online screening for mental illnesses: A systematic review. Paper Presented at the World Congress of Psychiatry Conference on Psychiatry and Mental Health: Global inspirations, locally relevant inspirations, Lisbon.

Hassem, T., & Laher, S. (2019). A systematic review of online depression screening tools for use in the South African context. South African Journal of Psychiatry, 25(1), 1–8. https://doi.org/10.4102/sajpsychiatry.v25i0.1373

Hertog, T.N., De Jong, M., Van der Ham, A.J., Hinton, D., & Reis, R. (2016). “Thinking a Lot” among the Khwe of South Africa: A key idiom of personal and interpersonal distress. Culture, Medicine, and Psychiatry, 40(3), 383–403. https://doi.org/10.1007/s11013-015-9475-2

IBM Corp. (2017). IBM SPSS Statistics for Windows, Version 25.0. IBM Corp.

International Test Commission. (2017). The ITC guidelines for translating and adapting tests (2nd ed.). Retrieved from www.InTestCom.org

Kaiser, B.N., Haroz, E.E., Kohrt, B.A., Bolton, P.A., Bass, J.K., & Hinton, D.E. (2015). “Thinking too much”: A systematic review of a common idiom of distress. Social Science & Medicine, 147, 170–183. https://doi.org/10.1016/j.socscimed.2015.10.044

Lal, S., & Adair, C.E. (2014). E-mental health: A rapid review of the literature. Psychiatric Services, 65(1), 24–32. https://doi.org/10.1176/appi.ps.201300009

Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

Leentjens, A.F. (2010). [Review of the book: Global Perspectives on Mental-physical Comorbidity in the WHO World Mental Health Surveys, edited by M. R. Von Korff, K. M. Scott, & O. Gureje]. Cambridge University Press. 2009. Psychological Medicine, 40(7), 1226–1227. https://doi.org/10.1017/S0033291710000632

Lenz, E.R. (2010). Visual analog scales. In C.F. Waltz, O.L. Strickland, & E.R. Lenz (Eds.), Measurement in nursing and health research (pp. 319–325). Springer.

Meyer, W., Moore, C., & Viljoen, H. (2003). Personology: From individual to ecosystem (3rd ed.). Sandton, South Africa: Heinemann Publishers (Pty) Ltd.

Mosotho, N.L., Louw, D.A., Calitz, F.J., & Esterhuyse, K.G. (2008). Depression among Sesotho speakers in Mangaung, South Africa. African Journal of Psychiatry, 11(1), 35–43. https://doi.org/10.4314/ajpsy.v11i1.30253

Mufamadi, J., & Sodi, T. (2010). Notions of mental illness by Vhavenda traditional healers in Limpopo Province, South Africa. Indilinga African Journal of Indigenous Knowledge Systems, 9(2), 253–264.

Nglazi, M.D., Joubert, J.D., Stein, D.J., Lund, C., Wiysonge, C.S., Vos, T., … Bradshaw, D. (2016). Epidemiology of major depressive disorder in South Africa (1997–2015): A systematic review protocol. BMJ Open, 6(7), e011749. https://doi.org/10.1136/bmjopen-2016-011749

Patel, V. (2001). Cultural factors and international epidemiology: Depression and public health. British Medical Bulletin, 57(1), 33–45. https://doi.org/10.1093/bmb/57.1.33

Patel, V., Saxena, S., Lund, C., Thornicroft, G., Baingana, F., Bolton, P., … UnÜtzer, J. (2018). The Lancet Commission on global mental health and sustainable development. The Lancet, 392(10157), 1553–1598. https://doi.org/10.1016/S0140-6736(18)31612-X

Patton, M. (1990). Qualitative evaluation and research methods (2nd ed.). Sage.

Polit, D.F., Beck, C.T., & Owen, S.V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199

SADAG, see South African Depression and Anxiety Group. Retrieved from https://www.sadag.org/

Stafford, G.I., Pedersen, M.E., Van Staden, J., & Jäger, A.K. (2008). Review on plants with CNS-effects used in traditional South African medicine against mental diseases. Journal of Ethnopharmacology, 119(3), 513–537. https://doi.org/10.1016/j.jep.2008.08.010

Statistics South Africa. (2018). General household survey. Retrieved from http://www.statssa.gov.za/publications/P0318/P03182018.pdf

Statistics South Africa. (2019). General household survey. Retrieved from http://www.statssa.gov.za/publications/P0318/P03182019.pdf

Stein, D.J., Benjet, C., Gureje, O., Lund, C., Scott, K.M., Poznyak, V., & Van Ommeren, M. (2019). Integrating mental health with other non-communicable diseases. BMJ, 364, 1295. https://doi.org/10.1136/bmj.l295

Sulmasy, D.P. (2002). A biopsychosocial-spiritual model for the care of patients at the end of life. The Gerontologist, 42(suppl_3), 24–33. https://doi.org/10.1093/geront/42.suppl_3.24

Terwee, C.B., Prinsen, C.A., Chiarotto, A., Westerman, M. J., Patrick, D.L., Alonso, J., … Mokkink, L.B. (2018). COSMIN methodology for evaluating the content validity of patient-reported outcome measures: A Delphi study. Quality of Life Research, 27(5), 1159–1170. https://doi.org/10.1007/s11136-018-1829-0

The Center for Epidemiologic Studies Depression Scale Revised. (n.d.). CESD-R website. Rerieved from https://cesd-r.com/

Tomlinson, M., Grimsrud, A.T., Stein, D.J., Williams, D.R., & Myer, L. (2009). The epidemiology of major depression in South Africa: Results from the South African stress and health study. South African Medical Journal, 99(5), 365–373.

Van Rensburg, A.J., Poggenpoel, M., Myburgh, C.P.H., & Szabo, C.P. (2015). Defining and measuring spirituality in South African specialist psychiatry. Journal of Religion and Health, 54(5), 1839–1855. https://doi.org/10.1007/s10943-014-9943-y

Wilson, F.R., Pan, W., & Schumsky, D.A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45(3), 197–210. https://doi.org/10.1177/0748175612440286

World Health Organisation (WHO). (2017). Depression and other common Mental Disorders: Global Health Estimates. Retrieved from http://apps.who.int/iris/bitstream/10665/254610/1/WHO-MSD-MER-2017.2-eng.pdf

Footnotes

1. The honours degree in South Africa is a postgraduate qualification that follows on from an undergraduate degree and precedes a Master’s degree

2. Depression in South Africa is diagnosed by clinical, counselling and educational psychologists, general practitioners as well as psychiatrists.

Original Research

Establishing the content validity of an online depression screening tool for South Africa

Tasneem Hassem

Abstract

Introduction and literature review

Methods

Study one

Test adaptation

Participants

Instruments

Procedure

Data analysis

Results of study one

Relevance and comprehensiveness

Depression according to the DSM criteria

Appropriateness of items

Cultural applicability of the tool

Time period

Response format

Comprehensibility

Appropriateness of instructions provided

Effectiveness of the language used

Length of the tool and item order

Scoring and feedback provided

Study two

Participants

Procedure

Data collection and instruments

Data analysiss

Results of study two

Discussion

Conclusion

Acknowledgements

Competing interests

Author’s contributions

Ethical considerations

Funding information

Data availability

Disclaimer

References

Footnotes