Mathematical literacy provides learners with an awareness and understanding of the role that mathematics plays in the modern world. Mathematical literacy is a subject driven by life-related applications of mathematics. It enables learners to develop the ability and confidence to think numerically and spatially in order to interpret and critically analyse everyday solutions and to solve problems. (p. 3)

ML differs from Mathematics in purpose and in content. In Mathematics, emphasis is placed on engaging with increasingly more complex and abstract mathematical concepts, the relations between them and some applications to problems. However, in ML the emphasis is specifically on the application of basic mathematics to understand situations in real life. There is some lack of clarity evident in the description of Mathematical Literacy noted above, as being mathematically literate requires a sound mathematical base of algebraic concepts and skills. The debate about the subject content of Mathematics and Mathematical Literacy, though regarded as critical, is not the focus of this article.

The juxtaposition of mathematics content with real life contexts has meant that many people are unclear about how competence or proficiency in ML may be demonstrated. It is clear that ML as a subject in its infancy requires much research in respect of teaching, learning and assessment, in order to generate debate and establish some consensus on the many contrasting perspectives within the ML field.

In this study the focus is the Grade 12 ML preparatory examination, which was set by a provincial Department of Education and is intended to prepare students for the final examination. We pay attention to one aspect, that is, assessment in ML, by identifying some issues arising from the analysis of the empirical data obtained from learners’ responses to this provincial preparatory assessment. The construct under scrutiny is the notion of proficiency in the subject ML. We apply Rasch measurement theory (RMT) to investigate the validity and accuracy of the test in providing a measurement-like representation of ML proficiency in terms of person proficiency and item difficulty.

A valid and reliable test would provide teachers with some indication of the levels of mastery of curricular elements and of developing proficiency in ML. It should also provide the Department of Education with an overview of the entire learner cohort taking ML. An application of the Rasch model will help us to identify anomalies and inconsistencies amongst these assessment items and the accompanying scoring rubrics and working memoranda.

In this article we consider the implications of considering the purpose of the test and the construction of rubrics so that they work coherently in the interests of valid measurement-like properties and consequently provide reliable information for teachers. Some concepts underlying the Rasch measurement theory are introduced to clarify the analytic process. The aim of the article is to investigate the domain of ML and offer some observations concerning the assessment of ML.

The Grade 12 KZN provincial preparatory ML examination paper is intended to assist Grade 12 learners in their preparation for the final examination. It is set by a team of examiners selected by the education department and written under examination conditions. For the purposes of this study the ML 2009 preparatory test was re-marked by the third author to ensure that the final version of the marking was entirely consistent with the marking memorandum supplied by the KZN Department of Education. A Rasch analysis supported the investigation of the test as a whole, the items and the ML learner responses. This analysis was conducted to identify factors that may have affected the coherence of the instrument for this sample of learners.

The first requirement for this analysis was to capture the score obtained in each of the 51 items for each of the 73 students.² The Rasch model offers various statistics to help diagnose where the data differs from what is expected by the model. Multiple means are applied to an analysis of this nature, to enable the subject expert to make an informed judgement. The output provides statistics on the test as a whole as well as the individual item statistics, in particular the fit residual statistic and the chi square probability statistic, which provide information on the fit of the items. In addition to these statistics we investigated the item characteristic curves (ICCs) to identify which items were misfitting in the ways to be discussed. The research question directing the study is: How may detailed attention to the scoring of the items in a Mathematical Literacy test through theoretical investigation and the application of the Rasch model contribute to a more consistent outcome?

From the classical theory of measurement, and measurement in the physical sciences (Wright, 1997), we note that the property of invariance of comparisons across the scale of measurement is a requirement. The application of the Rasch model enables the calibration of item measures and the estimation of person locations on a common continuum that together fit the criteria of invariance for a particular frame of reference (Rasch, 1960/1980; Humphry, 2005; Humphry & Andrich, 2008). The comparative difficulty of any two items should be constant regardless of the abilities of the persons responding to the items. Where the data do not conform to the measurement principles, the model will highlight the anomalies for further investigation. In the current study, the application of a Rasch analysis highlighted anomalies and inconsistencies that constituted threats to the construction of measures in the sense understood in classical measurement theory. In particular, the allocation of marks was inconsistent with the grading of proficiency along a continuum. In this article, we investigate the outcomes of both the initial scoring memorandum and the revisions, utilising both Rasch analysis and the educational considerations.

This explanation is presented as follows by Dunne, Long, Craig and Venter (2012):

Each outcome of an interaction between a person and an item is uncertain but has a probability governed only by these two characteristics, that is by person ability (β_n) and item difficulty (δ_i). The Rasch model avers that the arrays of numbers β_n and δ_i are on the same linear scale, so that all differences between arbitrary pairs of these numbers such as (β_n − δ_i) and hence also (β_n − β_m) and (δ_i – δ_j) are meaningful. Through these differences we may not only assign probabilities to item outcomes but also measure the contrasts between ability levels of items, and offer stochastic interpretations of these contrasts. (p. 7)

Applying Equation 1, we can see that if a person v is placed at the same location on the scale as an item i, then β_v = δ_i, that is, β_v − δ_i = 0, and the probability in Equation 1 is thus equal to 0.5 or 50%. Thus, any person will have a 50% chance of achieving a correct response to an item whose difficulty level is at the same location as the person’s ability level. Similarly, if an item difficulty is above a person’s ability location, then the person has a less than 50% chance of obtaining a correct response on that item, whilst for an item whose difficulty level is below that of the person’s ability the person would have a greater than 50% chance of producing the correct response. In Figure 1, the person location is represented on the horizontal axis, with the probability of a correct response located on the vertical axis.

FIGURE 1: Category probability curves (Item 2.2.3).

Figure 1 models the conditional probability of a score of 0, 1 or 2 for a polytomous item (Item 2.2.3a) with three categories. As the person ability increases, the conditional probability of a score of 0 decreases. By contrast, as ability increases the probability of obtaining a maximum score of 2 increases. Also on this graph is the curve that shows the probability of a score of 1. In summary, this curve shows that when a person has very low ability relative to the item’s location, then the probability of a response score of 0 is most likely; when a person is of moderate ability relative to the item’s location, then the most likely score is 1 and when a person has an ability much greater than the item’s location, then the most likely response score is 2 (see also Van Wyke & Andrich, 2006, p. 14).

[A] person having a greater ability than another person should have the greater probability of solving any item of the type in question, and similarly, one item being more difficult than another means that for any person the probability of solving the second item is the greater one. (Rasch, 1960/1980, p. 117)

In RMT it is expected that the data will accord well with the model. The notion of ‘fit’, that is, accord with the model, is defined as ‘the correspondence between a data set and a statistical model’ (Douglas, 1982, p. 32). The model provides indicators that alert researchers to where this principle of invariance of comparisons is not being met, which may result in item misfit. The fit residual is a measure of the difference between the observed response of each person to each item and that predicted by the model. The analysis process, whether showing a degree of conformity with the model or not, inevitably leads to greater understanding of the construct in question.

In a Rasch analysis test of fit, the learners are placed into class intervals of approximately equal size. We have used four groups. The mean ability of the four groups becomes the horizontal coordinate of points in the diagrams, depicting the probability of answering correctly.

Where the data conform to the model, the theoretical curve (the expected frequencies) and the observed proportions (the empirically established average of the actual item scores in the four chosen groups) are in alignment. Figure 2 shows the theoretical curve as expected by the model and the observed proportions, represented by black dots.

FIGURE 2: Item characteristic curve for Item 2.1.3, indicating fairly good fit.

Firstly, the observed proportions may align with the theoretical curve, in which case there is a good fit to the theoretical requirement. Figure 2 shows the item characteristic curve (ICC) for Item 2.1.3, in which the observed proportions are aligned fairly well with the theoretical curve. Note that the fit residual, 0.601, is relatively small tending towards zero and within an acceptable range of good fit (–2.5 to +2.5). This relatively small residual means that the difference between the observed response of each person to each item and the expected response is small.

A second phenomenon may be that the observed proportions are flatter than the theoretical curve, in which case the item does not discriminate enough. The pattern is labelled under-discrimination or underfit and is illustrated in Figure 3. This unexpected pattern indicates that learners of lower proficiency appear to perform better than expected on this item and consequently, because of the interactive nature of item difficulty and learner ability, the high proficiency learners are falsely estimated to respond to the item as if the item was easier than it really was. The qualitative analysis suggests that a possible explanation lies with the marking rubric, which gives scores between 0 and 3. The scoring rubric allocates an arbitrary method mark and an additional mark for presenting information provided in the instruction. The allocation of marks appears to be more generous for the poorly performing learners and too constrained for the higher performing learners.

FIGURE 3: Item characteristic curve for Item 3.2.2, indicating under-discrimination.

A third general category occurs when the observed proportions are steeper than the theoretical curves, in which case the discrimination is greater than expected, as shown in Figure 4. Over-discrimination in an item may unduly advantage high proficiency learners, whilst disadvantaging learners of lower proficiency. Whilst traditional test theory asserts the greater the item discrimination the better, the case in RMT is that highly discriminating items provoke a concern that there is a marked dependence amongst responses in one form or another. An example of such a misfit is that of Item 1.3.2, shown in Figure 4. Again we note that the fit residual is relatively high at –2.613. Both a high negative fit residual and a high positive fit residual signal poor fit to the model.

FIGURE 4: Item characteristic curve for Item 1.3.2, indicating over-discrimination.

FIGURE 5: Item characteristic curve for Item 4.1.1, indicating haphazard misfit.

FIGURE 6: The person-item location distribution prior to rescoring.

FIGURE 7: The person-item location distribution prior to rescoring.

Clearly this odd arrangement of thresholds and persons is out of alignment with what is expected of a balanced assessment. The items as originally conceptualised are not distinguishing the intended range of proficiency levels in this particular set of learners. The detailed discussion of thresholds, and the disordering of thresholds is not presented here. See Andrich (2012) for a detailed discussion.

Both the summary statistics (see Table 1) and the PITD (see Figure 7) indicate some disjuncture between the items and the persons suggesting that there are some anomalies in the data. Further investigation is required for both items and persons in terms of fit to the model⁸. In this article, and in the next section, we focus only on the possible anomalies that arise due to the scoring of items.

FIGURE 8: Item characteristic curve (Item 3.2.3a).

FIGURE 9: Category probability curves for Item 4.1.4.c.

The category probability curves for Item 1.3.2 (Figure 10) are used to illustrate disordered thresholds. Figure 10 shows that the location of the first threshold (τ₁) (the intersection of the curves for score 0 and score 1) has a difficulty of approximately 0.83 logits. However, the location of the second threshold (τ₂) (the intersection of the curves for a score 1 and score 2) has a difficulty of approximately –0.15 logits. The location of the third threshold (τ₃) (at the intersection of the curves of scores 2 and score 3) is approximately –0.68 logits.

FIGURE 10: Initial category probably curves for Item 1.3.2.

The disordering of the thresholds confounds the idea that thresholds between higher level categories are more difficult to attain than thresholds between lower order categories.

These initial analyses help us to identify possible anomalies and inconsistencies in the scoring rubrics, which can alert us to possibilities that should be considered when devising scoring rubrics. There are strategies we can use post hoc to adjust the item scoring in order to exhibit ordered thresholds as shown in Item 2.2.3a (Figure 1). The verification that scoring rubrics are functioning as expected, and subsequent revision where necessary, contribute to the reliability of the ML examination paper in the context of this set of 73 learners.

In the process of refining the instrument, iterative adjustments took place. The first step was to identify items that showed severe misfit according to the fit residual statistics and the chi square probability. In addition the ICCs were investigated. Where there were indications of anomalies we checked the item itself and the scoring rubric to identify any problems from both a mathematics education perspective and an assessment perspective. Where the qualitative investigation confirmed the anomaly and it was deemed proper to adjust the scoring of the item, this step was taken. The item statistics were then reinvestigated by reanalysing and by rechecking the fit to the model with the revised scores. If the fit had not improved we would reverse the change. If there was no theoretical reason to support the rescoring of an item, then no rescoring took place. The details of the various items, together with the marking memorandum provided by the Department of Education, and details of the rescoring appear in Appendix 2.

FIGURE 11: Initial category probability curve for Item 1.1.3.

FIGURE 12: Initial item characteristic curve for item 1.1.3.

The details of the item and rescoring appear in Box 1. In each case where the categories were not working according to the model, we applied multiple criteria before revising the scoring, keeping in mind principles of best test design (see Van Wyke & Andrich, 2006; Wright & Stone, 1979). After this rescoring process, both the category curves (Figure 5) and the ICC (Figure 6) indicated better fit according to the model.

Comparing Figure 11 and Figure 13, one can see that the categories are now working more appropriately.

Figure 14 shows the ICC for Item 1.1.3 after the rescore. When comparing Figure 12 and Figure 14, one can see that the fit between the observed and expected means for the persons in the four class intervals is much improved. A final check of the fit residual statistic and corresponding chi square values indicates that initially the fit residual statistic was –0.794 (chi square probability = 0.4317), whilst the final fit residual statistic is 0.082 (chi square probability = 0.8281), showing that the change affected the fit statistics positively. Also note that with RMT, a p-value higher than 0.01 for the fit statistics indicates that the difference between the observed and expected is not statistically discernible and hence that the model is working well.

FIGURE 13: Final category probability curve for Item 1.1.3

FIGURE 14: Final item characteristic curve for Item 1.1.3.

In a second process we investigated each ICC again, and then the category probability curves, and found that the thresholds were disordered for Items 1.2.1b, 1.2.2, 1.3.1, 1.3.2, 3.2.1, 3.2.3b, 4.1.4b, 4.1.4c, 5.1.4 and 5.3. (See Appendix 2 for details of these items.) At this stage, we re-examined the questions to see whether we could justify rescoring these items again. We also examined the fit residual statistics to help us decide whether a rescoring was necessary or not. A summary of these processes and decisions appears in Appendix 1.

We note as well that the differences observed in the new person-item location distribution (PILD) (Figure 15) are an improvement on the initial PILD (Figure 2).

FIGURE 15: Person-item location distribution after refinement to instrument.

Furthermore, one can see that the most difficult item, Item 5.1.3, was far too difficult for this group of learners. Its estimated difficulty level, calibrated at 5.448 logits, is much higher than the ability level of the most proficient learner in this test, who was estimated just above 1 logit. The items 4.1.2, 4.1.3, 4.2.1, 4.2.2, 5.1.4, 5.2, 2.1.2 and 3.2.3a were also beyond the ability level of the learners, but the difference δ_i – β_v for this group of items was not as large as it was in the case of Item 5.1.3.

Item 5.1.3 required learners to work out the price of an item before VAT, given the final price including VAT. The usual solution involves setting up an equation of the form A + 14% of A = Final price, and they were required to find A. This item required the setup and solution of an equation and involves applying algebraic techniques. The ML curriculum document makes it clear that skill in such algebraic manipulation is not a focus of ML. The document states: ‘As a rule of thumb, if the required calculations cannot be performed using a basic four-function calculator, then the calculation is in all likelihood not appropriate for Mathematical Literacy’(Department of Basic Education, 2011, p. 8).

Given such a stipulation, and noting that the item was far beyond the ability of this group of learners, it is recommended that a greater use of basic algebraic manipulation skills should be encouraged, to enable them to solve items such as Item 5.1.3. We note that this item could also be solved by informal methods based on reasoning about percentage change or using proportion and perhaps opportunities for such reasoning should also be encouraged.

On the other hand, three items, 4.2.3, 3.1.1 and 2.2.1a, were lower than the ability level of all the learners in the class, but the location of the items was less than 2 logits lower than the person with the lowest total score. This difference contrasts with the finding that in the higher ability ranking, one item was almost 5 logits higher than the location of the person with the highest total score, with two items being 2 logits or more higher and six other items being higher than the location of the person with the highest total score. We now present the person-item threshold distribution that was generated after the rescoring process.

There is a marked difference between the initial PITD (Figure 7) and the final PITD (Figure 16). The distribution of the thresholds prior to rescoring ranged between –22 and +22 logits (see Figure 7). Now after rescoring the threshold distribution has been limited to between –4 and 8 logits. The number of thresholds has been reduced from 150 to 75 by the rescoring processes (see Appendix 3).

FIGURE 16: Person item threshold distribution post-rescoring.

Another related issue was in the case of Item 4.1.4c, which asked for two causes of an observed change in the data. The scoring rubric allocated 2 marks per reason. The category curves in Figure 9 revealed that the categories 1 and 3 were redundant. No learner attained the first mark without getting the second mark and similarly no learner attained 3 marks without getting the fourth mark.

An important advantage of the Rasch analysis is that item difficulty and person proficiency are located on the same scale. Without checking individual item validity, we cannot take the results at face value; instead, we need to verify the item fit. Now that we have subjected the scoring of each item to Rasch analysis and qualitatively investigated the structure of the scoring rubrics, we argue that the new scoring procedures for the instrument allow greater precision than the original instrument scoring. It must be emphasised that the scoring rubrics were not the only threats to validity in this test.

The process that was conducted, which involved systematically rescoring each of the assessment items, has revealed the important role of scoring rubrics in contributing to the validity of assessments. Examiners need to ensure that each mark that is allocated can be justified educationally. Marks should not be allocated on the basis of time that is required to be spent during the examination. Neither should marks be allocated for guessing. If these checks are not taken into account, then the total score has diminished meaning. If the total score is indicative of a position on a unidimensional scale, then differences between the score must reflect differences between proficiency levels of learners.

A further process that could be followed is to study which items were experienced as more difficult and which items were experienced as easier in order to obtain a description of items on different levels of the scale. It may then be possible to identify different demands of the various questions. A study of a similar nature has been conducted by Long (2011) who presented a comprehensive application of RMT to the multiplicative conceptual field. In her study, Long used the Rasch model to develop clusters of proficiency zones along the continuum representing the alignment of person proficiency and item difficulty.

It may be the case that particular items are mathematically demanding because they require sophisticated use of algebraic tools (as in Item 5.1.3). Perhaps the increasing demand of the questions may be related to the contexts that were used. These analyses will be conducted and reported in a follow-up article.

Andrich, D. (1988). Rasch models for measurement. Newbury Park, CA: SAGE Publications.

Andrich, D. (2012). An expanded derivation of the threshold structure of the Polytomous Rasch Model that dispels any “threshold disorder controversy”. Educational and Psychological Measurement, 73(1), 78–124. http://dx.doi.org/10.1177/0013164412450877

Andrich, D., Sheridan, B., & Luo, G. (2011). RUMM2030 software and manuals. Perth, Australia: University of Western Australia. Available from http://www.rummlab.com.au/

Debba, R. (2012). An exploration of the strategies used by Grade 12 Mathematical Literacy learners when answering mathematical literacy examination questions based on a variety of real-life contexts. Unpublished master’s thesis. University of KwaZulu-Natal, Durban, South Africa.

Department of Basic Education (DBE). (2011). Curriculum and assessment policy statement (CAPS). Mathematics Grades 10–12. Pretoria: DBE.

Department of Education (DOE). (2003) National curriculum statements Grades R–9 (schools). Pretoria: DOE.

Douglas, G. (1982). Issues in the fit of data to psychometric models. Education Research and Perspectives, 9(1), 32–43.

Dunne, T., Long, C., Craig, T., & Venter, E. (2012). Meeting the requirements of both classroom-based and systemic assessment of mathematics proficiency: The potential of Rasch measurement theory. Pythagoras, 33(3), Art. #19, 16 pages. http://dx.doi.org/10.4102/pythagoras.v33i3.19

Humphry, S.M. (2005). Maintaining a common arbitrary unit in social measurement. Unpublished doctoral dissertation. Murdoch University, Perth, Australia.

Humphry, S.M., & Andrich, D. (2008). Understanding the unit in the Rasch Model. Journal of Applied Measurement, 9(3), 249–264.

Long, C. (2011). Mathematical, cognitive and didactic elements of the multiplicative conceptual field investigated within a Rasch assessment and measurement framework. Unpublished doctoral dissertation. University of Cape Town, Cape Town, South Africa. Available from http://web.up.ac.za/sitefiles/file/43/314/Long,_M__C__(2011)__The_multiplicative_conceptual_field_investigated_within_a_Rasch_measurement_framework_.PDF

Masters G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174. http://dx.doi.org/10.1007/BF02296272

Matters, G. (2009). A problematic leap in the use of test data: From performance to inference. In C. Wyatt-Smith, & J.J. Cumming (Eds.), Educational assessment in the 21st century: Connecting theory and practice (pp. 209−225). Dordrecht: Springer. http://dx.doi.org/10.1007/978-1-4020-9964-9_11

Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (expanded edn. with foreword and afterword by B.D. Wright). Chicago, IL: University of Chicago Press. (Original work published 1960)

Van Wyke, J., & Andrich, D. (2006). A typology of polytomously scored items disclosed by the Rasch model: Implications for constructing a continuum of achievement. Perth: Murdoch University.

Wright, B.D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, Winter, 33–45.

Wright, B.D., & Stone, M.H. (1979). The measurement model. In B.D Wright, & M.H Stone (Eds.), Best test design (pp. 1–17). Chicago, IL: MESA Press.

Wright, B.D., & Stone, M.H. (1999). Measurement essentials. Wilmington, DE: WIDE Range, Inc.

2. Missing responses were allocated zeros.

3. See Dunne et al. (2012) for details of the analysis of dichotomous items.

4. In this study we use the Rasch partial credit model, which is the default model in the RUMM 2030 software.

6. The discussion here will concern the dichotomous model. Extensions of the model have been derived from this model for partial credit scoring by Masters (1982) and rating scales by Andrich (1978).

7. This mismatch is in itself not a problem; however, more information could be gleaned from a test situation that is better targeted.