|
Orientation: This is the final article in the special edition on human resource (HR) programmes and evaluation. Its starting point is that programme evaluation is the application of a wide range of social science research methods that provide credible information about the need, use, planning, effectiveness and cost of a programme. Research purpose: The purpose of this article is to review the other articles in this volume, and to draw out general conclusions about their contributions to knowledge in the field. Motivation for the study: If evaluations are undertaken in the HR domain in South Africa, they remain mostly unpublished, and thus cannot contribute to a knowledge base for the field. Research design, approach and method: This article provides a theory-based approach to programme evaluation. The seven articles were analysed in terms of two major functions of programme evaluation, namely to ask ‘How does a programme work?’, and ‘Does it work?‘ Main findings: Eight overarching themes are identified in the articles included in this volume. Practical/managerial implications: The evidence discussed here can be used to make better decisions, promote organisational learning, improve practice, and enhance employee wellbeing. Contribution/value-add: The main contribution of this concluding article is its argument that research and theory in this field can enhance the work of HR professionals, by providing evidence about how ‘good’ a programme is, and why it is good. This adds substantial value in a world characterised by accountability and evidence-based practice.
Evaluations in the corporate sector are seldom made public. Reports of studies about the effectiveness of training programmes, mentoring and coaching exercises, employee wellness interventions, leadership development, and other interventions, more often than not remain in-house, and are not accessible to others outside the organisations that commissioned the studies. As a result, it is difficult to develop a knowledge base about these interventions. Without access to findings about them, it is impossible to synthesise or accumulate knowledge about good practices, dependable relationships, or strong effects. This volume, as the first collection of evaluations of human resource (HR) programmes in South Africa, is a major step to improve practice through evaluation. In this article I provide a critical reflection about the articles contained in this volume, identify common themes, and draw out potential contributions to human resource management. The value of this special edition lies in: • collecting these evaluations in a single volume for readers, that covers an area in which little evaluation has been undertaken
• containing user-friendly examples of how to go about evaluating HR programmes
• showing the variety of evaluation methods that can be used for evaluating HR programmes
• exhibiting the first examples of theory-driven evaluations in HR. As a result of reading these articles with contemporary programme evaluation knowledge in mind, at least seven themes stand out: • The authors follow a common approach (programme theory-driven evaluation), and this strengthens the usefulness of their work.
• All of the articles include careful consideration of programme activities, and this creates the possibility of getting to best practices.
• Literature (or research) reviews enable practitioners to accumulate practical knowledge of programmes.
• Human resource practitioners are not familiar with thinking in evaluation terms, and studies like these could make useful contributions to change this, and to provoke practitioners to
think evaluatively.
• The evidence base for selecting specific HR interventions is generally quite low.
• Programme evaluation can play a significant role in planning HR interventions, and this is not sufficiently recognised in the field.
• As a transdiscipline, programme evaluation can enrich applied fields such as HR practice. In the rest of this article I will expand on these themes, and argue more generally that programme evaluation reports of HR interventions, as described in the articles, can play a significant role in the improvement of practices in the field.
Current theoretical perspectives: A theory-based approach
The strength of all the studies included in this volume is that they stick to a programme theory-driven approach. Chen (1990, p. 415) gave a straightforward description of what programme theory is: ‘... a set of implicit or explicit assumptions of how the program should be organized and why the program is expected to work.’ Rossi, Lipsey and Freeman (2004) expanded on these two aspects: what Chen calls programme organisation they call process theory. This includes all aspects of the implementation of the programme: its target population; how these individuals will be reached; how they interact with the programme facilities, personnel, administration, facilities; and other aspects of implementation. The second aspect of Chen’s definition is the programme’s impact theory. For Rossi et al. (2004): the cause and effect sequence maps the proximal and distal outcomes of the programme’s desired ends.The authors in this edition generally follow Donaldson (2007), who gave detailed accounts of the complexities and challenges of systematically following a particular approach to evaluations in the real world. Thus, this volume, in a sense, is an HR companion to Donaldson’s more general work, and fills the gap in our understanding about what happens in evaluation practice in this field. (It might be more accurate to say ‘what COULD happen’, because these practices are far from routine in the field, as the studies show). The importance and usefulness of a theory-based approach to human resource programme evaluation are clear from these articles. For a start, they all emphasise how important it is simply to describe the intended activities (process) and desired outcomes (impact). In many of the programmes an explicit programme description was not present, and therefore it was difficult to know what was supposed to happen in the programme. In the contribution by Hendricks and Louw-Potgieter (this edition), for example, they show how a modest programme theory led to the development of sparse programme activities, which did not produce the outcomes envisaged by the programme manager. They re-direct programme planners to develop a more plausible programme theory and to extend programme activities to establish the link between intended outcomes. In most cases programmes gave very little attention to outcomes, and if they did, it was to relatively short-term outcomes (see Rundare & Goodman, in this edition). Brinkerhoff (2003) has correctly identified longer-range outcomes in human resource as higher-level business goals, often referred to as strategic goals or objectives. If HR professionals could show how their activities are connected to these overall business needs, via programme theory building, we believe their work will be better integrated with what the rest of the organisation is trying to achieve. The article by Field and Louw (this edition) provides a more extensive list of reasons why programme theory-driven evaluation is useful, as well as some arguments by critics of this approach. The authors follow the best advice on how to extract programme theory, which is namely to engage and interact with programme stakeholders first (Rossi et al., 2004, p. 162). In some instances this is all that they had to go on, as other sources of information about programme theory were non-existent (programme documents and records), or impossible to implement (site visits and observation). However, as I will discuss below, an important further step is to link the emerging programme theory to what is known about the topic in the literature, and in this the authors followed many of the good practices suggested by Donaldson (2007). A further major benefit of describing and systematising the programme in this manner is that it creates (or encourages) consensus amongst the stakeholders. Over time programme planners, staff members, management, and funders may drift apart in their understanding of how the programme is supposed to work, and this exercise brings them together again (for example, in some cases different stakeholders, in fact, were never in agreement about what the programme is supposed to accomplish). Engaging different stakeholders also enables them to participate meaningfully in the evaluation itself, and increases the chances that the evaluation results will be used by the organisation. Stakeholders, such as programme planners and managers, together with programme documentation, are important sources of programme theory. Another source, of course, is social science theories and research, and quite a few authors used the social science literature to ask questions about the programme theory. They asked, for instance, ‘Given the programme-as-designed, what does the literature tell us about the plausibility of this programme achieving its intended outcomes?’ This is a further illustration of the usefulness of taking the existing literature seriously and, in all instances, where the plausibility question was raised, the authors’ considerations were useful to the programme. The strength of social science research is that much of it exists in the HR and organisational psychology literature. If programme managers engage more with this research (or employ contractors to show them what the research says), they could generate plausible programme theories and design effective programme activities based on such theories. Thus, programme theory-based evaluation, in Donaldson’s (2007, p. 10) terms, is a versatile programme evaluation approach that has been used successfully to develop and improve programmes, aid decision-making, facilitate organisational learning, create new knowledge, and meet transparency and accountability needs.
Best practices
A few evaluations in the present edition went one step further than just elucidating programme theory, in that they compared the programme as intended (theory) to the programme as delivered (for instance, Buys & Louw; Salie & Schlechter, in this edition). In other words, they added an implementation component to the evaluation, namely how the programme actually works. Thus, some of them could show where implementation steps were missing, weakly implemented, or even wrongly conceptualised. As a result, all of the evaluations paid much attention to programme activities. This, at first glance, may seem rather obvious, but programmes are often remarkably vague about which activities constitute a programme. Thus, the simple exercise followed by the present authors, of constructing programme process theory by asking questions about programme activities and resources, clarified matters for the relevant stakeholders. There is, however, a more important motivation for focusing on these aspects of a programme. Ideally we would like to determine the activities that make up good practice in a field of study. For example, which activities should be included in training programmes or performance management programmes? In addition, we would like to know about the requirements for implementation, how the programme activities should be delivered (and aligned) to maximise chances of having an effect. Joseph, Emmett and Louw-Potgieter (this edition) for example used an implementation evaluation to show how a pay-for-pe rformance programme had little effect because of its flawed implementation. These authors were able to extract the essential variables for the successful implementation of such a programme.
Not reinventing the wheel
The potential benefits of carefully considering existing knowledge in a field are obvious, and this reflects the importance of reviewing the literature. Programme evaluators often focus on a slightly different knowledge base to HR professionals, and look for indications of effectiveness rather than instructions on how to perform things. All the articles in this volume include literature reviews that were narrowly focused on what the existing knowledge base can tell HR practitioners about what works for whom and under which circumstances.Brinkerhoff and Montesino (1995), for example, identified low rates of transfer from corporate education to the work environment as a major factor that limits the effectiveness of training. This, in itself, is a major finding from the literature, the knowledge of which will assist the training department to increase the effectiveness of training programmes. However, the literature is even more useful in this regard: it spells out what the factors are that encourage the transfer of training. These include the alignment of training programmes with the strategic direction of the organisation; the degree of connection between training professionals, the trainees themselves, and line managers of the trainees; and attention to the phases before and after the training. Therefore, instead of focusing so closely on the design and delivery of learning interventions, these authors argue that training professionals can increase the effectiveness of the training by attending to issues of the transfer of training. Beets and Goodman, and Rundare and Goodman (this edition) are cases in point. Literature (or research) reviews enable us to accumulate knowledge about what works in a field, and how and why it works. The information we have about HR interventions is too often of a piecemeal nature, difficult to collect and even more difficult to integrate. The publication of these studies is a significant step to counteract this tendency, and to promote the accumulation of practical knowledge.
Educating practitioners to think evaluatively
The experience of all the authors in conducting these studies was that HR people were not used to evaluations. In two rather extreme instances it even extended to the extent that data were not made available to evaluators, despite undertakings to the contrary (for instance Field & Louw; Salie & Schlechter, this edition). It is reasonable to expect some form of anxiety when programmes are evaluated, and it might be that a few programme managers were excessively anxious about this. Hopefully, this reaction will abate as HR stakeholders and programme managers become more familiar with evaluation, perhaps simply through participating in evaluation. This is often referred to as process use, ‘… cognitive, behavioral, program, and other organizational changes resulting, either directly or indirectly, from engagement in the evaluation process and learning to think evaluatively …’ (Patton, 2008, p. 108). Patton identified some of the positive consequences of this, for example: • the organisation’s culture becomes infused with evaluative thinking
• there are enhanced shared understandings about the programme’s model and expected outcomes
• the intervention is supported and reinforced
• other additional consequences.
In addition, evaluators working in the South African HR field may have to plan quite specifically to reduce extreme evaluation anxiety. Donaldson, Gooler and Scriven (2002) identified a number of signs that indicate possible excessive evaluation anxiety: • accusing evaluators of hidden agendas
• avoiding or refusing to work with evaluators
• stalling to use evaluation results
• amongst others signs. In addition, these authors provide a number of useful strategies to deal with this, such as expecting and accepting it right from the start, providing balanced continuous feedback for improvement, and providing role clarification on an ongoing basis. The lack of monitoring data for evaluation is evident in almost all of these programmes. Because of this, almost all evaluators had to collect post hoc data and stay with formative or implementation evaluations. Frequently, the evaluators were in fact responsible for collating this information in a meaningful way, and the impression remained that not much would have been undertaken with it were it not for the evaluation. As a result, feedback, in the organisation about the programme, is often unsystematic and haphazard. It leaves very little room to make recommendations about improving, for example, the number, type, timing, and presentation of programme activities. In other words, feedback is given on the basis of hunches or impressions, whilst we have argued that it is best based on what the literature says and findings from the evaluation. Programme staff must be convinced of the importance of keeping accurate records – for their internal evaluation, if for nothing else. The focus on programme theory in an evaluation draws attention to programme outcomes. Given the unsettled nature of the programmes included in this volume, no outcome evaluations were conducted (although a few evaluations used proximal outcomes – like Buys & Louw; Beets & Goodman, this edition). We have seen no evaluations of long-term outcomes and, despite our limited exposure to HR interventions in this volume, we see little evidence that this will materialise in the HR field in the near future.
The evidence base for human resource interventions
Reading through the articles, one cannot help but wonder: On what grounds were these HR interventions introduced in the organisations concerned? and, Based on what evidence were decisions made to continue or discontinue the interventions? Almost all of the authors commented on some or all of the following: • the absence of a clear problem definition,
• a conceptualisation of programmes and what it could do
• systematic data on implementation
• the specific outcomes that could be taken as an indication of success of the programme. In other words, in a world that increasingly values evidence-based decision-making, these cases cast doubt on the quality of information about South African HR practice. There is at least one international study that would strengthen this impression. Proudfoot Consulting (2008) found that South African companies train their personnel more extensively than their international counterparts, but do very little to find out whether or not the training is in any way useful. There is no need to make the case here for evidence-based management, but one has to ask, If business decisions are not based on empirical evidence, what then drives these decisions? Is it: • conventional wisdom?
• assumptions about the right decisions?
• copying what seems to work for other organisations?
• management fads?
• quick fixes based on success stories from management gurus? Whatever the case might be, these articles ask HR practitioners and management to pause and consider the evidence for introducing or continuing with a programme. For example, when a training programme is introduced, what evidence is there that it is the best answer to a well-defined and understood problem in the organisation? Which known alternatives were considered instead of training? Training normally assumes that poor performance is the result of a lack of knowledge, skills, or attitudes – but what if the problem is one of low intrinsic motivation, which would perhaps require a job enrichment scheme to improve performance? It is sometimes said that programme evaluators ask programme staff to hit the ‘pause’ button; to stop and consider what is being undertaken in response to a perceived problem. This is possibly the less obvious lesson to be learned from these empirical studies.
Evaluation as a planning tool
In all of the programmes discussed in the previous articles, evaluation was introduced at some time into the life of the programme. This is probably the default option, not just in HR interventions, but also in programmes of all types. Almost all introductory textbooks on programme evaluation (e.g. Rossi et al., 2004), however, argue for evaluators to become involved in a programme much earlier in its life cycle. Thus, evaluation is of significant value in terms of designing and delivering a programme rather than as an activity that kicks in halfway through delivery of the programme, or only at its end. In other words, evaluation can also be a prospective activity, to be used as a planning tool, and not just as an activity that comes at the end of the process. When designing a programme, for example, a programme evaluator can assist programme planners to think through the problem that the programme is supposed to address and consider: • the elements that should be included
• the way in which the activities should be delivered
• what can be expected to change after a specified period of time
• other related problems. Field and Louw (this edition) illustrated this well in their recommendations regarding design and implementation of a wellness programme.
Evaluation as a transdiscipline
Evaluation is increasingly regarded as a transdiscipline, and one ‘that has standalone status as a discipline and is also used as a methodological or analytical tool in several other disciplines’ (Scriven, 2008, p. 65, emphasis in original). In an earlier piece, Scriven argued that these ’… disciplines are notable because they supply essential tools for other disciplines, while retaining an autonomous structure and research effort of their own‘ (2003, p. 19).The articles in the present edition, firstly, showcase knowledge, skills and methods that are specific to the discipline of evaluation. Secondly, and more importantly, they show how the research and theory of this field can enhance the work of HR professionals, by providing evidence about how ‘good’ a programme is as well as why it is good. In a corporate environment, characterised by accountability and evidence-based practice, this is a substantial value added to their work. Such evidence can be used to make better decisions, promote organisational learning and effectiveness, improve practice, and enhance employee wellbeing (Donaldson, 2007, p. 240). To paraphrase Michael Scriven’s (2003) thoughts about the possibilities of evaluation to enrich applied fields like HR: The human resources field will divide into the progressive, evaluation-enriched school, and the conservative, evaluation-impaired school. The evaluation enriched group … will become the winner in nearly all bids for resources and contracts aimed at separating solutions from non-solutions of corporate problems. The evaluation-impaired branch will gradually wither on the vine, with its aging adherents exchanging stories about the good old days. (pp. 19–20)
Competing interests
The author declares that he has no financial or personal relationship(s) which may have inappropriately influenced him in writing this paper.
Brinkerhoff, R. (2003). The success case method. San Francisco: Berrett-Koehler.Brinkerhoff, R.O., & Montesino, M.U. (1995). Partnerships for training transfer: Lessons from a corporate study. Human Resource Development Quarterly, 6(3), 263–274.
http://dx.doi.org/10.1002/hrdq.3920060305 Chen, H.T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage. Donaldson, S.I. (2007). Program theory-driven evaluation science: Strategies and applications. New York: Lawrence Erlbaum Associates. Donaldson, S., Gooler, L., & Scriven, M. (2002). Strategies for evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation, 23(3), 261–273. Patton, M.Q. (2008). Utilization-focused evaluation. (edn.). Los Angeles: Sage. Proudfoot Consulting. (2008). A world of unrealised opportunities. Global Productivity Report. Retrieved August 30, 2011, from
www.alexanderproudfoot.com/WorkArea//DownloadAsset.aspx?id=254 Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation. A systematic approach. (7th edn.). Thousand Oaks, CA: Sage. Scriven, M. (2003). Evaluation in the new millennium: The transdisciplinary vision. In S.I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium, (pp. 19–42). Mahwah, NJ: Lawrence Erlbaum. Scriven, M. (2008). The concept of a transdiscipline: And of evaluation as a transdiscipline. Journal of MultiDisciplinary Evaluation, 5(10), 65–66.
|