|
|
| |||||||||||||||||||||||||||
• Introduction • Key focus of the study • Background to the study: Programme description • Literature review • Evaluation question • The potential value-add of the study • Research design • Research approach • Research method • Research participants • Measuring instruments • Research procedure • Statistical and other analyses • Results • Success cases • What knowledge, skills and attitudes were applied from the training? • What results were achieved as a consequence of the programme? • What good has the training done? • What has helped trainees to apply the knowledge, skills and attitudes that they acquired on the programme? • What barriers prevented successful application of the coach training knowledge, skills and attitudes? • How could the coaching training programme be improved? • Ethical considerations • Potential benefits and hazards • Recruitment procedures • Informed consent • Data protection • Trustworthiness • Reliability • Validity • Discussion • The scope of the success of the training intervention • Worthwhile results from the training • Suggestions for improving the programme • Environmental factors that hindered and helped the successful application of knowledge, skills and attitudes • Conclusion • Limitations • Suggestions for future research • Acknowledgements • Competing interests • Authors’ contributions • References |
|||||||||||||||||||||||||||
Orientation: The evaluation of training programmes provides methodological and logistical challenges to evaluators and human resource (HR) managers. The training of executive coaches is no exception in this regard. Research purpose: The study aimed to investigate one particular aspect of the results of an executive coach training programme, and the extent to which knowledge, skills and attitudes acquired during the programme were applied in practical settings. Motivation for the study: Too little is known in South Africa about the effectiveness of training programmes, including executive coach training programmes. There is a need to demonstrate methodological approaches that would provide valid and reliable data. Research design, approach and method: The success case method (SCM) was used to guide the study, consisting first of a survey of 80 participants in the training programme, followed by eight interviews to compare successful with less successful cases of skills transfer. Main findings: All six successful coaches were applying the proximal outcomes from the training with good results, with several valuable consequences resulting from the training. Barriers to successful implementation included personal circumstances and unfulfilled expectations of the programme content. Practical/managerial implications: Aspects of the training programme that could be improved included: the buddy selection system, more individualised feedback about self-development, closer supervision, and more support from programme managers. Contribution/value-add: This evaluation contributes to the evaluation literature by providing a documented exploration of a systematic application of the SCM. It also contributes to the coach training literature by providing a systematic evaluation of a coach training intervention in South Africa.
Key focus of the study Executive coaching is a rapidly growing field in management. Its practices are expanding and it is associated with an extensive quantity of literature. In this article we describe a training programme for executive coaches, and examine whether or not trainees report their use of the assumed benefits in their work. There is a substantial range of methods available to evaluate training programmes and, in this article, we utilise an innovative method to assess the successful and non-successful application of learning. Background to the study: Programme description The coach training programme (CTP) runs over six months and its target population is defined as people who want to become professional coaches. Qualified and experienced facilitators run the programme. Two philosophies underlie the programme: integral coaching and andragogy. Integral coaching refers to coaching that takes into account all aspects of a person’s life (not just work) (Flaherty & Handelsman, 2004). Andragogy is an adult educational philosophy which encourages learners to become autonomous and self-directed (Bedi, 2004; Ozuah, 2005; Zemke, 2002). The programme content can be categorised into two developmental pathways: personal development, and the development of coaching skills and competencies. Theoretical coaching content and skills are inculcated during three training sessions (86 hours in total). Interspersed between these three sessions, the trainees engage in self-development exercises, which could consist of readings, writing personal letters or written assignments. Briefly, the programme content comprises the following activities presented in sequence: • a two-day introductory course on coaching At the end of the programme the facilitators assess the competence of the trainees and decide whether to award a certificate or not. The programme activities of the CTP focus on building knowledge, skills and attitudes (KSAs) for coaching, plus the fostering of personal development through increased self-awareness. Figure 1 presents the impact model for the coach training programme, extracted from personal communications with programme staff and programme records.
In Table 1, the proximal outcomes of deepened coaching skills and expanded self-awareness are mapped.
Literature review Executive coaching originated in the USA and has become a rapidly developing and a fast growing industry (International Coaching Federation [ICF], 2007; Upton, 2006). In South Africa, executive coaching has come to the fore as a development tool over the past six years (Rostron, 2006). As with all new knowledge domains it has attracted its fair share of contesting groupings, from academics, to ‘professional’ coaches, to unscrupulous practitioners. In South Africa, executive coaching is not a statutory profession and there is little agreement regarding definitions, theories, models, constructs, standards and practice guidelines within the domain (Bluckert, 2006; Executive Coaching Forum, 2004). A body called the Coaches and Mentors of South Africa (COMENSA) was formed in 2004 to regulate coaching, develop credibility and awareness of coaching, empower the consumer and standardise the skill requirements for coaches (COMENSA, 2007a). Unit standards for coaching were registered on the National Qualifications Framework (NQF), in the field of Education, Training and Development, in 2003 (South African Quality Authority [SAQA], 2007) and, COMENSA‘s Standards of Professional Competence were developed in 2006 (COMENSA, 2007b). To date, no research has been conducted to ascertain whether or not coaching delivers the intended results described on websites and in advertisements for coaching training. By employing a rigorous method, the current evaluation sought to answer two questions: how does executive coaching training work; and does it work? Evaluation question The CTP is popular with a diverse group of recipients. We were interested in what the trainees were achieving with the KSAs and self-awareness that they attained on the programme, and we decided to focus the evaluation on the following question: To what extent are the trainees using the intended proximal outcomes to enhance or develop their coaching practice? The potential value-add of the study Training and development practitioners often firmly believe that what they are achieving is worthwhile and valuable, but they lack the methods to provide credible and valid evidence of the impact of their efforts. In this article we demonstrate the use of one method that could assist in this regard, namely the Success Case Method (SCM), developed by Brinkerhoff (1983, 2003, 2005, 2006a, 2006b). The SCM was used to evaluate the CTP described above, and provided practically useful, and reliable information about the programme.
Research approach The SCM follows a mixed method approach: a sample of success and non-success cases was selected by means of a quantitative questionnaire and then interviewed to find the reasons for their success or lack thereof. The findings of the interviews are reported as qualitative data. According to Brinkerhoff (1989) the real value of a training intervention lies in the application of the KSAs presented during the training. The SCM is designed to answer the following evaluation questions: a. What has actually happened since the intervention? (i.e. who is using the training in their work; what aspects of the training are being used; who is successfully using the training; who is not using the training successfully?) b. What results are being achieved post-intervention? (i.e. what outcomes are being achieved; are these outcomes associated with the original programme objectives; are there any unintended results?) c. What is the value of the results? (i.e. has the training produced meaningful business results?) d. How can the training be improved? (i.e. what aspects of the training are working well and what areas need to be changed; what barriers prohibit learning transfer; how can one increase the percentage of successful cases?) According to Brinkerhoff (2003, 2006a) the SCM evaluation process begins, firstly, by identifying the focus of the evaluation and mapping out the evaluation plan. Secondly, an impact model is designed that represents the intended outcomes of the training intervention for the trainees. Thirdly, once the impact model has been defined, and approved by different stakeholders, the evaluator designs a short survey aimed at identifying success and non-success cases. This is generally a succinct document containing a limited number of questions related to the intended outcomes of the programme, as defined in the impact model. Fourthly, the respondents who scored the highest (success cases) and lowest (non-success cases) on the survey are then selected for in-depth interviews. The interviews explore the nature of the interviewees’ experiences post-training and aim to develop a coherent picture of how the training was used, what results were achieved, as well as which factors supported and which barriers inhibited successful training transfer. Research method Research participants For the first stage of the research, a purposive sampling strategy was employed (Trochim, 2006). The entire population of 80 participants who completed the CTP in 2005 (n = 32) and 2006 (n = 48) were contacted to complete a survey. The final sample consisted of 55 accredited coaches, ranging in age from 32 to 61 years with an average age of 45. Forty-six percent of the sample was female and 53% was male. On average, coaches were well educated (64% had postgraduate degrees, 20% had first degrees, and the rest had diplomas or school leaving certificates). Participants were in job positions that they referred to as coaching positions (38.2%), or management positions (31%) or they were business owners (11%). Eighteen percent stated that they were in other positions than those mentioned above. Of the 55 respondents, 55% were coaching full-time and 44% were using coaching skills as part of their job. Most recipients completed the coach training with the intention of becoming full-time coaches (58%), whilst 47% indicated that they intended to apply the coaching skills as part of their current job. In line with the guidelines for the SCM, the survey was followed by interviews of success and non-success cases. Of the 55 respondents who completed the survey, 20 indicated that they were applying the training outcomes and obtaining positive results. Six of them were chosen for the high success case interviews, based on the highest survey scores, contact details supplied and the number of clients coached. Initially, three respondents were chosen as low success cases, but one had to be discarded as a brief telephone interview revealed moderate rather than low success. Measuring instruments A survey instrument, based on the proximal outcomes in Table 1, was designed to identify success and non-success cases. The survey included three sections: applications of specific proximal outcomes of the training, questions of general interest and a demographics section (a copy of the survey is available from the evaluators). The six success cases and two low success cases were interviewed by means of a Protocol Conceptual Model (Brinkerhoff, 2003, 2006a) of ‘filling buckets’. Each bucket represented a specific category of information. For high success cases people were asked: • What was used? Low success cases were simply asked about barriers to using the training, and suggestions for improvement. Research procedure An online survey was sent to all participants who had completed the 2005 and 2006 coaching programmes. Each interviewee was informed via e-mail of the upcoming interview and given information about the content that would be covered during this process. All interviews were conducted by telephone and the length of the success interviews ranged between 30 minutes to 50 minutes. Low success interviews were more focused and typically lasted 20 minutes. All interviews were digitally recorded and then transcribed. Statistical and other analyses The survey responses on the section, Application of proximal outcomes, were scored according to the format presented in Table 2.
Survey scores were used to select highest and lowest success cases. The interview data were reduced by means of naming codes, and these codes were further reduced by using themes (Miles & Huberman, 1994).
Table 3 presents the percentage of respondents who indicated that they had applied the course outcomes in the workplace with clear and positive results. The majority of respondents attained positive results with all the proximal training outcomes.
Success cases The qualitative data analysis revealed that all six success case coaches were successfully applying the proximal outcomes from the training. They saw the results and the value of this training. Furthermore, they were able to identify the aspects that helped and that hindered the results during the application of the knowledge, skills and attitudes (KSAs) and they could make suggestions for improving the training. This is summarised in Figure 2.
The results are reported in terms of the six themes that emerged from the data analysis of the interviews. What knowledge, skills and attitudes were applied from the training? All six coaches reported that they were using the process to successfully design and conduct coaching programmes. They also referred to applying structured coaching conversations as taught during the programme. This rigorous structure separates a coaching conversation from just another ‘pleasant conversation’ and helps to move the clients closer to their intended outcomes. Five coaches described how applying assessment models had helped them to make grounded assessments of where ‘the individual actually is’. Three success cases pointed to the usefulness of understanding and applying language as imparted on the programme. This quotation below illustrates how this outcome was used: My powers of observation are significantly higher and … I now see language, I feel language, whereas before perhaps I just heard, it’s more three dimensional now for me, and that is the power of observing. (Success Case 6) Two of the coaches highlighted the usefulness of the integral coaching paradigm: Because it is ontological coaching … it looks at you as a whole human being … I’m absolutely committed to having an integrated approach to how I deal with people. (Success Case 6) What results were achieved as a consequence of the programme? All six success case coaches claimed that they experienced significant personal development as a result of the programme. The following quotations illustrate this: The personal development aspect was very different from other training and studying that one does, because lots of the work that is assigned is to … it’s meaningful for me to be exposed to this type of … personal growth opportunity and … the consequence is I help Five success cases indicated that the insight they gained from the programme had improved their interactions with other people and specifically with their clients. They mentioned the skills of listening in a different way, being non-judgmental, using open-ended questions and refraining from having the last word. The following quotations illustrate these results: I have better relationships with people…the ability to listen differently to people, that … makes an impact on my clients…and I had always taken on everyone’s stuff and woes … which is really sapping. [The training] gave me a new set of skills for dealing Two of the success cases indicated that their practices had benefited from their link with the training provider. Another success case explained that the training had led to increased business opportunities. Three success cases mentioned shifts within their own careers, which they attributed partly to the training. One of these success cases changed jobs: from an information technology position to one in human resources. This move was attributed mainly to the personal development process experienced during and after the training. An unintended consequence of the training for three of the success cases was the ability to use the KSAs from the coach training as a leadership style. This involved a shift in managing people. This shift was described as follows: I think it’s quite different … coaching as leadership style versus leading without having been exposed to all the concepts [on the One of the success cases recommended that all managers should be exposed to one of the short coaching workshops, which the service provider offered in order to improve their people management skills. What good has the training done? All six success cases reported that, when applying the KSAs from the training programme, they had observed positive changes in their clients. The following examples illustrate this: They achieved their objectives and their performance improved. (Success Case 3) You can see other people’s lives changing as a result of being coached. (Success Case 1) You see a change in clients’ motivation, how they [present themselves] and the responsibility that they take for their own behaviour All six success cases indicated that they aim to assist their clients to become ‘self-correcting, self-generating, long-term excellent performers’. These are important outcomes of the training. Two success cases reinforced this idea by mentioning the career changes that their clients made. The following quotation illustrates this: His whole dream was to go into a different division at work and he got the job he wanted … it was a great feeling of success from a coaching All six success cases confirmed that their own observations of shifts in clients’ behaviour, documented coaching conversations, feedback from clients and feedback from clients’ colleagues provided evidence of the good that the training programme had done. Five of the six success cases reported that the training encouraged them to further their studies. For four of them this meant undertaking the next level of the coach programme, whilst for one of them it involved a decision to begin postgraduate studies. This success case said: The way that [he] does the training, it just makes you curious to want to learn more and that’s been a wonderful gift for me because All six success cases reported that the training improved their credibility as coaches. They indicated that this credibility was related to the accreditation of the programme: I think the training provides discipline and rigour in a market where entry [requirements] are very low and there are lots of people out there who Two success cases highlighted how the training had led to important networking opportunities. They met like-minded people on the programme, with whom they have remained in contact. What has helped trainees to apply the knowledge, skills and attitudes that they acquired on the programme? All six interviewees indicated that a background in a business-related field was advantageous for successful application of the programme’s KSAs. However, it was stressed that such knowledge was not a prerequisite: I think personally, if I didn’t have the industrial [psychology] background, I would [have experienced] a lot of gaps. It’s not that Two success cases were of the opinion that a business-related background was more useful than a clinical psychology background: [Clinical psychologists] don’t have a feel for the corporate stresses, politics, hierarchies and structures. That’s a huge part of Five of the six coaches discussed how reading coaching-related and personal development books had helped them develop as coaches. Not only did these readings improve their knowledge, but they also served as reference texts: When you’re working with someone you know a little switch flicks in your brain: oh, hang on, I think this might be good and you go and have Five of the success cases referred to the supervision provided during the programme as being helpful to maintain objectivity, or to provide benchmarks of excellence: Coaches can get caught up in our own structures of interpretation. (Success Case 5) One of the things about coaching is you can only go as far as you’ve been personally, and you only get to go places through personal Two success cases mentioned that the support they received from their managers to apply the coaching KSAs back at work, was ‘beneficial’ and ‘critical’. What barriers prevented successful application of the coach training knowledge, skills and attitudes? One low success case had not found the opportunity to apply the KSAs to any meaningful extent as a result of personal circumstances. She reported that the training was useful, but that other priorities had taken precedence since the course. The second low success case had expected that the training would be specifically tailored for working in a corporate environment, as she had understood from the promotional material. The coach expected to leave the programme feeling confident to enter the corporate world as a coach, but found that she had not gained this confidence: I know how to work with people. I’m not a business person; that is not my background. I wanted to leave this course feeling like I could go Having already had extensive training in psychology, she found that she was familiar with the material presented on the programme. This low success case would have chosen a different training intervention had she known of the overlap with her past training and the lack of corporate-related content to meet her learning needs. This low success case preferred an eclectic style, and also reported that she does not use integral coaching in its purest form, although she does make use of the coaching conversation structure and tools offered by the programme: I have seen results from coaching, but not that I would attribute directly to the [programme]. I certainly don’t think that I would ever How could the coaching training programme be improved? Improvement suggestions for the CTP were made by all six success and the two low success cases. For this sub-section of the results, these improvements will be reported for all eight participants. Four of the eight participants reported having disappointing experiences with the buddy coaching system (a pairing of participants to practise skills), mainly because of personality clashes. They did not suggest how buddy selection could be improved. Three participants indicated that more individual attention and detailed feedback during the contact sessions would have been valuable. This could be done by means of more one-on-one coaching or supervision by the programme presenters: With high costs, it’s necessary for people to feel that they have had some individual attention and individual reflection on what they are Two coaches reported that learning pods (small groups created for shared learning, communication of coaching experiences and project work) were not effective, because the pod degenerated into a social rather than a learning event. They suggested that a pod mentor might help to direct the group dynamics towards a learning experience. The following suggestions were made by single participants: • reading material should be provided at the start of the programme to enable recipients to prepare for the learning pods
The Ethics in Research Committee of the Faculty of Commerce, University of Cape Town, approved the evaluation. Potential benefits and hazards There were no benefits or hazards for participants who took part in the evaluation. Recruitment procedures The primary evaluator contacted all participants in the 2004 and 2005 coaching programmes and requested that they complete an electronic questionnaire. Success and non-success cases were identified from the questionnaire results and contacted by e-mail to explain the purpose and content of the telephonic interview that was to follow. Participants had to agree to the telephonic interview before the primary evaluator contacted them. Informed consent On the first page of the questionnaire the purpose, confidentiality of the data and the time that it would take to complete the questionnaire were described in detail. The e-mail sent to the success and non-success cases requesting a telephonic interview contained similar information. Data protection The primary evaluator was the only person who had access to the questionnaire and telephone data. Both success and non-success cases were promised anonymity and were not identified by the evaluators.
Reliability The questionnaire items and transcripts of the telephone interview are described in the Method section. Validity The evaluation focused on a specific programme and did not seek to generalise the findings to other similar programmes.
The SCM aims to discover what is working or not working in a training programme. It identifies best practices, produces results to motivate future participants and is a practical way to evaluate the success of an intervention (Brinkerhoff, 2003, 2006a). However, Brinkerhoff (2003, 2006a) clearly indicates that the SCM does not make an overall summative judgment. Instead its usefulness and value lie in describing what a training intervention can achieve. The SCM does this by capturing the specific and personal ways the training has been used to achieve successful results. What Brinkerhoff demands of an SCM evaluation is that the results provide sufficient evidence for a convincing argument that the training has been applied in the work place. This argument is framed in terms of the scope of the success of the training, worthwhile results achieved, suggestions for improving the programme and environmental factors which have hindered successful application. This framework is utilised below. The scope of the success of the training intervention Most of the trainees reported high levels of successful application of the proximal outcomes of the training. Even the low success cases reported relatively high application levels. Self-report data provided by the participants showed that the programme has had a positive influence on them and that their clients had also benefited indirectly from the training. Thus, it can be concluded that this cohort of 2005 and 2006 recipients was successful in applying the proximal programme outcomes in the workplace. Worthwhile results from the training From the results it is clear that the objectives have been achieved, namely to develop or enhance coaching skills, and self-awareness (see Figure 1). This dual outcome structure constituted the impact theory of the programme. Furthermore, the trainees were successfully applying the programme’s KSAs in conducting coaching programmes with clients. All the high success cases indicated obtaining positive results from applying the following proximal outcomes: using the assessment models; eliciting a client’s structure of interpretation; employing distinctions; developing practices; overcoming clients’ resistance; building relationships with clients; establishing mutual trust, respect and freedom of expression; tackling one’s own barriers to coaching; keeping in condition to coach; and enabling clients to become self-correcting, self-generating, long-term excellent performers. The programme appeared to have influenced participants in areas of personal development. Personal development is considered to be a critical competency in coach training that helps the trainee coach to develop ‘psychological mindedness’ (Bluckert, 2005; Lee, 2003). Bluckert (2005, 2006) suggests that personal development of trainee coaches is as important as coaching theory and skill training and should be given equal weight in the training process. Reports from the participants also indicated behaviour changes, over time, in their clients. This distal outcome is important and needs to be examined further, as client change is often cited as a mediating variable between the programme and organisational benefit (Olivero, Bane & Kopelman, 1997). All six success case coaches reported that they aim to help their clients become self-correcting, self-generating, long-term excellent performers, that is they are working towards the distal outcomes described in the programme literature. In this way, the participants are using a coaching intervention as a tool for client empowerment (Eggers & Clark, 2000). The success case coaches reported several valuable consequences from the training (e.g. studying further, career shifts, networking, building a more successful coaching business and enhancing their credibility as coaches). The quality of the coaching service offered (Bluckert, 2005) and self-regulation of the profession (Hamlyn, 2004) are often raised in this regard. Research (ICF, 1999a, 1999b) has indicated that clients regarded training as important when selecting a coach and that certification was important. An unintended consequence of the training was how participants used the KSAs for a specific coaching leadership style in their jobs. The recipients were applying the KSAs more broadly than the intervention intended by shifting their leadership styles. Goleman (2000) identified six distinct leadership styles, one of which is a coaching leadership style, which necessitates empathy and self-awareness in order to encourage others to improve and develop long-term-strengths. He indicated that this leadership style was used least often, because leaders reported that they did not have the time in their high-pressured environments to help people to develop. Suggestions for improving the programme The evaluation results provided evidence that the 2005 and 2006 coach training was effective, but several participants made suggestions for improving the training that offer valuable feedback to the programme staff. The evaluators are supporting these suggested improvements with further evidence from social science literature. As part of a coach training intervention, Bluckert (2006) recommends that participants should receive coaching themselves. Taking up the role of a client will enable the trainee coach to empathise with future clients regarding their feelings of vulnerability. However, four recipients found the programme’s buddy coaching system ineffective, which was a consequence of personality clashes and resulted in conflict that they considered to hamper their learning experience. We would like to suggest an improved buddy selection system, clear instructions about the intended outcomes of the buddy system, and explicit behavioural guidelines for buddies. The revolving triad method (two buddies and an observer who gives feedback about the process) could also be implemented. Bluckert (2006) recommends that trainee coaches should be part of an ongoing experiential group to enhance self-reflection and growth. The pod meetings were supposed to fulfil this function, but two participants suggested that pod mentors should be introduced to ensure that these meetings retain a learning function and not deteriorate into social events. Furthermore, three other participants suggested that more individual attention and interactive supervision should be provided during the contact sessions. All these suggestions indicate a need for more individualised feedback about self-development. Some of the other suggestions from the trainees might help in this regard. Class size has been shown to affect learning (Kokkelenberg & Dillon, 2002) and the addition of small group interventions could satisfy the need for individual attention. One coach suggested that the programme could offer more demonstrations of different types of coaching conversations. These could capture the development of a coaching relationship over time and the challenges that emerge for coaches. Video clips or multi-media training material could illustrate these high-level KSAs (Noe, 2003). Environmental factors that hindered and helped the successful application of knowledge, skills and attitudes All six success cases found that a business-related background was important in helping them carry out their coaching. Washylyshyn (2003) reported that business experience was one of the top three criteria reported by clients when selecting a coach (the others being training in psychology and the coach’s reputation). A lack of corporate experience could be a problem, as is the case when clinical psychologists enter the field without any business training (Filipczak, 1998; Lowman, 1998; Somerville, 1998). Garman, Whiston and Zlatoper (2000), in a review of coaching skills, reported that 15% of coaches had psychology backgrounds and 15% had business backgrounds, whilst the remainder was not reported. They found that the psychological skill set was usually (61% of the time) referred to as a unique skill base, separate to the coaching skill set. In 45% of these articles, the psychological skill set was thought to add clear value. We suggest that participants are made aware of this debate regarding the different skill sets.Closer supervision was mentioned as a need by the participants. Bluckert (2006) recommends that coaching supervision continues throughout the career of trained coaches, because it is important to provide the opportunity to reflect on one’s own coaching methods and experiences with a senior colleague in order to improve one’s coaching practice. Participants pointed out how important it was to have management support when conducting coaching programmes with in-house clients. Brinkerhoff and Montesino (1995) found that management support facilitated the transfer of KSAs from training to the work environment. Noe (2003) also describes the importance of managerial support in the transfer of training. The two low success cases reported that it was not environmental factors, but rather personal circumstances and choices that hindered the application of the programme’s KSAs. The one low-success case indicated that information on the programme website led her to believe that together with improved coaching skills, she would develop basic business-related skills. It is suggested that the programme managers examine the website and scrutinise the intended outcomes mentioned there. If the target population for the training programme is those who already have a business background, it should be stated unambiguously. Alternatively, the programme staff could provide additional reading to those who do not have a business background. A critical question here is: is business-related knowledge a knowledge component of the programme or not? If so, the evaluators recommend that it is included in the programme activities. Conclusion We conclude that the SCM proved to be a useful and practical evaluation tool for conducting this evaluation. Its main value is providing focused and realistic evaluation questions, such as whether or not KSAs are applied in the workplace? It equipped the evaluators with an evaluation tool that provided a robust structure for the evaluation process. For instance, the ‘buckets’ used in the interviews were particularly useful to conceptualise the interview process. These assisted the evaluators to create a detailed picture of the elements associated with the application of the training and its results. Finally, the SCM is a user-friendly method with a sensible design, which provided the programme staff with information that could be useful for refining the programme. Nevertheless, we have to acknowledge limitations to the method. The first phase of the SCM, the sampling phase, produces extreme groups (success and low success cases) of recipients. The average participants are not selected. Brinkerhoff (2003, 2006a) argues that the average participant might not exist. However, if he or she does, he or she may misrepresent the reality of a training intervention. Brinkerhoff is of the opinion that a few extreme cases can reveal much about the applicability of a training intervention. The SCM is indeed a practical, user-friendly evaluation method. It works well in a single organisation (or several similar organisations) where participants attend training programmes which are mandatory or have perceived value for the organisation. It is not recommended for evaluating training programmes with participants who come from diverse organisations or who are self-employed (see the point on proximal outcomes below). It also does not work well where the participants are a self-selected group who already ‘believe’ in the positive outcomes of the training. As this evaluation showed, these ‘believers’ do not generate sufficient low success cases and thus the opportunity for suggesting sufficient programme improvements might be lost. Limitations The evaluation itself had a number of shortcomings. For a start, it focused on the application of proximal and not distal outcomes. As the participants of the programme were self-employed or employed by different organisations, it was not feasible to focus on distal outcomes of the training. In order to establish the merit of coach programmes, evaluators would need to know whether or not the clients benefited from the coaching, and not just whether or not trainees applied the KSAs of the training. The danger of examining proximal outcomes only could very well result in premature conclusions about the merit of the programme. This evaluation concluded that the programme worked for the trainees. We do not know, however, whether or not the programme and application of the programme KSAs changed client behaviour and benefited organisations. The data for this evaluation consisted of self-reported data. The evaluators assumed that the data collected via this method were credible. Using more objective measures, such as client feedback of coaching experiences, would have been useful, but they were beyond the scope of this evaluation. Despite these short-comings, this evaluation contributes to the evaluation literature by providing a documented exploration of a systematic application of the SCM. In addition, it contributes to the coach training literature by providing the first systematic evaluation of a coach training intervention in South Africa. Suggestions for future research The SCM cannot be used to determine whether or not a specific training programme, and not anything else, caused the results obtained. The best evaluation designs in which causality can be inferred are experimental or quasi-experimental designs with random assignment (Graziano & Ravlin, 2000) and these were not practical to apply in this context. As such, it cannot be asserted with confidence that the coach training programme caused all the results discussed above, but it can be reported that the participants attributed the results to the training. We need to determine which other evaluation methods could be applied successfully in a coach training context. For example, it could be beneficial to conduct an experiment in which a cohort of participants, who have been through the programme, is compared to a group who has not yet been trained. Such an evaluation would provide causal evidence for the effects of the training and not rely solely on self-reported data provided by participants.
Competing interests The authors declare that they have no financial or personal relationship(s) which may have inappropriately influenced them in writing this paper. Authors’ contributions K.B. (University of Cape Town) completed this evaluation as a requirement for her Master’s degree in Programme Evaluation. S.G. (University of Cape Town) supervised the evaluation.
Bedi, A. (2004). An andragogical approach to teaching styles. Education for Primary Care, 15, 93–97. Bluckert, P. (2005). The foundations of a psychological approach to executive coaching. Industrial and Commercial Training, 37, 171–178. http://dx.doi.org/10.1108/00197850510602060 Bluckert, P. (2006). Psychological dimensions of executive coaching. Maidenhead: McGraw-Hill, Open University Press. Brinkerhoff, R.O. (1983). The success case: A low-cost, high-yield evaluation. Training and Development Journal, 37, 58–61. Brinkerhoff, R.O. (1998). An integrated evaluation model for HRD. Training and Development Journal, 42, 66–68. Brinkerhoff, R.O. (2003). The success case method: Find out quickly what’s working and what’s not. San Francisco: Berrett-Koehler Publishers. Brinkerhoff, R.O. (2005). The success case method: A strategic evaluation approach to increase the value and effect of training. Advances in Developing Human Resources, 7, 86–101. http://dx.doi.org/10.1177/1523422304272172 Brinkerhoff, R.O. (2006a). Telling training’s story. San Francisco: Berrett-Koehler Publishers. Brinkerhoff, R.O. (2006b). Getting real about evaluation. Training & Development, 60, 24–25. Brinkerhoff, R.O., & Montesino, M. (1995). Partnerships for training transfer: Lessons from a corporate study. Human Resource Development Quarterly, 6, 263–274. http://dx.doi.org/10.1002/hrdq.3920060305 Coaches and Mentors of South Africa. (2007a). About COMENSA. Retrieved March 03, 2007, from http://www.comensa.org.za/dotnetnuke/AboutUs/tabid/72/Default.aspx Coaches and Mentors of South Africa. (2007b). Standards of Professional Competence. Retrieved March 04, 2007, from http://www.comensa.org.za/COMENSAstandardsofprofessional competence.htm Eggers, J.H., & Clark, D. (2000). Executive coaching that wins. Ivey Business Journal, 65, 66–70. Executive Coaching Forum. (2004). The executive coaching handbook: Principles and guidelines for a successful coaching partnership. Retrieved 25 April, 2007, from http://www.executivecoachingforum.com Flaherty, J., & Handelsman, A. (2004). Integrating rigor, compassion and creative design. The promise of coaching and new venture west’s professional coaching course. In Organisational Psychology Master’s Programme Handbook. Unpublished manuscript, University of Cape Town. Filipczak, B. (1998). The executive coach: Helper or healer? Training, 35, 30–36. Garman, A.N., Whiston, D.L., & Zlatoper, K.W. (2000). Media perceptions of executivecoaching and the formal preparation of coaches. Consulting Psychology Journal: Practice and Research, 52, 201–205. http://dx.doi.org/10.1037/1061-4087.52.3.201 Goleman, D. (2000). Leadership that gets results. Harvard Business Review, 78, 78–90. Graziano, A.M., & Ravlin, M.L. (2000). Research methods: A process of inquiry. (4th edn.). New York: Pearson Education Company. Hamlyn, J. (2004). Executive personal coaching: A key driver to sustainable business success. HR Future, 28–29. Retrieved February 20, 2007, from http://www.comensa.org.za/COMENSAcommunity.htm International Coaching Federation. (1999a). Core Competencies. Retrieved February 28, 2007, from http://www.coachfederation.org/Downloads/Docs/Credentialing/ICF_Core_Competencies.pdf International Coaching Federation. (1999b). Corporate survey results: Analysis of the 1999 survey on coaching incorporate America. Retrieved July 18, 2007, from http://www.coachfederation.org/ICF/For+Current+Members/Member+Resources/Research/Research+Repository/Corporate+Survey+Results/ International Coaching Federation. (2007). ICF global coaching study: Executive summary. Retrieved May 16, 2007, from http://www.coachfederation.org/NR/rdonlyres/257CE43D-C730-4556-9D4B-F8B53061AA56/6809/ExecutiveSummaryFinal.pdf Kokkelenberg, E.C., & Dillon, M. (2002). The effects of class size on student achievement in higher education: Applying an earnings function. Retrieved December 06, 2007, from http://buoir.binghamton.edu/papers/Class_Size_ md2002.pdf Lee, G. (2003). Leadership Coaching: From Personal Insight to Organisational Performance. London: Chartered Institute of Personnel and Development. Lowman, R.L. (1998). Consulting to organizations as if the individual mattered. Consulting Psychology Journal: Practice and Research, 50, 17–24. http://dx.doi.org/10.1037/1061-4087.50.1.17 Miles, M.B., & Huberman, A.M. (1994). Qualitative Data Analysis. (2nd edn.). Thousand Oaks: Sage. Noe, R.A. (2003). Employee training and development. New York: McGraw-Hill. Olivero, G., Bane, K.D., & Kopelman, R.E. (1997). Executive coaching as a transfer of training tool: Effects on productivity in a public agency. Public Personnel Management, 26, 461–469. Ozuah, P.O. (2005). First there was pedagogy and then came andragogy. Einstein Journal of Biology & Medicine, 21, 83–87. Rostron, S.S. (2006). Business coaching in South Africa. Business Coaching Worldwide. Retrieved February 22, 2007, from http://www.wabccoaches.com/bcw/2006_v2_i2/feature.html South African Qualifications Authority. (2007). Registered Unit Standard: 113909. Retrieved May 03, 2007, from http://regqs.saqa.org.za/showUnitStandard.php?id=113909 Sommerville, K. (1998). Where is the business of business psychology headed? Consulting Psychology Journal: Practice and Research, 50, 237–241. http://dx.doi.org/10.1037/1061-4087.50.4.237 Trochim, W.M.K. (2006). Introduction to evaluation. Research Methods Knowledge Base. Retrieved April 03, 2007, from http://anatomyfacts.com/Research/ResearchMethodsKnowledgeBase.pdf Upton, P. (2006, August 10). Sport can learn from business. Finweek, pp. 48–49. Washylyshyn, K.M. (2003). Executive coaching: An outcome study. Consulting Psychology Journal: Practice and Research, 53, 94–106. http://dx.doi.org/10.1037/1061-4087.55.2.94 Zemke, R. (2002). A pocket guide to useful learning theories. Training, 39, 90–91. |