key: cord-0034779-pp32u4sz authors: Lindgren, Daniel; Budd, John title: A Framework for Social Development Assessment date: 2008 journal: From Customer Retention to a Holistic Stakeholder Management System DOI: 10.1007/978-3-540-77430-3_15 sha: 8b09d20f576956317aaa2c7a3f902a7e232476e2 doc_id: 34779 cord_uid: pp32u4sz Behavioural change is probably the single most difficult objective communicators are asked to undertake. In the private sector it is an area which falls under marketing, and in the context of development it could also be called social marketing. Nor is behavioural change communication confined to the developing world. Countries at all levels of development undertake mass awareness campaigns in order to influence behaviours, as does the private sector. Behavioural change is probably the single most difficult objective communicators are asked to undertake. In the private sector it is an area which falls under marketing, and in the context of development it could also be called social marketing. Nor is behavioural change communication confined to the developing world. Countries at all levels of development undertake mass awareness campaigns in order to influence behaviours, as does the private sector. In the private sector a huge research industry has developed to provide consumer insights and answers about behaviours. It is the same in the development context. Consumer research is called KAP surveysknowledge, attitude and practice. But such research is often narrowly focused and too often a one-way street. Impact studies are conducted but mainly to inform communicators about the effectiveness of their specific campaigns and messages. How beneficiaries regard the campaign and the issue as a whole and what they think is beneficial or even needed in the broader context of the services provided is often overlooked. So the question is, how can we use the beneficiary's perceptions of the effectiveness of a campaign in order for it to be more effective? In other words start a dialogue -make the beneficiary an active participant in the process. In Indonesia, UNICEF stumbled into this accidentally when ensuring that efforts in the reconstruction of tsunami ravaged Aceh were effective and being well received by the beneficiaries, and could be reported to donors around the world. Having successfully conducted research in 2004-2005 and then in 2005-2006 , it was decided to apply the same framework to This article was first published in Public Opinion Polling in a Globalized World, Carballo, Marita; Hjelmar, Ulf (Eds.), © 2008 Springer assess public awareness campaigns UNICEF was conducting on avian influenza in 2006-2007. Looking at the social research scene in Indonesia, most work that TNS and other international research agencies carry out is with large aid and donor organizations or through consultancies servicing these clients in a broader context. Polling through media remains under developed or is served by local agencies on a much smaller scale. The same applies to government work and means revenue streams from these alternative sources remains very limited. But then again, Indonesia, the world's largest archipelago as well as the most populous Muslim nation, presents plenty of work opportunities for donor organizations. Working closely with several of the world's most prominent aid and donor agencies, one of the main challenges that is evident is the need for good program evaluations. Aid organizations in particular, large and small, receive a large proportion of their funding for specific projects. As with any commercial operator aid organizations often have to compete for this, often limited, pool of funds. There is a tender process and the aid organization has to present a strong case for their project in terms of how it will contribute to the betterment for beneficiaries as well as how results will be evaluated. Donors, like investors, would reasonable like to know that the money was well spent. Hence, some, but far from all, social development projects have a research component for evaluation. Ideally there should be a baseline measure followed by an impact measure. But more often than expected perhaps, the evaluation is conducted internally in sometimes very ambiguous and arbitrary circumstances. Audits are part of daily life in commercial operations (Scharioth, Huber 2003) . But anyone dealing with large aid organizations will be aware of the frantic atmosphere in the client's office at the end of a large project. Yes, the auditors are coming and everyone is scrambling to come up with a good story to show that their contribution was indeed worth while. Having met with many program staff across an array of different organizations it is amazing to see their commitment to their work. Not to mention the field operators, who are often tucked away in the bush somewhere without access to modest, or even basic, conveniences. Their inspiration stems from caring about people in need and because they themselves believe a difference can be made. At the end of the day it would often be unfair to question their commitment to the cause, but donors want proof of their work in terms of measurable outcomes. Key Performance Indicators or KPIs is a term with which aid organizations are quickly becoming more familiar. Aid organizations are slowly beginning to realize that even the warm and fuzzy world of social development is becoming subject to accountability. No doubt many resist this but there seems to be no escape. It has been said that what is not measured cannot be managed. How true it is and perhaps one could argue that common sense also dictates this. But it has also been said that common sense is not common. In the case of social development work what one has to remember is that these are no ordinary projects, it is very complex often involving multiple organizations seeking to assist beneficiaries from impoverished, difficult to reach regions under adverse circumstances. From a research perspective, measurement becomes a daunting task both in terms of sampling as well as developing suitable survey instruments. Stakeholder Management is an area of expertise within TNS and those working in this field have a number of models and tools to their disposal. The question was whether appraisal of social development work can be viewed as a stakeholder management problem? Secondly, would it be possible to apply one of TNS's more popular models called TRI*M? The power of the TRI*M model is not so much in it's statistical rigor, nor the fact that it has been subjected to numerous validations across all sorts of markets and industries (Huber, Scharioth, Pallas 2004) . No, the attractiveness really lies in its flexibility and being able to strike a cord with clients in the most varied of circumstances. Still, the area of social development research seemed like a tough challenge but still possible, somehow. At the outset of developing the social assessment framework, the vision was clear. The framework should help to identify specific focal points that can help aid and donor organizations to achieve enhanced results. From a client perspective, the following benefits were seen to be relevant: To have the ability to map stakeholder needs geographically To focus on specific beneficiaries or stakeholder groups, in a more relevant way To be able to prioritize program initiatives and activities To assess the deployment of resources & effort To benchmark performance across activity areas In order to deliver on those benefits, the social assessment framework itself must adhere to some basic guidelines. This was necessary to ensure the framework can be adopted in a local as well as global context. Provide for a consistent format across: Time, communities / regions, programs, participating organizations Accessible & easily comparable understanding to facilitate learning and dissemination of results Build pro-active thinking in the form of a red-flag or early warning mechanism High level insight for strategy development down to specific task related information for program management But how is social development work related to corporate reputation? As has been pointed out, the truth is that many aid and donor organizations are not able to effectively measure the performance of social development programs. Within the area of corporate reputation, effective tools have been developed over the years to allow organizations to becoming more knowledgeable about their own industry (O'Gorman, Pirner, 2006) . Understanding ones corporate reputation provides for a holistic picture of how different stakeholder groups view an organization as well as how the company should align its resources to communicate more effectively (Hermann 2007) . It is all about building awareness, changing attitudes, and ultimately have people behave is such a way that it benefits the company. The same applies to social development work, which in most part is about making people aware and changing the way they think and behave. Looking at the stakeholders themselves beneficiaries are at the core, rather than customers. And just as corporate reputation is dependent on several stakeholder groups, social development programs rely on support from multiple parties often including various government departments. Looking at the stakeholders themselves beneficiaries are at the core, rather than customers. And just as corporate reputation is dependent on several stakeholder groups, social development programs rely on support from multiple parties often including various government departments. Communication is not about selling a product or service but rather to inform and educate as an end goal (selling ideas). Whether in a commercial or social context, media spending can be substantial and media channels employed can be similar. It is sometimes necessary to develop peoples' mind-sets by changing their attitudes. Just a like a mining company wants people to be forgiving towards its impact on the environment, a social development program on AIDS wants people to acknowledge there are people who live with this decease and they should not be ignored. Ultimately and ideally, social change is about engagement through participation rather than persuasion. But the ultimate goal of a social development program is not to build a reputation for the organization behind the program but rather for the program itself. The first basic building block of the social assessment framework is the Program Effectiveness Index (PEI). The idea behind the index is to capture the relevant elements that help to determine perceived effectiveness of a social development program. The table below shows the original TRI*M index dimensions for corporate reputation together with the modified dimensions for program effectiveness. Overall reputation Extent of effort behind the program Emotional Appeal Emotional engagement Favourability towards the organization Favourability towards the program Trust in the organization Trust in what is being communicated Competence Effective engagement Financial or economic success Success of the program to convince people to take action Products & service quality Quality of program communication The relevance behind each of the index dimensions can be highlighted with a simple example. The Avian Influenza that first appeared in Indonesia in early 2006 has been subject to a lot of attention globally. A collaborative effort is currently in place to inform the public in Indonesia about its risks, how to avoid contracting the virus, and how to prevent it from spreading further. First of all, are people aware and do they see a concerted effort behind the Avian Influenza program? This does not only involve UNICEF and the government of Indonesia but several other organisations, all working to combat the threat of a possible pandemic. Secondly, are people emotionally engaged? In other words, do they support the program and do they feel that what is being communicated is relevant and make sense? Finally, there needs to be a link to behavioural change through effective engagement. So, is there a sense that the Avian Influenza program is working in terms of changing behaviour and is communication of good quality? Having a standardised, independently measured PEI has an important advantage in that it can be used across countries and programs. This allows for effective benchmarking, a tool which is becoming more and more essential for global companies including aid and donor organisations. Further, the PEI allows for flexibility in terms of diagnosing the level of performance across time, geographical areas, stakeholder groups, and time. So it is a very effective monitoring tool and provides for a quick overview of where a particular program is working and where it is having less impact. The 2004 Tsunami disaster was an event that captured the attention of the entire globe. Indonesia had the biggest death toll with some quarter of a million people confirmed dead or missing. No less than 385,256 people were displaced, loosing their home, family members or both (UN Information Management Service 2005). The table below shows some of the worse hit regions. The 8.6 magnitude earthquake on the island of Nias, that struck shortly after the Tsunami made matters worse. The disaster was met by unprecedented support from the international community and it has been estimated that some 200 organizations set up base in Aceh alone to assist in the relief effort. The amount of donations contributed from the international community was so significant that it far exceeded what is needed for the reconstruction effort. Yet, the traumatized region is still struggling to get back on its feet. This may seem odd but the fact is, having excess funds only adds to the problem, especially in Indonesia, one of the world's most notorious nations for corruption. Too much money brings out the worst in people and means the relief effort has in many parts been brought down on its knees with very little progress, especially in the first 12 months (TNS Indonesia 2006) . Having so many organizations operating simultaneously adds to the complexity and coordination alone has become a gigantic task. Aceh Amongst other organizations operating in Aceh and Nias, UNICEF is playing a key role in the rebuilding of the disaster struck area. Focusing on the areas of health, education, water & sanitation and child protection, UNICEF has a fair challenge in coordinating its own activities. Research carried out by TNS in 2005 confirmed the notion that the relief effort, whilst moving in the right direction, was not moving ahead quickly enough. Feedback from beneficiaries was therefore needed to understand their perceptions as to how effective the relief effort has been. The brief from UNICEF was simple, what are the red button issues that IDPs want to see improved immediately? With this brief, the first opportunity to test the new social assessment framework had materialized. The performance effectiveness index was able to show that performance varied across regions. The index showed that the more easily reached regions, Banda Aceh and Aceh Besar, had the worst performance ratings whereas Aceh Barat and Aceh Jaya, some of the most badly hit remote regions, had significantly better performance ratings. However, when sharing the results with UNICEF staff in the respective regions said they were not surprised. Banda Aceh as the capital of the province has served as a hub for much of the relief effort. IDPs in this area have had relatively easy access to assistance without having to rely on their own efforts. Whilst the devastation was large, many parts of Banda Aceh and Aceh Besar were unaffected and meant IDPs were not totally stranded. This led to IDPs becoming somewhat complacent. In contrast, more remote regions have had relatively limited resources to begin with and meant that IDPs and their local communities had to put in significant effort to rebuild their lives, to survive. This logic became evident when looking at the extent to which IDPs have developed a mind-set that was coherent with the program objectives outlined by UNICEF. In short, an examination was done in relation to IDPs knowledge, attitude and behaviour to see how developed their mind-set was in order to cope and improve their livelihood. Figure 3 clearly shows that IDPs in Banda Aceh and Aceh Besar had a much less developed mind-set compared to Aceh Barat and Aceh Jaya. In other words, there was a very strong association between program performance and the actual ability of IDPs to cope and improve their livelihood. Apart from understanding performance across regions, UNICEF wanted to understand what activities were actually driving program performance. This brings us to the second part of the social assessment framework, the TRI*M Grid. In addition to the PEI, a number of specific performance attributes (e.g. program activities, communication channels, etc) are independently measured. In order to uncover the more important attributes in the minds of beneficiaries, one need to go beyond what is simply stated as being important, by making a distinction between must have factors and those that truly drive program performance. The Grid used for this analysis has three dimensions as shown in the diagram below. The first two dimensions include claimed importance of program attributes and perceived performance by the program to deliver on the same attributes. The third dimension looks at the impact each attribute has on the PEI. The position of the attributes on the Grid helps to understand the relevance of each activity in the minds of IDPs. Based on the three dimensions in the Grid, attributes can be classified into four groups depending on their location within the Grid (i.e. Motivators, Hygienic, Hidden Opportunities and Potential/Savers). The position on the Grid determines the relevance of each attribute. The Motivator quadrant is the area to observe most closely as any activity within this area is seen by beneficiaries to be a key driver for program effectiveness. Hygienics, on the other hand, are activities that IDPs expect to see and could relate to basic survival needs. Hidden Opportunities represent activities that can make a difference even though the IDPs themselves don't regard them as overly important. Finally, the Potential/Savers quadrant highlights activities that according to IDPs are at the bottom of the priority list. Attributes in this quadrant should be assessed in terms of being essential to the program and sometimes further education of beneficiaries is necessary. Importance on a given attribute is defined on the basis of how important it is rated in relation to all other attributes. As such, there will always be relatively important and relatively unimportant aspects to consider. Performance on a given attribute is defined on the basis of how it is rated in relation to all other attributes assessed in the questionnaire. As such, there will always be relative strengths and relative weaknesses. This is the calculated correlation between the individual attribute and the program effectiveness index. Strong correlation means the attribute will have relatively more impact on perceived performance as a whole. Importance on a given attribute is defined on the basis of how important it is rated in relation to all other attributes. As such, there will always be relatively important and relatively unimportant aspects to consider. Performance on a given attribute is defined on the basis of how it is rated in relation to all other attributes assessed in the questionnaire. As such, there will always be relative strengths and relative weaknesses. This is the calculated correlation between the individual attribute and the program effectiveness index. Strong correlation means the attribute will have relatively more impact on perceived performance as a whole. Looking at the 6 core program components it can be seen that UNICEF is perceived to perform well in the areas of Education and Health & Nutrition. The PEI for these activities is higher than the overall PEI of 49. The remaining program areas have a PEI lower than the average. It is interesting to note that child protection is generally not seen to be important by IDPs. Nor does it have any significant impact on program effectiveness. It is likely that more education is needed in this area and improved access to information is needed to achieve this effectively. For each program component a leverage score has been calculated which can range from 0-10. Leverage scores help to identify which program activities have the highest impact on program effectiveness, and therefore indicate the relevance for taking action. The leverage metric is intended to provide decisional support regarding resource allocation to different programs. Both Education and Health & Nutrition generate high leverage for UNICEF probably because these activities are the most visible for IDPs. On the other hand, Water & Sanitation represents a fundamental, basic need not currently met. This area became the single most important opportunity for improvement. It is also clear that the local Government in Aceh, the most important collaboration partner for UNICEF, may not be pulling their fair share of the work. . The development sector is a fragmented and complex beast, even within organizations. From the perspective of a communicator targeting both beneficiaries and donors, the research is an independent, valuable and measurable indicator of how work was perceived in Aceh. However the research was a little of a surprise for program staff and Government counterparts who by and large had never been subjected to beneficiary assessments of their effectiveness. They did grasp the survey's importance and at this point are still digesting the points made. It is a work in progress. But perhaps the most important perspective on the research was UNICEF's decision in Indonesia to take it further. The looming global catastrophe of an influenza pandemic was the motivating force -and Indonesia was bird flu central. Avian influenza (H5N1) is one of the very few examples of a development issue which affects everyone, no matter where we are -bird flu doesn't stop at borders, and it pays no regard to where you come from -poor country or rich country. So what people think about bird flu in the remote distant islands of Indonesia is going to directly affect us all no matter where we are because the odds are that it will be in Indonesia or another Asia country that the virus could mutate. It is globalization by the most unwelcome of definitions. It links the donor to the beneficiary in a challenge which will leave neither untouched. If the governments and agencies involved in combating AI in Indonesia, or Thailand, Vietnam or China succeed it will save lives everywhere. This thought motivated the Japanese Government to provide a staggering USD 40 million grant to UNICEF to conduct a global public awareness campaign to prevent a pandemic in 2006. About USD 3.5 million was allocated to Indonesia. The challenge was enormous. The poultry industry is massive and pre industrial. About 1.2 billion poultry in 17,000 islands spread across three time zones. Eighty percent of Indonesians raise poultry in their own backyard in a free range environment. It is an essential economic and nutrition issue for these people. Little wonder that AI was endemic throughout the country. From UNICEF's perspective it was also an issue killing young people and children, as looking after chickens was a chore for them. In early 2006 we were faced with almost universal ignorance and a complete black hole of reliable, usable information about the disease. Take this from early formative research undertaken in Indonesia by USAID (USAID 2006) . Under enormous pressure to do something, anything to improve public awareness the research from USAID was used to prepared a national mass media campaign. It was hurried and not ideal because it was not possible to obtain answers to a million questions about the issue. TNS was contracted to conduct a fast and focused survey of beneficiaries in September, 2006 as the mass media campaign was starting. That research confirmed that the USAID research was accurate and that the campaign was succeeding in providing four key prevention messages, cook, separate, report and wash. Cook 63 Separate 69 Report 44 Wash 25 The above figures are a percentage of the 500 people surveyed answering a question about whether they had seen an AI prevention message on television (TNS 2006) . As you can see the survey clearly reflects the mass media messages being disseminated. The first two were PSAs already being aired prior to and during the survey period. The third advertisement had been broadcast for about a week and the fourth had just started to be shown. The mass media campaign reached well over 100 million people and ran in two blocks, September/October 2006 and February/March 2007. All fairly crude but combined with social mobilization and advocacy campaigns in high risk areas everyone felt quietly confident that prevention information was out there and that it had reversed the overwhelming ignorance on the issue. But could any changes in behaviour be seen? Yes, in terms of institutional change. The Indonesian Government at all levels was focused on the issue but what of the individual? TNS was commissioned to conduct an impact survey using the newly developed methods presented here. The results confirmed what everyone on the campaign instinctively understood. The survey stated that: Overall the AI initiative has come off to a relative good start. The perception about the initiative so far is much better compared to the Tsunami Relief Effort in Aceh for example. However, regional differences do exist and there is a strong indication that the effort is not really generating changes in behaviour, especially in rural areas where also exposure to poultry is much higher. South Sulawesi, Yogyakarta and the Botabek area are critical regions to look at to see how the AI campaign can generate more impact. This applies to both urban and rural areas (TNS 2007). The worrying area for UNICEF was communication to people living in rural areas. Clearly there was success in providing information through national TV PSA campaigns. But, beneficiaries in the higher risk provinces of Java and South Sulawesi judged the community level/grassroots communication to need further work, especially in terms of government and community involvement. Also, resistance was anticipated based on economic imperatives overriding preventive messages, and this was again apparent in the beneficiary response. But by using the leverage column it is relatively easy to see that major work should be done to obtain further government involvement. This probably has a lot to do with the traditional relationships between Indonesian people and the Government. So part of the strategy for 2007 will be to support the government in establishing district level task forces on avian influenza and piloting a range of preventative measures in a village to see what initiatives will work. The TNS beneficiary research clearly points to a need for an adjustment in tactics, messages and targets for the remainder of the campaign in 2007-2008. There seems to be a great interest for a tool to monitor social programs at this point in time, the time of globalization. SARS, avian Influenza and terrorism are all examples of issues that affect people across borders creating a situation in which the developed and the developing world intersect. This in turn has opened up a two-way link between beneficiaries and donors. It is no longer a situation where donors passively donate funds to a cause in a far away country. The country may still be far away but the problem may be as much their problem as the beneficiaries. For example, whilst having the largest number of confirmed deaths, the avian influenza problem is not just an Indonesia problem, it is a global problem. Hence, donors no longer want to just feel good about contributing to a good cause, they now also want to see results. Already, the framework has established itself as a valuable monitoring and evaluating tool. It is the only independent method of assessing beneficiary attitudes to services being provided to them. But the tool should not be confined to development issues. Governments and social service providers in the developed world can use this Framework as a way of assessing their effectiveness and judging what people think is important. The Framework was extremely helpful for UNICEF's communication outputs. The other benefit is that it provides information at two levels. The model is a bit like an iceberg, the tip is the TRI*M Index which is accessible and allows you to quickly see the most valuable insights, and beneath the surface is a wealth of supporting detailed information. Indonesia is a country that has learnt to cope with the most testing of times. In the past 3 years it has suffered repeated natural disasters and also health crises. The Framework worked in providing the information we needed in that environment but among people who are more used to coping than others. The next step for the model is to be tested in different cultures and looking at other social issues in a less extreme situation. Perhaps a challenge for the social development assessment framework can be put forward. Can it be used to provide vital clues as to why some of the Millennium Development Goals are being met in many regions of theworld and not in others? What do beneficiaries, the poor or the vulnerable think of this extraordinarily ambitious promise to them? Changing their perceptions, influencing their attitudes to change negative or ignorant actions into positive behaviour is one of the main means by which cycles of poverty and high risk activities can be broken. The Framework has the potential to contribute to this process. Time is running out before the deadline has to be met, but it is not too late to try out a new approach. Measuring, Managing and Monitoring CSR as a Driver for Corporate Reputation Putting Stakeholder Management into Practice Achieving Excellence in Stakeholder Management Examine your engagement and monitor. Review activities and resources and make sure not to over invest. Build on strengths and eliminate dangerous weaknesses! Use strengths in order to communicate!