key: cord-0039311-0tjnnzoq authors: Lindgren, Daniel; Budd, John title: A Framework for Social Development Assessment date: 2008 journal: Public Opinion Polling in a Globalized World DOI: 10.1007/978-3-540-75753-5_15 sha: 571ec1458866ae8ce9f2753aa468d45b0627c2f3 doc_id: 39311 cord_uid: 0tjnnzoq This chapter presents a new model of thinking for evaluating social development work. The social assessment framework put forward is based on TNS stakeholder management model, TRI*M, and focuses on the critical area of how aid and donor organizations can assess the effectiveness of social development programs. The Millennium Development Goals have opened up a two way street between donors and beneficiaries and has provided for a need to look at social development assessment from a new perspective. The reader is taken through a journey of how the model came about, challenges faced during the process, and a vision for how the model will be used in the future. The two basic components of the model, the Program Effectiveness Index and the TRI*M Grid for understanding program drivers, are explained. A case study covering the Tsunami relief effort in Aceh, where the model was used for the first time, is included as an example of how TNS was able to deliver insight and clear direction in an otherwise complex environment. A client perspective is also provided both in terms of needs assessment and actions taken by UNICEF as a result. The chapter concludes with a reflection of what the future holds and the opportunity to apply the framework in other countries as well as for different situations. it could also be called social marketing. Nor is behavioural change communication confined to the developing world. Countries at all levels of development undertake mass awareness campaigns in order to influence behaviours, as does the private sector. In the private sector a huge research industry has developed to provide consumer insights and answers about behaviours. It is the same in the development context. Consumer research is called KAP surveys -knowledge, attitude and practice. But such research is often narrowly focused and too often a one-way street. Impact studies are conducted but mainly to inform communicators about the effectiveness of their specific campaigns and messages. How beneficiaries regard the campaign and the issue as a whole and what they think is beneficial or even needed in the broader context of the services provided is often overlooked. So the question is, how can we use the beneficiary's perceptions of the effectiveness of a campaign in order for it to be more effective? In other words start a dialogue -make the beneficiary an active participant in the process. In Indonesia, UNICEF stumbled into this accidentally when ensuring that efforts in the reconstruction of tsunami ravaged Aceh were effective and being well received by the beneficiaries, and could be reported to donors around the world. Having successfully conducted research in 2004-2005 and then in 2005-2006 , it was decided to apply the same framework to assess public awareness campaigns UNICEF was conducting on avian influenza in 2006-2007. Looking at the social research scene in Indonesia, most work that TNS and other international research agencies carry out is with large aid and donor organizations or through consultancies servicing these clients in a broader context. Polling through media remains under developed or is served by local agencies on a much smaller scale. The same applies to government work and means revenue streams from these alternative sources remains very limited. But then again, Indonesia, the world's largest archipelago as well as the most populous Muslim nation, presents plenty of work opportunities for donor organizations. Working closely with several of the world's most prominent aid and donor agencies, one of the main challenges that is evident is the need for good program evaluations. Aid organizations in particular, large and small, receive a large proportion of their funding for specific projects. As with any commercial operator aid organizations often have to compete for this, often limited, pool of funds. There is a tender process and the aid organization has to present a strong case for their project in terms of how it will contribute to the betterment for beneficiaries as well as how results will be evaluated. Donors, like investors, would reasonable like to know that the money was well spent. Hence, some, but far from all, social development projects have a research component for evaluation. Ideally there should be a baseline measure followed by an impact measure. But more often than expected perhaps, the evaluation is conducted internally in sometimes very ambiguous and arbitrary circumstances. Audits are part of daily life in commercial operations (Scharioth, Huber 2003) . But anyone dealing with large aid organizations will be aware of the frantic atmosphere in the client's office at the end of a large project. Yes, the auditors are coming and everyone is scrambling to come up with a good story to show that their contribution was indeed worth while. Having met with many program staff across an array of different organizations it is amazing to see their commitment to their work. Not to mention the field operators, who are often tucked away in the bush somewhere without access to modest, or even basic, conveniences. Their inspiration stems from caring about people in need and because they themselves believe a difference can be made. At the end of the day it would often be unfair to question their commitment to the cause, but donors want proof of their work in terms of measurable outcomes. Key Performance Indicators or KPIs is a term with which aid organizations are quickly becoming more familiar. Aid organizations are slowly beginning to realize that even the warm and fuzzy world of social development is becoming subject to accountability. No doubt many resist this but there seems to be no escape. It has been said that what is not measured cannot be managed. How true it is and perhaps one could argue that common sense also dictates this. But it has also been said that common sense is not common. In the case of social development work what one has to remember is that these are no ordinary projects, it is very complex often involving multiple organizations seeking to assist beneficiaries from impoverished, difficult to reach regions under adverse circumstances. From a research perspective, measurement becomes a daunting task both in terms of sampling as well as developing suitable survey instruments. Stakeholder Management is an area of expertise within TNS and those working in this field have a number of models and tools to their disposal. The question was whether appraisal of social development work can be viewed as a stakeholder management problem? Secondly, would it be possible to apply one of TNS's more popular models called TRI*M? The power of the TRI*M model is not so much in it's statistical rigor, nor the fact that it has been subjected to numerous validations across all sorts of markets and industries (Huber, Scharioth, Pallas 2004) . No, the attractiveness really lies in its flexibility and being able to strike a cord with clients in the most varied of circumstances. Still, the area of social development research seemed like a tough challenge but still possible, somehow. At the outset of developing the social assessment framework, the vision was clear. The framework should help to identify specific focal points that can help aid and donor organizations to achieve enhanced results. From a client perspective, the following benefits were seen to be relevant: • To have the ability to map stakeholder needs geographically • To focus on specific beneficiaries or stakeholder groups, in a more relevant way • To be able to prioritize program initiatives and activities • To assess the deployment of resources & effort • To benchmark performance across activity areas In order to deliver on those benefits, the social assessment framework itself must adhere to some basic guidelines. This was necessary to ensure the framework can be adopted in a local as well as global context. • Provide for a consistent format across: Time, communities/ regions, programs, participating organizations • Accessible & easily comparable understanding to facilitate learning and dissemination of results • Build pro-active thinking in the form of a red-flag or early warning mechanism • High level insight for strategy development down to specific task related information for program management But how is social development work related to corporate reputation? As has been pointed out, the truth is that many aid and donor organizations are not able to effectively measure the performance of social development programs. Within the area of corporate reputation, effective tools have been developed over the years to allow organizations to becoming more knowledgeable about their own industry (O'Gorman, Pirner, 2006) . Understanding ones corporate reputation provides for a holistic picture of how different stakeholder groups view an organization as well as how the company should align its resources to communicate more effectively (Hermann 2007) . It is all about building awareness, changing attitudes, and ultimately have people behave is such a way that it benefits the company. The same applies to social development work, which in most part is about making people aware and changing the way they think and behave. Looking at the stakeholders themselves beneficiaries are at the core, rather than customers. And just as corporate reputation is dependent on several stakeholder groups, social development programs rely on support from multiple parties often including various government departments. Communication is not about selling a product or service but rather to inform and educate as an end goal (selling ideas). Whether in a commercial or social context, media spending can be substantial and media channels employed can be similar. It is sometimes necessary to develop peoples' mind-sets by changing their attitudes. Just a like a mining company wants people to be forgiving towards its impact on the environment, a social development program on AIDS wants people to acknowledge there are people who live with this decease and they should not be ignored. Ultimately and ideally, social change is about engagement through participation rather than persuasion. But the ultimate goal of a social development program is not to build a reputation for the organization behind the program but rather for the program itself. The first basic building block of the social assessment framework is the Program Effectiveness Index (PEI). The idea behind the index is to capture the relevant elements that help to determine perceived effectiveness of a social development program. The table below shows the original TRI*M index dimensions for corporate reputation together with the modified dimensions for program effectiveness. The relevance behind each of the index dimensions can be highlighted with a simple example. The Avian Influenza that first appeared in Indonesia in early 2006 has been subject to a lot of attention globally. A collaborative effort is currently in place to inform the public in Indonesia about its risks, how to avoid contracting the virus, and how to prevent it from spreading further. First of all, are Having a standardised, independently measured PEI has an important advantage in that it can be used across countries and programs. This allows for effective benchmarking, a tool which is becoming more and more essential for global companies including aid and donor organisations. Further, the PEI allows for flexibility in terms of diagnosing the level of performance across time, geographical areas, stakeholder groups, and time. So it is a very effective monitoring tool and provides for a quick overview of where a particular program is working and where it is having less impact. The 2004 Tsunami disaster was an event that captured the attention of the entire globe. Indonesia had the biggest death toll with some quarter of a million people confirmed dead or missing. No less than 385,256 people were displaced, loosing their home, family members or both (UN Information Management Service 2005). The table below shows some of the worse hit regions. The 8.6 magnitude earthquake on the island of Nias, that struck shortly after the Tsunami made matters worse. The disaster was met by unprecedented support from the international community and it has been estimated that some 200 organizations set up base in Aceh alone to assist in the relief effort. The amount of donations contributed from the international community was so significant that it far exceeded what is needed for the reconstruction effort. Yet, the traumatized region is still struggling to get back on its feet. This may seem odd but the fact is, having excess funds only adds to the problem, especially in Indonesia, one of the world's most notorious nations for corruption. Too much money brings out the worst in people and means the relief effort has in many parts been brought down on its knees with very little progress, especially in the first 12 months (TNS Indonesia 2006). Having so many organizations operating simultaneously adds to the complexity and coordination alone has become a gigantic task. Amongst other organizations operating in Aceh and Nias, UNICEF is playing a key role in the rebuilding of the disaster struck area. Focusing on the areas of health, education, water & sanitation and child protection, UNICEF has a fair challenge in coordinating its own activities. Research carried out by TNS in 2005 confirmed the notion that the relief effort, whilst moving in the right direction, was not moving ahead quickly enough. Feedback from beneficiaries was therefore needed to understand their perceptions as to how effective the relief effort has been. The brief from UNICEF was simple, what are the red button issues that IDPs want to see improved immediately? With this brief, the first opportunity to test the new social assessment framework had materialized. The performance effectiveness index was able to show that performance varied across regions. The index showed that the more easily reached regions, Banda Aceh and Aceh Besar, had the worst performance ratings whereas Aceh Barat and Aceh Jaya, some of the most badly hit remote regions, had significantly better performance ratings. However, when sharing the results with UNICEF staff in the respective regions said they were not surprised. Banda Aceh as the capital of the province has served as a hub for much of the relief effort. IDPs in this area have had relatively easy access to assistance without having to rely on their own efforts. Whilst the devastation was large, many parts of Banda Aceh and Aceh Besar were unaffected and meant IDPs were not totally stranded. This led to IDPs becoming somewhat complacent. In contrast, more remote regions have had relatively limited resources to begin with and meant that IDPs and their local communities had to put in significant effort to rebuild their lives, to survive. This logic became evident when looking at the extent to which IDPs have developed a mind-set that was coherent with the program objectives outlined by UNICEF. In short, an examination was done in relation to IDPs knowledge, attitude and behaviour to see how developed their mind-set was in order to cope and improve their livelihood. Figure 1 clearly shows that IDPs in Banda Aceh and Aceh Besar had a much less developed mind-set compared to Aceh Barat and Aceh Jaya. In other words, there was a very strong association between program performance and the actual ability of IDPs to cope and improve their livelihood. Apart from understanding performance across regions, UNICEF wanted to understand what activities were actually driving program performance. This brings us to the second part of the social assessment framework, the TRI*M Grid. In addition to the PEI, a number of specific performance attributes (e.g. program activities, communication channels, etc) are independently measured. In order to uncover the more important attributes in the minds of beneficiaries, one need to go beyond what is simply stated as being important, by making a distinction between must have factors and those that truly drive program performance. The Grid used for this analysis has three dimensions as shown in the diagram below. The first two dimensions Importance on a given attribute is defined on the basis of how important it is rated in relation to all other attributes. As such, there will always be relatively important and relatively unimportant aspects to consider. Performance on a given attribute is defined on the basis of how it is rated in relation to all other attributes assessed in the questionnaire. As such, there will always be relative strengths and relative weaknesses. This is the calculated correlation between the individual attribute and the program effectiveness index. Strong correlation means the attribute will have relatively more impact on perceived performance as a whole. Importance on a given attribute is defined on the basis of how important it is rated in relation to all other attributes. As such, there will always be relatively important and relatively unimportant aspects to consider. Performance on a given attribute is defined on the basis of how it is rated in relation to all other attributes assessed in the questionnaire. As such, there will always be relative strengths and relative weaknesses. This is the calculated correlation between the individual attribute and the program effectiveness index. Strong correlation means the attribute will have relatively more impact on perceived performance as a whole. include claimed importance of program attributes and perceived performance by the program to deliver on the same attributes. The third dimension looks at the impact each attribute has on the PEI. The position of the attributes on the Grid helps to understand the relevance of each activity in the minds of IDPs. Based on the three dimensions in the Grid, attributes can be classified into four groups depending on their location within the Grid (i.e. Motivators, Hygienic, Hidden Opportunities and Potential/Savers). The position on the Grid determines the relevance of each attribute. The Motivator quadrant is the area to observe most closely as any activity within this area is seen by beneficiaries to be a key driver for program effectiveness. Hygienic, on the other hand, are activities that IDPs expect to see and could relate to basic survival needs. Hidden Opportunities represent activities that can make a difference even though the IDPs themselves don't regard them as overly important. Finally, the Potential/Savers quadrant highlights activities that according to IDPs are at the bottom of the priority list. Attributes in this quadrant should be assessed in terms of being essential to the program and sometimes further education of beneficiaries is necessary. Examine your engagement and monitor. Review activities and resources and make sure not to over invest. Build on strengths and eliminate dangerous weaknesses! Use strengths in order to communicate! Look for new opportunities to develop and exploit them, especially if they are strengths. Maintain performance level. These items are expected by stakeholders so must ensure performance meets an acceptable standard. Looking at the 6 core program components it can be seen that UNICEF is perceived to perform well in the areas of Education and Health & Nutrition. The PEI for these activities is higher than the overall PEI of 49. The remaining program areas have a PEI lower than the average. It is interesting to note that child protection is generally not seen to be important by IDPs. Nor does it have any significant impact on program effectiveness. It is likely that more education is needed in this area and improved access to information is needed to achieve this effectively. For each program component a leverage score has been calculated which can range from 0-10. Leverage scores help to identify which program activities have the highest impact on program effectiveness, and therefore indicate the relevance for taking action. The leverage metric is intended to provide decisional support regarding resource allocation to different programs. Both Education and Health & Nutrition generate high leverage for UNICEF probably because these activities are the most visible for IDPs. On the other hand, Water & Sanitation represents a fundamental, basic need not currently met. This area became the single most important opportunity for improvement. It is also clear that the local Government in Aceh, the most important collaboration partner for UNICEF, may not be pulling their fair share of the work. The development sector is a fragmented and complex beast, even within organizations. From the perspective of a communicator targeting both beneficiaries and donors, the research is an independent, valuable and measurable indicator of how work was perceived in Aceh. However the research was a little of a surprise for program staff and Government counterparts who by and large had never been subjected to beneficiary assessments of their effectiveness. They did grasp the survey's importance and at this point are still digesting the points made. It is a work in progress. It is globalization by the most unwelcome of definitions. It links the donor to the beneficiary in a challenge which will leave neither untouched. If the Governments and agencies involved in combating AI in Indonesia, or Thailand, Vietnam or China succeed it will save lives everywhere. This thought motivated the Japanese Government to provide a staggering USD 40 million grant to UNICEF to conduct a global public awareness campaign to prevent a pandemic in 2006. About USD 3.5 million was allocated to Indonesia. The challenge was enormous. The poultry industry is massive and pre industrial. About 1.2 billion poultry in 17,000 islands spread across three time zones. Eighty percent of Indonesians raise poultry in their own backyard in a free range environment. It is an essential economic and nutrition issue for these people. Little wonder that AI was endemic throughout the country. From UNICEF's perspective it was also an issue killing young people and children, as looking after chickens was a chore for them. Under enormous pressure to do something, anything to improve public awareness the research from USAID was used to prepared a national mass media campaign. It was hurried and not ideal because it was not possible to obtain answers to a million questions about the issue. TNS was contracted to conduct a fast and focused survey of beneficiaries in September, 2006 as the mass media campaign was starting. That research confirmed that the USAID research was accurate and that the campaign was succeeding in providing four key prevention messages, cook, separate, report and wash. The above figures are a percentage of the 500 people surveyed answering a question about whether they had seen an AI prevention message on television . As you can see the survey clearly reflects the mass media messages being disseminated. The first two were PSAs already being aired prior to and during the survey The survey stated that: Overall the AI initiative has come off to a relative good start. The perception about the initiative so far is much better compared to the Tsunami Relief Effort in Aceh for example. However, regional differences do exist and there is a strong indication that the effort is not really generating changes in behaviour, especially in rural areas where also exposure to poultry is much higher. South Sulawesi, Yogyakarta and the Botabek area are critical regions to look at to see how the AI campaign can generate more impact. This applies to both urban and rural areas (TNS 2007). The worrying area for UNICEF was communication to people living in rural areas. Clearly there was success in providing information through national TV PSA campaigns. But, beneficiaries in the higher risk provinces of Java and South Sulawesi judged the community level/grassroots communication to need further work, especially in terms of government and community involvement. Also, resistance was anticipated based on economic imperatives overriding preventive messages, and this was again apparent in the beneficiary response. try. The country may still be far away but the problem may be as much their problem as the beneficiaries. For example, whilst having the largest number of confirmed deaths, the avian influenza problem is not just an Indonesia problem, it is a global problem. Hence, donors no longer want to just feel good about contributing to a good cause, they now also want to see results. Already, the framework has established itself as a valuable monitoring and evaluating tool. It is the only independent method of assessing beneficiary attitudes to services being provided to them. But the tool should not be confined to development issues. Governments and social service providers in the developed world can use this Framework as a way of assessing their effectiveness and judging what people think is important. The Framework was extremely helpful for UNICEF's communication outputs. The other benefit is that it provides information at two levels. The model is a bit like an iceberg, the tip is the TRI*M Index which is accessible and allows you to quickly see the most valuable insights, and beneath the surface is a wealth of supporting detailed information. Indonesia is a country that has learnt to cope with the most testing of times. In the past 3 years it has suffered repeated natural disasters and also health crises. The Framework worked in providing the information we needed in that environment but among people who are more used to coping than others. The next step for the model is to be tested in different cultures and looking at other social issues in a less extreme situation. Perhaps a challenge for the social development assessment framework can be put forward. Can it be used to provide vital clues as to why some of the Millennium Development Goals are being met in many regions of the world and not in others? What do beneficiaries, the poor or the vulnerable think of this extraordinarily ambitious promise to them? Changing their perceptions, influencing their attitudes to change negative or ignorant actions into positive behaviour is one of the main means by which cycles of poverty and high risk activities can be broken. The Framework has the potential to contribute to this process. Time is running out before the deadline has to be met, but it is not too late to try out a new approach. Measuring, Managing and Monitoring CSR as a Driver for Corporate Reputation Putting Stakeholder Management into Practice Measuring and monitoring Stakeholder Relationships: Suing TRI*M as an Innovative Tool for Corporate Communication Achieving Excellence in Stakeholder Management Avian Influenza Campaign Measurement Social Development Assessment Framework: Evaluating the Performance of the National AI Initiative from a Beneficiary Perspective Tsunami Relief Impact Study UN Management Information Service in collaboration with Rehabilitation and Reconstruction Agency (BRR