key: cord-0068982-7cf8h7nh
authors: Osipova-Stocker, Yelena; Shiu, Eulynn; Layou, Thomas; Powers, Shawn
title: Assessing impact in global media: methods, innovations, and challenges
date: 2021-10-27
journal: Place Brand Public Dipl
DOI: 10.1057/s41254-021-00240-4
sha: 96bdc7805e19fe7a0020a2893a9ebf3c83c552de
doc_id: 68982
cord_uid: 7cf8h7nh

International broadcasting, defined as “the use of electronic media by one society to shape the opinion of the people and leaders of another,” is an area of public diplomacy research ripe for scholarly attention (Price in Media and sovereignty: the global information revolution and its challenge to state power, MIT Press, Cambridge, 2002). The primary agent for U.S. international broadcasting is the U.S. Agency for Global Media (USAGM), which operates in 62 languages in over 100 countries. While there is broad consensus in the literature that evaluation in public diplomacy is extremely difficult, USAGM has a unique challenge of assessing impact across such a broad and diverse geography of audiences and media environments. Moreover, as an independent public service media agency, USAGM is often called upon to show it is a responsible steward of resources by demonstrating the effectiveness of its programs to policymakers and key stakeholders (Metzgar in Seventy years of the Smith-Mundt Act and U.S. International broadcasting: Back to the future? CPD perspectives in public diplomacy, 2018). This paper describes and assesses USAGM’s Impact Model, which serves as the conceptual framework for aggregating, understanding, and communicating the Agency’s research. It illustrates the research-strategy-evaluation loop examining the case of Voice of America in the Democratic Republic of Congo, and discusses some of the challenges that USAGM faces in both research as well as implementation of the recommendations proposed based on that research.

International broadcasting, defined as "the use of electronic media by one society to shape the opinion of the people and leaders of another," is an area of public diplomacy research ripe for scholarly attention (Price 2002) . The primary agent for U.S. international broadcasting is the U.S. Agency for Global Media (USAGM), which oversees five news networks: Voice of America, Radio Free Europe/Radio Liberty, Radio Free Asia, Middle East Broadcasting, and the Office of Cuba Broadcasting. While there is broad consensus in the literature that evaluation in public diplomacy is extremely difficult, USAGM has a unique challenge of assessing impact in that its audiences are located in more than 100 countries with diverse and specialized media environments.

Moreover, as an independent public service media agency, USAGM is often called upon to show it is a responsible steward of resources by demonstrating the effectiveness of its programs to policymakers and key stakeholders (Metzgar 2018) . To address this challenge, USAGM conducts research to understand the composition of its audiences, how they consume media in general, the topics they find appealing, how and how often they consume USAGM media specifically, and whether its programming is influential. This information is collected via extensive third-party surveys in most of its markets and augmented by metrics from its digital and social media platforms, qualitative data, and a range of structured anecdotal indicators. 1 This paper describes and assesses the Agency's Impact Model, which serves as the conceptual framework for aggregating, understanding, and communicating this research. It illustrates the research-strategy-evaluation loop examining the case of Voice of America in the Democratic Republic of Congo, and discusses some of the challenges that USAGM faces in both research as well as implementation of the recommendations proposed based on that research.

Measurement of public diplomacy effectiveness has been "the holy grail" in the field (Wallin 2012) , both for practitioners as well as academics. Evaluation remains a daunting task not only because of the "elusive and contested nature of the practice" of public diplomacy (Sevin 2017a ), but also because there is little agreement on the very definition of the concept (Gilboa 2008; Gregory 2008) . While earlier conceptualizations of public diplomacy instrumentalized it as a clear tool of foreign policy-telling a country's story to the world-the debate later shifted towards a broader understanding that incorporated relational, collaborative, dialogic, and networked aspects as well, particularly with the emergence of digital communications and social media (Castells 2008; Hayden 2013; Manor 2019; Zaharna 2010; Zaharna et al. 2014) . In this study, public diplomacy is defined as a political actor's management of its image and the related international environment through communication of culture, values, and policies to foreign publics (Cull 2009; Nye 2004; Pamment 2013) . This definition incorporates the relational and discursive elements of public diplomacy necessitated by the rise of the global and digital public sphere, while also recognizing that public diplomacy is, ultimately, a foreign policy tool meant to promote the objectives of a political actor. Within this conceptualization, public diplomacy is also intertwined with soft power-"getting others to want the outcomes you want" through attraction, persuasion, and perceived legitimacy (Nye 2011) . Here, public diplomacy acts as the chief mechanism through which soft power resources and tools are mobilized and enacted to enhance the operational environment of an international actor and create the conditions necessary for the success of its foreign policy (Hayden 2012 (Hayden , 2017 Sevin 2017b) .

Within the public diplomacy literature, international broadcasting is conceptualized as a public diplomacy tool, along with other instruments such as culture, educational exchanges, or public affairs (Cull 2008; Gilboa 2008; Powers and El Gody 2009; Price 2002; Tuch 1990 ). Price's (2002) definition of international broadcasting emphasizes the shaping of opinion as its ultimate objective. He notes that international broadcasting is an "elegant term" comprising a combination of state-sponsored news, information, and entertainment directed at a foreign public (Price 2002) . Within the practice of Western states, international broadcasting and particularly the use of objective news, has mirrored the domestic culture of ethical and high standards journalism and has served as a mechanism for promoting and diffusing that culture abroad (Cull 2008) . While many of the reporters working in international broadcasting tend to object to the notion that international broadcasting is public diplomacy-they see themselves as professional journalists operating independent of any particular governmental interest-the effort is ultimately meant to promote the foreign policy objectives of state or non-state actors (Hacker and Mendez 2016; Rawnsley 2016) . Nevertheless, inasmuch effective international broadcasting might depend on truth and credibility, it also requires structural arrangementssuch as firewalls-that separate it from covert instruments, deception techniques, and other influence operations that political actors might utilize as they pursue their foreign policy (Gregory 2008) . What is more, the technological and professional requirements of international broadcasting have maintained its institutional separation from other, more traditional public diplomacy functions, which has also served to enhance its independence as well as credibility (Cull 2008) . Price (2002) notes that international broadcasting is not defined by the technology of distribution. While it started with shortwave radio, international broadcasting has moved in step with the times and has embraced all the technological innovations and changes happening in media, from television and satellite broadcasting to the internet, utilizing the advantages provided by social media as well as digital streaming. These advances in communication technology have led to the deterritorialization of news, shattering "forever the spatial and temporal boundaries that constrained their predecessors" (Rawnsley 2016) . This was also a transformation necessitated by globalization, the declining significance of national borders, and the increasing emergence of regional-if not global-information spheres (Castells 2008; Volkmer 2014) . Therefore, while targeting specific populations in their native or familiar languages remains important, international broadcasters are increasingly cognizant of the transnational nature of their audiences, working to reach diasporas, migrants and expats, as well as those who live in particular countries. 2 Another characteristic of international broadcasting is the variety of its styles and objectives, based on the sponsoring countries' histories, foreign policy goals, structures, as well as domestic political and cultural factors (O'Keeffe and Oliver 2010; Price 2002) . In their overview of various international broadcasters, O'Keeffe and Oliver (2010) identified five broad reasons driving governments around the world to use international broadcasting:

1. to provide credible alternative sources of information and ideas, particularly to nations which may have insufficient resources to support robust independent media 2. to access and influence the overseas residing diaspora 3. to preserve non-English languages 4. to counter Western and English-language media, cultural or political dominance, and 5. to project a country's culture, ideals, values and expertise.

For example, the BBC World Service, Radio Netherlands Worldwide, and Radio France International historically emerged to reach their colonial and former colonial populations with news in their home languages (O'Keeffe and Oliver 2010). Meanwhile, China's international broadcasting aims to improve the PRC's image by depicting it as a responsible and respected member of the global community, even as it promotes outright propaganda and disinformation (Brandt and Taussig 2020; Cook 2020; Varrall 2020) .

In the case of the United States, international broadcasting started with Voice of America (VOA) broadcasts during WWII, aimed at countering German and Japanese propaganda, and later expanded to include other Congressionally-mandated networks-Radio Free Europe/Radio Liberty (Eastern Europe and Eurasia), Radio y Televisión Martí (Cuba), Radio Free Asia (East and Southeast Asia), and Middle East Broadcasting Networks (Middle East and North Africa). These networks started and expanded with the onset of the Cold War as well as the transformations in Europe following its end, the deteriorating situation in Cuba and Asia, as well as the aftermath of the 9/11 terrorist attacks. Whether telling America's story, explaining U.S. policies around the world, or prioritizing the coverage of the local stories not covered by other media, USAGM networks provide independent and objective reporting in markets where media freedom is either lacking or nonexistent (USAGM 2021a). Over the decades, oversight of these networks moved from the U.S. Information Agency to the Broadcasting Board of Governors (BBG), which, in 2018, was rebranded as the U.S. Agency for Global Media as part of a larger modernization effort (USAGM 2018). As a federal, public agency, USAGM relies entirely on the federal budget process for its funding, requiring demonstration of effectiveness and viability to decision-makers in the executive and legislative branches of the government, as well as other stakeholders (Metzgar 2018) . Therefore, measurement and evaluation comprise a core part of USAGM's work, in order to be able to assess needs and opportunities across its range of operations in over 100 countries.

Even as USAGM advances a public service-oriented mission to "inform, engage, and connect people around the world in support of freedom and democracy" and its journalists' independence is protected by Congressional statute (USAGM 2021a), it has simultaneously faced accusations of producing propaganda in support of U.S. government interests. This debate regarding the role of international broadcasters as extensions of state interests is an important one (Youmans and Powers 2015) , but its relevance to the question of organizational assessment of international broadcasting is limited. 3 All government-supported news and information entities seek influence one way or another. The goal of this influence-in the case of USAGM, in support of freedom and democracy-should drive organizational assessment. Because governments invest in international broadcasting for a variety of purposes and their tactics align with those particular goals, each international broadcaster needs to implement an assessment regime specific to its operational toolkit. For USAGM and its funder Congress, this requires sweeping assessment of broadcast operations delivering independent news and information tailored for local audiences in 62 languages. Sevin (2017a) notes that "there is no universally accepted practice or academic model to assess the effectiveness of public diplomacy projects." Some of the reasons revolve around the very nature of public diplomacy-as noted earlier, there is little, if any, agreement on its definition, while public diplomacy tools represent such a broad range of activities that establishing a unitary approach to their evaluation might bring about major methodological flaws. It is, therefore, important to consider evaluation within the context of how and why public diplomacy is employed in each instance (Pamment 2014; Sevin 2015 Sevin , 2017a . Robert Banks (2011) , on the other hand, identified a long list of challenges with public diplomacy evaluation ranging from the intangible nature of the subject matter and the impossibility to establish direct causality between outcomes and certain public diplomacy interventions to the time, labor, and cost intensiveness of the evaluation exercise. Perhaps the most pointed critique came from Pahlavi (2007) , who noted:

Examination of major PD programmes reveals that, broadly speaking, these programmes are centred on immeasurable goals, tend to consider outputs rather than outcomes, concentrate on the wrong indicators, and suffer from a lack of appropriate methods for conducting audience analysis.

Despite these challenges, there have been several concrete suggestions for operationalizing and assessing the impact of public diplomacy projects (Pamment 2014) . One approach, suggested by Sevin (2017a) , focuses on three "layers" of impact-public opinion, relationship dynamics, and public debates-that are then assessed on two additional scales of focused and larger impact, reflecting the outcomes on the individual and societal levels. Another approach, proposed by Ji (2017) , conceptualizes impact within the context of soft power and suggests looking at the two main categories of soft power outcomes-cognitive and behavioralat micro, meso, and macro levels. At the micro level, the impact assessment focuses on the individual's opinion and behavior, at the meso level -group opinion and institutional policy, while at the macro level -public opinion and state policy-related outcomes (Ji 2017) .

While most of this criticism and suggested evaluation rubrics pertain primarily to traditional public diplomacy programs usually conducted directly by foreign ministries, these considerations and approaches can contribute to evaluating the impact of international broadcasting, as well. To evaluate the effectiveness of international broadcasting as public diplomacy, it is important to go beyond audience reach or ratings and put its longer-term objective at the core of the exercise -contextualize it specifically as a tool of foreign policy. Nevertheless, it is also important to keep in mind that USAGM is ultimately a media organization and, as such, its approach to evaluation closely resembles that of other public interest media as opposed to some of the more traditional public diplomacy evaluation methods adopted by the State Department.

USAGM is an independent federal agency, operating outside of the State Department and receiving a separate Congressional appropriation, with a mission to inform, engage, and connect people around the world in support of freedom and democracy. 4 Its five constituent news networks are mandated to not only bring America's story to the international public, but also provide independent reporting and much needed information in places lacking freedom of the press and democracy (USAGM 2021a). In 2019, a sixth entity -the Open Technology Fund (OTF) -was established as an independent grantee of the Agency with the targeted purpose of incubating open technologies to support free expression, circumvent censorship, and obstruct repressive surveillance (OTF 2021). Increasingly, the tools supported by OTF are also integral for audiences in repressive and closed societies to be able to access programming from USAGM's news networks.

Thus, USAGM funds public service journalism and open technologies with a primary aim of serving as an example of high-quality journalism and providing access to information in challenging environments around the world, ranging from North Korea and Russia to Cuba or Burma. The underlying foreign policy rationale is rooted in democratic peace theory, which suggests that free and democratic societies are more peaceful, prosperous, inclusive, and stable, conditions that fit squarely within America's national interest (Doyle 1983) . That is, by promoting democratic principles and values of freedom, USAGM promotes a more favorable and less hostile environment for the United States abroad, thus contributing to safety and security at home. This is also in line with our definition of public diplomacy, as USAGM supports networks that communicate American culture and values, in particular the foundational values embedded in free expression and a free press as a bedrock of an open society, further helping the U.S. manage its image in the international environment as a champion of democratic values. Furthermore, by covering current events and related policy debates, USAGM networks help explain U.S. domestic and foreign policy to the global public while also engaging that public in a direct conversation through "call-in" shows, usergenerated content, as well as interactions on social media.

Measuring and evaluating the effectiveness of such broad objectives, particularly across sixty-two languages and over one hundred countries, is a challenging task. However, over the years, USAGM, in collaboration with private sector partners such as Gallup, Kantar Public, M&C Saatchi World Services, Ipsos, and others, has operationalized these concepts and drawn up a methodology that goes beyond measuring outputs like "weekly reach" and includes medium and longer-term factors such as sustained engagement with the audience and influencing policy. BBG's initial Impact Framework was drawn up in 2013 and was grounded in the Logic Framework -activities, results, outcomes, and impacts necessary to achieve the stated mission -and a general theory of change, which identifies the causal chain of events required to achieve the desired impact, including intermediary outputs, outcomes and considerations (Gallup 2013) . That framework has evolved over the years into the current Impact Model that USAGM uses to evaluate the effectiveness of its programming and serves as the foundation of the annual, Congressionally-mandated Performance and Accountability Report. 5 To assess programming performance, the Impact Model draws from a diverse set of evidence comprising quantitative, qualitative, and anecdotal data. One key source of data comes from extensive third-party surveys. In the past, USAGM commissioned annual, nationally representative full surveys; however, due to the rising costs of in-person surveys and budget constraints, it has in recent years moved to adding targeted questions to larger omnibus surveys conducted by private market research firms. Both full-length and omnibus surveys gather data such as media brand awareness and use; technology ownership; and topics of interest, but also more generic information such as demographic data and general political outlook, as well as targeted questions aiming to gauge the extent to which USAGM helps improve the understanding of world events or U.S. policy. The survey data are augmented by digital metrics, using data from website and social media analytics as well as digital ratings and comparison tools that help provide accurate and real-time measurement of the effectiveness of Agency's digital assets.

These quantifiable measures are complemented by qualitative data from focus groups, interviews, and monitoring panels. The Agency implements focus groups and interviews with USAGM users, as well as non-users who represent potential target audiences. The Agency commissions focus groups and interviews interchangeably, selecting the method that is most suited to real-world political and cultural realities. Focus group discussions, when carefully designed to account for gender, language, and socioeconomic differences, can elicit rich insights and powerful ideas from target audiences; however, coordinating small group discussions in conflict regions and under COVID-19 restrictions can be challenging. At times, our audience members require less visible, less exposed ways to engage in research, and this is often where in-depth individual interviews serve as a more flexible, discreet method of obtaining structured and semistructured data as well as direct audience feedback. Monitoring panels, where a small subset of audience members access USAGM content over multiple, consecutive days before completing home exercises and participating in interviews, also serve as a tool to evaluate the range of programming offered by the Agency's media brands. Monitoring panel participants score USAGM content for clarity, appeal, and informativeness -in real-time and under the guidance of trained researchers who prompt participants on a daily basis. If quantitative research provides a broad evaluation of the Agency's performance among a certain population, then qualitative research serves as a detailed program assessment by the very consumer for whom it is created. Although not officially a part of the Impact Model framework, the Agency also oversees network-implemented program reviews, where internal analysts, journalists, and producers, as well as on occasion external experts, assess a random sampling of content for objectivity, accuracy, and audience appeal.

Furthermore, to provide a more comprehensive and holistic assessment of USAGM's programming, the Agency also collects structured anecdotal data that captures the impact of content produced by its networks, such as pick-ups of their coverage by other major news media, reactions from government officials, or visible policy change. Additional inputs from customer relationship management databases -for example, the list of affiliates in particular regions and their standing in the target media market -as well as internal quality reviews comprise yet another set of unstructured data that feed into the Model. Taken together, this body of evidence provides the opportunity to evaluate and measure the success of USAGM programming in specific markets, as well as worldwide.

The Model comprises over 30 indicators organized around USAGM's mission. Some of the indicators (highlighted in bold) represent required data and are always included in the impact assessment, while other -optional -data can provide a more nuanced understanding of performance, but might be more difficult or impractical to acquire. The optional indicators are included depending on the specific goals and objectives as outlined in the strategy for the particular market and depending on the availability of data in that specific context.

The ultimate goal of all USAGM programming is longterm and transformational -freedom and democracy -yet it would be impossible to attribute such transformation directly to USAGM programming. The other three objectives, however -informing, engaging, and connecting -can be assessed directly, and when combined with indicators capturing successful influence, provide a practical evaluation of the Agency's overall impact. The Model groups the indicators into three broad categories that correspond to Agency's overarching objectivesinform, engage/connect, and be influential -as well as more specific goals of reaching target audiences, providing value, engaging target audiences, engaging/connecting media markets, creating loyalty, and being influential with people, media, and governments (USAGM 2021b). The Model then operationalizes each of these goals by defining corresponding indicators that help measure the impact in each of the areas, outlined in detail in Fig. 1 . For example, in measuring impact for engaging and connecting with target media markets, the Impact Model considers indicators such as the number of affiliates in the particular market and their number of downloads of USAGM content, as well as whether there have been instances of creating shared content engaging local audiences with those affiliates. For the broader influence measure, the Model combines quantitative data from surveys on questions relating to self-rated understanding of current events or U.S. society with anecdotal evidence demonstrating prominent media pickups, attributable improvement of journalistic practices The Impact Model establishes benchmarks for each of the indicators while also comparing the performance to that of previous years. This approach enables a more comprehensive view of USAGM's impact, including hard-to-operationalize effects such as credibility or gradual political transformation, even if incremental. More importantly, it provides a multi-level, holistic assessment of impact, by evaluating it at the micro (individual audience members), meso (media organizations in target markets), and macro (broader public opinion; local and national officials and policy) levels. As such, the Model serves as a useful framework for setting the strategies for each of the language services as well as for their informal assessment at the end of each review period -usually every fiscal year. The Model highlights the major priority areas to help inform each service's efforts and clearly defines what these areas and categories entail, comprising an "overlay of success that [the Agency] is seeking" (Beatty 2021) .

The Model thus provides overall guidance and informs decisions at every step of strategy formulation. Nevertheless, its application can vary from service to service, depending on contextual factors and the priority areas for the specific market. It is practically impossible to include all of the indicators in the strategies or assessments of every service because their objectives and circumstances of operation can be very different around the world -for example, what VOA can do and hope to achieve in Mexico is very different from Mali or North Korea. What is more, the centrality of certain goals or indicators -or even their inclusion in the strategy in the first place -varies across services, whether based on circumstances of the local media market or depending on the salience of these markets for U.S. national security interests. Using the earlier example, VOA's priority objectives in Mexico might include engaging/connecting with the local media market or affiliates, or facilitating greater audience participation or dialogue on its platforms. These goals would be untenable in a market like North Korea, where the central priorities might instead include reach, providing value, and increased understanding of current events or U.S. policy.

Another caveat is that while the language of the Model is utilized extensively at the agency and network level, it does not always percolate all the way down to the service level. Since the Model serves as the framework for language service strategy and assessment, it naturally provides the foundation for the goals and operations of the services. However, the latter might not necessarily use the same language as outlined in the Model, or rigidly follow every goal or indicator set out in the strategy in their day-to-day work because that would slow them down, inhibit their agility, and render them unable to keep up with the news cycle. What is more, the nature and availability of data from indicator to indicator can vary greatly, limiting their utility for everyday consideration across the board. Resource and geopolitical constraints make it impossible to conduct full surveys or in-depth focus groups -which provide much of the most useful impact data -across all of USAGM's markets every year or even every other year. When available, these insights help services make big changes, such as introducing new shows or starting new language offerings. These data are also extremely valuable for tracking historical trends in the market and make assessments of USAGM reach and impact over time. Meanwhile, digital metrics provide real-time, immediately actionable information and services do rely on those extensively when making short-term editorial decisions, fine-tuning existing content, or considering their offerings on particular digital platforms (Donangmaye and Jaafari 2021) .

The case study of Voice of America's broadcasting in the Democratic Republic of Congo (DRC) outlined below illustrates how parts of the Impact Model as well as quantitative, qualitative, and anecdotal data are integrated into service strategies and how this contributes to their future planning. Nevertheless, it is also important to note that resource constraints and budgetary considerations -particularly the time it takes to make adjustments -can present a major obstacle in the timely application of the necessary changes made apparent by the data. The following section provides a brief discussion of the federal budget process and the challenges it presents to the ability of USAGM -or any federal agency -to be agile and responsive to the fast-changing environment where it operates.

Aligning resources based on performance has always been a challenge. The Clinton Administration ushered in the largest reform effort to try to tackle this problem through the Government Performance and Results Act (GPRA) of 1993 (ACUS 2020). This Act directed federal agencies to set goals and measure the performance of their goals to better inform Congress when making decisions on the allocation of funding through the appropriations process. In his testimony before Congress on GPRA, Donald Kettl (1996) stated the "American government has not one deficit but two: a budget deficit, and a performance deficit." He went on to state that the two are linked and that Congress cannot cut budgets without understanding the impact on performance.

Building on the Clinton Administration's initiative, The Obama Administration sought to improve GPRA through the GPRA Modernization Act of 2010, Public Law 111-352 (Title 5 2010). The key provision in this legislation required federal agencies to submit an annual performance plan covering each program activity set forth in the budget and providing an agency official assigned to ensure the progress of each goal. However, even with the modernization of GPRA, the long, drawn-out budget process does not enable government agencies to align resources with performance in any meaningful way. The budget process normally begins a year and half before receiving the appropriation. First, in the spring of the prior year, the agency must determine what their funding priorities are and develop strategic plans on realigning funding for programs for the next fiscal year (FY) to start in October of the following year. By September, the agency submits their budget to the Office of Management and Budget (OMB). Then, the agency negotiates and defends their strategic funding priorities with the OMB, which provides a final decision on funding around early December. Typically, the agency submits their final budget proposal, now approved by the OMB, on the first Monday in February. The agency then works with Congress to negotiate and defend the proposed budget. Both the House and Senate then provide their own versions of the budget, which often differ, sometimes drastically. Very rarely does Congress have an agreed upon budget by the beginning of the fiscal year -October 1. Without an agreed upon budget, Congress must pass a continuing resolution which only funds the agencies for a limited amount of time, normally two to three months. A continuing resolution only provides limited funding, which impairs the agency's ability to start any new initiatives tied to performance. Making things worse, there is always the danger that if Congress and the President do not agree upon the continuing resolution, the federal government might have to shut down, losing any momentum in making necessary adjustments or closing the performance gap.

With the budget process taking so long to work through the labyrinth of approvals and decisions, the performance data tied to the budget submission erodes with the time that it takes to approve the funding. Valid performance and evaluation data that speaks to the effectiveness of resources is key for decision makers to allocate funding to achieve the desired results (Joyce 2003) . However, the tedious and lengthy budgeting process means decision makers are necessarily working with outdated performance data. For example, if a program began to show some promise through more recent evaluations, that information would not be available for appropriators to allocate or redirect funding and performance may decline without a surge of funding.

Exacerbating the resource-performance challenge, President Trump's first budget submission in May 2017 proposed higher funding for defense and homeland security activities without increasing the overall federal deficit. In order to achieve that, the President's budget proposed to reduce the funding for foreign affairs activities, among them the U.S. State Department, U.S. Agency for International Development, and USAGM, by 28% (OMB 2017). Congress chose to deviate from the Trump Administration's approach and countered by actually increasing spending on foreign operations. Since the FY 2018 budget submission, the trend of a massive divergence between White House-proposed and Congress-enacted funding levels has continued. Though Congress has maintained a consistent funding level for USAGM, the Agency often does not know what its annual funding level will be until after both the White House and Congress settle on an annual appropriations bill. The possibility of a sizable reduction in funding directly affects USAGM's agility and its ability to respond to world events or make necessary adjustments based on fast-paced changes in the operational environment or research. Furthermore, if the Agency were to receive the proposed budget reduction, it would have to cut staff, limiting its ability to deliver the coverage and content that make up the foundation of its reliability and credibility around the world. Such divergence in budget plans transcends political parties -as, for example, in FY 2021, the Democratic-controlled Congress reduced USAGM's budget by $7 million -and makes it extremely difficult for the Agency to plan ahead (Fig. 2) .

In 2014, the Advisory Commission on Public Diplomacy (ACPD) noted that the Impact Model does not connect the data with strategic objectives of the then BBG (Powers et al. 2014 ). More importantly, the Model is not tied directly to resource allocations -whether for research that feeds the model or the necessary programming adjustments made apparent by that research. Thus, if the Impact Model is indicating some type of feedback, the drawn-out budget process does not allow USAGM to respond in a timely manner that would allow progression towards the Agency's goals. The uncertainty about future funding levels also limits the ability to plan for and commission additional research to ensure the consistent availability of performance data in the future. The following case study illustrates this process and demonstrates how budgeting adds another layer of complication for the effectiveness of the Impact Model.

Case study: VOA French to Africa service in the democratic republic of Congo DRC is the largest country in Sub-Saharan Africa and despite its wealth of natural resources, it has the third largest population of poor in the world. The country's development is hampered by endemic corruption, ethnic conflict, lack of public services, and disease, including Ebola, measles, and COVID-19 (Gallup 2020; World Bank 2021). As one of the largest and poorest states on the continent, DRC is an important country for U.S. foreign policy and national security in the region. In Sub-Saharan Africa, including DRC, the U.S. is working to promote stability through democratic governance and effective institutions, as well as to encourage economic growth through increased trade and investment (VOA 2017). Conflict, fragility, and food insecurity are also leading to a flood of migrants and refugees in the region and beyond, contributing to further instability around the world (State Department 2020). The United States is the DRC's largest bilateral donor and the single largest financial contributor to the U.N. Organization Stabilization Mission in the Democratic Republic of Congo (MONUSCO) (State Department 2020). USAGM's objectives of creating an informed and engaged citizenry contributes to the overall improvement of democratic governance in DRC, economic development, and greater awareness of health-related best practices (Donangmaye and Jaafari 2021; VOA 2017).

Launched in 1960, VOA's French to Africa Service produces a broad mix of news and interactive programming for radio, television, and the internet, primarily targeting the francophone countries in sub-Saharan Africa . Over time, in an effort to reach an even broader audience in these markets, the Service has launched additional programming in local vernaculars: Bambara, Lingala, Sango, and Wolof. On radio, the Service broadcasts 22.5 h of programming each week -through shortwave, a growing network of USAGM owned-and-operated FM stations in major cities, and an array of local FM affiliates throughout the region that rebroadcast French to Africa content. For TV, the Service produces 3.5 h of programming broadcast via a network of 72 affiliates across the region. All this content is also available online -on the Service's website and major social media platforms, as well as in the form of podcasts and a 24/7 online radio stream. 6 USAGM has conducted surveys in nine countries with sizeable French-speaking populations in recent years -Benin, Burundi, Central African Republic, Cote d'Ivoire, Democratic Republic of Congo (DRC), Gabon, Madagascar, and Mali -with DRC as the Service's largest and one of the primary markets. VOA French to Africa seeks to reach opinion leaders and young adults, and to enhance its reach and impact with women. Because French is only spoken by Africans with some education, the target audience tends to be more urban than rural, and better educated than the general population. Men tend to have higher education than women and many more of them understand French well enough to follow a broadcast, skewing reach among men higher than among women. Nevertheless, the Service has tried to mitigate these trends by featuring programming specifically targeting women -such as shows that focus on health or women's issues -as well as by launching programming in local vernaculars such as Bambara for Mali and Lingala for DRC.

For the following analysis, the paper highlights USAGM national surveys conducted in DRC in 2016 and 2020, and a 2017 monitoring panel for a specific radio program among respondents based in the country. An important caveat for the discussion of DRC data is that VOA broadcasts in four languages there -French, Lingala, Swahili, and English. The latter two are managed by the VOA Swahili and VOA English Services, while French and Lingala fall within the French to Africa Service. Strategies are set for each country or region, sometimes broken down by service and/or language. In countries or regions with multiple languages, there is naturally some overlap of audiences, but services usually report unduplicated numbers when it comes to weekly audience reach. However, there is limited service-specific data available for some of the impact metrics, such as "credibility" and "understanding," because of the way survey questions and reports are set up. In those cases, the paper will discuss the perception of the network/brand more broadly, i.e., VOA in general as opposed to VOA French to Africa or VOA Lingala (Table 1) .

Measuring micro-level changes in human perception and behavior as a result of news content necessarily begins with the establishing that an individual has "received" or "consumed" the information. Thus, the Agency's Impact Model begins with measuring the size of its audience. In order to limit the effect of recall error, the Agency counts individuals who report accessing its networks' content in the seven days immediately prior to survey participation. Assuming there are still limits to human ability to accurately remember all media activity even within the past week, reporting on the past seven days, or "weekly reach," reduces recall errors that are introduced with a larger timeframe that stretches back further. The 2016 survey showed that VOA's reach with the French-speaking audience that consumed VOA content in the past seven days in DRC was 9.0%, with 2.5% and 2.6% for Swahili and English respectively (Gallup 2016) . 7 Men comprised the large majority of this audience -VOA's total reach among men in 2016 was 14.2% vs. 3.8% among women. Reach among young people -the primary target of the services (15-34 age group) -was significant, at about 9% on average, but was not higher than among the older age groups (35 +) ( Table 2 ). 8

In terms of content distribution, the survey demonstrated that radio was still the dominant platform for VOA audiences, reaching 8.1% of the audience, followed by TV (5.0%) and the internet (2.1%). Among the French listeners within this radio audience, 71.5% said they listen to the Service through the FM radio band broadcast via local stations ( Table 3) .

The key impact metrics of interest in the quantitative surveys involve questions about "credibility" and "understanding." A good proxy for credibility is the question about perceived trust of the news received from the specified sources with available responses including "trust a great deal," "trust it somewhat," "do not trust it very much," and "do not trust it at all." In 2016, 89.8% of all VOA users in DRC said they trusted the network "somewhat" or "a great deal." (Fig. 3) .

In terms of understanding, 92.4% said VOA helped them "understand current events," 66.1% said it helped them "understand U.S. foreign policy," and 66.1% said the same about "American society." (Fig. 4) .

Quantified behavior and perceptions such as measured audience, how people access content, credibility, and understanding create the foundation for interpreting and integrating alternative data. A 2017 in-depth monitoring panel probed 12 DRC-based participants for opinions about six episodes of the Votre Santé, Votre Avenir radio show over the course of six weeks (Evrima Research and Survey Sarl 2017). 9 Votre Santé, Votre Avenir is a 30-min weekly (2016) Weekly reach refers to the percent of the adult population that reported accessing USAGM media content in the past seven days. The agency adheres to a strict "past seven days" standard in order to limit recall error 2016 (n = 2008) (%) VOA total weekly reach 9.0 VOA total in French 9.0 VOA total in Swahili 2.5 VOA total in English 2.6 In addition, parts of the provinces of Katanga and Orientale were excluded from the population frame because of unsafe conditions. In total, the geographic exclusions represented about 10 million people. 8 The "2016 Analytic Report for the DRC Media Use Survey" presented the data for non-Kivu provinces and the three Kivu urban areas separately. For purposes of this discussion, the paper is reproducing the non-Kivu data only. Given that the represented population in three Kivu cities is only 700,000 people, its inclusion in the 2016 figures would change impact measures only negligibly and within the margin of error. 9 The study required respondents based inside DRC to listen to and evaluate six episodes of the Votre Santé, Votre Avenir show between November 4 and December 9, 2017. The monitoring panel consisted of 12 panelists, all regular listeners of VOA programming, located in the non-Kivu DRC provinces, and in the Kivu cities of Bukavu, Goma and Beni. All panelists were regular listeners to VOA French to Africa Service in the past year and represented a diverse range of interactive radio show offering practical health tips, particularly regarding diseases such as AIDS and Ebola, and featuring listener questions posed to healthcare experts. Among other show-specific findings, the panel demonstrated that the practical and relevant advice, personal stories, and the inclusion of African or local health experts lend a high degree of authenticity and trustworthiness to the programming. Some of the panelists also indicated that what they learned from the show helped them change their behavior in relation to hygiene and the consumption of tobacco and alcohol, as well the reduction of other risky behavior -suggesting a link in their minds between their changed behavior and VOA programming.

When looking at these and other data, the Service identified key strengths and gaps in its programming and distribution strategy in the region. One key area of emphasis in the past five to ten years for the French to Africa Service, including in DRC, has been to grow its network of local affiliates -that is, local radio stations and TV channels that redistribute VOA-produced content. The Service launched the first of its own FM stations in DRC in 2016, a number that has grown to seven across some of the major cities in the country. Nevertheless, it also relies on other radio stations across the country to carry its programming. For TV, the Service is entirely reliant on affiliates in major cities -primarily in Kinshasa and Lubumbashi -to broadcast its content, making the establishment of close relationships and collaboration with these affiliates vital for the Service's success in DRC (Donangmaye and Jaafari 2021) .

The Service also identified the introduction of Lingalalanguage programming as another major area of importance, particularly in growing the total reach of the Service and serving the non-elite, non-French-speaking audiences. While the other international broadcasters in the region -particularly RFI and BBC -provide extensive French-language programming and have significant success among the audience, none of them produce programming in Lingala, giving way Fig. 3 How much do you trust the news from each of the following sources? Percentage saying "a great deal" or "somewhat" among weekly audience members for each network who offer an opinion (2016) Fig. 4 To what extent has your usage of VOA in any language increased your understanding of each of the following topics? Percentage "a great deal or "somewhat" among weekly VOA users who offer an opinion (2016) ages, educational backgrounds, occupations and listening histories. All of the panelists spoke fluent French.

Footnote 9 (continued) to a gap in the market that VOA could easily fill. In addition, 80% of the DRC population understands Lingala, twice the number of those who understand French. Over one-third of Congolese say Lingala is the language they speak most at home, with only 8.5% identifying French as their primary language at home. French-language skills also tend to be much lower among women than among men, meaning that Lingala-language programming had the potential of reaching more women, as well (Kamba 2021) . To that end, in November 2018, USAGM launched the Lingala programming with radio, and is currently offering two five-minute news updates on weekdays, featuring reporting from VOA's Africa Services and reporters in DRC, and a daily five-minute show focused on public and personal health (Inside VOA 2018) . The only Lingala offering for TV, added recently, is a versioning of the VOA60 one-minute news wrap specifically for DRC audiences. In early 2021, twenty-six affiliates -20 radio and six TV -carried Lingala programming in the country. 10 The most recent French to Africa and DRC strategy (FY 2019-2020) was written based on the data available in 2018 as well as the strengths and opportunities the Service identified using them. The data thus helped set the priority goals, operationalize the tactics, and specify targets for the Service, including in DRC, its largest market. The broader regional goals for French to Africa Service included the following:

Goal 1: Reach and engage young audiences by creating dialogue via broadcast and social and other digital media, focusing on information they need to improve their lives and societies. Goal 2: Capitalize on VOA's reputation as an unbiased, politically neutral news source and a hub for U.S. and pan-African news not otherwise available in its target markets to reach and engage opinion leaders and support the continent's democratization and development. Goal 3: Increase reach and engagement with women through targeted programming that brings women into the national dialogue and provides them with actionable information to improve their lives (VOA 2018).

Some of these goals were operationalized through program-specific plans such as continuing the coverage of certain topics in some of the shows or making specific adjustments to digital content or its presentation on the website. This could help increase reach among the audience, but also enhance trust and improve understanding of current events. Other planned tactics included closer collaboration with local affiliates to better understand their needs and audience preferences, serve as their "U.S. Bureau," and co-create programming to make it more authentic and engaging for the audience, which would then help improve their understanding of U.S. foreign policy as well as American society. In terms of reaching and engaging more women, the planned tactics included increasing the programming devoted to the coverage of health or women-specific issues, increasing the number of women experts and interviewees appearing in programs, and allocating a set budget for digital advertising targeting women. Specific performance targets for each of these goals were set accordingly.

The major DRC-specific component in the strategy comprised the consolidation of Lingala-language programming, with the primary goal of reaching underserved audiences to support the stabilization of the country (VOA 2018). The tactics outlined for the goal included two daily short news programs and a five-minute health program, accompanied by VOA-branded digital assets, including social media profiles. Performance targets were set prioritizing reach among Lingala-speakers and uniqueness of content, as well as the gradual growth in the size of and engagement with associated digital assets among target audiences.

The most recent survey in DRC, conducted in early 2020, showed significant growth in VOA's weekly audience reach for its French and Lingala programming, with French increasing from 9.0 to 14.0% over four years, and a measured audience of 6.2% for Lingala programming in a little over a year of its existence by that point (Gallup 2020) (Table 4 ). 11 Table 4 Measured weekly audience reach estimates for VOA language services in DRC (2020) 2016 (n = 2008) (%) 2020 (n = 2960) (%) VOA total weekly reach 9.0 16.2 VOA total in French 9.0 14.0 VOA total in Lingala N/A 6.2 VOA total in Swahili 2.5 1.7 VOA total in English 2.6 3.2 10 Data from USAGM's Business Development office, January 2021. 11 The nationally representative face-to-face survey was conducted between February 20 and March 29, 2020 among 2,960 Congolese adults. The target population was the non-institutionalized, civilian population, aged 15 and older living in nine provinces of the DRC (Bandundu, Bas-Congo, Equateur, Kasaï-Occidental, Kasaï-Oriental, Katanga, Kinshasa, Maniema and Orientale) as well as three urban areas in the North Kivu (Goma) and South Kivu provinces (Bukavu and Uvira). The survey was fielded in those three urban areas only as opposed to the entire geographic areas of the North and South Kivu province due to the Ebola epidemic, rebel activity, and severe insecurity issues. In addition, insecurity in parts of the provinces of Katanga, Orientale and Kasaï Occidental as of mid-October 2019 led to those areas being excluded from the population frame. In total, the geographic exclusions represented about 10 million people. As a result, the country coverage for this study was 84.5% of the estimated total population or 40,884,000 adults. Unlike the 2016 survey report, all data from the 2020 survey include the total sample in the country Most of the growth happened among audiences consuming content on radio and TV, indicating that the addition of USAGM's own FM transmitters and the continuous expansion of its local affiliates' network are paying off (Table 5) .

The 2020 survey showed that VOA had also achieved significant gains in reaching more women in DRC with 12.3% in 2020 vs. only 3.8% in 2016. Nevertheless, there is still a significant gap between male and female audiences, particularly evident for the French-language programming with nearly twice the reach among men as among women. Yet, as expected, the gender gap was significantly smaller for Lingala-language programming (7.3% vs 5.1%). The French and Lingala Services had also increased their overall reach among young audiences; however, the highest reach for both languages was among the 55 + age group, suggesting a need to realign the programming and its distribution to more closely reflect the focus on younger audiences outlined in the strategy (Table 6) .

Between 2016 and 2020, VOA saw its overall credibility slide slightly in the country from 89.9 to 82.4%. In terms of understanding, 81.1% of respondents in 2020 said VOA has increased their understanding of "current events" (vs. 92.4% in 2016), 53.5% for "U.S. foreign policy" (vs. 66.1% in 2016) and 52.7% for "American society" (vs. 66.1% in 2016) . Some of this decline is a typical function of expanding reach in the market, as the Service starts to attract new audiences who might not be as committed to the brand. The drop in these indicators can also be attributed to contextual factors, such as the overall decline of U.S. credibility around the world since 2016 or the growth in disinformation in the market leading to a general decline in trust in all media (Drew 2020) (Figs. 5, 6) . 12

Some of the apparent recommendations based on the latest set of data would include fine-tuning the content and expanding its distribution further across all platforms. While news and analytical shows constitute some of VOA's core reporting across the world, coverage of health issues has clearly been a major draw for African audiences more broadly and for the Congolese more specifically. This coverage is ever more important given the global COVID-19 pandemic, the continuing Ebola outbreak in parts of DRC, and the ongoing threat of HIV/AIDS on the continent. The Service can make this programming more appealing for youth by featuring more local and personal stories, African and Congolese experts, and explaining the complex issues involved in a way that can be easily understood, especially considering that in the case of French, it is usually a second -if not third -language for much of the audience. The Service should further reduce the gender gap, continuing with its previous strategy but also potentially experimenting with different formats and broadcast times to capture more women audiences.

The strategy of growing the network of local affiliates has been productive and resulted in a significant growth in the Service's overall reach. However, in an effort to improve its performance, the Service might focus on new opportunities for innovative collaboration or content co-creation with these local media organizations, using them not only to deliver VOA content to the audience but also to bring the audience -its preferences, but also its direct engagement -to VOA and its content (Table 7) .

Another avenue for further exploration and potential growth is the internet, where VOA had only 2.3% weekly reach in 2020. Even though the internet is much less popular or accessible in DRC than radio or TV, over one-in-five Congolese say they have used it in the past week (an increase from 15.0% in 2016). The percentage is even higher among the urban population (41.6%), the educated (25.9% secondary/vocational education and 67.5% among those with some college or more), as well as among the younger age groups (26.9% among 15-24-year-olds and 26.4% among 25-34-year-olds) -some of the primary targets of the Service. Expanding audience reach online, across platforms, will also provide the opportunity to test out new content formats -such as repackaging 30-min shows into three or four shorter videos for social media or radio broadcasts repurposed as audio files distributed through WhatsApp -as well as improve targeted advertising to reach more women and young people. The final major recommendation is to expand the programming in Lingala. As noted by the senior staff of the Service itself, the market for French, English, and Swahili-language programming in the country might be reaching a point of saturation, which is not the case for Lingala. VOA currently has the competitive advantage being the only international broadcaster to offer any content in Lingala and it should build on the success achieved in the past two years by expanding the length and variety of its broadcasts and adding original programming for TV, while continuing to build its digital strategy (Donangmaye and Jaafari 2021). Such expansion is dependent on resource availability -current Lingala programming is produced by a single reporter. Adding more core staff in D.C., as well as a network of stringers in DRC, is essential to ensuring the sustainability and continued growth of the Service and its impact on the ground. It would also need to invest in acquiring original content and translation services. Funding such an expansion would require the Service and VOA to go through the Congressional appropriations process, as USAGM cannot make significant modifications to the networks' broadcast hours without approval from the Congress.

Setting aside the requirement to secure Congressional approval, programming changes that have funding implications must also run a competitive gauntlet that includes other language services at VOA, other networks, and How much do you trust the news and information from each of the following sources? Percentage "a great deal" or "somewhat" among past-week users of each source with an opinion (2020) Fig. 6 To what extent has your usage of VOA in any language increased your understanding of each of the following topics? Results among past-week VOA users who had an opinion (2020) other Agency offices that are making their own pitches for these same resources. What is more, once the budget justification is submitted to OMB, Lingala's request -as part of the Agency's overall request -would be evaluated against all the other public diplomacy needs submitted by the State Department. If OMB decides to approve the expanded funding request for Lingala, USAGM would have to include an explanation for it in its Congressional Budget Justification, at which point it would formally become part of the appropriations bill to be considered by the House and Senate and signed by the President. Only then could the French to Africa Service start implementing the Lingala expansion. Another significant issue with the process is the length of time it takes to put such plans into action. Planning for the most recent, 2020 DRC survey began in early 2019, and the contract awarded in late 2019. Fieldwork took place in February-March 2020, wrapping up just as the whole world closed down due to COVID-related travel restrictions. Survey insights collected in 2020 fed into strategy planning sessions in 2021 to inform the FY 2022 strategy for the French to Africa Service; however, the earliest budgetary changes and expansion -even if approved -will not be possible before FY 2023. Lingala programming expansion would need to be included in the budget proposal to OMB by September 2021, and if approved, would be submitted as part of USAGM's FY 2023 Congressional Budget Justification in February 2022, to be deliberated in the fall of 2022. Based on the experience of the recent annual appropriation bills, the request for Lingala approval would likely come no earlier than December 2022 or January 2023 -almost three years after the survey was conducted and data has yielded actionable recommendations.

The Impact Model provides a clear framework for the operationalization of USAGM's strategy and the evaluation of its networks' performance across all markets where they operate. The Model combines the longer-term transformational goals with the more immediate objectives of informing, engaging, and connecting the audience. It captures indicators that go beyond just quantitative measures of reach or digital analytics to also include impact on individual audience members, local media partners, and the society more broadly. While it is difficult to make direct causal arguments about some of the outcomes or impact of USAGM programming, the Model enables the Agency to set specific performance targets, conduct a degree of assessment, and capture changes -however incremental -over time to evaluate its performance on the ground.

In terms of research and its application to enhancing performance, USAGM faces two major challenges -availability of data and availability of resources. USAGM operates in over 100 markets around the world and it would be challenging -for a variety of reasons, such as political instability or danger to participants, as well as resource constraints -to conduct audience research in all of these markets every year or even every two years. Nevertheless, the Agency is committed to developing a data-informed strategy and responding to the rapidly changing trends as well as challenges and opportunities on the ground. To that end, USAGM is exploring new approaches to ensuring research is recent, accurate, and actionable. One, already in place, is to supplement the lengthy, in-depth national surveys conducted every two to four years with annual or biannual modules that are part of larger omnibus surveys conducted by third-party vendors. These modules focus on the key metrics of interest to USAGM, such as reach, platform use, trust and understanding, and are then supplemented by other data, sometimes from other sections of that same omnibus survey -such as on the population's general media use -that can help provide a better understanding of the trends within that population.

The Agency is also working to modernize its standard questionnaire in order to ensure even greater accuracy of its data. Given how quickly technology and applications evolve, the instrument used to measure media consumption must reflect how individuals consume content today and in the future. The Agency is in the midst of revising the questionnaire that it implements globally -an effort comprising cognitive interviews and survey testing with Russian, Arabic, Spanish, and English speakers. In addition to aligning terminology with contemporary media consumption practices -e.g. capturing content access across platforms and devices, such as audio content on TV or television content over the internet (OTT) -the Agency must ensure consistency in the translation as well as in survey respondents' understanding of its research concepts across 62 languages. By updating core concepts, using less abstract terminology, such as "people and culture of the United States" vs. "American society," and detailed translation instructions, the Agency strives to maintain reliability and accuracy in measuring its impact.

Another potential approach is data modelling to measure audiences or approximate select impact indicators by estimating media use from digital metrics, based on in-depth research into the media consumption patterns of USAGM users. While attempts to build such a model have been experimental so far, data modelling is a promising area of research that could make it much easier for the Agency to evaluate its performance without having to conduct surveys in all markets every year. Related, data modelling could also be used to try and create a composite index of effectiveness, blending data from surveys and digital analytics to create a single evaluative score for each language service that could be temporally intercomparable regardless of market. As the science behind data modelling continues to advance, it is possible that a composite index of assessment could be achieved, though it would still need to account for the vast differences across over one hundred markets where USAGM's content is distributed.

The other challenge centers on the uncertainty of the budget and the time it takes to secure funding both for research as well as operational changes. The need to align resources with performance in real-time by responding to research findings and evaluation, as well as to rapidly changing circumstances on the ground, are not new to public diplomacy practitioners or federal government in general, but in USAGM's case it is amplified by the nature, scope, and range of its operations worldwide. There are several ways the Agency could be more responsive to these challenges and break its recent cycle of conducting research in USAGM markets only once every three to five years. This policy-borne out of the necessity of limited funds-threatens the relevance of research. USAGM knows where data is expiring years in advance and could develop research plans and request funding for them accordingly. More specifically, the Agency and network leadership could articulate a vision and goals for the next five to ten years, instead of the OMB-mandated four-year plans, integrating these goals into a performance framework with metrics collected at different milestones (e.g., at one year, three years, five years). For the medium-term, three-and five-year goals, the Agency could plan a budget that includes research funds for the next two years, which would ensure data in three to five years. Finally, to remedy the disconnect between the budget and the research agenda, USAGM, working with the OMB and Congress, should align the Impact Model with an appropriate funding model. The current funding model ties research and language services to the same type of appropriation as the annual operations and management of the Agency. The research-funding model should be an appropriation that does not expire annually and ensures a sufficient research budget for all language services, as planned. Breaking away from the annual appropriation cycle will provide the Agency the freedom to develop a multi-year research plan without being stymied by the cumbersome federal budget process.

The U.S. Agency for Global Media has made meaningful progress in building an organizational culture that supports improvement. For decades, USAGM has implemented quantitative and qualitative research on foreign audiences, to support effective broadcast operations. The Impact Model Framework discussed in this paper codifies a theory of change that begins with individual news consumers, who in turn make up informed publics demanding independent news from local distributors (or through circumvention tools). This is the bedrock of freer, more democratic societies. To monitor performance against this logical framework, the Agency and its networks maintain robust program review activities that integrate survey, interview and focus group data, digital metrics, audience feedback, and other anecdotal evidence.

Just as the Government Performance and Report Act directs, there is always an opportunity for the government to improve. Indeed, USAGM can continue building on achievements by enhancing its evaluation practice in key ways. Aligning budget and assessment processes would facilitate greater responsiveness. Continuing to investigate alternative, economical means of data collection will improve the consistency of data collection. And finally, investing in a modern data management system not only protects the Agency's investment in data, but would also support USAGM's commitment to transparency by efficiently ingesting data from thousands of sources, storing and processing that information for analysis, and providing the means for external parties to access that data. This would also enable greater collaboration with other government agencies, international public media organizations, and the research community, thus deepening the critical conversation around measuring public diplomacy's impact.

Government Performance and Results Act. Federal administrative procedure sourcebook

Assessing impact in global media: methods, innovations, and challenges gov/ wiki/ Gover nment_ Perfo rmance_ and_ Resul ts_ Act/ view

A resource guide to public diplomacy evaluation

Interview with the authors. Theresa Beatty, Deputy Director of Performance

The Kremlin's disinformation playbook goes to Beijing

The new public sphere: Global civil society, communication networks, and global governance

Beijing's global megaphone

Public diplomacy: Taxonomies and histories

Public diplomacy: Lessons from the past. CPD perspectives on public diplomacy

Interview with the authors. Timothee Donangmaye, VOA French to Africa Service Chief, and Salwa Jaafari, Executive Producer with VOA French to Africa Service

Kant, liberal legacies, and foreign affairs

suffers greatest global decline in trust

Strategic plan

VOA French to Africa Service Programming in DRC: Radio monitoring panel

Defining a BBG impact model: Assessment of performance measures and practices

Analytical report for media use survey of Democratic Republic of the Congo (DRC)

Gallup. 2020. 2020 analytic report for Democratic Republic of the Congo Media Use Survey

Searching for a theory of public diplomacy

Public diplomacy: Sunrise of an academic field

Toward a model of strategic influence, international broadcasting, and global engagement

The rhetoric of soft power: Public diplomacy in global contexts

Social diplomacy, public diplomacy, and network power

Scope, mechanism, and outcome: arguing soft power in the context of public diplomacy

VOA begins broadcasting in Lingala language

VOA Broadcasting in French to Africa

Measuring soft power

Linking performance and budgeting: Opportunities in the federal budget process

Interview with the authors. Ellington Kamba, Research Analyst, VOA Strategy and Audience Development

Implementation of the Government Performance and Results Act of 1993. Testimony delivered to the 104th Congress

The digitalization of public diplomacy

Seventy years of the Smith-Mundt Act and U.S. International broadcasting: Back to the future? CPD perspectives in public diplomacy

Soft power: The means to success in world politics

Future of power

International broadcasting and its contribution to public diplomacy. Lowy Institute

America first: A blueprint to make America great again

OTF. 2021. About. Open technology fund

Evaluating public diplomacy programmes

New public diplomacy in the 21st century: A comparative study of policy and practice

Articulating influence: Toward a research agenda for interpreting the evaluation of soft power, public diplomacy and nation brands

In Data driven public diplomacy: Progress towards measuring the impact of public diplomacy and international broadcasting activities

The lessons of Al Hurra television

Media and sovereignty: The global information revolution and its challenge to state power

Introduction to "international broadcasting and public diplomacy in the 21st century

Pathways of connection: An analytical approach to the impacts of public diplomacy

A multilayered approach to public diplomacy evaluation: Pathways of connection

Understanding soft power through public diplomacy in contrasting polities

GPRA Modernization Act of

Communicating with the World: U.S. Public Diplomacy Overseas

United States District Court for the District of Columbia

USAGM. 2020. FY 2020 performance and accountability report. U.S. Agency for Global Media

Who we are: Mission. U.S. Agency for global media

USAGM. 2021b. Impact and results: Measuring impact. U.S. Agency for Global Media

Behind the news: Inside China global television network. Lowy Institute

Sub-Saharan Africa Regional Strategy FY 2018

VOA French to Africa Strategy FY 2019-2020

The global public sphere: Public communication in the age of reflective interdependence

Tara sonenshine on measuring the effectiveness of public diplomacy

The World Bank in DRC: Overview

Remote negotiations: International broadcasting as bargaining in the information age

Battles to bridges: US strategic communication and public diplomacy after 9/11

Relational, networked and collaborative approaches to public diplomacy: The connective mindshift

The authors thank the three anonymous reviewers, as well as this issue's guest editors, Efe Sevin and Kadir Jun Ayhan, for their valuable comments and contributions. The authors also thank Theresa Beatty, Timothee Donangmaye, Salwa Jaafari, Ellington Kamba, Kate Neeper, James Pamment, and Kristin Searing for their time, insights, and feedback.

The authors of the article report that as employees of USAGM, which is the subject matter of the paper, they have a direct affiliation with the organization as well as potential financial and non-financial interest in the matters discussed in this manuscript. The authors have tried to mitigate this potential conflict of interest by fully disclosing this affiliation to the editors of the journal and the editors of the special issue, as well as through the double-blind peer-review process. Furthermore, none of the details or recommendations discussed in the paper involve any personal financial gain for either of the authors.