EVALUATION OF } CHILD HEALTH SERVICES: ne na SEP The Interface Between Research UBL and Medical Practice Samuel J. i Jaime Arias Scientific Edito Proceedings of a Conference Sponsored by American College of Preventive Medicine Pan American Community Health Association Bureau of Community Health Services, Health Services Administration, Office of Child Development, Office of Human Development, DHEW National Institute of Child Health and Human Development, NIH The John E. Fogarty International Center, NIH March 7-9, 1977 National Institutes of Health Bethesda, Maryland U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE Public Health Service National Institutes of Health DHEW PusLicaTiON No. (NIH) 78-1066 Information contained in this publication in no way reflects the opinions of the Fogarty International Center, the National Institutes of Health, the De- » partment of Health, Education, and Welfare, or any other agency of the federal government. For sale by the Superintendent of Documents, U.S. Government Printing Office Washington, D.C. 20402 (Paper Cover) Stock Number 017-046-00043-8 RJ 101 E93 PUBL The John E. Fogarty International Center for Advance : the Health Sciences is a federally established focal point for sharing America’s health research with more than 48 other nations of the world. The Center also functions as a forum for drawing together the talents and resources of other countries and engaging them in ways that bear on the many and varied health problems of the United States. Since 1968 when it was designated a memorial to Congressman John E. Fogarty of Rhode Island, the Center has pursued Mr. Fogarty’s vision of international scientific cooperation in the interests of mankind. The Fogarty Center’s programs, developed by medical, health, and administrative professionals, promote advanced study in biomedical and related sciences with a view toward improving health and physical well-being through practical means. Under the aegis of the Confer- ence and Seminar Program of the Center, scientists are brought together for study and discussion of topics such as liver disease, diabetes mellitus, and chronic childhood illness. The Geographic Health Studies Program sponsors analyses of the health care system and medical research of other countries to determine how their experiences might serve as models for health care in the United States. Moreover, scientists who have been singled out for their special concerns by their own countries can apply for fellowship awards through the Scholars and Fellowships Program. These awards enable them to work closely with U.S. scientists at NIH or at academic institutions throughout the country. In like manner, American researchers are sponsored for study abroad. During the years of its formation and growth, the Fogarty Inter- national Center has been dedicated to the use of science for peaceful purposes, for the good of all human beings. To this end, the present monograph has been prepared in cooperation with the American College of Preventive Medicine; the Pan American Community Health Association; the Bureau of Community Health Services, Health Services Administration; the Office of Child Development and the Office of Human Development, DHEW; and the National Institute of Child Health and Human Development. This monograph derives from a conference sponsored by the Fogarty Center which brought together 65 experts in evaluation, organizers of health care systems, directors of child health programs, FOREWORD iil 1v university-affiliated educators, social scientists, economists, policymak- ers, and consumer advocates from the United States and Latin America. Of primary concern was the need for a current evaluation of maternal and child health services delivery systems and the implications of this evaluation for health professional education in the Americas. It is hoped that the discussions in this volume will form a comprehensive report about a variety of issues related to the health of the world’s children. Miro D. LeavrTtT, JR., M.D. Director Fogarty International Center CONTENTS INTRODUCTORY REMARKS Milo D. Leavitt, Jr. Samuel J. Bosch PosiTioN PAPERS 1 Child Health in the Americas: A Historical and Global Perspective Joe D. Wray Child Health in Latin America and the Caribbean Jorge V. Rosselot Questions Behind the Answers: Why We Do Not Evalu- ate and Why We Should Abraham Horwitz Pragmatic Aspects of Program Evaluation Regarding Child Care and its Relation to Decisionmaking at the Political-Administrative Level Antonio Ordoriez-Plaja Evaluation of Child Health Care As Seen by Evaluators Jaime Arias CASE STUDIES 6 Nine Evaluation Case Studies: An Approach to Their Classification Samuel J. Bosch and Jaime Arias Sociomedical Indicators for Child Health Carlos J. M. Martini and Ian McDowell Discussion Milton I. Roemer and Gabriel Velasquez Application of the Tracer Method to Child Health Care David M. Kessner Discussion Maurice Backett and Barbara Starfield An Approach to the Evaluation of Child Health Care Through a Simulation Model Rodolfo A. Heredia, Alejandro Vivas, Luis Hernando Ochoa Vv Page xi xii 237 37 47 57 69 73 87 89 115 117 vi 10 11 12 13 14 15 Evaluation of Child Health Care in PRIMOPS: A Comprehensive Health Program in Cali, Colombia Gildardo Agudelo Gil (editorial summary) Goal Conflicts in a Child Health Station: A Problem in Action Research Jane Z. Moss, Renate Belville, Bernard Levenson Discussion Maurice Backett and Barbara Starfield Nutrition Indicators and the Barbados Preschool Child: Nutrition Intervention Program, 1967-76 Frank C. Ramsey Discussion Milton I. Roemer and Gabriel Velasquez Response of Indicators of Nutritional Status to Nutri- tional Interventions in Populations and Individuals Charles Yarbrough, Jean-Pierre Habicht, R. E. Klein, R. Martorell, A. Lechtig, G. Guzman Discussion Maurice Backett and Barbara Starfield Child Health and the Community: The Rochester Child Health Studies, 1967-77 I. B. Pless Discussion David Bersh and Carlos V. Serrano Health Accounting: An Outcome-Based Strategy of Health Care Quality Assurance John W. Williamson Discussion Milton 1. Roemer and Gabriel Velasquez Use oF EVALUATION IN EDUCATION 16 17 Mirror or Window: The Interface of Health Services Evaluation and Provider Education John L. Simon Evaluation as a Tool for Community Education Rolando Merino, Samuel J. Bosch, Ruth Zambrana Discussion: Education and Evaluation David J. Jones Page 145 153 166 169 193 195 207 209 227 229 244 249 257 266 CLOSING REMARKS Robert J. Haggerty Antonio Ordoriez-Plaja PARTICIPANTS INDEX vii Page 271 281 283 287 INTRODUCTORY REMARKS ~~ - WELCOMING REMARKS Milo D. Leavitt, Jr. In September 1975, an agreement was reached among officers of the Fogarty International Center, the American College of Preventive Medicine, and the Pan American Community Health Association to hold a workshop on child health care in the Americas that would focus primarily on methodologies for evaluating health services to children. The focus and target population were chosen because of the growing interest and activities in different countries in the Americas in evaluation of these services. This has provided an excellent opportunity for an international exchange of information. Dr. Samuel J. Bosch, of the American College of Preventive Medicine, and Associate Professor of Community Medicine, Mount Sinai School of Medicine, New York, and Dr. Antonio Ordonez-Plaja, representing the Pan American Community Health Association, and Director of the Colombian National Institute of Family Welfare, were given the responsibility for planning and chairing the workshop. Dr. Jaime Arias, of the Colombian Department of Public Health, has undertaken the important task of coordinating the reporting on evaluation projects in different countries as well as the development of that information for presentation and analysis at this workshop. Several agencies within the Department of Health, Education, and Welfare have program interest in this matter and are cosponsors of this workshop. In addition, a number of their key scientific staff are participants. The Fogarty Center staff has worked enthusiastically with the planning committee to put together what we hope will be a successful, informative conference. xi WELCOMING REMARKS Samuel |. Bosch I would like to welcome you to this meeting in the name of the American College of Preventive Medicine. The College, though basically a U.S. organization, has always recognized the importance of having close ties with professionals dedicated to preventive medicine in other parts of the world. Indicative of ties, interest, and feelings of collegiality is the fact that the College recently established a new category of international membership for physicians in the specialty of preventive medicine. In sponsoring meetings such as this, we are looking for new and better ways to develop knowledge about issues that are of common interest around the world. This particular meeting has been planned and organized by representatives of the sponsoring institutions. The planning has defined the following goals for the meeting: —To establish channels of communication between health care evaluators from different disciplines and from different countries in the Americas. —To promote an exchange of information and experiences in evaluation techniques: comparing approaches, methods, needed resources, difficulties, achievements, and failures. Evaluation of health care was selected as a topic because it was recognized that this problem is common to many countries, and one about which we still have very limited knowledge. The planning also stressed the fact that language and cultural barriers can be minimized through this topic. The team chose child care as the main focus because solving child health problems is a recognized priority in most of the American countries. We hope that this conference will provide each of us an opportu- nity for analysis of our own experience in evaluation and for thought about its implications for each of our countries. xiii Ee a Ea we TT er POSITION PAPERS 1 CHILD HEALTH IN THE AMERICAS: A HISTORICAL AND GLOBAL PERSPECTIVE Joe D. Wray Introduction Child health in the Americas is a reflection of the quality of health services provided. The health of children depends, of course, on other things as well. The socioeconomic status of their families is surely one of the most important factors. Many thoughtful observers believe, in fact, that the health status of children—especially in the first few years of life—is one of the best indicators of a country’s social and economic development. Those of us who are concerned, professionally or otherwise, with the welfare of children in the Americas know that the available data indicate extreme variation in the status of children in different countries of our hemisphere. We are fortunate in that this information, uneven as it may be in some respects, is far better in this hemisphere than in most other regions of the world. The Pan American Health Organization (PAHO) deserves credit for much of this information; among the most revealing of the PAHO studies is Patterns of Mortality in Childhood (Puffer and Serrano 1971). Mortality, of course, is a negative indicator of health, but it is the best we have and it is extremely useful. By examining the trends in mortality patterns over the last few years we can at least estimate the progress that has been made in the status of child health in the Americas. By studying the available data concerning certain other specific, major problems that afflict the children in this hemisphere, we can get some idea of where we should concentrate our efforts to improve child health in this region. Finally, by placing this study in both a historical and a global perspective, we can derive some estimates of what is likely to happen if present trends continue. Trends in Childhood Mortality in the Western Hemisphere Two simple but revealing indicators of the status of child health are the proportion of all deaths that occur in children under age 5, 3 4 EVALUATION OF CHILD HEALTH SERVICES 50 Lo 30 Percent 20 Under 1 year 1964 1968 1972 1964 1968 1972 1964 1968 1972 Northern Middle South America America America Figure 1. Percentage of deaths under age 5, Western Hemisphere, 1972 [Redrawn from PAHO 1974] and infant mortality rates. Figure 1 shows the changes in the proportion of deaths occurring in children under ages 1 and 5 in Northern America, Middle America, and South America since 1964. We know from Northern American data, as well as from similar European data, that when children are well nourished and adequately protected from infection, the proportion of deaths in children under age 5 is small. Evidence (figure 1) from a PAHO review of health conditions in the Americas (PAHO 1974) strongly suggests that the average child in Middle and South America is not yet receiving adequate nutrition, protection from infection, or proper medical care. Figure 2 shows the trends in infant mortality in Middle and South America, as well as the goals agreed upon at Punta del Este (PAHO 1974, 1971). Although the rates have declined in the past 15 years, it is clear that excessive numbers of infants are dying in the first year of life, both in Middle and South America. Some indication of the numbers of deaths for specific, mostly preventable, diseases, is given in figure 3. If literally thousands of children are dying needlessly, what can we conclude about child health services? As we contemplate such mortality rates, we should bear in mind a statement made by Dr. Cicely Williams (1953) many years ago: Few persons realized that “the survival of the fittest” is a misapplied cliché. It was not recognized that the same conditions that will kill 30% of the babies in the first year of life will also produce a large proportion of persons with damaged lives who will be a burden for years and perish at a later date. Child Health in the Americas 5 80 wn + reeee... = Sressestan, : [ep es, SOUTH AMERICA 2 60 — EEE] wi .- Teeg, Yc hsneveionnnsniane = SHS Mus . —— WI . S SISO] dren = Pa “ey o i Se —— ~ MIDDLE AMERICA <4 - ~ a, 7% «Lo ~ ees i “ISL Th wn [as 5 i 2 —illllGoals Im ~ oy 7 — w= Actual rates l 3 l L T v T T 1960 1965 1970 Figure 2. Goals for infant mortality reduction and rates actually achieved, Latin America, 1960-72 [Redrawn from PAHO 1971, 1974] Influenza, pneumo- Diarrheal Nutritional nia,and bronchitis diseases deficiency Whooping cough, All other measles, diphtheria, infectious and and poliomyelitis parasitic diseases Expected on basis oe of U.S. rate Dom. Exes deaths Figure 3. Numbers of deaths expected in children under age 5 on the basis of U.S. rates, and excess deaths, by various causes, Latin America 1970 [Redrawn from PAHO 1971] I11-defined causes 6 EVALUATION OF CHILD HEALTH SERVICES Table 1. Neonatal Mortality by Birth Weight, Ribeirao Preto and California Projects, Inter-American Investigation of Mortality in Childhood and United States 1960 Birth Cohort (deaths per 1,000 live births) Ribeirao Preto Birthweight California project (grams) project 1968-70 1,000 or less 940.0 890.6 1,001-1,500 434.8 696.3 1,501-2,000 106.9 391.0 2,001-2,500 28.2 68.1 2,500 or less 129.9 215.7 2,501-3,000 5.4 16.8 3,001-3,500 2.5 8.5 3,501-4,000 22 6.0 4,001 and more 3.1 8.7 2,501 and more 3.1 10.0 Total 12.7 28.2 Source: Puffer and Serrano, 1975. Reproduced with permission of Pan American Health Organization Fortunately, 30 percent of babies no longer die in the first year of life, but Dr. Williams’ admonition to remember the damage done to those who survive is surely as applicable today as it was 25 years ago. Low Birth Weight and Mortality A major cause of excessive infant mortality everywhere is low birth weight. A recent analysis showed, for example, that a significant proportion of racial differences in infant mortality rates in the United States could be accounted for by differences in birth weights: In those groups with higher mortality rates there is a higher percentage of babies with low birth weights (Habicht et al. 1974). Similarly, a recent study in England revealed that much of the variation in infant mortality is associated with only two factors: birth weight and social class—and the two are obviously interrelated (Neligan 1974). Variations in mortality with birth weight are striking and were clearly shown in the Pan American Health Organization study of mortality in childhood (Puffer and Serrano 1971, 1975; Serrano and Puffer 1974). Table 1 shows the mortality in the first month of life, by birth weight, in two of the PAHO study sites in Brazil and the United States. The high rates in the lowest weight groups are expected; the striking fact is that rates in infants weighing between 2,501 and 3,000 grams, a weight range usually considered “normal,” are approximately twice as high as those in infants with birth weights over 3,001 grams. Child Health in the Americas 7 CALIFORNIA BRAZIL MEX ICO EL SALVADOR 40 — Ad o Percent I) o l — I I Birth r 2501- 3001- 3501- weight [Jf < 2500 3000 Ef 1500 Ed” hoo [J sore (gms) Figure 4. Distribution of live births, by birth weight, various countries, 1972 [Redrawn from Serrano and Puffer 1974] The full significance of this becomes apparent when we examine other findings of the PAHO study showing the distribution of birth weights in various places. In figure 4, it is clear that the number of newborns with weights that place them at higher risk is much greater in the Latin American sites than in California (Serrano and Puffer 1974). The importance of this finding is tremendously increased by recent observations at the Instituto de Nutricion de Centro America y Panama (INCAP). Studies there have shown that dietary supplemen- tation of pregnant women can significantly increase birth weights (Habicht et al. 1974); Lechtig et al. 1975a; Lechtig et al. 1975b). The average weight of babies born to mothers who had received supple- ments providing at least 20,000 calories during either the second or third trimester was 200 grams higher than that of babies born to unsupplemented mothers. Thus, INCAP has demonstrated that a simple and not very costly intervention during pregnancy can increase birth weights and enhance the probability of survival. Malnutrition and Mortality The association between malnutrition and infection, and the fre- quently resulting mortality, is well known (Scrimshaw et al. 1968). If we wish to understand the high mortality rates that prevail in many parts of the Western Hemisphere, we should examine the data from Puffer and Serrano, whose study (1971) showed clearly the role of 8 EVALUATION OF CHILD HEALTH SERVICES BRAZIL All causes a Diarrheal wd diseases Recife Riberdo Preto Sao Paolo Measles COLOMBIA Percent 80 Respiratory infections Percent esesnand l 3 i Cali Cartagena Medellin Figure 5. Percentage of deaths in children under age 5, with malnutrition as associated cause, various underlying causes, Latin American cities, 1970 [Redrawn from Puffer and Serrano 1971] malnutrition in producing infant and early childhood mortality. From a wealth of thought-provoking data in their report, only a small sample can be presented here. Figure 5 shows the importance of malnutrition as an associated cause of death in children under age 5, as observed in six cities included in their study sample, three in Brazil and three in Colombia. Malnutrition was an associated cause of death in about 50 percent of all cases in the six cities. It was a consistently important associated cause in diarrheal diseases and measles, ranging from 50 percent to almost 80 percent, and was found in 30 to 45 percent of those children dying of respiratory infections. In figure 6 the percentage of deaths in children under age 5 in which malnutrition was found to be either an underlying or an associated cause is shown by age group. Such deaths were not too prevalent in the first month of life, but became more so between the first and the sixth months. From the sixth to the eleventh months, the proportion reached almost 50 percent and thereafter remained high. What is especially striking about these findings is that malnutri- tion increased in each succeeding year as an underlying (i.e. primary) cause of death throughout the first 5 years of life. Clearly, malnutri- tion is not a problem that children simply pass through in early childhood, but one that persists and, in fact, becomes even more important in later childhood. Child Health in the Americas 9 Associated cause : Percent ‘Underlying cause 1 mo 6 mos 1 yr 2 yrs 3 yrs 4 yrs 5 yrs Figure 6. Percentage of deaths in children under age 5, with malnutrition as underlying or associated cause, by age, Latin America, 1970 [Redrawn from PAHO 1971] We know from a number of surveys carried out in various parts of the Western Hemisphere that in many countries malnutrition in children under age 5 is indeed widely prevalent. Figure 7, for example, shows the results of surveys carried out among Central American countries (PAHO 1971). Nowhere is the overall rate significantly less than 50 percent and it is over 70 percent in three of these countries, where almost one third of the children have moder- ate or severe malnutrition. Percent 20 Lo 60 80 100 Guatemala El Salvador Honduras Nicaragua Costa Rica Panama D f ot 1a Severe Moderate 0] Mild ] Normal Figure 7. Percentage of children under age 5 with various grades of malnutrition, in six Latin American countries, around 1970 [Redrawn from PAHO 1971] 10 EVALUATION OF CHILD HEALTH SERVICES Even if we ignore the tragic results of malnutrition as reflected in the mortality data, we must not forget the long-lasting consequences of malnutrition in these children. Surely we must acknowledge that malnutrition is a major factor in determining the health status of young children in many parts of the Americas today. This concept is not new. Pediatricians, nutritionists, and others have been well aware of the problem of malnutrition in Latin America for the last 15 or 20 years. Dozens of national and international conferences have been held; thousands of articles and probably scores of monographs have been published. Yet in many parts of this hemisphere the problems remain essentially the same. Surely, eradicating malnutrition in children under age 5 poses the most serious of all challenges to those concerned with the health of children in the Western Hemisphere. Breast Feeding and the Health of Infants Malnutrition in early childhood is the result of a great many interacting factors. We know, however, that breast feeding in the very early months of life provides fully adequate nutrition, as well as protection from infections. Unfortunately, we also know that breast feeding is being abandoned in many parts of the Americas, especially in the urban areas. This trend must be a matter of serious concern to all pediatricians and others responsible for the care of children. The increased risks associated with artificial feeding of infants were Table 2. Morbidity and Mortality Rates per 1,000 Infants, Ages 1 to 9 Months, and Fatality per 100 cases, by Diagnosis and Type of Feeding, Chicago, U.S.A., 1924-29 Partially Ratios, Breast- Bottle- Breast: Diagnosis Breast-Fed Fed Fed Bottle Respiratory Morbidity 279.9 339.9 389.6 1:1.4 Mortality 0.4 5.1 53.9 1:134.7 Case fatality 0.15 1.5 13.8 1:92 Gastrointestinal Morbidity 51.8 120.4 158.8 1:3.1 Mortality 0.2 0.7 8.2 1:41.0 Case fatality 0.4 0.6 52 1:13.0 “Unclassified” Morbidity 33.0 59.9 81.4 1:2.5 Mortality 0.7 2.9 19.3 1:27.6 Case fatality 2.2 4.9 25.7 1:10.8 Infants at risk 9,749 8,605 1,707 Source: Grulee et al, JAMA 103: 735-748. Copyright 1934, American Medical Association. Further reproduction prohibited without permission of copyright holder. Child Health in the Americas il apparent in the late 19th century (Knodel and van de Walle 1967). Repeated studies since then have confirmed the early findings (Howarth 1905, Davis 1913, Woodbury 1922, Grulee et al. 1934, 1935, Robinson 1951). Table 2, for example, shows the variations in morbidity and mortality rates by type of feeding and by selected diagnosis during the first 9 months of life, as shown in a study carried out in Chicago in the late 1920s. There are striking differ- ences in mortality rates from respiratory infections, diarrheal diseases, and other infections, depending on the type of feeding. Morbidity rates are by no means so different, although they are of course higher in bottle-fed babies. Thus, breast feeding appears to provide modest protection from infection, but a great deal of protection from dying. Although comparable current information from Latin America is limited, it shows clearly that the excessive risk associated with bottle feeding is surely still present today in much of that area. In figure 8, the results of a study carried out in Chile a few years ago (Plank and Milanesi 1973) show that there is an excess of mortality among partially breast-fed and even more among bottle-fed babies at ages 4 60 SIE [Jereast-red i Fo] Partially breast-fed Bottle-fed Batatals w = wi o o o nN o Deaths per 1.000 infants 4 weeks 3 months 6 months Figure 8. Mortality rates during the first year of life in breast-fed, partially breast-fed, and bottle-fed infants, among those surviving at 4 weeks, 3 months, and 6 months, rural Chile, 1969-70 [Data from Plank and Milanesi 1973] 12 EVALUATION OF CHILD HEALTH SERVICES Table 3. Proportions of Infants Breast-Fed 6 Months or Longer in the Population, and Among Infants Dying at 6 to 11 Months, in 4 PAHO Study Areas, Around 1970 Percent Breast-Fed Ratio of : Mortality Infants Dying? Risk for Total Infant at 6-11 Mos. Breast-Fdg. Population < 6 mos. Study Areas < 6 mos. = 6 mos. < 6 mos. = 6 mos. = 6 mos. El Salvador” 20 80 78.0 22.0 14.2: 1 Kingston, Jamaica® 51 49 87.4 12.6 71:1 Medellin, Colombia? 61.8 31.2 91.3 8.8 6.4: 1 Sao Paulo, Brazil® 77.2 228 95.9 4.1 6.8:1 Sources: (a) Puffer and Serrano 1971, (b) Menchu et al. (1972), (¢) Grantham-McGregor and Beck, 1970, (d) Oberndorfer and Mejia (1968), (e) Iunes et al. 1975 Nn MEXICO VENEZUELA o [ec] Ratio of mortality rates to Canadian rates CUBA 1 - I m1 Age © group v 1-6d | — - - > > > > -— o~ on ~T 7-27d 1-11mo lyr 2yr 3yr Lyr =< 1d 1-6d 7-27d 1-11mo Figure 9. Ratio of mortality rates at various ages under 5 years to Canadian rates for those ages, four Latin American countries, 1972 [From data in PAHO 1974] Child Health in the Americas 13 weeks, 3 months, and even at 6 months. Further confirmation of this is shown in table 3, based on mortality data from the PAHO study and from four other studies of the prevalence of breast feeding in the total population of four Latin American cities (Menchu et al. 1972, Grantham-McGregor and Beck 1970, Oberndorfer and Mejia 1968, Iunes et al. 1975). It is clear that the risk of death between ages 6 and 11 months is 7 to 14 times greater among infants who were breast fed for less than 6 months. Indirect evidence of the protection provided by breast feeding is seen in figure 9, which shows the ratio of the 1972 mortality rates at various ages during infancy and childhood in a number of Latin American countries to those for the same ages in Canada (PAHO 1974). As Monckeberg pointed out some years ago (1970), where mortality ratios are high in the first year of life, we can expect that bottle feeding is common and produces early, severe malnutrition. High mortality ratios in the second through fourth years of life are more indicative of a poor environment as well as the limited availability of medical care. If Monckeberg’s conclusions are correct, we can assume, as he did, that in his country, Chile, bottle feeding is indeed a problem while health status and medical care are relatively better beyond the age of 1 year. In Venezuela, on the other hand, bottle feeding may be somewhat less common, as reflected by lower ratios under age 1 year, but health care after the first year of life is probably less adequate than it is in Chile. Diarrheal Diseases in the Children of Latin America For many years we have known that respiratory infections and diarrheal diseases are the leading causes of death in children under age 5 throughout the developing world. It is instructive to review the infant mortality patterns in New York City, by cause of death, between 1898 and 1931, as shown in figure 10 (McDermott 1966). During the 33-year period infant mortality fell from 140 per 1,000 live births to well under 60 per 1,000 live births. As is clear in the figure, much of this decline in mortality occurred because of the steady decrease in deaths from diarrheal diseases and from respira- tory infections. McDermott (1966), as well as Dubos (1966), has pointed out that at no time during this period did the health professionals have available any specific effective preventive or curative measures for either of these conditions that were killing so many infants. The number of deaths decreased, they suggest, because of a variety of improvements in the standard of living, particularly in nutrition, rather than because of any specific, easily transferred technology. 14 EVALUATION OF CHILD HEALTH SERVICES 100 [eo] o on o = o Deaths per 1,000 live births N o All other cause Phi Slay hanes 1900 1905 1910 1915 1920 1925 1930 ital mal format i Ome =e dee fee vee ene eeserd Figure 10. Infant mortality rate, by cause, New York City, 1898-1931 [Redrawn from McDermott 1966] Paraguay | | | +— New York rates | Guatemala EY rovorcion due to | » Vv Ecuador diarrheal diseases S$ go Chile > Peru ~—1972 country rates 1 9 Argentina <+—°Proportion due to > as Mexico I diarrheal diseases ] ° Be: i JCosta Rica g HHH . El Salvador! 2 Ng #, Venezuela | - : Dominican Republic! 8 Uruguay nw 40 Params ££ F Cuba 3 / a = ] 20 1915 1920 1925 1930 1935 1940 1945 1950 Figure 11. 1972 infant mortality rates and proportion of deaths due to diarrheal diseases in Latin American countries, plotted against infant mortality rates and diarrheal deaths in New York City, 1915-50 [Data from McDermott 1966, New York City Health Department 1976, PAHO 1974] Child Health in the Americas 15 The status of diarrheal diseases in the Americas today is shown in figure 11. The infant mortality rate and the proportion of that rate due to diarrheal diseases in New York City between 1915 and 1950 was drawn. Then, on the New York infant mortality curve the current infant mortality rate for a number of Latin American countries (PAHO 1974) was plotted and the proportion of that mortality due to diarrheal diseases indicated with a bar. As the figure shows clearly, current infant mortality rates in these countries are in a range comparable to levels in the United States between 25 and 60 years ago. It is interesting to note, furthermore, that the proportion of mortality due to diarrheal disease in contemporary Latin countries is approximately equal to that in the United States when infant mortality rates were equally high. Given the importance of diarrheal diseases as a cause of death in infants and young children, and the association with environmental factors including nutrition, it is clear that much remains to be done. Health Status of Children and the Economy For at least 100 years, physicians have recognized a strong correla- tion between the socioeconomic status of families and mortality rates in infants of those families (Newman 1906). Table 4, for example, shows the infant mortality rate by income group in Sweden over 50 years ago (Titmuss 1943). Note that in the highest income class, the infant mortality rate was already below 15 per 1,000 live births—a level yet to be reached in the United States (Wegman 1976). Some evidence of the exquisite sensitivity of child health to economic changes as reflected by mortality rates is shown in figure 12. There, we see changes in economic trends as indicated by unemployment rates, and changes in the trends of infant death rates within the first day of life, between 1915 and 1965 (Brenner 1973). These variations in mortality in infants in the first 24 hours obviously Table 4. Infant Mortality Rates, by Income Class, Stockholm, 1918-22 Income, Swedish crowns? 4,000- 6,000- < 4,000 5,999 9,990 > 10,000 Neonatal 24.0 15.1 19.7 11.4 Postneonatal 24.9 23.2 12.2 2.9 Infant 48.9 38.3 31.9 14.3 2US. $1 = Sw. cr. 3.3 (2) Source: Rietz, cited by Titmuss, 1943 16 EVALUATION OF CHILD HEALTH SERVICES | | Oo, Infant mortality rate under 1 day ° Fie / oe ? 3 + wn + — o “does . Ny 5 pd Percentage deviation from trends Economic trend* 1 N o 4 1920 1930 1940 1950 1960 Figure 12. Association between deviations in infant mortality rates in the first 24 hours and in economic trends, United States 1915-65 [Redrawn from Brenner 1973] cannot be due to differences in their external environment. They must instead be related to differences in the mother’s circumstances or physical condition, which can be due to economically related environmental circumstances. The findings suggest that as economic conditions worsen, indicated by increasing unemployment, something happens in mothers that increases the likelihood that infants they produce will die within 24 hours. Given the well-established impor- tance of birth weight as a factor in infant survival and the recently available evidence concerning the impact of maternal nutrition on birth (Habicht et al. 1974, 1975; Lechtig, et al. 1972, 1975), it seems altogether likely that what we are seeing reflects the impact of economic conditions on the nutritional status of the mother—which, in turn, affects the birth weight and thus the survival of her infants. Evidence that economic factors have as powerful an impact on infant mortality in Latin America as in the United States is presented in figure 13, which shows the infant mortality rate in Sao Paulo between 1950 and 1970 and real income during that same period (Leser 1973). As income improved between 1950 and 1960 there was a generally substantial decrease in infant mortality. As real income declined between 1960 and 1970, infant mortality rates began to rise and were finally as high in 1970 as they had been in 1950. Child Health in the Americas 17 110 Real income S o w -— L£ "100 i o £o a) 2 : 0 — w 8 90 8 2 S z - 8 © g a _ 80 2 2 Infant mortality rate—e © & a 70 1950 1955 1960 1965 1970 Figure 13. Variations in real income and infant mortality rates, Sdo Paolo, Brazil, 1950-70 [Data from Leser 1973] The health status of children, as noted earlier, is a powerful indicator of the economic status of the mass of ordinary people. The evidence, however, indicates clearly that economic factors are power- ful determinants of the health of children. As physicians we surely have an obligation to apply our medical knowledge and skills to improve the health of children in the Americas. If we truly wish to help the children of Latin America, we also have an obligation to do what we can to eliminate poverty. Eliminating poverty means more than merely increasing the gross national product. Figure 14 shows the relation between income per person and infant mortality rates in a number of countries (World Bank 1975, UNESCO 1975). There is a clear general trend: As income goes up, infant mortality goes down. More interesting, however, are the exceptions to the trend. Some countries with high incomes have relatively high mortality rates; others have low incomes with relatively low mortality. How can we account for these excep- tions? We do know that in many countries with higher income and higher mortality rates the richest 5 percent of the population collect from one fourth to one half of the income while the poorest 40 percent receive 15 percent of the income or less, sometimes much less. The available estimates of the income shares received by the 18 EVALUATION OF CHILD HEALTH SERVICES w 150 + - - elIndia o 125 2 oe Brazil ~ 100 S Ecuador i. * ,dGuatemala © 75 Colombia SChile ® Peru 2 a e Kerala State ; * Mexico 2 504——=e Sri Lanka fests Rice ® Venezuela | Ru 3 Cuba e Panama 25 o ' ' 1 T T T 200 Loo 600 800 1000 1200 Dollars per person per year Figure 14. Association between per capita income and infant mortality rates, various countries, 1972 [Data from World Bank 1975, UNESCO 1976] richest 5 percent and the poorest 40 percent in the Latin American countries included in the figure are shown in table 5 (Adelman and Morris 1973). The situation of the very poorest is worse. In Mexico, for example, the poorest 20 percent of the population attempt to survive on 1.3 percent of the income (Barnet and Muller 1974; see also Power 1975). The poor health status of many children in Latin America is not surprising. Table 5. Income Distribution Estimates in Various Latin American Coun- tries Percentage Share of Income Poorest Richest Country 40 Percent 5 Percent Brazil 12.9 38.40 Chile 15.0 22.60 Colombia 7.3 40.4 Costa Rica 13.3 35.0 Ecuador 16.9 33.7 Mexico 10.5 28.5 Panama 14.3 34.5 Peru 8.8 48.3 Venezuela 13.4 23.2 Source: Excerpted from Adelman and Morris, Economic Growth and Social Equity in Developing Countries. Stanford University Press, 1973 Child Health in the Americas 19 On the other hand, we know that where income is low and infant mortality is low, as is the case in Sri Lanka and Kerala, there has usually been significant land or income redistribution (Grant 1976, Ratcliffe 1976). There is more to it than that, of course. Both Sri Lanka and Kerala have extensive (even though not expensive) health services, and literacy rates are high, for example. But the result of such factors, coupled with more equitable income distribution, is a minimally adequate standard of living for the poorest families—and their children benefit. Conclusions This brief sampling of available data concerning the status of children in the Americas indicates very clearly that among infants and children in Latin America, there is still much suffering that is undoubtedly preventable and there are many thousands of needless deaths. As we have seen, many of these deaths are associated with malnutrition; many others are caused by diseases which are com- pletely preventable by immunization or prompt and often simple treatment. The health services required to prevent these needless deaths are neither complex nor terribly costly: a decent diet for mother during pregnancy and lactation; breast feeding early and an adequate weaning diet later; immunizations; simple, early treatment for respiratory infections and diarrhea. If this short list of health service measures—which are surely not mysterious, or even sophisticated—were provided, infant and early childhood mortality could easily be cut in half. A decent standard of living for those children and their families in the poorest population groups would have the same effect. What are we waiting for? REFERENCES Adelman, I., and Morris, C.T. (1973) Economic Growth and Social Equity in Developing Countries. Stanford: Stanford University Press. Barnet, R.J., and Miller, R.E. (1974) Global Reach: The Power of Multinational Corporations. New York: Simon and Schuster. Brenner, M.H. (1973) Fetal, infant, and maternal mortality during periods of economic instability. International Journal of Health Services 3: 145-159. Davis, W.H. (1913) Statistical comparison of the mortality of breast-fed and bottle-fed infants. American Journal of Diseases of Children 5: 234-247. Dubos, R., Savage, D., and Schaedler, R. (1966) Biological Freudianism: Lasting effects of early environmental influences. Pediatrics 38: 789-800. Grant, J.P. (1976) The changing world order and the world’s poorest billion: A fresh approach. Paper presented at the 25th Pugwash Conference, Madras, India. Grantham-McGregor, S., and Beck, E.H. (1970) Breast-feeding in Kingston, Jamaica. Archives of Diseases in Childhood 45: 404-409. Grulee, C.G., Sanford, H.N., and Herron, P.H. (1934) Breast and artificial feeding. Journal of the American Medical Association 103: 735-748. 20 EVALUATION OF CHILD HEALTH SERVICES Grulee, C.G. (1935) Breast- and artificially-fed infants. A study of the age incidence in the morbidity and mortality in 20,000 cases. Journal of the American Medical Association 104: 1986-1988 Habicht, J-P., Lechtig, A., Yarbrough, C., et al. (1974) Maternal nutrition, birth weight and infant mortality. In Size at Birth, pp. 353-377 CIBA Foundation. Amsterdam, Associated Scientific Publishers. Howarth, W.J. (1905) The influence of feeding on the mortality of infants. Lancet ii: 210. ITunes, M., Sigulem, D.M., and Campino, A. (1975) Estado nutricional de criangas de 6 a 60 meses no municipio de Sao Paulo. Sao Paulo: Escola Paulista de Medicina. Knodel, J., and van de Walle, E. (1967) Breast-feeding, fertility and infant mortality: an analysis of some early German data. Population Studies 21: 109-131. Lechtig, A., Habicht, J-P., Delgado, H., et al. (1975a) Effect of food supplementation during pregnancy on birth weight. Pediatrics 56: 508-520. Lechtig, A., Yarbrough, C., Delgado, H., et al. (1975b) Influence of maternal nutrition on birth weight. American Journal of Clinical Nutrition 28: 1223-1233. Leser, W. (1973) Crescimento da populacao de cidade de Sao Paulo, entre 1950 e 1970, e seu reflexo nas condicoes de sauda publica (Mimeo., unpublished). McDermott, W. (1966) Modern medicine and the demographic disease pattern of overly-traditional societies: a technologic misfit. Journal Medical Education 41: (suppl.), 137-162. Menchu, M.T., Flores, M., Lara, M.Y., Béhar, M., (1972) Lactancia y destete en el area rural de Centro America y Panama. Archivos Latino-americanos de Nutricién 22: 83-99. Monckeberg, F. (1970) Factors conditioning malnutrition in Latin America, with special reference to Chile. In Malnutrition is a Problem of Ecology, Gyorgy, P. and Kline, D.L. (eds.), pp. 23-33. Neligan, G., Prudham, D. (1974) The Formative Years. London: Oxford University Press. Newman, G. (1906) Infant Mortality: A Social Problem, London. Oberndorfer, L., and Mejia, W. (1968) Statistical analysis of the duration of breast- feeding (A study of 200 mothers of Antiquoia Province, Colombia). Journal of Tropical Pediatrics 14: 27-42. Pan American Health Organization (1974) Health Conditions in the Americas. 1969- 1972. Scientific Publication, No. 287. Washington, Pan American Health Organiza- tion. Pan American Health Organization (1971) Facts on Health Progress-1971. Scientific Publication, No. 277. Washington, Pan American Health Organization. Plank, S. and Milanesi, L. (1973) Infant feeding and infant mortality in rural Chile. Bulletin World Health Organization 48: 203-210. Power, J. (1975) The alternative to starvation. Encounter 45: 11-33. Puffer, R., and Serrano, C.V. (1971) Patterns of Mortality in Childhood. Scientific Publication No. 262. Washington, Pan American Health Organization. Puffer, R., and Serrano, C.V. (1975) Birth Weight, Maternal Age, and Birth Order: Three Important Determinants of Infant Mortality (Scientific Publication No. 294). Washington: Pan American Health Organization. Ratcliffe, J.W. (1977) Poverty, politics, and fertility: The anomaly of Kerala. Hastings Center Report, pp. 34-42, February. Robinson, M. (1951) Infant morbidity and mortality: A study of 3266 infants. Lancet i: 788-794. Scrimshaw, N.S., Taylor, C.E. and Gordon, J.E. (1968) Interactions of Nutrition and Infection. WHO Monograph Series No. 57. Geneva: World Health Organization. Serrano, C.V. and Puffer, R. (1974) Utilization of hospital birth weights and mortality as indicators of health problems in infancy. Bulletin of the Pan American Health Organization 8: 325-345. Titmuss, R.R. (1943) Birth, Poverty and Wealth: A Study of Infant Mortality. London: Jamish Hamilton. UNESCO (1975) Poverty, Unemployment and Development Policy. New York: United Nations. Child Health in the Americas 21 Wegman, M. (1976) Annual review of vital statistics. Pediatrics 58: 798-800. Williams, C.D. (1953) Kwashiorkor. Journal of the American Medical Association 153: 1280-1285. Woodbury, R.M. (1922) The relation between breast and artificial feeding and infant mortality. American Journal of Hygiene 2: 668-687. World Bank (1975) Health Sector Policy Paper. Washington, D.C.: World Bank. 2 CHILD HEALTH IN LATIN AMERICA AND THE CARIBBEAN Jorge V. Rosselot INTRODUCTION The. processes of growth and development which are the outstand- ing characteristics of infancy and childhood can be adversely affected by multiple factors. Prominent among these are socioeconomic and cultural backwardness and a population growth that outstrips the resources of the social sectors, including the health sector. Not only can the resultant problems affect child health and later stages of the life cycle, but the well-being of the family as a whole may also be impaired (Rosselot 1972, PAHO 1973). Conventionally, the portion of the population affected covers a generation, which may be considered to begin immediately following conception and to continue through the embryonic and fetal phases (gestation), and then, after delivery, through infancy, childhood, adolescence, and youth (Rosselot 1974, WHO 1974). The cycle is then closed, and a new cycle begins. THE RIGHTS OF CHILDREN Children and adolescents need protection, both in the special process of adaptation set in motion by development, which makes them especially vulnerable to the environment, and in their prepara- tion for a working life, which subsequently enables them to be effective participants in national development. At present, all countries are well aware that the commitments embodied in the Declaration of Human Rights must be honored and translated into an acceptable level of living, health, and well-being. This standard includes special care and assistance during infancy. Specifically, the principles regarding the protection of infants that were embodied in the 1959 United Nations Declaration of the Rights 23 24 EVALUATION OF CHILD HEALTH SERVICES of Children (Instituto Interamericano del Nino 1961) must be put into effect. These include: —Right to a name and nationality —Right to social security and health —Right to full and harmonious development —Right to family love —Right to education and recreation —Right of handicapped children to special treatment, education, and care —Preemptive right to aid —Right to protection in employment —Right to protection against social discrimination The responsibility for satisfying the needs of children falls on the family, the community, the State, and, to a significant degree, on international cooperation agencies. These needs vary greatly. Those for health, education, social welfare, vocational counseling, and employment are of high priority. Thus, fulfillment of these principles calls for a firm policy decision, which must be implemented through the effective coordination (at both the technical-administrative and intersectoral levels) of the programs of the government and private agencies concerned and through the continuing and active coopera- tion of the target communities. It is obvious that the indicators for evaluating these programs should be direct and comprehensive, allowing for the assessment of the various components of the health and welfare processes. STATUS OF CHILD HEALTH IN LATIN AMERICA AND THE CARIBBEAN Health Problems The estimated population of Latin America in 1976 was 326 million; the projected population around the year 2000 is 606 million. The basic demographic indicators are birth rate, 37.0 per thousand; general mortality rate, 9.0 per thousand; infant mortality rate, 75.0 per thousand; and annual population increase, 2.8 percent. As may be seen from table 1, 59 percent of the population lived in urban areas, and children under age 15 represented 42 percent of the population. The average per capita income was around $940. The table also shows that these parameters vary appreciably in the different subregions and countries of Latin America. Child Health in Latin America 25 At present, the stage of child health in Latin America and the Caribbean reflects a high prevalence of preventable diseases. This phenomenon is due, among other things, to the vulnerability of certain age groups to harmful environmental influences and to the current health policies. The status of child health is indirectly reflected in quantifiable risks of mortality and morbidity during childhood. Among the indicators normally used to evaluate the measurement of the child health status is mortality in the first year of life, which in underdeveloped countries is of limited validity because of the underregistration of deaths (Montoya 1974). These indicators do not bring out the subsequent risk of mortality in the various stages of infancy or the possible health risks to survivors arising from the disabling sequelae of disorders that may have affected them at an earlier age. Table 2 shows that in Latin America and the Caribbean the health risks are also often high in childhood. This fact and the foregoing considerations taken together reflect a manifestly bad situation, especially when it is compared with that in North America: In about 1974, deaths in children under age 5 in Middle America and South America accounted for 37.2 percent and 31.2 percent respectively of all deaths, as opposed to 3.3 percent in North America (PAHO 1977). In Latin America and the Caribbean, the leading causes of death in children under age 5 are enteric diseases (around 23.5 percent), respiratory diseases (between 9.0 and 34.5 percent), and diseases preventable by immunization (approximately 10 percent). It is not easy to gauge the role played by diseases included in Section XV of the International Classification (WHO 1967, pp 291-301), “Certain Causes of Perinatal Morbidity and Mortality,” whose rates, despite deficiencies in registration, have in recent years ranged between 6.3 and 39.5 percent of all deaths in children under age 5. The diseases mentioned are of greater importance in the neonatal period, even though they originate during the period of fetal life. This pattern holds particularly true in prematurity, which was the basic or associated cause of 22.6 percent of deaths (Puffer and Serrano 1973). These statistics do not bring out the role of nutritional deficiencies as the underlying cause of death in infants, although the Inter-American Investigation of Mortality in Childhood (Puffer and Serrano 1975, pp. 53-63) showed that it was a basic or associated cause of around 34.1 percent of deaths in children under age 5. Of course, the health risks of infants and children are significantly influenced by the health status of their mothers, especially during fetal development and the neonatal period. During the reproductive cycle, there are still serious risks (most of which are preventable) of dying. At the beginning of the decade they were reflected in maternal Table 1. Selected Demographic and Economic Data in the Americas 96 Per Capita Population Rate of Gross Estimate Population Infant Population Urban National Mid-1976 Birth Rate Growth Mortality Rate under 15 years Population Product Region or Country (millions) (per thousand) (%) (per thousand) (%) (%) (US$) North America 239.0 15 0.8 16 27 74 6.580 Canada 23.1 15 1.3 16 29 76 6.080 United States 215.3 15 0.8 17 27 74 6.640 Latin America 326.0 37 2.8 75 42 59 940 Middle America 81.0 45 3.4 65 46 56 900 Costa Rica 2.0 28 2.3 45 42 41 790 El Salvador 4.2 40 3.2 54 46 39 390 Guatemala 5.7 43 2.8 79 44 34 570 Honduras 2.8 49 3.5 117 47 28 340 Mexico 62.3 46 3.5 61 46 61 1.000 Nicaragua 22 48 3.3 123 48 49 650 Panama 1.7 31 2.6 44 43 49 1.010 Caribbean 27.0 31 2.1 71 41 43 820 Bahamas 0.2 22 4.2 32 44 58 2.460 Barbados 0.2 21 0.8 38 34 4 1.110 Cuba 9.4 25 1.8 29 37 60 640 Dominican Republic 4.8 46 3.0 98 48 40 590 Grenada 0.1 26 0.4 32 47 8 300 Guadeloupe 0.4 28 1.5 44 40 9 1.050 Haiti 4.6 36 1.6 150 41 20 140 Jamaica 2.1 31 1.9 26 46 37 1.140 Martinique 0.3 22 0.5 32 41 33 1.330 Netherlands Antilles 0.2 25 1.8 28 38 32 1.530 Puerto Rico 3.2 23 2.4 23 37 58 2.400 Trinidad and Tobago 1.1 26 1.5 26 40 12 1.490 SADIAYAS HLTVHIH ATIHO 40 NOILLVNTVAA Table 1. Selected Demographic and Economic Data in the Americas (Cont.) Per Capita Population Rate of Gross Estimate Population Infant Population Urban National Mid-1976 Birth Rate Growth Mortality Rate under 15 years Population Product Region or Country (millions) (per thousand) (%) (per thousand) (%) (%) (US$) Tropical South America 178.0 38 29 82 43 58 840 Bolivia 5.8 44 2.6 108 43 35 250 Brazil 110.2 37 2.8 82 42 58 900 Colombia 23.0 41 8.2 76 46 64 510 Ecuador 6.9 42 3.2 78 47 39 460 Guyana 0.8 36 22 40 44 40 470 Paraguay 2.6 40 27 65 45 38 480 Peru 16.0 41 29 110 44 60 710 Surinam 0.4 41 52 30 50 49 870 Venezuela 12.3 36 29 54 44 75 1.710 Temperate South America 39.0 24 1.5 67 32 80 1.540 Argentina 25.7 22 1.4 64 29 81 1.900 Chile 10.8 28 1.7 78 39 76 820 Uruguay 2.8 21 1.1 45 28 80 1.060 Source: World Population Data Sheet, Population Reference Bureau Inc., Washington, D.C. 1976 BOLIDWY UNE ul YI[edy pryn LG 28 EVALUATION OF CHILD HEALTH SERVICES Table 2. Death Rates Per 1,000 Population Under Age 19, by 5-Year Age Groups in 10 Latin American Countries Age, in years Country Year Total 0-1* 1-4 5-9 10-15 15-19 Argentina 1969 9.8 61.1 2.8 0.7 0.7 1.2 Colombia 1969 %:5 62.7 8.4 1.7 1.0 1.5 Costa Rica 1972 59 59.0 3.0 0.9 0.5 1.1 Cuba 1971 6.1 $5.9 0.9 0.5 0.5 0.9 Chile 1972 8.8 71.1 2.6 0.8 0.8 14 United States 1969 9.5 20.9 0.9 0.4 0.4 1.1 Guatemala 1971 14.1 84.1 25.9 6.5 3.0 5.1 Jamaica 1971 7.6 26.4 4.6 0.8 0.6 0.8 Mexico 1971 9.0 63.3 7.9 1.8 | 1.7 Venezuela 1971 6.6 49.8 5.3 1.0 0.7 1.1 * Per 1000 live births. Source: Pan American Health Organization 1974, Department of Health Statistics. death rates of 13.5 per thousand and 18.80 per thousand in Middle and South America respectively and of only 2.5 per thousand in North America. Furthermore, low birth weight, one of the most accurate predictive indicators of child health, is notoriously prevalent in most of the countries of the region (Puffer and Serrano 1975a, b). The risks of mortality after childhood and during adolescence are considerably less; during this period, the leading causes of death are accidents, suicide, homicide, certain types of infectious diseases, metabolic disorders, endocrinal diseases, and tumors. During this period, psychosocial disorders play a preponderant role and contrib- ute to and/or are the consequence of, family disruption. They take the form of drug addiction, smoking, drinking, sensory disorders, learning difficulties, and behavioral disorders that may lead to prostitution and juvenile delinquency. Especially worthy of mention during the period of adolescence is the importance of diseases connected with the onset of the reproductive cycle. These diseases occur as a consequence of the gradual change in roles and values and the lack of proper sex and family life education. This change is conducive to sexually transmitted diseases, undesired pregnancies, and illicit abortions (WHO 1976a, b, PAHO 1975, Rosselot 1977). The health problems of children and adolescents are closely related to changes of various kinds in the process of overall development. These vary from country to country and eventually produce impor- tant sociocultural changes in the community, especially in families. This pattern is particularly true of families exposed to major biological and/or socioeconomic risks, especially those affected by Child Health in Latin America 29 temporary or permanent migration from the rural areas to the cities. In these cases, the protective role of the traditional extended family is minimized, especially during the traditional phase of adaptation—a phase which usually takes place in the adverse environmental and acculturation conditions characteristic of shantytowns (Wray 1975). Ongoing Child Health Programs The health protection of children and adolescents must be part of general economic and social development plans and family health and welfare programs, which must be designed to enlist their conscious cooperation and thereby provide an impetus to the health services. This was precisely the purpose of the health goals embodied in the Charter of Punta del Este and approved by the First Special Meeting of Ministers of Health of the Americas (Washington, D.C,, 1963). One of those goals was a 50 percent reduction in mortality in children under age 5 in the 10-year period 1961-71. At the same time, it was also postulated that, to achieve this goal, the Ten-Year Health Plan should become part of the general development plans of the countries involved (PAHO 1964). In the past decade, because of the political and socioeconomic situation in Latin America and the Caribbean, limited progress has been made in implementing these plans and programs, bringing about structural changes, accelerating economic growth, and expand- ing social services, including health services. This limited progress explains why the goals agreed upon have not been fully achieved. An analysis of the health sector, and in particular of child protection, shows that, although the risk of dying at an early age has been reduced, it still does not meet the target set (table 3) (PAHO 1971). At the beginning of the present decade, the coverage of child health programs in most Latin American and Caribbean countries was still limited. Health activities were carried out without the necessary degree of coordination with the activities of related pro- grams—such as medical and dental care, epidemiology, nutrition, mental health, health education, and community development—for which the Ministries of Health and other institutions of the health sector and other sectors were responsible. The scope of maternal health programs, so closely connected with child health, has been limited. Thus in Latin America and the Caribbean the coverage of prenatal care, of care during labor and delivery, and of postpartum care by qualified personnel has averaged 30, 50, and 5 percent respectively. These types of care influence the biological quality of the fetus and the newborn child (high incidence of congenital anomalies and low birth weight) and has an important 30 EVALUATION OF CHILD HEALTH SERVICES Table 3. Reduction in the Death Rates of Children under Age 5, in Selected Countries, Compared with the Goals of the Charter of Punta del Este, 1969 0-1 1-4 Deaths per 1,000 live births Deaths per 1,000 population Average Goal Average Goal Country 1960-62 1969 1969 1960-62 1969 1969 Costa Rica 66.1 67.1 39.7 7.5 5.0 4.5 El Salvador 72.5 63.4 43.5 17.1 13.2 10.3 Guatemala 89.3 93.8 58.0 32.4 24.7 22.7 Honduras 48.4 34.0 31.5 14.1 12.0 9.2 Jamaica 49.1 39.0 29.5 6.8 5.4 4.4 Mexico 71.4 66.7 42.8 13.8 8.8 8.3 Nicaragua 63.1 52.6 379 8.6 8.7 52 Panama 51.1 39.2 30.7 7.9 8.1 4.7 Dominican Republic 94.1 63.9 56.5 10.4 %.5 6.2 Trinidad and Tobago 42.9 36.6 27.9 2.5 1.8 1.6 Argentina 61.0 54.9 42.7 4.3 2.6 3.0 Bolivia 103.0 101.6 67.0 16.8 — — Colombia 92.8 78.3 65.0 15.4 11.7 10.8 Chile 117.8 81.7 70.7 8.2 3.3 4.9 Ecuador 99.4 91.0 59.6 22.2 16.1 13.3 Paraguay 89.7 91.1 53.8 9.4 11.3 6.1 Peru 92.9 75.3 65.0 15.7 8.7 11.0 Uruguay 44.6 48.7 26.8 1.3 1.6 0.8 Venezuela 52.1 46.8 313 5.7 5.3 34 Source: Adapted from PAHO 1971. bearing on health in the later stages of childhood. In addition, the shortcomings in the above-mentioned programs reduce opportunities for providing proper training in basic mothering and counseling for family life, including birth control (PAHO 1973, WHO 1976). The care of children under age 5 is belated and limited, and sometimes the preventive aspects are neglected. An even worse situation is to be noted in adolescence, when health care is precarious (Rosselot 1972, PAHO 1973, Rosselot 1974). Furthermore, specific family welfare programs that have been formulated and are conducted in close coordination with other institutions and other sectors are rare, even though children can obtain undeniable health benefits from them. All this applies to the shantytowns and especially to the rural areas, where health problems, particularly in children, are very serious. At their meeting in Punta del Este in 1967, the presidents of the Americas recommended that maternal and child protection and family education programs be increased. In addition, participants at the Second Special Meeting of Ministers of Health of the Americas (PAHO 1968) (Buenos Aires 1968), when reviewing the results obtained in implementing the Ten- Year Health Plan mentioned above, made specific recommendations Child Health in Latin America 3] Table 4. Evaluation of the 10-Year Health Plan, 1971-80, Child Health and Family Welfare Mortality in- Children Ages Infant Mortality 1-4 Deaths in infants under age 1 Deaths in children ages 1-4 per 1,000 live births per 1,000 population Country 1971 1980 1971 1980 Brazil 105.0 70.0 60.0 30.0 Chile 70.5 40.0 2.9 1.8 Ecuador 78.5 47.1 15.7 6.3 Guatemala 89.0 71.2 24.0 12.0 Honduras 117.6 85.0 20.7 10.4 Mexico 66.0 44.4 10.1 5.6 Panama 37.6 18.8 7.4 4.4 Venezuela 48.7 42.4 5.0 4.2 Source: Adapted from PAHO 1976 in the area of child health that were aimed at expanding and integrating the programs, particularly in the underserved areas, and at strengthening the education and training of necessary personnel. More recently, the Third Special Meeting of Ministers of Health of the Americas (Santiago de Chile, 1972), which approved the Ten- Year Health Plan for the decade 1971-80, subsequently ratified by the governing bodies of the Pan American Health Organization, reviewed the most serious child health problems, set goals in reducing the risks and in expanding coverage, and outlined a strategy for achieving the above-mentioned purposes and overcoming the techni- cal and administrative problems impeding the smooth implementa- tion of the programs. Specifically, the document recommended a reduction of 40 percent on the average in maternal and infant mortality rates and of 60 percent in the mortality of children in the 1-4 age group. In the area of extension of program coverage, it recommended 90 percent coverage for health activities dealing with mothers and children under age 1 and 50 percent coverage for those in the 1-4 age group by the end of the decade. These goals are of course regional estimates, and the countries must adjust them to local conditions. The goals established for selected countries in the region are shown in table 4 (PAHO 1968). The goals of the suggested strategy and that mentioned earlier (PAHO 1973) are basically the same, and the degree of achievement has varied from country to country. Among these activities, the following warrant mention: 32 EVALUATION OF CHILD HEALTH SERVICES —The formulation of an intersectoral policy for the protection of families, mothers, and children that guarantees their civic, legal, and employment protection rights and promotes recreational, educational, and vocational guidance activities for children and adolescents. —The preparation of a program for the integration of child health activities into the general health services. This program will cover all children, will be accessible to all, and will be efficient. Its operations will encompass the various stages of children’s lives and include coordination with health activities to be carried out during the whole life cycle. In this regard, the processes of human reproduction will receive special attention, since they influence growth and development and, consequently, the health of children. Where appropriate, criteria will be established for the preferential and early treatment of cases that merit such treatment because of their biomedical-social risk. To accomplish this task, a system of regionalized health services which will provide progressively more complex care at each level must be established. In addition, activities must be carried out by multidisciplinary teams whose efforts are directed at different sectors of the population and whose members include (especially in rural areas) health assistants, lay midwives, and community leaders. —The establishment of a system for the education and training of all personnel involved in the program. The system will be based on methods that ensure continuing education and/or training geared to the problems and needs of each community. This system in turn will foster research for improving the efficiency of the service (table 5). —The involvement of the community and of the families themselves in improving health conditions, bearing in mind the political, administrative, social, cultural, and anthropological characteristics of each community and the coordination of that process with intersectoral programs that have a significant health impact. —The establishment of a system for the continuous and ongoing evaluation of the extent to which technical cooperation objectives are achieved. The system will include the study of indicators for evaluating the effectiveness and the cost-benefit ratio of the activities concerned and for making the necessary adjustments in the process where appropriate. In the last 5 years, some Latin American and Caribbean countries that have reasonably satisfactory health structures have made prog- ress in reducing the risks of mortality in infancy and childhood. There are many reasons for this progress, but it is primarily the Child Health in Latin America 33 Table 5. Priorities of Maternal and Child Protection Programs According to the Level of Socioeconomic Development* Health Determinants Socioeconomic Level Medium Low Prevalent Risks Maternal health Infant health Priority Groups Maternal programs Infant programs Priority Activities Family life education Primary prevention Medical care Mental health Social welfare Professional Personnel Available Specialized Basic Nonprofessional Personnel Available High Low maternal mortality Perinatal mortal- ity declining Primiparas, ado- lescents, unmar- ried women Newborn dren with birth weight chil- low Moderate Moderate Moderate High Limited Obstetrician-pe- diatrician Nurse and/or midwife (***) General practi- tioner and nurse (H**) Auxiliary (*) Volunteers (*) High maternal and neo- and postnatal mortality Primiparas and abortion repeaters Newborn, weaned infants un- High High High Moderate Moderate Obstetrician-pedia- trician Nurse and/or mid- wife (**) General practi- tioner and nurse (**) Auxiliary (**) Health promoters and untrained community workers Si Very high maternal mortality and child mortality All pregnant women All children High High High Limited High Obstetrician-pedia- trician Nurse and/or mid- wife (*) General practi- tioner and nurse *) Auxiliary (***) Health promoters and untrained community workers (HH%) *, *% *x** Increase in numbers SOURCE: Rosselot 1972. Reproduced with permission of Pan American Health Organization. 34 EVALUATION OF CHILD HEALTH SERVICES result of an expansion in medical care coverage and in community participation, especially in the rural areas. There is a good possibility that improvements in the statistical systems have obscured some of the possibly favorable changes that might have occurred in certain countries as regards basic indicators of child health, especially of infants. However, there is no doubt that the international financial situation and its effects on national eco- nomic and social development have made it difficult for most countries to obtain sufficient resources for the health sector in general and for child health programs in particular. Since future financial developments are difficult to predict and are not to be influenced greatly by the health sector, a more equitable sectoral distribution of the national income should be encouraged whenever feasible. The health sector must promote policy decisions designed to improve the mechanisms for inter- and intrainstitutional coordination and to find rational solutions to the problems of health service delivery. Solutions are needed that will permit an increase in the operational efficiency of the programs, while keeping the cost-benefit ratio of the tasks to be undertaken (PAHO 1968) at an.appropriate level. There is no doubt that at present much is to be expected from the innovative medical care approach in which child health problems are first analyzed and their control is then programed in a broader context that extends beyond traditional maternal and child health care to the promotion of the health of the entire family including the biologically most vulnerable family members (Rosselot 1974). It is promising that the concept of family health is not only being progressively accepted in Latin America and the Caribbean, but is also being gradually incorporated into the operations of the general health services. It appears to be increasingly better understood in the region that, if families are to perform their full biological, psycholog- ical, social, cultural, and economic functions, more satisfactory condi- tions for the promotion of child health must be established. This calls for the institutionalization of the above-mentioned process, based primarily on the motivation of the community (Newell 1975) and on the proper training and supervision of the health team responsible for the activities concerned (Flauhault 1970). In the rural areas, the promotion of family health must be the focal point of the primary health care system. It has been taken into account in the Sixth Program of Work of the World Health Organization for the period 1978-83 (WHO 1976a). The new strategy, which is strongly supported by the United Nations Chil- dren’s Fund, calls for the “expansion of the basic services for children Child Health in Latin America 35 in developing countries” (United Nations 1976). It is a formula which, given resolute community participation, will gradually put an end to the present underdevelopment in the social and health fields. When this task is accomplished, children will undoubtedly be the principal beneficiaries. With the technical cooperation of the Pan American Health Organization (1976), the Latin American and Caribbean countries are making headway in implementing that policy. It is the only feasible alternative for securing, within a reasonable period of time, a more adequate level of health and well-being for the underprivileged and needy communities and for their most valuable and potential human resource, the children of the Americas. REFERENCES Flahault, D. (1970) An Integrated and Functional Team for Primary Health Care. WHO Chronicle, Vol. 30. Geneva: World Health Organization. Instituto Interamericano del Nino (1961) Los Derechos del Nino. Asemblia General de Naciones Unidas, 20 noviembre 1959, Nueva York. Montevideo, Uruguay. Montoya, C. (1974) Levels and Trends of Infant Mortality in the Americas, 1950- 1971. World Health Statistics Report. Vol. 27. Geneva: World Health Organization. Newell, K.D., ed. (1975) Health by the People. Geneva: World Health Organization. Pan American Health Organization (PAHO) (1977). Supporting Services Division. Personal information. PAHO (1977) XXIV Meeting of the Directing Council. Final Report. Official Document No. 146, Mexico, D.F. PAHO (1976) Evaluation of the Ten-Year Health Plan of the Americas. 1971-1980. Doc. CD24/18. Washington, D.C. PAHO (1975) The Health Needs of Youth in Latin America and the Caribbean. Report of a Working Group of the Pan American Health Organization. Washington, D.C. PAHO (1974) Health Statistics Department. Personal information. PAHO (1973) Ten-Year Health Plan for the Americas. Final Report of the III Special Meeting of Ministers of Health of the Americas, Santiago, Chile, October 1976. Official Document 188. Washington, D.C. PAHO (1971) Facts on Health Progress. Scientific Publication No. 227. Washington, D.C. PAHO (1968) Special Meeting of Ministers of Health of the Americas, Buenos Aires, October 1968. Final Report. Official Document PAHO 89. Washington, D.C. PAHO (1964) Task Force on Health at the Ministerial Level. Final Report. Official Document 51. Washington, D.C. Pan American Union (1967) Declaration of the Presidents of America. In Meeting of American Chiefs of State (Punta del Este, Uruguay, April 1967). Publication 34.1E7480. Washington, D.C. Population Reference Bureau (1976) World Population Data Sheet. Washington, D.C. Puffer, R.R., and Serrano, C.V. (1975a) Results of the Inter-American Investigations of Mortality Relating to Reproduction. Scientific Publication No. 306. Washington, D.C.: PAHO Puffer, R.R., and Serrano, C.V. (1975b) Birthweight, Maternal Age, and Birth Order. Three Important Determinants in Infant Mortality. Scientific Publication No. 294. Washington, D.C.: PAHO Puffer, R.R., and Serrano, C.V. (1973) Patterns of Mortality in Childhood. Scientific Publication No. 262. Washington, D.C.: PAHO 36 EVALUATION OF CHILD HEALTH SERVICES Rosselot, J. (in press) Problematica de la Salud del Adolescente y del Joven en América Latina y el Caribe. Pediatria, Santiago, Chile. Rosselot, J. (1974) Formulacién e implementacion de programas de salud maternoin- fantil y bienestar familiar en Latinoamérica y el Caribe. (Publicacién Mimiogratiads) Organizacion Panamericana de la Salud. Washington, D.C. Rosselot, J. (1972) Maternal and child health in Latin America. Bulletin of the Pan American Health Organization, Vol. 6, No. 1. United Nations (1976) Economic and Social Council. Operational Activities for Development. The Expansion of Basic Services for Children in Developing Coun- tries. New York. World Health Organization (WHO) (To be published) Health Needs of Adolescents. Report of a WHO Expert Committee. Geneva. WHO (1967) Manual of the International Statistical Classification of Diseases, Injuries, and Causes of Death, Vol. 1, Geneva. WHO (1976a) Resolutions and Decisions. Annexes. Part I. Twenty-Ninth World Health Assembly. Geneva. WHO (1976b) New Trends and Approaches in the Delivery of Maternal and Child Care in Health Services. Technical Report Series No. 600. Geneva. WHO (1974) Human Development and Public Health. Technical Report Series No. 485. Geneva. Wray, J.D. (1975) Health Maintaining Behavior of Mothers in Traditional, Transitional and Modern Societies. (Mimeographed publication) AAAS Meeting. New York. 3 QUESTIONS BEHIND THE ANSWERS: WHY WE DO NOT EVALUATE AND WHY WE SHOULD Abraham Horwitz The fact that mortality in children under 5 years of age, along with the constellation of causes that bear on it, is the most significant health problem in the developing Americas has been brought out by the presentations of Dr. Wray and Dr. Rosselot. These children are historically poor and they live in physiological misery—which means that they are continually at risk of disease and interference in their physical and intellectual growth. They are the victims of underdevel- opment. Medical technology, essential as it is, is only one factor in the organized efforts to combat the deleterious effects of history, the environment, lifestyle, a cumulative genetic heritage, prevailing pol- icy, and the economy. The progress in this undertaking over the last 30 years is evident, but we have not evaluated it systematically, with an overall view of the problem. The volume of the investment has not been small, but it has always fallen short of the tangible need. We would have preferred, in the same period, fewer arms and more food, less costly health equipment and more proteins and vaccines. Still, we have failed to measure maternal and child health programs rationally in terms of their structure, functions, and effects. At best, we have done no more than register specific morbidity and mortality rates, and even this effort has been incomplete, carried out on the basis of data whose precision left something to be desired, as did the timeliness with which they were examined and disseminated. In general we have done it after it is too late to modify our course of action. We know perfectly well that statistics has an intrinsic value as an analytical language which describes phenomena or situations and, on the basis of experience, helps us to predict the way they are likely to evolve. It gives a picture of things as they are and helps to focus them more closely. It is essential for shedding light on the past and forecasting the future. Furthermore, as events take shape, statistical data are irreplaceable as a means of adjusting the decisionmaking process. In short, they are the basis for modern systems of evaluation 37 38 EVALUATION OF CHILD HEALTH SERVICES and information, which are part of the process of economic and social development planning. There can be no viable programming without reliable statistics. Some 25 years ago, Frost (1951) wrote it this way: ... for while the facts expressed in a statistical record constitute only a part of the evidence required, they constitute the part which can be most conveniently and forcibly presented and are essential to any quantitative statement of results achieved. If we really desire to submit our judgment of what is being accomplished to this final test of statistical evaluation we must set about collecting statistics which will serve the purpose. The fact is that we do not evaluate on a regular basis; if we do, it is exceptional—as in connection with research that has clearly circum- scribed objectives or with categorical problems, such as the control of a communicable disease. For the most part we fail to evaluate health programs and general services—and, by extension, complex social undertakings—as an overall system. And when we do, it is less often than we would like that our recommendations lead to their desired specific effects: namely, the creation of laws, standards, or proce- dures; the changing of a policy; improvement in the design of a program, the definition of objectives, or the planning of investments; or fulfillment, generally speaking, of the true purposes of evaluation in terms of giving priority to problems, reducing their magnitude, and increasing the efficiency and effectiveness of programs (Goldstein 1973, pp. 205-206). Flagle (1973) has said: “Information is potentially valuable only to the extent that it influences decisions and actions. It is actually valuable only to the extent that it influences outcomes favorably.” And he adds: “The value of information gained from the evaluation process is ultimately the value of change in programs in response to evaluation. If we are speaking of large-scale programs, we must add to the objective measures of improvement the subjective ones associated with basing policy on knowledge rather than on pressure, hope, or delusion. Also, we should weight the expected value of acting on the evaluation reports by the probability that such response will take place.” It is appropriate to ask why we do not practice evaluation in the developing countries. The answers to this question represent personal value judgments and generalizations in the complex world of health care that might be open to criticism or unfavorable comment. Nevertheless, the fact alone of enunciating them is reason enough for holding this conference, whose timeliness it is important to empha- size. Why We Should Evaluate 39 We do not evaluate because the basic information and the program- ing that results therefrom are deficient. The definition of targets is not always precise, so that the periodic measurement of their attainment tends to lead to rather subjective interpretations. Kane, Henson, and Deniston (1974, pp. 217, 230) maintain that objectives are rarely explicitly stated and often lack conceptual clarity. Health planning as a differentiated discipline is relatively new in the Americas. It has come into its own in the last 15 years as an instrument for inducing social progress, not as the expression of a given political structure in which decisionmaking is totally centralized. Planning has been officially accepted by the governments. How- ever, it is not done regularly in actual practice. Not all the countries have formulated a national health policy or identified their priority problems, setting goals for each that are consistent with the available resources. Moreover, their decisions have lacked continuity. What we have said can be summed up in the statement that evaluation is not possible without viable programing, and viable programing, in turn, depends on having measurable targets. This explains—although it does not justify—the fact that systematic evalu- ation to ensure the quality of health care is practiced only as an exception. Pamela Horst and coworkers (1974) have pointed out: Many federal programs are not evaluable: no measurable objectives are officially established; intended uses of evaluation studies have not been clarified by program managers; agencies are under pressure to spend substantial funds for evaluation yet lack the leverage for applying meaningful criteria to them; and the means of controlling evaluation funding are both ineffective and inefficient. Schmidt et al. (1975) say, for their part, that “while evaluation offices define criteria they fail to specify adequate measures of success.” We fail to evaluate systematically, it is maintained, because the “state of the art” is rudimentary (National Academy of Sciences 1974, p. 11)—which is the same as saying that there is no consistency yet in the principles and methods that go to make up the process. No consensus has been reached regarding the type of data that should be obtained and analyzed in order to evaluate health care. Lewis (1974) reminds us that in the United States the assessment of the quality of care has come full circle. Some 50 years ago it was the final results that were the principal subject of evaluation; subsequently, their place was taken by the resources and their characteristics, and by the processes. Today, once again, the results are evaluated, but bearing in mind that they depend on the methods used. 40 EVALUATION OF CHILD HEALTH SERVICES Some experts believe that information, which reflects structure (organization and resources) and process (health actions), is more important than the outcome (Donabedian 1966, Morehead 1974, pp. 53, 54). Others maintain that the quality of services should be measured in terms of the final results—that is to say, the recovery of health by the sick and their active reintegration into society (Roemer 1971, Williamson 1971, National Academy of Sciences 1974). Both approaches have, as it is to be expected, serious limitations, which have been analyzed by Brook (1974, pp. 183-210). From this difference of opinion there arises the urgent need—which can be expressed in another series of questions—to investigate the real correlations between data on structure and process, as well as between these two and outcome. It is recognized that what we know about structures and processes should be interrelated, but the precise nature of the relationship eludes us. This problem has led Barbara Starfield (1973) to propose an integrated research model and to point out that: Public pressure, exerted through legislation and administrative agencies, will increasingly require that the results of medical care be demonstrated ... If structural and process criteria are indeed to continue to be useful indicators of the “quality” of medical care, systematic efforts to relate them to outcome should be initiated without further delay. This, in my view, is the most important task if we are to move toward a rational approach for the evaluation of health systems. It lies at the very center of the interface between research and medical practice. In the meantime, I think that for the developing world it is useful to measure, both in institutions and in communities, the resources invested and the activities performed toward achieving given objec- tives for prevention and cure. As we record our accomplishments, we should point out that they are not the exclusive consequence of health technology; without it, however, results would probably be less tangible in many cases. We would like to cite, recognizing their intrinsic weaknesses, the three evaluations of the Ten-Year Health Plan contained in the Charter of Punta del Este (OAS 1961; PAHO 1966, 1968, 1971). They include measurements, for one region of the world, of steps taken and results achieved in regard to the goals for the respective priority problems agreed upon by all the governments. The fact alone that they served as an incentive and as a model for the formulation of the health plan for the current decade is worth pointing out. We are confident that this exercise will continue—each Why We Should Evaluate 41 time on an increasingly solid basis—with objectives set in terms of resources, activities, and results, their regular evaluation being an integral part of the health system in each country. We do not evaluate, it is claimed, because we lack definitions, indicators, and standard classification schemes at the national and international levels (WHO 1971). The methods that we now use are not always appropriate for measuring action and outcome. This, then, is another area of investigation in which a series of questions can be posed. It might well be asked why, despite the logic of the concept, evaluation is practiced in the health field more as an exception to the rule and on an experimental basis than as a regular activity. In my view, those who decide on and carry out programs react in a singular way to assessment of the expected results. They seem to have a psychological inclination to content themselves with whatever effort is made, rather than to be concerned with its immediate and long-range effects—that is, the degree and characteristics of the changes induced. It is a subconscious evasion of the responsibilities inherent in their work. If the real accomplishments do not coincide with what has been projected, the magnitude of the failure is commensurate with the rank of the official concerned, the reaction on the part of society and consumers, and the size of the investment. There are those who will say that this interpretation is baseless—and they may be correct. The fact remains, however, that the best planning models habitually fail to be explicit in regard to evaluation. At the same time, it is not hard to show that there are programs of questionable value—be it because they are unproductive, untimely, or unnecessary—which tend more to serve political, cultural, social, or administrative interests than the human beings they are designed in principle to benefit. The absence within the program design of clear-cut objectives and guidelines for measuring their attainment obviates the need to qualify their useful- ness and thus allows them to continue, despite their low apparent yield. Patton and coworkers (1976) have conducted an in-depth analysis of 20 evaluation studies in the health field, taken from an overall sample of 170 studies. They conclude that the recommendations are followed in direct proportion to what they call the “personal factor”: in other words, the manifest interest of those who are responsible for health decisions. Evaluation serves to reduce their uncertainty in the face of the complexities of social reality and to support their courses of action—not necessarily to exercise an impact on national policy or on the broad lines of programing. Thus, decisions are usually of a political nature and not always based exclusively on scientific tenets. We lack the necessary evaluators. Furthermore, we do not have a 42 EVALUATION OF CHILD HEALTH SERVICES clear understanding of whether evaluation is a separate discipline—a specialization within the health sciences—or only an attitude, a way of thinking and acting on the part of professionals, whose training, in any case, should be based on the principles of modern epidemiology (White and Henderson 1976). Developing countries also lack simple and reliable methods for organizing evaluation as a continuing, rather than an incidental, process. This problem can be traced to the demands on the part of certain sources of foreign capital for cost-benefit and cost-effect studies in order to set priorities in health—techniques whose practical applicability to the social sectors of the economy has yet to be demonstrated. Despite our insufficient experience and the series of questions that are still awaiting answers from research, we must evaluate. It is in our interest and it is our moral obligation to do so. The State, in order to protect society and guard its investments, is beginning to insist on concrete results. An experiment, unique in the world, to ensure the quality of medical care has already received legislative backing in the United States. Many governments are concerned about providing access to services for all their people, but none of them as yet has established a national system that guarantees the quality of such benefits. We recognize, of course, that it is a long-term process; its immediate goals should be limited at first, and its guiding procedures should be carefully established (National Academy of Sciences 1974, Haggerty 1974, Cambridge Research Institute 1976, pp. 131-136). Surely other countries will adopt legislation of this kind in the future. Evaluation is imperative, however. The growing competition within each government for the distribution of national income in relation to priority problems makes it necessary. Underdevelopment is char- acterized by a marked imbalance between vital needs and the resources that are available. At the same time, the social sectors are at a great disadvantage vis-a-vis those that are considered “directly productive.” We have not even learned, let alone used, the language of the economists to defend our proposals. We do not want to believe that our moral arguments in terms of the meaning of life—not necessarily its quality—and of disease and death fail to impress those who have the power to decide what order of investments in health and in food are compatible with a civilized society. If we do not evaluate, we will not be able to explain to the State and to the communities what the social effect of our efforts, and their money, has been. Nor will we be able to ask for increases, because we will be unable to justify the results of the preceding investment. If we do not evaluate, we will not be able to identify the defects in Why We Should Evaluate 43 the organization of preventive and curative services—nor the waste, the duplication of effort, the unnecessary examinations and interven- tions, the iatrogenic diseases, or, in sum, the ineffective administra- tion of institutional, ambulatory, and community care. It is claimed that in the United States the overall waste in the annual health budget of more than $130 billion is no less than 25 per cent (White 1976, p. 71). A similar proportion is estimated for the developing countries in the Americas, but there are many added problems as well: the inadequate utilization of scarce human resources—the example of nurses being a conspicuous case in point (Secretaria de Salud Publica, Asistencia Social, Republica Dominicana, 1974, Grupo de Trabajo de Enfermeria, 1973); an enormous expenditure on the construction of hospitals in which the quality of care is sacrificed for sumptuous, and not necessarily functional, design; an investment in complex equipment which we tend to purchase in excess of our real needs. If these funds were spent on maternal and child health, nutrition, and systematic immunization, they would generate greater health and higher levels of economic production. In any case, if the cost of medical care were reduced without a decline in quality, many more people could be treated. If we do not evaluate, we will not be able to improve the planning process and periodically readjust our objectives in terms of the human, material, and financial resources available. Nor will we be able to guide the decisionmaking process in line with the purposes of the health function. If we do not evaluate, we will not be able to argue intelligently for reforms in health sciences education. For we want the teaching process to give greater priority to the prevention and not just the cure of disease. To this end, we want it to become more community oriented and not to remain institutionally based. At the same time, we want the learning process to result in an understanding of the major problems and their feasible solutions, and not merely in an analysis of disciplines that reflect the program structure in the schools of health sciences. We want the physician who emerges from this educational experience to have a view of human beings as biological and social entities, and to have a genuine interest in the society that he is going to serve. If we do not evaluate, we will not find out what still needs to be investigated in order to more appropriately measure resources, functions, and effects in relation to the objectives. It will be more difficult for us to adapt methods that are applied in other societies, as it will be to identify those professionals whose effectiveness is less than what is expected in order to motivate and educate them. If we do not evaluate, we will not be able to adequately meet our 44 EVALUATION OF CHILD HEALTH SERVICES moral responsibility to contributé to social well-being and to the common good. In summary, the advantages of evaluation are evident, although the difficulties are still considerable. Our questions “Why don’t we evaluate?” and “Why should we evaluate?” suggest a series of other questions, in turn, in regard to this poorly defined area which reflects the sequence between science and practice. We do not yet have universal indicators and methods for evaluating general and specific health programs—a responsibility of research. But we do not want procedures that are excessively complicated, for they will inhibit the administrator who is responsible to the political powers and cause him to lose interest. In this sense, I hope that this conference will make some useful contributions. REFERENCES Brook, R.H. (1974) Critical issues in the assessment of quality of care. In Kane, R.L., The Challenges of Community Medicine. New York: Springer. Cambridge Research Institute (1976) Regulation of health service costs and utilization. In Trends Affecting the U.S. Health Care System. Prepared for U.S. Department of Health, Education, and Welfare (HRA) 76-14503. Donabedian, A. (1966) Evaluating the quality of medical care. Milbank Memorial Quarterly 44: 166-206. Flagle, C. (1973) The value of information for decisionmaking. In Evaluation of Health Services Delivery. Proceedings of an Engineering Foundation Conference. Frost, W.H. (1951) Rendering account in public health. American Journal of Public Health. 41:8, pt. II. Goldstein, H. (1973) Principles of evaluation in maternal and child health. In Maternal and Child Health Practices. H.M. Wallace, ed. Springfield: C.C. Thomas. Grupo de Trabajo de Enfermeria. (1973) Informe. Estindares, para la atencién de enfermeria del paciente hospitalizado en Centro América y Panama. Con la colaboracion de la Organizacion Panamericana de la Salud. Haggerty, R.J. (1974) Quality assurance—The road to PSRO and beyond. Pediatrics 54: 90-97. Horst, P., et al. (1974) Evaluation planning at the National Institute of Mental Health: A case history. Washington, D.C.: Urban Institute 962. Kane, R.L., et al.(1974) Program evaluation: Is it worth it? In Kane, R.L., The Challenges of Community Medicine. New York: Springer. Lewis, C.E. (1974) The state of the art of quality assessment—1973. Medical Care 12:799-806. Morehead, M. 1974. Comment and dissent. In Advancing the Quality of Health Care: Key Issues and Fundamental Principles. A Policy Statement by a Committee of the Institute of Medicine. Washington, D.C.: National Academy of Sciences. National Academy of Sciences (1974) Advancing the Quality of Health Care: Key Issues and Fundamental Principles. A Policy Statement by a Committee of the Institute of Medicine. Washington, D.C.: National Academy of Sciences. Organization of American States (1961) Resolucion A2. Plan Decenal de Salud Publica de la Alianza para el Progreso. Documentos Oficiales de la OEA, Ser./H/XIIL.1. Rev. 2 (esp). Pan American Health Organization (PAHO) (1966) Facts on Progress. Health Goals in the Charter of Punta del Este. Miscellaneous Publication No. 81. Washington, D.C. PAHO (1968) Facts on Health Progress. Scientific Publication No. 166. Washington, D.C. Why We Should Evaluate ~~ 45 PAHO (1971) Facts on Health Progress 1971. Scientific Publication No. 227. Washing- ton, D.C. Patton, M.Q., et al. (1975) In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research. Minneapolis: University of Minnesota. Roemer, M.I. (1971) Evaluation of health service programs and levels of measure- ment. HSMHA Health Reports 86:9:839-848. Schmidt, R.E., et al. (1975) The Market for Evaluation Services. Washington, D.C.: Urban Institute 962. Secretaria de Salud Publica y Asistencia Social, Republica Dominicana (1974) Estudio en 10 Hospitales para Establecer Estandares de Atencion de Enfermeria del Paciente Hospitalizade—Republica Dominicana. Col la colaboracion de la Organizacién Panamericana de la Salud. Starfield, B. (1973) Health services research: A working model. The New England Journal of Medicine 289:134. White, K.L. (1976) Opportunities and needs for epidemiology and health statistics in the United States. In Epidemiology as a Fundamental Science. Its Uses in Health Services Planning, Administration, and Evaluation. K.L. White and M.M. Henderson, eds. New York: Oxford University Press. White, K. L., and Henderson, M.M., eds. (1976) Epidemiology as a Fundamental Science. Its Uses in Health Services Planning, Administration, and Evaluation. New York: Oxford University Press. Williamson, J.W. (1971) Evaluating quality of patient care. A strategy relating outcome and process assessment. Journal of the American Medical Association 218:564. World Health Organization (1971) Statistical Indicators for the Planning and Evalua- tion of Public Health Programmes. WHO Technical Report. Series No. 472. Geneva. A PRAGMATIC ASPECTS OF PROGRAM EVALUATION REGARDING CHILD CARE AND ITS RELATION TO DECISIONMAKING AT THE POLITICAL- ADMINISTRATIVE LEVEL Antonio Ordonez-Plaja The Need for Evaluation One third of my professional life has been devoted to administra- tion and two thirds to the pursuits of research, teaching, and professional practice. With this in mind, it is obvious that the biases introduced by these diverse activities will affect my conceptualization of an issue. Since I share Gunnar Myrdal’s viewpoint (1967) that no one can completely eliminate his biases, I have decided to follow his advice and frankly declare mine at this point. Undoubtedly, the establishment of priorities is the main focus of an administrative effort, especially in developing countries, where the gap between resources and problems is more evident and where the magnitude, quantity, and severity of those problems are equally apparent—especially as related to children, who constitute a high percentage of the population. In dealing with this problem, the first question that could be asked is the following: Can an entire program be evaluated? In general, the immediate response would be affirmative; nevertheless, an absolute answer would probably be erroneous from a practical standpoint. We could try to answer the question after analyzing a few concrete experiences. A pertinent example is the history of endemic goiter in Colombia. About 150 years ago, M. Boussingault (1849), who had come to Nueva Granada to study other problems, observed the correlation between the presence of goiter and the lack of iodine in food, especially in salt. He also noticed the absence of goiter in places where salt contained sufficient iodine. Boussingault never quantified this phenomenon, because it was so obvious that such action seemed unnecessary. The governments existing in those days ignored his 47 48 EVALUATION OF CHILD HEALTH SERVICES recommendations in spite of his clear and convincing argument that iodizing salt would solve the problem. According to the “petite histoire” of the time, one ruler who understood and believed Boussingault’s word refused even to consider the possibility of iodizing salt because “it was easier to rule those who had goiter.” This example shows that sometimes appropriate decisions are not made, either because of reasons that have no relation to the problem itself, or because an administrator has a different perception of the problem. An evaluation conducted some 30 years ago (Parra 1948) pointed out that more than half of the school population in Colombia suffered from endemic goiter. It is highly probable that in our countries, given the socioeconomic structure of the population, the nonschool population at the time suffered from endemic goiter at a similar, or even higher, rate. A comparison of the costs of iodizing salt and those of treating goiter and its consequences was a determin- ing factor in the positive action taken to solve this problem. In this case, the administrator used the results of the evaluation as a basis for making a political decision; information resulting from a scientific study was utilized. The two examples are not strictly comparable, because other factors pertaining to the time, values, and technological advances were not the same. Still later, researchers wanted to perform an evaluation of the results of salt iodization, but by that time, the results were evident: Endemic goiter as a public health problem had disappeared. It is worth mentioning that sometimes a researcher confuses the means and the ends, either because of an excess of academic zeal, or simply because funds are available for research. Then the researcher may commit himself to an investigation without considering its relevance and usefulness for the country as a whole. Fortunately, the abovementioned project did not take place, and resources were channeled into the study of other problems of higher priority, such as malnutrition and the causes of high infant mortality rates. As shown in the above examples, factors contributing to a decision on the practicality of evaluating a program are multiple and should be considered before any commitment is made. We should look at the availability of resources, the practical utility of the results for the decisionmaking process, the usefulness of the study, and the oppor- tunity for making it. In view of these considerations, it is often necessary to have the courage to sacrifice academic zeal when priorities have changed or objectives have already been reached. In such cases, research and/or evaluation are no longer necessary, since they may only show that the level of achievement was 93.2 percent Pragmatic Aspects 49 instead of 93 percent—in such instances, these projects are no longer relevant. In order to reinforce this statement, we may add that water fluoridation has not been initiated in many countries because of an alleged lack of resources for the evaluation of such a program. In reality, further evaluation is completely unnecessary, since the results are easily foreseen (a 60 to 65 percent decrease in caries); further- more, the cost of the evaluation might be more than that of the program itself. Relationship Between the Administrator and the Evaluator The relationship between the administrator and the evaluator implies a process of communication and feedback. It is essential that the administrator trust the work of the evaluators—how they do it (methodology) and why they do it. Frequently, researchers complain that the results obtained are not considered by decisionmakers. This is sometimes the case, but there are usually good reasons for such behavior, and they are not the sole responsibility of the administrator. First of all, some researchers seem to ignore the times we live in and work on problems that have very low priority when compared with all the other problems of the country or region. These workers must learn that even if their studies are very interesting and their methodology perfect, the results will be of no use to the country because they are not timely. The researcher must live in his own time, within his historical moment, to understand it, and to focus on the solution of existing problems. If he locks himself in an “ivory tower,” he will be unable to influence the destiny of his country. If his armbition is to participate in and influence the change process, he must abandon his seclusion and become aware of reality. In this way, he can plan his research study so that the results yield alternatives from which the administrator can choose the most feasible and suitable one, that with the greatest potential impact on the population to which he is ultimately responsible. In dealing with those projects, surveys, and evaluations whose objective is the study of a specific situation in a given group, one must remember that they should be promptly implemented. Other- wise, the results might not be applicable, even though they may seem very interesting. For example, in a constantly changing situation, a survey that takes 3 or more years (1 or 2 years for analysis and 1 year for publication) becomes obsolete by the time it reaches the person who could make the necessary decisions because, by then, the group under study has changed. Decisions made with a delay of 3 or more years—when conditions are totally different—could be unacceptable 50 EVALUATION OF CHILD HEALTH SERVICES to or even rejected by the community, whose attitude would be justified since the decisions would have no bearing on conditions at the time. Now let us look at the other side of the coin. When the well-known _ “Human Resources and Morbidity Survey” was made in Colombia in 1964 (Ministeria de Salud Publica 1967), there were many criticisms. One of them was excessive cost. I had the opportunity to contest this opinion in Maracay, Venezuela, when the preliminary results and analysis of the study’s methodology were presented for the first time to researchers and administrators in this hemisphere. Just one of the findings has halted the construction of a hospital whose cost would have been 15 to 20 times that of the study. With this finding, the study was more than paid for, and anything left over was considered a marginal benefit or an investment profit. On the other hand, the review and analysis of the life expectancy table in 1967 permitted the Government to show the Colombian labor sector that calculation of the retirement age had been based on inaccurate life-expectancy figures (Ministerio de Salud Publica 1968). It is very difficult to determine the benefit of this discovery to the country’s economy; between that year and the present, the figures might have added up to billions of pesos. How can the administrator or politician-administrator and the researcher reach agreement on the objectives and content of a research project? As one approach to this difficult theme, I suggest that we start with the assumption that the politician-administrator is not necessarily pursuing ignoble ends. He wishes to succeed in order to satisfy his personal vanity, or to gain votes in upcoming elections, or for some similar reason. In my opinion, all these reasons are acceptable if they coincide with an intention to improve the living conditions of the population. Furthermore, if we have a researcher who is “in tune with the times” and is aware of the problems of his society, then it is not difficult to reach an agreement on which studies should be carried out and on the amount of time and resources that should be allocated for such purposes. The studies should deal with research that has immediate application and with research that is “pure”—in other words, research that may not have an immediate application but will give research methodology a more sophisticated approach and will also raise issues on problems that will be of major concern in 10 to 20 years. A study on the change in environmental pollution in our cities could be an example. Some years ago, a proposal to research this problem might have been received with skepticism. Today we realize that pollution is becoming worse, necessitating corrective measures that could have been avoided by appropriate location of factories and alternative solutions to mass transport needs, for example. Pragmatic Aspects 51 The important issue is that the administrator realize the signifi- cance of a successful research or evaluation study. He often fails to understand this significance, and we hear complaints from the public of skepticism toward evaluations and irrelevant information. Ongoing communication between the administrator and the evaluators is also necessary so that interim results can be utilized as appropriate. In an effective interrelation between an administrator, who must face the problem to be solved, and a researcher, who often ignores the practical aspects, certain facts can be identified so that the evaluative process can be assessed during the initial stages. In this way, appropriate activities are supported and problem areas can be readjusted or eliminated. For example, during the initial activities of a study on malnutrition and mental development (Mora, Ortiz, Florez, et al. 1976), it became apparent that 65 percent of the mothers did not possess a clock. Since Bienestarina! must be cooked for 15 minutes before being given to the child, a timepiece was necessary for accuracy. Once the project's director was informed about the problem, appropriate modifications were introduced— evidence of the benefits of good communication between evaluators and administrators. The evaluation process has been considered as a procedure which utilizes scientific methodology, but we must be warned that on many occasions the opinions of the community, as well as the participant- observation process, sometimes do not receive due credit. In many instances, the usefulness and relevance of the community’s opinions have been proved in making decisions that deal with reality—such were the cases which arose when the Rural Promoters Program was established in Colombia (Ordonez Plaja 1967, p. 42). Community Participation in the Evaluation Process It is impossible to discuss evaluation without considering the importance of the community in relation to the process. Its signifi- cance goes beyond the fact that in most cases the community is the “target population.” For true participation, the community must be involved during the first stages of programing and design. Commu- nity participation will add validity to the process. Furthermore, the gradual learning process that will occur as a result of the interaction will eventually demonstrate the usefulness of evaluation. Realizing the importance of giving correct answers when the surveys are carried out, of keeping appointments, of participating, and so on, is !Bienestarina is a mixture of high-protein-content cereals plus vitamins, minerals, milk, and so on. 52 EVALUATION OF CHILD HEALTH SERVICES part of a process which sooner or later will have repercussions on the community members in an increasingly favorable way, inasmuch as their participation was requested in a serious, honest, and timely manner. Their participation has other marginal advantages, such as developing the researcher’s awareness that truth is not his sole property—thereby dispelling an image of “the almighty know-it-all” who can solve all kinds of problems from one point of view while ignoring or even rejecting the values of the participants. We often hear that a certain action has been carried out with the community’s participation when, in reality, the community has been used or manipulated as the decisions are being implemented. It is obvious that the community feels that its human dignity has been affronted when a decision that affects its members is taken without their participation. But it must be even more irritating for the community members to see publicity announcing their involvement in the program when they have not even been initially consulted on their perception of the matter. These perceptions can be right or wrong; in most cases they are right. Where they often fail is in determining what measures should be taken to solve the problem. The following situation illustrates the need for community partici- pation. In a small village, the community requested a children’s hospital to combat the high infant mortality. The diagnosis was right, but the proposed solution was neither appropriate nor viable. Discussion with the community members showed them that maybe the installation of facilities for potable water, adoption of a few sanitary control measures, inspection of food, and formal or informal education in child feeding could drastically decrease mortality. Through a participation process, the community accepted solutions that were technically more feasible, but this acceptance was achieved on the basis of solid arguments and with the participation of the community in the decisionmaking process. It has been said, “Nobody knows better where the shoe hurts than the owner of the foot.” This was the case in these communities: They knew very well what their problem was, even though they did not know the appropriate solution. Evaluation and its Financing The funding of research or any other method of evaluation where external financing is involved creates situations that are worth analyzing. Quite often, persons with wide experience in other socioeconomic and cultural environments arrive in a country with preconceived ideas as to what should be investigated and how to evaluate the investigation. This point is very important, since the Pragmatic Aspects 53 evaluation, even though it may seem necessary from a scientific point of view, can overlook the human resources, habits, and traditions of each country. Thus an evaluation which is almost perfect in method- ology may not meet the real needs of the country. Galbraith (1971) has stated: “We are inclined to believe that we are becoming much more scientific about the whole business although in an established tradition of the discipline there is some tendency to identify scientific precision with mechanical elegance rather than reality.” When we complain about research and methodological systems that are proposed or imposed on us by donor agencies that are aware of our inability to fully implement these systems, we are facing a problem which is basically the same as the one mentioned before, even if it is on a different level. Proportion of Resources Dedicated to Evaluation How much of a program’s budget should be allocated to evaluation purposes is frequently discussed. The answer is not a simple one; a definite amount cannot be arbitrarily assigned, because there is interplay between a large number of variables. For example, when we are dealing with a program about which there are serious doubts, we might very well assign a sum that might seem quite high for comprehensive evaluation of the program’s usefulness, convenience, and acceptance by the community. In such a case, the investment is worthwhile, even though it may seem excessive. However, as men- tioned in an earlier example, it would be absurd to invest any amount, even 1 percent of the total budget of a water fluoridation program, to determine the usefulness of fluoridation. The same could be said about studying the importance of breast feeding—its successful results need no further scientific confirmation. In programs benefiting children, where the results are obvious and where action is the required element, I do not believe that a definite percentage figure ought to be assigned to program evaluation. Each program should be dealt with according to its circumstances. In this case common sense is of more value than any scientific formula I could propose. In the words of Einstein, “Imagination is more important than knowledge.” In summary, what the administrator wants is relevant, reliable, and understandable information that will allow for sound decisionmaking. The scientist should consider this need, especially when presenting results that may be very clear to him, but not to the administrator. The researcher—evaluator must remember that his conclusions will only be given credit or taken into account if they are in accordance with the real needs of the people, and he must remember that the 54 EVALUATION OF CHILD HEALTH SERVICES decisionmaker is genuinely willing to solve the problem and usually has the necessary resources to do so. Final Comments To close the discussion on this subject which in its very essence needs a balance among science, humanity, politics, and facts and cases that improve the quality of life, I do not find any more appropriate words than those of Eduardo Cote Lamus, one of my country’s greatest poets: That world became extinct thus had to burn up that way because the things dear to man were destroyed: his will, his faith, his attempt to become one with his fantasy. And he was left with only the ability to reason. We cannot pretend that science is the only form of knowledge or that its conclusions are the sole truth; they are important, but only as a part of something bigger and more complex—the quality of life of individuals and communities, their expectations, their visions of the world, and their values. These factors are such that the administrator cannot solve a problem without the help of the scientist and vice versa; nor can both solve a problem without the participation of the beneficiary (or the legal guardian, when we refer to children). In today’s world, the politician—administrator must acknowledge that he needs the help of scientists and technical advisers if his goal is a better chance of success. But the scientist must also understand that the extent of his participation in decisionmaking (a new phenomenon in the world’s history) (Drucker 1968) implies a great responsibility, and under no circumstance should the scientist consider himself to be above the politician, since the latter is ultimately responsible to the people. This advice is valid in any political system, and although this responsibility can and should be shared, it ultimately rests with the politician. REFERENCES Boussingault, M. (1849) Scientific Trips to the Equatorial Andes, or a Collection of Memoirs on Physics, Chemistry, and Natural History of the Nueva Granada, Ecuador, and Venezuela. Paris: Editorial Lasserre. Drucker, P. (1968) The Age of Discontinuity. New York: Harper and Row. Galbraith, J.K. (1971) Does knowledge have a future? Chapter 17 In Economics, Peace, and Laughter, A.D. Williams, ed. Boston: Houghton Mifflin. Pragmatic Aspects 55 Ministerio de Salud Publica, Asociacion de Facultades de Medicina (1968) Estudio de Recursos Humanos para la Salud y Educacion Médica en Colombia. 1967. III Trabajos e Informes de la Conferencia de Maracay—Conferencia Internacional sobre Recursos Humanos para la Salud y Educacion Médica. Maracay, Venezuela, 19-23 Junio. Ministerio de Salud Publica, Asociacion de Faculta des de Medicina (1968) Estudio de Recursos Humanos para Salud y Educacion Médica en Colombia. Investigacion Nacional de Morbilidad—Hechos Demogrificos, 3a edicion. Bogota: Imprinta INPES. Mora, J.O., Ortiz, N., Flore, A, et al. (1976) El Retardo Mental Asociado con Privacion Psico-Nutricional en Ninos Marginados. Posibilidades de Prevencion. Trabajo presen- tado en las primeras jornadas internacionales a nivel latinoamericano sobre retardo mental y 3a. jornados de Avepane. Caracas, Nov. 8 a 13. Myrdal, G. (1967) Objectivity in Social Research. Winner Lecture given at St. Vincent College, Latrobe, Pennsylvania. Ordonez Plaja, A. (1967) Informe al Honorable Congreso de la Republica, Agosti 1966 a Julio 1967. Bogota. - » “ x . ‘ B 3 - + 1 . < w - - . N * f . ) E . ~ hb = . . = Ed - BN ) - . k - - . . - . . . : Bh oe # i Bh “ i . oo . . - - . x - & - 5 EVALUATION OF CHILD HEALTH CARE AS SEEN BY EVALUATORS Jaime Arias Introduction There is a logical order to the three previous presentations. To continue along the same lines, I will present some ideas on evaluation of medical care from the viewpoint of the evaluator. Dr. Wray and Dr. Rosselot have described quantitatively the problems affecting child health in the Americas and have summarized the existing programs created to tackle those problems. They have identified the problem for us, allowing us to proceed to the conceptual, political, administrative, and technical considerations in order to evaluate both process and outcome. Although identification and quantification of problems and their related health activities are components of the evaluation process, these aspects cannot be considered evaluation in themselves unless other aspects are included, particularly those that permit comparison and judgment of data and information. Dr. Horwitz has pointed out the importance of child health programs compared with programs devoted to other risk groups. He has further directed our attention to efforts to measure and evaluate programs in the field of health care in each country of the Americas. One of the basic purposes of evaluation is to determine priorities between programs; the priorities may then be examined in relation to the overall economic and social development of the country or community under study. These considerations are more important when one is dealing with a country or community where scarce resources must be distributed optimally, avoiding waste and maximiz- ing the output of each resource. Before priorities among different health programs can be set, each country must determine its basic policies, such as the relative importance of its investments in education, nutrition, and medical care. Then, within the health sector, the policymakers should deter- mine if, for example, preventive medicine programs are justified in light of the country’s existing needs and resources. What is the 57 58 EVALUATION OF CHILD HEALTH SERVICES proportion of primary care required vis-a-vis secondary care, and how can we use evaluation tools to find out? The aforementioned considerations bring forth a multitude of questions: What aspects of a program are to be assessed, and how? Which activities should be evaluated, and what measures should be used? How can one assess efficiency, cost-benefit and cost-effective- ness, program efforts and inputs, quality of procedures, and, of course, short- and long-term impacts of efforts? Dr. Ordonez Plaja has discussed evaluation in terms of appropriate- ness. More importantly, he has shown that there must be an appropriateness in the use of results. He has also spoken of the factors that can decrease effectiveness in the relationship between administrator and evaluator. He has emphasized the time factor and the importance of clarifying investigative and evaluative goals. Community participation in the evaluation process has also been mentioned; the community should participate, not only as the subject of the evaluation, but also as a key factor in the evaluation process itself, from the programing and design of the evaluation to the implementation of change. Dr. Ordonez Plaja explained that external funding sources may influence the evaluation process according to their own interests rather than those of the program or agency under study. Hence a certain amount or proportion of funds should be set aside for evaluation in each particular agency or program. His viewpoint is that of the administrator at the political level for whom evaluation is just another activity belonging to a broad planning and administrative process. As a follow-up to these thoughts, I shall discuss three topics: —The definition of the evaluation process within the health care system; —The “evaluator,” his or her role, field, way of working, and relationships with politicians and administrators; —Technology transfer in evaluation between the countries of the Americas and the future of evaluation of health care programs in the region. Evaluation as Part of Medical and Health Care In the last decade evaluation has become so important it is in danger of becoming an end in itself, an independent system, when it should be an integral part of the planning of medical and health care. Evaluation As Seen by Evaluators 59 RISK FACTORS ————1 Population-At-Risk, by groups Disease Cases (Different Stages) Demand for Services (Felt need, interest, capacity for demand) FINANCING OF HEALTH CARE Clinical or Other Specific Procedures Formal or Nonformal Health Care Programs Manpower and Other Inputs Figure 1. Flow chart of the medical care process OUTCOME Using a so-called Health-Disease-Intervention (HDI) model, one can apply the different aspects of the evaluation process in order to clarify the relationship between health care and evaluation and to illustrate the relevance of the latter in that general health care context. Figure 1 is a flow chart of the overall medical care process, including a population at certain risk to disease and the requirements of that population for medical services; a network of individuals and 60 EVALUATION OF CHILD HEALTH SERVICES POPULATION Epidemiologic Research DISEASE CASES Impact Analysis DEMAND = OUTCOME Financing A SPECIFIC Quality PROCEDURES Assessment Effectiveness Cost- Studies Effectiveness PROGRAMS Studies Efficiency Analysis Effort INPUTS AND Assessment RESOURCES Figure 2. Normal relationships between health care and evaluation processes institutions organized to render services through a series of specific procedures for each patient and particular disease; a financial infrastructure that includes costs, prices, fees, and payment of services. The infrastructure is located at both the level of the population and the level of the institutions and individual providers. Finally, the interaction of demand and supply (or capacity of services) has an outcome affecting individuals, the total population, institutions of health care, individual providers, the specific health system, and the global social system. Figure 2 shows the different types of evaluation which have a clear relationship to the overall process of medical care in such a way that each of the components of that process can be assessed according to a complete and systematic scheme. We can see that some types of evaluation require only the consideration of one component of the medical care process. Others Evaluation As Seen by Evaluators 61 imply consideration of both demand for and supply of services. Consideration of outcome, as in the case of impact analysis or cost- benefit studies (including the comparison of outcome with the cost and effect of the programs), is also important. There never will be one ideal evaluation method, but we can hope to develop adequately conceptualized and executed methods that will vary according to the appropriateness of the process to the program to be evaluated. Only when the various evaluation aspects are carried out can we have adequate and sufficient knowledge of a program. In real life, complete and absolute knowledge is neither possible nor necessary. The accumulated knowledge about most components of the HDI process permits the application of programs and technical procedures. In many instances, the causes and mechanisms of diseases are well established, and duplication of basic research is not needed— particularly in the less-developed countries. Basic epidemiologic studies do not require duplication in each country or in each particular situation, whereas applied epidemiologic studies should be carried out for each different situation, even in less-developed countries, since the complexity and costs of such studies are not beyond the ability of such countries. One example of such studies would be “The distribution and prevalence of kwashior- kor in a given rural province of a country.” When one analyzes the secondary stages of the HDI process, it is clear that specific evaluation studies are needed—not only for each country, but also for each program, or at least for each particular type of program—since a number of social, cultural, economic, and psychologic variables influence the effectiveness and impact of the program or procedures. This type of evaluation is essential for each country, program, and institution. An example of this type of study would be “The effectiveness of program X (or procedure Y) in decreasing the prevalence of kwashiorkor among a given population of children.” Evaluation of clinical, surgical, laboratory, or other specific proce- dures is done at the tertiary stage of the process of organization of services. This sequence does not mean that effort evaluation or effectiveness analysis, for example, applied in previous stages is more important. Figure 2 and table 1 give us a better understanding of the primary, secondary, and tertiary stages of the HDI process. They help us to understand the relationships among basic medical research, epidemi- ologic studies, program evaluation, and clinical procedures evalua- tion. Traditional definitions of evaluation do not include basic research on causes and mechanisms of diseases, therapeutics, or epidemiology. 62 EVALUATION OF CHILD HEALTH SERVICES Table 1. Stages of the HDI Process and Different Types of Studies Applied to Each Stage Stage Demand Supply Studies Primary Healthy and Basic and epidemiologic unhealthy research population Secondary Demand and social ~~ Resources, cost Evaluation research outcome programs Program evaluation Tertiary Individual outcome Specific Evaluation of specific procedures procedures However, when these types of research are considered in relation to the complete HDI process, one can understand their links to traditional evaluation studies. In terms of evaluation methodologies, we will have the opportunity to hear and analyze nine cases, selected in such a way that we will have a sample of most of the best known methodologies for various program needs and circumstances. It is important to understand the adaptation requirements of evaluation methodologies to specific program needs. We cain identify at least three different levels: (1) experimental or quasi-experimental designs of varying complexity; (2) nonexperimental designs for program evaluation; or (3) very simple designs for self-control of performance, suitable for any practitioner. Table 2 is a diagram of the three possible stages of an experimental program or procedure. The first level requires experimental or quasi- experimental designs, with control groups, and so on. This stage has been called “evaluation research.” Once the programs or procedures have been evaluated, they can become “demonstration programs or procedures.” These in turn should be evaluated by means of a technical and sometimes complex design that is, in general, of the nonexperimental type. Finally, when demonstration programs or procedures have proved their appropriateness and feasibility, it is possible to multiply them as “extended programs or procedures.” These require program evaluation of simple design and easy applica- tion. This type of evaluation can be called “control evaluation” or “monitoring.” Table 2. Evaluation Designs for Each Type of Program or Procedure Program or Procedure Evaluation Design Experimental Experimental or quasi-experimental (Evaluation research) Demonstration Systemic nonexperimental Extended Control evaluation or monitoring Evaluation As Seen by Evaluators 63 The Evaluator—His Role, Work Setting, and Working Relationships Once the evaluation process is defined, it is necessary to decide who will carry it out. First, it should be ascertained whether there are available professionals who are devoted exclusively to evaluation and who have received formal training in the field, or whether there are individuals who have learned how to perform the functions of an evaluator. Second, the evaluator’s role, the setting in which he or she carries out that role, and the relationships between the evaluator and administrators or decisionmakers must be considered. A panel of three esteemed medical evaluators has reviewed the more important evaluation studies published in Bogota, the capital of Colombia, during the last 2 years, and it has selected 30 individuals who participated in the studies and who can be considered as evaluators, or at least temporary or partial evaluators. A simple survey was distributed among the group of 30, requesting informa- tion about their professional backgrounds, activities, and the setting and conditions of their work. None considered systematic evaluation to be his/her only professional activity, and only five (16.6 percent) said evaluation was the most important of their professional activities. Only three evaluators at the time of the survey were working on the discovery or testing of new indicators. Twenty individuals (66.6 percent) had received adequate postgrad- uate training in evaluation, almost always as part of training in public health, sociology, or statistics; the rest had been trained through their professional practice. Of those with formal training, 11 (33.0 percent) had studied in the United States or England for more than 2 years. In Bogota the majority of evaluators of health programs are medical doctors (23); the rest are sociologists (3), statisticians (2), and economists (2). Within the group of 23 medical doctors, 15 had different specialties in public health, social medicine, or community health, and the rest were clinicians. Taking into account the amount of professional time devoted to evaluation during the last 2 years, the survey revealed that 7 of the respondents had spent more than 4 percent of their time on evaluation; 4 spent between 20 percent and 30 percent; and the other 19 spent less than 20 percent of their time on evaluation. Twenty- three were carrying out some evaluation study at the time of the survey, and only 13 had newly designed and funded evaluation projects for the next 2 years. Twelve worked for governmental institutions (40 percent); the remainder, for universities or research organizations. It is clear, at least in Latin America, that there is not a special group of professionals devoted entirely to evaluation of medical care, 64 EVALUATION OF CHILD HEALTH SERVICES and one can predict that this situation will not change in the near future. Since the “felt need” for evaluation is increasing and most programs require it, it is likely that the number of individuals developing new methods and indicators, as well as those doing evaluation, will increase—but not to the extent to which the profes- sional can be considered purely as an evaluator. According to present trends, we can predict the need to train at least three types of evaluators for the future: —Those devoted to the experimentation and development of metho- dologies and indicators that will be applicable to extended pro- grams, projects, and individual practice. These professionals will work exclusively in evaluation, and their supply will grow more rapidly in industrial countries. —Those individuals devoted to the development and evaluation of experimental, pilot, and demonstration programs or to the evalua- tion component of extended or action programs. These individuals should have formal training in evaluation. —Finally, those professionals who will perform some sort of self- evaluation of their activities; therefore, it will be necessary to include the teaching of basic evaluation techniques in the curricula of most health professions. (Today, the health professional does not have information about evaluation, yet most medical and other health-related schools are not including information on evaluation in their curricula.) Now that I have briefly described the characteristics and profes- sional backgrounds of the present Latin American evaluators, I will attempt to identify their roles and the conditions of these roles. Evaluation is relatively new in Latin America, and the evaluator’s role is not well defined. It varies according to the type of evaluation, the program to be evaluated, the purposes of evaluation, and the institu- tion where the evaluation is taking place—that is, according to the evaluator’s working conditions. When evaluation takes place in an academic setting, researchers have more freedom and control over their work. The reverse occurs when professionals are doing their work for an agency devoted to health programs. The role also varies according to whether an evaluator is working on a single project or program or for an evaluation unit in charge of the coordination of many projects or programs. In the case of program evaluation, the role includes the evaluator’s participation in planning and program design, as well as in the implementation of changes through the application of evaluation Evaluation As Seen by Evaluators 65 results. In the case of control evaluation or monitoring, the evalua- tor’s role should focus mainly on the teaching and advisory aspects in order to help those individuals who are doing self-evaluation. In general one can summarize the evaluator’s functions as follows: —To promote evaluation techniques; —To participate in the design of experimental programs or projects; —To design an evaluation methodology to be applied to the program, or to select one which is appropriate; —To carry out direct collection and processing of data and informa- tion needed, or to advise on data and collection methods; —To analyze or to help in the analysis of collected information; —To describe, explain, and communicate the evaluation results; —In some cases, to motivate and persuade the administrators con- cerning the importance of the use of results in the change process. —To collaborate in change implementation, basing it on evaluation results; —To educate those individuals engaged directly or indirectly in the evaluation or planning processes. When the evaluator’s functions are taken into account, the impor- tance of the relationships between the evaluator and all those individuals who plan, manage, and make decisions on projects or programs becomes apparent. The success here depends not only on the adequate execution of the evaluation, but also on the capacity of the evaluator to “sell” the results and engage all the key persons in this complex process. Thus it is important to consider the management of interpersonal relationships. The evaluator should detect and analyze carefully the constraints and determine which individuals are more likely to obstruct sound relationships. Taking steps to avoid or to detach and solve these problems is part of the strategy that must be developed at the beginning of any evaluation effort. I would like to stress the important role of evaluators as educators. This role enables evaluators to multiply their efforts and permits the institutionalization and internalization of the evaluation concepts. The last two papers of this conference address the importance of evalua- tion as a tool for education. Technology Transfer of Evaluation Between the Americas It is clear that evaluation techniques are developing at different paces and in different directions in industrialized countries and in 66 EVALUATION OF CHILD HEALTH SERVICES preindustrialized ones. We mentioned before that basic and epidemi- ologic research should be the primary focus of developed countries, while evaluation of experimental, demonstration, and applied pro- grams, as well as the evaluation of specific procedures and applied epidemiologic studies, should be of interest to both types of countries. We should encourage technology transfer and communication in those studies common to either type of country. Transfer should take place both at the level of new methods and indicators and at the level of results of those analyzed programs. Transfer and communication should occur not only between countries at different stages of development and with different cultures but also, more importantly, between countries of similar development and similar problems, and between institutions or programs within a single country. It may seem obvious, but it is important to reiterate that in countries with technological and financial limitations, it is very important to avoid duplication of studies. The ready exchange of ideas on methodologies and results will eliminate this problem. In order to establish effective transfer, it is necessary to create new communication mechanisms: conferences such as this, publication of research, studies of different programs, and training of evaluators with participation of different countries. It is desirable, through an exchange of ideas, to devise simple methodologies and indicators that assure validity and reliability and are simple to execute. Such designs can be applied more universally. It should be noted that this paper does not make specific reference to evaluation of child health programs because, from the evaluator’s point of view, the principles and techniques of evaluation are similarly applicable to any population-at-risk group. CASE STUDIES 6 NINE EVALUATION CASE STUDIES: AN APPROACH TO THEIR CLASSIFICATION Samuel |. Bosch and Jaime Arias The team that planned this conference selected nine case studies in an attempt to indicate the diversity of current perspectives on health care evaluation in different parts of the Americas, as well as to reflect the overall state of the art in the hemisphere. This paper is an effort to classify these studies in a variety of ways to facilitate their organized analysis in the workshop discussions. Classification of evaluation studies is often difficult because every study involves multiple aspects of evaluation and because the defini- tions of different types of evaluation lack uniformity. Moreover, terminology often varies from one country to another and from one language to another. Over 60 studies from 14 countries were reviewed prior to selection of these 9 cases. Limitations in time and in resources necessitated the omission of a number of excellent studies. It is nevertheless fortunate that the authors of some of those omitted papers were among the moderators and participants in this confer- ence. Each of the cases possesses a series of characteristics that the planning team considered important to bring to the attention of the conference for analysis and discussion. Each case corresponds to a well-known academic effort in its own country and represents a different type of health care program and evaluation methodology. The following characteristics were taken into consideration in select- ing the studies: —Country or region of origin (of both the study and the principal investigator); —Size of the geographic area in which the program to be evaluated operates; —Type of population-at-risk analyzed; —Level or area of health care under study; —Type of program and program objectives; 69 70 EVALUATION OF CHILD HEALTH SERVICES —Purpose of evaluation and originators of the evaluation; —Type of evaluation: effort, efficiency, effectiveness, impact, cost- benefit, quality assessment; —Methodology and indicators used in each study; —Use and implementation of results. The workshops analyzed each study primarily in relation to four of the above-mentioned characteristics: the type of evaluation, particu- larly in relation to the objectives of the program; the problems encountered with the methodology; the difficulties in implementation of results; and the applicability of the method to other programs and cultures. There is no particular technical reason for the sequence of the case presentations. The chosen order intermingles Hispanic and English- speaking authors in an attempt to enhance communication among professionals with different languages and different cultures. Based on the above, the case studies can be classified as follows: —Country of origin. There are four studies from Latin America and the Caribbean (Agudelo, Heredia, Ramsey, Yarbrough); four from the United States and Canada (Belville, Kessner, Pless, Williamson); and one from the United Kingdom (Martini). The paper from the United Kingdom was presented by a Latin American investigator, while one of the Latin American studies was presented by a North American investigator (Yarbrough). —Size of the geographic area. There is substantial variation in the programs under study. Two national health care programs are evaluated (Heredia, Ramsey). The remainder focus on counties or provinces (Agudelo, Pless); limited communities (Kessner, Martini, Pless); different forms of practice (Kessner); health centers (Belville); and, in one case, a variety of settings (Williamson). —Type of population-at-risk. Almost all the studies are related to child populations. In those not directly related to children, the method- ology is applicable to child care programs. In one instance, the evaluation includes mothers as well as children (Heredia). —Level or area of health care under study. Some papers stress primary care (Agudelo, Belville, Kessner, Martini), and others focus on secondary care or on all three levels of health care (Heredia). Two cases refer specifically to nutrition (Ramsey, Yarbrough), and others use specific diseases as indicators (Kessner, Martini, William- son). —Type of program. In some case studies, the evaluation has been applied to “experimental programs,” and hence their design is of an experimental type (Kessner, Martini, Pless, Williamson, Yar- brough); other case studies are related to demonstration programs Classification of Case Studies 71 (Agudelo, Belville, Ramsey) or to the so-called extended or current programs (Heredia, Ramsey). —Purpose of the evaluation. A large proportion of the papers have an academic purpose, and their initiation must be credited to acade- micians. In many of these academically oriented studies, the application of research findings to program development is a clear aim. In some instances, the study goes beyond assessment of a particular program or project and seeks the discovery and devel- opment of new methodologies (Heredia, Kessner, Martini, William- son). Some of the papers have a clear administrative or policy- making purpose and attempt short- or long-term program changes (Belville, Heredia, Ramsey), whereas others rely more on educa- tional processes to achieve change (Kessner, Williamson). —Type of evaluation. Some cases address issues of quality assessment, including elements of effectiveness (Kessner, Martini, Williamson). Two case studies address program effort or inputs (Agudelo, Heredia), and other papers focus on studies of program efficiency and effectiveness (Agudelo, Belville, Pless, Ramsey, Yarbrough). One case includes cost-benefit and long-term impact analysis (Heredia). —Methodology. A wide range of methodologies and indicators is utilized in the nine case studies under analysis. There is a mathematical simulation model (Heredia), which uses known indi- cators of outcome, effectiveness, and efficiency. There are experi- mental designs with well-selected controls (Kessner, Martini, Pless, Williamson, Yarbrough) that use specific health conditions as indicators with a different methodological approach. Martini uses sociomedical indicators with data provided by the patients. Kessner uses a prospective approach for analysis of outcome which he combines with education. Williamson analyzes the diagnosis and treatment processes and their outcome for specific health condi- tions and relies on educational strategies to achieve early change. Pless presents a more global approach to the evaluation of a community health program using systemic surveys as a way to assess health status and services rendered, and Yarbrough presents a quasi-experimental design to measure the effectiveness of a nutritional program. Belville uses a method called “concurrent evaluation,” which attempts to produce ongoing change in the health care program through active participation in the study of those engaged in the operation of the programs. Agudelo and Ramsey combine the measurement of effort and effectiveness in demonstration programs using nonexperimental designs and rela- tively simple methodologies with a combination of indicators. 72 EVALUATION OF CHILD HEALTH SERVICES —Implementation of results. In some of the cases, the use of results is direct and immediate (Belville, Ramsey, Williamson); in others, the use of results is direct but requires some delay in their implemen- tation (Agudelo, Kessner, Martini, Pless). In the remaining cases, the use of results is indirect and requires a longer period of time before implementation (Heredia, Yarbrough). In some instances, the application of results is made directly by an individual practi- tioner or by a group practice (Kessner, Martini, Pless), whereas in other instances, the application is made by the Government or administrators (Agudelo, Belville, Heredia, Pless, Ramsey). Finally, in several papers the method in itself and the possibility of its application are the main focus of the study (Heredia, Kessner, Martini, Williamson). 7 SOCIOMEDICAL INDICATORS FOR CHILD HEALTH Carlos |. M. Martini and Ian McDowell Introduction Many indicators have been used to estimate the need for child health services. These include birth rates, the incidence of low birth weight, the proportion of the population under age 15, stillbirth rates and, most important of all, infant and perinatal mortality rates (Report of the Committee on Child Health Services 1976, Lowe 1976). In addition to these, a variety of other indicators related to the mother, such as the number of children she has, short pregnancy intervals, small physical stature, socioeconomic status, and certain behavioral variables, such as whether or not she smokes, are being used in Britain as bases for high risk strategies in planning care for children. Although these data are used frequently, some caution must be expressed over their use as indicators of need and for evaluating the outcomes of child care (Department of Health and Social Security 1976). While indices such as the sickness and death rates of children at birth or in the perinatal period may provide useful data for the overall planning and development of social and medical services, they are not specific enough to identify the effectiveness of health care alone. Infant mortality rates in general are known to reflect a number of social and environmental influences. They should therefore be considered as overall indicators of community poverty or deprivation (and therefore, perhaps, also as estimates of global needs for services), rather than as specific indicators of the outcomes of health services. We have examined the relative contribution of medical care provision and social factors in explaining the interregional variation in a number of indicators of child mortality and morbidity in England and Wales (Martini et al. 1977). Figure 1 gives a summary of the This project is financed by a grant from the Social Services Research Council, London. 73 74 EVALUATION OF CHILD HEALTH SERVICES Proportion of Medical care Socio-demographic Total variance Dependent variable provision variables Collinearity Explained (R2) Infant mortality/1000 live births, 0.18 0.39 0.06 0.63 Perinatal mortality/1000 live births. 0.26 0.29 0. 06 0.61 Figure 1. The relative contribution of medical care and sociodemographic variables to variation in two indicators of child mortality rates, England and Wales, 1970-72 typical findings, which show that the proportion of variance in infant mortality rates explained by sociodemographic characteristics is more than twice that explained by medical care.? For perinatal mortality, the relative contribution of the two com- ponents was virtually equal, but the data show that this indicator is still relatively insensitive to the provision of medical care. Similar results were also obtained for crude and specific mortality rates. However, one indicator—the overall case fatality rate—appeared to reflect the contribution of medical care more adequately. In this instance, the medical care contribution to variance explained was 0.56 (the total variance explained being 0.75). Let us assume that these findings express, albeit crudely, a real phenomenon—that the impact of medical care on many accepted indices of child health is subordinate to that of the socioeconomic and environmental circumstances of the population. The interpreta- tion of this finding will vary according to our expectations for the scope of medical care. If, for example, we argue that child health care should strive to improve the total environment of children, then these results suggest that such care is not succeeding in its goal, for its influence is still outweighed by that of social circumstances. Alternatively, if we take a more restrictive view of the role of medical care, then these indices are not adequately specific for its evaluation. To select more suitable indices for planning and evaluating care, it is necessary to describe in more detail the various components of the planning process. ! The variables considered were age of the population, social class distribution, level of urbanization, levels of income and education. To describe medical care, the following variables were included: health and social services expenditure, hospital resources and performance, and various indices of community care. Sociomedical Indicators 75 Indicators for Planning and High Risk Strategies The first stage in planning health care requires an overall descrip- tion of the actual health status of the population. The previous findings, in our opinion, do not invalidate the use of traditional indicators for this task, since such an assessment is not concerned with specific health care activities, but with reflecting medical, social, and environmental conditions. For this purpose, indices of child mortality rates remain the most readily available and general proxy measures of child health problems in a community. The second stage in planning care is to specify the type of intervention most suitable for the problems outlined by the overall description of health status. With the growing emphasis on preven- tion, an application for the indices of mortality rates again arises, since these do reflect social conditions. Many of these social correlates of mortality rates may therefore be measured in the population. Groups at high risk may also be identified (World Health Organiza- tion 1976). In this sense, the mortality rates provide a more useful basis for planning preventive interventions (high risk strategies) than for planning specifically medical activities. The third stage of planning care covers evaluation of the effective- ness of the programs implemented. In evaluating child health programs, we must address only those elements of the whole health/ sickness process which may reasonably be affected by care. The results shown above indicate that mortality and morbidity rates reflect social realities that are normally considered beyond the scope of health services. It is only, perhaps, at the level of community child health care that such indicators may provide relevant evaluative information; they are still relatively insensitive to the detailed inter- ventions of the health care team. This phenomenon is particularly noticeable where much of the work is not concerned with potentially fatal conditions. To overcome these problems of evaluative methods, alternative indices of child health levels must be considered. For example, if it were possible to develop a survey instrument to distinguish the contributions of social circumstances and of medical care to a child’s health, it would be possible to plan and subsequently evaluate health care interventions tailored more specifically to existing problems. Outcome indicators based on measurements of a child's functional abilities, in which the parents are asked about changes in their child’s and in their own behavior as a result of the child’s ill health, could provide a simple and yet specific means for assessing the less serious types of condition seen in primary health care (McDowell and Martini 1976). Measurements of this type would not be intended to replace 76 EVALUATION OF CHILD HEALTH SERVICES the traditional health indicators, but rather to complement them, first in the area of evaluation and then also in the assessment of the need for care. The Development of a Sociomedical Child Health Index We are at present developing and testing a measurement technique in which parents (preferably the mother) are asked to comment on alterations in the child’s normal activities and also to describe changes in their own normal lifestyle produced by the child’s illness. (Older children can of course comment directly on such changes.) These perceived changes or problems can be used in before- and after-care evaluation to indicate the short-term effects of medical intervention in reducing incapacity, as well as in restoring the patient’s normal functioning—social, emotional, and physical. The child health index is applied through interviews with which- ever parent has brought the child to care, using a standardized interview schedule. In the past, the parent has been extensively used as an informant; a number of behavioral inventories, such as the Peterson checklist (1961), the Achenbach index (1966), and Rutter’s screening instrument (1967), have exemplified this approach. The interview schedule is based on a collection of statements obtained from 536 interviews with parents whose children were attending the outpatient clinics at the Nottingham Children’s Hospital. As with the method used in Bergner’s Sickness Impact Profile (Bergner et al. 1974), all the statements describe changes or restrictions in behavior due to the child’s illness, relating either to the child or the parent. This procedure permits the development of two types of profile: one describing the child’s problems and another describing the parent's altered activity levels. The rationale for inquiring into the parent’s own behavior is based on the simple fact that the child’s illness will temporarily affect his parents’ quality of life, with possibly detrimental consequences for the child himself. Obviously a parents life is affected by children, whether they are healthy or sick, and this impact is illustrated in figure 2. The data shown in figure 2 come from a preliminary test of the child health index on two contrasting groups of women: 15 mothers with young children (under 12 months), and 12 women in at least the 28th week of their first pregnancy. With a sick child, the parent's normal activities will be further modified, so that establishing the patterns of activities of mothers with healthy children of a given age will make it possible to study the additional effect of the child’s illness. Alternatively, the questions can be clearly worded so that the respondent is asked to report on changes—over Sociomedical Indicators 77 Statements % of women who answered 'yes' to a particular statement | go out less to visit people 1 often act irritably towards people C= Mothers with young children - | worry about my state of health Pregnant women | get upset more easily | am cutting down on physical activities | do not finish work that 1 start | act irritably towards my family | express worries about the future | tend to cook simpler meals | eat less than usual 1 ti 1 1 1) freind 0 10 20 30 4 S50 60 70 80 9 100 Figure 2. Comparison of mothers with young children (» = 15) and pregnant women (n = 12) and beyond the normal work of caring for the child—brought on exclusively by the child’s illness. Results of Statement Collection Stage As the basis for constructing the child health index, 1,070 state- ments were collected. These were grouped by four judges into ten categories. General titles indicating the content of these categories are shown in table 1. In table 2, the types of statements describing changes in the parents’ behavior are shown, and table 3 gives examples of some of the statements collected. The age and sex of each child and the type of clinic being attended were recorded. The parent's age, occupational status, and the number Table 1. Titles of Categories of Statements Collected as a Basis for Child Health Index: Sections for Children Sleeping problems Eating problems Changes in play activities and hobbies Sociability and family relationships Increased dependency Irritable, moody, and uncooperative behavior “ Emotional and upset responses Indications of depression and boredom Languid and listless behavior Symptoms of ill health rEpommpoEp 78 EVALUATION OF CHILD HEALTH SERVICES Table 2. Titles of Categories of Statements Collected as a Basis for Child Health Index: Sections for Parents Sleeping problems Coping with daily routine Restrictions on going out Changes in social life and leisure activities Working patterns Family relationships Altered caring behavior and patience Increased concern Fatigue and tiredness Increased mental tension Feelings of depression Irritability Symptoms of ill health EORTC I DOmMEOOR p> Table 3. Child Health Index: Some Examples of Statements Collected Parents Children Caring Behavior and Patience I am more indulgent towards the child—allow bad behavior without scolding. When he is asleep, I check that he is all right more than usual. Fatigue and Tiredness I found difficulty in getting up. I become forgetful, confused and un- Sociability and Family Relationships He doesn’t want to know you. He doesn’t want to mix with other children—prefers to stay at home. Dependency He wouldn’t allow me to leave the room. He behaves younger. Irritable, Moody, and Uncooperative Be- methodical. havior Feeling Low and Depressed He’s nasty tempered and very hard to I tend to neglect myself a lot. please. I have feelings of complete inade- quacy. He's hard to control. of children in the family were noted, and these data were used to tabulate the statements provided. Rather than using individual statements, the general categories shown in tables 2 and 3 were used in these tabulations so as to provide adequate numbers in each cell. This method evidently provided a very rough indication of the types of material collected from different respondents. However, it did indicate how much the type of information given varied by the age and sex of the child. Figure 3 indicates very little variation in the overall number of categories of statements mentioned for children of differing ages—that is, the range of problems did not appear to vary by age. A similar finding for the type of clinic attended is shown in figure 4. Sociomedical Indicators 79 1 4 - < 8 <3 " po £ NON N MN g N NN N EL» L N NN a NN N N NG NN IN NG ¥ 1 + N NN IN AN Ed NN MN N N A N NG NG q uy <1 1 2 3 4 5 6 7 8 9 10+ Age of child Figure 3. Average number of dimensions in which children experienced problems, by age (n = 536) “1 1 5 fw 4 | 3 - Dimensions LLLL Neurology Fracture 777777 77777 a? 77777. Medical ([[/[[[[/] Clinics Figure 4. Average number of dimensions in which children experienced problems, by type of clinic (n = 381) 80 EVALUATION OF CHILD HEALTH SERVICES wr Pd 60 | 40 Percentage 10 |- LiL LLLL LLL] N B C DE F c H I J Dimensions vil LLL LlL LLL] Figure 5. Percentage of children experiencing problems in each of 10 dimensions (n = 536) Some categories, however, were mentioned much more frequently than others: Sleeping and eating were the two most frequently mentioned, while sociability was the category least affected by the child’s illness (figure 5). Certain categories showed some variation in response frequency by the age of the child: for example, younger children showed more changes in dependency and irritability than older children (figure 6). Among the parents’ reactions to the child’s illness there was no variation in the categories of statement mentioned by the age of the child, or by the parity or the age of the parent. The only clear trend was for parents of higher social class (using the Office of Population Censuses and Surveys [1970] British classification) to mention symp- toms of fatigue and tiredness more frequently than did lower social class parents (figure 7). These simple analyses give some information about the possible need for different versions of the questionnaire for children of different ages or for different types of condition. In the next stages of the project the most suitable statements for inclusion in a preliminary version of the index will be chosen and this Sociomedical Indicators 81 o———o Dependency (n=175) 0 & - = - 4 [rritable, moody and unco-operative (n=208) Percentage 0 1 1 1 1 1 L 1 1 1 1 J <1 1 2 3 4 5 6 7 8 9 10 Age in years Figure 6. Percentage of children experiencing problems within two dimin- sions, by age 100 \ \ 40 30 Percentage ol1 ] I ] ] J | i HIN 1lIM Iv Vv Social Class Figure 7. Percentage of women experiencing problems of fatigue and tiredness, by social class (n = 108) 82 EVALUATION OF CHILD HEALTH SERVICES will be tested for reliability and validity in a series of trials. Where a numerical assessment of the child’s health status is desired, weightings representing the severity of disability implied by each statement will be calculated. In this way the final index may be used either in a simple, unweighted manner, to provide a conceptual picture of the problems of a child or a group of children or to give a numerical indication of the extent of change before and after treatment. Applications of the Index There are at least three main applications for such indices. First, they provide an important means for improving communication between the individual patient and the health team. By providing a standardized way for expressing the patient’s main areas of concern, they specify accurately and concisely the demands that he is placing on the health care team. Inquiry into these problems is, of course, part of good medical practice, but the index nevertheless represents an important timesaving and summary device for the health care professional. Many patients find that the “checklist” approach re- moves the inhibiting stresses sometimes experienced in verbally explaining problems to the doctor. The second important application is in the area of medical education, since the information provided by these indices promotes a more holistic approach to patients’ problems. Although the interre- lationships between the individual and his social and physical environ- ment have been systematically explored for a number of years, the relevance of individuals’ adjustments to their life situation is still ignored in many medical curricula. The third and perhaps most significant application of the index is in the evaluation of care. It can be employed (and we are testing this use) in before- and after-care studies, to build profiles of individuals and groups, and thereby to show what impact is being produced on the quality of life of patients under treatment, so as to modify the management of the cases accordingly. A possible fourth application, not yet explored, is the use of the index in population surveys to contribute to the study of needs for services. In considering this type of application, note that most existing sociomedical indices lack predictive power and are of relatively low value in showing which of alternative strategies should be adopted in the long term. The knowledge of the increased risk of disease attributable, for example, to age, smoking, or obesity may be used by health practitioners for preventive care and maintenance of health. The future development of sociomedical indices should include the incorporation of transitional probabilities (Kaplan et al. Sociomedical Indicators 83 1976) between states of social dysfunction, thereby providing a fuller indication than exists at present of the risk that a patient will fail to recover. We believe that adding information concerning high risk factors in childhood, about possible asyptomatic disease (sometimes detectable through screening), and the actual amount of medical morbidity to the basic sociomedical index will improve its analytical power. The index will then combine the available information on functional status, morbidity, and prognosis. It is not yet known how much these risk factors, such as parity of the mother, birth weight, or socioeco- nomic status affect the functional status of the child, although some fascinating work has been published on the relationship of neonatal disease and low birth weight to subsequent behavioral disturbances, and to reading attainment (Daye et al. 1972). It may be that behavioral indices of the type described in this paper already reflect these other prognostic factors, and it is our intention to explore these possible relationships further. Figure 8 shows a possible model of how these variables could interrelate. Health status (indicated by the rectangle at the right) is shown as related to morbidity (as measured by conventional, biomedical indices), to functional levels (as measured by an index of the type outlined above), and to prognosis (as measured by information on predictive risk factors). The way in which these factors interrelate to influence “health” is very inade- quately specified, but the diagram shows that risk factors, such as neonatal disease, may influence morbidity and thereby influence health status. Alternatively, the risk factors may not produce overt, medical morbidity, but may influence a child’s functional status in areas such as scholastic attainment. The implications of this concept for strategies of intervention are important. Intervention may occur at four stages: We may intervene in modifying the risk factors, often in a program of social action (as was shown necessary to alter child mortality rates). Alternatively, strategies of screening may detect presymptomatic disease and, where earlier detection allows cure, may improve health status. Third, the conventional medical intervention may discover biomedical aspects of morbidity. Finally, it is also logically possible to intervene directly at the level of the patient’s functional abilities (as is done with the provision of prostheses). It is clear that conventional medical care represents only one of several possible stages for influencing the health status of the individual: Very plausibly, this is an alternative way of analyzing the results of variations in death rates outlined in the introduction. Clearly, death rates will not represent the full health status of a population, but they may reflect to some extent each of these three EXAMPLES OF RISK FACTORS: 1. PREMATURITY _ ~~ <___ IPRESYMPTOMATIC | CONDITION - | m=~—~_l HEALTH 2. NEONATAL DISEASE | (e.g. brain damage | STATUS | and mental retardation) | I 3+ (Depending on risks, MORBIDITY functional levels) (e.g. mongolism) HE 2 i ps oy: oy l FUNCTIONAL! ABILITIES 3. AGE OF MOTHER | | | \ 1 4. SOCIOECONOMIC STATUS BEHAVIORAL INDEX INTERVENTION MAY AIM AT MODIFYING RISK FACTORS, MORBIDITY, OR HEALTH STATUS Figure 8. Hypothetical model of influence of risk factors on child health ¥8 SADIAYAS HLTVAH ATIHD 40 NOLLVATVAY Sociomedical Indicators 85 components of morbidity, functional levels, and risk factors. We conclude that, as mortality rates are too general to indicate the outcomes of medical care, a pattern of indices covering aspects of risk factors, morbidity, and functional status will provide a more specific analysis of the outcomes of child health services and of the factors which may modify these outcomes. REFERENCES Achenbach, T.M. (1966) The classification of children’s psychiatric syndromes. Psycho- logical Monographs 80:7:1-37. Bergner, M., Bobbitt, R.A., Kressel, S., et al. (1974) The sickness impact profile: Conceptual formulation and methodology for the development of a health status measure. International Journal of Health Services 6:393-415. Committee on Child Health Services (Chairman, Sir S.D.M. Court) (1976) Fit for the Future. London: Her Majesty’s Stationery Office. Davie, R., Butler, N., and Goldstein, H. (1972) From Birth to Seven: The Second Report of the National Child Development Study. London: Longmans. Department of Health and Social Security (1976) Sharing Resources for Health in England. London: Her Majesty’s Stationery Office. Kaplan, R., Bush, D., and Berry, C.H. (1976) Validity of an index of well-being. Paper presented at Health Status Index Conference, Phoenix, Arizona, October 21-28. Martini, C., Allan, G.J.B., Davison, J., et al. (1977) Health indices sensitive to medical care variation. International Journal of Health Services 7:293-309. McDowell, I., and Martini, C. (1976) Problems and new directions in the evaluation of primary care. International Journal of Epidemiology 5:247-260. Office of Populations Censuses and Surveys (1970) Classification of Occupations, 1970. London: Her Majesty’s Stationery Office. Peterson, D. (1961) Behavioural problems of middle childhood. Journal of Consulting Psychology 3:205-209. Rutter, M. (1967) A children’s behaviour questionnaire for completion by teachers: Preliminary findings. Journal of Child Psychology and Psychiatry 8:1-11. West, R.R. and Lowe, C.R. (1976) Regional variations in need for and provision and use of child health services in England and Wales. British Medical Journal 2:843- 846. World Health Organization (1976) Reports of the Task Force in High Risk Approach in Maternal and Child Care. Geneva. September 21-26. 86 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION Milton 1. Roemer and Gabriel Velasquez The sociomedical indicators of health developed and presented by Dr. Martini impressed the conference as having great significance for several reasons. Rates of mortality and even disability, especially in children, are clearly influenced by many socioeconomic and environ- mental factors, in addition to health care intervention. By contrast, the several indicators of patient feelings and behavior developed by Dr. Martini are probably more sensitive reflections of the impact of health services. In these days when, in some circles (e.g., Ivan Illich and his followers), the entire value of medical care is being questioned—on the basis of the minor changes in mortality rates associated with medical resources—the importance of these more sensitive reflections of the effects of medical care on the “quality of life” is especially great. A worldwide demand for greater emphasis on primary health care currently exists. Dr. Martini’s sociomedical indicators are particularly sensitive to the effects of primary care, which are seldom measurable in mortality or even clear-cut morbidity rates. A high proportion of patient complaints—some estimate as much as 50 percent—cannot be defined in terms of specific diagnoses. Yet changes in patient behavior or feelings following medical intervention can be docu- mented by this technique. The method of measuring behavioral changes is also remarkably simple and efficient. The patient fills out a questionnaire while he is waiting for the doctor, which requires no extra time from either of them. Any changes in behavior on a before-and-after basis can be readily determined. The exact question to be posed must be varied with the character- istics of the patient (age, sex, occupation, and so on) and the cultural settings. Thus the questions on behavior posed in England, where the technique has been developed, would differ from those to be posed in Latin America or in the United States (or in Seattle, Washington, where the method may soon be tested). Moreover, one must consider the differences in answers on behavior or feelings that might be given by a child, a parent, or a school teacher. This method of evaluation is not meant to replace “hard” measure- ments such as mortality and morbidity rates. It is intended to supplement them, especially with respect to the influence of health services on the quality of life. Such impacts may appear, at first glance, relevant only in the highly developed and industrialized Sociomedical Indicators 87 countries, but they are also relevant in the developing countries of Latin America. Many children (as well as adults) in such countries obviously have conditions that do not necessarily result in death or disability, but in behavioral difficulties which can be reduced by health service. With increasing experience, users of Dr. Martini’s questionnaires have been able to standardize the questions posed to children or adults. After repeated trial of certain questions, a range of answers has been accumulated which reflects the various levels of change in the patient's behavior, for better or worse. Similar standardization will be necessary in various national settings. Dr. Martini’s method, of course, assumes that the quality of life can have a positive value. It is important to determine if the patient who has survived a disease, by either prevention or treatment, is living and functioning in a more satisfactory and socially effective manner. This appraisal includes “peace of mind,” as well as physical capabili- ties. The technique, therefore, can detect in the patient changes that are not identified by the conventional physical examination. Its application, furthermore, can help to make the doctor more aware of the sociopsychological and behavioral objectives of medical care. In this sense, this approach can also be useful in medical education. ems SE me 8 APPLICABILITY OF THE TRACER METHOD TO CHILD HEALTH CARE David M. Kessner SUMMARY This report describes the results of a field study designed to assess the health status of a defined population of children representing a range of socioeconomic characteristics and to compare the care delivered to these children by different types of medical organiza- tions. A set of tracer conditions was selected, and process and outcome data were analyzed by pertinent demographic, social, and medical care variables. The major conclusions of the study are that prevalence rates in the community were extremely high for all tracer conditions; socioeco- nomic status of the children influenced the prevalence rates of some, but not all, of the tracer conditions; and prevalence rates for the tracer conditions did not vary significantly by provider organization when children of similar social status were compared. The limitations of this methodology and field study are discussed. General quality assurance methods are considered in light of their cost, utility, and potential impact on the health of patients. PURPOSE OF STUDY In 1969, the Board on Medicine, later to become the Institute of Medicine of the National Academy of Sciences, began a program to study and compare the health services received by contrasting social 89 90 EVALUATION OF CHILD HEALTH SERVICES and economic populations. The committee responsible for this pro- gram' chose to evaluate the null hypothesis that there are no differences in specific measures of health status in children of comparable social and economic status, regardless of the source of their usual ambulatory medical care. The findings discussed in this paper have been reported in a monograph entitled Assessment of Medical Care for Children—Contrasts in Health Status, Volume 3, published by the Institute of Medicine, National Academy of Sciences, 1974. A large interdisciplinary staff participated in the various phases of this program; without their efforts this report would not have been possible. Carolyn Snow and James Singer, my coauthors on volume 3, are responsible for much of the data discussed in this presentation. To explore the question raised by the committee, it was necessary to (1) define different organizations for the delivery of ambulatory health services; (2) develop a methodology for studying the quality of health services furnished by these different health providers; (3) identify populations that represented a spectrum of social and economic characteristics, and that used a variety of ambulatory care facilities, and (4) undertake a field program that would allow the evaluation methodology to be tested, the results to be subjected to critical review and, if warranted, the methodology to be replicated by other investigators on different populations. The National Academy of Sciences is a nonprofit, private organi- zation, founded under a congressional charter to act as science adviser to the executive and legislative branches of the United States Government. This study was supported by a consortium of private foundations and government agencies. 2 1 Health Services Research Panel: S.M. Nabrit, Chairman, The Southern Fellowship Fund; K.W. Deuschle, Mount Sinai School of Medicine; A. Donabedian, University of Michigan School of Public Health; L.J. Duhl, University of California, Berkeley; R. Fein, Harvard Medical School; C.L. Hudson, The Cleveland Clinic; R.C. Long, Louisville, Kentucky; D. Mechanic, University of Wisconsin; C. Phillips, Cook County Hospital, Chicago, Illinois; N. Piore, Columbia University; E.W. Saward, University of Rochester School of Medicine and Dentistry; J.B. Turner, Case Western Reserve University; A. Yarmolensky, University of Massachusetts; A.S. Yerby, Harvard School of Public Health. 2 Major support for this study was provided by the Carnegie Corporation of New York. The work was also supported by the Fannie E. Ripple Foundation; Association for the Aid of Crippled Children; John Hancock Life Insurance Company; Office of Health Evaluation, Deputy Assistant Secretary, Evaluation and Monitoring, Depart- ment of Health, Education, and Welfare (Contract No. HEW-05-70-130 and HEW- 05-72-167); and Office of Planning, Research, and Evaluation, Office of Economic Opportunity (Contract No. BIC-5243). Applicability of the Tracer Method ~~ 91 AREA OF CHILD HEALTH CARE The basic methodology centered on the use of specific indicator health problems called tracer conditions. The tracer concept was based on the premise that carefully selected diagnoses, problems, symptom complexes, drugs, or procedures, when combined in sets, can provide manageable data to examine both the process and outcome of medical care. These specific indicators enable health care evaluators to assess the health status of a defined community population, to evaluate the process and outcome of care delivered by different practice settings, and to begin to consider the interaction needed among a community, its people, and the providers of health servces in order to improve the health of children. Before this field study began, criteria were developed for selecting tracers. An initial set of tracers was designated, and explicit medical management criteria for the individual tracer conditions were speci- fied. In addition, the epidemiology and functional impact of each tracer condition were summarized. For this field study, a set of tracer conditions that included the following common pediatric problems was investigated: middle ear infection, conductive hearing loss, iron-deficiency anemia, and visual disorders. Table 1 shows that each of the five categories of medical care activities presented—prevention, screening or case finding, evaluation, management, and follow-up—is highlighted by at least two of the tracer conditions. Health counseling and proper treatment Table 1. Aspects of the Process of Primary Ambulatory Health Care Highlighted by Accepted Tracer Conditions Tracer Conditions Middle Hear- Vision Iron deficiency ear ing defects anemia Process Activities infection loss Prevention + + Screening + + 8 Evaluation History and physical exam + Laboratory + Other testing + Management Chemotherapy + + Health counseling + + Specialty referral + + + Hospitalization Followup + + + 92 EVALUATION OF CHILD HEALTH SERVICES of middle ear infections can be effective in the prevention of iron- deficiency anemia, and follow-up is often essential for acceptable care of children with middle ear infection, conductive hearing loss, and iron-deficiency anemia. To use these conditions as measures of the health status of children and as indicators of the way physicians practice medicine, it was necessary to specify diagnostic and therapeutic management criteria. The criteria could be used for measuring the prevalence of the condition in a defined population and as standards for assessing the performance of a medical care organization. These criteria repre- sented a minimal care management plan that a group of practicing family physicians and specialists felt would be useful guidelines for evaluating ambulatory health services in a population with these conditions. The criteria were not proposed as a rigid treatment formula applicable to individual patients. GENERAL METHODOLOGY A major goal of this study was to assess the ambulatory health services delivered by different types of medical care organizations. The study was intended to encompass persons of different income levels, define their usual source of ambulatory care (including those who did not identify with any specific provider), and compare— within and among different social classes—the performance of these organizations and the health status of the children who received care from these providers. These objectives required that the families and children participating in the study be representative of the commu- nity, rather than the particular patients of specific medical care organizations. Logistically, the study was divided into four separate but related data collection efforts. First, a probability sample of families residing in two areas of Washington, D.C., and a random sample of enrollees of a neighborhood health center (NHC) were selected. Interviewers gathered data from the mothers concerning health attitudes and orientations, medical care utilization, problems in the receipt of health care, social and economic position, social participation, and tracer-specific medical histories of their young children. Second, the children, ages 6 months through 11 years, whose families were included in the study, were evaluated clinically for specific tracer conditions—middle ear infection, conductive hearing loss, visual disorders, and iron-deficiency anemia. Historical, physical, and laboratory data were gathered independently to provide clinical measures of the children’s health status and to provide data for outcome analyses. Applicability of the Tracer Method ~~ 93 Third, providers identified in the household survey as the “usual source of care” for the children were surveyed. Data were collected on the professional background and activities of the physicians, the organizational structure of their practices, and selected aspects of their usual process of medical care. Fourth, medical records were abstracted to record the medical care delivered to a sample of the children for the tracer conditions and selected outcome measures of that care. SAMPLE DESIGN Community Sample The sample population was restricted to families with at least one child age 6 months through 11 years who resided in households in two contrasting socioeconomic neighborhoods in Washington, D.C. The Shepherd Park-Takoma area comprised 10 census tracts in the northwest part of the city that included a majority of middle- to upper-income families. The seven census tracts that constitute the Congress Heights area in the southeast part of the city are made up predominantly of lower-income families. To ensure that the final sample would include a sufficient number of children using different medical care organizations, the study population was selected in a two-staged sampling process. In the first stage, a random sample of all residential addresses in the two neighborhoods were surveyed to identify families with children ages 6 months through 11 years and to ascertain the children’s usual sources of care. In the second stage, families with children in this age range were sampled at different sampling ratios, depending on their usual source of care. All families whose children used the less common source of care—prepaid group practice—were sampled, while only half of the families who took their children to more common providers, such as solo physicians, were included in the study. Figure 1 shows the various sampling stages, the number of families and children included at each stage, and the final number of families and children that are the subjects of this study. Households outside the survey boundaries, nonresidential proper- ties, and families with no eligible children were excluded from the one-in-four sample of residential addresses. The overall completion rate for the screening survey—the first stage of the sample—was 91.0 percent. A total of 2,909 families had children in the designated age range whose usual source of care could be identified. An additional sample of 711 families, for whom no identification was available on the family composition by age, was included. 94 EVALUATION OF CHILD HEALTH SERVICES 3738 Total Number Residential Addresses 9332 One-in-Four Random Sample of Residential Addresses 3,620 Families with Children 6 Months to 11 Years of Age BR 1.987 Famitios > Sampled 3,203 Children 1,680 Eligible 3,007 Eligible Families Sampled Children 1,436 Eligible Families Interviewed Figure 1. Simplified sampling scheme for selection of the community sample, selected geographic areas in Washington, D.C., 1971. Of the total number of families, eligibility by children’s ages could not be determined for 711 families. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] All families who reported taking their children to a prepaid group practice or a fee-for-service group or reported having no particular source of regular care were included in the sample. They were sampled, therefore, at the rate of one in four of the total population. The families who reported taking their children to public clinics, private individual or partnership practices, and hospital outpatient departments or emergency rooms were subsampled at the rate of one in two, or one in eight from the total population. The final eligible and interviewed sample included 1,436 families and 2,780 children. Neighborhood Health Center (NHC) Sample Neither of the defined geographic areas sampled for the commu- nity population was served by an Office of Economic Opportunity NHC. To include an NHC among the provider organizations, a supplemental sample of families using the NHC located in the inner city Cardozo area of the District of Columbia was added to the study. Families with at least one child age 6 months through 11 years were selected randomly from the enrollment records of the Cardozo NHC. A total of 775 eligible families was identified. Systematic sampling of this group at a ratio of 2 in 3 resulted in a population of 516 families who were initially contacted. Of this group, approximately one fourth could not be located. An additional one eighth were located but could Applicability of the Tracer Method ~~ 95 not be contacted, had left the area, or were deceased. Eight percent refused to participate. The final eligible and cooperating population included 289 families—56 percent of the original sample. HOUSEHOLD SURVEY QUESTIONNAIRE The household questionnaire was designed to gather the following kinds of information: —Medical care utilization, costs, and satisfaction; —Barriers to receipt of medical care; —Demographic and socioeconomic characteristics; —Health knowledge and attitudes; —Perceived health status; —General attitudes and lifestyles; —Tracer-specific medical history (eligible children only). All family information was obtained from the mother or mother surrogate. In households where more than one family with eligible children resided, supplemental interviews were conducted with each mother. The completion rates for the household field work are presented in table 2. Nonresponse rates varied from 10.3 to 14.5 Table 2. Household Interview Completion Status for Shepherd Park- Takoma and Congress Heights Number Percent Completed interviews 1,436 89.32 (85.5)° Noninterviews Refusals 139 Not at home 34 173 10.82 (10.3) Subtotal 1,609 Indeterminate eligibility status Moved, could not be located 58 Other 13 71 4.9 Subtotal 1,680 Nonsample cases No eligible children 309 Moved outside the area 43 House vacant 12 No such address 5 369 18.0¢ Total 2,049¢ ® Based on total sample of 1,609 eligible respondents. " Based on total sample of 1,680 eligible respondents. © Based on total of 2,049 sample and nonsample cases. ? Includes 62 families added from households where more than one family with eligible children resided. 96 EVALUATION OF CHILD HEALTH SERVICES percent. In the NHC sample, 100 percent of the eligible respondents who consented to participate were interviewed. CLINICAL SURVEY Methodology All children ages 6 months through 11 years in the families interviewed were eligible for clinical examinations. Of the mothers interviewed, 90 percent gave permission for their children to be examined. Because age-specific estimated prevalence rates varied for each tracer condition, and because it is difficult to perform sophisti- cated sensory examinations on preschool children, different sets of tracers were chosen for the two age groups: Age Group Tracer Condition 6 months through 11 years Anemia Middle ear infection 4 months through 11 years Middle ear infection Hearing loss Visual disorders To control for seasonal effects on the incidence of middle ear infection, all clinical examinations were completed between January and April 1971—the 3-month period of highest incidence of the acute condition. Protocols for ear, nose, and throat examination; audiometric and vision tests; and hematocrit determinations were developed with the assistance of consultants. A comprehensive otolaryngologic examina- tion was performed by six board-certified or board-eligible otolaryn- gologists. Any abnormality of the ear examination other than occlu- sion of the auditory canal was considered a criterion for failure. Children ages 6 months through 3 years who failed the otologic examination, plus a 10-percent random sample of those who passed, had impedance testing performed. Children ages 4 through 11 with otologic failures had pure tone hearing threshold and impedance tests. Older children who passed the otologic examination were tested with a pure tone hearing screen. Those who failed this screening test, plus a 10-percent random sample of those who passed, were also given the hearing threshold and impedance tests as shown in figure 2. AGE 6 MONTHS AGE 4 THROUGH wn < FAIL 100% G1 [ENT MIDDLE-EAR IMPEDANCE ™ PASS < 10% S Y90% Oo & [WCRGHEMATOCAIT FAIL I 2. [MICROHEMATOCRIT|———— END y PASS END 1. [VISION SCREEN FAIL 100% DIAGNOSTIC OPHTHALMOLOGY PASS 0 10% & Y 90% > END FAIL100% ro [50 HEARING |'%%% [MIDDLE-EAR } FAIL100% THRESHOLD IMPEDANCE PASS 100% PASS |10% 90% L— {HEARING SCREEN > END Figure 2. Flow of children through clinical examination stations. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] POYIR 122B1], 91 jo Anpqesrddy L6 98 EVALUATION OF CHILD HEALTH SERVICES Clinical Criteria Dynamic impedance measurements of the middle ear provide an objective assessment of middle ear functioning and have shown high sensitivity to subtle changes in the integrity of the middle ear network. The fail criterion for this field study was a negative middle ear pressure equal to or greater than 100 mm H,O. Pure tone hearing tests (screening and threshold measurements) were carried out by the staff of the Children’s Hearing and Speech Center, Children’s Hospital National Medical Center. The pure tone hearing screening test was done at test frequencies of 250, 500, 1000, 2000, 4000, and 8000 Hz. The screening level chosen for the study was 15dB (ISO, 1964). The fail criterion was no response to two or more test tones in either ear. Pure tone air conduction thresholds were obtained for test tones from 125 through 8000 Hz, and bone conduction was assessed from 250 through 4000 Hz. Failure criteria for these tests included an air conduction threshold of 20dB (ISO, 1964) or greater in either ear at any test frequency and/or a 20dB air-bone gap at any frequency regardless of ear. The vision screening examination for acuity, organic abnormalities, ocular motility, and fusion was performed by second-year ophthalmic assistant students. Discrete pass-fail criteria were established by consultant ophthalmologists, and all children who failed the screening test, plus a 10-percent random sample of those who passed, were given diagnostic ophthalmologic examinations with cycloplegia by board-certified or board-eligible ophthalmologists. Fingertip blood specimens were obtained for the microhematocrit test. All specimens were processed by the hematology laboratory of Children’s Hospital. A hematocrit of less than 34 percent for those children younger than 19 months, or less than 35 percent for those children older than 19 months, was considered a failure. FOLLOW-UP CARE Part of our obligation to those who participated in the study was to provide treatment for children whose families had limited financial resources and no regular source of medical care. Nonsurgical care was provided for acute and chronic otolaryngologic problems. Pre- scriptions were given for glasses, and all children with motility problems were referred to the ophthalmology clinic at Children’s Hospital. The results of all tests were sent to parents and/or their children’s physician upon parent request. Applicability of the Tracer Method 99 PROVIDER SURVEY The final phase of this study focused on the physicians who were the usual or primary source of care for the community sample of children (those residing in the Shepherd Park-Takoma and Congress Heights areas) as well as those children enrolled at the Cardozo NHC. Information obtained from the providers included the follow- ing: —Characteristics of the physicians, including medical training, satis- faction with patient and practice situation, and attitudes toward health care organizational and financing issues; —Characteristics of the organizational setting of the practice includ- ing facility, number and types of personnel, accessibility, and charges; —Medical process data from the medical records of a sample of children who did and did not have a history or clinical evidence of tracer-specific pathology. The provider study was divided into two stages. In stage one, data were gathered about physicians and their organizations; in stage two, data were abstracted from medical records. Survey of Organizations and Physicians In stage one, data were collected with a self-administered question- naire sent to all providers who had been identified as the primary and/or usual source of care for the total sample of children. This information was supplemented by data from the physician master file maintained by the American Medical Association’s Physicians’ Records Service. Sample Criteria for “usual provider” were based on information given by the mother during the household survey. All practices mentioned as the usual source of care were contacted. If the usual provider were a physician practicing in partnership, all physicians in the practice were included in the provider study. For hospital outpatient departments or emergency rooms and public and private clinics, all staff physicians who provided care to children were included. The total sample for stage one included 178 provider organizations and 312 individual physicians. The organizations and physicians were categorized into six predominant medical care organizations as shown 100 EVALUATION OF CHILD HEALTH SERVICES Table 3. Distribution of Provider Organizations and Individual Physicians by Child’s Usual Source of Care, Washington, D.C., 1972. Neigh- Prepaid Assoc./ borhood Solo Hospital Public Group Private Health Practice OPD/ER Clinic Practice Clinic Center Total Provider organizations Number 110 12 16 3 36 1 178 Percent 61.8 6.7 9.0 1.7 20.2 0.6 100.0 Individual physicians Number 110 81 25 14 79 3 312 Percent 35.3 26.0 8.0 4.5 25.3 1.0 100.0 in table 3. Organization questionnaires were completed by approxi- mately 85 percent, and physician data were obtained from 80 percent of the providers. Survey of Medical Records Stage two consisted of abstracting the outpatient medical records of a sample of children. The chart review had three objectives: to validate the utilization data obtained from the household interview; to describe the records system used by different medical care organizations; and to assess the process and outcome of care given for the tracer conditions to a sample of the children. Sample The sample was limited to the medical records of children whose only source of care was a hospital outpatient department or emer- gency room, the prepaid group practice, or the NHC. A subsample was selected from all eligible children whose parents gave permission to have their medical records reviewed. All children who were identified as using more than one provider type as the usual source of care were excluded from the sample. The remaining eligible children were then stratified by the following variables: type of practice, area of residence, and presence or absence of tracer-specific pathology. The final sample included two thirds of the users of the NHC, two thirds of the hospital outpatient department and emer- gency room users, and all of the eligible children enrolled in the prepaid group practice. Overall, 77 percent of the sample medical records were located, ranging from 97 percent in the prepaid group practice records to approximately 60 percent of records in the hospital outpatient Applicability of the Tracer Method 101 departments. Of the located records, 92 percent contained informa- tion pertinent to the time preceding our clinical examination of the child. RESULTS Examples of results given herein are limited to the major commu- nity sample. The data for the sample of NHC users are not considered. Characteristics of the Families and Children Approximately 94 percent of the 1,436 families in the final community sample were black. The explanation for the very high proportion of blacks in the community sample (census data for these two areas reported an 85 percent black population in 1970) may be our requirement that each eligible family included in the study have at least one child between ages 6 months and 11 years. The mean age of the mothers (respondents) was 31 years, with 3 percent under age 20, 45 percent between ages 20 and 29, and 16 percent age 40 or older. One third of these families were headed by separated, divorced, widowed, or single females. A broad range of family income was represented by the community sample. Approximately one fourth of the families had annual incomes of less than $5,000, one fifth had annual incomes of $10,000 to $15,000, and almost 15 percent had family incomes of more than $15,000. Analyses of the respondents’ educational attainment and occupation were generally consistent with the family income data. The distribution of the source of care received by the 2,780 children from the community sample is presented in table 4. By individual types of provider organization, almost 30 percent of the Table 4. Distribution of Eligible Children by Usual Source of Medical Care: Community Sample, Selected Areas in Washington, D.C., 1971. Eligible Children Usual Source of Care (% distribution) Solo physician 29.5 Hospital OPD/ER 28.7 Public clinic 15.8 Prepaid group practice 6.3 Association, private clinic 10.4 No source 3.3 Multiple provider types 6.0 Total 100.0 Total number (all sources of care) 2,780 102 EVALUATION OF CHILD HEALTH SERVICES children received care from solo practitioners and an additional 10 percent from physicians in small associations or in fee-for-service clinics; only 6.3 percent were children enrolled in the prepaid group practice. Slightly less than 30 percent identified a hospital outpatient department or emergency room as their source of care, and 16 percent used public clinics. Finally, children with no identified source of care accounted for 3.3 percent of the population, and 6 percent of children received care from multiple types of organizations. Health Status and Tracers A primary objective of this program was to use specific health problems as a means of contrasting the health status of a diverse group of children, ages 6 months to 11 years. The health status of the children was assessed by independent clinical examinations carried out by a team of otolaryngologists, audiologists, and ophthal- mologists. All children received standardized tests to determine the presence or absence of the tracer health problems. The criteria for deciding whether a child had a particular condition were set prior to the study and were applied uniformly to all of the children. Selected examples of measures of the children’s health status and provider process and outcome data from the children in the commu- nity sample are presented in the following sections. Anemia Iron-deficiency anemia was one of two indicators of the health status of children ages 6 months through 3 years. Because normal hematocrit values vary with age, our criteria for determining anemia were based on age-specific hematocrit values. A child was categorized as anemic if the hematocrit level was less than 34 percent for children younger than 19 months, and less than 35 percent for children 19 months or older. Of the 505 children tested, 26.3 percent were classified as anemic. Most of the anemias detected were mild. Hematocrit values ranged from 26 to 42 percent in those younger than 19 months and from 28 to 49 percent in the older group. There was no statistically significant difference by age, sex, or color in the proportion judged anemic. There was, however, a strong association between anemia rates and the educational attainment of the child’s mother (figure 3). Prevalence rates varied from less than 19 percent in children whose mothers had education past the high school level to almost 33 percent among those whose mothers had 1 to 11 years of schooling. Analysis shows the relationship between the Applicability of the Tracer Method 103 Figure 3. Proportion of children ages 6 months through 3 years with anemia, by education of respondent: community sample, selected areas in Washing- ton, D.C., 1971. Total chi square (2 df) = 7.982, p<0.025; regresssion within chi square (1 df) = 7.611, p<0.01. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] mother’s education and the anemia fail rate to be inverse, linear, and statistically significant. The relationship between the prevalence of anemia and the children’s usual source of care is clearly influenced by the association of the mother’s education and anemia, and the known differences in the mother’s educational level among different provider organiza- tions. Thus, the crude anemia rates by usual source of care varied from just under 18 percent for those who regularly went to associations of physicians to almost 32 percent (figure 4) for children Figure 4. Proportion of children ages 6 months through 3 years with anemia, by usual sources of care: community sample, selected areas in Washington, D.C., 1971. Total chi square (5 df) = 3.945, p>0.5. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] 104 EVALUATION OF CHILD HEALTH SERVICES Table 5. Prevalence Rates of Anemia Standardized by Respondent's Educa- tional Attainment Within Provider Group: Community Sample, Selected Areas in Washington, D.C., 1971. Usual Source of Care No Assoc./ Prepaid Source/ Private Group Public Hospital Multiple Anemia Clinic Solo Practice Clinic OPD/ER Source Total number examined 48 137 53 121 97 48 Observed cases 8 33 12 33 30 13 Expected cases 11.3 31.6 12.3 34.3 26.9 12.5 Observed to expected ratio 0.71 1.05 0.98 0.96 1.11 1.04 Chi square (1 df)? 1.021 0.052 0.004 0.042 0.441 0.000 ® Total chi square (5 df)=1.965, p>0.5. using hospital outpatient departments and emergency rooms. How- ever, when an indirect standardization method was used to adjust the anemia rates for differences in the mother’s educational attainment, virtually all variations in the prevalence of anemia among the providers were eliminated (table 5). These data show that the ratio of observed-to-expected cases for each usual source of care approxi- mates that found when the education of the mothers within each provider organization is considered—that is, essential differences between the number of observed and expected cases do not exist. Medical process and outcome data were obtained by reviewing a sample of patient records. As shown in figure 5, of 169 children ages Figure 5. Disposition of a population of children ages 6 months through 3 years based on record review for anemia: neighborhood health center, hospital outpatient department, and prepaid group practice samples, Washington, D.C., 1972. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] Applicability of the Tracer Method 105 6 months through 3 years, slightly more than 70 percent were screened for anemia. Of those children with a hematocrit or hemoglo- bin test recorded at any time in their charts, 36.4 percent had anemia, based on the criteria used in this study. Almost two thirds of the screened children whose laboratory tests indicated they were anemic had not been diagnosed or treated for the problem. Only 16 of the 44 children identified as being anemic had been appropriately diagnosed and/or treated with iron. Because so few children were actually treated for anemia, no outcome assessments could be made on a provider-specific basis. Visual Disorders Uncorrected visual disorders constituted one of three health status measures used for children ages 4 to 11 years. A comprehensive vision screening test was given to 1,670 children. Pass-fail categories were established for each of the three components of the screening test—acuity, ocular motility, and organic disorders. Children who failed one or more of the three components of the screening test were considered to have failed a summary category. Because the component measures are not mutually exclusive, the sum of the proportions failing each component add to more than the total failing the summary criteria. According to the summary criteria, 26.3 percent had uncorrected or inadequately corrected visual disorders, 19.1 percent had poor acuity, 4.6 percent had ocular motility disorders, and 6.6 percent had organic diseases. The proportion of children failing the visual acuity and organic disease categories was related to age. There were no sex- related differences in fail rates, nor were differences in fail rates noted between black and white children. No consistent associations were found between pass-fail rates for the component or summary vision measures and a variety of social class indicators, including the respondent’s educational attainment. Figure 6 demonstrates clearly that there are no significant differ- ences in the proportion of children ages 4 to 11 who failed the vision screening summary examination by the usual source of their primary medical care. Indeed, the striking feature of these data is the narrow range of fail rates—from a low of 24.3 percent in the children using associations of physicians to a high of 28.9 percent of children using public clinics as their usual source of care. Children with no regular source or with multiple sources of care had a fail rate of 27.6 percent, only slightly higher than the overall fail rate of 26.6 percent. These data certainly do not indicate that the proportion of children failing this comprehensive vision screening test was related to whether or 106 EVALUATION OF CHILD HEALTH SERVICES w Figure 6. Proportion of children ages 4 to 11 failing vision screening test— summary criteria—by usual sources of care: community sample, selected areas in Washington, D.C., 1971. Total chi square (5 df) = 4.075, p>0.5. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] not they had a usual source of primary ambulatory care or to the nature of the organization ¢hat provided the care. The physician questionnaires revealed that high proportions of physicians from some medical care organizations claim to screen routinely for visual disorders—90 percent of physicians in prepaid group practices and 50 percent of those in hospital-based practices. However, review of 489 patient charts did not confirm the physicians’ claims; four {ifths of the children studied did not have a recorded visual exam. The proportion of children without vision screening information on their charts varied from 60 percent in children who received care from the prepaid group practice to 95 percent in children receiving care from the hospital-based facilities. Among the children screened, one third had an organic disorder or an abnormality in visual acuity or motility. Although we did not attempt to assess directly the adequacy of the specialty care for children with visual disorders, data from our clinical evaluations provide information on the adequacy of correction in children who wore glasses. Overall, 7.5 percent of all the children ages 4 to 11 years tested in our community sample wore glasses. As expected, the proportion varied by age and the education of the mother. The adequacy of and need for glasses were assessed by comparing acuity test results with and without correction for children who wore glasses. As shown in figure 7, the adequacy of correction was unrelated to the mother’s education. A striking 72.4 percent of children wearing glasses were not corrected appropriately or adequately—these children passed the Applicability of the Tracer Method 107 Figure 7. Proportion of children ages 4 to 11 with glasses whose correction was inappropriate or inadequate, by education of respondent: community sample selected areas in Washington, D.C., 1971. Total chi square (2 df) = 0.789, p>1; regression within chi square (1 df) = 0.719m p>0.1. [From Kessner et al. (1974). Reproduced with the permission of the National Academy of Sciences] test without their glasses or failed, both with and without glasses. Results similar to these have been reported by Safir and his colleagues (1973) in a study of children in New York City. LIMITATIONS OF THE TRACER TECHNIQUE There are no problem-free methods for evaluating the quality of health services. All approaches have limitations and are often plagued by methodologic and/or analytic weaknesses. In considering the broad types of data used, we know that the assessment of structure relies on the assumption that the quality of the facilities and the qualifications of the health professionals are related to the process and to the outcome of medical care—a doubtful assumption in most instances (Payne et al. 1976). Process evaluation, based on predetermined or judgmental criteria, is known to produce results that differ from outcome studies (Brook 1973). Indeed, in some instances, a poor process has been associated with a good medical outcome. These weaknesses of structure and process evalua- tion have led many individuals—researchers, administrators, and politicians—to encourage reliance on outcome studies alone. This advice makes sense because, ostensibly, outcome evaluation directly measures a person’s “health” or factors related to it. On closer scrutiny, however, it becomes clear that outcome studies often cannot be performed unless criteria for process evaluation are also established: First we must define the population with the problems whose outcome will be studied. Further, the intrinsic limitations of outcome evaluation, including proximate or short term 108 EVALUATION OF CHILD HEALTH SERVICES outcomes, are significant. Frequently, the prevalence of the outcome to be studied is very low; the outcome may be significantly influenced by variables that are beyond the control of the medical care system— genetic, social, cultural, and economic factors. In addition, there is often difficulty in obtaining outcome information without followup patient interviews and tests—Ilogistically complex and expensive pro- cedures to carry out. In a recent comprehensive study of the outcome method, Brook and his colleagues (1976) summarized the state of the art in outcome evaluation as follows: . . .there exists a paradoxical situation, in which policy demands that operational quality assurance systems use the outcome method to assess quality of care, while there is a dearth of valid and reliable outcome criteria and standards and no method of proven feasibility by which they can be applied. As there are serious limitations with the types of data used in evaluation studies, so there are with the general methodologic approaches. The tracer method, as initially described, was based on the premise that one can carefully examine selected medical care activities and their outcomes to gain insight into the quality of general medical care that is being received by patients. The results of this study (Kessner et al. 1974), plus the work of Brook (1973) and Payne and coworkers (1976), indicate that, with the exception of preventive services, one should be cautious in drawing conclusions about the quality of general medical care when studying the management of a set of tracer problems, diagnoses, drugs, and so on. On the other hand, in assessing the quality of medical care for children, one undoubtedly should address such areas as the type of preventive care provided; the content of the case-finding programs; the adequacy of diagnostic evaluations; the appropriateness of the management, including therapy and follow-up care; and the effect, or outcome, of these medical care processes on the well-being of the patients. In this initial work on developing the tracer method, a process was specified by which tracers should be selected. In an attempt to rationalize the decisionmaking process by which sets of tracers are selected, we established a hierarchy of criteria for use in tracer selection. The criteria, in order of importance, were as follows: —A tracer should have a significant functional impact; —It should be relatively well defined and easy to diagnose, in both field and practice settings; —Prevalence rates should be high enough to permit the collection of adequate data from a limited population sample; Applicability of the Tracer Method 109 —The natural history of the condition should vary with utilization and effectiveness of medical care; —The techniques of medical management of the condition should be well defined for at least one of the following processes: prevention, diagnosis, treatment, or rehabilitation; —The effect of nonmedical factors on the tracer should be under- stood. These original criteria have been reassessed; for many purposes they are unrealistic and impractical because few medical problems meet these conditions. Thus the scrutiny of the evaluators is limited. In another sense, the criteria are impractical because they ignore or rule out much of the routine disease that consumes so much of the physician’s time. In recasting the criteria, we have tried to focus on medicine as it is practiced—the patients who are seen and their presenting complaints. The modified criteria are: —The problem or diagnosis should be of significance in terms of either its potential functional impact on the patient, the burden on the provider, or its potential for treatment resulting in more harm than good; —The prevalence of the condition should be high enough so that analyses can be carried out on the study population; —The condition should be well defined and easy to diagnose; —There should be general consensus on the medical management of the condition for the specific therapeutic process under scrutiny. Although we have revised and simplified the criteria for selection, the use of sets of indicators rather than a single condition is still important in attempting to evaluate a health care delivery system. This statement is particularly true given the lack of concordance in the outcome among a set of tracers. However, in the past 3 years, four articles have come to my attention in which the “tracer technique,” or a similar approach, was used to evaluate health services (McBride and Ralph 1976, Campbell 1974, Smith 1974, Novick et al. 1976). In all these published reports, a single problem or diagnosis was “the tracer” used. Several articles indicate that a set of tracers had been selected, but data were reported on only a single condition. It is possible that inadequate funds or personnel forced the authors to limit their study to one condition. LIMITATIONS OF THE FIELD STUDY The preliminary prospectus for this study was developed between 1969 and 1970. It was clear then, as it is now, that the study results 110 EVALUATION OF CHILD HEALTH SERVICES could not definitively answer the question “Is the health status of a child related to the organization that provides the child’s usual medical care services?” Rather, this study was designed primarily to measure the health status of a selected population; assess differences in provider performance, organization, and attitude; and provide information about the strengths and weaknesses of the tracer method. Because these data may be used inappropriately to make narrow comparisons of particular populations of children or providers or to draw very broad conclusions about the delivery of health services, the limitations of the study must be clearly stated. The most crucial limitations are its location in one city; its predominantly black sample; its representation of families from the community and not patients of a defined group of physicians; and its failure to include all types of providers in the record review. With the exception of the sampling procedures employed, these limitations arose from cost and logistic considerations. These factors should point out the need for caution in applying these results to other cities and other populations. It is important to note that while we divide providers into groups such as solo practitioners, associations of physicians, and public clinics, we recognize that within each of these categories there is heterogeneity. Our analyses, therefore, focus on categories of provi- ders and not on individual organizations or practitioners within a single category. Further, we cannot know that a particular child used one provider exclusively. We separated the data on those children whom we could identify as using two different kinds of provider organization; but within a provider category, notably the hospital outpatient departments, some of the children very likely may have used more than one facility within that provider group. In addition to these factors related to the broad design of the study, there are limitations to the data that were collected. Scrupulous attention was given to calibrating and checking instruments during the clinical examinations to avoid erroneous findings; a special effort was made to gather follow-up data on a random sample of children who passed the screening tests so that false-positive and false-negative rates could be calculated; and an effort was made to standardize the findings of the physician observers. In several initial meetings, the terms to be used for anatomic descriptions were discussed, and several children were examined by all members of the ear, nose, and throat staff. Beyond this initial training session, we were unable, during the course of the clinical examinations, to effectively measure interobserver or intraobserver error. It is necessary to clarify differences in the way prevalence rates were compiled between the clinical field study and the chart review Applicability of the Tracer Method 111 analyses. During the clinical examination, we determined point prevalence rates: that is, the number of children with the condition at the time they were examined, relative to the total number of children examined. During the medical record review, however, the data we collected defined the number of children with any evidence in their records of the condition at any time in the past. We anticipated that these latter rates, which we termed variable-period prevalence rates, would be higher than the point prevalence rates because the children could have had one or more episodes of the disease during the longer period of time and because the providers had several oppor- tunities to detect diseases. In the strict sense, point prevalence rates and variable-period prevalence rates cannot be compared. They are useful, however, as indicators of the case-finding activities of the health care providers. If, for example, the variable-period prevalence rates are lower, when logically they should be much higher than the point prevalence rates, one can be confident that many cases have not been detected by the providers or have not been entered in the patients’ charts. Complex problems arise from ascribing certain untreated condi- tions to a child's current provider, with both highly mobile popula- tions and new provider organizations; however, this is also true with relatively stable patient and provider groups. For example, if a child has a conductive hearing loss and scarring of the tympanic mem- brane, indicating that the loss is associated with previous middle ear infection—perhaps an infection that was poorly managed 4 to 6 years earlier—it may be inappropriate to assign poor management of the early infection to the present provider. Further, it must be recognized that, in this study, we measured the health status of the children at only one point in time. Thus, we cannot ascribe directly to any provider a change in health status of children to whom he delivered care Finally, knowledge of the nature and severity of a child’s health problem could have induced parents to take their child to one particular type of provider rather than to another. Such self-selection factors could result in the presentation to different provider groups of biased samples of children—an unusually large or small number with serious problems. This indeed would be a concern if the results of this study demonstrated that, among the types of provider organizations, there were significant differences in the health status of children of comparable social and demographic characteristics. These are some of the specific limitations of this large community- based evaluation program carried out in the early 1970s. But what is the status of quality assessment in 1977 in North America? Current interest in evaluation and pressures to develop quality assurance 112 EVALUATION OF CHILD HEALTH SERVICES programs have increased exponentially during the past 5 years in the face of mounting data indicating that feasible, valid, and reliable methods to assess health services have yet to be developed. Clearly, we are far from being able to integrate an effective and efficient quality assurance mechanism into a health care system. To begin with, we constantly face the nagging question—What is quality? In a recent paper, David Rutstein and his colleagues (1976) defined quality as being concerned with the outcome of care. However, in common parlance, “quality” means degree of excellence and degree of conformation to standards. The issues in medicine are excellence in whose terms and conformation to whose standards. Quality is not a static concept; it is dynamic and it relates to the reasons for the assessment, the methods employed, what will be measured, and how it will be measured. At a more pragmatic level, we must cope with the unreliability of recorded diagnoses. Criteria must be applied to specify who has a streptococcal sore throat or a behavioral disorder. Having applied these criteria, we are obligated to evaluate a sample of those who do not meet the criteria as well as those who do, for in the former group a high proportion of inappropriate care may be found. One of the most difficult issues in assessing child health care from existing medical records is defining an episode of illness. Is it possible, retrospectively, to put a time frame around an episode of anemia, otitis media, or a urinary tract infection? Of course these delineations can be made with extensive protocols and a prospective design. However, we must be concerned with the cost of evaluation, and thus we must ask what can be accomplished with existing data and a pragmatic methodology. Current inpatient audits in the United States are estimated to cost $250 million annually (New England Journal of Medicine 1977). The cost of a simple evaluation based on encounter form data was approximately $2.50 per encounter (Kessner et al. 1975). Even if such an approach were applied to a small sample of all outpatient encounters, the cost for the 1 billion ambulatory care visits made annually in this country would exceed $250 million, and this figure does not include developmental costs. I should add that such a “bare- bones” process evaluation would not include many of the broader approaches now advocated for evaluation studies—total episode of illness and inclusion of lifestyle, social, psychologic, and environmen- tal factors. These approaches are pipe dreams. Even with funds and physician and patient compliance, we still would not have reliable tools to measure many important social, psychologic, and environ- mental factors. Given that we could develop and implement a valid, reliable, Applicability of the Tracer Method 113 simple, and inexpensive quality assurance system; and given that the data will be analyzed in a timely fashion and in a format that is meaningful to physicians and administrators; and given that health professionals will change their organization or clinical practices based on these data—will there be a measurable impact on the health status of our patients? There is very little evidence to suggest that there will (McNerney 1976). Clearly, this is not the time to expand existing quality assurance systems or to initiate complex and new programs. Rather, we must now carefully assess the impact of our present efforts on both the cost of health services and the well-being of our patients. Child health may be served better now by expanding carefully studied and efficacious pediatric preventive and case-finding activities than by prematurely joining the rush to establish complex quality assurance systems. REFERENCES Brook, R.H. (1973) Quality of Care Assessment: A Comparison of Five Methods of Peer Review. DHEW Publication No. (HRA) 74-3100. Rockville, Md.: Department of Health, Education, and Welfare. Brook, R.H., Avery, A.D., Greenfield, J., et al. (1976) Quality of Medical Care Assessment Using Outcome Measures: An Overview of the Method Prepared for the Health Resources Administration, Department of Health, Education and Welfare. R-2021 11-HEW. Santa Monica: Rand Corporation. Campbell, H.R. (1974) Do physicians satisfy their own diagnostic and treatment criteria in the care of patients with urinary infections. Journal of Family Practice 1:19-23. Kessner, D.M., Snow, C.K., and Singer, J. (1975) Assessing Ambulatory Care: The Use of an Enriched Encounter Form. (Prepared for Office of Health Evaluation, Deputy Assistant Secretary, Evaluation and Monitoring, Department of Health, Education, and Welfare Contract No. HEW-100-75-110). Washington, D.C. Kessner, D.M., Snow, C.K., and Singer, J. (1974) Contrasts in Health Status, Volume 3. In Assessment of Medical Care for Children. Washington, D.C.: Institute of Medicine, National Academy of Sciences McBride, T.C., and Ralph, J.R. (1976) Assessing quality of care in a university health service using the tracer method. Journal of the American College Health Association 24:150-153. McNerney, W.J. (1976) The quandary of quality assessment. New England Journal of Medicine 295:1505-1511. New England Journal of Medicine. (1977) Quality—can we measure it. Editorial. 296:170-171. Novick, L.F., Dickinson, K., Asnes, R., et al. (1976) Assessment of ambulatory care: Application of the tracer methodology. Medical Care 14:1-12. Payne, B.C. et al. (1976) The Quality of Medical Care: Evaluation and Improvement. Chicago: Hospital Research and Educational Trust. Rutstein, D., Berenberg, W., Chalmers, T.C., et al. (1976) Measuring the quality of medical care— A clinical method. New England Journal of Medicine 294:582-588. Safir, A., Kukikowski, C., and Deuschle, K. (1978) Delivering Vision Care in Grade School. Presented at the annual meeting of the American Medical Association, New York, June 23-27, 1973. Smith, S.R. 1974. Application of the tracer technique in studying quality of care. Journal of Family Practice 1:38-42. 114 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION Maurice Backett and Barbara Starfield Dr. Kessner’s paper described a quality assessment study with a very clear objective: to examine the adequacy of different types of medical care services. Thus, the study clearly fits into the category of effectiveness studies, which examine the relationships among struc- tural, process, and outcome variables. The study was population- based rather than clinical in that the sample was drawn from the community. The major limitation of the method was its very high cost, exceeding $1 million for evaluation of care for only a few clinical “tracer” conditions. Implementation of the findings was problematic. Although the study clearly involved quality assessment rather than quality assur- ance, the investigators (and especially the sponsoring and funding agencies) appeared to have an underlying, much larger expectation that defects found in the assessment would be acknowledged and that these would lead to changes, either within individual practices or through actions at higher levels (such as professional associations). Although this expectation was never stated explicitly, Dr. Kessner believes it existed in this study as it does in all similar studies. If such expectations are justifiable (there were many who felt they are not) and if researchers are to avoid extreme frustration, much greater effort to involve practitioners or practitioner organizations earlier in the planning of the study concerning their practices is required. In the discussion of this study, there was general agreement that the method had applicability in different areas and cross-culturally, although the specific tracers would have to be different. However, two cautionary notes were sounded: (1) Efficacy of medical care interventions for the particular conditions must be documented before a study is mounted, and (2) The tracer concept must not be misused. It is not justifiable to draw conclusions about the overall quality of care by examining the care for just one condition, as there is as yet no evidence that the presence of one defect (or one type of defect) in a practice is associated with the presence of others. As a result of his study, Dr. Kessner has concluded that current quality-assurance activities lack justification because of the unavaila- bility of a valid, reliable, sensitive, and inexpensive measure of quality which can be applied on a broad scale. Instead, he would give priority to efforts at prevention and case finding. Viewing his suggestion in another light, perhaps it might be concluded that quality-assurance activities are misguided because they fail to address the most Application of the Tracer Method 115 important areas—these include the inadequate recognition of prob- lems in the community and failure to assess the outcomes of care in individual patients. All quality-assurance approaches focus on documenting the validity of diagnoses and the prescription of appropriate therapy (whether judged by validated or unvalidated criteria, or by explicit or implicit judgments) for diagnoses. Quality-assurance organizations rarely even consider the possibility that practitioners have misdiagnosed, as most choose patients with certain diagnoses as the basis for review of quality. Moreover, reviewing the extent to which desired health goals are attained as a result of medical care intervention is given little attention. Perhaps arrangements by which success and failure in achieving expected goals in patient management on a day-to-day basis can be determined would do more to improve the quality of care than sporadic, retrospective studies which demonstrate current de- fects. 0 A SIMULATION MODEL FOR THE EVALUATION OF THE MATERNAL-CHILD CARE PROGRAMS IN COLOMBIA Rodolfo A. Heredia, Alejandro Vivas, Luis Hernando Ochoa INTRODUCTION This paper describes a preliminary simulation model for integral evaluation of Maternal-Child Care Programs (MCCPs) in Colombia. The model includes both maternal and child aspects, reflecting the priority given them in public health and development programs by the Government. Integral evaluation is achieved by incorporating the Maternal-Child Care Model (MCCM) into a more general model, known as SERES! (System for Study of Economic, Social, and Demographic Relations— Sistema para el Estudio de las Relaciones Economicas, Sociales y Demograficas). This approach, in addition to controlling aspects that belong exclusively to the MCCPs, allows for the incorporation of other health-related elements, such as economic factors, population structures, and government activities. The idea of using a simulation technique to develop an evaluation instrument for the MCCP arose during the presentation of SERES at the Maternal-Child Care Division of the Ministry of Health. At that time it was agreed to test a new methodology in the field, using an interdisciplinary team of pediatricians, gynecologists, public health specialists, sociologists, and economists from both a private institution (CCRP) and the Ministry of Health. The group endeavor was to formalize the MCCP so that it would eventually materialize into an MCCM proposal. The model was to include the most acceptable methods and techniques, given the restrictions of available data and data collection procedures. ! SERES is a simulation model developed at the Corporacion Centro Regional de Poblacion (CCRP) (Regional Population Center Corporation) for analyzing (through comparative studies) the implications for development of different policy alternatives. Appendix 1 contains a brief description of this model. 117 118 EVALUATION OF CHILD HEALTH SERVICES The team began its task by analyzing the SERES health submodel and by studying in detail the different components of the Govern- ment MCCP. Unfortunately, the ongoing study was interrupted by a medical strike that affected the country during the last quarter of 1976. Up to that time, a good deal of work had been carried out, and the discussions dealing with a theoretical MCCM (similar to the SERES health submodel) had started. When the strike occurred, the CCRP group continued to work by itself, so that the MCCM and the information presented here essentially correspond to the CCRP effort. It is hoped that meetings with the MCCP team will start again in the near future. In this way, the great advantages perceived and the results obtained through the interaction between the “technicians and administrators” can be used. THE EVALUATION PROCESS The need for evaluation of Government programs in Colombia does not seem to be debatable. On the contrary, technicians and administrators favor such programs as much as they strive to design better instruments for the purpose of evaluation. This concern comes from an awareness that—in countries with large and rapidly growing populations, great accumulated deficits, and inadequate social and economic structures—it is necessary to set priorities among programs and to rationalize by using the limited resources available in order to make substantial short-term improvements. Interest in the evaluation aspects has increased with the priority recently given to the health field (DNP 1969, 1970, 1975; Ministerio de Salud Publica 1967a,1973), but as it will be shown, although the country has adopted the evaluation component in the programs, it has not set a clear definition of evaluation, nor has it included appropriate instru- ments for its implementation. Difficulties concerning evaluation do not belong exclusively to underdeveloped countries. Turner (1976, p. 26) states in general: “This rapid growth (of program evaluation) inevitably has been accompanied by confusion about purposes, methods, organizational arrangements and, not least, terminology.” Roemer (1971, p. 839) states: “Because of the complexities of evaluating methods of health service organization, there has been a great deal of confusion in even deciding what should be evaluated, let along how to go about doing it.” These observations have led to a partial review of available literature. This review was intended to provide a reference frame- work that would permit the analysis of the content of evaluations carried out or planned in the MCCP field in Colombia and would Simulation Model 119 determine the possible role of the proposed model in the evaluation process (Donabedian 1966, Freeborn and Greenlick 1971, FPEI 1974, Roemer 1971, and Turner 1976). The review indicated that it is possible to design a framework based on three general evaluation levels corresponding to the different program development stages. The first level assesses the need for a program with respect to other programs such as health versus education. A second evaluation level refers to the available design alternatives for the same program—for instance, the different com- bination of instruments for health care delivery. The third level deals with the program’s implementation (figure 1). The first two levels are basically analytical, and from them the standards and expected values of the program are produced. Their particular features fuse them with the planning process. The last level reflects the actual develop- ment of the program being implemented. The evaluation is achieved through comparison of the different levels and not, as is usually the case, through a description of results or their relationships. Using the definition given by Reynolds (FPEI 1974, p. 4): “Evalua- tion is a process for making judgments about selected objects and events by comparing them with specified value standards for the purpose of deciding among alternative courses of action.” The comparision between the different levels can be expressed in terms of evaluation subjects, each corresponding to one of the analytical units into which a program can be divided.? Those most frequently mentioned in the literature are inputs, processes, results, effects, and impacts. Inputs refer to resources used to render services— such as administrative personnel, physicians, nurses, equipment, and so on. Processes refer to the basic structure and regulations of the service system—administrative regulations, document flow, and orga- nizational ranks are some examples. Results deal with the type and frequency of services rendered, such as X-rays, medical examinations, and so on. Effects and impact refer to the repercussions of results on other aspects within or outside the .health system—for instance, the patient morbidity rate or the population age structure. The evaluation is carried out through indices derived for each level at each evaluation subject category. It may be said that there are two types of index: those of effectiveness and those of efficiency. The former link up with the outcomes of a given evaluation subject; the latter, with the relationships between outcomes of different evaluation of subjects. Indices of the first type are measures such as quality of service, percentage of population well treated, number of medical 2 There are no uniform definitions for the levels and evaluation subjects used in evaluation literature. Those used here are based on Reynolds’ definition. EVALUATION SUBJECTS EVALUATION au 5 EVALUATION LEVELS INPUTS PROCESSES | Achievements | EFFECTS IMPACTS PROCESS Needs T A N D A R D || = | l 3. 2 jie. Ll ’ ! 2 > - Design T A N D A R D oS < < B. > = | i | S [<7] Program E << > Implementation | v ' : ¢ 0 \ M 2 w Standards and measures can be constructed to evaluate program effectiveness and efficiency in each case at three levels: quality, quantity and time. Source: CCRP, Ares Socioeconbmica, MODELO MATERNO-INFANTIL, March 1977. Figure 1. Levels and subjects of evaluation [From CCRP, Area Socioeconomica, Modelo Materno-Infantil, March 1977]2 0al SHOIAYAS HLTVAH dTIHD 40 NOILVNTIVAY Simulation Model 121 examinations by different category of services, and so forth. As for the second one, we could mention the relationship between services rendered and inputs used and impact and expenditures. These indicators are usually part of cost-benefit studies. HISTORY OF EVALUATION IN COLOMBIA In order to understand to what extent the model satisfies MCCP evaluation needs in Colombia, it seems appropriate, within the given reference framework, to present a brief critical review of the types of evaluation and indexes used in evaluations that have been proposed or planned so far in Colombia. During the past years, the need to evaluate programs has been recognized in the health system. The first step toward the introduc- tion of evaluation techniques was taken when health system planning was institutionalized in 1968. This event took place as a response to the commitments the country had made to the First Decennial Plan for the Americas (Ministerio de Salud Publica 1967a). The Colombian health plan for the 1968-77 period was based on the Survey of Human Resources for Health and Medical Education (Ministerio de Salud Publica 1967b) and on the National Inquiry on Morbidity (Ministerio de Salud Publica 1966). The plan showed the magnitude of morbidity risks and mortality levels of mother and child populations. In response to this situation, the mother-child group was created as part of the Ministry of Health in 1969. Later, as part of the development of the 1972 health plan, mother-child care was given first priority. In 1974, family planning was added. As a result, the Maternal-Child Care and Population Dynamics Division was established in the Ministry of Public Health. Although planning of a Colombian health system began in 1968, it was not until 1975 that the first formal evaluation of the MCCP activities was attempted (Ministerio de Salud Publica 1975a). The second evaluation was carried out in the middle of 1976 (Ministerio de Salud Publica 1976a). Additional evaluations cover more specific aspects of the MCCP: for example, the followup studies of family planning acceptors undertaken since 1974. Up to the present time, general evaluations carried out in Colombia by the MCCP have concentrated on one evaluation level, that of implementation. The treatment of MCCP’s impact aspects on health and other socioeconomic elements was made on a speculative basis only. From the standpoint of comparison standards or norms, the 1975 evaluation made explicit only those aspects relating to geo- graphic and population coverage. In 1976 the standards for coverage 122 EVALUATION OF CHILD HEALTH SERVICES estimates by auxiliary nurse and fieldworker activities were also clearly stated. Because of design changes in the Colombian National Health System (NHS) begun in 1973, the importance of a permanent evaluation process for the ministry’s programs (Ministerio de Salud Publica 1976b), as well as a systematic compilation of data required to ensure feasibility, became apparent. The plan proposed by the NHS seeks to evaluate the outcome of programs for some of the evaluation subjects previously mentioned. The resulting system, completed in 1976 but not yet implemented, proposes the generation of both qualitative and quantitative indica- tors. The qualitative indicators have been assigned different labels: precision, satisfaction, emphasis, extent, efficacy, and effectiveness of health services. The quantitative indicators are the following: physical and monetary resources, program activities, persons who received care, population and service coverage, and activities per person (concentration). Efficiency indicators proposed by the NHS are as follows: cost indicators (monetary resources by activities) and yield indicators (activities by resources). The evaluation system proposed by the NHS represents important progress with respect to the 1975 and 1976 evaluations, because it explicitly states the indicators for each selected evaluation subject. However, there are still imperfections when the system is compared with the previously mentioned reference framework. —The evaluation is limited to program implementation and does not deal with need or design aspects. —The comparison standards or norms are mentioned only for coverage and concentration indicators. —Indicators selected for evaluation fundamentally emphasize the effectiveness of program achievements. Therefore, evaluation of effectiveness and/or efficiency of other aspects, such as the pro- gram’s inputs, effects, and impacts, is neglected. —Finally, the use of such terms as “effectiveness,” “efficiency,” and “efficacy” as indicator classifications in the evaluation system leads to confusion and differs from the use generally accepted in the field of evaluation. ” THE MATERNAL-CHILD CARE MODEL Model-building is not foreign to the field of health, where mathe- matical simulation models have been developed. Generally, however, such models have been directed toward specific aspects of the system, Simulation Model 123 such as hospitals, outpatient visits, and so on (Edwards 1971, Lindau 1972, Feldstein 1967). They have rarely been aimed at the study of health within the more general structure of the health system of a country. ? The preliminary version of the MCCM is based on the structure of a more comprehensive health model (CCRP 1974a,b), some aspects of which have already been integrated into SERES (CCRP 1975,1974c). Within the context of the integral evaluation of the MCCP, the MCCM is incorporated in SERES as a programatic unit (figure 2). The model for maternal-child care has been developed in two general stages. In the first, the different classes of demand* and service requisites are calculated. In the second, the standards or norms of inputs, results, and effects are determined. FirsT STAGE—CALCULATION OF REAL DEMAND AND SERVICES REQUIREMENTS Estimation of Demand Levels Selection of risk groups within the target population. The demographic model (on a dynamic yearly basis) generates data on population composition by simple age groups, births, and sufficient data to estimate the numbers of married and pregnant women. The target population is selected within the model, bearing in mind government priorities. It is then formed into risk groups by age, sex, and urban- rural residence. Establishment of potential demand (number of cases in the country, by morbidity rates). Based on risk groups, cases (individuals) are estimated by type of disease, using the appropriate morbidity rates which could be seen as a result of an epidemiologic transition process and of preventive programs. In estimating demand, two points are worth emphasizing with respect to the determination of morbidity rates. The first is related to the epidemiologic transition (Omran 1971), and the second refers to preventive programs. 3 Model-building in Colombia should take into account research by institutions focusing on aspects of the health system: the National Health Institute of the Ministry of Health, the Colombian Family Welfare Institute, the Department of National Planning, the Antioquia Health Planning Program of the School of Public Health, the PRIMOPS (Programa de Investigacion en Modelos de Prestacion de Servicios de Salud) program of the Department of Valle, and COLIMPLAS (Programa de Investigacion en Planes de Salud OMS-OPS). 4 The term “demand” is used instead of “need” to avoid cross reference with one of the evaluation levels. ve Fertility a, Fertility and mortality DEMOGRAPHY p= Population Population (7-23) (all ages) Zz T ® Ww i EDUCATION Policies and resources GOVERNMENT Policies and resources HEALTH | — [ MATERNAL FAMILY PLANNING AND § RAM = % $ \ § g 3 8 8 8 | |E y Distribution of income Distribution of income ECONOMY Categories of labor supply | "Production coefficient Figure 2. Basic relationships in the SERES model [From CCRP, Area Socioeconomica, Modelo Materno-Infantil, March 1977] SADIAYAS HLTVAH dTIHD 40 NOILVNTIVAA Simulation Model 125 The epidemiologic transition permits understanding the observed relationship between incidence and prevalence for certain diseases and the level of development of a country (Puffer and Serrano 1973, p- 309; United Nations 1973, p. 518). The ability to understand this phenomenon is of great importance in determining the demand for services—especially if one considers the precarious life conditions of the population, particularly as they affect children. The need to integrate the model (with indicators not included in the MCCP or in the health system) is derived from the nature of the described phenomenon: The health status of the population not only depends on direct medical attention but on other factors, such as levels of education and nutrition, housing conditions, the existence of potable water and waste disposal systems, and so on. Under certain conditions these infrastructure elements can have a greater effect on the morbidity-mortality of a group than do health programs. Preventive programs may eventually accelerate changes in the morbidity structure so that a massive vaccination campaign can affect the number of infectious disease cases independently of the level of development of the country. Felt demand (number of cases that perceive the need for health services) and real demand. From potential demand estimates, it is now possible to go into felt and real demand, using fixed coefficients which include the patient's perception of his/her health status and the geographic location of the services. Note that the term “real demand” is being applied to individuals who are conscious of their status, have favorable attitudes toward public or private health services, and have geographic access to them. (The cases which will become the respon- sibility of the MCCP are estimated in the next step.) Estimation of Requirements Calculation of physical requirements to satisfy real demand. Based on information concerning real demand, it is possible to convert this demand into inputs by applying a matrix of inputs and activities needed to treat each group of illnesses by cause. This combination of resources is defined by the existing or available technology for the country at a given moment. Specification of the technological proc- esses is of particular importance to planners. It is here that the activities and inputs required to treat a patient (case) are defined. Alternate resource combinations, such as increased participation of paramedical personnel and systems specialization, fall into this cate- gory. (Each alternative technology which is studied generates differ- ent pressures on the health system and its resources.) 126 EVALUATION OF CHILD HEALTH SERVICES Adjustment of physical requirements. The following step determines whether the society has available inputs or can secure them during the process. In order to accomplish this, the limitations of the country in terms of human resources, equipment, hospitals, and so on should be made explicit. This estimate leads to the adjustment of physical resources needed to meet real demand. The availability of human, physical, and other resources is of special importance for the achievement of MCCP objectives. Many of these resources are not under the control of the health system, but depend on the development of other sectors in the country. The integration of the model with SERES permits the evaluator to verify explicitly whether or not society has the required program inputs. Calculation of requirements in monetary units. Next, the physical requirements that were calculated previously are expressed in mone- tary units, using information on average cost for inputs. In this way, the model generates information on the total cost incurred to provide total maternal-child care services. Part of this total cost will be covered by families and employers, and the Government will finance the rest. This situation implies calculating non-Government expenditures and subtracting them from the total cost in order to obtain the cost for the MCCP—that is, estimating the purchasing power of the popula- tion and its access to certain health subsystems (social security, health insurance, prepaid medical care, private services, and so on). Given the capacity of SERES to generate information on income distribu- tion, consumption patterns, and employment, it is ‘possible to incor- porate the effects of changes in these variables on the demand for public services. Nevertheless, the difficulties implicit in estimating demand functions under public expenditures are recognized. SECOND STAGE—ESTIMATION OF EFFECTIVE DEMAND, MORTALITY, AND Days oF INcapaciTY (TABLE 5) The second general development stage of the model consists of determining those cases which can be effectively treated by the MCCP (effective demand) and calculating the effects of the program on maternal-child mortality and disability. The main steps are: Determination of Effective Demand and Achievements Effective demand is the result of the comparison of monetary requirements (to meet the real demand) with resources available to the MCCP (which come from the Government). If the resources are greater than the requirements (in monetary terms), the level of effective demand will be determined by real demand. If the reverse Simulation Model 19% is the case, then effective demand will depend on the limitations imposed by the volume of resources, as well as on the priorities among MCCP subprograms which have been set by Government policy. Effective demand, in terms of cases and resources, must also be interpreted. In this instance, the previously calculated require- ments are adjusted according to the policies and economic capacity of the Government. Case Classification and Type of Care (attended, poorly attended, and not attended) In the second step, cases are classified into two groups, those who were not treated and those who received poor care. The second group is made up of three types: —~Cases resulting from the difference between potential demand and actual demand (interpreted as cases without access to the MCCP system); —Cases obtained by subtracting effective demand from real demand (interpreted as cases that lack MCCP coverage); —Cases poorly attended (calculated by applying a coefficient of the effectiveness of attention to the cases attended by the MCCP). Estimation of Mortality and Disability Once the two case groups are defined, appropriate mortality rates are applied to generate child-maternal mortality levels. A coefficient of illness duration by risk groups is also applied to the two groups. In this way, the total number of days absent from work or from school can be obtained. The most sensitive aspect of this stage is the calculation of the morbidity and mortality rates and the duration of illness, especially for the group of cases not treated. The natural rates are known for only a few diseases, and these have no known effective therapy. In order to obtain a natural rate, an experimental population (subject to disease but not receiving medical care) would be required. An assessment of cause of death and duration of illness would also be necessary. These conditions are not possible in real-life situations. In order to measure morbidity and mortality in well-treated cases, rates corrected by the effects of treatment should be applied. These coefficients will be obtained from institutions with “captive popula- tions” which are receiving adequate health care. It is understood that such coefficients are numerically less than those for the rest of the 128 EVALUATION OF CHILD HEALTH SERVICES cases; therefore, disability and death estimates would also be lowe1 among these groups. INDICATORS OF THE MATERNAL-CHILD CARE MODEL Through its different stages, the MCCP can produce a set of results that can be converted into evaluation indicators. These indicators are subject to limitations of the MCCM and SERES structures, and they are produced by the chosen design alternative. This alternative, in turn, is defined in accordance with Government goals and technolog- ical considerations. Measures derived in this way can be seen as “expected values” of the different elements relevant in the study of the MCCP. Within the proposed evaluation scheme, these measures correspond to standards produced at the needs and design levels and should be compared with the indicators produced during the implementation level. The indicators may be grouped according to the different steps into which the model has been divided, as follows: Indices at the Demand Step Most indices estimated at the demand step (figure 3) belong to the needs level (figure 1) and can be interpreted as effect and impact goals. Demographic indices. At a risk-group level, these indicators permit the study of size and changes in age groups of the MCCP target populations at the national level. Changes in such indicators are related to changes in the demographic behavior of the population (fertility, mortality, and migration), some of which are caused by the MCCP (maternal-child mortality, family planning) and others of 3 SERES was specifically oriented toward the study of design alternatives through the so-called control elements (shown in figures 3, 4, and 5 by means of small, directed rectangles). In general, control elements can be divided into two groups: policy controls, which are specified in order to study the alternatives in Government decisionmaking, and parameter controls, which must be specified to calibrate the dimensions of the model. The latter are necessary to the extent that constraints exist on our knowledge of the behavior within the MCCM or of its interrelationship with the rest of the system. A When calculating demand, the first type of control embodies the parameter named “Government definitions on maternal and child policies for the country.” The second type deals with morbidity rates, epidemiological transition, preventive services, and the attitude and accessibility coefficients. When assessing requirements, policy controls are those parameters which define the input-activities matrix. Availability of resources, average cost of inputs, and non-Government expenditures are elements of parameter control that adjust to the model. Fl Policy-control parameters Adjustment parameters DEMOGRAPHIC - AGE STRUCTURE AND BIRTHS Demographic indices for MCCP population Size Growth rate Government health pol > RISK GROUPS Health status indices for MCCP population POTENTIAL DEMAND Clinical morbidity (BIOLOGICAL DEMAND) Clinical need (medical criteria) Health status indices for MCCP population Feltmorbidity ~~ FELT DEMAND Felt need (population criteria) Index for the effect on the demand of the cultural component Educational index MCCP coverage index after structural constraints and G REAL DEMAND are taken in account Figure 3. Estimation of demand levels in the maternal-child care model [From CCRP, Area Socioeconomica, Modelo Materno-Infantil, March 1977] [PPO uoneuIS 6361 130 EVALUATION OF CHILD HEALTH SERVICES which occur outside the MCCP (changes in education level, employ- ment, life expectancy, and so on). Health status indices. At the potential or biological demand estimation step, these indicators show data on clinical morbidity or clinical needs. They refer to the incidence of different diseases sustained by the target population, regardless of desire for or availability of health services. Cultural indices. At the felt demand estimation step, these indicators show information on the felt morbidity and needs, that is, the different diseases experienced by the individuals as perceived by the population once their criteria are considered. Coverage indices. At the real demand level, these indicators show the number of cases to be attended by the national MCCP; the number of cases is calculated by taking into account structural and cultural restrictions as well as Government health policies. Indices at the Requirements Calculation Step Useful indicators for evaluation generated at this step of the model (figure 4) are complementary to those obtained during the estimate of different demands (need evaluation level [figure 1]). They can be grouped according to the measurement units employed (either physical or monetary). —Physical indices (obtained during the calculation of the total and adjusted requisites). Inputs and activities indices. These indicate the resources necessary to satisfy real demand in terms of the quantity of inputs (profes- sional and paraprofessional personnel and auxiliaries, equipment and buildings, drugs, diagnoses, and administration) and activities (specialized, general, and simplified consultations; and specialized and general hospitalization). Bottleneck indices. These identify the pressures on necessary resources to satisfy real demand and those inputs which need to be promoted in the country. —Monetary indices. Indices of cost of inputs and activities. These give an idea of the maximum cost that could be incurred by the maternal and child - demands in the country, given the resource constraints and the level of technology. Indices of cost for the MCCP. These are similar to the maximum cost indicator but are applied to the MCCP. (It is assumed that the Government is responsible for absorbing the value of the residual services, once the non-Government expenditures are deducted). Indices for inputs and activities (required to satisfy real demand) REAL DEMAND Policy-control parameters Adjustment parameters PHYSICAL REQUIREMENTS TO SATISFY REAL DEMAND Activities Indices of technological change Indices for input bottlenecks [ ADJUSTED PHYSICAL REQUIREMENTS Input production Monetary indices Indices for maximum cost for the MCCP after bottlenecks are taken into account / ADJUSTED REQUIREMENTS IN MONETARY UNITS Income distribution Indices for maximum cost for the MCCP after taking into account non-Government expenditure | REQUIREMENTS FOR THE MCCP IN MONETARY UNITS Figure 4. Estimation of Requirements in Materno-Infantil, March 1977] the Maternal-Child Care Model [From CCRP, Area Socioeconomica, Modelo [9POJ uOnEMWIS I¢1 REQUIREMENTS IN MONETARY UNITS PROPORTION OF REQUIREMENTS THAT CAN BE COVERED BY THE GOVERNMENT Indices of requirements Adjustment parameters Input Activities EFFECTIVE DEMAND Physical Indices of Achievements FOR MCCP i Deficit or surplus MCCP expenditure (CASES TREATED) for real demand Cases treated Coverage indices CASES WELL TREATED Effect Indices Death rates Perinatal CASES NOT TREATED Infant OR POORLY TREATED Proportional mortality ages 0-5 Maternal MORTALITY Effect Indices Disability rae ! DAYS OF Days of certified incapacity INCAPACITY Rates by cause of incapacity Figure 5. Estimation of Effective Demand for Services and of Mortality and Days of Incapacity of the MCCP Maternal- Child Care Model [From CCRP, Area Socioeconomica, Modelo Materno-Infantil, March 1977] SADIAYAS HLTVAH A'TIHO AO NOILVATIVATd g¢gl Simulation Model 133 Indices at the Steps of Estimation of Effective Demand, Mortality, and Days of Incapacity At these steps of the model (figure 5) information on expected results (inputs, activities, and effects) is generated at the design level (figure 1). These indicators are very closely related to the different measurements normally used to describe the operation of programs. Deficit and surplus indices. These are derived through the comparison of the monetary requirements of MCCP and Government expendi- tures on the program. They point out the need for additional Government efforts to cover the full cost of the MCCP programs and to modify its expenditure priorities for MCCP subprograms. They also indicate the possibility of improving the quality of services where surplus of economic resources exists. Treated cases indices. These indicators can be seen as “positive” health measurements. They result from a process that first calculates demands, then estimates different resources available and finally compares both elements. These indicators are specified according to the different risk groups considered in the model or to the type of care provided (well and poorly treated cases). Requirement indices. These are some of the most commonly used standards for evaluation. They refer to the quantity of inputs (human resources, equipment, and fixed capital) required to satisfy the treated cases. Coverage indices. It is possible to obtain these indicators both for the volume of inputs and activities and for treated cases. They are derived from data obtained during the preceding step and from the calculation of demands. Because the MCCM is incorporated into SERES, an integral interpretation of the indices is possible, since they take into account the demographic, economic, and social conditions of the country. Mortality indices. Changes in the mortality rates have been used as the most frequent “negative” index of program results. As the MCCM has been specified, the mortality indices take into account differential mortality rates according to health services received by the population, defined in terms of coverage and classification of cases. These indices can be separated into perinatal, infant, and maternal mortality rates. Disability indices. The cost of illness to society is observed through days of absence from work or school. The model generates informa- tion for this index by applying coefficients of duration of illness to the “target” groups. 134 EVALUATION OF CHILD HEALTH SERVICES FinaL CONSIDERATIONS For evaluation, the model structure has been specified so as to include the indicators derived from the review of the 1975-76 MCCP evaluations and from the proposal incorporated into the Colombian health system. As it was shown above, the model responds to the different evaluation levels and objectives proposed in the framework. Incorporation of the MCCM into a more general structure such as the SERES model allows for dynamic feedbacks of MCCP with demographic, economic, and social determinants, thus permitting derivation of additional indices at the impact level. Since it is usually not possible to integrate the MCCM into a general model, the elements that define the relationship between the MCCM and the rest of the system must be isolated and treated as exogenous conditions; this task can be accomplished by using information generated by other models or by experience. Formulation of the MCCM in term of three steps—demand estimation, calculation of requirements, and adjustment between demand and available resources—provides a general framework that can be applied for formalization of other topics at different levels of complexity. In addition to program evaluation, a model like MCCM can be used by planners to compare project design alternatives. For this purpose it is necessary to specify the values of the political, technolog- ical, and service parameters that define each of the alternatives. Each alternative will generate a different set of values for the indices of the model, and comparisons among design alternatives are achieved by contrasting these sets. The possibility of developing the model as. an educational tool becomes an important issue during the work of the interdisciplinary team. Participation of the members increases their knowledge and interest in MCCPs. It seems that the specification of a clearly attainable goal; the systematic pattern of thought imposed by formal- ization; the joint review of the program; the need to identify the relationships between MCCP elements (both internal and external); and the incorporation of demographic, economic, and monetary resources as well as other limitations—all lead to a better understand- ing of what constitutes the MCCP system, illustrate the importance of trade-off considerations among policy alternatives, and make explicit the role of time and its dynamic implications. It is hoped that additional users of this proposed MCCM will continue to benefit from group work reinforcing its use as an educational tool and showing the advantages of a simulation model as an evaluation instrument—characteristics that today are perceived only on a speculative basis. Simulation Model 135 Table 1. Projected Government Expenditures on Health According to Two Health Policy Designs®—Colombia (billions of pesos at constant prices of 1970) Base Alternative 1964 1.5 1.6 1975 6.3 6.8 1985 13.2 14.7 1995 27.8 31.7 2002 49.1 58.7 * See definitions in text. Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 AN EXAMPLE USING SERES HEALTH SUBMODEL As an example of the use of data generated. by a simulation model, some numerical results produced by the existing SERES Health Submodel (SHS) will be summarized and interpreted. The SHS is used here because of the similarities of its structure to that of the proposed MCM; therefore, this example only partly reflects the use of a model such as the MCCM. According to the criteria presented in this document the SHS indices may be taken as evaluation standards. They may help in the diagnosis of health needs, in the definition of priorities for the country, and in the statement of alternative program designs. The example is formulated in terms of two alternative policy designs. The base policy assumes that public health expenditure follows the historical pattern observed in Colombia during the last 15 years. The alternative policy assumes an increase in the trend of public health expenditures. The alternative was obtained by simulat- ing the increase of Government income that would follow a tax reform such as the one approved in Colombia in 1974. Table 1 shows Government expenses for health for each of the alternative policies. The example compares the indices that result from the two designs, and in this way it can give an idea of the described uses of the MCCM. The indices included in the example refer to the achievements, effects, and impact subjects of evaluation; all of them are effectiveness standards. (Efficiency indices could also be easily constructed by combining the elements in which effectiveness indicators are based.) 136 EVALUATION OF CHILD HEALTH SERVICES Table 2. Child Clinical Morbidity Index, for MCCP, According to Two Health Policy Designs—Colombia (thousands of cases) Ages 0-1 Ages 1-4 Ages 5-14 Base Alternative Base Alternative Base Alternative 1964 2,135 2,135 5,361 5,361 2,578 2,578 1975 2,278 2,278 5,637 5,637 3,530 3,680 1985 2,318 2,292 5,539 5,501 4,837 4,832 1995 2,110 2,041 4,970 4,836 6,355 6,273 2002 1,962 1,936 4,310 4,242 7,204 7,000 Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 Health Status Indices As indices of the population health status and of cultural determi- nants, the SHS presents the sizes of different types of demand: biological (clinical morbidity), felt, and real. Table 2 indicates the biological demand and clearly shows the convenience of working with cases rather than with persons, since one person can become several cases with the same sickness during a given year. From 1964 to 2000, the infant population under age 4 increases, but the biological demand does not.® Basically this trend appears because of changes in the country’s development and because of epidemiological transition. Thus, if the public health organization assumes that the need for services will increase in direct proportion to population growth, it may incorrectly direct its resources. Data also show that because of changing demographic structures and death rates, the country must be prepared to treat an ever-increasing number of cases between ages 5 and 14. It must be noted that the decrease (with respect to base policy) in biological-demand cases from 1985 on, when the Govern- ment augments its public health education and family planning budget (“alternative” column in table 2), is due as much to greater coverage of health programs as to a lowered birth rate brought about by fertility control programs (family planning and education). Indices of Achievement The achievement indices generated by the model are the treated cases and the physical and human resources they use. Treated children are divided by age into three groups (under 1, between ages 8 This result assumes a decline in fertility slower than the one found during the last intercensal period for Colombia (1964 to 1973). Simulation Model 137 Table 3. Child Health Achievement Index (Effective Demand), for MCCP, According to Two Health Policy Designs—Colombia (thousands of treated cases) Ages 0-1 Ages 1-4 Ages 5-14 Base Alternative Base Alternative Base Alternative 1964 1,309 1,309 2,089 2,289 238 315 1975 1,992 2,061 4,616 4,789 2,577 2,663 1985 2,249 2,283 5,161 5,418 4,068 4,229 1995 2,106 2,041 4,949 4,836 5,916 6,148 2002 1,962 1,936 4,310 4,242 7,204 7,000 Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 1 and 4, and between ages 5 and 14); these categories correspond to those used by the Mother-Child Division of the Colombian Health Ministry (table 3). Note that with the increase in public health expenses, the number of treated cases grows (relative to the base alternative) in the early period, but diminishes toward the end of the century. Considering this trend, the need to understand the above- mentioned relationships between health, demographic change, and the determinants of birth rate changes is vital. In the early part of the period the percentage increase in treated cases is lower than the increase in expenses (table 1), reflecting an increase in average costs due to the enlargement of health facilities and to changes in the proportons of infectious and degenerative diseases. In the model, physical and human resources are calculated from the number of cases treated and from a description of the amount of resources required per case treated. It is evident, then, that the measurement of resources per case is crucial in estimating the total resources utilized by the health system. The importance of preparing an accurate description of the observable structures in the country also becomes evident. It is possible in the model to simulate reality as well as to incorporate alternative descriptions of the quantity of resources needed per case in accordance with technical norms. These norms could be fixed by a user of the model, permitting the study of the effects of changes in resources required due to changes in health service technologies. Indices of Effect The two major effect indices produced by the model refer to death rates and the duration of diseases. The model specifies that well- treated cases present lower mortality rates than poorly treated or 138 EVALUATION OF CHILD HEALTH SERVICES Table 4. Mortality Child Program Effect Index, for MCCP, According to Two Health Policy Designs— Colombia (thousands of deaths) Ages 0-1 Ages 1-4 Ages 5-14 Base Alternative Base Alternative Base Alternative 1964 85.0 85.0 45.0 45.0 14.0 14.0 1975 78.3 7.2 42.0 41.4 15.0 14.9 1985 75.0 73.1 41.0 38.7 19.0 18.7 1995 66.8 64.0 34.9 33.9 23.2 22.5 2002 59.9 58.9 30.0 29.5 25.1 24.4 Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 untreated cases; thus it is expected that a larger health budget and more treated cases will decrease the number of deaths. In table 4, death data (classified into three groups) are presented, and the changes resulting in these groups when Government health funds increase are also shown. The differing behavior of deaths in the three age groups during the period 1964-2000 (that is the decrease in the first two and the increase in the last) is produced by fertility changes in the model and by the specification of the function about epidemiological transition. The number of deaths in the 0 to 1 age group decreases from 34 percent of total deaths in 1964 to 18 percent in the year 2002. However, the statistics of the Department of National Planning (DNP) show that this percentage was between 28 and 32 in 1968. The differences between the DNP statistics and the results shown in table 4 must be studied further, taking into consideration special note of the large underestimation of death rates, especially for infants, in official statistics. The number of days of disability shown in table 5 could provide a basis for estimating the effects of child morbidity on the available time of parents and on the performance of their children in school (for the 5 to 14 age group). Indices of Impact The quantity of health services and their distribution among population groups affect the structure, size, and rate of growth of the population through changes in mortality and fertility. Thus, figures such as those on the size of the child population and the total population which appear in table 6 are examples of impact indices of health services. However, the differences in the demographic effects of the base and alternative policies are caused both by changes in Simulation Model 159 Table 5. Disability Child Program Effect Index, for MCCP, According to Two Health Policy Designs—Colombia (millions of days) Ages 0-1 Ages 1-4 Ages 5-14 Base Alternative Base Alternative Base Alternative 1964 16.7 16.4 41.4 41.0 32.4 32.0 1975 16.6 16.3 37.6 37.1 38.1 37.9 1985 17.0 16.7 37.1 36.3 50.9 50.6 1995 16.0 15.6 33.5 32.4 66.2 64.6 2002 15.7 15.5 29.6 29.2 74.3 72.1 Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 public expenditures for health programs and by changes in tamily planning and education activities, which also have impacts upon the entire population. Therefore the measured impact is not exclusively due to health programs. It is possible to isolate the impact produced by additional health services, but this calculation would require additional experiments with the MCCM. We expect these to be carried out in the near future. APPENDIX In recent years the interrelationships between population growth and economic development have been a topic of lively discussion and controversy throughout the world. Analysis of the problem is difficult because of the complexity of the issue. Before attempting any analysis, it is necessary to develop a conceptual framework which describes the interrelationships between demographic and economic processes. In a developing country like Colombia, the magnitude of these relationships is affected by Government policies, particularly in the fields of education, health, and family planning. SERES is a computer simulation model which is designed to provide a convenient way of studying these issues in Colombia. Its purpose is to analyze, by means of alternative computer simulations, the effects of changes in Government policies (particularly in the areas of education, health, family planning, social overhead invest- ment, and taxes) upon the economic and demographic characteristics of Colombia. The fact that the model has been estimated with Colombian data imposes unavoidable limitations, because of the scarcity and unrealia- bility of historical information in a developing country. The model is divided into seven submodels: government, economy, education, health, family planning, migration, and demography. The model is designed to make each of the submodels as independent as possible. Table 6. Population Size Impact Index, for MCCP, According to Two Health Policy Designs—Colombia (thousands of persons) ori Ages 0-1 Ages 1-4 Ages 5-9 Ages 10-14 Base Alternative Base Alternative Base Alternative Base Alternative 1964 697 697 2,531 2.531 2,891 2,891 2,269 2,262 1975 875 875 3,216 3,216 3,536 3,536 3,047 3,047 1985 1,030 1,018 3,874 3,842 4,383 4,371 3,914 3,915 1995 1,101 1,061 4,291 4,204 5,110 5,019 4,748 4,709 2002 1,168 1,155 4,413 4,355 5,313 5,136 5,159 5,040 Sources: CCRP, Estudio de Salud, November 8, 1975 CCRP, Estudio de Reforma Tributaria, November 9, 1975 SADIAYAS HLTVAH ATIHD J0 NOLLVATVAL Simulation Model 141 As a result, there are very few interrelationships among the sub- models. The Government submodel, based on general policy objectives and on previous model output, determines the official economic policy for each year: revenue by category and expenses by program. The economic submodel computes economic activity using information on Government policies, demographic processes, education, and eco- nomic activity in previous years. The health, education, and family planning submodels compute the educational structure of the popu- lation and calculate the number of deaths and the births prevented. The demographic submodel (using the educational structure of women of fertile age, the number of births averted, and the number of deaths) computes the age and sex structure of the population after migration. A nontechnical description of the structure and uses of the model may be found in CCRP (1974). A complete technical description of the structure, data estimation, and validation of SERES may be found in CCRP (1975 a,b,c,d). SERES was developed at the CCRP in Bogota by a team consisting of Rodolfo Heredia, Manuel Ramirez, Bernardo Kugler, and Jaime Arias. This team was assisted by Alejandro Vivas, Alvaro Rodriguez, Franca Casazza, Edgar Baldion, and Isaias Roa. Programing was done by Alfonso Gonzalez and Alfredo Gonzalez with the help of Guil- lermo Rojas and Alvaro Lopez. SERES was developed under a contract with the Population Division of General Electric TEMPO in Washington, D.C. REFERENCES Arias, J., and Roa, I. (1976) Efectos en Salud de Alternativas en Gastos Piblicos y Privados y Migraciones Rural-Urbana. CCRP Area Socio-Econémica. Documento Técnico No. 16. Bogota, Colombia. Corporacion Centro Regional de Poblacion (CCRP) (1975) Modelo SERES. Descripcion Técnica. Submodelo de Salud. Documento Técnico No. 8. Bogota, Colombia. CCRP (1974a) Descripcion del Modelo de Salud. Documento Interno de Trabajo No. 055. Modelo SERES. Bogota, Colombia. CCRP (1974b) Desarrollo de un Modelo de Salud. Documento Interno de Trabajo No. 061. Modelo SERES. Bogota, Colombia. CCRP (1974c) Modelo SERES: Estructura y Usos. Monografia No. 3. Bogota, Colombia. Departamento Nacional de Planeacion (DNP). (1975) Para Cerrar la Brecha. Plan de Desarrollo Social, Economico y Regional 1975-1978. Bogota, Colombia. DNP (1970). Plan de Desarrollo Economico y Social 1970-1973. Bogota, Colombia. DNP (1969). Planes y Programas de Desarrollo 1969-1977. Bogota, Colombia. Departamento Administrativo Nacional de Estadistica. 1969. XII Censo Nacional de Poblacion, Julio de 1964. Bogota, Colombia. Donabedian, A. (1966). Evaluating the quality of medical care. Milbank Memorial Fund Quarterly 44: 186-218. 142 EVALUATION OF CHILD HEALTH SERVICES Edwards, S. (1971). Toward a Health Services Simulator. Second Interim Report. U.S. Public Health Service Grant No. HS-00314. Family Planning Evaluation Institute (FPEI) (1974) A Handbook for Family Planning Program Evaluation. Washington, D.C.: University Research Corporation. Feldstein, M.S. (1967) An aggregate planning of the health care sector. Medical Care 5:369-381. Freeborn, D.K., and Greenlick, M.R. (1973) Evaluation of the performance of ambulatory care systems: Research requirements and opportunities. Medical Care 11: Supplement, 68-75. : Heredia, R., and Baldion, E. 1975. Programas de Planificacion Familiar y sus Efectos Directos sobre el Desarrollo Colombiano. CCRP. Documento Técnico No. 17. Bogota, Colombia. Heredia, R., and Baldion, E. (in press) Participacion de Algunas Variables Relacionadas con la Fecundidad en Descenso para Colombia 1964-1973. CCRP. Bogota, Colombia. Lindau, M. 1972. Nursing care requirements for chronic emphysema patients. Hospital Progress. 53:74-77. Ministerio de Salud Publica. 1976a. Evaluacion del Programa Materno-Infantil de Colombia, enero 1974 - mayo 1976 y Revision del Convenio Tripartito. Bogota, Colombia. Ministerio de Salud Publica. 1976b. Subsistema de Informacion (MAC-1). (Edicion corregida) Bogota, Colombia. Ministerio de Salud Publica. 1976¢. Prestacion de los Servicios de Atencion Materno- Infantil. Bogota, Colombia. Ministerio de Salud Publica. 1975a. Evaluacion del Programa Materno-Infantil del Ministerio de Salud Publica. (febrero-marzo) Bogota, Colombia. Ministerio de Salud Publica. 1975b. Sistema Nacional de Salud Plan Nacional de Salud. Primera Fase. Nivel Local. (MAC-1). Documento Oficial (ISNS-11). (noviembre Bogota, Colombia. Ministerio de Salud Publica. 1973. Rediseno del Sistema Nacional de Salud. Bogota, Colombia. Ministerio de Salud Publica. 1967a. Plan Nacional de Salud 1968-1977. Bogota, Colombia. Ministerio de Salud Publica. 1967b. Estudio de Recursos Humanos para la Salud y la Educacion Médica en Colombia. ACFM. Imprenta INPES. Bogota, Colombia. Ministerio de Salud Publica. 1966. Investigacion Nacional de Morbilidad. ACFM. Imprenta INPES. Bogota, Colombia. Omran, A. (1971) The epidemiologic transition: A theory of the epidemiology of population change. Milbank Memorial Fund Quarterly. October, 509-538. Puffer, R., and Serrano, C. (1973) Caracteristicas de la Mortalidad en la Ninez. Organizacion Panamericana de la Salud. Washington, D.C. Reyes, A. (1975) Efectos de la Migracion Rural-Urbana sobre el Desarrollo Economico y Demografico. CCRP. Area Socio-Economica. Documento Técnico No. 15. Bogota, Colombia. Robertson, R., and Zschock, D. (1967a) Conceptual Framework and Guidelines for the Analysis of Health Services Financing. Draft report (mimeo). Robertson, R., Zschock, D., and Daly, J. (1976b) Health Sector Financing in Latin America: Conceptual Framework and Case Studies. Office of International Health, Department of Health, Education, and Welfare. Washington, D.C. (mimeo). Roemer, M.IL. (1971) Evaluation of Health Service Program and Levels of Measure- ment. HSMHA Health Reports 86:839-848. Turner, H.D. (1976) Principles and methods of program evaluation. Focus: Technical Cooperation 3:26-30. United Nations (1973) Determinants and Consequences of Population Trends. United Nations, N.Y. USAID—Colombia (1974) 1974 Colombian Health Sector Analysis. Bogota, Colombia: USAID. Vivas, A. (1976) Implicaciones Demograficas y Economicas de Programas de Nutricion del Gobierno en Colombia. CCRP. Area Socio-Economica. Bogota, Colombia. Simulation Model 143 DISCUSSION David Bersh and Carlos V. Serrano By means of a simulation model, the Regional Population Center Corporation, in collaboration with the Ministry of Health and other institutions in Bogota, Colombia, is attempting to develop evaluation procedures to measure the impact and effectiveness of a comprehen- sive maternal-infant health program at the national level, as well as to measure the limiting role of other factors on the health sector. This method of evaluation is distinct from the others presented and discussed at this meeting in that it constitutes a method of long- range prediction and projection for national health planning. Perhaps its principal value lies in its imposition of a formal structure on the system: Key elements are identified and their interrelationships recognized and evaluated. Unlike other more restricted models and methods of evaluation, it takes a comprehensive approach which considers the system being evaluated in its totality, not as an isolated element. The groups considered the method most applicable to policy planning and formulation, given its ability to analyze and project the course of events. They stressed the program’s value as an educational tool for the project staff. They agreed as well that the method has more applications as a predictive tool. One possible application is to start from a demographic structure so that predictions can be made about which population groups will be at high risk. Potential demand can be calculated for the at-risk groups; perceived demand can be determined by the morbidity levels which are encountered or calcu- lated using potential demand as a base. The response of the population to perceived demand will define real demand, and the options for improving coverage and prevention will be multiple if the required data are obtained. Several major problem areas were identified. First, the method involves many elements, and data have to be obtained from multiple sources of varying quality. Many gaps undoubtedly will be encoun- tered, since data for independent variables will be missing in many instances. The data will be soft, and calculations will depend heavily on estimates and approximations, and at times on old data (for example, the 1965 National Morbidity Survey in Colombia). The successful operation of the model requires that all the data be obtained and that there be no major changes in national health status or health policy. All these problems are common to simulation models, which originated and were successfully applied in fields such as industry, where data are hard, soft, or nonexistent, and where conditions are unforeseen. 144 EVALUATION OF CHILD HEALTH SERVICES Secondly, it was pointed out that highly trained professionals are needed to adequately manipulate such a system or model and that, at a programmatic level, personnel are needed who can understand and incorporate the results into health policy and programs. This type of human resource is not always available in less-developed countries. The application of a single-faceted model to a country where conditions are heterogeneous represents a problem in itself. Finally, the most important problem foreseen in applying the model is that, because of its characteristics, it may elicit considerable resistance on the part of decisionmakers and policy formulators, since such people frequently prefer simpler, more direct, and more rapid evaluation mechanisms. Similarly, mathematical simulation models have not been received with the desired enthusiasm in political circles. 10 EVALUATION OF CHILD CARE IN PRIMOPS: A COMPREHENSIVE HEALTH PROGRAM IN CAL], COLOMBIA Presented by Gildardo Agudelo Gil! PRIMOPS? is a demonstration program that is a coordinated effort of the Ministry of Health of Colombia, the Health Department of the city of Cali, and Universidad del Valle. The city Health Department has overall responsibility for services and program operation, and the University performs a technical assistance role. A previous demon- stration in Candelaria, a semirural area, served as the basic model. PRIMOPS has the following goals: 1. To demonstrate an innovative system for the provision of efficient, effective services which can be adapted and replicated at the regional and national levels. To provide such services, the system must fulfill the following requirements: a. Low cost b. Achievement of a high degree of coverage of the population c. Reassignment of an ever-increasing number of functions to nonprofessional and auxiliary personnel; 2. To improve health conditions in the demonstration area, signif- icantly reducing current rates of mortality, morbidity, and fertility, especially in the most vulnerable group of the popula- tion; 3. To prepare, train, and utilize professional and nonprofessional personnel capable of mare realistically facing the health prob- lems of the area; 4. To carry out studies and develop operational methodologies which permit the evaluation of the efficiency of the system and ! As the paper presented at the conference had many tables and an extensive text, a summarized version has been prepared for this volume by the editor, in cooperation with the author. % Spanish acronym for “Research Program in Models for Providing Health Services.” 145 146 EVALUATION OF CHILD HEALTH SERVICES its impact on the health of the community and which offer data for the determination of alternatives to the solution of health problems. Characteristics of the Target Population The target area is a sector of the city of Cali, which has a total population of approximately 60,000 living in five “barrios.” The area and living conditions of the inhabitants are considered characteristic of marginal urban areas of the region. In 1973, 43 percent of the families had a monthly income that varied between $22 and $44; unemployment was approximately 18 percent; and illiteracy was 30 percent. The main target groups in the program are women of childbearing age and children under 5, who were considered the most vulnerable. It was found that 38 percent of the children in the target area suffered from malnutrition. Service Principles and Objectives In order to achieve the program goals, the service model was designed with the following operational principles: 1. Coverage of more than 80 percent of the population, particu- larly of mothers and children; 2. Ability to function within a system of urban regionalization with a high degree of decentralization; 3. Satisfaction of the following requirements: a. Maintenance of permanent contact with the family unit through home visits; b. Fulfillment of real and felt needs of the community; c. Maintenance of both centrifugal and centripetal activity; in other words, provision of care to patients visiting health agencies as well as to people in their own homes; d. Maximum use of resources in each community, with com- munity participation in the planning and solution of its problems; e. Use of professional personnel (physicians, nurses, and den- tists) with maximum efficiency, giving them more important responsibilities and more supervisory functions; f. Optimal use of auxiliary personnel through the reassignment of functions while redefining the role of such personnel in health programs; g. Ease of replication in other communities; Evaluation in PRIMOPS 147 h. Limitation of costs to the budgetary allowance of the health centers; i. Periodic evaluation of the system and its impact on the community. The following service objectives were established: 1. For the mother a. Decrease in maternal mortality; b. Decrease in frequency and severity of: — Obstetrical hemorrhage, including abortion — Toxemia during pregnancy — Obstetrical (amniotic) and puerperal infection 2. For the child Decrease in infantile mortality; Decrease in perinatal mortality; Decrease in mortality in preschool children (ages 1 to 4); Decrease in frequency and severity of; — Eye infection and umbilical infection in newborn babies — Prematurity — Diarrhea in children under age 5 — Malnutrition in children under age 5 3. For the couple a. Education and motivation in responsible parenthood; b. Provision and supervision of contraceptive techniques; c. Treatment of infertility; 4. For the community—education and motivation in the utilization of services offered. ao ow Evaluation Activities The external evaluation of the program is carried out through the implementation of the following six research studies and the devel- opment of area profiles for both the target area and a control group area, with international technical assistance coordinated by Tulane University. 1. In the first study, a control area, similar insofar as possible to the demonstration area, was selected in Cali; and sampling techniques and cross section samples to be used in comparing the two areas were established. The control area was selected on the basis of geographic characteristics and income, racial composition, and number of inhab- itants. The samples will provide data for comparison of the target and control barrios in 1975 (before the PRIMOPS intervention) and again in 1977 (after). The community profiles will provide detailed 148 EVALUATION OF CHILD HEALTH SERVICES descriptions of the target and control areas and will identify favorable community conditions worthy of being stressed in the replication phase of the program. Background information gathered in these studies will be helpful in understanding the reasons for possible unexpected program results. 2. In the second study, the degree of the community’s acceptance or rejection of the proffered health services is being measured, and knowledge, attitudes, and practices (KAP) are being assessed over- time. “Felt” morbidity and health service utilization are also being surveyed. Overall, the study attempts to measure the degree of consumer satisfaction with the health services offered by PRIMOPS or by the traditional system. This will be a crucial element in deciding the merits of replication of PRIMOPS. 3. The third study measures, over time, changes in morbidity, mortality, and fertility in the target and control areas. It is believed that this study will eventually validate the assumption that improving the efficiency of the official public health services in poor communi- ties does indeed result in improved health and survival as well as in decreased fertility. This is a worthwhile effort, since few studies have documented the real impact of health services on health status. Naturally, it takes into account concurrent improvements in water supply, sanitation, and housing conditions included in the area profiles, which complete the related environmental conditions neces- sary to establish cause-effect relationships. 4. The fourth study assesses the adherence of service personnel to PRIMOPS norms and the quality of such services which, although heretofore performed by physicians and nurses, have been delegated to others under PRIMOPS. One of the purposes of this study is to counter probable attacks on the quality of delegated health services. 5. The fifth study measures or defines the roles and role structure established within PRIMOPS and the acceptance of PRIMOPS and of their own roles by the personnel of the project. It also analyzes interpersonal relationships within the project. This study will provide useful information for detecting personnel problems and offers relatively rapid feedback to management for corrective modifications. The definition of role structure and of interpersonal relationships will provide important information for possible changes to improve the efficiency of PRIMOPS in its replication elsewhere. 6. The last study is aimed at the establishment of reliable cost- analysis procedures, allowing the measurement of cost-effectiveness and cost efficiency. Among the requirements to be met by PRIMOPS, the provision of low-cost health services is a most important one. Existing cost-accounting procedures do not allow for the detailed analysis of cost data. It will be necessary to design better ways of Evaluation in PRIMOPS 149 allocating costs (simultaneous equations, cost centers, and so on), and consequently to be able to determine service costs by subprograms as well as by operational and development costs. Otherwise it will be very difficult to establish the feasibility of the replication phase. The PRIMOPS services have been financed entirely by the Colom- bian Ministry of Health, based on its policy of strong support of regionalized services, decentralized administration, use of paramedi- cal personnel, priority for maternal and child health, and extension of services to the poor. 150 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION David Bersh and Carlos V. Serrano Before discussing evaluation, it is important to indicate that the Research Program in Health Services Delivery Models (PRIMOPS) is not exclusively an evaluation model. Rather, it has a more general focus, of which evaluation is only one part. The primary objective of PRIMOPS, a joint effort of the Universidad del Valle and the Ministry of Health, is to design, implement, and evaluate a model for comprehensive health services that can be replicated in other com- munities. The program, which until now has emphasized maternal and infant health, attempts to expand low-cost coverage, to deliver all levels of health care on a regional basis, and to foster maximum, active community participation. In the course of the presentation and discussions, it became very clear that PRIMOPS is not in itself a pilot project, but a program of experimentation and demonstration of normative models and meth- ods of evaluation. The program is expensive to operate, but within the context of a comprehensive system, the experience and outcome can be worthwhile. In the case of Colombia, the National Health System will adopt, refine, and replicate the most successful program elements on a broad national level. In other words, under the proper circumstances, the program and its methods need not be reproduced in their entirety to have a wide impact. In the course of the discussions, several limitations, aside from cost, were identified. PRIMOPS is, for example, an urban program oriented toward a poor population. The modifications needed for its application to other types of urban populations or to rural popula- tions must be identified. Also, the number of variables studied is so large that their satisfactory control appears difficult. As a result, large quantities of data must be gathered; this situation in turn causes data collection problems. Additionally, there is a possible disparity in the quality of the data for the study group and those for the control group. For example, the greater vigilance and meticulousness exer- cised in collecting morbidity data for the study group may mean that relative morbidity for this group appears higher than it actually is. Control group data also present certain problems in comparability. Although it is not easy to maintain equilibrium between the scientific and technical aspects of the program and its evaluation component, the political considerations and community expectations, this goal should be pursued. It was felt that without this balance, the occurrence of disturbing situations was possible. For instance, if the Government were interested in quick results and did not allow Evaluation in PRIMOPS 151 sufficient time for evaluation, the academic world would want to so amplify the theoretical implications of the program outcomes that results would be extremely difficult to implement because of their complexity or untimeliness. The program evaluation model was designed with solid technical assistance and was carefully conceived and tested. Although the difficulty in obtaining the required data and the desirability of having appropriate preexisting sources of information was noted, it was felt that the program could be replicated in other cultures. Also, despite the fact that the proposed methods are expensive, it was recognized that after several applications, it would be possible to select the most useful indicators and procedures and so reduce the cost to approxi- mately 5 percent of the total budget. The program as a whole already occupies a recognized place in the national and international arenas. It has begun to produce results that demonstrate that the benefits will significantly exceed expectations and will justify whatever initial investment was required. ape m— 11 GOAL CONFLICTS IN A CHILD HEALTH STATION: A PROBLEM IN ACTION RESEARCH Jane Z. Moss, Renate Belville, Bernard Levenson Introduction The objective of the research described here was to make recom- mendations for the solution of an organizational problem in a child health project. The problem was the unpredictable fluctuations in demand that affected the ability of the staff to perform according to the goals of the project. This issue was one in a series of organiza- tional problems with which the authors worked during a period of “concurrent evaluation” (Moss 1970) in the Wagner Child Health Project. Concurrent evaluation is based on the principles of policy research. The distinction and justification for policy research has been clearly stated by Coleman (1972, p. 3) as follows: ... policy research bridges two worlds with very different properties: the world of the academic discipline and the world of policy and action. The defining characteristics of policy research are two: the research problem originates outside the discipline, in the world of action; and the research results are destined for the world of action, outside the discipline. The special properties of policy research stem from the different properties of the disciplinary world and the world of action, and from the translation problems involved in moving between these two worlds. Using the methods and concepts of social science, concurrent evaluation conducts policy research within an organizational setting. The social scientists translate an organizational problem into a research problem. Findings from the analysis of the research problem are presented to the executive staff with recommendations for change. The executive staff can only accept recommendations which it considers realistic and an improvement to the organization. Conse- quently, concurrent evaluation must be cognizant of the options and 153 154 EVALUATION OF CHILD HEALTH SERVICES constraints of the organizational staff and setting. The knowledge from the academic discipline is used to inform the world of action about the nature of the issues and the realistic options available. Once implemented, concurrent evaluation has a built-in before- and-after research design. Data generated from the implemented decision can be compared with data used as a basis for the recommen- dation.! The process can be repeated as the implemented decision is modified. Policy research must also be cognizant of the social organization of the staff when recommendations are made. The staff that implements change must integrate that change into its role configurations. This change usually requires consultation from the research team. The concurrent evaluation approach has evolved into a series of steps for the research process within the organization: —Assessing the organizational structure or a substructure with respect to the goals of the organization, including redefinition or specification of goals where they are inappropriate or vague; —Setting priorities for the organizational problems located in the assessment; —Designing and implementing research questions based on the problems that have been ranked; —Analyzing research findings and putting the implications of such findings in the form of recommendations for those persons with authority to accept or reject them; —Counseling the staff in the implementation of accepted recommen- dations; — Reassessing the organizational structure; —Reviewing the next organizational problem appropriate for re- search. This approach emphasizes the integrative process involved in the application of research findings and the use of exploratory statistics? with the most appropriately available or easily collectible data. It assumes that any service is a dynamic process and not a static operation. As a process, changes introduced into one part of the organization produce a ripple effect in other parts of the organiza- tion. Until these effects have been integrated into the service and the ! Campbell states: “... thus economists, operations researchers and mathematical decision theorists trustingly extrapolate from past science and conjecture, but in general fail to use the implemented decisions to correct or expand that knowledge (1971). 2 The approach to exploratory statistics is explicated in the work of John Tukey (1970). Goal Conflicts in a Child Health Station 155 policy outcome determined, continuous monitoring for reassessment and realignment is required. Consultation with the responsible staff is continuous and detailed. Resolution of problems according to a priority of organizational needs leads to a sequential series of research questions, explorations, analyses, recommendations, and consultation—before, during, and after implementation. The problem reported in this paper is one research question in such a series. The staff of the child health station had already resolved a number of structural problems with new role definitions, new functions identified within the child health station, and new designs for data collection. At this point, the staff members were struggling to keep up with their own work plan. The unpredict- able fluctuations in demand for the services of the station® affected their ability to perform according to the goals they had set for themselves in the project. Defining the Project The goal of the experimental project in the Wagner Child Health Station was to treat the child in the context of family and neighbor- hood.? This project had three objectives: —To provide curative and preventive care to all children eligible to visit the station, by age and residence in the health area; —To educate and train parents in the health care of their children; —To contain costs after the developmental phase of the project to make replication in existing health agencies feasible. The codirectors of the special project at Wagner were a pediatrician and a nurse-clinician who had selected the Wagner station because it was a deprived area with no health services in the immediate vicinity. Confronted with the variety of problems for the publication and a working knowledge of the strengths and limitations of their own professional traditions, the codirectors realized the need for a new approach between the providers and consumers of health services. The new concepts for service were reflected by a greater number of clinic sessions than the station had previously held as a preventive care service and by new staffing at the station.? It included an expanded role for the nurse as nurse-practitioner and primary 3 The project was developed by the Department of Community Medicine at the Mount Sinai Medical Center through an agreement with the New York City Health Department. The codirectors of the project were Nicholas Cunningham, M.D., and Beatrice Thomstad, R.N., P.N.A. 4 For discussion of these new staffing concepts, see Thomstad et al. 1975. 156 EVALUATION OF CHILD HEALTH SERVICES caretaker, a restricted role for the physician as a consultant specialist, and new roles for community health workers as assistants to the nurses and as outreach workers. What developed from the experience was a new type of service geared toward a specific patient population that had its own social and cultural definitions of health, illness, and social problem solving. These families had not one but many problems at any given time. Problems with housing, landlords, unemployment, crime, and school dropouts, as well as those involving family and health, all competed for attention. Family resources for solving these problems were limited. Financial means were minimal. Agencies designed to provide help and services were distant, cumbersome, impersonal, and over- loaded with work. Faced with many problems and few resources, the parents tended to respond to problems only when they erupted into crises. Crises determined the priorities. The health teams offered help, advice, and services for any problem which the parents brought to their attention. The staff understood that it was necessary to respond to the immediate priorities of the parents.®> The resolution of higher-priority crises permitted parents to give attention to health care in turn. The acceptance of parentally defined priorities established communication bridges between staff and parents. This bridging of the communica- tion gap with the parents worked so well that parents and patients came to the child health station in large numbers. The staff of the station ‘was caught in an organizational dilemma. The success of the educational outreach goal brought about increased accessibility to the station. This heightened accessibility, in turn, resulted in increased demand in both the number and duration of patient visits. The number of patients presenting themselves for acute care services fluctuated widely from one day to the next. The time required for response to the number of patients was sabotaging further educational achievement and thorough preventive work. This crisis orientation of the population had interfered with the planning of the staff members and forced them to adopt a crisis response. The staff problem was translated into a problem for the research team. The Research Problem How could we account for the fluctuations in patient demand? If we could account for them, would the crucial variables lend them- selves to manipulation for planning? If they did, would we then be 5 Maslow (1954) has emphasized the importance of the ranking of needs by the individual. Goal Conflicts in a Child Health Station 157 able to smooth out the day-to-day fluctuations in demand so the staff could plan its activities in a rational manner, reducing the emergency atmosphere which tended to eliminate preventive and educational priorities? If we could solve the problem of the fluctuations, then the staff members might be able to preserve each of the goals they had set for the station in their integrated model of health care delivery. If we could not, then they would have to make a decision about setting priorities for the health station, making a conscious choice concerning which goals to eliminate or reduce. A very real alternative would have been to turn the child health station into a children’s emergency room emphasizing acute care to the exclusion of preventive and educational service goals. Inherently, the latter course of action would have changed the character of the station. Establishing Components of Patient Care Demand The data initially available were daily appointment sheets kept by the receptionist for the station. Appointment sheets classified the type of visit: scheduled preventive visit, scheduled follow-up visits, and walk-ins. Broken appointments were recorded. We explored a 4-month period from January through April of the current year as the first approach to understanding patient care demand. Patient demand included three types of services: —Acute care services; —Preventive care services; —Educational or interpretive services. In our present discussion of demand, we are concerned with the first two types, which have different forms of expression by the patient population. Walk-in patients indicate the need for acute care services. Patients with scheduled appointments generally indicate a need for preventive care services. Our two indicators represented different types of variables for planning: controllable and uncontrollable. For purposes of policy research, this distinction is important. Most of the scheduled appoint- ments were controllable by the staff. The walk-in appointments were uncontrollable. . Because it was a controllable variable, we examined scheduled visits first. The codirectors of the child health station had developed preventive protocols which defined the number of preventive visits for each year of life (Cunningham et al. 1975). We assembled data on 158 EVALUATION OF CHILD HEALTH SERVICES Follow-ups Preventive Visits Appointments ——) per day ——L 3 Discharge Walk-ins Approximate Daily Demand Type of Number Rate of Visits Visit Scheduled Kept Appointments Per Day Preventive Appointment 21 0.60 = 15 Follow-up Appointment 11 0.60 = 7* Walk-in None 14 Approximate Daily Total: 36 * This is a low estimate based on an average of only one follow-up visit where indicated. Figure 1. Visit formula the number of children by age from the census and from registration information. These data permitted us to calculate the total number of preventive visits for the station on a yearly basis. Given the existing staff and the time required for a preventive visit, we determined that the station would need an average of 15 visits per working day to maintain its preventive services. However, study of the appointment sheets from the 4-month period revealed an overall 40 percent broken appointment rate. Consequently, it was necessary to schedule 21 visits per day in order to realize an average of 15 actual visits. Acute care, by definition, was not a controllable variable. How many walk-ins could be expected? Based on our analysis of the first 4 months, there were, on the average, 14 walk-ins per day. Given the average figures for preventive visits and for walk-ins, follow-up demand could now be estimated. As described previously, follow-ups were generated from two sources. Follow-up appointments Goal Conflicts in a Child Health Station 159 were given to 15 percent of the preventive visits and to 67 percent of the walk-ins. Based on these calculations, a typical day—including both preven- tive and acute visits—could be constructed as shown in figure 1. This first approximation based on 4-month average figures provided a rule of thumb for establishing different types of demand. The formula took into consideration those factors common to most texts on health planning (Rosenfeld 1971, Gentry 1973). It reviewed demand by age for preventive protocols and utilization characteristics by type of visit and included standards set by the station for quality care. Overall, the formula characterized the optimal combination of preventive and acute visits offered by the available staff. The formula was not intended as a realistic plan for a visit quota, because it did not take into consideration any fluctuations in demand. These fluctuations, not the aggregate workload, interfered with planning for the delivery of services on a daily, weekly, or monthly basis. Far from prescribing visit quotas, this formula was a first step in providing the planner with the components for manipulating de- mand. We hoped that understanding the interrelations of the demand variables would permit a flexible response to the fluctuations in demand. To explore the fluctuations we decided to analyze a longer timespan of utilization. Analysis of Demand Components We explored the daily appointment sheets for a 9-month period from January through September. We began to compare the number of scheduled visits with the number of actual visits. Scheduled visits consisted of preventive and follow-up visits. Actual visits were both kept appointments and walk-ins (figure 2). Except for January and February, scheduled visits exceeded actual visits. If the volume of scheduled visits had actually materialized, the station would have been heavily overloaded. In the beginning of the year, both scheduled and actual visits showed the largest number of patients. Scheduled visits continued to fluctuate around a mean of 40 patients per day. Actual visits, however, exhibited a downward trend. The number of patients fell from a peak of 50 in January to a low of 25 in June and did not start up again until August. Although variation in the scheduled visits showed less dispersion, it was more than we had expected. We had anticipated that the scheduled visits would be very evenly distributed throughout the 9- month period. However, if we keep in mind that scheduled visits consisted of preventive and follow-up visits, then the variation is Number of Patients 50, - * Scheduled 404 i Apptate. Hi Bi id 3 1 . ‘YY 3 1% “ee FA o . iif oper 3Y TFTLIY A : 1" 1 Actual iy: lords} SH : {oLSvisies 30}: ta td i Ve : it ju xX ij ve : : FO i tJ : , 7 e’ R bs Teed A» 20H Tt January February March April May June July August September Figure 2. Variation of scheduled and actual visits. In order to highlight the trend and reduce the day-to-day fluctuations, running medians were used to calculate scheduled appointments and actual visits. The running median is a moving average which is best suited for smoothing variations if the period selected matches the data period. Based on the 5-day clinic week, running medians were established from 5-day periods. I 09 SADIAYAS HLTVAH ATIHO 40 NOILVNTVAA Goal Conflicts in a Child Health Station 161 better understood. From another study we had conducted,® we knew that 80 percent of the presenting conditions during January and February were diagnosed as upper respiratory infections or as otitis media. The acute conditions during this period were swelling the numbers of both walk-ins and scheduled follow-up appointments. The difference between the scheduled visits and the actual visits was accounted for by two components—the number of broken appointments, and the number of walk-ins. If walk-ins equaled broken appointments, then scheduled and actual visits would coin- cide. However, no correlation between broken appointments and walk-ins was found. Actual visits consisted of kept appointments and walk-ins. Since broken appointments were uncorrelated with walk-ins, kept appoint- ments logically also had to be uncorrelated with walk-ins. Conse- quently, we were able to look at these components independently. Figures 3 and 4 present the number of kept appointments and the number of walk-ins respectively, plotted for a 9-month period. Throughout the 9-month period, kept appointments showed no seasonal differences and were relatively stable. The number of walk- ins, on the other hand, was highly seasonal. The greatest number of walk-ins occurred in the first months of the year. They decreased considerably, beginning with April and continuing through August, and rose again in September. Means and standard deviations were calculated for both kept appointments and walk-ins. A comparison of their variability was more critical for planning than a comparison of means. The variabil- ity for the walk-ins more than doubled that of kept appointments. Seasonal variations of the walk-ins had a high of 17 visits in January and leveled off at 8 visits in July and August. In the same period, the number of kept appointments fell only slightly, from 26 to 22. Variations from the monthly average decreased for walk-ins through- out the summer; in contrast, the monthly average for kept appoint- ments showed no significant variation. Although we cannot control walk-in visits, we can predict them. They are higher in the fall and winter months. With an ability to predict the uncontrollable variable, we can manipulate the controlla- ble variable of scheduled visits as a flexible adaptation to the seasonal fluctuations. We recommended to the staff a reduction in the scheduling of preventive care appointments during the winter months, with a 8 The series of studies performed within an organization provides an accumulated knowledge base which gives any one of the studies a broader context for interpretation and insight. 162 EVALUATION OF CHILD HEALTH SERVICES One standard deviatiowm Mean Number of Patients 35 30 25 10 Jan. Feb. Mar. Apr. May June July Aun, Sept. Figure 3. Variation of kept appointments by month gradual increase in appointment scheduling from April through August. We anticipated that seasonally adjusting the preventive care visits to the acute care pattern should have made the visit volume smoother throughout the year. Even so, within the reduced and more manageable volume, day-to-day fluctuations still remained unpredict- able. Since we had modified the demand side of the equation as far as possible, further improvement could come only from the staffing side. The staff had been divided into two teams, each directed by a nurse-practitioner as primary caretaker, assisted by three community health workers. (The physician was a consultant, but not a member of the team.) The teams alternated each week between working in the clinic and the field. During the field week, the nurse-practitioner was the coordinator of the outreach program. Goal Conflicts in a Child Health Station 163 One standard deviation Mean Number of Paticnts T T rT T T ™ T T my ln, Feb. March Apr. May June July Au. Sept. Figure 4. Variation of walk-ins, by month Because the greatest pressure from patient queues in the clinic was on the nurses, we recommended that the nurse on the field team be considered as a routine backup person for the clinic team. If there had been no concern with fiscal problems, we might have recom- mended hiring an additional nurse. However, financial constraints required the use of existing staff only. In order not to disrupt the field week, we recommended that teams develop a more elaborate reporting system between community health workers and nurse-practitioners for the fieldwork. In this way, the fieldwork would continue unhampered when the nurse assigned to the field had to serve as a backup in the clinic. Technically difficult cases would be reserved for the nurse-practitioner. Routine field visits would be delegated to the community health workers. The develop- ment of a revised reporting system would initially require additional training for the community health workers and considerable effort on the part of the nurse-practitioners. However, if effective, the 164 EVALUATION OF CHILD HEALTH SERVICES system for reporting and delegating would increase the probability of accomplishing the multiple objectives of the station. At the same time, it would make the entire staff more aware and more articulate about its field week activities. The problem of supporting a multiple goal program, including preventive and curative care with educational outreach, requires constant alertness on the part of the codirectors. The dilemma continues, but with somewhat more control, when the problems of acute care demand are anticipated and incorporated into planning. We might add, as an epilogue, that the research team did an evaluation of the field week activities in conjunction with the service staff. This evaluation resulted in a redefinition of high-risk patients and families. It also produced impressive evidence that the fieldwork was a highly effective component of the health services offered by the child health station. At a time when nationwide immunization rates for preschool children had dropped to an alarming low (German et al. 1976), 95 percent of the children at the Wagner station had been immunized for diphtheria, pertussis, tetanus, and polio. Summary In this paper, we have detailed the organizational problem, the relevant research, and the recommendations for application to the problem. This organizational problem of the child health station stemmed from unpredictable, widely fluctuating daily demand for services. The unpredictability of the demand disrupted staff members in the organization of their work, thereby threatening the achieve- ment of their multiple goals, particularly the preventive and educa- tional ones. In order to cope with the problem of large and unpredictable fluctuations, the research team began a series of successive studies, each of which resolved some part of the problem in its application. First, the research team investigated appointment sheets available from the ongoing operation of the station. Based on the analysis of the data, an optimal visit formula was established according to the requirements of the staff and the age-specific protocols for the station. The next step in the research process involved a comparison by type of visit to study the demand for acute and preventive care over time. This analysis led to the recommendation that the staff adjust the controllable variable, namely preventive visits, around the sea- sonal fluctuations of the uncontrollable, but predictable variable of walk-ins. The effect of this shift in the planning of preventive Goal Conflicts in a Child Health Station 165 appointments was intended to smooth out the extreme variations and keep the total volume of visits more constant throughout the year. It would reduce, but not eliminate, the daily fluctuations. Since we had exhausted avenues for manipulation of demand, any further response to fluctuations had to be made by manipulating staff assignments. Specific recommendations for staff reassignment com- pleted this phase of the study.” REFERENCES Campbell, D.T. (1971) Methods for the Experimenting Society. Paper presented to th¢ American Psychological Association, Washington, D.C., September, 1971. Coleman, J.S. (1972) Policy Research in the Social Sciences. Morristown, N.J.: General Learning Press. Cunningham, N., et al. (1975) PCPs for the PNA—The development and use of primary care protocols (standing orders for pediatric nurse associates) in Advanced Medical Systems: Issues and Challenges, C.F. Flagle, ed. Miami: Symposia Specialists Medical Books. Gentry, J.T. (1973) The Organization, Financing and Delivery of Personal Health Services. Section III. Chapel Hill: University of North Carolina Press. German, P.S,, et al. (1976) Preventive and periodic health care of inner-city children. Journal of Community Health 2:92-106. Maslow, A.H. (1954) Motivation and Personality. New York: Harper. Moss, J.Z. Concurrent evaluation: An approach to action research. Social Science and Medicine (Supplement 1) 4:25-30. Rosenfeld, L.S. (1971) Ambulatory Care: Planning and Organization. (Chapters 4 and 6). Rockville, Md.: U.S. Department of Health, Education, and Welfare. Thomstad, B., Cunningham, N., and Kaplan, B. (1971) The doctor-nurse game— revising the ground rules. Nursing Outlook 23:422-427. Tukey, J.W., and Wilk, M.B. (1970) Data analysis and statistics: techniques and approaches. In The Quantitative Analysis of Social Problems, E.R. Tufte, ed. Reading, Mass.: Addison-Wesley. 7 Recommendations stemming from our study of fluctuations in demand could not be implemented at the station. To the regret of the staff, the parents and the research team, the New York City Department of Health during its fiscal crisis closed a number of child health stations, including the Wagner Child Health Station. 166 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION Maurice Backett and Barbara Starfield Categorization of this paper according to the type of research was difficult. “Operations research” or “efficiency analysis” seemed a more appropriate term than “research” because the study was aimed more at obtaining information to solve a problem in a particular situation than at providing new insights or new knowledge. The essential difference between the two types of inquiry is in the extent to which the various factors impinging on the dependent variables are known. When the study design deals only with a limited number of dependent and explanatory variables, it is not possible to generalize the findings because the degree of similarity of the facility in which the study is done to other facilities is not known. Although Ms. Belville stated that the work presented in the paper was only one aspect of the involvement of the researchers in this community health center, the relationship of this work to the other activities is not presented. This omission led several discussants to express concern that the narrow concentration on only one relation- ship (in this case, that between scheduling of appointments and clinic workload) might lead to a failure to recognize other (perhaps undesirable) effects of the intervention. In particular, there was concern that the objective of smoothing out marked variability of patient load, although desirable for providers, might adversely affect patients by making their care less continuous. An advantage of this type of study is that implementation of research findings is optimal. Those who are involved in providing care are also involved in defining the problem and setting the objectives, and implementation of the findings in that setting should follow naturally and easily. Unfortunately, effectiveness of implemen- tation could not be examined by the researchers, since the clinic was discontinued because of financial exigencies. Applicability of the findings to other situations is not possible, however, because the research was tailored to particular providers in a particular setting. However, the applicability of the method itself, whereby research is based upon observed needs, is potentially generalizable. But even here, transferability of the method would be limited in situations where financial support for researchers is lacking. These researchers were fortunate in having independent funding, which made it possible to respond to research needs as they arose. In most situations, definition of a problem is followed by the need to apply for a grant to address it, thus involving a long lag time between problem identification and research. Goal Conflicts in a Child Health Station 167 Thus, discussion of this paper showed how easy it is for researchers to trade generalizability for ease of implementation of findings (and vice versa). The more generalizable a result is, the less likelihood that it will lead to change in the setting where it has been generated, because the question may arise from the perception of a wider problem rather than from one perceived by the facility itself. Conversely, the less generalizable the result, the more likely that it will produce a change in the particular setting, because it arises from within the setting. Researchers should be able to respond to both goals. In order to do so, they must have the stability of financial support to respond to needs as they arise in a particular setting and must plan their research so that all the important variables are included in the design. 12 NUTRITION INDICATORS AND THE BARBADOS PRESCHOOL CHILD NUTRITION INTERVENTION PROGRAMME, 1967-76 F. C. Ramsey Summary In 1965, the children’s ward of the Queen Elizabeth Hospital (QEH) in Barbados was faced with overcrowding, cross-infection, and staff shortages. To determine the cause of the problem, 340 consec- utive admissions to the children’s ward under the author’s care were analyzed in that year.! This study showed that the overcrowding was due mainly to the admissions of such preventable diseases as gastroen- teritis and malnutrition. In 1966, the total admissions (1,394) were analyzed in order to determine the extent of the problem. This analysis showed that 143 cases (10 percent) were children with energy-protein malnutrition (EPM) and that the majority were under age 2. The readmission rate for children with EPM was high, at 25 percent. The management of these children was also costly. We determined that 89 percent of ’ these children were lost to followup. This significant finding empha- sized the need for a system of continuity of care. The Nutrition Intervention Programme, embracing continuity of care between the hospital and the community, was introduced in 1967. The main objective of this program was to reduce overcrowding on the ward through a significant reduction in: —Hospital-days for EPM; —Readmission rates for EPM; —Case-fatality rates for EPM in hospital; —Point prevalence rate of EPM in the community. ! The study reported here was supported in part by funds from the Macy Foundation, the Williams-Waterman Programme of the Research Corporation, OX- FAM, Milbank Memorial Fund, and CAFOD. 169 170 EVALUATION OF CHILD HEALTH SERVICES During the 10-year period from 1967-76, a marked decline was noted in: —Hospital-days, from 7,825 to 1,353; —Hospital readmission rate, from 25 percent (1966) to 0 percent (1976); —Case-fatality rate, from 13.6 percent to 0 percent (1976); —Community point prevalence rate, from 16.5 percent to 10 percent; —Mean daily bed occupancy, from 90 to 27 (in a 54-bed ward). The following indicators were considered the most suitable for measuring changes in the nutritional status of the Barbadian pre- school child: the infant mortality rate, specific mortality rate for malnutrition, weight and height in relation to age, classification of children according to the extent of malnutrition—using Gomez and World Health Organization (WHO) classifications, percentage of readmissions, number of days spent in hospital by children recuper- ating from malnutrition (hospital-days), and the number of admis- sions of malnourished children to the hospital. The utilization of results is discussed in relation to developing priorities, the establishment of staff positions, the development of an information system and road-to-health charts, the motivation of staff, the conducting of nutrition surveys, and the establishment of the Nutrition Centre of Barbados. Publications, training, program expan- sion, expenditure, estimates, research, and invitations to scientific meetings are also considered. In the discussion of results, it is evident that the nutrition indicators demonstrated a marked improvement in the nutritional status of the preschool child during the period. However, the absence of a control group made it impossible to carry out an effective, objective evalua- tion of this nutrition intervention program. Purpose This study was initiated in 1966 by the author in Barbados in order to create an appropriate, service-oriented nutrition intervention program that would alleviate the conditions of overcrowding (two or more infants in some cots) and cross-infection on the QEH children’s ward. These conditions were caused by the high admission rate and long duration of stay of malnourished children. The QEH is the only Government-sponsored acute general hospital on the island. It has 600 beds—>54 in the medical pediatric ward and 28 in the special baby unit. In 1967 the hospital established a teaching Nutrition Intervention Program—Barbados 171 program for medical students in their final year. In 1976 the program was extended to students in their second clinical year. The need for the program was clearly shown by the overcrowding on the ward (a mean daily bed occupancy of 90 children in a 54-bed ward), in the outpatient clinics, and in the emergency room. This condition contributed to difficulties in recruiting and retaining highly trained professional staff. This program, which was supported with funds from the Government and private foundations, was started to provide a special child health care service at a cost that Barbados could afford and to monitor that service by the use of certain nutrition indicators. Another intention of the program was to provide useful information from which Government planners could decide whether this program should eventually be incorporated into the island’s health services. Five years later, in 1972, as an outcome of the above program, the Nutrition Centre of Barbados was established by the Government within the QEH compound. One of the center’s ten basic functions, performed in collaboration with the staff of the children’s ward, is the management of all malnourished children before and during admission, as well as after discharge from the hospital. Area of Child Health Care The service offered by this program has been mainly in the area of nutrition intervention. However, from the start, the program was integrated into the health services by the author, who holds joint appointments in the departments of pediatrics and social and preven- tive medicine. In the hospital, treatment of malnourished children is conducted by a highly trained team of consultant pediatricians, junior supportive staff, and nurses. They, in turn, are supported by a home visiting program run by community health nurses. The latter program is designed to identify contributing factors and to institute innovative measures in the management of undernourished children before admission to or after discharge from the hospital. In the nutrition clinic, the child is measured and his anthropomet- ric measurements are recorded. The mother is given nutrition education by group discussion-demonstration and later by one-to-one teaching. At that time breast feeding and nutritionally relevant weaning techniques are emphasized. These procedures are followed by a clinical examination, counsel- ing, and treatment. Full immunization for school entry is performed. When necessary a subsidized food supplement and/or medicine are provided. Full laboratory investigations are possible through the 172 EVALUATION OF CHILD HEALTH SERVICES hospital laboratory service. Nutrition surveillance and rehabilitation services are available on an ongoing basis for these children from birth to age 12. Although the program attempts to be comprehensive in aim, it is focused on children at high risk for malnutrition. Efforts have been made to integrate this endeavor with the existing services for maternal and child health and family planning and health (including dental and mental health) education. Teaching the family how to cope with problems is very much a part of this program. It is part of the community health nurse’s job to try to find employment for the mother and to teach her how to budget wisely. Study Design of Nutrition Intervention Program A nurse interviews the mother or guardian of an infant with a case of EPM. Using a structured questionnaire (appendix 1), she elicits information pertaining to the infant's nutritional background. An- thropometric measurements, clinical examinations, and laboratory tests are performed. All weights are charted according to the Gomez Scale (appendix 2), and the road-to-health card (appendix 3) is retained by the mother. Home visits are made by nurses to gather data on environmental conditions, record anthropometric measurements, treat minor ail- ments, and provide health education. In the nutrition clinic, the nurses hold consultations with the mothers and children, record anthropometric measurements, examine and treat children with prepackaged medicines under the physician's supervision and dis- pense powdered whole milk at a price subsidized according to the child’s need. Immunizations, demonstrations, and health education at group and individual levels are also performed. A family planning service is provided in the nutrition clinic by a representative of the Family Planning Association, as well as by our trained nurses. This active association has 11 clinics throughout the island. In the family planning clinics, after conducting an interview and examination, the staff dispenses various contraceptive devices for a nominal charge, or free for indigent clients. At the end of the clinic session, a record is made of all mothers who failed to attend. A nurse visits each home on the following day to ascertain the reason for the absence. There she provides the same service the mother and child would have received if they had attended the clinic. A new appointment is made, and the mother is urged to keep all future appointments. The nurses have access to transportation, and a minibus is available to those mothers and Nutrition Intervention Program—Barbados 17% children who live several miles from the hospital or who do not have bus fare. When the child is ready for discharge from the hospital, mothers are instructed to bring the child regularly to the nutrition clinic or to return to the ward if the child becomes acutely ill. In addition to the main Thursday clinic of about 50 children, there are miniclinics comprising about 8 children on other days except Sunday, when the children visit the QEH emergency room. In the event of unsatisfac- tory clinical progress at home, the nurse takes the child to the ward for treatment or admission, as indicated. Thus the community nurses work in close collaboration with the ward nurses in the management of EPM infants. In this nutrition intervention program, the community nurse frequently has three categories of mothers and children—those whom she has to see daily during an acute illness, those who attend the nutrition clinic regularly for general health supervision, and those who have failed, or are unable, to attend the clinic and have to be followed at home temporarily. Consequently, home visiting occurs on a selective basis. In her role as homemaker, the nurse gives advice which is adapted to the realities of the situation in the home. She also provides nursing care and medical treatment when prescribed by the physician. Great emphasis is placed on home visiting, and in 10 years the nurses have failed to trace only 14 children, 6 of whom have emigrated. In this way good rapport is established between the mothers and the community nurses. Cases with difficult socioeconomic problems are discussed at staff meetings, and every effort is made to develop a team approach to the solution of these problems. The data on which the results are based have been taken from the program records, which are compiled separately from the hospital records. Simple monthly reports showing relevant data and trends are compiled by the nurse supervisor (appendix 4). The following indicators were considered most suitable for the measurement of changes in the nutritional status of the Barbadian preschool child: —Mortality rate in children of less than age 1; —Mortality rate in children ages 1 to 4; —DMortality rate in children ages 1 to 2; —The proportion of all deaths in children under age 5; —Specific mortality rate for malnutrition; —Specific mortality rate for gastroenteritis; —Weight and height in relation to age; 174 EVALUATION OF CHILD HEALTH SERVICES —Clinical indicators of nutritional status; —Classification of children according to the extent of malnutrition (Gomez scale and WHO classification); —Percentage of readmissions; —Number of days spent in hospital by children recuperating from malnutrition (patient-days); —Number of admissions of malnourished children to hospital; —Cost of rehabilitating malnourished children in hospital; —Cost of rehabilitating malnourished children in the community. Results Table 1 compares statistics of children admitted to QEH with EPM from 1967 through 1976. This table shows a general trend of improvement during this period in the indicators of admissions and readmissions, especially when these are compared with the base year of 1966 (the year before the program started). For that year, admissions were 150 patients and readmissions 25 percent. This trend is more marked in the number of patient-days and in the hospital case-fatality rate, which show a decline from 7,825 (1967) patient-days to 1,879 (1973) and from 13.6 percent to 1.9 percent respectively.? In 1974 and 1975 all these nutrition indicators showed deterioration, but in 1976 a remarkable recovery was posted, with the lowest figures on record for admissions, readmissions, patient-days, and hospital case-fatality rate. Two national nutrition surveys have been carried out in Barbados. Both were random stratified samples of the entire island population, but the 1969 survey comprised all age groups, while the 1975 survey was restricted to the most vulnerable group, from birth to age 5. The results are illustrated in table 2 with a comparison by age of the Barbadian preschool children in the community who fall into differ- ent levels around the Harvard standard weights for age. This table shows that the rate for those children who failed to thrive (below 80 percent of standard or the third percentile) declined from 16.5 percent in 1969 to 10.2 percent in 1975. The Gomez scale for moderate and severe malnutrition (below 75 percent of standard) shows a surprisingly low figure of 3.5 percent in 1975. The infant mortality rate declined from 54.1 (1967) to 29.1 per 1,000 live births (1975). The mortality rates in children ages 1 to 4, 2 The apparent rise in patient days of 3,463 (1971 J-D), as compared with 2,860 (1970 A-D) can be explained on the basis that the 1971 period is for 12 months and the 1970 period, for 9 months. The rise in case fatality rate for 1970 was associated with a temporary staff shortage. Table 1. Results of Hospital Statistics of Children Admitted with EPM, 1967-76 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 A-M A-M A-M A-D J-D J-D J-D J-D J-D J-D Admissions 125 138 135 88 89 96 57 69 73 51 Readmissions 3 1 2 2 2 2 1 0 0 0 Patient-Days 7,825 5,922 4,943 2,860 3,463 2,468 1,879 2,431 2,052 1,353 Hospital Case Fatality Rate 13.6 9.5 3.5 9 2 1. 1.9 2.9 4 0 Hospital Cost (BDS $) 211,275 159,894 93,461 85,800 121,205 86,380 65,765 102,102 90,234 56,826 A-M, Apri-March; A-D, April-December; J-D, January- December sopeqreg—ure180.1d UOIIUIAINU] UOHILIINN GL1 Table 2. Comparison by Age Group of Barbadian Children Surveyed in the Community and Falling into Different Levels Around the Harvard Standard Weights for Age, 1969 and 1975 Percentage Age Group in Different Levels Around Standard Weight for Age Age Group No. of Children (Months) Evauined 100 & over 80-99 60-79 Under 60 1969 1975 1969 1975 1969 1975 1969 1975 1969 1975 0-11 34 615 35 49.6 29 44.8 29 5.8 6 0.1 12-23 51 712 23 25.3 61 64.3 16 10.1 0 0.28 24-35 52 853 23 25.9 63 62.6 14 11.1 0 0.35 36-47 58 840 24 26.8 59 63.3 15 9.5 2 0.24 48-59 53 687 21 21.4 72 65.8 7 12.5 0 0.29 0-59 248 3,707 24.6 28.9 58.9 60.8 15.3 9.95 1.2 0.27 9L1 SADIAYAS HLTVAH A'TIHD 40 NOILLVNTVAA Nutrition Intervention Program—Barbados 177 ages 1 to 2, and the proportions of all deaths in children under age 5 from 1967 to 1976 have not been analyzed. However, research indicates that there are fewer severe grades of malnutrition on admission to the ward and that mothers are presenting earlier at the hospital with relatively more Grade I EPM children. The cumulative community case-fatality rate of 5 per 1,000 chil- dren ages 1 to 4 years recuperating from malnutrition (1969) compared favorably with the mortality in the general population over the same age group which was 2.2 per 1,000 children (1970) in Barbados and 4.5 per 1,000 (1968) in Jamaica. This rate is well below the published figures for community studies, which range from 15 percent to 30 percent. Utilization of Results Priorities. An analysis of admissions of children to the ward in 1966 demonstrated a pattern of disease that made us decide to concentrate on the treatment of malnutrition and gastroenteritis. Since then, frequent analyses of our results have assisted us in developing priorities. Staff. The presentation of interim reports at the outset of the program enabled the director to recruit staff, first from the Peace Corps, and finally from the Government. Information System. Monthly and annual reports have been submitted to the Ministry of Health, and politicians and administrators have released edited versions of data periodically through the communica- tion media to inform the public concerning improvements in the status of child health in Barbados. Columnists have been supplied with technical information arising from this program that has enabled them to prepare nutrition articles for the women’s page of the daily newspaper. Occasionally, feature articles have been written about the work of the nutrition intervention program as a result of interviews with its director or staff members. Road-to-Health Charts. A modified version of the original Morley Road-to-Health Chart has been used successfully in this program. The demonstrative success of this chart has convinced the Ministry of Health to adopt it. Its use is now mandatory in all Government child health clinics, and the chart is available free to all voluntary child health clinics as well as to general practitioners. This chart is also used to demonstrate to the principal that immunization for school entry has been fully completed. Motivation. Various nutrition indicators have shown a steady decline in malnutrition over the past 10 years. This visual demonstration of improvement in the health status of children, which we had felt 178 EVALUATION OF CHILD HEALTH SERVICES intuitively, has helped to promote and maintain a high sense of motivation among the staff. Surveys. The QEH nutrition-indicator statistics suggested a marked decline in the prevalence and severity of EPM in the early years. When the improvement was presented at various scientific meetings, a question was raised as to whether these figures reflected accurately the situation in the community. These discussions tended to expedite the community nutrition surveys that were carried out in 1969 and 1975. Establishment. of the Nutrition Centre in Barbados. As the nutrition intervention program progressed, the work increased in volume and complexity. With a steadily growing staff, the small space allocated in a corner of the ward and in the general children’s outpatient clinic soon became inadequate. These factors, coupled with the results of the interim reports and funds raised by voluntary efforts, local organizations, and overseas foundations, led the Government to establish the Nutrition Centre by providing the land and a matching grant. Publications. Data gathered in the program have been of use in the production of pamphlets, brochures, manuals, slide presentations, learning packages, and calendars. This educational material has been distributed to community nutrition groups, schools, teacher training colleges, and interested lay and professional persons. Training. Information from this program has been very valuable in designing demonstrations for mothers of malnourished children as well as in influencing the content of seminars for nurses, community groups, health aides, and medical students. St. Vincent Community Health Action Programme. Experience gained in this nutrition intervention program in Barbados has led to the implementation of a community health action program in two communities on the neighboring island of St. Vincent. After 3 years it is hoped that, with the Barbadian example, the government of St. Vincent will find it financially feasible to assume responsibility for this program. Estimates of Expenditure. In the draft estimates for the nutrition center, the results of the nutrition intervention programs have been used in relevant budgetary notes in order to justify requests for funds. This application has enabled us at Estimates Committee meetings to base our requests on factual information rather than on humanitarian grounds alone. Research. The need for further research into selected areas of the program has been highlighted by the results, and action has been taken where appropriate. Invitations to Meetings. The program and results have led to Nutrition Intervention Program—Barbados 179 invitations to attend various international meetings. In turn, discus- sions at these meetings have provided valuable information for improving the program, and contacts with professional colleagues have also led to direct inputs in some cases. Discussion The original purpose of this program was to apply effectively existing knowledge in reducing the overcrowding and cross-infection on the QEH children’s ward and not to do basic research. The original objectives were clear. In each of the next 5 years, the goals were to: —Reduce hospital-days for malnutrition by 10 percent; —Reduce the readmission rate for malnutrition by 10 percent; —Reduce the case-fatality rate for malnutrition by 10 percent; —Establish and compare the mortality rate of children ages 1 to 4 years recuperating from malnutrition with the rate of children in the same age group for the general population of Barbados and other Caribbean territories. Since the program was based in the only Government-sponsored, acute general hospital serving the entire island, the sample size was adequate. However, the treatment of malnourished children is widely known and accepted, and because treatment could not be withheld on humanitarian or political grounds, the use of control groups was not considered feasible in the study design. The program had satisfactory means for carrying out measure- ments and recording data, but not the processing of the data. The mechanisms for checking the validity of the procedures were super- vised internally by senior staff members of the program as well as by visiting colleagues. However, apart from the internal evaluation by our staff, no special staff members with skills for objective evaluation have been available to us. Our baseline studies have provided us with valuable information which has helped us measure our performance. Nevertheless, it has not been possible to consider and give detailed costs of a range of alternatives. More importantly, a detailed cost-benefit or cost-effective analysis, in terms of input costs and direct or indirect output costs, has not been carried out. Although causality cannot be proved, the target groups of families of malnourished children, politicians, administrators, and donor agencies were reached, and objectives were met. 180 EVALUATION OF CHILD HEALTH SERVICES Perhaps of great significance is the fact that apart from the 1974- 1975 period immediately following the energy crisis, there has been a marked decline in the nutrition indicators of this socioeconomic disease, in spite of rising economic costs. REFERENCES Gomez, J. (1956) Mortality in second and third degree malnutrition. Journal of Tropical Paediatrics 2:77. Jelliffe, D.B. (1966) The Assessment of the Nutritional Status of the Community. Monograph Series No. 53. Geneva: World Health Organization. Ministry of Health and Welfare. (1975) Annual Report of Chief Medical Officer, Barbados. Ministry of Health and Welfare. (1966-1972) Health Services in Barbados. Published annually. Morley, D.C. (1966) The weight chart for under-fives clinic. In Medical Care in Developing Countries, A Symposium from Makerere, M. King, ed. London: Ely House. Pan American Health Organization (PAHO) (1972) The National Food and Nutrition Survey of Barbados. Scientific Publication No. 237. Washington, D.C.: PAHO/WHO. PAHO (1966) Report of the Latin American Seminar on the Planning and Evaluation of Applied Nutrition Programs, Popagan, Colombia, November 10-17. Scientific Publication No. 160. Washington, D.C.: PAHO/WHO. Standard, K.L., Lovell, H.G., and Garrow, ].S. (1966) The validity of certain physical signs as indices of generalised malnutrition in young children. Journal of Tropical Paediatrics 11:100-106. WHO (1974) Report of currently available indicators in the field of health and their use for nutrition activities. PAG Bulletin 4:4. Geneva. Nutrition Intervention Program—Barbados 181 APPENDIX 1 MALNUTRITION PROFORMA NAME OF MOTHER: NAME OF CHILD: DETAILED ADDRESS: DATE OF BIRTH: AGE: OCCUPATION OF MOTHER: HOSP. NO.: AGE OF MOTHER ON LEAVING SEX: SCHOOL: AGE OF MOTHER AT BIRTH OF FATHER OF CHILD: CHILD: AGE OF FATHER AT BIRTH OF ADDRESS: CHILD: NEAREST TELEPHONE NO.: ADMITTED ON: DISCHARGED ON: DAYS IN QEH: DIED ON: P.M. YES NO IMMUNISATIONS MILESTONES 1 2 3 HELD UP HEAD: FIRST STEP: D.P.T. SAT UP UNSUPPORTED: FIRST WORDS: POLIO STOOD UP ALONE: WALKED ALONE: SMALLPOX DIETARY HISTORY F.B.F. P.B.F. B.F.N. N.B.F. SUPPLY OF B.M. BREAST FEEDING: WHY DID YOU STOP? BOTTLE FED: TYPE COMPOSITION FREQUENCY ? ” WHY? WEANING ABRUPT GRADUAL SENT AWAY YES NO PERSON DIFFERENT HOME 182 EVALUATION OF CHILD HEALTH SERVICES 0-3 Mths [3-6 Mths | 6-12 Mths | 1-2 Yrs EGGS FREQUENCY/ WEEK MEAT ” FISH ” CHEESE FRUITS ” VEGETABLES ” STARCHY FOODS 7 Sago, Arrowroot Cornmeal, Linseed BUSH TEAS Mode of preparation TYPE OF BUSH TEA AND FREQUENCY FOLKLORE AND CUSTOMS: IS IT HARMFUL TO BREAST FEED AFTER (1) Working in the sun (2) Cohabiting (3) Other reasons (4) Is milk too heavy after 6:00 p.m. (5) Are some “teas” “cooling” and others “heating”? If “yes,” make lists of cooling and heating teas: (6) Do some foods give worms? If “yes,” which ones? PREVIOUS HISTORY: BORN AT: PREGNANCY: V.D.R.L. TREATMENT GESTATION DELIVERY ATTENDED BY SINGLE MULTIPLE B.W. CRIED IMMEDIATELY Nutrition Intervention Program—Barbados 183 INFECTIOUS DISEASES: MEASLES WHOOPING-COUGH BRONCHOPNEUMONIA GASTROENTERITIS OTHER FAMILY HISTORY NO. OF CHILDREN, WITH AGE AND SEX PLACE IN FAMILY: DEATHS IN CHILDREN IN FAMILY WITH CAUSES: S.H. FINANCIAL SUPPORT: YES NO MARRIED UNMARRIED FATHER’S OCCUPATION: EXPENDITURE HOUSE—RENT PAID FREE EXPENDITURE/WEEK MOTHER'S OCCUPATION HOW MUCH ON FOOD MOTHER’S DRESS: WELL PLAIN MOTHER'S NUTRITION: SATISFAC- UNSATISFACTORY TORY AGE OF BABY WHEN MOTHER RETURNED TO WORK: HAS CHILD SPENT ANY TIME IN ALMSHOUSE? REGULAR ATTENDANCE AT CLINIC WITH INFANT: G.P. QEH NIL EDUCATION: PRIMARY Class SECONDARY Form FAMILY PLANNING: DID YOU PLAN TO HAVE THIS BABY? YES NO HAVE YOU ATTENDED A FAMILY PLANNING CLINIC? YES NO REFERRED? YES NO WERE YOU USING ANYTHING TO PROTECT YOU FROM BECOMING PREG- NANT? YES NO DOES THE FATHER ACKNOWLEDGE THIS BABY AS HIS OWN? YES NO 184 EVALUATION OF CHILD HEALTH SERVICES CLINICAL EXAMINATION! T. P. R. W.0.A. CC.O.A. W.0.D. HT.O.A. CC.O.D. HT.O.D. HC.O.A. ARMC.O.A. HC.O.D. A.C.O.D. SWELLING YES NO WASTED MUSCLES YES NO SUBCUTANEOUS FAT YES NO MISERY YES NO HAIR CHANGES: BROWN REDDISH GREY STRAIGHTNESS SPARSENESS SILKINESS EASILY PULLED OUT SKIN COLOUR: LIGHT REDDISH-BROWN OTHER NORMAL LOOSE STOOLS: YES NO “COLD” YES NO RASH FLAKY PAINT YES NO OTHER YES NO ULCERS AND CRACKS YES NO SORE ANGLES OF MOUTH YES NO BRIGHT RED TONGUE YES NO LARGE LIVER YES NO INVESTIGATIONS: Hb. Ht. MORPHOLOGY V.D.R.L. MANTOUX STOOLS URINE EYES edcba abcde TEETH ''W, Weight; HT, Height; HC, Head circumference; CC, Chest circumference; AC, Arm circumference; O.A., On admission; O.D., On discharge Nutrition Intervention Program—Barbados 185 FIELD NURSE'S FORM NAME OF MOTHER: NAME OF CHILD: DETAILED ADDRESS: DATE OF BIRTH: AGE: SEX: SURVEY NO.: HOME ENVIRONMENT HOS. NO. R. = RESPONSIBLE FOR CHILD D.C. = DAY CARE OF CHILD AGE 0-6 MTHS 3-6 MTHS 6-12 MTHS 1-2 YEARS GRANDMOTHER: MOTHER: FATHER: OTHER RELATIVE: PAID PERSON: CHILD UNDER 14: OTHER (Specify) PARTICULARS OF HOUSEHOLD: NO. OF ADULTS: NO. OF CHILDREN: NO. OF ROOMS: SEPARATE SLEEPING AREA FOR INFANT: RUNNING WATER: YES NO TOILET: PIT BUCKET W.C. KITCHEN FACILITIES: KEROSENE WOOD MATERNAL CAPACITY: MOTHER CRAFT: SATISFACTORY HEALTH EDUCATION: ” PROTECTION OF MILK AND FOOD 2 PRESENCE OF FLIES RATS IS BOTTLE STERILISED BEFORE EACH FEED? TYPE OF BOTTLE USED: UNSATISFACTORY ROACHES 186 EVALUATION OF CHILD HEALTH SERVICES FIELD NURSE'S CHART i DATE | AGE WT. HT. . . ‘ REMARKS KILOS Nutrition Intervention Program—Barbados 187 APPENDIX 2 BARBADOS Q. E. H. NUTRITION CLINIC 14 131 SW 430 NAME 13} BIRTH n 7] REG: NO: 0 SURVEY NO: ~~ LBS 1 1 1 1 1 1 1 0 2 4 6 8 10 20 30 36 MONTHS 188 KILOS 20 19 EVALUATION OF CHILD HEALTH SERVICES NAME BIRTH REG: NO: SURVEY NO: BARBADOS Q. E. H. NUTRITION CLINIC SEXES COMBINED 3-6 YEARS 2 A 1 1 1 18 1 1 8 so Sm—l— 2 4 6 8 g) 2 4 6 8 45 2 MONTHS LBS 23 22 2 19 18 — — wo on — += HILOGRAMS 13 12 11 10 “ww DIET 3-4 Years 4-5 Years BARBADOS MINISTRY OF HEALTH HEALTH RECORD SMALL POX IMMUNIZATION CONTRA INDICATED [_] DATE PERFORMED [| DATE [J oate.... [C] pate... SUCCESSFUL [|] DATE... NOT successFuL [] DATE... [O oate... 3 pate... DOCTOR'S NURSE'S SIGNATURE..... REVACCINATION AGAINST SMALL POX Date D Performed Result Reod REGISTERED AT DOCTOR CLINIC HOSPITAL DIPHTHERIA, WHOOPING COUGH, TETANUS, POLIO Dose D.P.T.Date | Polio Date |ooenma or" CHILD! s : ma NoTHERS FIRST SECOND IMMUNIZATION SCHEDULE 3Hi%D . 1st BOOSTER YOU SHOULD TAKE YOUR CHILD TO YOUR DOCTOR/ | 2nd BOOSTER CLINIC AS FOLLOWS: 3rd BOOSTER AGE IMMUNIZATION 4th BOOSTER 3 MTHS 1st DPT AND ORAL poLio Seseonne C8000 2000000000000 000000RRNGRNRS om" 2nd N.B. "TER AGE 5 D.T. TOXOID IS USED 6 3rd * ” $0000000000000000000000000000000000000000000 Joly Mall, pox VACCINATION OTHER IMMUNIZATIONS OR TESTS "Te 131 BOOSTER DPT AND ORAL POLIO tnd I Doctor's/Nurse’s 4 YRS 6 MTHS 2nd BOOSTER DPT AND 8) Nome pon Rei Signature ORAL POLIO 1 4 YRS 7 MTHS (HEAF TEST AND BCO HEAF IF NEGATIVE) BCG 4 YRS 8 MTHS 2nd SMALL POX VACCINATION : MEASLES | i This Book is a permanent Record of your Child's Road to Health. Take it with you whenever you take LANDMARKS OF DEVELOPMENT your child to any To Clinic, or Hospital. Te ror WTS SMILED SAT UP WALKED KEEP THIS BOOK IN A SAFE PLACE TURNED STOOD WITH 2SWERDS OVER SUPPORT € XIANAddV sopeqieg—uwei801J UONUIAINU] UONLIINN 681 REASONS FOR SPECIAL CARE (T) LOW BIRTH WEIGHT <5 LBS. OR 2270 GMS. [J (2) TWIN [J (3) PREM. TZ] (4) POOR EARLY WEIGHT GAIN eo ce (6), FREQUENT R.T.I. [J (7) 6TH CHILD + [J (8) ESNM. {T] (9) MIM [J (100 MOTHER DECEASED = (11) LOW MATERNAL WEIGHT GAIN [] (12) TEENAGE MOTHER J] 13) Pm OJ 0-1 Year 13 16 15 14 1-2 Years Ae A=i1- ties fer nieip 18 17 16 15 ul wap mapa fee femme fe daa] 12 10 11 — oo © Fe =1-=-jemde mfp mm rs fo ia - 4 ed wf wf = - ————— —fee pm ~~ -t beenilice dene pavmbumadeastesobinat s mutns ieee 0 B oo, -T |. 1. o a] I ITT - "tp oF "TT 1 1 n 4 LT] n[ | ol l N — — ~~ Feb Tbe mpm mbm bee — [=] T “wo KILOGRAMS KILOGRAMS ¥, — rfl oid CoAT ==p---b st eb - F--4---}--1-=1- re -1=F rt EL / . > os 8 Fy +1 8 L —1 E codes feng wl fe Ao edt eet SH yea eal pelfe HE ~~ re df] LL |= 36 LoL ALS m7 eet TL = ~+ £2 YXA | 4 SP la ceded _fe ju crite rma we ere face es yd] 5 4 ’ x ars I} 6 le=1 - 2 7 7 + ® AA === EY met sede deen —— -4--- dpe a [neil on ris moon i mmo wh nt nm Tm cm J i tts fe a¥Y Ls yd x 5 AA 2 Ft - 7 bp -1--- “T19 i a mma pwns we Ete == ey a wa DIET =~ DIET Breast milk is best, Iron and folic acid add zest, while multimiv will da tha raet 2-3 Years Record on chart all important illnesses and haemoglobin. I 06 SADIAYAS HLTVAH A'TIHD 40 NOILLVNTVAY Nutrition Intervention Program—Barbados 191 APPENDIX 4 BARBADOS NUTRITION PROJECT—MONTHLY REPORT » 197 M F Total Cumulative Total to End of Current Month This Year Last Year Admissions Marasmus Marasmic Kwashiorkor Kwashiorkor Grade 1 Grade 11 Grade 111 0-3 Months 3-6 Months 6-12 Months 1-2 Years 2-5 Years Ward Deaths Anaemia Readmissions: PCM Readmissions: Other Hospital Days Occupancy % Cost 192 EVALUATION OF CHILD HEALTH SERVICES Chil- dren First Visit This Year First Revisits Visit Ever Attend- ances Children Attend- ances Immunisa- tions Triple & Polio—1st Triple & Polio—2nd Triple & Polio—3rd Triple & Polio— Booster Small Pox Measles Home Visits Mileage Remarks Signed____ Nutrition Intervention Program—Barbados 193 DISCUSSION Milton 1. Roemer and Gabriel Velazquez The numerous improvements in the nutritional status of children in the island of Barbados have been clearly demonstrated by Dr. Ramsey’s report. The evidence is strong, not only from the reduced admissions of malnourished children to the island’s single large hospital, but also from a decline in the case-fatality rate of those children admitted. Probably more important epidemiologically has been the reduction in the number of underweight children found in population surveys. Nevertheless, how much of this improvement can be attributed to the energetic nutritional program developed and described by Dr. Ramsey is not clear. In the Barbados setting, a “control” population was not feasible, nor was any other Caribbean island suitable as a control. No data were presented on the trend in per capita income or the standard of living over the 8-year observation period. Price levels did indeed rise, but we do not know whether changes in earnings were greater or less than the inflation rate. The birthrate in the Barbados population declined from 35.4 per 1000 per year in 1955 to 21.9 in 1975; this change alone could be expected to improve the nutritional status of the fewer children in a family, even if family real income remaind unchanged. (The family-planning program, of course, was part of the total effort, but it is not the same as nutritional intervention.) Also, over the same 8-year period, the Government promoted a program of diversified agriculture, in contrast to the island’s previous exclusive dependence on sugar production. Tourism also brought to the island more money, some of which doubtless filtered down to the general population. These weaknesses in the evaluative research design do not disprove the value of the nutritional interventions. They must make one very cautious, however, in attributing to those interventions the full credit for the improvements in the nutritional status of Barbados children. After all, Dr. Ramsey's creative work was undertaken in response to a serious problem, not as a scientific experiment with evaluative strategies built in at the outset. In spite of these evaluative limitations, the Barbados experience has lessons to teach other developing countries in demonstrating indicators of nutritional change by way of both hospital-patient data and population surveys over time. The task in future research of this type will be to establish “control” or comparison populations which will permit ruling out (or holding constant) the intervening variables, 194 EVALUATION OF CHILD HEALTH SERVICES so that one can draw conclusions on the actual contribution of the nutritional intervention program to the changes found to occur in the nutritional status of children. 13 RESPONSE OF INDICATORS OF NUTRITIONAL STATUS TO NUTRITIONAL INTERVENTIONS IN POPULATIONS AND INDIVIDUALS C. Yarbrough, J-P. Habicht, R. E. Klein, R. Martorell, A. Lechtig, G. Guzman SUMMARY An intervention study designed to change the nutritional status of moderately malnourished preschool children illustrates the use of anthropometry as an indicator of nutritional status and shows that demonstrating the physiological efficacy of an intervention does not guarantee a significant total impact of that intervention on the population. Careful design makes evaluation of some interventions possible, but theory and knowledge in human public health nutrition and its relation to growth in children now limits the design and especially the evaluation of many nutrition interventions. This paper contributes to recent knowledge in this area by estimating the actual energy conversion rates to growth in moderately protein-calorie malnourished children aged 1 through 3 years under field conditions. These conversion rates are even lower than those which would be estimated from normative data in the literature. PURPOSE OF THE STUDY The present study was an intervention designed to change the nutritional status of preschool children in order to test experimentally the hypothesis that moderate protein-calorie malnutrition is related to impaired mental development (Canosa, Salomon, and Klein 1972; Klein, Habicht, and Yarbrough 1975, p. 61). Thus the principal aim of the project was not public health intervention but rather scientific "The present study has been funded for approximately 9 years by the National Institute of Child Health and Human Development in a series of contracts (currently NO1-HD-5-0640) to the Institute of Nutrition of Central America and Panama (INCAP), a research center of the Pan American Health Organization. 195 196 EVALUATION OF CHILD HEALTH SERVICES investigation of the causal effect of nutrition on certain variables. Whether or not the purpose of the study is scientific demonstration or public health intervention, evaluation of such a program requires a comparison between two groups. One group must have benefited from nutrition not received by the other group. Here we present appropriate comparison groups for scientific demonstration of a nutritional effect. We will show that such an effect is essential for evaluating public health intervention programs in nutrition, even though an intervention program that scientifically demonstrates the effect of nutrition does not necessarily produce a meaningful effect on nutrition of the population as a whole. The proposed test of the basic hypothesis consisted of improving the nutritional status of children in order to look for associated changes in mental development measures. Thus, the first step in analyzing the study is to show that such nutritional improvement did in fact occur, which in turn requires the measurement of some outcome universally recognized as being related to nutrition in malnourished populations. AREA OF CHILD HEALTH CARE Failure to achieve maximal mean growth rates during infancy and early childhood is probably the most frequently used indicator of poor nutritional health status for populations. The use of growth rates to assess communities is widely accepted (Jelliffe 1966) and rests on a solid scientific base: Early growth rates show no apparent ethnic differences and are affected by low dietary intakes and high morbid- ity rates (Habicht, Martorell, Yarbrough, et al. 1974; Marsden 1964; Martorell, Habicht, Yarbrough, et al. 1975; Martorell, Yarbrough, Lechtig, et al. 1975). Thus we shall be considering here whether the intervention described below had an impact on the growth of young children between the ages of 1 and 3 years. Other nutritional effects on children’s performance, health, and survival are not dealt with in this paper. The issues to be addressed are: (1) Does this study present convincing evidence that improved nutrition improves growth in malnourished children under field conditions, where other growth- retarding influences continue to be prevalent? Previous studies on this issue have been contradictory (Martorell, Yarbrough, Lechtig, et al. 1976). (2) Does this study present convincing evidence that the nutritional effect on growth can be physiologically substantial under field conditions? Previous studies, which claim an effect on growth, have been contradictory as to whether or not the effect was substantial (Habicht and Butz in press). (3) Does the information derived from Response of Nutritional Indicators to Interventions 197 this study about the effect of nutrition on growth explain the contradictory results of previous studies and suggest strategies for future public health intervention programs? STUDY DESIGN Target Population The study we are reporting was conducted in four rural Ladino? villages in eastern Guatemala. Since the inception of the research, free outpatient preventive and curative medical services have been made available to everyone in the villages (Habicht 1973, p. 24). The economy is subsistence agriculture; the local diet is based on corn and beans. Our principal concern is with the young child. In these villages there is considerable stunting in growth of these children, consistent with a diet in which it is estimated that the protein-calorie ratio is well above recommended limits (World Health Organization 1973, National Academy of Sciences 1974), but the energy level is at least 20 percent below the apparent requirements for children aged 1 to 3 years. Table 1 compares values for the four villages at the beginning of the study with values typical of developed countries. Note the absence of sex differences in incremental growth (Yar- brough, Habicht, Malina, et al. 1975) or diet. Although table 1 does not reflect the small dietary contribution from breast milk, we believe that the average daily intake, including breast milk, does not exceed 80 kcal/kg/day between ages 1 and 3. Design, Methods, and Data The research design presented here is a comparison of the growth of children who voluntarily consumed different amounts of supple- mentation food. In two of the four villages, “atole,” a protein-calorie preparation of Incaparina, dry skim milk, and sugar, is distributed. In the other two villages, “fresco,” a caloric preparation containing sugar, flavoring, vitamins, fluoride, and iron, is given. Table 2 shows the nutrient content of both preparations. Both provide the same vitamins, fluoride, and iron, but atole contains proteins and more calories. The supplementation, whether atole or fresco, is available twice daily in the local field station of each village, and attendance and consumption by all villagers are encouraged, but are entirely voluntary. A daily 2Of Hispanic culture, in contrast to American Indian culture. 198 EVALUATION OF CHILD HEALTH SERVICES Table 1. Growth and Diet of Study Population in Comparison with Values from Developed Countries Values Typical of Study Population Developed Countries Males Females Males Females Height (cm) 12 mo. 68.9 67.3 74.1b< 72.3 36 mo. 84.4 84.2 95.2 93.8 Net growth 15.5 16.9 21.1 21.5 Weight (kg) 12 mo. 8.0 7.4 10.0 9.4 36 mo. 11.7 11.4 14.6 13.9 Net growth 3.7 4.0 4.6 4.5 Dietary intake (kcal/kg/day) 15 mo. 6624 69 18 mo. 73 75 21 mo. 78 76 24 mo. 80 78 100-103¢ 99-106 30 mo. 80 75 36 mo. 76 75 ® Growth data from baseline studies prior to intervention. ® Data from McCammon, 1970. ¢ 12 month height adjusted by 1.7 cm to correct for differences in measuring techniques. 4 Dietary intake data taken during time period of intervention. ¢ Values apply to ages 1 to 3 and are from Table 7 p. 34 Energy and Protein Requirements. (1973) record of attendance and intake is kept for each individual. Because of the different relative concentrations of calories as related to the volume in the two supplements, and because of differing ingestions of supplement for similar attendance to the supplementation centers, these data can be analyzed separately for the relative contributions to growth of energy intake, attendance, and volume of supplement ingested. Beginning in February 1969, data were collected on all children under 7 years of age, and all children born to the village populations before March 1973 were added to the sample. Data collected on children until their seventh birthdays included anthropometry, dental eruption and skeletal maturity observations, dietary and morbidity surveys, sociocultural inventories, and a large battery of infant and preschool mental tests. Other data, not presented here, were collected on pregnant and lactating mothers. Here we are discussing only growth in height and weight of the 456 children for whom data were collected at both 12 and 36 months of age. Height was determined by measuring supine body length on a standard measuring table, and weight by a beam balance, both measurements taken by standard procedures (Martorell, Habicht, Response of Nutritional Indicators to Interventions 199 Table 2. Nutrient Content Per Cup? of Supplementary Feeding (180 ml) Atole Fresco Total calories (kcal) 163.0 59.0 Protein (g) 11.0 — Fats (g) 0.7 — Carbohydrates (g) 27.0 15.3 Ascorbic acid (mg) 4.0 4.0 Calcium (g) 0.4 — Phosphorus (g) 0.3 — Thiamine (mg) 1.1 1.1 Riboflavin (mg) 1.5 1.5 Niacin (mg) 18.5 18.5 Vitamin A (mg) 1.2 1.2 Iron (mg) 5.4 5.0 Fluoride (mg) 0.2 0.2 # Review date: October 11, 1973; figures rounded to the nearest tenth. Yarbrough, et al. 1975b). The observations were taken at the child’s birthday +7 days. Expected Impact of Nutrition on Growth Since there are no clinical or biochemical signs of severe vitamin- mineral deficiencies in the study population, we will immediately focus on the issue of proteins versus calories. Studies of home dietary intake, biochemical indicators, and anthropometric indicators of fat and muscle mass show a diet which appears to be limiting in calories but not in proteins (Martorell, Lechtig, Yarbrough, et al. in press; Martorell, Yarbrough, Lechtig 1976). Moreover, there is reason to believe that in such a situation caloric supplementation alone will shelter proteins and influence growth rates (Martorell, Lechtig, Yarbrough, et al.,, 1976); indeed analyses to date in this population show that caloric supplementation is related to birth weight (Lechtig, Habicht, Delgado, et al. 1975) and placental weight (Lechtig, Yar- brough, Delgado, et al. 1975). Thus we shall discuss the issue of growth as a response to ingested energy. This does not exclude a possible effect of other nutrients on growth in this population. Nor does it imply that the dietary problem of all populations is caloric. Many communities are in vastly different situations, and dietary interventions must be designed only after careful study of which nutrients are limiting. Indeed, interventions which are useful in one setting may be harmful in others. What impact will additional calories have on growth? Since a comprehensive discussion of all ages is beyond the scope of this paper, we will focus on ages 1 to 3 as the youngest period with a span 200 EVALUATION OF CHILD HEALTH SERVICES long enough for effects to be apparent and in which energy consumption from breast feeding is not important. What is the expected impact at this age of nutrition on growth under field conditions? We can make some estimates as follows: Approximately 50 kcal/kg/ day are necessary for basal metabolism and maintenance. In this range there is severe malnutrition and complete growth failure. The average consumption in healthy populations of the developed nations, 100 kcal/kg/day, leads to maximal growth. Thus growth occurs in the range of intake from 50 to 100 kcal/kg/day. In the four villages the typical diet of 80 kcal/kg/day is an increase of 30 kcal/kg/day above the basic need, or 67 percent of the total possible increase in consumption. Thus approximately two-thirds of the part of the diet used for activity and growth accounts for 75-85 percent of the possible growth increment from 1 to 3 years of age. In other words, the approximately 5.6 X 10° kilocalories consumed by a village child in this 2-year period is 2.1 X 10% kilocalories over basal metabolic needs. This energy ingestion above the basal meta- bolic requirements leads to a growth of approximately 7.7 mm/104 kcal in height and 182 gm/104 kcal in weight. In the developed world the typical child consumes a total of 8.8 X 10% kilocalories in 2 years—4.4 xX 10% kilocalories above basal needs—and uses this extra energy to grow at the lower net rate of 4.8 mm/10% kcal in height and 104 gm/10* kcal in weight. Finally, the approximately 1.4 x 10° kilocalories additional needed to bring the diet of a village child to 100 kcal/kg/day can give at most 4.6 cm of growth in height, i.e. a net rate of 0.3 mm/10* kcal. The comparable net rate for weight is 50 gm/10* kcal. This means that the greatest conversion rate of calories to growth that can be expected in the children is 25 times less efficient for height and 36 times less efficient for weight than was the conversion rate from calories to the children’s growth observed at the beginning of the study. Thus we would not expect extra food to be converted to extra growth at a rate greater than the baseline study values, and we would expect that at some point the rate of growth per unit of extra food must drop dramatically in order to bring the overall growth rate into line with the lower value of energy-to-growth conversion found in developed countries. We conclude, therefore, that the conversion of energy to growth must follow a function similar to that depicted in figure 1, which shows a curve of decreasing conversion of each unit of energy to growth with increasing total caloric consumption. The data for this curve do not, however, exist in the literature, so that no exact numerical values can be given. Thus the predictions are at best only Response of Nutritional Indicators to Interventions 201 Growth 1 Dietary Energy Intake Basal Metabolic Needs -J «———Activity and Growth ——— Figure 1. Hypothetical rate of conversion of dietary energy to growth approximate. Furthermore, there is no information about the effect of other nondietary, growth-stunting influences such as morbidity or endemic parasites on this curve. RESULTS Table 3 gives the relationship between four categories of calorie ingestion from the supplement by children aged 1 to 3 and their growth in height and weight. The increasing values of growth in height by category of supplementary energy intake are statistically significant (F = 20.8, df = 3,452, P = 0.001) and biologically important (figure 2). The lower categories are, as we might expect, essentially at the baseline levels given in table 1, while the top category shows a growth rate in which over half the deficit has been eliminated. Finally, we note that prevailing levels of morbidity make growth at the maximal rates unlikely (Martorell, Habicht, Yarbrough, et al. 1975a; Martorell, Yarbrough, Lechtig, et al. 1975). Essentially the same situation applies to growth in weight (F = 18.7, df = 3.452, P =< 0.001) (figure 3). Table 3 shows the overall rate of conversion of calories to growth by level of intake. These rates were obtained from multiple regression analyses in which we calculated the partial slope of growth on 202 EVALUATION OF CHILD HEALTH SERVICES Table 3. Relationship of Supplemental Caloric Intake and Growth from Ages 1to 3 Level of Ingestion (kcal/day) 101 0-10 11-50 51-100 and up Total Sample size 66 . 169 76 145 456 Height increment (cm)? 15.7 16.5 17.2 18.3 17.1 (*2.7) (2.6) (*2.4) (2.2) (*2.6) Weight increment (cm)? 3.67 3.85 4.05 4.50 4.06 0.98) (£0.89) (£0.76) (£0.94) (+0.95) Supplemental calorie ingestion (kcal/ day) 5.6 26.7 73.6 200.0 86.2 Slope of growth in height to calorie — 2.9 23 0.1 1.3 ingestion (mm/10* kcal)? (2.6) (*2.8) (17) (0.2 Slope of growth in weight to calorie _— 124 44 11 47 ingestion (gm/10* kcal)? (+87) (+84) (x14) 9) @ Mean (+ standard deviation) E 19} S £ 18} £ o 17+ QO £ 16 £ ® 151 ok. | | | 0 50 100 200 Supplemental Calorie Ingestion (kcal/day) Figure 2. Growth in height of children ages 1 to 3 as a function of supplemental calorie ingestion Response of Nutritional Indicators to Interventions 203 € [+] E 5 5D 4 —— . Ex S 3 2 1 1 ] 1 50 100 200 Supplemental Calorie Ingestion (kcal/day) Figure 3. Growth in weight of children ages 1 to 3 as a function of supplemental calorie ingestion calories, controlling for initial size. Controlling for initial size means that negligible influences of size at 1 year of age (correlation coefficient of weight at 1 year of age and subsequent gain in weight is r = —0.094; P = 0.10; and for height and subsequent growth in height, r = 0.003, P = 0.10) which might affect subsequent growth have been taken into account and, therefore, size at 1 year of age cannot be an explanation for the findings. Similarly, one would like to be sure that no factors other than the nutrient intake from the supplement could be the cause of the relationship between supplement ingestion and growth. Factors which influence a person to cooperate with a study could also affect growth quite independently of any benefits received from the supplementa- tion. To reject this possibility, one must introduce measures of cooperation into the energy-to-growth regression and show that they do not affect the findings in table 3. An unequivocal measure of cooperation with the supplementation program is the amount of energy ingested by drinking the supple- ment. Infant calorie ingestion from the supplement is associated (r = 0.72; P = 0.01) with calorie ingestion from 1 to 3 years of age and, therefore, infant consumption should reflect many of the influences which affect later supplement consumption. Including infant calorie ingestion from the supplement in the regression relating growth to the energy consumption from 1 to 3 years of age had no influence on the energy-to-growth conversion slopes in table 3. This was reflected by statistically nonsignificant partial correlation coefficients, r », of infant energy intake from supplementation into the energy-to- growth conversion equation (for growth in weight r, = 0.05, P = 204 EVALUATION OF CHILD HEALTH SERVICES 0.10; for growth in height r , = —0.02, P = 0.10). Other measures of cooperation are the number of days a child attended the supplemen- tation center and the amount of supplement ingested. Introducing these measures of cooperation separately into the regression analysis of the effect of calorie intake on growth revealed that neither days attended (r, = 0.01 for growth in weight, P = 0.10; r, = 0.06 for growth in height, P = 0.10) nor amount of supplement ingested (r = —0.05 for growth in weight, P = 0.10; r, = 0.00 for growth in height, P = 0.10) had any significant effect on the slopes in table 3. When all these indicators of cooperation were entered together into the regression, thus taking them all into account at once, the rates of energy conversion to growth were essentially the same as those shown in table 3. The energy-to-weight gain was 64 gm/10* kcal (P = 0.01) and the energy-to-height gain was 1.6 mm/104 kcal (P = 0.01) when all the above measured indicators of cooperation were taken into account. Therefore, the significant effect of calories on growth could not be explained by anything except the nutrients in the supplements themselves. These analyses assume that there is no substitution of the supple- ments for the home diets. In other words, every calorie ingested from the supplement is a calorie added to the dietary mean at every level of supplementary intake. This is in fact almost the case, because the dietary intake decreases only 1 calorie for every 10 calories of supplement ingested. Such a small dietary replacement effect of the supplement does not affect the figures in table 3. Finally, as expected, there are statistically significant nonlinearities in the overall rate (P = 0.002), which is reflected in the differences in the slopes with category of consumption as shown in figures 2 and 3. In other words, the more calories ingested from the supplement, the less the efficiency of conversion into growth. UTILIZATION OF RESULTS The purpose of the analyses was to confirm that the intervention used in this study did have an impact on the nutritional status of the child, an aim which would seem to be achieved. We must note, however, that what is being confirmed is that the supplemental food, when it is being consumed, has an impact on the child’s nutritional status. This is not the same thing as confirming that the intervention affected an important number of children. That would require a different study design, one contrasting the villages with other nonsup- plemented villages, which was not done. Even so, the small number of children in the highest supplement category makes it clear that our intervention was not very successful as an effort to treat Response of Nutritional Indicators to Interventions 205 populations, despite its success as a confirmation that an intervention had effect. Indeed it gave biologically meaningful utilization rates which were in accord with our prior notions of diminishing returns. DISCUSSION Three conclusions that are pertinent to the evaluation of child health services may be drawn from our presentation. The first is already well known: The evaluation of an intervention should depend on its purposes. As a confirmation of the biological effectiveness of the supplementary feeding, the study is a success. Were this program to be evaluated as a public health intervention to improve the growth rates of the children in the four villages, however, it would not rate very highly, either absolutely or cost- effectively. This is not surprising, because the program was not designed for this purpose. The second conclusion is that in the area of nutritional indicators the adequacy with which the efficacy of an intervention can be evaluated is sharply limited by the state of present knowledge. This results in uncertainties as to program effectiveness and efficiency. Careful design can perhaps in part face this problem, but most interventions will be practically rather than scientifically designed. For these, the lack of useful information as to what the response should be will severely limit evaluation. The lack of adequate baseline data is of course often a limiting factor as well, but currently the most serious limitation to evaluation is the absence of more theory based on experimental data from field studies which can be applied to make numerical predictions of response. This study, for instance, revealed a constraint on growth, due to nondietary factors, which corresponds in magnitude to the deleterious effect of diarrhea on growth (Martorell, Habicht, Yarbrough, et al. 1975a; Martorell, Yarbrough, Lechtig, et al. 1975). The third conclusion is that in many cases there is an inevitable tension between demonstration of an impact of a multifaceted intervention and the evaluation of its components. The confirmation that an agent has had an effect requires either an unambiguous prediction, which current theory cannot provide, or a range of variation in the treatment which will lead to less than maximal possible impact on the population as a whole. The fact that practical interventions tend to introduce simultaneously such variables as nutrition, education, medical care, and economic changes, increases the complexity of this problem manyfold. Thus we see the absence of a careful theory of nutrition and physiological response to nutrition as a fundamental practical limitation to the evaluation of programs, and not just a scientific nicety. 206 EVALUATION OF CHILD HEALTH SERVICES REFERENCES Canosa, C.A., Salomon, J.B., and Klein, R.D. (1972) Field studies of malnutrition and child development—The intervention approach: The Guatemalan study. In Nutri- tion, Growth, and Development of North American Indian Children, Moore, W. M., Silverberg, M. M., and Read, M. S. (eds.), pp. 185-199. DHEW Publication No. (NIH) 72-26. Washington: U.S. Government Printing Office. Habicht, J-P. (1973) Delivery of primary care by medical auxiliaries: Techniques of use and analyses of benefits achieved in some rural villages in Guatemala. In Medical Care Auxiliaries, PAHO/WHO Scientific Publication No. 278. Washington, D.C. Habicht, J-P, and Butz, W. In press. Measurement of health and nutrition in large- scale nutrition-related interventions. To be published in conference proceedings of Measurement of Nutrition and Related Health Programs in Latin America. PAHO. Habicht, J-P, Martorell, R., Yarbrough, C., et al. (1974) Height and weight standards for preschool children: Are there really ethnic differences in growth potential? Lancet 1:611. Jelliffe, D.B. (1966) The assessment of the nutritional status of the community (with special reference to field surveys in developing regions of the world). WHO Monograph Series No. 53. Geneva: WHO. Klein, R.E., Habicht, J-P, and Yarbrough, C. (1975) Some methodological problems in field studies of nutrition and intelligence. In Nutrition, Development and Social Behavior, D.J. Kallen, ed. DHEW Publication No. (NIH) 73-242. Washington, D.C.: U.S. Government Printing Office. Lechtig, A., Habicht, J-P, Delgado, H., et al. (1975) Effect of food supplementation during pregnancy on birth weight. Pediatrics 56:508. Lechtig, A., Yarbrough, C., Delgado, H., et al. (1975) Effect of moderate maternal malnutrition on the placenta. American Journal of Obstetrics and Gynecology 123:191. Marsden, P.H. (1964) The Sukuta project. A longitudinal study of health in Gambian children from birth to 18 months of age. Transactions of the Royal Society of Tropical Medicine and Hygiene 58:455. Martorell, R., Habicht, J-P, Yarbrough, C., et al. (1975a) Acute morbidity and physical growth in rural Guatemalan children. American Journal of Diseases of Children 129:1296. Martorell, R., Habicht, J-P., Yarbrough, C., et al. (1975b) The identification and evaluation of measurement variability in the anthropometry of preschool children. American Journal of Physical Anthropology 43:347. Martorell, R., Lechtig, A., Yarbrough, C., et al. (1976) Protein-calorie supplementation and postnatal physical growth: A review of findings from developing countries. Archivos Latinoamericanos de Nutricion 26:46. Martorell, R., Lechtig, A., Yarbrough, C. et al. In press. Relationships between food supplementation, home dietary intake and physical growth. American Journal of Clinical Nutrition. Martorell, R., Yarbrough, C., Lechtig, A., et al. (1975) Diarrheal diseases and growth retardation in preschool Guatemalan children. American Journal of Physical Anthro- pology 43:341. Martorell, R., Yarbrough, C., Lechtig, A. (1976) Upper arm anthropometric indicators of nutritional status. American Journal of Clinical Nutrition 29:46 McCammon, R.W. (1970) Human Growth and Development. Springfield, Ill.: Charles C. Thomas. National Academy of Sciences/National Research Council. (1974) Recommended Dietary Allowances. Food and Nutrition Board. 8th ed. Washington, D.C.: National Academy of Sciences. World Health Organization (1973) Energy and Protein Requirements. Report of a Joint FAO/WHO Ad Hoc Expert Committee. WHO Technical Report Series No. 522, FAO Nutrition Meetings Report Series No. 52. Geneva: World Health Organization. Yarbrough, C., Habicht, J-P., Malina, R. (1975) Length and weight in rural Guatemalan Ladino children: Birth to seven years of age. American Journal of Physical Anthropology 42:439. Response of Nutritional Indicators to Interventions 207 DISCUSSION Maurice Backett and Barbara Starfield The objective of this study was to examine the effect of a caloric supplement on height and weight of children whose anthropo- morphic measurements fall below standards for North American children. As such, it is an effectiveness study because it examines the relationship between a process of care and the result of that process on health status. The major limitation of the study was its possible inappropriate- ness. The goal of improving height and weight to North American standards was not one expressed by the community, but rather one derived from the researchers. Whether the failure of the research to “sell” the intervention (supplementary caloric feeding) resulted from lack of community participation or whether it was due to other factors is unclear. Thus the researchers were faced with a situation where the intervention was efficacious (it produced the desired result under carefully controlled conditions), but not effective (in the community setting, it could not be implemented). The applicability of the intervention was considered inappropriate, because the intervention did not appear to be a useful one in the community. There are, however, lessons to be learned about the method of research. If it is the intention of the researcher that the results of a study be applied, it is necessary that the goal be explicitly recognized as legitimate by those for whom it is intended. The goal of generalizability in this research was attained at the expense of implementability of the intervention. The latter might have been realized by participation of the target population in its planning. mn jal e——— Ew Fat © BE 14 CHILD HEALTH AND THE COMMUNITY: THE ROCHESTER CHILD HEALTH STUDIES, 1967-1977 I. B. Pless Summary The broad goal of these studies is to improve the health of all children in a defined community; more specific objectives are the reduction in mortality at all ages and the prevention (both primary and secondary) of morbidity from a wide variety of common childhood disorders. To achieve these ambitious aims, new health services were developed and evaluated and, through health educa- tion, attempts were made to change behavior in health and illness. The principal basis for these studies was continuous monitoring of the health needs of children and their utilization through periodic household surveys of a population sample in a single county in upstate New York. The results of these surveys were utilized by the department of pediatrics which sponsored them, by practitioners in the community, by local health planners, and by medical educators, practitioners, and policy makers elsewhere. Introduction The series of studies conducted in the Rochester region under the auspices of the University of Rochester Department of Pediatrics extended over a period of nearly 10 years. It is our only example in which a university department has conducted a program of research as opposed to a specific project or projects. This program is also special because it spanned a period of many years and was directed toward a common goal. It thereby provides an example of some advantages of this approach and also illustrates some of its limitations. If we agree that the main purpose of research is to bring about change, then this and other examples of child health studies must be examined in relation to the question: To what extent did the effort produce the desired change? 209 210 EVALUATION OF CHILD HEALTH SERVICES In the case of the Rochester studies, the change in question was the provision of more and better services. Ultimately they also aimed at the broader, albeit less realistic, goal of improving the health of all children in the community. It is difficult to explain how better health outcomes can be related to improving the processes of care, when much current evidence fails to support such a relationship in most advanced societies (Cochrane 1972). However, if one’s definition of health is broad enough, and if the entire context in which the program is conceived is also broad enough, we may argue that a community is healthier when its health services are more rational and more efficient and deliver a better product. This may be true whether or not the health of its citizens demonstrates improvement when traditional measures of physical or emotional well-being are used. Purpose of Study The impetus for the Community Child Health Studies Program came from Dr. Robert Haggerty when he became chairman of the Department of Pediatrics in 1964. Because of Dr. Haggerty’s commit- ment to community pediatrics, it was a logical first step for him to initiate studies that would provide both a sensitive assessment of areas of unmet needs and a baseline against which future programs could be evaluated. He realized that the best way to obtain the required information was to find a capable sociologist with special skills in survey methodology. He chose Dr. Klaus Roghmann, who has undoubtedly been the methodological backbone of the research program. The initial intention was simply to carry out a sample survey of all households in the county to study the health needs and utilization of services of children in those households. The community, in this and most of the subsequent studies, was defined as Monroe County—an interesting, heterogeneous area surrounding the city of Rochester, in upstate New York. In composition, the population—about three quarters of a million—is typical of many other communities of similar size in the United States. However, this community differs from others in some aspects of health planning and services. One particular feature of services planning that makes it atypical is its long record of community participation in health planning, particularly regarding hospital beds. Prior to 1964, a team of external advisers recom- mended that pediatric hospital beds be redistributed to avoid dupli- cation and obtain greater efficiency (Stokes 1965). This recommen- dation was implemented when Dr. Haggerty arrived. Several small units were closed, and specialized services were distributed among the community hospitals and the university hospital. In addition, a Rochester Child Health Studies 211 close relationship was forged between the practicing community and the department. Clinical faculty appointments were offered to many of the practitioners. Subsequently, efforts were made to involve them in collaborative research. The effect of this move was to diminish the “town/gown” separation that is so frequently a characteristic of university communities, thereby increasing the likelihood that changes initiated by the department would be acceptable to those in clinical practice. In the remainder of this case presentation I shall outline the methods involved in the surveys, describe in general terms what the program of research encompassed, illustrate some of the major findings, and indicate the extent to which they appear to have been implemented. Areas of Child Health Care This series of studies has dealt broadly with the entire spectrum of childhood morbidity and health services provided for children in one geographically defined area. Some aspects have received more atten- tion than others: organization and utilization of services, school problems, behavioral disorders, chronic illness, and health education. The focus emphatically has not been on a single problem, but on the relationship between several common problems in child health and the care provided. Target Population Over the first 7 years of the study, the child population of the country rose from 220,000 to 250,000, with the number of black, predominantly poor, children increasing from 37,000 to 65,000. As with many similar American communities there was a continuous migration of whites toward the suburbs, with blacks remaining in the city. Within the city, two poverty areas, defined both by census tracts and by sociopolitical subdivisions (or wards), were of particular interest. Here the black population rose from 61 percent to 77 percent in the 5-year period beginning in 1966. Also of interest was the western section of the county, to which about 1,000 migrant farm workers came annually, often with their children. Study Design The most common methodology utilized throughout these studies is the sample survey (Warwick and Lininger 1975, pp. 46-68). In 1967, 1969, 1971, and again in 1975, a sample varying between 1 212 EVALUATION OF CHILD HEALTH SERVICES percent and 0.5 percent of all households in the county was studied systematically. These household surveys focused primarily on chil- dren but varied in their emphasis. Generally, they dealt with the frequency of symptoms as an expression of health needs and then examined patterns of utilization. The variations included specific sections dealing with attitudes toward health care; the use, in one instance, of a 28-day health calendar or diary for recording stressful events and medical care utilization; school health; behavioral prob- lems; and a series of studies on the care of children with chronic illnesses. The core questions in each of the surveys were sufficiently similar to allow comparisons to be made over time. Accordingly, conclusions could be drawn about the effect of changes that had been introduced, either by the program itself, or by local or national events. Further- more, to evaluate specific programs such as the neighborhood health center (NHC), various sections of the community were oversampled and modifications in the design were made or experiments conducted as the need arose. The random sample and the household interview were used throughout. The interview contained questions on illness or injury sustained in the preceding 2 weeks, the use of prescribed and nonprescribed medications, the use of medical care services (during a 2-week and a 12-month interval), hospitalization, the presence of long-term or major illness and, in the most recent version, questions about reproductive behavior and attitudes. (Earlier versions also contained a series of questions on attitudes towards illness and doctors.) A number of related studies on specific topics were built around this common core. Figure 1 presents a grossly simplified picture of the core, or first generation, studies. The lower part shows the concurrent or subsequent substudies during the first 5-year period. Figure 2 illustrates the program of research during the second 5-year span. Initially this. work was supported by the U.S. Children’s Bureau (Grants H-104 and 148) and subsequently by the National Center for Health Services and Research Development (USPHS Grants HS-467 and HS-294). The bulk of the work was carried out from the Rochester Child Health Studies (RCHS) offices—a division which operated as a major component of the Department of Pediatrics research program. Dr. Haggerty was the principal investigator; Dr. Roghmann and I and three other senior faculty served as project directors. The team also included full-time research assistants, two field supervisors, many part-time interviewers and coders, and two full-time programers. The annual budget for the studies during their peak years averaged DATA BASE Main Surveys 1967, (69), 1971 Unmet health needs Distribution of services Stress and utilization Evaluation of Medicaid Evaluation of abortion law | COLLABORATIVE Nurse practitioner trial WBC in office practice Nurse practitioner program Figure 1. Rochester child health studies. CORE PROJECTS i y BEHAVIORAL CHRONIC ILLNESS Preschool Prevalence School Psychosocial adjustment Adolescent | SECONDARY (SPINOFF) STUDIES Psychodiagnostic lab APPLICATIONS Psychodiagnostic School health program Intervention trial Phase I: 1967-72 Family functioning index Family counselor trial Automobile safety Family counselor program SAPMIS YI[EIH PIIYD IA$Y20Y §1é 214 EVALUATION OF CHILD HEALTH SERVICES DATA BASE Household interview (1975, 1977) Medicaid files Vital statistics Etiology of respiratory allergies Encounter forms Effects of genetic counselling Census summary tapes Alpha-1 antitrypsin deficiency ER utilization H. influenza vaccine trial Hospital discharge abstracts CORE PROJECTS I EVALUATION STUDIES Attitudes of consumers (satisfaction) HMO enrollment choice Utilization and cost analysis Population projections 1 COLLABORATIVE RESEARCH Antibiotic compliance Telephone management Serous otitis decongestant trial Implementation of research ER trends SPECIFIC PROJECTS 1 I I 1 BEHAVIORAL HEALTH EDUCATION CHRONIC DISEASE Followup of 1969 2 year olds Seat restraint trial Followup 1968 group Parent education in practice settings Bicycle safety Nephrotic siblings SECONDARY (SPINOFF) STUDIES I | Role of pediatrician in mental health Study of followup methods Family knowledge Followup of difficult temperament Figure 2. Rochester child health studies. Phase 11: 1972-77 at least $100,000. An additional $200,000 or more was spent in support of the clinical training program aimed at residents and fellows in ambulatory pediatrics. The average direct cost per com- pleted household interview ranged from about $10 to approximately $40. Five or six projects were concurrently under way at any given time during a typical year. This was only possible because the senior investigators were full-time workers and had a staff devoted exclu- sively to these projects. Under other circumstances such as heavier clinical commitments or additional administrative duties, the output would have been much smaller. The program relied heavily on computer technology. During the latter years four programers were needed to carry the load. The major subsets of the studies, directly related to the principal theme, fell into several distinct areas: new manpower studies, particularly concerning nurse practitioners; a variety of other collaborative re- search efforts involving physicians in practice; studies of organiza- tional change, such as the NHC; and finally, studies of the care of children with chronic illnesses. In addition, attempts were made to develop a model for evaluating health education programs presented in a pediatric setting. Many of the studies were done in collaboration with nurses, other health practitioners, sociologists, psychologists, administrators, and health planners. They frequently provided an essential resource for colleagues with purely biological interests, such as the study of alpha- Rochester Child Health Studies 215 1 antitrypsin deficiency, genetic disorders amenable to amniocentesis, hemophilus influenza vaccine, or the etiology of allergic disorders. Methodology The surveys, by means of standardized interviews and random samples of families with children, provided the bulk of the informa- tion obtained. They were conceived as a series of linked interrelated studies based on the strategy of developing a broad system of data collection; the results were regrouped in order to answer a wide range of questions. Using the intermittent sequence of cross-sectional studies, we could measure the effects of both planned and unplanned changes in health care. A major design problem _in this type of research may arise when one wishes to generalize from the findings. Information obtained in the interviews with the 3,000 families was supplemented by data available in Medicaid and Blue Cross insurance payment (or claims) files, birth and death statistics, manpower inventories, and census summary tapes. In the latter years, we also relied increasingly on interviews or questionnaires from physicians in practice. Together the data provide a picture of the total system. The community itself, however, is both typical and atypical in many respects, and this must be taken into account when applying the results elsewhere. Although the major surveys do not tell the complete story of this program, they merit additional details. In the 1967 survey there were no available sampling frames of children apart from the annual school census conducted exclusively in the suburban districts. There- fore, in the city, a list of 100,000 households was made and four sequential samples were drawn, using every 3,000th address as a starting point of a 10-residence segment or cluster. Each home was visited to see if any children resided there. A maximum of two children were selected randomly from among those available in the home. The suburban list permitted a simpler approach, using every 200th child. Thus the city procedure yielded an unbiased family sample, whereas the suburban approach gave an unbiased child sample; hence correction procedures were needed in the analysis (Roghmann and Haggerty 1970). Because only two children were selected, a further sampling variation was introduced, since the fraction necessarily varied from family to family. The 1967 survey resulted in about 94 percent completed interviews at an average cost of $10 per interview and covered a total of 1,402 children. The 1969 survey was a smaller midpoint study (0.5 percent of all families with children under age 18). Some groups were oversampled (1 percent) to allow for special studies of particular interest. In this 216 EVALUATION OF CHILD HEALTH SERVICES study, Blue Cross and Medicaid lists were used as a sampling frame. In addition, we included all pediatric admissions to the largest area hospital who did not appear on either list. Systematic random sampling proceeded until 512 families were selected. By using an abbreviated child questionnaire, we could obtain information about all children in each family, although most analyses were confined to the mother and her youngest child. Each mother kept a 28-day health calendar (or diary) to record stress, medications, and symptoms (Roghmann and Haggerty 1972). Eighty-two percent of the interviews were completed in spite of the increased demands made on respond- ents. The average cost per interview was $40 because of extra travel, coding, and the payment of a nominal sum to each respondent for completing the health diary. The 1971 survey was similar to that in 1967, but the sampling made use of the housing unit list from the 1970 census. Five l-in- 1,000 systematic random sample clusters of addresses were selected as starting points, actual names and addresses being obtained from the computerized file of a local utility company. A total of 1,216 families with 2,952 children were interviewed; an 82 percent comple- tion rate was recorded at an average cost of $13 per family interview. The 1975 survey was again similar but was aimed at only 750 families with approximately 1,800 children. Data and Information Systems The results obtained were of many kinds and were interpreted in several different ways. The model of the community health system shown in figure 3 may help categorize them in a conceptually meaningful fashion (Haggerty, Roghmann, and Pless 1975, p. 8). The central issue is the ability of the care system to interact with health needs of the population so as to result in appropriate utilization, leading to compliance and hence to improved health. Accordingly, data were obtained about some aspects of social background and risk factors, such as stress and genetic abnormalities, health status (as assessed by disease, disability, and discomfort), perceived needs, and actual demands for care. We also assembled data about various health manpower resources, health agencies, their organizational forms, and the manner in which services were made available to those who required them. Indicators and Measurement Criteria Both simple and complex criteria were used. Simple criteria included birth and death rates and the prevalence of various PUBLIC HEALTH MEASURES INDIRECTLY INFLUENCING HEALTH IN THE COMMUNITY NO IMPROVEMENT OF HEALTH Manpower, Skills Physicians IMPROVEMENT OF HEALTH Other Health Professionals \ \ [ Non-Professional Genetic Objective Need as Conditional Conditional Environment Health Perceived Demand for Offer of Exposure to Status: in Services Services Organi- stress | disease | Populati w| space space zation of traffic “| disability La “1 time time Care hazards discomfort expense expense bad diets social social smoking, etc relations relations I Hospital Facilities Health Planning Agency NO NO NO UNMET UNUSED INEFFECTIVE Health Insurers ABNORMALITIES PERCEPTION TRANSLATION DEMAND SERVICES ORGANIZATION Provider OF NEED INTO DEMAND A ations Xe J J Yo Y Health Department THE SETTING AND THE PROBLEMS MEDICAL CARE FOR CHILDREN Figure 3. Model of the community health system [Reproduced with permission from Haggerty, R.J., Roghmann, K.J., and Pless, 1.B. Child Health and the Community, copyright 1975, John Wiley and Sons. Further reproduction prohibited without permission of copyright holder.] SAPS I[EIH P[IYD 12159420Y L1G 218 EVALUATION OF CHILD HEALTH SERVICES disorders or conditions. For the most part, parents’ reports that a disease or symptom was present were accepted at face value without confirmation from a physician. The use of complex indicators was usually a result of being faced with the need to operationalize some abstract or imprecise concept. Examples of the latter were health supervision (Hoekelman and Peters 1972), maladjustment (Pless, Roghmann, and Haggerty 1972), family functioning (Pless and Satterwhite 1973), total family impact, stress, and crisis (Roghmann and Haggerty 1973). In each case, various approaches to index information were employed, such as aggregating several indicators with or without simple weighting, and only occasionally using factor analysis or other statistical techniques to generate weights. At no point did we attempt to deal with the formidable challenge of attempting to develop a child health status index for the whole population, although such a measure would clearly have been of immense value in assessing the total impact of the program. Type of Evaluation Virtually the entire repertoire of evaluation strategies was used. At one stage in studying the utilization of services, both provider and survey data were linked to inputs and health manpower resources. Efficacy, effectiveness, and to a lesser extent, efficiency were consid- ered in the studies of the health center, the nurse practitioner, the family counselor, and several other programs of care. Examples of cost-benefit analyses, in the traditional sense, were isolated. The few direct attempts at measuring “quality” were based on assessments of short-term outcomes. However, for the most part, the primary objectives were directed more toward assessing quantitative aspects of the delivery of health services. Of interest in this regard are the findings that, based on the number of medical care contacts, utiliza- tion of ambulatory care services declined over the period from 1967 to 1971 and similarly, hospital inpatient occupancy rates fell steadily. Results The main results of these studies are described in a book, Child Health and the Community (Haggerty, Roghmann, and Pless 1975), as well as in approximately 50 scientific publications. Since it is impossi- ble to do justice to the wide range and depth of findings obtained, only a few of the more important have been selected for presentation in this section. In the first 5-year period the following results were achieved: Rochester Child Health Studies 219 —The surveys estimated the prevalence of unmet medical needs of children, and the uneven distribution of services in the community were documented with reasonable precision (Haggerty 1970). —The household surveys revealed the dimensions of the “new childhood morbidity” (Haggerty, Roghmann, and Pless 1975, pp. 94-113). —Health and illness behavior were related to family crises, with stress of various kinds increasing utilization of health services, provided that the site of care was easily accessible. If barriers such as appointment systems were imposed, stress appeared actually to decrease utilization; in both situations the effects differed in mothers and children (Haggerty, Roghmann, and Pless 1975, pp. 142-154). —An experimental study demonstrated the effectiveness of the pediatric nurse practitioner in the office setting, along with the acceptability, financial feasibility, and impact on quality of care of this new form of health manpower (Charney and Kitzman 1971). —The survey documented the distribution of chronic illness among children and the frequency of secondary psychosocial dysfunc- tion, particularly as a reflection of family functioning (Pless and Roghmann 1971). A regional survey of physicians in primary care who must deal with these children revealed several critical problem areas (Pless, Satterwhite, and Van Vechten 1976). —Methods were developed for identifying behavioral problems in children age 2, and the problems were correlated with parenting techniques (Chamberlin 1974). A controlled trial to test hypotheses about the reinforcement of such problems by inappropriate paren- tal responses was initiated. —Studies of suburban adolescents showed the need for specialized counseling programs. —Controlled studies of the NHC showed a decrease in hospitalization (Klein et al. 1973) and emergency room use for those registered (Hochheiser, Woodward, and Charney 1971). —Studies following the introduction of Medicaid in New York State showed few changes in the place, frequency, or patterns of care for the poor and little basic alteration in the pre-existing “two-class” system of care (Roghmann, Haggerty, and Lorenz 1971). —The continuous data base permitted the effect of change in the New York State abortion law to be studied, showing a decline in total births and in births of high-risk babies, thus reducing the incidence of prematurity and infant mortality (Roghmann 1975). —The collaborative research program assessed the value of white blood cell counts in predicting the results of treatment of children with pneumonia (Shuttleworth and Charney 1971); it examined the 220 EVALUATION OF CHILD HEALTH SERVICES correlation of clinical and other factors related to compliance with oral penicillin prescriptions (Charney et al. 1967); later it showed the effectiveness of using long-term sulfa in the treatment of recurrent otitis media, again through a controlled, double-blind experimental trial (Perrin et al. 1974). Utilization of Results In an attempt to evaluate our progress, we decided to determine the extent to which two of the program’s publications (Perrin et al. 1974, Kanthor et al. 1974) influenced the thinking and behavior of practitioners in the community. All practitioners received a question- naire about their awareness of the results of these studies (both had been published in reputable journals) and the extent to which those who had read them were prepared to implement the major recom- mendations. Fewer than half of the more than 70 practitioners surveyed were both aware of and implementing the results of either or both studies. Thus the second stage involved either a letter of encouragement from the department chairman, along with a copy of the reprint in question, or personal visits by our director of commu- nications to a second, randomly selected portion of the group. A follow-up questionnaire showed only a small increase in reported intentions to adopt the recommendations, with more of those who had been visited willing to do so than those who had not. These conclusions, which suggest that more than publication in widely read journals is needed to persuade physicians to utilize the findings of medical research, must also be interpreted as evidence of how little we know about other, more successful, methods of persua- sion. They indicate the problem at the “micro” level, that is within the immediate target community. It is conceivable that greater impact may have occurred elsewhere within the scientific community or among practitioners in other parts of the country. Clearly, however, we know little about the factors that influence the diffusion of new ideas, whether or not they stem from research findings. As investiga- tors, we appear to care little about this problem. Perhaps there is no way of knowing what influence any of the work in Rochester has had on the thinking of others. Counting reprint requests is a crude and unreliable indicator of anything other than the popularity of a title or a particular subject at a given time. Upon reflection, we suspect that there are a number of areas in which the impact of this program was probably significant in spite of the pessimism reflected in the preceding section. One such area is the introduction of the idea that there is a new morbidity in North American pediatrics (Haggerty 1968). We were not the first to call Rochester Child Health Studies 221 attention to the prominence of such common problems as school or behavioral disorders, or the difficulties related to the management of the chronically ill and handicapped; however, our having coined and popularized the term “the new morbidity” appears to have persuaded others to address these issues more intensively. Secondly, the work in Rochester, combined with that of others elsewhere in the area of health manpower (particularly the work of Dr. Charney in the nurse practitioner studies (Charney and Kitzman 1971) and the use of nonprofessional family counselors to assist in the care of the chronically ill (Pless and Satterwhite 1972), appears to have also aroused considerable interest. Third, at a time when the value of these centers was still debatable, the work demonstrating the worth of the NHC has undoubtedly influenced the thinking of others. Fourth, drawing attention to the special needs of the chronically ill, with respect to possibilities for prevention of secondary psychological and psychosocial problems, appears to have focused increasing attention on this previously neglected area. Finally, there was the achievement of the most important, but least explicit, objective: to provide an example to other pediatric depart- ments of how an academic department could develop a scientific program in community pediatrics. This result is the most perplexing and least clear-cut. On the one hand, it appears from an analysis of abstracts submitted to various public health or pediatric societies that this department has indeed played a leading role in the evolution of community pediatrics. More and more graduates are seeking fellow- ship level training in this specialty and it is reasonable to assume that this growing interest was stimulated in part by the achievements of the Rochester program. On the other hand, no other department has tried to emulate this example with respect to its research commitment. Whether this should be interpreted as an indication that, in their judgment, the effort is not worthwhile, or whether they lack the resources, financial or personal, we cannot say. At the clinical level, it seems unlikely that these efforts have substantially improved the health of children in the county itself, let alone that of children elsewhere. It has, however, provided some important tools for doing so and others for assessing changes in health and health care research methodology. It demonstrated that with ingenuity, effort, and some skill, it is possible to apply the randomized clinical trial to many unconventional problems. It dem- onstrated the usefulness of the 28-day calendar in studying relation- ships between stress and utilization of health services. This approach helped clarify some of the basic issues in utilization models such as those proposed by other investigators: Suchman (1967), Andersen 222 EVALUATION OF CHILD HEALTH SERVICES (1968), and Rosenstock (1966). As a research tool for assessing the quality of family life, the Family Functioning Index (FFI) may have clinical applications as well (Pless and Satterwhite 1973). The feasibil- ity of using Medicaid and Blue Cross data tapes for the analysis of utilization patterns over time and between groups has been demon- strated. Several basic problems in methodology involved in sampling children in households have also been clarified. Toward the end of the 10-year period the program has increasingly influenced local health planners. At least some of the ideas in the region are indirectly a product of data produced by the RCHS. The collaborative research program continues to flourish, and increasing numbers of practitioners are becoming involved in clinical projects that draw upon the expertise of the investigators in the RCHS group. At the national level it seems quite likely the papers dealing with the supposed “medical care crisis” (based on an analysis of utilization data), the effects of the Medicaid program on care patterns for children, and the effects of the health center have all had an impact on the thinking of those responsible for policy formation. Discussion This program of evaluation of child health care differs from other case studies presented at this conference in several important re- spects: In the first place, it is based exclusively on a defined community; second, it has originated from an academic department; and third, it has attempted to answer simultaneously both theoretical and practical questions (table 1). How well it has succeeded in meeting these potentially conflicting goals is a question of some importance for a conference dedicated to examining the interface between research and medical practice. This work is based on a belief that academic departments have an obligation to deal with issues of concern to the community which the university has been created to serve. It may be argued that in such a setting the sole commitment is to the “community of scholars.” In this case, answering basic, fundamental (as opposed to “applied”) ques- tions is all that matters, and this orientation prevails in most areas of academic endeavor. However, in the field of health care research, specifically community pediatrics—a vital, rapidly developing disci- pline—this ivory tower approach cannot succeed. Most questions of relevance to health care investigators must be posed in real-life terms. They cannot be divorced from the ultimate test of the likelihood that the solutions they yield will be applied. Recognizing this issue, however, does not make the task any easier. If anything, it heightens the dilemma that confronts those who, from an Rochester Child Health Studies 293 Table 1. Guiding Principles of the Rochester Program of Child Health Research 1. Focusing of studies on defined populations (sampling permits generalizations) 2. Use of experimental and quasi-experimental designs whenever possible to permit “causal” inferences 3. Linking of theoretical and methodologic studies to practical problems to permit “relevance” and “science” to co-exist 4. Linking of research to teaching and service to enhance acceptability and long-term benefits Combination of sociomedical and biomedical aspects to enhance value of each Application of results to solution of problems at many levels. oo academic perspective, try to address problems whose ultimate solution depends only in part (and perhaps in very small part) on the acquisition of knowledge alone. Clearly social, economic, political, and other forces are at work. What is learned from research must interact with these forces before any significant changes in policies can be expected. While many of us who engage in this type of research have expertise in the techniques required to achieve scientif- ically respectable results, few have the skills, knowledge, or even understanding of the complicated processes whereby social change is brought about. We are therefore operating in a peculiar vacuum— one that may or may not be unique to this field of investigation. Even at a more basic level of simply diffusing and implementing new ideas which may be of value to those in clinical practice, our knowledge and skills are woefully deficient. It has been estimated that between 50 percent and 75 percent of clinical advances are never implemented. The exact figure is unimportant; what is significant-is the increasing realization by funding agencies, as well as the public, that we must become more accountable for research expenditures. Investigators need to recognize the limitations in their ability to “market” research results and begin to try to rectify these shortcom- ings. More energy must be devoted to examining this problem. Part of the explanation may lie in the fact that most studies originate from social scientists and tend to examine the diffusion and implementa- tion of research findings in a theoretical rather than a practical vein. It is difficult to convey fully the degree of interdigitation that has existed between this research program, the department as a whole, and the community. These studies have played a central role in the department, particularly with respect to its research and ambulatory care teaching programs. This aspect makes the RCHS so unusual and hence so important as a model for other clinical departments. In doing so, we must be candid about some of the problems associated with operating a program of this kind in a university setting. It is probably fair to say that many of our colleagues in other 224 EVALUATION OF CHILD HEALTH SERVICES clinical disciplines either could not understand, or were skeptical about, the emphasis placed on the RCHS. Many may even have resented the support it received, both internally and externally, and the opportunity this support gave some of us to engage in research on a full-time basis. Coupled with these internal problems were other, more basic, questions about the place of such a community-oriented applied research program in a clinical department. The university itself questioned the involvement of any part of the medical school in developing programs of care beyond those essential to the fundamen- tal purpose of training physicians or nurses. We counterargued that it is only by developing programs in the community that we provide realistic models for future students. And only through research which examines those models scientifically will disciplines such as commu- nity medicine and epidemiology gain useful knowledge and wide- spread relevance. The fundamental belief that a department of pediatrics must be actively engaged in the development and evaluation of community programs remains unchanged. We hope that many colleagues now agree. Unfortunately, however, only a few other departments of pediatrics in North America have followed the Rochester example. Conclusion In summary, it should be clear that this program has dealt not with one study but with many, all of which relate to a central theme. One overriding purpose of this conference is to examine child health evaluation methods that are applicable in different settings. Based on our experience, there seems no doubt that heavy reliance on the survey approach provides at least part of the answer. Undoubtedly, thoughtful design of surveys allowing an examination of effects over time or the establishment of appropriate control groups, permits conclusions to be drawn about how health care changes or how the health of children is affected by that care. It is equally apparent, however, that no single approach can deal adequately with the many diverse issues encompassed by the term “child health.” A universal health status index for children may ultimately be of value in work of this kind, but efforts thus far have shown this to be extraordinarily difficult to conceptualize, let alone put into operation. For the time being, therefore, we conclude that to evaluate child health adequately we must employ many different philosophies and approaches, con- stantly attempting to unify them under a single conceptual frame- work. Rochester Child Health Studies 225 REFERENCES Andersen, R.A. (1968) Behavioral Model of Families’ Use of Health Services. Chicago: Center for Health Administration Studies, Research Series. Chamberlin, RW. (1974) Authoritarian and accommodative child rearing styles: Their relationship with the behavior patterns of two-year-old children and other variables. Journal of Pediatrics 84:287-293. Charney, E., et al. (1967) How well do patients take oral penicillin? Pediatrics 40:188- 195. Charney, E., and Kitzman, H. (1971) The Child Health Nurse (Pediatric Nurse Practitioner) in Private Practice. New England Journal of Medicine 285:1353-1358. Cochrane, A.L. (1972) Effectiveness and Efficiency: Random reflections on health services. London: Nuffield Provincial Hospitals Trust. Haggerty, R.J. (1968) Community Pediatrics. New England Journal of Medicine 278:15-21. Haggerty, R.J. (1970) Research Issues in Child Health II, Some Medical and Economic Issues. Pediatrics 45:702-712. Haggerty, R.J., Roghmann, K.J. and Pless, I.B. (1975) Child Health and the Community. New York: John Wiley and Sons. Hochheiser, L.I., Woodward, K. and Charney, E. (1971) Effect of the neighborhood health center on the use of pediatric emergency departments in Rochester, New York. New England Journal of Medicine 285:148-152. Hoekelman, R.A., and Peters, ENN. (1972) A Health Supervision Index to Measure Standards of Child Care. Health Services Report 87:537-544. Kanthor, H., Pless, 1.B., Satterwhite, B. et al. (1974) Areas of responsibility in the health care of handicapped children. Pediatrics 54:779-786. Klein, M., Roghmann, K., Woodward, K. et al. (1973) The impact of the Rochester Neighborhood Health Center in hospitalization of children 1968-1970. Pediatrics 51:833-839. Perrin, J.M., Charney, E., MacWhinney, J., Jr., et al. (1974) Sulfisoxazole as chemopro- phylaxis for recurrent otitis media: A double-blind crossover study in pediatric practice. New England Journal of Medicine 291:664-667. Pless, 1.B. (1974) The Changing Face of Primary Pediatrics. Pediatric Clinics of North America 21:223-244. Pless, I.B., and Roghmann, K.J. (1971) Chronic illness and its consequences: Observa- tions based on three epidemiologic surveys. Journal of Pediatrics 79:351-359. Pless, [.B., Roghmann, K.J., and Haggerty, R.J. (1972) Chronic illness, family functioning, and psychological adjustment: A model for the allocation of preventive health services. International Journal of Epidemiology 1:271-277. Pless, I.B., and Satterwhite, B. (1972) Chronic Illness in Childhood: Selection, activities and evaluation of non-professional family counselors. Clinical Pediatrics 11:403-410. Pless, I.B., and Satterwhite, B. (1973) A Measure of Family Functioning and Its Application. Social Science and Medicine 7:613-621. Pless, 1.B., Satterwhite, B. and Van Vechten, D. (1976) Chronic illness in childhood: A regional survey of care. Pediatrics 58:37-46. Roghmann, K.J. (1971) The Impact of the New York State Abortion Law on Black and White Fertility in Upstate New York. International Journal of Epidemiology 4:45- 49. Roghmann, K.J., and Haggerty, R.J. (1970) Rochester Child Health Surveys I: Objectives, Organization and Methods. Medical Care 8:47-59. Roghmann, K.J., and Haggerty, R.J. (1972) The Diary as a Research Instrument in the Study of Health and Illness Behavior. Medical Care 10:143-163. Roghmann, K.J., and Haggerty, R.J. (1973) Daily Stress, Illness and Use of Health Services in Young Families. Pediatric Research 7:520-526. Roghmann, K.J., Haggerty, R.J., and Lorenz, R. (1971) Anticipated and Actual Effects of Medicaid on the Medical Care Pattern of Children. New England Journal of Medicine 285:1053-1057. 226 EVALUATION OF CHILD HEALTH SERVICES Rosenstock, I.M., Haefner, D.P., Kegeles, S.S., et al. (1966) Public Knowledge, Opinion and Action Concerning Three Public Health Issues. Journal of Health and Human Behavior 7:91-98. Shuttleworth, D.B., and Charney, E. (1971) Leukocyte Count in Childhood Pneumonia. American Journal of the Disabled Child 122:393-396. Suchman, E.A. (1967) Preventive health behavior: A model for research on community health campaigns. Journal of Health and Human Behavior 8:197-209. Stokes, J. (1965) Preliminary Report on Pediatric Facilities in Monroe County, New York (unpublished). Warwick, D.P., and Lininger, C.A. (1975) The Sample Survey: Theory and Practice. New York: McGraw Hill. Rochester Child Health Studies 22% DISCUSSION David Bersh and Carlos V. Serrano The Rochester evaluation program consisted of a series of studies conducted between 1967 and 1977. Evaluation, however, was merely a single component of a broader demonstration program whose overall objectives were to determine how community pediatrics— including community-based research and health planning—could be introduced into a clinical department of pediatrics, and to evaluate the impact of such a program on the community and, in particular, on the health of children. The evaluation component depended primarily on data collected through a series of home interviews supplemented with data from other sources. The program itself did not develop original evaluation tools, but rather applied and integrated the data resulting from a wide variety of previously existing, validated procedures and metho- dologies. There was an attempt to make appropriate use of the collected data to improve health status, to assess the internal success of the program’s components, and to determine the ability of the health sector to comply with health needs. A number of valuable studies were carried out on various subjects, such as unmet health needs, effectiveness and acceptability of the pediatric health practitioner, and chronic illness among children. The investigators were satisfied with the outcome of the internal program evaluation; however, questions were raised regarding the long-term impact of the endeavor. Although medical students were particularly attracted to the program, members of other clinical departments did not understand it, or were skeptical and resistant to the program and the importance it attached to community activities and community/ consumer perceptions. No other pediatrics department in the United States has committed itself to community involvement in such a systematic way, and even the Rochester program has not been continued beyond the demonstration phase. Additionally, the Rochester program faced certain problems which would make replication difficult. The cost (approximately $10 to $40 per interview) was obviously quite high, although it was noted that cost would present less of a problem in Latin America, where suryeys can be conducted at less expense through the participation of students. It was also difficult for the program to maintain ideological and methodological continuity over a 10-year period, because of normal turnover in professional staff. Finally, the community did not participate in the program in any significant way, inasmuch as funds were provided through the medical school rather than through community organizations. 228 EVALUATION OF CHILD HEALTH SERVICES Despite these caveats, the meeting consensus was that the final impact of the program in terms of education and service will not become apparent until after those students who were exposed to it are in a position to affect the delivery of pediatric care. The workshops considered the Rochester program a classic example of applied research, providing a realistic and invaluable learning expe- rience for future doctors, nurses, and other members of the health team. 15 HEALTH ACCOUNTING: AN OUTCOME-BASED STRATEGY OF HEALTH CARE QUALITY ASSURANCE John W. Williamson Summary This paper describes an outcome-based strategy for integrating the functions of health care quality assessment and improvement called “health accounting.” This approach depends upon skillful use of small group judgment methods in a five-stage procedure that begins with a structured method for identifying cost-effective topics for study (stage 1), and concludes by reassessing outcomes to document impact in terms of improvement (stage 5). This approach is illustrated here by 1 of the 74 health care outcome studies that have been implemented in the past 14 years by the author in multispecialty group clinics and/or their associated hospitals throughout the United States. Finally, the results of extensive utilization by other investigators and intensive reliability and validity evaluation tests of health account- ing by the author and his colleagues have established its feasibility and provided substantial supportive evidence for recommending its more widespread application. Purpose of Study The following demonstrates an outcome-based strategy for assess- ing and improving the provision of health care. The concepts encompassed were developed as part of the Health Accounting Project, which has evolved over the past 14 years in 23 multispecialty group clinics and/or their associated hospitals throughout the United States. During this time, over 277 study topics have been formally proposed, and 74 outcome-based health care studies have been completed. This work represents the author’s personal initiative. It has been financially supported by grants and contracts from the National 229 230 EVALUATION OF CHILD HEALTH SERVICES Center for Health Services Research in the U.S. Department of Health, Education, and Welfare. Private foundations, such as the Commonwealth Fund and the Milbank Fund, have also made major contributions. The results of this research and development have had direct influence on the policy development of quality assurance activity in the United States, namely the major strategies of the Professional Standards Review Organization (PSRO) and the quality assurance procedures of the Joint Commission for Accreditation of Hospitals (JCAH). This paper outlines the project, as illustrated by data from the hypertension studies in two health maintenance organizations (HMOs) in Los Angeles, California. Next, the influence of this approach and its application by other investigators are summarized. Finally, the advantages and disadvantages of health accounting strategy are briefly enumerated. Area of Health Care The topic of hypertension was selected because it was one of the earliest studied and provides the most dramatic illustration of the total health accounting strategy. Hypertension is a common health problem with disastrous results in inadequately managed patients. Abundant evidence has indicated that nearly half the patients with this problem have not been diagnosed, and about half the ones who have been detected are not being controlled. Consequently, approxi- mately three out of four hypertensive patients are presently at high risk for serious complications including stroke, heart failure, myocar- dial infarction, or kidney disease. Finally, recent evidence indicates that hypertension may also be a seriously neglected health problem in children (Heyden et al. 1969). The seriousness of the hypertensive condition has been previously illustrated by the author in a study of an emergency room walk-in population in a city hospital affiliated with the Johns Hopkins School of Medicine (Williamson 1971). A 1-year follow-up of a sample of hypertensive working class patients whose average age was 49 revealed that 1 in 5 (20 percent of the study sample) had died (figure 1), compared to the maximum acceptable outcome assessment stand- ard of 10 percent set by the medical staff in this institution. Subsequent study revealed that three out of four of these deaths were probably preventable, since they were directly related to inadequate patient compliance with required medical management of hyperten- sion. A subsequent partial replication of this study by Brook (1973) confirmed these findings (figure 2). The blood pressure control of Health Accounting 231 50 P=.005 21% 10% STANDARD FINDING Figure 1. Case-fatality rate for hypertension found-in a city hospital affiliated with Johns Hopkins University in 1966. One-year followup by Williamson of 87 patients [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder] treated and untreated patients was not significantly different. Again, patient compliance seemed to be the major factor involved in lack of health improvement. In a subsequent study, Inui (1976) demonstrated that it is possible to improve patient health outcomes by instructing physicians to provide more effective health education to patients. Figure 3 indicates the results of a controlled trial in which one set of physicians was given focused tutorial instruction on health education of their own patients. A matched set of physicians was given placebo education in didactic hypertension facts. Because of such findings, hypertension was selected as one of the best topics to demonstrate the health accounting strategy in the following section of this report. Study Design Health accounting is a five-stage cyclic strategy emphasizing health outcomes, as is illustrated in figure 4. The first stage consists of the formulation of priorities for quality assurance topic selection by a team appointed from those most familiar with the clinic or hospital. 232 EVALUATION OF CHILD HEALTH SERVICES % 50 56% i 46% 2 ol i TREATED NOT TREATED 71 PATIENTS 34 PATIENTS Figure 2. Uncontrolled blood pressure among an independent cohort of 105 hypertensive patients in 1971. Five-month followup by Brook. There was no statistically significant difference in blood pressure control between the groups. (City hospital affiliated with Johns Hopkins University). [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder.] %[ PHYSICIAN PLACEBO EDUCATION 8 9 N.S. P=.005 36% 34% 29% 15% BEFORE AFTER BEFORE AFTER Figure 3. Uncontrolled blood pressure among 102 hypertensive patients in medical clinics of Johns Hopkins University in 1972 with and without the education of physicians. Two-month followup by Inui [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder.] Health Accounting 233 STAGE 1 2 3 4 5 — — ——— — ——— Ef —— —— — Pf —— ff — — ¥ PRIORITY INITIAL DEFINITIVE IMPROVEMENT OUTCOME SETTING OUTCOME ASSESSMENT ACTION REASSESSMENT ASSESSMENT (e.g. Continuing Education) | RECYCLE Figure 4. Stages of the Health Accounting Project in its strategy to assure an approved outcome [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder.] Using formal procedures for small group estimates, a priority-setting team selects those topics that have the most potential for improving the health of patients or reducing expenditures for medical care. The second stage initially assesses the outcome of medical care for the chosen topic. This assessment is designed by a second team of qualified clinic staff members with the aid ofan outside specialist in a field relevant to the chosen topic. The assessment design that is developed incorporates explicit estimates of the total benefit to health which can be causally related to the diagnostic and therapeutic medical intervention used at that facility. These estimates provide a basis for predicting the potential health impact of the study; they also provide standards for evaluating the outcome. Measures of the health results in patients are obtained in follow-up studies by an evaluation assistant called a “health accountant.” The health accountant can be a health professional, such as a nurse, or merely a high school graduate with perhaps 2 years of college, who has demonstrated the problem- solving skills and personality characteristics required to gather data and interview patients. The results measured by the health accountant are compared with the standards previously established by the team. If serious discrepancies are found, further action in the subsequent stages of the strategy is usually recommended. The third stage consists of more definitive evaluation studies by a variety of relevant methods to identify correctable determinants of the unsatisfactory outcomes. Again, the design is provided by the study team staff members, and the measures are made by the health accountant. The fourth stage consists of a replication of the original assessment of results in patients to determine whether health standards have now been met. If not, stages 3, 4, and 5 are repeated one or more times until acceptable improvement has been achieved or until it is clear that any further gain will not be worth the effort required. 234 EVALUATION OF CHILD HEALTH SERVICES Study Results In 1972, an independent health accounting system of quality assurance was organized by the California Medical Group, a subsidi- ary of HMO International in Los Angeles. The following report describes part of their experience with this approach in 2 of their 21 clinics; the results of studies in these clinics have been combined in this presentation. During stage 1, a team of physicians and administrative personnel was organized to identify those assessment topics in which quality assurance activity might produce the most improvement in the health of clinic patients. Hypertension was one of the five topics identified. In stage 2, initial assessments of the results of hypertension treatment were made. The clinical study team decided that it was unacceptable for more than 5 percent of their hypertensive patients to have uncontrolled high blood pressure. Figure 5 indicates that the finding of 36 percent of patients with uncontrolled high blood pressure was seriously deficient for the 248 consecutive walk-in patients found to have hypertension in a 1-month sample. During stage 3 the determinants of the deficient outcomes were analyzed more definitively by means of two questionnaires adminis- tered by the health accountants: One questionnaire was directed toward the physician, and the other focused on the patient. The % PATIENT PATIENT EDUCATION EDUCATION BY BY " HEALTH ACCOUNTANT 8 P=.05 - P=.005 01 39% 249% 29% 13% AUG 73 AUG 74 AUG 73 AUG 74 100 PATIENTS 148 PATIENTS Figure 5. Uncontrolled blood pressure found in the initial assessment of 248 patients participating in the study by HMO International [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder]. Health Accounting 235 Table 1. Hypertension Questionnaire of Physicians on Knowledge Items Physicians? Inadequate drug information 0 Unaware danger of hypertension not related to symptoms 4 Overestimated national control of hypertension and compliance 8 Overestimated control of hypertehsion or compliance by his own patients 10 Total with inadequate information® 11 “N=14 - Counting each physician only once. questions were few and were concentrated on those factors for which the efficacy of medical care interventions for essential hypertension was documented and established. Table 1 illustrates the results of some of the knowledge items on the physicians’ questionnaire. All 14 physicians queried had adequate textbook knowledge of hypertension pharmaceutics; all were well informed on that subject. However, 10 were not aware of the poor compliance or poor blood-pressure control of their own patients. Table 2 indicates that 11 of the 14 physicians failed to mention patient education when they listed necessary interventions for the ideal management of hypertension. The second questionnaire containing knowledge items was directed toward the patients. The results had to be projected from the combined clinic data to a total group of 248 patients, because all the patients (52) were studied in one clinic, while only a random sample (100 of 196) were studied in the second clinic. From the sample group of 61 who saw the health accountant in the second clinic, a total of 120 patients were extrapolated and added to the 28 patients who saw him in the first clinic, to obtain an extrapolated total of 148 Table 2. Hypertension Questionnaire of Physicians on Value and Attitude Items 4 Physicians? Omitted the education of patients among aspects of ideal care for hypertension 11 Questioned the criteria for hypertension determined in a study by the Veterans Administration 5 Rejected clinical diagnostic standards for evaluating the outcome of treatment ~ . 1 Rejected the blood-pressure outcome project or the role of the health accountant 1 Total who were possibly deficient 11 AN=14 236 EVALUATION OF CHILD HEALTH SERVICES Table 3. Hypertension Questionnaire of Patients on Knowledge Items Patients? Unaware that they had hypertension 23 Unaware that hypertension is a serious risk to health 14 Unaware of the form or dosage of the drug they were receiving to control hypertension 30 Unaware of the possible toxic effects of the drug they were receiving to control hypertension 120 Unaware that the danger of hypertension is not related only to symptoms 137 Total with inadequte information” 140 # N=148 (extrapolated) ® Counting each patient only once patients seen by the health accountant in both clinics. Similarly, the group of 100 remaining extrapolated patients who saw only the physician was obtained by adding the total number of patients in the first clinic (24) and the extrapolated sample of the second clinic (39 extrapolated to 76). Table 3 illustrates the projected results from knowledge items in the patients’ questionnaire. Of the extrapolated 148 patients who visited the health accountant and completed the questionnaire, 23 were not aware that they had hypertension; 14 were not aware that high blood pressure is a serious health risk; 30 were unaware of the form or dosage of the drug they were receiving to control hyperten- sion; 120 were unaware of the possible toxic effects of that drug; and 137 erroneously related the danger of hypertension to symptoms, not understanding that their first symptom might be a stroke or possibly death. Table 4 indicates that more than half these patients were not obtaining adequate therapy for their high blood pressure. These findings constitute a set of correctable determinants of uncontrolled hypertension—the unacceptable health outcome originally measured. Table 4. Hypertension Questionnaire of Patients on Behavior Items Patients? Not taking any medication for hypertension 39 Taking medication for hypertension sporadically 25 Taking medication for hypertension only if symptoms occur 16 Total obtaining inadequate medication 80 a N=148 (extrapolated) ® Counting each patient only once Health Accounting 237 During stage 4, an educational effort was implemented by the health accountant. Each physician was given a specific educational prescription based on the inadequacies shown in the questionnaire. Depending on his answers, the physician was made aware of the compliance and blood pressure control rates of his own patients. Alternatively, he was given reprints of the Veterans’ Administration cooperative studies about the efficacy of hypertension treatment or of studies based on data from the surveys of the National Center for Health Statistics which revealed the lack of relation of blood pressure levels to symptoms. Patients participated in a similar educational program based on those items that they answered incorrectly on the questionnaire. The health accountants had the patients return, first weekly and then every 6 weeks, for blood pressure checks and reevaluations of their understanding of and compliance with the critical items included on the questionnaire. New health problems or lack of response to therapy resulted in an immediate referral to the physician. In stage 5, a reassessment of health outcomes indicated that the educational effort had been an apparent success. Figure 6 indicates that the proportion of patients in the entire group whose blood pressure was out of control dropped from 36 percent in August 1973 to 19 percent in August 1974. Although this figure was a significant improvement, it still was higher than the acceptable standard set by the clinic staff. Consequently, these patients were recycled to stage 3 to identify other correctable factors. Here two independent groups were identified: (1) the 148 extrapolated patients who had responded to the health accountant’s request to return to complete the question- naire and participate in the educational program based on its results; and (2) the 100 extrapolated patients who refused to return to fill out the questionnaire, preferring to come in only for their regular appointments with their physicians; therefore, the physicians were the main sources of their education. As figure 5 has shown, the differences in improvement in the two groups were startling. Those patients educated by the physician showed a drop from 39 percent uncontrolled to 29 percent; those educated by the health accountant dropped from 34 percent uncontrolled to 13 percent. More than twice as many patients who saw the health accountant had their hypertension brought under control. This difference in the rate of improvement was statistically significant at the p = 0.05 level. Utilization of Results The results of the overall health accounting research and develop- ment program have been applied by other investigators and have influenced national quality assurance programs in the United States. 238 EVALUATION OF CHILD HEALTH SERVICES %r EDUCATION ¥ P=.005 36% 19% 2 fH oL Jus STANDARD AUG 73 AUG 74 Figure 6. Uncontrolled blood pressure found in a 1-year followup of 248 patients participating in the study by HMO International [From Williamson et al. 1975. Reproduced with the permission of the Bulletin of the New York Academy of Medicine. Further reproduction prohibited without permission of copyright holder] Clement Brown was among the first to apply the basic health accounting strategy of cyclic integration of patient care assessment and continuing medical education. He developed an elaboration of this strategy, known as the “bi-cycle concept” of quality assurance (Brown and Fleisher 1971). His version was implemented in over 265 hospitals in the United States and became the basis of the Quality Assurance Procedure of the American Hospital Association. Partly because of Brown's influence, the basic health accounting approach was later reflected in PSRO legislation in the United States. Ray Helfer was another early investigator who was influenced by the first health accounting studies. An article on his work at the University of Colorado Medical Center utilized the proficiency and efficiency health care quality indices developed by the author in early assessment development studies related to health accounting (1967). Joseph Gonnella developed a unique application of the health accounting outcome assessment strategy in his thesis work at the University of Illinois College of Medicine. This application was reported in a subsequent article (Gonnella et al. 1970). His contribu- tion to quality assurance, focusing on health problem staging, repre- sents the subsequent growth of his interest in outcome-oriented methods. The JCAH was directly influenced by health accounting in its policy decision to adopt an outcome approach to supplement its Health Accounting 239 hospital accreditation procedures. Its Performance Evaluation Proce- dure, now implemented throughout the United States, is a variation of the health accounting outcome strategy. The HMO Act of 1973 was directly influenced by health account- ing. In this legislative bill, now signed into law, outcome-based quality assurance strategies are mandatory. Alvin Mushlin and Daniel Barr (1975) were directly influenced by implementation of health accounting in the prepaid group practice in Columbia, Maryland. They applied this outcome strategy in the development of a unique “Problem Status Index” for assessing outcomes of health problems in ambulatory patients. Robert Brook (1975) applied health accounting quality assessment design as part of his doctoral thesis research. The form and group judgment approach of health accounting outcome standards devel- opment directly influenced Brook’s later research at Rand (Brook et al. 1976). Thomas Inui was directly influenced by health accounting in development of his thesis work, “Improved Outcome in Hypertension after Physician Tutorials—A Controlled Trial,” which was reported in a subsequent article (1976). Discussion The methods and procedures of health accounting have been studied over a 14-year period. During the past 3 years, rigorous reliability and validity evaluation has been applied to essential procedures with very encouraging results. The structured group judgment procedures for developing priori- ties regarding topics for quality assurance study might be cited as a health accounting technique that has proved reliable and valid. Eight matched teams of representative staff in health care institutions independently identified areas of their own performance strengths and weaknesses in terms of patient health outcomes. The observed concordance of judgment could be expected less often than 1 in 10,000 if only chance agreement were operating. The findings were based on over 16 consecutive observations of the procedure at work. Out of 18 projects implemented to validate judgment, 17 have proved valid. An interesting sidelight of this effort has been the comparison of study topics developed by previous nonstructured topic-selection methods applied by inexperienced teams (group A), compared to structured methods applied by equally inexperienced teams (group B), as well as by experienced teams (group C). All teams were asked for topics that encompassed health benefits not then being achieved in their own practice but which they considered most achievable. 240 EVALUATION OF CHILD HEALTH SERVICES Group A, using nonstructured methods, identified only 2 percent of high-priority topics related to pediatrics. Utilizing structured proce- dures developed in health accounting, this proportion increased to 10 percent in group B and 25 percent in group C. Examples of the diversity of these pediatric topics developed by health accounting methods include the following: —Achievement of improved parent compliance and follow-up by family practitioners, pediatricians, and ENT specialists for recur- rent otitis media in children up to age 12; —Diagnosis, reporting, and follow-up on abused children under age 9 by all primary health care personnel; —More specific diagnoses by physicians and school psychologists of learning disorders in school-age children; —Treatment by primary care physicians and mental health team of parenting problems. —Dental care to be arranged by administrators of all enrolled Medicaid patients age 4 to 16; —Improved provision of sex and contraception counseling by physi- cians, nurses, and health educators for adolescents age 10 to 18; —Detection by physicians and nurse practitioners of inadequate immunization status in patients age 2 months to 16 years; —Administrative arrangements to avoid exposure of well babies to acute illness in pediatric waiting rooms. Another important finding from an overall evaluation of health accounting procedures is that prospective follow-up ministudies, as applied in this strategy, have proved feasible and inexpensive. Of the first 56 projects completed, 51 involved assessment of health out- comes; 45 of these involved direct patient follow-up interviews from an aggregate total of 5,857 patients. Of this latter group, 4,714 patients were successfully contacted and complete data were obtained, yielding an overall response and completion rate of 80.5 percent. This rate varied from 71.5 percent for poverty class patients, to 89.3 percent of working class, and 91.6 percent for middle class patients sampled. Among the first 56 projects completed, 29 involved assessment of diagnostic outcomes based mainly on recorded medical record data. Among 20 studies for which adequate denominator data were available, complete information was obtained for 14,020 of the 14,672 patients sampled, yielding a data completion rate of 95.6 percent. This rate varied from 95.4 percent for poverty class patients to 93.4 percent for working class, and 98.4 percent for middle class. These findings confirm the feasibility of conducting minisampling surveys of clinic and hospital patients for purposes of health care Health Accounting 241 outcome assessment. This capability is crucial for focusing quality assurance study on the critical problems, freeing the physician or researcher from the narrow constraints of studying only those problems identifiable in accessible data such as those found in patient charts. The health accounting procedure has several important advantages as a strategy to assess and improve health care: —Health accounting is a strategy that identifies deficient outcomes of health care and the deficient processes related to those outcomes. This strategy has the major advantage of considering all correcta- ble determinants of inadequate health results, not just those traditionally considered as part of the medical care. —Health accounting encourages provider consideration of a broad range of problems, from diet and lifestyle to inadequate parenting. Encouraging certain children to learn better diet habits, to achieve swimming skills, or to clarify values regarding smoking and drugs may have far greater health benefits than most traditional immu- nization procedures. —By utilizing encounter forms that are independent of medical records, health accounting can focus on topics having the greatest potential for effecting health improvement. In other words, this approach seeks data related to the most important problems, whereas present methods seek problems related to the most available data. —Health accounting stresses quality assurance clinic staff participa- tion in an educational planning process that requires a series of cost-effectiveness decisions at key points in every project. In this process the staff must examine its assumptions regarding: —Prevalence of the health problem in their own patient popula- tion. —Scientific evidence of efficacy of health care interventions. —Level of health or economic outcome deficiency in their present practice. —Probability of achieving significant improvement if the above deficiencies are verified. —Probability that the improvement achieved will be worth the total quality assurance effort and costs involved. —Health accounting requires active personal involvement of the staff in a rewarding process that entails a focus on the validity of their present knowledge of health science information, as well as clarifi- cation of their values and their assumptions about those patient or consumer values that are encompassed in establishing quality assurance priorities and outcome standards. 242 EVALUATION OF CHILD HEALTH SERVICES On the other hand, health accounting entails several disadvantages: —The concepts and procedures required by health accounting, such as epidemiology, efficacy, and sampling and survey methods, are often foreign to health care personnel, and must therefore be learned and understood. —Developing skills in these procedures requires contact with trained personnel who are in short supply at present. In the interim it may be necessary to hire a coordinator who is familiar with research and evaluation methods. —The act of exposing cherished health management assumptions held by staff members to open group critique can be an emotionally threatening experience. Assumptions regarding the validity of diagnostic methods or efficacy of the therapeutic modalities often prove to be ill founded. —Tracking down unavailable patients that fall in study samples can be a frustrating as well as a challenging experience. —Involvement with this method requires considerable internal moti- vation of the participating staff. This requirement is often difficult to meet when the staff is concerned more with productivity than with the improvement of health benefit. In conclusion, experience to date indicates that there are some financial and staff costs involved in implementing the health account- ing strategy. However, these costs do not seem inordinate and are probably much less than those for alternative systems. On the other hand, the potential health impact of this strategy seems to be significantly greater than has been documented by other systems. The educational rewards and enhanced personal staff involvement alone seem worth the total effort. Finally, the extensive and favorable reliability and validity testing that this method has undergone makes it unique and worthy of important consideration. REFERENCES Brook, R.H. (1972) Quality of Care Assessment: A Comparison of Five Methods of Peer Review. (Doctoral Thesis) Baltimore: The Johns Hopkins University School of Hygiene and Public Health. Brook, R.H. (1973) Quality of Care Assessment: A Comparison of Five Methods of Peer Review. DHEW Publication No. (HRA) 74-3100. Rockville, Md.: Department of Health, Education, and Welfare. Brook, R.H., Avery, A.D., Greenfield, S., et al. (1976) Quality of Medical Care Assessment Using Outcome Measures: An Overview of the Method. R-2021/1-HEW. Santa Monica: The Rand Corporation. Brown, C.R., and Fleisher, D.S. (1971) The bi-cycle concept—Relating continuing education directly to patient care. New England Journal of Medicine 282:Suppl:88- 97. Health Accounting 243 Gonnella, J.S., Goran, M.H., Williamson, J.W., et al. (1970) Evaluation of patient care—An approach. Journal of the American Medical Association 214:2040-2044. Helfer, R.F. (1967) Estimating the quality of patient care in a pediatric emergency room. Journal of Medical Education 42:244-248. Heyden, S., Bartel, A.G., Hames, C.G., et al. (1969) Elevated blood pressure levels in adolescents, Evans County, Georgia. Journal of the American Medical Association 209:1683-1689. Inui, T., Yourtee, E.L., and Williamson, J.W. (1976) Improved outcome in hyperten- sion after physician tutorials—A controlled trial. Annals of Internal Medicine 84:646-651. Mushlin, A.L,, Appel, F.A., et al. (1975) The use of patient reported problem status in evaluating emergency room care. Clinical Research 23:267. Williamson, J.W. (1971) Evaluating quality of patient care: A strategy relating outcome and process assessment. Journal of the American Medical Association 218:564-569. Williamson, J.W., Aronovitch, S., Simonson, L., et al. (1975) Health accounting: An outcome-based system of quality assurance: Illustrative application to hypertension. Bulletin of the New York Academy of Medicine 51:6:727-738. 244 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION Milton I. Roemer and Gabriel Velazquez The combination of a method of quality assessment with strategies for promoting health care improvements, based on the findings of Dr. Williamson's “health accounting,” is especially noteworthy. It appears that the personal involvement of a broad range of doctors and other members of the health care team is the crucial feature of the process. It was noted that topics selected for evaluation were predominantly practices for which there was a broad consensus on efficacy—such as drug therapy of hypertension or certain immunizations. Thus, the task is only to determine if the specific service is being effectively delivered. In the early stages of development of the health accounting strategy, Dr. Williamson and his colleagues took the initiative in approaching selected group practices. As the work became better known, various health care organizations came to him for evaluation. In all cases, the final definition of problems to be investigated was made by the personnel in the cognizant organization, not by the “outside” evaluators. This seems to assure that corrective actions will be taken in response to the findings. Dr. Williamson stresses that the findings, however, are not overextended beyond the specific problem investigated. The management of certain disease entities has nearly always been the topic initially chosen for evaluation in any one health care organization. With time and experience, more attention tends to be directed to issues in prevention and in the organization or administra- tion of health services. A key problem has been the training of competent “health accountants.” These persons must be carefully chosen and skillful in their interpersonal relationships. The training of health accountants has now been standardized, but it involves a cost—not to mention the cost of the operation of the evaluation program over time. So far, the method has only been applied in organized medical care settings, mainly group practices. It is hoped to apply it eventually also to solo medical practices, perhaps by assembling “consortia” of private practitioners in an area. Since the method is prospective, rather than retrospective, it should be feasible to collect the necessary information (through records), once a decision has been made to undertake the evaluation. The health accounting strategy for assessing quality through outcome measures has been tested recently in Holland. It might be Health Accounting 245 more difficult to apply in Latin America, but it would seem to be worth trying in a program like that of PRIMOPS in Cali, Colombia, where a strong, technically competent staff is assembled. eu =p ==t eta a gg, n= ee Si eet S—————— SE mm, USE OF EVALUATION IN EDUCATION a = a 16 MIRROR OR WINDOW: THE INTERFACE OF HEALTH SERVICES EVALUATION AND PROVIDER EDUCATION John L. Simon INTRODUCTION In a very important sense, provider education is the interface between the evaluation of health services and medical practice. Its responsibility is to use evaluative data to keep its curriculum “healthy” in the hope that the medical practice of its graduates will be improved. The extent to which this interface can be characterized as a mirror or as a window may indicate how good a fit there will be between what we are training our providers to do, and what our providers will need to be able to do. If we were preparing to dress appropriately for the weather outside, we would be more likely to look through a window, than a mirror. If the sun were shining, some of us would dress for warm weather. However, we might telephone the weather service for a temperature reading, because the sun can shine in 32° or 52° weather. Some of us might still not venture out for a long trip without some weather forecast. Will we need an umbrella 4 hours from now? After collecting all these data, we might dress appropriately and then stand in front of a mirror. Do we look presentable or ridiculous? Can we modify our clothing combination, or is this choice the best we can make? In summary, both the window and mirror are important tools if we are to dress appropriately for weathering the environment. This simple analogy can also be applied to evaluation of health services. Health services evaluation holds the potential to be a useful window to the environment into which we send our students, as well as a mirror reflecting how well we have prepared them for it. While such evaluation may yield imperfect and imprecise information, it can reflect to some extent whether our graduates are presentable or incongruous in terms of their training; or whether we need to modify the clothing combination or curriculum. If my understanding is correct, there are two major issues concern- ing the delivery of child health services in the United States today. 249 250 EVALUATION OF CHILD HEALTH SERVICES The first is identified by using evaluative research as a window. Haggerty and others (1975) have looked through the window and described a “new morbidity.” This fresh and perceptive look at the child health environment indicates the need for new knowledge, and therefore must impact on the education of child health providers. The second major issue is identified by using evaluative research as a mirror. Richmond (1977) and Kessner (1974) have made us face the mirror to discover that even our existing knowledge is not being adequately applied. This information, too, must have an impact on the education of child health providers. In essence, I am suggesting the rather simple notion that the educational objectives of a curriculum for health providers ought to be derived from the professional tasks and responsibilities these future providers will have in their professional settings. Moreover, this use of actual experiences for planning educational objectives must be dynamic. The results from health service evaluation must be continually assessed and translated into educationally useful data. The fact that curricula in health science schools usually represent a measurement of the relative strengths of department chairmen, expressed in terms of hours, suggests that this “simple” notion is not quite so simple to implement. . BACKGROUND Our project was an embryonic attempt to use several types of research data as a source for a minicurriculum based within a Health Maintenance Organization (HMO). The project took place at George- town University from June 1975 through August 1976. At that time, I was a faculty member in the Department of Community Medicine and International Health.! The Georgetown University School of Medicine is a private, urban medical school located in Washington, D.C. There are 205 students enrolled in each class. Except for a 2-to-3-week stint in a hospital outpatient department or physician’s office during medical and pediatric clerkships, no primary care experiences are required. The Georgetown University Community Health Plan (GUCHP) is a prepaid, group-practice-type HMO which opened for services in late 1972. Originally sponsored by the Department of Community Medi- cine and International Health, today it is an independent corporation serving an economic cross section of the Washington metropolitan community. At the time of the project, there were over 10,000 ! Portions of this paper are based on activities which were, in part, supported by the Association of American Medical Colleges and the Kellogg Foundation. Interface of Evaluation and Education 251 enrollees and an additional 6,000 “active” fee-for-service patients who were served by the HMO’s three primary care centers: one in densely populated northeast Washington; a second in the new town of Reston, Virginia; and a third in Kensington, Maryland. Community hospitals were utilized by each primary care center, and Georgetown University Hospital was utilized as a tertiary care referral center. Our goal was to use GUCHP as a setting for teaching aspects of primary care to undergraduate medical and nursing students. The message 1 would like to convey, however, deals with the process we used to plan the minicurriculum, rather than the specific setting. We began by asking the following kinds of questions. If “child health provider” is substituted for “primary care physician,” the following questions will still be the relevant ones to ask when planning a purposive educational experience: —What is a primary care physician? —What constitutes the content of primary care? —What responsibilities or competencies are, or should be, required of the primary care physician? —What effect does professional setting have, be it HMO, solo practice, partnership, etc., on the role of the primary care physi- cian? The answers to these questions provided the educational objectives for the curriculum. Obviously, it would have taken several years and a major study to answer these questions with precision. We had neither the resources nor the luxury of time to carry out such a study, so we approximated one. We did some evaluation of our own, as well as using research reported in the literature. In an effort to look through the appropriate windows, we thought it was very important to understand: —The perspective of the learner, i.e., medical and nursing students; —The perspective of the potential preceptor, i.e., the primary care providers of the HMO; —The perspective of the consumer, i.e., the HMO enrollee. THE PERSPECTIVE OF THE LEARNER We hired three sophomore medical students and three senior nursing students for a summer. They formed three interdisciplinary teams, each consisting of one nursing and one medical student. Under faculty guidance, these student teams searched out and read literature on issues in primary care. They studied the content of 252 EVALUATION OF CHILD HEALTH SERVICES primary care practice and the roles of the primary care physician and nurse practitioners in a health team. They also read about successful and unsuccessful curricula attempted elsewhere for training under- graduates in primary care. The learning that occurred during this phase prepared the student teams for the next phase. THE PERSPECTIVE OF THE FUTURE PRECEPTOR The students’ next task was to discern certain aspects of primary care in our HMO setting. Because of the students’ level of clinical and technical expertise, they concentrated on the more generic tasks and responsibilities of primary care physicians and nurses in GUCHP. Each interdisciplinary team was assigned to one of GUCHP’s three primary care centers. The teams first developed observation guides and spent 1 week observing physician—nurse, physician—patient, and nurse—patient interactions. Then, based on their literature reading and their observation results, the students developed a staff interview questionnaire. Each team interviewed the primary care physicians and primary nurse practitioners in their centers individually. Since the practitioners were our students’ potential preceptors, the faculty thought it was crucial to understand the practitioners’ perceptions of primary care, practicing within an HMO, and working on a health care team. In addition, their views as to the adequacy of their previous training in preparing them for their current roles were also sought. Some of the questions asked of staff were: —What do you find satisfying about working in primary care? —What do you find unsatisfying about working in primary care? —What kinds of things have you encountered in primary care practice that your education didn’t prepare you for? —What do you feel are the most important factors which influence your role relationship with the physician/nurse? How can the answers to these questions help in planning a curriculum?An example is our finding that many physicians believed they had not been sufficiently prepared for their supervisory respon- sibilities. The addition of supervising students on top of other supervisory tasks augmented their need for such skills. Therefore, one of our planning needs became the development of a preceptor preparation course. Virtually all full-time primary practitioners vol- unteered to participate on their own time in a 3-session, 6-hour course. A major component of the course included real and simulated training in a teaching technique called “clinical supervision.” The curriculum, the resources, and some of the exercises used in this Interface of Evaluation and Education 253 course are currently being published for use in other preparation courses for clinical preceptors (Simon, in press). THE PERSPECTIVE OF THE CONSUMER Each of the GUCHP primary care centers had a Consumer Advisory Board. The faculty initially discussed the project with these boards at the HHIMO centers. Both faculty and students thought it was essential to learn what patients (in this case, HMO enrollees) expected from the physician. It was felt that patient expectations of the physician and methods for dealing with such expectations should definitely be included in a training program for primary care. Therefore, the students developed a questionnaire and interviewed enrollees both before and after their encounters with the physician or nurse practitioner. Examples of questions asked of patients before their encounter included: —What do you want to happen during your visit today? —Do you have any questions on your mind you want to ask the doctor or nurse? Examples of questions asked of patients after their encounter included: —Did you ask all of your questions? —Did you get all of your questions answered? —What things were satisfying about your visit with the doctor/nurse? —What things were not satisfying about your visit with the doctor/ nurse? In this case, we were not specifically interested in whether or not the patient was satisfied. Rather, we were concerned with what incidents were considered critical by the patient in determining whether the visit had been satisfying or unsatisfying. PRIMARY CARE COURSES Patient responses to these questions were being summarized when I left the project. A preliminary look at these data, however, has already generated ideas for the following courses, which were being planned. 254 EVALUATION OF CHILD HEALTH SERVICES A Course in Developing a “Good Chairside Manner” During an interview with a student, one patient responded that the factor which satisfied her the most about her visit with the doctor was his “good chairside manner.” Evidently, the bedside manner of inpatient care has a corollary in the chairside manner of ambulatory care. Other patients’ responses helped to define a good chairside manner; based on this information, a course is to be developed for first-year students. It will concentrate on the patient—physician rela- tionship and involve interviewing techniques, such as putting the patient at ease and managing encounter time so that patient and physician do not feel rushed. It will also include techniques for answering patient questions. A Course in Telephone Medicine One outcome of our evaluation of the professional tasks and responsibilities of primary care practitioners related to telephone encounters. We found that a growing proportion of primary care is being delivered over the telephone. Physicians in the United States are spending 1 to 2 hours per day consulting with patients by phone. Child health providers are perhaps the heaviest utilizers of telephone medicine. In most instances, these providers invariably not only have to make clinical judgments without the patient’s being visibly present, but also must elicit critical information in an efficient way from a third party—usually a parent or guardian. Although pediatric clerks or house officers probably participate in telephone duty or night call, participation does not equal education (Ott et al. 1974). The skills and knowledge required to adequately “treat by telephone” need to be documented and analyzed. It is a faculty’s responsibility to systematically plan out an educational experience through which a learner’s competence in telephone medicine can be gained and assessed. Electives in Primary Care Team Practice One outcome of the medical-nursing student team experience was data for planning a sequence of two primary care team electives. The first elective would be targeted jointly for sophomore medical students and junior nursing students. This course would match the students into MD-RN student dyads or teams and concentrate on developing skills required for effective nurse practitioner-primary care physician collaborative practice. Role negotiation and the practi- cal exploration of each other’s professional skills would be a course Interface of Evaluation and Education 255 goal, and methods of simulating team practice would be heavily utilized. The major goal of this course would be to train student physician-nurse teams just to a level where they could treat patients with minimal disruption to normal HMO operations. The second experience, involving senior medical and senior nurs- ing students, was piloted last September. This course is a 6-week clerkship in the Health Plan. The medical students and nursing students. again constituted primary care teams, precepted by mentor primary care teams within the HMO. At this point, let me summarize what we have accomplished. Instead of meeting as a group of faculty to plan a series of courses in primary care, we first went to —the practitioners of primary care —the recipients of primary care —the potential students of primary care to help answer the questions originally stated. We attempted to utilize existing and newly generated evaluative results as a basis for planning educational objectives. ACHIEVEMENTS AND PROBLEMS Achievements —The types of questions we were asking are still being asked and answered in the HMO. The providers themselves have “taken off” on some of the data produced by the students to improve their own performance. —The students, while evaluating only an aspect of health practice, were treated to an intense learning experience, the product of which will be educational objectives for future students. —The positive attitudes of the providers toward evaluation served as a good role-modeling experience for the students. —The process of learning about professional roles in this setting has generated A Medical Students’ Guide to Health Maintenance Organiza- tions (Simon et al. in press). This guide is targeted for classroom use. Essentially written by medical students for medical students, it is currently being revised to include the HMO Amendments of 1976 ’ Problems —The project did not last long enough to institutionalize the process within the department. 256 EVALUATION OF CHILD HEALTH SERVICES —Many of the educational objectives to be derived from our data are probably more relevant to the teaching programs of other depart- ments within the medical school, where the mission for training clinicians is more legitimized traditionally. CONCLUSION The approach for curriculum development described above repre- sents one way to utilize the interface of research and medical practice as a window and mirror for provider education. A similar process has been applied on a broader scale by others. In his recently published book, An Inventory for Primary Health Care Practice, Dr. Archie Golden (1976, p. 7) writes: There is little evidence today that curricula for all levels of health workers are based on the real world of the performance of health personnel in the field. There is a need to relate the disparate areas of education for health careers and the delivery of services. I have attempted to make a case for utilizing data from health services evaluation to develop a set of educational targets for student learning. There should be better examples of how best to do this. Perhaps you will help to find them. REFERENCES Golden, A.S. (1976) An Inventory for Primary Health Care Practice. Cambridge, Mass.: Ballinger. Haggerty, R.J., Roghmann, K.J., and Pless, I.B. (1975) Child Health and the Community. New York: John Wiley and Sons. Kessner, D.M., Snow, C.K., and Singer, J. (1974) Assessment of Medical Care for Children. Washington, D.C.: Institute of Medicine, National Academy of Sciences. Ott, J.D., Bellaire, J., and Machotka, P., et al. (1974) Patient management by telephone by child health associates and pediatric house officers. Journal of Medical Education, 49:596-604. Richmond, J.B. (1977) The needs of children, in Doing Better and Feeling Worse: Health in the United States. Daedalus (Winter) pp 247-259. Simon, J.L. in press. A Role Guide and Resource Book for Clinical Preceptors, Washington, D.C.: U.S. Government Printing Office. Simon, J.L., Creelman, W.L., Gocker, M., et al. in press. A Medical Student's Guide to Health Maintenance Organizations, Health Resources Administration, Washington, D.C.: U.S. Government Printing Office. 17 EVALUATION AS A TOOL FOR COMMUNITY EDUCATION! Rolando Merino, Samuel |. Bosch, Ruth Zambrana Introduction During the last decade in the United States, consumers and community groups have expressed clear demand for participation in health institutions and, more importantly, in the decisionmaking processes of those health organizations that affect their lives (Gartner and Riessman 1974, Metsch and Veney 1976). This demand has not led to participation in established health organizations, but rather to the development of new, community-based organizational modalities, such as neighborhood health centers, most of which were launched by the Office of Economic Opportunity (OEO) and later supple- mented with centers funded by the Department of Health, Education, and Welfare (DHEW). Neighborhood health centers (NHCs) were intended to exert a-direct influence on the way medical care was provided to persons in low-income areas. NHCs were to provide a broad range of services, including outreach services by indigenous workers and legal services not usually within the purview of a health facility (Zwick 1972, Klarman 1974). They were also meant to assume, with consumer participation, a measure of responsibility for the continuous health care of the populations that they register (OEO 1968). Evaluation of established health organizations has been man- dated by several Federal programs, thereby presenting difficulties in the selection of measurable criteria, as well as the implementation and assessment of programs. Community-based health centers do not have the advantages of the established medical community. For this very reason, it is vital that community organizations be capable of effectively evaluating their efforts to provide health services in previously underserved and needy areas. Evaluation data were used as the principal tool in an educational program planned and developed by a community organization in ! Study supported by the Health Services Improvement Fund, Inc. 257 258 EVALUATION OF CHILD HEALTH SERVICES partnership with a school of medicine. This paper presents the preliminary findings of that program. The program was implemented to increase the health care knowledge and decisionmaking skills of the consumer governing board in a recently established NHC that serves children and their families in an underprivileged area of New York City. Background In East Harlem, 48 percent of the population is of Hispanic origin (Johnson et al. 1972). On July 1, 1975, the East Harlem Tenants Council (EHTC), a Puerto Rican community-based organization, took over a New York City Health Department child health station and converted it into an NHC with an HEW 314-E grant. The center now serves the children previously served by the health station, their families, and a growing number of new families in the area. The EHTC received technical assistance in the planning and development of the center from the Department of Community Medicine (DCM) of the Mount Sinai School of Medicine (Bosch and Deuschle 1977). Since then, the center has operated under the supervision of the EHTC’s lay board with the participation of consumers (registered patients) on the board. The DCM maintains its role in the evaluation of the center’s program. On April 1, 1976, the EHTC and the DCM decided to undertake a joint educational program to enhance the EHTC board’s capacity to govern the center. The EHTC board is composed of 18 persons (from various disciplines and with different educational backgrounds) who are not health care providers. The majority are East Harlem residents and consumers of services in the health center. The board is the policy maker for the NHC. Its policies are communicated through the project director, who is responsible for their execution. Board leadership belongs to its Executive Committee, composed of a chairperson, a vice president, a treasurer, a secretary, and two members-at-large. There are two standing committees—Membership and Personnel—and ad hoc committees are formed as needed. Elections to all committees are held annually or as vacancies occur. The Need for An Educational Program In the original legislative mandate for NHCs, participation was required in those federally funded programs where policymaking responsibilities were specified for the consumers in planning, man- Community Education 259 agement, and evaluation activities (OEO 1968). The requirement is based on the premise that consumers are more sensitive to and respond better to community health needs. Therefore, if they could participate in policymaking, the goals of a program would be better met and the NHCs would become more readily accepted in the communities they serve. Experience has demonstrated that participa- tion can occur if consumers and community organizations take the necessary steps to acquire the knowledge and skills that are pertinent to their particular area of decisionmaking. During the planning of the center's first-year operational phase, 1975-76, the active participation of community members through the EHTC board produced a health care plan tailored to meet the health and medical care needs of the East Harlem community. Nevertheless, during the first year of services there was little participation by the EHTC board in the management—in the definition of overall policies and in the periodic evaluation of the program. During this period, the board realized that it was gradually losing control of the program and recognized the need for more expertise in health care policy in order to articulate the health needs of the community more effec- tively. At the same time, the authors of this paper, functioning as the DCM planning and evaluation team for the center, also recognized that the board members’ inadequate understanding of the center’s service goals was encouraging political interests to take priority over program objectives. We also observed that the absence of basic health care knowledge on the part of the board was permitting the center’s administration to decide on matters that were rightfully under the jurisdiction of the consumer board. In other words, our observations confirmed that the governing board's power over the center was gradually being shifted from the consumers to the providers of care. An analysis of our observations enabled us to conclude that this shift was occurring not as a result of a power conflict between the board and the administration, but primarily because of a deficiency in the technical capability of the board to define health care policies. Such cooptation of the community organizations’ decisionmaking power by administrators is frequently encountered. Since a power conflict was not the issue here, an explanation was sought in other areas. Several discussions with the project director identified the need’ for more active consumer participation in the governance of the center. It was also hypothesized that active board participation could best be attained if two basic criteria were followed: (1) The board should learn how to read and analyze the basic utilization and cost data used in the periodic evaluation of the center; and (2) the board should learn how to understand basic effectiveness and efficiency measures, so that its decisions would be more relevant to community needs. 260 EVALUATION OF CHILD HEALTH SERVICES The Development of the Educational Program The planning and development of the curriculum were carried out by four DCM health planners, who represented different disciplines, and the NHC project director. As trainees in the program, the members of the EHTC board actively participated in the definition of the educational objectives, selection of the educational methods, and development of evaluation procedures. The educational objectives of the program were defined through —Analysis of the results of a questionnaire administered to the board members to identify their perceptions of learning needs; —Observations by a bilingual Puerto Rican sociologist of three monthly board meetings that preceded the program; —Direct participation of the center’s project director in the planning process. The educational objectives were defined to ensure that board members would have acquired certain skills and knowledge at the end of 1 year. The maximum 15 hours available for formal training sessions was also taken into consideration. It was agreed that at the end of the training program the board members should have the skills to: —Read monthly utilization reports critically. These reports included data on the efficiency and effectiveness of the health center. —Recognize utilization and productivity trends. —Utilize data reports to decide among alternative recommendations made by the center’s director. —Critically analyze the budget in relation to the health needs of the community and the health center’s objectives; analyze data related to the effectiveness and efficiency of the health center's operation and to its financial sources of income. It was also agreed that at the end of the same period, members of the board should have acquired knowledge of the terminology and basic concepts involved in the following specific topics: —Community health needs, demand, and utilization of health serv- ices; — Utilization and productivity trends and related factors; —Program budgeting; —Ongoing evaluation of the center’s performance, with emphasis on program efficiency and effectiveness; and Community Education 261 —Definition of health policies and intermediate and long term planning. The training methods were defined on the basis of suggestions made by the chairperson and executive committee members of the board. It was decided that the most appropriate teaching procedures would be —Presentation of educational materials in an incremental fashion through group discussion with the trainers and trainees; —TFocusing discussions on analysis of the center’s actual evaluative data; —Presentation of these data by the director of the health center to the board members; —Explanation of selected topics for that session using actual data, following the project director’s presentation; —Encouragement of discussion between the board members and the trainers based on questions and answers. The main educational resources were the DCM health planners, who assumed the responsibility of educators throughout the program. The summative procedure utilized in the evaluation of this pro- gram’s results was the administration of three multiple-choice ques- tionnaires to board members in order to measure their knowledge gain. The final questionnaire will be administered upon completion of the 12-month training period. Results of this procedure are therefore not incorporated in this paper. The main formative procedure was the field observation of “a group at work” in order to evaluate process, particularly in relation to participation in decisionmaking. The field observations were made by the same sociologist who participated in the identification of educational objectives. In this particular training program, the sociol- ogist attended (as an observer) all board meetings and executive sessions, in addition to the curriculum development meetings, to collect detailed descriptive data and to analyze them in terms of the program’s objectives. The major thrust of the analysis was to identify the crucial variables of group process—such as intragroup conflict and hidden agendas of individuals—which have an impact on the decisionmaking capability of the board. The principal methods used were qualitative analysis of data based on the following criteria: increased level of participation; use of appropriate terminology; better understanding of the relationship between income and expenses; and application and use of information presented in the training sessions in order to make policy decisions. The predetermined categories were developed in consultation with education experts. 262 EVALUATION OF CHILD HEALTH SERVICES Preliminary Findings In general, the EHTC board members are demonstrating a favorable attitude toward the educational training process. The following preliminary results indicate some positive outcomes of the training program: —Increased participation in discussions. An increasing number of board members have gradually begun to participate in the different sessions. Further, the formulation of pertinent questions, opinions, and suggestions in relation to the management of the health center has become more sophisticated. —More appropriate use of terminology and concepts. Board members have also made progress in using appropriate terminology and concepts related to the operation and evaluation of the performance of the health center. Of utmost importance has been their ability to use and interpret the measures of effectiveness, for example, compari- son between attained and planned objectives, activities, and re- sources. This ability has in turn enabled them to recognize the concept of efficiency: the relationship between ongoing activities, cost per activity and administrative costs, and the cost per unit distribution in operations. The relationship between the unit cost of a specific activity and outcome in terms of health has also been explored by board members. —Better understanding of relationship between income and expenses. The board members have acquired an understanding of the notion of limited resources for unlimited health needs. During the last two sessions, they have initiated in-depth discussions of priorities in the formulation of objectives for the health center. There have also been several discussions on the identification of activities which would enable the center to operate more effectively: the cost of pediatric care as compared to adult care and the impact of both on the health status of the population, as well as the cost of preventive care as opposed to diagnostic care and treatment. The need to establish health priorities has been recognized as a fundamental planning concept. —Request for a standardized information system for all programs operated by the council. An important outcome of training has been the interpretation and use of monthly statistical data pertaining to the health center’s functions. The board has suggested the possibility of standardizing a periodic information system for all the programs presently operated by the institution. This would involve defining indicators that would demonstrate efficiency, effectiveness, and quality of services delivered by the different programs under the auspices of the EHTC. Community Education 263 —Inclusion of topics not originally defined in the curriculum. There has been a persistent demand from the board members to engage in detailed discussions regarding planning, monitoring, and evalua- tion of the health center program. The assessment of the quality of care, for example, has been identified as an area which should be addressed in a supplemental training session. —~Extension of the designated time for the training sessions. Initially it was decided, at the suggestion of the board members, that each session would last no longer than 1 hour. However, the sessions have generally been prolonged to 1'/>-2 hours because of the interest generated by the materials presented. Even so, it has become necessary to set limits on the questions and answers raised by board members, especially when the questions were not related to the material being covered in a particular session. This limitation was set so that the trainers could systematically develop the curriculum in its totality. —Request for additional training sessions. The board chairperson has made a formal request for additional training sessions. These sessions would be provided for those members who desire to further expand their knowledge in the areas already presented in the regular sessions and for the new members who joined the board after the initiation of the training program. —Improvement of the attitudes of the board members toward the DCM. Another important outcome has been an improvement in the relationship between the EHTC and the DCM. Interaction during the training sessions has helped to establish better communication between them. Finally, the board members decided to review in February, at a specially designated meeting, a final draft of the 1977-78 NHC proposal which was submitted to DHEW in March 1977. The review of the grant proposal by board members is a standard procedure which is considered part of their responsibility as mandated by the legislation for NHCs. The board members in that meeting were able to review the goals and objectives of the program specifically in terms of personnel and activities, review and redefine the activities neces- sary to achieve the goals and objectives, and analyze and approve’the budget as proposed by the project director. Discussion and Conclusions The major problems in the development of this project were encountered in the identification and application of evaluation instruments that could measure with precision the process, the 264 EVALUATION OF CHILD HEALTH SERVICES acquisition of knowledge, and the learning of decisionmaking skills. As expected, the heterogeneity of the board in terms of education, and professional background made this task difficult. One of the first problems was the board’s initial reluctance to grant the sociologist permission to attend meetings as an observer. In essence, the members felt uncomfortable with the fact that an “outsider” would be privy to all their deliberations. Several discussions were held in order to clarify the role of the sociologist, particularly in terms of confiden- tiality. Ultimately, a letter of agreement was written to ensure confidentiality and the right of the board to request that the observer leave the board meetings at any time the members considered it necessary. The administration of the multiple-choice questionnaire was not a major problem to overcome; nevertheless, like any traditional method of evaluation, it raised questions by the project director and members of the board regarding the advisability of doing research “on them.” The board members, all very busy persons in their usual occupations, have only a limited amount of time to devote to nonprofit activities. A time-consuming procedure was out of the question. An agreement was reached whereby a short questionnaire would be administered during the meetings. The admission of new members to the board midway in the program complicated measurement even further. Consequently, the impact of our intervention cannot be measured as a cause-effect interaction. Our preliminary findings merely indicate that a change has occurred as a result of the program. More importantly, the introduc- tion of the program has served as a catalyst for change and has accelerated a process whose effect is not immediately measurable. Without doubt, one of the principal outcomes of this program has been the improvement of interinstitutional, as well as interpersonal, relations. The process at this point appears irreversible, mainly because of the mutual cognitive and affective learning that is taking place among trainees and trainers. Further, the interaction which facilitates that learning is influencing attitudes and strengthening trust and rapport between the two groups and the institutions they represent. Probably one of the most important factors involved in the success of the process was the initial clear definition of role and responsibility for both institutions. The recipient of the project grant was the community organization. This organization, through a contractual agreement with the medical school, “hired” the training group. The basic responsibilities of each organization and of the trainers were explicitly outlined. The rationale underlying the agreement was the recognition of a need for the exchange of resources such as funds Community Education 265 and specialized skills which would be complementary to the achieve- ment of the goals of the organizations involved. It would also ensure the maintenance of the social distance necessary to avoid conflicts and the feeling of one organization's being “swallowed up” by the other (Levine et al. 1963, Litwak and Meyer 1966). The selection of evaluation data as a tool for education has proved in this instance to have been an appropriate strategy. The real-life nature of the tool and the relevance of the information for board governance were a source of motivation for the learners which probably encouraged attendance and triggered meaningful discus- sions. The subject matter enabled the trainers to remove the “mys- tique” surrounding health care delivery and health statistics. Further- more, the use of evaluation data has generated a sense of security in the board members concerning the decisions they are making; indeed, they now wish to extend this approach to other service programs for which they are responsible. REFERENCES Bosch, S.J., and Deuschle, K.W. (1977) The role of a medical school in the organization of health care services. Presented at New York Academy of Medicine, Fifth Annual Symposium on Medical Education, October 14, 1976. Bulletin of the New York Academy of Medicine 53:449. Gartner, A., Riessman, F. (1974) The Service Society and the Consumer Vanguard. New York: Harper and Row. Johnson, L., Lynch, A., Rivera, O. (1972) East Harlem Community Study. Mount Sinai School of Medicine of The City University of New York. May. Unpublished. Klarman, H.E. (1974) Major public initiatives in health care. Public Interest 34:120. Levine, S., White, P.E., Paul, B.D. (1963) Community interorganizational problems in providing medical care and social services. American Journal of Public Health 53:1183. Litwak, E., and Meyer, H. (1966) A balance theory of coordination between bureau- cratic organizations and community primary groups. Administrative Science Quar- terly. June. Metsch, J.M., Veney, J.E. (1976) Consumer participation and social accountability. Medical Care 14:283. OEO (1968) Comprehensive Neighborhood Health Services Program: Guidelines. Washington, D.C.: Health Services Offices, Community Action Program. March. Zwick, D.I. (1972) Some accomplishments and findings of neighborhood health centers. Milbank Memorial Fund Quarterly 50:387 266 EVALUATION OF CHILD HEALTH SERVICES DISCUSSION: EDUCATION AND EVALUATION David |. Jones The two papers dealing with evaluation, providers, and the com- munity stimulated a discussion to which the subjects of community involvement, communication between consumers and providers, and the complexity of these activities were central. Community involvement was described in its historical-political context. One speaker cautioned the group to remember that in the Western Hemisphere, community participation predated Columbus— suggesting further that it was civilization that had ruined the process! Another speaker characterized political and cultural differences as the weak links in communication among institutions, their boards, and the community. This “interrupted communication,” as one participant labeled it, underscored the importance of increasing the accuracy and objectivity of evaluative data from the community, so that we might understand the problems better. This observation was countered with the question “In any political system, can we ever really know how to determine what people want or need?” Neverthe- less, there was consensus that constant negotiation was essential in determining wants and needs. One discussant brought up an even more complicated issue—that community-based boards sometimes want to deal with problems which are not amenable to health strategies. But one can ask in return, “Who defines health problems?” The discussion was brought full circle by a related observation that evaluation (to determine what constitutes a health problem) is ultimately based on values, and that values are critical to that negotiating, communication process. Several persons cautioned the conference about the difficulty which can arise if we organize around patient or consumer needs or wants rather than clinical problems—the lack of structure. Similarly, with reference to the paper on provider education, if we derive practice from existing providers when the context is not known or specified, we again face the problem of no structure—either for consistent delivery or for evaluation. Participants noted the conflict between dynamism and structure, but no specific resolution was proposed. The communication between consumers and providers focused on the common ground on which they frequently meet: the boards of community organizations or institutions. A question frequently im- plied, but not specifically asked, was whether the traditional role of board members has now changed. Historically, selection of members for community organization boards has been based on specific skills or knowledge possessed by the individual named—the executive, the Community Education ~~ 267 lawyer, the accountant, the engineer. The other consideration has been prestige, with board membership reflecting recognition of position, power, wealth, or past contribution to the community. One person commented that the political nature of boards has changed “from prestige to work”! This change in the composition and political nature of community boards has intensified communication problems between consumers and providers. One participant suggested that there are two lan- guages which must be learned and spoken. One is the “cultural language of health,” reflecting consumers’ interests in services which typify the community as they know it; the second, the jargon of health care. Consumers need to learn this “tongue,” just as providers must learn the different cultural languages of health. Discussion of the two languages led to another exchange concern- ing the mystification (through professional education) and the demys- tification (through consumer—provider communication and familiar- ization with the jargon) of the health field. There may be an inherent threat to professionals in the demystification process, but that threat may be lessened with an awareness and acceptance that trust is a more mutually satisfying response than blind faith. Such a view suggests that providers can trust consumers when they do not understand the difference in the cultural language of health, and that consumers can learn to trust providers when they do not always understand the professional jargon of health. At heart throughout the discussion was an implicit question of the risks we are willing to take with each other in trying to achieve better health for our communities. Perhaps no one summarized better the challenge to consumers, providers, community, and communication than one commentator who suggested that we should join in this risk venture, since, in his words, “We can no longer lie; the people understand authenticity.” rm x malay Ei -—— St memes epee Ae 0 S— ie Seley Sm— - : ) 1 . i - in BN . . . , E - BN b I . N . -— — —— A CLOSING REMARKS “i tet a SUMMING UP Robert |. Haggerty I would like to begin by making a general point on semantics. We have used a lot of words that have different meanings to different ones of us. Some of these words are listed in table 1. In this whole field of evaluation, for the last 2/2 days, we have included epidemiol- ogy, clinical trials, the issue of efficacy, and a series of what could be labeled “types” of evaluation, all under the general rubric, evaluation. Dr. Yarbrough insisted on the word “assessment” if there was no intervention, and on “evaluation” if there was. I think he makes too fine a distinction, and I consider the two words as synonymous. Within evaluation, we have had several types reported, with very nice distinctions made between effort, effectiveness (namely impact on populations), efficiency, quality, and policy. In addition, some of the papers have described the methods themselves. Now to me this is a spectrum, and I have tried to classify the papers that have been presented along this spectrum. Dr. Yarbrough’s study was essentially a beautiful, classical clinical trial. We had several presentations of the effectiveness type (Pless, Ramsey, and Agudelo-Gil). Belville’s paper was a little more difficult to classify because it was a kind of “operations research.” It did not have the cost aspect that efficiency studies usually have; however, it was on the efficiency side. Then we had several studies on methods (Kessner, Martini, and Heredia). I found the rather tongue-in-cheek classification that Dr. Velasquez gave us on evaluation research very useful, and I have added this to table 2 to illustrate how these other words can be used. I do not know whether Dr. Velasquez would want to acknowledge the fact that academic research produces only papers, Government research pro- duces policy, providers are interested in monitoring operations, consumers are interested in satisfaction, professional evaluators are interested in methods, and multisectoral evaluators are concerned with change. At least this outline put a lot of the words that we have bandied about here into some sort of perspective, and that may help us as we continue. Next, I want to mention three or four major limitations concerning evaluation. The first is the lack of methods. Clearly we have had a few exciting methods presented to us, but an enormous number of problems remain unsolved because we do not have methods for solving them. For example, the monitoring of a community's health needs is to me one of the high priority items. Yet the methods we 271 272 EVALUATION OF CHILD HEALTH SERVICES Table 1. The Field of Evaluation Papers Presented Epidemiology Clinical Trials Yarbrough Efficacy Studies Evaluation-Assessment Effort Ramsey, Pless Effectiveness Agudelo-Gil Efficiency Belville Quality Williamson Policy Kessner, Martini Methods Heredia Table 2. Velasquez Classification of Health Services Evaluation Research Sponsor Product Academic Research Papers Government Policy Provider Monitoring of Operations Consumer Satisfaction Profession of Evaluators Methods Multisectoral Research Change used in Rochester to monitor or measure community needs and to make the community diagnosis are too expensive to apply in most communities. We need simpler methods for measuring these needs. Some researchers suggested that all we have to do is ask the public about community needs. I am not convinced that one can query only the self-appointed leaders in the community. There has to be a mechanism to reach the disenfranchised and the group that is inarticulate about its needs as well. Key informant interviews should be based on a representative sample of the public. Secondly, one has to face the trade-offs between different levels of sophistication in methodology. The academic researcher may well aim at obtaining a small standard error in his method, yet that may not be necessary in the operating agency. We must see to what degree error can be tolerated in order to make the kinds of decisions that we need to make. The next limitation in the field of evaluation—and one that is clearly made by Dr. Kessner—is the lack of efficacy studies. We really have relatively few medical interventions where the data on the efficacy—that is, on the benefit of any given treatment—are clear. Without that information, most evaluation programs must resort to measuring the extent to which people use the services, the ability of the community to bear the cost, and the satisfaction of the people. These are important issues, but in the absence of efficacy, I think they are very limiting. Community Education 273 The minimal role that health services play in producing health is another limitation. Dr. Wray emphasized the relation of the economy and its changes to child health; Dr. Martini also alluded to this relationship in his paper. Unfortunately, we lack good measures to indicate those components of health that can be influenced by health services. Therefore, in evaluation studies, we are frequently measur- ing or trying to measure the outcome of a variety of specific treatments that have been put together as a program. The lack of cost data presents an additional limitation. I was impressed by this lack in the case studies. Perhaps this is a reflection of my views in 1977 in the United States, but I have been guilty of ignoring costs for the last 15 years. I think it is very clear that we will no longer have this luxury in the future. We are all facing limited resources, and evaluation must show how wisely these resources are being used. At the next conference I would expect to see much more sophisticated analysis of the costs of these programs in dollars, in people, and in the trade-offs that these dollars and people might buy when used in other areas. Finally, the issue of controls or comparison areas came up in several of the discussion groups. This area is a subset of the problems of methods. We are not going to have the luxury of random assignment of patients or programs to communities in most settings or in most medical care studies. We are going to have to use some methods of multivariant matching in order to know whether we have a comparison group that is similar to the experimental one. The sophistication of that technique has certainly not been advanced very much up to now. Lest we in health services think that we are the only ones with these problems, I would like to quote from an introduction in a recent book on evaluation studies by Dr. Glass (1976). It is rather rhetorical in wording, but I hope the message comes through. He said, The methodologic and philosophical literature on evaluation is distressingly repetitious. Writers in one field exhume the same dead horses that writers in a second interred years before. One who reads widely is in jeopardy of gagging from repeated swallowing of dry banalities about how evaluation can be distinguished as either process or outcome. Or how relationships between program person- nel and evaluators must be handled sensitively. Or how evaluation must serve decision makers. Scholars in each new area of evaluation need not lumber xenophobically up the evolutionary ladder, acting as though the problems in their field are so unique that no one can possibly have addressed them before. 274 EVALUATION OF CHILD HEALTH SERVICES I will probably labor up some of those same ladders in this discussion by relating some of the practical issues in evaluation research. However, I do not think I will be alone. Those who have tried to evaluate everything from defense to education, to other human services have faced the same kind of problems. The concep- tual framework for all evaluation is so simple—it is to try and look at input as related to output or to outcome and to see what intervening variables may modify that outcome. Workers in the field have made a useful distinction between formative and summative evaluation. Formative evaluation is the development of the actual project as you go along, with feedback to change the program, while summative evaluation is an overall evaluation, once the project has been completed. By and large, the problems of evaluation research are not in the conceptual framework. That is simple, but there is no doubt that we have difficulty carrying out the research because of some limitations that I have mentioned before. Now I would like to go through what I consider the seven steps in evaluation research and comment on each of these (table 3). The first step—to get the program to specify its goals and objectives and its underlying assumptions in measurable terms—is an exceedingly difficult one. One of the differentiations that people make between other research and evaluation research is that in the latter, one often can tell only whether a program has in fact met the objectives set by the program. One cannot say whether some other method of meeting those objectives would be better. The first step, obviously, is to get the program to state its purpose very specifically. The researcher may want to do something in a different way, but as an evaluator, he must have the program say what it wants to achieve. The next step— describing the program or defining the independent variable—is also a very difficult task. Whether one implements the program personally or whether one tries to evaluate a program that is in place, one of the very great difficulties is to say specifically, from one place to another, what was accomplished. How does one control for this independent variable? A man who is doing a considerable amount of evaluation research in the education field recently told me that the use of onsite recorders to determine what was actually going on in the program is one useful way. He suggested that we use people who were jointly responsible to the program and to the evaluators to describe what, in fact, was delivered by the program and the details of how the program was carried out, thereby specifying the independent varia- ble. He found this procedure very useful in evaluating some of the Title I educational programs. Very often the differences that we find in outcomes between one program and another—when the programs are ostensibly the same— Community Education 275 Table 3. Steps in Evaluation Research Specify goals, objectives, underlying assumptions of program. Describe program (independent variables). Define criteria for goal achievement (dependent variables). Develop and carry out measures of achievement. Describe intervening variables—the political and social milieu. Analyze results. Implement findings. MS Th LR BO ~~ are because they have really delivered something different. The independent variable is not the same. I might say that when we tried this approach in a program in Rochester—putting an onsite person in our neighborhood health center to describe what actually went on—there was a strong tendency on the part of the observer to . gradually identify with the staff. She felt increasing loyalty to the providers, and we lost access to these data. So I am not at all sanguine about getting this kind of material. Defining the setting in which the program develops presents another difficulty in determining appropriate treatment. It is not just what is delivered, but the whole milieu in which it is delivered that becomes important in evaluation. The implementation of the U.S. nationwide Quality Assurance Program—the PSRO Program—is a good example of this problem. We will have some 200 of these programs in place; approximately half are already in place. Several of us who were charged with some responsibility for evaluating this program have argued that we needed some kind of planned experi- ment. We had to know what was being done, and we had to have some control over where it was being done so that the milieu in which one demonstration was being carried out could be compared to another. But we were overruled for political reasons, and the program was allowed to develop wherever people had the most interest and therefore developed it most rapidly. Under these circumstances, it would be foolish to compare areas with and without a PSRO. Of course, the very reasons people developed these pro- grams first made their settings very different. I am reminded of Lewin’s comment that one way to understand the system is to try to change it. I think that is one way in which we can move from the passive evaluation of what is going on to a more active intervention— the one Dr. Yarbrough defines as evaluation, rather than assessment. At the other end of the spectrum is another type of research— secondary analysis. In my opinion, evaluation research has suffered from too little secondary analysis of data. Most of the large-scale programs have an enormous amount of data that others have collected. In our Rochester study, I know we have data that are 276 EVALUATION OF CHILD HEALTH SERVICES untouched by human thoughts! We offered the computer tapes to people by advertising in Medical Care, but very few have ever taken up the offer to do secondary analysis of the tapes. That is part of the academic paper business—you have difficulty getting papers pub- lished that consist of only data from secondary analysis. However, it is an area where evaluation research could profit from the example of some of the other sciences, such as economics, where secondary analysis has in fact become an honored approach. The third step of evaluation research is to define the criteria for goal achievement, that is, the dependent variable. Perhaps our major need here is to develop some proxies for outcome; this need was certainly expressed by Dr. Pless in the discussion on chronic disease outcomes. It is very difficult to measure specific functions, and there are trade-offs between one function and another. I think we must be very creative in developing skills to find proxies for outcomes, instead of relying on direct outcomes. The next step is to define the intervening variables—the political and social milieu. This step has been given too little emphasis in this meeting. Changes in the political system or social milieu affect the most carefully planned evaluation. In Boston some years ago, we carried out a very well-controlled medical care study, randomly assigning families to the new program and watching the control groups go through the system in the old fashion. In the midst of this study, however, a local program of great importance—a change in the welfare program of reimbursement of the poor—was instituted. This change had a great effect on the utilization of the health services by both groups of families. It overwhelmed the research intervention. In Rochester we were monitoring the child health services, and in midstudy there occurred the most profound effect on child health that we have experienced: liberalized abortion in New York State. The number of children born each year dropped by 35 percent in the first year and 40 percent in the second. The drop in the birth rate had far more influence on child health than did the neighbor- hood health centers, the chronic disease programs, or anything else we were doing and studying. Similar types of intervening variables are very likely to occur. The issue is to have a monitoring system, so that when the natural experiment occurs (something you never can really plan for), you can document what has happened and the real effect of this intervening variable. The next step is the analysis of results. Briefly, it is obviously an important issue, but we have not given it much attention at this conference. However, with the modern methods of multivariant regression analysis, one can extract relationships of great importance from data that on the surface do not seem capable of providing a clear answer. Community Education 277 The final step is the implementation of findings. We have alluded to this process several times during the course of the meeting. Essentially, evaluation is action research, and it should lead to some change or implementation. I think we all agree with this statement. Therefore, before the project is completed, we must see just how the results will be used—or at least plan for them. I must say I have been guilty of ignoring this step in the past. In a current project we are just starting—the evaluation of child health services in the United States by selecting a series of communities and studying the total picture of child health in those communities—we are engaged with the Kennedy School of Public Policy at Harvard in planning for implementation of the program ahead of time. We plan to recruit a faculty member jointly with that school, and his or her primary responsibility will be the implementation of the program. The plans call for a series of meetings so that as the results emerge they will be transmitted to local officials, planning agencies, health departments, political groups, and to those responsible for decisions at the national level through staff meetings with congressional health workers and other people in the political process who are key to setting up the medical care system in this country. I have no personal experience with this type of planned attempt, although members of the political science and government departments seem to use it quite expertly, and it is an important aspect of medical care evaluation research that we have ignored in this conference. It is difficult to determine what programs are worth evaluating. Table 4 may give the evaluator some guidelines in making his choice. Those in charge of the program do need to be able to state the objectives of the program, and these objectives have to be measurable. There must be some testable assumptions that link the program activities to the accomplishment of these objectives. Sometimes you run into a study group that says, “We're doing this, and it will result in this,” and you say, “Well why?” and the group says, “Well, I haven't a clue, you know.” I think there ought to be some sort of assumption that A will lead to B or that A will at least have some relationship to B. It is most important for those in charge of the program to have some motivation, ability, and authority to run the program—that is, to alter the resources—and to meet the objectives of the program. In other words, the group conducting the program must have some sort of motivation to change what it is doing as a result of the evaluation. These problems I have raised may make evaluation research appear to be a very risky business and one more involved with minor adjustments in ongoing programs than in measuring outcomes of real importance to health. While this observation is often true, I also believe that creative and innovative efforts in evaluation can lead to 278 EVALUATION OF CHILD HEALTH SERVICES Table 4. Guidelines in Selecting Programs for Evaluation Those in charge of a program must set objectives that are measurable. A program must have testable assumptions linking program activities to accom- plishment of objectives. 3. Those in charge must have the incentive, motivation, ability, and authority to run the program (to alter resources, task, or objectives). No — In the absence of these, performance analysis may be all that is possible since overall program impact is not possible. significant insight into what works and what does not. It may seem that such research is very ad hoc and empirical, but I think the best of evaluations can occasionally lead to the development of new theories and to major change in human services programs. For example, our finding of decreased use of services by children at a time of increased availability of services and decreased financial barriers was to us an unexpected result in Rochester. At least, my economist friends all look at the outcome and say, “That's wrong; it doesn’t happen that way.” In turn we are led to ask, “Well, what is wrong with the traditional theory? Have we found something new here?” At least we should be thinking how we can reformulate the theoretical basis of the project in evaluation research, and sometimes I think that effort is important. I would like to bring up a few additional points in conclusion: the relation of evaluation to education and the issue of values. The relation of evaluation to education. The attitude of how one looks at the services that he is being trained to provide is important. I personally believe that a critical, questioning view of what health providers can do is an important part of medical education. For too long we have given the impression that providers are omnipotent and that we are great healers and savers of mankind. Evaluation research can be a very potent antidote to that impression, but there is a danger here. The problem is one of balance. I have had pediatric residents look at their work so critically that they come away very negatively: “Why stay in medicine? Nothing we do does any good.” You have to weigh this attitude on what I call the doubt/confidence axis. In dealing with the individual patient, physicians and other providers really need to project a kind of confidence—I am convinced that is a large part of the therapeutic benefit. At the same time, physicians must not be allowed to feel so omnipotent that they never question what good they are doing. In most places, I think we have failed to achieve that balance between doubt and confidence. Good evaluation research could be a vehicle for teaching that balance in medical education. Community Education ~~ 279 The issue of values. We have had a spirited discussion in one of our groups about values. I think it was a communication problem rather than a difference in opinions. But clearly values do dictate the fact that we evaluate a program in the first place. A lot of people with different kinds of values will say the service provided is enough of a reason for having the service and that evaluation is unnecessary. The solution is to try to make these values explicit at the start of any program. As one speaker said at the beginning of the conference, we should “put the biases out on the table” at the beginning of any program. My own value is that evaluation research has modest contributions to make. We cannot expect evaluation research to tell us if we should have a national health program in this country. We cannot expect evaluation research to tell us whether equity of access is a value that society should seek to achieve. However, once the body politic, or indeed once we as individuals in our nonprofessional roles, have lobbied for such effects (and, it is to be hoped, achieved them), then evaluation research can identify how well those goals are being met and can compare alternative ways to achieve those goals. Evaluation research is being asked to achieve too much if it is asked to set the goals. It can help outline the options, the alternatives, and particularly the priorities, or at least the cost of different ways of meeting the priorities. By placing excessively high expectations on evaluation research, we also face the danger of putting too much of our resources into evaluation. The trick is to tease out those few instances where evaluation can measure something and effect change. That selection process is still an art, in my view—one that we can hope is learned by performance. Perhaps conferences like this one advance this skill. In conclusion, I would say that evaluators will not inherit the earth, but they may help to make it a somewhat better place in which to live. REFERENCES Alpert, J.J, Robertson, L.S., Kosa, J., et al. (1976) Delivery of health care for children. Report of an experiment. Pediatrics 57:917. Glass, G.V. (1976) Education Evaluation Studies. Review Annual, Vol. 1. Beverly Hills: Sage Publications. Haggerty, R.J., Roghmann, K. J., and Pless, I.B. (1975) Child Health and the Community. New York: Wiley Interscience Series. Struening, E.L., and Guttenog, M., eds. (1975) Handbook of Evaluation Research. Beverly Hills: Sage Publications. r- oo Lowe A of = - ¥ . » . ™~ . EY = - kd . - - - = ln a? an Su TEEN en Sami wes