LC \ 213.22 C2 D38 1988 IGSL UC B IUCATIDN Policy Paper No. PP88-4-3 Peninsula Academies Replications: 1986-87 Evaluation Report Charles Dayton, Alan Weisberg, E David Stern, John Evans April 1988 ‘ " . NHL TE 0? GEBVERRHRE mgmémmes Lia“?! Directors M .13“? :3 :3: 3:59;: w. G thrie WGRNM Unifgsfify of Cglifornia QNNERSITY 0%: {M Berkeley Michael W. Kirst Stanford University Policy Paper No. PP88-4-3 Peninsula Academies Replications: 1986-87 Evaluation Report Charles Dayton, Alan Weisberg, David Stern, John Evans April 1988 Charles Dayton is a policy analyst with PACE. Alan Weisberg is a policy analyst with PACE. David Stern is an associate professor of education at the University of California, Berkeley. John Evans is a senior policy analyst with PACE. This paper was sponsored and published by Policy Analysis for California Education, PACE. PACE is funded by the William and Flora Hewlett Foundation and directed jointly by James W. Guthrie and Michael W. Kirst. The analyses and conclusions in this paper are those of the authors and are not necessarily endorsed by the Hewlett Foundation. This report was funded with special grants from the Edna McConnell Clark Foundation and the William and Flora Hewlett Foundation. The authors wish to acknowledge the assistance of Josh Hirshstein, Estelle Barber, Dana Hollister, Il-Woo Paik, and Susan Sather; Robert Rowe of the State Department of Education; James Guthrie, Jacob Adams, and Linda Humphrey of PACE; and all site managers, who provided helpful criticisms and assistance in gathering data. Additional copies of this paper, PP88-4-3, are available by sending $10 per copy to: PACE School of Education University of California Berkeley, California 94720 CHECKS PAYABLE TO THE REGENTS OF THE UNIVERSITY OF CALIFORNIA (California residents add appropriate sales tax.) Policy Paper No. PP88-4-3 Policy Analysis for California Education (PACE) Berkeley, California April 1988 ii Executive Summary In the fall of 1985, 10 academy programs were established by the State of California as replications of the Peninsula Academies. Policy Analysis for California Education evaluated these 10 academies in 1985-86. This report presents findings from a second evaluation covering the academies' 1986«87 school year. Student Outcomes The most important finding from this second-year evaluation is that the academy program model is capable of producing substantial positive effects on participating students, but that developing an effective program is difficult. This has been clearly accomplished in three sites. Attitude improvements among students were registered at most sites: program students noted a strong connection between their schoolwork and outside employment, liked the academy better than the regular school program, found their academy classwork more interesting than their regular school program, highly rated the academies' equipment and materials, and improved their self esteem by participating in the program. In 3 of the 10 sites studied, academy students registered significant improvements on objective measures of school performance. When viewed against matched comparison groups not participating in the academy programs, academy students recorded higher attendance rates, earned more academic credits, achieved higher grade point averages, and failed fewer courses. There are also preliminary indications that these successful sites may be reducing the dropout rate, but the final assessment on this question must await subsequent data on high school graduation rates for academy and comparison-group students. Significantly, the ability of the academy model to accomplish these outcomes is not limited to socially and economically congenial settings. The three clearly successful sites included a small, suburban, predominantly white high school; an average—sized, urban, predominantly black high school; and a large, urban high school with an ethnically diverse population. Two sites presented evidence of not succeeding. One never implemented anything approaching the academy program model, and outcome data were not gathered there. This site closed in June 1987. The other has a record of findings favoring comparison rather than academy students, although not generally at a statistically significant level. Rigorous statistical procedures were used in this study to judge the academy programs' effectiveness. These procedures offer an ambiguous picture at five sites. While the vast majority of differences at these sites favor academy students, most are not statistically significant. Because the programs become fully operational only during the 1987-88 school year, when all three classes of students are in place, evidence from the next evaluation, covering the 1987-88 school year, should allow clearer judgments regarding these sites. Another way of approaching the academies’ performance to date is by viewing them as a group. Overall they have been successful in helping students compile better academic records, as reflected in larger numbers of course credits, better grades, and fewer courses failed. For attendance and dropout rates, there is no clear evidence that similar gains have been made by academy students, though at the three successful sites there are preliminary indications of fewer dropouts. These findings are generally consistent with those reported by PACE in the 1985- 86 evaluation of these programs, but they indicate a somewhat higher level of effectiveness, and stronger data underlie the findings. In particular, more and better data were secured on comparison students, and a more sophisticated analysis of these data greatly strengthened the evaluation's conclusions. Overview of the Academies Model During the 1986-87 school year, the academies (now called Partnership Academies) operated in 10 high schools: four in the San Francisco Bay Area, three in or near Sacramento, two in Stockton, and one in Bakersfield. Each received a $50,000 grant from the state under Assembly Bill 3104, with matching funds required from both school district and cooperating business supporters. An academy involves a modified high school program with a number of specific elements, including: - A student selection process designed to enroll students with academic potential, but whose past performance indicates they are in danger of dropping out - A school-within-a-school administrative structure, such that academy students in grades 10-12 enroll as a group in three core academic subjects, including English and mathematics, with selected teachers iv o A technical course in grades 10-12 designed to provide students with basic job skills in a promising labor.market field (e. g., computers, electronics, and health) located nearby 0 Support from local businesses, including curriculum development, guest speakers, field trip sites, mentors, and work experience positions 0 Both high school and district support for the program, providing the necessary teacher preparation time, facilities, equipment, curriculum development, and counseling support Evaluation Design and General Findings The 1986-87 evaluation focused on two areas. First, the evaluation addressed program implementation and the degree to which programs followed the academy model. There was considerable variation from site to site in this regard. On a 25-point scale, programs ranged from 7.5 to 23 in the degree to which they fully implemented the model. Program expenditures also ranng widely, with well implemented programs spending more money. The second focus of the evaluation addressed student outcomes and the degree to which these were influenced by the programs. The evaluation employed a comparison group design, in which a group of nonacademy students similar to those in each academy program was selected and tracked along with the academy students. Comparisons were made in terms of retention in school, attendance, credits earned, courses failed, and grade- point averages. The statistical model used to test for differences between program and comparison groups on these dimensions corrected for differences in prior school performance, gender, race or ethnicity, and date of birth. For the most part, implementation observations correlate well with what outcome data demonstrates about the academies' success or lack thereof. The three academies with the "best numbers" also rated highly in the extent to which they fully implemented the academy model. While the correlations were not perfect, the other sites with better implementation also tended to have better student outcomes. While full development of the model does not guarantee success, it appears to be a prerequisite to achieving significant positive differences in student outcomes. Recommendations Legislation (Senate Bill 605) is already enacted and in place to expand the number of academy replications up to 15 additional sites per year for four years, as well as to continue funding of existing programs. If the quality of implementation across sites can be improved and new programs can be replicated with greater consistency than has occurred with previous expansion efforts, the proportion of sites with clearly positive outcomes for participating students can be raised. A number of actions are needed to provide the best chance for such consistent implementation. 1. 1 he requirements 9155 all: shgulgl be carefully followed in establishing new academies. Grants should be provided only to districts with appropriate target populations that are willing to meet the matching requirements. Further, the programs should be implemented in full accordance with key features of the model. 2 i nt tr inin l r v' d t ll w it Such training should target district and school administrators as well as teachers. It should provide a full understanding of the academy model as well as address the necessary private-sector involvement. This training should occur during the planning period and be followed up during the first year of implementation. 3. Senate Bill 605 is designed around a performance- -based system of funding. For this system to Operate as intended MEMQZWMW car refully in acggr rdange with provisions of the legislation, A clearly articulated mechanism for such tracking must be established by the State Department of Education as part of its grants administration. 4. The State Department of Education should monitor program implementation and W The department should take action to correct inconsistencies where found and terminate the grants for sites that do not take corrective action. 5. The evaluation should be continued through at least June 1990. allowing for assessment of the academies’ effects on their first two classes of graduates. The academies offer a rare opportunity to widely replicate a program that is based upon a clear model that, when properly implemented, can lead to improved academic and career performance for at- risk students. It also offers a well structured, productive way of bringing together the education and business communities toward this end. If SB 605 is to be implemented, and many new academies created, further knowledge about their effectiveness and its causes is needed. vi Contents Executive Summary ...................................................................... iii Foreword .................................................................................. ix Policy Analysis for California Education .............................................. xi Part I: Overview of the Academy Program and History 1 Introduction ................................................................................ 3 Part 11: Case Studies 2 The Health Careers Academy, Bakersfield ........................................... ll 3 Edison Financial Academy, Stockton .................................................. 23 4 The Franklin Food Services Academy, Stockton .................................... 29 5 The Corporate Academy, Sacramento ................................................. 39 6 Computech Academy, Mountain View ................................................ 50 7 The Health Academy, Oakland ......................................................... 62 8 The Foothill Computer Academy, El Dorado Hills .................................. 73 9 The Rio Cazadero Academy, Elk Grove .............................................. 84 10 The East Side Electronics Academies, San Jose ...................................... 96 Part III: Cross-Site Analyses and Conclusions 11 Cross—Site Analyses and Conclusions ................................................ 117 Appendix A: Student Outcomes Tables, by Site ................................... 137 Appendix B: Academies Bill SB 605 ............................................... 147 Appendix C: Multivariate Analysis of Academy Program Effects ............... 153 vii Foreword The search for better ways to educate is as old as education itself. But perhaps more than in fields such as medicine and engineering, the history of efforts to improve education has been guided more by hope and belief than by data and research. The study reported here, designed and carried out by PACE analysts Charles Dayton, Alan Weisberg, David Stern, and John Evans is an important contribution to an emerging discipline in education which seeks to change tradition and to assess educational innovations and reforms by the most rigorous research methods possible. In this evaluation of the Peninsula Academies, a school-within-a-school program designed to give disadvantaged high school students employment skills and prevent them from dropping out, Dayton and colleagues have utilized a longitudinal, quasi-experimental design to assess the program's effectiveness. They found that when the academy model is properly implemented, it can significantly improve student attendance, grades, and motivation. The findings of this evaluation are important to educators and policy makers in several ways. First, they provide evidence for the potential effectiveness of an important educational program for disadvantaged high school students, a group which if not aided effectively will join the list of dropouts and the unemployed in high numbers. Second, the PACE evaluation makes clear to administrators of the academy program that full adherence to the program model is essential if its benefits are to be achieved. Finally, because of the strong methodology on which this evaluation rests, we can regard its findings with uncustomary confidence. This is crucial if, in education as in other traditionally more rigorous disciplines, we are to avoid wastefully pursuing unproductive programs and methods, and instead concentrate education's limited resources on ones we know can produce the improvements that are so critically needed. James W. Guthrie, Professor of Education , and co—director, PACE University of California, Berkeley Policy Analysis for California Education Policy Analysis for California Education, PACE, is a university-based research center focusing on issues of state educational policy and practice. PACE is located in the Schools of Education at the University of California, Berkeley and Stanford University. It is funded by the William and Flora Hewlett Foundation and directed jointly by James W. Guthrie and Michael W. Kirst. PACE operates satellite centers in Sacramento and Southern California. These are directed by Gerald C. Hayward (Sacramento) and Allan R. Odden (University of Southern California). PACE efforts center on five tasks: (1) collecting and distributing objective information about the conditions of education in California, (2) analyzing state educational policy issues and the policy environment, (3) evaluating school reforms and state educational practices, (4) providing technical support to policy makers, and (5) facilitating discussion of educational issues. The PACE research agenda is developed in consultation with public officials and staff. In this way, PACE endeavors to address policy issues of immediate concern and to fill the short- term needs of decision makers for information and analysis. PACE publications include Policy Papers, which report research findings; the Policy Forum, which presents views of notable individuals; and Update, an annotated list of all PACE papers completed and in progress. Advisory Board Mario Camara A. Alan Post Partner - California Legislative Analyst, Cox, Castle & Nicholson Retired Constance Carroll Sharon Schuster President, Saddleback Executive Vice President Community College American Association of University Women Gerald Foster Eugene Webb Region Vice President Professor, Graduate School of Business Pacific Bell Stanford University Robert Maynard Aaron Wildavsky Editor and President Professor of Political Science The Oakland Tribune University of California, Berkeley xi Part I Overview of the Academy Program and History chapter 1 Introduction For the past two years, 10 academy programs have been operating in California under state sponsorship. These programs are replications of the Peninsula Academiesf which themselves have been operating since 1981. The academies are directed at reducing dropout rates among "at-risk" high school youth. They combine a modified high school curriculum and structure with exposure to local companies and jobs. Their intent is to improve students‘ motivation to perform well in school, graduate, and make a successful transition from school to work. The 10 academies receive state funding from Assembly Bill 3104, passed during the 1984 legislative session. That bill also included a requirement to "evaluate the Peninsula Academies model program and report to the Legislature regarding issues which shall include, but not be limited to, the cost-effectiveness and educational quality of the program." This 1986-87 evaluation report addresses that requirement. The evaluation has been conducted with joint support from the Edna McConnell Clark Foundation and the William and Flora Hewlett Foundation. Academy History The two Peninsula Academies originated in the fall of 1981 in the Sequoia Union High School District They were designed from extensive research and modifications of model programs begun elsewhere. Their purpose was to reduce high dropout rates among at-risk students in that school district. Three years later, their performance was sufficiently compelling for the California State Department of Education to declare them a model program, and Assembly Bill 3104 was passed to replicate them in 10 high schools around the state. What is an academy? It involves a modified high school program with a number of special elements, including: - A student selection process designed to enroll students with potential, but whose past performance indicates they are in danger of dropping out * Sequoia Union High School District, Redwood City, Califomia 4 PENINSULA ACADEMIES 1986-87 EVALUATION ' A school—within-a-school administrative structure, such that academy students in grades 10-12 enroll as a group in three core academic subjects, including Enghsh and mathematics, with selected teachers 0 A technical course in grades 10-12 designed to provide students with basic job skills in a promising labor-market field (e. g., electronics or health) located nearby 0 Strong support from local businesses, including curriculum development, guest speakers, field trip sites, mentors, and work experience 0 High school and district support for the program, providing the necessary teacher preparation time, facilities, equipment, curriculum development, and counseling support The academies address a significant problem. The high school dropout rate in California is approximately 30 percent. Among urban schools and minority youth, this figure often surpasses 50 percent. The loss to these young dropouts is enormous, both in terms of their self esteem and financial future. At the same time, due to the "baby bust," there is a declining number of graduates and an estimated 40 percent drop in the number of youth who will be in the work force by the year 2000. In many areas, a shortage of trained entry-level employees is occurring already. Thus, on one hand, youths are leaving school with no diploma and limited job skills; on the other hand, the economy is suffering from a lack of well prepared entry-level workers. Given today's intensely competitive international marketplace, high school performance and students' transition to work are issues important not just for California youth or local businesses, but for the nation as a whole. The current academies represent threeeway partnerships among the state, local school districts, and supporting companies. The state provides a $50,000 grant to each district, which must be matched in direct or in-kind support by both the receiving district and local business community. Thus, the funding mechanism is designed to encourage cooperation among school districts and private-sector companies in preventing dropouts. During the 1987 legislative session, Senate Bill 605 was passed to expand the academy replications up to another 15 sites per year for four years. The bill was signed by the governor, but funds were not appropriated for the replications. The current academies continue to be supported under the established system. However, the governor has included the necessary funding in the proposed 1988-89 education budget, and six new programs are currently being selected; additional sites may be selected this fall. Replication sites will receive planning funds in the fall of 1988, leading to the creation of new academies by the fall of 1989. INTRODUCTION 5 Senate Bill 605 changes the system by which academics are funded. Instead of a grant provided at the beginning of a school year, districts will receive an amount based on the number of students kept in school who are making normal progress toward graduation, as determined at the end of the school year. Thus, the programs will survive if they are successful; they will be phased out automatically if they are not. The required matching support from sponsoring districts and local business communities will continue. (A copy of SB 605 is attached as Appendix B.) Site Descriptions The 10 academy programs are located in eight school districts: four in San Francisco Bay Area high schools, three in or near Sacramento, two in Stockton, and one in Bakersfield All sites were selected through a grants competition conducted in early 1985 by the State Department of Education. The "Academy School Profile" chart on the next page provides information on these 10 high schools, including enrollment, percentage of limited-English-proficient (LEP) and AFDC enrollment, number of graduates, and student ethnicity. Evaluation Design The evaluation includes both process and outcome components. The process evaluation examines successes and problems regarding implementation. This information derives from site visits; interviews with school staff, students, and business , representatives; and questionnaires completed at the end of the school year by all involved parties. The process findings are presented in the first part of each case study in Part II and summarized across sites in Part III. The outcome evaluation employs a comparison group design. This means that a group of students not participating in the academy program was selected at each high school and matched with academy students, and the performance of both groups was tracked. The matching was done in terms of descriptive indicators (age, grade level, gender, and ethnicity) and school records (attendance, credits, courses failed, and grade point averages) from the year prior to the academy students' entry into the program. Both academy and comparison group students were then tracked on these same variables, as well as retention in school. This enabled evaluators to better measure the academy programs' effects on students. ACADEMY SCHOOL PROFILES STU [ZNT ETHNICITY (7o) SANTA CLARA COUNTY , GRADES ENROLL« % GRADS “7° AMER 0A0 SCHOOL/COUNTY SERVED MENT LED 1985 AFDC IND ASIAN 15m FIL. HISP. BLACK WHITE WOUNTA‘N WW ““3“ SCHOOL 9-12 1,326 354 255 1.87 0.00 10.94 000 4.37 8.52 4.60 71.5.7 5A111A CLARA 0001111 ERANKL'NSEN'OR W3“ SCHOOL 9-12 2,228 23.61 262 30.87 0.85 12.52 0.00 1.97 37.21 11.94 35.50 DAN JOAQUIN COUNTY EASTBAKERSF'ELD H’GH SCHOOL 9—12 1,836 316 227 11.26 0.71 0.38 :200 0.44 42.21 9.04 47.22 KERN COUNTY 1210 CAMDERO H'GH ‘5CHOOL 11 — 12 198 6.57 37 N01 3.03 0.51 0 00 3.54 20.20 20.71 52.02 SNZPAMENTO 0001111 AVA1L H'RW WOHNSON H‘GH SCHOOL 9-12 3019 865 486 41.03 1.09 17.30 000 3.25 14.05 18.82 45.49 54111811151110 0001111 WLANDTECHN'CALH'GH SCHOOL 9—12 1,784 14.97 181 35.75 0.39 5.10 8.02 1.29 2.24 78.42 4.54 ALAMEDA 0001111 W R'DGE H'GH SCHOOL 9— 12 950 0.00 160 2.04 0.00 0.21 1.05 0.00 2.11 0.00 96.65 EL DORADO 0001111 59'5”” SEN'OR “'6“ SCHOOL 9-12 2,272 23.02 253 36.24 0.22 15.05 0.00 10.83 37.72 21.39 14.79 8411 JOAQUIN 0001111 S'LI‘ER CREEK “'9“ SCHOOL 9—12 2,352 23.34 298 15.79 1.11 18.03 0.51 8.06 30.87 12.71 28.61 541114 CLARA 00111111 INDEPENDENCE ”'9“ SCHOOL 9—12 4079 31.14 704 9.70 1.03 19.83 0.29' 11.28 28.68 7.50 31.38 NOILVH’IVAEI LEE-9861 $311413on V’IflSNlNEd INTRODUCTION 7 In addition, the outcome evaluation includes information from two additional sources on academy students. The first was a student questionnaire that collected information on attitudes toward careers, school, and self, and tracked changes from year to year. Questionnaires were administered before students participated in the program and at the end of each school year. A series of items related to students' socioeconomic status was included in these questionnaires. In addition, a standardized achievement test was administered to academy students each spring, tracking their progress from year to year on this dimension. The 1985-86 report uncovered a number of weaknesses in the evaluation itself. Among these were lack of matched comparison groups at four sites, the apparent lack of comparability of some existing comparison groups, and failure to collect all intended data in some sites. These problems were due largely to inadequate staffing and support for the evaluation. In contrast, due to two private foundation grants for the 1986- 87 evaluation, these problems have been corrected, so that the data base and quality of analysis have been substantially improved. While certain problems still remain, on balance the information available this year allows for considerably firmer conclusions than could be made a year ago. Report Overview Part I includes an overview of the academy program and history. Part H includes individual site case studies. There are nine case studies; the two academies in the Eastside Union High School District are jointly administered and identically structured, and they are treated as one case. Each case study contains several sections, including: - A program description, which includes sections on setting, student profile, program management, and program resources - A summary of student outcome data on retention, attendance, credits earned, courses failed, and grade point averages, comparing academy and comparison group performance 0 Questionnaire responses from students, school personnel, parents, private-sector representatives, and mentors - A summary 8 PENINSULA ACADEMIES 1986-87 EVALUATION The one exception to this pattern is Edison High School in Stockton, vvhere the program was so incompletely implemented that a full evaluation proved impossmle. Part 111 provides a cross-site summary of the findings with respect to program implementation, student outcomes, and questionnaire responses. It also offers some conclusions about what is needed to implement a successful academy° Finally, a brief section of recommendations is included. Part II Case Studies chapter 2 The Health Careers Academy East Bakersfield High School Kern Union High School District Bakersfield, California The Health Careers Academy in Bakersfield, California struggled with substantial administrative and personnel problems during its first year. While problems remained in 1986-87, the program stabilized and progressed. Two dedicated full-time teachers, who had a common planning period and worked together effectively, provided a solid foundation. A "home base" classroom was allocated to the program, enhancing its identity. The school supported the program administratively. Private-sector support developed, including speakers, field trip sites, and mentors. Students testified that the program is having a positive influence on them. While there is still room for improvement, the program seems to be establishing itself at East Bakersfield High School, and talk of adding another academy in the disuict has begun. Setting Bakersfield is located in the California central valley, north of Los Angeles and south of Fresno. Its population is approximately 100,000, and its economy is based largely on agriculture and oil. East Bakersfield High School, one of 10 high schools in the district, was built in the late 19303 and is located on the east side of the city in a low- to middle-income neighborhood. Its student population of approximately 2,000 is composed of 46 percent white, 43 percent Hispanic, 9 percent black, and 2 percent other. Its parent education index puts it at the 26th percentile statewide. Eleven percent of its students come from families receiving AFDC. Student Profile In year one, academy planners conducted a substantial "advertising" campaign in 9th grade social studies classes, which resulted in 250 applicants, far more than the program could accommodate. In the spring of 1986, students themselves made these presentations, resulting in a list of interested students substantially smaller but more 11 12 PENINSULA ACADEMIES 1986-87 EVALUATION appropriate for the program. Program teachers then examined the records and backgrounds of applicants in the following respects: Aptitude in science, health, and ability to get along with people Patterns of underachievement in 9th grade, in terms of attendance, grades, and credits Emotional and family problems, including economic disadvantage (in order to include those with such problems as long as they were not too severe for the program to handle) Involvement in Gifted and Talented Education (GATE) classes or college preparation tracks (these students were eliminated from consideration) A search for at-risk students whom the program could serve effectively Table 2.1 presents a profile of academy students for 1986-87 by grade level, race/ethnicity, and gender. Data for comparison- group students are also presented. TABLE 2.1 Student Profile, The Health Careers Academy Academy Comparison Group Grade Level 10 39 62 1 1 48 53 Race/Ethnicity Asian 0% 0% Black 16% 9% Hispanic 50% 49% White 32% 41% Other 2% 1% Gender Female 54% 39% Male 46% 61% THE HEALTH CAREERS ACADEMY 13 Program Management Perhaps the academy's most serious problem arising in its first year concerned staffing. The two teachers involved in developing the program decided at the last moment not to participate. Thus, two teachers who were not involved in initiating of the program became its chief staff members, and neither was particularly committed to its purpose. One of these was pregnant and left during the fall semester. Her replacement, an experienced teacher with a strong health careers background, proved to be a strong addition, and this year became the lead teacher, handling the science classes. Additionally, a new English teacher was hired in 1986-87 who was committed to the program and its students. These two teachers proved to be a strong nucleus around which to build the academy. The program at East Bakersfield High follows the school-within-a-school academy model, but only in part. During its first year, students enrolled in three academy classes— English, science/lab, and math. This year, the 10th grade program was reduced to just two classes, English and science/lab. Eleventh graders had three classes—English, science, and science/lab. At both grade levels, the science/lab courses included some job preparedness training. English classes were also coordinated to some degree, including lessons related to medicine that employed literature, films, and vocabulary. While the senior curriculum was not fully determined at the time of this writing, it will probably include enrollment in a Regional Occupational Program (ROP) health-related training program, plus one academy lab class. Other program components were developed in 1986-87 as well, including: 0 Guest speakers, including an anesthesiologist, nurse practitioner, surgical transplant nurse, and dental assistant 0 Field trips to a dialysis center, birthing center, home for the deaf, and career day program - Social activities, including a welcoming picnic organized by juniors for sophomores, building a float for homecoming, baby showers for two academy students, and a teachers' birthday party planned by students . Improved incentive programs, including certificates for students who made the honor roll In addition, the school provided a temporary structure located on the campus as the science/lab classroom, which gave the program a physical identity and home base. Teachers provided personal and career counseling through close contacts with students. Parental involvement was developed, and parents supported various program activities. School and district administrators seem committed to the program, even to the point of continuing it without state funding if necessary. Indeed, the district is exploring the l4 PENINSULA ACADEMIES 1986-87 EVALUATION possible establishment of a Small Business Academy, evidence of the value it attaches to the academy approach. Attempts to develop private-sector support were also made during 1986-87. These had been largely abandoned the first year because of school-based problems. While no advisory committee was established, this option was being explored. About half the juniors had mentors. Speakers and field trips were organized, as detailed above. Jobs were sought for the summer. And the 1987-88 schedule was structured to allow one of the teachers to spend a half-day each day promoting private-sector support. Program Resources During the 1986-87 school year, the two full-time academy teachers spent a total of 2,690 hours teaching. Teachers also spent an additional 205 hours on curriculum development and student recruitment. School and district administrators (including the program director) contributed 335 hours of their time to the academy. Counselors devoted 30 hours, and secretaries devoted 295 hours. During the course of the year, participants from local businesses made an estimated contribution of 108 hours to the program. Two—thirds of this time was spent giving talks. The remaining one-third was used mainly in helping to manage and coordinate the program. Only 10 hours were reportedly spent in one-to—one mentoring activities with students. The Health Careers Academy used only one special facility: a computer room used for 10 periods during the year. Equipment and supply items included a monitor, VHS, and cart ($2,000), lab kits ($1,100), and books and magazines ($1,930)° These were reportedly purchased with school district general funds rather than the academy grant. No material contributions by businesses were reported. Student Outcome Data The most objective form of data by which to judge the effectiveness of an academy is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in this report address attitude changes and other subjective responses from program participants. THE HEALTH CAREERS ACADEMY 15 Table 2.2 presents differences in outcomes between academy students and comparison groups, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natural" units, i.e., the actual difference in attendance percentage, credits earned, grade point average, and courses failed. TABLE 2.2 Adjusted Differences Between Academy and Comparison Groups, by Class, The Health Careers Academy Courses Dropout Attendance Credits GPA Failed Probabilitv Class Entering Fall 1985 1985-86 Outcomes 5.5* 0.7 0.1 «0.2 —0.03 1986-87 Outcomes -3.7 1.9 0.1# -0.3 -0.08 Class Entering Fall 1986 1986-87 Outcomes 2.6 1.4 0.1 -0.2 I * Statistically significant difference # Estimated from separate regressions for academy and comparison groups I Insufficient number of dropouts for analvsis Only one difference in Table 2.2 is statistically significant: the first class of academy students had a higher attendance rate than the comparison group in 1985-86. Although the other 13 differences shown in the table are not statistically significant, 12 of them indicate better performance by academy students. Questionnaire Responses The following sections provide an overview of responses received through individual questionnaires. Those providing feedback included students, academy 16 PENINSULA ACADEMIES 1986-87 EVALUATION personnel, parents of academy students, private-sector participants, and mentors working directly with students. Student ue tionnaires Table 2.3 presents data from questionnaires administered to academy students. A brief version is administered to all students as they enter the program (pre), asking about their career-related plans and attitudes toward careers, school, and themselves. A longer version is administered at the end of each school year, asking again about these topics. This version also requests information about program activities and asks students to rate the program. Data are presented by class, so that one can see differences between this year's 10th and 11th grade classes. For the first class (11th graders), responses are presented separately for each year. Numbers reflect either the percentage of students responding (as indicated), or the mean number, depending on the form of the item, TABLE 2.3 Student Questionnaire Responses, The Health Career Academy (means [decimals] or percentages of respondents, as indicated) Class 1 Class 2 Pre* Spring 86 Spring 87 Pre Spring 87 Program Descriptive Information How many academy classes are you currently taking? n/a 3.8 3.0 n/a 2.3 Who have you asked about career or job information this year? a. Academy teacher n/a 60% 100% n/a 65% b. Other high school teacher n/a 10% 0% n/a 8% c. Parents or other relatives n/a 80% 40% n/a 31% d. Friends n/a 40% 30% n/a 46% e. Academy mentor n/a 0% 20% n/a 4% f. Employer n/a 10% 10% n/a 15% g. School counselor n/a 40% 20% n/a 35% h. School career-center staff n/a 20% 0% n/a 42% THE HEALTH CAREERS ACADEMY l7 Class 1 Class 2 continued Pre* Spring 86 Spring 87 Pre Spring 87 Students who had a mentor this year n/a 0% 30% n/a 11% Of these, those who found it useful n/a 0 ° n/a 0 Students who had a job arranged through the program a. Summer job n/a 0% 0% n/a 15% b. School-year job n/a 0% 10% n/a 0% Of these, those who found it useful n/a 0 0 n/a 0 How many guests from industry spoke in your classes this year? a. Fall semester n/a 5.6 6.6 n/a 4.6 b. Spring semester n/a 5.1 4.6 n/a 4.4 Those who found speakers helpful n/a 90% 100% n/a 81% How many field trips did you attend with the academy? a. Fall semester n/a 2.9 200 n/a 0 b. Spring semester n/a 3.2 1.6 n/a 0 Those who found field trips useful n/a 78% 89% n/a n/a Did you get the extra help you needed with your school work during the past year? n/a 80% 70% n/a 70% Who gave you the help you needed? a. Teacher in class with the problem n/a 90% 60% n/a 73% b. Another academy teacher n/a 60% 70% n/a 8% c. A peer tutor n/a 0% 10% n/a 0% d. Another student or friend n/a 60% 90% n/a 19% e. Parent(s) n/a 40% 40% n/a 27% 18 PENINSULA ACADEMIES 1986-87 EVALUATION Class 1 Class 2 continued Pre* Spring 86 Spring 87 Pre Spring 87 Program Ratings Academy equipment was good or excellent n/a 50% 30% n/a 1 1% Academy materials were good or excellent n/a 70% 30% n/a 22% Academy schedule better than regular one n/a 60% 60% n/a 52% Felt strong connection between what was studied in school and post- graduate plans n/a n/a 93% n/a 85% Liked academy better than regular program n/a 89% 89% n/a 78% Career-Related Plans Have definite plans for the first two years after graduation 10% 40% 60% o . If have plan, it includes a. A job 78% 67% 56% . . b. Junior college, college, or university 33% 78% 67% . 0 0. Vocational or technical training 11% 33% 33% - 0 d. Military service 11% 22% 1 1% ° ° e. Marriage and housekeeping 11% 0% 0% ° ° Have a long-term career goal 67% 78% 89% ° ° THE HEALTH CAREERS ACADEMY 19 Class 1 Class 2 continued Pre* Spring 86 Spring 87 Pre Spring 87 Attitudes Related to Career, School, and Self Feel it is very important to make career plans while in school 80% 90% 100% 0 0 Feel it is very important to gain job skills while in school 70% 90% 90% o . Found classwork interesting during past year 50% 70% 80% ° ° Feel good about self after past year 80% 100% 80% ° ' Would not want to be someone different 78% 100% 89% v - Feel success is not mainly due to luck 80% 80% 90% ° 0 * Adminstered as students enter program ~= Missing data n/a= Not applicable School Personnel Responses At the end of the school year, five members of the school staff completed questionnaires, providing their responses to the program. These included the school principal and vice principal, the two main program teachers, and a counselor. Their responses included the following: 0 The academy's facilities, instructional materials, and curriculum were all rated "good" to "excellent," while equipment was rated "poor." - Nonacademy teachers at the high school were rated as "slightly supportive" of the program, counselors as "somewhat supportive," and administrators as "very supportive." . Program planning was rated from "good" to "excellent." . Cooperation and support from the private sector was rated "not very effective." 20 PENINSULA ACADEMIES 1986-87 EVALUATION ' On a seven-point scale (1=very weak, 7 =very strong), the academy received an average rating of 5.4 "in concept" and 5.8 "as implemented." 0 The program's greatest weakness was its lack of private-sector support, public visibility, and equipment. ° The program's greatest strengths were its administrative support, high-quality staff, and supportive relationships between staff and students. P ntR ne Four parents of academy students completed questionnaires at the end of the school year. Their responses included the following: 0 Three parents rated the program as "excellent," the fourth as "good." ° All four reported that their children were "much more interested in school and learning" since joining the academy. - Parents were involved with the program in several ways, including attending school events, helping with social events, meeting with teachers at school, and helping their children with homework. - In response to how well they had been kept informed of their children's progress in school, all four parents responded "very well." - Suggestions for improvements included widening the program beyond health and obtaining more equipment. Private—Sector Responses Three private-sector representatives associated with the academy completed questionnaires at the end of the school year. Their responses included the following: 0 Asked whether the academy's curriculum would prepare students for jobs locally, one respondent said "yes," and two "don't know." 0 Asked about communications with the program, one rated this as "good," one "poor," and one "don't know." THE HEALTH CAREERS ACADEMY 2 1 On a seven—point scale (1=very weak, 7=very strong), the academy received an average rating of 5.5 "in concept" and 5.5 "as implemented." The program's greatest weakness was poor student attendance. The program's greatest strengths were teacher dedication and added direction for students' lives. Mentor Responses Five academy mentors completed questionnaires at the end of the school year. Their responses included the following: On average, mentors met 1.4 times with their students and talked with them twice by telephone. Asked if the program was providing students with job skills needed by area companies, all five mentors responded "don't know." Asked to rate the mentor program, one responded "highly useful," one "O.K.," one "not useful," and two "don't know." The program's greatest weaknesses were lack of a coordinating body outside school, student motivation, and scheduling. The program's greatest strengths were the concern of teachers and the potential for improving the workforce. Summary The Health Careers Academy made progress during 1986-87. A home base classroom was established, and two capable and dedicated teachers took charge of the program. The main weaknesses were insufficient private-sector involvement and support and the need for improved materials and equipment. Outcome data presented only one statistically significant difference between program and comparison group students (i.e., attendance during 1985-86). Yet of 13 differences between the two groups, 12 favored academy students. Responses from both student and adult questionnaires were also largely positive. 22 PENINSULA ACADEMIES 1986-87 EVALUATION This program seems poised for success, if it can marshall the additional resources needed for full implementation. In particular, more support is needed for the two lead teachers, both within the school and in developing private-sector involvement. If this is forthcoming, there is good reason to believe this will be a very successful academy. chapter 3 Edison Financial Academy Edison High School Stockton Unified School District Stockton, California The Edison Financial Academy, in the estimates of its own staff, began as less than a "real" academy, and it never overcame this shortcoming. It lacked many of the basic components that define an academy, including a school-within-a—school structure, coordinated technical and academic classes, a three-year sequence of courses, and business involvement and private-sector supports. Clear district and school administrative support for the academy concept and components was also missing. Indeed, when the district was asked by the State Department of Education this year to demonstrate certain of these features or lose its funding, the district chose the latter, and the program was terminated at the end of its second year. Setting Stockton is located south of Sacramento and north of Modesto in the California central valley. It has a population of approximately 150,000 and an economy based largely on agriculture and, more recently, services. Edison High School is an older facility located on the city's south side, in a low- to middle-income neighborhood. It has an ethnically mixed student population of approximately 2,300 (38 percent Hispanic, 26 percent Asian/Pacific Islander, 21 percent black, and 15 percent white). Its parent education index ranks it at the 12th percentile statewide. Thirty-six percent of its students come from families receiving AFDC. Student Profile A problem during the program's first year was that no clear student selection system was developed. The intent was that students with the usual academy profile would be identified, told about the program, and given an opportunity to enroll if interested. Instead, students were simply assigned to the program by school counselors who had little 23 24 PENINSULA ACADEMIES 1986-87 EVALUATION understanding of the academy's nature or purpose. As a result, the program became largely a repository for students whom counselors "didn't know what else to do With. This problem was recognized in 1986—87, and attempts were made to enroll appropriate students, using a voluntary process that included interviews, Average to below-average performing students were sought, particularly those with high unexcused absence rates. Unfortunately, by this time almost all the first-year students had left the program, so no multiple-year effect was detectable. Further, there was no attempt to limit new enrollment to the 10th grade, thus to begin a new three-year program. In fact, the fall 1986 enrollees came from all four grade levels. A profile of these students and their comparison group counterparts is presented in Table 3.1. TABLE 3.1 Student Profile, Edison Financial Academy Academv Comparison Grout) Grade Level* Fall 1985 entrants 30 -= Fall 1986 entrants 54 105 Race/Ethnicity** Asian 6% 5% Black _ 25% 17% Hispanic 47% 44% White 8 % 1 3 % Other 14% 22% Gender** Female 55% 61% Male 45% 39% * Few students that enrolled in the program in 1985-86 continued in 1986-87, so the number involved in each cohort, rather than a grade-level breakdown, is listed here, Each cohort contained students from several grade levels. There was no comparison group for the first year. ** Data on ethnicitv and gender are for the second cohort onlL EDISON FINANCIAL ACADEMY 25 Program Management Edison High School seemed a good choice for an academy because of its large population of minority students and its choice of banking as a vocational focus. Initial efforts to establish private-sector support proved promising, with several banks demonstrating interest in the program. Initial efforts at developing a curriculum were also encouraging—employability-skills training materials were obtained from the California Industry-Education Council and contributions were received from interested banks. A wing of the high school with four classrooms was refurbished to give the program a home. However, a number of problems hindered the program from the beginning, and these proved to be substantial. First, the vice principal and teachers involved in developing the grant application were not substantially involved in program implementation. With one exception, teachers were assigned to the program, rather than choosing it. The lead teacher was skeptical about the program and accepted her position under a threat of being laid off. The school-within-a-school structure with block scheduling and coordinated technical and academic courses never developed. Students had a business class, covering basic employability and banking skills, but only a few had any academic courses as part of their program. While these problems were recognized, few of them were solved in 1986-87. The business class continued, but coordinated academic classes with block scheduling were developed for only about a fourth of the academy students. In addition, few of the other activities called for in the model were developed. There were two business speakers, no field trips, no mentors, no summer jobs for juniors, few parent contacts, and no private« sector advisory body. In fact, by this point most private-sector interest had disappeared. A newsletter begun in the fall closed because of lack of interest. There was not enough student interest to initiate an academy club. Two teachers new to the high school were assigned part-time to the program (in English and history), and the two remaining full-time teachers (in business and math) tried to carry on. While they remained supportive of the program's intent, they recognized the degree to which this replication failed to meet it. As the math teacher said, "It is a program in name only." The history teacher confessed that the relationship between history and the program's business focus was unclear to him. Also, there was an almost complete absence of administrative responsibility for the program. The vice principal who wrote the grant application assigned implementation responsibilities to another vice principal, who was new to the school and who knew nothing of the program's origin or purpose. She was overwhelmed with other responsibilities, and in her own honest words, she "didn't do a thing. I touched base when 26 PENINSULA ACADEMIES 1986-87 EVALUATION I could." While the four teachers were given a common preparation period, they made no use of this to meet. The district administration also provided no guidance, and, in fact, was leaderless itself much of the year, with continual staff and superintendency changes. By the fall of 1986, the State Department of Education became aware of the degree to which the program deviated from the academy model and wrote to the district administration to insist that at least the school-within—a—school model be followed, with block-scheduled technical and academic classes. The district was given several weeks to show evidence that it met this requirement. When a site visit by the state administrator failed to demonstrate the required changes, the disuict itself decided to end the program at the end of the 1986—87 school year. Thus, the Edison Financial Academy was terminated. In spite of this, the school plans a program with a similar structure and purpose for next year, under new teacher leadership. Student Outcome Data The most objective form of data by which to judge the effectiveness of an academy is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. The evaluation design calls for tracking each of these indicators on a year-by-year basis, for both academy students and a matched comparison group. This provides a picture of developments over the three-year course of the program and provides a reasonable standard by which to judge academy students' progress. At Edison High School a number of factors made it difficult to implement this evaluation design. First, in year one the program did not follow the academy model of enrolling only 10th gaders, but accepted students at several grade levels. Almost all of these students left the program by the end of the first school year. A new group of students was admitted in the fall of 1986, again at several grade levels. This made year-to-year tracking impossible. Further, during the first year, the program chose not to track a comparison group of students (an option it was given). Thus, no comparison group data were collected. While a comparison group was identified and tracked in 1986-87, its data relates only to this year's academy students. A third problem was that a significant number of this year's academy students were 9th graders, therefore, the only available pre-program data was from 8th grade. Since students attended various junior high schools, there is no assurance that data from 8th grade are either standard across schools or comparable to those in high school. This makes both comparison group matching and year-to-year tracking nearly impossible. EDISON FINANCIAL ACADEMY 2 7 For all these reasons it was decided not to attempt an analysis of outcome data for the Edison Financial Academy. What little data were available for analysis suggested a negative program influence on students. For example, this year's academy students have higher dropout rates, earned fewer credits, have lower grade point averages, and failed more courses than their matched comparison group counterparts. None of these differences are very large, but there is certainly no evidence of positive program impact. Questionnaire Responses Student g zgrestjennaires In addition to many other problems at this site, no incoming student questionnaires were administered in the fall to new students. While the year-end versions were administered to those students still enrolled in the spring, given the inability to analyze other types of data and the fact that the program had already been terminated, these were not analyzed. Thus no student questionnaire responses are available. School Personnel Responses At the end of the school year, the four teachers involved in the academy completed questionnaires, providing their responses to the program. Their responses included the following: o The academy's facilities, equipment, instructional materials, and curriculum all received average ratings of "good." - Nonacademy administrators, teachers, and counselors at the high school all received average ratings of "slightly supportive." - Program planning was rated from "average" to "poor." - Cooperation and support from the private sector was rated as "nonexistent." - On a seven-point scale (l=very weak, 7=very strong), the academy received an average rating of 5.0 "in concept" and 1.75 "as implemented." 28 PENINSULA ACADEMIES 1986-87 EVALUATION ° The program's weaknesses were staff apathy, student disinterest, lack of administrative and counseling support, lack of teacher orientation and direction, and lack of private-sector coordination. ° The program's strengths were the academy's potential effect on at-n'sk students and the promise it showed, No parent, private-sector, or mentor questionnaires were received. Summary The Edison Financial Academy was, by almost any reckoning, a failure. It failed to develop almost all the essential elements of an academy and, as a result, it had no measurable positive effect on students. Indeed, it may have had some negative influence. It offers a case study of what can go wrong when a new site fails to understand and implement a model program developed elsewhere. chapter 4 The Franklin Food Services Academy Benjamin Franklin High School Stockton Unified School District Stockton, California The Franklin Food Services Academy encountered a number of serious problems in 1986—87, mostly relating to administrative support. While it was well implemented in terms of technical training in food services, and enjoyed moderate private-sector support, it failed to develop the school-within-a-school structure and block scheduling that defines an academy administratively. When confronted by the State Department of Education on this point, district administrators made some efforts to comply with the model but never fully succeeded. Thus, like its sister program at Edison High School, this program was terminated at the end of the 1986-87 school year. However, an even more ambitious program, based on the academy model but under different funding, was planned for the fall of 1987. Setting Stockton is located south of Sacramento and north of Modesto in the California central valley. It has a population of approximately 150,000 and an economy based largely on agriculture and, more recently, services. Franklin High School is an older facility located on the east side of the city, in a low- to middle-income neighborhood. It has a student population of approximately 2,200, composed of 37 percent Hispanic, 35 percent white, 13 percent Asian, 12 percent black, 2 percent Filipino, and 1 percent American Indian. Its parent education index places it at the 13th percentile statewide. Thirty-one percent of its students come from families receiving AFDC. Student Profile The Franklin academy developed a precise student selection procedure during its planning period and has followed this consistently. As a consequence, the academy enrolled students appropriate for the program. This procedure includes several steps: 29 3 0 PENINSULA ACADEMIES 1986-87 EVALUATION ° Notifying 9th grade teachers and counselors of the program and its intended target group and seeking nominations of students who meet the target- group profile (record of poor attendance, low achievement, poor motivation, and minimum 6th grade standardized test scores) 0 Circulating informational flyers to students themselves ° Holding informational meetings for interested or nominated students ° Conducting interviews with interested students who qualified 0 Holding informational meetings for parents of interested students in which they must Sign permission or support forms for their son or daughter The profile in Table 4.1 demonstrates how program students compare with their matched comparison groups (grades 10 and 11) in terms of race/ethnicity and gender for the past year. TABLE 4.1 Student Profile, The Franklin Food Services Academy Academv Comparison Group Grade Level 10 23 16 l l 40 19 Race/Ethnicity Asian 2% 6% Black 1 9% 6% Hispanic 34% 50% White 3 8% 33 % Other 8 % 6% Gender Female 55 % 48% Male 45% 52% THE FRANKLIN FOOD SERVICES ACADEMY 3 1 Program Management Responsibility for management of the program at Franklin High School was never fully assumed by any administrator. The principal was supportive of the program but was involved only when particular problems arose. Program administration was delegated to a teacher in charge of high-risk youth. This teacher attended advisory board meetings and maintained a loose connection to the program, but he had many other responsibilities and did not view the academy as his primary responsibility. Thus, the only person with operational responsibility for the program was the lead teacher, who everyone feels did a fine job, but who was limited by the lack of higher-level support. The program never fully implemented the school-within-a—school structure that is part of the academy model. When the state recognized this in the fall of 1986 and confronted the district, attempts were made to comply. However, the main change was to place certain students in competency—based math and reading classes, which contained nonacademy students as well. In some cases, students were placed in classes too basic for their abilities, causing understandable hard feelings. Also, the district never accepted the idea of small classes, because of their expense, undermining this element of the academy model from the start. One well implemented component was the food services technical classes and training. The lead teacher taught these classes, and she related well to the academy students. Further, students gained sufficient skills (cooking, waiting on tables, preparing trays) to be in constant demand to cater luncheons at the school and a variety of outside events. The lead teacher estimated that academy students performed 200 jobs during the school year. Other school-based components were instituted as well. A counselor assigned to the program became involved in helping academy students address both personal and school-related problems. A career center coordinator worked with students, helping to develop career plans, a resume, and interviewing skills. Parental contact continued through the school year. The food center classroom served as a program base, providing academy students a sense of identity. Private-sector involvement was modest. Several business speakers visited the school during the course of the year. Students were taken on field trips to Red Lion Inn in Sacramento and the Delta Food Exposition in Stockton. Eight juniors had mentors, although the frequency of meetings with them was limited by transportation problems. Through the efforts of involved employers, students worked at a variety of events outside school. A lO—member advisory board met monthly and included representatives 32 PENINSULA ACADEMIES 1986—87 EVALUATION from seven area companies. Delta College was also involved, helping to develop curriculum and providing a speaker. Unfortunately, when word of the program’s cancellation reached the staff and students (it was never personally communicated to academy staff by a district representative, nor were they consulted in the decision), much of the program's momentum dissipated. Private-sector representatives felt betrayed. In some cases, this confirmed their reluctance to get involved with the Stockton Public Schools in the first place. Parents also felt betrayed, having signed forms to support their children through three years of the program. Efforts to develop summer jobs evaporated. Staff morale plummeted. Program Resources During the 1986-87 school year, 740 teaching hours were allocated to the Franklin Food Services Academy. This represents the single academy teacher spending 4.11 hours per day in academy classes for 180 days. In addition, that teacher spent 370 hours during the year on program management, curriculum development, and student recruitment. School and district administrators (including the academy director) devoted 325 hours to the program; counselors, 200 hours; secretaries, 50 hours; and an aide, 750 hours. Participants from local businesses donated an estimated 1,100 hours to the academy during the year. Of this total, 400 hours were spent on job placement and supervision of academy students, 300 hours on one-to-one mentoring, 150 hours giving talks, and 250 hours on program management and coordination. The Food Services Academy made use of a kitchen/classroom belonging to the school district, with an estimated rental value of $15,000 per year. New equipment (stoves, ovens, a dishwasher, etc.) was purchased for $75,000 with money from the state grant or donated by local businesses. Materials (uniforms, texts, films, paper) were purchased for $5,000 with state grant or local district funds or donated. One thousand dollars spent on student recognition and social events also came from these three sources. Student Outcome Data The most objective form of data by which to judge the program's effectiveness is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in THE FRANKLIN FOOD SERVICES ACADEMY 33 this report address attitude changes and more subjective responses from program participants. Table 4.2 presents differences in outcomes between academy students and the comparison groups, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natural" units, i.e., the actual difference in attendance percentage, credits earned, grade point average, and courses failed. TABLE 4.2 Adjusted Differences Between Academy and Comparison Groups, by Class, The Franklin Food Services Academy Courses Dropout Attendance Credits GPA Failed Probabilitv Class Entering Fall 1985 1985-86 Outcomes 7.0 14.4* 0.5* -O.7 -0.03 1986-87 Outcomes 10.9 9.4 0.0 0,7 053* Class Entering Fall 1986 1986-87 Outcomes 7.7 14.8*# 1.0* 0.3# I * Statistically significant # Estimated from separate regressions for academy and comparison groups I Insufficient number of dropouts for analysis Table 4.2 portrays mixed results. Both classes of academy students in the first year accumulated significantly more course credits and achieved significantly better grades than comparison groups students. However, students in the first academy class had a significantly higher probability of dropping out of school than did comparison students in 1986—87. Of the nine differences that were not statistically significant, eight indicated better performance by academy students. 34 PENINSULA ACADEMIES 1986-87 EVALUATION Questionnaire Responses The following sections provide a general overview of responses received through individual questionnaires. Those providing feedback included program students, academy personnel, parents of academy students, private-sector participants, and student mentors. Student uestionnaires Table 4.3 presents data from questionnaires administered to academy students. A brief version is administered to all students as they enter the program, asking about their career-related plans and attitudes toward careers, school, and themselves. A longer version is administered at the end of each school year, asking again about these topics. This version also requests information about program activities and asks students to rate the program. Data are presented by class, so that one can see differences between this year's 10th and 11th grades. For the first class (11th grade), responses are presented separately for each year. Numbers reflect either the percentage of students responding as indicated, or the mean number, depending on the form of the item. TABLE 4.3 Student Questionnaire Responses, The Franklin Food Services ‘ Academy (means [decimals] or percentages of respondents, as indicated) Class 1 Class 2 Pre Spring '86 Spring '87 Pre Spring '87 Program Descriptive Information How many academy classes are you currently taking? n/a 1.0 1.6 n/a 2.1 Who have you asked about career or job information this year? a. Academy teacher n/a 80% 80% n/a 33% b. Other high school teacher n/a 0% 20% n/a 7% c. Parents or other relatives n/a 40% 20% n/a 40% (1. Friends n/a 20% 20% n/a 7% e. Academy mentor n/a 0% 0% n/a 0% f. Employer n/a 20% 0% n/a 13% g. School counselor n/a 40% 60% n/a 33% h. School career-center staff n/a 40% 60% n/a 60% THE FRANKLIN FOOD SERVICES ACADEMY 35 Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring '87 Students who had a mentor this year n/a 0% 100% n/a 7% Of these, those who found it useful n/a - 0 n/a - Students who had a job arranged through the program a. Summer job n/a 20% 20% n/a 7% b. School-year job n/a 0% 0% n/a 0% Of these, those who found it useful n/a - - n/a ‘ - How many guests from industry spoke in your classes this year? a. Fall semester n/a 2.6 2.6 n/a 1.8 b. Spring semester n/a 8.6 2.2 n/a .7 Those who found speakers helpful n/a 100% 100% n/a 62% How many field trips did you attend with the academy? a. Fall semester n/a 0 0 n/a 0 be Spring semester n/a 2.4 1.2 n/a .9 Those who found field trips useful n/a 80% 100% n/a 55% Did you get the extra help you needed with your school work during the past year? n/a 100% 100% n/a 53% Who gave you the help you needed? a Teacher in class with the problem n/a 60% 60% n/a 47% b. Another academy teacher n/a 20% 100% n/a 20% c. A peer tutor n/a 0% 0% n/a 7% (1. Another student or friend n/a 20% 20% n/a 53% e. Parent(s) n/a 40% 60% n/a 40% 36 PENINSULA ACADEMIES 1986-87 EVALUATION Class 1 Class 2 continued PIC SDI‘iUg '86 Spring '87 P13 Spring '87 Program Ratings Academy equipment was good or excellent n/a 60% 100% n/a 60% Academy materials were good or excellent n/a 60% 80% n/a 47% Academy schedule better than regular one n/a 100% 80% n/a 67% Felt strong connection between what was studied in school and postgraduate plans n/a n/a 100% n/a 60% Liked academy better than regular program n/a 100% 100% n/a 80% Career-Related Plans Have definite plans for the first two years after graduation 33% 33% 67% 33% 67% If have plan, it includes a, A job 100% 67% 67% 67% 22% b. Junior college, college, or university 67% 67% 67% 67% 56% 0. Vocational or technical training 33% 33% 33% 0% 22% d. Military service 33% 0% 0% 0% 11% e. Marriage and housekeeping 0% 0% 0% 11% 0% Have a long-term career goal 67% 67% 67% 44% 56% Attitudes Related to Career, School, and Self Feel it is very important to make career plans while in school 67% 67% 100% 100% 78% Feel it is very important to gain job skills while in school 100% 67% 100% 89% 100% THE FRANKLIN FOOD SERVICES ACADEMY 37 Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring '87 Found classwork interesting during past year 33% 100% 67% 33% 11% Feel good about self after past year 67% 67% 100% 78% 89% Would not want to be someone different 67% 100% 100% 67% 56% Feel success is not mainly due to luck 100% 100% 100% 44% 67% .— Missing data n/a= Not applicable School Personnel Responses At the end of the school year, five members of the school staff completed questionnaires providing their responses to the program. These included the principal, career center counselor, program counselor, and two academy teachers. Their responses included the following: The academy's facilities, equipment, instructional materials, and technical curriculum were all rated from " good" to "excellent." Nonacademy teachers, counselors, and administrators at the high school were rated from "slightly" to "very" supportive. Program planning received an average rating of "good." Cooperation and support from the private sector were rated from ”somewhat" to "very" effective. On a seven-point scale (1=very weak, 7=very strong), the academy received an average rating of 6.6 ”in concept" and 4.2 "as implemented." The program's greatest weaknesses were its lack of district support and lack of core academic classes as called for by the academy model. The program's greatest strengths were the support the program gave its at-risk students and the quality of the lead teacher. 38 PENINSULA ACADEMIES 1986-87 EVALUATION Wm Three academy parents completed questionnaires at the end of the school year. Their responses included the following: 0 Two parents rated the program as "excellent," the third as "average." - All three reported improvements in their child's attitude toward school and learning since joining the academy. ° The only way parents were involved in the program in 1986—87 was by helping their children with homework. - In response to how well they had been kept informed of their child's progress in school, two parents said "very well," the third "poorly." 0 No suggestions were offered for program improvements. No private-sector mentor questionnaires were received, Summary The academy's main problem of 1985—86 continued in 1986-87, namely, the absence of a full school-within-a—school set of academic classes. Due to this problem and the district's inability to meet the academy model, district administrators decided to end the program in June 1987. Thus, evaluation findings became largely irrelevant, at least to this program. In spite of this, there are indications of positive program impact. Eleven of 13 differences between program and comparison group students favor academy students, and four of these are statistically significant (credits and grades, both classes, their first program year). Responses from both student and adult questionnaires were also predominantly positive. chapter 5 The Corporate Academy Hiram Johnson High School Sacramento City Unified School District Sacramento, California The Corporate Academy at Hiram Johnson High School is a large, ambitious program with a full complement of classes. It is set in a large, inner-city high school with a diversity of students. It operated with four full-time and five part-time teachers in 1986-87 and gained substantial acceptance at the high school. It is well equipped with AT&T computers, having spent much of its first-year grant on this equipment. Private-sector support developed moderately, with a partial mentor program, a few speakers and field trips, and summer jobs for approximately 15 juniors. While the program has strong support and leadership at the high school, its greatest weakness is lack of district support, where it tends to get lost in the bureaucracy and is viewed mainly in budget terms. Setting Sacramento, the state capitol, is the largest city in the California central valley, with a population of approximately 300,000. It is a fast- growing city with a healthy, diverse economy, including agriculture, state government, and services. Hiram Johnson is one of 10 high schools in the district (the city includes other districts). It is a relatively modern facility located on the city's south side in an inner-city neighborhood. Its student population of approximately 3,000 is composed of 46 percent white, 19 percent black, 17 percent Asian, 14 percent Hispanic, 3 percent Filipino, and 1 percent American Indian. Its parent education index places it at the 5 lst percentile statewide. Forty-one percent of its students come from families receiving AFDC. Student Profile The program used a well defined selection system the first year. Ninth grade students were told about the program, and those who expressed interest applied. Participants were then selected using several criteria: 39 40 PENINSULA ACADEMIES 1986-87 EVALUATION ° Standardized test scores that are low but not more than three years behind grade level (5th grade level when tested in 8th grade) 0 Poor attendance, specifically more than 20 percent absences ' Credit deficiencies of 10 credits or more (of 60 expected in 9th grade) Parents were asked to attend an informational meeting and to sign a permission and support form. In 1986-87, the same general plan was used but somewhat less consistently. Many students learned of the program through word of mouth, rather than through structured presentations. While students still applied for the program and were filtered through the above criteria, the program's manager felt that students were screened less thoroughly. Plans were in place to correct these problems for the third year. Teachers stated that they can best serve students who genuinely want to be in the program, and since most students in the school are "at-risk," teachers asserted that interest is the best criteria on which to distinguish appropriate candidates. Table 5.1 presents a profile of academy students this year by grade level, race/ethnicity, and gender. Figures for the comparison groups are also presented. TABLE 5.1 Student Profile, The Corporate Academy Academv Comparison Group Grade Level 10 53 1 l9 1 l 70 86 Race/Ethnicity Asian 1 1% 8 % Black 30% 26% Hispanic 21% 23% White 37 % 36% Other 1% 8 % Gender Female 5 l % 43 ‘70 Male 49% 57% THE CORPORATE ACADEMY 4 1 Program Management The program director is also the school's outreach consultant. He taught the 10th grade academy computer classes last year, but this year his time was spent on school administrative duties, one of which was managing the academy. He was unable to secure the extra preparation period provided for teachers the first year. This was missed, but teachers maintained looser ties on their own, meeting over lunch or whenever they could. All program teachers were volunteers. The staff this year included: - A full-time English teacher, who taught both 10th and llth grade 0 A full-time history teacher, who taught two 10th grade world civics classes and three 11th grade U.S. history classes . A full-time math teacher, who taught two pre-algebra classes, two algebra classes, and one business math class - A four-fifths time computer teacher, who taught both sophomores and juniors (sophomores are taught computer literacy and basic programming, juniors are taught business applications, seniors conduct individual projects) 0 Several other classes to supplement the above, including two science, one language arts, and two computer While all classes were block scheduled during the morning the first year, this was not possible in 1986-87, and classes were spread through the day, making it harder to arrange field trips and other programwide activities. There was some coordination between academic and vocational curriculum. Partly this involved using computers for assignments in other subjects. Teachers related academic subjects to employment realities, and English students prepared resumes, practiced interviewing, and so on. Parental contact was maintained—~teachers called home when students were absent, failed to complete homework, or had behavior problems. Most counseling was handled by the director, who had an academy office which made this feasible. He dealt with a variety of student needs, encompassing academic, scheduling, school, career, and personal topics. Private-sector support was developed this year, primarily by the director, with help from other teachers on major activities. It was difficult to fit this time-consuming responsibility into a school schedule, given the schedules most employees have. However, support was demonstrated on several counts: 42 PENINSULA ACADEMIES 1986-87 EVALUATION 0 While there is an advisory board, it met infrequently in 1986-87. It played a more important role last year, helping to develop curriculum. ° There were three speakers (fewer than last year). This activity was complicated by the lack of morning block scheduling and a common preparation period for teachers. 0 Only two field trips occurred, for the same reason. 0 Approximately 25 juniors had mentors. This was voluntary, and all students who asked for a mentor got one. Interest in this aspect of the program increased as students became familiar with the concept and spread its reputation by word of mouth. 0 A few summer jobs were provided for juniors, although only about 15 students met all the program's requirements for this, and most who worked got jobs through their mentors or on their own. Much of the private-sector support was obtained through the Chamber of Commerce Education Committee. While the program seems well regarded by corporate sponsors, the need for more time to coordinate this component is strongly felt by the director. Probably the most serious problem facing this academy is the school district's waning support. Evidence of this was manifested in several respects: - Average academy class size was allowed to increase from 20 to 25 this year, due to the district's concern with the expense of smaller classes (average class size in the district is about 30). ° The teacher's extra preparation and coordination period was dropped, even though it is part of the academy model. 0 The district would not provide substitutes to allow academy teachers to have staff development time, particularly to explore the use of computers in other subjects. The Corporate Academy was also the last site from which a district superintendent sent a (requested) letter to the State Department of Education expressing support for the academy and its basic components and promising to use the grant funds exclusively for the program. This letter was not provided until the end of the summer, causing uncertainty about the program's continuation and making it difficult to plan for the next year. THE CORPORATE ACADEMY 43 Program Resources Five teachers spent a total of 4,500 hours instructing academy classes during the 1986-87 school year. This represents an average of five hours per day per teacher over a ISO-day year. In addition, teachers reportedly spent 900 hours on cun‘iculum development and student recruitment. School administrators (including the academy director) contributed an estimated 684 hours to the program, counseling (including testing at the career center) occupied 186 hours, and secretaries devoted 36 hours to the academy during the year. Participants from local businesses were estimated to have contributed a total of 196 hours to the academy program during the year. Of that total, 102 hours were spent in one- to-one mentoring with students, 64 hours were devoted to program management and coordination, and 30 hours were spent giving talks. The school district expended $3,000 modifying the school's computer facility (providing additional outlets and air conditioning) and $1,000 remodeling an office for academy use. Local businesses donated 10 complete Hewlett Packard computer systems at $3,500 each. The state grant was used to purchase 10 printers at $350 each and 20 AT&T 6300 computers at $1,450 each. Student Outcome Data The most objective form of data by which to judge the program's effectiveness is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in this report address attitude changes and more subjective responses from program participants. Table 5.2 presents differences in outcomes between academy students and the comparison groups, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natural" units, i.e., the actual difference in attendance percentage, credits earned, grade point average, and courses failed. 44 PENINSULA ACADEMIES 1986-87 EVALUATION TABLE 5.2 Adjusted Differences Between Academy and Comparison Groups, by Class, The Corporate Academy Courses Dropout Attendance Credits GPA Failed Probabilitv Class Entering Fall 1985 1985-86 Outcomes NA 7.6 * 0.3 -1,1* I 1986-87 Outcomes 1.3 ~3.0 -0.3 0.5 008 Class Entering Fall 1986 1986-87 Outcomes 7.1 * 5.3 0.4 * -1.1 I * Statistically significant # Estimated from separate regressions for academy and comparison groups I Insufficient number of dropouts for analvsis Four of the differences in Table 5.2 are statistically significant, and all four reflect better performance by academy students than by the comparison group. The first class of academy students accumulated significantly more course credits and failed a smaller number of courses than its comparison group in 1985-86. The second class of academy students had significantly better attendance and grades than its comparison group in 1986- 87. Of the eight remaining differences shown in the table, four indicate better performance by academy students. Questionnaire Responses The following sections provide a general overview of responses received through individual questionnaires. Those providing feedback include program students, academy personnel, parents of academy students, private-sector participants, and student mentors. Student uestionnaires Table 5.3 presents data from questionnaires administered to academy students. A brief version is administered to all students as they entered the program, asking about their career-related plans and attitudes toward careers, school, and themselves. A longer version is administered at the end of each school year, asking about these topics again. This THE CORPORATE ACADEMY 4 5 version also requests information about program activities and asks students to rate the program. Data are presented by class, so that one can see differences between this year's 10th and 11th grade classes. For the first class (11th grade), responses are presented separately for each year. The numbers reflect either the percentage of students responding as indicated, or the mean number, depending on the form of the item. TABLE 5.3 Student Questionnaire Responses, The Corporate Academy (means [decimals] or percentages of respondents, as indicated) Classl ClassZ Pre Spring '86 Spring'87 Pre Spring '87 Program Descriptive Information How many academy classes are you currently taking? n/a 4.0 3.7 n/a 3.7 Who have you asked about career or job information this year? a. Academy teacher n/a 67% 42% n/a 47% b. Other high school teacher n/a 4% 0% n/a 16% 0. Parents or other relatives n/a 33% 29% n/a 47% (1. Friends n/a 25% 33% n/a 34% e. Academy mentor . n/a 0% 13% n/a 8% f. Employer n/a 0% 4% n/a 5% g. School counselor n/a 8% 4% n/a 13% h. School career-center staff n/a 21% 25% n/a 47% Students who had a mentor this year n/a 5% 27% n/a 21% Of these, those who found it useful n/a - ° n/a 75% Students who had a job arranged through the program a. Summer job n/a 0% 0% n/a 8% b. School-year job n/a 0% 4% n/a 3% Of these, those who found it useful n/a - 0 n/a 50% 46 PENINSULA ACADEMIES 1986-87 EVALUATION Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring '87 How many guests from industry spoke in your classes this year? a. Fall semester n/a 1.9 93 n/a .4 b. Spring semester n/a 2.2 .8 n/a .4 Those who found speakers helpful n/a 71% 57% n/a 36% How many field trips did you attend with the academy? a. Fall semester n/a 1.9 4 n/a 0 b. Spring semester n/a 2.7 6 n/a 0 Those who found field trips useful n/a 81% 75% n/a n/a Did you get the extra help you needed with your school work during the past year? n/a 57% 48% n/a 45% Who gave you the help you needed? a. Teacher in class with the problem n/a 57% 44% n/a 66% b. Another academy teacher n/a 30% 35% n/a 29% Ce A peer tutor n/a 4% 0% n/a 3% d. Another student or friend n/a 9% 30% n/a 26% e. Parent(s) n/a 22% 26% n/a 32% Program Ratings Academy equipment was good or excellent n/a 96% 83% n/a 59% Academy materials were good or excellent n/a 71% 72% n/a 62% Academy schedule better than regular one n/a 57% 17% n/a 31% Felt strong connection between what were studying in school and postgraduate plans n/a n/a 54% n/a 65% THE CORPORATE ACADEMY 47 Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring '87 Liked academy better than regular program n/a 96% 70% n/a 65% Career-Related Plans Have definite plans for the first two years after graduation 21% 21% 33% 27% 24% If have plan, it includes a. A job 32% 42% 21 % 46% 42% b. Junior college, college, or university 68% 68% 68% 54% 58% 0. Vocational or technical training 11% 5% 16% 8% 17% (1., Military service 16% 26% 42% 4% 13% e. Marriage and housekeeping 0% 0% 5% 9% 0% Have a long-term career goal 57% 62% 42% 48% 56% Attitudes Related to Career, School, and Self Feel it is very important to make career plans while in school 63% 83% 71% 57% 57% Feel it is very important to gain job skills while in school 74% 70% 74% 70% 67% Found classwork interesting during past year 33% 58% 54% 20% 40% Feel good about self after past year 86% 90% 67% 55% 72% Would not want to be someone different 65% 80% 80% 55% 66% Feel success is not mainly due to luck 86% 100% 77% 59% 66% -= Missing data Ma: Not applicable 48 PENINSULA ACADEMIES 1986-87 EVALUATION Parent Responses Six parents of students in this academy completed questionnaires at the end of the school year. Their responses included the following: Three parents rated the program as "excellent," two as "good," and one said "don't know." Five reported that their children were "much more interested in school and learning" since joining the academy; the sixth reported "a little more," Parent involvement with the program included. three parents who helped their son or daughter with homework, three who met with an academy teacher at school, and one who attended an open house. In response to how well they had been kept informed of their child's progress in school, four parents responded "very well" and two "moderately well." No suggestions for program improvements were offered. Mentor Responses Two academy mentors completed questionnaires at the end of the school year. Their responses included the following: On average, mentors met twice with their students and talked with them once by telephone. Asked if the program was providing students with job skills needed by area companies, one mentor said "yes" and one "don't know." Asked to rate the mentor program, one responded "highly useful" and one ".OK." The program's greatest weaknesses were lack of assured future funding and lack of information from the school about the students. The program's greatest strengths were committment of program staff and students and support from the business community. No school personnel or private—sector questionnaires were received. THE CORPORATE ACADEMY 49 Summary There are two main problems associated with The Corporate Academy at Hiram Johnson High School: insufficient district commitment and little private-sector involvement. Implementation of the program itself is generally good. If the necessary district support is forthcoming and means can be found to develop needed Speakers, field trip sites, mentors, and jobs, the program will be well implemented. Analysis of data provides some evidence of positive program effect. Of 12 differences between program and comparison groups, nine favor academy students, and four of these are statistically significant. Questionnaire responses from students and adults is also largely positive. While there is clearly room for improvement, if the remaining needs can be met, this program has a good potential for success. chapter 6 Computech Academy Mountain View High School Mountain View-Los Altos Union High School District Mountain View, California Computech Academy faces its third year of operation in 1987-88, challenged by a number of problems, including declining enrollment, changes in teaching personnel, limited private-sector involvement, and the perceived negative image of the program within its host high school. Mountain View High School, where the program is housed in a portable classroom far enough away from the center of the school so that bells ring between periods cannot be heard, is among the most affluent academy replication site schools. Mountain View High is a largely white, middle class, suburban, and relatively small (1,300) high school. While academy students also can be drawn from Los Altos High School and Shoreline Continuation School, most Computech Academy students come from Mountain View High. The number of students in the district who meet the general profile of an academy student is relatively small, and this, in part, explains low enrollments in the academy program. In 1986-87, there were sophomore and junior academy classes of 20 and 28 students, respectively, though by year's end the number of juniors had dropped to 14. By December of the 1987-88 school year, there were 19 sophomores, 21 juniors, and 11 seniors in Computech Academy. These low figures do not reflect a lack of recruiting effort by the school and district. Ninth grade students who fit the academy profile of poor attendance and underachievement are identified at all three schools—Mt. View, Los Altos, and Shoreline. They are contacted individually and through group presentations. But increasingly smaller numbers are choosing to enroll in the academy. Students are block programmed in the morning, sharing a two classroom portable which was formerly used for business classes and is one of the few air conditioned classrooms on campus. Sophomores take academy English, math, science, and computer literacy. Juniors take academy English, math, and electronics. The plan for the 1987-88 seniors is to have four blocked periods in the morning: English, computer repair, small business preparation, and practical economics. Other required and elective classes are 50 COMPUTECH ACADEMY 5 1 taken in the afternoon outside of the academy. Current plans call for all seniors to work part time during their senior year. Setting The combined population of Mountain View and Los Altos is approximately 80,000. Both communities are located in Santa Clara County with its Silicon Valley economy. Unemployment countywide is under five percent. Although the electronics industry fluctuates, the overall trend is toward continuing growth. And growth in electronics has spawned a parallel growth in service industries and jobs. Mountain View High is a sprawling, one-story suburban campus with an educational program stressing college admission. School officials estimate that 80 percent of the school's graduates continue on to college. Black and Hispanic students compose 13 percent of the school's total population. According to the State Department of Education, parents from the school rank in the top seven percent in education of all parents statewide. Fewer than two percent of the families in the school receive AFDC. Student Profile In the first year of program operation, there was no matched comparison group. Two such groups were identified in 1986-87. Table 6.1 compares program students with their matched comparison groups (10th and 11th grades combined) in terms of race/ethnicity and gender for the past year. 52 PENINSULA ACADEMIES 1986-87 EVALUATION Table 6.1 Student Profile, Computech Academy Academv Comparison Group Grade Level 10 20 62 1 1 28 55 Race/Ethnicity Asian 2% 0% Black 15% 8% Hispanic 9% ' 13% White 72% 74% Other 2% 5% Gender Female ' 25% 43% Male 75% 57% Program Design and Management The director of pupil personnel is ultimately in charge of Computech Academy, though day-to-day operations are administered by a lead teacher who has one period per day of released time. The personnel director acts as liaison with the State Department of Education and independent evaluators and also prepares the program's budget. But, as his title implies, he has many other district responsibilities. Within the high school, limited attention has been paid to the academy by administrative staff, except for the dean, who was assigned to help oversee the program and who volunteered to teach in the academy part time. The relative lack of attention by school administration has, according to several Computech teachers, reinforced the program's isolation from the rest of the school. Virtually all program staff expressed a concern about inadequate electronics equipment provided for technical classes. For the most part, the academy model has been closely followed at Computech. Students are block programmed, and most teachers in the program share a common planning period. But only the lead teacher has an extra planning period. Since only the COMPUTECH ACADEMY 5 3 lead teacher and the electronics teacher spend more than 50 percent of their schedule with academy students, it is not easy to use the common planning period effectively. Classes are small. Though, as already noted, classes are smaller than planned because of enrollment and attrition problems. Except for the lead and electronics teachers, none of the 1986-87 academy teachers returned in 1987-88. There are several reasons for this. For example, one teacher was a Stanford intern whose assignment at the school ended. But some of the teachers asked not to teach Computech students. They cited as reasons a lack of administrative support, inadequate equipment for the electronics class, and "burnout" from working with difficult students. For the 1987-88 year, the district recruited a new math/science teacher with both experience and interest in working with at—risk students. Private-sector involvement in the first two years was organized in part through assistance from the Stanford Mid Peninsula Urban Coalition. Both mentors and summer jobs were arranged by Urban Coalition staff, with each junior in 1986-87 being matched with a mentor. Of the 13 eligible juniors (those whose attendance and grades were up to the standards set by the program), all had summer jobs in 1987. About half the placements came from the Urban Coalition and the other half were secured by students themselves. In 1987-88, private-sector relations are managed by the district itself, without assistance from the Urban Coalition. The program has a business advisory committee, but participation has been limited to a handful of electronics and computer company representatives. In the spring, the advisory group focused on the planned computer repair class for seniors and ways in which local electronics and computer firms could assist. Computer and electronics equipment has been donated by several firms. Several speakers and field trips were arranged during the year, and there was an overnight camping trip for many Computech students. But some program staff questioned the value of the field trips because of poor student attitude. In general, staff perceive Computech students as the most difficult in the school and feel that not enough resources are provided for them. Some of the mentors agree with this perception. Staff felt that the strongest component of the program was the informal counseling provided, working with students facing pregnancy, drug problems, and abuse in their home lives. Program Resources Six teachers reported spending 1,430 hours instructing academy classes during 1986-87, an average of 1.3 hours per day for each teacher during a l80-day year. In addition, teachers reported spending 432 hours on program management and coordination. 54 PENINSULA ACADEMIES 1986-87 EVALUATION District and school administrators (including the academy director) contributed an estimated 560 hours to the program. Counselors reported spending 130 hours With academy . students; aides devoted 170 hours to the program; custodians, 270 hours; and secretaries, 97 hours. Participants from local businesses donated 877 hours to the academy. Of this total, 525 hours were used for one-to-one mentoring activities with students, 167 hours for program management and coordination, 175 hours for job placement and supervision of academy students, and 10 hours for presentations. Computech Academy used 14 IBM computers and printers belonging to the school district, with a total estimated rental value of $5,600 for the 10-month school year. The school district also spent $2,148 for electronic kits, $1,310 for textbooks, $1,567 for instructional supplies, and smaller amounts for travel, publications, and student recognition. Businesses donated $3,000 in electronic equipment and test devices. Student Outcome Data The most objective form of data by which to judge the program's effectiveness is measurable indicators of performance in school. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in this report address attitude changes and more subjective responses from program participants. Table 6.2 below demonstrates differences in outcomes between academy students and the comparison group, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natur " units, i.e., the actual difference in attendance percentage, credits earned, grade point average, and courses failed. COMPUTECH ACADEMY 5 5 Table 6.2 Adjusted Differences Between Academy and Comparison Groups, by Class, Computech Academy Courses Dropout Attendance Credits GPA Failed Probability Class Entering Fall 1985 1985—86 Outcomes 4.1 0.6* =O.4 I 1986-87 Outcomes 0.6 -O.8 0.1 —O.3 ~0.01 Class Entering Fall 1986 1986-87 Outcomes -O.4 -2.8# 0.3 0.7 -0.89 #:Estimated from separate repressions for academy and comparison groups. I=Insufficient number of dropouts for analysis *=Statisticallv significant Only one difference in this table is statistically significant: the first class of academy students achieved significantly better grades than their comparison group in 1985-86. Of the remaining 12 differences shown in the table, eight indicate superior performance by academy students. Questionnaire Responses The following sections provide an overview of the responses received through individual questionnaires. Those providing feedback include program students, academy school personnel, parents of academy students, private-sector participants, and student mentors. Student Questionnaire Responses Table 6.3 presents data from questionnaires administered to academy students. A brief version was administered to all students at the point they entered the program, asking about their career—related plans and attitudes toward careers, school, and themselves. A longer version was administered at the end of each school year, asking about these topics S 6 PENINSULA ACADEMIES 1986-87 EVALUATION again. This version also requested information about program activities and asked students to rate the program. Responses are presented separately for each year. Numbers reflect either the percentage of students responding as indicated, or the mean number, depending on the form of the item. Unfortunately, two pages from the year-end questionnaire for 1985-86 were not administered, and, as a result, there is missing data for many items. Table 6.3 Student Questionnaire Responses, Computech Academy (means [decimals] or percentages of respondents, as indicated) Classl Class2 Pre Spring '86 Sprin2'87 Pre Spring '87 Program Descriptive Information How many academy classes are you currently taking? n/a 3.9 3.9 n/a 4.0 Who have you asked about career or job information this year? a. Academy teacher n/a 69% 77% n/a 38% b. Other high school teacher n/a. 0% 0% n/a 0% 0. Parents or other relatives n/a 15% 46% n/a 38% do Friends n/a 15% 31% n/a 25% e. Academy mentor n/a 0% 9% n/a 0% f. Employer n/a 8% 15% n/a 25% g. School counselor n/a 8% 54% n/a 13% h. School career-center staff n/a 0% 31% n/a 25% Students who had a mentor this year n/a 0% 100% n/a 0% Of these, those who found it useful n/a - 0 n/a ° Students who had a job arranged through the program a. Summer job n/a 0% 0% n/a 0% b. School-year job n/a 85% 0% n/a 0% Of these, those who found it useful n/a - 0 n/a ° COMPUTECH ACADEMY 5 7 Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring ’87 How many guests from industry spoke in your classes this year? a. Fall semester n/a 5.6 66 n/a 4.6 b. Spring semester n/a 5.1 4.6 n/a 4.4 Those who found speakers helpful n/a 90% 100% n/a 81% How many field trips did you attend with the academy? a. Fall semester n/a 3.8 3.0 n/a 4.5 b. Spring semester n/a 3.6 3.5 n/a 35 Those who found field trips useful n/a 85% 77% n/a 75% Did you get the extra help you needed with your school work during the past year? n/a 0 0 n/a 75% Who gave you the help you needed? a. Teacher in class with the problem n/a - ~ n/a 88% b. Another academy teacher n/a ° ° n/a 0% c A peer tutor n/a 0 0 n/a 0% d. Another student or friend n/a ' - n/a 25% e. Parent(s) n/a - 0 n/a 0% Program Ratings Academy equipment was good or excellent n/a 0 - n/a 75% Academy materials were good or excellent n/a - 0 n/a 38% Academy schedule better than regular one n/a 0 - n/a 38% Felt strong connection between what were studying in school and postgraduate plans n/a n/a 75% n/a 63% 58 PENINSULA ACADEMIES 1986-87 EVALUATION Class 1 Class 2 continued Pre Spring '86 Spring '87 Pre Spring '87 Liked academy better than regular program n/a ° ° n/a 75% Career-Related Plans Have definite plans for the first two years after graduation: 22% 46% 31% 0% 25% If have plan, it includes a. A job 8% 53% 46% 75% 50% b. Junior college, college, or university 54% 69% 69% 38% 50% 0, Vocational or technical training 23% 31% 23% 25% 50% (1. Military service 15% 39% 15% 13% 0% e. Marriage and housekeeping 8% 8% 8% 13% 13% Have a long-term career goal 46% 62% 62% 38% 50% Attitudes Related to Career, School, and Self Feel it is very important to make career plans while in school - ~ 0 25% 38% Feel it is very important to gain job skills while in school - ° - 50% 88% Found classwork interesting during past year - o o 13% 50% Feel good about self after past year c o v 57% 86% Would not want to be someone different 0 - - 63% 75% Feel success is not mainly due to luck . o . 75 % 75% -= Missing data nla= Not aDDlicable COMPUTECH ACADEMY 5 9 fighggl Personnel Responses At the end of the school year, five members of the school staff completed questionnaires. These included a school administrator, the program coordinator, a teacher specializing in computer technology, and two classroom teachers. Their responses included the following: While the facilities were "average," the program would benefit from more space for the much needed increase in equipment. The curriculum was rated as " good," though it could be more challenging, and an increase in the amount of books and learning materials available is necessary. Nonacademy teachers at the high school were rated as only "slightly supportive," and counselors and administrators were seen as just a little more supportive. Program planning was rated "average" with the comment that it would be helpful for the teachers to have more time to plan together. Private-sector cooperation and support was rated "somewhat effective," again stressing the need for more time to be spent on planning. It was mentioned that mentor cooperation was excellent. On a seven-point scale (1=very weak, 7=very strong), the academy received a very strong 6.5 rating for the program "in concept" and a 4.5 "as implemented." The program's greatest weaknesses were lack of time, lack of reward and support for teachers in the program, and attitudinal problems on the part of students. The program's greatest strengths, despite problems, were that it provided a more positive learning environment and a greater potential for individual instruction and educational success than the students had previously experienced. Parent Responses Fourteen academy parents completed and returned questionnaires at the end of the school year. Their responses included the following: Eight of the 14 parents rated the program "excellent," and the remaining five responded that it was "good." 60 PENINSULA ACADEMIES 1986-87 EVALUATION ° Thirteen parents reported that their children's interest in school markedly improved as a result of the program. 0 Parents were involved in the program in several ways. The majority of them attended the open house and assisted with extracurricular activities. They also took a more active part in helping their children with homework and kept in closer contact with teachers. ° Ten of 14 parents felt they were "very well informed" regarding their children's progress in school, and two felt they were "moderately well informed." 0 Suggestions for improvement were to expand the program so that it covered more educational requirements (e. g., history), hire more teachers, form a parent support group, and provide parents the opportunity to have more input on program planning and activities. Private-Sector Responses Four private-sector representatives completed questionnaires at the end of the school year. Their responses included the following: - Asked whether the academy's curriculum would prepare students for jobs in the community, three said "yes" and one "don't know." 0' When asked to rate communications with the program, one felt it was "excellent," two said "good," and one said "average." ° On a seven-point scale (1=very weak, 7=very strong), the academy received a very strong 6.0 rating for the program "in concept" and a 4.5 "as implemented." ° The program's greatest weaknesses were not enough coordination with the private sector, a lack of skilled teachers, and insufficient equipment for the students. 0 The program's greatest strengths were dedication on the part of teachers and the program's effective method of teaching—keeping the children interested and enrolled in school. COMPUTECH ACADEMY 6 l Mentor Responses Six academy mentors completed questionnaires at the end of the school year. Their responses included the following: - Mentors met with students 10 to 20 times and spoke at least 20 times by telephone. ° Asked whether the academy's curriculum would prepare students for jobs in the community, all six said "yes." 0 Five of the six rated the mentor program as "highly useful," and one rated it as "somewhat useful." 0 The program's greatest weakness was lack of help for students finding summer jobs. 0 The program's greatest strengths were increased involvement of nonteachers— private business representatives, mentors, parents—in education. They also felt the program demonstrates to students that skills learned in school are important and do relate to their goals, thereby increasing a student's desire to participate and stay in school. Summary Only one of 13 outcome data measures reported here is statistically significant The first class of academy students achieved significantly better grades than their comparison group in 1985-86. On eight of the other 12 measures, academy students outperformed the comparison group, but the differences were not statistically significant. Thus, based strictly on outcome measures, the program has not made a major difference in the performance of students. Both parents and staff, however, believe strongly that Computech Academy is an important alternative for the small number of district youth who do not fit into the mainstream of the school. Parent satisfaction with the program is high. Still, the program faces significant obstacles to improving student performance. There have been some changes in teachers for the program in 1987-88. Several staff members in interviews stated that district—level support needs to be increased greatly. Enrollment is low, and private-sector participation is weak. chapter 7 The Health Academy Oakland Technical High School Oakland Unified School District Oakland, California The Oakland Health Academy enjoys strong school-level leadership and, although it operates in a difficult setting, is a well run program. Due to reduced enrollments and faculty cutbacks at the school, the academy began the year with the loss of one of its first- year teachers. However, it also moved into a newly renovated and expansive classroom, giving it a home base, and acquired several new teachers. As one of four academics in the Oakland school district, it also has a strong base of district support. The program follows the academy model fully—-—students take four classes each day as part of the program in both 10th and 11th grades. Private-sector support developed during the year, as students went on a number of field trips and, in many cases, were provided summer jobs. Toward the end of the school year, progress was made in developing the mentor program. And the program acquired a major three—year grant for curriculum development, giving it a strong boost for 1987-88. Setting With a population of approximately 350,000, Oakland is the third largest city in the San Francisco Bay Area. Located on the east side of San Francisco Bay, Oakland includes a relatively high minority population and weak economy, particularly compared to wealthy areas that surround it. Oakland Technical High School is one of six comprehensive high schools in the Oakland Unified School District. It is housed in a recently refurbished and attractive older two-story building in an integrated section of North Oakland not far from the border with Berkeley. Its student population of approximately 1,800 consists of 78 percent black, 8 percent Pacific Islander, 5 percent Asian, 5 percent white, 2 percent Hispanic, and 2 percent other. Its parent education index places it at the 27th percentile statewide. Thirty-six percent of its students come from families receiving AFDC. 62 THE HEALTH ACADEMY 6 3 Student Profile There are few students at Oakland Technical High School who are not "at risk" in some sense. Thus, the academy selection process involves not so much finding students who are unusually at risk, but rather average students who will benefit from the program because of its motivational features and focus on health careers. Much of the selection proceeds from recommendations of teachers and counselors familiar with 9th graders. The school uses a tracking system, and, with rare exception, those in Gifted and Talented Education (GATE) are eliminated from consideration. Thus, participants come from the "survey" track, which is composed of the school's average students (one or two years below grade level in reading), and the "skills" track (three or more years below grade level). Among these, students are sought who appear to be performing below their ability, as reflected in low credits or poor attendance. Interest in the program and health careers is also an important consideration. Table 7.1 presents a profile of the 1986-87 program students by grade level, race/ethnicity, and gender. Comparison group figures are also presented. Table 7.1 Student Profile, The Health Academy Program Comparison Group Grade Level 10 32 42 1 1 49 41 Race/Ethnicity Asian 12% 1% Black 73% 94% Hispanic 0 1 % White 8% 4% Other 7% 0% Gender Female 67% 60% Male 33% 40% 64 PENINSULA ACADEMIES 1986-87 EVALUATION Program Management The lead teacher at the high school serves as the program director, as well as its sophomore and junior English teacher. The academy is a separate department at the high school. The program receives good support from the principal and school counselors. There are five other teachers involved in the program, all part-time—a sophomore biology lab (three days/week) and junior physiology lab (five days/week) teacher, sophomore computer lab teacher (two days/week), a sophomore biology teacher, a junior physiology teacher, and a math teacher who teaches several courses (pre-algebra, algebra, geometry), depending on the academy students' abilities. Three program teachers had a common planning period. Last-minute scheduling changes due to staff cutbacks eliminated this for other teachers. Nevertheless, teachers tended to meet regularly, often over lunch or "catch as catch can." All academy teachers volunteered to participate in the program. Private-sector support developed satisfactorily in 1986-87. A program advisory committee met three times. With three other academies in the district, there were also meetings among the four academy directors. There was also talk of establishing an academywide steering committee. There were several speakers and field trips, including one to a medical school where students saw their first cadaver, a highlight of the year. About 20 juniors had summer jobs arranged for them at Children's Hospital. Both sophomores and juniors arranged other jobs as well, although most of these were not related to the health field. Efforts to develop the mentor program during the year progressed slowly. A series of three breakfasts in June, however, elicited a strong response, and the mentor aspect of this program appeared to be crystallizing over the summer and for next year. Nursing and medical students will be used as mentors, as well as local hospital employees. At the end of the spring semester, the program won a highly competitive "California Academic Partnership" grant, which is administered by the California State University system. It provides forties with a school of higher education, in this case the Samuel Merritt College of Nursing, and will give credit to nurses of that school for teaching in the program. Ties were established as well to a Kaiser Hospital labor union, whose members planned to adopt the program and provide support. THE HEALTH ACADEMY 65 Program Resources In addition to a full-time lead teacher, the Oakland Health Academy is budgeted for another 1.8 full-time-equivalent teachers. District and school administrators spent an estimated 785 hours on the Health Academy; this includes one-fourth of the time of a full- time academy coordinator, who also serves three other academy programs in the district (which are not part of the state-funded Peninsula Academies replication). Counselors contributed approximately 120 hours; secretaries, 110 hours; and teachers who were not officially part of the academy staff spent 35 hours on the Health Academy program. The Oakland Health Academy provided 18 pages of documentation describing resources used by the program. More than one hundred specific contributions of time, money, or materials were listed. These came from nine hospitals and 25 other health- or education-related organizations. Estimates of the amount of time or the value of material contributions were given in most instances. It is apparent that the amount of time contributed to the Health Academy by individuals from these organizations exceeded 1,000 hours during the year. Roughly half was spent giving talks or hosting field trips for academy students. Approximately one- third of the total time was spent on curriculum development or program planning and coordination. The remainder, approximately 150 hours, was used for one-to-one mentoring and job "shadowing." The school district used approximately $32,000 of its own money to remodel a classroom for use by the academy, with laboratory facilities included. Local hospitals and businesses donated $11,300 in equipment and materials in 1986-87. Student Outcome Data The most objective form of data by which to judge the program's effectiveness is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in this report address attitude changes and more subjective responses from program participants. Table 7.2 presents differences in outcomes between academy students and the comparison group, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. 66 PENINSULA ACADEMIES 1986-87 EVALUATION In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natur " units, i.e., the actual difference in attendance percentage, credits earned, grade point avergae, and courses failed, Table 7.2 Adjusted Differences Between Academy and Comparison Groups, by Class, The Health Academy Courses Dropout Attendance Credits GPA Failed Probabilitv Class Entering Fall 1985 1985-86 Outcomes 9.3* 27.0* 1.1#* -3.4#* I 1986—87 Outcomes 15.6 26.2* 0.6# -3.7* -0.29* Class Entering Fall 1986 1986-87 Outcomes 8.0* 213* 1.0* -3.1* I #=Estimated from separate repressions for academy and comparison groups. I=Insufficient number of dropouts for analysis *=Statisticallv significant Twelve of the 13 differences in Table 7.2 are statistically significant, and all 12 indicate better performance by academy students. The first academy class in both years, and the second academy class in its first year, had higher attendance rates, failed fewer courses, and earned more course credits than comparison groups. Also, both academy classes achieved significantly better grades in their first year than comparison groups. Finally, the first academy class had a significantly lower probablility of dropping out of school than did students in the 1986-87 comparison group. The one difference in the table that was not statistically significant also indicated better performance by academy students. Questionnaire Responses The following sections provide a general overview of the responses received through individual questionnaires. Those providing feedback included program students, THE HEALTH ACADEMY 67 academy school personnel, parents of academy students, private-sector participants, and student mentors. Student uestionnaires Table 7.3 presents data from student questionnaires. A brief version is administered to all students at the point they enter the program, asking about their career- related plans and attitudes toward careers, school, and themselves. A longer version is administered at the end of each school year, asking about these topics again. This version also requests information about program activities and asks students to rate the program. Data are presented by class, so that one can see differences between this year's 10th and 11th grade classes. FOr the first class (11th graders), responses are presented separately for each year. The numbers reflect either the percentage of students responding as indicated, or the mean number, depending on the form of the item. TABLE 7.3 Student Questionnaire Responses, The Health Academy (means [decimals] or percentages of respondents, as indicated) Class] ClassZ Pre Sprin2'86 Spring'87 Pre Spring '87 Program Descriptive Information How many academy classes are you currently taking? n/a 3.9 3.8 na 3.3 Who have you asked about career or job information this year? a. Academy teacher n/a 83% 100% n/a 100% b. Other high school teacher n/a 0% 0% n/a 0% c. Parents or other relatives n/a 67% 58% n/a 44% d. Friends n/a 42% 17% n/a 33% e. Academy mentor n/a 0% 33% n/a 28% f. Employer n/a 0% 0% n/a 17% g. School counselor n/a 33% 33% n/a 22% h. School career-center staff n/a 0% 8% n/a 17% 68 PENINSULA ACADEMIES 1986-87 EVALUATION Class 1 Class 2 continued PIC Spring 'Sé Spring '87 PI'C Spring '87 Students who had a mentor this year n/a 13% 82% n/a 53% Of these, those who found it useful n/a ° ' n/a 100% Students who had a job arranged through the program a. Summer job n/a 8% 17% n/a 11% b. School-year job n/a 0% 42% “/3 6% Of these, those who found it useful n/a 0 0 n/a o How many guests from industry spoke in your classes this year? a. Fall semester n/a 1907 90 n/a 15 b. Spring semester n/a 150 67 n/a 49 Those who found speakers helpful n/a 92% 92% n/a 88% How many field trips did you attend with the academy? a. Fall semester n/a 21 09 n/a ., b. Spring semester n/a 40 6.3 n/a 6.1 Those who found field trips useful n/a 82% 91% n/a 94% Did you get the extra help you needed with your school work during the past year? n/a 60% 50% n/a 71% Who gave you the help you needed? a. Teacher in class with the problem n/a 90% 60% n/a 88% b. Another academy teacher n/a. 80% 50% n/a 47% c. A peer tutor n/a 10% 0% n/a 6% d. Another student or friend n/a 50% 60% n/a 47% e. Parent(s) n/a 30% 30% n/a 29% THE HEALTH ACADEMY continued Program Ratings 69 Class 1 Class 2 Pre Spring'86 Spring'87 Pre Spring '87 Academy equipment was good or excellent n/a 75% 92% n/a Academy materials were good or excellent n/a 75% 92% n/a Academy schedule better than regular one n/a 82% 82% n/a Felt strong connection between what were studying in school and postgraduate plans n/a n/a 82% n/a Liked academy better than regular program n/a 90% 90% n/a Career-Related Plans Have definite plans for the first two years after graduation 27% 36% 45% 90% If have plan, it includes a. A job 22% 44% 44% o b. Junior college, college, or university 100% 100% 89% 0 0. Vocational or technical training 0% 0% 0% ° d: Military service 0% 0% l 1% 0 e. Marriage and housekeeping 0% 0% 11% 0 Have a longwterm career goal 89% 67% 67% 0% Attitudes Related to Career, School, and Self Feel it is very important to make career plans while in school Feel it is very important to gain job skills while in school Found classwork interesting during past year 82% 82% 82% 0 82% 91% 73% 79% 55% 91% 82% 100% 88% 100% 88% 100% 94% 40% 60% 93% 75% 70 PENINSULA ACADEMEES 1986-87 EVALUATION Class 1 Class 2 continued Pre Sonng '87 Pre Spring '87 Feel good about self after past year 90% 90% 80% 100% 100% Would not want to be someone different 89% 78% 67% 17% 83% Feel success is not mainly due to luck 70% 80% 80% 0% 92% -= Missing data n/a= Not applicablL School Personnel Responses At the end of the school year, three members of the school staff completed questionnaires, providing their responses to the program. These were the school principal, academy lead teacher/director, and one other academy teacher. Their responses included the following: 0 The academy's facilities, equipment, curriculum, and instructional materials were all rated "good" to "excellent." 0 Nonacademy administrators, teachers, and counselors at the high school were all rated as "very supportive" of the program. 0 Program planning was rated "good" or "excellent," 0 Cooperation and support from the private sector was rated from "somewhat" to "very" effective. - On a seven—point scale (1=very weak, 7=very strong), the academy received an average rating of 7 "in concept" and 6 "as implemented." 0 No program weaknesses were listed by respondents. - The program's greatest strength was the lead teacher. THE HEALTH ACADEMY 7 1 Parent Responses Four parents completed questionnaires at the end of the school year. Their responses included the following: All four parents rated the program as "excellent." Three of four parents reported that their children were "much more interested in school and learning" since joining the academy, the fourth "a little more." Parents had been involved with the. program in several ways: attending school events, meeting with teachers, and helping their children with homework. In response to how well they had been kept informed of their children's progress in school, three parents responded "very well," the fourth "moderately well." The only suggestion for program improvements was to "keep only those in the program who work hard." Priv - trR n Eight private-sector representatives completed questionnaires at the end of the school year. Their responses included the following: Asked whether the academy‘s curriculum would prepare students for jobs locally, all eight respondents said "yes." Asked about communications with the program, all eight rated these as "excellent." The effectiveness of the program's steering committee was rated as " good." Ratings of students who had worked for the companies represented by these respondents were either "high" or "very high" on all dimensions listed, including dependability, appearance, attitude, language, job preparation, and job performance. The program's greatest weakness was school district bureaucracy and weak initial support (which has improved) and the hospitals’ inability to provide program funding. 72 PENINSULA ACADEMIES 1986-87 EVALUATION - The program's greatest strengths were the partnership approach between local hospitals and the Oakland schools, and the positive attitude of students in the program 0 Suggestions for improvements included "to continue to increase academic expectations of students," "increase the amount of medical terminology," and "add microbiology lab work." No mentor questionnaires were received. Summary Visiting this program is an interesting experience. The program's director and lead teacher offers example after example of student problems and program shortcomings. Yet this is done with a sense of devotion to the tasks and affection for the students, and it becomes clear that attention is being given to every task and student. Given the size of the tasks and nature of the students in the program, such an ambitious approach cannot but lead to frustrations and stress. But caring this much and paying this price has resulted in remarkable success. There is virtually no dimension addressed by the evaluation in which this success fails to appear. Virtually all features of the academy model have been replicated. The school-within—a-school model is almost fully in place. Support for the program within the school is strong, and the same seems largely true at the district level. Private-sector support is developing well, including speakers, field trip sites, mentors, and jobs. Indeed, both hospital management and a hospital labor union provide support. And the award of a "California Academic Partnership" to develop a partnership with the Samuel Merritt College of Nursing provides strong ties to related higher education. Student outcome measures examined here reflect this success. On 12 of 13 tests of differences between academy and comparison- group students, academy students perform at a significantly higher level. Responses from student questionnaires are also largely positive, and responses from adult questionnaires (school staff, parents, and private-sector representatives) are consistently positive. In short, this program is working. chapter 8 The Foothill Computer Academy Oak Ridge High School El Dorado Union High School District Diamond Springs, California The Foothill Computer Academy struggled with a number of problems in 1986-87. The program's coordinator, who had been at Oak Ridge half-time the first year, moved full-time to another school in the district, weakening the program's leadership. The two teachers who composed the academy's principal staff were new to the program, and one was a first-year teacher, reducing continuity with the first year. The academy school— within-a—school model was followed, but only in part. And the district Board of Education, struggling with budget problems, provided little support. Private-sector support developed little as well—there was no private-sector advisory body, no curriculum contributions, and the number of speakers declined. While there was a successful mentor program the first year, it was not continued in 1986-87. Summer jobs for juniors were not developed. In spite of these problems, there continued to be some encouraging results from the school-based part of the program. Setting El Dorado Union High School District is located just east of Sacramento and covers an area ranging from suburban to rural. It contains several small cities, including El Dorado Hills and Diamond Springs. The economy is largely service-based, with almost no industry and few large employers. Oak Ridge High School is a modern facility located in a picturesque semi—rural location, with housing developments springing up around it. Its student population of approximately 950 is composed of 97 percent white, 2 percent Hispanic, and 1 percent Asian/Pacific Islander. Its parent education index places it at the 90th percentile statewide. Two percent of its students are from families receiving AFDC. 73 74 PENINSULA ACADEMJES 1986-87 EVALUATION Student Selection A well defined student selection process was developed during the planning phase of the program. This involves examining records of 9th grade students, looking for students who fit the following criteria: 0 10 or more absences ° Lower than a 2.0 GPA in either the student's core or overall classes - A deficiency of 5 or more credits (of 60 expected to be earned) ° Standardized test scores no more than two years below grade level (6th grade for the test administered in 8th grade) Students who match the criteria are informed of the program, and those interested are invited to apply. These students are then rated in terms of the likelihood they will benefit from the program, and the best candidates are selected. However, it appears that this system is inconsistently followed, and students with academic problems are at times placed in the program during the fall semester to fill out the program's enrollments. Table 8.1 presents a profile of 1986-87 program students by grade level, race/ethnicity, and gender. Comparison group figures are also presented. Table 8.1 Student Profile, The Foothill Computer Academy Academv Comparison Group Grade Level 10 22 45 l 1 28 27 Race/Ethnicity Asian 0% 2% White 100% 98% Gender Female 45% 49% Male 55% 51% THE FOOTHILL COMPUTER ACADEMY 75 Program Management Academy staff consisted primarily of two teachers, one in business English and computer lab and one in English and math. The second teacher was the only one whose schedule was fully dedicated to the program, and he was a first-year teacher initially unfamiliar with the program's purpose, history, and students. The history and science teachers, who had mixed classes of academy and nonacademy students, were sympathetic to the program's purposes but did not really view themselves as academy staff. The program‘s originator and coordinator was assigned full-time at another district ., high school this year and had very limited involvement in the program. The principal, while supportive of the program and its purposes, had little day-to—day direct involvement except when problems occured. Responses from academy teachers and students suggest that the program has a low profile and a remedial image. The program is structured around the school-within-a-school model, with students assigned a core of classes and teachers. However, while the model calls for one technical/laboratory class and three academic classes within the academy, this was achieved only in part. Sophomores took computer laboratory, English, and math classes as a group. They were also grouped for history and science, although in these classes nonacademy students were involved as well. In their junior year, academy students took a life skills/computer laboratory and were grouped in mixed English and history classes. An optional tutorial period before the regular school day was offered to both groups, but few students attended. Enrichment activities were included in the 1986-87 program. At awards luncheons at the end of quarters or semesters, every program student received some kind of award. Students were shown movies approximately every three weeks if they performed well enough to earn this. Students' homework was checked daily and rewarded if well done. In an attempt to encourage students who were performing poorly, students were allowed to repeat assignments for a higher grade if they did poorly. A one-day trip was made to San Francisco, specifically to visit a planetarium and the Exploratorium. A friendly open atmosphere was maintained in academy classes so that students felt free to discuss problems and trust their teachers. Little private-sector support developed, though it was not totally absent. The program's administrative structure hindered this support. In addition, there are few large businesses near the high school, which is located in a semi-rural area removed from industry. Nevertheless, there were three business speakers. Juniors participated in mock interviews, which were videotaped so they could learn from their performances. A Junior Achievement program was initiated during the fall semester for program students. A few 76 PENINSULA ACADEMIES 1986-87 EVALUATION students had pan-time school-year or summer jobs related to the program's training, which they arranged on their own. There was no private-sector advisory board, busmess contribution to the curriculum, mentors, or program-arranged JObS. Program Resources Information available on resources used by the Foothill Computer Academy is sketchy. The two academy teachers reportedly spent a total of 1,200 hours on the program during 1986-87. Administrators (including the program director) spent 390 hours on the program; secretaries, 35 hours; and counselors, 20 hours, Participants from local businesses contributed a total of .55 hours. A reported $10,000 was expended on computers and instructional materials, of which $6,000 was school district money and $4,000 came from the state grant. Material contributions from businesses came to $80. Student Outcome Data The most objective form of data by which to judge the program's effectiveness is measurable indicators of student performance. These include attendance, credits earned, courses failed, grade point average, and retention. While these are not the only indicators of the program's effect on students, given its emphasis on motivation and career preparedness, they provide the most statistically convincing measures. Later sections in this report address attitude changes and more subjective responses from program participants. Table 8.2 presents differences in outcomes between academy students and the comparison groups, when these are adjusted to account for differences between the two groups in age, gender, race or ethnicity, and academic performance in the preceding year. In this table, positive numbers in the attendance, credits, and GPA columns mean that academy students compiled better records than the comparison group. Negative numbers in the columns for courses failed and school dropout probability also mean that academy students outperformed the comparison group. Most of the numbers represent "natural" units, i.e., the actual difference in attendance percentage, credits earned, grade point average, and courses failed. THE FOOTHILL COMPUTER ACADEMY 77 Table 8.2 Adjusted Differences Between Academy and Comparison Groups, by Class, The Foothill Computer Academy Courses Dropout Attendance Credits GPA Failed Probability Class Entering Fall 1985 1985-86 Outcomes 2.2 7.8* 04* ~03 I 1986-87 Outcomes NA 0.8 0.3 0.1 -0.02 Class Entering Fall 1986 1986-87 Outcomes NA 2.3 0.7* =1.0 I I=Insufficient number of dropouts for analysis *=Stan 9 « N Appendix C Multivariate Analysis of Academy Program Effects Underlying this statistical model is the notion that a student's performance in a given year can be predicted to some extent by that student's performance in the preceding year and by the student's gender, race or ethnicity, and date of birth. The question is: when differences between academy and comparison students in these predictor variables are taken into account, are academy students performing better than their counterparts in the comparison group? The exact relationship between predictor variables and this year's performance may or may not be the same for academy students as for students in the comparison group. Therefore, the first step in the analysis is to estimate the relationship between predictor variables and student performance in the academy and comparison group at each site and to test whether that relationship is the same. Specifically, let (1) ’iij,t= a + inj,t—1 + CXij denote the estimated relationship between predicted student performance and predictor variables. Here 9m denotes the predicted performance (attendance, credits, grades, or courses failed) of the ith individual student in the jth program (either an academy or comparison group) in year t. Yij,t-1 is the previous year's value of the same performance measure for the same student in the same program. Xij stands for a set of predictor variables (gender, race or ethnicity, and date of birth) that pertain to that same student but do not change from year to year. Finally, a, b, and c represent regression coefficients estimated from the data. In equation ( 1), these coefficients have the same value for academy and comparison- group students at each site. That is, the model in equation (1) assumes that the relationship between predictor variables and student performance does not differ between the two groups of students at each site° Equation (1) is estimated by pooling data for both groups at each site and computing the ordinary least-squares regression coefficients. To test the assumption that the relationship is the same for both groups, two alternative models are estimated. (3) 1%; = aj + inj,t-1 + CXij Equation (2) is the same as equation (1) except that the intercept coefficient, a, is now allowed to take a different value for the academy group than for the comparison group. However, in equation (2) the slope coefficients, b and c, are still constrained to be the same in both groups. Equation (2) 153 l 5 4 PENINSULA ACADEMIES 1986-87 EVALUATION is estimated by pooling the data for both groups, but adding a binary variable which denotes whether a student belongs to the academy or comparison group. A (3) Yij,t = aj + ijij,t-1 + Cinj In equation (3), both the slope coefficients, b and c, and the intercept coefficient, a, are estimated separately for the academy and comparison groups. Estimating equation (3) requires computing a separate regression for each of the two groups, Equation (3), which "custom tailors" the model to fit each group of students, can account for more of the observed variance in student performance than equations (1) or (2). However, estimation of separate coefficients for each group of students requires that a larger number of coefficients be estimated. The statistical criterion for deciding whether to use model (3) instead of model (1) or (2) is whether the better fit obtained with equation (3) is "worth" the degrees of freedom that are used up in estimating separate coefficients for each group. Formally, this question is answered by performing an F test. To test the null hypothesis that slope coefficients have the same value for both groups of students, while the intercept coefficient may have different values, the F statistic is computed as follows. (E2~E3)/(n3-n2) F(n3—n2;N-n3) = E3/(N-n3) where E2 = the sum of squared errors around regression equation (2); E3 = the sum of squared errors around the separate regressions represented by equation (3); n2 the number of regression coefficients estimated in equation (2); n3 = the total number of coefficients estimated in the separate regressions represented by equation (3); and N = total number of students in the academy and comparison groups combined. If this F-statistic is smaller than the critical value, one concludes that it is appropriate to pool the data on academy and comparison students in order to estimate a predictive model. The choice then is between equation (1) and equation (2). That choice simply depends on the estimated coefficient on the binary variable denoting group membership in equation (2). If that coefficient is APPENDIX A l 5 5 significantly different from zero, equation (2) is the best model; otherwise, equation (1) is best. To say that equation (2) is better than equation (1) is equivalent to saying that being in the academy makes a statistically significant difference. The size of that difference is the estimated coefficient on the variable denoting academy membership. If the F—statistic above is greater than the critical value, then it is apropriate to use separate regressions for the academy and comparison groups. One then uses the following procedure to test whether performance of the average student in an academy program was better than would be predicted if that student had been in the comparison group instead. Let ’y‘A A = predicted value of outcome (attendance, credits, grades, or courses failed) for a student whose predictor variables had values equal to the mean for students in an academy program, where the predicted outcome is computed from regression coefficients estimated for the academy group; 37 AC predicted value of outcome for a student whose predictor variables had values equal to the mean for students in the academy program, where the predicted outcome is computed from regression coefficients estimated for the comparison group; §CC predicted value of outcome for a student whose predictor variables had values equal to the mean for students in the comparison group, where the predicted outcome is computed from regression coefficients estimated for the comparison group; and predicted value of outcome for a student whose predictor variables had values equal to the mean for students in the comparison group, where the predicted outcome is computed from regression coefficients estimated for the academy group. A YCA Comparison of [y A A and ’37 AC tells us whether a student with the average characteristics of the academy group did better in the academy than he or she would have been predicted to do in the comparison group. Likewise, comparing yCC and 9C A indicates whether a student with the average characteristics of the comparison group did better in the comparison group than would be predicted if he or she had been in the academy program instead. If both of these comparisons favor the academy program, it is apparent that the academy program is having a positive effect. Statistical significance of those differences is tested by determining whethery~ AC lies outside the 95 percent confidence interval for predictions around y A A , and whether 37C A lies outside the 95 percent confidence interval for predictions around 9C0 LEY LIB AR ulluiflhjgfgifjnnlmlulHim Minn “11073'4