721 Analysis of an Open Textbook Adoption in an American History Course: Impact on Student Academic Outcomes and Behaviors Penny Beile, Aimee deNoyelles, and John Raible* Textbook costs can have a significant impact on the purchasing behaviors and aca- demic success of higher education students. Open textbooks promise significant cost savings, yet perceptions about quality and efficacy still linger. This study explored the impact of an open textbook adoption in an American history course on student academic outcomes and behaviors. Using a mixed-methods design, significant sav- ings were realized with no decrease in student academic outcomes. Further, students reported having a positive experience using the open textbook, perceived the textbook as being of high quality, and expressed gratitude about the free cost. The authors describe the respective roles of the librarian/instructional designer team and note the importance of working collaboratively with instructors to ensure successful implementation of open textbook adoptions. Analysis of Bureau of Labor Statistics data indicate that textbook costs have almost doubled during the past 20 years, even when controlling for a 55 percent inflation rate.1 Some commer- cial publishers have effectively priced textbooks out of the market for many college students. For example, in 2016, the Florida Virtual Campus (FLVC) administered a statewide survey to Florida higher education students to examine how the cost of textbooks impacted their educa- tion, purchasing behaviors, and academic success.2 Approximately 22,000 students from across the state responded to the survey, and the authors were able to obtain survey responses spe- cific to the institution, the University of Central Florida (UCF). Of the 1,975 UCF students who responded to the FLVC survey, 53 percent indicated that they “frequently” or “occasionally” had not purchased a textbook due to cost, and 19 percent attributed obtaining a poor course grade to not having the textbook. Penny Beile is Associate Director of Research, Education, & Engagement in UCF Libraries at the University of Central Florida; email: pbeile@ucf.edu. Aimee deNoyelles and John Raible are Instructional Designers in the Center for Distributed Learning at the University of Central Florida; email: aimee@ucf.edu, john.raible@ucf.edu. This study was conducted as part of the OER Research Fellowship program, which is administered by the Open Education Group and sponsored by the William and Flora Hewlett Foundation. Findings were not influenced by this association. The authors would like to acknowledge John Hilton III for his unending dedication and support of all things open education. Appreciation also is extended to Andre Watts, Associate Director of UCF Institutional Knowledge Management, for his guidance and analysis support. ©2020 Penny Beile, Aimee deNoyelles, and John Raible, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:pbeile@ucf.edu mailto:aimee@ucf.edu mailto:john.raible@ucf.edu https://creativecommons.org/licenses/by-nc/4.0/ 722 College & Research Libraries May 2020 Additionally, local survey research conducted at the institution reveals that 82 percent of students have delayed purchase of a course textbook “frequently” or “occasionally” due to cost. Other large-scale research studies suggest that students’ lack of access to course materials from day one may place them at an academic disadvantage, while students enrolled in courses using open textbooks tend to enjoy higher GPA, increased retention, greater satisfaction, take more courses and therefore graduate earlier, and leave with less student debt.3 Escalating textbook costs, and the potential impact on student behaviors and academic outcomes, has sparked a response from legislators and higher education administrators as well as on the teaching frontlines from instructors and campus academic support units. For example, the federal 2008 Higher Education Opportunity Act contains a section that requires institutions to communicate the price of course materials to students at the time of registra- tion for the purpose of ensuring “that students have access to affordable course materials by decreasing costs to students and enhancing transparency and disclosure with respect to the selection, purchase, sale, and use of course materials.”4 In 2015, the Affordable College Textbook Act (S.2176) acknowledged that the high cost of college textbooks was a barrier for many stu- dents in achieving a higher education and specifically called for expansion in the use of open textbooks to achieve savings for students.5 Recently, Congress funded a $5 million grant to create open textbooks and renewed funding for next year.6 Complementing federal activity, more than half of all US states have legislation that contains provisions for efforts to reduce the cost of course materials for students.7 Within Florida, Statute 1004.085, Textbook and Instructional Materials Affordability,8 explicitly authorizes institutional boards of trustees to adopt policies for the use of innovative pricing techniques and payment options for textbooks and instructional materials. A requirement of the legisla- tion includes an annual report submitted by boards of trustees to the respective chancellors of the college and university systems, highlighting cost variances and institutional initiatives designed to reduce the cost of textbooks. In a series of focus groups held at UCF, a large public research-intensive institution, fac- ulty confirmed that many students don’t purchase course materials and complain about the high cost of textbooks. Faculty further reported pursuing a variety of creative ways to lower costs, including using older editions of the traditional course text, using the same high-priced textbook across multiple courses, and putting the course text in a centralized location for students to access—usually the library’s course reserve collection. It was within this environ- ment that librarians and instructional designers formed a small working group with a com- mon purpose in mind, which was to explore what could be done to increase student access to affordable course materials. Comprising three librarians and two instructional designers, the working group initially came together to discuss “cease and desist” notices received from university administrators. The library received such notice when a librarian had posted a guide on how to save money on textbooks, and the instructional design team when they investigated platforms for hosting open or library-sourced materials for purposes of making course materials more affordable. When librarians met with institutional attorneys to further describe their rationale for the guide—and the legislation that backed them—they were told that the state legislation was not written strongly enough to overturn the “no compete” clause in the contract with the bookstore. However, it was determined that open (and therefore freely available) and library-sourced materials (already held by the institution) were outside the bounds of the “no compete” clause, Analysis of an Open Textbook Adoption in an American History Course 723 which allowed the working group members to advocate for adoption of affordable course materials and help faculty transition from commercially produced textbooks to no/low-cost alternatives. Initially, the working group was composed of two instructional designers who had explored the platforms and two librarians: the librarian who posted the guide and an as- sociate director of the library. Due to potential heavy involvement with copyright concerns, the Scholarly Communication librarian was recruited and joined the team shortly thereafter. Since summer 2016, the textbook affordability working group has presented at various venues around campus, ranging from faculty department meetings to faculty development conferences and professional development days. One option promoted by the working group is the adoption of course materials that are freely available to students through an open license. Often called Open Educational Resources (OER), these openly licensed course materials can include videos, websites, or textbooks, among other types of artifacts.9 Faculty adoption of existing OER has been most successful at the General Education Program (GEP) level, and OpenStax texts (http://openstax.org) are by far the most adopted. Work with the Faculty Cen- ter for Teaching and Learning, a faculty development center, has been especially fruitful in reaching GEP-level instructors. Describing how textbook costs can negatively impact student academic behaviors and performance and offering examples of open textbooks have resulted in successful faculty adoptions of open course materials. An American history instructor was one early adopter. The instructor attended a presentation by librarians and instructional designers and was intrigued enough to transition from using a commercial course textbook to an OpenStax US history textbook. As the introductory American history course was no longer a requirement of the GEP, the instructor wanted to explore whether adopting an open textbook could have a positive impact on access to course materials, declining enrollments, and student evaluation ratings. While this served as the primary impetus for embarking on the study from the perspec- tive of the instructor, the librarians and instructional designers also were interested in gath- ering institution-specific data related to student perceptions, use, and performance. Student demographics vary across institutions, so it was deemed especially important to see what was happening locally rather than relying on information about what is taking place at other institutions. The research team also hoped for increased credibility of open educational re- sources, in that the “you get what you pay for” mentality still lingered, making the quality of free course materials suspect.10 A final reason for undertaking the study was that the authors wanted to monitor the implementation of the adoption and be able to immediately address any issues that came up. The remainder of this paper explains the research framework, summarizes the literature, reports findings of a study conducted to examine the impact of the adoption of an open textbook on student behaviors and academic outcomes, and describes how results were acted upon. Research Framework The research on the impact of open textbook adoption in higher education environments has dramatically increased in the last two years, and many researchers rely upon the COUP Framework to explore this impact. The COUP Framework, offered by the Open Education Group, consists of four strands: Cost, Outcomes, Usage, and Perceptions.11 The Cost strand explores the magnitude and direction of the financial impacts of OER adoption on students and institutions, and can range from student savings to changes in tuition revenue due to http://openstax.org 724 College & Research Libraries May 2020 changes in enrollment intensity, drop rates, and persistence.12 The Outcomes strand measures the magnitude and direction of the learning impacts of OER adoption and provides evidence of changes in learning proxies, such as the percentage of students receiving a “C” or better, rates of completion, persistence, and graduation rates.13 The Usage strand is often employed to explore how students use OER in novel ways and instructors engage in new pedagogical practices, while Perceptions provides a framework to investigate what faculty and students think about OER.14 Review of Literature This review is structured around the four strands that make up the COUP Framework. It explores the studies that came before and where gaps still exist, especially in the context of discipline-specific research and the sparse literature focused on American History. Cost Student savings can be a persuasive factor to encourage faculty to adopt open textbooks. The most common way to track savings is by comparing the cost of previously assigned textbooks to the newly adopted open textbooks and calculating savings based on number of student enrollments. Depending on level of involvement, savings can range significantly. For example, a statewide initiative in Georgia reports student savings of more than $31 million since its inception in 2014–2015,15 while individual faculty who have transitioned to open textbooks can save students thousands to hundreds of thousands of dollars over time for one course.16 While the Cost strand focuses on savings, reducing the cost of course materials, in and of itself, can contribute to increasing access and implications thereof and reducing college debt. These are explored in the Outcomes strand. Outcomes Most of the research in open textbooks focuses on student academic outcomes. This could be a change in grade percentages or rates of completion and withdrawals, among other per- formance indicators. In a meta-analysis of 22 studies about open textbook adoption that had been published in the last three years, Hilton concluded that 95 percent of the studies cited same or better outcomes for students who used open textbooks versus commercial course materials.17 In a study that compared open textbooks to purchased materials in an American Government course, Lawrence and Lester concluded that …the outcome of this study is something of a mixed bag for the effectiveness of, and satisfaction with, OERs in political science. While the findings do not support the notion that OERs represent a dramatic improvement over commercial texts, nor do they indicate that students perform substantially worse when using open content texts either.18 Specific to American history, one study examined eight undergraduate course sections (two of which were introductory American history) that switched from commercial to open textbooks.19 The authors found that there were significant positive differences for students enrolled in the open textbook courses, as measured by final grades. Another study, which compared five sections of an introductory history course using an open textbook with five sec- Analysis of an Open Textbook Adoption in an American History Course 725 tions that used a commercial textbook, found a moderately positive relationship between the use of the open textbook and student academic achievement. Although the study controlled for mode of delivery, the authors acknowledged that it did not control for teacher effect, which is the measurable difference a particular teacher has on the outcomes of students.20 However, it’s difficult to study the true effectiveness of OER adoption without consider- ing the many nuanced factors that are involved. For instance, if two course sections are being studied, and the instructor is not the same, teacher effect could influence student performance in a course.21 In an analysis of 37 course sections in several General Education courses, Win- itzky-Stephens and Pickavance found that, when controlling for certain factors such as the instructor, students who were new to college experienced a small positive impact on their course grade when using open textbooks.22 As such, they suggest controlling for instructor-, course-, and student-level variables to more accurately gauge the impact of OER adoption on student outcomes.23 Usage Usage is often measured by how instructors edit, supplement, and/or delete material in open course materials, but it can also include an in-depth look at how students actually use the open textbook. In our search of the literature, there were limited studies that explored student usage, specifically access and usability of open textbooks. However, the few studies that were found looked at frequency of use compared to traditional textbooks, how students accessed and tended to use the open textbook, and how they rated readability of the text. For example, students us- ing an open textbook in a physics course were asked about their frequency of use of the open text compared with how often they used commercial textbooks in other courses, with students responding that they used the open text approximately the same as with a commercial textbook.24 Results of another survey designed to explore students’ use of open textbooks, including how they access them, reported that most students used digital versions of the open textbook, with 72 percent downloading a PDF file and 50 percent reading course materials online on the com- puter. Of the students who preferred print format, the majority of them simply printed pages or chapters as needed.25 Other studies have focused on issues of usability and readability. For instance, 64 percent of students rated the readability of the open biology textbook as “good” or “very good” in a study conducted by Watson, Domizi, and Clouser.26 Perceptions Gauging perceptions of open textbooks is usually accomplished by asking students to rate the effectiveness of the text on factors such as credibility, quality, and alignment with the course. Methodologically, surveys and focus groups are most often used for this line of inquiry. In his meta-analysis, Hilton reported that students (and instructors) are generally positive about OER.27 For example, one study reported that 90 percent of student respondents indicated OER were the same or better quality as commercial textbooks.28 Others also have reported on the quality of open textbooks, with open textbooks typically perceived by students to be the same or better than commercial textbooks.29 In a math course that offered an open textbook, 83 percent of students agreed that the OER supported their work in the course.30 The study described herein adds to the scant literature found on the impact of open text- book adoption on student academic behaviors and performance in an introductory American history course. It also controls for teacher effect by analyzing outcomes of students enrolled 726 College & Research Libraries May 2020 in the course taught by the same instructor and compares outcomes of students both prior to and after adoption of the open textbook. Questions and Methods To investigate the impact of an open textbook adoption on student academic behaviors and performance in an introductory American history course, the investigators relied on research questions from the COUP Framework, which was described earlier in the literature review section. Questions by COUP strand included: Cost: How much money did students save due to the adoption of the open textbook? How does the cost of traditional textbooks impact student academic behaviors? Outcomes: How did the adoption of the open textbook impact student academic performance, defined as course GPA, pass rate, and drop/fail/withdrawal rate? Usage: How did students report using the open textbook? Perceptions: How did students perceive the quality of the open textbook com- pared to traditional textbooks in other courses? Did students perceive the open textbook as credible, relevant to the course, and supportive of their performance in the course? Each strand of the framework called for its own unique methodological approach, ulti- mately including analysis of quantitative, survey, and focus group data. Student data were analyzed by course GPA and retention indicators, comparing students enrolled in the course the two major semesters prior to adoption with students from the two major semesters after adoption. All students enrolled in sections since adoption of the open textbook were invited to complete a survey, designed to gain insight into their usage and perceptions of the open textbook. Survey responses were further explored with student focus group sessions. Results Cost Since adoption of the open textbook, the instructor has taught 14 sections of the introductory American history course: US History 1877 to Present. Course enrollments during this time period have ranged from 23 to 150 students, with an average of 98 students in each section. The cost for a new copy of the commercial textbook was $80, while the open textbook is freely available online or available for purchase in print at the campus bookstore. Since adoption of the open text version, the bookstore has sold one copy in print. Potential savings were cal- culated based on total number of students enrolled in all sections after adoption (n = 1,370) multiplied by cost of the traditional textbook ($80), less cost of one print open textbook ($52), for total potential savings of $109,548. Although not every student would purchase a new textbook—or purchase the textbook at all—the calculation does indicate significant student savings. Analysis of an Open Textbook Adoption in an American History Course 727 As noted earlier, a local survey was administered to students who used the open textbook in the history course. Students were asked how textbook costs have influenced their behav- iors throughout their academic career; of the 278 respondents, 82 percent indicated that they frequently or occasionally had delayed purchasing a textbook due to cost, 60.8 percent had not purchased a textbook due to cost, and 25.2 percent had taken fewer courses due to text- book cost. These results were comparable to findings reported from the 2016 FLVC statewide survey (see table 1). Survey questions also asked who was primarily responsible for paying tuition and for course materials, with options including scholarships or financial aid (support that typically is not paid back), loans (typically paid back), parents, or self, with multiple selections allowed. In general, responses to who pays for tuition were divided fairly equally among scholarships or financial aid, parents, and loans, with “self” selected at about half the rate of other options. However, the trend is markedly different when students responded to the question of who pays for textbooks, with parents and students fairly equally divided, followed by scholarships and financial aid or loans (see table 2). A chi-square test of independence was performed to examine the relation between who pays for textbooks and how it affected decisions to not purchase the course textbook, delay the purchase of the textbook, or take fewer courses. The relationship between how textbooks are paid for and not purchasing a textbook was statistically significant at the .05 level (x2 = 15.68, 8 df, p = .05), as was not taking a course due to textbook cost (x2 = 16.56, 8 df, p = .04). Students who are at least partially responsible for purchasing their own textbooks and course materials, approximately 30 percent of respondents, are less likely to purchase the course textbook and more likely to not take a course with an expensive textbook than students who rely on other TABLE 1 Impact of Textbook Costs on Student Academic Behaviors; Statewide and Local Surveys 2016 FLVC Survey (n = 20,557) Locally Administered Survey (n = 278) n % n % Delayed purchasing a required textbook Not asked Not asked 228 82.0 Did not purchase a required textbook 13,670 66.5 139 60.8 Took fewer courses due to textbook cost 9,785 47.6 70 25.2 TABLE 2 Purchasing Responsibility for Tuition Compared to Textbooks Tuition (n = 508) Textbooks (n = 480) n % n % Scholarships/Financial Aid 147 28.9 98 20.4 Loans 143 28.1 87 18.1 Parents 145 28.5 151 31.5 Students/“Self ” 76 15.0 144 30.0 728 College & Research Libraries May 2020 sources (parents, financial aid, or scholarships) for purchasing their course textbooks. Delaying purchase of a textbook (x2 = 11.77, 8 df, p = .16) was not statistically significant at the .05 level. This finding was further explored during a focus group session, and students over- whelmingly commented that they were less likely to buy a course textbook when they were responsible for purchasing it, and often did not purchase a textbook when they felt it put a financial burden on their parents to purchase it. During the session, one student commented, Textbooks are well down the list. I get financial aid and have to pay tuition and fees first. If I don’t, then I don’t need the textbook, right? Then I look at how much housing and food cost. And finally, I take whatever money is left over, if any, and decide how to apply it to textbooks. I have never been able to afford to purchase all of my textbooks. Every student in the focus group acknowledged that they did the same thing, prompting the authors to create a graphic depicting the hierarchy of school needs in relation to available funds. (see figure 1). An unexpected finding was the theme of gratitude, which was identified in the open- ended survey comments and the focus group responses. For instance, one student wrote, “The textbook was provided via free pdf files by the professor. Extremely helpful on a college stu- dent budget and very accommodating and kind of the professor to do.” Another commented, “Having free and easy access to the textbook is the best thing I could have asked for!” Students were grateful for day-one access to no-cost course materials. FIGURE 1 Hierarchy of School Needs as Suggested by Student Spending Priorities Analysis of an Open Textbook Adoption in an American History Course 729 Outcomes This section reports on an analysis of student academic outcomes by comparing course reten- tion and grades of students enrolled in the American History course prior to and after adoption of the open textbook. Student data from fall 2015 and spring 2016, the two major semesters prior to adoption of the open textbook, were aggregated and compared to aggregated fall 2016 and spring 2017 (postadoption) student data. This pre- and postadoption comparison controls for pedagogical differences (teacher effect) by exploring patterns involving the same instructor. A total of 517 students enrolled in this instructor’s course during the two major terms prior to implementation of the open textbook. Among them, 54 (10.4%) students withdrew during the Add/Drop period, leaving 463 who are included in the Pass/DFW analysis. Pass rate is the number of students who receive a course grade of A, B, C, or D, while DFW is the number of students who drop after the Add/Drop window, receive a failing grade of F, or withdraw for other reasons. A total of 471 students were enrolled in the course during the two major terms after implementation. Of them, 53 (11.3%) students using the open textbook withdrew during the Add/Drop period, leaving 418 who are included in the Pass rate/DFW analysis. As an indicator of course persistence, the number of students who dropped or withdrew from the course after the Add/Drop window were compared to those who completed it. No statistically significant difference was found between the proportion of students who completed (p = 0.700), dropped (p = 0.714), or withdrew (p = 0.904) from the course prior to or during the terms where open textbooks were used by this instructor. Further, no statistically significant difference was found between the number of students receiving DFW grades in the open text- book class compared to students using the traditional publisher course textbook (see table 3). Likewise, no statistically significant difference was found with average course GPA when comparing students enrolled in the pre–open textbook adoption classes with students using the open textbook (see table 4). In sum, no changes in student academic performance were found. Students who used the open textbook performed equally well as students using the traditional textbook. TABLE 3 Drop, Pass, and DFW Rates; Pre- and Post-OER Adoption Comparison* Pre-OER Adoption Post-OER Adoption n % n % Dropped during Drop/Add Period 54 10.4 53 11.3 Pass Rate 440 85.1 382 81.1 Drop/Fail/Withdrawal Rate* 23 4.5 36 7.6 517 100.0 471 100.0 *includes drops, fails, and withdrawals after the Drop/Add deadline TABLE 4 Average Course GPA; Pre- and Post-OER Adoption Comparison Pre-OER Adoption Post-OER Adoption n mean SD n mean SD 494 2.85 1.1 418 2.82 1.2 730 College & Research Libraries May 2020 Usage The third area of investigation explored how students reported using the open textbook. The survey asked students whether they used the textbook primarily in digital format or if they printed or obtained a print version. Of the 293 students who responded to this question, 271 (92.5%) preferred to use the digital format and 22 (7.5%) preferred to use the textbook in print. As one student noted in the survey, “It was much easier to be able to take the book anywhere, whether it be a phone, laptop, tablet, etc. Also, if you’re like me, and you like to physically hold paper and write notes, you could easily print out the PDF pages.” An open-ended survey question that asked students to comment upon how the textbook or experience could be improved yielded a rich set of responses that were further explored during the focus group. Two main themes emerged from the analysis, with one related to perceptions of the open textbook, described further in the next section, and the other related to use. Students were especially frustrated with what they viewed as a disconnect between textbook readings and the rest of the course, noting that the textbook didn’t “pair well” with the quizzes and homework. Several mentioned that the quizzes appeared to be drawn from the class presentations and not the text, but they had relied on the text to study for the quizzes. One student commented, “There was apparently a miscommunication that led to the quizzes not matching the content of the text. This was in no part the fault of how the textbooks were handled and a free textbook was incredibly useful.” One of the goals of the study was to monitor use and be able to respond quickly. Once the instructional designer/librarian team reviewed the survey results, which was administered in the fourth week of the course, the issue was immediately brought to the attention of the course instructor. Apparently, the instructor had continued to use quiz questions from class lectures and had not adapted them from readings. Additionally, some of the ancillary quizzes provided by OpenStax for the American history text were not aligned to the correct chapters. The researchers worked with the instructor to ensure that clearer quiz questions were writ- ten for future quizzes and that the content had been covered in the textbook. Responses from earlier administration of the survey were compared to those after the issue was addressed, with the result that student satisfaction was greatly improved. Students were asked to rate aspects of how “easy” the open textbook was to acquire, use, read, or study from. Of the 283 students who responded to the question, 255 (90.1%) strongly agreed or agreed with the statement that the open textbook was easy to acquire; 243 (85.9%) strongly agreed or agreed that the open textbook was easy to use; 236 (83.4%), that the open textbook was easy to read; and 203 (71.7%), that it was easy to study from (see table 5). One student commented, “I really liked how you were able to download the pdf of this textbook instead of paying for the textbook. It was easy to use and I really liked how you TABLE 5 Survey Responses to Question of Ease of Use of the Open Textbook Strongly Agree/Agree Disagree/Strongly Disagree n % n % Easy to Acquire 255 90.1 28 9.9 Easy to Use 243 85.9 40 14.1 Easy to Read From 236 83.4 47 16.6 Easy to Study From 203 71.7 80 28.3 Analysis of an Open Textbook Adoption in an American History Course 731 can look up a keyword when you are studying, rather than having to flip through pages and pages to find something while you are studying.” However, the students who did not rate the textbook so highly were quite vocal in their desire for the digital open textbook to be more user-friendly, matching the way the respondents studied with print textbooks. The research- ers reviewed the comments and the online text and surmised that some students were not familiar with the functionality offered by the online text. Consequently, user instructions were created and offered in the learning management system. Perceptions Finally, student perceptions of the general quality of open textbooks, as well as the course open textbook’s credibility, relevance to the course, and support of their performance in the course were explored. First, students were asked how they would rate the quality of free educational materials, in general, as compared to traditional materials (defined as printed or digital materials for purchase). Results were positive, with 73.7 percent (n = 205) of 278 respondents indicating that the quality of OERs are about the same or better. Exploring their perceptions specific to the open American History textbook, 85.8 percent (n = 253) strongly agreed or agreed that the textbook was high in quality. Students also were asked the degree to which they agreed or disagreed with the value statement that the open textbook was credible, relevant to the course, and supported their performance in the course. Of the 283 responses, 83.7 percent (n = 237) strongly agreed or agreed that the textbook was credible, 79.2 percent (n = 224) strongly agreed or agreed that it was relevant to the course, and 66.4 percent (n = 186) strongly agreed or agreed that it sup- ported their performance in the course (see table 6). Based on student comments, one could surmise that the rating for supporting course performance was impacted by the relationship of the assessments to the textbook. Specifically, several students noted that the assessments (quizzes) did not always align with the content in the open textbook, with one student explaining, “I loved the textbook overall, but a hand- ful of the questions on the quizzes were not mentioned in the textbook at all or they were in chapters ahead and put in the quizzes prematurely.” Students cited convenience, portability, and accessibility as positive aspects of the digital textbook. Discussion Incentives for adoption of the open textbook from the instructor’s perspective included reduc- ing the cost of course materials to enhance student access and to see whether the adoption of the open textbook impacted student enrollments or student evaluation ratings in his course. TABLE 6 Survey Responses to Dimensions of Open Textbook Strongly Agree/Agree Disagree/Strongly Disagree n % n % Credible 237 83.7 46 16.3 Relevant 224 79.9 59 20.1 Supported Performance 186 66.4 97 33.6 732 College & Research Libraries May 2020 The instructional designer/librarian team was most interested in exploring whether the adop- tion impacted student outcomes to establish credibility of OER use and, as one of the earlier adoptions at the institution, to monitor implementation so that any needed adjustments could be responded to quickly. Other unanticipated outcomes of the study emerged that proved to be as significant as the planned drivers. These can best be expressed as gratitude, collabora- tion, and visibility. Each of these outcomes and themes is expanded upon below. First, the cost of textbooks does appear to be a barrier to student success. Survey responses suggest that, while scholarships and financial aid are relied upon to pay for tuition, students and parents primarily bear the cost of textbooks. Students who are at least partially responsible for purchasing their own textbooks and course materials, approximately 30 percent of respon- dents, were less likely to purchase the course textbook. Hence, the high cost of textbooks, and associated behaviors of delaying or not purchasing textbooks, may potentially disadvantage lower-income students. The course instructor was able to reduce the cost of course materials to $0, thus ensuring that each student had access to readings from day one. The instructor also was interested in exploring impact of an open textbook adoption on student enrollments and evaluations. The instructor reported that there has been an uptick in the student evaluations of instruction since adopting the open textbook, and we have since heard other faculty express the same. In one faculty focus group, an anthropology instructor confirmed that students regularly commented on the free textbook in her evaluations. Course enrollments have continued to increase over time, with student focus group attendees com- menting that they had heard about the “free textbook” course from other students. What was unexpected was the theme of gratitude for the instructor, which emerged in responses to open survey questions and focus group discussion. Students appreciated that the course instructor offered a free textbook, leading one to write, “Thank you for assigning a free textbook! I was able to purchase gas and groceries and pay my electric bill.” The instructional designer/librarian team was most interested in how adoption of an OER might impact academic behaviors and outcomes. As described in the Results section, no appre- ciable differences were seen across any student academic metrics when comparing performance of commercial textbook users with open textbook users. When “no statistically significant differences” in academic outcomes was shared across the institution, the team fielded ques- tions from an associate vice president as to why using OER had not resulted in more favorable results. Two possible explanations quickly came to mind; the first is that it is likely that having zero-cost materials in only one course doesn’t provide enough scale to make a difference, and the second is whether a textbook that costs $80 new is expensive enough to be a significant barrier to purchasing it. If this study is replicated, we will investigate higher-cost textbooks in GEP courses that tend to have high drop/fail/withdrawal rates. The team also has been asked to continue to investigate possible impacts of OER adoption on student performance. Textbook affordability is of increasing concern on our campus, and institutional support has recently been devoted to extending adoptions of open course materials. Specifically, a vice provost is dedicating staff resources to facilitate and support adoptions and celebrate faculty efforts through recognition awards and ceremonies. The provost has named textbook affordability a primary initiative of her office31 and recently allocated funding for faculty creation, adoption, or adaptation of open textbooks. And the library has received support for a dedicated textbook affordability librarian to continue to facilitate adoptions and track sav- ings for performance funding and legislative reports. Although these activities are not solely Analysis of an Open Textbook Adoption in an American History Course 733 due to this study, results of the study have been used to advocate for increased attention to the topic and increased adoptions by course instructors. And while these are all excellent outcomes, an even greater incentive is to more fully understand the role that open textbooks play in student success and to further explore how librarians and instructional designers can play a key role in that effort. Certainly, providing open access and open data services positions librarians to play an active role in the open textbook movement. Okamoto summarizes librarian involvement with OER into three areas: advocating for and promoting OER adoption, collecting and providing access to OER, and facilitating the curation or creation of OER.32 The landscape of librarian support of OER is still in its infancy, and librarians have an opportunity to continue these roles while investigating new ones. As such, the authors suggest a fourth role, which is work- ing as a team with the course instructor and instructional designers to provide the necessary support for successful adoption, from inception through implementation. This paper describes several ways that the librarian/instructional designer team collabo- rated with the course instructor to support successful adoption of an open textbook in an American history course. This was especially important as the team had originally advocated for the adoption and wanted to ensure a positive and somewhat seamless transition to the open textbook. From this experience it is suggested that instructional designers and librarians, working in collaboration with the course instructor, are a critical component of open textbook adoptions, especially during the initial semester of implementation. It is important to note that an instructor cannot simply adopt an open textbook outright, with no change in other course materials or assessments, and both instructional designer and librarian expertise is useful throughout the process. On the front end of the adoption, librarian input can range from reviewing assignments and ensuring that resources are avail- able to complete them, to infusing information literacy skills throughout the course, and on to anticipating questions that might arise from student use of an online textbook. Similarly, instructional designers work to ensure that course content and materials align to goals and outcomes and that educational processes are optimized. Throughout the semester, the instructional designer/librarian team also continuously monitored student use of the online textbook, responding to issues as they arose. For example, the team became aware of the disconnect between course assessments and textbook content early on, then brought it to the attention of the course instructor and worked with him to resolve the issue. Later, through focus group comments, the team learned that some students were dissatisfied with functionality of the online textbook, such as the inability to annotate and underline. These features actually were available, yet some students were not aware of them. To that end, more information could be made available about the features of the online books and reader platforms, possibly with librarians creating guides or information on fea- tures and instructional designers embedding the resources within the learning management system. Going forward, it is further recommended that the instructional designer/librarian team deploy “listening posts” at dynamic or frequently timed intervals to learn of and address any issues that arise. Although the research team used surveys and focus groups, having a librarian or instructional designer embedded in the learning management system may allow for an even quicker response when issues arise. Course instructors, instructional designers, and librarians all bring unique roles and knowledge to a project of this type, and working collaboratively ensured that the adoption was, if not flawless, successful. 734 College & Research Libraries May 2020 Notes 1. Barry Ritholz, “Price Changes (Jan. 1997–Dec. 2017): Selected US Consumer Goods and Services, and Wages,” available online at https://ritholtz.com/wp-content/uploads/2018/02/pricechanges.png [accessed December 2018]. 2. Florida Virtual Campus [FLVC], 2016 Student Textbook and Course Materials Survey: Results and Findings (Tallahassee, FL: Florida Virtual Campus Office of Distance Learning & Student Services, 2016), available online at www.openaccesstextbooks.org/pdf/2016_Florida_Student_Textbook_Survey.pdf [accessed March 2019]. 3. See, for example, Lane Fischer et al., “A Multi-institutional Study of the Impact of Open Textbook Adoption on the Learning Outcomes of Post-secondary Students,” Journal of Computing in Higher Education 27, no. 3 (2015), which reported on an analysis of more than 4,900 students using open textbooks; and Nicholas B. Colvard, C. Edward Watson, and Hyojin Park, who analyzed performance of more than 10,000 students using open text- books in “The Impact of Open Educational Resources on Various Student Success Metrics,” International Journal of Teaching and Learning in Higher Education 30, no. 2 (2018), available online at http://microblogging.infodocs.eu/ wp-content/uploads/2018/07/IJTLHE3386.pdf [accessed December 2018]. 4. Public Law 110-315, 110th Congress, Higher Education Opportunity Act, Washington, DC: GPO, Section 133, (2008) available online at https://www.govinfo.gov/content/pkg/PLAW-110publ315/pdf/PLAW-110publ315. pdf [accessed September 2019]. 5. Introduced by Senator Richard Durbin, D-IL, the Affordable College Textbook Act was proposed to expand the use of open textbooks for the purpose of achieving savings for students. Senate Bill 2176, 114th Congress, Affordable College Textbook Act, Washington, DC: GPO, Section 2 (2015), available online at https:// www.congress.gov/bill/114th-congress/senate-bill/2176/text [accessed July 2019]. 6. SPARC [Scholarly Publishing and Academic Resources Coalition], “Congress Renews $5 Million Open Textbook Pilot for Second Year,” available online at https://sparcopen.org/news/2018/open-textbooks-pilot-fy19/ [accessed December 2018]. 7. SPARC, “OER State Policy Tracker,” available online at https://sparcopen.org/our-work/state-policy- tracking/ [accessed December 2018]. 8. 2017 Florida Statutes, Textbook and Instructional Materials Affordability, Tallahassee, FL: Florida Senate 1004.085 (2017), available online at http://flsenate.gov/Laws/Statutes/2017/1004.085 [accessed December 2018]. 9. For a more expanded definition of OER, see the Hewlett Foundation website: “Open Educational Resourc- es,” Hewlett Foundation, available online at https://hewlett.org/strategy/open-educational-resources/ [accessed December 2018]. 10. Among barriers to faculty adoption of OER is the perception of quality, in that you get what you pay for. See summary and commentary at EducationDive, “Survey: OER Adoption in Higher Ed Still Slow,” available online at https://www.educationdive.com/news/survey-oer-adoption-in-higher-ed-still-slow/514978/ [accessed October 2019]; and Edscoop,“The Evolution of Educational Publishing: Does OER Have a Quality Problem,” available online at https://edscoop.com/the-evolution-of-educational-publishing-do-we-have-an-oer-quality- problem/ [accessed October 2019]. 11. The Open Education Group, based at Brigham Young University, is composed of faculty and research- ers dedicated to expanding student access to open course materials and researching impact of those adoptions on student behaviors and outcomes. To facilitate this exploration, they have devised the COUP Framework for Evaluating OER, which is composed of four strands with a variety of questions under each strand; see OpenEd Group, “The COUP Framework,” available online at http://openedgroup.org/coup [accessed January 2019]. 12. OpenEd Group, “The COUP Framework.” 13. “OpenEd Group, “The COUP Framework.” 14. OpenEd Group, “The COUP Framework.” 15. Affordable Learning Georgia, “ALG Statistics, Research, and Reports,” available online at https://www. affordablelearninggeorgia.org/about/reports [accessed December 2018]. 16. Christina Hendricks, Stefan A. Reinsberg, and Georg Rieger, “The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions,” International Review of Research in Open and Distributed Learning 18, no. 4 (June 2017), available online at www.irrodl.org/index.php/irrodl/article/ view/3006/4220 [accessed December 2018]. 17. John Hilton III’s summary of reported research, “A Synthesis of Research on OER Efficacy and Perceptions Published between September 2015 and September 2018,” presented as a paper presentation at the 15th Annual Open Education Conference, Niagara Falls, NY (October 11, 2018), includes findings from studies conducted by Gregory Allen et al. (2015), Rajiv Jhangiani et al. (2018), Virginia Clinton (2018), and Nicholas Colvard et al. (2018), all of whom reported that students using open textbooks performed equally well or better than students using https://ritholtz.com/wp-content/uploads/2018/02/pricechanges.png http://www.openaccesstextbooks.org/pdf/2016_Florida_Student_Textbook_Survey.pdf http://microblogging.infodocs.eu/wp-content/uploads/2018/07/IJTLHE3386.pdf http://microblogging.infodocs.eu/wp-content/uploads/2018/07/IJTLHE3386.pdf https://www.govinfo.gov/content/pkg/PLAW-110publ315/pdf/PLAW-110publ315.pdf https://www.govinfo.gov/content/pkg/PLAW-110publ315/pdf/PLAW-110publ315.pdf https://www.congress.gov/bill/114th-congress/senate-bill/2176/text https://www.congress.gov/bill/114th-congress/senate-bill/2176/text https://sparcopen.org/news/2018/open-textbooks-pilot-fy19/ https://sparcopen.org/our-work/state-policy-tracking/ https://sparcopen.org/our-work/state-policy-tracking/ http://flsenate.gov/Laws/Statutes/2017/1004.085 https://hewlett.org/strategy/open-educational-resources/ https://www.educationdive.com/news/survey-oer-adoption-in-higher-ed-still-slow/514978/ https://edscoop.com/the-evolution-of-educational-publishing-do-we-have-an-oer-quality-problem/ https://edscoop.com/the-evolution-of-educational-publishing-do-we-have-an-oer-quality-problem/ http://openedgroup.org/coup https://www.affordablelearninggeorgia.org/about/reports https://www.affordablelearninggeorgia.org/about/reports http://www.irrodl.org/index.php/irrodl/article/view/3006/4220 http://www.irrodl.org/index.php/irrodl/article/view/3006/4220 Analysis of an Open Textbook Adoption in an American History Course 735 traditional textbooks. In “Evaluating the Effectiveness of the Open-access ChemWiki Resource as a Replacement for Traditional General Chemistry Textbooks,” Chemistry Education Research and Practice 16 (September 2015), https://doi.org/10.1039/C5RP00084J, Chemistry faculty Gregory Allen, Alberto Guzman-Alvarez, Amy Smith, Alan Gamage, Marco Molinaro, and Delmar S. Larsen compared academic performance of students in a Chem- istry course using an open wiki (ChemWiki) with students using a traditional textbook and found groups did not substantially differ. The authors ultimately surmised that the ChemWiki serves as a viable alternative to traditional textbooks. Similarly, Rajiv Jhangiani, Farhad Dastur, Richard Le Grand, and Kurt Penner reported that cost savings to students through use of open textbooks does not come at the expense of resource quality or student performance in “As Good or Better than Commercial Textbooks: Students’ Perceptions and Outcomes from Using Open Digital and Open Print Textbooks,” Canadian Journal for the Scholarship of Teaching and Learning 9, no. 1 (April 2018), https://doi.org/10.5206/cjsotl-rcacea.2018.1.5. Virginia Clinton, comparing student performance in two Introduction to Psychology courses, one using an open text and another a commercial textbook, reported the open textbook saved students money without negatively impacting their learning in “Savings without Sacrifices: A Case Study of an Open-source Textbook Adoption,” Open Learning: Journal of Open, Distance, and e-Learning 33, no. 3 (2018), https://doi.org/10.1080/02680513.2018.1486184. Moving from assessing impact of instructor-level adoptions to institutionwide changes, Colvard, Watson, and Park, in “The Impact of Open Educational Resources on Various Student Success Metrics,” analyzed 21,822 student performance from eight courses to arrive at the conclusion that not only does open textbook adoption save students money, it also improves course grades and decreases D, F, and Withdrawal grades. This held true across all student populations, but the trend was even more pronounced for Pell recipients, part-time students, and populations historically underserved by higher education. 18. Christopher Lawrence and Julie Lester, “Evaluating the Effectiveness of Adopting Open Educational Resources in an Introductory American Government Course,” Journal of Political Science Education 14, no. 4 (Feb- ruary 2018): 563, available online at https://www.tandfonline.com/doi/full/10.1080/15512169.2017.1422739?scroll= top&needAccess=true [accessed December 2018]. 19. Colvard, Watson, and Park, in “The Impact of Open Educational Resources on Various Student Success Metrics,” analyzed impact of an open textbook adoption across eight courses, including two introductory His- tory courses. Although the authors did not disambiguate History results from the overall analysis, they did report significantly better academic outcomes for students using the open textbooks. 20. In one of the few studies exploring impact of open textbook use in an History course, Kim Grewe and Preston Davis reported a moderately positive correlation (r = .41, p < .001) between open textbook versus com- mercial textbook use and student achievement, even when controlling for prior college achievement, in “The Impact of Enrollment in an OER Course on Student Learning Outcomes,” International Review of Research in Open and Distributed Learning 18, no. 4 (June 2017) available online at www.irrodl.org/index.php/irrodl/article/ view/2986/4211 [accessed December 2018]. 21. See, for example, a RAND Education and Labor report that acknowledges while there are many factors that contribute to students’ academic performance, research suggests that, among school-related factors, teachers will influence outcomes the most, in RAND, “Teachers Matter: Understanding Teachers’ Impact on Student Achieve- ment,” available online at https://www.rand.org/education-and-labor/projects/measuring-teacher-effectiveness/ teachers-matter.html [accessed December 2018]. When researching impact of open textbook adoption on student performance, it is particularly important that “teacher effect” be controlled for. 22. Jessie R. Winitzky-Stephens and Jason Pickavance, “Open Educational Resources and Student Course Outcomes: A Multilevel Analysis,” International Review of Research in Open and Distributed Learning 18, no. 4 (June 2017) available online at www.irrodl.org/index.php/irrodl/article/download/3118/4224 [accessed October 2019]. 23. Winitzky-Stephens and Pickavance, “Open Educational Resources and Student Course Outcomes.” 24. Hendricks, Reinsberg, and Rieger, “The Adoption of an Open Textbook in a Large Physics Course.” 25. Rajiv Jhangiani and Surita Jhangiani, “Investigating the Perceptions, Use, and Impact of Open Textbooks: A Survey of Post-Secondary Students in British Columbia,” International Review of Research in Open and Distrib- uted Learning 18, no. 4 (June 2017), available online at www.irrodl.org/index.php/irrodl/article/view/3012/4214 [accessed December 2018]. 26. C. Edward Watson, Denise Domizi, and Sherry A. Clouser, “Student and Faculty Perceptions of OpenStax in High Enrollment Courses,” International Review of Research in Open and Distributed Learning 18, no. 5 (August 2017), available online at www.irrodl.org/index.php/irrodl/article/view/2462/4299 [accessed December 2018]. 27. Hilton, “A Synthesis of OER Research.” 28. T.J. Bliss et al., “An OER COUP: College Teacher and Student Perceptions of Open Educational Resources,” Journal of Interactive Media in Education 2013, no. 1 (February 2013), available online at https://jime.open.ac.uk/ articles/10.5334/2013-04/ [accessed December 2018]. https://doi.org/10.1039/C5RP00084J https://doi.org/10.5206/cjsotl-rcacea.2018.1.5 https://doi.org/10.1080/02680513.2018.1486184 https://www.tandfonline.com/doi/full/10.1080/15512169.2017.1422739?scroll=top&needAccess=true https://www.tandfonline.com/doi/full/10.1080/15512169.2017.1422739?scroll=top&needAccess=true http://www.irrodl.org/index.php/irrodl/article/view/2986/4211 http://www.irrodl.org/index.php/irrodl/article/view/2986/4211 https://www.rand.org/education-and-labor/projects/measuring-teacher-effectiveness/teachers-matter.html https://www.rand.org/education-and-labor/projects/measuring-teacher-effectiveness/teachers-matter.html http://www.irrodl.org/index.php/irrodl/article/download/3118/4224 http://www.irrodl.org/index.php/irrodl/article/view/3012/4214 http://www.irrodl.org/index.php/irrodl/article/view/2462/4299 https://jime.open.ac.uk/articles/10.5334/2013-04/ https://jime.open.ac.uk/articles/10.5334/2013-04/ 736 College & Research Libraries May 2020 29. See, for example, Cailean Cooney, who reported that 97 percent of 67 students surveyed about the qual- ity of the open text found it the same, somewhat, or much better than a comparable commercial textbook, in “What Impacts Do OER Have on Students? Students Share Their Experiences with a Health Psychology OER at New York City College of Technology,” International Review of Research in Open and Distributed Learning 18, no. 4 (June 2017), available online at www.irrodl.org/index.php/irrodl/article/view/3111/4216 [accessed December 2018]; Hendricks, Reinsberg, and Rieger, “The Adoption of an Open Textbook in a Large Physics Course,” who found that 93 percent of 143 students surveyed reported quality of the open text to be the same or better than the commercial textbook; Barbara Illowsky et al., who stated that 87 percent of 231 students found the open text as good or better than the commercial text, in “Examining Student Perception of an Open Statistics Book,” Open Praxis 8, no. 3 (July–September 2016), available online at https://openpraxis.org/index.php/OpenPraxis/article/ view/304/218 [accessed December 2018]; and Jhangiani and Jhangiani, “Investigating the Perceptions, Use, and Impact of Open Textbooks,” who found that 96.5 percent of 307 students found the open text average, above average, or excellent in terms of quality. 30. John Hilton III et al., “The Adoption of Open Educational Resources by One Community College Math Department,” International Review of Research in Open and Distributed Learning 14, no. 4 (September 2013) available online at www.irrodl.org/index.php/irrodl/article/view/1523/2652 [accessed December 2018]. 31. University of Central Florida, Office of the Provost, “Initiatives,” available online at https://provost.ucf. edu/initiatives/ [accessed October 2019]. 32. Karen Okamoto, “Making Higher Education More Affordable, One Course Reading at a Time: Academic Libraries as Key Advocates for Open Access Textbooks and Educational Resources,” Public Services Quarterly 9, no. 4 (November 2013), available online at https://www.tandfonline.com/doi/full/10.1080/15228959.2013.842397 [accessed October 2019]. http://www.irrodl.org/index.php/irrodl/article/view/3111/4216 https://openpraxis.org/index.php/OpenPraxis/article/view/304/218 https://openpraxis.org/index.php/OpenPraxis/article/view/304/218 http://www.irrodl.org/index.php/irrodl/article/view/1523/2652 https://provost.ucf.edu/initiatives/ https://provost.ucf.edu/initiatives/ https://www.tandfonline.com/doi/full/10.1080/15228959.2013.842397 _Hlk10560929 _GoBack