key: cord-0750863-8b99y13s
authors: Rice, Mary Frances; Ortiz, Kelsey R.
title: Evaluating Digital Instructional Materials for K-12 Online and Blended Learning
date: 2021-10-06
journal: TechTrends
DOI: 10.1007/s11528-021-00671-z
sha: 6c3594c254f5ddf09ed549f6fe6ddad001ac8925
doc_id: 750863
cord_uid: 8b99y13s

With the large increase in online instruction, including remote instruction with online materials during the COVID-19 pandemic, there also was an increase in the use of instructional materials that were made to be displayed online or were digitized for online use. However, teachers have not had access to guidance about how to select and evaluate online instructional materials for classroom use. The lack of guidance has the potential to harm historically excluded populations of students and could frustrate teachers as they learn to teach with digital materials. The purpose of this paper is to share the 4A Framework for evaluating online instructional materials. The framework is organized around the premise that quality online instructional materials are accessible, promote active engagement, advocate for inclusion, and are accountable for their relationships to standards and data privacy. Each feature is discussed and examples of teacher work in applying the framework are shared.

Instructional materials that are developed for and hosted in online spaces are distinct from traditional materials (Choppin & Borys, 2017) . Digital instructional materials have the potential for increased individual interactivity (Choppin & Borys, 2017) . Also, digital instructional materials contain a broader array of adaptivity and personalization features, and they have more potential to be accessible to students with disabilities (Choppin & Borys, 2017) . Finally, digital instructional materials have a greater occurrence of built-in assessment programs (Choppin & Borys, 2017) .

The mere fact of being a digital material does not guarantee these perceived benefits (Greer et al., 2014; Rice, 2018) . Even so, teachers are expected to use these digital materials in teaching, even when most instruction occurs in person (Kalolo, 2019; Selwyn, 2007; United States [U.S.] Department of Education, 2016). With the increased pressure to acquire and use them, digital and digitized instructional materials are a rapidly expanding market within the educational technologies sector. This market is expected to grow from $85.5 billion (USD) in 2020 to $181.3 billion in 2025 (Markets Insider, 2020) . Examples of digital instructional materials include videos, content presentations, animations, educational games, online articles, and educational materials from traditional print texts that have been scanned and uploaded. The increased awareness and availability of such materials was likely aided by the onset of the COVID-19 pandemic, which closed school buildings in the United States and abroad (Education Week, 2020; UNESCO, 2020) . During this time, remote online learning replaced in-person learning in many schools. Teachers who may have been infrequent or non-users of digital instructional materials suddenly had to gather and use them for their lessons.

As students and teachers return to in-person learning, the educational technology sector is unlikely to relinquish their profits or decrease their product offerings. Some schools will continue to use fully online learning, but many more will use forms of blended learning-where instruction can be provided through both traditional and online modalities in ratios that meet the needs of the students and match the resources in the school (see Martin et al., 2020) .

The surge in popularity and availability of online educational resources and the necessity to use them during the pandemic required teachers to make quick decisions about what digital instructional materials to use to teach their students (Gallagher et al., 2019) . Selecting materials off the internet simply because they are inexpensive or even free is unlikely to result in materials that embody the benefits of using digital materials (Reidenberg & Schaub, 2018) . Moreover, it could encourage content developers and vendors to make materials that meet corporate ends without considering what teachers and students need to have educative experiences together. Using digital materials with deliberation and agency requires teachers to ask very basic but very important questions, such as what materials will I teach with? And why these materials?

Before the pandemic, much growth in digital learning came from traditional public schools who had technology integration initiatives (Digital Content Goes to School, 2016; Digital Learning Collaborative, 2019) . In many cases, teachers made decisions about digital materials for online teaching with little to no guidance from initial teacher preparation programs or professional learning (Crouse et al., 2018; Gallagher et al., 2019) . Teachers in these studies described the stress involved in not knowing how to find digital instructional materials or make them (Kundu & Rice, 2019; Kundu et al., 2021) . In such contexts, teachers made decisions based on a few factors such as compatibility with current software, the perceived technological simplicity of the materials, or faith in a recommendation from a colleague (Fang et al., 2010) . While none of those reasons are inherently problematic, teachers deserve stronger guidance for informing their decision-making about the quality of instructional materials. Moreover, this guidance should be efficacious without being too complex to remember and apply (Cheon & Grant, 2008; Chiu & Churchill, 2015; Kim et al., 2017) .

In response to the need for practical guidance about choosing instructional materials, the authors of this paper developed a conceptual tool based on implications from research and policy guidance for high quality digital instructional materials in K-12 settings. This tool is called the 4A Online Instructional Material Evaluation Framework. It is abbreviated in this document as the 4A Framework (see Fig. 1 

The 4A Framework was built using principles of generalized invariance structure theory (GIST; Vigo, 2013) . Humans organize concepts by determining a minimal number of useful but distinct rules they can apply across an array of situations (Feldman, 2000) . In GIST theory, conceptual understandings emerge as qualitative patterns are detected 

Digital instructional materials attend to multiple dimensions of active engagement (i.e., behavioral, cognitive, emotional) when they invite students to invest effort and energy into learning concepts.

Digital instrucitonal materials promote advocacy when they represent diverse peoples with contexual nuance, compassion, and respect.

Digital instructional materials demonstrate accountability when they are transparent about their origins and purposes, based on standards or principles, and are open about personal information and user data collection sharing processes.

and judged to resonate with humans based on how they have made sense of their experiences. When new knowledge is encountered, it can be applied to the present framework and placed in an existing category or determined to belong to a new one.

To make a qualitative framework of 'rules' for digital instructional materials, we, the authors of this paper located information about what constituted quality in digital instructional materials in K-12 online and blended learning settings. This included research studies in K-12 settings where digital instructional materials were being used, the U.S. Section 508 law, and the Web Content Accessibility Guidelines by the Web Accessibility Initiative of the World Wide Web Consortium (2020). Then, we made lists of features of quality instructional materials based on that literature. Next, we used the lists to organize the items into groups based on conceptual matching (see Table 1 ). We alternated roles of organizing items and giving each other feedback, striving to reduce the number of broad categories until we had the smallest number possible with minimal overlap (Watkins, 2017) . This left four categories: Accessibility, Active Engagement, Advocacy for Inclusion, and Accountability. During this process, we also compared our findings to previous frameworks to ensure such a framework did not already exist (see Table 2 ) (California State University Center for Distributed Learning, 1997; Chu, 2000; Vargo et al., 2003; Jung et al., 2016; Information Technology Industry Council, 2020) . The 4A Framework is the only framework to include both issues of accessibility for students with disabilities and advocacy for inclusion of groups in addition to students with disabilities. Also, no other framework that was reviewed addressed research on the need for materials to be accountable for the user data and personal information they are collecting and using for various purposes. While the other frameworks that were reviewed called for materials that were aligned to standards, they did not make space for educational goals that were not academic (Nesbit et al., 2009; Vargo et al., (2003) . Finally, other frameworks that included concerns about user experiences lacked descriptions of multiple types of engagement (e.g., behavioral, cognitive, emotional) (Chu, 2000; Jung et al., 2016) .

Multiple audiences for the 4A framework are necessary and acceptable in the digital realm because of the many different groups that design, choose, teach with, and purchase digital instructional materials. Other groups have also named multiple audiences in their frameworks for reviewing instructional frameworks (see Table 2 ). For example, Jung et al. (2016) specifically addressed their framework to educators and developers. Also, Chu (2000) indicated that the critical content approach to evaluating materials could be used by anyone who uses multimedia and then lists librarians, teachers, and parents.

Classroom teachers who use digital instructional materials can use the 4A Framework to evaluate materials created by others, support the modification of materials for new purposes, or as guidance to make their own materials. Because K-12 teachers are not always allowed to choose which materials they use (Archambault & Crippen, 2009; Crouse, et. al, 2018) , Course Designers might also benefit from using the 4A Framework. This might be especially important for the asynchronous, teacher-less or teacher-low instruction that is becoming increasingly popular in the K-12 context (Rice & Ortiz, 2020) . In such contexts, the only instruction students receive is from the materials. Finally, since Administrators are often the ones who hold the purchasing power, they might also be interested in the 4A Framework to support teachers as instructional leaders.

The rest of this paper conceptualizes the 4A Framework. Each section begins with a definition of each of the elements. Then, a justification using previous research and/or policies is provided. After that, examples appear as vignettes the authors of this article developed for educators to use in professional learning and future research. Appendix contains a checklist of example questions that further clarify the four elements of the 4A Framework.

Definition: Digital instructional materials are accessible when they adhere to applicable legal standards, and users can open, view, and interact with digital material.

Students cannot learn from materials that they cannot open, use, view, and interact with (Rose, 2018) . To ensure students can access digital materials, specific design features and supports must be present (Rose, 2014) . The purpose of supports is to ensure that all students, regardless of disability status, can access the same information and have the same interactions with similar amounts of effort (Dear Colleague Letter from the Office of Civil Rights, 2011). In many countries, students have the right to equal opportunities to benefit from using school-adopted technologies and digital materials (Wisdom et al., 2007) . (2017) Decrease access points and task steps to increase accessibility (e.g., logging on, finding assignment, submitting assignment) de Witte et al. (2018) Assistive technologies should be affordable and available to students who will benefit from them Edyburn (2004) Assistive technologies include text-to-speech and speech-to-text, but the definition of assistive technologies is wide Federal Communications Commission (2021) Under U.S. law, electronic devices and technologies must be accessible Information Technology Industry Council (2020) Digital devices and technologies should be easy to obtain and use Rice (2017) Teachers should evaluate text complexity in online instructional materials and add support for comprehending complex text Rice (2018) Accessibly evaluation requires more than relying on the accuracy of Voluntary Product Accessibility tables; it requires information from actual users with disabilities Rice and Deshler (2018) Teachers should provide vocabulary support for online texts even when definitions are available by 'hovering' or 'clicking' Rose (2018) Students can and do make complaints to the U.S. Office of Civil Rights about online instructional materials when they are not accessible [U. S.] Office of Civil Rights (2011) Technology-based learning opportunities should be available with equal ease to those with and without disabilities W3C (2020) Accessible content is perceivable, operable, understandable, and robust Active Engagement Rice et al. (2019) Parents locate links to digital instructional materials, including activities and videos to promote engagement Boettcher and Conrad (2016) Teachers need to be proactive about monitoring the completion of online work and providing engaging materials Borup et al. (2020) Online engagement is an ecology of various types of engagement with many actors, rather than an individual psychological activity California State University Center for Distributed Learning (1997) Digital instructional materials should lead to positive cognitive outcomes Cheon & Grant (2008) Digital instructional materials should avoid presenting large amounts of content because of the risk of cognitive overload Chiu and Churchill (2015) Digital instructional materials should contain some variability of presentation and behavioral response Crouse et al. (2018) Online teachers used technological tools and programs to engage students, such as allowing children to use drawing tools Jung et al. (2016) Digital instructional materials should support cognitive engagement and outcomes Kundu et al. (2021) Elementary children in India engaged quickly with blended curriculum when teachers changed the materials to meet students' interests and needs Shapiro (2008) Too many choices of activities may overwhelm students and diminish engagement ter Vrugte et al. (2017) Computer-mediated games, simulations, and worked examples facilitated knowledge acquisition Tonks et al. (2020) Students engaged online when emotional needs were met Watson et al. (2017) Feedback is critical for online learners' cognitive engagement Advocacy for Inclusion Archambault et al. (2021) Educational marketplaces contain online materials rife with misrepresentations, bias, and stereotyping

According to Rose (2018) , most legal complaints about accessibility claim that digital instructional materials: (1) lack alternative text (short descriptions embedded into images), (2) have been posted in formats that are inaccessible, (3) lack captions for video materials, (4) include improperly structured data tables, (5) use improper formatting and labeling of information, and (6) use improper contrast between background and foreground.

In addition, most digital instructional materials require many steps to access them (Behling, 2017) . For example, a student who needs to do a simple math worksheet online might have to (1) enter a learning management system, (2) locate the correct folder, (3) open the document, (4) print the document, (5) do the math problems, (6) scan the document with their answers, (7) upload the document, and (8) attach the document in a place where it might be graded. The more steps there are, the less accessible the materials are.

Some materials might not be accessible because the text is too linguistically complex for students. Text in online instructional materials can shift from low complexity to high complexity without notice to the student. These shifts in complexity can occur between lessons or web pages (Rice, 2017) . Support for navigating text complexity, such as vocabulary support, is often unavailable or inaccessible (Rice & Deshler, 2018) . Thus, much of the text in instructional materials can be too difficult for students to comprehend.

Supports to increase accessibility in digital instructional materials include text-to-speech/speech-to-text software or features, display change options (e.g., changing the background color, font type, or size), and alternative text (i.e., explanatory information added to describe embedded images) (Edyburn, 2004; Rose & Meyer, 2006) . Students should not be required to sign in multiple times or engage in many successive tasks (e.g., downloading, filling in, (2000) Cultural bias is common in digital multimedia materials; biases are reduced by including multiple perspectives and support for thinking Galia & Mulyana (2020) Online English language teaching materials need to be revised to include gender groups other than heterosexual male Gallagher et al. (2019) Teacher education should support teachers in noticing racism and other bias in online instructional materials Lewis Ellison (2017) Learners need to find identity and representation in digital learning experiences Mann (2019) Identifies patterns of whiteness and economic advantage in multiple aspects of online schooling Shelton et al. (2020) Teachers should not buy materials from other teachers on websites unless they evaluate them first for racism and other biases Accountability

Attai (2020) Many teachers do not understand issues of educational data privacy Beienberg (2020) Online instructional materials should disclose information to parents about content development, ideological orientations, and data Boniger et al. (2017) Data privacy is a pressing issue that needs to be addressed in schools Hodges and Barbour (2021) Digital assessment is enabled by digital tools and strategic planning by teachers who are seeking to meet goals and standards Lazar et al. (2017) Students with disabilities are more vulnerable to data misuses and breeches Selwyn (2007) The digital learning marketplace should not exploit learners Wang et al. (2019) Obtaining content expert and user feedback are important for ensuring ethical data use Vargo et al. (2003) Instructional media should align with local or state content standards (2000) Librarians, teachers, parents, and others who are responsible for selecting and providing multicultural multimedia materials (text, sound, and graphics)

The proposed criteria require the examination of four components: objectivity, language, subject mastery, and resources. These criteria are designed to be applied broadly in evaluating multicultural content of any multimedia, for any audience and level of coverage, and in any language

Focuses solely on advocacy criteria for multimedia without acknowledging other considerations; operates from a multimedia orientation but not necessarily an online one

Learning Object Review Instrument 

Corey teaches eleventh grade economics and government to students who are currently learning remotely due to a state public health order. Corey feels that discussion, even when it occurs over a videoconferencing platform, is important for developing key ideas and understandings of historical events. Corey wanted students to use an educational website with information about economic systems to answer questions and prepare for a discussion. While learning about accessibility, Corey acted on advice to teachers recommending that they place links to educational websites and the discussion questions in a shared document so students do not have to look in multiple places. With the shared document, Corey's students could follow one link, open the website in a new tab, and toggle between the website and the document, or they could open a new window and place the website and the questions side-by-side. Corey also realized the website had both dark colors and dark font. There was no way to change this. Corey found a different website with the same information that had high contrast between the background and the font. This website had a cleaner appearance, more headings, and was not disrupted by advertisements and images. Finally, Corey learned that the screen readers could not read screen captured questions as a pasted picture file. To ensure that students using screen readers could access the materials, Corey decided to type the questions directly into the document. In this scenario, Corey made strategic decisions about what digital materials to use with a focus on accessibility. Rather than memorizing a long list of accessibility technicalities, Corey focused on key ideas about access points, provided visual and auditory support by enabling screen reading technology, and anticipated problems students might have. Although institutions need to provide some supports (e.g., when human captioning is required), teachers can make many efforts to ensure instructional materials are accessible on their own.

Definition: Digital instructional materials attend to multiple dimensions of active engagement (i.e., behavioral, cognitive, emotional) when they invite students to invest effort and energy into learning concepts.

Engagement is a commonly used term in educational research that refers to the effort and energy students invest to learn the concepts presented in the learning environment (Fredricks et al., 2004; Hughes et al., 2008) . Digital instructional materials can be designed to elicit engagement from students (Cheon & Grant, 2008; Mayer, 2019) . Engaging digital materials are especially important when in-person learning contact is reduced or eliminated, which is common in remote, fully online, and even some blended learning environments (Boetcher & Conrad, 2016) . Three major types of engagement from research include (1) behavioral, (2) cognitive, and (3) emotional (Lim & Lerner, 2013; Zimmerman & Schunk, 2001) .

Behavioral engagement occurs when learners complete measurable and observable actions to learn the content (Zimmerman & Schunk, 2001) . Students actively engage in online learning activities by opening pages, clicking on links, giving answers, writing comments, submitting assignments, using emoji, and more (Chapman, 2003; Lawson & Lawson, 2013) . When teachers use a variety of digital instructional materials, such as pre-recorded videos, screencasts, games, simulations, and interactive webpages, students can engage with the materials at times convenient to them and as many times as they want (Chiu & Churchill, 2015) . Also, students should be able to choose from among multiple types of materials, instead of having to use every type of instructional material in a lesson (Rose & Meyer, 2006) . However, merely providing many materials and activities does not universally result in engagement. Students can become overwhelmed by many choices (Shapiro, 2008) . They may also fail to behaviorally engage without a clear understanding of the purpose of lesson and may even stop logging on to class (Archambault et al., 2013) . Teachers should choose a few types of materials that meet their purposes and satisfy the other elements of the 4A Framework to maintain engagement (see Fig. 1 ).

Cognitive engagement is "the expenditure of thoughtful energy needed to comprehend complex ideas in order to go beyond the minimal requirements" (Finn & Zimmer, 2012, p. 102) . Cognitive engagement often emerges alongside a behavioral response, such as asking a poignant question or completing an assignment. Cognitive engagement is supported when students have access to information about all the relevant parts of the concept rather than overly simplified definitions or incomplete explanations (Meyer, 1993; Sanders & Makosta, 2016) . Worked examples and simulations have also proven useful for learning concepts with digital materials (Roblin et al., 2018; ter Vrugte et al., 2017) .

Emotional engagement requires the "presence of taskfacilitating emotions such as interest and the absence of task-withdrawing emotions such as distress" (Reeve, 2012, p. 150) . Sometimes, students have negative emotions in response to digital instructional materials about tragic events. Presenting content that consistently evokes negative emotions, like distress, impedes cognitive and behavioral engagement (Nabi, 1999) . How learners are treated by teachers and peers also affects emotional engagement with instructional materials (Ruzek et al., 2016) .

Content presented on the internet with powerful visual media also provides more opportunities for students to exhibit unexpected emotional responses to materials (Chen et al., 2015; Lim & Childs, 2020) . Teachers should understand that students' emotional responses might not be predictable because of past experiences, culture, disability, language, or other factors. For example, students might respond to a video of the falling World Trade Center Towers with cheers and laughter, rather than with horror and sadness, because they are intrigued by the visual images of large structures collapsing. They might also be confused by many adults' negative reactions to footage of the towers collapsing because there are many videos on the internet of building implosions that were safely planned and carried out. The fact they were not yet born in 2001might make it more difficult for young people to emotionally respond to the building collapse in ways adults expect.

Highly engaging instructional materials simultaneously attend to multiple types of engagement (Chiu & Churchill, 2015) . Such materials tend to lead with intellectually interesting questions (cognitive engagement), require a recorded response (behavioral engagement), and invite appropriate feelings about the topic (emotional engagement). Materials designed for engagement also can include elements from games, such as earning points, gathering materials, or going on a quest, and using feedback (Reeves & Reed, 2009; Watson et al., 2017) . Feedback can support engagement when it includes features like leaderboards, interactive task lists, and partial credit (behavioral); information about performance, including missed items and worked examples (cognitive); and upbeat sounds, inspirational images, or short positive phrases, like "good job" (emotional) (Watson et al., 2017; Yeo & Fazio, 2019) .

Avery teaches seventh grade English. Avery's students use blended rotations to complete some of their work online. The students also engage in person with each other in small groups and as a whole class. Avery wanted to include a short video in an online lesson developed to introduce key concepts about the industrial revolution in England. Several colleagues recommended a website that contained an online video about the topic and included a free worksheet with many specific questions about minor facts and events in the video. When previewing these materials, Avery found the video very long with slow narration. Avery became concerned the students would become disinterested before the main content of the video was presented. The students might also find it difficult to attend to the detailed points of the worksheet while watching the video, and they might give up on both. Avery located a shorter video with more lively dialogue and images. Instead of a worksheet asking for many details, Avery prepared a simple organizer for students to download and record information while they watched. The organizer asked a few general questions and left space for short answers (1-3 words) . The students could also respond to the questions using emoji from a bank at the top of the sheet. Also, Avery worried that some of the images of children covered in soot, looking tired, or suffering with major physical injuries may be emotionally disruptive. Avery decided to pre-record a short video segment to prepare the students for the fact that images of some of these children might make them sad. Avery sent this video to parents using a school email list feature, so parents would know that their children were being presented with sensitive images.

In this example, the goal was for students to engage with key ideas and not just view images of children in distress. Avery did not merely find a video that was short enough to hold the learners' attention. Instead, Avery made efforts to monitor engagement with digital instructional materials to ensure students had multiple ways to behaviorally demonstrate what they were learning cognitively while also receiving appropriate emotional support.

Definition. Digital instructional materials promote advocacy for inclusion when they represent diverse peoples with contextual nuance, compassion, and respect.

Instructional materials presented to students should advocate for inclusion. Materials represent an inclusive mission when they present people as having complex lives, motivations, and interests (Chu, 2000; Lewis Ellison, 2017) . Inclusive instructional materials promote engagement, particularly among students historically from excluded groups, whose stories and perspectives are often absent or mischaracterized in school spaces (Bishop, 2012) .

Digital instructional materials reflect the hidden curriculum that permeates teaching and learning at all levels of education (Alsubaie, 2015; Margolis, 2001) . Hidden curriculum creates conditions where the materials and activities students are exposed to advance an implicit argument that characteristics such as whiteness, heteronormativity, and hegemonic masculinity are preferable to other characteristics (Selvi & Kocaman, 2020) . Students also receive messages about what languages and dialects are valued, what their bodies should look like physically, what counts as intelligence, and which emotions are valid.

Digital instructional materials allow for high quantities of easily shared content that has been encoded in multiple simultaneous modes (e.g., visual and auditory) to be transmitted to nearly limitless audiences with little oversight (Chu, 2000) . Because there is no governance ensuring that instructional materials will fairly represent all groups, teachers must ensure they are identifying and evaluating the materials for racist and biased messaging (Gallagher et al., 2019) . For example, many materials for sale on sites where teachers sell materials to other teachers presented content from racist perspectives (Archambault et al., 2021; Shelton et al., 2020) . Another recent study of online materials for English language instruction found gender bias, such as stereotypes, under-representation, and gender-exclusive language use (Gailea & Mulyana, 2020) .

When instructional materials promote exclusion, they are not quality materials, even if the materials are otherwise accessible and engaging for students. Teachers must actively review and choose materials that support valuable opportunities for challenging deficit views of social groups and promoting community in the classroom and beyond. Teachers should not assume students will automatically make connections in their thinking that will provide the opportunity to notice and denounce social injustices on their own. Instead, an ethos of inclusion is built by presenting multiple perspectives on events, providing multiple opportunities for student response, and then supporting students in examining and dismissing perspectives using various approaches to reasoning (Gallagher et al., 2019) . Below is an example of how teachers might confront online educational texts that are not inclusive.

Harper teaches first grade using a blended learning format where the students are divided in half, and they alternate weeks in coming to school in person and staying home to learn online. Harper's school district serves over 8,000 students from Native American groups. These students in the district represent over 100 indigenous communities from all over the world, but many students and their families claim membership in one or more of the 20+ local groups. Harper wanted to represent Native American peoples in her teaching and make strong school-to-home connections. Therefore, Harper searched for materials about Native Americans to use with students when they were learning online at home. Most of the materials she found featured images of native peoples who were depicted as children with feathers in their hair. Harper also found a few sites about Native American storytelling. Harper immediately dismissed the feather images as misrepresentative but decided to use some of the Native American stories about animals and pair them with a science unit about animals. In continuing to search, Harper found information from a local indigenous people's cultural center and learned that many of the groups in her community only engaged in storytelling during the winter months. To do so outside of that time of the year, even in school, was considered problematic and disrespectful by many families. Further, Harper learned that some of her students claimed membership in groups that had restrictive expectations for storytelling about certain animals, such as spiders and snakes. Considering this information, Harper decided to use the digital materials provided by the local cultural center rather than write new ones or buy them from a site that was not sponsored or endorsed by native peoples. Harper also decided to search for additional materials that were developed by, or in collaboration with, members of these groups.

The example above illustrates the complexity involved in finding and evaluating digital instructional materials and the need for collaboration with local communities. Sometimes teachers might wish that there was a concise list of Dos and Don'ts somewhere that they could memorize or have on hand. Other teachers might become exasperated by what they perceive as too many rules. Instead of an annoyance, prioritizing inclusion in selecting digital instructional materials should be conceptualized as a shared responsibility and part of an on-going effort to align educational practices with community ethics.

Definition: Digital instructional materials demonstrate accountability when they are transparent about their origins and purposes, based on standards or principles, and are open about personal information and user data collection sharing processes.

Teachers should seek digital instructional materials designed to foster a learning experience that aligns with learning goals and objectives (Vargo, et. al. 2003) . Explicit alignment with goals is an important element of digital assessment practice (Hodges & Barbour, 2021) . Standards alignment is common in other digital materials evaluation frameworks (California State University Center for Open and Distributed Learning, 1997; Chu, 2000; Jung et al., 2016; Nesbit et al., 2007; Vargo et al., 2003) . In the 4A Framework, accountability is more than ensuring that students are accountable for learning standards and teachers are accountable for teaching the standards. Digital materials must also be held accountable to the students and families for what is done with the information gathered during the learning process.

Instructional materials should reveal their goals and values to users (Beienberg, 2020) . For example, the Learning for Justice organization (2021) created instructional materials for both online and in-person learning. It was easy for teachers to ascertain the organization's values and goals because there is a page on their website titled "About Learning for Justice," where the group clearly reveals their connection to the Southern Poverty Law center. They do not claim they are politically or socially neutral in their aims. Instead, they are transparent about their alliances.

Instructional materials that connect to local, state, or national standards should describe how they are aligned, instead of merely stating that they are (National Academies of Science, Engineering, and Medicine, 2018) . Not all materials have to be aligned to state standards to be considered high quality, and digital assessment is much more than standardized testing (Rice, 2021) . Some digital instructional materials might have been created to support non-content goals, such as social/emotional development. Other materials might operate from a position that content standards represent a baseline of knowledge only, and students should be given the opportunity to go beyond the standards (Noddings, 2013) . Such extensions might be considered important advantages in online learning settings so long as these are explained to students by the developer, the teacher, or hopefully both.

Digital instructional materials should reveal what personal information and user data they are collecting (Boninger et al., 2017) . When students create accounts, they give their personal information to a company who collects data about them (Attai, 2020) . Collecting some data might be important for vendors to support and maintain the application. However, sometimes account agreements allow vendors access to search histories and other information that is outside of the original application and not important to the functioning of the program or its materials.

Educators, including teachers and administrators, need to carefully consider what data they are asking young people and their families to give to companies when they require registration for accounts as part of a class. Sometimes, teachers might get excited about new applications and send their students links to create accounts before they have considered how to use these materials (Beienberg, 2020) . Students might end up with many user accounts for materials that the teacher never uses or only uses a few times. For a student with a disability or who has trouble remembering and keeping track of passwords, this can add frustration. It also creates a need for an adult to assist in the storage and retrieval of all these passwords and makes data more susceptible to misuse (Lazar et al., 2017) . While single sign-on options alleviate the problem of remembering passwords, they also increase the need for vendors to share what data they are gathering within and across sites and what they plan to do with the data they collect (Pangrazio & Cardozo-Gaibisso, 2021; Pötzsch, 2021) .

Vendors might also give schools access to students' personal information beyond what the school already collects in addition to user data about how the students perform. In such cases, schools should also be transparent about how they are using student data to distribute opportunities, contribute to databases, make decisions about resources, and so forth. Some district user agreements explain the rules for students and families, but many might not yet include reciprocal protections where the district promises to protect personal information and user data. Policies at local and state levels regarding the terms schools can enter with vendors to share students' data is emerging but incomplete (Krueger, 2014; Reidenberg & Schaub, 2018; (Unauthorized Release of Personally Identifiable Information, 2015) . The goal should be to give students and their decision-makers as much information as possible and allow them to make decisions about their personal information and data. In the meantime, schools and families need to navigate accountability for data privacy and related protections together as best they can (Reidenberg & Schaub, 2018) . Below is an example of a teacher working to address accountability challenges in online instructional materials.

Jan needed to adopt a math textbook for students in a distance learning school that pairs physical textbooks with some online instruction. The school where Jan worked offered three choices. The first book had a seal on the cover that said it was aligned to state standards recommendations, but it had no description of how the instruction was linked to the standards. The second book offered worked examples for students online through the company website, but the school needed to pay a licensing fee for students, or families could pay separately. The publisher of the third book was also offering worked examples and other activities online; some of these online resources were free and did not require registration, while others did require registration and did charge a fee. When Jan contacted the publishers of choices two and three about data collection practices, the publisher of choice two said that privacy information was not available, but assured Jan that they only used the data they needed. The publisher of the third choice sent a disclosure about what data was collected, how it was used for program improvements, and how it was stored. The disclosure also indicated that both the free and the fee-based materials collected some user information, but the fee-based materials gathered data that could be shared with students about their progress and offered students more options to opt-out of some data collection features. Jan also saw that the book had been endorsed by American Mathematical Society for its strong documentation of standards alignment. Jan decided to recommend that the school principal purchase the third book and budget enough money to pay the fee to register students, so they could have access to and make more decisions about their data.

The example above demonstrates the importance of mutual accountability in evaluating digital instructional materials. While it is important to advocate for open materials, it is problematic when schools choose free materials from vendors that gather students' personal information and user data, use it, and then sell it, while denying students access to and control over the data. Students lose potential benefits from working with digital standards-based instructional materials when the materials do not provide useful information to them and when the students' data as a commodity takes primacy over the learning experience.

The 4A Framework enables decision-makers to evaluate individual items intended for use in instruction (e.g., a single video or website) and make informed decisions about using the material as it is, adding additional support, or modifying the materials. Educators could also use the 4A Framework in groups to make decisions about what instructional material to use with classes, small groups, and individuals. While some users of the 4A Framework might see how these elements can apply to instructional materials that are not online, it is important to remember that the tool was developed using research and policies that considered the affordances of digital materials in K-12 settings across a variety of digital settings (e.g., fully online, remote, blended). In fact, existing conceptual overlaps may make the tool highly suitable for learning spaces where both traditional and online materials are used.

Teachers deserve access to instructional materials that they can use with students with confidence (Kim et al., 2017) . The 4A Framework for evaluating digital instructional materials provides teachers and those who support them with important guidance for making decisions about teaching and learning (Mayer, 2019) . Even after teachers choose materials, they must continue to be vigilant. Informing teachers' judgement about major concerns can help them make decisions for their students (Chu, 2000) . Of course, more research is needed that focuses on supporting teachers and other decision makers in evaluating learning materials for children with different needs, strengths, and interests. Specifically, more research is needed to investigate how teachers use the 4A Framework and whether and how it improves their ability to evaluate instructional materials. Even so, this framework adds a needed update and consolidation of ideas about quality online instructional materials in K-12 settings.

rhhlt erm= histo ry#t= MERLOT_ Peer_ Review_ Infor mation. htm& rhsea rch= histo ry

Alternative approaches to assessing student engagement rates

Effects of different video lecture types on sustained attention, emotion, cognitive load, and learning performance

A cognitive load approach to metaphorical interface design: Reconsidering theoretical frameworks

Exploring the characteristics of an optimal design of digital materials for concept learning in mathematics: Multimedia learning and variation theory

Trends in the design, development, and use of digital curriculum materials

Hear, and speak no evil: A content approach to evaluating multicultural multimedia materials

Learning to serve students with disabilities online: Teachers' perspectives

Assistive technology provision: Towards an international framework for assuring availability and accessibility of affordable highquality assistive technology

Retrieved from www. everg reene dgroup. com/ keepi ng-pace-repor ts Education Week (2020). Map: Coronavirus and school closures

Rethinking assistive technology. Special Education Technology Practice

Explain the factors of primary school teachers choosing digitized teaching materials with theory of planned behavior

Minimization of Boolean complexity in human concept learning

Student engagement: What is it? Why does it matter?

School engagement: Potential of the concept, state of the evidence

Gender equality in online English book in reading and writing materials

Yet another (Mis)representation of disability: A critical content analysis of blindness/vision impairment in young adult literature

Hidden curriculum as one of current issue of curriculum

American Psychological Association (APA), & National Council on Measurement in Education (NCME)

K-12 distance educators at work: Who's teaching online across the United States

Cyber-truancy: Addressing issues of attendance in the digital age

Teacher-buyers beware and vet with care: Searching for expanded perspectives on online educational marketplaces

Student data privacy: Managing vendor relationships

Accessibility considerations for hybrid courses

De-escalating the curriculum wars: A proposal for academic transparency in K-12 education. Goldwater Institute

Reflections on the development of African American children's literature

The online teaching survival guide: Simple and practical pedagogical tips

Asleep at the switch: Schoolhouse commercialism, student privacy, and the failure of policymaking--The Nineteenth Annual Report on Schoolhouse Commercializing Trends

Communities of Engagement: an expansive lens for examining support structures in blended and online learning

MERLOT information and peer-review process

Pinning" with pause: Supporting teachers' critical consumption on sites of curriculum sharing

Applying principles of text complexity to online learning environments

Assessing learning during emergency remote education

Teacherstudent support, effortful engagement, and achievement: A 3-year longitudinal study

Do you live in a teepee? Do you have running water?" The harrowing experiences of American Indians in North Carolina's urban K-12 schools

Information Technology Industry Council

Framework for assessing fitness for purpose in open educational resources

Digital revolution and its impact on education systems in developing countries

Building teacher competency for digital content evaluation. Teaching and Teacher Education

10 Terms you must include in contracts with online service providers: A responsible approach to student data privacy requires careful scrutiny of the agreements that schools make with vendors

Indian educators' perceptions of their preparation and resources to support inclusion in secondary schools

Time to engage: Implementing math and literacy blended learning routines in an Indian elementary classroom. Education and Information Technologies

New conceptual frameworks for student engagement research, policy, and practice

Information privacy and security as a human right for people with disabilities

About learning for justice

Digital participation, agency, and choice: An African American youth's digital storytelling about Minecraft

Interrelations of behavioral, emotional, and cognitive school engagement in high school students

Visual storytelling on Instagram: Branded photo narrative and the role of telepresence

Whiteness and economic advantage in digital schooling: Diversity patterns and equity considerations for K-12 online charter schools. Education Policy Analysis Archives

The hidden curriculum in higher education

Global EdTech and smart classroom market (2020 to 2025) -Increasing spending on the education sector presents opportunities. Business Insider

Bichronous online learning: Blending asynchronous and synchronous online learning. Educause Review

Thirty years of research on online learning

Recognizing and changing students' misconceptions: Instructional perspective

Design, selection, and implementation of instructional materials for the next generation science standards: Proceedings of a workshop

Learning object review instrument (LORI), Version 1.5. E-Learning Research and Assessment (eLearn) and the Portal for Online Objects in Learning

Education and democracy in the 21st century

Your data can go to anyone": The challenge of developing critical literacies in children

Bringing materiality into thinking about digital literacy: Theories and practices of critical digital education in a digital age

A self-determination theory perspective on student engagement

Total engagement: How games and virtual worlds are changing the way people work and businesses compete

Critical digital literacies: Boundary-crossing practices. Critical literacy teaching series: Challenging authors and genres

Supporting literacy with accessibility: Virtual school course designers' accessibility planning for students with disabilities

Analyzing text cohesion in online learning environments: Implications for students with reading difficulties

Too many words, too little support: Vocabulary instruction in online earth science courses, International Journal of Web-Based Learning and Teaching Technologies

A case study of a foster parent working to support a child with multiple disabilities in a virtual school

What are critical features of science curriculum materials that impact student and teacher outcomes?

A practical reader in universal design for learning

iNACOL-Access-and-Equityfor-All-Learn ers-in-Blend ed-and-Online-Educa tion-Oct20 14

How teacher emotional support motivates students: The mediating roles of perceived peer relatedness, autonomy support, and competence. Learning and Instruction

Cultural content of three EFL textbooks: teachers' and learners' cultural preferences and cultural themes of textbooks

The possible influence of curriculum statements and textbooks on misconceptions: The case of evolution. Education as Change

Mis-/Under-) Representations of gender and sexuality in locally-produced ELT materials

Curriculum online? Exploring the political and commercial construction of the UK digital learning marketplace

Hypermedia design as learner scaffolding

Lesson platforms for teachers have a racism problem

Computer game-based mathematics education: Embedded faded worked examples facilitate knowledge acquisition. Learning and Instruction

Future ready learning: Reimagining the role of technology in education: 2016 National Technology Education Plan

Learning object evaluation: Computer-mediated collaboration and interrater reliability

The GIST of concepts

Iterative user and expert feedback in the design of an educational virtual reality biology game

Rapid and rigorous qualitative data analysis: The "RADaR" technique for applied research

Instructional strategies to help online students learn: Feedback from online students

Systems limitations hamper integration of accessible information technology in northwest US K-12 schools

World Wide Web Consortium (2020)

The optimal learning strategy depends on learning goals and processes: Retrieval practice versus worked examples

Self-regulated learning and academic achievement: theoretical perspectives

The authors declare that there are no conflicts of interest in this work.

There were no human participants in this research, therefore it was exempt from Institutional Board Review. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Since there were no human/animal participants in this research, no informed consent was needed for this research.