URLs in this document have been updated. Links enclosed in {curly brackets} have been changed. If a replacement link was located, the new URL was added and the link is active; if a new site could not be identified, the broken link was removed.

Informing Food Protection Education: A Project to Define and Classify Resources for a Cross-Disciplinary Expert Community

Donna Schenck-Hamlin
K-State Libraries
Kansas State University
Manhattan, Kansas
dschenckhamlin@gmail.com

Jennifer Pierquet
School of Public Health
Minneapolis, Minnesota
jpierquet@foodshield.org

Chuck McClellan
MLIS Program
St. Catherine University
St. Paul, Minnesota
chucktmcclellan@gmail.com

Copyright 2011, Donna Schenck-Hamlin, Jennifer Pierquet, and Chuck McClellan. Used with permission.

---

Abstract

In the wake of the September 2001 attacks, the U.S. government founded the Department of Homeland Security (DHS) with responsibility to develop a National Infrastructure Protection Plan for securing critical infrastructures and key resources. DHS established interdisciplinary networks of academic expertise administered through Centers of Excellence across the country, each addressing a different aspect of homeland defense.

The National Center for Food Protection and Defense (NCFPD), created in 2004, organized over 150 experts around five theme groups: Agent Behavior, Event Modeling, Systems Strategies, Risk Communication, and Educational Programs. The fifth theme group, Educational Programs, was the focus of a special library project to assist experts in delivering "high-quality education and training programs to develop a cadre of professionals equipped to deal with future threats to the food system." An initial database provided National Center for Food Protection and Defense experts with a listing of educational resources and their uniform resource locators (URLs).

In 2009, NCFPD formed a team of librarians and subject experts to finalize the framework for a working collection of food protection educational resources. The framework consisted of 1) a scope of work for the database, 2) descriptive metadata elements and classification decisions, and 3) a vetting instrument to maintain the utility and currency of selected information. This paper describes the work performed by the database team to create criteria for inclusion and classification of resources consistent with NCFPD end-users' evolving needs. A content validity method is described that was used to gauge whether the team's conceptual definition of the domain of information was congruent with that of the expert community being served.

Introduction and Background

Librarians perform an important task for expert communities: that of extracting, aggregating, and classifying knowledge resources for specific end uses. Special librarianship has long been dedicated to providing information demanded by expert communities to fulfill their special functions or missions, but addressing cross-disciplinary communities' information-sharing needs has recently become more of a priority during a decade marked by national crisis response.

This paper describes one project to assist a national, cross-disciplinary and cross-sector community of experts in food protection. It details key decisions and a content validity method that librarians, collaborating with subject specialists, undertook to address a critical information need. The project illustrates how librarians can test their consistent application of classification criteria, along with the end-users' understanding of that criteria. It underscores the necessity of frequent interaction with end-users as partners, in order to articulate the problems and goals to which information is applied.

2001 is remembered for the September attack on U.S. targets that exposed security vulnerabilities in what had been considered, for all its complexity, a remarkably safe transportation system. Subsequently, the U.S. government founded the Department of Homeland Security (DHS) with responsibility to develop a National Infrastructure Protection Plan (NIPP) for securing critical infrastructures and key resources (CIKR). NIPP established 18 CIKR sectors for protection, of which the Agriculture and Food sector ranked high, not only for its prominence in the national economy (accounting for one-fifth of all U.S. economic activity), but also for its interdependence with other sectors, such as Water, Transportation, Health care and Public Health (U.S. Department of Homeland Security 2011a).

Recognizing the role of universities in national research and education, the Department of Homeland Security encouraged interdisciplinary networks of academic experts administered through Centers of Excellence across the country, each addressing a different aspect of homeland defense (U.S. Department of Homeland Security 2011b). One of them, the National Center for Food Protection and Defense, created in 2004, organized over 150 national experts around five theme groups: Agent Behavior, Event Modeling, Systems Strategies, Risk Communication, and Educational Programs (National Center for Food Protection and Defense 2011b).

The fifth theme group, Educational Programs, was the focus of a special library project to assist experts in delivering "high-quality education and training programs" and "to develop a cadre of professionals equipped to deal with future threats to the food system" (National Center for Food Protection and Defense 2011a). The information required by Educational Programs included topics covered by the other four groups, but in the form of curricular resources (course offerings and course materials). A working collection or database of these curricular resources was planned to serve postgraduate or advanced professional educators as they developed and delivered cross-disciplinary education in food protection.

In 2006, as an initial task, the Educational Programs group began an inventory of curricular resources in higher education that contribute to advanced knowledge in food protection. The group consisted of information and communication experts as well as scientists and other specialists. Three challenges were evident from the outset. 1) Consensus across disciplines was needed for concepts, terminology, and competencies in food-protection education. 2) In university classes, often only parts of a given course addressed themes emphasized by the National Center (e.g., agent behavior, event modeling). 3) U.S. university programs were just beginning to bridge disciplinary boundaries to formulate certificates and degrees focused on food protection as conceived in the CIKR framework.

While one task force worked at defining educational competencies for curricula, another task force from Michigan State University (MSU) surveyed diverse public, private, government, and academic web sites and databases for educational resources and began identifying end-user groups needing specific levels of education and training. An initial Food Protection Education database developed at MSU provided NCFPD experts with an inventory of the collected resources. In 2008, a review of those resources focused on controlled vocabulary for descriptors. In 2009, NCFPD formed a database team to finalize the framework for a working collection of food-protection educational resources. The framework consisted of 1) a scope of work for the collection/database, 2) decisions on metadata elements and subject descriptors, and 3) a vetting instrument to maintain the utility and currency of the educational information. The following sections describe how a team of library and subject specialists collaborated to accomplish each of these components.

Defining the Scope of Work for the Collection/Database

The challenge of systematically collecting curricular resources in food protection is a subset of a larger challenge identified in the report "Harnessing Knowledge to Ensure Food Safety: Opportunities to Improve the Nation's Food Safety Information Infrastructure" (Taylor and Batz 2008). The authors described the food-safety information infrastructure as a vast, decentralized array of "public and private institutions, programs and processes through which data and other information are collected, made accessible and actively shared." They outlined the complexity of this system in terms of diverse topics, sources, information/data types, tools, institutions, and roles. At the federal governmental level alone, national leaders in food safety from the Food and Drug Administration (FDA), the U.S. Department of Agriculture (USDA), Environmental Protection Agency (EPA), Centers for Disease Control and Protection (CDC), and Department of Homeland Security address topics as diverse as surveillance, hazard analysis, risk communication, decontamination, and recall strategies.

What Is Food Protection?

A primary task critical to establishing the scope of work was the definition of "food protection" itself, as distinct from, but related to food safety, food defense, and food security concepts. Food safety information has long been developed and disseminated by government, industry, and non-profit institutions to monitor and mitigate damage from natural hazards, errors, or failures in the food system. Most educational resources on food safety are clustered around training in Hazard Analysis Critical Control Point management (HACCP ) or consumer education, where prevention, detection, and response occurs at a local level.

Food defense involves a larger-scale response to intentional damage to the food system, whether economically or politically motivated; i.e., an "attack" having potential implications beyond local control. Most educational resources in food defense are aimed, therefore, at professionals having a broader scope of food system surveillance.

Some government publications after the 9/11 attacks used the term "food security" as synonymous with "food defense," though in agricultural economic literature "food security" has long designated a nation's ability to provide sufficient access to food for its population's essential nutritional needs. (Food and Agriculture Organization of the United Nations, and the U.S. Congress (U.S. Agency for International Development 1992).

Food protection encompasses both food defense and food safety, with professional interests converging from multiple disciplines and sectors. Food Protection covers the entire array of food safety and food defense information serving awareness, prevention, response, and recovery. A brief listing of key contributing disciplines and sectors is given below, followed by a list of some of the sources of food-protection education (Tables 1 and 2).

Table 1: Key disciplines and sectors contributing to Food Protection Knowledge

Food Safety professional arenas	Food Defense professional arenas
Environmental health Epidemiology Food science Public health Veterinary science Food production Food retail Food service Food manufacturing	Border security Import regulation Bioterrorism Vulnerability assessment Risk analysis Risk communication Supply chain management Emergency management Crisis planning

Table 2: Sample of Institutions Providing Educational Content

Government	Universities	Other
U.S. Environmental Protection Agency	University of Minnesota, School of Public Health	American Veterinary Medical Association
U.S. Department of Homeland Security	Michigan State University, Food Science and Human Nutrition	Rand Corporation
U.S. Federal Emergency Management Agency	San Diego State University	American Industrial Hygiene Association
U.S. Department of Health and Human Services	University of Purdue Extension	Extension Disaster Education Network
United States Department of Agriculture	University of Eastern Kentucky	Maryland Emergency Management Agency
U.S. Centers for Disease Control and Prevention	University of Georgia	American Institute of Baking (AIB International)
U.S. Food and Drug Administration	Louisiana State University, National Center for Biomedical Research and Training	World Health Organization
U.S. Government Accounting Office	University of Southern California, Viterbi School of Engineering	Association of Food and Drug Officials

Who Is the End-User Population for Food Protection Education?

For any issue that a federal agency is required to address, a group of experts must develop, employ, and transfer that topical knowledge to acting professionals. Professionals in the field could range from a food inspector with minimally a Bachelor's Degree to a public health official with a Ph.D. Since advanced formal education beyond a bachelor's degree inevitably involves research, the definition of "educational resources" beyond textbooks and training manuals, including higher level research publications, was likely to require much discussion and refinement as the project proceeded. For example, professional association conference proceedings could include continuing education workshops to update experts with new knowledge. Short of considering any informative offering an "educational" resource, a definition was required based on some form of curriculum. The primary users were assumed to be those developing parts of an evolving food protection curriculum for the end-user professionals.

What Is a Curricular Resource?

For this project, a curricular resource is defined as one of two types: a course or a document. Providers of advanced education and training consult other courses as they develop their own offerings or refer others for continuing education. Educators need to compare current course offerings, including online or classroom; credit or non-credit; procedural, theoretical, or experimental settings.

Sample Course

Type:	Course - Module
Title:	Protecting Our Food System from Intentional Attack: Hay Bot Tox Exercise
Creator:	Shutske, John
Publisher:	University of Minnesota, School of Public Health

Educators also consult reading materials used in food protection coursework, so these were included as well.

Sample Document

Type:	Document - Book, Section
Title:	Biosecurity: Food Protection and Defense
Source:	Food Microbiology: Fundamentals and Frontiers
Creator:	Kennedy, Shaun P.; Busta, Frank F.
Contributor:	Doyle, Michael P.
Publisher:	American Society for Microbiology

Given the annual variable schedule of course offerings, it is anticipated that a yearly review will occur, and with it the opportunity to archive, or separate inactive courses from active ones.

Database, Collection, or Repository?

Today many information resources are born digital and reside in various devices or platforms. When this project began, the majority of online information resources were identified by a URL, although there was no guarantee that the resource creators or Internet location managers would permanently maintain the resources there. In the last two years, increasing use of digital object identifiers (DOIs) assigned to information resources is helping to establish efficient, unique, and permanent, "persistent" identification for them (Paskin 2011). Since there is rarely an indefinite commitment to store an information resource, the DOI and the object's metadata, if not the object itself, is the best guarantor of sustainability.

Enterprise inventories may evolve over time into more complex, searchable databases. Databases, when populated with records' full-text objects, may evolve into digital repositories. Leaders in business and government dealing with rapid changes in the food-protection agenda, and resulting effects on information resources, may hesitate in committing to a large or complex information system, particularly if its data originates from other databases and repositories.

This project team aimed at the middle ground of a "working collection" as an achievable and necessary commitment to the demand for food-protection curricular resources. Working collections consist of resources employed through the life of a project which may cease to be housed physically together beyond its completion, but continue as a database for future reference. The "working collection" for this project consists of a database in Microsoft Access, with resources cited by DOI and/or URL. Where copyright permits, a collection of full-text resources has been archived as backup files in order to avoid the "404 Not Found" error code when a persistent DOI is not available.

Decisions about the database platform have yet to be finalized as of this writing. Early communication with project participants who have a FDA-funded project with parallel goals (International Food Protection Training Institute 2011) permitted the team to swap records and begin collaborating in the spirit of Taylor and Batz's proposed "network of food safety networks." The database may eventually reside on the information gateway FoodSHIELD, with links to the proposed food-safety information infrastructure "hub" of collection activities and information resources (Taylor and Batz 2008). Collection- and/or article-level metadata may also link these food-protection education records to similar collections developed with other, related DHS Centers of Excellence. To optimize such collaboration, the project team addressed record-level metadata, deciding how to define the data in a way that would enable future exchange and collaboration across platforms.

Establishing Metadata Elements and Descriptors

Choosing a Metadata Standard

Dublin Core Metadata Initiative (DCMI) is "an open organization engaged in the development of interoperable metadata standards that support a broad range of purposes and business models" (Dublin Core Metadata Initiative 2011a). Dublin Core standards were chosen for cataloging entries in the food-protection education database for several reasons. First, the 15 Dublin Core (DC) descriptive elements (Dublin Core Metadata Initiative 2011b) offer a generalized model that could be used for both curricular events and curricular materials in the same database (Table 3). The 15 elements are easy to learn and to apply by both librarians and trained staff.

Table 3: 15 Dublin Core Elements Illustrated by a Sample Course Record

Dublin Core Element	Example
DC.title	Surveillance of Zoonotic Pathogens in Animals
DC.type	Class, graduate
DC.date	May 2010
DC.creator	Wells, Scott
DC.source	Public Health Certificate in Food Protection
DC.subject	Hazards Analysis; Research/Investigation
DC.rights	/
DC.format	Classroom
DC.relation	{https://web.archive.org/web/20100710044357/http://www.sph.umn.edu/ce/institute/docs/syllabi/2010/7200-113Wells.pdf}
DC.publisher	University of Minnesota, School of Public Health
DC.language	English
DC.identifier	PubH 7200-113 Class #89519
DC.description	/
DC.contributor	/
DC.coverage	/

But there were even more important reasons to employ Dublin Core standards for the long-term utility of the database. Taylor and Batz (2008) pointed to differences in how food-safety data is collected and stored as a factor inhibiting the sharing of information among stakeholders, and suggested standardizing collection and storage methods to enhance data-sharing within this community. The authors also acknowledged significant impediments to standardization between diverse federal databases. In answering this challenge, Dublin Core elements can be mapped to or from other metadata models, thus demonstrating interoperability essential to collaborative efforts with other institutional databases. But aside from "cross-walking" or converting records from one set of definitions to another, another advantage to Dublin Core exists through the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH).

OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting 2011) is a widely recognized protocol for record exchange among open-access repositories, employing Dublin Core as the metadata standard of choice. This type of common denominator allows for a wide range of repositories to expose information about their records' contents to a wide array of metadata harvesters, including Google. The protocol enables repositories to be "data providers" (those that expose their DC-structured metadata) and/or "service providers" (those that make service requests to harvest the metadata). A syntax of verbs for request and response between repositories has been developed that permit exchange of metadata sets among those "OAI-compliant" (i.e., following the protocol). The list of over 1,000 worldwide OAI-compliant repositories is growing, and includes university as well as government repositories (OCLC 2011).

Discussions regarding the metadata model to use began at Michigan State University's National Food Safety and Toxicology Center in consultation with OAIster database managers. A July 2007 two-week work session addressed working definitions of crucial metadata elements through repeated inter-coder reliability tests among three participating institutions: a DHS visiting scholar at the National Food Safety and Toxicology Center; a librarian from Kansas State University; and a research assistant in Public Health, MSU. The three examined collected food protection education documents, discussed them together, and independently classified them with descriptive metadata elements in the following categories: 1) audience, 2) genre, and 3) purpose (see below).

During the two-week session, statistical tests were conducted of the coders' independent classification of documents using Cohen's kappa as measure of inter-coder reliability (Lombard 2010). Cohen's kappa is a conservatively high standard for agreement among coders, because it factors in the possibility of randomness when one chooses to select or not select among numerous classification terms. The team hoped to achieve .6 kappa scores, considered moderate to good for agreement, but with each new test, new titles and genres were introduced to the coders that added challenges to their understanding of the code definitions. Although .6 kappa scores were not achieved by the end of two weeks, the group arrived at refinement and adjustment of language for the 3 metadata elements, and coders found the process of classifying easier and faster by the end of the session. At the conclusion, they recommended descriptors for DC.subject and DC.audience (not one of the 15 "Core" DC elements, but accepted subsequently as "Qualified" DC) (Dublin Core Metadata Initiative 2011c). Later, these descriptors were re-examined and refined again in 2009 when responsibility for the database was transferred from Michigan State University to the University of Minnesota's School of Public Health.

Developing DC.subject and DC.audience Descriptors for Food Protection Education

"Subject" is recognized in library science as a key access point in database searching, and is also the point at which controlled vocabulary, especially for cross-disciplinary collections, can help or hinder the end-user. Given that the goal of the collection was to facilitate advanced education for professionals responding to potential food-protection emergencies, the concept of "subject," rather than being discipline-centered, was reframed as two overarching categories: stage [of protection] and purpose [of educational resource].

Tables 4 and 5 present the recommended vocabulary developed from 2009 to 2010 to articulate these subject parameters for a given educational resource. The descriptors were tested for inter-coder reliability over a one-week work session with the authors and a second library graduate student from University of Wisconsin-Milwaukee. With approval from the database project' principal investigators, the descriptors were employed subsequently in cataloging records of the Food Protection Education database.

Table 4: DC.Subject as Stages in the Life-cycle of Emergency Preparedness

Descriptor	Watchwords for descriptor selection
Hazards Analysis	names of agents, symptoms, risk factors or diseases; HACCP; agent behavior; risk analysis, threat/vulnerability assessment
Prevention	practices; guidelines; methods; steps; roles/responsibilities; procedures; systems strategies; regulatory; facility or site security
Detection/Occurrence	incidence, prevalence; symptoms; outbreak, event, chronology
Emergency Management	response, first-responders, risk communication, SWAT, Event Modeling
Recovery	notification, recalls, decontamination; disposal; re-supply

Table 5: DC.Subject as Purpose -- End-use of an Educational Resource

Descriptor	Watchwords for descriptor selection
Public Awareness	fact sheet, FAQs, interview, news, poster, press/media release
Guidance/Procedural	after-action/enforcement report; best practices; plan; guidelines; recommendations; interventions; methods, standard operating procedures
Policy Development/Enactment	case study; policy directive; discussion paper; laws/legislation; lessons learned; report card; white paper; testimony; expert witness
Research/Investigation	abstract; case study; thesis/dissertation; clinical trials; inspections/surveillance; audit check; journal article; needs/cost/risk assessment; patent; evaluation; survey
Reference	bibliography; database; glossary; acronyms; index; directory

The Dublin Core Metadata Initiative introduced the concept of "qualified Dublin Core" in 2005 as a means of achieving additional specificity for information managers and end-users. Beyond the initial 15-element set, a few introduced elements included "Audience" (Dublin Core Metadata Initiative 2011d). The focus of food-protection education on public and private sector professionals in distinct categories necessitated use of such an element.

Table 6: DC.Audience -- primary end-user community for educational resource

Descriptor	Watchwords or genres for descriptor selection
Agri-Food Supply Chain	[food/feed] producers/production; process*, handl*, retail/wholesale, food service, restaurant, supply chain, etc.
Consumer Education, Public Interest Groups	"how-to", FAQs, consumer, public, etc.
Government, Law Enforcement	policy, legislation, white paper, regulation, etc.
Information, Communications	risk communication, press/media release, etc.
Health & Medicine	disease*, symptom*, illness*, epidemiolog*, veterinar*, etc.
* denotes truncation characters for words beginning with the character string

Defining and Testing the Vetting Instrument

As often happens in multi-institutional collaborations, the database records previously collected at one institution required a review by the adopting institution for inclusion or deletion before adding new records or applying further metadata elements and descriptors. The Education Theme Group obtained a grant from NCFPD to construct a vetting instrument that would inform on-going resource selection and review.

Method

First, the librarian and two scientists drafted a decision-tree approach to ranking food protection education resources that could be easily learned by librarians or information managers without advanced degrees in the disciplines served by NCFPD. To assure a future, the database would be managed by college-educated but non-specialized personnel under the supervision of one subject expert. The library interns and scientists conducted a one-week study to learn and apply the vetting model, based on relevance, currency, and criteria for quality, sponsorship, and peer review. The rubric was derived, in part, from document-selection standards maintained by the ERIC Clearinghouse (Education Resources Information Center 2010).

Training non-experts to evaluate the relevance of a course or document to food-protection education began with examples from the database illustrating the wide array of disciplines and problems. Then use of the vetting rubric began by judging the resource's relevance to food protection using two criteria: 1) the contents' relation to a risk-management framework (risk assessment -- preparedness -- detection/occurrence -- response -- recovery) and 2) the contents' primary target audience (agri-food supply chain -- consumer public -- governance/law enforcement -- health & medicine -- information & communication).

Next, the content's relevance to education was assessed. For course resources, a standardized checklist of types of curricular offerings is sufficient to establish this. For document resources, a more subjective judgment is required, since the range of training to education relies on more genres than textbooks or procedure manuals. For a document to offer curricular utility, it should summarize; synthesize; test readers' understanding with questions or exercises; advance users to the next level of comprehension; or offer detailed instruction on process or procedure. Once a resource has met the criteria for relevance to food protection and to education, the vetting decisions are based on:

• currency: a course within three years; a document within ten years
• sponsorship: a course by accredited, governmental, or professional organization; a document by a peer-reviewed publication
• quality: verifiable author expertise; perceived objectivity and completeness of the educational object).

The scientist, librarian, and two library students worked as a group, in pairs, and separately, to practice evaluating courses and documents based on the criteria. During the same week, a survey was sent to NCFPD experts with six randomly selected courses and documents from the database of 496 resources, requesting that they examine the food-protection education criteria and use them to rate the resources as "essential," "useful," or "peripheral" to the collection. Testing the selection criteria by surveying members of the primary end-user community would capture a portrait of NCFPD members' agreement or disagreement on the concept of food protection itself. A conceptual definition with high agreement is important for managers of the shared information resources. To gauge agreement among NCFPD experts, a content validity test was employed using the sampled resources (Table 7).

Measuring Content Validity

The team employed a measurement of content validity known as Expert Panel Validity, developed by C. H. Lawshe (1975). It involves a process in which persons with expertise in the domain being tested are asked to use a conceptual definition to determine whether a set of sample resources should be rated as "essential," "useful," or "not necessary".

Employing this test would establish whether our conceptual definition is sufficiently coherent for experts to produce a statistically significant ranking of essential resources. A Content Validity Ratio (CVR) from -1 to +1 is calculated for each examined resource, reflecting experts' agreement on how essential it is:

"N" is the total number of experts doing the rating and "ne" is the number of experts rating the item as "essential". Any positive value indicates that over half of the experts rate the item essential. High CVR values would assure managers of the collection that a given resource has an agreed-upon potential value to surveyed end-users, while low CVR values would indicate a lack of agreement on its potential value.

The conceptual definition of the Food Protection Education domain was described as follows:

For a resource to qualify for the collection, it must be both:

1. directly applicable to food protection and
2. readily useable in a curricular context

If neither of these criteria can be met, a resource is considered "peripheral" to the collection. This term was substituted for Lawshe's "not necessary" by the researchers to indicate that a resource would not be discarded, but marginalized if it did not meet the criteria. The reason for doing this was to avoid the influence of bias if surveyed experts were actually creators of any of the randomly selected resources. The respondents were not told how their judgments would be used; they were only told that their opinion was sought on "their potential utility to a curriculum in food protection." To judge a resource as "essential" or "useful" would both be positive responses, and "peripheral" a negative response, rather than offering a middle category designating a neutral or undecided value between selection or omission.

If both criteria are met, a resource is either "useful" or "essential" depending on the extent that it is:

1. Verifiable
2. Current
3. Offered by reputable sources
4. Suited to education of end-user communities

After pre-testing the survey with on-site scientists at the University of Minnesota, it was determined that a mailed survey to the wider NCFPD population would generate more results if no more than six resources had to be evaluated by them. Three documents and three courses were selected randomly from the FPE database of 496 resources.

Table 7. Samples for the Content Validity Test

Identifier Date Type	Title, URL
362 2004 document	Hitting America's Soft Underbelly: The Potential Threat of Deliberate Biological Attacks Against the U.S. Agricultural and Food Industry http://rand.org/pubs/monographs/2004/RAND_MG135.pdf
307 11/07/2002 document	Food Emergency Preparedness & Response Planning {http://web.archive.org/web/20110316140042/http://www.mass.gov/Eeohhs2/docs/dph/environmental/foodsafety/food_safety_security_resources.pdf}
288 N.D. document	How to Talk to Kids about the Threat of Biological Warfare or Terrorist Attack {http://web.archive.org/web/20101009091516/http://www.fema.gov/kids/terrism.htm}
480 05/07/2009 course	Introduction to Incident Command System, ICS-100 {https://training.fema.gov/EMIWeb/IS/courseOverview.aspx?code=IS-100.b}
256 Fall 2003 course	Network & Computer Security {http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-857-network-and-computer-security-spring-2014/}
381 Fall 2009 course	Food Safety Risk Communication and Education {https://web.archive.org/web/20100528072729/https://www.ndsu.edu/ndsu/rr/schedule/fall/safe.pdf}

The Content Validity Ratio results for the selected titles are as follows (Table 8):

Table 8: CVR Results

Results

The survey was sent to 133 NCFPD member researchers, augmented by volunteers from the Institute of Food Technologists, totaling 153 solicited participants, 54 of whom responded. Since the CVR value is dependent on the number of expert raters, a minimally statistically significant CVR value will be highly dependent on the number of experts providing ratings. Lawshe offered a table of minimum CVR values demonstrating that for 40 experts, only a CVR value of .29 or greater is statistically significant, so .29 was used in this study as a benchmark.

Table 8 demonstrates that no positive CVR value was achieved for the six sampled resources, i.e., not even half of the respondents agreed to a ranking of "essential" for any resource; in two pre-tests two out of the six achieved a consensus of preference. Comparing the numeric differences between "useful" and "peripheral" with "useful" and "essential", all resources except #381 are shown to have a closer association of "useful" with "essential" resources. As might be expected, individual differences were fewer between two positive categories than between positive and negative categories.

The researchers then questioned whether a CVR based on de-selection of peripheral resources might approximate the process of vetting just as well as the preferential selection of essential ones. Substituting np (peripheral) for ne, (essential), the following rankings resulted (Table 9):

Table 9: CVR Results using np

Here a clear preference emerged for de-selecting two "peripheral" resources (288: "How to Talk to Kids about the Threat of Biological Warfare or Terrorist Attack" and "256: "Network & Computer Security"). The positive CVR values .37 and .22 could be employed to give the database team a de-selection benchmark, although originally a positive benchmark for inclusion had been sought. This approach to vetting was approved by NCFPD project leaders, who requested periodic reviews with new sample courses and documents by additional subject experts in food protection.

Conceptual disagreements among professionals in this cross-sector, cross-disciplinary arena had been uncovered before. An NCFPD project leader cited the CVR data to illustrate the need for continuous interaction among experts as the particular educational needs among sectors evolve in response to specific threats. The small sample size of documents should be enlarged, and a check of new acquisitions and classifications should be conducted using a wider pool of experts to improve the quality of information collecting.

Conclusions

Librarians and information specialists who aggregate cross-disciplinary information should examine their end-users' level of agreement on key concepts as they collect and classify resources. Expert panel validity offers a useful formula for testing experts' agreement, which can be used to benchmark collection and classification decisions. Even validity scores that show significant disagreement over key concepts may be instructive to managers of shared information resources. In the case of food protection education, as collaboration among specialists continues, repeated validity tests may show convergence on concepts of what constitutes an "essential" information resource. Inter-coder reliability is another useful measure that should be used more by librarians for training and periodic validation of classification schemas.

These tools, along with Dublin Core metadata, equipped a team to assist a national effort to extract and aggregate curricular resources to inform a new generation of experts in food protection. Due to annual changes in course offerings and food-protection priorities, a means of reviewing and vetting FPE records was supplied that resulted in a manageable, realistic collection for the intended end-users. The selection of Dublin Core metadata elements to describe the records enabled the database to be exposed through multiple search engines, enlarging its accessibility to a wide public. The records' subject terms addressing specific dimensions of audience, end-use, and the risk management spectrum offer the most pertinent information to professionals seeking or offering advanced education in FPE. On-going growth and maintenance of this collection, in concert with other networked food-safety information, is both a feasible and necessary contribution to the food-protection information infrastructure.

References

Dublin Core Metadata Initiative. 2011a. [Internet]. [Cited February 28, 2011]. Available from: http://dublincore.org/

Dublin Core Metadata Initiative. 2011b. Dublin Core Metadata Element Set, Version 1.1. [Internet]. [Cited February 28, 2011]. Available from: http://dublincore.org/documents/dces/

Dublin Core Metadata Initiative. 2011c. Using Dublin Core -- Dublin Core Qualifiers. Date Issued: 2005-11-07. [Internet]. [Cited February 28, 2011]. Available from: http://dublincore.org/documents/usageguide/qualifiers.shtml

Dublin Core Metadata Initiative. 2011d. 4.16 Audience. In Using Dublin Core -- The Elements. Date Issued: 2005-11-07. [Internet]. [Cited February 28, 2011]. Available from: http://dublincore.org/documents/usageguide/elements.shtml#audience

Education Resources Information Center. 2010. ERIC Selection Policy. Reformatted 1/22/2010. [Internet]. [Cited February 28, 2011]. Available from: http://www.eric.ed.gov/ERICWebPortal/resources/html/news/ERIC%20Selection%20Policy.pdf

International Food Protection Training Institute. 2011. [Internet]. [Cited February 28, 2011]. Available from: http://www.ifpti.org/

Lawshe, C.H. 1975. A quantitative approach to content validity. Personnel Psychology 28, 563-575.

Lombard, M. 2010. Intercoder Reliability. In Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects. Last updated: June 1, 2010. [Internet]. [Cited February 28, 2011]. Available from: http://astro.temple.edu/~lombard/reliability/

National Center for Food Protection and Defense. 2011a. Program Summary. [Internet]. [Cited February 28, 2011]. Available from: {http://www.ncfpd.umn.edu/about/overview/}

National Center for Food Protection and Defense. 2011b. Research and Education Teams. [Internet]. [Cited February 28, 2011]. Available from: {http://www.ncfpd.umn.edu/index.cfm/investigators/research/}

National Center for Food Protection and Defense. 2011c. What is Food Defense?. [Internet]. [Cited February 28, 2011]. Available from: {http://foodprotection.umn.edu/about/overview/what-food-protection}

OCLC. 2011. OAIster. [Internet]. [Cited February 28, 2011]. Available from: http://www.oclc.org/support/documentation/firstsearch/databases/dbdetails/details/OAIster.htm

Open Archives Initiative Protocol for Metadata Harvesting. 2011. [Internet]. [Cited February 28, 2011]. Available from: http://www.openarchives.org/pmh/

Paskin, N. 2011. Digital Object Identifier (DOIŽ) System. [Internet]. [Cited February 28, 2011]. Available from: http://www.doi.org/overview/DOI_article_ELIS3.pdf

Taylor, M.R. and Batz, M.B. 2008. Harnessing Knowledge to Ensure Food Safety; Opportunities to Improve the Nation's Food Safety Information Infrastructure. FSRC Report 08-01. Gainesville, FL: Food Safety Research Consortium; Washington DC: George Washington University, School of Public Health and Health Services. ]. [Cited February 28, 2011]. Available from: {http://hsrc.himmelfarb.gwu.edu/sphhs_policy_facpubs/477/}

U.S. Agency for International Development. 1992. USAID Policy Determination; Definition of Food Security. PD-19. April 13, 1992. [Internet]. [Cited February 28, 2011]. Available from: http://www.usaid.gov/policy/ads/200/pd19.pdf

U.S. Department of Homeland Security. 2011a. Critical Infrastructure and Key Resources (CIKR); Understanding the Agriculture and Food Sector. [Internet]. [Cited February 28, 2011]. Available from: {https://www.fema.gov/pdf/emergency/nrf/nrf-support-cikr.pdf}

U.S. Department of Homeland Security. 2011b. Homeland Security Centers of Excellence. [Internet]. [Cited February 28, 2011]. Available from: http://www.dhs.gov/files/programs/editorial_0498.shtm

Previous

Contents

Next