510 Assessing Library Scholarship: Experience at a Land Grant University Andrea A. Wirth, Maureen Kelly, and Janet Webster Andrea A. Wirth is Geosciences & Environmental Sciences Librarian in the Valley Library at Oregon State University; e-mail: andrea.wirth@oregonstate.edu. Maureen Kelly is Head Librarian & Extended Campus Librarian in Cascades Campus Library at Oregon State University; e-mail: maureen.kelly@oregonstate. edu. Janet Webster is Librarian in Guin Library at Hatfield Marine Science Center; e-mail: janet.webster@ oregonstate.edu. © Andrea A. Wirth, Maureen Kelly, and Janet Webster Library faculty are often tenure track, requiring a record of significant publishing and service before achieving tenure. However, meaningful tools and methods for evaluating the scholarship of librarianship fre- quently fall short of providing an accurate picture of the scholarship of any particular candidate. The authors conducted a case study analyzing the research output of Oregon State University (OSU) Libraries’ faculty using the Boyer Classifications1 and Blake’s research methodologies.2 Broadening our view of acceptable formats and outlets and learning how to communicate and assess our scholarship within the academy are key issues that require continued exploration. ibrary faculty are often tenure track, requiring a record of significant publishing and service before achieving ten- ure. In land grant universities, 42 percent of librarians are on tenure track in profes- sional ranks and reviewed for promotion and tenure along with other faculty in those institutions.3 This percentage in- creases in those land grant universities granting doctorates to 68 percent, com- pared to 50 percent in the general faculty librarian population.4 At times, convincing university tenure review committees of the value of library scholarship presents challenges. Hill makes a convincing argument for faculty status but understands that the review process has not been adapted to our needs or approaches.5 St. Clair and Miller believe academic librarians need to better articulate the value of our scholarship and “rais[e] the profile of librarians within the academic community.”6 This is evident at the university-level review where li- brary faculty face scrutiny by colleagues unfamiliar with our discipline and our scholarship. Christiansen, Stombler, and Thaxton give a cogent overview in their sociological discussion of library-faculty relations of some of the reasons.7 In their study, they point out that the terminal degree can reinforce status differences on a campus. They find that many faculty members do not understand what librar- ians do and see them as more service- oriented and themselves as “focusing on the production and dissemination of knowledge.”8 Various studies discuss the challenge librarians face as scholars. Many describe the lack of time dedicated to research or crl-51r1 Assessing Library Scholarship 511 the paucity of research skills gathered during MLS studies. Others focus on giving advice for approaching the ten- ure process and tips for success, recent examples being Harris, Garrison, and Frigo,9 as well as Houdyshell.10 While these many studies and articles explore the issues around promotion and tenure for librarians, few address valuing our scholarship. At Oregon State University (OSU), a medium-sized land, sea, space, and sun grant university, librarians are tenured faculty members. The formal promotion and tenure review process follows the university guidelines used by all faculty members. The process is rigorous; and, while tenure rates vary throughout the university’s units, the libraries are not as successful as desired. OSU’s tenure rate for three years 2005–2007 was 90 percent, while the libraries’ was 75 per- cent.11 Looking at ten years of data, the university rate stays consistent and the libraries’ rate improves to 83 percent, still not good enough. One would hope for close to 100 percent tenure rates, given regular annual reviews, a comprehensive third-year review, and ongoing invest- ment in professional development—all intended to help faculty members track performance, address inadequacies, and, in some instances, recognize a less than perfect fit with the institution. The OSU Promotion and Tenure Com- mittee described the libraries’ dossiers as “problematic in terms of how to evaluate scholarship.”12 In addition to ongoing discussions within the libraries about re- search and writing, the University Librar- ian tasked a working group of the library faculty to review and describe where OSU Libraries faculty had published in the past ten years. She requested information on audience, review process, acceptance rate, and impact factor. This compilation of data would possibly provide a succinct way to describe the value of the published scholarship. During this project, questions, con- cerns, and challenges arose. What should be readily available information of inter- est to all library faculty (such as journal lists with acceptance rates) was not so. Much of the work done on research outlets, policies, and patterns has not been regularly updated.13 Describing publication outlets by impact factor alone is simplistic and does not reveal the rich- ness of library scholarship. The interplay between scholarship and service is rarely addressed in the tenure review, especially where scholarly output is directed toward practitioners and audiences beyond aca- demia.14 Finally, documenting the value of new and nontraditional approaches to scholarship remains challenging to many in the university setting.15 Librar- ians continue to grapple with describing their scholarly output succinctly and com- municating its value effectively to those outside the discipline. The Carnegie Foundation studies in the 1990s provide a means of redefin- ing scholarship within the academy as well as initial steps toward assessing it.16 Sometimes referred to as the Boyer model, the redefinition was highly touted as a catalyst for broadening the academy’s perspective on scholarship. Table 1 describes the model and two interpreta- tions specific to librarianship. Extension faculty in tenure positions have worked with these definitions to develop means to better describe their scholarship.17 Li- brarians have been encouraged to do the same.18 We used the scholarly output of OSU librarians as a case study to see how the Boyer model and other factors could be used to better validate and describe faculty output. We examined faculty publications over the past ten years to provide insight into output trends, valida- tion of outlets, and the character of library faculty publications. Finally, we looked at blogs created by OSU librarians as ex- amples of a new form of scholarship that blends service, research, and communica- tion. Throughout our study, we sought to find how the value of our scholarly output could be best described to the university tenure committee. In doing so, we an- 512 College & Research Libraries November 2010 ticipate that many tenure-track librarians and library review committees may look at their description of scholarship differ- ently and more creatively. Defining Scholarship for Librarians Central to the task of validating library scholarship is describing it. In 1995, OSU adopted the Boyer model into its pro- motion and tenure guidelines to better articulate the character of scholarship in a land grant institution.22 The adoption reinforced the concept that scholarship is creative, peer validated, and communi- cated appropriately. It explicitly considers that scholarship is “broader than results of research published in a peer refereed journal.”23 Within the OSU Libraries, faculty members were encouraged to view their scholarship as the aspects of their work that involved creativity, peer validation, and communication. The inte- gration of the Boyer concepts in the OSU Promotion and Tenure Guidelines begins to address concerns about adapting the tenure process to the library faculty. Lowry, St. Clair, and Miller explain how the Boyer model can be applied to librarianship broadly24 (see table 1). Boyer describes four categories of scholarship: Discovery, Integration, Application, and Teaching.25 Traditionally, the research community understands discovery the most readily, as it represents new contri- butions to knowledge. Librarians work- ing in this category are often Library and TABLE 1 Examples using Boyer’s Classification of Scholarship Boyer (1990) General: Weiser19 Libraries: St. Clair & Miller20 Libraries: Lowry21 DISCOVERY • Generates & communicates new knowledge & understanding • Develops & refines methods • Information organization • User needs • Preservation & access • Navigating evolving information landscape • New knowledge INTEGRATION • Synthesizes & communicates new or different understanding of knowledge or technology & its relevance • Develops & refines methods • Learning theory applied to bibliographic instruction • Communication theory to study reference work • Management techniques as implemented in the library • Applying management & organizational theory to library • Working with computer science to develop tools APPLICATION • Develops & communicates new technologies, materials, or users • Fosters inquiry & invention • Develops & refines new methods • Best practices in daily work • Experiments with what worked & did not • Advances best practices • Create bibliographies, Web sites TEACHING • Develops & communicates new understanding & insights • Develops & refines new contents and methods • Fosters lifelong learning behavior • Create and improve teaching techniques • Effectiveness of online tutorials • Best practices in user interfaces • Learning styles in teaching • Ensure information literacy • Develop innovative methods Assessing Library Scholarship 513 Information Science (LIS) faculty as op- posed to practicing librarians with faculty status. Research on new ways to organize or search for information are examples of Discovery. Integrative scholarship brings together ideas and concepts across disci- plines and consequently interprets them or finds new perspectives. Examples include collaborations with computer sci- ence or business management examining library processes and services. Scholar- ship of Application addresses problems and issues encountered in library work and professional service commitments. The development of software, compila- tions of bibliographies, and assessment of library services and products are examples. Finally, Teaching as a form of scholarship involves the development of innovative methods, articulation of outcomes, assessment of efficacy, and best practices. Raber and Connaway26 suggest that the Boyer model helps bridge the gap be- tween librarians as scholars and librarians as practitioners because it “incorporates research, teaching and service, but in a holistic and practical context.” For faculty who are on tenure track with a research component to their position description, the tension between research and practice can make it difficult to define and execute a research agenda because “research” can be seen as removed from the prob- lems and issues raised from daily work. Raber and Connaway make a strong case for valuing all categories of scholarship as they inherently address the needs of librarian-scholars as well as practitio- ners. They also stress the importance of teaching as a process of discovery and education. They encourage flexibility in faculty evaluation criteria and stress the importance of the practitioner as audi- ence for scholarship. The Association of College and Research Libraries 199627 task force on redefining scholarship also voices support for use of the Boyer model because many activities of academic librarians do not fall “neatly into the traditional model of faculty performance expectations.” St. Clair and Miller dem- onstrate the applicability of the Boyer model with their analysis of one volume of Journal of Academic Librarianship.28 They show that articles fall into the categories of Discovery, Integration, Application, and Teaching in patterns similar to other disciplines. The Value of the Journals Even given OSU’s acceptance of the Boyer model, OSU librarians still face the chal- lenge of how to effectively evaluate our scholarship and communicate that to oth- ers. Usually, we use the publication record in peer-reviewed journals as validation in itself, a shared practice among disciplines. We look to impact factor, newer metrics such as Eigenfactor (http://eigenfactor. org), and expert opinion to assess the val- ue of a journal. Noting the problems with the Institute for Scientific Information’s Journal Citation Reports, Cameron explains the limitations of the ISI impact factors and citedness including journal size, type, and frequency, as well as language and lack of coverage.29 He reminds us that “it is misleading to assess the value of the ar- ticle based solely on the journal in which it has been published.” Nissonger agrees that “a journal’s position in the various rankings for its discipline helps define the journal and delineate its role in scholarly communication.”30 He is concerned with flaws in the ranking caused by blending information and library science journals into a single ranking as well as combining titles aimed at different audiences (such as LIS faculty versus library faculty). More generally, Browman and Ster- gious remind us that the use of perfor- mance metrics without considering their weaknesses and strengths runs counter to the purpose.31 Quantifying performance without context has limited value. Cole- man echoes this in her study examining the multifaceted value of specialized library journals.32 She contends that a journal may have a low impact factor, yet serve a primary role among a specialized audience. Rousseau describes several 514 College & Research Libraries November 2010 indicators used to evaluate scholarship including citation factor, timeliness of publication, quality of the editorial board, review process, and where the journal is indexed.33 He encourages using an array of tools to evaluate library journals. Rous- seau is one of the few to discuss shifts in evaluation given electronic journals. He suggests that the increased emphasis and use of individual articles “erode[s] the value given to journal impact factors.” Via examines the meaning of peer- review and the level of acceptance as indicators of value.34 She finds a lack of consistency in the manuscript review process that suggests difficulty in using peer review as a term. Peer review ranges from editorial to full double blind; hence knowledge of the process for each jour- nal is important. She encourages review committees to consider the quality of the articles including its writing, contribution to the field, timeliness and accessibility in addition to where it is published. This last consideration is particularly pertinent with changes in patterns of communica- tion and shrinking library collections. Ali, Young, and Ali reinforce this approach with their two checklists for review deci- sions.35 The first checklist addresses the quality of the articles; the second checklist assesses the quality of the journal. They encourage thoughtful and multifaceted assessment rather than reliance on one or two popular measures. Value is a judgment call. The opinions of experts are often used to compile rankings. Nissonger and Davis continue this practice, comparing the opinions of ARL library directors with those of ALA- accredited LIS programs.36 They show a hierarchy of LIS journals, but that compo- sition is slowly changing as LIS becomes more diverse and interdisciplinary. They also point out that a low-ranking journal may be important for a specialized audi- ence. Campbell, the editor-in-chief of Nature, reminds us that articles may “fail” the citation test because they are in fields with low rates (such as LIS) or are good papers “but not ‘hot’.”37 The OSU Case Study O’Meara reminds us that “tenure and promotion are the valuing of people’s professional lives.”38 The publication record is one element of those lives. This case study examines scholarly output by describing it in terms of the outlet, the topic, the methodology, and the Boyer Classification. In the process, we found ways of more thoroughly describing our scholarship. Methodology We created an EndNote library of OSU Library faculty publications, using sub- scription databases (Library Literature and Library and Information Science and Technology Abstracts) and the Internet (Google Scholar) as the primary tools. Since none of these resources provide an author affiliation field, we used several search terms reflecting the names of our main and branch campus libraries in the subject and keyword fields to find publications. In addition, we searched individual author names as they were dis- covered and maintained a list of authors. To further identify authors, we reviewed the OSU Archives collection “Library Memorabilia.”39 Several documents in the collection included references to pub- lications, and others included names we had previously missed in our searching efforts. We looked for articles authored by at least one OSU librarian and published between 1998 and 2007. We eliminated book reviews and abstracts of conference proceedings from the list. Using the research notes field in EndNote, the entries were tagged by the type of publication (such as chapter, peer reviewed, magazine, newsletter, proceed- ings, report) and whether their audience was local (such as a report to the OSU faculty senate), state, national, or inter- national. Some liberties were taken with the audience classification, as it was not always easy to distinguish international from national. We reviewed each pub- lication’s online description including place of publication and editorial board Assessing Library Scholarship 515 members to help determine audience geography. We also identified the acceptance rates, review process, impact factor from Journal Citation Reports, and SCImago Journal and Country Rank (www.scima- gojr.com/). We were surprised that an accurate, current list of library science journals was not readily available that had at least some of this information. This type of guide would be useful, as past examples illustrate.41 We each reviewed the subset of 87 peer- reviewed articles and tagged each with three additional descriptors identifying the Boyer form of scholarship, the meth- odology, and the topic (see table 2). The methodology terms were derived from Blake42 as described by Lowry43 and the topics were devised by us. Where Do OSU Librarians Publish? Peer-reviewed publications with an in- ternational or national reach comprise the majority of OSU librarians’ scholarly output over the last ten years (see table 3). This is congruent with the trends expect- ed in a library whose faculty members are tenure track and where publishing in venues that provide the most impact is essential for achieving tenure. The sec- ond highest number is in the magazine category, where the geography is defined as “state.” The number of peer-reviewed articles published by OSU librarians between 1998 and 2007 fluctuated from year to year, but generally trends upward (see figure 1). This is opposite of Wibberly, Hurd, and Wellar’s findings, which de- scribe a downward trend in publication output by U.S. academic librarians.44 They recognize that the research libraries have the most productive librarians with eight of the top twelve being land grant institu- tions. For OSU, the upward trend reflects the variation in where faculty members are in the tenure process, rather than the number of librarians. In fact, the number of library faculty has decreased slightly in the past ten years. As described earlier, our original task was to find and present traditionally accept- able metrics for the journals in the list of faculty publications. These metrics include such facets as impact factor and ac- ceptance rates. With consistent data, a higher impact factor and a lower acceptance rate could TABLE 2 Categories Used to Describe OSU Library Publications Boyer Scholarship Blake Methodology40 Topics Discovery Teaching Integration Application Descriptive Case study Bibliographic (such as essays, review articles) Historical/Biographical Survey Bibliometric (citation studies) Experimental Quasi-experimental Theory Instruction Cataloging Collections Management Profession Public Services Scholarly communication Systems TABLE 3 OSU Publications by Publication Type and Publication Audience Type of Publication International National State Local Peer Review 44 43 0 0 Magazine 3 8 25 0 Chapter 1 0 1 0 Proceedings 10 3 0 0 Newsletter 2 6 0 1 Report 2 0 0 1 516 College & Research Libraries November 2010 provide one way to measure a journal’s value and likewise OSU librarians’ scholarship. However, the peer-reviewed journals OSU librarians published in most frequently (see table 4) did not consistently provide acceptance rates or appear in Journal Citation Reports. Other measures such as the SCImago Journal Rank also did not offer a clear indication of accepted value. Mukherjee’s analysis using GoogleScholar of open-access LIS journal offers initial measures for the impact of some electronic journals.45 The “standard” metrics called upon to evalu- ate the quality of library publications fall short, except for the use of peer review. Not a single research journal in which OSU librarians published most frequently had all of the desired metrics available, and only three had an ISI Impact Factor. Table 4 shows each title in which OSU librarians published twice or more over the last decade. In addition to some of the metrics discussed previously, information on the open-access status of the publica- tion is included.46 Schwartz’s work on uncitedness versus usefulness questions the concept of cited- ness and whether it reflects usefulness of the journal or of specific articles.47 He also stresses the value of practitioner- oriented information, as validated by the 2000 survey of ACRL members.48 Via and Schmiddle describe the purpose of the journals as either a means of “sharing trends and reporting on improvements in operations and services” or “publishing research findings that advance theories of information science.”49 These findings im- ply that the value of scholarship has more dimension than simply being published in a high-impact journal. Besides the peer-reviewed publica- tions, publishing locally is important to OSU librarians. OLA (Oregon Library As- sociation) Quarterly is not peer reviewed and has a smaller geographic range than the national and international library sci- ence journals. Publishing locally reflects the OSU commitment to the broader Or- egon community through its role as a land grant institution. It helps the community of librarians develop, communicates with partners in other endeavors such as local consortial initiatives and programs, and is a good way to get started in publishing. Most of the articles in OLA Quarterly ad- dress the librarian’s service mission rather than scholarship. Every discipline has a “top tier ” of journals. Yet many more journals have audiences ranging from those focused on very specific topics to those looking for good ideas. Powell, Baker, and Mika stress the role of popular journals in the dis- semination of scholarship.53 They examine how practitioners discover research and hence integrate it into the profession and their understanding of their work. They found that many practitioners “do FIGURE 1 Peer-Reviewed Publications Published 1998–2007 at OSU Libraries Assessing Library Scholarship 517 TA B L E 4 P ee r- re vi ew ed J ou rn al s P ub lis he d in ≥ 2 T im es b y O SU L ib ra ri es F ac ul ty (N um be r of O SU L ib ra ri es P ub lic at io ns b et w ee n 19 98 a nd 2 00 7) P ub lic at io n T it le IS I Im pa ct F ac to r (2 00 7) A cc ep ta nc e R at e Sc im ag o (2 00 7) E ig en fa ct or A rt ic le In flu en ce O pe n A cc es s5 0 P ee r R ev ie w O pe n A cc es s Jo ur na l R an ki ng by N um be r of C it at io ns 51 N is so ng er & D av is Ta bl e 2 R an ki ng s5 2 A ri ad ne (5 ) N ot ra te d 55 % –6 0% N ot ra te d N ot ra te d FU L L E di to ri al 7 N ot ra te d E le ct ro ni c Jo ur na l o f A ca de m ic & S pe ci al L ib ra ri an sh ip (2 ) N ot ra te d 70 % N ot ra te d N ot ra te d FU L L B lin d 15 N ot ra te d In te rn et R ef er en ce S er vi ce s Q ua rt er ly (2 ) N ot ra te d 40 % –5 0% .0 41 N ot ra te d Pr ep ri nt B lin d N ot a pp lic ab le N ot ra te d Is su es in S ci en ce & T ec hn ol og y L ib ra ri an sh ip (1 0) N ot ra te d 74 % (r ef er ee d) 90 % (n on re fe re ed ) .0 39 N ot ra te d FU L L B lin d 8 N ot ra te d L ib ra ry C ol le ct io ns , A cq ui si tio ns a nd T ec hn ic al Se rv ic es (3 ) .2 50 75 % .0 37 .1 9 Po st pr in t D ou bl e bl in d N ot a pp lic ab le 12 L IB R E S: L ib ra ry & I nf or m at io n Sc ie nc e R es ea rc h E le ct ro ni c Jo ur na l ( 3) N ot ra te d 21 % –3 0% .0 39 .6 3 FU L L B lin d 11 N ot ra te d Jo ur na l o f A ca de m ic L ib ra ri an sh ip (2 ) .5 51 40 % .0 44 .4 8 Po st pr in t B lin d N ot a pp lic ab le N ot ra te d Jo ur na l o f L ib ra ry A dm in is tr at io n (2 ) N ot ra te d 40 % –5 0% .0 39 N ot ra te d Pr ep ri nt B lin d N ot a pp lic ab le 4 O C L C S ys te m s & S er vi ce s (8 ) N ot ra te d 70 % .0 39 N ot ra te d Po st pr in t D ou bl e bl in d N ot a pp lic ab le N ot ra te d po rt al : L ib ra ri es & th e A ca de m y (5 ) .8 85 N ot a pp lic ab le * .0 50 .3 9 Po st pr in t D ou bl e bl in d N ot a pp lic ab le N ot ra te d R ef er en ce L ib ra ri an (6 ) N ot ra te d 65 % .0 35 N ot ra te d Pr ep ri nt B lin d N ot a pp lic ab le N ot ra te d R es ea rc h St ra te gi es (5 ) N ot ra te d N ot a pp lic ab le .0 35 N ot ra te d Po st pr in t B lin d N ot a pp lic ab le N ot ra te d Se ri al s L ib ra ri an (3 ) N ot ra te d 66 % –7 5% .0 55 N ot ra te d Pr ep ri nt D ou bl e bl in d N ot a pp lic ab le N ot ra te d Te ch ni ca l S er vi ce s Q ua rt er ly (2 ) N ot ra te d N ot a va ila bl e .0 35 N ot ra te d Pr ep ri nt B lin d N ot a pp lic ab le N ot ra te d 518 College & Research Libraries November 2010 engage in and care about research,” but many limit what they read (for example, 15 percent read more than four research journals regularly). What Do OSU Libraries Publish? Where OSU librarians publish suggests a leaning toward being practitioners who write rather than scholars who practice. Does the content reveal the same pattern? Hildreth and Aytac found little differ- ence between the two camps, although “practitioners conduct more library- specific studies and academics conduct more use and user studies.”54 The authors expressed concern in the low use of quali- tative research methods by practitioners. The 2003 report also noted this weakness but stated that the output is usually well structured and readable.55 OSU librarians publish on topics from think pieces to the edges of digital librarianship to usability studies for Web interfaces to collection assessment case studies. A critical look at the type, methodology, and content of peer-review publications over the past ten years suggests that, while the librarians are practitioners, they are also thoughtful researchers with a breadth of approaches that refute stereotyping. This examination was not straight- forward in part because there is little discussion of this approach.56 Initially, there was disagreement among us after individually reviewing and classifying the OSU Libraries’ faculty peer-reviewed articles. Out of the 87 articles, we disagreed on Boyer clas- sification on 26 of them, 29 on methodology, and 12 on topic. It took discussion to arrive at a shared understanding of the Boyer classifications and the different research methodol- ogy terms. In particular, the differences between Integra- tion versus Application, and Teaching versus Application in the Boyer classification required thoughtful conver- sation. Integration focuses on bringing together knowledge and practic- es from various perspectives and possible disciplines. Application is more directly working to communicate new approaches. Teaching works with instruction methods and theory. In light of our deliberations, we reconsidered our choices and pro- duced the categorized list of articles used in the analysis that follows. More than half (52%) of the articles fell into the Boyer classification of Ap- plication. The remaining articles classed fairly evenly across the remaining three categories, with Integration leading at 19 percent, followed by Discovery and Teaching (see figure 2). Case Study (36%) was by far the dominant methodology used. Surveys, Quasi-experimental, Bibliographic/Com- mentary, and Descriptive research had similar numbers, about 15%–16% each, and represented the majority of other articles. Bibliometric (5%), Historical/Bio- graphical (4%), and Theory (2%) reflected approaches employed in a small number of articles (see figure 3). No one topic dominated, although those reflecting expertise and interest in direct services to users (Public Services, Instruction) comprised nearly 40 percent of the publications. Some of the distribu- tion can be attributed to the expertise of librarians in the tenure process (Public Service, Cataloging, Collections). Other topics reflect emerging interests (Systems, Scholarly Communication) (see figure 4). FIGURE 2 Boyer Scholarship Classifications Teaching, 13% Integra�on, 19% Discovery, 16% Applica�on, 52% Assessing Library Scholarship 519 Analyzing the articles in light of Boyer, research methodologies and even topic areas was very challenging. The exercise raised the question of how complex as- sessing the content of publications is when done by those familiar with the field, let alone those unfamiliar. Applying a lens of Boyer classification, methodology, and content requires effort, yet doing so reveals more about the breadth and depth and value of scholarship. Our results indi- cated that, like the St. Claire/Miller study, librarian scholarship is not so different from that of other disciplines when ana- lyzed by the Boyer classification.57 Assessing New Forms of Scholarship We can assess value when published in a peer-reviewed, high- impact outlet. We can examine it by looking closely using Boyer’s model. However, the library community as well as others in academia confront the evolving nature of scholarly commu- nication and pres- sure to fit new formats and methods into the traditional value system. Born-digital and open-access journals, while not new, are slowly gaining recognition along with their older print coun- terparts. The journals used by Nisonger and Davis in their survey of ARL library directors and LIS deans in- cluded several e-only jour- nals (such as D-Lib and First Monday), but there was not agreement on their value.58 Mukherjee’s study on open- access journals suggests the growing validity of open- access e-journals in the library field.59 Beyond journals, blogs are emerging as a method for communicating to patrons, for keeping up in one’s profession, and as instructional design tools. Their value as scholarship is less clear. Recent publica- tions on this topic include development of blogs for specific audiences.60 Murray and Bell recommend librarians track faculty (nonlibrarian) blogs as a way of keeping up with their scholarly interests and perspectives in their subject areas and their roles as educators.61 Martindale and Wiley describe the use of RSS feeds FIGURE 3 Research Methodologies Theory, 2% Survey, 16% Quasi- experimental, 16% Historical/ Biographical, 4% Bibliographic/ Commentary, 15%Bibliometric, 5% Descrip�ve, 13% Case Study, 36% FIGURE 4 Publication Topics Cataloging, 9% Systems, 21% Scholarly Communica�on, 4% Profession, 8% Public Services, 20% Management, 5% Instruc�on, 19% Collec�ons, 15% 520 College & Research Libraries November 2010 (http://oregonstate.edu/~reeset/blog/), provides users of his software programs with critical updates and advice, describes for friends his latest cycling adventures, and then delivers insights on current developments in the library world. The first is service, the second just fun, but the third opens a window into a creative, thoughtful mind in ways that journal articles do not. Another example at OSU Libraries is Anne-Marie Deitering’s blog Info-fetishist (http://info-fetishist.org/). It combines observations on teaching and information visualization with personal information. Deitering’s communications challenge the reader to engage with the topic and leave comments if so inclined, making the process of scholarship more interactive. This type of engagement is hard to measure in terms of value, yet it should be seriously considered. Blogs are vehicles to teach and com- municate to both broad and specific audi- ences. Their format precludes them being taken seriously as scholarship in current tenure review processes, but their con- tent often demonstrates engagement and suggests impact in ways rarely seen in the print library journal. This raises questions about the concept of format and vehicle. Expanding acceptance of new forms of communication along with reconsidering what constitutes scholar- ship will benefit librarianship as a whole. A first step is accepting open-access, peer-reviewed journals as outlets of high impact and validity. The next step will be integrating nontraditional peer-reviewed work such as blogs that have an active readership and generate comments and discussion. Valuing Our Scholarship: Changes in Approach This study demonstrates the challenges of describing and valuing library schol- arship. OSU librarians are publishing more, yet not in journals with the highest impact. There is breadth in approaches, topics, methodologies, and outlets. The librarians are contributing in a meaning- to track blogs as a tool that makes keep- ing up with development in one’s own field easier than in the days of surfing from Web page to Web page.62 Via and Schmiddle also mention blogs and dis- cussion lists as timely and cost-effective means of sharing information.63 Hard- esty and Sugarman look at how librarian practitioners stay abreast of the field.64 Their findings show mixed opinions on the usefulness of blogs but high use of discussion lists and journals (84% and 95% of respondents respectively). The blog as a form of academically acknowledged scholarship in the library and higher education communities has not been adequately studied. In the rare article in which blogs are mentioned in proximity to scholarly pursuits, they seem to be dismissed out of hand as not having scholarly impact but instead accepted as part of academic service requirements— which often is also considered in promo- tion and tenure processes.65 Lawson and Pelzer are one of the few to focus on as- sessing technology projects as scholarship but do not mention blogs or open-access/ born-digital journals—not surprising given the date of publication.66 Rather, they seek to validate Web-based and other technology projects as scholarship. Several examples from OSU highlight the complex nature of assessing blogs as scholarship or service. OSU librarians started Infodoodads as a forum to share “existing and new tools, services, and technology for finding information on the [I]nternet” (www.infodoodads.com). Most of the short reviews are personal impressions and encounters with the technology. This informal style belies the testing and thought that goes into recom- mending these tools to others. This blog is partly service, but it reflects the scholarly interests of its creators—how people are coping with new technology in the infor- mation realm. Individual blogs vary, but they can have a thread of scholarship intertwined with opinion, practical advice, and whim- sy. Terry Reese’s blog, Terry’s Worklog Assessing Library Scholarship 521 ful way to the profession. We can do a better job of adequately explaining this breadth and impact of our contribution. The value of library scholarship is not measured solely by where one publishes. It includes assessment of the author ’s scholarly thinking and ability to com- municate to the appropriate audience. Glassick, Huber, and Maeroff describe a framework for assessing scholarship in its variable forms.67 They focus on the steps shared by those involved with scholarship and the qualities of a scholar. Both can be used in libraries as criteria for assessing the value of a librarian’s work. Integrity, perseverance, and cour- age—the qualities of a scholar—should be reflected in the methods, willingness to collaborate, curiosity, focus, and will- ingness to take on difficult work. The resulting works can then be assessed by considering the Boyer classification, the research methodology, and the topic. Tenure committees may use these three as a consistent means of describing the scholarly works. At OSU, the Library Review Committee discusses the quality and value of a candidate’s scholarship in light of the above. The university recently revised its guidelines in recog- nition of differences in types of scholar- ship among units. The three categories provide a more disciplined approach to review that can be shaped to both the libraries’ and the university’s view of scholarship. Value is also found in the impact of the work, where something published or communicated should align with the in- tended audience. It may be a high-impact journal, a specialized e-journal, or a blog. Each outlet has value. Recent candidates in the OSU Libraries have included more description about their publications including articulating their roles in the research and writing, and documenting the impact (such as journal rank, accep- tance rate, or number of downloads). This in turn aids the University Promotion and Tenure Committee in understanding the value of an article or report. Journals emerge, disappear, and change focus and relevancy. New ven- ues for scholarly communication are evolving. Yet the library community lacks comprehensive tools to describe those outlets and keep the descriptions current. The Association for College and Research Libraries would be doing the community a great service if it developed a resource listing current information on policies, acceptance rates, audience, and impact of the various outlets for library scholarship. Finally, the intersection of service and scholarship needs to be explored more completely in the review process. It is simplistic to partition a librarian’s work into teaching, administration, service, and scholarship. To truly embrace the Boyer concept, library faculty should explain how their work contains elements of scholarship throughout. Communicating to a local audience often has value as both service and scholarship. This needs to be better explained by librarians involved with such activities. One approach is better definition of position descriptions, a task undertaken across OSU. Position descriptions are expected to accurately describe what a candidate has done while working toward tenure. Recently, faculty members are revising their position de- scription to define what percentage of their duties result in scholarship. This gives librarians an opportunity to link teaching and administrative duties, for example, to scholarly outcomes. This study enabled us to better under- stand how to assess and value scholar- ship. Understanding suggests dialogue about what makes good scholarship, including the relevant issues to research, the possible audiences, and the appropri- ate mechanisms of communication. Using the Boyer classification provides a means to assess our scholarship within the acad- emy. Only time will reveal if thoughtful and consistent evaluation will help ar- ticulate the value of library scholarship in the context of the tenure review process at both the library and university level. 522 College & Research Libraries November 2010 Notes 1. Ernest L. Boyer, Scholarship Reconsidered: Priorities of the Professoriate, Special Report (San Francisco: The Carnegie Foundation for the Advancement of Teaching, 1990). 2. Charles B. Lowry, “The Status of Faculty Status for Academic Librarians: A Twenty-Year Perspective,” College & Research Libraries 54, no. 2 (1993): 163–72. 3. Mary K. Bolin, “A Typology of Librarian Status at Land Grant Universities,” The Journal of Academic Librarianship 34, no. 3 (2008): 220–30. 4. Janet Swan Hill, “Constant Vigilance, Babelfish, and Foot Surgery: Perspectives on Faculty Status and Tenure for Academic Librarians,” portal: Libraries and the academy 5, no. 1 (2005): 7–22; Charles B. Lowry, “Research and Scholarship Defined for Portal: Libraries and the Academy,” portal: Libraries and the academy 4, no. 4 (2004): 449–53. 5. Janet Swan Hill, “Wearing Our Own Clothes: Librarians as Faculty,” The Journal of Aca- demic Librarianship 20, no. 2 (1994): 71–76. 6. Glorianna St. Clair, Rush G. Miller, and P. Michelle Fiander, “Academic Librarianship and the Redefining Scholarship Project,” The Serials Librarian 38, no. 1/2 (2000): 66. 7. Lars Christiansen, Mindy Stombler, and Lyn Thaxton, “A Report on Library-Faculty Relations from a Sociological Perspective,” The Journal of Academic Librarianship 30, no. 2 (2004): 116–21. 8. Ibid., 119. 9. Laura Harris, Julie Garrison, and Emily Frigo, “Bringing the Tenure and Promotion Process into the Digital Age,” College & Research Libraries News 70, no. 8 (2009): 465–68. 10. Mara L. Houdyshell, “Ten Tips towards Tenure: Advice for the Professional Journey,” College & Research Libraries News 70, no 8 (2009): 469—70. 11. Office of Academic Affairs, 3-Year Executive Summary (2005, 2006, 2007): Promotion and Tenure Review Outcomes (Corvallis: Oregon State University, 2008. 12. Promotion and Tenure Committee, Annual Report 2001–2002 (Corvallis: Oregon State University). Available online at http://oregonstate.edu/senate/committees/ptc/ar/2001-2002. html. [Accessed 21 May2009]. 13. Leslie Haas, Suzanne Milton, and Aimee Quinn, “Surviving the Publishing Process: A Beginner’s Guide,” RQ 36, no. 2 (1996): 230–46; Mohammed Mury and Mitchel Walters, “Writ- ing for Journals in Library and Information Science: A Report of a Survey,” The Serials Librarian 31, no. 4 (1997): 23–40. 14. Kerry Ann O’Meara, “Uncovering the Values in Faculty Evaluation of Service as Scholar- ship,” The Review of Higher Education 26, no. 1 (2002): 57–80. 15. Joan M. Leysen and William K. Black, “Peer Review in Carnegie Research Libraries,” College & Research Libraries 59, no. 6 (1998): 512–22. 16. Charles E. Glassick, Mary Taylor Huber, and Gene I. Maeroff, Scholarship Assessed: Evalu- ation of the Professoriate (San Francisco: Jossey-Bass Publishers, 1997). 17. Roger G.J.R. Adams et al., “A Diversified Portfolio of Scholarship: The Making of a Successful Extension Educator,” Journal of Extension 43, no. 4 (2005): 4COM2; eXtension, “The Scholarship of Extension,” available online at http://about.extension.org/wiki/The_Scholar- ship_of_eXtension [Accessed 6 June 2009; Barbara O’Neill, “Promotion, Tenure, and Merit-Based Pay: 15 Keys to Success “ Journal of Extension 46, no. 4 (2008): 4TOT2. 18. Lowry, “Research and Scholarship Defined for Portal: Libraries and the Academy,” 449–53; St. Clair, Miller, and Fiander, “Academic Librarianship”, 63–67. 19. C.J. Weiser, “The Value System of a University: Rethinking Scholarship.” Available online at www.adec.edu/clemson/papers/weiser.html. [Accessed 29 April 2009] 20. St. Clair, Miller, and Fiander, “Academic Librarianship,” 63–67. 21. Lowry, “Research and Scholarship Defined for Portal: Libraries and the Academy,” 449–53. 22. C.J. Weiser and Lyla Houglum, “Scholarship Unbound for the 21st Century,” Journal of Extension 36, no. 4 (1998). 23. Ibid. 24. Lowry, “Research and Scholarship Defined for Portal: Libraries and the Academy”, 449–53; St. Clair, Miller, and Fiander, “Academic Librarianship”, 63–67. 25. Boyer, Scholarship Reconsidered. 26. Douglas Raber and Lynn Silipigni Connaway, “Two Cultures, One Faculty: Contradictions of Library and Information Science Education,” Journal of Education for Library and Information Science 37, no. 2 (1996): 125. 27. W. Bede Mitchell et al., Academic Librarianship and the Redefining of Scholarship Project: A Report from the Association of College and Research Libraries Task Force on Institutional Priorities and Assessing Library Scholarship 523 Faculty Rewards (Chicago: Association of College and Research Libraries, 1998): 2, 7. 28. St. Clair, Miller, and Fiander, “Academic Librarianship,” 63–67. 29. Brian D. Cameron, “Trends in the Usage of ISI Bibliometric Data: Uses, Abuses and Implications,” Portal: Libraries and the academy 5, no. 1 (2005): 112. 30. Thomas E. Nisonger, “Jasis and Library and Information Science Journal Rankings: A Review and Analysis of the Last Half-Century,” Journal of the American Society for Information Science 50, no. 11 (1999): 1004. 31. Howard I. Browman and Konstantinos I. Stergiou, “Factors and Indices Are One Thing, Deciding Who Is Scholarly, Why They Are Scholarly, and the Relative Value of Their Scholar- ship Is Something Else Entirely,” Ethics in Science and Environmental Politics 8 (2008): 1–3. 32. Anita Coleman, “Assessing the Value of a Journal Beyond the Impact Factor,” Journal of the American Society for Information Science and Technology 58, no. 8 (2007): 114–61. 33. R. Rousseau, “Journal Evaluation: Technical and Practical Issues,” Library Trends 50, no. 3 (2002): 418–39. 34. Barbara J. Via, “Publishing in the Journal Literature of Library and Information Science: A Survey of Manuscript Review Processes and Acceptance,” College & Research Libraries 57, no. 4 (1996): 365–76. 35. S. Nazim Ali, Harold C. Young, and Nasser M. Ali, “Determing the Qualityof Publications and Research for Tenure or Promotion Decisions: A Preliminary Checklist to Assist,” Library Review 45, no. 1 (1996): 39–52. 36. Thomas E. Nisonger and Charles H. Davis, “The Perception of Library and Information Science Journals by LIS Education Deans and ARL Library Directors: A Replication of the Kohl- Davis Study,” College & Research Libraries 66, no. 4 (2005): 341–77. 37. Philip Campbell, “Escape from the Impact Factor,” Ethics in Science and Environmental Politics 8 (2008): 6. 38. O’Meara, “Uncovering the Values,” 77. 39. Memorabilia Collection (MC-Library) (Corvallis: Oregon State University Archives). 40. Ruth Fraley and Barbara J. Via, “Survey of Library & Information Science Journal Pub- lishers,” in Librarian/Author: A Practical Guide on How to Get Published, ed. Betty-Carol Sellen (New York: Neal-Schumann, 1985): 117–214; Mury and Walters, “Writing for Journals,” 23–40; Carol F. Schroeder and Gloria G. Roberson, eds., Guide to Publishing Opportunities for Librarians (New York: The Haworth Press, 1995); Norman D. Stevens and Nora B. Stevens, eds., Author’s Guide to Journals in Library and Information Science (New York: Haworth Press, 1982). 41. Blake, “Research Methods,” 2513–23. 42. Virgil L.P. Blake, “Research Methods in LIS Dissertations,” in Encyclopedia of Library and Information Science, ed. Miriam A. Drake (New York: Marcel Dekker, 2003): 2513–23. 43. Lowry, “Research and Scholarship Defined for Portal: Libraries and the Academy,” 449-53. 44. Stephen E. Wiberley, Julie M. Hurd, and Ann C. Weller, “Publication Patterns of U.S. Academic Librarians from 1998 to 2002,” College & Research Libraries 67, no. 3 (2006): 205–16. 45. Bhaskar Mukherjee, “Do Open-Access Journals in Library and Information Science Have Any Scholarly Impact? A Bibliometric Study of Selected Open-Access Journals Using Google Scholar,” Journal of the American Society for Information Science and Technology 60, no. 3 (2009): 581–94. 46. Sherpa and University of Nottingham, Sherpa/Romeo Publisher Copyright Policies & Self- Archiving. Available online at www.sherpa.ac.uk/romeo.php. [Accessed 19 June 2009]. 47. Charles A. Schwartz, “The Rise and All of Uncitedness,” College & Research Libraries 58, no. 1 (1997): 19–29. 48. Melissa Cast and Shannon Cary, “Members Assess ACRL: Results of the 2000 Member- ship Survey,” College & Research Libraries News 62 (2001): 623–37. 49. Barbara J. Via and Deborah J. Schmidle, “Investing Wisely: Citation Rankings as a Mea- sure of Quality in Library and Information Science Journals,” Portal: Libraries and the academy 7, no. 3 (2007): 339. 50. Sherpa, Publisher Copyright Policies. 51. Mukherjee, “Open-Access Journals,” 581–94 . 52. Nisonger and Davis, “Perception of Library and Information Science Journals,” 341–77. 53. Ronald R. Powell, Lunda M. Baker, and Joseph J. Mika, “Library and Information Science Practitioners and Research,” Library & Information Science Research 24 (2002):49–72. 54. Charles R. Hildreth and Selenay Aytac, “Recent Library Practitioner Research: A Meth- odological Analysis and Critique,” Journal of Education for Library and Information Science 48, no. 3 (2007): 254. 55. Charles R. Hildreth, “How Are They Going about It? A Comparison of Research Methods by Lis Academic and Practitioner Researchers,” in Annual Meeting of the Association for Library and Information Science Education (Philadelphia, 2003). 524 College & Research Libraries November 2010 56. St. Clair, Miller, and Fiander, “Aca- demic Librarianship,” 63–67. 57. Ibid. 58. Nisonger and Davis, “Perception of Library and Information Science Jour- nals,” 341–77. 59. Mukherjee, “Open-Access Jour- nals,” 581–94. 60. J. Blair and A.V. Level, “Creating and Evaluating a Subject-Based Blog: Plan- ning, Implementation, and Assessment,” Reference Services Review 36, no. 2 (2008): 156; D. Murray and S. Bell, “Exploring the Faculty Blogoverse: Where to Start and What’s in It for Academic Librarians,” College & Research Libraries News 68, no. 9 (2007): 576–79; D.L. Schrecker, “Using Blogs in Academic Libraries: Versatile Information Platforms,” New Library World 109, no. 3/4 (2008): 117–29. 61. Murray and Bell, “Exploring the Faculty,” 576–79. 62. T. Martindale and D.A. Wiley, “Us- ing Weblogs in Scholarship and Teaching,” TechTrends Linking Research and Practice to Improve Learning 49, no. 2 (2005): 7. 63. Vi a a n d S c h m i d l e , “ I n ve s t i n g Wisely,” 333–73. 64. Skye Hardesty and Tammy Sugar- man, “Academic Librarians, Professional Literature, and New Technologies: A Sur- vey,” The Journal of Academic Librarianship 33, no. 2 (2007): 196–205. 65. H. Farrell, “The Blogosphere as a Carnival of Ideas,” Chronicle of Higher Education 52, no. 7 (2005); B14–B15. 66. Karen G. Lawson and Nancy L. Pelzer, “Assessing Technology-Based Proj- ects for Promotion and/or Tenure in Arl Academic Libraries,” College & Research Libraries 60, no. 5 (1999): 464–76. 67. Glassick, Huber, and Maeroff, Scholarship Reassessed. Connect Theory to Practice Published by the International Society for Technology in Education (ISTE), the premier membership association for education leaders engaged in improving learning and teaching by advancing the effective use of technology. ISTE journals help you make the connection. ◾ Stay on top of current trends and challenges in educational technology ◾ Apply specific, research-based applications ◾ Discover a forum for sharing research and developments For pricing and more information, visit www.iste.org/publications Journal of Research on Technology in Education www.iste.org/jrte Journal of Digital Learning in Teacher Education www.iste.org/jdlte Formerly the Journal of Computing in Teacher Education.