Evidence Based Library and Information Practice

Setting

The University of Illinois at Chicago is a public research university in the United States serving more than 33,000 students enrolled in its 16 colleges. The University Library provides collections (2.2 million print volumes) and services in support of campus instructional and research programs on the Chicago campus and at regional health sciences campuses in Peoria, Rockford, and Urbana. The Richard J. Daley Library in Chicago holds collections in the arts, humanities, social sciences, and engineering. The Library of the Health Sciences supports all of the health sciences colleges with materials for medicine, dentistry, nursing, public health, pharmacy, and applied health sciences. The University Library employs 150 faculty and staff members as well as more than 100 student workers.

Problem

The University of Illinois at Chicago Library is committed to enhancing its outcome-oriented initiatives. To that end, the University Library’s Steering Committee developed a strategic plan using a logic model, which is a visualization of a program or project showing the relationships between investments, activities, and intended results (W.K. Kellogg Foundation, 2006). The Assessment Coordinator, who is a member of the Steering Committee, has the primary responsibilities of planning and implementing library assessment initiatives, leading the Assessment Coordinator Advisory Committee, conducting surveys, and consulting with library units to determine their assessment needs. Another responsibility of the Assessment Coordinator is creating professional development opportunities for library staff related to assessment. While the Assessment Coordinator plays a role in providing resources and expertise in assessment, it is not practical for one person to look into every project. At the University Library, some librarians who possess strong assessment skills are involved in committees and working groups related to assessment in the library. Assessment responsibilities must involve team members across departments to overcome assigning responsibilities to one specific group. Therefore, this paper aims to demonstrate how assessment tools were used to create professional development opportunities to contribute to an assessment culture.

Evidence

A fundamental task of librarians with assessment responsibilities is to “help librarians demonstrate their library’s value to the institution” (ALA, 2017). Another key responsibility is to provide mentoring, training, and coaching in order to “build a culture of assessment and organizational capacity for assessment” (ALA, 2017).

To foster an assessment culture, the Assessment Coordinator at the University Library decided to develop and offer a workshop designed for library staff to illustrate the use of a logic model followed by individual consultations. The logic model was chosen as the workshop topic because logic models have been used as a framework in grant proposals (e.g., IMLS), strategic planning (Dubicki, 2011), library assessment (Stoddart & Lajoie, 2014), program evaluation (Markless & Streatfield, 2017), and program intervention (Kletter, Mendelez-Torres, Lilford & Taylor, 2018). Additionally, logic models are an important training topic (e.g., CARLI Counts, 2019).

During September - October 2018, the Assessment Coordinator had several informal interviews with department heads and senior staff to determine the best way to develop the logic model workshop. Based on the feedback from the department heads, the Assessment Coordinator created two pilot workshops during regularly scheduled monthly department meetings prior to launching the logic model workshop to all staff. Two department heads at the University Library agreed to offer 30 minutes of their meeting time for the pilot workshop.

Three days before the pilot workshop, the Assessment Coordinator distributed to participants a pre-workshop survey and pre-test asking them about their previous experience and knowledge of logic models. According to the pre-workshop survey results, more than 65% of participants were not aware of logic models, and more than 80% of participants had no experience using a logic model. Therefore, the workshop focused on the basic concepts of the logic model, as well as emphasizing the benefits of adapting logic models to their projects. To develop the workshop content, the Assessment Coordinator reviewed logic model training guides developed by Abdi and Mensah (2016), Taylor-Powell and Henert (2008), Community Tool Box (n.d.), and the W.K. Kellogg Foundation (2006).

During the workshop, the Assessment Coordinator shared the results from the pre-survey, played a video from the dean about the importance of learning the logic model and her expectations, and led activities to enhance the participants’ engagement. After the pilot workshop, library staff and faculty received a post-workshop survey and post-test. The pre- and post-assessment tools aimed to answer five major questions:

1. What were the participants’ previous knowledge and experience in using logic models?

2. What is the overall evaluation of the pilot logic model workshop?

3. What were the most interesting aspects of the workshops?

4. Did the participants’ knowledge and awareness of logic models increase after the workshops?

5. Which areas of the workshops could be improved?

A total of 26 library staff and faculty attended two pilot workshops (a 65% attendance rate). The first pilot workshop was conducted with the Resource Acquisition and Management Department in November 2018 (n = 16), and the second pilot workshop was conducted with the Research Services and Resources Department in December 2018 (n = 10). One half of participants completed the workshop evaluation survey. The key findings from the workshop evaluation survey (Figure 1) demonstrate that participants generally found the workshop to be helpful. All of the participants (n = 13) replied “strongly agree” and “agree” with respect to the content’s usefulness, the workshop’s level and format, the speaker’s understanding of the material, and their willingness to recommend the workshop to others. Only the workshop’s length was an issue. More than 15% of the respondents (n = 2) “disagreed” that the length of the workshop was appropriate, and 20% of the participants (n = 3) in the open-ended questionnaire provided further information saying that the workshop was too short.

Figure 1

The Evaluation Results of the Pilot Workshops (N = 13)

Participants were asked to name the most interesting aspect of the workshop. Participants indicated that the most interesting aspect was the game activity where groups worked together to identify the basic concepts of the logic model and put them in the correct (n = 5). One respondent commented that “the video from the dean was great to show she supports these efforts.” The results from the tests indicate that the mean scores of participants’ knowledge of the logic models in the post-test (M = 6.14) were higher than in the pre-test (M = 4.25). This result confirms the effectiveness of the workshop.

Implementation

Based on the findings from the two pilot workshops, the Assessment Coordinator has been providing the logic model workshop on an ongoing basis to groups or individuals involved in a project that requires measuring outcomes. To date, the workshop has been conducted with three departments and one committee. Because participants reported that the length of the pilot workshop was not sufficient, the workshop was expanded from 30 to 60 minutes. Additionally, depending on the group’s previous experience and knowledge as assessed through the survey and pre-test, the Assessment Coordinator tailored the workshop activities and the level of contents.

During the logic model workshops, many participants understood the concepts; however, when applying the logic model to their work or a project, it was not easy to develop on their own. As a result, follow-up support was offered through one-on-one consultations. After presenting the logic model workshops, several participants requested assistance with either developing logic models for their programs and projects or reviewing their previous logic models to check if the measurable outcomes were appropriate and to verify how the intended outcomes are met.

Outcome

Within half a year, this initiative had positive results. Immediately after the logic model workshop for the library’s human resources department in February 2019, the Director of Human Resources requested a follow-up consultation and expressed their desire to develop an onboarding program using a logic model. Since then, the Assessment Coordinator has been collaborating with the Human Resources department as they develop an onboarding program for new employees. As a result of the Assessment Coordinator’s six-month consultation with this department, the onboarding program using the logic model was completed and presented to the library steering committee.

Another example is the Undergraduate Engagement Program of the Research Services and Resources department, which was developed by the outreach engagement faculty using the logic model prior to the logic model workshop. During the consultations, the Assessment Coordinator and the outreach engagement faculty revised the outcome statements and discussed the possible measures that enable faculty to measure the desired outcomes. The updated logic model was also shared with the library steering committee and in a library faculty meeting. Some faculty who attended the presentations commented to the Assessment Coordinator that it was helpful for them to better understand the program and the program goals, and they wanted to develop their projects using logic models.

The last example is a faculty member who participated in the logic model workshop and wanted to write a grant proposal using the logic model. After the workshop, the Assessment Coordinator and the faculty member reviewed the draft grant proposal and focused on how to articulate measurable outcomes using the logic model. Afterwards, the faculty member submitted her grant proposal and received the $20,000 grant.

Reflection

Developing valuable library staff training requires innovative strategies and a significant investment in time and resources. Offering logic model workshops to the University of Illinois at Chicago Library staff was one example of a successful training to build a culture of assessment. It was a useful place to start because it shows participants the value of considering assessment from the very beginning of designing a project or initiative. Additional workshops are needed to provide librarians with skills for actually measuring whether the outcomes set in the logic model are being achieved. In the end, each member of the library staff who participates in the workshop will have the ability to assess and evaluate their own project and program which, in turn, establishes and reinforces the culture of assessment.

Acknowledgements

This article would not be complete without acknowledging the guidance and support of several colleagues at the University of Illinois at Chicago Library: Sandra De Groote, Linda Naru, Paula Dempsey, Gwen Gregory, Glenda Insua and Carl Lehnen.

References

Abdi, S., & Mensah G. (2016). Focus On: Logic models-a planning and evaluation tool. Toronto: Queen’s Printer for Ontario. Retrieved from https://www.publichealthontario.ca/en/Pages/default.aspx

American Library Association (2017, January). ACRL Proficiencies for Assessment Librarians and Coordinators, Retrieved from http://www.ala.org/acrl/standards/assessment_proficiencies, Document ID: 498fde98-4c4b-fc94-a5f7-9b6decd8a2e9

Community Tool Box (n.d), chapter 2. Other models for promoting community health and development. Retrieved from https://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development

Consortium of Academic and Research Libraries in Illinois (CARLI) (2019), CARLI Counts overview. Retrieved from https://www.carli.illinois.edu/products-services/carli-counts

Dubicki, E. (2011). Strategic planning in college libraries. [Chicago]: College Library Information Packet Committee, College Libraries Section, Association of College and Research Libraries.

Institute of Museum and Library Services (n.d). Retrieved from https://www.imls.gov/research-evaluation/evaluation-resources

Kletter, M., Mendelez-Torres, G. J., Lilford, R., & Taylor, C. (2018). A Library of logic models to explain how interventions to reduce diagnostic errors work. Journal of Patient Safety, 00(00), 1. https://doi.org/10.1097/pts.0000000000000459

Markless, S., & Streatfield, D. (2017). How can you tell if it’s working? Recent developments in impact evaluation and their implications for information literacy practice. Journal of Information Literacy, 11(1), 106–119. https://doi.org/10.11645/11.1.2201

Stoddart, R., & Lajoie, E. W. (2014). The possibilities are assessable: Using an evidence based framework to identify assessment opportunities in library technology departments. Evidence Based Library and Information Practice, 9(4), 29–43. https://doi.org/10.18438/B8MK6G

Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide. Retrieved from https://fyi.extension.wisc.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf

W.K. Kellogg Foundation (2006). W.K. Kellogg Foundation Logic Model Development Guide. Retrieved from https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide