College and Research Libraries MEREDITH BUTLER AND BONNIE GRATCH Planning a User Study- The Process Defined Library literature is replete with articles describing library user studies. How- ever, little has been written about the planning process necessary to undertake a tas.k ~complex~ a user.stud¥ that measures the effectiveness of an aca- demtc ltbrary s servtces. Thts arttele focuses on the planning process that was used .by the staff of a college.library to develop and implement a user study. Outltned here are the planmng steps necessary to coordinate disparate staff concerns and interests to undertake a complex study and accomplish stated objectives in a specified time. THE LITERATURE Planning is certainly not a new concept to the library profession, but it is a concept of increasing importance to today's rapidly changing academic libraries. However, even a cursory glance at the literature of library planning makes it painfully obvious to there- searcher that literature on planning is still in its infancy in the library profession. Bibliographic control of the literature on planning for libraries is problematical. Sub- ject headings, such as "Library Planning," lead most frequently to literature on the planning of library buildings. More general headings relating to library administration and management or planning library ser- vices are too broad to be satisfactory ap- proaches when one wants information on planning theory and models. The researcher on this topic must rely heavily on the litera- ture of systems analysis and planning theory from business and other related disciplines in the social sciences, and be prepared to do a lot of reading and sifting. Many librarians have found the literature relating to planning in academic and re- search libraries issued by the Association of Research Libraries and the Council on Li- Meredith Butler is assistant director for plan- ning, State University of New York at Albany, and Bonnie Gratch is reference librarian , State Univer- sity of New York College at Brockport. 320 I brary Resources extremely helpful. This lit- erature tends to focus on library self-studies, management self-analysis, and developing goals and objectives. The self-studies present a planning model and offer the expert assis- tance of professional consultants trained to facilitate the planning process.l Other librarians have looked to the litera- ture of business, operations research, and sys- tems theory to provide planning and evalua- tion models. One excellent article that appeared several years ago in library litera- ture is McClure's article on planning as a strategy for action. 2 A third area of planning literature con- cerns public library planning. Palmour's handbook3 has relevance to the academic li- brarian and is the most recent in a longtradi- tion of planning manuals for the public sec- tor. Bibliographic control of the literature on library surveys and user studies is even more problematic to the researcher. The literature is too enormous to be tackled by any one re- searcher, as evidenced by numerous bibliog- raphies and reviews devoted to these topics in Library Literature, Information Science Ab- stracts, etc. Terms such as "survey," "user study," and "evaluation study" are fre- quently used interchangeably, and no dis- tinction is made in subject headings and bib- liographic citations between types of user studies or points of view. Research that ex- plores the use made of an information system is lumped together with research on users and their characteristics. In addition, the bulk of the literature is descriptive rather than analytic and therefore of limited use to the library planner. Zweizig4 •5 •6 and Dervin 7 have pointed out the deficiencies of the current literature on user studies and have suggested research al- ternatives of interest to the library planner. The task of this article is to present a basic planning model, apply it to a planning activity- that of developing a user study- and detail the process involved. Of the wealth of material on planning and user studies reviewed, cited here are only those materials that relate specifically to the plan- ning process of a library survey or a user study. THE PLANNING MoDEL The specific elements of a planning process may differ from model to model; but the ba- sic stages of planning are readily identifiable, relatively similar from model to model, and can be applied to the study of any library problem. All of these stage!:! can be taken with or without benefit of outsid~ consul- tants, options that are not always available to many smaller academic libraries. Planning a User Study I 321 Perhaps the basic planning model is best visualized as similar to a general model of the scientific method of inquiry found in any ele- mentary science or social science text. The left column of figure 1 (Busha and Harter) 8 illustrates the steps involved in the scientific method. On the right are the roughly equiva- lent stages of planning which were used in the planning process of the present study. As is apparent from figure 1, the process of inquiry is a regenerative process in which in- formation derived contributes to the genera- tion of new hypotheses that must be re- searched and tested. The creativity of this process comes in the early stages of problem definition and the design of an appropriate methodology to address the problem. The more mechanical tasks of gathering and ana- lyzing the data and reporting results neces- sarily follow. However, no analyses, no mat- ter how sophisticated, can make up for a badly formulated hypothesis or a poorly de- signed methodology. THE PLANNING PROCESS Illustrated now will be the application of this general planning model through a dis- cussion of planning as a process that provides a structure for problem-solving activities in a Stages of planning l. Problem definition 2. Goal identification & needs assessment 3. Definition of the scope and objectives of pro- gram a. Budget projection 4. Literature review 5. Program development a. Strategy b. Tactics c. Testing d. Implementation 6. Program evaluation a. Analysis b. Publication 7. Redefinition of problem and identification of new research areas Fig.l Model of the Scientific Method 322 I College & Research Libraries • july 1982 library. An example of a complex activity- the development and implementation of a user study- has been used to illustrate and highlight certain aspects of the planning pro- cess. Outlined here are the planning steps necessary to coordinate disparate staff con- cerns and interests in order to undertake a complex study and accomplish stated objec- tives in a specified time. Experience and insight form the founda- tion of any well-conceived plan. The experi- enced planner incorporates these into a flexi- ble but goal-directed process in which organizational goals and objectives are trans- lated into plans of action. The resulting ser- vice programs are evaluated by obtaining measurable evidence of the extent to which organizational objectives and provided ser- vices correlate .with user needs. All stages of planning- problem definition, goal identifi- cation, needs assessment, program develop- ment, program evaluation, etc.- must be present before talk of a planning process can begin. This type of planning is not an end in itself, but a means of achieving an end- a continuous process requiring constant reap- praisal as conditions, needs, and services change. Three major phases may be identified in any planning process: 1. The normative phase, in which the main activities are to identify the problem and justify the need for a study; to diagnose and define the objectives of the study- first in general terms, then in more specific terms; and to specify alternative methodologies and options. 2. The operational phase, in which a strategy to ensure the best use of resources is developed; the methodology is determined; and data are gathered and analyzed. Issues to be considered in this phase include: a. budget availability b. personnel c. limitations of time, money, access, etc. d. public and staff receptivity e. available expertise f. available resources, e.g., computer- based data analysis and printing fa- cilities g. political considerations 3., The evaluative phase, in which results are analyzed and reported; costs are com- pared to benefits derived; results are com- pared to program objectives and adjustments are made. The above process will now be illustrated through an examination of the planning of a user study by a group of librarians, library ~taff, and faculty at a medium-sized college library. The library of the State University of New York College at Brockport is part of a rela- tively new institution serving approximately 10,000 students and faculty. It is a very ac- tive and busy library which enjoys a good re- lationship with its clientele and has been rea- sonably well supported by the college over the past decade of rapid growth. Forty-eight staff members (eighteen librarians) provide the basic services one has come to expect from an academic library, as well as an ambitious program of bibliographic instruction. IDENTIFYING PROBLEM, JusTIFYING NEED, DEFINING OBJECTIVES Perhaps the first question to ask when be- ginning a planning effort is why. In this case, Why do a user study? The answer to that question is critical and provides the raison d'etre for all subsequent planning activity. In general terms, the purpose of a user study is similar to market research in busi- ness. It is an attempt to discover patterns of use and levels of awareness of users toward li- brary services, to determine success or non- success of services, and to identify what ad- justments are needed in service strategy. Martin's article, "User Studies and Library Planning," describes several kinds of use studies and attempts to appraise what they contribute to the planning process. 9 He labels the most common study "who, what, when investigations," and judges them beneficial within limitations of a clearly identified need. Their results can directly facilitate the decision-making process. Martin states that studies of how libraries are used (and with what success rate) tend to be less frequent, because they are complex and time- consuming. Evaluative studies that look at why libraries are used are rarer still, because they fall into the category of experimental rather than survey research. Line defines one type of user study, the survey, as "a systematic co1lection of data concerning a. library, its activities, opera- tions, staff, use and users, at a given time." A survey gives an overview, "a map rather than a detailed plan," and it may be motivated by the need to improve, to provide information, to bring complex situations under control, or to satisfy curiosity .1o Specifically, the staff at SUNY Brockport Library lacked data about who used the li- brary, for what purposes, and to what extent their needs were satisfied. The staff needed information about users to assist in the analy- . sis of problems, and wanted to support budget requests with statistical measures of outcomes of services. The library was enter- ing an extended period of financial con- straints in which the college would compare every resource allocation to benefit derived. In addition, the college was anticipating .a required self-study and self-evaluation in the near future. The library could not afford to be passive. The library staff decided to seek out the users .and question them about the services offered; to test hypotheses based on experience, in- sight, and observation; to provide data for the college's self-study; and to provide data to help the library make decisions about budget and service cuts, should such cuts be- come necessary in the future. The plan was for a modest beginning, to develop a "who, what, when" survey. Once the justification for a survey is de- fined and the type of study and the focus are clearly identified, the normative phase of the process moves from the general to the spe- cific. In this stage, it is necessary to analyze in precise terms what the specific objectives of the survey are and what costs will be and what the expected outcomes are. Alternative methods to accomplish the stated objectives must also be identified and given full consid- eration. DETERMINING BuDGET The importance of a budget as a planning document cannot be overemphasized. 11 Whether one develops an all-inclusive budget that gives details of both direct and indirect costs (e.g., cost of committee mem- bers' time, overhead, etc.) or narrows the scope to include only the direct costs of pa- per, printing, mailing, computer time, or student assistance, these costs must be care- fully identified and projected before further Planning a User Study I 323 planning takes place. The availability or lack of money is a key consideration in determin- ing what methodology will be used and other critical decisions in the planning process. Budget limitations made it necessary for the authors to narrow the scope of the original survey plans and to rely on "free advice" from knowledgeable campus faculty. Once plans were formulated and costs projected, money was requested from the library ad- ministration. Plans were modified in re- sponse to the budget allocated. SETTING UP THE WoRK GROUP OR CoMMITTEE Because the major objectives of this study were administrative in nature (i.e., focused on measuring effectiveness of service rather than learning more about the information- gathering behavior of users or user perfor- mance in our library), it was important to supplement administrative input with points of view from various levels of staff and a vari- ety of service departments. Because the task was complex and time-consuming, a special committee was created. Its charge was to de- velop, implement, analyze, and evaluate a survey of library users within a ten-month period. The composition of a committee, identifi- cation of its charge, and determination of its reporting relationships and timetables are absolutely critical to the success of any plan- ning process, and careful thought must be given to each of these considerations before potential committee members are ap- proached. Committee members for this study were chosen for their interest and ex- pertise in the subject, their professional com- patibility, and their reputation for successful completion of work. The authors also tried to represent the diverse interests of various li- brary departments. Leadership must be pro- vided by someone with good leadership skills as well as knowledge of communication and group-dynamics skills. It is important to have a balance between people who are "product" oriented ("let's get the job done") and people who are "process" oriented ("let's get agreement on this issue before we move to the next item"). The committee's role in planning should be clearly spelled out, both to committee members and to the rest of the library staff. Discussion and publicity about the commit- 324 I College & Research Libraries· July 1982 tee work should begin at the formative stage. A realistic discussion and appraisal of the complexities of the task ahead and what might be expected is essential in the early planning stages. Martin warns that "when the question is raised concerning what the li- brary wants to know about its users, and how the information will be applied to planning, ambiguities appear. " 12 After all, "user needs" are hard to define, even by the user, and user "interests" are even more nebulous. Edwin E. Williams summed it up nicely when he said of surveying, I was sometimes discouraged by the complexities of surveying and the questions it raises. Recalling the hitherto competent centipede who became help- lessly entangled when asked how he managed to coordinate so many legs, I wondered if I were be- coming so conscious of the complexities that I should never be able to do any more surveying.l3 Problems arise from the inexperience of the committee, the members of which may never have done any survey research; from expectations of both library staff and users about what the survey will do; and from the limitations of eliciting information from hu- man beings and all the complexities inherent in that process. The measurement of atti- tudes is extremely complex and the limitation and potentialities of the task must be dis- cussed frankly from the beginning if the com- mittee is to become a cohesive work group. DEFINING THE ScoPE AND OBJECTIVES OF THE STUDY The committee must analyze in precise terms what information it wants to find out, what information it can realistically find out within the constraints of the study, and what it will do with the information. It must plan the survey before formulating specific ques- tions to be asked of users. As Wood points out in his article on user surveys, knowing what you want to find out and what you already know will help you determine "the scale of the investigation, the method to be used, the timing of the study, and the type of questions to be asked or the type of information to be recorded. "14 Once the committee defines the general hypothesis and specific problem statement, it can begin to identify the objectives of the study. Is the objective to measure work load or user satisfaction? Will the survey be lim- ited to library users, or will it include nonus- ers? How large is the population to be sur- veyed, and what are the possibilities and limitations the committee must consider? Decisions about point of view (administra- tive, user, etc.), depth of detail, and popula- tion (user, nonuser) must be made and ad- hered to. All of these decisions will help focus the task and will force a careful review of op- tions and limitations. After the objectives of the study are devel- oped; a certain amount of background work remains to be done to help develop a frame- work. The committee must: 1. Identify what data about the organiza- tion and its services are currently available and what is known through experience and observation. Communication with col- leagues is critical at this stage and will save much time and work. 2. Review the literature and look for models of similar projects. Seek guidance from colleagues at the home institution or at other institutions who have experience in similar projects. Identifying a guide or model can save enormous amounts of work and may contribute to the research validity of the results. 3. Pull together institutional mission and goal statements and develop a hierarchy of goals as a basis for the development of, and later, the evaluation of, the survey. Remem- ber, performance will be measured against stated goals and objectives. It is very helpful to involve other appropriate staff in this goal- identification step to ensure a valid outcome and gain acceptance of and publicity about the committee's work. 4. Identify campus policies regarding re- search on human subjects and integrate ap- propriate procedures in the planning. Such policies may place restrictions on activities and will have to be taken into account from the beginning. Only when the objectives are clearly delineated and referenced to the goals of the organization and its various service units is the committee ready to move from the normative phase to the operational phase. DEVELOPING A PLANNING CHART At this stage of planning, or possibly even earlier, it is useful to develop a planning chart and timetable on which all activities to be performed are listed and responsibilities and deadlines are assigned. A Gantt chart, in which time is plotted on the horizontal axis for each part of a project, allows for indica- tion of progress at regular intervals and pro- vides a comprehensive illustration of the pro- cess as a whole. 15 Things to be done are entered in symbols (as shown in figure 2 be- low), and descriptions may be included un- der the portions of the calendar in which the activity is planned. New work can be added to take precedence over work already planned, and no erasure is necessary. The obvious advantages of a planning chart are that it makes a definite plan for Date Activities Assigned to Begun Define objectives Dept. heads 3/7/79 of each service dept. 1. Circ. J.B. II 2. Reserve R.L. II 3. ILL K.W. II 4. Reference, etc. S.P. II Identify and Cormnittee 3/14/79 collect presently members available service (list) statistics Literature revie1v 3/14/79 Develop budget, etc. Cormnittee chairperson working w/ Asst. Dir. Planning a User Study I 325 each project necessary, and that it forces the "thinking through" of things that will be en- countered and must be provided for. This type of chart both illustrates simultaneous activities and allows for the addition of steps that may have been overlooked or that may develop while the process is under way. DETERMINING THE METHODOLOGY AND SAMPLING DESIGN The planning committee is now ready to consider the methodology and design of their proposed survey. An effective and practical way to look at various methodologies is to weigh the advantages and disadvantages of a Month).,..~ Month ~ i'