EDITORIAL | TRuITT 3 Marc TruittEditorial W elcome to 2009! It has been unseasonably cold in Edmonton, with daytime “highs”—I use the term loosely— averaging around -25°C (that’s -13°F, for those of you ITAL readers living in the States) for much of the last three weeks. Factor in wind chill (a given on the Canadian Prairies), and you can easily subtract another 10°C. As a result, we’ve had more than a few days and nights where the adjusted temperature has been much closer to -40°, which is the same in either Celsius or Fahrenheit. While my boss and chief librarian is fond of saying that “real Canadians don’t even button their shirts until it gets to minus forty,” I’ve yet to observe such a feat of derring-do by anyone at much less than twenty below . Even your editor’s two Labrador retrievers—who love cooler weather—are reluctant to go out in such cold, with the result that both humans and pets have all been coping with bouts of cabin fever since before Christmas. n So, when is it “too cold” for a server room? Why, you may reasonably ask, am I belaboring ITAL readers with the details of our weather? Over the week- end we experienced near-simultaneous failures of both cooling systems in our primary server room (SR1), which meant that nearly all of our library IT services, including our OPAC (which we host for a consortium of twenty area libraries), a separate OPAC for Edmonton Public Library, our website, and access to licensed e-resources, e-mail, files, and print servers had to be shut down. Temperature readings in the room soared from an average of 20–22°C (68–71.5°F) to as much as 37°C (98.6°F) before settling out at around 30°C (86°F). We spent much of the weekend and beginning of this week relocating servers to all man- ner of places while the cooling system gets fixed. I imag- ine that next we may move one into each staff person’s under-heated office, where they’ll be able to perform double duty as high-tech foot warmers! All of this happened, of course, while the temperature outside the building hovered between -20° and -25°C. This is not the first time we’ve experienced a failure of our cooling systems during extremely cold weather. Last winter we suffered a series of problems with both the systems in SR1 and in our secondary room a few feet away. The issues we had then were not the same as those we’re living through now, but they occurred, as now, at the coldest time of the year. This seeming dichotomy of an overheated server environment in the depths of winter is not a matter of accident or coincidence; indeed, while it may seem counterintuitive, the fact is that many, if not all, of our cooling woes can be traced to the cold outside. The simple explanation is that extreme cold weather stresses and breaks things, including HVAC systems. As we’ve tried to analyze this incident, it appears likely that our troubles began when the older of our two systems in SR1 developed a coolant leak at some point after its last preventive maintenance servicing in August. Fall was mild here, and we didn’t see the onset of really severe cold weather until early to mid-December. Since the older system is mainly intended for failover of the newer one, and since both systems last received routine service recently, it is possible that the leak could have developed at any time since, although my supposition is that it may be itself a result of the cold. In any case, all seemed well because the newer cool- ing system in SR1 was adequate to mask the failure of the older unit, until it suffered a controller board failure that took it offline last weekend. But, with the failure of the new system on Saturday, all IT services provided from this room had to be brought down. After a night spent try- ing to cool the room with fans and a portable cooling unit, we succeeded in bringing the two OPACs and other core services back online by Sunday, but the coolant leak in the old system was not repaired until midday Monday. Today is Friday, and we’ve limped along all week on about 60 percent of the cooling normally required in SR1. We hope to have the parts to repair the newer cooling system early next week (fingers crossed!). Some interesting lessons have emerged from this incident, and while probably not many of you regularly deal with -30°C winters, I think them worth sharing in the hope that they are more generally applicable than our winter extremes are: 1. Document your servers and the services that reside on them. We spent entirely too much time in the early hours of this event trying to relate servers and ser- vices. We in information technology (IT) may think of shutting down or powering up servers “Fred,” “Wilma,” “Betty,” and “Barney,” but, in a crisis, what we generally should be thinking of is whether or not we can shut down e-mail, file-and-print ser- vices, or the integrated library system (ILS) (and, if the latter, whether we shut down just the underlying database server or also the related staff and public services). Perhaps your servers have more obvious names than ours, in which case, count yourself for- tunate. But ours are not so intuitively named—there is a perfectly good reason for this, by the way—and with distributed applications where the database Marc Truitt (marc.truitt@ualberta.ca) is Associate Director, Bibliographic and Information Technology Services, University of Alberta Libraries, Edmonton, Alberta, Canada, and Editor of ITAL. 4 INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2009 may reside here, the application there, and the Web front end yet somewhere else, I’d be surprised if your situation isn’t as complex as ours. And bear in mind that documentation of dependencies goes two ways: Not only do you want to know that “Barney” is hosting the ILS’s Oracle database, but you also want to know all of the servers that should be brought up for you to offer ILS–related services. 2. Prioritize your services. If your cooling system (or other critical server-room utility) were sud- denly only operating at 50 percent of your normal required capacity, how would you quickly decide which services to shut down and which to leave up? I wrote in this space recently that we’ve been thinking about prioritized services in the context of disaster recovery and business continuity, but this week’s incident tells me that we’re not really there yet. Optimally, I think that any senior member of my on-call staff should be empowered in a given critical situation to bring down services on the basis of a predefined set of service priorities. 3. Virtualize, virtualize, virtualize. If we are at all typi- cal of large libraries in the Association of Research Libraries (and I think we are), then it will come as no surprise that we seem to add new services with alarming frequency. I suspect that, as with most places, we tend to try and keep things simple at the server end by hosting new services on sepa- rate, dedicated servers. The resulting proliferation of new servers has led to ever-greater strains on power, cooling, and network infrastructures in a facility that was significantly renovated less than two years ago. And I don’t see any near-term likeli- hood that this will change. We are, consequently, in the very early days of investigating virtualization technology as a means of reducing the number of physical boxes and making much better use of the resources—especially processor and RAM— available to current-generation hardware. I’m hop- ing that someone among our readership is farther along this path than we and will consider submit- ting to ITAL a “how we done it” on virtualization in the library server room very soon! 4. Sometimes low-tech solutions work . . . No one here has failed to observe the irony of an overheated server room when the temperature just steps away is 30° below. Our first thought was how simple and elegant a solution it would be to install duct- ing, an intake fan, and a damper to the outside of the building. Then, the next time our cooling failed in the depths of winter, voila!, we could solve the problem with a mere turn of the damper control. 5. . . . and sometimes they don’t. Not quite, it seems. When asked, our university facilities experts told us that an even greater irony than the one we currently have would be the requirement for Can$100,000 in equipment to heat that -30°C outside air to around freezing so that we wouldn’t freeze pipes and other indoor essentials if we were to adopt the “low-tech” approach and rely on Mother Nature. Oh, well . . . n In memoriam Most of the snail mail I receive as editor consists of advertisements and press releases from various firms providing IT and other services to libraries. But a few months ago a thin, hand-addressed envelope, post- marked Pittsburgh with no return address, landed on my desk. Inside were two slips of paper clipped from a recent issue of ITAL and taped together. On one was my name and address; the other was a mailing label for Jean A. Guasco of Pittsburgh, an ALA Life Member and ITAL subscriber. Beside her name, in red felt-tip pen, someone had written simply “deceased.” I wondered about this for some time. Who was Ms. Guasco? Where had she worked, and when? Had she published or otherwise been active professionally? If she was a Life Member of ALA, surely it would be easy to find out more. It turns out that such is not the case, the wonders of the Internet notwithstanding. My obvious first stop, Google, yielded little other than a brief notice of her death in a Pittsburgh-area newspaper and an entry from a digi- tized September 1967 issue of Special Libraries that identi- fied her committee assignment in the Special Libraries Assocation and the fact that she was at the time the chief librarian at McGraw-Hill, then located in New York. As a result of checking WorldCat, where I found a listing for her master’s thesis, I learned that she graduated from the now-closed School of Library Service at Columbia University in 1953. If she published further, there was no mention of it on Google. My subsequent searches under her name in the standard online LIS indexes drew blanks. From there, the trail got even colder. McGraw-Hill long ago forsook New York for the wilds of Ohio, and it seems that we as a profession have not been very good at retaining for posterity our directories of those in the field. A friend managed to find listings in both the 1982–83 and 1984–85 volumes of Who’s Who in Special Libraries, but all these did was confirm what I already knew: Ms. Guasco was an ALA Life Member, who by then lived in Pittsburgh. I’m guessing that she was then retired, since her death notice gave her age as eighty-six years. Of her professional career before that, I’m sad that I must say I was able to learn no more.