College and Research Libraries By J O H N A L B E R T S A N F O R D and F R E D E R I C K R. T H E R I A U L T Problems in the Application of Uniterm Coordinate Indexing Dr. Sanford is deputy chief, Technical Library Division, National Security Agency, and Dr. Theriault is chief of the documentation branch. T' H E L I B R A R Y of the National Security Agency has completed the organiza- tional and experimental work necessary for the creation of a large-scale Uniterm coordinate index. Production is now on a r o u t i n e basis. Over 70,000 documents have been cataloged. T h i s report is writ- ten at this time to make our experience available to other librarians who may be considering the use of this system. W e wish we could answer all the ques- tions that have been raised about co- ordinate indexing in the literature. Many earnest librarians with very con- siderable professional experience have been deeply troubled by its potential pit- falls. Perhaps we have been very lucky. Perhaps the pitfalls will vanish in any other large-scale test. W e d o not know. W e can report only that our system works. W e do not know of any other means to gain such tight control of large masses of documents so economically and rapidly. T h e r e have been problems, however; a n d some of them were formidable. O u r version of the Uniterm system of coor- dinate indexing is certainly not the last word in desirable development. It con- tains some of the whimsical invention a n d much of the rough-and-ready crudi- ties of Henry Ford's Model " T " auto- mobile. But, like the Model " T , " it runs. As will be a p p a r e n t in the account that follows, we have had to introduce many adaptations and changes of the system as originally outlined in the literature. Since we were pioneering, matching our wits against the new system day by day has been challenging. W e hope that in solving our own problems we have made a contribution to the developing science of documentation. F o r a n y r e a d e r u n f a m i l i a r w i t h t h e U n i t e r m s y s t e m of c o o r d i n a t e i n d e x i n g , t h e s c h e m e is c o n c e i v e d a s f o l l o w s : T h e i d e a s p r e s e n t e d i n t h e t i t l e of a d o c u - ment, plus additional ideas embodied in the text if t h e t i t l e is n o t s u f f i c i e n t l y d e - s c r i p t i v e , a r e b r o k e n u p i n t o s e p a r a t e w o r d s , d u b b e d " u n i t e r m s . " T h e d o c u - m e n t is a s s i g n e d a n a r b i t r a r y n u m b e r . A 5 " x 8 " m a s t e r i n d e x c a r d is p r e p a r e d f o r e a c h U n i t e r m , a n d t h e n u m b e r as- s i g n e d t o a d o c u m e n t is r e g i s t e r e d o n a l l of t h e U n i t e r m c a r d s t h a t d e s c r i b e t h a t d o c u m e n t . T h u s , t h e U n i t e r m c a r d b e a r - i n g t h e h e a d i n g C A T A L O G w i l l h a v e i n - s c r i b e d o n i t t h e d o c u m e n t n u m b e r o f e v e r y r e p o r t h a v i n g a n y t h i n g of m o m e n t t o d o w i t h c a t a l o g s o r c a t a l o g i n g . A d o c - u m e n t t h a t is a c a t a l o g of s p a r e p a r t s f o r a u t o m o b i l e w i n d s h i e l d w i p e r s w i l l h a v e i t s n u m b e r r e c o r d e d o n e a c h of t h e f o l - l o w i n g c a r d s : C A T A L O G , A U T O M O B I L E , W I N D S H I E L D , W I P E R , P A R T S . T O f i n d t h i s d o c u m e n t , t h e a b o v e c a r d s a r e c o m - p a r e d . W h e r e v e r t h e i d e n t i c a l n u m b e r a p p e a r s o n t w o o r m o r e c a r d s , t h a t n u m - b e r r e p r e s e n t s a d o c u m e n t w h e r e i n t h e i d e a s i n t e r s e c t , i.e., c o o r d i n a t e . W e thought our first problem was t h e creation of a list of these uniterms for our subject matter. Here, however, o u r experience recommends three immedi- JANUARY, 1956 19- ate departures f r o m the system as pro- posed by Documentation, Inc. 1. Let the documents themselves gen- erate their own uniterms. Catalog 1,000 documents. T h e y will produce about 1,000 uniterms. Weed this list carefully, combining synonyms. W i t h this core, catalog another 1,000 reports, using the "approved" basic list where possible, and then repeat the weeding operation. Once some 8,000 uniterms have been chosen, the rate of addition falls off very rapid- ly, even in highly varied subject matter. T h e curve begins to grow nearly flat when between 40,000 a n d 50,000 docu- ments have been cataloged. T h e useful limits of a Uniterm vocabulary are so soon reached that above 10,000 terms only highly specific items, such as new trade names a n d e q u i p m e n t designa- tions, need to be added. 2. Forget all about "free" and " b o u n d " terms as set f o r t h in the literature of the subject. " B o u n d " terms almost inevita- bly free themselves sooner or later, and the intermediate step serves only to make extra work. Multiple words, how- ever, should be used for exact descrip- tion of concepts, wherever the idea ex- pressed is a unit. 3. F r o m t h e s t a r t , u s e see a n d see also r e f e r e n c e s o n t h e h e a d i n g s of U n i t e r m c a r d s , i n a c c o r d a n c e w i t h s t a n d a r d li- b r a r y p r a c t i c e . W e h a v e f o u n d n o o t h e r s a t i s f a c t o r y s o l u t i o n f o r p r o b l e m s of n e a r s y n o n y m s , f o r s y n o n y m s - i n - s o m e - m e a n - i n g s of w o r d s , a n d f o r a l l t h e o t h e r p e r - p l e x i t i e s b o r n of t h e f a c t t h a t U n i t e r m c o o r d i n a t e i n d e x i n g u s e s t h e l i v i n g f a b - r i c of l a n g u a g e f o r i t s b a s e . O u r next problem was the discovery that we needed to develop satellite in- dexes around our coordinate index. Here the needs of libraries will n o d o u b t dif- fer, b u t we soon f o u n d that in the co- ordinate index we were building a heavy-duty machine unsuitable for light work. We decided to employ traditional library methods for all types of docu- ment reference questions they served best, a n d to t u r n to the coordinate index where traditional methods broke down when laden with the peculiar burdens which documents engender. T h e com- bination of old a n d new methods has turned out to be an unexpectedly har- monious arrangement. Problems of work flow came next. O u r system as it finally evolved repre- sents the solution of a series of prob- lems in practical operation and hence may be of interest. Because the approach lends itself so readily to r a p i d process- ing, attention paid to "time a n d mo- tion" pays large dividends in total pro- duction. O u r basic requirements per- haps differ little f r o m those of a great many libraries: A collection of at least 200,000 tech- nical reports a n d other documents need- ed improved information control. T h e y were scattered t h r o u g h o u t the organiza- tion in several dozen informal collec- tions. A good many individual office files also needed to be surveyed. Each collection, small and large, had been or- ganized according to a scheme chosen by its compilers. No professionally built catalogs existed. Large numbers of dup- licate copies of reports were known to be wasting file space among various collec- tions. T h e total n u m b e r of reports to be processed probably approached one mil- lion. T h e task was to weed out the dupli- cates, select items from the r e m a i n i n g originals which were worth keeping, and to create an index for them without as- sembling a definitive central collection. A central index was desired but a central file could not be contemplated: among the wealth of duplicates, too many items were u n i q u e and were required for f r e q u e n t reference use in the depart- ments then holding them. O u r organization is built on four teams of three members each, with a support group of twelve people located at the central cataloging department. T h r e e teams operate in the "field," visit- 20 COLLEGE AND RESEARCH LIBRARIES i n g a n y d e s i r e d c o m p o n e n t of t h e a g e n - cy's o r g a n i z a t i o n a n d c a t a l o g i n g i t s d o c - u m e n t s . T h e f o u r t h , a " h o m e " t e a m , o p - e r a t e s i n t h e c e n t r a l d e p a r t m e n t i n as- s o c i a t i o n w i t h t h e s u p p o r t g r o u p , a n d is r e s p o n s i b l e s o l e l y f o r c a t a l o g i n g n e w l y i s s u e d r e p o r t s . E a c h t e a m h a s a " l e a d e r " w h o is r e s p o n s i b l e f o r i t s e n t i r e , i n d e - p e n d e n t o p e r a t i o n , i n c l u d i n g p u b l i c r e - l a t i o n s w i t h t h e p e o p l e w h o s e files h e is o r g a n i z i n g . H e is a l s o t h e U n i t e r m c a t a - l o g e r f o r h i s t e a m . H e is a s s i s t e d b y a d e s c r i p t i v e c a t a l o g e r a n d b y a c l e r k . A t t h e b e g i n n i n g of t h e o p e r a t i o n p e r - f o r m e d o n a n y file d r a w e r of d o c u m e n t s , t h e c l e r k of t h e t e a m c o p i e s o n l y t h e ti- t l e s i n i n f o r m a l l i s t s . O n c e d a i l y h e r e - t u r n s t o c h e c k t h e s e a g a i n s t t h e a u t h o r i - t y t i t l e f i l e i n t h e c e n t r a l c a t a l o g d e p a r t - m e n t . D u p l i c a t e s of d o c u m e n t s a l r e a d y c a t a l o g e d a r e n o t e d . U p o n r e t u r n i n g t o h i s t e a m , t h e c l e r k r u b b e r - s t a m p s t h e s e i t e m s " D u p l i c a t e C o p y . " H e n c e f o r w a r d , t h e s e m a y b e d e s t r o y e d w i t h c o n f i d e n c e w h e n n o l o n g e r n e e d e d l o c a l l y . T h e r e - m a i n d e r a r e s t a m p e d " R e c o r d C o p y " w i t h a s p a c e p r o v i d e d f o r r e g i s t e r i n g a p e r m a n e n t i n d e x n u m b e r . T h e s e o r i g - i n a l s p r o v i d e t h e t e a m w i t h t h e m a t e r i a l f o r t h e e n s u i n g d a y ' s w o r k . B e c a u s e d e s k s p a c e is u s u a l l y l i m i t e d i n t h e office b e i n g v i s i t e d , e a c h t e a m is r e s t r i c t e d t o o n e t y p e w r i t e r , n o r m a l l y o p e r a t e d b y t h e d e s c r i p t i v e c a t a l o g e r . T h e d e s c r i p t i v e c a t a l o g i n g is p e r f o r m e d d i r e c t l y o n f a n f o l d s . B e c a u s e of t h e to- t a l n e e d s of t h e s y s t e m , t h e p r o c e s s is s i m p l e a n d s w i f t . W e r e c o r d (a) t i t l e , ( b ) c o r p o r a t e a u t h o r , (c) p e r s o n a l a u - t h o r , ( d ) s e r i e s n u m b e r , (e) c o n t r a c t n u m b e r , if a n y , (f) c o l l a t i o n . N o t r a c - i n g s , s u b j e c t h e a d i n g s , o r o t h e r t i m e - c o n s u m i n g n o t a t i o n s a r e r e q u i r e d . T h e y a r e c a r e d f o r , u s i n g s i m p l e s h o r t c u t s , e l s e w h e r e i n t h e s y s t e m . D o c u m e n t a n d f a n f o l d a r e t h e n p a s s e d t o t h e t e a m l e a d e r , w h o v e r i f i e s t h e ac- c u r a c y of t h e d e s c r i p t i v e c a t a l o g i n g . H e s c a n s , s t u d i e s , o r d i s s e c t s t h e d o c u m e n t as i t s i m p o r t a n c e o r d i f f i c u l t y s e e m s t o r e q u i r e , a n d w r i t e s o u t i n l o n g h a n d i n a s p a c e p r o v i d e d o n t h e f a n f o l d a l l t h e u n i t e r m s h e b e l i e v e s t h e d o c u m e n t r e - q u i r e s f o r i n d e x i n g " i n d e p t h . " T h i s m e a n s t h a t h e a t t e m p t s t o r e c o r d all of t h e s u b j e c t s c o n c e r n i n g w h i c h t h i s d o c - u m e n t c o u l d c o n c e i v a b l y b e u s e f u l . A l - w a y s , if t h e d o c u m e n t c o n c e r n s s o m e s u b o r d i n a t e t o p i c — a p a r t of a l a r g e r m a c h i n e , a s t e p i n a p r o c e s s — t h e n e x t l a r g e r c o n c e p t is s e t d o w n as t h e first U n i t e r m . T h e n c o m e a l l t h e w o r d s t h a t a n s w e r t h e c l a s s i c r e p o r t e r ' s q u e s t i o n s : " W h o ? " " W h a t ? " " W h y ? " " W h e n ? " " W h e r e ? " " H o w ? " T h e n , u n i t e r m s t o c o v e r a n y i d e a s g i v e n s p e c i a l t r e a t m e n t i n t h e d o c u m e n t o r w h i c h a r e i m p o r t a n t i n t h e c o n c l u s i o n s . T h e t e a m l e a d e r is n o t a f r a i d t o s c r i b b l e o u t a l o n g l i s t . H e k n o w s (a) t h a t t h e e n s u i n g p r o c e s s e s i n i n d e x i n g t h e s e t e r m s i n t o t h e s y s t e m a r e so e c o n o m i c a l of t i m e t h a t i t is d e s i r a - b l e t o e r r o n t h e s i d e of o v e r - c o m p l e t e - n e s s , a n d ( b ) t h a t o n a n y w e e k ' s w o r k h i s l i s t s of u n i t e r m s w i l l a v e r a g e n i n e t e r m s p e r d o c u m e n t . H a v i n g f i n i s h e d h i s l i s t , h e e x a m i n e s i t c r i t i c a l l y . T h e b e s t t e s t w e h a v e f o u n d f o r g o o d u n i t e r m i n g is t h i s : D o t h e w o r d s , r e a d c o n s e c u t i v e l y , c o m e c l o s e t o f o r m i n g a c o m p l e t e a n d i n t e l l i g i b l e s e n - t e n c e ? If so, n o e s s e n t i a l h a s p r o b a b l y b e e n o m i t t e d . N e x t q u e s t i o n : D o t h e t e r m s c o v e r a l l t h e i d e a s f o r w h i c h t h i s d o c u m e n t c o u l d b e w a n t e d ? H e r e , of c o u r s e , t h e h u m a n f a c t o r e n t e r s h e a v i l y — t h e c a t a l o g e r ' s k n o w l e d g e , b a c k - g r o u n d , a n d p l a i n b r a i n p o w e r . W e k n o w of n o o t h e r s y s t e m , h o w e v e r , w h e r e o v e r l y - c a r e f u l a n d t o o - d e t a i l e d i n d e x i n g c a n b e so c h e e r f u l l y a p p l a u d e d b y t o p m a n a g e m e n t . I t is c e r t a i n l y t r u e t h a t p e r f e c t l y s a t i s f a c t o r y i n d e x i n g c a n b e p e r f o r m e d b y c a t a l o g e r s w i t h m u c h i n - f e r i o r t e c h n i c a l s u b j e c t b a c k g r o u n d t h a n is r e q u i r e d i n a n y t a x o n o m i c s y s t e m of c l a s s i f i c a t i o n . T h e t e a m l e a d e r ' s final c h o r e is t o as- s i g n a p e r m a n e n t a c c e s s i o n n u m b e r t o t h e d o c u m e n t a n d t o r e c o r d i t o n b o t h JANUARY, 1956 21- d o c u m e n t a n d f a n f o l d . H e c h o o s e s t h i s n u m b e r f r o m a b l o c k of " o p e n " n u m - b e r s c u r r e n t l y a s s i g n e d f o r h i s u s e . W h e n t h e d o c u m e n t is r e f i l e d b y h i s c l e r k , h i s p a r t of t h e o p e r a t i o n is c o m p l e t e d . T h e r o u t i n e i n t h e c e n t r a l office e m - p l o y s c o p i e s of t h e f a n f o l d s f o r v a r i o u s n e e d s . T h e o r i g i n a l a n d o n e c a r b o n g o f i r s t t o t h e U n i t e r m c o n t r o l o f f i c e r , w h o m u s t a p p r o v e a l l n e w t e r m s , a d j u s t c r o s s r e f e r e n c e s , a n d e l i m i n a t e u s e l e s s s y n o - n y m s . T h e o r i g i n a l t h e n g o e s t o t h e c l e r k w h o t y p e s t h e M u l t i l i t h s t e n c i l s , a n d finally t o t h e d e s k w h e r e t h e b i - w e e k l y d o c u m e n t a c c e s s i o n s l i s t is p r e - p a r e d . T h e c a r b o n is r o u t e d t o t h e p o s t - i n g c l e r k s . T h e s e c o n d c a r b o n is f i l e d i m m e d i a t e l y i n t h e t i t l e a u t h o r i t y f i l e ; t h e t h i r d i n t h e a c c e s s i o n - n u m b e r f i l e u n t i l r e p l a c e d b y t h e p e r m a n e n t p r i n t e d c a r d . F o r e c o n o m y , s t e n c i l s a r e c u t w i t h a m i c r o - e l i t e t y p e w r i t e r o n c o m m e r c i a l l y a v a i l a b l e M u l t i l i t h m a t s of n a r r o w g a u g e h a v i n g p e r f o r a t e d s p r o c k e t e d g e s w h i c h p r e v e n t s l i p p i n g , s i n c e t h e t y p e w r i t e r is e q u i p p e d ( a t v e r y s m a l l c o s t ) w i t h s p r o c k e t g u i d e s a b o v e t h e p l a t e n . W h e n t h e s e s p r o c k e t e d g e s a r e t o r n off a l o n g t h e p e r f o r a t i o n , t h e s t e n c i l is t h e c o r r e c t w i d t h t o p r i n t 3 " x 5 " c a r d s o n l o n g s h e e t s . T h e p r e s s w i l l a c c o m m o d a t e t w o of t h e s e m a s t e r s s i d e b y s i d e s o t h a t p r e s s t i m e is r e d u c e d t o h a l f . T h e s t e n c i l s a r e p r e - p r i n t e d , a g a i n f o r e c o n o m y , w i t h w h a t e v e r l e g e n d s a r e s t a n d a r d f o r t h i s l i b r a r y ' s c a r d s . T h e f i n i s h e d s h e e t s of p r i n t e d c a r d s , b e i n g c o m p l e t e l y u n i f o r m i n r e g i s t e r , c a n b e m a c h i n e c u t , r e a d y f o r f i l i n g . S a t e l l i t e f i l e s a r e m a i n t a i n e d b y t i t l e , c o r p o r a t e a u t h o r , p e r s o n a l a u t h o r , s e r i e s n u m b e r , a n d c o n t r a c t n u m b e r . T h e " p o s t i n g " o p e r a t i o n , a s t h e p r o c - ess of r e c o r d i n g d o c u m e n t n u m b e r s o n U n i t e r m c a r d s is c a l l e d , c a u s e d r e a l t r o u - b l e . H e r e l a y t h e m o s t f o r m i d a b l e p r o b - l e m w e e n c o u n t e r e d i n t h e a p p l i c a t i o n of c o o r d i n a t e i n d e x i n g . T h e p r o c e s s s e e m s s i m p l e e n o u g h , b u t o n c e i t is b e - g u n d i f f i c u l t i e s m u l t i p l y . E a c h c a r d m u s t b e p u l l e d , r e c o r d e d u p o n , a n d r e f i l e d . T h e w o r k is b o r i n g a n d f a t i g u i n g . E r - r o r s a r e e a s y t o m a k e a n d d i f f i c u l t t o d e - t e c t . W o r k e r s g e t i n e a c h o t h e r ' s w a y . W h i l e p o s t i n g is g o i n g o n , a n y r e f e r - e n c e u s e of t h e i n d e x u s u a l l y m e a n s t h a t o n e o r t h e o t h e r o p e r a t i o n m u s t s t o p . P o s t i n g w a s h o p e l e s s l y s l o w i n r e l a t i o n t o t h e s m o o t h n e s s a n d s p e e d d i s p l a y e d i n a l l o t h e r s t e p s . I t is n o t a n e x a g g e r a t i o n t o say t h a t t h i s b o t t l e n e c k t h r e a t e n e d t h e c o l l a p s e of o u r e n t i r e s y s t e m . T h e s o l u t i o n p r o v e d t o b e a s i m p l e o n e . W e i n s t a l l e d a n I B M p u n c h i n t h e c a t a l o g d e p a r t m e n t a n d e q u i p p e d i t w i t h t w o s t a n d a r d " p r o g r a m s . " A d o c u m e n t n u m b e r p u n c h e d ( a n d v e r i f i e d ) o n t h e f i r s t c a r d is a u t o m a t i c a l l y r e p r o d u c e d o n a l l e n s u i n g c a r d s u n t i l t h e o p e r a t o r w i s h e s t o c h a n g e i t . I n c h a n g i n g f r o m o n e d o c u m e n t n u m b e r t o t h e n e x t h i g h - e r o n e , t h e o p e r a t o r t o u c h e s o n l y t h e f i n a l d i g i t k e y s . O n e t y p i s t w o r k i n g t w o h o u r s a d a y c a n k e e p u p w i t h t h e p u n c h - i n g f r o m a l l f a n f o l d s g e n e r a t e d b y a l l f o u r t e a m s w o r k i n g a t f u l l p r o d u c t i o n . A t t h e e n d of e a c h w e e k t h e a c c u m u l a t e d I B M c a r d s a r e d r o p p e d i n t o a m a c h i n e s o r t e r . N o w t h e p o s t i n g o p e r a t i o n is a q u i t e d i f f e r e n t m a t t e r . O u r c o o r d i n a t e i n d e x is h o u s e d i n t h e f a m i l i a r l i b r a r y " K a r - d e x " t y p e file. B e g i n n i n g w i t h t h e f i r s t o n e r e q u i r e d , t h e p o s t e r w i t h d r a w s o n e t r a y a t a t i m e , d i s t u r b i n g r e f e r e n c e w o r k - e r s a n d o t h e r p o s t e r s n o m o r e t h a n d o e s a n y o t h e r c a t a l o g d e p a r t m e n t w o r k e r w h e n s h e r e m o v e s a d r a w e r f r o m t h e m a i n c a r d c a t a l o g i n a n y l i b r a r y t o file a n e w c a r d . O n t h e U n i t e r m c a r d f o r A U T O M O - B I L E t h i s c l e r k p o s t s t h e n u m b e r f o r t h e d o c u m e n t o n w i n d s h i e l d w i p e r s , and all others concerned with automobiles that the library has cataloged that week. T h e I B M s o r t i n g m a c h i n e h a s e v e n p l a c e d a l l t h e a u t o m o b i l e e n t r i e s i n c o r r e c t n u m e r - i c a l o r d e r . T h e p o s t i n g o p e r a t i o n is s w i f t a n d h i g h l y a c c u r a t e . W e h a d o u r fingers c r o s s e d c o n c e r n i n g t h e r e a c t i o n of o u r l i b r a r y u s e r s t o w a r d 22 COLLEGE AND RESEARCH LIBRARIES the coordinate index, b u t we soon dis- covered that our misgivings were groundless. Unless this agency's em- ployees are miraculously different f r o m the general public, no one else needs to worry either. It is true, however, that the most enthusiastic response has been from our engineers and others with training in some academic discipline. Use of the system numbers several hun- dred questions each week. O n the premise that our customers could not care less whether the answer to their reference question came f r o m a book, a serial, or a document, we placed the coordinate index and its satellite catalogs right beside the library card catalog. All who will may use them. H a b i t u a l library customers almost in- variably prefer to consult the coordinate index unaided after their first five-min- ute indoctrination course in the system. Reference personnel are available, of course, to help any newcomer, or any- one else with a problem. W e think it is sound public relations to offer to help everyone. Everyone, including the ref- erence staff, is taught to think of the co- ordinate index as his heavy artillery. Where author, title, or serial n u m b e r of a document is known, the satellite cata- logs provide quick reference. Much has been printed speculating on the a m o u n t of "noise" a large installa- tion of coordinate indexing would pro- duce; that is, the n u m b e r of false co- ordinations of the man-bites-dog variety which would interfere with effective re- search work. T h e gloomy predictions have not been borne out by our year of operational use of the system. Now, it may well be that there are subject fields in which "false hits" would embarrass the reference librarian. W e can only re- port that, in our library, a little care a n d forethought in the catalog d e p a r t m e n t has kept the n u m b e r of false coordina- tions so small, in our subject matter, that the annoyance is negligible. W e have f o u n d that: (a) the more specific the subject field we are cataloging the tighter is the information control gained; (b) the more specific the uni- terming the fewer are the false hits cre- ated; (c) skillful uniterming is a logical fractionating process, not a mere slicing of a document's title into separate words —this is especially true in the exact sciences; (d) wherever the man-bites- dog difficulty can be foreseen by a cata- loger, the addition of a simple delta sign after the index word will signal the ref- erence user which one is the correct reading, e.g., G E R M A N Y A — I M P O R T S — F R A N C E means only German imports f r o m France, not French imports f r o m Germany. W e have used these "delta flags" freely for words which cause us trouble. T h e i r total number, however, has remained small. T h e r e are, no doubt, more problems which we shall encounter as time goes on, b u t these are all the difficulties which we have met so far. A potential problem, that of an unwieldy pile u p of numbers on " p o p u l a r " uniterms, was solved just as it arose with us by a timely p a p e r from the Office of Basic Instrumenta- tion of the U. S. National Bureau of Standards.1 O n the question of "brows- ability" of the system we refer the reader to the excellent discussion of "brows- ability and suggestability" in the same paper. I n their conclusions we heartily concur. W e have discovered no completely valid method to test the reliability, or the percentage of completeness of re- trieval of information, of our index. W e can testify that to date it has never failed to produce any "known" document. T h e expressions of pleasure we receive con- cerning the quality of our reference service leads us to conclude that the per- centage of retrieval is high, perhaps even very high. 1 W i l l i a m W i l d h a c k , a n d o t h e r s , " D o c u m e n t a t i o n i n I n s t r u m e n t a t i o n , " American Documentation, V ( 1 9 5 4 ) , 223-37. T h i s a r t i c l e c o n t a i n s a u s e f u l b i b l i o g - r a p h y o n d o c u m e n t a t i o n e x p e r i m e n t s r e p o r t e d a b r o a d . W e e m p l o y s t a n d a r d U n i t e r m c a r d s f o r all e x c e p t t h e m o s t h e a v i l y u s e d t e r m s . JANUARY, 1956 23-