\)5; e5a..7t/7: 77-1 The Professional and Administrative Career Examination: Research and Development m United States · Civil Service Commission Bureau of Policies and Standards · SUNY AT BUFFALO THE LIBRARIES . DEPOSITORY· COP't 1 PRR-77-1 THE PROFESSIONAL AND ADMINISTRATIVE CAREER EXAMINATION: RESEARCH AND DEVELOPMENT Richard H. McKillip Marvin H. Trattner Daniel B. Carts Hilda Wing Research Section Personnel Research and Development Center United States Civil Service Commission Washington, D.C. 20415 April 1977 THE PROFESSIONAL AND ADMINISTRATIVE CAREER EXAMINATION: RESEARCH AND DEVELOPMENT ABSTRACT The work of the United States Civil Service Commission's Personnel Research and Development Center to develop a basis for the written test portion of the Professional and Administrative Career Examination (PACE) is described. The objective of the research was the identification of a construct valid set of cognitive abilities tests appropriate for selection of applicants for entry level positions in Federal professional, administrative, and technical occupations. The occupational coverage of the examination was defined and a set of 27 occupations selected for intensive study. These occupations represented .approximately seventy percent of annual appointments in the occupations to be covered by the PACE. Duties of the 27 occupations were rated by subject matter experts for importance and relative amount of time spent in their performance. Subject matter experts also rated a specially developed set of 31 knowledges, skills, abilities, and other worker characteristics (KSAO's) in terms of their importance for overall job performance. Six cognitive abilities were hypothesized as important for duty performance and were rated by personnel research psychologists. A method was devised for combining subject matter expert ratings with psychologist ratings to determine the relative weight of each ability within each occupation. Patterns of ability weights to be applied to subtests for each job resulted from this process. Test question types were identified from the professional literature as measures of the abilities to be included in the test. Factor analysis of the subject matter expert ratings of the 31 KSAO's·provided support for the abilities. Research needed to provide additional technical support for the test was outlined. THE PROFESSIONAL AND ADMINISTRATIVE CAREER EXAMINATION: RESEARCH AND DEVELOPMENT CONTENTS Historical perspective on abilities measurement 1 Summary of examining for entry-level professional, administrative, and technical positions in the Federal Servi~e 3 The need for a new written test 4 Strategy of validation --construct validity 5 Research objective 6 METHOD 6 Selection of occupations for study 6 Identification of major duties 8 Identification of knowledges, skills, abilities, and other worker characteristics 8 Rating of duty importance and time spent in performance 8 Rating of KSAO's 10 Identification of abilities proposed for the written test 10 Definition of abilities 11 Rating of abilities proposed for the written test 12 Analysis of data 12 Choosing acceptable measures of abilities 14 RESULTS AND DISCUSSION 16 Ratings of duty importance and time 16 Rating of abilities proposed for the written test 16 Development of test weights for occupations 20 Ratings of KSAO's 25 Assignment of the remainder of PACE occupations to occupational categories 29 SUMMARY AND PROSPECTUS 30 REFERENCE NOTES 31 REFERENCES 31 APPENDIX A Thirty-one Knowledges, Skills, Abilities, and Other Characteristics Required for Successful Performance in Twenty-seven Occupational Series 33 APPENDIX B Means and Standard Deviations of Ratings of the Importance of Duties (I) and Time Spent in Performance (T) by Subject Matter Experts in Twenty-seven Occupations 38 APPENDIX C Mean Importance Ratings of Knowledges, Skills, Abilities, and Other Characteristics for Each Job Series and for All Job Series Combined THE PROFESSIONAL AND ADMINISTRATIVE CAREER EXAMINATION: RESEARCH AND DEVELOPMENT This paper describes the research and deveiopment work undertaken to provide the construct validation basis for the written test portion of the Professional and Administrative Career Examination, a broad-band examination for certain entry-level positions in the Federal service. 1 One function of the United States Civil Service Commission, as the central personnel agency of the Federal Government, is the development of systems for selecting Federal employees into the competitive service. This includes most positions in the executive branch of the government. Civil Service competitive examinations must by law and regulations be "practical in character and as far as possible relate to matters that fairly test the relative capacity and fitness of the applicants for the appointment sought" (5USC 3304). In order to meet this requirement of law, examinations must be job related. This and other statues (e.g., 5USC 3313, 5USC 3317, 5USC 3318) require candidates to be ranked, certified, and selected in their relative order of fitness. This latter provision distinguishes the Federal examining system and other merit systems from the common industrial practice of selecting from among those who have met some minimum qualifications. Because of the large number of Federal occupations (currently about 2200) and the large number of applicants, the examining function itself is complex. Practical constraints, discussed below, make it necessary that, for examining purposes, occupations be grouped in terms of the abilities which underlie successful performance, and that these abilities be validly assessed in the examination system. Historical perspective on abilities measurement. Research underlying written test development in the Civil Service Commission is based in large part on the findings of leading theorists and researchers in ability testing. Following is a brief review of the relatively short but rich history of the measurement of cognitive abilities. The scientific study of these abilities began with Sir Francis Galton (Tuddenham, 1962; Chaplin & Kraweic, 1968). "Intelligence'' was assumed to be a combination of simpler elements, specifically of simple psychomotor skills such as reaction time and pitch discrimination. This elementalistic theory was discredited when J. McKeen Cattell's student, Wissler, tried to use measures of such skills to predict the grade-point averages of Columbia University freshmen in 1901, and failed. Contemporaneously, Charles Spearman, a student of Galton's, was developing the mathematics of correlational analysis, postulating that "mental ability" was really a unitary factor, called "g" for "general." In France, Alfred Binet was dealing with the practical task of predicting which children needed special assistance b~cause they were unable to learn in the school environment. After reviewing the relevant research, he concluded that the ability to profit from schooling was a complex entity and should be assessed by tasks which represented 1In Federal examining practice, the word "examination" refers to the complete set of procedures by which selection for employment is made. "Test" refers specifically to the written test. An examination may or may not include a written test. In broadband examining, several occupations with similar knowledge, skill, ability, or other worker characteristic requirements are included under one examining procedure. 1 the complexity of the underlying variable without calling for knowledge which could be acquired only in the situation for which the ability was necessary. By 1905, Binet and Theodore Simon, using these principles, had developed a test to accomplish the assigned task. Within a few years, psychologists from other countries translated the test for their own use. Lewis Terman published a revision of Binet's test, the Stanford-Binet, ~n 1916, shortly before the United States entered World War I. Psychologists wished to help w~n this war and determined that they could assist ~n the classification of draftees in the armed services (Korman, 1971). The StanfordBinet was, however, inappropriate, as it was administered on an individual basis, hardly suitable for the task of assessing hundreds of thousands of new soldiers. A committee of psychologists was formed to help with the war effort (Goodenough, 1949). This committee developed and put into use the Army Alpha test to screen over one and a half million men for acceptance as enlisted men, for officer training, and for placement in various jobs. The test measured simple reasoning, following directions, arithmetic, and information, although only the total score, not a score for each ability tested, was used. The utility of ~he total score was assumed, and one finds little reference in current sources (for example, Guilford, 1967) as to how useful the Army Alpha actually was. Apparently it had drawbacks as well as benefits. After the war carne a great increase in the production of tests and test items; and in test usage for occupational and school selection as well as for individual guidance (Korman, 1971). There also developed a concern with what was being measured and why. There appeared to be noncognitive abilities, such as musical talent and mechanical aptitude, that could be more important than cognitive abilities in certain situations. There appeared to be more than just the one underlying ability, the ''g" of Spearman. In addition, the course of the development of multiple cognitive abilities was not well understood. Concurrently, the statistical technique of factor analysis, originated by Spearman, was developed as a means of studying the interrelations of tests (Chaplin & Krawiec, 1968). Factor analysis is a method for identifying the common dimensions which underlie such interrelationships. Reference to the work of Louis L. Thurstone, the originator of modern factor analysis, serves to illustrate the development of tests and factor analysis during the 1930's. Thurstone did not accept Spearman's hypothesis of a general factor of ability; rather, he postulated multiple factors of ability underlying performance on cognitive tests. In a major study, Thurstone (1938) administered 56 psychological tests to volunteers at the University of Chicago. He intercorrelated the scores on these tests and factor analyzed the resulting correlation matrix. There were six underlying dimensions, or "Primary Mental Abilities," for performance on these tests: verbal, number, spatial, word fluency, memorizing, and reasoning (Thurstone & Thurstone, 1941). If "factor-pure" tests, measuring one ability each, could be developed, then an accurate prediction of performance on the tests in the original battery could be made from performance on these six "factor-pure" tests. The implication, extensively developed in military selection during World War II, was that a small number of pure measures could be used to predict performance that depended upon a large number of the abilities combined in a complex fashion. Dramatic changes in technology had made. the task of accurate selection and placement of soldiers in World War II even more critical than in World War I. There was a great deal of testing using batteries measuring a number of abilities. The Army Air Corps Aircrew Classification Battery used scores weighted differentially for different jobs. This battery was constructed to measure psychological constructs adjudged to be important for success in pilot training. The tests used were not factor-pure but each measured a subset of the relevant abilities (Guilford & Hoepfner, 1971). The battery was constructed along factorial lines (Guilford, 1967). 2 After the war, test development efforts continued, some emphasizing "factors" of ability which could be combined in different ways for different predictions. These tests measured "factors" in the manner of the Aircrew Classification Battery, not in the factor-pure manner of Thurstone. The United States Employment Service, using Thurstone as a consultant, developed the General Aptitude Test Battery (GATB) for a variety of jobs (Guilford, 1967). In the Civil Service Commission, Ernest S. Primoff constructed a multiple-abilities battery to measure "elements" found in trades and industrial jobs. This battery was later expanded to measure additional elements found in clerical jobs (Primoff, Note 1). The Differential Aptitude Test was developed for use in high school as an academic and vocational counseling tool (Cronbach, 1970). With the increased number of tests and test batteries came attempts to analyze and synthesize research findings in human abilities testing. One comprehensive description of factor analyses of aptitude and achievement tests dealt with hundreds of tests in 69 factor analytic studies carried out in diverse settings (French, 1951). In 1963, a set of reference tests was identified for cognitive abilities based on factors so identified (French & Ekstrom, 1963). Various other authors have hypothe sized over one hundred cognitive ability factors. Although much basic research is still in process, a considerable number of abilities underlying human behavior have been identified and reliably measured. Summary of examining for entry-level professional, administrative, and technical positions in the Federal service. Many professional, administrative, and technical jobs require the same k1nds of knowledges, skills, abilities, and other worker characteristics. In 1939 the United States Civil Service Commission established the Junior Professional Assistant Examination (JPA) as a comprehensive program to select applicants for entry-level positions in 28 different Federal occupations. The requirements for passing this examination were two: specific experience or college education relevant to the designated occupation, and performance at a high level on a written test. The test consisted of a verbal comprehension ability section and a specific test of knowledge of the subject matter required for the occupation. An individual would apply separately, and be considered separately, for each of the 28 occupations. After World War II, changes were made 1n both the written test and the background requirements. The subject matter sections of the written tests were dropped, as it became more and more difficult to maintain up-to-date test material for such a wide variety of occupations. However, work experience or education now had to be related directly to the occupation pursued. Quantitative reasoning and abstract reasoning abilities questions were added to the written test. This change was generally satis factory, but several examining issues remained. First, the demands for such specialized preparation proved to be a barrier to liberal arts graduates and to others who, despite their lack of specific experience, had high levels of basic ability to learn and to perform the duties of the jobs. Second, many Federal occupations have no specific experience requirements or have no industrial counterparts where people can gain experience. Internal Revenue Officer, Customs Inspector, and Social Insurance Claims Examiner are examples. Third, employment in the Federal service provides the oppor tunity for mobility in professional and administrative assignments, often requiring the ability to change and to keep up with advances in technology as well as ability to adapt to shifts in social attitudes and practices. The ability to learn new material appeared common to many occupations. It could be appropriately measured by one examination for those occupations. Such an examination would be convenient to both the applicant and to the Federal Government as it would 3 expand the candidate's opportun1t1es for employment and career development. Thus the Federal Service Entrance Examination (FSEE) was introduced in 1955. The FSEE had two basic requirements: appropriate experience and an adequate level of ability to deal with the complexity and difficulty of the journeyman or career-level demands of the occupa tions covered. The experience requirement could be met with three years of relevant work experience or a bachelor's degree. A written test was used, here assessing verbal, quantitative, and abstract reasoning abilities. The occupations had an entrylevel of GS-5 or GS-7, and with certain exceptions, the normal progression for successful employees was to a full performance level of GS-11 or above. Occupations requiring both the stated experience and the necessary configuration of abilities assessed by the written test were filled primarily through the FSEE. Some years later, in 1963, the occupational coverage was reviewed. Certain scientific and professional positions which required highly specific and extended academic preparation were determined to be more properly filled through other techniques of examination, and were excluded from FSEE coverage. Simultaneously, the written test was reviewed for the remaining occupational coverage. The measurement of abstract reasoning ability was dropped, so that the test now measured verbal and quantitative abilities. The need for a new written test. The test was kept under continual review, and by 1970 it was clear that changes were desirable. The examination itself was almost fifteen years-old. The test was working well but it was now technically feasible to introduce significant improvements. Advances in data processing capability made differential weighting of test scores of large numbers of applicants practical. The FSEE written test measured two abilities, verbal and quantitative, determined by job analyses to be necessary to learn and to progress to full performance levels in the occupations covered by the examination. Further, the verbal section carried approximately twice the weight of the quantitative section, a weighting which, while it was optimal for a majority of occupations and hence useful for the mobility concept, might not be optimal for all FSEE occupations. While the written test of the FSEE was designed to measure two abilities which were and are common job requirements, technical advances made it feasible to measure additional abilities to reflect more accurately and comprehensively real, reasonable, and important qualifications underlying effective job behavior. It was recognized that a systematic process was necessary to ''determine the relative importance of the qualification requirements for the job in order to identify those of greatest relevance for measurement and, if appropriate, for later weighting when measures are combined" (USCSC 1972). It was also recognized that existing methods of job analysis did not provide efficient identification of underlying abilities common to large numbers of occupations. For these reasons the United States Civil Service Commission decided in late 1972 to undertake the development of a new examination for the entry-level positions of p~ofessional, administrative, and technical Federal occupations. 4· Strategy of validation --construct validity. Test development begins with the choice of a strategy for validating the test --the way in which the degree to which the test measures what it is supposed to measure will be shown. Professionals in testing have concurred in stating their goals for assessment of human characteristics, as codified in Standards for Educational and Psychological Tests (American Psychological Association, American Educational Research Association, & National Council on Measurement in Education, 1974). They agree that "questions of validity are questions of what may properly be inferred from a test score" (Standards, p. 25) and propose three types of strategy to explicate the relationship between the test as predictor and the inferred criterion-related final product or criterion: criterion-related validity, content validity, and construct validity. Criterion-related validity depends upon the empirical statistical relationship between test scores and measures of some criterion, such as academic grade or job performance. Content validity implies that the test measures some representative sample of the content of the criterion, such as stenographic skills or knowledge of lathe operating procedures. Construct validity requires hypothesizing of various psychological constructs important for performance both on the test and on the criterion; the. development of measures of these constructs; and the gathering of evidence to support both the relationship of the predictor and the criterion via the constructs as well as to explicate the constructs themselves. It is obvious that "these aspects of validity can be discussed independently, but only for convenience. They are interrelated operationally, and logically ..." (Standards, p. 26). In determining the process to use in validating the new examination, it appeared that content validity was least relevant. Selection would be of individuals who could progress tp successful performance in occupations for which they currently had no knowledge or experience, occupations that could easily change from the time of their selection to the time of their achieving journeyman status. The case for criterion-related validity was more plausible, but it could not be accepted as the sole strategy either, for two reasons. The first is the technical infeasibility of collecting enough job incumbents together to perform satisfactory statistical studies. The occupations to be covered are diverse. For most, there are relatively few incumbents; these workers are often scattered among many geographical areas and Federal installations. Occupations with larger numbers of incumbents might or might not emphasize the same duties depending upon the requirements of their employing agencies. The second reason, mu~h more critical, is that this validation strategy has limited generalizability. Criterion-related studies in those occupations where data collection is feasible would assess whether scores on the written test were statistically correlated with measures of job performance criteria. Such studies in.themselves can explicate neither the methodology used in job analysis, nor the procedures used in written test specification and construction, nor the scientific judgment required to relate the written test to the job analysis. Further, such studies individually would say little if anything about the proper use of this written test for occupations where criterionrelated studies were not feasible but job analysis was. What must be validated for this examination are the psychological constructs posited. A series of criterion-related validity studies can be the means of providing much of the evidence required for construct validity. The first step 1n the validation process was to select the constructs, specify their assessment both in the job and in the written test, collect preliminary data supporting the identity of the constructs and the relationship of test to job via the constructs, and plan the subsequent research required for complete documentation. Empirical evidence is vital to any validation strategy, but the gathering of it need not be limited to criRa-ther, empirical evidence is necessary to elaborate terion-related statistical studies. 5 and define the nomological network, the scientific theory of which the psychological constructs are basic entities (Cronbach & Meehl, 1955). And, as one of the deans of Amer~can psychometrics has said: The point which we want to make is that rational analysis and judgment are the fundamental basis for the use of tests in personnel selection, and that statistical procedures are of value as a guide and supplement to, and not as a substitute for, professional judgment. (Thorndike, 1949, p. 212). Research objective. In view of the above considerations, the objective of the PACE research was the 1dentification of a set of construct valid cognitive abilities tests appropriate for selection of applicants for a broad range of professional, admin istrative, and technical occupations. These occupations have a usual entry level of GS-5 or GS-7 and a full performance (journeyman) level of GS-9 or above. The outcome of the research was to include the following: 1. Identification of a number of cognitive abilities, or psychological con structs, required for successful performance in the occupations to be covered. 2. Development of a systematic method.of job analysis, which could provide an assessment of the relative importance of the abilities for successful performance. 3. Specification of a new written test which would assess the performance of applicants in each ability and which would provide differential weighting of ability test scores for differing job requirements. 4. Specification of need for future research studies to provide additional support and documentation of the construct validation strategy used to develop the new examination. METHOD To reach the objective, the research team had to choose a manageable number of occupations to study, then list the duties of each, and then define the abilities underlying successful performance of these duties. Next, the researchers had to find ways to rate the duties for importance in each occupation, to rate the abilities for their importance to performance of each duty, and to combine these two ratings so as to provide weights for each ability in each occupation. Finally, tests had to be developed to measure the abilities. The detailed procedures by which these steps were accomplished are described below. Selection of occupations for study. Since some 120 Federal occupations fit the scope of coverage requ1rements descr1bed above, it was necessary to reduce the jobanalysis problem to a manageable size. Manpower need projections were not available,but records of placements in 1970 and 1972 were. The researchers assumed that while some fluctuation in hiring rates would probably occur, a reasonably large sample of the occupations in demand by agencies would embody more than half the hires to be made under the PACE. Accordingly, 27 such occupations were selected for intensive analysis.Table 1 lists these occupations with the numbers of placements in 1970 and 1972. In all,selections in these occupations account for over 69 percent of hires from the Federal Service Entrance Examination in those years. Inspection of Table 1 reveals the wide diversity of the occupations. Many of them are found only in the Federal service, some 6 in only one Federal agency. With few exceptions, it 1s unreasonable to require either specific ac~demic preparation or work experience of applicants for these occupations. It was apparent that ranking of candidates would have to depend on measures of those abilities which were basic to successful performance in them. Series 993 1890 1169 105 334 526 1811 560 201 1102 343 110 996 570 1816 2001 187 967 1854 1710 962 1082 1810 1081 1712 685 180 Note. Data for TABLE 1 "In-Demand" High Use Occupational Series for 1970 and 1972 Combined Occupation Social Insurance Claims Examining Customs Inspection Internal Revenue Officer Social Insurance Administration Computer Specialist Tax Technician Criminal Investigating Budget Administration Personnel Management Contract and Procurement Management Analysis Economist Veterans Claims Examining Financial Institution Examining Immigration Inspection General Supply Social Services Passport and Visa Examining Alcohol, Tobacco and Firearms Inspection Educational and Vocational Training Contact Representative Writing and Editing General Investigating Public Information Specialist Training Instruction Public Health Program Specialist Psychology 1971 were not available at the time of the study. Placements 1697 997 960 882 880 770 619 540 533 493 389 373 364 346 340 266 235 211 202 193 181 143 135 105 98 65 52 12,069 Data Source: Bureau-of Recruiting and Examining, U.S. Civil Service Commission. 7 Identification of major duties. The Civil Service Commission conducts intensive analyses of occupations to develop classification and qualification standards. These analyses are performed by skilled occupational analysts in accordance with accepted pos1t1on classification practices. Classification standards identify and group jobs with common content into occupations for job evaluation and pay purposes. These standards provide a broad description of duties performed by incumbents of the occupa tions. As such, they were an appropriate starting point for the special job analysis to be employed in the PACE research. Qualification standards delineate education, experience, and other requirements for establishing minimum eligibility at various grade levels within occupations. The PACE test was to be designed to identify applicants who would be superior performers on the job. The researchers, then, were not interested in measuring the characteristics needed for barely acceptable performance. Therefore, little use was made of the qualification standards in this study. The classification standards were reviewed, and the major duties performed at the grade level occupied by the greatest number of employees were extracted. This was taken to be the "journeyman" or "full performance" level to which most of those entering the occupation at the GS 5/7 levels could expect eventually to advance. The number of major duties varied depending upon the content of the occupation. Identification of knowledges, skills, abilities, and other worker characteristics. As major duties were extracted from the classification standards, the researchers educed a set of 31 knowledges, skills, abilities, and other worker characteristics (KSAO's) underlying successful performance in the occupations. These variables were defined in terms of the behaviors associated with them, and, where appropriate, the definition included examples of job situations in which the KSAO's are important. The aim was to list KSAO's that had been described in the measurement literature. With respect to content and level of specificity, the list of 31 KSAO's resembled lists previously developed for selection purposes by the American Institutes for Research (Theologus, Romashko, & Fleishman, 1970) and the Civil Service Commission (Primoff, Note 2). The KSAO's were to provide a framework for describing the occupations in terms of the attributes of successful performance and to provide a basis for selecting abilities to be measured by the written test. A complete list of the KSAO's, with their definitions, is provided in Appendix A. Rating of duty importance and time spent in performance. A sample of employees in senior-level and supervisory positions was selected for each occupation. These "subject matter experts" (SME's) were chosen from 26 Federal agencies, in locations throughout the country, that were major employers in the occupations. Seventy-six separate groups of SME's, a total of 1241 persons, participated in the research. The rating sessions were each conducted by one of three of the members of the PACE team of researchers, each of whom specialized in several of the occupational series. Table 2 shows characteristics of each occupation and the distribution of SME's among the occupations. The SME's were asked to review the list of duties for their occupation. The first group to review an occupation was encouraged to amend the_list of duties or add to it as appropriate. Considerable discussion under the guidance of the researcher often characterized this part of the procedure, with resultant changes in wording of existing or proposed duty statements. Subsequent SME groups in the occupation did not change the list but were free to add to it. Subject matter experts next rated each duty for its importance 1n the context of successful performance of the total job. They then rated each duty 1n terms of relative amount of time spent in its performance. g The following rating scales were used: Time Spent on Duty Compared to Importance of Duty for Job Success Other Duties Performed 0 No Importance 0 No time spent 1 Insignificant 1 Very Much Below Average 2 Slight Importance 2 Below Average 3 Some Importance 3 Slightly Below Average 4 Important 4 Average 5 Very Important 5 Sligntly Above Average 6 Great Importance 6 Above Average 7 Critical 7 Very Much Above Average TABLE 2 Characteristics of 27 PACE Occupations and Distribution of Subject Matter Experts No. of Modal No. of No. of SME Series Occupation Grade Duties SME's Groups 105 Social Insurance Administration 10 20 68 4 110 Economist 14 12 40 3 180 Psychology 13 10 9 1 187 Social Services 7 6 16 1 201 Personnel Management 13 16 77 5 334 Computer Specialist 12 11 73 4 343 Management Analysis 12 13 60 4 526 Tax Technician 9 9 53 3 560 Budget Administration 13 14 69 5 570 Financial Institution Examining 12 9 20 1 685 Public Health Program Specialist 13 6 20 2 962 Contact Representative 9 7 40 2 967 Passport and Visa Examining 10 7 16 1 993 Social Insurance Claims Examining 10 10 78 4 996 Veterans Claims Examining 12 8 40 2 1081 Public Information Specialist 13 10 25 2 1082 Writing and Editing 12 9 30 2 1102 Contract and Procurement 12 8 58 4 1169 Internal Revenue Officer 12 11 71 4 1710 Education and Vocational Training 9 16 39 2 1712 Training Instruction 9 6 40 2 1810 General Investigating 11 8 36 2 1811 Criminal Investigating 13 14 60 4 1816 Immigration Inspection 9 6 36 2 1854 Alcohol, Tobacco, & Firearms Inspection 9 10 34 2 1890 Customs Inspection 9 10 68 4 2001 General Supply 9 19 65 4 1241 76 9 Rating of KSAO's. After completing the ratings of duties, the SME's studied the list of 31 KSA0 1 s described earlier and shown in Appendix A. They then rated each KSAO for its overall importance to successful job performance. The same 0-7 importance scale that was used for duty importance rating was employed for KSAO importance rating. Identification of abilities proposed for the written test. The identification and rating of duties and underlying KSA0 1 s supplied a great amount of information essential to developing a sound examination. However, this inf9rmation was not sufficient. There remained the problems of identifying testable abilities that applied to all the jobs covered and df choosing appropriate item types to measure these abilities. To accomplish this, the researchers examined the KSAO's and the factor analytic literature relating to cognitive ability measurement. Attention was focused on cognitive abilities because of their known usefulness in predicting job performance, and because of regulatory and practical restrictions on other measures. It was not possible, for example, to measure several aspects of personality, as the Federal Personnel Manual (Chapter 337, Subchapter l-Si) expressly prohibits the use of personality tests except as part of a medical-determination of fitness (U.S. Civil Service Commission, 1969). Certain other KSAO's, such as object perception and physical stamina, were given relatively low importance ratings by SME's, and were therefore not considered further. Several sources of information were consulted in the effort to identify appropriate testable abilities. The most comprehensive compilation of evidence concerning cognitive abilities measurement was found in French's monograph (1951). Despite its age, this monograph remains a major source of information about factors extracted from aptitude and achievement tests. It integrated results obtained in 69 factor analytic studies, and identified 59 factors. Item types included in these studies form the basis of most of the abilities tests 1n use today. The cognitive components of the 31 KSAO's described earlier were compared with the 59 factors in the monograph. Neither the set of KSAO's nor the factors in the monograph were designed to consist solely of cognitive abilities, so it is not surprising that only six matches were found. The content of KSAO's has already been discussed in this report. The French monograph included many noncognitive factors, such as ambidexterity and liberal-conservative. Table 3 lists the matched abilities and the number of 1nstances the factor was 1dentified in the monograph. Certain of the matches do not reflect exact correspondence. General reasoning ability relates to the factors of deduction and induction, while arithmet1c computat1on and quantitative reasoning relate to the number factor described by French. It should be noted that quant1tative reasoning is often found to load on deduction and verbal comprehension factors. Further, the literature relating to cognitive abilities testing is complex, and cognitive factors are generally somewhat interdependent. Six abilities, for which there were reasonable matches with those identified in the monograph, were selected for closer analysis of their relationship to job performance. The six abilities as they were redefined, constituted the set of cogn1t1ve ability constructs for which PACE test items would be developed. These abilities are well established, they appeared to be reasonably related to successful performance of the kinds of jobs to be filled from PACE, and they appeared to be a manageable number to include in an examination for employment in which large numbers of applicants are assembled for examination administration.2 2The test which ultimately was developed measures only five abilities, but takes almost four hours to administer. 10 TABLE 3 PACE Cognitive Abilities Matching Factors Described by French Cognitive Ability Factor General Reasoning Deduction (37) General Reasoning Induction (9) Judgment Judgment (5) Memory Associative Memory (16) Arithmetic Computation Number (35) Quantitative Reasoning Number (35) Reading Comprehension Verbal Comprehension (46) Note. Numbers 1n parentheses indicate the number of studies in which the factor was found. Definition of abilities. The abilities were defined to reflect both the mental operat1ons 1nvolved 1n the jobs and the work processes to which they were most applicable. The ability definitions which follow were given one-or two-word labels. This was done with great reluctance, for editorial convenience in making reference to specific abilities in the remainder of this report. Short labels, such as these, are inadequate to describe the abilities, and can result in misunderstanding when used in communication with persons who are not thoroughly familiar with the fuller descriptions presented in this paper. Readers are cautioned to avoid using such labels without ascertaining that their audience is familiar with the full descriptions of the abilities. 1. The ability to reason from principles to the implication of these principles in specific situations. The ability to reason from given premises to their necessary conclusions. The ability to think clearly about the implications of given facts. This ability would be very important in developing a system, plan, or procedure. For convenience, it is referred to as deduction in the remainder of this report. 2. The ability to generalize from specific data to general facts. The ability to examine specific facts and to arrive at an understanding of their underlying relations. This type of reasoning includes the formation and testing of hypotheses. It would be important in solving problems. Hereafter referred to as induction. 11 3. The ability to solve a presented problem when all the facts for solution necessary are not given. Solution to the problem involves making some reasonable assumptions or anticipating what the most likely of several possible occurrences might be. This is the ability to make good decisions in such circumstances. It is important when decisions must be reached based on incomplete evidence. Hereafter referred to as judgment. 4. The ability to retain a large quantity of information. This ability is important when a large quantity of information must be quickly recalled, or when the reference system for storing needed information is inadequate. Hereafter referred to as memory. 5. Facility in manipulating numbers in any form. This ability is important where numbers are utilized. They are either calculated or numerical problems are solved. Hereafter referred to as number. 6. Knowledge of, understanding of, and the ability to manipulate the English language. Effective command of the English language. This ability is important when the duty requires use of the English language. Hereafter referred to as verbal comprehension. Rating of abilities proposed for the written test. A group of six personnel research psycholog1sts not prev1ously assoc1ated w1th the study was assembled to rate the importance of each of the six known ability constructs for measuring performance of the duties performed in the 27 jobs. Classification standards were studied, duty lists were reviewed and understood, and ratings were made in strict accordance with the definitions of the abilities. These psychologists were all experienced in the use of tests for selection of employees. They used the same rating scale that was used by the SME's to rate duty importance. Analysis of data. For each occupation, ratings had now been obtained on eight-point scales for the 1mportance of each duty for successful job performance, the relative amount of time spent on each duty, the importance of each of the six abilities for performance of each duty, and the importance of 31 KSAO's for overall job performance in each occupation. Reliability of the various ratings was measured by dividing the rater groups into random halves and calculating the product-moment correlation coefficient of the mean ratings of each half for each duty or ability being rated. A critical link in the chain of procedures leading from job requirements to the test was the mathematical procedure for combining ratings of duty importance and time spent with ratings of ability importance for duty performance. It was this combination that would provide an economical determination of the relative importance of the abilities for job performance in each occupation. Relative importance of the abilities was to provide the basis for weighting subtests designed to measure the abilities. 12 For each of the 27 occupational series, the importance of each ability constructwas calculated. Means of SME ratings of duty importance and time spent were summed,divided by two, and multiplied by the psychologists' ratings of the importance of theability for performance of the duty. The resultant products for all the duties in the occupation were summed and divided by the total number of duties in the occupation. The formula for accomplishing this is: N I· + T· J J }'; A·. 2 1-J j=l AI·1, (1) N where: Aii The importance of ability i for the occupation; The mean rating of importance of duty j; The mean rating of time spent on duty j; The mean importance rating of ability i for performance ofduty j; j 1, 2, N duties of the occupation i 1, 2, 6 abilities N Number of duties in the occupation While this formula yielded valuable information concerning the importance of theability constructs, there remained the problem of determining the relative weight ofeach ability to be applied to test scores for each occupation. The large number ofoccupations to be included in PACE made it impracticable to conduct criterion-relatedvalidity studies to obtain test weights through multiple correlation techniques. How ever, the comprehensive rating process of duties and abilities for the sample of occupations could provide weights with one additional step. This was done by obtainingfor each ability its proportion of the total of all the ability importance (AI) valuesin each occupation. The formula for this is simply: AiiAWi ~-6~----}'; Aiii=l where: AWi Weight of ability i for the occupation; Aii Importance of ability i for the occupation, from (1); i l, ... , 6 abilities 13 These values were rounded to whole numbers for exam1n1ng convenience, since applying fine weights adds practically nothing to the predictive validity of a test battery (Dawes & Corrigan, 1974). Rounded weights were expected to provide a manageable number of weighting patterns, each including one or more occupations. Rounded weights were derived by multiplying the AW values by ten and then rounding to a whole number. These weights could then be used differentially to predict success in various occupational categories. It is often desirable to classify a large number of elements according to some presumed relationships. To accomplish some objective groupings of the 31 KSAO's and the 27 jobs sampled in this research, factor analysis procedures were employed. Factor analysis is a procedure used "to assign each entity or element to a group such that there is a well-defined basis' for 'belonging to a group' and that the groups are clearly distinguishable one from another" (Harman, 1972). A factor analysis of ability importance would group the occupations according to the relative importance of each ability for each occupation. The occupations had already been grouped in terms of the similarity of their AI weighting patterns. These two procedures should provide similar results, because the basic data, ratings of importance of abilities, are the same. A second factor analysis was undertaken to group the 31 KSAO's into categories which could be compared to the abilities. Choosing acceptable measures of abilitie.s. Identification of specific test quest1on types to measure the ab1l1t1es was deferred until data analysis was completed and the final determination of abilities to be tested was made. Selection of question types was based on both technical and administrative considerations. Where feasible, two question types would be employed for each ability. The monograph of French (1951) and the Manual for kit of reference tests for cognitive factors (French, Ekstrom, & Price, 1963) served as the primary sources for identification of test question types. Since the test was to be administered to some 200,000 persons per year, question types had to be amenable to group administration with machine scoring of answer sheets. Accordingly, it was decided that questions would be of five-alternative mult~ple-choice format, but otherwise following the "reference kit" question types as closely as possible. Where deviations from the reference kit tests were to be made, care was to be taken that the mental processes required to answer the questions would conform to the mental processes required to answer questions in the corresponding reference kit tests. The descriptions of the question types which follow are adapted from the booklet of sample questions given to each PACE competitor. The ability to reason from general principles to the implications of these prin ciples in specific situations, to reason from given premises to their necessary contabular completion questions. The clusions, is measured by inference questions and inference question type presents a statement which is to be accepted as true and should not be questioned for purposes of the test. The correct alternative must derive from the statement without drawing on additional information not presented. Incorrect alternatives rest, to varying degrees, on the admission of new information. Inference is a test for the syllogistic reasoning factor in French, Ekstrom, and Price (1963). Tabular completion questions present charts or tables in which some entries are missing. The examinee must deduce the missing values. This question type has a mean loading of .46 on the deduction factor in two studies noted in French (1951). Letter series and figure analogies question types measure the ability to gener alize from specific data to general facts, identifying underlying relations or anal ogies in the data. Letter series questions consist of a set of letters arranged in a definite pattern. The examinee must discover what the pattern is and determine the letter which should occur next in the series. Letter series occurs in four studies included in French (1951), in which the mean loading on induction factors was .49. 14 Figure analogy questions each consist of two sets of symbols where a common character istic exists among the symbols in each set and where an analogy is maintained between the two sets of symbols. A symbol is missing from one of the sets. The examinee must discover which alternative fits the missing symbol in such a way as to preserve the characteristics common to the second set and to preserve the analogy with the first set. This question type fits the description and is similar to the Figure Classifi cation Test in the reference kit of French, Ekstrom, and Price (1963). The ability to solve problems when all the relevant facts are not given, to infer missing facts in data and missing events in out-of-order sequences, is measured cur rently by a test of comprehension. In fiscal year 1977, logical order of events ques tions will also be used to measure this ability. In compre~ension questions the examinee is required to determine the most plausible or reasonable alternative which might explain or follow from a given statement. Selection of the best alternative requires general knowledge not included in the original statement. While more than one alterna tive may be plausible, the correct answer is the most plausible of the alternatives. Comprehension had a mean loading of .54 on the judgment factor in two studies reported in French (1951). Logical order of events questions require the candidate to determine the most reasonable sequence of occurrence of a set of events. This determination may require application of general knowledge to infer missing concepts or events that are essential to the sequencing of the elements of the set. This question type is suggested by the Practical Judgment and Sequence of Maneuvers tests identified as measuring the ability as exposited in French (1951). The ability to perform or check arithmetic operations and to solve quantitativeproblems of varying complexity is measured by computation and by arithmetic reasoning questions. Computation questions require straightforward calculation and may include decimals, fractions, and percentages. Arithmetic reasoning questions are word problemswhich require quantitative reasoning processes for their solution. These question typeshave mean loadings of .48 to .82 on the number factor in studies noted 1n French (1951). Verbal comprehension ability is measured by reading comprehension and vocabularyquestions. Reading comprehension questions require the examinee to read a given paragraph and to select an answer on the basis of comprehension of the conceptual content of the paragraph. The correct answer is either a reworded statement of the main conceptsin the paragraph or a conclusion so inherent in the paragraph content that it is equivalent to a restatement. Reading comprehension tests had a mean loading of .66 on the verbal comprehension factor in six studies noted in French (1951). Each vocabularyquestion contains a key word and five alternative choices. The examinee is to select the alternative word that is closest in meaning to the key word. The incorrect alternatives may have a more or less valid connection with the key word. In some cases, the correct choice differs from the others only in the degree to which its meaning comes close to that of the key word. Vocabulary had a mean loading of .80 on the verbal comprehension factor in two studies noted in French (1951). No adequate paper and pencil test of the ability to retain a large quantity of information could be found. Tests of short-term memory are available, but the professional, administrative, and technical jobs included in this study were adjudged to require long-term memory. Therefore this ability was not included in the test development plan. 15 RESULTS AND DISCUSSION Ratings of duty importance and time. Subject matter experts rated the importance and time spent for each duty 1n the1r occupation in the manner described earlier. Appendix B contains means and standard deviations of importance and time spent ratings for each duty for each occupation. Number of SME raters is indicated for each occupation. Inspection of Appendix B suggests that duties varied considerably within occupations in both importance and time spent. This suggests the raters undertook their tasks seriously and made well-reasoned judgments, as halo and inflationary rating tendencies are not apparent. As might be expected, occupations with greater numbers of duties tended to have lower average ratings than did occupations with fewer duties. That is, the greater the number of duties, the less important any single duty tended to be, and the less time spent on it. The correlation between number of duties in the occupation and mean importance rating was -.70. This relationship makes it improper to attribute any significance to differences in AI values between occupations. AI values appear to be useful in this study only to ascribe relative weights within occupations. Reliability estimates were obtained by computing product moment correlation coefficients betwen mean ratings of randomly selected halves of the rating groups. Correlations were corrected by the Spearman-Brown prophecy formula. Table 4 lists reliability coefficients for subject matter expert ratings of duty importance, duty time spent and KSAO importance. All but five of the 81 coefficients reached statistical significance. This is especially remarkable in view of the small numbers of duties in many of the occupations. In fact, there were six duties in each of the two occupations in which nonsignificant coefficients (p > .05) occurred for both importance and time spent ratings. In the occupation where the other nonsignificant coefficient was obtained there were seven duties. These results indicate that the SME's understood their tasks and rated from a common frame of reference. The duties and KSAO's were well defined and carefully rated. The relation of duty importance to relative amount of time spent was explored by correlating mean SME ratings of these variables. Table 5 lists these correlations for the 27 occupations. Twenty-five were statistically significant (p < .05). Most of the correlations were of high magnitude, fully 20 of them above .90~ This very strong relationship between importance and time suggests the measures can be used interchangeably. This finding supports the position of Christal (1974) that time spent ratings can be used in task analysis systems. According to Christal, relative time spent ratings are preferable to duty importance ratings in calculating degree of similarity between positions. Rating of abilities proposed for the written test. The psychologist ratings of the six abilities proposed for the written test were to be combined with SME ratings of duty importance and time spent to provide the basis for weighting test parts. Table 6 presents reliability estimates of these ability ratings. The coefficients were obtained by dividing the six psychologists into two equal sized groups, calculating the correlation of the mean ability ratings over the duties in the occupation, and correcting for a double-sized group by the Spearman-Brown formula. The psychologists appear to have rated the abilities quite reliably, especially 1n view of the small number of raters, the difficulty in making the required inferences, 16 TABLE 4 Corrected Pearson Product Moment Split-Half Reliabilities of SME Rating of Duty Importance, Duty Time Spent, and KSA0 1 s Reliability Estimates No. No. Series Occupation of of Impor-Time KSAO's Raters Duties tance 105 Social Insurance Administration 68 20 97** 98** 98** 110 Economist 40 12 91** 94** 97** 180 Psychology 9 10 92** 84** 89** 187 Social Services 16 6 71 48 94** 201 Personnel Management 77 16 96** 96** 98** 334 Computer Specialist 73 11 96** 95** 97** 343 Management Analysis 60 13 92** 91** 98** 526 Tax Technician 53 9 98** 98** 98** 560 Budget Administration 69 14 92** 92** 96** 570 Financial Institution Examining 20 9 94** 95** 95** 685 Public Health Program Specialist 20 6 43 77 94** 962 Contact Representative 40 7 96** 98** 97** 967 Passport & Visa Examining 16 7 13 90** 81* 993 Social Insurance Claims Examining 78 10 96** 99** 98** 996 Veterans Claims Examining 40 8 94** 86** 95** 1081 Public Information Specialist 25 10 74* 94** 93** 1082 Writing & Editing 30 9 86** 86** 95** 1102 Contract & Procurement 58 8 94** 97** 98** 1169 Internal Revenue Officer 71 11 99** 100** 98** 1710 Education & Vocational Training 39 16 94** 96** 95** 1712 Training Instruction 40 6 97** 98** 98** 1810 General Investigating 36 8 100** 99** 96** 1811 Criminal Investigating 60 14 92** 98** 97** 1816 Immigration Inspection 36 6 88* 98** 97** 1854 Alcohol, Tobacco, & Firearms Inspection 34 10 94** 94** 97** 1890 Customs Inspection 68 10 84** 95** 97** 2001 General Supply 65 19 95** 95** 94** Note. Decimal points have been omitted. * p < .05 ** p < .01 17 TABLE 5 Correlations Between Mean SME Ratings of Duty Importance and Relative Time Spent Series 105 110 180 187 201 334 343 526 560 570 685 962 967 993 996 1081 1082 1102 1169 1710 1712 1810 1811 1816 1854 1890 2001 Note. Occupation Social Insurance Administration Economi&t Psychology Social Services Personnel Management Computer Specialist Management Analysis Tax Technician Budget Administration Financial Institution Examining Public Health Program Specialist Contact Representative Passport and Visa Examining Social Insurance Claims Examining Veterans Claims Examining Public Information Specialist Writing and Editing Contract and Procurement Internal Revenue Officer Education and Vocational Training Training Instruction General Investigating Criminal Investigating Immigration Inspection Alcohol, Tobacco, and Customs Inspection General Supply Decimal points omitted. * P < .OS ** p < .01 Firearms Inspection Duties 20 12 10 6 16 11 13 9 14 9 6 7 7 10 8 10 9 8 11 16 6 8 14 6 10 10 19 Correlation 98** 88** 91** 99** 99** 98** 98** 98** 95** 91** 82* 94** 54 95** 92** 94** 99** 90** 98** 96** 35 95** 85** 80** 70* 94** 99** 18 TABLE 6 Corrected Pearson Product-Moment Split-Half Reliabilities of the Means of the Psychologists• Ratings of Abilities I Series Occupation () t=: :l 0 "Cl •.-4 Q) -1-1 A 105 Social Insurance Administration 20 78** 110 Economist 12 74** 180 Psychology 10 51 187 Social Services 6 74 201 Personnel Management 16 74** 334 Computer Specialist 11 75** 343 Management Analysis 13 92** 526 Tax Technician 9 82** 560 Budget Administration 14 84** 570 Financial Institution Examining 9 89** 685 Public Health Program Specialist 6 69 962 Contact Representative 7 87** 967 Passport & Visa Examining 7 81* 993 Social Insurance Claims Examining 10 91** 996 Veterans Claims Examining 8 77* 1081 Public Information Specialist 10 95** 1082 Writing & Editing 9 80** 1102 Contract & Procurement 8 -13 1169 Internal Revenue Officer 11 76** 1710 Education & Vocational Training 16 67** 1712 Training Instruction 6 85* 1810 General Investigating 8 95** 1811 Criminal Investigating 14 81** 1816 Immigration Inspection 6 68 1854 Alcohol, Tobacco, & Firearms Inspec. 10 69* 1890 Customs Inspection 10 88** 2001 General Supply 19 73** Note. Decimal points omitted. * p < .05 ** p ~ .01 74** 91** 92** -22 92** 65* 67* 44 41 96** 95** 89** 96** 73* 63 51 45 73* 90** 68** 92** 72* 88** 86* 80** 41 15 73** 96** 64* 58 82** 76** 90** 88** 81** 87** 90** 92** 95** 90** 88** 96** 81** 75* 81** 84** 95** 97** 48 88* 94** 88** 69** 48* 89** 95** 00 62** 91** 91** 60 35 77* 92** 87** 92** 78** 88** 67* 69* 74* 56 89** 88* 70 86** 63 82** 87** 64** 77** 80** 83** 93** 86** 26 82** 88** 67** 94** 95** 90** 74 87** 84** 90** 98** 92** 95** 80** 85* 92** 40 91* 91** 91** 86** 74** 92** 85** 95** 67** 88** 75** 89** 91** 93** 98** 92** 68 81** 70 77** 91** 81* 91** 88** -56 94** 95** 76 85** 84** 87** 19 and the tedium of the task. It should be recalled that the six raters became familiar with the jobs through the use of classification standards. They were each required to make 1710 separate ratings, that is they rated six abilities for each of the 285 duties of the 27 occupations. Of the 162 reliability coefficients, 134 reached statistical significance. The median value of the 162 coefficients value was .84. Development of test weights for occupations. The subtest weights in Table 7 were developed by application of Equations (1) and (2) and the rounding procedure described in the Method Section. Occupations are ordered in the table according to similarity of weighting pattern. Seven categories accounted for all 27 occupations studied. Since memory could not adequately be measured, the first two categories shown collapsed into one, leaving six categories to cover the 27 occupations. Category A includes fifteen occupations; Category B, eight; and the remaining categories, one each. In the classification system for General Schedule occupations, series number digits to the left represent the group of occupations in which a particular occupation belongs. For example, the 560 series (Budget Administration) belongs to the 5, or Accounting and Budget, occupational group; the 1811 series (Criminal Investigating) belongs to the 18, or Investigation group. Generally, this research found that occu pations in the same group had the same weighting pattern. Abilities received weights of either two or one, with the exception of a weight of three for the verbal comprehension ability for the Writing and Editing occupation. In order to provide another view of the way the jobs tend to cluster according to patterns of AI which they might share in common, an iterative factor analysis with an orthogonal varimax rotation procedure was carried out on the intercorrelation matrix of the ability importance values.3 This analysis was made across the six weighted constructs, that is, correlations in the matrix were based upon N=6. Factors were then interpreted using factor loadings, communalities, and percentages of variance accounted for by each factor. Table 8 shows the results of this factor analysis. Five significant factors factors were extracted, accounting for a total of 100 percent of the variance in the factored matrix. All the variance was accounted for because there were five degrees of freedom and five factors. The factor analysis, therefore, could not properly be used to test hypotheses. The values were based on mean ratings and were, therefore, quite stable. It is recognized that the factor analysis cannot be considered statistically strong. However, the analysis can be useful in describing the way these occupations tended to group together in comparison with the ability weighting patterns. Because all variance in the factored matrix was accounted for in the five factors, all communality values were 1.00. Obviously no items would be omitted in subsequent factor analyses. The factors were named according to guidance provided by Harman (1967, p. 133): "The coefficients of a factor pattern indicate the correlations of the variables with the respective factors and furnish the basis for naming them," and "The investigator is guided by the magnitude of the factor weights in the selection of appropriate names for the factors. The name selected is usually suggested by the nature of the variables having the largest correlations with the factors under consideration. This name should be consistent with the nature of the remaining variables which have low corre lations with the factor." Factor names are considered arbitrary. They are generally necessary 1n research such as this for understanding the nature of the factor. 3The factor analysis started with ones used as first estimates of communalities. The matrix was refactored a number of times; first, until the number of significant roots had been decided, then until the diagonal estimate had stabilized, that is, the sum of squares of the factor loadings for a variable fell within .02 of the diagonal value in the f~ctored correlation matrix (Buhler, 1973). 20 TABLE 7 Rounded Weights of Abilities by Occupation Series Occupation ~ 0 •o-1 ~ c:.l ::l "0 Q) Q ~ 0 •o-1 ~ c:.l ::l "0 ~ H ~ ~ Q) ~ "0 ::l..., :>. 1-1 0 m ~ 1-1 Q) ..0 § z I ~ Q) 0 r-1 1-1 •o-1 Ill P..Cil ..o a ~ 1-1 0 Q) Q)U,.C:: > :>. 1-1 .0 bO Q) ~ Ill u 201 967 993 1712 1810 1811 1816 1890 Personnel Management Passport & Visa Examining Social Insurance Claims Examining Training Instruction General Investigating ·Criminal Investigating Immigration Inspection Customs Inspection 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 A 105 187 343 685 996 1081 1710 Social Insurance Administration Social Services Management Analysis Public Health Program Specialist Veterans Claims Examining Public Information Specialisth Education & Vocational Traini g 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 A' 110 180 526 560 570 1102 1169 2001 Economist Psychology Tax Technician Budget Administration Financial Institution Examining Contract &Procurement Internal Revenue Officer General Supply 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 B 334 Computer Specialist 2 2 2 1 2 1 c 962 Contact Representative 2 1 2 2 1 2 D 1082 Writing & Editing 2 1 2 2 1 3 E 1854 Alcohol, Tobacco, & Firearms Inspection 2 1 2 2 2 2 F 21 No grouping of jobs was hypothesized, so these factors, made up of jobs, are viewed purely in an investigative frame of reference, Factor 1. Table 8 shows the loadings on the first factor. This factor accounted for 49.5 percent of the total variance. The following occupations, with their grade levels and factor loading, are representative of the factor: Educational and Vocational Training (9) .96 Personnel Management (13) .96 Public Information Specialist ·(13) .96 Passport and Visa Examining (10) .95 Writing and Editing (12) .94 Training Instruction (9) .94 The factor appears to have classified jobs together which have verbal, nonquantitative aspects in common. Analysis of duties of these occupations suggests the name for this factor: Individual and organizational examining, training, and analysis. Factor 2. The second factor shown in Table 8 has loadings which group:ijobs having quant1tat1ve applications in common. It accounted for 28.4 percent of the total var1ance accounted for by the analysis. Typical jobs loading highly on this factor were: General Supply (9) .95 Tax Technician (9) .94 Contract and Procurement (12) .92 Budget Administration (13) .88 Internal Revenue Officer (12) .87 Financial Institution Examining (12) .83 Because of the quantitative nature of the work, and the additional common denominator of fiscal and material accountability, this factor was titled: Quantitative application to fiscal, material, and research requirements, Factor 3. This factor accounted for 10.3 percent of the total variance, Important jobs loading on it were: Computer Specialist (12) .99 Economist (14) .58 Psychology (13) .58 Management Analysis (12) .46 This appears to be a research and analytical factor with a significant social component. The factor was named: Systematic research and analysis, The lesser loadings and bipolarity (See Table 8) emphas1zed a soc1al aspect of those jobs. ' Factor 4. A fourth factor, representing 5.9 percent of the variance, grouped the follow1ng jobs: Customs Inspection (9) .57 Criminal Investigating (13) .56 Internal Revenue Officer (12) .41 Immigration Inspection (9) .36 General Investigating (11) .35 The inspection, investigation and enforcement aspects of these jobs provided the obvious factor name: Inspection, investigation, and enforcement, 22 TABLE 8 Varimax Factor Analysis Performed on Weighted Construct Ratings for 27 Job Samples, Giving Jobs Identified by Series Factor Loadings and Communalities with Accompanying Summary Information about Contributions to Variance Ser1es Factors and Communality and Job Sample Journeyman Fl F2 F3 F4 Fs h2 Grade Education and Vocational Training 1710-09 96 ~3 07 06 10 1.00 Personnel Management 0201-13 96 03 14 19 14 1.00 Public Information Specialist 1081-13 96 26 -02 03 10 1.00 Passport and Visa Examining 0967-10 95 -05 16 25 03 1.00 Writing and Editing 1082-12 94 13 -30 04 06 1.00 Training Instruction 1712-09 94 -13 -25 -18 -05 1.00 Immigration Inspection 1816-09 89 21 07 36 13 1.00 Public Health Program Specialist 0685-13 89 41 04 00 18 1.00 General Investigating 1810-11 88 -15 22 35 -19 1.00 Social Insurance Claims Examining 0993-10 86 28 26 08 32 1.00 Social Insurance Administration 0105-10 85 49 -05 -15 11 1.00 Management Analysis 0343-12 83 29 46 -08 06 1.00 Contact Representative 0962-09 83 -31 -44 -07 15 1.00 Criminal Investigating 1811-13 81 -03 16 56 03 1.00 Veterans Claims Examining 0996-12 80 37 13 25 39 1.00 Customs Inspection 1890-09 65 31 -05 57 40 1.00 Social Services 0187-07 64 59 41 26 -04 1.00 General Supply 2001-09 -06 95 -12 -03 28 1.00 Tax Technician 0526-09 03 94 26 20 10 1.00 Contract and Procurement 1102-12 27 92 -21 -14 -13 1.00 Budget Administration 0560-13 -18 88 17 -30 25 1.00 Internal Revenue Officer 1169-12 24 87 04 41 16 1.00 Financial Institution Examining 0570-12 41 83 20 20 26 1.00 Psychology 0180-13 18 79 58 -07 -02 1.00 Economist 0110-14 20 76 58 12 -17 1.00 Computer Specialist 0334-12 -03 11 99 04 11 1.00 Alcohol, Tobacco, & Firearms Inspec. 1854-09 41 36 08 07 83 1.00 Percent Contributions to Variance 49.5 28.4 10.3 5.9 5.8 Cumulative Percent 49.5 77.9 88.2 94.2 100.0 Note. Decimal points are omitted for all entries in the body of the Table. The number of weighted construct ratings for each job in the sample was 6. 23 Factor 5. The fifth factor had the weakest array of factor loadings and accounted for 5.8 percent of the accountable variance, Table 8 shows one loading to be very high, and three others to be weak but supportive: Alcohol, Tobacco, and Firearms Inspection (9) .83 Customs Inspection (9) .40 Veterans Claims Examining (12) .39 Social Insurance Claims Examining (10) .32 Few jobs would be expected to fall into a category such as this. Study of the duties and responsibilities of these occupations suggest> the name: Monitoring and servicing. It was not expected that the grouping of the jobs by factor analysis would match the grouping arrived at by the other method. The great majority of jobs would probably group into a few categories where a standard weighting would be applied, Other jobs would group into a few other c~tegories where the required weighting systems would be somewhat unusual, and at times, unique, Jobs commonly group this way, even when other aspects of the job are used as a basis for grouping. A factor analysis of the type shown in Table 8 provides a cross check of the. groupings of jobs with the same ability weighting patterns. A factor analysis was applied to the same data underlying the weighting patterns to determine whether an objective grouping could be obtained using an independent methodology. As can be seen in a comparison between the weighting patterns in Table 6 and the factor loadings in Table 8, there was perfect agreement between jobs .in occupational category A. All had their highest loadings on Factor 1. Every job in occupational category B loaded .76 or higher on Factor 2. Computer Specialist, the only job in occupational category C, loaded only on Factor 3; a very logical factor in terms of the requirements of the other jobs loading on that factor. Contact Representative, a job requiring personal interactions, and the only job in occupational category D, had a high loading on Fac tor 1, but also had negative loadings on Factors 2 and 3. It is less quantitative and analytical, and this is reflected in the lower weights on number and induction, The Writing and Editing job was in a unique weighting category because of the high verbal comprehension requirement as compared to inductive reasoning requirements. Alcohol, Tobacco and Firearms Inspection had its own abilities weighting pattern, and had the highest loading in Factor 5. Interestingly, 18 of the 27 occupations shown in Table 8 loaded above .30 on more than one factor, The occupations loading on one, two, three, or four factors are shown below: One Two Three Four 1710 967 962 1890 201 1816 996 1081 685 187 1082 1810 1854 1712 993 2001 105 526 343 1102 1811 334 560 1169 570 180 110 24 Examination of Table 8 reveals a logical correspondence between jobs loading on various factors and the known requirements of the occupation. At times, loading on more than one factor indicates jobs of a more complex nature. Some factors reflect more complex job requirements than others; therefore, an analysis of job complexity must involve a review of the nature of the factors as well as their number. Also, jobs can involve a number of factor activities, but at lower levels of those activities. At any rate, the loadings suggest the proper emphasis of abilities 'within occupations. A job such as Public Information Specialist loads only on the verbal factor, (F 1), as the duties of the majority of those positions loading on that factor would suggest. On the other hand, the occupation of Customs Inspector loads on the verbal (FJ), quantitative (F2), inspection/enforcement (F4), and service factors (F5). The duties of the position require interaction with the public, inspection, seizure, arrest, import duty calculation, and learning of many regulations. Again, this diverse activity is not at the highest levels of all factors, but the diverse nature of the occupations is revealed by the factor analysis. The categorization of the occupations makes sense 1n terms of factor names which were assigned to them. (1) Individual and organization examining, training, and analysis. (2) Quantitative application to fiscal, material, and research requirements. (3) Systematic research and analysis. (4) Inspection, investigation, and enforcement. (5) Monitoring and servicing. These groupings could possibly have been obtained without formal analysis. They do not violate present knowledge of job categorization, and can conveniently be used for reference to job groupings having unique requirements. Ratings of KSAO's. Reliability coefficients for the ratings of the importance of each of 31 KSAO's for overall job performance were presented in Table 4. The magnitude of these coefficients indicates that the interrater reliability was high for all rater groups. Appendix C lists the mean rating of each KSAO for each occupation as well as the overall mean and standard deviation for each KSAO. Fully seventeen of 31 KSAO's were rated very important across all 27 occupations; that is, mean ratings over the 1241 SME's were 5.0 or higher. Ten were considered important (~4.0), and four were of some importance (~3.0). All of the KSAO's were therefore considered to be of at least some importance for the occupations. While there was variation from occupation to occupation, the ratings strongly suggest that the KSAO's are relevant to occupations covered by the PACE. Since the KSAO's were rated with respect to overall job performance it was not possible to relate them to specific duties. It was considered important to determine whether the 31 KSAO's could logically be consolidated into fewer, more general groups. It was hypothesized that the reduced number of more general KSAO's would corroborate the six constructs rated by the psychologists. A second iterative factor analysis was therefore carried out using the mean ratings of the 31 KSAO's by SME'-s. This analysis was done across the 27 jobs in the sample; that is, correlations in the input matrix were based upon N = 27. Table 9 shows the results of the factor analysis. Eight significant factors were extracted. They accounted for 83.7 percent of the variance in the factored matrix. Communalities ranged from .52 to .99, indicating that in subsequent analyses none of the abilities need be omitted. 25 Notice that eight of the first 11 KSAO's listed in Table 9 are noncogn1t1ve or personality variables and 17 of the last 20 are cognitive variables, The factor analysis confirms this distinction since the two sets of variables generally load on different factors, The only exceptions to this are factors 2 and 3 which have both cognitive and noncognitive components, Factor 1. The first factor, which contributed 21.1 percent of the variance, was interpreted as an "interacting with people" factor. The abilities which loaded highly· on this factor are: Deal with People .95 Emotional Maturity .87 Tact .87 Empathy .75 Oral Communication .75 The factor appears to _have classified abilities together which call for relating with people on jobs. Table 9 also shows Listening, Persuasiveness, Self Control and Forcefulness as having high loadings on this factor. The title for the important ability reflected by this factor was: Ability to influence and work considerately with others; to interact as a developed, mature employee. The ab1l1ty was not related to any of the ab1l1ty constructs measured by the PACE, Factor 2. The second factor accounted for 11.5 percent of the total variance. It was a factor emphasizing keenness, or perceptiveness, and the following abilities typify the factor: Object Perception .95 Physical Stamina .77 Alertness .63 Integrity .59 Perseverance .51 This factor was considered related to the ability construct of induction as defined in this research, It emphasizes the examination of specific data, recognizing relationships, and solving problems. The title for this factor is: Mental and physical alertness to perceive important aspects of the work environment, Factor 3. This factor represents 11.2 percent of the total variance and is lnterpreted as an innovative planning factor, The following abilities are typical of those loading on the factor: Originality .94 Planning .90 Curiosity .63 Quantitative Reasoning .48 Self Control -.32 Memory -.32 This factor was seen to relate to the ability construct of deduction which emphasizes thinking clearly about the application of principles and developing a system, plan, or procedure. It was titled: Planning and executing original and exploratory work. 26 TABLE 9 Varimax Factor Analysis Performed on Ability Ratings by Subject Matter Experts Giving Abilities, Factor Loadings and Communalities with Accompanying Summary Information about Contributions to Variance Factors and Communalit;! Abilities Fl F2 F3 F4 F5 F6 F7 F8 h2 Deal with People 95 10 -07 -05 -20 02 07 -02 97 Emotional Maturity 87 13 -12 -22 -11 05 16 15 91 Tact 87 29 -31 08 -09 06 08 04 96 Empathy 75 -19 05 07 35 -21 -04 33 89 Oral Communication Listening 75 71 08 24 09 -11 -11 08 08 -20 -02 05 02 27 -02 33 60 81 Persuasiveness 68 10 36 21 12 49 -08 -21 96 Self Control Forcefulness 65 65 44 21 -32 36 -04 03 -06 23 -02 46 40 -13 -04 -25 89 94 Object Perception 02 95 03 05 25 06 05 -09 99 Physical Stamina 28 77 10 -16 -18 02 01 18 77 Alert Integrity 52 39 63 59 20 -05 10 05 -15 -09 04 43 34 -05 -03 14 85 72 Memory 18 49 -32 13 41 18 43 15 80 Originality -10 03 94 16 07 10 03 04 95 Planning 05 -02 90 -15 -05 06 25 04 90 Curiosity 00 34 63 10 -41 20 -24 18 82 Grammar -01 -05 01 97 -04 03 00 06 94 Spelling 14 14 -18 87 09 -16 26 04 93 Written Communication -19 -09 36 79 -01 24 -09 00 85 Quantitative Reasoning -25 20 48 -30 63 09 -04 -32 93 Reading Comprehension -26 10 -05 55 62 14 08 02 78 Letter Writing 10 -17 -08 -06 61 21 22 -11 52 Fund of Information 02 20 -18 37 57 40 19 24 78 Judgment 14 26 04 -06 12 83 19 15 86 General Reasoning Attention to Detail -19 -12 -09 07 34 13 13 10 34 41 76 10 -02 68 09 -30 88 77 Adaptability Perseverance 40 16 03 51 05 30 05 19 16 -12 -02 32 63 60 -02 -03 59 89 Objectivity Arithmetic Computation 36 24 10 -05 02 -25 -02 -25 02 20 23 -01 -06 -26 84 -57 90 62 Percent Contributions to Variance 21.1 11.5 11.2 10.3 8.5 8.2 7.1 5.8 Cumulative Percent 21.1 32.7 43.9 54.2 62.7 70.9 78.0 83.7 Note. Decimal points are omitted from factor-loadings. The number of subject mat~expert ratings for each ability was 27. 27 Factor 4. Factor loadings of the fourth factor indicated the presence of a verbal factor, It accounted for 10.3 percent of the variance. Abilities typical of those loading on this factor were: Grammar .97 Spelling .87 Written Communication •79 Reading Comprehension .55 Verbal comprehension was an ability construct used in the current research, It emphasized knowledge of, understanding of, and ability to manipulate the English language. This factor was titled: Writing, reading, and general verbal ability. Factor 5. This factor appeared to be a general reasoning factor. It accounted for 8.5 percent of variance. Of the 31 abilities, the following loaded highly on this factor: Quantitative Reasoning .63 Reading Comprehension • 62 Letter Writing •61 Fund of Information .57 Curiosity -.41 All of above abilities are generally associated with the concept of "intelligence" as it is ordinarily defined. This factor was titled: Intelligent and knowledgeable reasoning ability. Factor 6. The loadings for Factor 6 strongly indicate a factor of judgment, It accounted for 8.2 percent of the variance, Abilities loading on this factor were: Judgment .83 General Reasoning • 76 Persuasiveness .49 Forcefulness .46 Integrity • 43 The ability construct Judgment, used in this PACE research emphasized solving problems when some facts are om~tted, making reasonable assumptions, and producing reasonable decisions. This factor, which supports the use of a judgment ability in PACE, was titled: Ability to make and act upon effective judgments, Factor 7. The loadings for the seventh factor in Table 9 highlight the characteristics of a problem solver. It accounted for 7.1 percent of the variance. The highest loadings of KSAO's for this factor were: Attention to Detail .68 Adaptability ,63 Perseverance .60 Self Control .40 This factor is logically related to goal orientation and inductive reasoning with implications for productivity, However, it is recognized that problem solving is an important aspect of most jobs, as was the personal interaction aspect of Factor 1. This factor was named: Persistent and adaptive attention to details of the work. 28 Factor 8. Factor 8 appears to be an objectivity factor as opposed to a factor of quantitative ability. It accounted for 5.8 percent of the variance. Abiiities loading highly on Factor 8 were: Objectivity .84 Empathy .33 Listening .33 Quantitative Reasoning -.32 Arithmetic Computation -.57 This factor is distinctly bipolar and could indicate a sensitive counseling type of objectivity apart from anything computational or quantitative. It was not directly related to any abilities measured by the PACE. It was named: Empathetic and receptive objectivity, vs. quantitative reasoning and computation. The factor analysis provided support for four of the s1x abilities studied in this research for possible inclusion in the PACE. Two of the cognitive abilities were factorially complex; that is, they loaded significantly on a number of factors. Quantitative reasoning and arithmetic computation, which would be equivalent to the ability, number, loaded high on Factors 3, 5, and 8. The complex character, memory, while not emerging as a factor in its own right, was present in four factors as one of the KSAO's which had a reasonably high loading on each. Table 9 shows memory to load significantly on Factors 2, 3, 5, and 7. The characteristics of self control and perseverance are the only other characteristics of the 31 to load significantly on as many as four factors. The factor analysis points up the need for measurement of the ability to interact with people. The SME raters consider that many of the noncognitive characteristics reflect this one underlying factor. These data indicate the need to develop a valid measure of interpersonal skills. Past research has shown it to be a desirable but elusive variable to measure. Factor 7 is also interesting in that it relates to induction, and the KSAO's having major loadings are characteristics of productive problem solvers. Research is needed to develop measures of persistence and problem solving abilities. The factor analysis supported the six abilities on which a test could be based. Four of the abilitie's emerged as factors in their own right.. A fifth ability, number, is a component of several of the factors, and can be readily measured. Memory also appeared in several factors, but it was not feasible to include it in the test. Assignment of the remainder of PACE occupations to occupational categories. Since some 120 occupations meet the PACE scope of coverage requ1rements, and since the occupational categories were developed on a sample of 27 of these occupations, it was necessary to ascribe the remaining occupations to categories so that the examination could be used to meet the requirements of the Government. This became a task of the Standards Division of the Commission's Bureau of Policies and Standards. Occupational specialists in that division are responsible for development of classification and qualification standards for all occupations. The Standards Division developed a plan to obtain ratings of SME's to determine duty importance and time spent for their own occupations and to obtain ratings of occupational specialists of the importance of the abilities abilities for performance of duties. Weighting patterns would then be established as with the present study of 27 occupations. As a test of the agreement in ability rating between psychologists and occupational specialists, seven occupations rated by psychologists were also rated by a group of six occupational specialists. Only two 29 differences in the weighting patterns were found, and they could be explained by rounding error. It appeared that both groups could rate abilities reliably and consistently. As an interim measure, a senior occupational specialist who was thoroughly familiar with PACE occupations evaluated each remaining occupation in the context of the five abilities. He placed each occupation into an existing occupational category pending completion of the more comprehensive approach described above. SUMMARY AND PROSPECTUS The objectives of the project were met: 1. Five cognitive abilities were identified as being required for successful per formance in the occupations to be covered. These abilities were carefully defined, in order to enable analysis of their relations to performance of job duties. 2. A systematic method of job analysis, which could provide assessment of the an relative importance of the abilities for successful performance, was developed. Duties of an occupation were determined from classifi~ation standards and substantiated by subject matter experts. The duties were rated for importance and time spent. Psycholo gists evaluated the importance of each cognitive ability for successful performance of each duty. These three sets of ratings were mathematically combined to determine the the relative importance of each ability for the occupation. 3. A new written test to assess the cognitive abilities was planned and developed. Test scores could be weighted in six different patterns, for six different categories of occupations. 4. It was determined that future research should be directed toward several goals. First, the written test should be assessed to ensure the accuracy with which it measures the abilities. New item types may be required. Second, the abilities themselves should be studied further, within the test and on the job, to evaluate their independence. Third, means of assessing new abilities, such as the very important ability to interact with people should be developed. A fourth line of research will elaborate the construct validity of the new exami-nation. Of major importance to such research are criterion-related validity studies,where they are technically feasible. A plan has been developed and implemented to carryout such studies. These will assess the definitions of the abilities both in the written test and on the job, and the linkage between abilities in the test and abilities on the job. They will illuminate the problem of deciding what abilities are required in the job and how they may be assessed most appropriately. They will also substantiate the precision of the job analysis method. 30 REFERENCE NOTES 1. Primoff, E. S. Personal communication, September 1976. 2. Primoff , E . S . Second Edition. Test selection by job analysis. Test Technical Series No. Washington, D.C., U.S. Civil Service Commission. 1955. 20, REFERENCES American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. Standards for Educational and Psychological Tests. American Psychological Association: Washington, D.C., 1974. Buhler, R. P-Stat -A computing system for file manipulation and statistical analysis of social science data. Princeton, N.J.: Princeton University Computer Center, 1973. Chaplin, J.P., & Krawiec, T. S. Systems and theories of psychology. (2nd edition). New York: Holt, Rinehart and Winston, 1968. Christal, R. E. The state of the art in occupational research and development. The U.S. Air Force Occupational research project AFHRL-TR-73-75, AD-774. Lackland AFB Tex.: Occupational Research Division, Air Force Human Resources Laboratory, January 1974. Cronbach, L. J. Essentials of psychological testing. (3rd edition) New York: Harper and Row, 1970. Cronbach, L. J., & Meehl, P. E. Construct validity 1n psychological tests. Psychological Bulletin, 1955, 52, 281-302. Dawes, R. M., & Corrigan, B. Linear models 1n decision-making. Psychological Bulletin, 1974, ~. 95-106. French, J. W. The description of aptitude and achievement tests in terms of rotated factors. Psychometric Monographs, 1951, No. 5. French, J. W., Ekstrom, R. B., & Price, L.A. Manual for kit of reference tests for cognitive factors. Princeton, N.J.: Educational Testing Service, 1963. Goodenough, J. P. Mental testing: its history, principles, and applications. New York: Rinehart, 1949. Guilford, J. P. The nature of human intelligence. New York: McGraw-Hill, 1967. Guilford, J. P., & Hoepfner, R. The analysis of intelligence. New York: McGraw-Hill, 1971. 31 Harman, H. H. Modern factor analysis (2nd edition). Chicago: University of ChicagoPress, 1967. Harman, H. H. How factor analysis can be used in classification. In A. Kockova'-Kratochvilova', and E. Minarik (Eds.) Yeasts models in science and technics. Proceedings of the First Specialized International Symposium on Yeasts: Smolenice, June 1971, Publishing House of the Slovak Academy of Sciences, Bratislava, 1972. Korman, K. Industrial and organizational psychology. Englewood Cliffs, N.J.: Prentice-Hall, 1971. Theologus, G. C., Romashko, T., & Fleishman, E. A. Development of a taxonomy of human performance: a feasibility study of ability dimensions for classifyinghuman tasks. Technical Report No. 5. Washington, D.C.: American Institutes for Research, January 1970. Thorndike, R. L. Personnel selection. New York: Wiley, 1949. Thurstone, L. L. Primary mental abilities. Psychometric Monographs, 1938, No. 1. Thurstone, L. L., & Thurstone, T. G. Factorized Studies of Intelligence. Psychometric Monographs, 1941, No. 2. Tuddenham, R. D. The nature and measurement of intelligence. In L. Postman (Ed.)Psychology in the making. New York: Knopf, 1962. U.S. Civil Service Commission. Tests and other applicant appraisal techniques. Federal Personnel Manual System, FPM Supplement 271-2. Washington, D.C.: Author, 1972. U.S. Civil Service Commission. Federal Personnel Manual, Chapter 337, Section l-Si. Washington, D.C.: Author, 1969. United States Code, Title 5, Sections 3304, 3313, 3317, 3318, 1970 Edition: Washington, D.C., U.S. Government Printing Office, 1971. 32 APPENDIX A Thirty-one Knowledges, Skills, Abilities, and Other Characteristics Required for Successful Performance in Twenty-seven Occupational Series 1. Adaptability To be able to deviate from routine work methods. The ability to change a longstanding work method or a personally designed procedure based on a change in circumstances or valid criticism. This ability is important when the changes are sporadic and have far-reaching consequences. 2. Alert To be prompt to perceive and act; to maintain a constant state of watchfulness. Ability to maintain sustained attention to immediate happenings. This ability is important when an employee is required to monitor signals or events and react quickly to them. 3. Arithmetic Computation The ability to perform routine arithmetic operations quickly and accurately. The ability to quickly and accurately add, subtract, multiply, and divide whole numbers, fractions, and percentages, etc. This ability is important when the employee is required to perform or check a substantial amount of arithmetic operations. 4. Attention to Detail The ability to pay attention to many details in designing or implementing a system, procedure or program so that there are few or no unanticipated contingencies. It is the ability to handle many details so that they work harmoniously towards the desired end. It is also a willingness to become immersed in and challenged by a large amount of detail. 5. Fund of Information The ability to amass, keep current and integrate a large fund of information. The employee should have the ability to keep current in a large number of fields of knowledge. The knowledge need not be profound but should satisfy the work requirements and should provide a basis for accumulating more advanced information. 6. General Reasoning The ability to think clearly in factual, symbolic, or figural terms; to draw the correct implications from the stated conditions or produce a plan. This ability is important when an employee is required to utilize sufficient but complex information to reach a conclusion or produce a plan. 33 7. Grammar Knowledge of the structure of words and their customary arrangements into sentences. Knowledge of the formally accepted conventions with respect to the English language. This ability is particularly important in composing or editing written material and where a large amount of formal conversation is required. 8. Judgment The ability to reason, using incomplete or conflicting information. The employee, based on given information, can in many cases correctly infer that unspecified events have probably occurred or are likely to occur. The unspecified events are not comr pletely determined by the events that are stated, but represent the most likely occurrence, based upon past experience. This ability is particularly important where action must be taken on incomplete information. 9. Letter Writing . The ability to quickly compose clear and concise correspondence relating to routine office matters. The correspondence clearly conveys the intended message. The correspondence generally deals with routine matters for which somewhat standardized procedures have been developed. This ability is important when a substantial amount of time is devoted to composing correspondence relating to standardized operations. 10. Listening The ability to obtain oral information; to pay attention to what another person is saying in order to obtain needed information. The ability not to intrude with irrelevant information or not to misinterpret what is being said. 11. Memory The ability to retain a large quantity of information. The information need not be integrated into related fields of knowledge. This ability is important when a large quantity of information must be quickly recalled, or where the reference system for storing information is inadequate. 12. Objectivity To be able to view a situation without the interjection of any personal bias. To be able to think clearly about emotionally laden material. The ability to see things without any personally produced distortion. This ability is important when the employee is required to reason clearly concerning matters in which he has some sort of personal involvement. 13. Oral Communication The ability to convey information orally. The employee orally transmits information so that others understand what he is saying. He can anticipate where his listeners might have difficulty in understanding because the material is difficult or ambiguous and he makes the material intelligible. This ability is important where oral information is complex or is routine but must be presented quickly. 34 14. Originality The ability to devise new solutions to problems; to see connections between apparently unrelated events in order to produce an improved product. To uncover unusual relations among items of information instead of the more commonplace. This ability is important in positions which involve the development of programs, scientific research, and other functions involving the solution of difficult and unique problems. 15. Object Perception The ability to note subtle visual cues in objects or forms; to be able to detect slight changes or deviations from the usual appearance of objects. This ability is especially important where the employee is required to scan objects quickly and reach some kind of decision based on their appearance. 16. Planning The ability to design a method for producing a desired end. The method requires the integration of several actions. The employee must anticipate the possible consequences of each of the acts in the series and accommodate them to the desired end result. 17. Quantitative Reasoning The ability to solve problems by performing complex manipulations with quantitative symbols. The manipulations are efficient in eliminating unnecessary steps in solving the problem. It is also the ability to solve quantitative problems using alternate approaches; in this way answers can be checked. This ability is important when an employee is required to solve complex quantitative problems where the approach to be utilized is not specified. "Quantitative Reasoning" is a special form of "General Reasoning". 18. Reading Comprehension The ability to understand complex written materials. It is the ability to read and understand written materials quickly, and to retain the content for at least a limited period of time. 19. Spelling The ability to form words from letters according to accepted English usage. The written words are correct since they conform to common practice/ This ability is important when written materials are produced in final form. 20. Written Communication The ability to impart complex information in writing. The employee produces written information so that others understand the presented material. The employee can antici pate possible misinterpretations of the written material that he prepares and eliminate them. This ability is important where an employee is responsible for producing complex written material that must be easily understood. 35 21. 22. 23. 24. 25. 26. 27. 28. 29. Curiosity A desire to understand the reasons for happenings or behavior. A dissatisfaction with superficial or arbirtary explanations which leads to probing for more basic causes of phenomena. This trait is especially important in research positions and in positions which require deep and detailed familiarity in a rapidly changing field or specialization. Deal with People The ability to deal with people effectively and equitable, and to be able to work with others toward a mutually positive end. The ability to get along with people in maintaining a mutually productive relationship. Emotional Maturity To have emotional reactions appropriate to their stimulating conditions and to be able to control their expression so that they have productive consequences. To react to difficult situations so that behavior is appropriate in effecting a desirable change or result. Empathy The ability to see things from the other person's point of view. It is the abi~ity and desire of an individual to put himself in another person's shoes in order to one understand his difficulties. ·It is an habitual way of reacting in which is at least partially successful in understanding the behavior of others. Forcefulness To have enterprise and initiative in interpersonal relations. To be able to advocate a position, point of view, or opinion despite opposition. To exert influence over others. Integrity The quality of being dependable and not violating a legitimate trust. A person who has integrity will discharge his duties so that rules are followed and he conforms to the commonly accepted standards of right behavior. Is incorruptible. Perseverance The ability to persist in a task, problem, or undertaking in spite of difficulties, opposition, or discouragement. A persevering person will not permit difficulties to change his productively oriented approach to a task or problem. He attacks his work tenaciously until it is completed. This ability is important when the work may present many obstacles or distractions to successful completion. Persuasiveness The ability purposefully to influence the behavior of others. Can by effort cause The ability to be individuals to adopt a point of view or modify their behavior. personally compelling. Persuasiveness is a specific form of forcefulness. Physical Stamina The ability to expend a great deal of physical energy. The employee can respond with a large amount of physical force and strength. This refers both to a con tinuing requirement for the.expenditure of energy or a sporadic use of an extraor dinary amount· of physical force. 36 30. Self Control The ability to concentrate on the work task at hand regardless of any adverse emotional, mental, physical consequences. The ability to stifle nonproductive reactions in order to focus on the work to be performed. ,This ability is important where the employee is exposed to provocative or frustrating conditions which could significantly interfere with effective work performance. "Self Control" is a more specific form of "Emotional Maturity". 31. Tact The ability to avoid offense in dealing with people. It is the ability to deal with potentially unpleasant interpersonal situations in a manner such as not to produce antagonism to "Deal with People" and both are related to "Emotional Matu rity" and Self Control". 37 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. APPENDIX B Means and Standard Deviations of Ratings of the Importance of Duties (I) and Time Spent in Performance (T) by Subject Matter Experts in Twenty-seven Occupations Mean Mean SD SD Dut I T I T GS-105 Social Insurance Administration (N=68) Administers and/or participates in a public information and public relations program relating to Federal Social Security programs. 2.94 2.31 2.20 2.08 Participates in leadership capacity in community activities related to Social Security programs. 1.54 .99 1. 78 1.26 Adjudicates Social Security claims. 4.32 4.18 3.11 3.09 Initially develops Social Security program claims. 4.21 4.03 3.25 3.17 Analyzes Social Security program to assure that it meets the needs of the region or locality. 1.71 1.10 2.19 1. 79 Analyzes contractors and/or State agencies for performance. 1.66 1.54 2.61 2.41 Develops and maintains contacts and liaison with specific contractors and/or State agency personnel. 1.81 1.59 2.50 2.23 Receives and analyzes beneficiary complaints concerning program fraud and abuse. 3.51 2.56 2.16 1.96 Supervises employees, forms internal office policy and establishes relationships with other offices. 2.01 1.56 2.06 1.80 Visits certain classes of providers to explain program needs and requirements. 1.12 .84 1. 74 1.46 Performs technical analysis and review of health insurance-related reports. 1.96 1.59 2.15 1.96 Provides technical information and assistance to district offices, State agencies, and the public on program matters. 3.85 3.47 2.33 2.31 Plans and conducts training programs. 3.09 2.25 1.93 1.69 Develops and writes procedural instructions. 2.45 2.02 2.43 2.14 38 APPENDIX B (Continued) Mean Mean SD SD Duty I T I T GS-105 Social Insurance Administration (N=68) (Continued) 15. Coordinates with other elements in SSA and other government agencies. 16. Project management of responsible functions for which procedures have been developed. 17. Writes letters, memos, and detailed reports. 18. Designs and revises forms. 19. Establishes policy based on the laws and regulations. 20. Prepares regulations and manuals. GS-110 Economist 1. Conducts research into economic phenomena tracing cause and effect relations to solve practical problems or administer programs. 2. Applies professional knowledge of economic principles and methods to the solution of a broad range of practical problems and/or the administration of a program. 3. Collects and/or interprets economic data. 4. Utilizes statistical methods. 5. Keeps abreast of latest findings in area of specialization and in general field. 6. Learns and applies large store of technical knowledge to his work. 7. Recommends course of action or reaches conclusions based on evidence that has significant impact on theory or practice. 8. Supervises other economists. 9. Makes policy recommendations based on economic data and analysis. 10. Briefs policy officials on economic matters. 11. Represents agency and U.S. Government with respect to economic matters. 12. Writes reports for publication. 2.56 2.43 4.54 1.69 1.04 .52 (N=40) 4.77 5.40 5.28 4.10 5.35 5.00 4.85 4.42 4.20 3.75 3.95 5.29 2.15 2.13 4.02 1.27 .67 .39 3.85 4.67 4.85 3.44 3. 72 4.38 3.89 3.69 3.15 2.56 2.51 4.17 2.37 2.19 1.69 1.81 2.04 1.39 1.67 1.43 1.22 1.30 1.19 1.54 1.64 1.88 1.84 1.68 1.89 1.35 2.09 2.08 1.67 1.30 1.45 1.20 1.99 1.59 1.46 1.80 1.61 1.94 1.87 1.92 1.99 1. 76 1.68 1.66 39 01A B (Continued) Mean Dut I GS-180 Psychology (N=9) 1. Experiments with organisms to develop laws of behavior. 3.56 2. Applies knowledge of psychological principles and methods to the solution of a broad range of practical problems. 5.33 3. Utilizes statistical methods. 4.78 4. Keeps abreast of latest findings in area of specialization and in general field. 5.78 5. Learns and applies large store of technical knowledge to accomplish work. 4.78 6. Recommends course of action or reaches conclusions based on evidence that has significant impact on a theory or practice. 5.00 7. Designs and conducts research studies. 5.11 8. Evaluates research designs. 5.56 9. Designs and conducts educational training programs. f.oo 10. Evaluates educational training programs. 2.11 GS-187 Social Services (N=l6) 1. Provides assistance to individuals and families served by social welfare programs by serving as intermediary between client and welfare agency. 5.75 2. Investigates and determines clients' resources. 4.50 3. Determines kind and degree of personal problems of client in order to prescribe needed assistance. 4.63 4. Determines kind and amount of social service entitlement. 4.44 5. Prescribes unique social services where required. 3.13 6. Modifies social service plan where changes in circumstances warrant. 3.13 Mean T 2.13 5.25 3.88 4.25 4.63 5.38 4.88 5.50 1.63 1.88 5.19 4.19 4.25 3.88 2.63 2.75 SD I 2.96 1.22 1.48 1.48 2.05 1. 73 1.62 1.24 1.80 1.83 1.65 2.61 1.09 2.42 1.36 1.63 SD T 2.59 1.28 1.96 1.98 2.00 1.60 1. 73 1.20 1.!.9 1.46 1.87 2.68 1.29 2.50 1.50 1.77 40 Dut APPENDIX B (Continued) Mean I Mean T SD I SD T GS-201 Personnel Management (N=77) 1. Develops and/or evaluates personnel regulations, policies, or programs for one or several sub-areas of the total personnel program. 5.16 4.43 1.98 2.16 2. Implements personnel regulations and programs in an operational setting for one or several sub-areas of the total personnel program. 4.55 4.13 2.37 2.44 3. Participates with operating management in long range planning with respect to personnel aspects of organization program. 3.83 3.13 2.27 2.30 4. Advises management with respect to merit principles and personnel management concepts. 4.62 3.96 2.27 2.09 5. Deals with both management and labor in an advisory and adversary relationship. 3.37 2.70 2.47 2.29 6. Keeps abreast of the latest personnel regulations. 5.30 4.10 1.67 1.90 7. Matches applicants with vacancies applying knowledge of personnel qualifications, job requirements, and merit principles. 2.80 2.34 2.34 2.20 8. Classifies positions according to type of work, difficulty level, and degree of responsibility. 3.20 2.36 2.64 2.34 9. Resolves relationship problems existing between individual supervisors and employees. 2.95 2.24 2.35 1.93 10. Advises personnel on human_relations in the work setting. 3.12 2.41 2.27 2.00 11. Advises personnel on employee benefits programs and options. 2.14 1.61 1.98 1.72 12. Represents management in negotiating with employee organizations. 1.92 1.28 2.44 1.82 13. Serves as organizational consultant in field of employee development and training,providing knowledge of facilities available, permissible practices. 2.55 2.05 2.15 2.03 14. Conducts employee training on selected subjects. 2.57 2.03 1. 75 1.50 15. Determines training needs of personnel and arranges for conduct of training. 2.61 1.86 2.18 1.68 16. Develops and/or implements social action program. 5.00 4.26 1.91 1.91 41 Duty GS-334 ·Caii!puter ·SpeCialist 1. Writes efficient computer programs using specialized computer language. 2. Develops requirements for computer based information processing system. 3. Tests program to insure that it produces desired information within the constraints of the information system. 4. Corrects incorrect or inefficient program. 5. Learns subject matter field in order to develop computer applications. 6. Operates computer and peripheral equipment. 7. Develops detailed programs or systems speci fications and related documentation. 8. Provides training for trainee programmers. 9. Coordinates with users/customers, and other organizational levels. 10. Assists in developing the logic for data systems. 11. Evaluates equipment utilization and makes recommendations for increasing efficiency of operations. GS-343 Management Analysis 1. Develops and improves the effectiveness of work methods and procedures by applying management tools and techniques. 2. Analyzes and evaluates the effectiveness of work methods and procedures. 3. Keeps abreast of latest technical findings in field of management analysis in order to perform duties. 4. Recommends changes in management operations that have significant impact on organization and its personnel. APPENDIX B (Continued) Mean t1ean I T (N=73) 5.21 4.55 4.78 3.78 5.99 4.68 5.14 3.59 5.08 3.94 1.25 .65 5.15 4.29 3.51 2.43 5.12 4.09 4.65 4.18 2.24 1.71 (N=60) 4.43 3.90 4. 72 4.18 4.25 3.15 4.45 3.62 SD I 2.24 1. 62 1.56 2.03 1.42 1.33 1.83 2.05 1.27 1.49 1.89 1.60 1.39 1.47 1. 73 SD T 2.33 1.68 1.68 1.90 1.71 .91 2.03 1. 74 1.65 1. 70 1.61 1.64 1.57 1.46 1. 87 42 APPENDIX B (Continued) Dut GS-343 Management Analysis Mean I (N•60) Mean T SD I (Continued) SD T 5. Consults with operating management both in adversary and advisory capacity. 4.78 4.10 1.47 1.54 6. Thoroughly learns the functions and specific activities of the organization to be analyzed in order to perform duties. 5.38 4.50 1.57 1.77 7. Administers management improvement programs. 2.68 2.00 1.50 1.56 8. Recommends changes in operating policies, plans, and objectives. 4.65 3.95 1. 63 1. 73 9. Participates in design of management systems. 3.78 3.17 1.62 1.66 10. Develops management tools. 3.22 2.85 1. 74 1. 78 11. Develops and conducts training programs field of management analysis. in 2. 77 1.98 1.58 1.36 12. Keeps abreast and applies knowledge of puter processing. com 4.07 3.34 1.92 2.17 13. Serves as troubleshooter when management problems arise. 4.80 4.07 1.86 1.87 GS-526 Tax Technician (N=53) 1. Examines tax returns to verify their conformance with Internal Revenue regulations, sometimes involving personal contact with taxpayer. 6.66 6.53 .88 1.05 2. Applies knowledge of elementary accounting in order to perform duties. 4.92 4.28 1.80 1.57 3. Interprets precedent cases with respect to their relevance to the case under investigation. 5.25 4.43 1.30 1.60 4. Reaches decisions tax returns. on substantive issues in 6.42 5. 72 .93 1.54 5. Decides on best way to conduct examination; involves subtle distinctions in line of questioning. 6.06 5.25 1.13 1.70 43 APPENDIX B (Continued) Duty Mean I Mean T SD I SD T GS-526 Tax Technician (N=53) (Continued) 6. Assists taxpayer in preparation of returns during filing session. 3.74 2.81 1. 70 1.82 7. Answers taxpayers' inquiries involving tax planning and complex research issues. 3.40 2.45 2.18 2.00 8. Classifies and selects returns for examination. 4.83 3.44 1.62 1.71 9. Guides new tax auditors in technical and procedural problems. 4.27 3.06 1.56 1.84 GS-560 Budget Administration (N=70) 1. Makes budget estimates for organizational component. 3.97 3.57 2.38 2.37 2. Presents budget estimates to responsible Federal officials or to Congressional committee. 4. 34 . 3.23 2.47 2.17 3. Determines allowable funds after funds have been appropriated. 3.99 3.23 1. 75 1.65 4. Develops a system of management controls over the obligation and expenditure of funds. 3.93 2.86 2.03 1.80 5. Maintains a system of management controls over the obligations and expenditure of funds. 4.50 3.78 1.98 2.06 6. Simultaneously works budgets. on present and future 5.12 4.79 1.59 1.49 7. Modifies budget allocations based on actions taken by Bureau of Budget and by Congress. 4.31 3•.12 2.05 1.81 8. Coordinates budget of diverse operating programs. 4.11 3.57 2.83 1.82 9. Develops budget compilation and review process. 4.63 4.20 1.82 1.95 10. Critically analyzes and reviews proposed budgets to highlight issues for review and analysis by decision makers. 5.36 4.76 1.90 1.98 11. Supervises other budget personnel. 2.57 2.18 2.40 2.56 (continued) APPENDIX B (Continued) Mean Mean SD SD D I T I ------------------------~G~s~-~5~6~0~--JB~u~d~g~e~t~A~d~m~i~n=i2s.t~ra~t~1~·o~n~--~~----(~C~o_n~t~i~n~u~e~d~)________ 12. Critically analyzes and reviews proposed and pending legislation to highlight issues for review and analysis by decision makers. 4.15 3.31 2.32 2.28 13. Reviews agency rules, programs, and statistical forms to determine conformance with executive program. 3. 77 3.15 2.34 2.29 14. Develops public information and relations material. 2.96 2.32 2.26 2.04 GS-570 Financial Institution Examining (N=20) 1. Plans and performs examinations of financial institutions in order to determine financial condition and extent of compliance with regulations. 6.30 5.90 .86 1.33 2. Applies accounting knowledge. 4.15 3.50 .81 1.05 3. Deals with individuals in financial institutions in order to obtain cooperation. 5.00 4.35 .97 1. 69 4. Decides whether some malpractice occurs and prescribes corrective steps. 4.80 3.40 1.10 1.43 5. Applies specialized knowledge of real estate, agricultural, small business, economic conditions, or other factors affecting institution examined. 5.55. 5.45 1.19 1.32 6. Improvises special examining procedures where standard procedures inadequate. 4.35 2.95 .93 1.19 7. Serves as working leader for a team of examiners. 5.20 5.15 1.11 1.53 8. Investigates applications for branch offices, new banks, relocation, and mergers. 4.50 2.90 1.15 1.59 9. Evaluates competency of bank management. 6.00 5.30 1.03 1.53 GS-685 Public Health Program Specialist (N=20) 1. Advises on or conducts studies relating to community public health needs. 4.50 3.70 1.05 1.53 2. Applies knowledge of medical, financial, statistical, and legal procedures. 5.00 4.70 1.38 1.45 45 APPENDIX B (Continued) Mean Mean SD SDDut I T I T GS-685 Public Health Program Specialist (N=20) (Continued) 3. Advises on best tactics to overcome logistic, medical, or legal obstacles. 5.00 4.60 1. 34 1.10 4. Develops programs which motivate groups and individuals to take appropriate health action. 5.00 4.70 1.59 1. 38 5. Meets and deals with professional and political groups and individuals in order to further program objectives. 5.55 4.90 1.00 1. 74 s. Provides information and advice on requirements, .policies,and procedures for obtaining Federal grants. 4.95 4.00 1.23 1.52 GS-962 Contact Representative (N=40) 1. Explains and advises on benefit provisionsof regulations and procedures. 6.32 5.82 .86 1.20 2. Learns applicable rules, regulations, andprocedures; must know them in depth. 6.32 5.80 .92 1.26 3. Assists public in preparing forms andapplications. 6.02 5.85 1.07 1.41 4. Motivates clients to follow prescrtbedprocedures. 5.65 5.38 1.25 1.25 5. Represents claimants in both formal andinformal appeals before rating boards. 4.35 2.42 1.73 1.95 6. Has broad knowledge of city, State, andFederal agencies' benefits and other communityresources. 5.22 4. 72 1.27 1.50 7. Discusses benefits before private and public groups, appears on radio and TV, and conducts orientation lectures. 5.10 3.30 1.19 1.77 GS-967 Passport and Visa Examining (N=l6) 1. Issues passports and visas by applying statutesrelating to citizenship and nationality. 5.75 4.50 1.81 1.97 2. Adjudicates loss of citizenship and nationalityby applying appropriate statutes. 5.50 4.88 2.25 2.39 46 APPENDIX B (Continued) Dut Mean I Mean T SD I SD T GS-967 Passport and Visa Examining (N=l6) (Continued) 3. Applies broad knowledge of previous and still applicable citizenship laws, domestic relations laws, and foreign citizenship laws in order to issue passports and visas. 6.25 5.94 .86 .77 4. Grants recognition of citizenship. 5.80 4.40 1. 74 1.68 5. Keeps abreast of latest laws relating to immigration and U.S. and foreign citizenship. 6.44 5.63 .81 1.36 6. Examines documentary evidence relating to applicants' identity and birth. 5.67 4.80 1.91 1.82 7. Drafts correspondence relating to citizenship and visas. 5.53 5.60 .83 .99 GS-993 Social Insurance Claims Examining (N=78) 1. Develops, examines, adjusts claims for retirement, survivor, or disability benefits, health insurance, supplemental security income. 6.58 6.11 .71 1.26 2. Adjudicates and authorizes claims for retirement, survivor, or disability benefits, health insurance, supplemental security income. 6.62 6.39 .69 1.11 3. Learns and applies laws and implementing regulations for social security and/or railroad retirement insurance programs. 6.21 5.59 1.01 1.31 4. Learns and applies other pertinent legal matters relating to domestic relations, descent, and distribution of property. 5.17 4.49 1.40 1.53 5. Interprets and applies law in diverse circumstances and prepares special determinations on complex questions of fact and law. 5.27 4.40 1.44 1. 74 6. Researches legal interpretations and precedents. 4.92 3. 72 1.57 1.72 7. Applies lay medical knowledge with respect to disability claims. 2.90 2.19 2.05 1.89 8. Participates in training. 4.45 3.15 1.76 1.35 9. Examines corporate, partnership, and individual tax returns. 5.11 3.94 1.23 1.63 10. Develops evidence, medical and lay, in order develop and determine proper payees. to 5.44 3.83 1.10 1.47 47 APPENDIX B (Continued) Mean Mean Dut I T GS-996 Veterans Claims Examining (N=40) 1. Develops, examines, adjusts claims filed in connection with veteran's benefits by veterans, their dependents and beneficiaries. 5.10 4.57 2. Adjudicated and authorizes claims filed in connection with veteran's benefits by veterans, their dependents and beneficiaries. 6.63 6.15 3. Learns and applies laws and implementing regu lations for veteran's benefits and insurance programs. 6.42 5.70 4. Learns and applies other pertinent legal matters such as domestic relations, descent, and distribution of property. 4. 72 4.05 5. Interprets and applies veterans entitlements in wide range of circumstances and claims. 5.85 5.40 6. Refers to interpretation and legal precedents. 5.40 4.40 7. Acquires lay medical knowledge with respect to disability claims. 4.95 4. 72 8. Acts as a member of a hearing panel where veteran or claimant may present evidence in behalf of his claim. 4.57 2.90 GS-1081 Public Information (N=25) 1. Organizes and presents to interested parties, information about the work of the Government agency. 5.28 4. 72 2. Participates with management in deciding how the agency and the public can best be served through the public information program. 3. 72 2.84 3. Writes and edits material used in public information program. 5.16 5.20 4. Determines the best way to further the agency program through the use of the information media. 4.24 3.36 5. Works with organized outside groups in order to present agency program. 3. 72 3.17 SD I 1.45 •67 .98 1.43 1.35 1.24 1.88 1.48 1.65 2.34 1.52 1.92 1.49 SD T 1. 75 1.19 1.29 1.47 1. 37 1.45 1.85 1.69 1.84 1. 75 1.61 1.87 1. 76 48 APPENDIX B (Continued) Dut Mean I Mean T SD I SD T GS-1081 Public Information (N=25) (Continued) 6. Anticipates adverse reactions and plans the best way to further the work of an agency through the use of information media. 4.04 2. 92 1. 93 1.82 7. Supervises and trains lower grade employees. 2.76 2.08 1.42 1.47 8. Researches and analyzes subject matter materials. 4.36 3.68 1.68 1. 70 9. Keeps abreast of communication techniques. 4.40 2.96 1.04 1. 27 10. Writes contract specifications and monitors contracts. 2.25 1.33 2.01 1.56 GS-1082 Writing and Editing (N=29) 1. Writes and edits verbal material which appears in publications or reports to the information media. 5. 72 5.48 2.19 2.23 2. Acquires information about diverse subjects about which writing and editing. 6.20 5.67 1.24 1.52 3. Develops material appropriate to for which designed. the group 5.40 4.97 1.85 2.03 4. Conducts and edits radio interviews. 1.37 .87 2.08 1.41 5. Adapts and/or translates material for foreign audiences. 2.47 2.17 3.01 2.74 6. Researches and obtains already published material for dissemination or republication. 3.90 3.31 2.27 2.29 7. Plans format, typography, for publication. and illustrations 4.18 3.55 2.04 1.92 8. Reviews manuscrip1ts department policy. for conformance with 4.55 3.73 2.25 2.10 9. Researches and writes speeches. 2.45 2.36 2.73 2.69 49 APPENDIX B (Continued) Mean Dut I GS-1102 Contract and Procurement 1. Arranges to purchase materials and/or services by applying statutes, regulations, decisions, directives, and procedures for procurement. 5. 71 2. Negotiates with business and industry representatives to agree on terms of contract and/or price. 6.31 3. Analyzes current market and price trends. 3.52 4. Analyzes cost and price elements in contracts to determine equity. May require extensive contact with contractor. 5.53 5. Signs or prepares final agreement committing government to settle or procure material and/or services. 3.35 6. Contracts and procures new merchandise for which there are no or few precedents. 4.16 7. Acts as team leader in administration of contract. 4.12 8. Administers the terms and conditions of the contracts to see that the government's interest is protected. 4.98 GS-1169 Internal Revenue Officer 1. Deals with delinquent individual and business taxpayers in an attempt to obtain payment. 6.65 2. Investigates business books to determine whether taxes are due. 4.96 3. Decides whether business is insolvent. 5.30 4. Imposes penalties and invokes administrative and judicial remedies to collect taxes; e.g., may seize property. 6.41 5. Takes testimony under oath. 3.24 Mean T (N::;:58) 5~05 5.26 2.88 4.66 2.23 3.76 3.53 4.28 (N=71) 6.42 4.03 4.42 5.66 2.41 SD I 1.45 1.01 1.60 1.06 2.79 1.95 2.30 2.06 \ •61 i.25 1.39 .89 1.56 SD T 1.61 1.47 1. 79 1.45 2.31 2.07 2.30 2.18 .89 1.44 1.54 1.39 1.35 50 APPENDIX B (Continued) Dut Mean I Mean T SD I SD T GS-1169 Internal Revenue Officer (N=71) (Continued) 6. Applies knowledge of bookkeeping and accounting and specific business practices and conditions. 4.91 4.42 1.43 1.58 7. Conducts full compliance of tax laws. 6.24 5.65 .96 1.49 8. Instructs and assists in training lower grade personnel in Series 1169. 4.79 3.63 1.35 1.87 9. Acts as group manager as required. 4.24 3.04 1.89 1.82 10. Deals with sophisticated attorneys and accountants in resolving the most complex and sensitive cases of large dollar amounts. 6.48 6.08 .73 1.15 11. Has full responsibility to make independent decisions to determine what course of action would best serve the government's interests. 6. 71 6.46 .61 .93 GS-1710 Education and Vocational Training (N=39) 1. Teaches elementary and secondary school classes by applying professional educational knowledge. 2.97 2.59 2.82 2.91 2. Performs other professional duties in the education field; e.g., as guidance counselor, club advisor, or school principal. 2. 72 2.18 2.18 1.93 3. Maintains discipline in classroom. 3.41 2.46 2.69 2.36 4. Confers with students and in many cases parents on students' progress. 2.85 1.67 2.37 1.94 5. Keeps abreast of literature affecting areas of specialization. 5.08 3.87 1. 78 2.05 6. Keeps track of pupils' needs. progress and unique 3.87 3.08 2.46 2.34 7. Develops curricula and/or lesson plans based on educational practice and subject matter knowledge. 4.38 3.67 2.40 2.43 8. Reviews, evaluates, and critiques course materials for instructional adequacy. 6.03 5.54 1.33 1.67 9. Establishes standards for materials to be used in extension co~rse programs. 4.62 3.64 1. 76 1.86 51 APPENDIX B (Continued) Duty Mean I Mean T SD I SD T GS-1710 Education and Vocational Training (N=39) (Continued) 10. Develops and issues guidelines relative to these standards and reviews for conformance to standards. 11. Rewrites and revises course materials as necessary, or returns materials to preparer for rewording. 12. Assures that visual aids, including illustrations and schematics, are consistent with textual discussions, and review items; and examination items are adequately correlated with these for instructional purposes. 13. Answers student correspondence relating to instructional aids. 14. Coordinates and consults with course authors, their supervisors, and other subject matter specialists on problems relating to programming, preparation, and development of course rnaterials. 15. Reviews and evaluates statistical data relating to instructional aids and recommends revisions. 16. Constructs original test items as necessary. GS-1712 Training Instruction 1. Teaches classes by applying pratical knowl edge of educational principles. 2. Is a subject matter expert in a technical field. 3. Develops curriculum, training, and job aids based upon knowledge of technical field and teaching environment. 4. Prepares and grades examinations in subjects taught in order to assess studen.ts and the effectiveness of his teaching. 5. Keeps abreast of literature and new develop ments in subject matter field taught. 6. Keeps track of students' progress, counsels them, and modifies instruction to conform to student characteristics. 4.85 5. 79 5.62 4.29 5. 26 4.97 4.95 5.57 5. 72 4.17 4.82 5.38 4.97 4.26 5.26 4.92 3.08 3.87 4.03 3.69 (N=40) 5.70 3.13 3.45 3.60 4.30 4.25 1.61 1.22 1.11 1.68 1.37 1.91 1.92 1.13 1.20 1.11 .93 .98 1.33 2.05 1.60 1. 74 2.07 1.79 2.08 2.20 1.09 2.61 1.38 1.39 1.59 1.66 52 APPE~~IX B (Continued) Mean Dut I GS-1810 Investigator (General) 1. Plans and conducts investigations pertaining to personal character, suitability of persons or organizations for Federal benefits or employment. 4.69 2. Prepares verbal and written reports of findings. 6.03 3. Interviews individuals in order to uncover facts, solicits and obtains cooperation. 6.19 4. Connects clues as a result of disparate evidence and research. 5.89 5. Learns rules and regulations pertaining to type of investigations performed. 5.67 6. Works with other investigators in team situations. 3.31 7. Works closely with the legal staff in preparation of cases and serves as expert witness as required. 2.94 8. Prepares detailed schedules and/or charts based on documents obtained during the investigation. 3.17 GS-1811 Investigator (Criminal) 1. Plans and conducts investigations relating to suspected violations of criminal law. 6.45 2. Learns and applies rules of evidence and court procedures, precedent court decisions, constitutional rights, criminal laws, etc. 5.60 3. Uncovers, recognizes, develops, and presents evidence that reconstructs events and establishes relationships and legal liabilities. 5.93 4. Maintains surveillance and performs undercover work. 4.87 5. Interviews and interrogates individuals. 5.58 Mean T (N=36) 4.33 5.53 5.86 4.92 4.14 2.64 2.47 2.86 (N=60) 5.92 3.82 5.22 3.83 4.58 SD I 2.94 1.13 1.04 1.24 1.41 1.80 2. 77 2.96 •85 1.20 1.04 1.35 1.19 SD T 2.97 1.38 1.31 1.63 1.97 2.10 2.67 2. 77 1.31 1.45 1.15 1.60 1.38 53 APPENDIX B (Continued) Duty Mean I Mean T SD I SD T GS-1811 Investigator (Criminal) (N=60) (Continued) 6. Works with other investigators in team situations. 4.90 4.54 1.28 1.41 7. Prepares verbal and written reports, times testifying in court. some 6.07 5.02 .95 1.16 8. Develops, controls, and directs information. sources of 5.17 4.22 .98 1.33 9. Arrests violators, and/or seizes contraband and illegal merchandise. 5.57 4.03 1.57 1.94 10. Conducts searches of persons, places, things with and without warrant. or 4.92 2.92 1.59 1.53 11. Supervises other Criminal Investigators. 3.92 3.20 2.23 2.17 12. Coordinates with other agencies and prosecuting attorneys. 4.97 3.29 1.14 1.13 13. Makes recommendations to prosecute. to prosecute or not 3.08 1.54 3.48 1.66 14. Provides armed protection in potentially hazardous situations. 5.15 2.62 1.46 .96 GS-1816 Immigration Inspection (N=36) 1. Inspects and examines individuals and their credentials with respect to their right to enter the country. 6.47 5. 72 .91 1.55 2. Detains questionable individuals suspected of attempting illegal entry. 6.00 3.74 1.15 2.11 3. Learns and applies immigration and naturalization laws. 6.25 5.15 1.42 1. 75 4. Reports findings both orally and in written form to designated authority. 5.14 3.81 1.33 1. 79 5. Adjudicates applications and petitions for various benefits under immigration laws. 5.94 5.22 1.29 1. 74 6. Performs research and interpretation of laws to solve individually presented problems not falling into standard or common solutions. 5.06 3.73 1.30 1.53 54 APPENDIX B (Continued) Dut GS-1854 Alcohol, Tobacco, Mean I Mean T and Firearms Inspection SD I (N=33) SD T 1. Inspects alcohol, tobacco, and firearms producers and importers to assure compliance with applicable government regulations. 4.64 3.82 2.57 2.47 2. Audits business records. 5.42 5.03 1.06 1.15 3. Learns and applies relevant rules and regulations for industry regulation. 5.97 5.06 .92 1.39 4. Learns and applies production, distribution, and other relevant practices for industry and establishments regulated. 4.78 4.03 1.68 1.90 5. Inspects explosive manufacturers, dealers, users, and importers to insure compliance with applicable Federal laws and regulations. 4.30 2.50 2.54 2.19 6. Inspects alcohol dealers and users and firearms dealers to insure compliance with applicable Federal laws and regulations. 4.67 4.15 2.54 2.54 7. Conducts original investigations to determine eligibility to engage in alcohol, tobacco, firearms, or explosives industries. 5.36 3.24 2.85 2.25 8. Performs supervision of distilled spirits plant operations. 4.45 3.55 2.55 2.36 9. Conducts investigations of claims filed by industry members. 4.61 2.94 2.61 2.08 10. Trains less experienced inspectors and/or examiners (on the job training). 5.13 3.53 .74 1.13 GS-1890 Customs Inspection (N=68) 1. Inspects cargo and luggage being imported and exported to collect applicable duty. 6.29 5.97 .92 1.13 2. learns and applies knowledge of regulations and procedures concerning the examination, classification, and assessment of merchandise. 6.18 5.50 1.02 1.44 3. Makes judgment as to whether to hold merchandise or charge individuals who violate customs or other laws with fraud or just collect additional duty. 5.74 4.07 1.23 1.55 55 .DI.X B (Continued) Mean Mean SD SD Dut I T I T GS-1890 Customs Inspection (N=68) (Continued) 4. Collects duties and taxes on merchandise imported for personal use or for commercial use entered on an informal entry. 5.57 4.26 1.10 1.38 5. Verifies that cargo is truly and correctly invoiced and that the description is adequate for classification and value determination. 6.40 6.04 .81 .90 6. Is responsible for maintaining maximum security of all areas where imported and exported merchandise is held in customs custody. 5.53 4.32 1.18 1.56 7. Examines preliminary entrance and clearance of incoming vehicles, vessels, aircraft, and/ or railroads requiring the presentation of a proper customs manifest. 5.38 3.87 1.36 1.58 8. Examines aircraft, vehicles, cargo, luggage, and persons entering the U.S. in order to detect and seize narcotics and other prohibited merchandise. 6.29 5.66 .81 1.22 9. Learns and applies knowledge of laws and regulations of other Government agencies (Federal, State, and local) concerning restrictions and prohibitions of imported merchandise and articles. 5.39 4.63 1.12 1. 53 10. Testifies in court as required and aids Customs Agency and/or U.S. Attorney in preparation of fraud, pilferage, and narcotics cases. 5.19 3.81 1.42 1.80 GS-2001 General Supply (N=64) 1. Plans and coordinates material purchase and distribution actions to assure properly phased support. 3.55 3.30 2.57 2.49 2. Reviews, validates, and revises basic data and decision rules of automated supply systems. 4.23 3.60 2.05 2.08 3. Determines supply requirements. 3.25 3.06 2.60 2.65 4. Plans the distribution of material among supply stations and/or contractors. 3.20 2.95 2.31 2.34 5. Determines amount of funds material procured. required for 2.59 2.09 2.37 2.07 56 APPENDIX B (Continued) Mean Mean SD SD Duty I T I T GS-2001 General Supply (N=64) (Continued) 6. Prepares recommendations for the procurement of material. 2.83 2.48 2.45 2.23 7. Develops material control program. 2.83 2.59 2.32 2.33 8. Utilizes knowledge of EDP procedures and applications. 4. 72 4.34 1.82 1.99 9. Utilizes knowledge of the technical features and uses of items of material. 3.94 3.63 2.19 2.07 10. Utilizes knowledge of the relationship of procurement, fiscal, transportation, and storage activities. 4.38 3.90 1.82 2.05 11. Maintains effective personal contacts in coordinating the supply distribution system. 4.63 4.48 1.83 2.03 12. Performs technical or managerial work concerned with physically receiving, handling, storing items within a distribution system. 2.30 2.09 2.13 2.08 13. Plans and designs standardized packages to protect material between the time of purchase and use. 1.05 .86 1. 79 1.45 14. Utilizes knowledge of transportation management and materials handling equipment. 2.56 2.30 2.13 2.06 15. Provides standardized description and gories for material supply catalogs. cate 2.13 1. 78 2.19 1. 99 16. Utilizes knowledge of systems for cataloging and classifying material. 2.69 2.48 2.22 2 • ..:.~ 17. Maintains accountable records. 3.69 3.66 2.79 2.83 18. Coordinates with foreign and/or domestic customers on supply and technical problems. 4.80 4.54 2.27 2.42 19. Analyzes supply performance and other related reports. 4.69 4.50 2.31 2.21 57 APPENDIX C Mean Importance Ratings of Knowledges, Skills, Abilities, and Other Characteristics for Each Job Series and for All Job Series Combined KSAO'S Job Series 105 110 180 187 201 334 343 526 560 570 685 Adaptability 5.4 5.0 4.4 5.5 5.4 5.1 5.4 5.5 5.2 3.9 5.9 Alert 4.5 4.6 3.6 4.9 4.9 4.8 4.8 5.2 5.1 4.2 5.6 Arithmetic Computation 4.0 2.5 2.7 4.9 1.9 3.7 3.9 5.6 4.7 3.9 3.6 Attention to Detail 5.4 4.2 4.2 4.6 4.8 6.3 5.2 5.2 5.2 4.4 4.9 Fund of Information 5.6 4.9 4.8 4.3 5.4 4.8 4.9 5.8 5.0 5.1 5.4 General Reasoning 5.6 5.9 5.9 4.9 6.1 5.9 5.9 6.3 5.7 6.1 5.7 Grammar 4.9 5.1 4.0 4.6 5.3 4.1 5.1 4.8 4.7 5.0 4.6 Judgment 5.4 5.8 5.2 5.1 6.2 5.3 5.6 6.4 5.7 5.5 5.9 Letter Writing 4.4 3.9 4.1 4.1 4.3 3.8 4.7 4.5 4.1 4.0 4.6 Listening 6.0 4.6 5.2 5.9 5.6 5.2 5.7 6.2 5.3 5.4 5.8 Memory 5.3 4.0 3.9 4.5 4.7 4.6 4.6 5.2 4.9 4.6 4.9 Objectivity 5.1 5.3 6.0 5.7 5.8 4.3 5.6 5.7 4.5 5.1 5.4 Oral Communication 6.0 5.0 5.4 5.3 5.6 4.9 5.4 6.0 5.2 5.4 5.8 Originality 4.0 5.4 4.6 2.9 5.1 5.2 4.6 4.0 4.3 4.1 5.2 Object Perception 2.2 1.7 1.9 0.6 1.9 3.0 2.5 3.5 3.4 2.9 3.4 Planning 4.0 5.2 5.3 4.2 4.8 5.5 5.2 4.3 4.9 4.1 5.8 Quantitative Reasoning 2.8 4.1 4.2 1.1 2.6 4.2 3.4 3.8 3.9 2.8 3.7 Reading Comprehension 5.6 5.2 5.2 4.1 5.6 5.3 5.3 5.9 5.0 4.7 5.1 Spelling 4.3 3.7 3.3 4.5 4.1 3.7 4.2 4.5 3.9 4.2 4.5 Written Communication 4.9 5.7 4.8 4.3 5.7 4.8 5.7 5.0 5.0 5.8 5.1 Curiosity 3.3 5.2 4.8 3.4 4.8 4.6 4.8 4.3 4.3 4.7 4.5 De?l with People 6.0 5.3 4.9 6.6 5.9 5.1 5.9 6.6 5.7 5.9 6.7 Emotional Maturity 5.3 5.0 5.0 6.1 5.3 4.6 5.2 6.0 5.0 5.0 5.9 Empathy 4.9 4.5 5.0 5.6 5.3 4.3 5.1 5.5 4.7 4.6 5.8 Forcefulness 4.1 4.6 3.9 3.9 5.1 4.3 5.0 5.8 4.8 4.9 5.3 Integrity 5.5 5.4 6.0 5.3 6.0 5.1 5.7 6.7 5.5 6.0 6.2 Perseverance 5.3 5.0 5.1 5.1 5.2 5.8 5.5 5.7 5.4 5.3 5.7 Persuasiveness 4.0 4.5 3.3 3.7 5.2 3.9 4.8 5.7 4.4 4.8 50 3o Physical Stamina 2.3 2.2 2.9 2.9 2.4 2.6 2.8 3.4 3.6 3.0 3.5 Self Control 5.1 4.1 4.4 6.0 4.8 4.9 4.8 6.0 5.2 4.9 5.3 Tact 5.8 4.7 4.6 6.2 5.6 4.7 5.5 6.5 5.4 5.9 6.1 58 APPENDIX C (Continued) KSAO'S Job Series 962 967 993 996 1081 1082 1102 1169 1710 1712 Adaptability 5.6 5.4 5.3 5.0 4.7 5.1 5.0 5.7 5.3 5.1 Alert 5.2 5.1 4.1 4.4 5.0 5.4 4.9 5.9 4.7 4.9 Arith. Computation 4.6 2.6 4.8 4.2 2.6 2.2 4.4 5.3 4.5 4.0 Attention to Detail 5.2 5.4 5.3 5.4 4.7 4.9 4.9 5.8 5.6 4.5 Fund of Information 5.9 5.6 6.0 6.0 5.0 6.1 4.7 6.1 5.0 4.5 General Reasoning 5.4 6.0 6.0 6.0 5.4 5.8 5.7 6.3 5.6 5.3 Grammar 5.0 5.8 4.7 5.2 6.0 6.4 4.8 5.0 6.0 4.9 Judgment 5.3 6.0 5.8 6.2 5.4 5.2 5.7 6.6 5.1 4.7 Letter Writing 5.2 5.4 4.7 5.3 3.9 2.8 4.8 4.8 4.5 3.6 Listening 6.2 5.3 4.7 4.9 5.7 5.9 5.3 6.0 4.9 5.3 Memory 5.8 6.2 5.3 5.4 4.6 4.9 4.5 5.5 4.4 4.7 Objectivity 5.7 5.4 5.1 5.1 5.3 5.7 4.9 5.8 4.6 4.8 Oral Communication 6.3 5.3 4.4 4.7 5.1 4.7 5.4 6.1 5.5 6.3 Originality 4.0 4.5 3.7 3.7 4.• 5 4.8 4.1 5.2 4.9 4.1 Object Perception 3.3 4.7 2.7 2.7 2.9 3.2 2.3 4.0 3.6 3.0 Planning 4.4 4.7 4.1 3.9 4.8 4.4 4.8 5.7 5.0 4.3 Quant. Reasoning 3.3 4.3 3.9 3.2 2~5 2.0 2.9 4.4 3.7 3.7 Reading Comprehension 5.6 6.1 6.2 5.9 5.5 6.3 5.1 5.8 6.0 5.4 Spelling 4.9 5.1 4•.6 4.6 5.2 5.9 4.1 4.7 5.4 4.5 Written Communication 4.9 6.1 5.3 5.1 6.1 6.5 5.3 5.5 6.1 5.0 Curiosity 3.9 4.1 3.4 3.8 4.8 5.1 3.8 5.0 4.1 4.3 Deal with People 6.6 5.1 4.4 5.1 5.8 5.5 6.0 6.5 5.4 5.9 Emotional Maturity 6.3 4.6 4.6 5.0 5.0 4.9 5.4 6.0 5.0 5.5 Empathy 6.1 4.9 4.0 4.9 5.0 5.1 4.7 5.1 4.6 5.1 Forcefulness 4.8 4.7 3.3 4.6 4.3 4.4 5.1 6.1 4.2 4.8 Integrity 6.0 5.9 5.5 5.9 6.0 5.9 6.4 6.7 6.2 5.6 Perseverance 5.5 5.9 5.3 5.2 5.5 5.7 5.6 6.2 5.6 5.0 Persuasiveness 4.7 4.4 3.2 4.4 4.4 4.4 4.9 5.7 4.1 4.6 Physical Stamina 3.1 3.1 2.2 2.9 3~9 3.2 2.8 3.7 2.7 3.1 Self Control 6.3 5.4 4.9 5.0 5.1 5.0 5.3 6.0 5.3 5.3 Tact 6.5 5.4 4.6 5.3 5.8 5.4 5.7 6.2 5.4 5.1 59 APPENDIX C (Continued) / KSAO'S Job Series Mean SD All All 1810 18ll 1816 1851 1890 2001 Series Series Adaptability 5.7 5.2 5.0 5.4 5.1 5.4 5.26 1.33 Alert 5.3 6.0 5.3 5.0 5.7 5.2 5.02 2.48 Arith. Computation 3.0 3.1 1.5 4.9 5.0 4.6 3.96 3.49 Attention to Detail 4.7 5.0 4.8 4.7 4.6 5.3 5.ll 1.93 Fund of Information 4.8 5.1 5.5 5.1 5.2 4.7 5.27 1.55 General Reasoning 5.6 5.8 5.6 5.4 5.1 5.4 5.75 1.24 Grammar 5.2 4.6 4.8 5.2 3.9 4.6 4.90 1.59 Judgment 5.9 6.4 6.2 5.8 5.5 5.3 5. 72 1. 39 Letter Writing 3.3 3.9 4.5 4.0 3.4 5.0 4.31 2.01 Listening 6.3 5.9 5.9 5.5 5.5 5.3 5.52 1.24 Memory 5.3 5.3 5.3 4.6 5.4 4.6 4.98 1.58 Objectivity 5.7 5.8 5.9 4.4 5.3 4.7 5.22 2.14 Oral Communication 5.8 5.6 5.4 5.4 5.3 5.2 5.41 1.41 Originality 4.3 4.7 3.6 3.5 4.9 4.7 4.40 2.23 Object Perception 3.1 4.4 5.0 2.9 5.4 3.5 3.16 4.20 Planning 4.7 5.6 3.8 3.8 4.0 4.9 4.70 2.36 Quant. Reasoning Reading Comprehension 1.9 4.8 3.3 4.6 2.8 5.5 2.9 . 5.5 3.7 5.3 4.2 5.1 3.43 5.43 3.41 1.35 Spelling 4.6 4.1 4.4 4.6 3.8 4.4 4.37 2.00 Written Communication 4.9 5.6 4.6 5.4 3.9 4.9 5.22 1.82 Curiosity 5.3 5.1 4.4 4.5 4.6 4.7 4.42 2.78 Deal with People 6.2 5.9 6.2 6.0 6.0 5.6 5.78 1.44 Emotional Maturity 5.7 6.0 5.7 5.4 5.4 5.1 5.32 1.58 Empatl:l.y 4.3 4.5 5.0 4.2 3.6 4.8 4.82 1.90 Forcefulness 4.0 4.9 4.6 4.6 4.7 4.6 4.68 2.00 Integrity 6.6 6.8 6.8 6.3 6.5 5.6 5.98 1.59 Perseverance 6.1 6.1 5.4 5.5 5.4 5.5 5.51 1.17 Persuasiveness 4.1 4.9 4.0 4.4 4.1 4.3 4.49 2.24 Physical Stamina 3.5 5.3 4.0 2.1 4.6 3.2 3.14 3.29 Self Control 5.4 6.0 5.6 5.3 5.7 5.0 5.25 1.55 Tact 6.2 5.7 6.3 5.7 6.0 5.4 5.59 1.38 60