U:5 C:5 L1l/4: 77 -I BISOft Job Information Test Development for Claims Authorizers in the Social Security Administration m United States Civil Service Commission Bureau of Policies and Standards SUNY AT BUFFALO 1t9890 THE LIBRARIES DEPOSITORY \~ ··. ": TM-77-1 JOB INFORMATION TEST DEVELOPMENT FOR CLAIMS AUTHORIZERS IN THE SOCIAL SECURITY ADMINISTRATION Daniel B. Carts Research Section Personnel Research and Development Center U.S. Civil Service Commission Washington, D.C. 20415 February 1977 JOB INFORMATION TEST DEVELOPMENT FOR CLAIMS AUTHORIZERS IN THE SOCIAL SECURITY ADMINISTRATION ABSTRACT Sequential steps in the development of a content valid job information test for GS-10 Social Security Claims Authorizers in Series 993 are described. These steps outline the procedure for test construction, beginning with the item pool, through editing and use. Items in the completed test were designed to cover to the greatest extent possible the major duties of the job and to measure the amount of job knowledge of Claims Authorizers. Principles used in construction of the test and documents used for recording judgments of subject matter experts are provided. Data are provided showing the relationship of items to major job duties. The adequacy of the job information test as a validation criterion for measuring the relative job performance of Claims Authorizers is discussed. PREFACE The assembly of the completed form of this job information test was facilitated by the assistance of other psychologists in the Personnel Research and Development Center, U.S. Civil Service Commission. Sincere appreciation is extended to Paul Van Rijn for obtaining items from the Social Security Administration in Baltimore, Maryland, and to Marvin H. Trattner and Alice M. Nelson for conducting the trial of the job information test in Kansas City, Missouri. Daniel B. Corts CONTENTS Page Introduction 1 Procedure for Construction of the Job Information Test 3 Statistical Characteristics of the Job Information Test 8 Conclusions and Recommendations 13 References 14 Appendix A: Principles for the Development of New Test Items Used to Instruct the Subject Matter Experts 16 Appendix B: Duties of Job Series 993 -Social Security Claims Authorizer (Preliminary Version) Major Duties of Job Series 993 -Social Security Claims Authorizer (Final Version) 23 Appendix C: Directions for Editing and Evaluating Job Information Test Items 25 Appendix D: Subject Matter Expert Judgments Concerning Relevance and Usefulness of Items 27 Appendix E: Cover Sheet and Directions for the Job Information Test Claims Authorizer -Series 993 Social Security Administration 28 Appendix F: Directions for Conducting the SSA Claims Authorizer -993 Job Information Test (JIT) 3'1 Appendix G: Job Information Test Questionnaire 35 Tables Table 1: Relationship Between Items in the Job Information Test for Social Security Administration Claims Authorizer and Major Duties of the Job 7 Table 2: Item Analysis of the Job Information Test 10 INTRODUCTION The construction and development of a job information test for Social Security Administration (SSA) journeyman Claims Authorizer positions was undertaken as part of a larger research effort associated with the introduction of the Professional and Administrative Career Examination. The examination has been developed by the U.S. Civil Service Commission to measure the abilities of applicants for a wide variety of Federal jobs. Scores on the PACE subtests are linked to major duties of the jobs for which the test is used. Therefore, it is also important that the relative .proficiency of Claims Authorizer participants in the research be accurately measured in terms of some or all of the major duties of their jobs. Such job performance criteria should focus on the accurate measurement of job knowledge in order to establish the relative proficiency of research participants. Adequate criteria based upon major duties enables researchers to determine statistical relationships between predictor subtest scores and the performance of journeymen working in the positions. A number of criteria were constructed for measuring job performance of Claims Authorizers duties. A work sample was developed which consisted of an actual, detailed claim for Social Security benefits to be adjudicated by research participants. Supervisors will rate and rank the Claims Authorizers in terms of various aspects of their performance on job duties. Official performance ratings which strongly relate to job and duty performance will be used as criteria. A job information test, the topic of this report, was developed which was intended to measure the amount of job knowledge of journeymen Claims Authorizers in primarily three of the most important duties, although items selected represented two additional duties to a lesser extent. The Standards for Educational and PsychoLogicaL Tests (1974, p. 34) provides the following rationale for the use of a job information test as validation criteria: "For many employment and educational purposes, the ideal criterion may be an achievement test or work sample judged acceptable in terms of content validity.'' This report provides evidence of the content validity of the job information test to support its use as a criterion of job performance. 1 Development of such an instrument owes its success to the extent to which the research psychologist, acting as measurement or psychometric expert, can enlistthe cooperation of a team of subject matter experts who provide the knowledge necessary to develop valid tests in their occupational field. The requisite number of subject matter experts is a part of the responsibility of the research psychologist, as is the r~quirement for judging that point at which the test is sufficiently sound technically for meaningful administration to Claims Authorizers. Of course, the psychologist must assess the qualifications of subject matter experts participating in the selection and construction of items (Career Evaluation Branch, U.S. Air Force, 1973, p. viii). Procedures used for test development and evaluation, and the selection and coo~dination of subject matter experts who wrote, edited and/or revised items for use in this test are presented in the follo\ling sections. 3 Procedure for Construction of the Job Information test Multiple choice test items developed for use in training courses were collected from the Bureau of Retirement and Survivors Insurance (BRSI) and the Bureau of Disability Insurance (BDI) training divisions of the Social Security Administration. Items were considered for use if they related to the journeyman Claims Authorizer's job. Only multiple choice items were selected, since the responses to items in the job information test would need to be machine scored. Training staff from BRSI provided guidance relating to job structure and content. They provided training materials which would constitute the substance of any new items to be written and would further clarify the structure of the Claims Authorizer's job. Staff membersl in BRSI training wrote, edited and revised items at various points during the development of three versions of the job information test. The cooperation of subject matter experts was enlisted in the Northeastern Program Center, Flushing, New York in June, 1974. The research on PACE was explained by measurement psychologists, and the ·cooperation of an Operations Analyst, a supervisory Claims Authorizer, Technical Assistants, Claims Authorizer Instructors, and Claims Authorizers was enlisted in the construction and refinement of all criterion measures as well as a job task inventory.2 Information about the construction of new test items contained in Appendix A (extracted from Specialty Knowledge Test Branch U. S. Air Force, 1963, pp. 14-19) was read and discussed by the measurement psychologist and subject matter experts who would select items for inclusion in the first version of the job informa tion test (see Footnote 2). This provided the technical knowledge necessary for subject matter experts to deal effectively with test items. Standards for item construction were discussed and agreed upon (Adkins, 1947; Plumlee, 1974). 1 special appreciation is extended to Jack Gillard and Susan Parfrey of BRSI Training Branch, Social Security Administra tion, Baltimore, Maryland for assistance in the collection, wr:Lting and development of test items. 2Excellent and professional cooper~tion was provided by Conrad Sandler, Senior Operations Analyst, and Isaac Faighes, Super visory Claims Authorizer in the selection, editing and major modifications of test items assembled for trial in the North eastern Program Center, Flushing, N.Y. Two subject matter experts3 then studied and discussed the major duties of the job. These duties are shown in two versions in Appendix B. The Preliminary Version list of job duties shown first was derived from a complete listing of over 500 job tasks. The Preliminary Version was edited and later evolved to those shown in the second list in Appendix B labeled Final Version. However, judgments about the relevance of items to job duties were made using the first list shown in Appendix B. When understanding of the meaning of the duties was achieved, the two subject matter experts were prepared to judge the adequacy of each potential item for use in the job information test. They had previously reviewed the complete listing of all tasks which made up the major duties of the Claims Authorizer's job. The Final Version used in the research included 528 tasks under the eight duty headings. The items considered for inclusion in the test were evaluated in terms of many important aspects of test items as they relate to the job duties. These judgmental aspects of test item selection and evaluation, developed by the author for use in this research, are shown in Appendix C. They deal mainly with the relevance, importance, duty-relationship, datedness, difficulty, ambiguity, and alternatives of eachitem. The two subject matter experts and the measurement psychologist cooperated in this preliminary evaluation of potential test items. Judgments about the item adequacy were recorded on a form devised by the author, and shown in Appendix D. Items were placed on individual item cards and judgments made by the three test developers were entered on the reverse of each item card. They'reviewed, edited, revised, and judged the adequacy of each item in terms of standards shown in Appendixes A and C. Special attention was given to the relationship of the items to job duties, the importance of the topic ot the item, item wording, item stem, and the alternatives or distractors. Of all items in the pool judged to be acceptable, 46 possessed characteristics which met the standards outlined in Appendixes A and C, and were judged to be particularly relevant to the level of a journeyman Claims Authorizer's job. See Footnote 2, preceding page. These were assembled into an experimental version of the test, coded NYC A, and administered to three employees4 in the Northeastern Program Center, who were given sufficient time to complete all items in the test. In a group session, after testing, each item was again reviewed with respect to its content, wording, and judgmental data associated with it. The measurement psychologist, the three employees, the operations analyst, and the supervisory claims authorizer participated in this review. The items were further edited, opinions about the usefulness of the items were gained, and time estimates for working each item were collected. · After completion of the development of the first version of the job information test in New York, additional items were written in BRSI training division and were added to a modified version of the test coded KC B. It was administered at the Mid-America Program Center, Kansas City, Missouri, to ten Claims Author·izers and four Technical Assistants. Upon completion of this non-timed version, all participated in a discussion and review of the test items. Items were again edited or revised. The four Technical Assistants (senior Claims Authorizers who occupy a position between journeyman and first line supervisor), completed the detailed review on all items using the principles and procedures shown in Appendixes A through D. A simple item analysis was tallied on the upper and lower 50 percent groups using eight of the completed tests. Item responses of the Technical Assistants were not used in this analysis. Of the ten remaining research participants, one did not finish the test and one did not provide usable data. The results of this tally were entered on the reverse of the item cards and would provide a preliminary and very rudimentary idea of the discriminatory power of the test items. A final version of the test, coded C in draft and X PACE Research Instrument No. 9 for operational use, was developed using items with the most favorable characteristics, based upon the data gained in the two previously described trials. 4special thanks to Barbara Conroy, Instructor, Fred Kotteck, Technical Assistant, and Richard Vail, Claims Authorizer, all of the Northeastern Program Center, Flushing, N.Y., for their valuable participation in this testing and review. The 42 items selected provided reasonable coverage of duties 1-3, shown in the first list in Appendix B, and covered duties 4 and 5 to a limited extent. An inspection of duties 6-9 in this list will make apparent the reasons for not writing items to cover these duties. Items are randomized throughout the test in terms of duties to which they are related. The coverage of duties by test item number can be seen in Table 1. It was estimated that it would take the average Claims Authorizer between 60 and 90 minutes to complete all of the items in this operational version of the job information test. The test was considered basically a power test but reasonable speed was considered necessary to complete all of the items. Appropriate instructions were written for test administration, which are shown with the test cover in Appendix E. Directions for conducting the test are shown in Appendix F. Only rights are scored and there is no penalty for guessing. The test is open book because the use of the Claims Manual is a critical part of the Claims Authorizer's job. Most item cards maintained by the U.S. Civil Service Commission show the paragraphs of the Claims Manual which support the correct answer. All items have four alternatives, and item responses will be collected on a standard 125 item multiple-choice answer sheet, coded esc Form 790, dated February 1965. TABLE 1 Relationship Between Items in the Job Information Test for Social Security Administration Claims Authorizers apd Major Duties of the Job Duties 1. Identification of the Issues 2. Evaluating the Evidence and Making a Determination 3. Computation and Calculation 4. Maintenance of Individual Claims Folders 5~ Communication Note --Some items were judged as related to more relationship is shown for each item. No items were found or written which were Items 3,7,11,16,19,28,29,34, 37,39,41 1,5,8,13,15,18,22,23,25, 31,33,35,36,38,40,42 2,4,9,12,14,20,21,24,26, 27,30,32 6,17 10 than one duty. Only one appropriate for the following duties from the first list in Appendix B: 6. Training 7. Quality Review 8. Maintenance of the References 9. Use of Forms Statistical Characteristics of the Job Information Test After the 42-item final criterion instrument was printed and ready for administration, a last trial administration was carried out using nine Claims Authorizers in the Southeastern Program Center in Birmingham, Alabama; They were given sufficient time to complete all 42 items in order that preliminary item information would be available for analysis. An average of 105 minutes was required for completing the 42 items. Special problems were identified concerning two of the items in the final printed version of the test. Because of complications having to do with Social Security benefit entitlements when railroad work experience is involved, approximately 230 research participants in the main validity study will be instructed not to attempt item number 37. In addition, although trial participants and subject matter experts believed that item number 24 was a difficult, challenging and excellent item, it was found that some trial participants worked between 15 and 30 minutes on the item. It requires considerable calculation, and while some research participants in the main validity are completing this item, others may be gaining score credit on a number of other items. As a result, it was decided to include special instructions to the validity study research participants not only to omit item number 37, but to avoid attempting item number 24 until they.have considered all other items in the test. In saving item 24 until last, and eliminating item 37 from the test, it is expected that the average amount of time required to complete the remaining 40 items will be approximately 80 minutes. Therefore, the average research participants in the validity study should complete about three-fourths, or 30, of the items. A number of research participants are expected to complete 40 items and a few will attempt or complete item 24. The mean score of the nine Claims Authorizers tested in Birmingham was· 29.11. This may drop approximately seven points when the test is timed. The median score was 26, the variance was 43.36, and the standard deviation was 6.58. The scores ranged from a low of 20 to a high of 37, and a bimodal 9 frequency distribution of research participants' scores is shown below: Score Frequency Score Frequency 37 2 28 36 27 35 1 26 1 34 1 25 233 24 32 23 1 31 22 30 21 29 20 1 Table 2 shows an analysis of the 42 test items when the Kansas City and Birmingham samples were divided into upper half and lower half groups based upon total test score. Because there were nine research participants tested in Birmingham, the upper group contained four research partici pants and the lower group included five. The tallies of correct and incorrect responses for the small, but randomly selected, groups show most of the items discriminating effec tively between high and low scorers. Some items were answer ed correctly by nearly everyone. Easy items had been includ ed, in an effort to maintain the motivation of the Claims Authorizers, and contribute to the preservation of their self-image with respect to personal job knowledge. The p values were calculated for both the Kansas City and Birmingham samples, and tallies.of right and wrong responses areshown for both. Since the sum of the p-values for a sampleequals the mean test score for that sample, the means de~ived through this method areshown for both groups. TheX=29.15 approximates the value of 29.11 calculated in thestandard manner and reported earlier for the Birminghamsample. Since the Kansas City sample was administered a 59item version, which was later modified, all items shown inTable 2 were not common to both groups. Where a p-value wasmissing for the Kansas City group, the p-value calculated for the Birmingham sample was used. It was considered to beour best estimate of the true p-value based upon this relatively limited and preliminary information. The mean scoreof 31.96 shown for the Kansas City sample is therefore considered only an estimate, and was not found to be significantlyhigher than the Birmingham mean (t=1.56; df=40; p>.05). Therewas a considerable difference in performance between the twogroups on a few test items. In particular, performance onitems 22, 33, 37, and 40 contributed to a higher mean for theKansas City sample. ' TABLE 2 Item Analyoio of the Job Information Teat Item Right Upp~r Wrong Group Right Lower Wrong 1 1111 1111 1 2 111 1 11 111 3 0000 1111 000 111 0 11 4 0000 1111 000 111 0 11 5 1111 11111 6 0000 111 1 000 111 . 0 11 00 11 00 11 11 0000 111 8 0000 1111 000 1111 0 1 9 0000 111 1 000 111 0 11 10 11 1111 0000 111 1 ll11 00 1 1 00 1111 12 000 111 0 1 00 111 00 11 13 0000 111 1 00 00 11111 1~ 0000 1111 00 1111 00 1 15 0000 111 1 000 1111 0 1 16 oooO 1111 0000 1111 1 17 0000 1111 000 11 0 111 18 0000 1111 00 111 00 11 19 000 111 0 1 0 11 000 111 20 0000. 111 1 00 11111 00 21 000 111 0 1 0 11 000 111 Note--0 • 1 • Kanaaa City Birmingham • 56 23 . Item .89 22 • 78 .88 24 • 78 .88 25 1.00 26 .67 .88 27 .44 .25 28 .89 .88 29 .67 .88 30 .89 31 .44 • 75 32 .67 .63 33 .33 • 75 34 .89 • 75 35 • 78 .88 36 .89 1.00 37 .67 .88 38 .78 .75 39 .56 .so 40 .89 • 75 41 .56 .so 42 Right 0000 in 000 111· 000 11 0000 111 0000 1111 0000 1111 0000 111 0000 1111 0000 1111 0000 1111 0000 1111 0000 111 000 1111 000 1111 111 000 1 1111 1111 0.000 1111 11 0000 1111 Upper Wrong 1 0 1 0 11 1 1 1 0 0 1 0 111 11 Croup Right 00 1 00 111 00 0 1 0 11 00 1111 000 11111 000 1111 00 11111 . 000 11 00 11111 000 1 00 11 000 111 1 000 1 11111 111 0000 1 111 000 1111 Lower w~ong 00 1111 00 11 . 00 11111 000 1111 000 111 00 1 0 0 1 00 0 111 00 0 1111 00 111 0 11 1111 0 1111 11 1111 11 ·o 1 .44 .67 .22 .44 .67 .89 .89 .89 1.00 .67 l.OO .44 .67 .78 .44 .22 1.00 • 78 .56 .56 .89 -29.15 • 75 .63 .63 .63 .63 • 75 .88 .88 • 75 • 75 .88 .88 .63 • 75 .75 1.00 .88 31.96 PKC. • p-value for Kansas "Ci'ty Sample Pa • p-va1ue for Birminsham Sample A Pearson product moment correlation coefficient was calculated between the two sets of p-values as an estimate of the stability of the item functioning. Considering the small size of the two groups, the item p-values were considered surprisingly stable (r = .37; df = 40; p< .05). Table 2 shows most item p-values to be far more stable than this coefficient indicates. An estimate of the internal consistency reliability was based on statistics calculated for the Birmingham sample, since only those nine research participants were administered the 42 item test intended for use. During subsequent research, coefficients of this type will amount to estimates of the reliability of this instrument as a criterion measure. The magnitude of this reliability determines or limits the magnitude of the relationship between the predictor instrument and the criterion. As applied to this research, this coefficient affects the magnitude of any criterion-related validity coefficient that could be obtained for the PACE. The formula used for calculating this estimate of reliability is a variation of the Kuder-Richardson formula 20: 'i.pq ( 1-a~ ) Where: n = number of items in the test p = proportion passing an item q = 1'-p o2 variance of the total test t As applied to the 42-item job information test, the reliabil ity was calculated as shown below: Ptt -(42) {1-7.05 ·) = .86 -FJ ' 43.36 This coefficient was significantly different from zero (df=40; p<.Ol). The internal consistency reliability of the job information test was considered to be highly satisfactory. A criterion reliability coefficient of .86 for the job information test would make possible a validity coefficient of .93 between it and the composite of PACE subtests if they were "assumed to be perfectly reliable" (Thorndike, 1949; p. 108). The formula for this estimate of potential validity is shown in Thorndike: Where: rAA = reliability of the criterion If an estimate of the reliability of the predictor instrument is available, the following upper limit of validity can be established: Where: rAA = reliability of the criterion = reliability of the test composite r 88 CONCLUSIONS AND RECOMMENDATIONS It was concluded by the personnel research psychologist, the subject matter experts, the Technical Assistants and super visors of Claims Authorizers that the items included in the job information test adequately and validly represent the content ·of the journeyman Claims Authorizer's job. It was believed that the most important duties had been covered, and that the items would present a formidable challenge for Claims Authorizers who will be demonstrating their skills in their jobs. It was judged that the test would adequately distribute the scores of the sample of Claims Authorizers according to their relative abili-. ties and knowledge of the job. Responses were elicited in Kansas City, Missouri to a short questionnaire, shown in Appendix G. The following responses were collected from some subject matter experts in answer to the question, "What's your reaction to the test, is it a good one?" "It is a good test except as noted in booklet." (Recommended item improvements) "Seems very thorough" "Yes" "Yes" "Good test" "Time consuming. Person needs to be familiar with the Claims Manual." The Operations Analyst and supervisory Claims Authorizer who selected the original set of items in the Northeastern Program Center concluded that the set of items would task experienced Claims Authorizers and commented that the items constituted "••• a good representation of what we do." Discussion of the test content with Claims Authorizers in Birmingham revealed that the test adequately covered the major duties of the job. Based upon this description of procedures and the accompanying conclusions, it is recommended that the job information test be used with confidence as a criterion measure representing the typical and relative job performance of Social Security Administration Claims Authorizers. REFERENCE NOTE Plumlee, L. B. Principles and ·;eahniques of job knowledge test development (Working Draft). Washington, D.C.: Personnel Research and Development Center, U.S. Civil Service Commission, August, 1974. REFERENCES Adkins, D. C., Primoff, E. S., McAdoo, H. L., Bridges, C. F.,& Forer, B. Construction and analysis of achievement tests. Washington, D.C.: U.S. Government PrintingOffice, 1947. American Psychological Association. StandaPds fop educational and psychological tests. Washington, D.C.: Author, 1974. Handbook foP construction of the Specialty Knowledge Test and Associated tests. Career Evaluation Branch, 3700 Occupational Measurement Squadron: Lackland Air Force Base, Texas, 1973. Handbook foP subject-matteP specialists. Specialty KnowledgeTest Branch, Personnel Research Laboratory: Lackland Air Force Base, Texas, 1963. Thorndike, .R. L. PePsonnel selection, New York: Wiley, 1949. APPENDIX A Principles for the Development of New Test Items Used to Instruct the Subject Matter Experts1 There are several rules for the construction of good multi ple-choice test items. Most of these are reasonable and simple and probably represent what you would do in any case if you had the time to consider the problem at length. Look at the sugges tions and the examples carefully and you will be able to save considerable time in writing new items and will have a better test. The whole purpose of these rules is to make sure that your tests will be practical and useful. Rules for Test Items 1. Write the item about an important on-the-job situation. If your test is in the communications or electronics area, an is the value of radio?" is not appropriate. item which asks "What This type of information is applicable to a general test, but not to airmen working in communications. Mak~ your item realistic and practical, have it concerned with something you actually do on the job, in the way you do it on the job. As a general rule, don't ask an airman to remember a technical order number, since his ability to memorize is not what we are testing. 2. Avoid controversial material. If an item deals with something which involves an opinion, someone is bound to disagree. If you ask "Which branch of the military service is the best?" you are dealing with controversial material. In the same way, if you ask "Which meter is the best for measuring current?" you are forcing your opinion on the examinees that one of the meters is better tha~ the others. . 3. Write the item as clearly as possible in the working language of the job. Some examinees have larger vocabularies than others, but this is not what we are testing. We want to know how much an airman knows about his job, not if he undercan stand difficult ~ords. The item must mean the same thing to everyone who reads it. Simple, straightforward language is the best. Think about the airmen who will take the test when you select the words you use. Sometimes manuals and other directives use difficult language. The stem of the item shown below was ' Extracted from Chapter IV, Handbook for Subject Matt~r Specialists U.S. Air Force. Reprinted by permission of the United States Air Force Occupational Measurement Center, Letter of 19 February 1975 from Col. Lyle D. Kaapke, Commander. taken directly from a manual. Which one of the following awards is authorized and issued in lieu of a decoration for each succeeding award of the same decoration to the same person? A) Chevron *B) Oak-leaf cluster C) Brassard D) Bronze bar Read the stem carefully and try to see what it means. The average airman will have to read it several times before he understands it at all. If the item is written as a problem and the language is changed to everyday words, it might look like this: An airman won the Bronze Star Medal in 1945 and again in 1952. What device should he wear on his Bronze Star Medal Ribbon? A) Two bronze stars *B) Oak-leaf cluster C) Silver star D) Bronze arrowhead You will notice that the distractcrs have also been changed and are more sensible. We will discuss these in more detail later. 4. Have one central problem in each item that is clearly and accurately stated. The central problem is the statement of the who, what, where, when, why, or how in the stem. The problem must be completely stated. If you can cover up the distractors with your hand and still tell what problem is posed by the stem, the item probably has a central problem. The B-52 aircraft is A) the fastest used in the Air Force *B) equipped with tandem landing gear C) designed for use on low-level missions D) painted black Try covering the distractors with your hand and see if you , can guess the answer the original item writer used. As you can see, it is almost impossible to do so. The central problem of the item is the type of landing gear used on the B-52. It could be written 18 this way: With what type of landing gear is the B-52 aircraft equipped? A) Tricycle landing gear B) Tractor landing gear C) Tandem landing gear D) Tracked landing gear When the cent1al problem is clearly stated, the distractors have to be related to each other, another important principle of item construction. Difficulties with the central problem of an item are not usually as obvious as the example given, but the same test for the presence of a central problem will usually work. 5. Each item must have one and only one correct answer. Partly right answers or partly right distractors confuse the airman who really knows the item more than the airman who does not. You must make sure that the answer is correct and that all the distractors are wrong. Every item must have a reference to some document to support the accuracy of the item. The reference must be an official Air Force document, such as an AFR, AFM, TO., comparable document of some other service, or civilian publication which is available to airmen studying for the test. 6. Negative items should be avoided. Sometimes it seems easier to write an item that has three right answers and one wrong one and asks a question about what should NOT be done. This merely succeeds in confusing the airman taking the test as he has to twist the way he has been thinking completely around. Worst of all is the item which uses a negative in the stem and a negative in each of the distracters. No one should be expected to answer such a question. This is an example of a negative item, the type which should be avoided: Which item of equipment is not issued to aircrew members? A) Flashlight B) Oxygen mask C) Hunting knife *D) First-aid kit Notice that three of the alternatives A, B, and C are things which are issued to aircrew members while the correct answer is an item which is not issued. The right answer is a wrong answer, a confusing situation. There is also a type of item which uses a NOT in the stem, but does not set up a negative situation. The item has three wrong answers, and one right answer. This is a perfectly usable type of item. This is an example of a usable item with a negative word in the stem: When an aircraft is NOT in flight, who is permitted to operate the flight controls? A) Any crew member assigned to the aircraft *B) Any rated officer or aircraft mechanic C) Any commissioned officer D) The line chief only 7. Put just the right amount of information in the stem to state the central problem, no extra, but just enough. l!;xcess material is an item slows down reading time, confuses the examinee and may ruin an otherwise good item. If there is too little information, the item will not have a central problem. Here is an example: · The B-58 is a very fast bomber which was first flown by the Air Force in 1958. What kind of engines does it have? As you can see, the whole first sentence is wasted. The stem should read: What kind of engines are used on the B-58? 8. G.ood test items should not be too hard or too easy. I~ an item is too· easy, almost all the examinees will get it right, and it will not separate those who know their jobs from those who do not. A very easy item is usually concerned with general know ledge, rather than something specific to the job. If an item is too hard, almost all the examinees will get it wrong, and we have wasted an item. Here is an example of an easy item: What are milk bottles made of? A) Wood B) Steel ">'(C) Glass D) Stone This item is better fitted for elementary school children in a test of general knowledge than in an SKT. It can go the other way too: What are milk bottles made of? A) 647. silica, 337. potash, and 3% lead B) 62% silica, 357. potash, and 37. lead C) 607. silica, 377. potash, and 37. lead D) 48% silica, 49% potash, and 37. lead As you can see, this is equally silly the other way.ceramic engineer would know the answer to this item. Only a 9. Distractors must be plausible. A distractar must look like a possible correct answer to the examinee who is not sure of the answer. If the distractor is not reasonable, the guesser is given help in getting the item right. All three distractors must look possible. Here is an ex~mple: Which condition could cause overheating of an automobile engine? A) Fouled spark plug B) Loose door handle *C) Faulty water pump D) Advanced ignition timing The B choice is given away. The examinees who does not know now has only to guess among three choices instead of four. 10. Distractors must fit the stem grammatically. In an openstem item, each of the four choices must combine with the stem to make a complete sentence in good English. If only one choice fits, almost all examinees will identify the right answer, even though many do not know the information. On a military map a main highway is shown by a i~A) single heavy line B) two red dotted lines C) parallel broken lines D) parallel solid lines As you can see, only A) fits the stem and the item is given to examinees who do not know the correct answer. It is fairly easy to correct this problem. On a militiary map a main highway is shown by a A) single heavy line B) pair of red dotted lines C) set of parallel broken lines D) set of parallel solid lines ll. Distractors must be parallel in form and idea. Look at the example under rule 4. The distractors have no relation to what the central problem is supposed to be. To a certain extent, the same is true of the example under rule 3. You can see in each case that changing the distractors to make them parallel in idea has improved the item. In the same way, making distractors parallel in form avoids throwing distractors away. 21 Who administers punishment under Article 15, UCMJ? *A) The squadron commander or higher commander, whoever is appropriate B) The chief of security and law enforcement C) The administrative officer D) The judge advocate The correct answer stands out because it is much longer and much less specific. In the same way, a short, precise, correct answer with long and complex distracters will stand out and give the answer avay. All choices should be about the same length and in the same general form. 12. Avoid giving a clue to the correct answer. It is possible to give a clue to the correct answer within an item. The primary function of the Air Training Command is to A) monitor the supply program *B) provide formal technical training C) test new equipment and material D) administer the reserve program The word "training" in the stem and in the correct answer gives the item away. This can be corrected. The primary function of the Air Training Command is to provide A) academic training B) cross training *C) technical training D) on-the-job training Now all the choices are concerned with different types of training and the answer is not clued. It is also possible to have infomation in one item give away the answer to another. For example: Which action is permitted after the command of execution AT EASE has been given? A) Movement of .the right foot *B) Movement of the arms C) Smoking D) Talking Which command is a command of execution? *A) At ease B) Forward C) Column right D) Count cadence The information in the stem of the first item gives the answer to the second item. If both items are in the same test, the second item is wasted. The Ideal Test Item The ideal test item for a SKT is concerned with a problem which actually occurs on the job. The correct answer is the way the problem should be solved and the distractors are mistakes which airmen have made in trying to solve the problem. The problem be in troubleshooting, administrative work er any other aspect of the job as long as it is realistic and practical and tests the application of an airman's knowledge rather than remembering some principle, rule or number without being able to apply it. Summary of Test Item Rules Look again at the list of rules for writing good test items. 1. Write the item about an important on-the-job-situation. 2. Avoid controversial material. 3. Write the item as clearly and concisely as possible in the working language of the job. 4. Have one central problem in each item that is clearly and accurately stated. 5. Each item must have one and only one correct answer. 6. Negative items should be avoided. 7. Put just the right amount of information in the stem to state the central problem, no extra, but just enough. 8. Good test items should not be too hard or too easy. 9. Distractors must be plausible. 10. Distractors must fit the stem grammatically. 11. Distractors must be parallel in form and idea. 12•. Avoid giving a clue to the correct answer. APPENDIX B DUTIES OF JOB SERIES 993 -SOCIAL SECURITY CLAIMS AUTHORIZER (PRELIMINARY VERSION) 1. Identification of the Issues 2. Evaluating the Evidence and Making a Determination 3. Computation and Calculation 4. Maintenance of the Individual Claim Folders 5. Communication 6. Training 7. Quality Review 8. Maintenance of the References 9. Use of Forms APPENDIX B MAJOR DUTIES OF JOB SERIES 993 -SOCIAL SECURITY CLAIMS AUTHORIZER (FINAL VERSION) A. Identifying the Issues B. Evaluating the Evidence and Making a Determination C. Computing and Calculating D. Maintaining the Individual Claim Folders E. Communicating F. Training G. Maintaining the References H. Using Forms APPENDIX C DIRECTIONS FOR EDITING AND EVALUATING JOB INFORMATION TEST ITEMS Read Chapter IV of the USAF Manual for Subject Matter Experts entitled New Test .Items. Read the duties and functional areas of job series 993 -Social Security Claims Authorizer. Review each item to see whether they conform to the rules for construction of test items. Use the codes shown below for identifying certain important characteristics of the items which will need to be known when the job information test is constructed. 1. Item is relevant to the job of Claims Authorizer. N No R = Relevant HR = Highly Relevant 2. Knowledge of the item contents is critical to the work. If the item was relevant, a competent Claims Authorizer: S Should know the· correct answer M = Must know the correct answer 3. This item relates primarily to which duty shown in the list of duties. Enter the number 4. Item is up-to-date. N = No Y = Yes 5. Liability of the item going out of date: N = Not liable to go out of date Y = Yes, liable to go out of date 6. Rate the difficulty of the item. E = easy A = average in difficulty V = very difficult 7. Will the item mean the same thing to every Claims Authorizer: that is, not be ambiguous? N No Y = Yes 8. Is there more than one right answer; either shown in the alternatives or not. N = No Y = Yes Test No. Item No. Relevance to the Job Critical to the Work Relstl!s to Duty Number Up to Date Liable to Go Out of Date Difficulty Mean the same to everyone More than one right answer Decision on Item Useful """" Remarks "' & ..... i ('> " .... I iif ! .... i .... I '1 " ! i $ I '1 " I .... I ......... I " I t-'Cl. ! "" i I ~ .... I .." I I ..... .... b' .... ('>" " '1 .... " ~ ~ § ~ "" N " APPENDIX E Job Information Test Claims Authorizer -Series 993 Social Security Administration PROPERTY OF THE UNITED STATES CIVIL SERVICE CO.t--IMISS ION Limited Official Use This hooklet is lent for official use subject to restrictions regarding the use of Goverrunent documents. It should not be reproduced in whole or in part without the express approval of the Civil Service Commission, nor should it be used by anyone for any private purpose whatsoever. Violations may be punished by fine, imprisorunent, or both. United States Civil Service Commission X PACE Research Bureau of Policies and Standards Instrument No. 9 29 PACE RESEARCH Subobjective B 102.1 August 1974 JOB INFORMATION TEST Claims Authorizer -Series 993 Social Security Administration U. S. Civil Service Commission Directions to the Examinee -Read Carefully This examination is designed to test your knowledge of some important aspects of a Claims Authorizer's job. The score you obtain will not be available to any persons other than U.S. Civil Service Commission research personnel. Scores will not be used for any official personnel or promotion action. The examination is open book -that is, you may refer to the Claims Manual and other necessary technical material any time you wish. The test is designed to measure how much you know about your job. Working with the Claims Manual is an essential part of that job. There is a one hour time limit for the examination. Work rapidly and accurately. Complete as many test items as you can in the time allowed. The questions within each part of this test need not be taken up in order. Answer first those that you can answer without delay. Then use the remainder of the time on the questions you have passed over. This test will be scored solely on the number right and there will be no penalty for guessing. The examiner will tell you when to begin and when to stop. Write your name and identification number on the top of the answer sheet that was supplied with the test. You will need scratch paper for making necessary calculations. When you are told to turn the page and begin, read the items and select the answer you think is correct. Select the one correct answer for each item from the four alternatives presented. Mark your answer on the separate answer sheet in the numbered space which corresponds with the test item number. Mark between' the lines with a number two pencil. When you are finished, or are told to stop, turn your test booklet and answer sheet in to the examiner. Also, turn in any scratch paper used to make calculations. Please defer any phone calls that are not of an emergency nature until a later time. All other noise and activity will be kept to a minimum by the examiner. Breaks will not be scheduled during this one hour period. 31 APPENDIX FDFC 871 October 1974 United States Civil Service Commission Directions for Conducting theSSA Claims Authorizer -993Job Information Test (JIT) Time Required: Instructions and Questions 10 min.JIT 60 min. Sensitive -must be administered simultaneously with other JITsessions or back to backAdminister at work site -access to Claims Manual required Materials Required for the Examiner:- 1. Copy of these instructions 2. Watch or clock 3. Supply of #2 pencils for those who do not have them 4. Scratch paper for those who do not have it 5. Copy of the Job Information Test and answer sheet Materials Required for Each Participant:- 1. One copy of the Job Information Test -X PACE ResearchInstrument No. 9 2. One machine scored answer sheet -CSC Form 790, Feburary 1965 3. Pencil -#2 4. Scratch paper 5. Claims Manual and other relevant reference materials Specific Directions for the Examiner The directions you are to say or read are enclosed in boxes. Youmust be standing while reading the directions to the participants. When everyone is assembled and you are ready to begin, SAY: Please clear your desk of all unnecessary materials except for your Job Information Test, your blue answer sheet, your pencil and some scratch paper. Have your Claims Manual and other relevant reference material ready for use. If you have a calculator, put it aside or in a drawer. All calculations will be made by hand, using the scratch paper. Do not begin until I tell you to do so. I am now going to read you the directions for completing your Job Information Test. Please read them to yourself while I read them aloud to you. READ the following instructions: This examination is designed to test your knowledge of some important aspects of a Claims Authorizer's job. The score you obtain will not be available to any persons other than U.S. Civil Service Commission research personnel. Scores will not be used for any official personnel or promotion action. The examination is open book -that is, you may refer to the Claims Manual and other necessary technical material any time you wish. The test is designed to measure how much you know about your job. Working with the Claims Manual is an essential part of that job. There is a one hour time limit for the examination. Work rapidly and accurately. Complete as many test items as you can in the time allowed. The questions within each part of this test need not be taken up in order. Answer first those that you can answer without delay. Then use the remainder of the time on the questions you have passed over. This test will be scored solely on guessing. the number right and there will be no penalty for The examiner will tell you when to begin and when to stop. (continued) Write your name and identification number on the top of the answer sheet, that was supplied with the test. You Jill need scratch paper for making necessary calculations. When you are told to turn the page and begin, read the items and select the answer you think is correct. Select the one correct answer for each item from the four alternatives pre-· sented. Mark your answer on the separate answer sheet in the numbered space which corresponds with the test item number. Mark between the lines with a number two pencil. When you are finished, or are told to stop, turn your test booklet and answer sheet in to the examiner. Also, turn in any scratch paper used to make calculations. Please defer any phone calls that are not of an emergency nature until a later time. All other noise and activity will be kept to a minimum by the examiner. Breaks will not be scheduled during this one hour period. Then SAY: Are there any questions about these instructions? Answer any questions and clarify misunderstandings. Then SAY: If there are no more questions, there is one final special instruction. Make a note to omit Item No. 37. We have found the correct answer to be questionable. Also make a note to attempt Item No. 24 only after you have attempted all other items. It is time consuming and should be saved for last. You may draw a line through Item 37 on your answer sheet and write "last" beside Item 24 if you wish. A!fter allowing some time for the participants to write notes, SAY: :~·,d.J 1o q .. :::; BL·i.:-1· Ji:CJ :i..S·dnrun ac,J Jl).:.i.t1l:iri:~~h~~: .brr.t: ..::·tf't·:.n ~~xJoy ~; __-;J. ;'f~J h~~:,_')n J..Ii:'.~/ Ho~{ ( .. 1:::~.'2~~-s;C:!· rfj_J:-\r? b.C)_~~-!:q,_::};{-~ a:.-;~:-~ ·sf..~-d:~ ,. :1~:;;,s.c;;.~ ·x~;·\~rf~E.n:~. Remember not to converse with others while you are working on th'esel: itemstHo:'3 Ifdthere fi.s.Ga'='fieedcO•tie diseuss aiiy:tmatters'(. iiSifl'l other-"'-tthafl.ri.testDi-tems; :lbring:;themAtodtheoatten:tiOn:'O!df t:he.Lsos hrcr,; persofisCJ"motfitoraiigJin yout-wdrk1flarear'"":lNow:;..turn.oto·:r.p-a:ge:s4:?.:1u·y;-:;o:> and~l:)eg;in WOi"Kin~:S:tfheDi-tefil'SF.-qS8 s;J"j SW "lcJv!<:Ii.G J!.J(.: 1,~ ;;J:·n~~·~ , b9:JnSiegiil.",, andhaftersdrteohout hasdlapsedr,rs ' S~Y: , ,:!;.rJ 1)_tj.r~J~~f~JlJS..J ;;.;::rJJ:Yn o.J: bs-:::.~.I..f J:~:;q.rdf u---~.-... ~--·-· ~·"·~-~--------_................. ~."':; ·• l..Ilv.r y__:i·Jvl:J:~.s hr.rs =~·~?-.lor~ ~arljo 1IA .smll ~sJsl B Stop worl.o:tJand: ~~f~:::,rl5 :1!Jodg ~JJf)l:~:~c:Sti..rp s-:r:.s.dj ~-J·I/;.. J "\(f1f5 Administer-the---next·-research-·inst rument-·or--schedule--a·break-·depending1 upon the research circumstances. I .Cs.t.r.J:::f. ;~J..1{} ~~t -=-:·.:~.~~·.r.l'J ~ ,e.r.xo.I::; '3.!:-'f.r~? ;;;,·:~:t..)ffi -::;rx ~)~:r.g ::·::r::iif:~ }_I .} I .. ~.:!f11e J i ·:-?:·s£1 :J'c; J:.l:s b.~J,:J qnx::~ ;-::Sf\ .9-·If:t ri· v r)·r,:: :fB :J =-~-.g t ·~·r.r::rb 't(E5ii1 Lru .iz.EJ ·xo1 .bs~;-£;;.;, sd bJ.t..:o.dE .hn;-~ .<:~1f1J: ::;· ;aj: ! SJJ.~ J J-3 i APPENDIX G JOB INFORMATION TEST QUESTIONNAIRE l. Take test (Time). 2. Keep time (Time). 3. What's your reaction to test--is it a good one? Questions for SME l. Any questions you don't understand? Pl..ny questions have two correct answers? Any questions have no correct answers? Any questions have answers given by another question? 2. Organize by subject matter area? 3. Any important areas omitted? 4. Any questions trivial, i.e., have no real bearing on ability to perform job? 5. Make comments on form• .. U. S. GOVERNMENT PRINTING OFFICE' 1977-727-401/1057