±WS^> ■■■:■■■ ::/■:'■ THE UNIVERSITY OF ILLINOIS LIBRARY 370 ie6e no.36-47 fxr^r°-s b ^ ,he University of Illinois Library UNIVERSITY OF ILLINOIS BULLETIN Issued Weekly Vol. XXIII May 11, 1926 No. 36 [Entered as second-class matter December 11, 1912, at the post office at Urbana, Illinois, under the Act of August 24, 1912. Acceptance for mailing at the special rate of postage provided for in section 1103, Act of October 3, 1917, authorized July 31, 1918.] EDUCATIONAL RESEARCH CIRCULAR NO. 44 BUREAU OF EDUCATIONAL RESEARCH COLLEGE OF EDUCATION OBJECTIVE MEASUREMENT OF INFORMATION By Charles W. Odell Assistant Director, Bureau .of Educational Research m umm o» m APR 2 u 1926 mivmw)/ m ill; PUBLISHED BY THE UNIVERSITY OF ILLINOIS URBANA 3^0 OBJECTIVE MEASUREMENT OF INFORMATION Recent discussion of objective tests. The so-called "new examina- tion" 1 has been one of the most frequently discussed topics in educational literature during the last four or five years. Many writers and speakers have advocated that objective tests 2 be employed either in conjunction with examinations of the traditional type 3 or in place of such examina- tions. Although many of those who have done so have based their opinions largely upon theoretical arguments, others have presented experimental data to support their judgments. Among the latter are Wood 4 and Ruch, 5 both of whom strongly recommend the use of ob- jective exercises. Another experimenter, 6 however, has concluded that under satisfactory conditions there is little difference in the general merit of the two kinds of measuring instruments. The bulk of opinion, however, seems to be in favor of some use of the newer type. Purpose of this circular. It is the purpose of the writer to discuss very briefly the place and merits of the new examination, to follow this by illustrating and explaining different forms of objective and near- objective tests, and to give directions for administering and scoring them. Function and limitations of objective tests. Although some en- thusiastic advocates of objective tests have urged that they replace dis- cussion and essay examinations entirely, the writer does not believe that this viewpoint is justified. They have certain definite advantages 7 over traditional examinations and, in his opinion, should be used fre- lr rhe expression "new examination" is commonly used as a collective designation for tests or exercises which may be objectively scored and which require little writing on the part of those being tested. 2 A test or measuring instrument is said to be objective when different persons using it obtain the same result. In other words, objective scoring eliminates differences due to the fact that the standards of different teachers and also those of the same teacher at different times vary. In this circular the expressions "objective tests" and "objective exercises" will frequently be used instead of "new examination." 3 The expression "traditional" will be applied to examinations of the essay or discussion type which have been in common use for many years. 4 Wood, Ben D. Measurement in Higher Education. Yonkers: World Book Com- pany, 1923. 337 p. b Ruch, G. M. The Improvement of the Written Examination. Chicago: Scott, Foresman and Company, 1924. 193 p. "Brinkley, Sterling G. Values of New Types of Examinations in the High School. Teachers College Contributions to Education, No. 161. New York: Teachers College, Columbia University, 1924. 121 p. 7 The chief of these advantages will be presented later. [3] quently when one desires to test information, especially separate rather than connected or organized facts. Objective exercises undoubtedly secure some measure of other achievements than mere knowledge, such as the ability to connect cause and effect, to organize, to summarize, and so forth, but the measures so secured are not at all adequate. Indeed, in many cases, tests which on the surface appear to yield measures of such abilities will be found upon careful analysis to measure mere memory or knowledge of facts. For example, if the question "Do you think Grant was a better general than Lee?" is asked, it is likely that many pupils will answer by giving a statement remembered from textbook or teacher. In case a pupil does not remember a satisfactory statement, or perhaps even if he does, he may really form his own opinion as to the respective ability of Grant and Lee from the pertinent facts he knows. Therefore the writer maintains that tests of the types described in this circular should be used, for the most part, when one desires to measure memorized facts or information, and that their use for this purpose should be merely part of a complete testing program which also includes examinations of the traditional kind. The propor- tionate amount of testing to be done by each of the two methods in a particular situation depends chiefly on the kind of subject-matter cov- ered and the objectives of the course. Chief advantages of the new examination. The new examination has at least three important advantages over the traditional type. Probably the chief of these is that pupil responses can be scored upon an objective or almost objective basis. This can be accomplished by following proper directions in administering and scoring the exercises employed. The fact that objective or near-objective scoring is possible renders it probable that reliability 8 will be high. A second chief advantage is that the amount of writing done by pupils is reduced to a minimum. This practically eliminates the advan- tage of rapid and fluent writers over those not so gifted; the fact that they can write more material in the same length of time should not result in their receiving higher marks in most subjects. Also there is much less opportunity for the teacher to be influenced, frequently un- consciously, by good or bad writing, spelling, punctuation, and other similar qualities, which have no connection with achievement except in the field of English. 8 The reliability of a test is the degree to which a second application yields scores equivalent to those obtained by its first use. In other words, it is a measure of the variable errors present in the scores. [4] The third advantage, which is to some extent due to the fact that little writing is required, is that a much wider range of achievement can be measured or a much larger sample can be secured than is possi- ble by a discussion or essay examination requiring the same amount of time. This minimizes the likelihood that pupils just happen to know or not to know a large enough proportion of the answers to the ques- tions asked that their scores are largely the result of chance. Other minor advantages might be named but it seems to the writer that these three major ones are enough to justify the use of objective exercises in preference to those of the traditional type when one desires to measure detailed factual knowledge. Disputed points concerning the new examination. In addition to the advantages which this type of examination is rather generally con- ceded to possess there are certain other claims for it concerning which there are marked differences of opinion and, in some cases, of experi- mental evidence. One of these is the question of reliability. On the whole the evidence presented on this point indicates that the new types of exercises are somewhat more reliable than those of the traditional type. 9 There has also been some argument concerning validity. 10 One writer 11 claims that the validity of new examinations is somewhat higher than that of the traditional ones but his data do not prove this. An- other 12 gives results which indicate that there is practically no difference in the validity of the two kinds except that as a measure of mere infor- mation the new type ranks slightly higher. On the whole one is probably fairly safe in assuming that there is no great difference in validity when compared with such criterion measures 13 as are now available. An advantage very frequently claimed for the new examination is that it requires the expenditure of much less time on the part of the teacher. Wood 14 states that its use is an economy of time in large classes but not in small ones. He, however, fails to state how many pupils a class must have to be considered large. Brinkley 15 states that it takes 9 Wood, op. cit., p. 188-93. Ruch, op. cit., p. 114. "The validity of a measuring instrument is the degree to which it measures what it purports to measure. "Wood, op. ch., p. 188-91 and elsewhere. "Brinkley, op. cit., p. 58-59. "A criterion measure is one taken as a basis for determining the validity of some other measure. "Wood, op. cit., p. 198-99. "Brinkley, op. cit., p. 59. [5] two and one-half times as long to make and score a new examination as a traditional one if the two are of equal length and involve equal numbers of pupils. The numbers of pupils used in his experiments were, however, fairly small. There appears to be general agreement that it requires more time for the teacher to prepare the new type than the old. It can readily be seen, therefore, that if only a few pupils are tested any economy in scoring will be balanced by the added time required for making. On the basis of his own experience in using such tests the writer believes that for classes as large as forty, and in some cases for those of only twenty-five or thirty, the total amount of time required of the teacher is no greater for the new than for the old type of examination. In a few such cases he has even found it to be less. Furthermore, it should not be forgotten that in most cases a compara- tively unskilled clerk can, if it is desired, score exercises of the new examination type, whereas this is impossible with practically any tradi- tional examination. General principles of constructing and using objective exercises. Before proceeding to the construction of the different types of objective tests it will probably be well to state a number of general principles which should guide one in their construction and use. 1. Such tests should be preceded by explicit directions as to just what is to be done. If the pupils are not familiar with the kind of exercise to be used, the directions should usually include examples. Also the amount of time to be allowed should be stated. 2. It is important that a fairly large number of items 16 be included. If tests are given frequently and as a more or less regular part of the procedure during class time twenty or twenty-five items may be enough. Under other conditions fifty should probably be the minimum and one hundred is a still better number. 3. Care should be exercised in the selection of items. In some cases practically all the important items can be included. In others, it will be necessary or desirable to make a satisfactory sampling from a large number. 4. Care should be exercised in the wording of the statements used so that ambiguity will be avoided. 5. The exercises should be so worded and, in case answers are suggested, these should be so selected that the correct answers are not 18 The term item is used to refer to the smallest unit or part of a test which calls for a distinct answer or pupil response. Thus a single direct recall or yes-no question, a single pair of expressions in a matching exercise, a single word to be denned or a single term to be connected with the proper part of a figure, is an item. [6] too evident. In other words, the incorrect answers should not be so absurd or far-fetched that they can be ruled out without a satisfactory knowledge of the subject-matter being tested. 6. The single items and also the whole test should be of such a degree of difficulty that no or practically no perfect scores or zero scores will be made. 7. The items included should be so arranged that there is no reg- ular sequence of answers. For example, the positive and negative statements or questions in an alternative test should be arranged in random order. The same is true of the suggested responses in multiple- answer exercises, of one of the two lists in matching exercises, and so forth. 8. The directions should require the pupils' responses to be re- corded so that scoring is as easy as possible. 9. The principle of variety should be observed, that is, in any one course or subject tests of a number of different types should be em- ployed. Furthermore, in a single long test it is usually better to use several types of exercises. 10. Pupil's scores should be tabulated in terms of points of credit and after this has been done for a whole class or group these scores should be turned into marks. 17 Basis of choosing examples. The examples given in the following pages have been chosen to show that the various types may be used widely in different school subjects. In fact, almost any of the kinds illustrated may be used in any of the commonly taught school subjects. An analysis of them, however, makes it apparent that some are much better suited to use in certain subjects, others in others. However, this fact should not be considered sufficient cause for limiting very narrowly the types used in any one subject, and thus violating principle 9. Use of multigraphed copies. It is advisable in the use of all of the types of exercises described in this circular and necessary in the use of most of them that mimeographed or otherwise multigraphed copies be placed in the hands of the pupils. If this is impossible or impracti- cable there are several of the types which teachers may employ by 17 For a discussion of turning scores into marks, see any of the following: Monroe, W. S. An Introduction to the Theory of Educational Measurements. Boston: Houghton Mifflin Company, 1923, p. 292-94. Monroe, W. S., Df.Voss, J. C, and Kelly, F. J. Educational Tests and Measure- ments, Revised and Enlarged Edition. Boston: Houghton Mifflin Company, 1924, p. 425-28. Paterson, Donald G. Preparation and Use of New-Type Examinations. Yonkers: World Book Company, 1925, p. 70-71. [7] reading the exercises and having the pupils record the answers on blank sheets of paper. The types with which this is practicable are recall or single answer, alternative, definition, enumeration, opposite and genus- species varieties of association, and abbreviation or formulae. The writer does not wish to be understood, however, as recommending that these kinds of exercises be given in this manner, unless it is impracti- cable to place copies in the hands of the pupils. Of course any of the types may be given by writing the material on the blackboard, but in view of the labor involved it is rarely desirable to do so. Examples of objective exercises. In the following pages thirty- seven varieties of objective or near-objective exercises have been grouped under fifteen headings. Following the directions for giving and the examples of each variety will be found brief discussions and scoring directions. The varieties named and illustrated do not constitute an absolutely complete list, but do include practically all with which the writer is familiar except very minor variations or types which are unsuited to use in ordinary circumstances. Many slightly different forms can be made by combining or mixing the features of two or more of the kinds given. Several of the types given illustrate such combina- tions or mixtures, but many more can be devised by teachers who are familiar with objective exercises. In selecting the examples which follow no effort was made to choose the best possible ones in the various subjects. They are not based upon any study of the subject-matter commonly taught or the questions commonly asked. In other words, the examples in any one subject, if gathered together, do not constitute a test on the minimum essentials of that subject. Practically all of them are original with the writer, in so far as their exact form is concerned. Many standardized tests, a number of text-books, and several teachers were consulted, how- ever, in the search for suitable content. [8] I. RECALL EXERCISES OR QUESTIONS WITH SINGLE CORRECT ANSWERS Directions: 18 Answer each of the following questions with a single word or expression. Write the answer immediately under the first word or words of the question. 1. Who was President of the United States during the war of 1812-14? 2. What is the product of x 2 , x 3 and x 4 ? 3. What is the dative plural of homo? 4. What kind of placket should be used in a tailored woolen dress? This type, which is one of the easiest to prepare and which may be employed in all subjects, consists of questions having a single word or expression as the only correct answer. Therefore it may be made perfectly objective, though in practice this ideal is not always attained. The score is merely the number of correct answers. II. ALTERNATIVE EXERCISES A. Yes-No Questions, Answers to be Underlined Directions: Approximately half of the following questions should be answered by "y es " and half by "no." If you know or think you know which answer is correct, underline the proper one of the two words in front of the question. If you do not think you know, do not guess. Yes No 1. Did the South favor the Mexican War? Yes No 2. Did Browning write "The Idylls of the King?" Yes No 3. If 3x + 8= 16 - x, does x = 2? Yes No 4. Is the ulna located in the lower arm? 18 The directions for pupils and the examples will be given just as they should appear on the multigraphed copies actually used. [9] B. Yes-No Questions, Answers to be Written Directions: Approximately half of the following questions should be answered by "yes" and half by "no." Write "yes" in front of each ques- tion to which you think it is the answer, and "no" in front of each to which you think it is the answer. If you have no idea which answer is correct, do not guess. 1. Should corn immediately follow clover on the same ground? 2. Does "vita" mean "error?" 3. Is the hemlock an evergreen? 4. Are invoices commonly made once a year? C. True-False Statements, Answers to be Underlined Directions: Approximately half of the following statements are true and half are false. Underline the proper word before each statement concerning whose truth or falsity you know or think you know. If you do not think you know, do not guess. True False 1. The elm is hardier than the maple. True False 2. Water is an element. True False 3. A brush drawing should be outlined in pencil. True False 4. A rip saw is used to cut with the grain. D. True-False Statements, Answers to be Written Directions: Approximately half of the following statements are true and half are false. Place a plus mark ( + ) before each statement which you think is true, a minus sign ( — ) before each which you think is false. Do not guess unless you are reasonably sure. 1. One board foot contains 1728 cubic inches. 2. Baking requires more heat than boiling. 3. Most banks use double-entry bookkeeping. 4. Mechanical drawings should usually be done in pencil before be- ing inked. [10] As the examples show, there are several varieties of alternative exercises which may be used. Questions to be answered by either "yes" or "no" or definite statements which are to be marked either "true" or "false" may be employed. In either case, pupil responses may consist of writing in the proper word or underlining the proper one of the two words already given. The actual question form is better than the state- ment form, since it does not present false statements which may possibly interfere with learning. In preparing lists of questions or statements of this type, one should exercise care that approximately half of them call for the one answer, half the other. This type of exercise is one of the most widely used and also one of the easiest to prepare and score, but it has less merit than a number of the other types. Its reliability is probably lower than that of most of the other kinds. 19 Despite instruc- tions to the contrary, it is liable to encourage mere guessing. There have been some differences of opinion as to how tests of this type should be scored. Although the method of subtracting wrongs from rights yields a score little if any more reliable than that obtained by using merely the number right, the validity of the former method, especially in connection with instructions not to guess unless one is fairly sure of the answer, appears to be higher than that of the latter method. 20 Furthermore, if those taking the test know that it is being used, mere guessing will probably be discouraged. It also makes the best theoreti- cal correction for guessing and for erroneous answers. Therefore the writer recommends that on all such tests the score be the number right minus the number wrong, no account at all being taken of those omitted. III. INCORRECT STATEMENTS A. Alternative Statements to be Corrected Directions: Approximately half of the following statements are true and half are false. The latter have been falsified by the insertion of an incorrect expression or the use of an incorrect word instead of a correct one. Cross out the expression in each false statement which makes it incorrect. If the expression crossed out takes the place of a correct one, write the correct expression immediately above the one crossed out. 19 Ruch, op. cit., p. 118. 20 Brinkley, op. cit., p. 61. Ruch, op. cit., p. 114-21, is somewhat favorable to taking merely the number correct as the score, but later, in a paper delivered before the Society of College Teach- ers of Education in February, 1926, advocates subtracting the number wrong. [11] 1. Lead is heavier than gold. 2. Abraham Lincoln was born in 1812. 3. If x — 3, x 2 — 2x + 5 = 2x 2 + x — 13. 4. The T-square is not needed in drawing a human head. B. Incorrect Statements Directions: Each of the following statements is rendered incorrect by the presence of a certain expression. In some cases this is inserted in an otherwise true sentence, in others it takes the place of a correct one. In either case, the incorrect expression should be crossed out and in the second case the correct one should be written in immediately above the one crossed out. 1. His mother said he could go to town. 2. Water freezes at 0° Fahrenheit. 3. A United States senator serves four years. 4. A peck contains 2150 cubic inches. The first of the two varieties given also involves the alternative element and might have been classified under II. Some persons object to the use of this type because of the danger of erroneous learning. The writer believes that this danger is present, but not very great. This type of test should be scored by counting the number correct. It is permissible to give half credit when an incorrect expression has been crossed out, but the correct one not written in. IV. COMPLETION EXERCISES A. Simple Completion Exercises Directions: The following statements are to be completed by writing one and only one word or a number in each blank. 1. The is an invertebrate. 2. " fleur means "the red flower.'' 3. Water boils at degrees Fahrenheit. 4. The Congress of the United States consists of members, of whom are senators and members of the lower house. Senators are elected for years and representa- tives for years. The presides over the senate, whereas a is elected to preside in the house of representatives. [12] B. Completion Exercises with Suggested Answers Directions: Complete the following statements by writing one of the words found in the list at the right of the page in each blank. Do not use the same word twice. Adams 1 won the battle of New Orleans. Clay 2 was first Secretary of the Treasury. Hamilton 3 was a leader for Nullification. Haynes 4. was President during 1797-1801. Jackson The completion type of test is one of the most commonly used of the newer objective forms. From the standpoint of pupil response it is probably one of the most valuable tests of this kind. It is very difficult, however, to construct a simple completion test so that the scoring is highly objective. Pupils are almost certain to give answers which it is difficult to evaluate as either absolutely right or wrong. This difficulty can be eliminated by requiring answers to be chosen from a given list. Although this type generally consists of a number of sen- tences with one or more blanks in each, connected paragraphs contain- ing a number of blanks may be used instead. Completion tests are scored by counting the number of statements correctly completed or the number of blanks correctly filled in. Some- times half credit is allowed for answers which are not entirely correct and yet cannot be classified as absolutely incorrect. V. MULTIPLE-ANSWER OR RECOGNITION EXERCISES A. Multiple-Answer Exercises with only one Correct Answer Directions: Each of the following statements can be correctly com- pleted by the use of one and only one of the expressions contained within the parenthesis. Underline the correct expression in each sentence. 9h h 1. A side of an equilateral triangle equals (J£\/fr h\/5?^ — 7=' _ a/3)- 2. The President must be (21, 25, 30, 35, 40) or more years of age. 3. Discounts are (never, rarely, sometimes, usually) in percents. 4. If — represents the earth's distance from the sun, that of Jupiter from the sun is best represented by ( , , , ). [13] B. Multiple-Answer Exercises with Answers of Varying Degrees of Merit Directions: Underline the one expression in the parenthesis of each sentence which makes the best statement. 1. (Copper, iron, paper, rubber) is a good conductor. 2. The (pine, oak, elm, pear, ash) is a desirable shade tree. 3. Twelve year old children should have (8, 9, 10, 11, 12) hours sleep. 4. (Very few, few, some, most) final consonants are sounded in French. C. Multiple-Answer Exercises with one or more Correct Answers Directions: Underline all of the expressions in each parenthesis which make true statements when taken with the rest of the sentence. 1. (Poe, Longfellow, Scott, Macaulay, Irving) were Americans. 2. Louisiana raises much (corn, oats, rice, cotton, sugar-cane). 3. A (chisel, nail-set, lathe, plane) is needed in making a drawer. 4. (Alto, campo, magro, polmo, lado) are adjectives. D. Compound Multiple-Answer Exercises Directions: Underline the one expression in each line which is closely associated with the expression given in the first line. 1. Raleigh. Virginia, North Carolina, South Carolina, Alabama, Georgia. (Population) 15,000, 25,000, 40,000, 50,000, 75,000. 2. Henry Clay. "Tennessee, Kentucky, Virginia, Ohio. Whig, Democrat, Federalist, Republican. Governor, Vice-President, Attorney-General, Congressman. E. Multiple-Description Exercises 1. Directions: Place a check mark (V ) in front of the paragraph which best describes the city of Pittsburgh. [14] a. Pittsburgh is a city of about 1,000,000 population, located in the western part of Pennsylvania on the Ohio river. Situated in the center of a prosperous agricultural region, it is a leading center in the manu- facture of flour and dairy products. b. The city of Pittsburgh, with a population of more than half-a-million, is located at the junction of the Allegheny and Monongehela rivers. Be- cause of its proximity to coal fields and its transportation facilities, it has become one of the centers of the iron and steel industry. c. Pittsburgh, the second largest city of Pennsylvania, is one of the leading live-stock markets of this country. It is also noted for its iron and steel manufactures. It is often called the "Smoky City." 2. Directions: Place a check mark (V) in front of the statement which best describes the human heart. a. The heart is a sort of pump, located in the upper left part of the chest. It is about the size of a quart measure, but conical, the larger end being up. It consists of two auricles, on the right, and two ven- tricles, on the left, with valves between. b. The heart is the center of the circulatory system and is located within the lungs. It resembles a large pear in size and shape. The two ventri- cles receive the blood from the veins and pass it on to the auricles which force it into the arteries. c. The human heart is essentially a group of muscles which operates as a pump to cause the blood to circulate. It consists of four chambers, or tanks, the right and left auricles above, and the right and left ventricles below, with valves to prevent the blood from flowing in the wrong di- rection. It is about the size of one's fist. d. The heart is a double pump which receives the blood from the body, purifies it, and sends it back again. The work is accomplished by an elaborate system of reservoirs, valves and filters. The lungs supply the energy, or fuel, necessary to the heart's work. Everything considered, multiple-answer exercises are among the best of those mentioned in this circular. If one is careful in selecting the suggested answers, scoring can be made absolutely objective. Fur- thermore, in many cases one can make the degree of discrimination which must be exercised by pupils as fine as is desired. It is probably [15] most common to suggest either four or five answers though it is not unusual to see two, three, six or seven and occasionally even more than seven, used. Usually each answer is a single expression but sometimes whole sentences or even paragraphs are used, as in E. The methods of scoring the various types of multiple-answer ex- ercises differ. Type A is ordinarily scored by counting merely the number of correct responses. Sometimes, however, because of the pos- sibility of guessing correctly, a correction is made by using the fol- lowing formula: Score = R — In this R — the number of N - 1 right answers, W the number of wrong ones and N the number of suggested answers in each exercise. Thus if three possible answers are given one-half of the number wrong is subtracted from the number right, if four possible numbers, one-third the number wrong is sub- tracted, if five possible answers, one-fourth, and so on. When only three or four answers are given, it is probably wise to use this formula, but for more than four the increase in the validity of the scores is scarcely great enough to justify the required computations. 21 The scoring of type B is somewhat more complicated than that of the previous one. Each exercise should contain one best answer which is valued at, say, five points, at least one absolutely incorrect answer for which no credit is allowed, and one or more answers worth varying amounts between the best and the worst. A pupil's score consists of the total number of points of credit given for the expressions he underlines. The score for type C is usually the number of correct words under- lined. A correction may be applied by subtracting the number of in- correct ones underlined, however. Type D, compound multiple-answer exercises, may be scored in either one of two ways. An exercise may be counted correct only if the proper expression in each line is indicated or partial credit may be given if the correct expressions are underlined in some lines but not in W all. In the latter case the formula, Score = R •— , may be em- ployed. Variety E may be scored like either A or B. N - 1 1 Ruch, op. cit., p. 114-21. [16] VI. MULTIPLE-REASON EXERCISES A. Multiple-Reason Exercises with only one Correct Reason Directions: One and only one of the reasons given after each state- ment is correct. Indicate the correct one by making a check mark (V) just before it. 1. We know that animal life has existed millions of years because a. Existing forms could not have developed in less time. b. Human records from early times state it. c. It is proven by fossils which have been discovered. d. We know that proper food for animals existed. 2. Caesar conquered Gaul because a. The Gauls were poor fighters. b. His army was better armed, disciplined and led. c. The Romans were larger and stronger than the Gauls. d. He had a larger army than his foes. B. Multiple-Reason Exercises with Reasons of Varying Degrees of Merit Directions: Place a check mark (V) in front of the best reason for each of the statements given below. 1. The North won the Civil War because a. It had better generals. b. It had greater resources in men, money and materials. c. Its soldiers were braver and hardier. d. Its navy blockaded southern ports. e. Lincoln was a better statesman than Davis. 2. Coal is a better fuel than wood because a. It burns longer. b. It yields more heat. c. It is easier to procure. d. It costs less. [17] C. Multiple-Reason Exercises with one or more Correct Reasons Directions: Place a check mark (V) in front of each of the reasons given below which apply to the statement under which it is found. There may be only one correct reason for each statement or there may be more than one. 1. Great Britain developed more commerce than France because a. It was more necessary to her prosperity. b. France did not attempt to stimulate commerce. c. France had little access to the sea. d. Great Britain acquired more colonies. 2. Literature developed slowly in America during 1700-1800 because a. Its development requires leisure. b. The intellectual level here was low. c. European writers were imitated too much. d. The material demands of daily life were heavy. e. Books could not be printed here. This kind of test is very similar to the multiple-answer type, the difference being that instead of statements to be completed by the se- lection of the proper expressions it contains statements followed by sev- eral possible reasons of which the correct ones are to be indicated. The three varieties of this type are similar to the first three varieties given under the multiple-answer type. The three varieties of multiple-reason tests just given are scored in the same method as the corresponding varieties of multiple-answer tests. VII. MATCHING EXERCISES Directions: Each word in the list to the right is the English translation of one of the Latin words in the list to the left. Place the letter pre- ceding each English word before the Latin word of which it is the trans- lation. aequor a. shoulder ora b. shore umerus c. couch clipeus d. shield cubile e. sea advena f. stranger [18] Directions: Each of the phrases in the right-hand column is descriptive of one of the men named in the left-hand column. Make the proper connections by writing the number preceding the phrase in front of the name of the man with whom it should be associated. Tennyson 1. Modern American novelist. Kipling 2. "The Quaker Poet." Whittier 4. Modern English writer of eastern tales. Wright 4. American poet, wrote much for children. Riley 5. Poet laureate, author of "In Memoriam." In the use of this type one should take care not to make the cor- responding lists contain too many items, since so doing results in con- siderable waste of time in looking up and down the list to find the de- sired expressions. From ten to twenty items is probably the optimum number for a single list. If it is desired to give a longer test than this the material should be grouped into parts. An objection sometimes made to this type of test is that if a pupil knows and has marked most of the items it is easy to mark the few remaining correct ones by a pro- cess of elimination and careful guessing. The writer does not believe that this objection is very serious. It can be eliminated, however, by placing more items in one list than in the other. The score is the number correct. VIII. DEFINITIONS A. Simple Definitions or Explanations Directions: Define or explain each of the following words as concisely and clearly as possible. 1. a. protozoa 2. a. siderite b. felidae b. compound c. eugenics c. osmosis d. embryo d. ion B. Same or Opposites Directions: The two expressions in each of the following pairs have either practically the same meaning or directly opposite meanings. In- dicate those which mean the same by writing an "s" in front of the pair and those which have opposite meanings by writing an "o" in front of the pair. [19] 1. a. treaty, protocol 2. a. bon, mauvals b. plenary, restricted b. aller, marcher c. repeal, abrogate c. cesser, continuer d. executive, administrative d. venir, arriver C. Distinguishing Exercises Directions: Indicate clearly the difference or distinction between each of the following pairs of words. Do so in as few words as possible. 1. a. vapor, steam 2. a. fry, broil b. heat, light b. boil, stew c. sound, hearing c. protein, fat d. static, current d. bake, roast These kinds of exercises are not new, nor are A and C highly objec- tive. They are included here, however, because they do offer the opportunity of marking on a definite point system. It is almost always possible to grade a definition either as right or wrong, or as right, half right or wrong. The second and third varieties given do not call for definitions but probably belong under this general heading. The score on types A and C may either be taken as merely the number right or two points may be given for each entirely correct and- one for each partially correct definition. For type B the score is the number right minus the number wrong. IX. ENUMERATION A. Complete Enumeration 1. Name the presidents from Lincoln to Coolidge. 2. Give the nominatives of all the personal pronouns. 3. Name the different varieties of maples. 4. List all the factors of a 4 — a 2 b 2 . B. Partial Enumeration or Giving Examples 1. Name five foods rich in starch. 2. Mention four breeds of dairy cattle. 3. Name six confederate generals. 4. List five common coal-tar products. This also is not a new type of test exercise but one which has been in fairly common use. With a little care it can be made entirely ob- [20] jective. Furthermore, it has one of the widest ranges of usefulness of any of the objective tests. In both varieties of enumeration one point credit should be allowed for each item correctly given. Thus the total number of points of credit on each exercise should equal the number of items to be named in that exercise. X. ASSOCIATION EXERCISES A. Opposites Directions: Write the opposite of each of the following words imme- diately after it. 1. a. senor 2. a. sharp b. bianco b. pianissimo c. bien c. crescendo d. corto d. allegro B. Genus-Species 1. To what family does each belong? a. cat b. wolf c. whale d. earthworm e. eagle 2. Name a leading operatic composer of each country. a. France b. Prussia c. Hanover d. Bohemia C. Connected Terms Directions: Underline all words in each parenthesis which are in some way closely connected with or related to the word preceding the paren- thesis. 1. Noun (case, conjugation, number, voice, subject) 2. Italy (Mazzini, Thaddeus, Cavour, Kossuth, Garibaldi) 3. Cattle (Hereford, Jersey, Poland China, Ayrshire, Arab) 4. Dress slip (satin, wool, silk, sateen, jersey) [21] D. Disconnected Terms Directions: Cross out the one word in each parenthesis not closely con- nected with or related to the word before the parenthesis. 1. Electricity (fermentation, ohm, potential, volt, static) 2. Triangle (vertex, right, equilateral, radius) 3. Egg (fat, sugar, protein, calcium, iron, phosphorus) 4. Future (hablare, vivira, comia, estudiaran, sentiras) E. Disconnected Terms Directions: Cross out all words in each parethensis which are not closely connected with or related to the word before the parenthesis. 1. Compound (hydrogen, air, water, oxygen, carbon, salt) 2. Georgia (Atlanta, Columbus, Jackson, Tampa, Savannah) 3. Arm (femur, tibur, ulna, radius, carpal, phalanges) 4. Wool (dimity, broadcloth, calico, serge, tweed) These exercises involve the association of two or more words in any one of several ways. The first type included might have been classified under definitions but it seemed to the writer slightly better to place it here. Unless considerable care is exercised in selecting the items to be included in type A, scoring will not be perfectly objective, since it will be possible for pupils to give words which are not entirely opposite to the ones given and yet come fairly near to being so. It will be noted that variety B is in no sense new. For types A and B the score is the number right. The usual method of scoring C and E is to count the number of words correctly underlined. Sometimes, however, the number incor- rectly underlined is subtracted from the former number. It is probably better to make this correction. Type D is scored like the multiple-answer type with only one cor- rect answer. [22] XI. ARRANGE-IN-ORDER EXERCISES 1. Arrange in chronological order, beginning with the earliest, a. Xerxes b. Cyrus c. Philip d. Pericles e. Ptolemy 2. Arrange in order of protein content, from least to most, a. oatmeal b. round steak c. milk d. white bread. 3. Arrange in order of area, from smallest to largest, a. Siberia b. China c. Japan d. India e. Persia. 4. Arrange in order of specific gravity, from lowest to highest. a. lead b. iron c. aluminum d. gold e. copper. This is one of the types which may be made entirely objective. There are, however, several possible methods of scoring. The simplest way is merely to give one point credit for each term placed where it should be regardless of where the other terms are placed. A more accu- rate method but one which increases the labor of scoring considerably is as follows: Let each item count for one point less credit than the number of items. In other words, if there are six items let each count five points. Deduct the difference in rank between where each item should be and where it has been placed from the number of points allowed for each item. For example, if an item should be second and has been placed fifth, three points should be deducted from the number of points allowed for each item. The score is then the sum of the number of points left after these deductions have been made. XII. IDENTIFICATION EXERCISES A. Picture or Diagram Numbered or Lettered 1. Directions: Each of the numbers on the triangle indicates the loca- tion of one of the parts named in the list of terms at the right. Make the proper connections by writing each number in front of the appro- priate term. right angle external angle acute angle hypotenuse vertex leg [23] 2. Directions: Place each of the letters found on the figure in front of the name of the bone upon or nearest to which it is found, if this name appears in the accompanying list. radius humerus femur ulna tibia carpals tarsals B. Picture or Diagram to be Numbered or Lettered 1. Directions: Place the letter found in front of each term at the right on the part of the figure to which the term refers. a. spur b. comb c. hock d. ear-lobe e. wing-covert f. tail-covert [24] 2. Directions: Place the number found in front of each Canadian city in the list at the right on the map at the spot where the city is located. 1. Denver 2. Toronto 3. Sydney 4. Quebec 5. Halifax 6. Manchester 7. Ottawa 8. Vancouver 9. Winnipeg 10. Montreal This type is one which it is practically impossible to use in several subjects. It involves the use of a picture, figure or diagram and the identification of various parts thereof as is shown by the examples. The identification may be made by having the picture or figure num- bered or lettered and connecting the numbers or letters with the proper terms or the terms may be numbered or lettered and the symbols placed in the proper place on the picture or diagram. This type may also be varied by giving a list which contains fewer, just as many, or more terms than the picture or diagram appears to call for. Exercise 2 under A combines the first and last ideas by omitting from the list the name of one of the lettered parts and at the same time including several terms not connected with the figure. Type B is not entirely objective since it is frequently difficult to determine whether the letter or number has been placed on exactly the right part of the picture or not. The score on identification exercises is the number correct. XIII. ABBREVIATIONS OR FORMULAE A. Abbreviations, Symbols, or Formulae to be Given Directions: Give the chemical symbols for each of the following a. hydrogen chloride b. copper sulphate c. carbon dioxide d. sodium [25] 2. Directions: Give the formulae for each of the following: a. area of circle b. volume of sphere c. area of hexagon d. surface of sphere B. Abbreviations, Symbols, or Formulae to be Expanded 1. Directions: Give the word or words for which each of the following stands: a. C.O.D. b. F. O. B. c @ d. " 2. Directions: Give the musical term for which each of the following stands: a. ff b. ppp c. M. D. d. m. f. This is another kind of objective test which is not at all new. It has little place in some subjects but in others it has a wide range of use. The scoring may be made entirely objective. The score is the number correct. XIV. CLASSIFICATION A. Exercises with one Extraneous Word Directions: In each line there is one word which is unlike or of a dif- ferent sort from the rest. Cross it out. 1. radish, salsify, onion, beet, parsnip. 2. noun, verb, pronoun, phrase, conjunction. 3. hasta, tempora, porta, silva. 4. circuit, supreme, district, township. B. Exercises with one or more Extraneous Words Directions: Cross out all words in each line which are unlike or of a different class from the majority of the words in that line. 1. Grant, Jackson, Lee, Sheridan, Johnston, Early. 2. Cedric, Bois-Gilbert, Saladin, Rowena, Bertha, Rebecca. 3. calico, gingham, flannel, serge, madras, dimity. 4. oak, fir, walnut, mahogany, gum, pine. [26] This, and also the following type, seems to be among the least valuable for the purpose of testing achievement, since exercises of this kind are probably in too large a degree measures of general intelligence. They do, however, yield some measure of achievement and may receive a limited use for this purpose. The scoring of these two varieties is the same as that of varieties D and E of the association type. XV. ANALOGIES A. Analogies with no Answer Suggested Directions: Write on the blank line the proper word or expression to complete each of the following statements: 1. a 2 is to a 6 as x 3 is to 2 is to ceder as hablaron is to hablar. 3. Lain is to lie as is to sit. 4. Sharp is to flat as crescendo is to B. Analogies with Suggested Answers Directions: Underline the one of the answers in the parenthesis which makes the truest statement. 1. Square is to cube as circle is to (radius, solid, sphere, circumference). 2. Gallons per minute are to pressure as watt hours are to (volts, ohms, amperes, mhos). 3. Red is to green as (orange, brown, violet, yellow) is to blue. 4. Je is to moi as ils is to (il, elles, leur, eux). This type of exercise is much used in intelligence tests and usually belongs there rather than in an achievement test even though it deals with subject-matter commonly taught. On the other hand, there are probably some cases in which such a test of subject-matter may be used with good results. At least two varieties of this type may be used, one of which is essentially of the single correct answer or completion type, the other of the multiple-answer type. The score on type A is the number of correct answers, whereas B is scored in the same manner as the multiple-answer type with only one correct answer. [27] CIRCULARS OF THE BUREAU OF EDUCATIONAL RESEARCH, COLLEGE OF EDUCATION, UNIVERSITY OF ILLINOIS, URBANA, ILLINOIS No. 19. Streitz, Ruth. Provisions for Exceptional Children in 191 Illinois Cities. No. 20. McClusky, Frederick Dean. Place of Moving Pictures in Visual Education. No. 21. Monroe, Walter S. Announcement of the Bureau of Educational Research for 1923-24. No. 22. Odell, Charles W. Provisions for the Individual Differences of High School Pupils. No. 23. Monroe, Walter S. Educational Guidance in High Schools. No. 24. Nolan, Aretas W. The Project in Education with Special Reference to Teach- ing Agriculture. No. 25. Monroe, Walter S. and Clark, John A. Measuring Teaching Efficiency. No. 26. Barton, H. J., Clark, E. L., Pence, Helen, and others. Notes on the Teaching of Latin in High Schools. No. 27. Streitz, Ruth. Educational Diagnosis. No. 28. Staley, Seward C. The Program of Sportsmanship Education. No. 29. Odell, Charles W. The Use of the Question in Classroom Instruction. No. 30. Odell, Charles W. The Evaluation and Improvement of School Buildings, Grounds and Equipment. No. 31. Monroe, Walter S. The Planning of Teaching. No. 32. Miller, F. J., Flickinger, R. C, Sargent, Rachel L., Luke, Ethel J., Thompson, Glenna D., and others. Latin in High Schools. No. 33. Odell, Charles W. Educational Tests for Use in Elementary Schools, Revised. No. 34. Odell, Charles W. Educational Tests for Use in High Schools, Revised. No. 35. Monroe, Walter S. The Making of a Course of Study. No. 36. Reagan, George W. Principles Relating to the Engendering of Specific Habits. No. 37. Herriott, M. E. How to Make a Course of Study in Arithmetic. No. 38. Odell, Charles W. The Assignment of Lessons. No. 39. Prescott, Henry W., Flickinger, Roy C, Woodruff, Laura B., Whaley, Irene G., and others. Appreciation of Latin. No. 40. Orata, Pedro T. Adaptation of Subject-Matter and Instruction to Individual Differences in the Elementary School. No. 41. Herriott, M. E. Modifying Technique of Instruction for Gifted Children. No. 42. Herriott, M. E. How to Make a Course of Study in Reading. No. 43. Monroe, Walter S. Projects and the Project Method. Xo. 44. Odell, Charles W. Objective Measurement of Information. A limited number of copies of these educational circulars are available for free distribution to superintendents and teachers in Illinois. We shall be glad to add to our mailing list for these circulars the names of any teachers or superintendents who care to receive them regularly. We shall be glad also to send additional copies of any circular to superintendents or principals for distribution among their teachers. Address all communications to the Bureau of Educational Research, University of Illinois.