hh lilHlli iHHI iisni ilfl J Mill 1HBH llffii ill— will ill" e llini IMilli IHE9i!!li99 ■ JI HW 111 111 ill" LIBRARY OF THE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN 510.84 no."77 quit this exercise and go on. ■u might 1 ike to -try about 1J@ problems iSTuPl Figure 6. Penalty for an Early Exit 19 You. haven ' t worked I ~*rri?=, tt' J_ 1 20 79 percent of the integer constants presented were correctly identified 91 percent of the real constants presented were correctly identified 56 percent of the invalid constants presented were correctly identified Mistaken for an integer constant: real constants: 11 invalid constants: 12 Mistaken for a real constant: integer constants: 26 invalid constants: 56 Mistaken for an invalid constant integer constants: real constants: h Average time for drill: Minimum time for drill: Maximum time for drill: 1U5 seconds 25 seconds 286 seconds Table 3 Constant Identification Drill Statistics 21 the code. In the original version of the drill, a new number was randomly generated each time a student was asked to classify a constant. While this was a nice feature, it was extravagent of lesson space and not worth the effort since the data gathered showed that practically all students worked the required ten problems correctly and immediately proceeded on through the lesson. The new version of the drill uses a random selection-without-replacement method, selecting one of thirty constants from a list of ten integer constants, ten real constants, and ten invalid constants. The statistics on this particular drill were skewed by the behavior of one student. The student eventually classified a grand total of 79 constants; no one else ever classified more than 20. Moreover, he classified 71 of the 79 numbers he was presented as real constants. His behavior partially explains the high percentage of real constants correctly identified and the large number of integer and invalid constants mistaken for reals . 5 -^- The variable identification drill The variable identification drill (see Figure 7) is very similar in form to the constant identification drill. A student is asked to classify a given variable name as integer in type, real in type, or neither (invalid for one reason or another) . As in the constant identification drill, the student must correctly work ten problems before he can escape from the drill. The random selection-without-replacement method for choosing which variable names a student is asked to classify, was first implemented in this drill. It worked so nicely and required such a minimal amount of lesson space that the constant identification drill was later modified to use the same selection procedure. From a list of thirty variable names, ten 22 Figure 7 . Variable Drill 3C) VARIABLE DRILL This is a drill on FORTRRN variable names. You i,ui [] be shown a variable name and asked to classify it Type : 1 to classi iy it as _i_nteger r to classify it as real or n to classify it as neither 1 nteeei . i ;•-_■■ i :-.:-or ~ : 1 4 C> f Right 1 It is neither integer nor real sin-: 1 1 bee. i ns w i t h a d 1 g 1 1 „ Press: ( next ) for a new problem. Press: ( stop ) to qu.it this exercise and go on. You might like to try about 1 J0f problems 23 integer, ten real, and ten invalid, a variable name is randomly selected to be presented to the student for him to classify. Since the particular variable name selected varies randomly from student to student, it is statistically highly improbable that students will be presented the same variable names to classify in exactly the same order. Indeed, unless a student works all thirty problems and exhausts the list of variable names, the subset of variable names he will be presented to classify will probably be different from the subset of variable names another student is presented. If a student should be ambitious enough to exhaust the list of variable names by working thirty problems in a row (data collection showed no students so inclined), the list of variable names is restored and the student will begin reclassifying variable names he has already seen, though the names will appear in some new random order. The statistics for this drill (see Table h) seem to indicate that students recognize valid integer and real variable names fairly well, but have considerably more difficulty discerning that illegal variable names are indeed invalid. Since the lesson text gives examples of valid integer variable names and valid real variable names, but no examples of invalid names perhaps this is to be expected. At any rate, since the lesson carefully explains why an invalid name is illegal, no changes were made to the lesson as a result of this drill. The students' ability to use the given variable name rules to identify valid names was judged to be adequate while the non-ability of students to use the given rules to identify invalid names was thought to be of little importance. After all, any compiler given an illegal variable name will produce error messages. 2k 91 percent of the integer variable names presented were correctly identified 92 percent of the real variable names presented were correctly identified 29 percent of the invalid variable names presented were correctly identified Mistaken for an integer variable name: real variable names: 3 invalid names: 16 Mistaken for a real variable name integer variable names: 7 invalid names: 21 Mistaken for an invalid name: integer variable names: 5 real variable names: k Average time for drill: 110 seconds Minimum time for drill: seconds Maximum time for drill: 3&0 seconds Table h. Variable Identification Drill Statistics 25 5 .5 • The built-in function drill The "built-in function drill (see Figure 8) evidently caused students little difficulty so the built-in function section of the lesson was left unchanged. Only four wrong answers were recorded and most of these seemed to be due to not understanding the difference between the square of a number and the square root of a number, a subject not within the scope of this lesson. 5 .6 . Which-order drill #1 The section on operator precedence contains two exercises where a student is shown a FORTRAN assignment statement and asked in "which order" are the operations performed. The first of these which-order drills was so easy every student answered the questions correctly (see Table 5). This being the case, the exercise was judged to be too simple and a new element of difficulty was added by changing the statement X=R+S**T/U (the original problem) to X = R - SQRT(S) / T ** U to include a built-in function (see Figure 9)« Average score: 2 of 2 Average time for drill: 10 seconds Minimum time for drill: seconds Maximum time for drill: 22 seconds Table 5. Which-Order Drill #1 Statistics 26 Figure 8. Built-in Function Drill ' !] id ;. , alu.ei i =d --,+. 5£ igned to variabl fter I— I I— I Hi — i— i I ; i J f P. - ; +- , - 27 Figure 9. Revised Which-Order Drill WHICH ORDER? Given the following i-ORTRRN assignment statement ' f 1 i i ( I -1 1 J k ;-) T . 1 . . * . II i "i ■' V '■ ■' '1' '■ ■=l. i 4 rh d Jh 1 ch operat 1 or K ,- ■ f\\~- dl 1 ■=, ►"•:.■=■'!■"■ t ' '■'■•V' nrifd first" b Bu 1 1 1 - 1 n f u.nct 1 ons h 1 ghest precedence o. orrect ! Exponent 1 at 1 on next third? C orrect ! "D 1 v ■ s 3 on t h 1 rd find by process of elimination, the the last operation performed. subtraction a must 28 5. 7- Which-order drill #2 The second which-order type drill also tested operator precedence, particularly for operators of equal precedence (see Figure 10). When statistics revealed (see Table 6) that students had little difficulty with the drill, the first statement was made more difficult by adding an unary operator. This was done not only to make the drill more challenging, but because other drills revealed students to be weak on unary operators. Average score: 5.2 of 6 Average time for drill: 39 seconds Minimum time for drill: seconds Maximum time for drill: 67 seconds Table 6. Which-Order Drill #2 Statistics 5.8. Multiple choice drill The final drill of the operator precedence section of fortarith is the multiple choice drill. In this drill, students are shown a mathematical expression and three FORTRAN statements. They are then asked which statement accurately calculates the mathematical expression shown (see Figure 11). Some of the multiple choice questions gave students a great deal of trouble and this prompted a rethinking of both the drill and the preceding material (see Table 7). Originally in the drill, students were told which answer was correct and, if the answer they gave was wrong, the reason why their answer was incorrect. This all seems reasonable but one major assumption is made. The assumption is that, if the students' answers are wrong and they are told why it is wrong and shown the correct answer, the students Figure 10. Revised Which-Order Drill #2 5D) PRECEDENCE DRILLS !n the FORTRRN assignment statement be low: L: Si 4 N 29 inici h operat i on (a , b , is per for f l r st ? Correct ! Exponentiation has The left • the right i v i s l on be lore z\ Corr ect l \ lie r i ght :\ i v i s i on n The unary m i nus , last. Remember tha -,-t- and as above or j i nary as i n add i t i o i s o i course done , whether unary "i and subtract ioT always have the lowest precedence of all op 1+. 30 Figure 11. Multiple Choice Drill MULTIPLE CHOICE e the lettei ih i ch ...-> i f the i 4. "RTRAN statement she UJT | ates the following, mat he ■ ,-j - - -,-i RWfc ■f th 31 will immediately see why the correct answer is correct. This does not always seem to be the case. Therefore, the drill was rewritten to tell not only why a student's incorrect answer is incorrect, but why the correct answer really is correct. Data collection was also useful in showing which incorrect answers were favored. When writing a multiple choice drill, it is sometimes difficult to write wrong answers that look vaguely plausible rather than totally absurd. As it turned out, students chose rather evenly between "distractors" in this drill, but if they had not, less favored incorrect answers could have been replaced by equally wrong answers designed to "appear" more correct. Average score: 3-8 of c ; Average time for drill: 182 seconds Minimum time for drill: kl seconds Maximum time for drill: 368 seconds Table 7. Multiple Choice Drill Statistics 5-9* Integer division drill After a section of text describing the workings of integer arithmetic and in particular integer division, students were presented with a drill consisting of five FORTRAN statements to be evaluated (see Figure 12) Originally, this drill had a list of lettered answer choices and a student specified his answer by entering the letter next to his choice. This seemed unnecessarily complicated and restrictive, however, so the drill was revised Figure 12. Revised Integer Division Drill INTEGER DIVISION DRILL 32 What va statements ?=■ =, < t~ tVw=» t-, : I rax i i ow i risr R I P •}" I .!.!*=■>- *=■ ■r b or 33 to allow a student to type in his answer directly. This was the only change made to this section of the lesson since students really did rather well on the drill (see Table 8). Average score: k .3 of 5 Average time for drill: 56 seconds Minimum time for drill: seconds Maximum time for drill: 195 seconds Table 8. Integer Division Drill Statistics 5 . 10 . Mixed mode drill The mixed mode drill presents a student with two integer and two real variables whose values have already been assigned. The student is next shown an assignment statement with the given variables combined in an arithmetic expression to the right of the equal sign. The student is then asked \«rhat value will be assigned to the variable left of the equal sign (see Figure 13) • Students made quite a few errors in this drill (see Table 9) particularly on one statement with a unary minus. This led to several changes in the lesson. First, to give students an additional encounter with unary operators, the second "which order" drill was modified to contain an unary minus. Secondly, in keeping with the idea that a student should fully understand why a right answer is right, the drill's answer explanations were modified to show the intermediate steps in the evaluation of the arithmetic expression. The revised drill shows a student with an incorrect 3>+ Figure 13 . Revised Mixed Mode Drill 6E) MIXED MODE DRILL Assume the following in made : 55 1 arnment s have a 1 r e a d 1 - H R = 1 T What value w i rig st at ement s = 4. ,.j R i pht ! e assigned to X in the follow- I v\ 35 answer exactly where he went astray. Meanwhile, a student who answered correctly is able to see graphically the step-by-step procedure he went through mentally to arrive at the correct answer. The new answer explanations are particularly effective on PLATO since the sequence of the intermediate results -- the fact that the intermediate calculations are performed one at a time in a certain order -- is emphasized by displaying the intermediate results one at a time in sequence on the screen. Average score: 2.7 of U Average time for drill: 169 seconds Minimum time for drill: 7^ seconds Maximum time for drill: 559 seconds Table 9. Mixed Mode Drill Statistics 5-11. Time spent in the different sections of fortarith Fortarith can be divided rather naturally into eight sections: the title and introduction to the lesson, the table of contents, and the six major topics taught in the lesson. Code was installed to measure how long a student stayed in each area. The length of time spent in a particular section (see Table 10) turned out to be roughly proportional to the amount of code comprising the section. Not surprisingly, a student tended to spend more time in the sections with several drills and less time in the sections with no or only one drill . 36 Title and introduction Table of contents Assignment statements Constants Variables Built-in functions Operator precedence Real, integer, and mixed mode arithmetic Average Minimum 12 Maximum 35 175 21 7 50 138 19 k-jk 702 113 1,1+85 228 7 668 111 6 328 568 272 901 553 12 l,*+35 Table 10. Time (in seconds) Spent in the Different Sections of Fortarith 5 • 12 . Help sequence usage Help sequences in PLATO are special sections of code branched to when a student presses the -HELP- key. There are two such sequences in fortarith. The first explains the concept of scientific notation and is available from the section of the lesson that introduces real constants in E-form. Scientific notation is mentioned and students are told that a HELP section is available if they wish to review exactly what scientific notation is. The other help sequence in fortarith is a "universal" help sequence that is branched to when no specially written help sequence is available. It tells the student no special help sequence exists for the particular section he is currently studying and suggests that he use the term "comment" to leave a comment about the lesson if he is having difficulty understanding something. 37 Scientific Notation Help Sequence Number of times requested: 8 Number of different students requesting: 8 Universal Help Sequence Number of lines requested: 6 Number of different students requesting: 3 Table 10. Help Sequence Usage (for 23 students) 5.13- Use of the -TERM- key When a student on PLATO presses the -TERM- function key, he is given an arrow at which he can enter an 8-character "term* A lesson can be easily programmed to recognize certain terms and take appropriate action While the -TERM- key facility provided by PLATO is quite general, fortarith uses terms primarily as a jumping mechanism. The terms "index" and "contents" return the student to the table of contents. The term "notation" jumps the student to the help sequence explaining scientific notation. The term "help" gives the universal help sequence. The term "term" shows the list of terms available in the lesson. The term "guide" returns a student to his or her router lesson, while "talk" and "comment" do jumpouts to lessons cstalk and cscomments, respectively. Students did not use terms extensively but when they did, most of the terms they typed in were programmed for (see Table 11). Three times students entered terms that were unrecognized. One of these "scientif , " probably the 8-character truncation of "scientific notation, " was made into an expected term as a result of the data collection. The other two unrecognized terms, "cs 103" and "foot of" were probably misguided attempts 38 by students to get the list of CS 103 lessons or to literally "go to the foot of the class." Terms Found Term Frequency index k guide 2 term 1 help 1 Terms Not Found Term Frequency scientif 1 cs 103 1 foot of 1 Table 11. Usage of the -TERM- Key " 39 6 . CONCLUSION Data collection as applied to fortarith seems to have been a feasible and fruitful lesson improvement technique. It -would seem to be an excellent method for improving any lesson of the common "teach a little, quiz a little" genre. Whether the data collection method would prove as useful in evaluating a more fancifully written lesson is an open question, but one would hope that with an appropriate approach, good results could be obtained for any lesson. Whenever any new technique is developed, some methods are chosen for its initial implementation that later prove to be less than optimal. In applying data collection to fortarith mistakes were certainly made, most minor but some fairly major. The logistics of preparing the data collected in the datafile for analysis turned out to be one of the worst problems. Obviously, one wants a program to do the analysis , but PLATO is hardly the ideal system on which to run a large, time-consuming analysis program. In the first place, PLATO is designed for interaction, not number crunching. Secondly, TUTOR is not the language one would choose for such a program. Thirdly, an analysis program of any complexity would require a fairly large amount of free lesson space, a resource often not readily available. With such drawbacks as these to data analysis on PLATO and with an IBM 360 handy across the street at the Digital Computer Laboratory, the idea of doing the data analysis on a computer other than PLATO ' s CDC CYBER 70 sounded highly reasonable . However, neither PLATO nor TUTOR offered any ko generally available facility by which an author can write the contents of a datafile to a tape or a card punch. Consequently, the author decided to simply print the contents of the datafile and type up data cards from the listing. Though this sounds easy enough, the effort involved in insuring an accurate transcription of the data came to hours and hours of excrutiatingly boring drudgery. Pestering the PLATO systems personnel for some machine readable portable copy of the datafile (i.e., cards or a tape) would have taken time and been trouble, but ultimately would have been far less painful. Another problem with data collection in fortarith was the particular data collection mechanism installed in the lesson. Designed into the data collection mechanism was the concept of "packing, " collecting several pieces of information in a one or two word buffer before outputting the data to the datafile. At first glance, packing would seem to be a thrifty way to conserve space in the datafile. However, if the buffer into which items are being collected is partially full and the student decides not to take the normal path of the lesson, the data items in the buffer are lost. One solution to this problem, although it is wasteful of datafile space, would be to dispense with the packing mechanism entirely and output each data item immediately as it is collected. With hindsight, of course, it is easy to see things that should have been done that were not. It would have been most interesting to record the units from which students requested the universal help sequence, for example. Statistics on the number of shift -NEXT- and shift -BACK- keypresses might also have been nice. The time statistics, on the other hand, were most uninteresting and in future data collection experiments might well be kl eliminated. Students taking fortarith were totally unaware their responses were being monitored and recorded. In retrospect, this seems unethical; a student should at least know he is being used as a guinea pig. Future data collection experiments should begin with a notice informing the student that the lesson he is about to study contains code that will monitor and record his responses. But enough about the things the fortarith data collection mechanism failed to do or, worse yet, did wrong. What did it do right? It pointed the way to specific changes. For example, in the mixed mode drill, 77-8 percent of the students missed the fourth problem. On the second most difficult problem in that drill, only 25. 9 percent of the students answered incorrectly. The key difference between the fourth problem and the other problems of the mixed mode drill was this: to answer the fourth problem correctly a student had to know that an unary minus has the same operator precedence as addition and subtraction. The students evidently had not picked that up from the lesson. A special section was added to fortarith to specifically state that, in FORTRAN, an unary plus or minus has the same operator precedence as a binary plus or minus. To emphasize the point still further, an earlier drill shown by the data collection statistics to be too easy, was modified to contain a problem with an unary minus. That is just one example of the sort of changes data collection statistics generated throughout the lesson. In conclusion then, data collection seems to be an effective lesson improvement technique. Statistics gathered on students' use of a lesson point out a lesson's strengths and, even more clearly, point out k2 its weaknesses. Once a problem area is identified, the collected data indicate, sometimes with startling specificity, what changes need to be made to improve the lesson. to LIST OF REFERENCES Alpert, D. and Bitzer, D. L., "Advances in Computer-based Education," Science , No. 167 (March 20, 1970), pp. 1582-1590- Bitzer, D. L-, Sherwood, B., and Tenczar, P., "Computer-based Science Education, " CERL Report X-37> Computer-based Education Research Laboratory, University of Illinois at Urbana-Champaign, December 1972. Cress, P., Dirksen, R., and Graham, J. W., FORTRAN IV with WATFOR and WATFIV, Englewood Cliffs, New Jersey: Prentice-Hall, Inc., I97O kk APPENDIX A. SAMPLE QUESTIONNAIRE PLATO lesson evaluation form Lesson name Date taken Your namedf requested) Year in school (circle one) Fr. Soph. Jr. Sr. Grad. Other(or nonstudent) PLATO experience( circle a or b) : a. student using PLATO b. author. No. of lessons taken Among all the lessons you've taken, this one would rate Excellent (upper 20%) good average poor very poor ( last 20%) How much time did you spend in completing the lesson? minutes Please answer each question below by placing an X on the space after your response, or on a space between each set of adjective phrases describing some part of the lesson, amplifying your response whenever possible (either in the space provided, or on back). The continued improvement of PLATO lessons depends on accurate feedback to the author. 1. What was your previous knowledge of the lesson material. None some — 2. Does the lesson fit in well with others in the sequence? yes somewhat no don't know a lot COMMENTS on sequence Z>. What grade did the lesson give you? COMMENTS on grading or none given Examples too many very ?lear relevant COMMENTS on examples too few unclear irrelevant Exercises for the student too many very helpful very clear COMMENTS on exercises too few worthless unclear 6. Text material COMMENTS on text too much very clear too little unclear 7. Displ^s (diagrams, flow charts , graphs) excellent COMMENTS on displays poor ^ C. Organization well organized ™™.i,r to poorly organized too restrictive + rsn „„„+ * COMMENTS on organization t0 ° unst ™ ct ^ed Total lesson learned a lot learned nothing eas y to use hard to use too much material + nr , -,- +4 .-, too little much better than a bonk „,, .,, much worse much better than a lecture v, COMMENTS on total lesson mUCh W ° rSe Please use the space below and/or on the back to describe any problems you had witn the lesson. How could the lesson be improved? List any additional criteria which should be used for evaluation. Clonal criteria IBLIOGRAPHIC DATA HEET 1. Report No. uiucdcs-r-75-777 3. Recipient's Accession No. Title and Subtitle DATA COLLECTION AS AN IMPROVEMENT TECHNIQUE FOR PLATO LESSONS 5- Report Date December 23, 1975 6. Author(s) Jeffrey Alan Barber 8. Performing Organization Rept. No - UIUCDCS-R-75-777 Performing Organization Name and Address University of Illinois at Urbana -Champaign Department of Computer Science Urbana, Illinois 6l801 10. Project/Taslc/Work Unit No. 11. Contract /Grant No. 2. Sponsoring Organization Name and Address Department of Computer Science University of Illinois at Urbana -Champaign Urbana, Illinois 6l801 13. Type of Report & Period Covered Master of Science Thesis 14. >. Supplementary Notes i. Abstracts The University of Illinois' computer-assisted instruction system, PLATO, currently has a library of several thousand lessons. Each of these lessons attempts, through the medium of an interactive computer terminal, to present instructional material to students. This thesis proposes a method for improving the effectiveness of PLATO lessons and describes how this method was applied to a particular PLATO lesson written by the author. This improvement procedure has three steps. First, statistics are gathered about students' interaction with the lesson. Secondly, the data collected is analyzed and areas in the lesson where students had difficulty are identified. Finally, areas identified as deficient are rewritten to more clearly present material the average student found puzzling. \ Key Words anil Document Analysis. 17o. Descriptors PLATO, Fortarith b. Uentil icrs Open-Ended Terms 'c. ( OSATI Field/Group '.Availability Sratemenr Release Unlimited >RM NTIS-35 ( 10-70) 19. Security ("lass (This Report ) UNCLASSIFIED 20. Security ( lass (Tin Page UNCLASSIFIED 21. No. o( Pases h9 22. i' USCOMM-DC 40329-P7I ^ I *$ #