BHHHBmSSa poDOD Wfl P ttc cmfl OQ Q e o cw BWEa R9GB § BBWannicWaJiLlM gHJHlgI|H r5»m« Eras 111111 BBBB99r p* ■■--,, Sh Efi9 I J H KB mWSSsm HOBBoHraHoB WHHI LIBRARY OF THE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN 510.84 IX&r no. 824 ~82£> COJ9. Z The person charging ^ ™$£j \ T Z Latest Date stamped below L16 l_O-1096 CDCS-R-76-827 /n^c%- USER'S MANUAL AND GUIDE TO THE ACSES QUIZ SYSTEM by Richard Ivan Anderson \A Ril October 1976 DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN URBANA, ILLINOIS Digitized by the Internet Archive in 2013 http://archive.org/details/usersmanualguide827ande UIUCDCS-R-76-827 USER'S MANUAL AND GUIDE TO THE ACSES QUIZ SYSTEM by Richard Ivan Anderson October 1976 Department of Computer Science University of Illinois at Urb ana- Champaign Urbana, Illinois 6l801 TABLE OF CONTENTS Page 1. INTRODUCTION 1 2 . SYSTEM PURPOSE AND FUNCTION 1 3. FULFILLMENT OF SYSTEM PURPOSE 1 k. SCHEME OF QUIZ ADMINISTRATION 5 5 . HOW TO USE THE QUIZ SYSTEM MONITOR 5 6 . CURRENT STATUS OF THE QUIZ SYSTEM MONITOR 13 7 • QUIZ SYSTEM ADMINISTRATOR 13 8 . EVALUATION OF INSTRUCTIONAL LESSONS l£ 9. FINAL NOTE 20 APPENDIX A - GUIDELINES FOR DEVELOPING COMPUTER SCIENCE QUIZZES 22 APPENDIX B - QUIZ SYSTEM ADMINISTRATOR OPTIONS 32 APPENDIX C - MISCELLANEOUS SYSTEM INFORMATION 37 APPENDIX D - EXISTING QUIZZES kO TABLE OF PLATO DISPLAYS Page Page Presented Each Student Prior to Quiz 6 Sample Quiz Question 7 Sample Corrected Quiz Question 8 Sample Page Presented Each Student After Review of Corrected Quiz 9 Title Page of Quiz System Monitor As It Appears to Unauthorized Users 10 Option Page of Authorized System Monitor Users 12 Sample Data Accumulated for a Quiz Question lk Sample Accumulated Course Data for a Quiz 15 Option Page of Quiz System Administrator 17 Response Data Accumulated During Initial Trial of System 18 Response Data Accumulated Following Instructional Lesson Alterations 21 Sample List of Instructional Lessons with Quizzes 33 Sample Display During Addition of a Quiz-Producing Lesson to the System 35 Sample Display During Deletion of a Quiz-Producing • Lesson From the System 36 TABLE OF DIAGRAMS Simplified View of Interactions That Occur During Quiz Administration 3 1. INTRODUCTION The following document describes various aspects of the ACSES quiz system as it is currently implemented on the PLATO CAI system. With the intent of an introductory manual, the content herein is directed primarily to the potential quiz system user. Appendices conclude the manual by presenting selected system specifics to allow a user a more comprehensive view of how the system works. A close investigation of this paper should thus yield sufficient details of the quiz system's operation to enable knowledgeable utilization of that which the system offers. 2. SYSTEM PURPOSE AND FUNCTION The quiz system was designed and implemented by this author to fulfill two purposes as proposed by R. G. Montanelli, Jr. : 1) to provide a student taking a PLATO computer science lesson with both the means to assess how well he or she learned the material that the lesson is intended to cover and a tool to utilize for aid in learning the topic at hand; 2) to provide members of the Department of Computer Science staff with a means to assess how effective and how thorough a PLATO computer science lesson is at teaching its topic. 3. FULFILLMENT OF SYSTEM PURPOSE To fulfill this dual purpose, a pool of PLATO quizzes (see Appendix D) has been assembled for administration to students at the conclusions of appropriate computer science lessons. Each quiz -1- has been designed to pertain to some well-defined topic within the computer science field, and each quiz question has been selected by the quiz author to test pertinent details of the topic's content. Incorporated into each quiz is a data collection facility to record question responses as made by students during their initial attempts at the quiz. Access to the pool of quizzes is provided via the quiz system monitor to authorized individuals employing PLATO computer science lessons for instructional purposes. Such a user can select a quiz of the desired content area, interactively design the quiz to best suit his or her chosen lesson's needs, view the quiz exactly as a student will, and finally, via only a few keypresses, actually "attach" the quiz to the chosen instructional lesson. The latter task does, though, require a bit more effort as changes must be made to the code of the instructional lesson to enable an interface with the quiz. However, these changes are only minor and are clearly outlined to the user when quiz attachment is arranged. A look at Diagram 1 should be adequate to explain how the instructional lesson/quiz interface operates. At the time a quiz is to be administered to a student, control is transferred from the instructional lesson to a quiz system program that functions as a link to the quiz (arrow A). This program determines which quiz of those available in the system is to be presented for this particular instructional lesson, and control is transferred to it (arrow B). Interaction between the system program and the lesson that produces the quiz occurs at various points during quiz administration (arrow C). Following a review of the corrected quiz, a student is returned to the system program a final -2- quiz INSTRUCTIONAL LISSOM --.4-* <-§.-- QUIZ SYSTEM PROGRAM QUIZ QUIZ QUIZ quiz • • DIAGRAM 1 • • • SlrlfLIFItO VIBtf OF XWTPttCTtOitt THAT OCCUR DURING QUIZ AOttlrtTtfnMTXON -3- time (arrow D) where return to the appropriate instructional lesson is provided (arrow E). After the few minor changes are made to the instructional lesson and a quiz has been attached via the system monitor, quiz administration is ready to commence. Each student from then on will have unrestricted access to the quiz and will be informed of the average score obtained on the quiz by other members of his or her course. Each system user from then on will have unrestricted access (via the quiz system monitor) to data that is accumulated for each quiz question and will be informed (for all courses that have used the computer science lesson) of both the average quiz scores and the average amount of time needed for taking the quiz. And each authorized member of the Department of Computer Science staff from then on will have unrestricted access also to the data accumulated for each quiz question to facilitate analysis of the effectiveness and completeness of both the instructional lesson and the quiz. At the conclusion of course instruction, utilized quizzes should be detached from instructional lessons to clear accumulated course data. This procedure, which can be accomplished through the quiz system monitor, leaves all such data irretrievable. Thus important information should be recorded prior to the fatal keypresses. During the time that a quiz is detached from an instructional lesson, no reversal of the quiz-accomodating-alterations that were made earlier to the lesson are necessary. Transfer of control to the quiz system program so to administer the quiz (arrow A of Diagram 1) will simply result in the display of the message: "No quiz currently exists for this lesson". Control is then immediately returned from whence it was yielded (arrow E of Diagram l). -h- k. SCHEME OF QUIZ ADMINISTRATION All quizzes are designed to be of similar structure (see Appendix A), thus some aspects of the interaction that occurs between a student and a quiz are identical in all cases. The common sequence involved is briefly sketched below (examples are taken from a quiz on FORTRAN character manipulation): 1) once the quiz system program verifies the existence of a quiz for a particular instructional lesson, a page detailing the quiz's purpose is presented to the student (Display l) 2) quiz questions are then administered; questions may be skipped if the student so desires, and all questions may be reanswered in case an error was made (Display 2) 3) when the student decides that his or her attempt at the quiz is complete, he or she may advance to the presentation of the corrected quiz which is accompanied by clarifying explanations (Display 3) k) lastly the student is informed of his or her final quiz score as well as the average score received by others in his or her course (Display k) 5. HOW TO USE THE QUIZ SYSTEM MONITOR Entrance into the quiz system monitor is achieved via execution of program "csqdata". An individual must receive authorization from the quiz system administrator (see section 7) before use of the system's powers is permitted (Display 5). -5- Dear anderson, You a.r v e .about to take a quiz over the material in lesson fort char. This quiz, which was not prepare by the author (s) of fort char, will NOT count a a part of your grade. It lias instead the following two pi i r poses: 1) to give you, the student, somewhat of an idea of how well you. learned the material that the lesson is intended to cover. When you. complete the quiz, you' ] 1 be shown how wel 1, on "che average, previous students have clone i the staff ^ and tho = ,-, rri ^i„ha-l- Rnoui i edge is dt: : '. 1 gT ied tO t€ct wl Id. i t f ln y< 'U.r part . NOTE: The correct answers to each question will not be revealed until you complete the quiz. Press: -NEXT- to begin the quiz -BftCK- to postpone the quiz until later Display 1: PAGE PRESENTED EACH STUDENT PRIOR TO QUIZ -6- Quest ion 3 o f 9 i" -■rtchar qui Lpaili^ Fo 1 1 ow i ng execut i on o f t he f o ] 1 ouj i n;; RERL D/'HIV CHARACTER G , H# 9 , B * 7 / ' Chi I CRGO INTEGER A/ 'ILL. INI V G=B H=B triable G c< 1) CHIC 2) CHIC f. M'-jf ntaini RGO ;tl n< -n* ,--■, +' Jenotes answer you've given b denotes a blank space. Press: -SHIFT DRTR- to have qui -BACK- for question 2 -NEXT- for question 4 z graded Display 2: SAMPLE QUIZ QUESTION -7- I Quest i on 3 < • i. 1 :>rtchar qu 1 z nts Fo i ; :>wing execution of the following code-- S i nee t he length o acter var f char- lable G CHRRRCTER G, B* 7/' CHICAGO', i s not exp I i c i t 1 >-• stated, i t defaults to one byte. So when 7-1 G=B character- long B is assigned to G,j j the right -most 6 characters 3 re] | truncated.; v*r i ab 1 e G cont a. i ns exact 1 y : 1 ] 2 1 i. HICAGG HIC H'.-,U a b * denotes answer you've -given X indicates correct answer b denotes a blank space Press: -SHIFT DRTR- to exit -BACK- for question 2 -NEXT- for question 4 Display 3: SAMPLE CORRECTED QUIZ QUESTION •8- Total score received on quiz; 16/18 Average score of the 3 people of course cs 1 1 ws who have taken the quiz: 9.333X18 To review the quiz again, press -BftCK- . To exit, press -NEXT-. Display k: SAMPLE PAGE PRESENTED EACH STUDENT AFTER REVIEW OF CORRECTED QUIZ -9- You have just entered the X 1 2; -J v m Authored by Rich Finder so SORRY You are not allowed access to this lesson. To be considered for such, state who you 3.re. and whv you d e s 1 r e a c c e s s v 1 a p - n o t e s t c • a n d er son of c s a Copyright ycj. 1 -"'^5 ■:'•--> -J ■•■■* t>-. ,.= + It ■ 1 -. pf-c 1 , ! . . 1 T1 Tl 1 inoi Display 5: TITLE PAGE OF QUIZ SYSTEM MONITOR AS IT APPEARS TO UNAUTHORIZED USERS •10- All the work of an authorized system user is accomplished via manipulation of five options (Display 6). A brief description of each option follows below: Option "l) Attach a quiz to a lesson" To be chosen when a quiz is to be "attached" to an instructional computer science lesson. User receives access to the pool of available quizzes from which one may be selected, designed, and "attached". If desired, a password can be specified to protect the quiz from detachment by other system users. A complete list of changes that must be made to the lesson to enable proper functioning of the lesson/quiz inter- face is supplied. Option "2) Detach a quiz from a lesson" To be chosen when a quiz is to be "detached" from an instructional computer science lesson. User must supply appropriate password (if one was specified earlier via option l) in order to deal the lesson/quiz interface a fatal blow. Option "3) View list of lessons with quizzes" To be chosen to inspect a listing of all computer science lessons currently utilizing a system quiz. Lessons with a quiz attached that for some reason has been disabled are identified. Option "h) View a quiz's accumulated data" To be chosen to inspect student data accumulated by both the quiz and the quiz system. Available data includes responses -11- Choor>e what you want t i< 1) Attach a quiz to a lesson 2) Detach a quiz from a lesson 3) View list of lessons with qui: 4) View a quiz's a< cumulated dat- 5) Exit ■ ] \-\r - - 13.18.14. DRTE - - 03X12 / 7 6 Display 6: OPTION PAGE OF AUTHORIZED SYSTEM MONITOR USERS -12- made by students to each quiz question (Display 7), number of students of each course that have taken the quiz, average quiz score of the students of each course, and the average amount of time required by students of a course to complete the quiz (Display 8). Option "5) Exit" To be chosen to end a visit to the quiz system monitor. 6. CURRENT STATUS OF THE QUIZ SYSTEM MONITOR Currently only quizzes pertaining to selected topics of the FORTRAN programming language have been implemented (see Appendix D). As the first quizzes of the system, these were all designed and developed from objectives used to develop existing FORTRAN instructional lessons; thus very little versatility was provided for a system user to manipulate the quiz design to satisfy other lessons' needs. More quizzes are being developed, however, and those quizzes existing are continually being improved. The availability of a quiz that suits a user's lesson's needs should therefore be investigated via entrance into the quiz system monitor. 7. QUIZ SYSTEM ADMINISTRATOR The one individual who has complete information about every- thing involving the quiz system is the quiz system administrator. This person (currently this author) is responsible for the total upkeep of the system and should be held accountable for all quiz system activities, -13- Quest ion 3 o f fort char 'quiz ntfi Today is 08. 12/7 Fo 1 I ow i ng eyecut i on of t he f o 1 1 ow l ng cede - - RERL D/'HIV CHRRRCTER G , H# 9 , B* 7 / 'CHI CflGO ' , INTEGER R/' ILL INI V G=B H=B variable G contains exactly: ■i i,i CHICAGO 1 2) CHIC i i hQi -. — - . . , . , i" te numoer at tne text o! each possible resf: onse; i nd i ca.tes the frequency that students chose it . X indicates correct answer b denotes a. blank space Press: -SHIFT DRTR- to exit -BACK- for question 2 -NEXT- for que- t ion 4 Display 7: SAMPLE DATA ACCUMULATED FOR A QUIZ QUESTION -Ik- fort subex qu 1 z dat a - 6 / 15/76 t hrough 8 / 1 2 / 7 6 no . who ' ve ftve COURSE ta}cen_gu_iz _s£2E^_ CS101WS 13 11.69,16 cslflljl 15 11.47,16 ge293 1 6.00/16 CS400 ' 3 6 9.39,16 wdl06g 6 7 . 3 3, 1 6 * approximate average in minutes Press -NEXT- to choose another opt l or Display 8: SAMPLE ACCUMULATED COURSE DATA FOR A QUIZ ■15- fiv e . 1 1 me tak lDg_3J=iiz* 6. 6 2 t . 20 4. 00 36 q 8 3 Specific tasks that this person must perform are reflected in the list of options available for a quiz administrator's work (Display 9)« Options 1, 2, k, and 5 are identical in function to the same-numbered options that are available to a quiz system user; the other five options involve tasks exclusive to the quiz system administrator to ensure smoothness of system operation. (see Appendix B) Any problems that develop or any questions that arise concerning the quiz system should always be brought to the attention of the quiz system administrator. The identity of this indidivual as well as information detailing how he or she may be reached will always be specificed on the title page of the quiz system monitor. 8. EVALUATION OF INSTRUCTIONAL LESSONS The potential behind the analysis of accumulated quiz question data for purposes of locating instructional lesson weaknesses is well- reflected in a partial examination of the effects of the initial trial of the quiz system which occurred during the fall semester of 1975. One quiz which was prepared to test selected knowledge of character manipulation in the FORTRAN programming language was presented following an instructional lesson of the same topic. The instructional lesson had been used in two courses by approximately 800 students prior to this trial and had under- gone various revisions based on information obtained during the courses. The accumulation of question responses for this trial was restricted only to those resi^onses made by students of a particular introductory computer science course. Display 10 exemplifies the data as accumulated. Since all questions were designed to be answered correctly by all students if all were intelligently attentive while taking a thorough and effective -16- Choose what you want to do: 1) Attach a quiz to a lesson 2) Detach a quiz from a lesson 3) View list of lessons with quizzes n t" Z M.-ir qu 1 z T .:■, Jc 12 19 ■ Following execution of the following u RERL D/'HI ' ■ CHARACTER G,hk 9,B* INTEGER A/' ILL INI' G=B H=B CUK RGC a r ] ab I e G conta i ns exact 1 HICRGO "* ! ■ •?- X indicat' >rr answei b denotes a blank spa- P'r e : r'-~ > SHIFT DATA- to exit BACK- for quest i on NEXT- for que? -t ion Display 10: RESPONSE DATA ACCUMULATED DURING INITIAL TRIAL OF SYSTEM •18- FORTRAN character manipulation lesson, some problem seemed evident due to a split of responses among three of the five alternatives. Although eight students responded correctly, more appeared to conclude that the length of variable G would be the length of the variable assigned to it, and many could find no satisfactory choice among the first four possibilities, Examination of the instructional lesson revealed two irregularities: l) nowhere did the lesson mention that the default length of a variable declared to be CHARACTER in type is one; 2) a response to a question requiring students to type a declaration statement for a variable that could accomodate only one character at a time was not judged as acceptable unless the variable length was explicitly specified in the declaration. Both irregularities clearly represented lesson deficiencies; in fact, the latter clearly represented a lesson error. The instructional lesson immediately underwent a major "face- lift"; the above-mentioned deficiencies and others that were discovered through similar analyses of other quiz question data* dictated a complete lesson restructuring. Display 11 and display 7 exemplify data as accumulated during spring semester 1976 and summer session 1976, respectively, following the instructional lesson's surgery. The effects of the changes are clearly noticeable. * Other deficiencies discovered included no use of special characters and no discussion of the use of character strings in variables declared other than CHARACTER. -19- 9. FINAL NOTE Various enhancements to and improvements of the quiz system are planned for the future. Such alterations that are foreseen for the "near" future should be of a scope major enough to invalidate any information presented in this document. All occurrences of major system changes will be accompanied by explanatory manual amendments. -20- Quest ior t' ^-rtc.har quiz Jay is 5/2 s/ 76 £_J^ii ni 0< let 1 Following execution of the following code--- REAL D/'HIV CHRRfiCTER G , H* 9 , B*7/ ' CHICRGO ' INTEGER R/' ILL INI'/ G = B H=B ar 1 ab 1 e G cent a 1 ns exact 1 1 7 1) (_ HICRGU j 2.1 r HIC .j - ! ) ■-- HUjU \ .-' H \ - j K- C" '•; n< ne i the above ? r—> number - 3 ± the left i"' j- _ _ 1 - — ' [- 1 r ess: r ?~-e | indi :1a tes ■ j" - ;^ frequency > that students cr _ .._.. __ -.4- nd 1 cates correct answer lenotes a blank space Pr- -SHIFT DftTfi- to exit -BRCK- for question 2 -NEX T - for question 4 Display 11: RESPONSE DATA ACCUMULATED FOLLOWING INSTRUCTIONAL LESSON ALTERATIONS -21- APPENDIX A The following is a handout as prepared by this author for use by individuals in implementing a quiz-producing lesson. Its content clearly does not encompass information necessary for the development of a "good" criterion-referenced quiz; it is rather intended to encompass the needed basics behind the structure of such a quiz. Since the hand- out was intended to be a self-contained document, some information presented in it duplicates information as presented in the body of this manual. GUIDELINES FOR DEVELOPING COMPUTER SCIENCE QUIZZES (Direct questions or comments to Rich Anderson via phone or send a p-note to anderson of csa. ) 1. GENERAL COMMENTS These guidelines exist since all quizzes written for use with computer science instructional lessons are tied into one organizational system that monitors and controls quiz use. Each quiz must be constructed to conform to certain requirements which will allow such an interface. Two major purposes of each quiz: (1) to provide a student with both a means to assess how well he or she learned the material the instructional lesson is intended to cover and a tool to utilize for help at learning the topic at hand. (2) to provide members of the Department of Computer Science staff with a means to assess how effective and how thorough the instructional lesson is at teaching the topic -22- Each quiz should be designed to test pertinent aspects of some well-defined topic in the field of computer science. A quiz author should avoid viewing an instructional lesson on the topic since doing so may bias the choice of quiz questions. Contact R. I. Anderson for direction in determining question content. Quiz questions should neither be "tricky" nor heavily dependent upon knowledge of other materials for complete student understanding. A well-formulated quiz question should be responded to correctly by all students if each was intelligently attentive during a complete and effective instructional lesson. 2. ORGANIZATION OF A QUIZ Each quiz -producing lesson must perform the four main tasks which are discussed below. Each task must begin at a distinct location in the lesson's code and end via a -jumpout- to the quiz system. Task 1 - Administration of the quiz to a student Each question must be presented to a student to allow entry of a response. If a student has previously attempted a question and decides to try it again, his or her original answer (if one had been entered) should be displayed with the question, and a new response should be accepted if the student desires to enter one. A student is not to be told during this task whether or not his or her response was correct. When a student decides to quit, the quiz score must be tallied and if this is the first time this student has taken this quiz, student responses must be stored. Exit is then made via a -jumpout- to the system. Following this exit, a student can decide to return and perhaps -23- change more answers; provisions must thus be made so the final score and stored student responses reflect any possible changes. Task 2 - Review of the corrected quiz Each question must in some manner be again presented to a student designating both the correct response and the response the student entered (if indeed the student did respond). The student's scores and response explanations should also be presented if deemed useful to a student. New answers are not to be accepted during this task. When a student decides to quit, exit should be made via a -jumpout- to the system. Following this exit, a student can decide to return to again review the corrected quiz. Task 3 - Presentation of student responses All final student responses should have been retained exactly (if possible) in a "permanent" location such as (and probably) lesson common. Thus, each question's responses should in some manner be displayed allowing analysis by the staff to locate weaknesses in the instructional lesson, the quiz, and the students. Task h - Accepting quiz specifications from system user Comments about the quiz questions' content should be presented along with any options available to a user to allow him or her some say in what comprises the quiz. A keypress should be all that is necessary to: take a system user into the quiz as a student would view it; take a system user into reviewing the corrected quiz as a student would review it; take a system user back to the system via a -jumpout-. -2k- Much of the content of the pages that follow is derived from a handout prepared by L. R. Whitlock (1976). The basics of the inter- face utilized in the exam system to enable a problem generator /grader to be a part of an exam have been incorporated into the interface utilized in the quiz system to enable a quiz to administered following an instructional lesson. This thus allows a quiz -producing lesson to be used as a part of an exam or a problem generator /grader to be used as a quiz. 3. USE OF STUDENT VARIABLES All student variables are available for use by a quiz-producing lesson except nl31 through nl50; never use student variables nl31 through nl50. A "quiz header" will occupy the first four student variables. Portions of it that are of importance to the quiz-producing lesson are described on the next page. '"Problem Generator/Grader for the Generative Exam System" contents of which can now be mostly found in "Documentation on the Generative Exam System", 6/22/76. -25- nl .quiz ////e ~ Quit IIIq^IP (20 iz. Score, hU. l>oSS/d/e quiz number $$quiz flag $$score on quiz $$total possible points $$quiz specs size (^student's work size k. FORMAT OF QUIZ PAGES The top of each quiz page should appear similar to the following example : quiz number from quiz header — > 1 Question 7 of 9 quiz title from quiz header,*" — v fortfmt2 quiz 4/ 2 points At the bottom of each page should be information stating what a student can do next. For example: Press: -NEXT- for the next question -BACK- for the previous question -SHIFT DATA- to quit -HELP- for some aid -27- 5. KEY CONVENTIONS To maintain uniformity among all quizzes, the following key conventions should be adhered to as closely as possible: For tasks 1 & 2: -SHIFT DATA-' •SHIFT HELP- -HELP- -NEXT- -BACK- 3 -jumpout- student to quiz system present explanation about whole quiz present explanation of current question advance student to next quiz page take student to previous quiz page For task 3: An appropriate subset of the keys of tasks 1 & 2, For task k: ■SHIFT DATA- ■ SHIFT LAB- ■LAB- -jumpout- user to the quiz system review corrected quiz as a student administer quiz as to a student 6. RECIPES FOR THE FOUR TASKS The processing necessary in each of the four tasks that a quiz-producing lesson must perform is presented below in a step- wise manner. Minor variations may be necessitated depending upon the particular quiz. These keys should all accept a student response for task 1. -28- Task 1 Processing (1) quiz flag should be checked to determine whether or not a student is returning to reanswer questions; if so, previous student responses follow the quiz header; otherwise quiz specifications follow the quiz header and should be placed in other student variables (2) present question showing previous response (3) receive new response (k) save precise response so showing previous response (2) is possible (5) judge the response (but don't tell student) (6 ) if the value of the system-defined variable -lscore- equals -1, update permanent statistics to reflect the student's answer (be sure only to record the student's last answer to the question) (7) go to 2 The loop above continues until student decides to quit. Then: (8) quiz score should be tallied and stored in quiz header (9) place student responses in student variables which immediately follow the quiz header (i.e. beginning with n5) (10) execute these commands: -inhibit dropstor- -jumpout return, return - -29- Task 2 Processing (1) [in case used in exam system] quiz flag will be if student did not enter task 1; otherwise ^0. If 0, quiz specs follow quiz header; otherwise student responses follow quiz header (2) present question, student's response, and correct response (3) offer explanations (h) go to 2 The loop above continues until a student decides to quit. Then: (5) execute these commands: -inhibit dropstor- -jumpout return, return - Task 3 Processing (1) present accumulated student responses to each question (2) when user finishes viewing data, execute the following command: -jumpout return, return - Task k Processing (1) provide information about the quiz questions (2) allow instructor to view and review quiz as would a student (3) allow user to make available quiz specifications -30- (k) place specifications in student variables immediately- following the quiz header (i.e. beginning with n5) (5) assign values to maximum points possible, quiz specs size, and student's work size (all in quiz header) (6) when user finishes, execute: -jumpout return, return - -31- APPENDIX B Below is a list of the nine options available to the quiz system administrator through the quiz system monitor. Each option is accompanied by a brief discussion of what each involves. Option "l) Attach a quiz to a lesson" To be chosen when a quiz is to be "attached" to an instructional computer science lesson. User receives access to the pool of available quizzes from which one may be selected, designed, and "attached". If desired, a password can be specified to protect the quiz from detachment by other system users. A complete list of changes that must be made to the lesson to enable proper functioning of the lesson/quiz interface is supplied. Option "2) Detach a quiz from a lesson" To be chosen when a quiz is to be "detached" from an instructional computer science lesson. User must supply appropriate password (if one was specified earlier via option l) in order to deal the lesson/quiz interface a fatal blow. Option "3) View list of lessons with quizzes & attachers" To be chosen to inspect a listing of all computer science lessons currently utilizing a system quiz. Lessons with a quiz attached that for some reason has been disabled are identified. The identities of the individuals who attached the quizzes are displayed as well as the date that the attachment occurred (Display 12). -32- Lesson Nam -- RTTRCHED BY -- f < -COURSE- ■ on ■ -DATE fort do ■•rt array 2 ortsubex or t char or-fc funct anderson mont anderson anderson dan i e 1 son 07.' 05 ■ 7t 06 /l 5/7 6 06/ 15/7 6 7 ,/0 7 • 7 6 .6 8 ••- 12 7 6 indicates quiz is disabled and requires deletior end of 1 is-* Display 12: SAMPLE LIST OF INSTRUCTIONAL LESSONS WITH QUIZZES -33- Option "5) Exit" To be chosen to end a visit to the quiz system monitor. Option "6) View special system data" To be chosen to inspect details of the status of the quiz system to ensure its smooth operation. If detectable problems had at some time developed, appropriate warning messages are displayed. Option "7) View list of quiz lessons available" To be chosen to inspect a listing of all the lessons currently a part of the system that are available to system users to produce quizzes. Each lesson name is accompanied by information specific to the lesson that enables proper coordination between the quiz-producing lesson and the quiz system monitor. Option "8) Add a new quiz lesson (qg/g)" To be chosen to add a quiz-producing lesson to the pool of quizzes available in the system. All information necessary to incorporate the new lesson into the system must be entered (Display 13 ). Option "9) Delete a quiz lesson (qg/g)" To be chosen to remove a quiz -producing lesson from the pool of quizzes available in the system. An alert is issued if deletion of a specific quiz -producing lesson would disable an existing instructional lesson/quiz inter- face (Display lU). 5 qg/g is an abbreviation for "quiz generator/grader", a phrase which technically describes the lesson's function. -3k- Supply th«2 following information about the quiz lesson (qg/gj ( 1 ) Lesson name fort f rnt 1 q (2) Twenty character description fort ran formatting Four entry points: (3) to attach quiz attch (4) to give quiz £ (5) to review quiz revw (6) to analyze quiz responses data "ype the number of an item to change it OR >ress -SHIFT NEXT- to add to the system Display 13: SAMPLE DISPLAY DURING ADDITION OF A QUIZ- PRODUCING LESSON TO THE SYSTEM -35- Name the quiz lesson (qg: g) you wish deleted from the system 'p to: if Li. net q rtr^ you. sure you want to delete this quiz lesson (qg/g) ? It appears you'll disable 1 ouiz (res) i f you do! Press -SHIFT HELP- to delete (The qu i z (zes) < 1 1 sab 1 ed w i 1 1 be f 1 agged on the list of lessons with quizzes. Option 3) Press - BfiCK - to ex i t w ' t hout de 1 et i ng Display lk: SAMPLE DISPLAY DURING DELETION OF A QUIZ- PRODUCING LESSON FROM THE SYSTEM -36- APPENDIX C Certain facts about the quiz system's organization and limitations remain to be mentioned to permit a more complete view of the system and its operation. Since many of these tidbits are quite unrelated, all are presented as elements of a list: -The maximum number of instructional lessons that can be issued a quiz by the system is fifty. However, due to variations that are possible in the amount of storage space required to handle different instructional lesson /quiz interfaces, the actual number of lessons that can be accomodated may actually be less than fifty. -The maximum number of quiz-producing lessons that can be made available to system users is fifty. -Data to indicate average quiz scores, numbers of students who have taken a quiz, and average lengths of time required for students to take a quiz is accumulated separately for each course as long as the quiz system's storage space holds out. Since a vast amount of space is reserved to hold this type of data, space scarcity should be rare. Nevertheless, when space limitations permit no further new course total accumulations for a quiz, all such data is accumulated in general chunks. Data accumulated in this manner is reported as such whenever quiz data is displayed. -37- -Detachment of a quiz from an instructional lesson or deletion of a quiz-producing lesson from the pool of such lessons available in the system immediately frees all the released system space for other system use. No garbage collection routines are necessary. -All internal modifications to the system (e.g. storing gathered student data; saving information supplied by a system user to enable an instructional lesson/quiz interface) are protected both against abort (-SHIFT STOP-) attempts by students or system users and against simultaneous modifications (e.g. if more than one system user attempts to attach a quiz to an instructional lesson at the same time). -Data made available by the administration of a quiz is retained only the first time a student attempts a quiz. This is achieved by saving the student's quiz score in his or her individual router variables enabling a check to see if a quiz score has yet been recorded. An instructor of a course thus has access to individual student quiz scores, but no additional score can be issued by the instructional lesson itself. -When a student completes a quiz and control is transferred back to the instructional lesson, all student variables are reinstated exactly as they existed prior to the keypresses that commenced quiz administration. • -38- -The quiz system in its entirety consists of only two programs (i.e. disregarding the individual quiz- producing lessons), "csqdata", as mentioned before, comprises the quiz system monitor that is accessed directly by both the quiz system users and the quiz system administrator, "csxquiz" is the smaller system program that handles the interface between an instructional lesson and a quiz. -39- APPENDIX D The quizzes listed below either are currently available in the quiz system or are currently being developed for such availability. The authors listed are those individuals responsible at least for most of the code that produces the quizzes; in most cases, substantial aid and consultation in such areas as question development and quiz structure was provided by either/both this author or /and R. G. Montanelli, Jr. Quiz Topic FORTRAN character manipulation FORTRAN 2-dimensional arrays FORTRAN DO loops FORTRAN subroutines FORTRAN arithmetic FORTRAN functions FORTRAN formatting (elementary) Quiz Author R. I. Anderson D. Gamage J. Miller T. Puterbaugh J. Miller J. Glauert R. Bogue The above list could possibly be expanded due to the existance of L. R. Whitlock's generative exam system. Details of the interface that enables a quiz's administration following an instructional computer science lesson were derived from details of the interface that enables the administration of an exam in the PLATO generative exam system. Certain aspects of both interfaces were implemented sufficiently identical such that a problem set L. R. Whitlock, "interactive Test Construction and Administration in the Generative Exam System", University of Illinois, Department of Computer Science, Report No. UIUCDCS-R-76-821. -1*0- available for use as a portion of an exam in the exam system could actually be made available for use as a quiz in the quiz system (and vice versa). Since all problem-set-producing lessons do not incorporate question response gathering facilities as do the quiz -producing lessons, alterations to the exam system lessons -would be necessary prior to such an exchange. The full potential of these exchanges is yet to be investigated. Whitlock refers to these as problem generator/graders. -1+1- BIBLIOGRAPHIC DATA HEET 1. Report No. UIUCDCS-R-76-827 3. Recipient's Accession No. 5. Report Date October 1976 Tide and Subtitle USER'S MANUAL AND GUIDE TO THE ACSES QUIZ SYSTEM 6. Author(s) Richard Ivan Anderson 8. Performing Organization Rept. No. Performing Organization Name and Address Department of Computer Science University of Illinois at Urbana-Champaign Urbana, Illinois 6l801 10. Project/Task/Work Unit No. 11. Contract /Grant No. 2. Sponsoring Organization Name and Address Department of Computer Science University of Illinois at Urban a- Champaign Urbana, Illinois 618OI 13. Type of Report 8c Period Covered Technical 14. S. Supplementary Notes 5. Abstracts This document describes the ACSES quiz system as implemented on the PLATO CAI ;ystem. The quiz system allows an instructor of a computer science course access to i pool of PLATO quizzes for use with PLATO instructional computer science lessons ,hat will be utilized by the instructor's students. The use of a quiz with an .nstructional lesson provides both a study aid for each student and a means to assess iow effective and thorough a computer science lesson is. This latter provision is .ccomplished through the collection of quiz question data which, after collected, an be analyzed in search for instructional lesson (or quiz) weaknesses. Presented herein is a description of aspects of the quiz system from both an nstructor's standpoint (e.g., instructions on how to use the system) and a student's tandpoint (e.g., the interaction between the quiz and the student), some discussion f how the system is monitored and maintained, and an example of the potential behind he analyses of collected quiz question data to evaluate an instructional lesson. l7o. Descriptors Accompanying tne document (in an appendix; is a self-contained paper entitled "Guidelines for Developing Computer Science Quizzes" for use by authors of quizzes to be added to the system pool 1. Key Words and Document Analysis. Achievement tests Evaluation Education Instructions Computer programs 'b. Identifiers/Open-Ended Terms 'c. COSATI Field/Group 1. Availability Statement RELEASE UNLIMITED 3RM NTIS-38 ( 10-70) 19. Security Class (This Report) SS1EI1 UNCLASSIFIED curity Class ( 1 hi 20. Security Class (This Page UNCLASSIFIED 21* No. of Pages ^3 22. Price USCOMM-DC 40329-P71 5.or« v r s,TvoF,u,No,s -^A NA 30112088402992 BjKJi ^M ■wi I •yvv % P SB WBWr