LIBRARY OF THE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN 510.84 cop 2. Digitized by the Internet Archive in 2013 http://archive.org/details/effectsofmethodo727sega f Q UIUCDCS-R-75-727 EFFECTS OF METHOD OF ERROR INTERRUPTION ON STUDENT PERFORMANCE AT INTERACTIVE TERMINALS by Barr Zion Segal May 1975 IHE LIBRARY OF THE. JUN 24 1975 UNIVERSITY OF ILLINOIS UIUCDCS-R-75-727 EFFECTS OF METHOD OF ERROR INTERRUPTION ON STUDENT PERFORMANCE AT INTERACTIVE TERMINALS by Barr Zion Segal May 1975 Department of Computer Science University of Illinois at Urbana- Champaign Urbana, Illinois This work was supported in part by the Department of Computer Science and was submitted in partial fulfillment for the Master of Science degree in Computer Science, 1975. Ill ACKNOWLEDGMENT I wish to thank Professor W. J. Hansen for his suggestions and guidance in conducting this study and Professor R. G. Montanelli for his valuable advice on statis- tical design and analysis. In addition I am grateful for help on various statistical matters provided by Linda Schrom of the Computing Services Office of the University of Illinois and Michael Valente of the Department of Speech and Hearing Sciences. IV TABLE OF CONTENTS Page 1. INTRODUCTION 1 1.1 Background 1 1.2 Overview of Study 3 1.3 Limitations of Study 4 1.4 General Goals 4 2. METHOD AND PROCEDURES 6 2.1 Selection of Subjects 6 2.2 Equipment 6 2.3 Presentation 8 2.4 Data Collected 19 3. RESULTS AND DISCUSSION 21 3.1 Statistical Analysis 21 3.2 Results 27 3.3 Discussion 36 4. SUMMARY 43 REFERENCES 47 Appendix A RAW DATA COLLECTED BY PLATO LESSON ... 48 B MEASURES DERIVED FROM RAW DATA 56 C T-TEST RESULTS 65 D ONE-WAY UNIVARIATE ANOVA SUMMARY TABLES. . 6 7 E TWO-WAY UNIVARIATE ANOVA SUMMARY TABLES. . 71 F CORRELATION MATRICES 77 G CLASSIFICATION OF PERFORMANCE FOR MULTIVARIATE ANOVA 83 H PERSONALITY TRAITS MEASURED BY THE 16 PF TEST 84 LIST OF TABLES v Table Page 2.1 2.2 3. 2-a -b -c -d -e -f -g Primary Source Traits Covered by the 16PF Test, Group and Task Order Distribution by Subject Type Summary of Performance Data for States Task . Summary of Performance Data for Permutations Task Analysis of Variance Summary Tables for States Task. Analysis of Variance Summary Tables for Permutations Task Multivariate ANOVA — States Task Multivariate ANOVA--Permutations Task Major Significant Correlations between Personality Traits and Performance Measures — States Task. Major Significant Correlations between Personality Traits and Performance Measures — Permutations Task. 14 29 30 31 32 34 34 37 38 VI LIST OF FIGURES Figure Page 2.1 Screen Layout for Typing Test .... 12 2.2-a Sample Screen Layout for Immediate Error Interruption .... 16 -b Sample Screen Layout for End Error Interruption during Initial Response Entry 17 -c Sample Screen Layout for End Error Interruption during System Guided Response Correction 18 3.1 Two Factor Split Plot Design .... 23 1. INTRODUCTION 1.1 Background Computer systems are playing a steadily increasing role in every aspect of society. Education, business, and government are just a few of the areas where people find it necessary to communicate with digital computers, and the lat- ter are very often represented by interactive terminals. Sur- prisingly little is known, however, about the human factors involved in communication between human beings and computer terminals. R. S. Nickerson [8] pointed out in 1969 that ". . .there is remarkably little evidence of research that has been undertaken for the express purpose of. . .increasing our understanding of man-computer interaction. ..." Al- though some research has been carried out and some suggestions have been made since then (see Martin [7] for examples) , there is still much to be learned in the realm of man-computer com- munication. The study described in this report looks into one specific aspect of man-computer interaction--student perform- ance at on-line terminals under varied methods of error inter- ruption. Presently there are quite a few students learning to program through the use of interactive computer systems, and many of the "interactive compilers" which accept student pro- grams typed at a terminal have one feature in common: when a student commits an error, he is immediately interrupted by the system and forced to correct his error before continuing. (See Wilcox [9] for full description of such a system.) This type of error interruption may damage the stu- dent *s train of thought and thus adversely affect his success at the terminal. The communication process from human being to computer terminal can be looked at as consisting of two parallel processes — a thought process, in which the human generates ideas in his mind, and an output (typing) process in which he transfers these ideas to the terminal. A student might be typing one idea into a terminal, while preparing further ideas mentally. At this point, a system interruption could disturb the balance of thinking and typing, and there- fore it might be better for the student if the system were to wait until he finished a task or subtask before announcing errors . In addition, since human personality cannot be di- vorced from any form of man-machine interaction, it is pos- sible that personality traits are an important factor in the effect method of error interruption has on student perform- ance. Certain types of people may benefit from a specific type of error interruption. (Consider a nervous individual being upset by immediate system interruption — or would he be reassured by it?) The study described here explores some of these problems. 1.2 Overview of Study Two groups of students (equally matched in typing skill and experience on the interactive system used) were each given two simple tasks to perform at the terminal. Sub- jects in one group, upon making an error, were interrupted immediately by the system and forced to make corrections be- fore going on. Subjects in the other group were allowed to finish each task without interruption until they signalled completion of a task, at which time the system announced errors and required corrections. The tasks used were simple word listing problems: 1. List 25 of the 50 states. 2. List 20 permutations of ' abcde * such that the 'c' occurs somewhere before the 'a 1 and the *b' occurs somewhere before the 'd'. Numerous data were collected on subject performance at the terminal, and a standard personality test was adminis- tered to the subjects to obtain data on basic personality traits. The following experimental questions were considered: Does method of error interruption affect subject performance at an interactive terminal, and if so, how do these effects show up? What relationships do personality traits have to performance under different error interruption methods? 1.3 Limitations of Study The study described has several limitations. For example, it does not deal directly with the programming task. Such tasks were avoided for several reasons. First, the need to teach the subjects a programming language or ascertain their level of skill in a known language would have made un- reasonable time demands on the study. Second, programming tasks would introduce a large number of human variables which would be difficult to control, since so little is known about the way human beings tackle programming problems. The man- terminal communication process is the focus of this experiment, and such communication goes on regardless of the task being undertaken. An additional limitation is the fact that this study does not examine the learning process of subjects studying at computer terminals. The question of which method of error interruption provides a better learning environment for the student is certainly an important one. However, an examina- tion of this question would require a much larger and longer study (for instance, a semester long study) . Also, the learn- ing process, like programming, would introduce many additional extraneous variables requiring careful control. 1.4 General Goals In addition to examining the research questions presented in Section 1.2, it is important to find out whether we can, in fact, conduct sensible and useful experiments in man-computer interaction. This study presents examples of what factors may be considered and what measures may be used in evaluating human performance at interactive computer ter- minals. Hopefully, it will serve as the basis for further research. 2. METHOD AND PROCEDURES 2.1 Selection of Subjects The population under study consists of undergraduate students learning to program computers. Seventy subjects were chosen from students enrolled in 100 level computer science courses at the University of Illinois. Approximately one- half of the subjects were members of one course, which was run through the experiment during a scheduled class hour. The rest were volunteers from other basic courses. In this manner a suitable random sample was achieved. To prevent bias toward one error interruption method or another, subjects were not told the exact nature of the experiment. They were told instead that they would partici- pate in a study of common sense memory at computer terminals, which would require completion of several simple tasks. After finishing the experiment subjects were informed of its actual purpose. 2.2 Equipment The computer system used in the experiment was the PLATO IV system [1] f developed at the University of Illinois. PLATO (Programmed Logic for Automatic Teaching Operations) is a computer-aided instruction system consisting of a CDC CYBER-7 3 mainframe supporting sophisticated graphics terminals with plasma screens. This system is used in a variety of departments at this university, including the Department of Computer Science. An interactive program written for PLATO administered the terminal session of the experiment. The program collected background information on the subjects, assigned them to treatment groups, presented the tasks to them, and collected performance data. The error-checking algorithm of the program (also called a "lesson") is described briefly: The program kept an array of correct answers to the tasks, and in addition, a pointer table containing values such that table (i,j) pointed to the first answer in task j beginning with the i letter of the alphabet. (The value of table (27,j) was one more than the location of the last answer on task j.) When a subject began typing a response, the point- er table was used to determine the region in the answer array where the response might reside. As each character was typed, the bounds of the current "correct response region" were placed on a stack. When the subject erased a character the stack was popped so that the bounds as of the previous charac- ter became the new current bounds. These structures plus a counter for number of characters typed so far allowed for ac- curate error checking in each response. For an Immediate group error, the lesson responded immediately, while for the End group the response number and position of the erroneous character were saved for later announcement. In addition, the Sixteen Personality Factors Ques- tionnaire (16PF) , published and distributed by the Institute for Personality and Ability Testing, of Champaign, Illinois, was used to collect data on personality traits. The test, developed at the University of Illinois, measures sixteen basic personality traits. These are listed in Table 2.1. Form C of the test was used in this study. 2.3 Presentation The experimental procedure consisted of two phases. The subjects were first given the 16PF Test, which took approx- imately one-half hour. Following this they proceeded to a PLATO classroom, where the terminal session took place. A PLATO lesson taking about a half hour presented the actual experiment. The session had four steps: a. Background Information The lesson began by collecting the following background information for each subject: Major — 1 - Computer Science 2 - Other PLATO Experience - Has never taken or written a lesson 1 - Has taken or written a lesson Table 2.1 Primary Source Traits Covered by the 16PF Test Factor Low Score High Score B C E H M N Reserved , detached, critical, aloof, stiff Low intelligence Affected by feelings , emotionally less stable, easily upset, changeable Humb 1 e , mild, easily led, docile, accom- modating Sober , taciturn, serious Expedient , disregards rules Shy , timid, threat- sensitive Tough-minded , self- reliant, realistic Trusting , accepting conditions Practical , "down-to- earth" concerns Forthright , unpreten- tious, genuine but socially clumsy Self-assured , placid, secure, complacent, serene Conservative , respect- ing traditional ideas Outgoing , warmhearted, easygoing, participating High Intelligence Emotionally stable , mature, faces reality, calm Assertive , aggressive, competitive, stubborn Happy-go-lucky , enthusia- stic Conscientious , persistent, moralistic, staid Venturesome , uninhibited, socially bold Tender-minded , sensitive, clinging, overprotected Suspicious , hard to fool Imaginative , bohemian , absent-minded Astute , polished, socially aware, shrewd Apprehensive , self-re- proaching, insecure, worry- ing, troubled Experimenting , liberal, free-thinking 10 Table 2.1 (continued) Factor Low Score High Score Q. Group dependent , a "joiner" and sound follower Self-sufficient , resource- ful, prefers own decisions Q, Undisciplined self- conf lict , lax, follows own urges, careless of social rules Relaxed , tranquil, torpid, unf rustrated, composed Controlled , exacting will power, socially precise, compulsive, following self-image Tense , frustrated, driven, overwrought Table was taken from the manual for the 16PF test [6] . 11 Number of Years Programming Number of years (rounded to the nearest integer) subject has programmed On-Line Interactive Systems Experience 1 - Has previously typed a program into an interactive terminal 2 - Has never typed a program into an inter- active terminal b. Typing Test Next, the lesson administered a short typing test to the subjects. Ten fairly large words were displayed on the screen and subjects were asked to type the words as quickly and accurately as possible (see Figure 2.1). The total time elapsed (in tenths of seconds) and the total number of keypresses in error were collected for each subject. c. Group Assignment and Randomization of Task Order In order to control for effects of typing skill and previous experience on PLATO, assignment to method of error interruption groups and order of task presen- tation were determined on the basis of these two variables. Four types of subjects were defined: No PLATO Experience — Bad Typist No PLATO Experience — Good Typist 12 x: -p c •H I I -P £ ■H Vi CD X -P +J rd CO Q) U rd a to cd X -P O -P C ■H U rd o -P o M-) rd (U G CD rd -p nd CD -P CD fd a CO c £ X o -p X! O CO X! CO u T3 O P MH O 5 >i P, CD Eh X! -P CD U CU CD >iT3 Eh P O CD g rd to /N HfMn<*mvo>oo(j\ia. CO to CD P fd •H H fd P -p to 3 -P C CD tO S 3 O CD C o p, p CO to 3 3 o O H P 3 fd ft CD CD /\ o •H -P CO -H C O •H a fd to -P fd 3 T3 P C U CD CO <-\ c td 3 u CO -H XI CD CO -H p rd n a p c g rd rd H & IS1 -P rd C CD > •H tn CD H3 • P P SH rd CD P & S -P CD CO to -P CD C CD OH Id -P rd o P X! -P P fd £ x X fD m U rd C CO td o C to -P CD fd •H to p G -P fd CD U p 3 CD P -P tn -P o CO -P W C W -P H "Si w w • en 1 Eh -P fZ, Eh X K En H CO CD WHS Eh ffl i tn i tr> C ■H • a c >i CD Eh CD P P U CO m CU -P X! 3 -P >»UH rd M" a C CD -P CD P CD o x C/3 -P rHCNC0^1 X! M-l -P cd xi i O Eh 3 X co^ir>vDr~oo<7*TS. cn H HHHHHrlHfN 2 (D 1 T3 u in X! w (T3 CD 5-1 Mh cm — en G • TJ •H +J CD rd }-( 4-> 3 . 3 CN U U CD TJ CD r£ 4-> O T) CD 13 CD CO fd rd X! xjEI •H £ ^2 ^3 rd fd rd hn" U U O O U 'd c fd /\ cu CD cn • cn C 4J -H 1 4-1 CU a J w K cn 1 cn CD cn 54 en ft dJ >i 54 CD a, x rd rHCNf ¥ 1^in^Or^C» 3 ft O 3 o 5-1 5-1 CD - 4J X! C - H 5-1 G 5-1 CD 5-1 CO W p rd QJ O 4-> CD fd & ■H 13 CD CD cn g C g H ft cn 5-1 CD 54 U-( X! 4J 4-> 3 m ■H >iMh rd HH CD Xi C 4-> CD CD G 54 •H O cn t CD - CD 5-1 T3 r-\ U - ft P g O CD fd O Xi w -P rd i CN O) Q) 5h 3 tP ■H fa 17 U CD 0) C rd X! u rd CD • -p -p (0 M-l -P rd (0 I m e-» w 4-> •H in CN CO rd cn C rd M /\ ro^Lnvor^cxjcTiis.rHrsirO'^' iHrHrHrHrHrHrHCNCNCNCNCN rd ■H >1 c cd Q) u cn >,M rd 0) rd cn -P •H rd -H M U -P M > H (0 X 6 O O O i w £ W -r-i cn 6 CD x» -h +j tP rd c x •H CD }-i cn CD 3 u cn • T3 i CD T3 U CD H C T) (U C C cn rd W X -P M tPO O 3 C 4-1 X cn -P -P rd 3 rH X O i C rd J • cn >i cn C M CD CD -P rH CD C H W CD CJ X CO CD -P cn CD G - h o>d a, a cd G cn a rB CD >, CO PS -P X 1 cn • CN CD U 3 cx> •H b 18 ■"d - CD rd X G •H g- -H tr> CD rH n 4-> C 5H o CO rd CD 0) >i Tl tr Xi G C -d co • •H CD -H LD 5-1 .* CN 1 u CO 4-> CD CD CO 4-> 1 • CO c •H 4-> erruption du wers are mar wrong answer ■H -H 4-) (X, ft 4-> CO a C XJ CD hH- c g X ft CO CO W H rd O •H -P ft 3 m CO rd rd co -P rd tP g X! fd i CO 5-1 Cn CD co G co C C C C fd O 13 CD C rd rd -H rd -H 0X0 rd -H CO 5-1 5-1 C CO -P CO g -H O X5 Cn X CO 5-1 CO ft M 5h -H C C co 5h *d -H co CD £ rd CO O CD >i W 3: rd -H CD C ,£ rd M /CD T3 rd rH 5h CD 5h ^ee>-Hos G -H g *H P-i X -d 0) G • 4-1 X or E tion arac m'tfinvor-.oocrita.rHCN n «3< rH H iH H H H rH CN CN CN CN CN 4-4 o xi CD o 4J H ^^ G U -P p o u >iU CD d) rd 5-i £ • %» r^ Q) 54 CO CD CO O C C7> G G O rd • • rd • (DOC ft-H xi 5-1 xi 4-> 5h CO U o •H O CD 4-> rd H-l CO Pi CO CD rtf ,_) 54 • 5-1 5-1 CD CD fd -H en u CD O H 0)IH CD CD £ o e ftfd 4-> -p CO CD rd g -H «d fd m rd c 5h o rd G G 4-> rd •H >i i^ fd CD rd CO 4-> •H £ 4-> 4-) ld Eh rd -H ,* 5-1 4-> 5h > 5-1 rd X CD • o cm X g O U fd rHiCD 4-> CD c ID 1 I 1 W rd c 3 >t co £ CO -n CO g CD Eh « CN 4-> 2 X5 -H 4-> CD|rd Xi C C CD =t+= X u • CO 1 rd rH C £ C H fd c £ CD CD rd CD fd >i s m J CO td h >i c g ro s ft C S)\G £ En i 1 d) CO d) 54 X rS rS 5H G tn •H rH CN ro -^ LT) VD r- oo a\ ^s.rH cN w H rH rH P4 19 previous responses. This option was also available after all the responses were typed. Having signalled to PLATO that he had completed the task, the End interruption subject was then informed which of his responses were in error, and shown the first erron- eous character in each erroneous response. (This was an attempt to even up the End error announcement facility with that of the Immediate group, for whom the erroneous character was boxed.) The lesson then required the subject to retype each incorrect re- sponse, as shown in Figure 2.2-c. 2.4 Data Collected During execution of the tasks the PLATO lesson collected a large amount of data on subject performance. A complete list of data collected is given in Appendix A. The list below contains only those items used to calculate the dependent variables described as performance measures in Section 3.1. Each item given below was collected for each subject on each task. Total Error Keypresses — Total number of erroneous keypresses during 20 * completion of a task. For Immediate interruption this is by definition the number of error interruptions by PLATO. For End interruption this is the number of error presses which were not preceded by uncorrected error presses in the same response. In other words, for the latter group, error checking in a response was suspended whenever an error press occurred, and reactivated if the error was corrected by the subject. This was done be- cause of the difficulty in defining errors in an already erroneous response. Total Time — Time elapsed (in tenths of seconds) from presenta- tion of the task layout page to successful completion of the task. Total Number of Keypresses — Total number of keypresses from beginning to end of a task. Final Answers — The final, corrected set of answers arrived at by each subject on each task. Error Bit Array — A flag array denoting each response in which an error was made. * Does not include special function keys such as super- script or subscript, which were inactive during the experiment, 21 3. RESULTS AND DISCUSSION The aim of the study was to explore the following two basic questions: 1. Does method of error interruption affect stu- dent performance at interactive terminals, and if so, how do these effects show up? 2. What role, if any, do personality traits play in student performance under different error interruption methods? The following sections describe statistical analyses conducted and results obtained. 3.1 Statistical Analysis The two factors considered in the experiment are given below with their respective treatment levels: Factor A--Method of Error Interruption Level 1 — Immediate interruption Level 2 — End interruption Factor B — Task Level 1 — States task Level 2 — Permutations task Levels of each factor are discussed in detail in Section 2.3. Two distinct groups of subjects were used, each 22 being presented with a different method of error interrup- tion. Each subject completed both tasks, however, regardless of group. This led to a Two Factor Split Plot Design (also called Repeated Measures Design) . In this design, levels of one factor (error interruption method, in this case) are divided among subject groups, while all levels of the other factor (task) are presented to subjects in each group. Fig- ure 3.1 shows the design for this experiment. (The split plot design is fully discussed in Kirk [5].) Subject groups were matched on the basis of PLATO experience and typing skill, and the order in which the tasks were presented was randomized. These procedures are described in Section 2.3. Two reasons explain the use of separate subject groups for Immediate and End interruption. First, presenta- tion of both methods of error interruption to one group might have introduced bias due to transfer effects from one method to the other. Second, more tasks would have been necessary and thus the experiment would have become awkward in length. In order to examine subject performance five depen- dent variables were used. These performance measures were derived from raw data collected during the experiment. Each subject produced two scores (one per task) on each measure. The performance measures are given below with computational definitions (see Section 2.4 for raw data relevant to these variables) : 23 Method of Error Interruption States Task Permutations Immediate End g x (35) g-L(35) g 2 (35) g 2 (35) g. and g 2 represent the two subject groups; group size is given in parentheses. Figure 3.1 Two Factor Split Plot Design 24 Percent Error Keypresses Percentage of error keypresses on the task. This was calculated as 100 * total keypresses in error total number of keypresses Total Time (seconds) Total time for task completion converted to seconds Two Consecutive Responses in Error Number of occurrences of two consecutive responses in which the subject made an error. Number of Responses in Error Number of responses in which subject committed an error. Keypress Efficiency (percent) Defined as 100 * number of keypresses needed for error free solution total number of keypresses The numerator of the fraction was calculated from the final corrected set of answers collected for each subject on each task. It represents a "perfect" solution, i.e., the number of keypresses necessary to type the final answer set without making any errors. The denominator of the fraction was collected with raw data. 25 Numerous other variables were derived from the data collected during the experiment, but these five were chosen as the most reliable performance measures. A complete list of derived measures is given in Appendix B. The various statistical analyses conducted on these measures are described next. Some of these analyses were vital, while others were used either to check other results or to insure completeness. The analyses are described below: T-Tests — Group means were compared on typing test time and errors, and on number of years of programming to check that these variables were randomly distributed in the two error interruption groups. One-Way Univariate Analysis of Variance (ANOVA) — This simplest type of analysis of variance was conducted across method of error interruption for each performance measure on each task. Thus, ten such anal- yses were done. Results show whether error interruption method affects any of the performance measures consid- ering each task separately. Two-Way Univariate ANOVA — This analysis was used to examine the combined ef- fects of error interruption method and task on each of 26 the five performance measures taken alone. Five two- way analyses were done, mainly to check for interac- tions between the two factors. Multivariate ANOVA — Multivariate analysis was used to examine the ef- fects of method of error interruption on all five per- formance measures taken together. One such analysis was conducted for each task. In multivariate ANOVA a dis- criminant function is determined such that its applica- tion to the dependent scores results in maximum group differentiation. The coefficients of this function, called discriminant function weights, indicate the de- pendent variables which separate the groups most effec- tively. These analyses serve as a check on univariate results in that a significant multivariate result in- sures that univariate effects are not washed away when the dependent variables are considered together. In addition, multivariate ANOVA determines whether the levels of a factor affect certain combinations of the dependent variables. (More information on this type of analyses can be found in Finn [4].) To examine relationships between personality and performance a two-way multivariate ANOVA was conducted on the personality data collected by the 16 PF Test. 27 Subject performance was estimated as a function of total time and total errors on each task and was sorted into four categories, ranging from excellent to poor. (See Appendix G for details.) Method of error interruption and performance category were then used as factors in the analysis, with the sixteen personality scores on each subject as dependent variables. Correlations — To gain additional insight into personality- performance relationships, Pearson product-moment corre- lations of the performance measures with the personality scores were computed. In these correlations and in the multivariate analysis on personality data, raw scores from the 16 PF Test were used rather than scores stan- dardized by norms produced by the general population. 3.2 Results The T-Tests for differences in the group means of typing test time, typing test errors, and number of years of program- ming are summarized in Appendix C. None of these tests yielded significant results, so it can be assumed that typing skill and programming experience were randomly distributed across error interruption groups. The tests were done as a check on experimental distribution and their lack of signi- ficance lends credibility to further results. 28 Group means and standard deviations for each per- formance measure are shown in Table 3.2-a (States) and 3.2-b (Permutations) . (Summary data for all measures computed is given in Appendix B.) Univariate ANOVA revealed that method of error interruption significantly affected total time and keypress efficiency on both tasks. At a .05 significance level, the End interruption group took longer to complete each task and was less efficient at the keyboard than the Immediate group. These results are not independent, however, due to the high negative correlation between time and key- press efficiency. Tables 3.2-c and 3.2-d contain ANOVA sum- mary tables for significant results on each task. (Refer to Appendix D for summary tables of all ten univariate anal- yses. ) For the Permutations task the direction in the re- sults decisively favors the Immediate interruption group. (The only result on this task favorable to the End group was percent error keypresses, but the difference in group means (0.0 6) was negligible.) On the States task, however, direc- tion of results was mixed. The End group had fewer percent error keypresses and fewer responses in error. Results of the two-way ANOVA supported those of the one-way analyses and turned up no interactions between error interruption method and task. It turned out, however, that task made a difference in subject performance. Although 29 Table 3.2-a Summary of Performance Data for States Task Method of Error Interruption Immediate End Percent Error Keypresses Mean 2.55 1.9 9 St. Dev. 2.45 2.19 * Total Time (seconds) Mean 234.0 300. 3 St. Dev. 94.8 147. 6 Two Consecutive Re- sponses in Error Mean 1.17 1.17 St. Dev. 1.46 1.65 Number of Responses in Error Mean 4.29 3.77 St. Dev. 3.11 3.17 Keypress Efficiency (percent) Mean 90.94 84.91 St. Dev. 8.06 13.11 Based on sample size of 35 per group. Significant at the .05 level. 30 Table 3.2-b Summary of Performance Data for Permutations Task Method of Error Interruption Immediate End Percent Error Keypresses Mean 4.54 4.48 St. Dev. 2.21 3.4 5 * Total Time (seconds) Mean 408.8 546.4 St. Dev. 174.9 232.4 Two Consecutive Re- sponses in Error Mean 1.09 2.77 St. Dev. 1.40 5.03 Number of Responses in Error Mean 4.4 6 4.83 St. Dev. 2.14 5.10 * Keypress Efficiency (percent) Mean 83.09 75.57 St. Dev. 9.50 18.34 Based on sample size of 35 per group, Significant at a .05 level. 31 Table 3.2-c Analysis of Variance Summary Tables for States Task Total Time (seconds) Source of Variance Sum of Squares Degrees of Freedom Mean Square F Ratio ME I Error 76778.730 1076415.2 1 68 76778.730 15829.636 4.850 p < .05 Keypress Eff iciency (per* cent) Source of Variance Sum of Squares Degrees of Freedom Mean Square F Ratio ME I Error 636.014 8290.629 1 68 636.014 121.921 5.217 p < .05 i Method of Error Interruption. 32 Table 3.2-d Analysis of Variance Summary Tables for Permutations Task Total T ime (seconds) Source of Variance Sum of Squares Degrees of Freedom Mean Square F Ratio ME I Error 331437.00 2960801.3 1 68 331437.00 43541.196 7.612 p < .05 Keypress Eff iciency (perc :ent) Source of Variance Sum of Squares Degrees of Freedom Mean Square F Ratio ME I Error 988.129 14937.314 1 68 988.129 219.666 4.498 p < .05 Method of Error Interruption. 33 effects of task are not relevant to the original research questions, it is interesting to note that the Permutations task took longer to complete and caused a higher percentage of error keypresses and lower keypress efficiency than did the States task. (Level of significance was .05; see Appen- dix E for complete summary tables.) Multivariate ANOVA on the performance measures re- vealed significant effects of error interruption method on combined performance measures in each task. This backs up the isolated findings of univariate analyses. Discriminant function weights are shown in Tables 3.2-e and 3.2-f. The standardized weights are computed from the raw coefficients, taking into account the different standard deviations of the dependent variables. The signs of the discriminant weights generally reflect the directions in performance on the two tasks, while their absolute values indicate which variables are most effective in separating the groups. On the States task, number of responses in error and keypress efficiency seem to be the important variables, while on the Permutations task these two along with two consecutive responses in error are singled out. It might seem that total time, which was signi- ficant in univariate tests, has vanished from importance, but actually it was negatively correlated with keypress 34 Table 3.2-e Multivariate ANOVA — States Task Based on 5 and 64 degrees of freedom, the F ratio was cal- culated as 5.3861 (p < .05). Discriminant Function Coefficients Variable Raw Coefficient Standardized Pet. Error Keypresses -0.293085 -0.6915 Total Time (seconds) 0.001590 0.2000 Two Consecutive Re- sponses in Error -0.095394 -0.1508 Number of Responses in Error -0.273706 -0.8727 Keypress Efficiency (pet.) -0.143781 -1.5876 Table 3.2-f Multivariate ANOVA — Permutations Task Based on 5 and 64 degrees of freedom, the F ratio was cal- culated as 13.0327 (p < .05). Discriminant Function Coefficients Variable Raw Coefficient Standardized Pet. Error Keypresses -0.035850 -0.1054 Total Time (seconds) 0.002440 0.5091 Two Consecutive Re- sponses in Error 0.766626 2.8728 Number of Responses in Error -1.079513 -4.2855 Keypress Efficiency (pet.) -0.101125 -1.4988 35 efficiency, so its effect is mostly described by the discri- minant weight for the latter. The discriminant functions indicate that method of error interruption has an effect on a linear combination of the variables with high weights. This effect is stronger than that on some of the measures alone, as in the case of number of responses in error and two consecutive responses in error, for which univariate results yield no significant differences. Multivariate ANOVA conducted on personality traits produced no significant differences — neither on error inter- ruption method or on performance category. Nor were there any interactions between these two factors. Correlations between personality traits and perform- ance scores did, however, produce some significant results. Four correlation matrices were produced based on the follow- ing breakdown : Immediate interruption — States task (sample size = 31) Immediate interruption — Permutations task (sample size = 31) End interruption--States task (sample size = 30) End interruption--Permutations task (sample size = 30) Sample sizes used were not the group sample sizes due to nine subjects who either could not take the 16 PF Test, or refused to place identification on their answer sheets. 36 Major significant correlations are reported in Tables 3.2-g and 3.2-h. (See Appendix H for brief descrip- tions of personality traits involved.) On the States task, more intelligent subjects in both groups performed better. Immediate interruption subjects who were more tense and End interruption subjects who were less emotionally stable also did better. Possible explanations are presented in the next section. On the Permutations task only shrewdness and intel- ligence remained important to good performance in the Immed- iate and End groups, respectively. In all correlation results, although some of the coefficients shown in Tables 3.2-g and 3.2-h are not signi- ficant at a «05 level, direction of these personality- performance correlations is always consistent. 3.3 Discussion Analysis of the performance data indicates that the Immediate interruption group did a better job than the End group. The latter group was significantly slower and less efficient (in terms of keypresses) on each task. There are several probable reasons for these re- sults. First, the instant feedback built into Immediate error interruption may have given that group an advantage by 37 Table 3.2-g Major Significant Correlations between Personality Traits and Performance Measures — States Task Sample Size = 31 _, , . Performance Measures Personality Factor 12 3 4 Immediate Error Interruption B: Intelligence -0.36330* -0.28256 -0.26581 -0.22648 0.32929 Q.: Tension -0.39430* -0.34104 -0 . 53568*-0 . 40673* 0.43559 4 End Error Interruption B: Intelligence -0.33980 -0.46204* -0.22813 -0.25229 0.46954' C: Emotional * * * Stability 0.40539 0.31506 0.42508 0.43151 -0.31536 Performance Measures 1. Percent Error Keypresses 2. Total Time (seconds) 3. Two Consecutive Responses in Error 4. Number of Responses in Error 5. Keypress Efficiency (percent) * Significant at a .05 level; Critical value (from Ferguson [3]), with 29 degrees of freedom, was .355. 38 Table 3.2-h Major Significant Correlations between Personality Traits and Performance Measures — Permutations Task Sample Size = 30 _, , . . Performance Measures Personality Factor Immediate Error Interruption N: Shrewdness -0.44503* -0.29648 -0.52688 -0.53060 0.44530 End Error Interruption B: Intelligence -0.24363 -0.56538* -0.38138 -0.42832 0.32757 Performance Measures 1. Percent Error Keypresses 2. Total Time (seconds) 3. Two Consecutive Responses in Error 4. Number of Responses in Error 5. Keypress Efficiency (percent) * Significant at a .05 level; critical value (from Ferguson [3]), with 28 degrees of freedom, was .361. 39 allowing the subjects to catch their errors early and avoid making the same type of mistake more than once. In fact, several End interruption subjects made one consistent error on the Permutations task, such as omitting the letter ' e' in each response. These subjects did not learn of their mis- takes until they completed their responses, and thus produced data reflecting a very poor performance. Immediate interrup- tion subjects under a similar misconception would have been made aware of it at the very first error. Also, in terms of keypresses, error correction in the Immediate group required the retyping of one character whereas an error in the End group required that the whole response be retyped at the task's end. This could partially explain the lower keypress efficiency of the End group. A second reason for the better performance of the Immediate group could be that the End error interruption method allowed for more individual differences among subjects to surface. As shown in Tables 3.2-a and 3.2-b, the stan- dard deviations of scores in the End group are consistently higher than those for the Immediate group. This suggests that the rather free nature of End interruption causes it to be quite sensitive to individual differences. The resulting rise in group variability may have washed out some meaningful effects of the different types of error interruption. 40 Considering the mixture of directions in performance measures on the States task, it is quite possible that the task itself is the key to determining which error interrup- tion method is best. The Permutations task, the more diffi- cult and mathematical of the two, may have been ripe for Immediate interruption, whereas the States task, more related to individual experience, produced some good performance under End interruption. (As mentioned in the previous section, multivariate ANOVA indicated that error interruption method affected a combination of certain performance measures. However, since no meaningful interpretation was found for these combined ef- fects it is unlikely that these multivariate findings can shed additional light on the present discussion.) Correlations of personality traits with performance measures also support the importance of task as a factor. On the States task, Immediate group subjects who were more tense performed better. Possibly the instant feedback provided by Immediate interruption helped to reassure the more fretful subjects. In the End group, lower emotional stability led to better performance. According to the Handbook for the 16 PF [2] , high tension and low emotional stability are sometimes difficult to distinguish, so possibly the same basic traits are reflected by both measures. In fact for the Immediate group, tension and emotional stability were negatively 41 correlated. Motivation may explain some of this. Tense and excited subjects may have been concerned about the possibil- ity of making errors and therefore may have done better than subjects who were less concerned with their performance. The Handbook for the 16 PF [2] points out that ". . .in simple performances or in trial-and-error performances, where high activity level is important, performance has been found posi- tively related to Q. [Tension]." The "high activity level" might be the result of an excitable subject faced with a test-like situation at a computer terminal. Fewer significant correlations occurred on the Permutations task. For the Immediate group, more shrewdness implied better performance, while in the End group intelli- gence was the dominant factor. Based on their definitions these two factors are quite similar, so once again the same qualities are affecting performance in both error interrup- tion groups. It is dangerous, of course, to assume that the 16 PF Test accurately measures intelligence — whole tests have been designed to grapple with this trait alone. It does seem reasonable, however, since intelligence correlations remain significant in both tasks, to claim that more intelligent subjects do better at interactive terminals under any condi- tions. It is notable that many significant correlations on the States task disappear on the Permutations task. Only 42 intelligence/shrewdness seems to remain important in the latter task. Listing permutations gave the subjects con- siderably more difficulty than did listing states and so once again the tasks seem to make the difference. The lack of personality — performance correlations on the Permutations task implies that in more difficult tasks, personality fac- tors (with the exception of intelligence) play little role in subject performance at the terminal. Although Immediate error interruption was favored in this study, further experimentation in this area would probably be useful. For example, effects of error interrup- tion on programming might be tackled in a semester long study which would compare two classes learning to program through the use of interactive terminals. All in all, studies such as ours, which examine a part of man-computer interaction, are both feasible and use- ful. A wide range of data can be collected by the computer during such an experiment, and the system can provide accurate control of many variables. The university is an excellent place for such research because of the availability of sub- jects, equipment, and statistical services. 43 4 . SUMMARY If a student makes an error while typing material into an interactive computer terminal, the system can respond to the error in several ways, ranging from immediate inter- ruption of the student's work to delay of the interruption until the student decides that his work is finished. These two methods of error interruption — Immediate and End — were studied to determine whether the method used affects perform- ance. It was hypothesized that Immediate interruption would damage performance by jarring the student's train of through. The relationship of human personality to performance was also examined. The PLATO IV system was used to implement the major part of the experiment, with subjects chosen from basic com- puter science courses. The experimental procedure consisted of two phases: First, subjects were given the 16 PF Test, a personality test developed at the University of Illinois at Urbana- Champaign, which measures sixteen basic personality traits. Next, subjects took a PLATO lesson which asked them to com- plete two simple tasks: 1. List 25 of the 50 states. 2. List 20 permutations of 'abcde' such that the 'c' occurs somewhere before the 'a' and the 'b' occurs somewhere before the *d'. 44 Half the subjects experienced Immediate interruption when making an error, while the other half experienced End inter- ruption. PLATO experience and typing skill were taken into consideration by the lesson both in assigning subjects to treatment groups and in randomizing the order in which the tasks were presented. Various comprehensive data were collected on each subject during the PLATO phase of the procedure, and five dependent variables were chosen as most indicative of per- formance. Each subject produced two scores (one for each task) on each of the following variables: Percent Error Keypresses Total Time (seconds) Two Consecutive Responses in Error Number of Responses in Error Keypress Efficiency (percent) Significant differences appeared in the multivariate analysis of variance on the dependent variables, and the discriminant function weights generated verify results on the univariate analyses. Two variables showed significant dif- ferences in simple analysis of variance — total time and key- press efficiency. The direction in both of these favored the Immediate interruption group, i.e., they took significantly less time to complete each task and were significantly more efficient in their use of the keyboard (at a .05 level). It 45 should be noted, however, that these results were not indepen- dent, since the two variables were highly correlated in a negative direction. Although these results imply that the Immediate interruption group performed better, the direction of overall results was mixed. For example, on both tasks, the End group had a smaller percentage of error keypresses, and on the States task, this group had fewer responses in error. In general, results on the Permutations task favored the Immed- iate group, while results on the States task were mixed. The multivariate ANOVA on personality scores dis- played no significant differences, neither across error in- terruption method nor across performance categories. Nor did any interactions turn up in this analysis. There were, however, significant correlations be- tween personality measures and the five performance measures mentioned above. Results indicated that in the States task tension and instability, along with intelligence, led to better performance. Calmer, more stable subjects may have been less motivated to do well than more excitable subjects who, when faced with a test-like situation, were more keyed up and alert. On the Permutations task, however, most per- sonality relationships disappeared. Only intelligence re- mained important to performance, as it had also been in the States task. Apparently, as the task becomes more difficult, 46 basic intelligence becomes the more dominant performance- affecting trait. In conclusion, the superior performance of the Immediate interruption group indicates that this form of interruption has no adverse affect on performance, and in fact may have had some beneficial effect. In addition, per- sonality seems to affect subject performance at a computer terminal under certain conditions. Results as a whole, however, imply that different tasks and different personalities require different methods of error interruption in order to maximize effectiveness at the terminal. Therefore, further studies to examine the ef- fects of error interruption method on more complicated tasks such as programming at the terminal are recommended. 47 REFERENCES [1] Alpert, D. and Bitzer, D. L. , "Advances in Computer Based Education," Science , Vol. 167 (20 March 1970) , pp. 1582-1590. [2] Cattell, Raymond B., Eber, Herbert W. , and Tatsuoka, Maurice M. , Handbook for the 16 PF (Sixteen Personality Factor Questionnaire ) , Institute for Personality and Ability Testing, Illinois, 1970. [3] Ferguson, George A., Statistical Analysis in Psychology and Education , McGraw Hill, Inc., New York, 1971. [4] Finn, Jeremy, A General Model for Multivariate Analysis , Holt, Rinehart, and Winston, Inc., New York, 1974. [5] Kirk, Roger E. , Experimental Design Procedures for the Behavioral Sciences , Wadsworth Publishing Co., California, .. 1968. [6] Manual for the 16 PF Test , Institute for Personality and Ability Testing, Illinois, 1972. [7] Martin, James Thomas, Design of Man-Computer Dialogues , Prentice-Hall, New Jersey, 1973. [8] Nickerson, R. S., "Man-Computer Interaction: A Chal- lenge for Human Factors Research," IEEE Transactions o n Man Machine Systems , December 1969, Vol. M. M.S. -10, No. 4, pp. 164-180. [9] Wilcox, T. R. , "The Interactive Compiler as a Consultant in the Computer Aided Instruction of Programming," Proceedings of the Seventh Annual Princeton Conference in Information Sciences and Systems, March 1973. 48 Appendix A RAW DATA COLLECTED BY PLATO LESSON The following is a description of all the data collected by PLATO during the experiment. Of the data items listed, only total time (scaled to seconds) was used directly as a dependent variable. The other items were either used to derive other measures or were biased toward one of the error interruption groups. The term "initial entry" is used to describe the period during which subjects first typed their responses. "Answer correction" denotes the period dur- ing which End subjects were allowed to change answers after initial entry. "System guided correction" refers to the period after answer correction during which End subjects were guided through corrections by the system. Task Data (one score per subject per task) See Tables A-l and A-2 for summaries of results on these items. Total Error Keypresses: Total number of error keypresses made on the task (in End interruption, error checking in a response was suspended at the occurrence of an error, until that error was corrected or the response completed. 49 Table A-l Summary of Raw Task Data — States Task Method of Error Interruption Immediate End Total Error Keypresses Mean 7.00 6.37 St. Dev. 7.38 7.76 Min.* Max.* 35 37 Total Time (sec.) Mean 234.0 300.3 St. Dev. 94.8 147.6 Min. 126.7 119.3 Max. 525.1 738.2 Total Erase Presses Mean 9.31 10.03 St. Dev. 9.65 10.27 Min. Max. 39 43 Total Erase Operations Mean 7.69 4.51 St. Dev. 7.93 4.03 Min. Max. 37 16 Total Keypresses Mean 255.94 281.83 St. Dev. 23.86 54.90 Min. 217 216 Max. 319 44 8 50 Table A-l (continued) Method of Error Interruption Immediate End Number of Fooling-Around Times Mean 0.20 0.00 St. Dev. 0.71 0.00 Min Max. 4 Total Editing Time (sec.) Mean 0.0 44.5 St. Dev. 0.0 74.8 Min. 0.0 0.0 Max. 0.0 318.3 Number of Times Edit Occurred Mean 0.0 0.69 St. Dev. 0.0 1.17 Min. Max. 5 * Min. is minimum score; Max. is maximum score. 51 Table A-2 Summary of Raw Task Data — Permutations Task Method of Error Interruption Immediate End Total Error Keypresses Mean 7.00 9.31 St. Dev. 4.38 9.56 Min.* 2 Max.* 2 3 3 Total Time (sec.) Mean 408.8 546.4 St. Dev. 174.9 232.4 Min. 168.4 194.7 Max. 911.4 1212.6 Total Erase Presses Mean 10.91 6.51 St. Dev. 8.74 6.58 Min . 3 Max. 46 33 Total Erase Operations Mean 8.34 4.00 St. Dev. 5.36 4.13 Min. 3 Max. 27 20 Total Keypresses Mean 146.77 173.91 St. Dev. 21.19 59.89 Min. 127 122 Max. 2 36 36 7 52 Table A-2 (continued) Method of Error Interruption Immediate End Number of Fooling-Around Times Mean 0.14 0.00 St. Dev. 0.35 0.00 Min. Max. 1 Total Editing Time (sec.) Mean 0.0 69.7 St. Dev. 0.0 147.9 Min. 0.0 0.0 Max. 0.0 708.6 Number of Times Edit Occurred Mean 0.0 2.77 St. Dev. 0.0 4.09 Min. Max. 20 * Min. is minimum score; Max. is maximum score. 53 Total Time (tenths of seconds) : Time from presentation of task layout screen to successful completion of task. Total Erase Presses: Total number of keypresses made to erase char- acters . Total Erase Operations: One erase operation equals N consecutive erase presses (N being an integer greater than zero) . Total Keypresses: Total number of keypresses made during task, excluding presses of special function keys which were not active. Number of Fooling-Around Times: Fooling-around was defined as making three con- secutive error keypresses at the same position in the response. The purpose of this measure was to determine whether Immediate interruption subjects were using trial and error to figure out which characters the lesson would accept. Total Editing Time (tenths of seconds) : (End group only) total time from completion of responses to successful completion of task. Number of Edit Occurrences: (End group only) number of times subject 54 changed a previously typed response during initial entry and before system guided correc- tion. Bit Arrays (one bit per response per subject per task) (Note: The first six arrays apply only to the initial entry period. ) Error Correction: For the Immediate group — occurrence of an interruption by the lesson. For the End group — erasure of an erroneous character. Multiple Error Correction: More than one occurrence of an error correction as defined above. Unnecessary Erasure: Erasure of a correct character. Multiple Unnecessary Erasures: More than one occurrence of an unnecessary erasure. Error Occurrence: Occurrence of an error in the response. Interruption of a Previous Response: (End group only) interruption of the current response to change a previously typed response. 55 Such interruptions could only occur at the very- beginning of the current response. Wrong at Finish: (End group only) response was wrong when sub- ject signalled completion of task. Edit Occurrence: (End group only) response was later changed. Edit Error Occurrence: (End group only) error occurred while response was being changed. Other Multiple Data Response Times (tenths of seconds) : Times for each response of each subject on each task. Does not include changing of answers by End group after responses were typed. Final Answers: Complete list of answers arrived at by each subject on each task. 56 Appendix B MEASURES DERIVED FROM RAW DATA Various measures were derived from the raw data given in Appendix A. Summaries of the derived task data are given in Tables B-l and B-2. Definitions of the derived data follow: Interruption Bit Array (one bit per response per subject per task) Denotes whether the response was interrupted in any way. Computed as the logical OR of elements in the error correction/ unnecessary erasure, and interruption of previous response arrays. Task Data (one score per subject per task) Number of Responses Interrupted: Computed from interruption bit array. Number of Responses in Error: Computed from error bit array. Two Consecutive Responses Interrupted: Computed from interruption bit array. Two Consecutive Responses in Error: Computed from error bit array. 57 Table B-l Summary of Derived Task Data — States Task Method of Error Interruption Immediate End Number of Responses Interrupted Mean 4.40 3.43 St. Dev. 3.12 2.32 Min.* Max.* .12 9 Two Consecutive Responses Interrupted Mean 1.2 3 0.80 St. Dev. 1.48 1.04 Min. Max. 5 4 Number of Responses in Error Mean 4.29 3.77 St. Dev. 3.11 3.17 Min. Max. 12 12 Two Consecutive Responses in Error Mean 1.17 1.17 St. Dev. 1.46 1.65 Min. Max. 5 6 Background Time (sec. ) Mean 7.5 7.9 St. Dev. 2.3 2.1 Min 4.5 3.4 Max. 13.8 11.9 58 Table B-l (continued) Method of Error Interruption Immediate End Damage Time (sec. ) Mean St. Dev. Min. Max. 10. 5 4.8 3.6 24.5 10.7 5.8 2.9 32.5 Interruption Damage Mean St. Dev. Min. Max. 39.8 48.5 -37.9 217.8 35.8 73.2 -40.2 322.1 Number of Unnecessary Erasures Mean St. Dev. Min. Max. 1.11 1.21 4 3.14 2.23 9 Perfect Number of Keypresses Mean 2 31.09 St. Dev. 11.33 Min. 209 Max. 260 232.54 11.84 211 257 Keypress Efficiency (percent) Mean St. Dev. Min. Max. 90.94 8.06 70 100 84.91 13.11 55 100 Percent Error Keypresses (NR) Mean St. Dev. Min. Max. 28.00 29.53 140 25.49 31.03 148 59 Table B-l (continued) Method of Error Interruption Immediate End Percent Error Keypresses (PNK) Mean 3.07 2.70 St. Dev. 3.32 3.21 Min. 0.00 0.00 Max. 16.05 14.85 Percent Error Keypresses (NKP) Mean 2.55 1.99 St. Dev. 2.45 2.19 Min. 0.00 0.00 Max. 11.62 11.11 * Min. is minimum score; Max. is maximum score. 60 Table B-2 Summary of Derived Task Data — Permutations Task Method of Error Interruption Immediate End Number of Responses Interrupted Mean 5.00 3.17 St. Dev. 2.23 2.16 Min.* 1 Max.* 12 11 Two Consecutive Responses Interrupted Mean 1.29 0.69 St. Dev. 1.58 1.39 Min. Max. 6 7 Number of Responses in Error Mean 4.46 4.83 St. Dev. 2.14 5.10 Min. 1 Max. 11 20 Two Consecutive Responses in Error Mean 1.09 2.77 St. Dev. 1.40 5.03 Min. Max. 5 19 Background Time (sec. ) Mean 14.0 15.0 St. Dev. 6.4 4.6 Min. 5.1 6.3 Max. 41.7 25.8 61 Table B-2 (continued) Method of Error Interruption Immediate End Damage Time (sec. ) Mean St. Dev. Min. Max. 21.0 11.0 6.6 52.6 25.2 18.0 3.7 74.9 Interruption Damage Mean St. Dev. Min. Max. 64.5 84.4 -61.2 329.6 71. 3 110.3 -60.6 406.1 Number of Unnecessary Erasures Mean 2 . 14 St. Dev. 2.07 Min. Max. 9 2.23 1.76 9 Perfect Number of Keypresses Mean 120.00 St. Dev. 0.00 Min. 120 Max. 120 121.00 0.00 121 121 Keypress Efficiency (percent) Mean 83.09 St. Dev. 9.50 Min. 51 Max. 94 Percent Error Keypresses (NR) Mean St. Dev, Min. Max. 35.00 21.91 10.00 100.00 75.57 18.34 37 99 46.57 47.78 0.00 165.00 62 Table B-2 (continued) Method of Error Interruption Immediate End Percent Error Keypresses (PNK) Mean 5.83 7.6 9 St. Dev. 3.65 7.90 Min. 1.66 0.00 Max. 16.66 27.27 Percent Error Keypresses (NKP) Mean 4.54 4.4 8 St. Dev. 2.21 3.45 Min. 1.48 0.00 Max. 10.61 11.95 * Min. is minimum score; Max. is maximum score. 63 Background Time (tenths of seconds) : Average of all times for responses which were the third of three consecutive uninterrupted responses (excepting the first three re- sponses) . This represents an average response time during smooth response flow, when no re- cent interruptions have occurred. The first three responses were not considered part of the normal flow. (The counts used to compute these averages were also collected.) Damage Time (tenths of seconds) : Average of all times for responses (excepting the first three) which follow interrupted responses. This measures the gross effect an interruption has on the following response. (Counts used to compute averages were also collected. ) Interruption Damage: Computed as inn * Da ma 9 e Time - Background Time % Background Time measures the net damage to speed caused by interruptions . Number of Unnecessary Erasures: Computed from unnecessary erasure bit array. 64 Perfect Number of Keypresses: Number of keypresses necessary to produce subject ' s final task answers without making any errors . Keypress Efficiency (percent) : Calculated as inn * P er f ect Number of Keypresses Total Number of Keypresses Percent Error Keypresses (NR) : Percent error keypresses based on number of responses; calculated as inn * T °tal Error Keypresses Number of Responses Percent Error Keypresses (PNK) : Percent of error keypresses based on perfect number of keypresses; calculated as inn * Total Error Keypresses Perfect number of Keypresses* Percent Error Keypresses (NKP) : Percent of error keypresses based on number of keypresses; calculated as inn * Total Error Keypresses Total number of Keypresses (This measure was used as a dependent variable.) 65 Appendix C T-TEST RESULTS T-Tests were performed to determine whether group means on typing test time, typing test errors, and number of years of programming were significant. Results of these tests are shown below. The critical value of T at a .05 level of significance for the two-tailed test is 1.6676, taken from Ferguson [3]. Sample size for each group was 35, degrees of freedom was 68. i Typing Test Time (.1 sec.) Group Immediate End Mean 941.5 923.7 St. Dev. 277.5 266.1 Value of T was 0.001 (not significant), p < . 05 Typing Test Errors Group Immediate End Mean 2.46 2.20 St. Dev. 1.99 1.67 Value of T was 0.318 (not significant) p < . 05 Number of Years of Programming 66 Group Immediate End Mean 0.77 0.83 St. Dev. 1.20 1.52 Value of T was 0.183 (not significant), p < .05 67 Appendix D ONE-WAY UNIVARIATE ANOVA SUMMARY TABLES Results of ANOVA on each performance measure for each task appear on the following pages. States task re- sults are given in the first five tables, followed by Permutations task results in the last five. (MEI stands for Method of Error Interruption.) 68 Analysis of Variance Summary Tables Percent Error Keypresses — States Source of Sum of Variance Squares DF Mean Square F Ratio Probability MEI 5.601 1 5.601 1.006 0.319 Error 378.577 68 5.567 Total Time (seconds) — States Source of Sum of Variance Squares DF Mean Square F Ratio Probability MEI 76778.730 1 76778.730 4.850 0.031 Error 1076415.200 68 15829.636 Two Consecutive Responses in Error—States Source of Sum of Variance Squares DF Mean Square F Ratio Probability MEI 0.000 1 0.000 0.000 1.000 Error 169.943 68 2.499 Number of Responses in Error — States Source of Sum of Variance Squares DF Mean Square F Ratio Probability MEI 4.629 1 4.629 0.455 0.502 Error 691.314 68 10.166 69 Analysis of Variance Summary Tables (continued) Keypress Efficiency (percent) — States Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 636.014 1 636.014 5.217 0.026 Error 8290.629 68 121.921 Percent Error Keypresses — Permutations Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 0.061 1 0.061 0.007 0.934 Error 587.777 68 8.644 Total Time (seconds) — Permutations Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 331437.000 1 331437.000 7.612 0.007 Error 2960801.300 68 43541.196 Two Consecutive Responses in Error — Permutations Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 49.729 1 49.729 3.541 0.064 Error 954.914 68 14.043 70 Analysis of Variance Summary Tables (continued) Number of Responses in Error — Permutations Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 2.414 1 2.414 0.153 0.697 Error 1071.657 68 15.760 Keypress Efficiency (percent)- — Permutations Source of Sum of Variance Square DF Mean Square F Ratio Probability MEI 988.129 1 988.129 4.498 0.038 Error 14937.314 68 219.666 71 Appendix E TWO-WAY UNIVARIATE ANOVA SUMMARY TABLES Results of two-way analyses are shown in the fol- lowing pages. The summary table is given for each performance measure. The Geisser-Greenhouse Conservative F Test shown with each summary table is a check for homogeneity of co- variance in the data. The test made no difference to the results shown here. P >1 CD CO p CO CD •H 00 • • 5-1 -H u o o O -p CM 1 rd c M > CD M •H CO CD P CO CO a •H C • • t 3 CD Pm rH rH 00 m O CJ Q <^> m i c P H -H ■H o 00 CM 00 5-1 •H <3< o O O XI in o ■^ h fd .h O «£> 5-1 XI in O in w u • o • o • o <4H cm o T3 cm in 00 O 00 •H -=!< CO "ST O CM XJ ^ P <* o CO OO P co 1 U rH CM 00 in 00 Pn~ CD fd rr r- co CM •<* q m P P w 3 — * O U O 1 co r^ in CM *x> 'd fd fd 5-1 CO r» CD P Cm Ph 1 1 5-1 5-t fd rH -H U HH CD •HM-I CD W CD TJM-I u a CD e 4-> e (C C c •H CD Pn rH 00 rH rH 00 a\ 5-l-H 5-1 O p V£> *£> CO fd fd 5-1 rH «w e > CD 04 C 5-1 m rHIW 00 r-> 00 00 C^ o •H co M-) CO co o rH 00 r- co CO >i •H O CD rH CM 00 in CM o CLP (0 u ■^ CM CO CM rH «* H-H >1 co rd rH rH g 3 co in in CM rH r> CD-H fd 3 D 1 CN i> ^P «;r CO XI 3 CO CO m rH «tf rH 3 td rH 0X1 X! O CD c u X! CD CM p O CD 5^ mh -H O r< CQ P U 1 5-1 >1 fd CD u p P u CO m M-l u U CD CO Ej CD ^-^ CD p ^-* •H P CD «w < M-) a PQ CD 3 CO U HH m H *-* O CO CD C W w u fd u m 5-1 H U -H G c fd 3 5h •H u •H X U p o rd fd u fd U c/ ) > a w a < W Eh 72 -P >1 0) Cfl -p co a) •H rH o 3 Eh rH o r- •H o r- •C Cm ja o H c (0 o CT> a) oj XJ o rH qj > • • P -H P o O o -p 04 i fa c p > CD P •H Cfl CD -p Cfl CO a •H C • • • p a) o CM rH rH 00 P o u Q >X> P. a) •P >1 C -p H •H rH o i-H O O P •H CT> O r» XI H O rH p (0 O O p J3 O O rH w O u • o • o • O M-l CM o T3 CN ■^ 00 TJ 03 fO E C/> o ID en CN rH QJ -P -P Cm Cm •H \Q ro m *£> o •H U U 1 En c m m o •^ *X> M-l QJ QJ 1 (d ro m ■^T CN •H M-l M-l d) rH CD ro in T3 M-l M-l o ig 2 rH QJ QJ e p 6 03 c c •H Eh Cm rH 00 rH rH 00 X> <£> ro C P P M-l o o o O O O o o O O O O rH MH M-l o o O O O O •H w M -1 CO o o o O O o W >i >i •H Q.P -P co P o ■T KO ^1" 00 CN W -H -H >i CO o o m CN ^J" rH rH rH iH \ ^ X) a w vo U3 in ■^ V£> CTi D CO (T3 S m CN CN in rH rH m X» -Q ^ o 0) C P P 5 QJ Cm Cm -p G o QJ P M-l •H o s W s < w Eh 73 +J >1 cu co -p CO cu •H r>- CN 3 Eh rH C^ ro •H o m X! fa £> m ro G fd • • U -H 5-1 o o U -P fa G i m u > •H CD u -P CO CU a CO CO 3 •H G • • • 5-) CU o fa t-\ rH CO 5h o u Q >£> Q) ■4-1 G >i H 4-> ■H ^ r>- CN 5-1 rH vo (T» ro O •H o o m 5-1 X) «* in ro u fd o o> <^> w rH • o • O • IW 5h o o o fa -d 5-1 o o -G M 5-1 •H ro 00 00 o -P to 5-1 -P in CT\ vo o CD fd W td <£> r^ "sf o S El c •H fa fa CN CM ro o o • • # •• rH <3 CO CO (1) CO II 5-1 H CO CU c u fa +J +J fd ro a\ cu T3 G -H £ fd -P e ■H 3 n u fa H 00 rH rH 00 u fd Q) Q VD *£> ro > (0 G rH <4-l m c U rH to ro CN in ro o •H ■H £ n -1 CO "* LD ^ ^ rH o CO to EH 1 D CU ID 00 VD W3 r>> in CU >1 5-1 00 CN O 00 in VD w rH a fd fd 3 P •^ r- O ^ r» «* 0) c 3 tr CN ro CN ■^r rH rH 3 CU £ XI c •P g CU CU m 5-i ■H ■p 1 >i *Jj fa u 5-1 5-1 fd CU to «■ M -P U -P u 5-1 CU CO CO g CU ^^ CU -p *~s •H 3 CU 4-1 < m c m CU co w u a) g fd u H u rH o U -H c g o id d n ■H 5-1 ■H X u ■P i o fd fd 5-1 fd u c o > s W a < w Eh I 74 -p >i 0) w -P CO CD •H 10 00 o e rd r^ <0 CD CD XI (N m CD > O • • P -H P o o O -P Cm fi 1 f0 P > •H CD P -P CO CD a CO CO 3 •H C • • • p CD Cm rH rH 00 p O U a <0 CD -P C >i H -P -H ro 00 o P cd o r^ 10 W Xi CTl • in • M-l p o o o O fn TJ O x: ^ •H CN "tf o o -P co P -P ■H CP> CN o cd (d id O rH >£> o g EH P en • • • o P o r-\ o o W Cm • • • •• rH < OQ ■H CO II P P CD O CO p Cm 4-> -P cd id LD en o in rH a o u 10 3 00 o r-~ ^ in 03 f0 c o 1 r* vo o vo <0> T3 Cm Cm CO rH CO CN 00 o CD 1 a •H 1 CO c O •^r ro ^ rH iW CD CD id <-\ rH rH •H U on CD T) C g id m-i 6 ■H • • • • • • P Cm rH oo <-\ rH 00 a\ P rd p a CD rH MH M-l "§ C z rH CO IT) o o in in O ■H •H MH CO 00 (N [*» ■^r CO r-\ CO CO O CD r^- •^r o vo CN CN Cm >1 P rH LD CN 00 •^ CN W rH (O cfl • • • • • • (0 i 3 o o ro >0 CN ro CD 5 3 o 1 rH rH in 00 CO CO CO O f» r* 3 rH rH a) x: A e -P c CD CD M-l P ■H -p 1 >i < CQ u P p id CD cd p -P p CO P mh u O CD CO P o CD ^-v CD -P •H 3 CD M-l 6 *P c CD co CO u CD C m-i W M-l W M O O g w g >1 CD CO -p CO CD •H co CO 3 Eh ■H CO CM -H rH O XI Cm XI o CO G rd o rH CD CD X! o r^ CD > , P -H P o o O P a, c 1 rd P > •H CD P -p CO CD Cu CO CO p ■H fi • • • p CD Cm rH rH 00 u CD -P O u P \o C >i H -P •H CM co CO P rH O CO CM •H co rH O - - ■ P XI vo o CO P rd o o rH H XI O o • o t • M-l P o o o Ck T) CO CM CO X! ,* --^ ■H in CO CO OlOCO 4-> CO -p -P r^ co co CTi CO rH CD rd c fd (T\ ^r rH o^ *x> o S E-t . P o P -H -H rd > CD Q 1 GPP CD i< O O o m o O HPIH 10 r^ O r^ CM o o -H ■H 4-1 CO CO O co rH in CO W >i >i CO O CD i P W -H -H H CO fd ■^f CM CO CJ\ ID o rH rH id s p o CO CO rH •^ ^ CJ -H -H s p D 1 VD kO in in "^ CO XI XI co CO rH co CM I >1 < CQ O P p fd CD rd 4-1 -p u 4-> u P CD CQ CO c CD ^■^ CD 4-> *■— *% •H 3 CD 4-1 6 iw C m CD co CO O 4-1 4-1 H O CD c Cm" W U fd P CQ p rH P •H C c rd P p •H P •H X p -P O fd fd P rd p o to > a cq £ < w Eh 76 77 Appendix F CORRELATION MATRICES Four complete correlation matrices of performance measures and personality traits are given here: Immediate interruption — States task Immediate interruption — Permutations task End interruption — States task End interruption — Permutations task The variables are numbered as follows: Variable Number Meaning (one through five are performance measures) 1 Percent Error Keypresses 2 Total Time (seconds) 3 Two Consecutive Responses in Error 4 Number of Responses in Error 5 Keypress Efficiency (percent) * (six through twenty-one are the 16 PF traits) 6 Reserved vs. Outgoing 7 Less Intelligent vs . More Intelligent 8 Affected by Feelings vs. Emotionally Stable 9 Humble vs. Assertive * More detailed descriptions of those traits which were important are given in Appendix H. This list is taken from the Manual for the 16 PF. [ 6l 78 Variable Number (continued) Meaning (continued) 10 Sober vs. Happy-go-Lucky 11 Expedient vs. Conscientious 12 Shy vs. Venturesome 13 Tough Minded vs. Tender Minded 14 Trusting vs. Suspicious 15 Practical vs. Imaginative 16 Forthright vs. Shrewd 17 Placid vs. Apprehensive 18 Conservative vs. Experimenting 19 Group Dependent vs. Self Suf- ficient 20 Undisciplined Self Conflict vs Controlled 21 Relaxed vs. Tense 79 M m fO En w +) •4-1 in i i c o •H +J & M U o rH CO fl aj O 0) •H M +J &1 •H a) U a U o m *r oif> ^m in • #•••••• ~< o o ooooa ii ii 00><=mf^'M-rr-^ oa>m-* O-^^rjJB^^^r- • •••••»•• — < O O o ooooo I Oo •'^O n * 0>Hffl OulnO-t •»■ 000 ooooo I III O ~- O ~i O -O <6 W (M -O *4 Ol^O^Oo^^O^C^ -< o o o o o ooooo i T i 0'^'0-ooo*--*mo-*» OO^^-^CreemrgcO'***** O <^ O ^ -« r\) 0* ^Hin -o -o ^ • »••-••••••• ** O O O O O O OOOOO I I III II 0^>(7'©<0'*"^5'Niir»^»lM«0*4 OOr-co*rOO (a r4mNr^iA Of-'A"»''0' r >h.lf\Ov/>ll>ir>^» O (*l O N -* >0 rsjm -.^(^ T3 — « r> ^ '.-«,-»O 00-0 * rg »-' J "* r Jr-lM'©r-« , o -*• »nir*r- -^ ^i ^ ri r « -t oOn 4 -o O u"\ O O ** *»• rsl *M .-« CO o* in O r 1 * *» •*" OrgOO*^'^'*>rgrvi*™IO O ^ in O rw r<<«OOOOOOOQOQOOOOO II III O 0* (Mr^o-o ao u*» n- -D^-0 r\i ** C/ >0 tn Of f r x :»-«o « O^OfM r -* *+ ~* r+ r s .-« ~4 O r*l (M ^* -* O *f *-> o o o B o O C O Ci o ci o o o o <■> o III II I I I I I II O r- O <*> » > fini* , gO)rfnrooiM , 'iri*t* O ■** -r> r~~ v -■< o >C rj o* -»1 h- -T» 00 v7" U* 0* •* -O o r- in m «o yur^. o o *• « O o* h r*i m fc »*> O ? ID rM (M M H N O O "I ^ O l\ ^ »< H O fl .-I OOOC CO. 3CJOCJ OOO ooooo III I I I I I I I I ©.-400<*J>OoOr-co^'-*,.«wjcoO>r-«m»j O — < J ^ n it •? r O m <\j y -o oj r>. or>o o--«^r*-^ru^'-io.fCr o m r- • r-t o rt •+ t i • • ~* o 1 1 • Om a- o • Oh- r- ^ O TO o f- » cn 7 t> »n r- .-I o _■ » • o rj • • • -«o o o 1 » • o o 0> m * o r^ rn o m • CJ ao o ro f\J f- o ^ i*> * m • r-< >r 41 • a -a in r- o o o .1 -0 M f\J r- • o a y .-« o- GD o r-« o -r J-O (M r-> » I I I I I I -i o oo o o O f\» f*> \»C*f- ^r^rxTiT-^noOoOOlMf^'OOO O-roOrMU-Wf.. -OOn, coxv^o-"'-«OX^'r>> C O J M ; f* in C i) O ^i O rt ^ J) -f u ly U •* & ■IO^ K fNin w £^0«\j OO— *O.si.-«tr'rj-l-X0 > O o CO ^ ^ l"' rt, ^ - i r\j »^ o M ^ o fvj ^ rj ri , t m i O O O •"> o o I I I JOoOOOOOOOoOOO II I I I I I O J- rg CO -r ^« m o h" J- r\j r» U^ in o m — « #• >^i l ^ og IfN l- o h- u^ ■J B> ff M o t~t •-* -r o r^ o • • • • •"' o o l 1 oo o o 1 l r4*" lf» -O f*CD > O H m-flN (0^ Oh 80 M CO rd • rH in in w ro c • •H CO -P •H rrj -p in d o e • n CD V ft 1 1 cu 1 u •H 4-1 -P a ■P 3 G ^ a) m -H a) U -p •H a m H 4-1 a cd -p o fd •H c T5 a) •H g -P g H W rtJ CD O CD -H H -P Cn-H CD M Q U Oo^ommnj^-^ • ••«•••• •*OOOOOoO I I ! OO 1 8K1N (M« Mil 0»• r» ^ »» 00 * >0 r~ m ,-«(«» *M 0-< •torn ^iho« *«oooooooo I ou^^o '»■■<■»<'•*♦ O ootm^co ^o^ \v Or r *rMcr>Cf -O ^ C* -^ I*tr4 AM or- 4-iCO * mn J)rf ©in cj-oo-J.-iOoo-r OOOrt HJ-NNIVH ~ h in ^ » t »•§«■•••••■• r-IOOOOOOOOOOO I I III II o>o>nNorgnon''is-Mn 0Of-a3'f00(DN ,tl (y'-** or-m-r.omr'-mommif** OU\(MN(i)l»l NlHI»)r4 0*»< • #••••« *••••■ T i iii i Oinommrncomr-.-a'O^O'C^ o n^^ >t -* m oopsjo* no ^ * oivi'-'o-J'»-<-«i*iir»-*' "rfnxi • •»9*»« •».«•*» ^40000000000000 > I I II I I Oin fl i-Oir\o , mo< f i^N'Oin^rt owoOr^-< o in co^-ioiAso r ro o O O l/MT H fO iJ tft .* 1*; !/\ o >n >0 'O ^ C "O ^ ^ \ Off , NHvOWNO*ly , 0*WC3N'7>0'f'01rt Ocd-O Oo*»"0 ^■r q r- o h a5 j of*.o'-n-*"f r t'-4c^'A(jovfo*oCT*^-' , -'00*r OOOOOOOOOOO OOOOOOOOO I I I I I I I O-o vOT^- J) O O fj nJ fl (O ^ ui ro O r\j <-* m *- ^ o ^ r-j '") r- m ;nou ^ <:■ m ii> o«a ^ -o ^ r\i *r ■OO ■* o -o ^r-ino-cirjO't <*o -*■ 4- rj o r-< c-4 O m (fl 0> r- h H r, n m H (i o •)• o N h .-< r* r-iOoOOOOOOOoOOOOOoOOOO I I I I I I o o o o o Ollt O 0> OO O m O fM I t OPO O r-^ .© l/> tn r- *H 000 375 1.0 6 79 -0.1 766 -O.i co O 0* **! f* 1.0 0.1 0.0 -0.2 O o» <<■> •*• O mnoi" O a> r*j «M r» O ■om^-m r-t O -1- o>n • • • • • u O 1 1 O >o P- Vfl •*■ O O -0 iAr- 00 O ft ■O Psl CM r» ■O O CT- H C* OS ^ r-1 O ■f -tONrt # * » * * * * > » * * * » * » * » * * * » * » * * — « o o o o o O & rvi co co h ro O f- .*■ rsl r- j\ m * in —« ^h rsl r\j r>j •nc»o h-m>f 00 * t-> .-1 r* t mo » ** ° O O 1 T 1 1 ft ir .0 p- 00 O O *-« • ■-< w h H n M N 81 VD n M • W «J w En •H W lO CD rH W -H d) P Q U r-l O O O © "* O *>l •* • •••••»• -hOOOOOOO til • or i-ii>«« »« Tin -i O O O » O O M — • J" O -* • •••«••«•• ~«ooooooooo • III III «H 0^trN90 |w ^'|0'V ^ OO— . W _4 _ . O .«l O N O • •••••••••« -^OOOOOOOOOO II II O oian 4 O ** — «— «o~*— «~«~4 • ••••••••••• —oooooooooo© I I I * I I I o» Oi-iO'Oi'i-O'TrnmfMO'vm OfMOOi O^ONOrt^lMH >4oOOOOoOOOOOO • I I I O O O *** O* "1 h *^O^ k O(« *-«fN4-« —ooooooooooooo • III l _lOOOOO<3c>QOCOOOO I I III III O 3K •SNlP'ON* +NN *^"' > J r *' T *0 O r- O rsj co C* o o o ^ iN^oAivia^ftt^Ni^ -«OOOOOOOOOOOOOoO II I I I III I O ■■• C> ^ C3 •-* ^ •*! 0*-*f\l* r *l'>Jrt>Or\J-0 •-lOOOCOOOOcOOcOOOO I I II II I II oc*(\ia»-*«oytrs(-\ of"j\rgo^^coi*v 00*>(\i— «sj-0»/^*»C3**»r»>r— r\i-r^iAcNJ ^ O p -irif«^fno A r fc -'Oco,^cu*.0.-«f>joooc>ooc>ooooooo III • il I O * -o -« r^ o m ^ ^i ri -o ^ -fi -o ^ "»i * T O^^rvjf^T Co ■-<»-• O ^ji>jO*N„(f^O O -« O O O CiOOOO-J »>OOo<-^ O ' 3 OO III I II I C3J-r-(T *-ro-t-0-«'7'iOtr-«T>«Mo - *'>» O o •-* u^ -u 'j o —■ «r -• -» o-"-■ o — < a o r- oooooooooo 200000000 III III II I o >* n o o tf* a rt o ^^-^^..o^^^d.- />of^ o eo o r*- r- •/> o* .n cm o rv p • r> -r o* o ~ ^ ■*>>!■.- ** O O ~ ** or ~» •» O •* *n j i"' j* -■ * ^r ^ c*> *■" o t x O C ffi (? N ^ t « sfHO -* O t ^i -^ --**-*-* rr» <3 i-l N .-JOi^OOOOoOOOO oOOOOOOOO III I I II I p« «4 m« r- •"*"*^-"-»-^-^*>*o* • • • • • • o o • o o • o • • r-< * o h- » o o( O O -< CT" x o m »-.m o • • • * j — « o o o o I I o a> od sf- o> co O •» -O — « O— • ^^ O ^ — 00 (J* O — « r.rf»HrtNIM 82 M rH W «£> flj n Zn • w w C •H o •H LO -P O ffl • ■P 3 V e m a CD cu u 1 1 m c •P •H fi +J CD CU •H 3 U u ■H u 4-1 Q) 4-1 -P CD c H U 73 C C W •H -M r-i w rd CD u CD -H H 4J Cn-H CD u Q U o'M'-^J-'Or-oio 0000 < -*0<\i<-< • #■••••• -JOOOOOOO III I O0»CMif\.fr.-«.s-C0O0!M-J-< • •■«.««« • • • -•OOOOOoOQ » I I r* O O O I a o o o o o o t T i iii 000»cAr"-cs ( ^"'«0 > t^r» • •••••■•••ft -coqoocqqooo ? c I t OOc*>lA r *" , * T f r V - *'*-fM > Q^ L n'Mj)^C\|i/\0.ir ■* *■< O »*0©OOOC»00000 I till I I OmcOOfN^O^-I^^^COrll O NOmOKlONOlflHlMH •-•ooooaooooooo III I fry ©in ry ? iA (^ N tf m sfl fl N OCJ'r-<-trM'*r-*i'r'-:T'O lf *^"f*l -« o oooooooooooo i i I I I O-o^-^or-^f; co ea ,3* p- r- co "O 0-ct"<10'>*000>c\ir»sn,«».Cp-f— o »-• o rH'-'fjoaotMm.-j-^^o ^ooooooooooooo© 1 I i il ill Or^in(Mso fw i^"o-£ , >OcMcocr O 1*1 tc CO CO O PI ^ :0 lAn 'J\ [/ -OfV^-»fNjC7» Oh m*h l^^iO Hrt cm -♦ >., o cm -*■ co eo O0-cm-4"O.-<.-i---iO -t r-lOoOOOOc^oOOCOOOOO III III I O CD 'OO^Oi/w>i^o»^K>o»-tco^aocoor- O*OC0fWMU'E^0 > [>'0ON^( , lf , ' , OU"l^^-!^C0(»'O'QNC0rj'1.*^33IAO i 0"OfNjccoo(vjco^o»-totri\^>NO^ afcoNPijHr.o^ONOocoorfO ^OOOOooOOOOOOOOOOOO till 11(1 I I OcoNr-inco.oHm(\jOc>4>0(rN>oryo*^oco OO v, f'fni1^00t M A'ONi , l(flmOflm f i ODWf^pylOfDO C^ ry^Nf-iMcD ^ftitA^N » o cm o co cm ^ •-• cr r<-, o — « ro co o r- >^ o -t la m xr\ cr* «© <<> c.i ^ ^- h (m n r^ (^ f. o -^ C 1 -o OO worn *0 i •'. .o r- o >n cc *^N cm •*; ^o .j r-- »n. i o co s c\j ( o co -j* lt> o m m -O r*j m co in. ;."* ^ D* m r ■ Of-">i— ofMor:-oi\j.-'0'- , «" o o o o »-»i-«-^^4»-*^-4»-4 f* ,-*«-• CM fM a o o o o O r- O CM O t\l O >D O CM • • rM O I OOh o ~l .*■ O00H O -T X\ aoN • • • r-t O O O4r-to OHHC3 Olflo* cm in in OClMrl • • • • -< o o a I OU> HO'f o en en a* s O CO ,_< -o o o ro .-« m o • ■ • • t -< o o o o 1 I O co co sj- ty\ CO O fM -O CM C* IT o ~* m r<> co e«i -O O cm O ro o CM .1 O r-» -* O CM .H • • • » ■ I ►-€0 o o o I I I CM .0 -< a- VH -O f- -J- -. o t • o o I O r-i 0 S o o cm ro m O Cj -f CM ... m o o