LIBRARY OF THE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN vno.451 -4S<© cop. 2- The person charging this material is re- sponsible for its return to the library from which it was withdrawn on or before the Latest Date stamped below. Theft, mutilation, and underlining of books are reasons for disciplinary action and may result in dismissal from the University. UNIVERSITY OF ILLINOIS LIBRARY AT URBANA-CHAMPAIGN JUfl 6 1974 JAN 1 2 3 jaw 2 7 m L161 — O-1096 Digitized by the Internet Archive in 2013 http://archive.org/details/investigationofg541mont €T/C. &¥ 7 JO, Report No. k51 AN INVESTIGATION OF T] 'DNESS OF OF THE MAXIMUM LIKELIHOOD ESTIMATION PROCEDURE IN FACTOR ANALYSIS by Richard G. Montanelli, Jr. JUNE 1971 •THE LIBRARY OF THE NOV 91972 DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN URBANA, ILLINOIS Report No. U51 AN INVESTIGATION OF THE GOODNESS OF FIT OF THE MAXIMUM LIKELIHOOD ESTIMATION PROCEDURE IN FACTOR ANALYSIS by Richard G. Montanelli, Jr. JUNE 1971 Department of Computer Science University of Illinois at Urbana-Champaign Urbana, Illinois 6l801 This work was submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology, 1971. X f b ' lli ACKNOWLEDGMENT The author wishes to thank all of those individuals and organizations who contributed to one or more phases of the research presented in this thesis. The members of the thesis committee who gave many helpful suggestions when the thesis was in the planning stage and who carefully read and criticized the earlier drafts of the thesis deserve special thanks. The committee members were: Professor Lloyd G. Humphreys, Chair- man, and Professors Ledyard R. Tucker, Maurice M. Tatsuoka, Kern W. Dickman and William F. Stout. The author is also indebted to the Office of Naval Research, contract N0001U-67-A-0305-0012, which paid for the computer time. The work is part of a project "Development of Objective Factor Analytic Procedures" with Dr. Humphreys as the principal investigator. The author also wishes to thank the entire Statistical Consulting (SOUPAC) group of the Department of Computer Science of the University of Illinois for their programming aid. Special thanks go to Joseph Kolman, Paul Chouinard, and Bill Walter. A special thanks is also due to Pam Rosborough who deciphered the author's handwriting and typed all of the drafts of the thesis. Thanks also to the Offset Department of the Department of Computer Science for printing and collating the thesis. And finally, a very special thank you to the author's wife, Dale Montanelli for her patience and encouragement. IV TABLE OF CONTENTS Page I . INTRODUCTION 1 A . Background 1 B. Algebraic Methods of Factor Analysis 2 C . Maximum Likelihood Factor Analysis 3 D. The Tucker, Koopman, and Linn Study k E. Goodness of Fit 6 II . METHOD 11 A . Data Used 11 B. Generation of Sample Correlation Matrices 12 C . Factor Analysis lU D. Analysis of Variance ik III . RESULTS 16 A. Reliability Coefficients p and p 16 B. Other Goodness of Fit Measures 18 C . Use of the Likelihood-Ratio Test 21 D . Analyses of Variance 23 IV. DISCUSSION 27 LIST OF REFERENCES 31 APPENDIX A . Means of p and p 33 B . ANOVA summary tables 37 I . INTRODUCTION A. Background Factor analysis is a frequently used model building technique, especially in sciences where a large number of variables need to be studied. Unfortunately, little work has been done on ways of testing the goodness of fit of the model to the data. Several techniques for testing this goodness of fit have been evaluated in this investigation. In addition, these techniques were used to evaluate the maximum likelihood estimation procedure as a factor analytic method. For many years, factor analysis has been used as a research tool for finding the major or meaningful influences on a set of variables. Many different methods for finding these influences (or factors) have been proposed and used (Harman, 1967), but most are strictly non-inferential in nature. That is, they treat the observed correlations as though they are population values, and the resulting factor loadings are calculated directly from the sample correlations. Thus, the statistical problems of the sampling of individuals are ignored, and results are usually considered as though they are population values. On the other hand, the method of maximum-likelihood estimation provides a statistical procedure for the estimation of the parameters of the factor analytic model, based on the assumption that the original variables follow a multivariate normal distribution. The basic assumption of factor analysis is that x = Af + u + e where x is a random vector of p variables, f is a random vector of m < p common factor scores, e is a random vector of p uniquenesses, A is a p x m matrix of factor loadings, and u is a p vector of means. It is further assumed that E(f) = 0, E(e) = 0, E(ff') = I, E(ee') = D 2 , and E(fe') = 0. This model is usually written P = AA' + D 2 where P is the correlation matrix. Here, x is assumed to be multivariate normal and the rows of A are proportional to the rows of A as Lawley (19^0) has shown that the results are independent of the scale of measurement. i B. Algebraic Methods of Factor Analysis """" The principal axis method of factor analysis, developed by Hotelling i (1933), is today probably the most commonly used factor analytic technique. Its basic objective is to determine factors which account for the- -maximum amount of variance of the observed variables. The first factor is that linear combination of the original variables which accounts for maximum variance. Each subsequent factor accounts for a maximum amount of the remaining variance, while remaining uncorrelated with all previous factors. Thus, the factors are derived by an algebraic rule from the sample correlations, and can be strongly influenced by sampling variability. 1/2 The factors are determined by the matrix equation A = V A where V contains the normalized eigenvectors of the sample correlation matrix as columns, and A is a diagonal matrix containing the eigenvalues, i.e., R = V A V . Minres (minimum residual) factor analysis (Harman and Jones, 1966) is another algebraic approach to the problem of obtaining factors. Its aim is the best possible reproduction of the_ observed correlations, where best is defined in terms of a least squares fit. Thus the factor matrix A is determined such that the sum of squares of the off diagonal elements of R - AA' is minimized. C. Maximum Likelihood Factor Analysis In contrast to the algebraic techniques mentioned previously, the method of maximum-likelihood estimation requires the writing down of the density function of the observations, given the population parameters. Then the sample values are considered as fixed, and the parameters are considered as the variables. The resulting function, called the likelihood function, is then maximized with respect to the parameters. The values of the parameters which maximize the likelihood function are termed the maximum-likelihood estimates of the population parameters (Lehmann, 1959)- The possibility of the use of the maximum-likelihood method for the estimation of factor loadings has existed for at least thirty years (Lawley, 19^+0). The method has always required an iterative procedure with a large number of calculations to be performed on each iteration. Thus it seemed natural that the development of computers would encourage the use of the method, but Lawley and Maxwell (1963) reported that in some cases convergence of the likelihood function to its maximum was a very slow process or might not even be attained, unless good initial estimates for the factor loadings were used. Joreskog (19&7 a and b) has developed a new computational method which has the advantage that the iterative procedure always converges. In other papers, Joreskog (1969, 1970) has extended the maximum-likelihood estimation method to cover a wide variety of models, including factor analytic ones. In conjunction with these efforts, Joreskog, Gruvaeus, and van Thillo (1970) have developed a general computer program to calculate maximum-likelihood estimates. The maximum-likelihood estimation procedure also provides a likelihood-ratio test of the number of factors, and this test has been made available in their general computer program. In applying the method of maximum likelihood to the general factor analytic model (P = AA' + D ), one writes the log of the likelihood function, omitting a function solely of the observations (Lawley and Maxwell, 1963) log L = -f [log |P| + tr(RP -1 )] where R is a sample correlation matrix based on a sample of size n + 1. This expression is then maximized with respect to the elements of the matrices A and D, to obtain a maximum-likelihood solution. Since the method of maximum-likelihood estimation is an established statistical procedure, it is desirable to see how well it does in practice. Browne (1968) has already shown that maximum-likelihood estimates are preferable to many other types when dealing with sample correlation matrices drawn from populations which exactly satisfy the factor analytic model, but it remains to be seen how well it will work with data more like real data. D. The Tucker, Koopman, and Linn Study Tucker, Koopman, and Linn (1969) generated 5U population correlation matrices in order to study factor analytic methods. One of their simulated correlation matrices was defined by R - VA + B 2 P 2 B £ ♦ B 3 P 3 B 3 where B , B , and B were diagonal matrices with real positive diagonal elements b , b , and b (j = 1, 2,...,p, the number of variables), -'-J '—J -J J 2 respectively. Since b . was the proportion of variance of variable j 2 due to the major factors, b the proportion due to the minor factors, and 2 b„ . the proportion due to the unique factors they had: 2 2 2 b, . + \>t. + b, . =1. lj 2j 3j 1 2 Thus, b + b^,. equaled the communality (common variance) of variable j. Three different relationships between these coefficients defined the three types of population matrices used by Tucker et al . Correlation matrices 2 2 2 with b_ . = and b = (l - b ) exactly fitted the mathematical factor 2 analytic model (common factors + uniquenesses). With b = and -^ J 2 2 b_ . = (l - b ), the correlation matrices constituted the simulation model, as P contained the accumulated effect of 180 minor factors. It was hoped that these simulation matrices would approximate real data population-correlation matrices which could be thought of as arising from a few major and many minor influences. They also employed a third model which contained influences of both minor factors and uniquenesses. For 2 2 1 - b 2 this middle model, they had b = b . = lj All correlation matrices contained 20 variables. Each P matrix s (s = 1,2,3) was constructed from the relationship P = A* A* ' where s s s A* was obtained by adjusting the rows of A to be of unit length, s s The A matrices were generated by random processes and contained either three or seven columns, representing the number of major factors in each correlation matrix. The A matrices were generated by another random process so that the effect of V was as though there were 180 minor factors in it. P was an identity matrix, as the factor analytic model assumes a unique factor for each variable, and that these unique factors are uncorrelated. Tucker et al also used three levels of entries in the B matrices; hi (.6, .7, .8), wide (.2, .3, A, .5, -6, .7, .8), and low (.2, .3, .h) . Thus their design was three (models) x two (number of major factors) x three (levels of B coefficients), and they generated three correlation matrices for each of the eighteen cells. Tucker, Koopman, and Linn were interested in comparing several factor analytic techniques, but the data they generated are useful for studying any procedures related to factor analysis. In the Tucker, Koopman, and Linn (1969) study, the authors used a random process to generate conceptual input factor loadings K for the major factor domain. They combined these with random normal deviates, applied a skewing function, and multiplied by the matrices B , in order to get to actual input factor loadings A . (A = B A* where A* A* ' = P ). The authors used joint rotations of actual input factors with output factors, and also rotations of output factors only, to assess the degree to which actual input factors were found on output. Thus there were two methods of comparison used, and each resulted in a separate index (coeffi- cient of congruence) for each actual input factor. Although the raw data of the Tucker, Koopman, and Linn study consisted of population-correlation matrices and not samples, some of their results can serve as standards for some of the results of the current study. In general, the reproduction of the actual input factors in the output factors was very good for the formal model, and poorer for the simulation model. 2 The reproduction was good with a high level of b . and poorer for a low -*~ J level. Thirdly, results were better for three factors than for seven. 2 Finally, the combination of simulation model, low b ., and seven factors produced extremely poor results. These results led Tucker et al to conclude that the quality of factor analytic results depended heavily on the design and conduct of the study. E. Goodness of Fit The set of factor analytic methods can be divided into two parts: exploratory methods which are used in early investigations in an area, with the purpose of reducing a large number of variables to a smaller number of factors when the investigator has no a priori hypotheses as to the composition of the factors; and confirmatory methods which are used by investigators with specifiable hypotheses about the factors. The present study considers confirmatory factor analysis only, and a major interest is in the discovery or development of a measure which would reflect the degree of fit of the final solution to the specified hypothesis. It is possible to test this hypothesis via the likelihood- ratio technique, and although the distribution of the likelihood-ratio 2 statistic has not been tabled, it is distributed approximately as a x in large samples (Lawley and Maxwell, 1963). Unfortunately, this test sets up the hypothesis as a null hypothesis, and as the sample size increases, it is more likely to be rejected, as no hypothesis is exactly /true. Thus, this test is of little use to many researchers who are ^interested in how well their data agree with their model, fully realizing that their model cannot be exactly true in the population. Therefore what is needed is a measure to assess the goodness of fit of the model to the data. Thus, the problem in this study is different from the one considered by Tucker, Koopman, and Linn as they were interested in factor matching, while here, the aim is to have one index to measure the total goodness of fit. Tucker (personal communication) has suggested a measure 2 (*-- 1) v df ; -1 m p lm = 1 ~ — (*-- 1) df 2 where \ / d.f. is the c hi- square approximate test criterion for the likelihood-ratio test statistic divided by its degrees of freedom, taken after zero factors and after m factors have been extracted. This measure is analogous to a percent of variance accounted for by the model, as the 2 expected value of a x random variable divided by its degrees of freedom is one. More recently, Tucker and Lewis (1970) have developed a second reliability coefficient, U n - M m P, 2* M. - 1 n' m 1 2 where n' = N - 1 - 7 (2p + 5) - T m 5 P = number of variables, M = F /df , m 6 3 mmm F = minimum value of F (A, D) = log |p| + tr(RP _ ) - log |r| - P m m °e ' ' °e ' ' (Jb'reskog, 1967b) for m factors, and df = degrees of freedom for m factors. It was hoped that this coefficient would be independent of the sample size and would provide an estimate of the goodness of fit of the factor analytic model in the population. Tucker and Lewis calculated p for the number of major factors for some of the population-correlation matrices of Tucker, Koopman, and Linn. These values (Table l) can serve as targets for the current study. These two measures (p n and p_ ) are similar (as lm 2m 2 can be seen by substituting Y = n' F in p. , but not identical. It m mm lm is hoped that one or both of them are good indicators of goodness of fit for maximum-likelihood factor analysis. Table 1 Values of p_ Obtained by Tucker and Lewis from Eight of the Tucker, Koopman, and Linn Population Correlation Matrices (N = °°, p = 20). 3 Factors in Major Domain, Reliabilities for 3 Common Factor Models Formal Model Simulation Model high b 2 . 1.00 .83 lowb lj 1.00 .55 7 Factors in Major Domain, Reliabilities for 7 Common Factor Models Formal Model Simulation Model high b 2 1.00 .71 lj low b?. 1.00 .1+8 Another possible measure of goodness of fit is the sum of squares of differences between the correlations implied by the model and those reproduced by the actual output factors. Browne (1968) suggested this measure of goodness of fit: pi p c, = Z Z [**' - AA f ] . 1 i=l j=l 1J where A is the sample factor matrix, $ is the population factor matrix, and p is the number of variables. Of course, another possibility is to exclude the diagonal elements. This would emphasize reproduction of the correlations, while ignoring the communalities: P i-1 P c = Z Z [$$' - AA'] •• 2 i=2 j=l 1J 10 Both measures were scaled by the total sum of squares in order to produce coefficients, r and r , with upper limits of 1.00. In most cases they should vary "between zero and one. r 1 = l- c l P Z i=l 1 2 Z [']T. J-l 1J r 2 -l- C 2 P 1-1 r ,2 Z Z [$$»]7 i=2 j=l 1J These measures (c , c , r , r ) are all invariant under orthogonal rotation of the sample factor matrix A, and of the hypothesis factor matrix $. All six measures (including p and p ) were obtained for all 96 sample correlation matrices. 11 II. METHOD A. Data Used Due to limitations on computer time, it was necessary to use only some of the population correlation matrices from the Tucker, Koopman, and Linn study. In order to preserve the effects due to the independent vari- ables used in generating those matrices, it was decided to randomly select one matrix from each of eight cells in their design. The eight cells were created by using two levels of each of the three independent variables used by Tucker et al; i.e. model (formal vs simulation), level of B, (high vs low), and number of factors in the major domain (3 vs 7)- The eight matrices used are identified in Table 2. The level of battery (l, 2, or 3) was used by Tucker, Koopman, and Linn to designate a particular correlation matrix, as they had three such matrices in each cell in their design. In the current study, one battery was randomly selected from each of the eight cells of interest. In order to include the parameter of sample size, it was decided to draw samples of size 100, UOO, and 1600 from each population-correlation matrix. To achieve some stability of results, four sample correlation matrices were drawn from each population-correlation matrix, at each level of sample size, yielding 96 sample correlation matrices. Table 2 Number of Battery 2 2 3 3 ■ 3 1 1 1 Level Number of Matrix of b n . lj Model Factors 1 high formal 3 2 high formal 7 3 high simulation 3 h high simulation 7 5 low formal 3 6 low formal 7 7 low simulation 3 8 low simulation 7 12 B. Generation of Sample Correlation Matrices The intuitive way to generate sample correlation matrices is to generate samples of random variables from a multivariate normal distribu- tion "with a specified correlation matrix (Kaiser and Dickman, 1962) and to calculate the sample correlation matrices directly from this raw data. However, this method requires a large quantity of random numbers and a large amount of computer time, especially when large sample sizes are required. To avoid this problem, a more economical procedure, described by Odell and Feiveson (1966) and used by Browne (1968), was used in this study . In order to compute a sample correlation matrix R when given the population correlation matrix P, one uses R = (Diag [A])" 1 ' 2 A (Diag [A])" 1 ^ 2 and A = (fiT)(fiT) ' where P = ffi' and the elements of T( lower triangular) are chosen as independently distributed variables: t.. is distributed as N(0,l) (i > j) X J t.. is distributed as Chi with (N-i) degrees of freedom t.. = i 30) where v = degrees of freedom and X is a pseudo random normal deviate. The P value for h is gotten from the relation h = — h^ where h,-„ is tabled v v v 60 60 against values of X from -3.5 to +3.5 by Abramowitz and Stegun. A cubic equation was used to interpolate between the tabled values of lv n . h. Cr , = -.000924X - .000159X 2 + .000308X 3 + . OOOI89 60 p p p The correlation between lv and h/-_ (for the 15 tabled values) was 1.0000. C . Factor Analyses Each correlation matrix was factored using Joreskog's (1967a) maximum-likelihood factor analysis program. The maximum number of iterations was set to 100 and the probability of chance occurence was set to 1.0 so that all solutions were obtained. Solutions were obtained for the number of factors in the major domain. Additionally, the likelihood ratio tests of the number of common factors were obtained from zero up to the number of factors in the major domain, so that p and p p could be calculated for each possible number of factors. The coefficients c , c p , r , and r were calculated for each factor matrix. D. Analysis of Variance In order to determine the effects of the four independent variables (model, level of b , number of factors, and number of observations) on 15 the measures of goodness of fit, six separate fixed-factor analyses of variance were performed, each being 2x2x2x3- 16 III. RESULTS A. Reliability Coefficients p and p The means of p and p , across the four samples of the same size for each population-correlation matrix, are presented in Appendix A. There were only small differences between the two coefficients. For very large sample sizes, the formulas yield quite similar results, as is illustrated by the samples of size 1600. The correlation between p and p for the number of factors in the major domain, across all 96 sample factorings was .998. All results are discussed mainly in terms of p , as it is the later, published version. p was an excellent measure of goodness of fit for the factor matrices obtained from samples from the population-correlation matrices (hereafter called sample factor matrices) of the formal model. In the three factor matrices, with high or low b . (Table 3) P 9 was very close to 1.00, for 2 all sample sizes. With seven factors and high b . (Table h) the results 2 were as good. However, with seven factors but low b , p p went above 1.00 after four factors with only 100 observations. The average value of p~ after seven factors were obtained was 1.4285, and the individual values were 1.0956, I.OU28, 1.4710, and 2.1046. This value, 1.4285, was much larger than the population value of 1.00 obtained by Tucker and Lewis (1970) While this result was probably due to the small sample size of 100, it reflected an undesireable property for a reliability coefficient. However, with 400 and 1600 observations, results were much better. Thus, the method of maximum likelihood resulted in good solutions as measured by p_ when the populations - exactly - f Itted— the— factor analyt ic _mo del . The only exception was with variables of low communality, in which case more observations were necessary to obtain a good fit. IT Table 3 Means of p ? for Matrices with 3 Factors in the Major Domain, After 3 Factors Have Been Obtained Formal Model Simulation Model Sample size 100 1.0011 .8535 >f\ 1*00 1.0018 .8253 J 1600 1.0001+ .8319 ioo .9929 .6157 i+oo 1.0127 .5690 1600 1.0023 .5623 Table h Means of p for Matrices with 7 Factors in the Major Domain, After 7 Factors Have Been Obtained Formal Model Simulation Model Sample size 100 1.0139 .731+8 High b . 1+00 1.001U .6829 U l600 .9997 .7118 100 1.1+285 .6022 1+00 1.01+82 .5312 1600 .9981+ .1+982 Table 5 Means of p p for Matrices with 3 Factors in the Major Domain, After 1+ Factors Have Been Obtained Formal Model Simulation Model Sample size 100 1.0116 .8778 High h . 1+00 1.0051+ .81+22 lj 1600 1.0011 .81+1+2 100 1.01+16 .671+3 Low h . 1+00 1.0257 .6029 lj 1600 1.0050 .6108 18 Results from the simulation model matrices were not nearly as good. In no case did p p reach the value of 1.00. The largest values were 2 obtained with high b . and three factors (Table 3), but the highest was .8535. There was a trend for p to decrease with increased sample size for simulation model matrices with low b . (Tables 3 and k) . p also became smaller as the number of factors in the major domain increased and as b., . decreased. The calculation of p_ was extended to four factors in the three factor matrices, in order to see how it behaved. It was thought that there might be some leveling off, after three factors. This did occur for the formal model (Table 5), after p had already reached 1.000. There was some tend- 2 ency for the values of p to level off for the simulation model, high b ., as the increase from two factors to three factors was much greater than that from three factors to four factors (Appendix A) . In the simulation 2 model with low b . , there were no signs of a leveling off of p after three factors. B. Other Goodness of Fit Measures The results for c and c (Table 6) were very similar, as were' the results for r and r . To get an idea of the degree of similarity, the coefficients were correlated across all 96 matrices. Since c correlated .968 with c , and r correlated .99^- with r^, results will be discussed in terms of c and r only. The coefficient c behaved exactly as expected. For all eight matrices, c got smaller as the sample size increased. In all cases, increasing the number of factors, while holding model, level of 2 b ., and sample size fixed, caused an increase in c . In all cases, moving from the formal model to the simulation model while holding the other three independent variables fixed caused an increase in c. . Finally, in all 19 Table 6 Means of the Coefficients c , c Q , r , and r , After the Number of Factors in the Major Domain Have Been Extracted Matrix Sample c c r r size 1 100 1+00 1600 .9587 .21+72 .0U61+ .9050 .2296 .01+26 .9782 .99!+!+ • 9990 • 9737 .9933 .9988 2 100 1+00 1600 1.6355 .1+625 .1035 1.1+322 .1+336 .091+3 .91+13 .9831+ .9963 .9201 • 9758 .991+7 3 100 1+00 1600 1.1+689 .5290 .3523 1.3211+ .U51+7 .2865 • 9705 .9891+ .9929 .9671 .9887 • 9929 k 100 1+00 1600 2.5336 .9252 .6677 2.11+20 .61+90 .^593 .8915 .960^ • 971^ .81+03 .9516 .9657 5 100 1+00 1600 1.5298 .3131 .071+9 1.2331 .2732 .0623 .85I+I .9701 .9929 .8520 .9672 .9925 6 100 1+00 1600 3.7076 .7221 .2576 1.8881 • 385U .1078 .1256 .8297 .9392 .21+23 .81+53 .9567 7 100 1+00 1600 5.0938 2.9050 2.31+7!+ 1+.1732 2 . 0l6l 1.79l!+ .5351 .731+9 .7858 .5258 • 7709 • 7961+ 8 100 1+00 1600 6.783U 5.1870 U.7623 1+.1999 3.0895 2.71^3 -.5998 -.2233 -.1231 -.685I+ -.2398 -.0893 20 Table 7 Standard Deviations of the Coefficients c Number of Factors in the Major Domain Have Been Extracted , c , r , and r , After the Matrix Sample size 1 100 1+00 1600 .2668 .0696 .0066 .2555 .0632 .0071 .0061 .0016 .0001 .0071+ .0018 .0002 2 100 1+00 1600 .1+390 .1100 .0230 .5500 .1057 .021+0 .0177 .0039 .0008 .0307 .0059 .0013 3 100 1+00 1600 .9731+ .01+16 .053^ .9287 .01+59 .0U57 .0196 .0008 .0011 .0231 .0011 .0011 k 100 1+00 1600 .1+661 .1811+ .1701+ .2975 .1938 .1382 .0200 .0078 .0073 .0222 .011+8 .0103 5 100 1+00 1600 .2350 .1363 .0130 .0933 .1325 .0061 .0221+ .0130 .0012 .0112 .0159 .0007 6 100 1+00 1600 .21+19 .1819 .13^8 .1978 .051+6 .0111 .0571 .01+29 .0318 .079!+ .0219 .001+5 7 100 1+00 1600 2.329U .1106 .^228 2.2251+ .0656 .1850 .2126 .0101 .0386 .2529 .0075 .0210 8 100 1+00 1600 .6732 .61+1+9 .3716 .5566 .6975 .1726 .1503 .1521 .0876 .2231+ • 2799 .0692 21 2 cases, c increased when the b went from high to low. (Note, there was one reversal of this last finding with c , which decreased as the level of 2 b went from high to low from matrix 2 to matrix 6.) -*- J Also, r increased with increased sample size, increased values of 2 b, ., fewer factors, and from the simulation model to the formal model. lj 2 For high b ., formal model, the values of r were good for three factors, ■^ J -*- all sample sizes, while seven factors required a sample size of U00 for a 2 satisfactory result. With the simulation model, low b ., and three factors, r reached only .796^ with 1600 observations. In matrix 8 (simulation 2 model, low b ., seven factors), the values of r were actually negative. _L J J- This was partly due to the low total sum of squares in the model correlation matrix, but this result indicated, much better than did p , the inaccuracy of these solutions. The standard deviations of the coefficients c , c , r , and r were calculated (Table 7). There was a tendency for the standard deviations to be smaller with better fit, but there were more reversals than with the means. Also, since r and r had upper limits of 1.0, their standard deviations were forced to decrease as the means increased because the upper limit was being approached. C . Use of the Likelihood-Ratio Test Joreskog (1967b) used the likelihood-ratio technique to test the hypothesis that the number of factors m was a given number. The exact distribution of the likelihood-ratio test statistic is not known, but for 2 large N its distribution is approximately a x distribution with degrees 1 2 of freedom — [(p - m) - (p + m)]. If the hypothesis of m factors was rejected (due to a statistically significant value of the test statistic), Joreskog refactored the matrix for m+1 factors. It was thought that by 22 Table 8 Range of Probability Levels for x 2 Statistics 3 Factor Matrices Matrix Sample size 2 Factors Min . Max . 3 Factors Min . Max . 7 Factors Min. Max. 1 100 ^00 1600 .0000 .0 .0 .0000 .0 .0 .1173 .2697 .2028 .8533 .9211 .8257 .9890 .9820 • 9776 .9976 1.0000 .9968 3 100 1+00 1600 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0000 .0 .0191+ .0000 .0 5 100 1+00 1600 .0066 .0000 .0 .6501+ .0000 .0 .2060 .1201 .5070 .8027 .9712 .9186 .9376 .8897 .9616 .9960 .9982 .9923 7 100 1+00 1600 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0007 .0000 .0 .0051+ .0000 .0 7 Factor Matrices 1+ Factors Min . Max . 6 Factors Min . Max . 7 Factors Min . Max . 100 1+00 1600 .0000 .0 .0 .0000 .0 .0 .0000 .0000 .0 .0683 .0000 .0 .5381 .2903 .201+3 .81+65 .791+8 .7330 100 1+00 1600 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 100 1+00 1600 .0723 .0000 .0000 .9560 .0000 .0000 • 5019 .081+0 .0005 .9995 .6060 .0112 .6065 .7133 .2279 .9998 .9681+ .5371 100 1+00 1600 .0000 .0 .0 .0000 .0 .0 .0000 .0 .0 .0000 .0000 .0 .0000 .0000 .0 .0000 .0000 .0 Note: In the above table, the entry .0000 means that the number "was a zero, when rounded to 1+ decimal places. The entry .0 "was an exact zero, to the accuracy of the computations (about 7 decimal places). 23 looking at the probability levels (probabilities of the chance occurence 2 of the observed x values) of the test statistic for various numbers of factors, one might be able to determine the correct number of factors. In Table 8 are presented the ranges of these probability levels, for selected numbers of factors. For the formal model (matrices 1 and 3 for three factors, and matrices 2 and 6 for seven factors), in all cases one would accept (at any reasonable probability level from .001 to .100) the hypothesis of the number of factors in the major domain. The probability 2 levels ranged from a low of .1173 for one high b . , three factor matrix -*- J 2 to a high of .9998 for a low b ., seven factor matrix. However, in one 2 case, with low b ., three factors, and a sample size of 100, the hypothesis of two factors was also accepted (p = ,6^0h) . With seven factors, and low 2 b . (matrix 6), an hypothesis of only four factors was supported with 100 observations and an hypothesis of six factors was supported with ^00 observations . For all simulation model matrices, however, the hypothesis that the number of factors was equal to the number of factors in the major domain, was rejected. Even the hypothesis of seven factors for a sample 2 factor matrix with high b . and only three factors in the major domain -'-J was rejected (although one matrix of sample size 100 did have a p = .019^- which would not have been rejected at the .01 level). Thus, this test is appropriate for testing the hypothesis that the factor analytic model V f \y, Ihblds exactly in the data, but it is of no use as a measure of goodness sf fit for data that do not fit the model. D. Analyses of Variance Separate analysis of variance summary tables for the six measures p, , p„, c , c_, r , and r are presented in Appendix B. These analyses o 2^ were performed in order to discover the relative sizes of the effects of 2 the four independent variables, level of b , model, number of factors in the major domain, and sample size. Since the assumption of normality of analysis of variance was possibly violated, especially with p , p , r, , and r , border line significant F ratios should not be taken too seriously. The results for p and p_ were again very similar, so results are discussed in terms of p p . The main effect of model accounted for 61.19% of the total sum of squares for p . The average value of p for the formal model was 1.0^2, while for the simulation model, it was only .668. The only other large contributor to the total sum of squares (except for within cell) was the interaction between model and level of b , which accounted for 9.07% of the total sum of squares. formal simulation high b 2 U low b 1J 1.003 • 773 1.080 .563 Four other small but statistically significant effects were also found. The average value of p was .888 for sample factor matrices with 2 2 high b . and .822 for those with low b .. There was also a significant trend for p to decrease with increased sample size. The averages were .905, .83^, and .826, for sample sizes 100, U00, and 1600, respectively. The BXF and MXF interactions were also significant at the .01 level. 3 Factors 7 Factors high b low b lj 2 lj .919 .857 .792 .851 formal simulation 3 Factors 7 Factors 1.002 1.082 .710 .627 Although the results were similar for c and c p , only 6.68% of the total sum of squares was attributable to error for c , whereas 13.02% was 25 error for c_. This supported the earlier decision to discuss results in terms of c only, as it was less variable, within cells. The main effect of level of b., . accounted for 25.17% of the total sum of squares. The 1J 2 2 mean value of c was .827 for high b . and 2.807 for low b .. Model accounted for 2k. 62% of the total sum of squares, and the mean for the formal model was .838 while for the simulation model, it was 2.796. Sample size accounted for YJ .21% of the total sum of squares, with c dropping as sample size increased. The means were 2.96U, l.Ull, and 1.077 for the sample sizes 100, U00, and 1600, respectively. The level 2 of b . by model interaction accounted for 13.58% of the total sum of -'-J squares . formal simulation high b low b 2 u .576 1.079 1.101 1+.513 The main effect of numbers of factors in the major domain (c = 1.322 for three factors, c = 2.312 for seven factors) and the three interactions shown below were also significant at the .01 level. high b ] _ . low b lj 3 Factors 7 Factors .600 1.055 2.0UU 3.570 high b low b 2 lj 2 lj 100 Uoo 1600 1.6U9 .5U1 .292 U.279 2.282 1.861 formal simulation 3 Factors 7 Factors .528 1.1U8 2.116 3.U77 Unfortunately, due to negative values for low b ., simulation model, and 7 factors (summed across sample size), every main effect and interaction 26 which did not involve sample size was highly significant for r and r . While the negative values (r = -.338, r = -.315) indicated how poor results were for that combination, the effect was apparently strong enough to influence most other effects. The main effect of sample size did account for ^ .lk% of the total sum of squares, with r increasing as sample size increased. The means .587, -780, and .819 for 100, U00, and l600 observations, respectively, 27 IV. DISCUSSION A major goal of the present study was to find or develop a measure of goodness of fit for the factor analytic model. One such measure studied was p . 6l.l9% of the sum of squares of p was accounted for by the main effect of model. For samples from population-correlation matrices constructed to exactly fit the factor analytic model, p worked 2 exceedingly well. Only in the case of seven factors and low b . was it ^- J necessary to have a sample size greater than 100. p also had the desirable property of approaching unity (or nearly so) as the sample size increased, for the matrices developed from the formal model. The samples from the simulation model behaved quite differently. Even in the best case (three factors, high b ), the average value of p was .8535. -*- J c — Thus, in all cases, p p reflected the presence of the minor factors. There was also a significant decrease in p for sample sizes 400 and 1600, when compared with 100. This is not a good property for a proposed measure of goodness of fit, as intuitively one would expect the fit to a good model to improve with more observations. However, this decrease is due in part to p coming down to 1.000, after going over that value for samples of 100. There was a significant decrease in c with increased correlations implied by the model better with more observations. This was further illustrated by the fact that the p values were approaching the population values obtained by Tucker and Lewis. This can be seen by subtracting the population values (Table l) from the sample values (Tables 3 and k) . 28 Ta"ble 9 Sample Size 1+00 1600 .0018 .001-1+ .0001+ -.0003 -.001+7 -.0271 .0019 .0018 .0127 .0U82 .0023 -.0016 .0190 .0512 .0123 .0182 Differences Between Sample Values and Population Values of p 100 2 High b ., Formal Model, 3 Factors .0011 3 7 Factors .0139 2 High b ., Simulation Model, 3 Factors .0235 7 Factors .021+8 2 Low b , Formal Model, 3 Factors -.0071 J 7 Factors .1+285 2 Low b ., Simulation Model, 3 Factors .0657 J 7 Factors .1222 Since these values (Table 9) are only accurate to two decimal places (as 2 the Tucker and Lewis figures are to two places), all except the low b ., simulation model matrices were within rounding error of the population values for samples of size 1600. Thus, the decreases in p with increasing sample sizes were toward the population values. An important result was pointed out by the significant interactions 2 between level of b . and the model for p and c . In both cases, the 2 results in the simulation model, low b., . cell were much poorer than would lj have been predicted from the main effects alone. These results, an average p of .563 and an average c of 1+.513 (over four times greater than the next largest cell), showed that one cannot expect to support one's hypothesis with variables that have low percentage of variance accounted for in the major factors. It was interesting to note that while the values of c were 2 2 about the same in the two cells high b n ., simulation model and low b n ., lj lj formal model, the former had p = .773 and the latter had p = 1.080. Thus the model correlations were reproduced as well for simulation model, 2 2 high b., . as for formal model, low b n .. lj lj 29 The fact that level of b {23.11%) accounted for as large a per- -*- J centage of the total sum of squares of c as did model (2k. 62%) was encouraging. Thus, if the simulation model is a better model of the world, it is still possible for an experimenter to improve his results by constructing measures with high proportions of variance accounted for by the major factors. 2 The x statistic was useful for sample factor matrices for the formal model only. Even then, it lead to the acceptance of too few factors 2 in some cases, with low b . and/or too few observations. The measure r did well for the formal model matrices, although more observations were necessary before it neared its maximum of 1.00. 2 Also, with seven factors, low b . and 1600 observations it only attaimed 2 •9392. However, the simulation model matrices with high b . also gave -'-J high values of r . Thus the maximum-likelihood estimation procedure was doing a good job of reproducing the model correlations, but this was not 2 reflected in p . The results on matrices 7 and 8 (simulation model, low b .. confirmed the importance of controlling the relationship between the major and minor influences on one's results. The major factors should predominate over minor factors in any study. The results did indicate that it is easier to reproduce a small number of factors in a poorly designed study. Thus r was shown to be useful as a measure of goodness of fit. It does require the writing down of an hypothesized factor matrix , so it can not normally be used in exploratory studies. It has the advantage that it could be used with any factor analytic procedure. On the other hand, p could be used with any study, as long as the maximum-likelihood estimation procedure is used. However, it seems somewhat less useful than r when the factor analytic model does not hold in the population. 30 An investigator should strive to develop variables "which strongly represent his major factors. He should have a large sample size (a ratio of five observations per variable was not always sufficient to insure 2 good results, even with high b . and the formal model). If he uses "*- J maximum-likelihood factor analysis, he should usually stop factoring 2 when the X becomes non-significant, if his sample size is large enough. However, with real data, there may be statistically significant minor 2 factors which are not of interest. In this case the x cannot give an indication of when to stop factoring. However, the investigator can use p as an estimate of how well the formal factor analytic model holds in the population from which his data was taken. Although the statistical properties of this estimate are not known, and it may be high for small samples, it does have the desirable property of having a value of 1.00 in populations which exactly fit the formal factor analytic model. Small values of p probably indicate a poorly controlled study, and the investigator may be able to improve his results by using better controls over minor factors, by having variables with high percents of variance in the major domain, and by having a higher ratio of variables to major factors. Finally, in a confirmatory study he should write down an hypothesized factor matrix and use c and r to determine its goodness of fit. 31 LIST OF REFERENCES Abramowitz and Stegun. Handbook of Mathematical Functions . U. S. Department of Commerce, National Bureau of Standards Applied Mathematics Series, 1966, p. 9U1. Box, G. E. P., and Muller , M. E. A note on the generation of random normal deviates. Annals of Mathematical Statistics , 1958, 29, 6l0-6ll. Browne, Michael W. A comparison of factor analytic techniques. Psychometrika , 1968, 33, 267-33^. Dwyer, P. S. The square root method and its use in correlation and regression, Journal of the American Statistical Association , 19^5, ^0, ^93-503- Harman, Harry H. Modern Factor Analysis . Chicago and London: The University of Chicago Press, 1967- Harman, H. H. and Jones W. H. Factor analysis by minimizing residuals (MINRES). Psychometrika , 1966, 31, pp. 351-368. Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of Experimental Psychology , 2U(l933), UlT-^-l, U98-520. Joreskog, K. G. UMFLA A computer program for unrestricted maximum likelihood factor analysis. Research Memorandum RM-66-20 (revised edition). Princeton, New Jersey, Educational Testing Service, 1967- Joreskog, K. G. Some contributions to maximum likelihood factor analysis. Psychometrika , 1967, 32, U33-U82. Joreskog, K. G. A general approach to confirmatory maximum likelihood factor analysis. Psychometrika , 1969 , 3^, 183-202. Joreskog, K. G. A general method for analysis of covariance structures with applications. Part I: General methodology, Part II: Applications. Biometrika , 1970, in press. Joreskog, K. G., Gruvaeus, G. T., and van Thillo, Marielle. ACOVS: A general computer program for analysis of covariance structures. Research Bulletin 70-15. Princeton, N. J.: Educational Testing Service, 1970. Kaiser, H. F. and Dickman, K. Sample and population score matrices and sample correlation matrices from an arbitrary population correlation matrix. Psychometrika , 1962, 27, pp. 179-182. Lawley, D. N. The estimation of factor loadings by the method of maximum likelihood. Proceedings of the Royal Society of Edinburg, Section A, 19U0, 60, 6U-82. 32 Lawley, D. N. and Maxwell, A. E. Factor Analysis as a Statistical Method . London: Butterworths, 1963. Lehmann, E. L. Testing Statistical Hypotheses . New York: John Wiley & Sons, Inc. , 1959- Muller, M. E. A comparison of methods for generating normal deviates on digital computers. Journal of the Association for Computing Machinery , 1959, 6, 376-383. Odell, P. L. and Feiveson, A. H. A numerical procedure to generate a sample covariance matrix. Journal of the American Statistical Association , 1966, 6l, 199-203. Richardson, B. C. Random number generation on the IBM 360. Report No. 329, Department of Computer Science, University of Illinois at Urbana- Champaign, 1969* Tucker, L. R, Koopman, R. F., and Linn, R. L. Evaluation of factor analytic research procedures "by means of simulated correlation matrices. Psychometrika , 1969, 3^, ^21-^59- Tucker, L. R and Lewis C. A reliability coefficient for maximum likelihood factor analysis. Unpublished Manuscript, 1970. 33 APPENDIX A MEANS OF p AND p Matrix 3>* Ta"ble 10 Means of p for 3 Factor Matrices Sample size 1 Factor 2 Factors 3 Factors 1+ Factors 1 100 i+oo 1600 .1+321 .4385 .1+323 • 771+1 .71+71 .7575 1 . 0011 1 . 0017 1.0001+ 1.0135 I.005I+ 1 . 0011 3 100 1+00 1600 .1+776 .1+6U8 .1+679 .6626 .61+58 .6522 • 8570 .8262 .8321 .8818 .81+32 .81+1+1+ 5 100 i+oo 1600 .6739 .658I+ .6581 .8879 .8639 .861+1 .9932 1.0126 1.0023 1.0397 1.0255 1.0050 7 100 1+00 1600 .1+719 .1+1+13 .1+577 .5591 • 5205 • 5177 .626^ .5711+ .5629 .6863 .6058 .6112 Table 11 Matrix Sample size Means of p for 3 Factor Matrices 1 Factor 2 Factors 3 Factors 1+ Factors 1 100 1+00 1600 .1+273 .1+375 .1+320 .7703 .71+62 • 7573 1.0011 1.0018 1.0001+ 1 . 0116 1.0051+ 1 . 0011 3 100 1+00 1600 .1+731+ .1+639 .U676 .6571 .61+1+6 .6519 .8535 .8253 .8319 .8778 .81+22 .81+1+2 5 100 1+00 1600 .6702 .6578 .6579 .8852 .8631+ .8655 .9929 1 . 0127 1.0023 1.0l+l6 1.0257 1.0050 7 100 1+00 1600 .1+670 .1+1+03 .1+571+ • 5509 .5187 .5173 .6157 .5690 .5623 .671+3 .6029 .6108 35 t- OJ H Eh w 0) o ■H S Sh O -P o o O w S VO W *h o LA ■P O cd O 0) on CM H 0) H 0) ft N 3 cn •H -P o on t— _3" CO VO fOJ-4 ltn on t— onno\ o vo oj VO t— CO O t— On H O ON LTN CO H VO J" ON on on on O O ON t— vo t*- on o on vo mj rH H H H OJ VO OO co om H0J4 oj On t"- t-onn t-VO Ol onvo c— ONO LA HCO CT\ vo _=r vo vo co vo vo t— _=r 0\CO CO vo vo vo OJ ON ON LAJ-J- H OJ On co CO LA NO LA ON LA ON LA J" >vD LA t- -4- on cm vo on t— -3- C--CO o no vo vo -=t- on j- o t— co on h CO t— t— LTN LTN LTN H ON CO -3- J- -=)• rH 4COH 0OCO t— VO CO -3- On t— on H OJ VO ONVO -=J- LfNVO OJ on on co O t- t— vo -4- on H ONVO OJ H ON t— VO VO J- -=J- -=!■ o t- t- -=Jr _=)■ on H VQ t- H t— on vo CO t--* H CO VO t— la on O OJ CO t- OJ -=f la on on on On h -3- LA J" t— co on vo f-vo LA LAVQ on on on co vo vo on on on LAt^-O On ON_j- H LTNVO LA CO OJ CO CO H H O ON CO -=J- t— CM OJ CA co On h LAVO LTN vo t— oj o j- n -J" .J" LA OJ OJ OJ MD LA LA on on on on on h OJ OJ OJ LTN LAND LA J- CO t-LAH o on o H OJ ON CO OJ CO ON C— CO ltn oj on J- Ol o o f— _=r OJ OJ OJ H H H -3- -=t _=!- OJ OJ OJ o o o o o o o o o o o o o o o o o o o o o o o o H-=r vo H -J- VO H J" VO H -=J- vo H H H H OJ J- vo CO 36 oo EH m 0) o •H P a O -P o a3 t- o Ch CM CL O en cd a t- VD W u o -p CJ O 0) 6 CO C\J CD H 0) Pn N § CO CO X ■H H P a3 a on H o\ H O ON O O On C— 00 o WWH H 00 On ON0O OO VDM3 VD UN J" CM t— oo on ONVD VD VO LTNVO OOIA ON t— t- VDVDM3 On LTnVO VO 00 CM CO On H ltn LTNVO J- ONVD J- VO O t— On H J" J" LTN oo OncO 4- CM H COCO H t— VD C- CM CM O CM LTN_3- VD CO oco H4-VO ON t— CO CM CM CM o o o o o o H -=1- VO CM LTN CM -=t CO GO OO CM -4- On J- O ON H H vo o on HVOl- O OO VD OO ON ON t— ^t VD lAO\H j- OO OO LTN LTN LTN ON CO CO O CM OO LTN_3- OO J" _=* -=t- VD ON CM LTNOO VD O t— H On CO CM CM CM CM H 00 O OO ON VD LTN-^t CM O CM H LTN_4- JVOJ- LTN J" ^t H VD O O ON LTN CM H ONVD O t- OO CM CM t— CM t— LTN OO H J- -=f J" VD OO OO 00 OO t— w j- J- OO OO 00 H LTNVO ON CM O 00 t— OO OO CO VD VD H CM OO On CO CO OO LTN LTN CM CM CM O t— OO OO H ON J- CM CM H H H O O O O O O H4-VO 4VO t- VD -4" OO LTN LTN CM VD LTN LTN J" OO OO H ON ON 00 H O J- _3" -=f _3- o o o o o o rH J- VO H VD 00 00 CM LTN On O O On _=f J- OO H oo oo LTN O CM J" l— VD OO OO OO o oo CO O OO CO J- oo CM OO OO _=t- O LTN O H CO O t— -=f CM CM CM O O O O O O H4VO CO 37 APPENDIX B ANOVA SUMMARY TABLES 38 H 0> EH F-h 0) Pi ^ -onvDonHOC\io\VDono L/NCO O H VO \£> v£> CO l>- OAVO LTNrH-H-,^- ,_| ojovDONcooniAonoHmwaiH HOHOO-H-OOHOOOOOOOO on m o\o j- t- co on o t— on h vo t— o o\ r\j 0\HO(MOmroO\WJ-HiAHO\COJ- H H (M ChrH H OJ CO-H-Ht^lAt-VOMDt-fnOLAOOJVDCVl iJAcx30c\ivovoc\]co_=r-covooona\aoa\ oj- o t-3 Ph u O --* -P S O IS! O CO 0) H ft eg ><; >< |xj bx! co pq PQ PQ 3 Pn S S X M X S S § | g Pn PQ pq pq' on on o o o o OO H J- OA C0 -p o Eh V ft * ko VD H [in 2 O -P o !m Ch to O O cq cr cn o CD O T3 0) !h P"h O o en ******** lAH J t-J I>-CO ChVOcO J fO H [-Vfl h moo ltnooooj--=i- o on t- -J-_ o t-^- ^ H iAt--mvD o\b-a\ cm co cm co t—VO VD ON-3" H CM CM H (MCO J- COt— CM t— t— -3" VO -=f O t— O HCO OMAOOWJWJ-HOWt-CACJCvlfn^ CO H CM LTN ITN ON ON ON CM 0O C^ J- O VD 0OJ; OOlA4hCOlAWOOWHOO\W(n J- CM CO ON On CM CM CO O VD LTN CM CO H H H t— CM o t— oo -=t onoo H LTN LTN CO O HVOCO ir\ \.n vn 1 HVDcocoLrNcocoooOLr\cooOLr\VDVD LTN.-3- VD C— CO H H CM CM r-\ r-{ VD CMCOJ-VD t- CM LTN t— ON CO J" rH U"N H COON OMACOJJ- CM ON H O Lr\t-CO LTN-^ VD O cnH CMOLT\0\COONJ-VDO\0\OCMVDt- OOlA0\t-00H0lOOW(MO0\4O\ J-CMCO-4-OVDLCNCO ON ON CM VD LTN CM H H CM -=T CM CO H H VD O H CO HHH CMHHCMHCMCMHCMCMCMCMCM pq pq O w H H Pn m !h O ^ ■P S o - — En N O CO 1-5 H CO [>S r*S PS cq pq pq ffl CO CO VD CO t— CO LTN ON H O V 1+1 0> H EH fc Eh O w o 0) tj a) a; ^ o O (D o u o CO CM ON CO -3- H On on oo ONN£> c— mvoowj t— -j- h wtno HO\t- H VOCMOnr— UAOOOncOVOVOHOOOCMOLrN OCMAJ-maiJ-frnAOOOOlJCOOOO cot-tDt-j-HiAJ-moHOroaiiAtn o on on on on on _3- h w VDHO4t~-t-^V0mO\i)mH44W OHmiA0\O\DC\l4HOO4m\DO oo vo c\j on on H CM CM H VDOJPOlAtACOyDcomWHOt^lAOCO on_=t-oiiAON_=t-oo-=i-onLiAoocMoncM_=fco OOMAONOnCMCOPOH^OJ-CO t-b-J en t~co ^t ^t HO_=rt— hhovdltnoco o on on on on on_=f- on cm H H CM HHHCMHHCMHCMCMHCMCMCMCMCM Ph H H CO CM -H- CM H O O O O 00 CO VO H LTN On Qi -P O En V ft * k2 CO H EH Pn pi S ol CO O CO CO -P - LT\ rH 0\m\o IAIA 00 H r-{ o\ \Q CM CO CM OJ H CO OJ On CM J-OO 0\_3- LTN^t LTN LTNVO ON OJ O OJ VO 0\r|00W(\]OVDlAOrUOVDC0t-C0 C— t— LT\ LTN ON £— OOOLlAOJOt— OHO OJLTNCOLOvHHOOOOHCOOOOOO LT\ OJ OJ OJ OJ cooj^-ooonoonhcocoh t-04\OHO\r|OO^HO CO o VO P0-=t UTN H O CO LT\ OJ H H H H H -=t CO CO OJ -Cj- VO ON_=f H-^T O O VO CO CO ON LTN OJ ChHCOiACVJOPOtAHMJ-HW 1^4- H t-t-lAOOM-HOOO(MOIArinW OJLTNCOHHHVOOOCOCOOHOOVO LT\ OJ OJ H OJ OJ HHHOJHHOJHOJOJHOiOJOJOiOl Ph CO .St OJ o CO o o o o LTN o c— ON H LCN On a3 -P O EH *tf* 2,2. \tf*