LIBRARY OF THE UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN 510.84 IC63c VN0.l\l-ia.O ENGtNLCRir^^''^°°^'''''^ l ihe person charging this material is re- sponsible for its return to the library from which it was withdrawn on or before the Latest Date stamped below. Theft, mutilation, and underlining of books are reasons for disciplinary action and may result in dismissal from the University. UNIVERSITY OF ILLINOIS LIBRARY AT URBANA-CHAMPAIGN i -. - . ,.. ,, '^UL ^W 9 juH 1 1 m FEB Z1 HAP 5 1991 tECB L161 — O-1096 Digitized by the Internet Archive in 2012 with funding from University of Illinois Urbana-Champaign http://archive.org/details/towardkeystonesf113rayr ^^^s^ . m Advanced Computation UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN URBANA, ILLINOIS 61801 CAC Document No. 113 TOWARD KEYSTONES FOR THEORIES OF NATURAL COMMUNICATIONS ^y Robert M. Ray ^I-.v^ August 197^W ^\^1^ 01 '"'"^'^ \\rh»na- CAC Document No. II3 TOWAED KEYSTONES FOR THEORIES OF NATURAL COMMUNICATIONS by Robert M. Ray III Center for Advanced Computation University of Illinois at Urbana- Champaign Urbana, Illinois 6180I August 197^+ This work was supported in part by the National Aeronautics and Space Administration and the EROS Program of the Department- of Interior through NASA Grant NGR 14-005-202, and in part by the Advanced Research Projects Agency of the Department of Defense through Contract DAHCOU-72-C-OOOI. ABSTRACT Considering synthetic patterns as the most elementary units of semantic information^ pattern encoding and decoding mechanisms are discussed as the basic components of semantic communication systems. Employing mathematical concepts long used "by social scientists for modeling spatial interaction patterns within complex social systems, a particular representation of pattern information is presented and, with respect to this notation, a numerical measure of pattern correlation is developed that permits transducer recognition of internally stored patterns maximally correlating with patterns received from an external, noisy information source. Possible generalizations of the model lead to brief discussion of the potential utility of the concepts employed within the development of macro theories of natural communication processes. Introduction Attempting to formalize in both philosophical and mathematical terms the primacy of information flow within systems exhibiting purposive behavior, Wiener termed cybernetics the study of all communication and control processes, whether mechanical, biologica], or social in nature. ' '^ Not independently of Wiener, Shannon established an outline adequate for conceptual analysis of any communication system and, distinguishing sharply between message semantics and message encodings, formulated a mathematical theory of transmission efficiency for communication channels linking mes- (k) sage encoding and decoding operations. Inspired by the utility and elegance of Shannon's concepts and mathematical formulations, Weaver (and others) immediately speculated that the contributions of Wiener and Shannon would generalize toward a more comprehensive theory of communi- (5) cations treating message semantics as well as transmission efficiencies. Despite considerable development of Wiener-Shannon information theory to (67) date,^ ^ '^ as well as parallel developments in psychology, linguistics, and language philosophy, ' ' there remains lacking any viable prin- ciple for integration of existing theory toward a conceptual framework of sufficient coherence to support hypotheses concerning semantic com- munications within natural systems. '. Employing Shannon's premise that "the fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point," below we revise Shannon's diagram of a general communication system to emphasize the nature 2 of the encoding and decoding operations that occur at either end of a conmunication channel. Messages conveyed from source to destination, however complex, are necessarily organized in terms of a finite vocab- ulary of semantic elements common to "both encoding and decoding oper- ations alike. Except for labels and schematic indications of information flow, the symmetry of the diagram again reflects the bi-directional nature of communications--processes occurring always in the presence of noise. Information / Source Information Destination G Encoder Decoder Figure 1. Shannon's schematic diagram of a general communication system. If we use the terra pattern generally to encompass all basic organizational units of semantic information (signs, symbols, characters, words, etc.), we may regard a fundamental goal of pattern Information processing research to be the specification of' formal models of the message encoding and decoding operations that enable semantic information transmission between message source and destination. A primary theoretical problem of message encoding requires a formal representation of quantized semantic patterns appropriate for message organization and transmission. A corresponding decoding problem requires specification of models of pattern information recognition of sufficient efficacy and generality to support at least theoretical estimation of the degree of system complexity necessary for message comprehension. At a minimum level of complexity, the problem requires definition of measures of pattern information correlation, similarity, or association that may be employed to effect accurate transducer addressing of those patterns internal to Its own vocabulary that best communicate the semantics of the information transmitted to it by an external message sovirce. The diversity of techniques currently employed as pattern recognition models makes difficult any simple typology of methodologies. (See references (ik) and (15).) . For the purposes of this paper, It will be sufficient to distinguish only two broad approaches to pattern recognition which we will refer to as analytic and synthetic. By analytic, we mean any methodology for which pattern classification depends either upon analysis of component pattern parts, attributes, or features (e.g., statistical methods), or upon analysis of structural relationships between pattern components (e.g., syntactic methods). By synthetic, we mean simply any methodology that treats patterns as Integral units of information and that categorizes patterns In accordance with similarities determined by methods of direct comparison (e.g., template matching techniques and Wldrow's "rubber-masking" approach.^ ' ) The inadequacies of traditional correlation techniques for quantification of pattern similarities have discouraged greatly to date the research and development of pattern recognition models "based on methods of direct comparison. > ^^ ' On the other hand, the rationale underlying the various analytic methodologies so widely adopted seems contrary to the basic notion of pattern gestalt-- that the whole is more than the sum of its parts. The perplexities of feature extraction and selection processes within so many analytic models of recognition may well attest to the synthetic nature of pattern information in general. One purpose of this paper is to suggest that pattern information processing research may have abandoned too soon the search for quantitative measures of synthetic pattern similarity. Relying greatly on mathematical concepts long employed by social scientists for modeling certain aspects of organization exhibited by systems of social interaction within community spatial structure, be- low we suggest a particular representation of patterns as spatial dis- tributions of information and develop a mathematical index of pattern information correlation expressible for any two patterns so defined. The measure of pattern correlation is first developed geometrically for pictorial patterns as a coefficient of configuration similarity that may be estimated invariant with respect to individual pattern intensities, sizes, positions, and proximate orientations. This particular mathematical formulation of pattern correlation then suggests a general network model of pattern information processing that allows representation of pattern associa- tions as network distributions of information. These generalizations lead to consideration of the particular conabination of concepts employed as a mathematical basis for macro theories of information processing within natural communication systems. Patterns and Pattern Dissimilarities While the generality surrounding the concept of a pattern makes difficult any single verbal definition, to assist the present mathematical discussion we offer the following: a pattern is a collection of quantized information distributed over a set of spatial elements characterizing some more complex information source. If we adopt at least provisionally the above definition, we may represent mathematically a particular pattern f as a partitioned array (w|x) , as tall as there are elements of f, where W is a matrix of coordinates indicating the relative spatial positions of all elements of f and X is a vector of positive reals indicating the propor- tional distribution of quantized units of information — pattern quits (quantization u nits ) — across all pattern elements. For mathematical convenience we will assume normalization of pattern intensity so that T x. =1. Thus X may be considered a discrete i 1 probability distribution of pattern quits over pattern elements. Also, we will assume normalization of patterns with respect to positions and sizes so that pattern centroids are coincident at a common origin, ) .x, w ,, = for each spatial dimension k, and second moments about the origin equal unity, With such mathematical notation we consider the following geometric pattern recognition problem: given a set of normalized patterns F, determine a symmetric non-negative scalar index of pattern dissimilarity, ^^ ^ = 6[(w|x) , (y|z) ], pairwise computable for all feF and geF, such that -■-SB I g A approaches zero as the similarity of f and g increases. Below we develop such a criterion of pattern dissimilarity that is by normalization invariant with respect to individual pattern intensities, sizes, and positions and by computation optimal with respect to small displacements of individual pattern orientations. For any two normalized patterns f and g we establish an index of pattern dissimilarity in the following manner. We determine simultaneously a weighted correspondence of elements between f and g and a registration of f with respect to g such that there exists maximal spatial congruence of corresponding elements between f and g. We then take as our criterion of pattern dissimilarity the weighted sum, of squared distances between all pairs of elements corresponding between f and g where the weights of the sum reflect the degree of correspondence between each element pair. Let the two patterns f and g consist of m and n elements respectively and let their normalized representations be denoted (w| X) „ and (y|z) . We represent a particular weighted correspondence of elements between f and g as a matrix Q (m^n) satisfying m (i; L q. . = z. i i;0 J a = ^, ..., n n (2) ^^1,0^=^1 i = 1, . . ., m (3) q. . > i = 1, .. ., m J = 1, . . . , n . LetiTT denote the set of all Q, matrices satisfying (l), (2), and (3^ for given X of f and Z of g. Now by normalization of pattern intensities E.x. - Y,.z . = 1, hence E. >j.q. . - 1. Since also q. . > for all i and j, we may consider any Q e 7t„ a discrete bivariate probability density function characterizing a joint distribution of corresponding quits over the elements of f and the elements of g. Let a particular spatial registration of f with respect to g be denoted oWR + JT' where J is the vector (l, ...» 1 )', T is a m translation vector, a is a scale factoij and R is a rotational trans- formation. For a given registration of f with respect to g, let S (m X n) be the resulting matrix of squared Euclidean distances between the elements of f and the elements of g. j = 1,. .. ,n Let I denote the set of S matrices obtainable for f and g by (k) over all positive scalars a, all translations T, and all legitimate rotations R. Our criterion of pattern dissimilarity may then be formulated: m n (5) A = min I I ^' . s. = min tr(Q'S). le-n SeJ i j ^"^ ^"^ Qstt SeJ Note that such an index of pattern dissimilarity may be interpreted as a minimal mean squared error of registration between all corresponding pattern quits of f and g. , Now for Q given, consider all possible translations of f with respect to g and write m n (^' '^i|%.,p"i,.-V--j,.i' To determine the particular T that minimizes A over all translations of f with respect to g, differentiate (6) with respect to t, to obtain m n from which it may be determined that t^ = where /. / . q. . w. , = l-l- % . y. , , or where L x. x-r. ^^ = L z y . This Implies that an optimal registration between f and g requires a coincidence of pattern centroids. Since by normalization pattern centroids already occur at a common origin and since these centroids remain invariant over pattern rotations and scale .changes, we may conclude that no further consideration of pattern translations is necessary in minimizing A Now consider all possible positive scale factors a applied to f so that m n Differentiating A with respect to a we find that the particular a minimizing (8) is given by m n m n p (9) o = ill -Ivy ]/[l I q ^.J ]. If o scales optimally W with respect to Y^ then by symmetry, a scales optimally Y with respect to W. By an identical analysis we may determine for a the expression -, m n m n (10) a-l= [lUi , IvU^^^l l-if, I^lk^l rI- 1 J 1 J ^ Henc eo = a- =1 where IJ^^ q^^^ l^ v.^j^ - IJ^ q^^j l^ t^^^, or where YyTw =yz. /nV^i- Again, by normalization this condition is ^i ^ ^k i,k ^j j ^k "^J.k ^ met, since pattern second moments about the origin are set to unity. Hence, in minimizing A we may also exclude all further consideration of pattern scales. The foregoing analysis demonstrates that the pattern positions and sizes yielded by normalization are optimal with respect to the minimal A independent of whatever correspondences Q are defined between the f >6 elements of f and g and independent of whatever rotation R may be chosen to effect maximum spatial congruence of corresponding elements of f and g. It remains to determine simultaneously an optimal correspondence matrix Q and a rotation R of f yielding an optimal registration of f and g such that the minimal A may be determined. In the following sections we present a computational solution to (5) via Iterative solution two interdependent subproblems — the correspondences problem, which requires minimization of A over all ■*" • ft Qeir for fixed S, and the transformation problem, which requires f >8 determination of a rotation R that minimizes A„ over all SeT for f»g ijg fixed Q. The Correspondences Problem Consider f and g as two patterns taken from a set of quantized patterns that may be assumed to be normalized with respect to Intensity, position, and size and standardized with respect to orientation, e.g., a font recognition problem. Since In such a case we may assume no rotational freedom for either pattern, we may consider I to consist of a single S matrix determined directly by 10 j = 1, . . . , n. In such an instance, our formulation of pattern dissimilarity reduces to m n (12) A = rain T y q. . s. . = min tr (Q'S) f,g I ^. i,J i,J where again ti^ is the set of all Q matrices satisfying (l), (2), and (3). 1 »g The optimization prohlem given by (l), (2), (3), and (l2) , where y X = y z = Y but Y J^o't necessarily unity, may be recognized as the ^i 1 ^J J (21 22 2^) Hitchcock or transportation problem of linear programming. ' ■> ~>J Typically, the problem requires determination of a matching between a spatially distributed set of economic supplies and a spatially distributed set of demands such that the total cost of all material movements from suppliers to buyers is minimal. For such problems, computational algorithms are well known and solution properties well documented. While any of these algorithms may be used for computation . of A in the present case, the characteristics and computational f ,g requirements of pattern information processing in general compel us to look further for mathematical techniques more appropriate for parallel processing systems. A related problem known to transportation planners is called (2^ 25) the entropy network distribution problem . ' The problem arises where it is desirable to simulate traffic flows within a metropolitan region given data describing distributions of populations and economic activities over some set of analysis zones subdividing the region, zone- to-zone travel times, and estimates of mean travel times for specific 11 types of trips within the region. Borrowing the notation of our pattern correspondences problem, let A represent the mean travel time for all home-work commuting trips, let X be the probability distribution of workers over m residential zones, let Z be the distribution of jobs over n employment zones, and let S be a matrix of network travel times between any residential zone and any employment zone. The problem requires determination of a most probable, mean, or maximum entropy joint probability distribution Q with marginals X and Z such that each element q. repre- sents the forecasted proportion of all trips occurring between the i-th residential zone and the j-th employment zone. Mathematically, the problem is formulated m n (13) max H = - J J^ q log q i j ^"^ ^"J subject to (l), (2), and (3) and the additional mean travel time constraint m n Note that constraint (l^) may be considered simply an a priori specification of overall network distribution efficiency or energy expenditure. The solution to the problem is given by ^15^ "li.j = \ \ ^j ^j ^^P ^- ^ ^i.,j^ ^ = ^' •••' ^ j = 1 , . . . , n where 3 represents the Lagrange multiplier associated with constraint (ik) and the u and v are functions of the Lagrange multipliers associated with constraint sets (l) and (2). It has been shown that corresponding 12 to any real 3 there exists a unique Q maximizing (13 ) and satisfying (l), (2), and (3) given by (15) where the parameters u. and v. may be J determined by iterative solution of the equations (l6) u. =[y z.v. exp (-3s. .)] i=l, ...,m ^ 1 (IT) V. = il X. u exp (- 3 s. .)] j = 1, ..., n. .1 X -L 1 % fl i Additionally, it has been shown that there exists a monotonia mapping between all 3 and all feasible A such that as 3 approaches -oo^ A approaches A , and as 3 approaches +°°, A approaches A . , where A and A . max mm max mm respectively denote the maximum and minimum values of A possible for given (27 28) S, X, and Z. * Together these results yield theoretical justification for iterative determination of the unique Q maximizing (l3) and satisfying the network distribution efficiency constraint (lU) as well as constraints (l), (2), and (3). Since A . of the entropy network distribution problem mm ^'' -^ is analogous to the pattern dissimilarity criterion of o\ar present pattern recognition problem, these results also imply that we may at least theoreti- cally formulate a maximum entropy correspondence matrix Q*, unique and optimal with respect to the minimal A , via equations (15), (l6!^ and (iT) with the parameter 3 set to °°. (29 ) Using theorems developed elsewhere and well known properties (28 ) . ^ . of the Hitchcock model Evans demonstrates several features of the matrix Q* and suggests a strategy by which it may be computed. Let E denote a binary matrix such that e. . = 1 for all subscript pairs (i,j) where q* > and e. =0 elsewhere. (Properties of- the Hitchcock model imply that E will be sparse). The desired Q* and the matrix E then interrelate arithmetically in the form 13 (18) q*_j - x^ u* z^ V* e^_j where the u. and v^ satisfy the relations (19) u* = i[.j ^j n.jr^ i = i. •••.■» * "^ * 1 (20) V = [I X u e ]■ J = 1, ..., n. J -;-'--'■ 1 » J Evans' method for exposing E, and hence Q*, requires initially a solution to the Hitchcock problem (l2) determined presumably by traditional methods. If for our particular entropy network problem ve may assume that convergence of •(l5)j (l6), and (IT) is well-conditioned for large 3, then a matrix Q* may be computed heuristically in the following manner. Establish the matrix E such that e. = exp(- 3s. ) where S is S scaled linearly to have elements within a specified interval and 3 is chosen as large as computational considerations permit. Determine estimates of U* and V* via iterative solution of n ^ ^ (21) u = [I z V e r i = 1. ... ^ ffl ^ ^ _-, (22) -j = qx, u. e.^.r J=l, ... , m 1 and then estimate Q* via (23) q. . = X. u. z V e i = 1, ..., m i»J 1 1 j j i,j j = 1, . . . , n. Now assuming that 3 is sufficiently large such that Q is close to Q*, then we may expect small elements of Q to correspond to zero elements of Q*. Hence we may select for each q. some threshhold value. li^ say q. . = x. * z . and approach E by resetting e. _, = whenever q. . < q for any particular solution to (2l), (22), and (23)j that is, 1 5 J exp(- 6s. .)ifq. ._^q. . [ i = l. , m e . . ""'"^ ' if q < q I j = 1, ..., n. Note that resetting e. . = at any point is equivalent to the assumption that pattern elements i and j have been determined non-corresponding and thus q =0. Since initially e. . is taken always as exp(- 6s. .), i,j ^'^ ^ ^"^ this assumption is also equivalent to the condition that s. .is large relative to other elements of S, hence e. . approaches zero faster than other elements of E as 6 approaches °°. Iterative solution of (2l), (22), (23), and (2U) in the above manner should lead to stable values for Q and E. At convergence, we may equate heuristically Q and Q*. Setting all remaining positive elements of E to 1 we have a similar approximation of E. Q* and E should be related via (l8), (l9), and (20). We have gained some evidence that such a procedure is sufficient for determination of optimal pattern element correspondences, at least where elements of S are taken as squared Euclidean planar distances and m and n are small. While there is the possibility that the above procedure will prove ineffective (or inefficient) in other cases, the properties of the Hitchcock and entropy network models offer much promise for the development of parallel algorithms practical for specific applications. While we have indicated a possible course for the development of such algorithms, the primary intent of this section has been simply to introduce the conceptual utility of the entropy network model to pattern information processing research. 15 In this context. It ahoiild &lio b*. noted that minljnu»-dl stance classification of patterns using the distance criterion(5) does not necessarily require computation of the limiting Q . For most applications, we have found that a Q matrix determined directly via equations (l5), (l6) and (17), "with 3 set to some large positive integer, yields approximate values of pattern dissimilarities via (5) completely satisfactory for recognition purposes. The Transformation Problem Now assume f and g taken from a set of patterns normalized with respect to intensity, position, and size but not standardized with respect to orientation. On the contrary, in this section we will assume given quanti- zations of f and g, again (w|x) and (y|z) , to include small proper rotational o displacements of individual pattern orientations that must be determined and accounted for in the computation of A^ via (5). Therefore, in comparing f and g, our problem is to determine a rotation Re e , the set of all proper rotations, of one pattern with respect to the other, say f with respect to g, yielding an S via (k) such that A^ via (5) is minimal over all S€Z_p as well as all Qeit . This transformation problem may be solved in the following manner. Assuming small displacements of orientations for f and g, we may determine an initial correspondences matrix Q, via the method just presented using provisionally for S the value given by (U) . Then with Q, fixed, our new estimate of pattern dissimilarity may be formulated (25) A = min tr (Q'S) S€Z„ 16 where our problem is again to determine a rotation R yielding a new estimate of S via (k) stepwise optimal with respect to Q. Define Q, and _S as mn X mn diagonals such that .(26) (%,i'a2,2 •••' V,mn^= ^^1,1'%2' ••'' \,J and (27) (s^ ^ , s^ ^, . . . , s )= (s s^ „ . . . , s ) ^-1,1' —2,2' -mn, mn' 1^1 J- ,2 > ' m,n then we may express (25) alternatively as (28) . A - min tr (Q'3) = tr (QS) where the elements of S remain to be determined as a function of the unknown R. Also define W to be mn X k where the first row of W is repeated n times as the first n rows of W, the second row of W repeated as the next n rows of W" and so forth. Define Y to be mn X k where the entire matrix Y is simply repeated vertically m times. Now note that ordered diagonal elements of the matrix [ (WR-Y) (WR-Y)' ] are identically equal to the elements of S^, hence we may write (29) tr[(WR-Y)(WR-Y) '] = tr(s) and since Q_ is also diagonal we may restate (28) as (^°^ A = min trf Q(WR-Y) (WR-Y ) ' ] . Re0 -- - - - .1 1 jL Defining ^, such that Q ^Q,^ ^ Q, we may write 1 i_ (31) A.= min tr[^2(WR-Y)(¥R-Y)'92] ReG ind after manipulation, (32) , 1. i i i A = min tr [ ig2m-^^t)'{g;^m-^Y)] fow substitute W = Q^^ and Y = g^y into (32) to obtain (33) rhich may be written A = min tr [ (WR-Y)'(WR-Y)] Re0 (3ii) A = min tr (R'W'WR - 2R'W'Y + L'Y} Re0 )r, since the trace of a sum equals the sum of the traces, 17 (35) A = min [tr(R'W'WR) - 2tr(R'W'Y) + tr(Y'Y)]. Ree Now consider the first and last terms of (35). Clearly both are idependent of R. Furthermore, by normalization and our definitions E" W, Y, W and Y we may write 1 1 (36) tr(R'W'WR) = tr(W'W) = tr [ (qS^) ' (Q^f )] = H ^,- . I ^? v = ---- ij^'-Jk^'^ 1 k nd (37) tr(Y'Y) = tr[(Q2Y).(Q2Y)]= I I q^^j J yf^^ = I ^j I yf^k = ^' 18 1 1 Since also the middle term of (35) maybe written -2tr [R' (Q^l^) ' (Q Y)] = -2tr(R'W*Q,Y), we have shown that the transformation problem may be formulated equivalently as (38) A := 2 - 2 max tr(R'W'QY) Re0 or, (39) A = 2 - 2 max tr(R'W'QY). Re0 With reference to (39) we notice that solution of (5) is equivalent to solution of (^0) p = max tr (R'W'QY) Re0, QeiT^ f.g where there exist the inverse monotonic mappings A = 2 - 2P and P = 1 - 7^ A^ . Since A is formulated as a mean sum of squared f,g 2 ^^S f^g distances, its lower bound is zero hence, the upper bound for p is unity. We may thus refer to P„ as the pattern correlation of f and g and solve (^0) as an alternative to (5)' Now if we allow R to be any orthogonal transformation, i.e^ either a proper or improper rotation, the optimization problem given by any of the above formulations of our pattern transformation problem may be recognized ( 30 31 ) as the Procrustes proble m of psychometric s. ' The problem arises in factor analysis and multi-dimensional scaling applications where it is desirable to compare two sets of factor coordinates independently determined for the same set of variables by rotation of one set to maximum spatial 19 congruence with the other to maximize between-set factor correlations. Mathematically, the problem is closely related to the canonical correlation problem of multivariate analysis. It has been shown then that for our present problem where W and Y and hence W, Y, W and Y are of full-column rank k, an optimal orthogonal transformation R is given by 1 (^1) R = (HL ^H') W'Y where H (k x k) and L (k x k diagonal) represent respectively the eigen- vectors and the eigenroots of the matrix (W'YY'W). ^ ^ Since both W and Y are of rank k, all roots of (W'YY'W) are positive and we may take as the elements of L the reciprocals of the positive square roots of (30) the elements of L. While it is true that computation of R via (^1) optimizes A over all orthogonal transformations, definition of element correspondences Q via (U) (or via (4) where R has been chosen as a proper rotation) makes it extremely improbable that the maximum of tr(R'W'Q,Y) will now occur for an improper rotation--that is, a reflection of f about some axis as well as a proper rotation of f with respect to g. Exceptions to this rule occur when comparing patterns whose coordinate matrices, W or Y, are only weakly of full-column rank, i.e. patterns of nominal full-column rank k whose spatial geometries can be accomodated with only slight distortipn in a subspace of dimensionality k ' < k. For example, where two patterns being compared represent quantized left and right parentheses, "(" and ")", and where Q, has been given by (ll), we would expect the R given by (^1) to contain a horizontal reflection. On the other hand, if the two patterns being 20 compared are "M" and "W", both patterns strongly two-dimensional, we would not expect an R computed via the same method to contain a vertical reflection. In any case, where pattern reflections are significant, det R may be computed to detect improper rotations and further action may be undertaken appropriate to the specific application. The Procrustes formulation of our pattern transformation problem is interesting, since it provides a general solution applicable for comparison of patterns of any dimensionality. In the case of planar pictorial patterns, however, the problem may be resolved in a more direct fashion. Restricting R to be a proper rotation, let C = (W'QY) for a given Q and write max tr(R'W'QY) in (39) as fi42) f(a) = max tr a cos a - sin a sm a cos a c c ^ 1,1 1,2 c c 2,1 2,2 or, equivalent ly, (^3) f(a) = max (c^ 2 " ^2 I'* ^^^ ^ ^ *^^1 1 "^ ^2 2"^ ^°^ ^ o. ' ' ' ' Then let A - (c^ 2 " ^2 l'^ ^"^^ ^ " ^^1 1 ^ ^2 2^ ^"^^ -write (^3) as (^^) f(a) = max (a sin a + B cos a). a 1 2 2 2 Also, let K = (a + B ) so that A = K sin ^ and B = K cos cj) and (45) f(a) = max (K sin ^ sin a + K cos cj) cosa) = max [K cos (a - (}>)]. a ^ 2 p ^ The maximum occurs when a = ((> and is equal to K = [(c - c ) + (c -i+'^o 2^"^ 21 The proper rotation maximizing (39 ) is then determined by the relations sin a = A/K = (c^ ^ " ^2 1^^^ ^^^ ^°^ a = B/K = (c^ i + ^2 2''/^' ^^^^efore, in comparing any two planar patterns f and g, a proper rotation R, stepwise optimal with respect to a given correspondence matrix Q, can always be « determined directly as a function of the four elements of the matrix C = (W'QY). In the last section, we presented a method for determining Q, optimal with respect to an assumed S. In this section, we have shown how an optimal transformation R, and hence an optimal S, may be determined with respect to a given Q. Since both subproblems are formulated to optimize the same criterion A^ (or P„ ), iterative solution of both yields a value of A_p optimal at least locally over all SeZ^ and all Qeir . Thus given a set of quantized patterns G for which rotational displacements can "be assumed small, a numerically expressible procedure exists for determining A^ for all f e G and all g e G. The Present Context of Developing Communications Theory The particular approach to pattern communication systems presented above results from a cross-fertilization of two separate applied mathematics research efforts conducted by the author in the areas of metropolitan land use-transportation systems modeling and computer-assisted earth resources image analysis. Such a cross-fertilization of seemingly divergent efforts has been made possible largely by the research environment provided by the Center for Advanced Computation (CAC) of the University of Illinois at Urbana- Champaign — a widely diversified interdisciplinary team of scientists and technicians seeking appropriate applications of emerging 1 1 i 1 >4. > 1 1 4 \ / / 8 <4 i e \ L. 4°J° 'i '^ ^- ';' J". C v". <^^ :5 •J J"8 .-J 3 B J H H S^ i H 1 9 (DOO 8 S M 1/ t 4 * 1 8 H H 14 H 5 <•:> ODCP h 5 5 s ■3 5 S s C 6 Xi 8"° 6 s S 6- 6"° °0 % ■?; °7 8 »• % ,«»>»«* 7 f' !' 7 "> ? -I \ 8 8.^ % 6' J'" 8% i'i P t* O' 9 C4 :P •5> 9 s .9 ^ 6°°o 5' 5 ^»^5 ° 8 c» \} Q c7 % (1 // i'"8 8 S '5- \b J 6 Figure 2. To evaluate the above numerical procedures, sixteen noisy- copies of ten prototype patterns (extreme left) were generated. Despite considerable rotational displacements, all l6o patterns were correctly classified using a Fortran implementation of the above techniques. Figure 3' A computer graphic showing the rank order of prototype-to-pattern dissimilarity measures computed for the sixteen noisy "6's" of Figure 2. Individual blocks have been plotted proportional to 1/a^ • Also, proto- types have been ordered from left to right in accordance' with overall prototype similarity with all example patterns depicted. 22 information processing and conaaunication technologies to problems of social concern. That commionication systems , especially in the form of networks of transportation facilities, represent the prime determinants of regional and urban development patterns has long been emphasized by geographers and spatial economists. Since Wiener and Shannon,, moreover, commimicatlon theory concepts have been embraced enthusiastically by urban analysts, both for the construction of descriptive theories of communities as evolutionary socioeconomic spatial systems and for the development of normative planning methodologies through which society may actively influence the determination of its future environments. Likening the various forms of social interaction that bind together the community to the bonding forces that organize atoms within molec\£Les, molecules within cells, and cells within organisms, Meier has proposed description of the community as a complex, self-organizing open system whose viability depends throughout on a hierarchy of vital processes struggling to conserve negative entropy. ~^ Emphasizing the global impact of twentieth-century transportation and communication technologies, and the socioeconomic specialization that they afford, Webber has argued that our communities can now only be viewed as cultural clusters of individuals intersecting at nxjmerous levels within a world hierarchy of urbanization and that, within any comprehensive description of our communities, non- spatial structures of social, economic^ and institutional activity realms i3h) must become as important as our past notions of geographic places . Exploring in depth the process of creative environmental design, Alexander has proposed a systematic methodology for planners that requires 23 explicit a priori specification of all decision criteria and, where appropriate, exploits the neutrality of the computer in organizing with respect to stated values the planner's progression from problem analysis (35) to solution synthesis. Within the scope of current efforts to model via computer simulation the sensitivity of community land developments patterns to public policy-making, and hence to enhance the rationality of (36) public decision-making with respect to future urban environments, the present author has pursued mathematical programming techniques that would allow a computer-assisted search for future urban land use-transportation alternatives maximally congruent with projected social activity and (37,38,39) communication systems. It was in this context that the above pattern correlation measures were developed. To date, the primary motivation for pattern information processing research at CAC has been a cooperative commitment with the Laboratory for Applications of Remote Sensing (LARS) of Purdue University to explore (^0) applications of the ILLIAC IV parallel computer at NASA's Ames Research (Ul) Center and the nationwide ARPA Network in support of the earth resources monitoring objectives of the ERTS/EROS programs of NASA and the U. S. (I42) Department of Interior. Multispectral image processing software previously researched at LARS has been redeveloped in parallel computation mode to allow evaluation of the potential efficiencies of ILLIAC IV interpretation of large quantities of natural resource, agricultural, and urban land use data now obtainable by current remote sensing technologies such as the Earth Resources Technology Satellite (ERTS). Additionally, we are exploring the practicalities of computer communications networking as a mechanism to insure decentralized access by government resource management 21^ agencies to whatever centralized data processing facilities may prove efficient. We hope to explore the applicability of the pattern correlation techniques presented above to a number of data processing tasks within this research. Throughout all of this work, we find ourselves living Wiener's thesis of I95O: ". . . that society can only be imderstood through a study of the messages and the communication facilities which belong to it; and that in the future development of these messages and communi- cation facilities, messages between man and machines, between machines and man, and between machine and machine, are destined to play an ever-increasing part. " ( 3 j The mechanical Information processing and communication technologies in service to man today rest firmly on the foundations of encoded Information theory established by Wiener and Shannon more than two and a half decades ago. Much of our current pattern information processing research is motivated by a continued enthusiasm to comprehend better the nature of human communication processes in order to be able to program the machine to simulate — in some primitive fashion — under- standing of man's own communications and thus to make convenient the machine's subservience to man on man's own terms. A more important outcome of this research, we think, will be a more complete appreciation for the unique characteristics and capacities of human communications and an acute recognition of the primacy of human communications for the continued realization of the potentialities of man. From the standpoint of modern psychology, the present model of pattern commimications seems promising in that it suggests possible new 25 approaches for development of macro mathematical theories of the information encoding, transmission, and decoding processes that take place within our own brains and hence control the complex organization of pattern information we experience as language, thought, and awareness. Since our model defines patterns as distributions of quantized information over sets of discrete, information storage elements, by- analogy, we may speculate the pattern information of the brain to exist as pattern-specific biochemical substances distributed throughout intricate networks of neurons. Such a conceptualization of information within the brain would seem apt not only for representation of afferent and efferent memory traces within the modal centers of the brain, but also for repre- sentation of complex internuncial associations of patterns within and between modal areas throughout highly structured pathways as well as random cortical networks. The possibility that the processes of information organization within the brain that give rise to language and thought, however parallel and complex, might be scientifically demonstrated to be governed by some dynamic compromise between the natural laws of energy conservation and entropy maximization — concepts so fundamental to our present understanding of the physical world around us and becoming increasingly important for our description of our social environment — is indeed exciting. Such a demonstration should contribute greatly toward man's confidence in the uniqueness within nature of his own capacity for information acquisition, assimilation, and organization--pattern synthesis--a capacity then mathematically expressible only as some joint product of the factorial complexity of possible synaptic transmissions and the enormous capacities 26 for information coding particular biochemical substances within the brain (i^3,UU,i+5) are known to process. Referencing continually throughout his own work the ideas of men such as Leibniz, Lock^ and Bergson, Wiener viewed our present paradigm of cybernetics simply as a logical continuation of man's universal inquiry into the nature of his own being and his destiny. Suimnarizing a succession of individual perspectives on human commimication, Matson and Montagu have more recently termed this inquiry the "unfinished revolution" of communications. That present communication technologies now make practical for this inquiry a platform of planetairy dimensions, and that our theories of community and communications continue to develop in harmony, perhaps both as extensions of a natural psychology of man, give us much hope that this inquiry will be carried into the future by and for a global order of man, an order continually rising above a diversity of cultural languages and national institutions. As we seek for our own present generation participation within this social dialogue, we must proceed with a reverence for human commimication similar to that expressed by Dewey: "Communication is uniquely instrumental and uniquely final. It is Instrumental as liberating us from the otherwise overwhelming pressure of events and enabling us to live in a world of things that have meaning. It is final as a sharing in the objects and arts precious to a community, a sharing whereby meanings are enhanced, deepened and solidified in the sense of communion. . .When the instru- mental and final functions of communication live together in experience, there exists an intelligence which is the method and reward of the common life, and a- society worthy to command affection, admiration, and loyalty, "^^i-' 27 Acknowledgements The author wishes to express his appreciation for the Interdisciplinary support provided by numerous colleagues at the Center for Advanced Computation in the preparation of this paper. Special acknowledgements are due Professor Daniel L. Slotnick for numerous discussions concerning communication systems in general, as well as for introducing the author to the potential advantages of parallel processing concepts for social systems modeling. The Algorithm Development Group of CAC offered much assistance with mathematical formulations and notation. For example, the special solution to the planar orthogonal rotation problem was pointed out to the author "by Dr. George B. Purdy. 28 References 1. Rosenblueth, A., Wiener, N., and Bigelow, J., "Behavior, Purpose, and Teleology, " Philosophy of Science, Vol. 10, 19^3, pp. l8-2l^■. 2. Wiener, N., Cybernetics, The Technology Press of M.I.T. and John Wiley and Sons, Inc., New York, 19^1-8. 3. Wiener, N., The Human Use of Human Beings , Houghton Mifflin, Boston, 1950. h. Shannon, C E., "A Mathematical Theory of Communication, " Bell System Technical Journal, Vol. 27, 19^8, pp. 379-^23, 623-656. 5. Weaver, W., "Recent Contributions to the Mathematical Theory of Communication, " in Shannon, C E., and Weaver, W., The Mathematical Theory of Communication, University of Illinois Press, Urbana, 19^9* 6. Quastler, H. (ed.). Information Theory in Psychology (Proceedings of a Conference on the Estimation or Information Flow, Monticello, Illinois, July 5-9^ 195^)^ The Free Press, Glencoe, Illinois, 1955. 7' Garner, W. R., Uncertainty and Structure as Psychological Concepts, Wiley, New York, I962. 8. Miller, G. A., Galanter, E., and Pribram, K. H., Plans and the Structure of Behavior, Holt, Rinehart and Winston, Inc., New York, i960. 9. Miller, G. A., The Psychology of Communication , Basic Bopks, Inc., 1967. 10. Chomsky, N., Syntactic Structures, Mduton and Co., The Hague, 1957. 29 11. MorriS; C, Signification and Significance, The M.I.T. Press, Cambridge, 196^. 12. Cherry, C, On Human Communication: A Review, A Survey, and A Criticism (3rd ed.). The M.I.T. Press, Cambridge, 196ij-. 13. Pierce, J. R., "Communication, " Scientific American, Vol. 227, No. 3, September 1972, pp. 31-iil. - 14. Watanabe, S. (ed.). Methodologies of Pattern Recognition, Academic Press, New York, 1969. 15. Watanabe, S. (ed.). Frontiers of Pattern Recognition, Academic Press, New York, 1972. 16. Widrow, B., "The Rubber-Mask Technique -- Pattern Measurement and Analysis, " Journal of the Pattern Recognition Society, Vol. 5; No. 3, September I973, pp. I75-I97. 17. Widrow, B., "The Rubber-Mask Technique -- Pattern Storage and Recognition, " Journal of the Pattern Recognition Society, Vol. 5, No. 3, September I973, pp. 199-211. 18. Selfrldge, 0. G., and Neisser, U., "Pattern Recognition by Machine, " Scientific American, Vol. 203, No. I8, August I960, pp. 60-79i reprinted in Feigenbaum, E., and Feldman, J. (eds.). Computers and Thought , McGraw-Hill Book Co., New York, 1963* 19- Minsky, M., "Steps Toward Artificial Intelligence, " Proceedings of the Institute of Radio Engineers, Vol. k9, January I96I, pp. 8-3O; reprinted in Feigenbaum, E., and Feldman, J. (eds.). Computers and Thought, McGraw-Hill Book Co., New York, 1963* 30 20. Nagy, G., "State of the Art in Pattern Recognition, " Proceedings of the IEEE, Vol. 56, No. 5, May I968, pp. 836-862. 21. Hitchcock, F. L., "The Distribution of a Product from Several Sources to Numerous Localities, " Journal of Mathematics and Physics, Vol. 20, 19^1, pp. 22J^-230. 22. Dorfman, R., Samuelson, P. A., and Solow, R. M., Linear Programming and Economic Analysis, McGraw-Hill Book Co., New York, 1958* 23. Dantzig, G. B., Linear Programming and Extensions, Princeton University Press, Princeton, 1963* 2U. Wilson, A. G., Entropy in Urban and Regional Modelling, Pion Limited, London, 1970* 25. Potts, R. B., and Oliver, R. M., Flows in Transportation Networks, Academic Press, New York, 1972. 26. Evans, A. W., "Some Properties of Trip Distribution Methods, " Transportation Research, Vol. k, 1970^ PP« 19-36. 27. Evans, A. W., "The Calibration of Trip Distribution Models with Exponential or Similar Cost Functions, " Transportation Research, Vol. 5, 1971, pp. 15-38. 28. Evans, S. P., "A Relationship between the Gravity Model for Trip Distribution and the Transportation Problem in Linear Programming, " Transportation Research, Vol. 7, 1973, PP* 39-61. 29. Bacharach, M., Biproportional Matrices and Input-Output Change, Cambridge University Press, Cambridge, 1970. 30. Schonemann, P. H., "A Generalized Solution of the Orthogonal Procrustes Problem, " Psychometrika, Vol. 3I; No. 1, March I966, pp. 1-10. 31 31. Cliff, N., "Orthogonal Rotation to Congruence, " Psychometrika, Vol. 3I, No. 1, March I966, pp. 33-^2. 32. Green, B. F., "The Orthogonal Approximation of an Oblique Structure in Factor Analysis, " Psychometrika, Vol. I7, No. k, December 1952, pp. i^29-i^40. 33. Meier, R. L., A Communications Theory of Urban Growth, The M.I.T. Press, Cambridge, I962. 3^1-. Webber, M. M., "The Urban Place and the Nonplace Urban Realm, "in Webber, M. M., et al.. Explorations into Urban Structure, University of Pennsylvania Press, Philadelphia, 196^> PP« 79-153* 35- Alexander, C, Notes on the Synthesis of Form, Harvard University Press, Cambridge, I966. 36. Heramens, G. C. (ed.). Urban Development Models, Special Report 91} Highway Research Board, National Academy of Sciences, 2101 Consti- tution Avenue, Washington, D. C, I968. 37' Ray, R. M., "Heuristic Spatial Allocation, " Urban System Latoratory Working Paper, Department of City and Regional. Planning, Univer- sity of North Carolina At Chapel Hill, Chapel Hill, North Carolina 2751^, January I969. (mimeo) 38. Ray, R. M., "A Conceptual Framework for Urban Spatial Planning, " Urban System Laboratory Working Paper, Department of City and Regional Planning, University of North Carolina at Chapel Hill, North Carolina 2751^+, May 1970. (mimeo) 39- Ray, R. M., "A Generalized ILLIAC IV Urban Land Use Planning Model, " CAC Document No. 5I, Center for Advanced Computation, University of Illinois at Urbana- Champaign, Urbana, Illinois 6180I, October 1972. 32 40. Slotnick, D. L., "The Fastest Computer, " Scientific American^ Vol. 224, No. 2, February 1971^ PP- 76-87. ill. Roberts, L. G., and Wessler, B. D., "Computer Network Development to Achieve Resource Sharing, " AFIPS Proceedings, 1970 Spring Joint Computer Conference, Vol. 36, AFIPS Press, 1970, pp. 5^3- 5^9 • 42. Ray, R. M., Thomas, J. D., Donovan, W. E., and Swain, P. H., "Implementation of ILLIAC IV Algorithms for Multispectral Image Interpretation, Final Report, Covering the Period March 1973 - February 197^, " CAC Document No. 112, Center for Advanced Computation, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801, June 197^. k3' Koffka, K., Principles of Gestalt Psychology, Harcourt, Brace and World, Inc., New York, 1935- 44. Deese, J., The Structure of Associations in Language and Thought , John Hopkins Press, Baltimore, 1965* 45. Rose, S., The Conscious Brain, Alfred A. Knopf, Inc., New York, 1973. 46. Matson, F. W., and Montagu, A., "Introduction: The Unfinished Revolution, " in Matson, F. W., and Montagu, A., The Human Dialogue --Perspectives on Communication, The Free Press, New York, I967. 47- Dewey, J., Experience and Nature (2nd ed.), The Open Court Publishing Co., La Salle, Illinois, I929.