UNIVERSITY OF ILLINOIS LIBRARY AT URBANA-CHAMPAIGN BOOKSTACK' ;s CENTRAL CIRCULATION BOOKSTACKS The person charging this material is re- sponsible for its renewal or its return to the library from which it was borrowed on or before the Latest Date stamped below. You may be charged a minimum fee of $75.00 for each lost book. Theft, mutilation, and underlining of books are reasons for disciplinary action and may result in dismissal from the University. TO RENEW CALL TELEPHONE CENTER, 333-8400 UNIVERSITY OF ILLINOIS LIBRARY AT URBANA-CHAMPAIGN JAM 2 1 1997 4 2000 When renewing by phone, write new due date below previous due date. L162 . %3£I"1;t... FACULTY WORKING PAPER NO, 11.27 On the Correspondence Between 'Primary' and 'Secondary' Measures of Business Economic Performance: An Attempt at Methodological Tnanguiatscn N. Venkatraman Vasudevan Ramanujam • WBWsrffjsaa? CoMegs of Commerce 3pg Business Administration Bureau of Economic mc Business Research University of Illinois, Urbana-Cnampaign DJL;J> FACULTY WORKING PAPEP NO. 1127 College of Commerce and Business Administration University of Illinois at Urbana- Champaign March, 1985 On the Correspondence Between 'Primary' and 'Secondary' Measures of Business Economic Performance: An Attempt at Methodological Triangulation N. Venkatraman, Visiting Lecturer Department of Business Administraion Vasudevan Ramanujam Case Western Reserve University An earlier version of this paper has been accepted for publication in the 1985 Midwest Academy of Management proceedings. Comments provided by Charles Schwenk and Meera Venkatraman on the earlier version, and Patrick Gaughan's assistance in data collection are greatly appreciated, Digitized by the Internet Archive in 2011 with funding from University of Illinois Urbana-Champaign http://www.archive.org/details/oncorrespondence1127venk On the Correspondence Between 'Primary' and 'Secondary' Measures of Business Economic Performance: An Attempt at Methodological Triangulation Abstract This study is an attempt at assessing method convergence between two different operationalizations of Business Economic Performance — viz., managers' assessment of their organization's relative perfor- mance and secondary analysis based on published sources. An evaluation of the convergence of three indicators of performance — sales growth, profit growth, and ROI — provided strong results, indi- cating that managers are generally not biased in their assessments. On the Correspondence Between 'Primary' and 'Secondary' Measures of Business Economic Performance: An Attempt at Methodological Triangulat ion Introduction Concern with enhancing organizational performance is at the heart or most organizational research (Campbell, 1977; Chakravarthy , 1984; Ford oc Schellenberg, 1982; Hofer, 1983; Kanter 6< 3ri nkerhof f , 1981; Kirchoff, 1977; Seashore & Yuchtman, 1967; Steers, 1975, 1977). Perhaps more than any other branch of the organizational sciences, the field of strategic management is centrally focused on issues of orga- nizational performance (Schendel & Hofer, 1979). A normative theory of strategic management awaits not only the clarification of the term "strategic performance," but also the development of reliable and valid approaches to the measurement of the performance construct. Two major issues need to be addressed in dealing with the opera- tionalization of performance construct. One pertains to the choice of an appropriate set of operational indicators reflecting the construct's domain, while the other is concerned with the method of data collection The first issue is plagued with controversies and debates (see espe- cially Campbell, 1977; Connolly, Conlon, & Deutsch, 1980; Goodman Ct Pennings, 1977; Hanna, Freeman, & Meyer, 1976; Steers, 1975; 1977), and we will have little to say on this here. In relation to tne second issue researchers typically are faced with the choice of obtaining performance data either from 'secondary* sources — i.e., data collected from sources external to the organization, or from 'primary' sources — i.e., data collected from the organizations themselves. While opera- tionalizations based on secondary data permit replicabili ty , primary data could introduce respondent bias and nay not serve the interests of replicabi li ty . On the other hand, secondary data on performance may not be available at the desired level of detail for some applications (e.g., for an SBU-level focus). .lost researchers choose one of these approaches but seldom provide evidence of convergence witn the other operationalization. Given that both approaches, wnen considered individually, may have questionable measurement properties, it is necessary to address the issue of "method convergence" (Campbell & Fiske, 1959) in operationalizing organiza- tional performance to ensure that the variance reflected is that of trait and not of method. Such attempts reflect the philosophy under- lying methodological triangulation (Denzin, 1978; Jick., 1979) — the use of complementary methods to enhance researcher's beiief in results. Thus, this study seeks to assess the extent of convergence in operationalizations of the construct of Business Economic Performance (BEP) by collecting data from two different methods — viz., primary data from organizations themselves, and secondary data from published sources, external to the organization. Such an approach reflects Campbell and Fiske's (1959) call for using 'maximally different' methods to assess convergent validity of operationalizations. As noted by Bouchard, con- vergence between two methods "ennances our belief that the results are valid and not a methodological artifact" (1976; p. 268). When maximally- differing methods are used, the approach is termed ' between-raethods' triangulation (Denzin, 1978; Jick, 1979), which rests on the assumption that the two methods do not share the same weakness or potential bias (Rohner, 1977). -3- Thus , this research attempts a methodological triangulation of the correspondence between two different operationalizations of BEP — managers' assessment of their organization's relative competitive per- formance and secondary analysis of relative performance based on published sources. Research Method Indicators of BEP : Three indicators — viz., sales growth, net income growth, and return on investment (ROI) — were chosen to reflect BEP. These three indicators correspond to the key dimensions of per- formance distilled by Woo and Willard (1983) based on their analysis of PIMS data — viz., (i) profitability; (ii) relative market position; (iii) change in profitability, and (iv) growth in sales and market share. Hofer (1983) also found these indicators to be among the most commonly used measures of BEP. Hence, an examination of the method convergence of these indicators should be of interest to strategy researchers operationalizing business performance. Primary measures : For each of the three indicators, managers were requested to indicate their positions, not of their absolute perfor- mance but their performance relative to their major competitors. This reflects the "relative" nature of the performance concept stressed by many, including the PIMS-based strategy studies. A five-point inter- val scale ranging from -2 (much worse than competition) to +2 (much better than competition) with the neutral point indicating a level of performance equal to that of competition was employed. Data were collected from senior-level managers (either presidents/vice presi- dents of functional areas or vice presidents of corporate planning) as a part of a larger project during Feoruary-May 1984. Alchough the larger project had a response rate of over 33£ (207 out of bOO), only 8b cases are used in this study. Since anonymity was to be ensured, the respondent's name, and corporate affiliation was voluntary. 8b respondents indicated their organizational affiliations which was necessary to collect secondary data on them. Table 1 lists some key characteristics of the sample employed in this study. INSERT TABLE 1 AiiOUT HERE Secondary measures : For each of the three indicators, secondary measures were assembled from Business Week magazine's "Inflation Scorecard" for the year 1983, as reported in the March 21, 1984 issue. Business Week compiles these data from Standard & Poor's COMPUSTAT tapes, and was a convenient and easily accessible source of data. Relative performance was operationalized as "firm performance relative to industry" — where industry referred to the principal SIC industry classification in which the firm was normally placed. It was measured as the difference between the value of the indicator for the firm and the industry. For example, relative sales growth was the sales growth of the focal firm minus the sales growth of its primary industry. Results Table 2 presents the descriptive statistics as well as the analy- sis in the form of Campbell and Fiske's MultiTrait, MultiMethod (MTMM) matrix which is one of the analytical schemes of methodological triangu- lation. Entries in the MTMM matrix are Pearson's Zero-order correlations. -5- INSERT TABLE 2 ABOUT riERE The first of Che four criteria of an MTMiM matrix (Campbell & Fisfce, 1959) refers to convergent validity and requires that all the diagonal coefficients in the lower left quadrant of the matrix (termed, "validity coefficients") be "sufficiently large" and statistically significant (Campbell 6 Fiske, 1959). Table 2 indicates that all the three valid- ity coefficients are greater than 0.4 and statistically significant at a p-level better than 0.01. The other three criteria relate to discriminant validity, viz., whether the three traits are different from one another or not. While these criteria are not directly relevant for the attempt at checking for correspondence, they imply that measures of different concepts should share little common variance, since a high level of covariation casts doubt on the uniqueness of measures and/or the concepts. The second criterion requires tnat each validity coefficient should be larger than the "different trait-different method" correlations (which are in the same row or column as the validity coefficients in the dashed triangles adjacent to the validity coefficient). As shown in Table 2, this condition is satisfied in all three cases. The third criterion requires that each validity coefficient should be larger than the "different trait-same method" correlations (which involve tne same variable as that of the validity coefficient in the lower right and upper left quadrants). This condition is satisfied in two of the four cases for the sales growth measure, in one out of the four cases for tne profit growth measure, and in three out of the four -o- cases for the RUI measure. The general support for this criterion appears to be "moderate." The fourth and final criterion requires that the pattern of corre- lations present in each of the four triangles (both solid and dashed) in the matrix should be similar. A test of this similarity can be accomplished by ranking the correlations in each triangle and deriving a measure of the rank, correlation across the triangles. J-'riedman two- way test was conducted for this purpose. Its associated chi-squared statistic was 6.50 (df=2), statistically significant at a _p value of 0.039. Thus, we conclude that the relative rankings of the correla- tions is preserved within the four triangles, thereoy satisfying the fourth criterion. Discussion The results (especially, the first criterion of the MTMiM matrix) indicate that there exists a strong degree of "method convergence" when performance data was obtained from two 'maximally different' methods. It appears that respondents tend to be less biased in their assessments of their organizational performance than researchers have tended to give them credit for. The main implication of our finding is that per- ceptual data from senior managers, which tend to correlate well with secondary data, can be employed as acceptable operationalizations of BEP. A previous study by Dess and Robinson (1984), using self-reported 'objective' data and suDjective assessments of two performance ndicators — return on assets and sales growtn, reported a close corre- dence between the two operationalizations. Their two approaches -7- are conceptually similar in the sense of employing data collected from only primary source, and represent 'within-method ' type of triangulation (Denzin, 1978). The limitations of this type of triangulation are noted by Denzin, "Observers delude themselves into believing that .. .different variations of the same method generate .. .distinct varieties of triangu- lated data. But the flaws that arise using one method remain...." (1978, pp. 3U1-302). in contrast, the present study, wnich reflects a ' between-method ' type of triangulation rests on the assumption that the weaknesses in each single method will be compensated oy the counter- balancing strengths of another. Thus, this study can De seen as a study which moves the operationalizations of BEP towards the 'between-methods ' approach to triangulation, which "allows researchers to be more confi- dent of their results" (Jick, 1979; p. 608). Although the study established correspondence across two maximally different methods, a potential limitation of this study should be recognized. Data for this study were collected from a single respondent in each responding unit. Hence, the possibility of functional or response bias cannot be entirely ruled out. It would have been desir- able to collect data from multiple managers within a unit so that inter- manager consistency could have been assessed. However, the size of the target population and resource limitations prompted us to trade off in favor of larger sample size rather than multiple responses per unit. Further, based on results obtained in earlier studies which have employed tne "multiple respondent design" (e.g., Dess & Robinson, 1984; Snow & Hreoiniak, 1980) it can be argued tnat tnere is generally less variability within raters of a particular firm than raters across firms. -8- Moreover, respondents in our sCudy were senior-level managers (e.g., vice president-strategic planning, president or functional vice president) who can be argued to be key members of the dominant coali- tion of tne firm, and tnus can be considered as "'representatives" of the organization. Thus, while Uess and Robinson's study addressed the measurement theme of inter-judge reliability in performance assessment, this study focused on a different measurement issue, viz., convergence across "maximally different methods" — which is a key requirement for construct validity of measures (Campbell 6 Fiske, 19 59 ) . Nevertheless, the issue of using multiple-respondents to measure organizational-level constructs such as BEP needs to be addressed by strategy researchers. The use of MTMM framework, enabled us to address a related issue of "uniqueness" of the tnree traits considered for operationalizing BEP. The support received for the three criteria of discriminant validity imply that the tnree indicators considered here tap different "traits" of SEP. This is in agreement with the findings of Woo and Millard (1983) — who employed a different data-analytic framework (factor analy- sis) and a different data base (the PIMS program) in arguing for a multi-dimensional operationalization of performance. However, the results of this study should not be taken to indicate that these are the only dimensions. Nor is it implied by us that these are tne key dimensions of organizational performance. Based on the results reported here and previous theoretical arguments (e.g., Campbell, 1977; Steers, 75; 1977) and empirical results (e.g., Woo & Willard, 1983), we argue that the use of any single indicator (dimension) to capture the rela- ely complex construct of performance should be viewed with disfavor. -9- hxtensions Measurement of organizational performance in general, and BEP in particular is central to research in strategic uanageraent . Towards this end, Woo and Willard's (19tf3) study, Dess and Kooinson's (1984) study and the results reported here are to be viewed as starting points for further refinement and extensions. Future research directions on performance measurement can be broadly grouped under three streams. One is to employ multiple managers cnosen to represent different func- tional areas, hierarchical level, or length of tenure with the company, since these variables may have an impact on the aDility of respondents to make complex judgments on assessing organizational performance. Given managers' differing f rames-of-ref erence , such an analysis could provide interesting and useful pointers for the choice of respondents in the design of field studies. Two , since the domain of organiza- tional performance extends beyond 8EP, similar methodological triangula- tion attempts to assess convergence should be undertaken for broader conceptualizations of organizational performance which include both financial and operational indicators. The third stream of extension relates to the issue of superiority of one operationalization over another. This is important since researchers examining convergence between methods to assess the quality of their operationalizations may elect to use one or the other, but not necessarily both. In this context, Dess and Robinson in concluding their study expressed their preference for 'objective' data by noting that subjective performance data are good substitutes -10- for objective data whenever "(.J.) accurate objective measures are un- available, and (2) trie alternative is to remove tne consideration of performance from the research design" (1984, p. 271). This was not based on any specific analysis of the super iori ty of one method over another, but merely reflects their note of caution. It should prove useful to systematically assess the relative superiority of one metnod over another, by analytical approaches such as the analysis of co- variance structures (Joreskog & Sorbum, 1979), which provides a basis to decompose the variance in measurement into key components such as trait, method, and random error. Strategy researchers in particular need to take cognizance of these conceptual and measurement issues in view of tne embryonic nature of their field. Although a paradigm of strategic management is at hand (Schendel & Hofer, 1979), a normative theory of strategic management can- not be developed unless the crucial issues of conceptualizing and measur- ing organizational performance are more fully researched and understood. Sumraarv Data on three commonly employed indicators of performance — sales growth, profit growth, and ROI — were collected by two different methods — (i) perceptual assessments by senior executives and (ii) secondary data sources. An evaluation of their convergence provided positive results, indicating that managers are generally not biased in their assessments of organizational performance. In addition, it was observed that those indicators tap different traits of performance, thus raising some important measurement issues in relation to the Lmensionality of organizational performance. -11- REFERENCES Bouchard, T. J. Jr. 197b. Unobtrusive measures: An inventory of uses. Sociological Methods and Research , 4:2t>7-300. Campbell, D. T. , & Fiske, D. W. 1959. Convergent and discriminant validation Dy the raultitrait multimethod matrix. Psychological bulletin , 56:81-105. Campbell, D. T. , & Stanley, J. C. 1963. Experimental and quasi- experimental designs for research . Boston: Houghton-Mifflin. Campbell, J. P. 1977. On the nature of organizational effectiveness. In Goodman, P. S., Pennings, J. M. , & Associates. New perspectives on organizational effectiveness . San Francisco: Jossey-Bass. Chakravarthy , B. S. 1984. Re-search of excellence: Resolving the performance puzzle. Working paper 84-02 , The Reginald H. Jones Center, The Wharton School, University of Pennsylvania. Connolly, T. , Conlon, E. J., & Deutsch, S. J. 1980. Organizational effectiveness: A multiple constituency approach. Academy of Management Review , 5:211-217. Denzin, N. K. 1978. The research act . 2nd Ed., New York: McGraw-Hill Dess, G. G. , & Robinson, K. B. 1984. Measuring organizational perfor- mance in the absence of objective measures: The case of the privately-held firm and the conglomerate business unit. Strategic Management Journal , 5:265-273. Ford, J. D., & Schellenberg , D. A. 1982. Conceptual issues of linkage in the assessment of organizational performance. Academy of Management Review , 7:49-58. Goodman, P. S., & Pennings, J. M. (Eds.) 1977. New perspectives on organizational effectiveness . San Francisco, CA: Jossey-Bass. Hannan, M. T., Freeman, J., & Meyer, J. W. 1976. Specification of models for organizational effectiveness. American Sociological Review , 41:136-143. Hofer, C. W. 1983. ROVA: A new measure for assessing organizational performance. In R. Lamb (Ed.). Advances in strategic management . Vol 2, 43-55. New York: JAI Press, Inc. Jick, T. D. 1979. Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly , 24: 602-611. -12- Joreskog, K. G., & SorDura, D. 1979. Advances in factor analysis and structural management . Mass.: Abt Books. Kanter, R. M. , & Brinkerhoff, D. 1981. Organizational performance: Recent developments in measurement. Annual Review of Sociology , 322-349. Kirchoft, B. 1977. Organization effectiveness measurement and policy researcn. Academy of Management Review , 2:347-355. Rohner, R. P. 1977. Advantages of the comparative method of anthro- pology. Behavioral Science Research , 12:117-144. Seashore, S. E., & Yuchtman, E. 1967. Factorial analysis of organiza- tional performance. Administrative Science Quarterly , 377-395. Schendel, D. E. , & Uofer, C. W. 1979. Strategic management: A new view of business policy and planning . Boston: Little, Brown & Company. Snow, C. C, & Hrebiniak, L. G. 1980. Strategy, distinctive com- petence, and organizational performance. Administrative Science quarterly , 25:317-335. Steers, R. 1975. Problems in the measurement of organizational effec- tiveness. Administrative Science Quarterly , 20:546-558. Steers, R. 1977. Organizational effectiveness: A behavioral view . The Goodyear series in managemenc and organizations. Santa Monica, Calif.: Goodyear Publishing. Woo, C. Y., & Willard, G. 1983. Performance representation in strate- gic management research: Discussions and recommendations. Paper pre- sented at the Academy of Management meeting in Dallas. D/296 ■■> ,4 a 2, ,4 4, .7 4, .7 35, .9 -13- Table 1 Key Characteristics of the Sasple (n=8b) 1 . Sales level $ 5U - 100 Million $101 - 250 Million $251 - 500 Million $501 Million - $ 1 Billion over $ 1 Billion 2. Industry Category Consumer Goods 21.2 Capital Goods 31.8 Raw or serai-finished materials 22.4 Components for finished goods 12.9 Service 11.8 3. Respondent's Responsibility Staff responsibility (e.g., V.P. - Strategic Planning) 79.1 Operating responsibility (e.g., President or Functional V.P.s) 20.9 a All figures are percentages. Non-responses are excluded in the percentage calculations. -14- Table 2 Primary Versus Secondary Measures of Business Economic Performance: An MTMM Analysis PRIMARY SECONDARY DESCRIPTIVE STATISTICS Primary SG a PC RO [ SG PG RO I MEAN SD SC PG ROI L.OO 0.47\ 1.00 0.36 0.74\l.0O 3.32 3.47 3.24 0.91 1.05 1.06 Secondary — SG kp«44 s v 0, i4 0.151 1 N N ' 1 \ v 1 PG ' 0.32 s 0.42M).33 ! 1 \ x , ' \ N ROI |0.10 0.3b n0. 51 U.OO 0.69 \ 1.00 0.02 0.28\ 1.00 1.13 (-)2.64 (-)0.54 12.54 21.71 9.96 _ , — ■ __^_____^______ __^ Convergent Validity: Criterion 1 All validity coefficients (SG: 0.44; PG; 0.42; ROI: 0.51) are statistically significant at p<0.01. Discriminant Validitv: Criteria 2 and 3 Validity Coefficient 0.44 0.42 0.51 Criterion 4: Criterion 2 Z satisfied 100 100 100 Criterion 3 % satisfied 50 25 75 Chi-squared statistic for Friedman's non-parametric test for the rankings of the correlations within the four triangles: 6.50 (df=2) , statistically significant at p=0.039. Sales growth; PG: Profit (net income) growth; ROI: Return on itment. Entries in the matrix are Pearson's zero-order correlations. mary data are based on five-point Likert-type scale, secondary data are actual values. HECKMAN |±| BINDERY INC. |g| JUN95 tmd -To .PI M5( ? N.MANCHESTER INDIANA 46962 '