Summary by James R. Martin, Ph.D., CMA
Professor Emeritus, University of South Florida
Balanced Scorecard Main Page | Performance Measurements Main Page
The main purpose of this article, as stated by the authors, is to review the need for multiple performance criteria to measure long-run profitability, the potential difficulties of using such measures, and to present the analytic hierarchy process and its potential use in synthesizing multiple performance measures. The authors also describe a case and apply the analytic hierarchy process to illustrate the way in which it can be used to transform multiple criteria into a composite performance measure.
Multiple Performance Criteria vs. the Single Indicator
Performance evaluation has been a difficult problem for businesses that can be viewed from several perspectives.
1. The shareholders’, Board of Directors’ or CEO’s perspective: The perspective of the organization as a whole.
2. The upper management perspective: The perspective of
the organization or its segments as it pertains to the effectiveness
of the unit or the individual managing the unit.
3. The operational perspective: The perspective of the management and supervisors as it relates to the efficient
day-to-day operations.
Performance evaluation has been treated by management accountants as a by-product of their connection with the accounting system in the organization. If profit maximization is the main economic goal of the firm, then the profit measure will be the most important measure of performance. Kaplan (1984), believes that while financial measures are important, other measures such as product innovation, product leadership, employee skills and morale, or customer loyalty, can be much better indicators of future profitability than annual profits. The authors believe that a firm is effective in the long run only if it achieves its goals and that performance evaluation is closely connected to goal setting in that it feeds back information to the system on how well strategies are being implemented. Several possible reasons given by the authors for why businesses have not embraced the multiple performance criteria performance evaluation include:
1. The cost of designing and collecting multiple criteria is high,
2. Non-profit performance criteria are difficult to interpret and evaluate because of their subjectivity, and
3. A single comprehensive measure is often needed for decision-making purposes.
The Analytic Hierarchy Process
The authors define the analytic hierarchy process as a participation-oriented methodology that can aid coordination and synthesis of multiple measures. It models the way in which the human mind structures a complicated problem. It has been applied in a variety of decision and utility contexts to help groups arrive at decisions where varying values are held by group members. It is ideally suited to help resolve problems that arise when multiple criteria are used in performance evaluation.
An Example of the Analytic Hierarchy Process
The authors provide an example scenario where a senior executive must decide on which of three candidates to promote. The decision maker will compare three criteria (leadership, human relations, and financial management ability) in pairs to develop a ranking. The comparisons would be:
1. Is leadership more important than human relations skills for this job?
2. Is leadership more important than financial management ability for this job?
3. Are human relations skills more important than financial management ability for this job?
A ratio scale of 1 to 9 is added to the ordinal ranking provided by the responses to these questions to provide the relative importance of one criterion over another. The pairwise comparisons are summarized in a square matrix and the eigenvalues and preference vector are then computed to determine the relative ranking of the criteria. Then each of the candidates is compared on the basis of the criteria as follows:
1. Is candidate A superior to Candidate B in leadership skills?
2. Is candidate A superior to Candidate C in leadership skills?
3. Is candidate B superior to Candidate C in leadership skills?
Contributions of the Analytic Hierarchy Process to Performance Evaluation
The analytic hierarchy process can allow for multiple viewpoints to be incorporated into the priority ranking by being used at many levels of the organization. Its most important contribution to performance evaluation is that it provides a systematic approach for weighting performance to provide a comprehensive performance measure. This measure can be used to assess the overall performance of the organization, to rank organizations or segments of organizations, to serve as input to incentive compensation schemes, and as input into decisions about the organization.
Performance Evaluation and the Analytic Hierarchy Process - A Case Example
The authors provide an example of a hypothetical decentralized organization with division located in several countries. This decentralization meant the need for a means of monitoring and evaluating the division managers. The corporation had three indicators to assess divisional performance and to set bonuses:
1. Return on investment.
2. Net income per unit of labor.
3. Change in market share.
After analyzing these indicators, they decided to reaffirm the companies goals:
1. To sustain growth in market share through the production of quality products which meet customer needs.
2. To enhance profitability through efficient production and distribution of the products.
3. To provide a healthy working climate for employees.
4. To maintain the reputation of flexibility and innovation in the industry.
Since the divisions of the organization were located in several countries, the effects of foreign currency fluctuations had to be taken into consideration. They added six new performance indicator measures:
1. Operating margin ratio.
2. Proportion of net income contributed by foreign exchange gains or losses.
3. Percentage of employee turnover in the year.
4. Customer satisfaction.
5. Product and technology innovation.
6. Total operating cost variance.
To apply the analytic hierarchy process, a questionnaire was given to the division managers and two Divisional Vice-Presidents to compare the relative importance of the nine evaluation criteria. The results formed a 9 x 9 matrix from which the preference vector (eigenvalues) was computed. Based on the performance vector, the nine evaluation criteria were ranked as follows: profitability, product and technology innovation, hedging effectiveness, operating efficiency, marketing effectiveness, customer satisfaction, productivity, operating effectiveness, and employee morale.
After the priorities were established, the two vice presidents were asked to complete a second questionnaire which requires pairwise comparisons between the divisions on the basis of the actual values of each of the nine evaluation criteria separately.
The results of the analytic hierarchy process showed that one of the divisions was the best in terms of some factors while another was the best in terms of others. It was difficult from the information provided to determine which division has the best overall performance. These results were different from the company’s original evaluation scheme. The authors believe that the analytic hierarchy process experiment was successful for this firm even though the results were different from the company’s original three-factor weighted scheme.
Advantages of this Methodology
1. Both quantitative and qualitative measures can be included in the evaluation scheme.
2. It allows for multiple input in the setting of priorities of the evaluation criteria.
3. Subjectivity in setting priorities for evaluation criteria and assessing divisional performance is reduced.
4. Consistency in judgment is improved.
5. The evaluation of performance against standards can be incorporated in the process.
6. The resultant vector from the analytic hierarchy process provides a composite performance measure which can be used for other
purposes (e.g., the allocation of bonuses to divisions).
Limitations of This Methodology
1. It requires substantial time and effort.
2. It can be difficult and complicated to understand because of its quantitative nature.
3. Although it is systematic, it still requires managers to make subjective judgments about interpretation of the qualitative criteria.
4. The relative ranking of the original alternatives may be reversed when an identical alternative is added to the list.
____________________________________________________
Note: This study was somewhat confusing to us because the company sounds like a real organization although Chan and Lynn include the word hypothetical after the company name in parentheses on page 69. Kasanen, Lukka and Siitonen (1993) list this paper as a study in which a construction is developed further, but which lacks, in one way or the other, practical implementation. For a case study where the authors used the technique in an insurance company, see Chan, Y. L. and B. E. Lynn. 1993. Organizational effectiveness and competitive analysis: An analytic framework. Advances In Management Accounting (2): 85-108. I suppose the 1993 study passes the semi-strong market test of a managerial construction mentioned by Kasanen, Lukka and Siitonen on page 253 of their paper mentioned above.
Related summaries:
Bacidore, J. M., J. A. Boquist, T. T. Milbourn and A. V. Thakor. 1997. The search for the best financial performance measure. Financial Analysts Journal (May/June): 11-20. (Summary).
Clinton, B. D. and S. Chen. 1998. Do new performance measures measure up? Management Accounting (October): 38, 40-43. (Summary).
Dierks, P. A. and A. Patel. 1997. What is EVA, and how can it help your company? Management Accounting (November): 52-58. (Summary).
Epstein, M. J. and S. D. Young. 1999. Greening with EVA. Management Accounting (January): 45-49. (Summary).
Fera, N. 1997. Using shareholder value to evaluate strategic choices. Management Accounting (November): 47-51. (Summary).
Ittner, C. D. and D. F. Larcker. 1998. Innovations in performance measurement: Trends and research implications. Journal of Management Accounting Research (10): 205-238. (Summary).
Jalbert, T. and S. P. Landry. 2003. Which performance measurement is best for your company? Management Accounting Quarterly (Spring): 32-41.(Discussion of EVA, tracking stock and balanced scorecard). (Summary).
Lev, B. 2004. Sharpening the intangibles edge. Harvard Business Review (June): 109-116. (Summary).
Wallace, J. S. 1998. EVA® Financial systems: Management perspectives. Advances in Management Accounting (6): 1-15. (Summary).