Martin, J. R. 1998. Evaluating faculty based on student opinions: Problems, implications and recommendations from Deming’s theory of management perspective. Issues in Accounting Education (November): 1079-1094.

Summary by James R. Martin, Ph.D., CMA
Professor Emeritus, University of South Florida

Deming Main Page | Education Issues Main Page

The purpose of this paper is to present the case against using student opinions to evaluate teaching effectiveness and to recommend an alternative method consistent with Deming’s theory of management.¹ Briefly, the thrust of the argument is that student opinions should not be used as the basis for evaluating teaching effectiveness because these aggregated opinions are invalid measures of quality teaching, provide no empirical evidence in this regard, are incomparable across different courses and different faculty members, promote faculty gaming and competition, tend to distract all participants and observers from the learning mission of the university, and insure the sub-optimization and further decline of the higher education system. Using student opinions to evaluate, compare and subsequently rank faculty members represents a severe form of a problem Deming referred to as a deadly disease of western style management. The theme of the alternative approach is that learning on a program-wide basis should be the primary consideration in the evaluation of teaching effectiveness. Emphasis should shift from student opinion surveys to the development and assessment of program-wide learning outcomes. To achieve this shift in emphasis, the university performance measurement system needs to be redesigned to motivate faculty members to become part of an integrated learning development and assessment team, rather than a group of independent contractors competing for individual rewards.

This paper is organized into five sections. The first three sections add more specificity concerning the deficiencies of student opinion surveys as outlined above. The fourth section includes a discussion of the underlying causes of the misuse of student evaluations and the final section provides a set of recommendations concerning systemic changes in higher education derived from Deming’s theory of management perspective.

The Invalidity of Student Evaluations

The first step in selecting and validating a measurement is to develop an operational definition of the construct or phenomenon to be measured (Bipboye, Smith and Howell 1994, 59). This includes identifying the desired characteristics (e.g., inputs, processes and outputs) associated with the phenomenon involved. Subsequent steps include identifying and examining the relationships between possible measurements and these characteristics, as well as the relationships between the measurements and the characteristics of other related phenomenon. The purpose of these steps is to develop a theory concerning the relationship between the phenomenon to be measured and the measurements. For example, the underlying theory of using student opinions to evaluate faculty teaching effectiveness is that the characteristics of high quality teaching cause high student ratings and that the characteristics of low quality teaching cause low student ratings. However, if some of the characteristics of high quality teaching tend to cause low student ratings and some of the characteristics of low quality teaching tend to cause high student ratings, the theory is flawed and the measurement is invalid. Examining student evaluations from the perspective outlined above helps reveal several problems related to the invalidity of these measurements.

First, although the American Assembly of Collegiate Schools of Business (AACSB 1994, 47) requires assessments of teaching that ensure the effectiveness of instruction, and a five-dimensional model of effective teaching has been developed by the Accounting Education Change Commission and modified by an AAA committee (Calderon, Gabbin and Green, 1996), an explicit operational definition of high quality teaching does not exist. In other words, the very first step in the process of validation is missing. However, the use of student opinions to evaluate faculty implies that the mean student opinion represents the evaluator’s implicit definition of quality teaching. The underlying assumption is that collectively, any group of students will accurately define and definitively measure quality teaching, automatically.

Second, even if an explicit comprehensive operational definition of high quality teaching can be developed, it cannot be fully represented on a student evaluation form because students are not qualified to judge many of the aspects involved.² For example, what was the depth of student knowledge obtained in this class? Is that knowledge current, relevant, integrated and applicable to the solution of problems within the discipline studied? How long will the students retain this knowledge? Do the students understand the underlying theory and issues associated with the topics covered? Did the students enhance their critical thinking ability, creativity, oral and written communication skills, ability to work together and ability to learn? Did the instructor adequately conceptualize, organize and sequence the subject matter? Does the instructor keep up with the profession and relevant research in the areas associated with this class? Did the instructor integrate this course with related courses, disciplines and current research? Does the instructor share his or her teaching methods, techniques and successes with other instructors through publications and presentations at academic meetings? ³ Obviously students cannot answer these questions.

A third problem with student evaluations has been identified by a variety of researchers (e.g., Newton 1988; Wright et. al. 1984; Crumbley 1995). As mentioned above, student evaluations are not valid if some characteristics of high quality teaching tend to cause low student ratings and some characteristics of low quality teaching tend to cause high student ratings. For example, eliminating the most challenging material from a course, or from the exams, and generously awarding partial credit for incorrect answers may tend to generate higher student ratings, but it is doubtful that many faculty members would classify these actions as indicative of high quality teaching.⁴ Similarly, providing entertaining lectures without much, if any substance, may produce high student ratings as in the Dr. Fox episode⁵ , but this cannot logically be defined as high quality teaching. In fact, entertaining lectures, with or without much substance, may produce high student ratings in any class, but the "knowing-all-telling-all" teacher-centered approach is inconsistent with the current academic movement to transform students from passive recipients into interactive constructors of their own knowledge (Christensen, Garvin and Sweet, 1991). ⁶ This new student-centered active learning model changes the orientation of a high quality teacher from the relatively narrow master of content and dynamic oration to an infinitely more complex master of content, organization, facilitation and guidance in the process of building the students’ capacity for self-discovery.⁷ Moving from the old teacher-centered lecture mode to the student-centered active learning mode requires experimentation which is extremely risky in an environment where faculty are evaluated, compared and ranked on the basis of student opinions.

The Incomparability of Performance Evaluations

Now, to consider a different issue, assume that a valid measure of quality teaching could be obtained for each faculty member. Then, it would be appropriate to evaluate, compare and subsequently rank the faculty sequentially on the basis of these measurements. Right? Wrong! The misconception that a group of faculty members (or any group of workers) can be meaningfully compared and ranked is based on two faulty assumptions. The first assumption is that each worker can control his or her actual performance. However, system variation (frequently referred to as random variation) is inevitably present in any process, operation or activity. Deming (1986, 315) estimated that 94 percent of the variation in any system is attributable to the system, not to the people working in the system. The second assumption is that any system variation will be equally distributed across workers. According to Deming (1986, 353) there is no basis for this assumption in real life experiences. The source of the confusion comes from statistical (probability) theory where random numbers are used to obtain samples from a known population. When random numbers are used in an experiment, there is only one source of variation, so the randomness tends to be equally distributed. This is because samples based on random numbers are not influenced by such things as the characteristics of the inputs and other real world phenomena. However, in real life experiences, there are many identifiable causes of variation, as well as a great many others that are unknown. The interaction of these forces will produce unbelievably large differences between people (Deming 1986, 110) and there is no logical basis for assuming that these differences will be equally distributed.⁸ Evaluating, comparing and subsequently ranking a group of workers is a common misapplication of the concept of random numbers. Deming taught that ranking a group of workers is merely an exercise in ranking the effects of the system on the workers.

Confounding Variables in an Academic System

Now, consider some of the variables in an academic system. Table 1 provides a variety of variables that are related to the environment and individual course. Although no attempt was made to produce an all-inclusive list of environmental and course related variables, twenty-six easily came to mind.

Many of the variables listed in Table 1 have been the subject of research, while many others have not been studied. For example, several researchers have attempted to measure the relationship between student evaluations and class size, class level and class time (e.g., Wright et. al. 1984; Mulford and Schneider 1988). Others have attempted to measure the effect on student evaluations caused by different courses within the accounting major (Deberg and Wilson 1990).

The variables listed in Table 1 are perhaps less influential than those related to the students and the teacher. Consider the representative list of student characteristics that appears in Table 2. Some studies have examined the effects of student grade expectations (e.g., Wright et. al.1984 cite four) and male versus female preferences (Wright et. al. 1984 cite one), but it appears that most of the student characteristics listed in Table 2 have not been studied even though behavioral researchers have found cultural differences, such as masculinity and individualism, to be significant determinates of human behavior (e.g., Baker 1976; Hofstede 1986; McGregor 1960). ⁹ In addition, student characteristics, such as knowledge of prerequisite material, learning styles and attitudes and biases toward the course and the instructor, are perhaps critical to the student ratings. These are system-related variables and the effects are virtually impossible to measure.

Most of the research associated with student evaluations is directed toward teacher-related variables such as those presented in Tables 3 and 4 below. Many of the teacher-related characteristics listed in Table 3 are uncontrollable, or only partially controllable by the instructor. Most of these variables have not received much, if any, attention from researchers, although some have received considerable attention. For example, the personality of the instructor has been found to have a significant effect on student evaluations. Some researchers have suggested "that a lecturer’s authority, wit and personality can seduce students into the illusion of having learned, even when the educational content of the lecture is missing" (Wright, Whittington and Whittenburg 1984). Others have reported a connection between the instructor’s sex and student ratings (Bledsoe 1971 cited by Wright et. al. 1984). Even though all of the characteristics listed in Table 3 may influence student ratings of faculty, most of these variables have not been researched.

A separate list of variables that tend to be controllable by the teacher appears in Table 4. The purpose of this table is not to display an all-inclusive list of practices, but to suggest that a very large number of possible practices are available that might significantly influence student ratings. Most of the practices listed in Table 4 are related to how students are evaluated and graded. Although the relationship between student evaluations and grades has received more attention in the literature than any other variable discussed, the emphasis in most research studies has been on expected grades, actual grades and grading leniency (Wright et. al. 1984 cite six studies in this category), not on specific testing and grading practices. However, there are a wide variety of ways to be more lenient without appearing to be too lenient. For example, consider the effect of using non cumulative or non comprehensive exams throughout a course rather than using a more challenging group of comprehensive exams. Also consider the effect of using exams patterned after practice exams that are given to students in advance, or generously awarding partial credit for incorrect answers. ¹⁰

Gaming to Improve Student Ratings

The term "gaming" as used here refers to attempts by faculty to influence the student ratings in their favor by using practices that distract from learning rather than enhance learning. There are two issues related to faculty gaming. The first issue is that although gaming techniques represent low quality teaching, faculty use of the techniques may improve the student opinion ratings. This potential inverse relationship between good teaching and good student ratings is the invalidity issue discussed above. The second issue is that using student opinion surveys as the basis for evaluating teaching effectiveness causes the very problem that a good system’s design would prevent, i.e., a decline in the quality of teaching.

Many authors have observed that evaluating teaching effectiveness based on student opinions tends to motivate faculty gaming to improve the student ratings (e.g., Newton 1988; Wright et. al. 1984; Crumbley 1995). Perhaps many of the practices listed in Table 4 fall into the gaming category, but it is unlikely that a very large group of faculty could reach a consensus on very many of the items listed. Although most faculty would probably agree that frequently dismissing class early to take one’s students on a bar hopping escapade is gaming of the worst kind (Crumley 1995), it might be more difficult to achieve a consensus on giving non comprehensive exams, "I" grades and retesting students when they fail a test, or keeping grade expectations high until the student surveys are obtained and then "hosing" the students with a hard final exam (Crumbley 1995). Even so, as more faculty use a particular technique, even more are encouraged by students to follow suit. Many students use what one faculty member referred to as "a disgruntled consumer approach" (Wiesenfeld 1996). Student comments such as "My other teachers let me take exams early, or take retest, or drop the low exam grade, or give "I" grades for failing, or give practice exams or take home exams, or no final exam" are fairly common. The point of this discussion is simply that evaluating teaching effectiveness based on student opinions encourages game playing, which becomes more and more competitive and compelling as more gaming techniques are used, many to the detriment of quality education. Using student opinions to evaluate faculty promotes a system where students move forward unprepared for the next level. This promotes even more gaming behavior at the higher levels. Unfortunately, this behavior is inadvertently driven by an indeterminable number of university administrators who pretend to evaluate teaching effectiveness with little or no empirical evidence.

It is important to note that opinions do not qualify as empirical evidence. ¹¹ In fact, opinions, particularly anonymous opinions, cannot even be criticized because the basis for the opinion is unknown. Opinions do not require a theory as a justification and may be based on a gut feeling, whim, bias, infatuation, revenge, or psychic phenomenon. The realization that opinion is unreliable evidence was frequently reinforced by Deming who said, "In God we trust. All others require data" (Walton 1986, 96). Deming’s "management by data" concept means that accurate and timely data concerning the output of a process is needed to evaluate and subsequently improve the process.

Underlying Causes of the Misuse of Student Evaluations

The misuse of student evaluations is a symptom of a much larger and far more serious problem. The higher education system is based on a whole set of flawed assumptions that include the view that concentrating on the performances of individuals is the key to optimizing the performance of the system ¹² , that each individual is responsible for any variations in his or her performance, that individual performances are independent and can be measured separately (Scholtes 1998), and that these measurements can be compared to form a ranking of the individuals within the system. Additional misconceptions include the view that people mainly work for money and that individual incentive (zero-sum) pay plans improve the performance of individuals (Pfeffer 1998). These faulty assumptions and misconceptions are the antithesis of Deming’s theory of management and "systems thinking" in general (Senge 1990, 78).

Recommendations

So what is the solution? How should teaching effectiveness be measured so that the evaluation method promotes the optimization and continuous improvement of the higher education system? Unfortunately, there are no quick or easy solutions. Before a solution can be developed and implemented, the underlying flaws in the system must be recognized and eliminated. Optimizing the higher education system (as any other system) depends on the concepts of interdependence and synergy, not on the concept of individualism.

The following recommendations are based on Deming’s theory of management. First, the university should conduct a series of seminars on Deming’s theory of management and the issues addressed in this paper. Most universities have some faculty that are familiar with these concepts, but for some reason they have no credibility within their own institution. Therefore, outside help may be needed. ¹³ These seminars should emphasize what teamwork is, why teamwork is important and why the university performance evaluation system must be changed to promote teamwork.

Second, and a prerequisite for all that follows, the university performance measurement system must be changed dramatically to conform to Deming’s theory. This means that administrators should stop evaluating, comparing and ranking the faculty because ranking promotes competition, destroys cooperation, and accomplishes nothing positive. Judging and ranking people is not leadership. It does not provide a method for improvement. Leadership involves helping people to improve, and providing an environment where people see themselves as part of a system and are motivated to help each other to optimize the system (Deming 1993, 128-131).

Third, the faculty and administrators need to stop treating each course, each faculty member and each functional area as if they were self-contained private ¹⁴ stovepipes. ¹⁵

Fourth, faculty members must take collective responsibility for the overall educational outcome (Bailey 1994, 7; AACSB 1994, 23 & 48). This should involve frequent faculty collaboration to improve the content, pedagogy and delivery within each course and across courses. Student opinion surveys can be incorporated into the collaboration and review process if these instruments are designed to identify potential improvements and do not include a summary question concerning the overall rating of the faculty member. ¹⁶ The faculty collaboration process should definitely not include faculty committees that are used to produce an annual evaluation and subsequent ranking of each faculty member for merit pay or other salary adjustments. ¹⁷ To do so is to commit one of Deming’s seven deadly diseases (See appendix). Faculty members should not judge each other. Instead they should frequently interact, share, council, facilitate, teach and coach each other for the improvement of the system.

Fifth, cross-functional teams of faculty members (including other business and perhaps nonbusiness faculty) should identify as many of the interdependencies between courses as possible so that these relationships can be considered in the continuous improvement process.

Sixth, ranking students within each course is counterproductive and should be stopped. An alternative with considerable precedent from a wide variety of professional exams is simply to use a pass-fail system where the pass line is drawn at a demanding, but achievable level. Students should be able to demonstrate that they know and can perform whatever the collective faculty determines is required. With these changes, emphasis can be placed on maximizing learning in each class with a view toward the student’s overall program, not on each student’s position within the class ranking. ¹⁸ Then instructors can and should stop pushing unprepared students to the next level, and perhaps students will learn to cooperate with each other, rather than competing for a better position in the ranking. ¹⁹

Seventh, the cross functional faculty teams need to operationally define what students should know and be able to do at each stage of their program and at the end of their university program. A student’s acceptance into upper level courses should require the satisfactory completion of an entrance exam on the foundation material. Defining requirements across courses may require that faculty members sit in on other courses and team-teach some courses so that they have a better grasp of the entire major.

Eighth, the cross functional teams should develop comprehensive oral and written field exams and administer them at the end of the student’s program. ²⁰ This step is time-consuming, but it focuses attention on the student’s entire program rather than the individual parts, and it forces students to think in terms of their education holistically as a cumulative building process rather than as a series of isolated hurdles. This step also shifts the emphasis from a time-based program (120 or 150 hours) to a competency-based program ²¹ and provides a way to determine whether changes in the education process actually represent improvements in the learning outcomes (i.e., "management by data").

Finally, the cooperative efforts of administrators, faculty, students, and all other stakeholders will be needed in these efforts to change the higher education system from a competitive individualistic system to a system that promotes cooperation and teamwork. From Deming’s perspective, the current system produces approximately three percent of the potential of a system redesigned around the concepts he recommended. In Deming’s system, everyone wins and the potential benefits are quite extraordinary.

Footnotes

* Developed by the author based on the works of W. Edwards Deming (1986 and 1993).

____________________________________________

References and summaries

American Assembly of Collegiate School of Business. 1994. Achieving Quality and Continuous Improvement through Self-Evaluation and Peer Review.

Addy, N., and C. Herring. 1996. Grade inflation effects of administrative policies. Issues in Accounting Education (Spring): 1-13.

Bailey, A. R.1994. Accounting education: Gradual transition or paradigm shift. Issues in Accounting Education (Spring): 1-10.

Baker. C. R. 1976. An investigation of differences in values: Accounting majors vs. non-accounting majors. The Accounting Review (October): 886-893.

Bipboye, R. L., C. S. Smith, and W. C. Howell. 1994. Understanding Industrial And Organizational Psychology. Fort Worth, TX: Harcourt Brace College Publishers.

Bledsoe, J. C. 1971. Factors related to pupil observation and attitudes toward their teacher. Journal of Educational Research (November): 119-126.

Buckley, J. W., M. H. Buckley, and H. Chiang. 1976. Research Methodology & Business Decisions. National Association of Accountants. (Summary).

Burke, M. J., and R. R. Day. 1986. A cumulative study of the effectiveness of managerial training. Journal of Applied Psychology: 232-245.

Bures, A. L., J. J. DeRidder, and H. M. Tong. 1990. An empirical study of accounting faculty evaluation systems. The Accounting Educators’ Journal (Summer): 68-76.

Calderon, T., A. L. Gabbin, and B. P. Green. 1996. Summary of promoting and evaluating effective teaching. The Accounting Educator. The Newsletter of the Teaching and Curriculum Section, American Accounting Association. Supplement.

Caskin, W. E. 1983. Concerns about using students ratings in community colleges. In A. Smith (Ed.), Evaluating Faculty and Staff: New Directions For Community Colleges. San Francisco, CA: Jossey-Bass.

Christensen, C. R., D. A. Garvin, and A. Sweet. (Eds.) 1991. Education for Judgment. Boston, MA: Harvard Business School Press.

Coffman, W. E. 1954. Determining student’s concepts of effective teaching and their ratings of instructors. Journal of Educational Psychology: 277-286.

Crumbley, D. L. 1995. The dysfunctional atmosphere of higher education: Games professors play. Accounting Perspectives.

Deberg, C. L., and J. R. Wilson. 1990. An empirical investigation of the potential confounding variables in student evaluation of teaching. Journal of Accounting Education (Spring): 37-62.

Deming, W. E. 1986. Out Of The Crisis. Cambridge, MA: Massachusetts Institute of Technology Center for Advanced Engineering Study.

Deming, W. E. 1993. The New Economics For Industry, Government, Education. Cambridge, MA: Massachusetts Institute of Technology Center for Advanced Engineering Study. (Summary).

DeMong, R. F., J. H. Lindgren Jr., and S. E. Perry.1994. Designing an assessment program for accounting. Issues in Accounting Education (Spring): 11-27.

Elliott, R. K. 1992. The third wave breaks on the shores of accounting. Accounting Horizons (June): 61-85. (Summary).

Hofstede, G. 1986. The cultural context of accounting. Accounting and Culture. Annual meeting plenary session papers. American Accounting Association: 1-11. (Summary).

Hooper, P., and J. Page. 1986. Measuring teaching effectiveness by student evaluation. Issues in Accounting Education (Spring): 56-64.

Kau, J. B., and P. H. Rubin. 1976. Measurement techniques, grades and ratings of instructors. Journal of Economic Education: 59-62.

Martin, J. R. Not dated. What is the red bead experiment? Management And Accounting Web. Demings Redbeads

McDaniel, E., and J. F. Feldhusen. 1971. College teaching effectiveness. Today’s Education (March): 27.

McGregor, D. 1960. The Human Side of Enterprise. New York, NY: McGraw-Hill. (1957 Article Summary).

Meier, R. S., and J. F. Feldhusen. 1979. Another look at Dr. Fox: Effect of stated purpose for evaluation, lecture expressiveness, and density of lecture content on student ratings. Journal of Educational Psychology (June): 339-345.

Moore, T. E. 1997. The corporate university: Transforming management education. Accounting Horizons (March): 77-85.

Mulford, C. W., and A. Schneider. 1988. An empirical study of structural and controllable factors affecting faculty evaluations. Advances in Accounting: 205-215.

Naflulin, D. H., J. E. Ware, and F. A. Donnelly. 1973. The Dr. Fox lecture: A paradigm of educational seduction. Journal of Medical Education: 630-635.

Newton, J. D. 1988. Using student evaluation of teaching in administrative control: The validity problem. Journal of Accounting Education: 1-14.

Nichols, A., and J. C. Soper. 1972. Economic man in the classroom. Journal of Political Economy: 1069-1073.

Pfeffer, J. 1998. Six dangerous myths about pay. Harvard Business Review (May-June): 109-119.

Powell, R. W. 1977. Grades, learning, and student evaluation of instruction. Research in Higher Education. 7: 193-205.

Rayder, N. F. 1968. College student ratings of instruction. Journal of Experimental Education (Winter): 76-81

Reckers, M. P. 1996. Know thy customer. Journal of Accounting Education (Summer): 179-185.

Ryan, J. J., J. A. Anderson, and A. B. Birchler. 1980. Student evaluations: The faculty responds. Research in Higher Education. 12(4): 317-333.

Scholtes, P. R. 1998. Total quality or performance appraisal: Choose one. www.iqpic.org/per.htm.

Selden, P. 1984. Changing Practices in Faculty Evaluation. San Francisco. Jossey-Bass.

Senge, P. M. 1990. The Fifth Discipline: The Art & Practice of The Learning Organization. New York, NY: Doubleday Dell Publishing Group Incorporated. (Note).

Smith, A.1937. An Inquiry into the Nature and Causes of the Wealth of Nations, edited by E. Cannan, New York: Random House. Modern Library edition.

Stewart, C. T., and L. F. Malpass. 1966. Estimates of achievement and ratings of instructors. Journal of Educational Research (April): 347-350.

Stumpf, S. A., and Freedman, R. D. 1979. Expected grade covariation with student ratings of instructors. Journal of Educational Psychology. 71: 273-302.

Walton, M. 1986. The Deming Management Method. New York, NY: The Putnam Publishing Company.

Weaver, C. H.1960. Instructor ratings by college students. Journal of Educational Psychology: 21- 45.

Whitehead, A. N. 1929. The Aims of Education and Other Essays. New York, NY: Free Press.

Wiesenfeld, K.1996. Making the grade. Newsweek (June 17): 16.

Winsor, J. L. 1977. A’s , B’s, but not C’s: A comment. Contemporary Education. 48: 82-84

Worthington, A. G., and Wong, P. T. P. 1979. Effects of earned and assigned grades on student evaluations of an instructor. Journal of Educational Psychology. 71: 764-775.

Wright, P., R. Whittenton, and G. E. Whittenburg. 1984. Student ratings of teaching effectiveness: What the research reveals. Journal of Accounting Education (Fall): 5-30.

Wright, P. W., and T. Wotruba. 1978. Models of student evaluations. Paper presented to A.A.C.T. E., Chicago, February.

Yunker, J. A. and J. W. Marlin. 1984. Performance evaluation of college and university faculty: An economic perspective. Educational Administration Quarterly (Winter):9-37.