Summary by James R. Martin, Ph.D., CMA
Professor Emeritus, University of South Florida
Lab and Experimental
Research Main |
Decision Theory Main Page
The purpose of this article is to explain how a relatively small investment in training and software can help an organization develop a testing culture, or test-and-learn mind-set. Many business experiments lack the investigative rigor needed to provide useful information. However, the tools are now available to support small-scale experiments that provide scientifically valid information to support tactical decisions. A variety of software exist that can be used to determine what kind of samples are needed, where an experiment or test should be conducted from a control perspective, and whether the changes tested are statistically significant. In addition to explaining the steps required for a valid investigation, Davenport includes a number of examples of firms using experiments to test innovations including banks (PNC, Toronto-Dominion, Wells Fargo), retailers (CKE Restaurants, Famous Footwear, Food Lion, Sears, and Subway), and online firms (Amazon, eBay, and Google).
When Testing Makes Sense
Theoretically, experiments can be useful where a change in any part of a business can lead to an improvement. But in practice, many innovations cannot be tested on a small scale. Therefore, the most useful insights come from testing tactical changes such as a new store front, or marketing promotion, not a major change such as a new business model. Experiments are useful only where desired outcomes can be defined and measured, and a logical hypothesis has been developed about the effect of a proposed change. In addition, tests are most reliable where the settings are roughly equivalent, e.g., alternative versions of a website.
The Process of Testing
The process of testing includes six steps:
1. Develop a logical, testable hypothesis as indicated in the graphic below. The hypothesis should include desired outcomes that can be measured to indicate whether a proposed change passes or fails the test. The desired outcomes should represent significant economic value.
2. Design the experiment including choosing sites or units to be tested, the control groups, and the test and control situations. Make sure the number of test and control sites are sufficient to provide statistical significance. Use statistical analysis to minimize the test sites needed.
3. Execute the test. Have personnel report abnormal events, and remove sites that are affected.
4. Analyze the test using statistical software to measure the results, to indicate appropriate actions, and to determine whether additional testing is needed. Place a summary of the hypothesis, test dimensions, key results, and interactions in a "learning library".
5. Plan the rollout. Study the analysis to determine whether the rollout should be universal or differentiated.
6. Rollout the appropriate actions in a staggered fashion to test the rollout approach. Modify the approach if indicated by the desired results of early rollouts.
Building a Testing Capability
Developing a standard process of testing is the first step needed to establish an organization test-and-learn capability. In addition, organizations need to create an infrastructure to support the testing initiative. This includes training programs, software to structure the tests and analyze the results1, a way to capture what is learned from the tests (e.g., as part of their knowledge management system), a process to determine when to repeat the tests, and a centralized core resource group for expert support to insure that testing methods are sufficiently rigorous to produce statistically valid results.
Creating a Testing Mind-Set
In addition to the changes in process, technology, and infrastructure, creating a test-and-learn mind-set requires establishing a testing culture. A testing culture includes the realization on the part of everyone in the organization that no major change in tactics should be implemented without previously being tested by individuals who understand experimental or testing methodology. Although testing is not appropriate for every business initiative, the testing-and-learning mind-set should become a central part of every organization's decision making process.
_______________________________________________________
Footnote:
1 A web search for experimental design software produces millions of results from broad statistical packages to software designed for specific problems or industries. The graphic illustration above was originally adapted from Applied Predictive Technologies, a firm that provides test-and-learn predictive analytics software to analyze experiments. See predictivetechnologies. For many other A/B and multivariate testing tools see MAAW's Experimental Research Tools.
Related summaries:
Anderson, E. T. and D. Simester. 2011. A step-by-step guide to smart business experiments. Harvard Business Review (March): 98-105. (Summary).
Appelbaum, D., A. Kogan and M. A. Vasarhelyi. 2017. An introduction to data analysis for auditors and accountants. The CPA Journal (February): 32-37. (Summary).
Appelbaum, D., A. Kogan, M. Vasarhelyi and Z. Yan. 2017. Impact of business analytics and enterprise systems on managerial accounting. International Journal of Accounting Information Systems (25): 29-44. (Summary).
Davenport, T. H. 1998. Putting the enterprise into the enterprise system. Harvard Business Review (July-August): 121-131. (Summary).
Davenport, T. H. and J. Glaser. 2002. Just-in-time delivery comes to knowledge management. Harvard Business Review (July): 107-111. (Summary).
Kohavi, R. and S. Thomke. 2017. The surprising power of online experiments: Getting the most out of A/B and other controlled tests. Harvard Business Review (September/October): 74-82. (Summary).
Spear, S. J. 2004. Learning to lead at Toyota. Harvard Business Review (May): 78-86. (Summary).
Spear, S. and H. K. Bowen. 1999. Decoding the DNA of the Toyota production system. Harvard Business Review (September-October): 97-106. (Summary).
Thomke, S. and J. Manzi. 2014. The discipline of business experimentation. Increase your chances of success with innovation test-drives. Harvard Business Review (December): 70-79. (Summary).
Tschakert, N., J. Kokina, S. Kozlowski and M. Vasarhelyi. 2017. How business schools can integrate data analytics into the accounting curriculum. The CPA Journal (September): 10-12. (Summary).
