The Uncertainty Principle in Incentive Design
When great emphasis is placed on one element of an evaluation, people will focus on that measure, sometimes with good results that are in line with a desired objective, and sometimes to game the system and cheat.
New York Times Columnist Eduardo Porter says that phenomenon is best known as Goodhart's Law, after British economist Charles Goodhart. Porter adds that Luis Garicano, of the London School of Economics, calls it the "Heisenberg Principle of incentive design," likening it to the Heisenberg uncertainty principle in quantum mechanics. As Porter summarizes, "a performance metric is only useful as a performance metric as long as it isn't being used as a performance metric."
Porter's recent column cites several examples of Goodhart's Law. In 2004 the Chinese government decreed that there should be far fewer accidental deaths, and provincial authorities began a "no safety, no promotion" campaign, which tied bureaucrats' fates to accidental death rates. In seven years the death rate dropped in half. But scholars who studied the figures found that local officials had gamed the system. People severely injured in traffic accidents were counted as accidental deaths if they died within seven days. Officials who arranged to have the victims kept alive for eight days improved their statistics. Porter writes that U.S. hospitals have been known to do "whatever it takes" to keep fragile patients alive at least 31 days after an operation, to beat the Medicare 30-day survival yardstick. Further, Porter writes, Chicago magazine found that the city was able to report a reduced crime rate because some incidents were reclassified as non-criminal.
American education has begun an experiment in incentive design, Porter says, in which most states have established teacher evaluation systems based on gains their student make on standardized tests, along with some more conventional criteria such as evaluations by principals. The relevance of testing is based on sophisticated research. A study by Columbia Professor Jonah Rokoff and two Harvard colleagues, Raj Chetty and John Freidman, found teachers who improved student test scores-called high value added teachers-raised chances for student success in higher education and careers. But heavy reliance on testing has been extremely controversial and generated heated debate about the impact on children, and about whether education becomes less meaningful if there is relentless focus on testing success.
Porter doesn't take sides on specific matters of research and testing. But he says it's a good idea to keep Goodbhart's Law in mind because when the fate of individual teachers and schools depends on high stakes testing, the temptation for bad behavior is high. Several districts across the country have been accused of blatant cheating on tests, and others have used more subtle manipulations to create illusions of improvement. A recent New York Post story and Diane Ravitch's blog showed state test scores artificially elevated by deletion of four questions many students got wrong or left blank.
So will schools face massive unintended consequences as states institute more fully developed teacher evaluation systems? Porter quotes educators who say while evaluations are necessary, any system needs to be examined for unintended consequences. He quotes Professor Rockoff, who has defended the results of his own study, on preconditions for successful evaluations: "The obvious answer is do not put too much weight on any single measure." Read Porter's column here.
Friday, March 27, 2015
1-2 PM Eastern Time
Action Networks: Global to Local Structures for Change
Guests: Steve Waddell and Tom Bigda-Peyton
This is the second of two calls on system change. (This call is rescheduled from the original date of March 6, when we had to cancel for technical reasons.) While change always requires local action, change efforts are almost always heavily influenced by large contexts such as national conditions and-with increasing globalization-global ones. New approaches to organizing change move beyond traditions of hierarchy and build on multi-stakeholder strategies and inter-organizational networks. This discussion will build on 20 years of work with such strategies, highlighted in the book
Global Action Networks,
and will provide examples from U.S. healthcare reform.
PhD, focuses on collaboration and networks among organizations and institutions in business, government and civil society to produce innovation, enhance impact and build new capacity.His clients and project partners have included The Global Knowledge Partnership, the UN Global Compact, the World Bank, Global Reporting Initiative, the Ford Foundation, Humanity United, Civicus, International Youth Foundation, USAID, International Research and Development Centre, and the Forest Stewardship Council. He is a principal of Networking Action and lead steward for the
large change systems.
He has a PhD in sociology and a master's in business administration. He is author of
several articles and other publications
, including the books Societal Learning and Change: Innovation with Multi-Stakeholder Strategies and Global Action Networks: Creating Our Future Together.
EdD, is a system coach and consultant working to catalyze innovation and whole-system engagement in large organizations and networks striving for collective impact. Tom uses methods such as collaborative problem-solving, action learning, and positive deviance to promote culture change in industries such as healthcare, government, and forestry. As a practitioner-researcher for 25 years, and currently as a Partner at
Second Curve Systems
in Boston, his clients have included multiple healthcare systems, the Federal Aviation Administration, Fidelity Investments, the government of Ontario, and the Forest Safety Council of British Columbia. He is co-author of the
From Innovation to Transformation: Moving Up the Curve in Ontario's Healthcare System and Safety Culture: Building and Sustaining a Cultural Change in Aviation and Healthcare. Tom holds a doctorate in Organizational Behavior and Intervention from the Harvard Graduate School of Education, where he worked with two pioneers in the field of organizational learning and system dynamics, Chris Argyris and Don Schon. He also holds master's and bachelor's degrees from Harvard.
Anyone interested is welcome to participate in these calls. Use the social share links at the top of this email to share with your Facebook, Twitter, or other networks or to others.
Please use the forward link in this email to avoid the recipient unsubscribing you. You will not subscribe them. The feature only sends this individual message
. For information about Plexus Institute and more details about this call series and others, visit
See all upcoming PlexusCalls on the Plexus Calendar. Subscribe to the PlexusCalls or Healthcare PlexusCalls podcasts. Or, visit the Community section of plexusinstitute.org for the audio archive.