Decision analysis: past, present and future

Workshop attendees urged to “surf on the sea of uncertainty instead of drowning in it.”

By Ali E. Abbas

Decision Analysis

Workshop attendees included a Who’s Who of decision analysis, economics, political science and other fields. Participants included (l-r): Brad Powley (Stanford), John Pratt (Harvard), Carl Spetzler (SDG), Jim Dyer (UT Austin), Ross Shachter (Stanford), George Hazelrigg (NSF), Eric Horvitz (Microsoft Research), Ron Howard (Stanford), Jim Matheson (SmartOrg), Elisabeth Pate-Cornell (Stanford), Siegfried Hecker (Stanford), Ali Abbas (University of Illinois at Urbana-Champaign), David Bell (Harvard), Craig Boutilier (Waterloo), Ken Arrow (Stanford), Jon Leland (NSF), John Weyant (Stanford), Ralph Keeney (Duke), Detlof von Winterfeldt (USC), Vicki Bier (Wisconsin), Gordon Hazen (Northwestern University), Jason Merric (Virginia Common Wealth), Lauren Klak (Stanford), Robin Keller (University of California, Irvine), Muhammed Aldawood (Stanford).

Last October, several members of the Decision Analysis Society of INFORMS met in Palo Alto, Calif., with world-renowned economists, political scientists, industry researchers and experts in the fields of climate change, artificial intelligence and national security. Three Ph.D. students and two program directors from the National Science Foundation were also present. Workshop attendees were asked to reflect on the history of decision analysis and to think of the challenges and opportunities DA faces in both research and practice. Another objective was to identify how decision analysis can be used to handle the new problems that have emerged, problems characterized by large data sets, long and uncertain time horizons and group decisions, as well as complex models of beliefs and preferences.

The workshop started with a presentation by John Weyant about climate change. Ronald Howard then discussed the axioms of expected utility theory (the five rules of “actional” thought as Howard calls them) and how tempting it can be to deviate from these rules when faced with large problems. Howard cautioned that there are many fundamentally flawed approaches to decision-making. “Decision analysis requires mastering uncertainty, the main source of difficulty in decision-making, and the concomitant elements of alternative generation and preference assessment,” he said. “When other approaches to decision-making analyze decision problems, they attempt to minimize or ignore uncertainty. Decision analysts surf on the sea of uncertainty instead of drowning in it. They can determine the cost of uncertainty in making the decision and what would be profitable steps to reduce it.”

Howard went on to ask, “If you take the decisions out of any profession, what is left?” Yet few professional fields have decision-making as a core process. Howard also commented on how people often forget the most fundamental distinction in decision analysis: the difference between a decision and its outcome. Put simply, he said, “Good decisions never become bad, and bad decisions never become good.”

Next, John Pratt discussed one of the fundamental challenges in decision-making: group equity decisions. He presented unpublished work on the topic of fair and unfair division. On the topic of uncertainty, Pratt also mentioned that, over the years, he has noticed that an elaborate analysis’ persuasiveness depends more on the neutrality of its auspices than the appearance of perfection in its details.

“If you really want to learn and not just confirm a preconceived view, don’t explain away the results post facto but ask yourself how likely you would have thought them to be, honestly and beforehand, under various hypotheses,” Pratt said. That gives you the likelihood ratios you need to update in your analysis.

Ken Arrow treated the group to his personal reflections on the history of Arrow’s Impossibility Theorem. His reflections illustrated the importance of seeing connections between different fields.

“In college, I was interested in mathematics and logic,” Arrow said. “During my undergrad, at City College [of New York], I took a course on the calculus of relations. This is how I learned about transitivity and orderings. When I became a graduate student under Harold Hotelling, I began learning the theory of the firm. It was preference ordering that mattered in this case, not necessarily utility. It was very hard to even accept von Neumann-Morgenstern’s theory at the time. It took a year or so for us to really grasp its meaning.

“The main problem was that a firm maximizes profits. But real firms have many owners. So you might have two different stakeholders who make decisions. If you take two investment projects, A and B, I may prefer A because according to my calculations A is better, and you may prefer B. So how do we decide? Well, majority voting, weighted by the number of shares owned is the natural thing. Is this an ordering though? That took about 15 minutes with pencil and paper to realize that you get intransitivity. Now, I was trying to get a theory of the firm, not trying to look for paradoxes. This is a big nuisance!

“Then I worked at the RAND Corporation, which had a lot of emphasis on game theory. Of course the game players were the Soviet Union and the United States. But the United States and Soviet Union are abstractions; a lot of people live in the U.S. and Soviet Union. Then there was the idea of social welfare. They were trying to get away from the sum of utilities, a linear form, which we still apply today. I decided to write an expository paper. In writing it, I realized in effect it was kind of like a voting problem. Earlier, I had shown that pair-wise majority voting doesn’t work – transitivity is violated – but there are many voting methods, preferential ballots for example. But none of them seemed satisfactory.

“Some of the methods worked all right because they rank order all of the conceivable candidates, not the actual candidates. That is transitive, but it involves making judgments about candidates that aren’t running. If you put these together, I worked out a proof of impossibility. Of course I found out years later that the original intransitivity regarding voting had been developed in 1785, so I had slightly anticipated. I’m glad I didn’t publish that. I vaguely thought someone had already done that.”

Arrow also presented new work on decisions with long and uncertain time horizons, a topic much needed when dealing with problems like climate change.

Next, Ralph Keeney presented some promising new axioms that led a group of decision-makers to use an additive group utility function. Keeney also reflected about the connection of decision analysis with the operations research field: Operations research (O.R.) is thought of as the science of decision-making, so developing and using models to analyze decisions is critical to O.R. Hence, the impact of O.R. could be enhanced if the model provides more directly relevant insights for the decision-maker(s). Some of the most critical components of decision-making are the objective(s) to be achieved and the alternatives from which to choose. Decision analysis explicitly focuses on these components by constructing an objective function incorporating the decision-maker(s) preferences. O.R. could enhance its stature and impact by focusing a bit more time on these components in research and especially in the analysis of important decisions.

Siegfried Hecker, former emeritus director of the Los Alamos National Laboratory and a national expert on negotiations with North Korea, illustrated how decision analysis can be used to tackle national security decisions. Both he and Bill Perry, former secretary of defense, use decision analysis in their national security class at Stanford. Hecker also discussed the need to look for early signs of change when working in national security and decision analysis. He discussed how new innovations in technology may be useful early indicators of change. He gave an example of how the prevalent use of cell phones in North Korea as of December 2010 is an indication that “the winds of change are blowing.”

Detlof von Winterfeldt, Elisabeth Pate-Cornell and Vicki Bier also presented their perspectives on practical applications of decision analysis in addressing homeland national security decisions.

The workshop demonstrated that problems with long and uncertain futures – and those spanning a large domain of consequences, such as climate change – might require new models of preferences. David Bell discussed the new notion of one-switch utility independence and a variety of new conditions that result in one-switch multi-attribute utility functions (joint work with Ali Abbas). Jim Dyer discussed distinctions between objectives, attributes, preferences and the prediction of preferences. Ali Abbas discussed the notion of a utility tree and the notions of single-sided and double-sided utility copula functions that construct utility functions using univariate assessments.

The workshop also included a talk on medical decision-making by Gordon Hazen. Hazen also discussed promising new tools for replacing the widely used QALYs (quality adjusted life years) to include personal goals. Hazen then commented on the importance of encouraging and supporting cross-field collaboration, particularly with optimization researchers and practitioners.

Eric Horvitz mentioned that people in the MS/OR community might not realize the extraordinary influence of decision analytic methods and ideas on the science of machine intelligence. “Computer scientists trained in decision analysis were at the root of a veritable revolution in computer science,” he said. “Decision-theoretic ideas now play a central role in artificial intelligence (AI) research, where, for example, scientists wrestle with the challenges of autonomous agents that learn and reason about ideal actions under uncertainty. Probability, preferences and expected utility are front and center in AI today, and numerous concepts, such as the value of information and sensitivity analysis, can be found in leading solutions. There is great opportunity for more interaction between the communities beyond relying on the small number of people with feet in both places.”

Horvitz also noted that moving beyond consultative models for fielding decision analyses is the long-term dream of automating various aspects of decision analysis.

“Online services, coupled with laptops and smart phones bring to the fore multiple opportunities,” he said. “A broad spectrum of possibilities is on the horizon – to date the surface has only been scratched. New questions coming to the surface include: What more might be done to bring richer decision analyses to the fore to enhance peoples’ daily lives? Where is the “sweet spot” for value-added given the overhead of effort, and how might overhead of effort be reduced? As an example, people face potentially fine-grained decisions about how to balance their privacy with the value of sharing some personal data with a company in return for enhanced services. Deeper reflection about the tradeoffs is likely to be valuable.”

Carl Spetzler discussed decision-making in practice and the attention given to descriptive decision science. “This is demonstrating how big of an opportunity exists to improve decisions,” he said. “The value that can be created by closing this gap is being confirmed by the performance of entities such as Chevron that adopt DA as an organizational discipline. We need to support building a real profession of practitioners to advance and unify the field. Then, the research will get tremendous leverage in the real world.”

Robin Keller also stressed the connections to behavioral decision-making. Keller identified some questions to ask that can extend the value of decision analysis. For example, how can DA research results be used to help real people in real (daily) life? Can we combine decision theory with neuroeconomics, physical body measures (pulse, cortisol levels), vision research, etc? If nonconformity with expected utility and probability theory is a “bias,” is it possible to “de-bias” people? Can we train people to conform with models/axioms?

Vicki Bier noted that success stories of decision analysis such as the case of Chevron should be given more publicity, and that more of these cases should be developed. Bier, George Hazelrigg and many others noted that decision analysis courses and Bayesian probabilistic reasoning need to be more available at universities around the country and that decision analysis should be taught to design engineers. On education, Dyer emphasized that decision analysis is more than just decision trees.

Many undergraduate and MBA students are being introduced to decision tree software and to Monte Carlo simulation. However, these tools are often introduced with little attention paid to the important issues of problem structuring and of risk management. The context of risky decision-making is extremely rich, and the concepts of decision analysis provide ways of thinking about the complexity of real-world decision-making in ways that still are not exploited in many companies and that are not conveyed to many students. The group brainstormed courses for an ideal MSc., Ph.D. and MBA curriculum in decision analysis.

Elisabeth Pate-Cornell reflected on the DA field and its development. “Decision analysis is a great tool that everyone should have,” she said. “It provides a structured way of thinking about many problems.”

Several challenges, however, remain. First, many people remain uncomfortable with probability assignment when statistics are not available. The implementation of probability encoding for the population at large will require effort and is one area that needs to be addressed. People remain ambiguity averse, preferring lotteries where probability assignment comes from large evidence to those where insufficient evidence is available. While we can see normative issues in this type of preference, the fact is that the bulk of the population still has this preference even after explaining it to them.

Another issue is the quantification of preferences for utility assignment. People are reluctant to quantify trade-offs and make the types of utility assessments needed for decision analysis. To reach out to these different groups, researchers and practitioners in the field need to be cognizant of these facts and be able to interact with such decision-makers when they exhibit ambiguity aversion. Along those same lines, practitioners need to present their methodology in a simpler way to have people use our methodology in their everyday decision-making.

John Weyant would like to see more collaboration between the decision analysis audience and people working in the fields of climate change and energy.

Ross Shachter mentioned that that there has been a significant but limited role for DA and O.R. in public and private decision-making. The revolution in computing that provides us with so much information, online tools and complex systems presents an opportunity to extend the application of DA and O.R. principles to much wider audiences.

Jason Merrick observed that the workshop made clear two major points. First, the field of decision analysis is a vibrant and fruitful area of research. Second, we have our work cut out for us to address some of the challenges and modern problems faced by society today. Many fields can contribute to this research effort, including optimization, simulation, economics and psychology, but decision analysis must frame the problem and be the central technique to which other methods contribute.

The mix of young researchers and the founders of the field sitting together in a common room to discuss the future of the discipline and reflect on its past was a great experience for everybody. The group plans to meet again to discuss some of these topics in more detail.

To summarize:

  • The field of decision analysis needs to maintain and nurture its deep foundations and at the same time reach out to new fields. We are fortunate to have a method that can handle uncertainty and place a value on resolving it. This is seldom obtained using other methods of decision-making.
  • Easier methods of decision-making may be tempting but many are also arbitrary.
  • Every field centers on decision-making. Decision analysis is applicable and would be valuable in a broad spectrum of realms. Multidisciplinary research is crucial to making sure other fields gain the benefits of its coherent methodology.
  • Great opportunities lay ahead for research on leveraging combinations of decision analysis, artificial intelligence, data mining and visualization to guide action under uncertainty in such realms as healthcare, climate change, energy and national security. Beyond being experts in decision analysis, people working in these areas will need to learn scientific content of other domains to make strong contributions.
  • Group decision-making remains an interesting problem that should be further investigated, and has traditionally been addressed using incentives, game theory, facilitation and other methods.
  • The impact of O.R. could be enhanced if the model provides insights that have more direct relevance to decision-makers. More models are still required for large problems.
  • We should support the collaboration of researchers and practitioners to provide practice-oriented research and document successful case studies from practice. We should also strive to spread decision analysis and Bayesian probabilistic courses into college curricula.
  • Modern technology presents many early indicators and resources that should be used in building more representative models.

Ali E. Abbas (aliabbas@illinois.edu) is an associate professor in the Industrial & Enterprise Systems Engineering Department at the University of Illinois at Urbana Champaign. This workshop was supported by the National Science Foundation Career Award NSF-DRMS 0846417. The author thanks Jon Leland, Jack Mezaros and Bob O’Connor for their support and thanks Eric Horvitz for several edits, comments and suggestions. The author also thanks Lauren Klak, Brad Powley and Muhammed Al-Dawood for taking careful notes during the workshop. For more details about the workshop or lecture slides, contact Abbas.