Probability Management 2.0

New standards, free tools and growing adoption boost a new generation of interactive simulation.

By Melissa Kirmse and Sam Savage

Probability Management

“Nearly all organizations worship at the altar of the false god of determinism,” says Steve Roemerman, the CEO of Lone Star Analysis, a firm that creates analytical systems for a number of government agencies. “Until our budget process punishes this behavior, only real leaders will embrace truthful probability.”

To achieve this, organizations must rethink the way they communicate, calculate and make credible estimates with uncertainties. These are the three guiding principles of the discipline of probability management, which is ushering in its second generation with new standards, free development tools and growing acceptance.

Savage, Scholtes and Zweidler defined probability management in a 2006 OR/MS Today article [1]. Using the advent of electric lighting as a metaphor, the authors compared the incandescent light bulb to a new generation of interactive simulation, powered by auditable libraries of stochastic data (electricity). Rooted in methods developed in stochastic optimization, (see [2], [3], [4]), probability management represents uncertainties as arrays of simulation outputs called SIPs.

“SIPs are an ideal means for modeling and conveying uncertainty in a standardized fashion,” says Eric Wainwright, co-founder and chief technology officer of Oracle’s Crystal Ball. He believes SIPs “will play an increasing role in the way organizations manage uncertainty through their informational and predictive systems.”  

A SIP representing the roll of a die, for example, might consist of a data set of 10,000 integers between 1 and 6. These may be used in stochastic calculations through vector arithmetic (SIPmath), which illuminates systematic errors incurred in the face of uncertainty.

The 2006 article describes how top decision-makers at Royal Dutch Shell used SIP libraries to interactively swap projects in and out of their portfolio and instantly see the distribution of results [5]. Since 2006, technology has evolved to the point that developers using nothing but Excel’s Data Table command can create interactive simulations in which thousands of trials are run before the user’s finger leaves the <Enter> key [6]. According to Warren Hausman, professor of Management Science & Engineering at Stanford University, this “represents a true revolution in the world of simulation that will ultimately help non-technical managers avoid the flaw of averages in a wide variety of settings.”

Nonprofit ProbabilityManagement.org is standardizing and codifying this freely available technology to promote “truthful probability” at all levels of management.

New Standards

Communication is easiest when everyone speaks the same language. To improve the communication of uncertainty, ProbabilityManagement.org has developed a new version of their open SIPmath™ standard, which will be released in November at the INFORMS Annual Meeting in San Francisco.

Marc Thibault, chair of ProbabilityManagement.org’s standards committee, describes version 2.0 of the standard as “a simple, adaptable data architecture that makes it easy to create and use SIP libraries by piggybacking on common data formats: Excel worksheets, XML and CSV.”

According to Lone Star’s Roemerman, data used in government contracts really belongs to taxpayers and should not be stored in a proprietary format. “SIPs and SIPmath are powerful because they are easy to access,” says Roemerman, whose organization offered guidance in the development of the 2.0 standard. “They support the most common and inexpensive ways to share information, and they do it in a way that both humans and machines can understand. We don’t know of another alternative with these attributes.”

Figure 1: SIPmath Modeler Tools ribbon.

Figure 1: SIPmath Modeler Tools ribbon.

Free Tools

Calculating with SIPs is called SIPmath, and it can be done in any platform that supports arrays. In particular, native Excel performs arithmetic on uncertainties just as it does on numbers through the use of the Data Table function. To help simplify the process, ProbabilityManagement.org has developed SIPmath Modeler Tools, a set of macros that facilitate the development of Data Table driven models, which do not themselves require macros and may be freely shared with collaborators. The tools run in either Windows or Mac environments.

Also available are SIPmaker, a traditional Monte Carlo simulator derived from XLSim that outputs SIP libraries, as well as add-ins that create SIP libraries from both Oracle Crystal Ball and Palisade’s @RISK. In the future, the organization hopes to release additional translators for sharing libraries between Matlab, R, SAS, etc. through its common XML format. ProbabilityManagement.org offers numerous videos, tutorials and example models to help organizations develop SIPmath applications.

Growing Adoption

According to Daniel Zweidler, senior fellow at the Wharton School and co-author of the 2006 article, “The issue with SIPs will be the same as data – how can we ensure quality?” Actually he may understate the problem, in that many organizations do not even have permission to be uncertain. However, trusted sources of distributions are now appearing in numerous areas. For example, the Federal Reserve models the uncertainty of inflation. The U.S. Geological Survey models uncertain earthquake magnitudes for any latitude and longitude in the United States. Financial data firms offer details on the uncertainty of future asset values. Credible sources such as these give management the “permission” to be uncertain within auditable limits.

David Cashbaugh, director of Navy Personnel Research, Studies and Technology, says probability management “provides a level of clarity previously unavailable to senior Navy decision-makers. Interactive stochastic modeling can alter the conversation from ‘What’s the right answer?’ to ‘What’s the right question?’ ”

For security reasons, the Navy may use only a limited set of software packages, which includes Microsoft Excel. Lt. Cmdr. Connor McLemore, a member of the military faculty in the Operations Research Department of the Naval Postgraduate School in Monterey, Calif., was scheduled to teach simulation in Excel in a few weeks but did not know how to proceed without add-in software. After attending a workshop on SIPmath organized by Cashbaugh and using the tools, McLemore says, “I’ll definitely implement them in the spreadsheet modeling course here.”

Companies are experimenting with probability management in numerous settings. “Many of us involved in planning at Chevron believe SIPmath can be a breakthrough in the way we think about and analyze uncertainty,” says Brian Putt, a decision analyst there. “Applications vary from evaluating the portfolio of development wells, understanding exploration risk, preparing parts of a probabilistic business plan, and conducting value of information studies on appraisal wells.”

Lockheed Martin is another company beginning to adopt probability management. As Phil Fahringer, a Lockheed Fellow, reports, “Lockheed has established an internal Community of Practice to educate, share examples and store reusable SIPmath libraries.” He cites application areas that include “improving insights related to program costs, schedule and performance and budget and requirements decisions both internally and externally.”

Probability management is even being applied to accounting and tax issues. Michael Salama, lead tax counsel for Walt Disney, recently published a book on managing uncertain tax positions that makes use of several SIPmath models. It was developed for subscribers of Bloomberg BNA [7], a leading source of legal, tax, regulatory and business information for professionals.

Several vendors have begun to explicitly support the discipline of probability management in their software products including Analytica from Lumina Systems, Crystal Ball from Oracle Corp. and Risk Solver Platform from Frontline Systems.

Stefan Scholtes, professor at Cambridge University’s Judge Business School and one of the authors of the 2006 article, is excited by the latest developments. He and his co-authors hoped to replace huge monolithic risk management models, which they referred to in the article as “dinosaurs,” with small interactive models, called “rats.” Scholtes says the new open standards and tools will provide “free food to feed the many rats that will kill the average dinosaur.”

Read the sidebar: Teaching modern portfolio theory to 10-year-olds, by Sam Savage

Melissa Kirmse (melissa@probabilitymanagement.org) is director of operations of ProbabilityManagement.org. She has 20 years of experience in project coordination, administration and technical writing at tech companies including Microsoft and TiVo.

Sam Savage (savage@stanford.edu) is executive director of ProbabilityManagement.org. He is the author of “The Flaw of Averages: Why we Underestimate Risk in the Face of Uncertainty,” and a consulting professor of Management Science & Engineering at Stanford University.

References

  1. Sam Savage, Stefan Scholtes and Daniel Zweidler, “Probability Management,” OR/MS Today, February 2006, Vol. 33, No. 1.
  2. Ron S. Dembo, “Scenario optimization,” Annals of Operations Research, 1991, Vol. 30, Issue 1, pp. 63-80 (http://link.springer.com/article/10.1007%2FBF02204809).
  3. Gerd Infanger, “Planning Under Uncertainty,” Boyd & Fraser, 1994.
  4. Ben C. Ball and Sam L. Savage, “Holistic vs. Hole-istc Exploration and Production Strategies,” Journal of Petroleum Technology, September 1999, pp. 74-84.
  5. Sam L. Savage, “The Flaw of Averages: Why we Underestimate Risk in the Face of Uncertainty,” John Wiley & Sons, 2009, 2012.
  6. Sam L. Savage, “Distribution Processing and the Arithmetic of Uncertainty,” Analytics, November/December 2012 (http://viewer.zmags.com/publication/90ffcc6b#/90ffcc6b/29).
  7. www.bna.com