Robust Decision Making under Deep Uncertainty

Abstract

The uncertainty is highest in the beginning of a project, but it is also here decisions of crucial importance for the project's outcome are made. Furthermore, the world is constantly changing, making it impossible to make reliable decisions based on prediction of what the future holds. There is a need to move away from prediction of the future to instead calculating the most robust decisions.

This article describes how to approach decisions under high uncertainty as a project/program/portfolio manager or decision maker using robust decision making (RDM). RDM is a set of methods and tools developed over the last decade, primarily by researchers associated with the RAND Corporation .

After touching upon the different levels of uncertainty, the article moves on to the core ideas of RDM. RDM combines decision analysis, assumption-based planning, scenarios, and exploratory Modelling to stress test strategies over a great number of possible futures. To provide a hands on understanding the method, the article guides the reader through the five iterative steps of RDM. After this, an illustrative example about long term planning at a water utility is explained . Limitations and challenges of the method are touched on, including the challenge of shifting to a new way of dealing with uncertainty.

It is inevitable to encounter uncertainties in projects, but you can do your best to investigate which strategies will perform best under these uncertainties. RDM is an iterative framework that offers a data-based assessment on future scenarios on which stakeholders and decision makers can base their decisions.

Deep Uncertainty and Project, Program and Portfolio Management

Figure 1: Uncertainty and Significance of decisions

A paradox of project planning is that when uncertainty is at its highest – in the beginning of a project – is also when the most crucial and determining decisions need to be taken. When time goes by and the project performer knows more about the project, there are less degrees of freedom to influence it. All projects face varying degrees of uncertainty which is often mentioned as a reason for their failures.

Uncertainty characterizes situations where the outcome of a project is likely to deviate from the estimated outcome. There are several levels of uncertainty and many operate with four levels. Level 1, 2 and 3 ranges from almost certainty of the outcome to a limited set of plausible outcomes with unknown probabilities. Level four represents the highest level of uncertainty, also called deep uncertainty. Level four can be distinguished into two types: a) where the future is bound around many plausible futures, and b) where the only thing that is known about the future is that it is not known. Type a is often due to lack of information, whereas type b is due to unpredictable events . We see the tendency that important problems faced by decision makers are characterized by a higher level of uncertainty and cannot be reduced simply by collecting information, because the uncertainties are unknowable at the time. Such situations are defined as decision making under deep uncertainty

Normally when speaking about uncertainty in projects, program and portfolio management, risk management tools are discussed and applied . These tools often provide a framework for addressing and managing risks – or uncertain event/condition. But to be able to identify risks and thereby manage risk in a traditional way, the ambiguity needs to be low and the information available relatively high . Furthermore, when taking a general approach to risk management, portfolio, program, and project managers need to identify the risk probability and impact. .

Different approaches are needed for handling decisions in deep uncertainty (level 4) than in level 1, 2 and 3. This is because these types of uncertainties involve factors of which probability distributions and possible outcomes are not known. There is a need for a new paradigm for decision making under deep uncertainty that is not based on predictions of the future (known as the “predict-then-act” paradigm). The “monitor and adapt” paradigm is more suited for deep uncertainty, which recognizes the need for taking the uncertainty into account and prepares and adapts for uncertain events .

Overview of RDM

Robust Decision Making (RDM) is a set of tools, methods, and processes that are developed for Decision Making under Deep Uncertainty when predictions of the future are not possible or risky. RDM was originally developed to support decision making related to policy challenges. One of the first applied examples of RDM was in 1996 about abating climate change .

The RMD methods are suitable within both projects, programs and portfolios, but due to the ‘deep uncertainty’ aspect, there are often a longer time horizon and bigger complexity involved. Examples of application of the method are as mentioned within climate change related issues, water management or infastrcuture, where policy related decisions need to be taken. This means that RDM seem most usable under certain conditions. First of all when the uncertainty is deep and not well understood, secondly when there are several decision options, and third of all, when the decision challenge is complex enough that computer models are needed to help map out the potential consequences of different actions over mulitple scenarios .

Robust decision making can help by:

- Identifying which strateg(ies) are most robust. This means which stratigy’s satisfactory per-formance is relatively insensitive to all or most of the uncertainties – compared to the other

- Identifying vulnerabilities of the robust strategies. This means combinations of model formulations and parameters where the strategy performs poor compared to other

- Suggesting new strategies which might better combat these vulnerabilities

- Characterizing the trade-offs involved in the choice among the options

Foundation of RMD

RDM uses computation and quantitative models in a new way to provide better decisions under uncertain conditions. It combines decision analysis, Assumption-Based Planning, scenarios, and Exploratory Modelling to stress test strategies over a great number of possible paths into the future. By that, policy-relevant scenarios and robust adaptive strategies can be determined. RDM are often embedded in a decision support process called “deliberation with analysis” that promotes learning and consensus-building among stakeholders .

Decision Analysis Empirical research shows that people make better decisions when using well-structured decision aids. RMD represents a type of decision analysis that relies on decision structuring frameworks, evaluation of consequences of alternative actions, identification of trade-offs among alternative options, and tools for comparing decision outcomes . In RDM, the decision analysis is based on considering several strategies over a broad range of possible futures. The results are used to characterize weaknesses of the strategies and to identify and evaluate potential response to those weaknesses.

Assumption based planning RDM draws on assumption based planning. This implies the development of an organization’s plan and identification of load-bearing assumptions in this plan. This means identifying the potential break points that could cause the plan to fail. After identifying the assumptions, ABP considers shaping actions (actions designed to make the assumptions less likely to fail), hedging actions (those that can be taken if assumptions begin to fail), and signposts (trends and events to monitor in order to detect whether any assumptions are failing) .

Scenarios The concept in RDM of multiplicity of plausible futures is drawn from the scenario analysis (Lempert et al. 2003). Scenarios are projected futures that require less confidence than probabilistic forecasts and that seek to represent different ways of looking at the world without a ranking of relative likelihood (Wack 1985). RDM draws from scenario analysis by analysing information about the future into small numbers of different cases that can help people to explore and communicate the deep uncertainty

Exploratory Modelling The concepts of decision analysis, assumption based planning and scenarios are combined in the exploratory modelling. Exploratory modelling provides RDM with a quantitative framework for stress testing and scenario analysis

Iterative steps of RDM

Figure 2: RDM Iterative Steps

RDM follows the “deliberation with analysis” process. The deliberation with analysis means that parties to a decision consider their objectives and options; decision-relevant information is generated using system models; and the parties revisit their objectives, options, and problem framing influenced by the new quantitative information . The process is best used in situations with different decision makers who face a changing environment and whose goals can change during collaboration with other parties.

Step 1: Decision Framing

In the decision making framing step, stakeholders define the key factors of the analysis: “The decision makers’ objectives and criteria; the alternative actions they can take to pursue those objectives; the uncertainties that may affect the connection between actions and consequences; and the relationships, often instantiated in computer simulation models, between actions, uncertainties, and objectives”. The information is often filled into a 2X2 matrix called XLRM  :

X: Uncertainties L: Policy Levers R: Relationships M: Measures of performance

Figure 3: XLRM Matrix

Step 2: Evaluate Strategies across Futures

In the second step, the decision framework is executed through a simulation process which is used to evaluate proposed strategies across the multiple plausible futures. This means that the models (R) are run over the strategies (L) and uncertainties (X) to output the performance (M). To create the futures, different assumptions of the uncertainties outcome are combined. This can give extreme amounts futures if the uncertainties are many and broad, which is why this step generates a large database of results, when the strategies are evaluated. Depending on the number of strategies and the number of uncertainties, it might be more than thousands of cases .

Step 3: Vulnerability Analysis

To analyse the vulnerabilities, the data is visualised in the third step. RDM commonly uses Scenario Discovery (SD) algorithms (s. 31). By doing that, it is possible to visualise which factors distinguish futures where strategies hit or miss their goal (s.31). In other words, under which set of uncertain conditions / in what futures does a particular strategy fail to meet stakeholder goals? This way of assessing the futures can remove the bias that can be present when analysing scenarios qualitatively. This process is often helped out by a data mining process .

Step 4: Tradeoff Analysis

In this step, the scenarios can be used to reveal the trade offs among strategies. This can for example be done by plotting the performance of a strategy as a function of the likelihood of the relevant scenario to assess the futures when choosing one strategy over another .

5. New Futures & Strategies

The analysis made can then be used to propose new, more robust strategies that provide better trade offs .

Example of application

The following illustrative example of application of RDM is drawn from RAND Corporation, 2014: THE ROBUST DECISION MAKING FRAMEWORK .

The focus of the example is the long term planning of a small water utility. The aim is not to provide results but to show how the methodology can be utilized to make the method more hands-on and tangible.

Step 1:

The stakeholders and decision makers held a participatory scoping exercise and agreed on the following XLRM matrix:

Figure 4: Example of XLRM Matrix

(M) Measure of performance: Safe yield being raw water storage above 75% capacity 90% of the time

(R) Relationships: A model that simulates daily demand and supply and tracks water storage in the system

(X) Uncertainties:

- 10 different climate projections

- customer demand between 75 and 120 mgd

- contract size being small, medium and big.

(L) Policy Levers: The performance of the current infrastructure

Step 2:

Following RDM, the utility evaluated a wide range of futures using its systems model. Twenty futures were specified that uniformly sample across the uncertainties; climate projections, projections of future customer demand, and the sizes of possible wholesale contracts. The performance of its existing system were evaluated across the futures and the performance was measures through the safe yield. The information were put into a spreadsheet. See the figure 5 for example of future in the spread sheet. For that exact future, the safe yield was calculated to be 94 percent.

Figure 5: Example of a plausible future

Step 3:

Figure 6: Graph of the plausible futures

The graph shows the result of each future with respect to each of the uncertain factors. Climate projections are on the x axis, demand estimates are on the y axis, while the size of the circle indicates the wholesale size. The number below the mark is the safe yield result for that specific combination of factors. By using datamining, the vulnerable conditions were detmermined:

- Demand ≥ 95 million gallons per day (mgd)

- Climate projection: 6-10 (warmer/drier projections)

- Wholesale type: Medium or Large

These vulnerable spectres define a subset of the futures – the upper right corner. In this case it is 5 out of the 20 futures that have the vulnerable conditions. 4 out of these are actually vulnerable, meaning that they do not meet the goal of safe yield of 90%. 4 out of 7 failing futures are described by the vulnerable conditions.

Step 4:

In this example, the utility might be able to find a more robust strategy than the current baseline shown in the figure, and by that making it possible to meet the goals under higher demands and worse climate conditions. But this more robust strategy might also be more costly, why the decision makers needs to decide if they want to trade off cost for robustness or the opposite.

Step 5:

This short description of the largest vulnerability was included in a discussion regarding what strategies could be implemented to improve reliability under these specific conditions.

Limitations and Challenges

Even though RDM offers many benefits for making decisions to wicked problems with a high degree of uncertainty, there are also some limitations to the methods. These involves change of mindset, data requirements, conservative decision making, limited applicability for surprise events.

Change of mindset. When using RDM, one must move away from one way of managing uncertainties to another. Many analysts are used to working with assumptions and preductions of the future. RDM does not imply predictive thinking but instead be decision oriented thinking and the analysts needs to start using the quantative data in a new way .

The data requirements are big. Even though RDM is fairly simple and can be done with the help of computers, challenges still emerge. To analyse potential consequences of many scenarios, it is necessary to have large data requirements, computational capability, model simulation and visualization. These skills and data might not be existing in all organisations nor countries

Decisions tend to be conservative. Because the decision needs to be robust, they automatically become more conservative . This is because the strategy needs to perform semi well on a broad spectrum of cases and not necessarily very well – or bad – in any cases.

Still not completely suited for surprise events. Even though RDM is created to work under deep uncertainty, it still has some challenges adapting to the situations where the uncertainties are not known (the level 4b of uncertainty). It is to some degree possible to implement surprise events in the RDM model, but the robust strategies might not be fit for dealing with these kinds of completely unexpected events .

Annotated Bibliography

Marchau, V. A. W. J., Walker, W. E., Bloemen, P. J. T. M., & Popper, S. W. (2019). Decision Making under Deep Uncertainty: From Theory to Practice (1st ed. 2019 ed.)

This open access book is book is funded by the Radboud University, the RAND Corporation, Delft University of Technology, and Deltares in 2019. It focuses on both the theory and practice associated with the tools and approaches for decision making in the face of deep uncertainty. One of the approached being unfold is RDM, but also Dynamic Adaptive Planning, Dynamic Adaptive Policy Pathways, Info-Gap Decision Theory, and Engineering Options Analysis are covered. It explores how the approaches and tools support the design of strategic plans under deep uncertainty, and their testing in the real world, including barriers and enablers for their use in practice.

Bhave, A. G., Conway, D., Dessai, S., & Stainforth, D. A. (2016). Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world. Climate Risk Management

This paper provides an interesting perspective of the RDM in the context of developing countries. It discusses how to apply RDM in developing countries and the strengths, weaknesses and potential pitfalls that might be experienced in doing so.

Lempert, R. J., Popper, S. W., & Bankes, S. C. (2003). Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis

This paper is from 2003 and out springs from the fact that the choices that are made in the beginning of this century will affect the next hundred years. This paper reframes the question "What will the long-term future bring?" to “How can we choose actions today that will be consistent with our long-term interests?”. Robust Decision making methods are covered and it is described and examplified how these methods provide analytic support to humans desire to ask "what-if" Despite the paper being 20 years old it is still relevant with e.g. the important decisions yet to be made regarding sustainable development.