Robust Decision Making under Deep Uncertainty

Revision as of 00:44, 21 February 2022

Abstract

The uncertainty is highest in the beginning of a project, but it is also here decisions of crucial importance for the project's outcome are made. Furthermore, the world is constantly changing, making it impossible to make reliable decisions based on prediction of what the future holds. There is a need to move away from prediction to calculating the most robust decisions.

This article describes how to approach decisions under high uncertainty as a project manager or project stakeholder using robust decision making (RDM). RDM is a set of methods and tools developed over the last decade, primarily by researchers associated with the RAND Corporation. The article explains the XLRM Matrix for RDM and the Iterative Steps of RDM as well as how to apply these through exemplification. Limitations and challenges of the method are touched on, including the challenge of shifting to a new way of dealing with uncertainty.

It is inevitable to encounter uncertainties in projects, but you can do your best to investigate which strategies will perform best under these uncertainties. RDM is an iterative framework that offers a data-based assessment on future scenarios on which stakeholders and decision makers can base their decisions.

Deep Uncertainty and Project Management

A paradox of project planning is that when uncertainty is at its highest – in the beginning of a project – is also when the most crucial and determining decisions need to be taken. When time goes by and the project performer knows more about the project, there are less degrees of freedom to influence it. All projects face varying degrees of uncertainty which is often mentioned as a reason for their failures (source).

Uncertainty characterizes situations where the outcome of a project is likely to deviate from the estimated outcome. There are several levels of uncertainty and many operate with four levels. Level 1, 2 and 3 ranges from almost certainty of the outcome to a limited set of plausible outcomes with unknown probabilities. Level four represents the highest level of uncertainty, also called deep uncertainty. Level four can be distinguished into two types: a) where the future is bound around many plausible futures, and b) where the only thing that is known about the future is that it is not known. Type a is often due to lack of information, whereas type b is due to unpredictable events (Decision Making under Deep Uncertainty, V. A. W. J. Marchau et al.). We see the tendency that important problems faced by decision makers are characterized by a higher level of uncertainty and cannot be reduced simply by collecting information, because the uncertainties are unknowable at the time. Such situations are defined as decision making under deep uncertainty (source).

Different approaches are needed for handling decisions in deep uncertainty (level 4) than in level 1, 2 and 3. This is because these types of uncertainties involve factors of which probability distributions and possible outcomes are not known. There is a need for a new paradigm for decision making under deep uncertainty that is not based on predictions of the future (known as the “predict-then-act” paradigm). The “monitor and adapt” paradigm is more suited for deep uncertainty, which recognizes the need for taking the uncertainty into account and prepares and adapts for uncertain events.

Overview of RDM

Robust Decision Making (RDM) is a set of tools, methods, and processes that are developed for Decision Making under Deep Uncertainty when predictions of the future are not possible or risky. Robust decision making can help by:

- Identifying which strateg(ies) are most robust. This means which stratigy’s satisfactory per-formance is relatively insensitive to all or most of the uncertainties – compared to the other

- Identifying vulnerabilities of the robust strategies. This means combinations of model formulations and parameters where the strategy performs poor compared to other

- Suggesting new strategies which might better combat these vulnerabilities

- Characterizing the trade-offs involved in the choice among the options (source)

Foundation of RMD

RDM uses computation and quantitative models in a new way to provide better decisions under uncertain conditions. It combines decision analysis, Assumption-Based Planning, scenarios, and Exploratory Modelling to stress test strategies over a great number of possible paths into the future. By that, policy-relevant scenarios and robust adaptive strategies can be determined. RDM are often embedded in a decision support process called “deliberation with analysis” that promotes learning and consensus-building among stakeholders (Lempert et al. 2003).

Decision Analysis Empirical research shows that people make better decisions when using well-structured decision aids. RMD represents a type of decision analysis that relies on decision structuring frameworks, evaluation of consequences of alternative actions, identification of trade-offs among alternative options, and tools for comparing decision outcomes. (R. J. Lempert, 2019, s. 26) In RDM, the decision analysis is based on considering several strategies over a broad range of possible futures. The results are used to characterize weaknesses of the strategies and to identify and evaluate potential response to those weaknesses.

Assumption based planning RDM draws on assumption based planning. This implies the development of an organization’s plan and identification of load-bearing assumptions in this plan. This means identifying the potential break points that could cause the plan to fail. After identifying the assumptions, ABP considers shaping actions (actions designed to make the assumptions less likely to fail), hedging actions (those that can be taken if assumptions begin to fail), and signposts (trends and events to monitor in order to detect whether any assumptions are failing) (s. 28).

Scenarios The concept in RDM of multiplicity of plausible futures is drawn from the scenario analysis (Lempert et al. 2003). Scenarios are projected futures that require less confidence than probabilistic forecasts and that seek to represent different ways of looking at the world without a ranking of relative likelihood (Wack 1985). RDM draws from scenario analysis by analysing information about the future into small numbers of different cases that can help people to explore and communicate the deep uncertainty (s.28.)

Exploratory Modelling The concepts of decision analysis, assumption based planning and scenarios are combined in the exploratory modelling. Exploratory modelling provides RDM with a quantitative framework for stress testing and scenario analysis (s.29.)

Iterative steps of RDM

RDM follows the “deliberation with analysis” process. The deliberation with analysis means that parties to a decision consider their objectives and options; decision-relevant information is generated using system models; and the parties revisit their objectives, options, and problem framing influenced by the new quantitative information (NRC 2009). The process is best used in situations with different decision makers who face a changing environment and whose goals can change during collaboration with other parties.

(missing figure)

Step 1: Decision Framing

In the decision making framing step, stakeholders define the key factors of the analysis: “The decision makers’ objectives and criteria; the alternative actions they can take to pursue those objectives; the uncertainties that may affect the connection between actions and consequences; and the relationships, often instantiated in computer simulation models, between actions, uncertainties, and objectives”. The information is often filled into a 2X2 matrix called XLRM (Lempert, 2003 https://www.rand.org/pubs/monograph_reports/MR1626.html ):

X: Uncertainties L: Policy Levers R: Relationships M: Measures of performance

(missing matrix)

Step 2: Evaluate Strategies across Futures

In the second step, the decision framework is executed through a simulation process which is used to evaluate proposed strategies across the multiple plausible futures. This means that the models (R) are run over the strategies (L) and uncertainties (X) to output the performance (M). To create the futures, different assumptions of the uncertainties outcome are combined. This can give extreme amounts futures if the uncertainties are many and broad, which is why this step generates a large database of results, when the strategies are evaluated. Depending on the number of strategies and the number of uncertainties, it might be more than thousands of cases.

Step 3: Vulnerability Analysis

To analyse the vulnerabilities, the data is visualised in the third step. RDM commonly uses Scenario Discovery (SD) algorithms (s. 31). By doing that, it is possible to visualise which factors distinguish futures where strategies hit or miss their goal (s.31). In other words, under which set of uncertain conditions / in what futures does a particular strategy fail to meet stakeholder goals? (https://www.rand.org/pubs/tools/TL320/tool/robust-decision-making.html) This way of assessing the futures can remove the bias that can be present when analysing scenarios qualitatively. This process is often helped out by a data mining process.

Step 4: Tradeoff Analysis

In this step, the scenarios can be used to reveal the trade offs among strategies. This can for example be done by plotting the performance of a strategy as a function of the likelihood of the relevant scenario to assess the futures when choosing one strategy over another.

5. New Futures & Strategies

The analysis made can then be used to propose new, more robust strategies that provide better trade offs.

Example of application

The following illustrative example of application of RDM is drawn from RAND Corporation, 2014: THE ROBUST DECISION MAKING FRAMEWORK.

The focus of the example is the long term planning of a small water utility. The aim is not to provide results but to show how the methodology can be utilized to make the method more hands-on and tangible.

Step 1:

The stakeholders and decision makers held a participatory scoping exercise and agreed on the following XLRM matrix:

(missing matrix)

(M) Measure of performance: Safe yield being raw water storage above 75% capacity 90% of the time (R) Relationships: A model that simulates daily demand and supply and tracks water storage in the system (X) Uncertainties: - 10 different climate projections - customer demand between 75 and 120 mgd - contract size being small, medium and big. (L) Policy Levers: The performance of the current infrastructure

Step 2:

Following RDM, the utility evaluated a wide range of futures using its systems model. Twen-ty futures were specified that uniformly sample across the uncertainties; climate projections, projections of future customer demand, and the sizes of possible wholesale contracts. The performance of its existing system were evaluated across the futures and the performance was measures through the safe yield. The information were put into a spreadsheet. See the figure for example of future 5 in the spread sheet. For that exact future, the safe yield was calculated to be 94 percent.

(missing figure)

Step 3:

(missing graph)

The graph shows the result of each future with respect to each of the uncertain factorsCli-mate projections are on the x axis, demand estimates are on the y axis, while the size of the circle indicates the wholesale size. The number below the mark is the safe yield result for that specific combination of factors. By using datamining, the vulnerable conditions were detmermined: - Demand ≥ 95 million gallons per day (mgd) - Climate projection: 6-10 (warmer/drier projections) - Wholesale type: Medium or Large These vulnerable spectres define a subset of the futures – the upper right corner. In this case it is 5 out of the 20 futures that have the vulnerable conditions. 4 out of these are actually vulnerable, meaning that they do not meet the goal of safe yield of 90%. 4 out of 7 failing futures are described by the vulnerable conditions.

Step 4:

In this example, the utility might be able to find a more robust strategy than the current base-line shown in the figure, and by that making it possible to meet the goals under higher de-mands and worse climate conditions. But this more robust strategy might also be more cost-ly, why the decision makers needs to decide if they want to trade off cost for robustness or the opposite.

Step 5:

This short description of the largest vulnerability was included in a discussion regarding what strategies could be implemented to improve reliability under these specific conditions.

Limitations and Challenges

Missing section

Annotated Bibliography

Missing section

Bibiliography

Missing section