Robust Decision Making (RDM)

Revision as of 21:39, 15 March 2022

Abstract: In some situations, project, program or portfolio management decisions must be made in the face of deep uncertainty. ^[1]. In this case, it is not possible to define probabilities for possible futures. Therefore, classical risk management tools are not adapted. ^[2]. Robust decision making (RDM) is one solution that can be used to properly account for this high level of uncertainty. Unlike traditional risk management tools, RDM helps decision makers choose not the optimal solution, but the solution (or set of solutions) that is the least bad compared to the set of possible futures. ^[1]. After presenting the general idea and main principles of RDM, this article aims to give a guideline so that readers can apply it to their project. This includes a step-by-step methodology and a very brief presentation of the type of tools needed. The relevance and limitations of RDM are discussed at the end of this article. In this article, if nothing is specified, the concepts used are defined according to the PMI standards.^[3].

Presentation of the RDM

General Idea

As defined by Dr Alistair Hunt , Robust decision making is “ a methodology which aims to identify adaptation options or strategies which can perform well over a wider range of possible futures. “. ^[4]. The particularity of this method compared to classical risk management tools is that the goal is not to find the optimal solution to a problem but to create a robust solution that is satisfactory for a large number of potential futures.^[5].

In other words, RDM is not about trying to find the most plausible future and adapting one's strategy to it. Instead, the essence of this method is to consider many potential future situations (see the following sections for more details) and evaluate strategies under all the different scenarios. Then, the most robust strategies are valued over those that are optimal in a specific situation. ^[1]. This method leads to choosing not the best but the least bad strategy. The general foundations and a step-by-step approach are described later in this article.

Fields of application

A situation of uncertainty and/or ambiguity implies having limited and imprecise knowledge of future events.^[3]. When these uncertainties are deep, it becomes so difficult to accurately anticipate the future that traditional prediction or risk management tools cannot be used effectively. Indeed, a high level of uncertainty implies that it is not possible to put reliable statistics or probability distributions on possible future scenarios. And classical approaches need this according to project management standards^[3].An alternative is to consider as many possible futures as possible in the evaluation process in order to reduce the risk that their solutions will not fit the actual future that will occur.

As already mentioned, one of the particularities of RDM is that it takes into account a very large number of scenarios in the decision-making process. It is therefore specifically adapted to the context of deep uncertainties. More precisely, these deep uncertainties can appear mainly in three situations: ^[1]:

- First, if the contextual uncertainties are profound. That is, if the characteristics of the context in which the project, program or portfolio will evolve cannot be precisely defined. This is the case, for example, when the prediction of the future requires climate modeling. Indeed, global warming will have impacts that are very difficult to describe and anticipate precisely. ^[6] .

- Second, uncertainties increase when “the set of policies has more rather than fewer degrees of freedom” ^[1]. This means that the more policies can change and influence your project, the more difficult it is to anticipate future situations.

- Finally, another source of uncertainty is the disagreement that may exist between different experts on the fields concerned by your projects. Indeed, if there is such a strong disagreement, it is impossible to establish a single reliable hypothesis. And so you have to consider many possible futures.

The fields of application that we can find are numerous. Here is a non-exhaustive list: ^[7] :defense, higher education, insurance, science and technology planning, counter-terrorism... In the current state of knowledge, RDM is very often used when a project, program or portfolio needs to include climate change in the decision. ^[4] .

RDM fondations

As mentioned in the literature ^[8] , RDM does not use computer models and data as a predictive tool to anticipate the best estimate of the future. Instead, RDMs use the principle of exploratory modeling. ^[9] . This means that a large number of possible futures with many different paths are generated to test the proposed strategies. The objective is not to rank the plausible scenarios but to explore as many as possible. Comparisons of the pre-decided strategies with a wide range of scenarios are then used to create a database of key performance indicators and outcomes. In this database, there are two main types of data.

The first is the set of data relevant to assessing whether a strategy is suitable for a specific future regarding the achievement of predefined goals. The data used to say whether a strategy is suitable or not for a situation really depends on that situation. There is an extensive literature that describes how to choose KPIs and gives examples of data that may be relevant. RDM can use these results and methodologies to include them in the process.

The second type of data is a description of the path used to generate the range of possible futures. That is, all the characteristics of one path that are relevant to differentiate it from another. This data is mainly generated by coupling algorithms capable of generating scenarios and software capable of exploring these scenarios. ^[5] - see steps-by-steps section for more info.

With this database, data scientists can then use analytics to identify common characteristics of futures that cause a strategy to fail or achieve its goals. ^[1]. They can also highlight the strengths and weaknesses of the strategy against a range of different scenarios. These first two steps can be repeated for a wide variety of input strategies and assumptions. Unlike many traditional risk assessment tools, the outcome of the analysis in the RDM process is not a predefined solution, but a set of key characteristics that identify under which assumptions, policies, and scenarios a strategy achieves its objectives.

When all of these characteristics are known, they can be passed on to decision makers who will perform a trade-off analysis to find a robust strategy. The strategies can then be modified or combined to find a robust one. That is to say “ones that meet multiple objectives over many scenarios “^[1]. The process described above can be repeated several times to reveal new strategies and result in an even stronger decision.

Because RDM is useful for defining strategy, it is used early in the project/program or portfolio life cycle.

Collaboration Human/machine

An important point of RDM is that it is based on human-machine collaboration. ^[1]. The strength of this method is that it combines the power of human decision making with the computational power of computers. Computers are used to generate and explore many different scenarios which are in turn used to stress the strategies devised by humans. When it comes to deep uncertainties with a large number of hypotheses to explore, computing power is absolutely necessary to cope with the mass of data to be processed. On the other hand, the decision-making part and the creation of potential trade-offs then rely solely on the decision-makers who can use the results of the previous computational steps as support. This mix between computer science and human intelligence implies the use of a different set of tools. As we can see from the different applications of the RDM method, it only defines the general type of tool you need for each step. The exact tools to use depend on the context and must be defined for each new project. A very quick overview and advice on the type of tools to use are detailed in the "step by step" section.

Step by Step methodology

The purpose of this section is to provide a detailed guide for the reader to apply MDM to a new project, program or portfolio management situation. Each subsection presents a step in the process and briefly discusses the tools useful for that section. The order of presentation is chronological. The different steps are based on the methodology defined by Robert Lempert. ^[8]

Illustration of RDM process by Swann Roussillon and inspired by Robert Lempert ^[8]

Define Strategies

As with any project, program, or portfolio management situation, decision makers must define a set of strategies. These strategies will likely evolve over the course of the RDM process, but a baseline is needed to start the process. ^[10]. For a new project with a high level of uncertainty, it may be useful to define a set of strategies to test rather than a single one. Indeed, later on, the process may lead to a combination of different strategies to obtain a robust strategy.

Define the uncertainties as much as possible

One of the crucial aspects of using RDM correctly is to be as complete as possible in describing the uncertainties. This is a huge basis for simulation modeling to be unable to generate a large number of plausible solutions. The sources of uncertainties that need to be explored are primarily: ^[1]:

- Contextual uncertainties. These are basically all the physical things included in your scope that you are not able to predict. One of the main examples of this part is the set of strategies that must include climate or environmental simulations. Since it is not yet possible to have precise predictions, the level of uncertainty is very high.

- The policy uncertainties. This section includes all elements related to decision making that may influence the outcome of your strategies and that are beyond your control. For example, future taxes, future environmental laws, future local restrictions, etc. may fall into this category.

- Uncertainties related to assumptions. Experts may disagree on certain assumptions related to your strategies. If the disagreement is significant, these assumptions create uncertainties that you must consider.

Set objectives and metrics for the Strategies

Later in the process, the goal will be to subject the different strategies to a wide range of possible futures and evaluate whether or not they are appropriate. But a preliminary step is to define how these evaluations can be done. ^[8]. At the end of this step, it is crucial to be clear about what is to be measured, how it is to be measured, and what values define an acceptable outcome. Objectives and measures are used to assess the degree of success of the project, program or portfolio. A typical example might be the mayor of a city who has plans to build a new dam to protect the urban area from flooding in the context of global warming. He/she might decide on the following set of objectives and actions:

- The future dam should reduce property damage in the event of a flood. The total property damage in dollars could be defined as the metric. And a maximum acceptable level can be decided ($100,000 for example).

- The future dam should protect the citizens. A goal of zero people injured could be defined.

- The future dam must be affordable for the city. The total price of construction and maintenance in dollars could be defined as the measure. And a maximum of $10 million might be an acceptable outcome.

Define a plan for conducting the simulation modeling

The simulation model is a key factor in RDM. There are two things that must be considered during the simulation.^[10]. First, the model must be able to generate a wide range of potential futures based on the uncertainties defined above. Second, the model must provide access to the different steps it creates in all potential futures. This is essential to allow humans to understand the impact of the assumptions made at each stage on the strategies. The first point requires a scenario generator and the second requires exploratory modeling software. It is then necessary to choose a software adapted to the domain and the scope of your project.

In addition, your simulations will generate a lot of data. In order to make it usable for the next steps, it is crucial to define how this data will be collected and stored.

A final point that needs to be considered during the planning phase is the computational power required by the scenario generation. As mentioned in the previous article (you can find the part about computing power in one of the previous articles), this can require a lot of computing performance. And even if computers are making more and more progress, too many scenarios can exceed the capacities you have or create a very long waiting time. It is therefore necessary to anticipate these aspects and to reduce the complexity of scenario generations if necessary.

More information can be found in the references dedicated to exploratory modeling. ^[9]

Define a strategy of exploration and analyze of the data produced by the simulation modeling

When all the scenarios have been run, you will end up with a large database. And it is the exploration and analysis of this database that will create value for RDM. So it's important to have predefined strategies to leverage that data effectively. RDM's methodology helps you define what you are looking for. In fact, the goal is to identify and visualize the key factors in the scenarios that make your strategies work or not.^[1] To do this, you can use classic data analysis and machine learning tools. You can also decide how you want to present your results. An important point to keep in mind is that you are not directly looking for an option to select. But you want to compare your different options. ^[1]

Run the simulation

This step consists only of applying the strategy defined in step D. The expected result is the database mentioned above and a way to visualize the path through the different futures.

Analyze and characterize the database produced

This step consists of applying the methodology defined in point e. The expected result is an identification and visualization of the key factors in the scenarios for which your strategies do or do not work. The goal here is not to make a conclusion but to create a support that will help decision makers make the trade-off analysis described below.

Realize a tradeoff Analysis

It is in this step that the value of the RDM is highlighted. In this phase, decision makers should use the results of the previous step to compare the different strategies. Specifically, the strengths and weaknesses of each method can be compared by considering the different scenarios. The goal is not to directly choose a strategy. Instead, decision makers may want to see if two or more strategies can be combined to produce a more robust strategy.^[1]. That is, a strategy that is better than the others when considering all possible futures.

Iterate the process if necessary

After analyzing the trade-offs, decision-makers can propose new strategies they would like to try or revisit some of the assumptions used to create the different scenarios. They can then start the whole process again, hoping that this will help them find a more robust solution. These new iterations take less time because all the methodologies and tools are the same as in the first iteration.

Limitations

As presented previously, one of the particularities of RDM is that it is well adapted to situations with a high level of uncertainty. But the relevance of using this method and not the classical probabilistic risk analysis is really limited when we go outside this context. ^[8]. Indeed, two main points are often mentioned in the literature:

- RDM requires high computing power because generating and exploring many possible futures has a high computational cost. ^[1]. Thus, if the uncertainties are not deep and can be handled by the probabilistic approach, RDM will use unnecessary computing resources.

- The result of RDM is to provide a robust strategy. This strategy is then not necessarily an optimal solution for a specific future, but a compromise that minimizes the negative aspects of many possible futures. In a context of low uncertainty, i.e. in a context where statistical tools can classify futures by probability, decision-makers will prefer to find an optimal solution for the most probable futures. The idea of finding the least bad strategy then becomes much less interesting than the idea of finding the best strategy. And then RDM is no longer the preferred method.

Another limitation of RDM is the complexity of the tools needed to create and explore the different scenarios.^[9] . First, these tools can be difficult to use and require an expert who should be part of the project. Second, it may seem that suitable scenario generators do not exist for very specific types of projects. In this case, they must be implemented before proceeding with RDM. And this requires significant human and financial resources.

Annotated Bibliography

Only the titles of the references are listed in this section. Links to the articles can be found in the references section..

• Decision Making under Deep Uncertainty

This book offers a comprehensive examination of the approaches and tools for designing plans under deep uncertainty and their application. It identifies barriers and enablers for the use of the various approaches and tools in practice. More specifically, a section is dedicated to the analyse of Robust Decision Making.

• Project Management Institute, Inc. (PMI). (2019). Standard for Risk Management in Portfolios, Programs, and Projects. Project Management Institute, Inc. (PMI)

This book is an update and expansion upon PMI's popular reference, The Practice Standard for Project Risk Management. This standard describes the concepts and definitions associated with risk management and highlights the essential components of risk management for integration into the various governance layers of portfolios, programs, and projects.

• Comparing Robust Decision-Making and Dynamic Adaptive Policy Pathways for model-based decision support under deep uncertainty

This paper compares Robust Decision-Making (RDM) and Dynamic Adaptive Policy Pathways (DAPP)-which is another approach of designing plans under deep uncertainty. To do so, it first describes the two tools before underlining their strengths and weaknesses.

• Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world

This review paper examines the potential to expand the geographical and sectoral foci of RDM. The Strengths and Weaknesses, data requirement and limitations of RDM are discussed. The article examines potential entry points for RDM approaches through Environmental Impact Assessments and Strategic Environmental Assessments.

• Ensuring Robust Flood Risk Management in Ho Chi Minh City

Ho Chi Minh City faces significant and growing flood risk. This study demonstrates how robust decision making can help Ho Chi Minh City develop integrated flood risk management strategies in the face of such deep uncertainty.

References

1. ↑ ^{1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12} Vincent A. W. J. Marchau Warren E. WalkerPieter J. T. M. Bloemen Steven W. Popper (2019)Decision Making under Deep Uncertainty https://doi.org/10.1007/978-3-030-05252-2
2. ↑ Project Management Institute, Inc. (PMI). (2019). Standard for Risk Management in Portfolios, Programs, and Projects. Project Management Institute, Inc. (PMI). Retrieved from https://app.knovel.com/hotlink/toc/id:kpSRMPPP01/standard-risk-management/standard-risk-management
3. ↑ ^{3.0 3.1 3.2} Project Management: A guide to the Project Management Body of Knowledge (PMBOK guide), 7th Edition (2021)
4. ↑ ^{4.0 4.1} Dr Alistair Hunt ROBUST DECISION MAKING https://econadapt-toolbox.eu/robust-decision-making
5. ↑ ^{5.0 5.1} Jan H. Kwakkel, Marjolijn Haasnoot, Warren E. Walker,Comparing Robust Decision-Making and Dynamic Adaptive Policy Pathways for model-based decision support under deep uncertainty,Environmental Modelling & Software, Volume 86, 2016, Pages 168-183, ISSN 1364-8152, https://doi.org/10.1016/j.envsoft.2016.09.017
6. ↑ Bhave, Ajay Gajanan, Conway, Declan, Dessai, Suraje and Stainforth, David A. (2016) Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world. Climate Risk Management, 14 . pp. 1-10. ISSN 2212-0963 DOI: 10.1016/j.crm.2016.09.004
7. ↑ Robert Lempert, Steven Popper, Steve Bankes. ROBUST DECISIONMAKING (RDM) https://millennium-project.org/wp-content/uploads/2020/02/22-Robust-Decisionmaking.pdf
8. ↑ ^{8.0 8.1 8.2 8.3 8.4} Robert Lempert, Nidhi Kalra, Suzanne Peyraud, Zhimin Mao, Sinh Bach Tan, Dean Cira, Alexander Lotsch (2013) Ensuring Robust Flood Risk Management in Ho Chi Minh City https://openknowledge.worldbank.org/bitstream/handle/10986/15603/WPS6465.pdf?sequence=1&isAllowed=y
9. ↑ ^{9.0 9.1 9.2} Agusdinata, Datu Buyung. (2008). Exploratory modeling and analysis: a promising method to deal with deep uncertainty.
10. ↑ ^{10.0 10.1} Lempert, Robert J., Steven W. Popper, and Steven C. Bankes, Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, Santa Monica, Calif.: RAND Corporation, MR-1626-RPC, 2003. As of March 05, 2022: https://www.rand.org/pubs/monograph_reports/MR1626.html