RDM
(→RDM in general) |
(→The foundation of RDM) |
||
| (160 intermediate revisions by one user not shown) | |||
| Line 1: | Line 1: | ||
| − | + | ''Written by Rasmus Engberg'' | |
| − | ''' | + | |
| − | |||
| − | == | + | '''Robust decision-making (RDM)''', is a relatively new and innovative methodology in the science of decision support. The RDM framework is a key tool in decision-making processes under deep uncertainty and provides decision-makers with a method to make a plan for the future without having to predict it <ref name="Lempert2000"> Lempert R.J., Schlesinger M.E. (2000), Robust strategies for abating climate change. Climatic Change, Springer Netherlands, Volume 45, Page 387–401, DOI: 10.1023/a:1005698407365 </ref>. Decisions are taken everywhere and on all levels, whether it comes to project, program or portfolio management, there will always be decisions to make. RDM is focused on supporting [[Decision Making Under Deep Uncertainty (DMDU)]] concerning decisions with long-term consequences, which according to The standard for portfolio management more often applies within portfolio management compared to program or project management <ref name = "Standard2017"/>. As RDM is used to support groups of decision-makers in finding adaptive long-term strategies that make it possible to succeed despite the level of complexity or uncertainty associated with the future, RDM is more applicable within portfolio management, however, whether it is used for project, program or portfolio management, it will most often be in relation to policy challenges. RDM has e.g. been used to analyze and support decision processes concerning climate change, infrastructure building, water management and coastal planning which all are very complex and long-term issues involving a high degree of uncertainty<ref name="Rand.com"> Lempert R.J., Fichbach J.R. (2021), Robust Decision Making: A Tool to Help Address Climate Change, https://www.rand.org/pardee/projects/rdm-and-climate-change.html </ref>. When decision-makers have to make long-term decisions they're often called upon to anticipate future needs, resources and circumstances. The problem with the traditional methods based on predictions is that decisions made on predictions are less reliable the farther the prediction reach forward in time, as time entails the prediction to become more vulnerable to uncertainty <ref name="Lempert2010"/>. This uncertainty could e.g. be unforeseen economic crashes, terrorism attacks, political instability, climate change, chains of actions or reactions leading to more possible outcomes than a single prediction can handle. RDM uses computer simulations and advanced modelling techniques to stress-test strategies not only against one future but against thousands or millions of possible futures, in order to acquire knowledge from a wide range of diverse futures <ref name="Lempert2019"> Bloemen P., Lempert R.J, Marchau V., Walker W., Popper S. (2019), Robust Decision Making in Decision Making under Deep Uncertainty. Springer, Cham. https://doi.org/10.1007/978-3-030-05252-2 </ref>. The main purpose of RDM is not to make a better prediction but through its concepts and processes to contribute with knowledge for the decision-makers to be able to design more robust strategies, that perform no matter what the future holds.<ref name="Lempert2010">Lempert R.J., Groves D.G. (2010), Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west, Technological Forecasting and Social Change, Volume 77, Issue 6, Pages 960-974, https://doi.org/10.1016/j.techfore.2010.04.007.</ref>. In this article, the concept of RDM, its purpose, and RDMs foundation will be analysed. Further guidance on how RDM is implemented and used, as well as an analysis of when RDM is applicable, will be presented. Finally, a critical reflection on RDM will seek to find and enlighten the limitations of RDM. This article intends to provide a general insight into RDM, its capabilities, application and limitations but as the execution of RDM is quite complex compared to other APPPM tools and methods, interested readers are suggested to read the annotated bibliography for more in-depth information on RDM. |
| − | + | ||
| − | |||
| − | |||
| − | According to R.J Lempert, the traditional “predict-then-act” framework has proven to be successful for decades, which both a large number of theories and mathematical techniques is proof on. RDM was original developed to make policymakers able to make better decisions about issues that could have consequences in the long-term. RDM | + | == Handling decisions under deep uncertainty == |
| + | === History of RDM === | ||
| + | Decision-making under uncertainty is not a new problem, but the way to approach and handle these often complex issues has changed over time. RDM has become one of the more common approaches to handle DMDU. The first person to describe the RDM methodology was Robert J. Lempert from the RAND cooperation in 1997 in "When we don't know the costs or the benefits: Adaptive strategies for abating climate change "<ref name="Lempert1997"> Lempert R.J., Schlesinger M.E., & Bankes S.C. (1996), When we don’t know the costs or the benefits: Adaptive strategies for abating climate change. Climatic Change, Volume 37, Issue 1, 207–208. </ref>. But R.J. Lempert was not the first one who tried to develop a new method to handle the increasing number of decisions under deep uncertainty. The company Envolving Logic had in the 1990s, parallel with the RAND corporation and R.J. Lempert's development of RDM, developed the first software tool for DMDU. Their software was called the Computer-Assisted Reasoning system (CARsTM) and they described it as a tool to support decision-makers in the area of the [[Robust Adaptive Planning (RAP)]] methodology, while the RAND corporation and R.J. Lempert instead adopted the Robust Decision Making (RDM) methodology <ref name="1990s"> Bankes S.C., Lempert R.J., Popper S.W. (2001), Computer-assisted reasoning. Computing in Science & Engineering, Volume 3, Issue 2, Page 71–77 </ref>. | ||
| + | |||
| + | According to R.J Lempert, the traditional “predict-then-act” framework has proven to be successful for decades, which both a large number of theories and mathematical techniques is proof on <ref name="Lempert2010" />. RDM was original developed to make policymakers able to make better decisions about issues that could have consequences in the long-term. RDM was mostly focused and used for decisions about climate changes at the beginning of its lifetime. It was not a coincidence that RDM mainly were used for decisions regarding climate changes, because it was particular climate changes commonly known property of being embraced with deep uncertainty, that made RDM suitable for this area specifically <ref name="Lempert2019"> Marchau V.A.W.J., Walker W.E., Bloemen P.J.T.M, Popper S.W. (2019), Decision Making under Deep Uncertainty, From Theory to Practice, Springer Cham, DOI: https://doi.org/10.1007/978-3-030-05252-2</ref>. With the arrival of the RDM, it was no longer necessary to determine the probabilities of the future consequences of climate changes in order to make a strategy, which many also believe is something near impossible. The need for RDM is thus targeted more towards complex decision processes under deep uncertainty, where relatively simple prediction analysis is not enough and often will be misleading <ref name="Rand.com"/>. R.J. Lempert thought it would be better to seek a robust solution rather than trying to predict the future and then choose the solution that fits the predicted future best. He disagreed with the traditional framework for assessing alternatives in decision processes, in which he thought predictions were imposed too great a significance. He argued that the reason for decision-makers to use predictions were to make the future less scary and that ''"the quest for prediction probably fills some deep human need. Even though the accuracy of most predictions has proven to be poor"''"<ref name="Lempert2000"/>. A combination of new technologies and the need for a method to handle more complex issues resulted in the development of RDM, that instead of finding the alternative that seems to be the best based on specific predictions and probabilities, uses modern technology to analyse and evaluate alternatives against thousands or even millions of possible scenarios. The RDM method has given decision-makers a tool to identify trade-offs by testing strategies against different scenarios. Decision-makers can now determine which strategies that perform best under a large range of possible scenarios. If used correctly, RDM should make the decision-makers able to develop the most robust strategies that can handle the future, regardless of how the future ends up being. | ||
| + | |||
'''RDM consist of the following key elements:''' | '''RDM consist of the following key elements:''' | ||
| − | * Testing of multiple plausible futures. The combination of scenarios for the future should be significantly diverse. The reason for this is that the main idea of RDM is that you are interested in analysing how different strategies perform in as many different futures as possible. These multiple scenarios can also correspond to different persons in the decisionmaking groups view of the world. Instead of spending time debating which prediction of the future too chose, as one would do in the traditional “predict-then-act” framework, the decisionmaking group can use their time more effective while testing everybody's idea of the future at the same time. | + | * Testing of multiple plausible futures. The combination of scenarios for the future should be significantly diverse. The reason for this is that the main idea of RDM is that you are interested in analysing how different strategies perform in as many different futures as possible. These multiple scenarios can also correspond to different persons in the decisionmaking groups view of the world<ref name="Lempert2003"/>. Instead of spending time debating which prediction of the future too chose, as one would do in the traditional “predict-then-act” framework, the decisionmaking group can use their time more effective while testing everybody's idea of the future at the same time. |
| − | * The goal is not to find the best strategies but to find the most robust ones. The main focus in RDM is to find strategies that perform well over most if not all possible scenarios. | + | * The goal is not to find the absolute best strategies but to find the most robust ones. The main focus in RDM is to find strategies that perform well over most if not all possible scenarios<ref name="Lempert2007"/>. |
| − | * Apply adaptive strategies. One way to secure that strategies are robust is to choose some who are adaptive. Adaptive means that when the future comes, the strategies can adapt to the new circumstances | + | * Apply adaptive strategies. One way to secure that strategies are robust is to choose some who are adaptive. Adaptive means that when the future comes, the strategies can adapt to the new circumstances and still perform well under the new conditions<ref name="Standard2017"/>. |
| − | * The computer is used to ease the human job, not to give the final recommendation of which specific strategy | + | * The computer is used to ease the human job, not to give the final recommendation of which specific strategy should be used<ref name="Lempert2003"/>. |
| − | |||
| − | |||
| − | |||
| − | |||
| + | === The foundation of RDM === | ||
| + | The primary theoretical focus lays around uncertainty theories more specifically in the area of [[Decision Making Under Deep Uncertainty (DMDU)]], the reason for this is that the RDM methodology is used for developing strategies under deep uncertainty. As stated in ''The standard for portfolio management'', uncertainties are in general just the result of a lack or limited knowledge about a certain state in time and are almost impossible to avoid when making decisions in our modern world <ref name="Standard2017"> The standard for portfolio management — fourth edition. (2017), ProQuest Ebook Central, Page 91-94, https://ebookcentral-proquest-com.proxy.findit.dtu.dk/lib/DTUDK/detail.action? </ref>. Many books have been written about decisionmaking under uncertainty, but only a very limited amount includes DMDU. Even though the literature about DMDU is limited, however increasing, several tools and different analytical approaches exist. They share the underlying paradigm that they all seeks to reduce strategies vulnerability against the uncertainties associated with the future, which in some literature is defined as [[Assumption Based Planning (ABP)]]<ref name ="ABP"> Dewar J.A., Builder C.H. W., Hix M., Levin M.H. (1993), Assumption-Based Planning: A planning tool for very uncertain times. MR-114-A, RAND, Santa Monica, California. </ref>. | ||
| + | RDM is developed based on some already existing methods, paradigms and approaches, the most important ones will be reviewed in the following section. The foundation of RDM is visualised in the figure below. It shows that RDM is based on three main concepts, Decision Analysis, Assumption-Based Planning and Scenarios which then are united in the forth concept, Exploratory modelling. | ||
| − | <ref name ="Lempert2007"/> | + | [[File:Foundation1.png|500px|thumb|center|The foundation of RDM]] |
| + | |||
| + | ==== Decision Analysis ==== | ||
| + | A part of RDM's foundation is [[Decision Analysis (DA)]]. The reasoning behind DA is that the human, by including decision tools, make better decisions. It applies whether we are a group or an individual. DA provides the decision-makers with tools to enlighten what affect certain decisions will have on different situations under uncertainty.<ref name="NRC"> National Research Council (2001), Theoretical Foundations for Decision Making in Engineering Design, Washington, DC: The National Academies Press. https://doi.org/10.17226/10566</ref>. DA can take the form as e.g. [[Decision Trees]] or [[Expected Value (EV)]] calculations, but for more complex decisions under deep uncertainty, it is necessary to include more advanced methods and computer modelling. It could e.g. be RDM which is categorised as a quantitative DA method<ref name="Lempert2019"/>. DA can be divided into two different approaches, the first approach is called "agree-on-assumption" or "predict-then-act" and the second one, which RDM follows, is an "agree-on-decisions" approach. The idea of stress testing is one concept RDM has derived directly from the "agree-on-decision" approach. | ||
| + | The first approach seeks to rank different strategies based on agreed probabilities of how likely it is that the world is in a certain state in the future, known from the traditional DA framework. The second approach test different strategies against multiple scenarios of the future, which is used to find vulnerabilities of the strategies and mitigating actions to handle these vulnerabilities <ref name="Kalra"> Kalra N., Hallegatte S., Lempert R.J., Brown C., Fozzard A., Gill S., et al. (2014), Agreeing on robust decisions: A new process of decision making under deep uncertainty, Policy Research Working Paper. World Bank </ref> <ref name="Lempert2004"> Lempert R.J., Nakicenovic N., Sarewitz D., Schlesinger M. (2004), Characterizing climate change uncertainties for decision-makers — An editorial essay. Climatic Change, Volume 65, Issue 1–2, Page 1–9</ref>. According to C. Helgeson (2018), the "agree-on-decision" approach as RDM follows, can be summarized as an approach to support the decisionmaking group in the process of framing decisions under deep uncertainty <ref name="Helgeson"> Helgeson C. (2018), Structuring decisions under deep uncertainty. Topoi, Page 1–13 </ref>. | ||
| + | |||
| + | ==== Assumption-Based Planning ==== | ||
| + | RDM uses the properties of stress testing to challenge a variety of scenarios and strategies, to reduce the possibility of failure by improving the decision-makers understanding of when, why and what can go wrong. More specifically, RDM takes advantage of a certain practice of this, called [[Assumption-Based Planning (ABP)]] <ref name="Dewar2002"> Dewar J.A. (2002), Assumption-based planning—A tool for reducing avoidable surprises, Cambridge: Cambridge University Press. </ref>. ABP takes a starting point in a written plan and then seeks to identify the underlying assumptions on which the existing plan is based on. Only crucial assumptions, as if not goes as expected can lead to failure, is relevant in ABP. Further, ABP structures the crucial assumptions and uses them in the framework seen in RDM to develop adaptive strategies. Adaptive strategies or adaptive planning consist of several key elements, which all share that they in order to uncover uncertainties, uses computational power <ref name="Walker2013"> Walker W.E., Haasnott M., Kwakkel J.H. (2013), Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty, Sustainability, Volume 5, Page 955-979, doi:10.3390/su5030955 </ref>. The ABP approach secures that decision-makers using RDM tests a large number of uncertainties and develops both actions to be taken in the short-term and long-term, but with the possibility for long-term actions to change according to how situations develop. This includes developing signposts/signals that if observed in the future, will result in different actions to secure that the adaptive strategies won't fail <ref name="Walker2001"> Walker, W.E., Rahman, S.A., Cave J. (2001), Adaptive policies, policy analysis, and policymaking. European Journal of Operational Research, Volume 128, Page 282–289 </ref>. | ||
| + | |||
| + | ==== Scenarios ==== | ||
| + | Scenario analysis is another concept RDM uses. It is a way to represent different descriptions of plausible future states. Scenario analysis contributes to dividing different future states into multiple scenarios which makes it easier for the decision-makers to analyse and discuss. In contrast to ranking future states based on probabilities, scenario analysis analyses different views of how a situation will develop. Focusing on possibilities instead of probabilities gives the decision-makers the possibility to analyse more diverse events, compared to if they had to focus on only the events that most likely would happen. At the same time, the decision-makers are faced with more extreme and unexpected future events which help to test the relevant strategies against unlikely and more extreme scenarios than one might normally do with traditional frameworks <ref name="Schoe1993"> Schoemaker P.J.H. (1993), Multiple scenario development: Its conceptual and behavioural foundation. Strategic Management Journal, Volume 14, Issue 3, Page 193–213 </ref> <ref name="Gong2017"> Gong, M., Lempert, R. J., Parker, A. M., Mayer, L. A., Fischbach, J., Sisco, M., et al. (2017), Testing the scenario hypothesis: An experimental comparison of scenarios and forecasts for decision support in a complex decision environment. Environmental Modeling and Software, Volume 91, Page 135–155 </ref>. There exist different types of algorithms to do scenario analysis and the ones RDM uses is a quantitative approach of [[Scenario Discovery (SD)]] algorithms. In RDM, the decisionmakers use the final scenarios from the scenario analysis to capture the most important findings from the ABP stress test, which at the end of the RDM forms the basis for the decision-makers to make adaptive strategies<ref name="Lempert2014"> Groves D. G., Lempert R.J. (2007), A new analytic method for finding policy-relevant scenarios, Global Environmental Change, Volume 17, Page 73–85 </ref>. | ||
| + | |||
| + | ==== Exploratory Modeling ==== | ||
| + | RDM would not exist without Exploratory Modeling (EM). EM is particularly beneficial when it is not possible to use a model, because of e.g. missing data or if the future is associated with great uncertainty. It unites DA, ABP and SD and combines these concepts, by running the same model with sometimes only small changes in the input, even though many would normally argue that it is not good practice to run a model many times. In RDM this repeating is one of the key factors to achieve the targeted knowledge and to develop a large database of results. Another benefit of EM is that it does not have any starting point, and thus there are no restrictions for the possibilities of exploring different plausible futures and strategies <ref name="Lempert2019"/>. | ||
| + | EM can be used to analyse a variety of topics but for RDM, the goal of using EM is focused on finding strategies with a high degree of robustness<ref name="Weaver2013"> Weaver C.P., Lempert R.J., Brown C., Hall J.A., Revell D., Sarewitz, D. (2013), Improving the contribution of climate model information to decision making: The value and demands of robust decision frameworks. WIREs Climate Change, Volume 4, Page 39–60 </ref>. Besides being preferred because of the ability to support decisions under deep uncertainty, EM's computational simplicity is a big advantage, especially compared to alternative approaches that include Dynamic Programming or different optimization methods<ref name="Lempert2019"/>. | ||
| + | |||
| + | == Application of RDM == | ||
| + | RDM has been used to support a wide range of decisions, but RDM is not suitable in any decision process. RDM is most suitable if all or some of the following characteristics apply to the problem that needs to be handled. The primary condition is that there is a high degree of uncertainty (deep uncertainty) associated with the problem, other conditions that to some extent is related to deep uncertainty is if the problem is very complex, or if the decision-makers have a wide range of possible options to choose from <ref name ="Lempert2007"/>. It also includes if it is a problem that needs computational power in order to be solved, e.g. if it is necessary to test a large number of strategies against multiple scenarios. In general, RDM applies to most problems regarding DMDU as long as the decision-makers are not bound to a small number of mitigating possibilities <ref name="Weaver2013"/>. For less complex issues, it will normally be preferred to use a "predict-then-act" methodology, even though RDM in most cases also would be able to support the decision process in these cases. However, it would be an over-complication of the decision process, which is why RDM is not used in all decision processes. | ||
=== Steps in RDM === | === Steps in RDM === | ||
| − | [[File: | + | [[File:Figur.png|500px|thumb|right|Steps in an RDM analysis<ref name="Lempert2019"/>]] |
| − | ''' | + | Depending on the literature, the framework for an RDM may vary slightly, but overall the content is similar. The figure shown here is the latest version of R.J. Lempert's frameworks (founder of RDM) and is based on previous versions of the framework <ref name="Lempert2010"/>. A more in-depth go-through of the steps can be found in R.J. Lempert's book ''Robust Decision Making in Decision Making under Deep Uncertainty''<ref name="Lempert2019"/>. |
| − | + | ||
| + | It is important to keep in mind that RDM is an iterative process, which the figure also shows, and that the new knowledge acquired in each step of the RDM can lead to a necessity for re-framing the problem and its challenges. | ||
| + | ====Step 1 Decision Framing==== | ||
| + | |||
| + | As shown in the figure, the stakeholders start with a decision framing exercise, in which they have to define the key factors relevant for further analysis: | ||
* Objectives and criteria | * Objectives and criteria | ||
* Strategies/Alternatives that successfully can handle the objectives | * Strategies/Alternatives that successfully can handle the objectives | ||
* Uncertainties and their consequences | * Uncertainties and their consequences | ||
| − | * Relationship between strategies, objectives and uncertainties | + | * Relationship between strategies, objectives and uncertainties |
| + | ====Step 2 Evaluate Strategy Across Futures==== | ||
| − | + | The next step is to use an "agree-on-decision" approach"<ref name="Lempert2019"/>. To do so, RDM uses simulations of the strategies found in Step 1, against a variety of plausible scenarios of the future. These simulations generate a large number of results, which is merged into a large database and the results are evaluated across the different futures <ref name="Lempert2003"/>. | |
| − | + | ||
| − | + | ====Step 3 Vulnerability Analysis==== | |
| − | + | ||
| + | Step 3 tries to identify, explore and characterize vulnerabilities <ref name="Lempert2003"/>. The decision-making team can use visualisations and data analytics based on the database made in step 2. In order to identify the key factors that contribute to the success or failure of the strategies, it is common practice to use [[Scenario Discovery (SD)]] algorithms. The use of SD algorithms results in clusters of futures that will enlighten which factors the strategies are most vulnerable to<ref name="Lempert2019"/>. | ||
| − | |||
| − | |||
| + | ====Step 4 Tradeoff Analysis ==== | ||
| − | + | A tradeoff analysis is performed on the scenarios from step 3. In step 4 different strategies are evaluated in order to analyse the tradeoffs among different strategies<ref name="Lempert2000"/>. This is often done by making multi-objective tradeoff plots in which tradeoff curves can be used to e.g. compare cost against reliability for different scenarios. The tradeoff curves can be used by the decision-makers to determine what balance they evaluate to be most suitable among the objectives defined in step 1<ref name="Lempert2019"/>. | |
| − | + | ||
| − | === | + | ====Step 5 New Futures and Strategies==== |
| − | + | ||
| + | Finally, the decision-making team can then use the findings from step 4 to evaluate if they have found any robust strategies, with better tradeoffs, compared to existing strategies <ref name="Lempert2003"/>. The final strategies from the RDM will often be what you call [[Adaptive Decision Strategies]], which are strategies that adapt depending on how the future develops. The adaptive strategies consist of a short-term strategy with actions to be taken in the near future and so-called signposts which are predesignated signals that when observed, certain actions will be taken. These adaptive strategies can either be established from a jury of experts <ref name ="Lempert2003"> Bankes S.C., Lempert R.J., Popper S.W., (2003), Shaping the next one hundred years: New methods for quantitative, long-term policy analysis, Santa Monica, CA, The RAND Corporation, Technological Forecasting and Social Change — 2004, Volume 71, Issue 3, Page 305-307 </ref>, or the most suitable combination of near-term actions, signposts and actions to be taken if the signals are observed, can be determined with the help of optimization algorithms<ref name="Lempert2007"> Collins M., Lempert R.J., (2007), Managing the risk of uncertain threshold responses: Comparison of robust, optimum, and precautionary approaches. Risk Analysis, Volume 27, Issue 4, Page 1009–1026 </ref>. | ||
| − | == | + | == Limitations of the RDM Framework == |
| − | |||
| − | |||
| + | === Limitations === | ||
| + | RDM is not an easy method to follow and it certainly has it's requirements in order to be performed correctly, but as the philosophy behind RDM is ''" that it is better to be roughly right than precisely wrong"'', the models used to perform the RDM is often relatively simple and fast models. The problem is that even if a model solely is simple, problems can occur when it is necessary to run the model thousands or even millions of times, with a large number of different strategies and scenarios, which RDM involves. It sets requirements for the computer power and amount of data which in some cases, e.g. in developing countries, maybe doesn't exist causing it to be almost impossible to perform RDM <ref = name"philosophy"> Bhave A.G., Conway D., Dessai S., Stainforth D.A. (2016), Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world, Climate Risk Management, Volume 14, Pages 1-10, ISSN 2212-0963, </ref>. Besides data and computing requirements, then expert knowledge on the field of concern could also be needed. If the decision-making group does not possess this knowledge, they have to hire external experts which can be costly. It is also worth bearing in mind that even with the involvement of expert knowledge, the execution of RDM is based on subjective assessments, which is influenced by the perceptions of the decision-makers<ref name="econadapt"> Econadapt - Toolbox, Robust Decision Making, Retrieved February 16, (2021). https://econadapt-toolbox.eu/robust-decision-making </ref>. It means that RDM is only as good as the data used for it. However, performing thousands or millions of simulations significantly minimizes the risk but will never be able to eliminate it. | ||
| − | + | A limitation could also be the necessity of external involvement of experts to perform the modelling, there even though the models are simple, acquires some computer modelling skills to perform. Another limitation, as the annotated literature on RDM shows, is that the RDM methodology is mostly used to support DMDU for policy challenges. Besides the fact that it has mainly been used in connection with long-term issues. Among other things, this is due to the fact that the uncertainty associated with a given decision is largely related to time. Short-term strategies will only very rarely be associated with a high degree of uncertainty (deep uncertainty), which is why RDM is primarily limited to long-term decisions <ref name="Lempert2019"/><ref name="Lempert2003"/>. RDM is thus not a tool you decide to use without first familiarizing yourself with the requirements the execution of an RDM place on the decision-makers, their knowledge and competencies. | |
| + | == Annotated bibliography == | ||
| + | '''Bloemen P., Lempert R.J, Marchau V., Walker W., Popper S. (2019), Robust Decision Making in Decision Making under Deep Uncertainty''' <ref name="Lempert2019"/>. | ||
| − | + | This book is some of the latest literature about DMDU. It contains various approaches, applications, and implementations to handle DMDU, including RDM. It provides an in-depth guide on how RDM is performed and a concrete example of the implementation of RDM, both are very useful to look into if one wants to perform RDM. Besides being one of the newest books it is also one of the most detailed so for those who want to make use of RDM this book is a must-read. | |
| − | |||
| − | |||
| − | + | '''Collins M., Lempert R.J. (2007), Managing the risk of uncertain threshold responses: Comparison of robust, optimum, and precautionary approaches.''' <ref name="Lempert2007"/> | |
| − | + | This article studies different approaches for decision-makers to find robust strategies. It discusses the tradeoffs between different approaches which can be interesting for those who consider using RDM. Two of the three approaches is directly related to R.J. Lempert's work while the last one is another very used approach described by J. Rosenhead. The article can contribute to a deeper understanding of the tradeoff between performance vs. sensitivity which is key in the RDM framework. | |
| + | '''Bankes S.C., Lempert R.J., Popper S.W. (2003), Shaping the next one hundred years: New methods for quantitative, long-term policy analysis''' <ref name="Lempert2003"/>. | ||
| − | + | ''Shaping the next one hundred years'' was groundbreaking in relation to the hypothesis it was built around back in 2003. The authors stated that with the help of computers, it was possible to change the way long-term futures is predicted. They suggested scenario generator- and exploratory modelling software. Among other things, they mention in the book that the long-term future is defined by deep uncertainty. They also talk about robustness and adaptive strategies, all something that can be related to RDM. | |
| − | |||
| − | |||
| + | '''Lempert R.J., Groves D.G. (2010), Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west''' <ref name="Lempert2010"/> | ||
| + | |||
| + | This article is a case study that employs RDM in a real-world case regarding ''adaptive policy responses to climate change for water management agencies in the American west''. Both authors work for the RAND cooperation, who are experts and partly founders of the RDM framework. The article contributes with an in-depth analysis of how RDM is used on a real-world case, while is also discusses the benefits of using RDM instead of a traditional "predict-then-act" framework, in the process of creating adaptive decisions strategies. | ||
== Bibliography == | == Bibliography == | ||
<references/> | <references/> | ||
| − | |||
| − | |||
| − | |||
| − | |||
Latest revision as of 19:47, 28 February 2021
Written by Rasmus Engberg
Robust decision-making (RDM), is a relatively new and innovative methodology in the science of decision support. The RDM framework is a key tool in decision-making processes under deep uncertainty and provides decision-makers with a method to make a plan for the future without having to predict it [1]. Decisions are taken everywhere and on all levels, whether it comes to project, program or portfolio management, there will always be decisions to make. RDM is focused on supporting Decision Making Under Deep Uncertainty (DMDU) concerning decisions with long-term consequences, which according to The standard for portfolio management more often applies within portfolio management compared to program or project management [2]. As RDM is used to support groups of decision-makers in finding adaptive long-term strategies that make it possible to succeed despite the level of complexity or uncertainty associated with the future, RDM is more applicable within portfolio management, however, whether it is used for project, program or portfolio management, it will most often be in relation to policy challenges. RDM has e.g. been used to analyze and support decision processes concerning climate change, infrastructure building, water management and coastal planning which all are very complex and long-term issues involving a high degree of uncertainty[3]. When decision-makers have to make long-term decisions they're often called upon to anticipate future needs, resources and circumstances. The problem with the traditional methods based on predictions is that decisions made on predictions are less reliable the farther the prediction reach forward in time, as time entails the prediction to become more vulnerable to uncertainty [4]. This uncertainty could e.g. be unforeseen economic crashes, terrorism attacks, political instability, climate change, chains of actions or reactions leading to more possible outcomes than a single prediction can handle. RDM uses computer simulations and advanced modelling techniques to stress-test strategies not only against one future but against thousands or millions of possible futures, in order to acquire knowledge from a wide range of diverse futures [5]. The main purpose of RDM is not to make a better prediction but through its concepts and processes to contribute with knowledge for the decision-makers to be able to design more robust strategies, that perform no matter what the future holds.[4]. In this article, the concept of RDM, its purpose, and RDMs foundation will be analysed. Further guidance on how RDM is implemented and used, as well as an analysis of when RDM is applicable, will be presented. Finally, a critical reflection on RDM will seek to find and enlighten the limitations of RDM. This article intends to provide a general insight into RDM, its capabilities, application and limitations but as the execution of RDM is quite complex compared to other APPPM tools and methods, interested readers are suggested to read the annotated bibliography for more in-depth information on RDM.
Contents |
[edit] Handling decisions under deep uncertainty
[edit] History of RDM
Decision-making under uncertainty is not a new problem, but the way to approach and handle these often complex issues has changed over time. RDM has become one of the more common approaches to handle DMDU. The first person to describe the RDM methodology was Robert J. Lempert from the RAND cooperation in 1997 in "When we don't know the costs or the benefits: Adaptive strategies for abating climate change "[6]. But R.J. Lempert was not the first one who tried to develop a new method to handle the increasing number of decisions under deep uncertainty. The company Envolving Logic had in the 1990s, parallel with the RAND corporation and R.J. Lempert's development of RDM, developed the first software tool for DMDU. Their software was called the Computer-Assisted Reasoning system (CARsTM) and they described it as a tool to support decision-makers in the area of the Robust Adaptive Planning (RAP) methodology, while the RAND corporation and R.J. Lempert instead adopted the Robust Decision Making (RDM) methodology [7].
According to R.J Lempert, the traditional “predict-then-act” framework has proven to be successful for decades, which both a large number of theories and mathematical techniques is proof on [4]. RDM was original developed to make policymakers able to make better decisions about issues that could have consequences in the long-term. RDM was mostly focused and used for decisions about climate changes at the beginning of its lifetime. It was not a coincidence that RDM mainly were used for decisions regarding climate changes, because it was particular climate changes commonly known property of being embraced with deep uncertainty, that made RDM suitable for this area specifically [5]. With the arrival of the RDM, it was no longer necessary to determine the probabilities of the future consequences of climate changes in order to make a strategy, which many also believe is something near impossible. The need for RDM is thus targeted more towards complex decision processes under deep uncertainty, where relatively simple prediction analysis is not enough and often will be misleading [3]. R.J. Lempert thought it would be better to seek a robust solution rather than trying to predict the future and then choose the solution that fits the predicted future best. He disagreed with the traditional framework for assessing alternatives in decision processes, in which he thought predictions were imposed too great a significance. He argued that the reason for decision-makers to use predictions were to make the future less scary and that "the quest for prediction probably fills some deep human need. Even though the accuracy of most predictions has proven to be poor""[1]. A combination of new technologies and the need for a method to handle more complex issues resulted in the development of RDM, that instead of finding the alternative that seems to be the best based on specific predictions and probabilities, uses modern technology to analyse and evaluate alternatives against thousands or even millions of possible scenarios. The RDM method has given decision-makers a tool to identify trade-offs by testing strategies against different scenarios. Decision-makers can now determine which strategies that perform best under a large range of possible scenarios. If used correctly, RDM should make the decision-makers able to develop the most robust strategies that can handle the future, regardless of how the future ends up being.
RDM consist of the following key elements:
- Testing of multiple plausible futures. The combination of scenarios for the future should be significantly diverse. The reason for this is that the main idea of RDM is that you are interested in analysing how different strategies perform in as many different futures as possible. These multiple scenarios can also correspond to different persons in the decisionmaking groups view of the world[8]. Instead of spending time debating which prediction of the future too chose, as one would do in the traditional “predict-then-act” framework, the decisionmaking group can use their time more effective while testing everybody's idea of the future at the same time.
- The goal is not to find the absolute best strategies but to find the most robust ones. The main focus in RDM is to find strategies that perform well over most if not all possible scenarios[9].
- Apply adaptive strategies. One way to secure that strategies are robust is to choose some who are adaptive. Adaptive means that when the future comes, the strategies can adapt to the new circumstances and still perform well under the new conditions[2].
- The computer is used to ease the human job, not to give the final recommendation of which specific strategy should be used[8].
[edit] The foundation of RDM
The primary theoretical focus lays around uncertainty theories more specifically in the area of Decision Making Under Deep Uncertainty (DMDU), the reason for this is that the RDM methodology is used for developing strategies under deep uncertainty. As stated in The standard for portfolio management, uncertainties are in general just the result of a lack or limited knowledge about a certain state in time and are almost impossible to avoid when making decisions in our modern world [2]. Many books have been written about decisionmaking under uncertainty, but only a very limited amount includes DMDU. Even though the literature about DMDU is limited, however increasing, several tools and different analytical approaches exist. They share the underlying paradigm that they all seeks to reduce strategies vulnerability against the uncertainties associated with the future, which in some literature is defined as Assumption Based Planning (ABP)[10].
RDM is developed based on some already existing methods, paradigms and approaches, the most important ones will be reviewed in the following section. The foundation of RDM is visualised in the figure below. It shows that RDM is based on three main concepts, Decision Analysis, Assumption-Based Planning and Scenarios which then are united in the forth concept, Exploratory modelling.
[edit] Decision Analysis
A part of RDM's foundation is Decision Analysis (DA). The reasoning behind DA is that the human, by including decision tools, make better decisions. It applies whether we are a group or an individual. DA provides the decision-makers with tools to enlighten what affect certain decisions will have on different situations under uncertainty.[11]. DA can take the form as e.g. Decision Trees or Expected Value (EV) calculations, but for more complex decisions under deep uncertainty, it is necessary to include more advanced methods and computer modelling. It could e.g. be RDM which is categorised as a quantitative DA method[5]. DA can be divided into two different approaches, the first approach is called "agree-on-assumption" or "predict-then-act" and the second one, which RDM follows, is an "agree-on-decisions" approach. The idea of stress testing is one concept RDM has derived directly from the "agree-on-decision" approach. The first approach seeks to rank different strategies based on agreed probabilities of how likely it is that the world is in a certain state in the future, known from the traditional DA framework. The second approach test different strategies against multiple scenarios of the future, which is used to find vulnerabilities of the strategies and mitigating actions to handle these vulnerabilities [12] [13]. According to C. Helgeson (2018), the "agree-on-decision" approach as RDM follows, can be summarized as an approach to support the decisionmaking group in the process of framing decisions under deep uncertainty [14].
[edit] Assumption-Based Planning
RDM uses the properties of stress testing to challenge a variety of scenarios and strategies, to reduce the possibility of failure by improving the decision-makers understanding of when, why and what can go wrong. More specifically, RDM takes advantage of a certain practice of this, called Assumption-Based Planning (ABP) [15]. ABP takes a starting point in a written plan and then seeks to identify the underlying assumptions on which the existing plan is based on. Only crucial assumptions, as if not goes as expected can lead to failure, is relevant in ABP. Further, ABP structures the crucial assumptions and uses them in the framework seen in RDM to develop adaptive strategies. Adaptive strategies or adaptive planning consist of several key elements, which all share that they in order to uncover uncertainties, uses computational power [16]. The ABP approach secures that decision-makers using RDM tests a large number of uncertainties and develops both actions to be taken in the short-term and long-term, but with the possibility for long-term actions to change according to how situations develop. This includes developing signposts/signals that if observed in the future, will result in different actions to secure that the adaptive strategies won't fail [17].
[edit] Scenarios
Scenario analysis is another concept RDM uses. It is a way to represent different descriptions of plausible future states. Scenario analysis contributes to dividing different future states into multiple scenarios which makes it easier for the decision-makers to analyse and discuss. In contrast to ranking future states based on probabilities, scenario analysis analyses different views of how a situation will develop. Focusing on possibilities instead of probabilities gives the decision-makers the possibility to analyse more diverse events, compared to if they had to focus on only the events that most likely would happen. At the same time, the decision-makers are faced with more extreme and unexpected future events which help to test the relevant strategies against unlikely and more extreme scenarios than one might normally do with traditional frameworks [18] [19]. There exist different types of algorithms to do scenario analysis and the ones RDM uses is a quantitative approach of Scenario Discovery (SD) algorithms. In RDM, the decisionmakers use the final scenarios from the scenario analysis to capture the most important findings from the ABP stress test, which at the end of the RDM forms the basis for the decision-makers to make adaptive strategies[20].
[edit] Exploratory Modeling
RDM would not exist without Exploratory Modeling (EM). EM is particularly beneficial when it is not possible to use a model, because of e.g. missing data or if the future is associated with great uncertainty. It unites DA, ABP and SD and combines these concepts, by running the same model with sometimes only small changes in the input, even though many would normally argue that it is not good practice to run a model many times. In RDM this repeating is one of the key factors to achieve the targeted knowledge and to develop a large database of results. Another benefit of EM is that it does not have any starting point, and thus there are no restrictions for the possibilities of exploring different plausible futures and strategies [5]. EM can be used to analyse a variety of topics but for RDM, the goal of using EM is focused on finding strategies with a high degree of robustness[21]. Besides being preferred because of the ability to support decisions under deep uncertainty, EM's computational simplicity is a big advantage, especially compared to alternative approaches that include Dynamic Programming or different optimization methods[5].
[edit] Application of RDM
RDM has been used to support a wide range of decisions, but RDM is not suitable in any decision process. RDM is most suitable if all or some of the following characteristics apply to the problem that needs to be handled. The primary condition is that there is a high degree of uncertainty (deep uncertainty) associated with the problem, other conditions that to some extent is related to deep uncertainty is if the problem is very complex, or if the decision-makers have a wide range of possible options to choose from [9]. It also includes if it is a problem that needs computational power in order to be solved, e.g. if it is necessary to test a large number of strategies against multiple scenarios. In general, RDM applies to most problems regarding DMDU as long as the decision-makers are not bound to a small number of mitigating possibilities [21]. For less complex issues, it will normally be preferred to use a "predict-then-act" methodology, even though RDM in most cases also would be able to support the decision process in these cases. However, it would be an over-complication of the decision process, which is why RDM is not used in all decision processes.
[edit] Steps in RDM
Depending on the literature, the framework for an RDM may vary slightly, but overall the content is similar. The figure shown here is the latest version of R.J. Lempert's frameworks (founder of RDM) and is based on previous versions of the framework [4]. A more in-depth go-through of the steps can be found in R.J. Lempert's book Robust Decision Making in Decision Making under Deep Uncertainty[5].
It is important to keep in mind that RDM is an iterative process, which the figure also shows, and that the new knowledge acquired in each step of the RDM can lead to a necessity for re-framing the problem and its challenges.
[edit] Step 1 Decision Framing
As shown in the figure, the stakeholders start with a decision framing exercise, in which they have to define the key factors relevant for further analysis:
- Objectives and criteria
- Strategies/Alternatives that successfully can handle the objectives
- Uncertainties and their consequences
- Relationship between strategies, objectives and uncertainties
[edit] Step 2 Evaluate Strategy Across Futures
The next step is to use an "agree-on-decision" approach"[5]. To do so, RDM uses simulations of the strategies found in Step 1, against a variety of plausible scenarios of the future. These simulations generate a large number of results, which is merged into a large database and the results are evaluated across the different futures [8].
[edit] Step 3 Vulnerability Analysis
Step 3 tries to identify, explore and characterize vulnerabilities [8]. The decision-making team can use visualisations and data analytics based on the database made in step 2. In order to identify the key factors that contribute to the success or failure of the strategies, it is common practice to use Scenario Discovery (SD) algorithms. The use of SD algorithms results in clusters of futures that will enlighten which factors the strategies are most vulnerable to[5].
[edit] Step 4 Tradeoff Analysis
A tradeoff analysis is performed on the scenarios from step 3. In step 4 different strategies are evaluated in order to analyse the tradeoffs among different strategies[1]. This is often done by making multi-objective tradeoff plots in which tradeoff curves can be used to e.g. compare cost against reliability for different scenarios. The tradeoff curves can be used by the decision-makers to determine what balance they evaluate to be most suitable among the objectives defined in step 1[5].
[edit] Step 5 New Futures and Strategies
Finally, the decision-making team can then use the findings from step 4 to evaluate if they have found any robust strategies, with better tradeoffs, compared to existing strategies [8]. The final strategies from the RDM will often be what you call Adaptive Decision Strategies, which are strategies that adapt depending on how the future develops. The adaptive strategies consist of a short-term strategy with actions to be taken in the near future and so-called signposts which are predesignated signals that when observed, certain actions will be taken. These adaptive strategies can either be established from a jury of experts [8], or the most suitable combination of near-term actions, signposts and actions to be taken if the signals are observed, can be determined with the help of optimization algorithms[9].
[edit] Limitations of the RDM Framework
[edit] Limitations
RDM is not an easy method to follow and it certainly has it's requirements in order to be performed correctly, but as the philosophy behind RDM is " that it is better to be roughly right than precisely wrong", the models used to perform the RDM is often relatively simple and fast models. The problem is that even if a model solely is simple, problems can occur when it is necessary to run the model thousands or even millions of times, with a large number of different strategies and scenarios, which RDM involves. It sets requirements for the computer power and amount of data which in some cases, e.g. in developing countries, maybe doesn't exist causing it to be almost impossible to perform RDM [22]. Besides data and computing requirements, then expert knowledge on the field of concern could also be needed. If the decision-making group does not possess this knowledge, they have to hire external experts which can be costly. It is also worth bearing in mind that even with the involvement of expert knowledge, the execution of RDM is based on subjective assessments, which is influenced by the perceptions of the decision-makers[23]. It means that RDM is only as good as the data used for it. However, performing thousands or millions of simulations significantly minimizes the risk but will never be able to eliminate it.
A limitation could also be the necessity of external involvement of experts to perform the modelling, there even though the models are simple, acquires some computer modelling skills to perform. Another limitation, as the annotated literature on RDM shows, is that the RDM methodology is mostly used to support DMDU for policy challenges. Besides the fact that it has mainly been used in connection with long-term issues. Among other things, this is due to the fact that the uncertainty associated with a given decision is largely related to time. Short-term strategies will only very rarely be associated with a high degree of uncertainty (deep uncertainty), which is why RDM is primarily limited to long-term decisions [5][8]. RDM is thus not a tool you decide to use without first familiarizing yourself with the requirements the execution of an RDM place on the decision-makers, their knowledge and competencies.
[edit] Annotated bibliography
Bloemen P., Lempert R.J, Marchau V., Walker W., Popper S. (2019), Robust Decision Making in Decision Making under Deep Uncertainty [5].
This book is some of the latest literature about DMDU. It contains various approaches, applications, and implementations to handle DMDU, including RDM. It provides an in-depth guide on how RDM is performed and a concrete example of the implementation of RDM, both are very useful to look into if one wants to perform RDM. Besides being one of the newest books it is also one of the most detailed so for those who want to make use of RDM this book is a must-read.
Collins M., Lempert R.J. (2007), Managing the risk of uncertain threshold responses: Comparison of robust, optimum, and precautionary approaches. [9]
This article studies different approaches for decision-makers to find robust strategies. It discusses the tradeoffs between different approaches which can be interesting for those who consider using RDM. Two of the three approaches is directly related to R.J. Lempert's work while the last one is another very used approach described by J. Rosenhead. The article can contribute to a deeper understanding of the tradeoff between performance vs. sensitivity which is key in the RDM framework.
Bankes S.C., Lempert R.J., Popper S.W. (2003), Shaping the next one hundred years: New methods for quantitative, long-term policy analysis [8].
Shaping the next one hundred years was groundbreaking in relation to the hypothesis it was built around back in 2003. The authors stated that with the help of computers, it was possible to change the way long-term futures is predicted. They suggested scenario generator- and exploratory modelling software. Among other things, they mention in the book that the long-term future is defined by deep uncertainty. They also talk about robustness and adaptive strategies, all something that can be related to RDM.
Lempert R.J., Groves D.G. (2010), Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west [4]
This article is a case study that employs RDM in a real-world case regarding adaptive policy responses to climate change for water management agencies in the American west. Both authors work for the RAND cooperation, who are experts and partly founders of the RDM framework. The article contributes with an in-depth analysis of how RDM is used on a real-world case, while is also discusses the benefits of using RDM instead of a traditional "predict-then-act" framework, in the process of creating adaptive decisions strategies.
[edit] Bibliography
- ↑ 1.0 1.1 1.2 Lempert R.J., Schlesinger M.E. (2000), Robust strategies for abating climate change. Climatic Change, Springer Netherlands, Volume 45, Page 387–401, DOI: 10.1023/a:1005698407365
- ↑ 2.0 2.1 2.2 The standard for portfolio management — fourth edition. (2017), ProQuest Ebook Central, Page 91-94, https://ebookcentral-proquest-com.proxy.findit.dtu.dk/lib/DTUDK/detail.action?
- ↑ 3.0 3.1 Lempert R.J., Fichbach J.R. (2021), Robust Decision Making: A Tool to Help Address Climate Change, https://www.rand.org/pardee/projects/rdm-and-climate-change.html
- ↑ 4.0 4.1 4.2 4.3 4.4 Lempert R.J., Groves D.G. (2010), Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west, Technological Forecasting and Social Change, Volume 77, Issue 6, Pages 960-974, https://doi.org/10.1016/j.techfore.2010.04.007.
- ↑ 5.00 5.01 5.02 5.03 5.04 5.05 5.06 5.07 5.08 5.09 5.10 5.11 Bloemen P., Lempert R.J, Marchau V., Walker W., Popper S. (2019), Robust Decision Making in Decision Making under Deep Uncertainty. Springer, Cham. https://doi.org/10.1007/978-3-030-05252-2
- ↑ Lempert R.J., Schlesinger M.E., & Bankes S.C. (1996), When we don’t know the costs or the benefits: Adaptive strategies for abating climate change. Climatic Change, Volume 37, Issue 1, 207–208.
- ↑ Bankes S.C., Lempert R.J., Popper S.W. (2001), Computer-assisted reasoning. Computing in Science & Engineering, Volume 3, Issue 2, Page 71–77
- ↑ 8.0 8.1 8.2 8.3 8.4 8.5 8.6 8.7 Bankes S.C., Lempert R.J., Popper S.W., (2003), Shaping the next one hundred years: New methods for quantitative, long-term policy analysis, Santa Monica, CA, The RAND Corporation, Technological Forecasting and Social Change — 2004, Volume 71, Issue 3, Page 305-307
- ↑ 9.0 9.1 9.2 9.3 Collins M., Lempert R.J., (2007), Managing the risk of uncertain threshold responses: Comparison of robust, optimum, and precautionary approaches. Risk Analysis, Volume 27, Issue 4, Page 1009–1026
- ↑ Dewar J.A., Builder C.H. W., Hix M., Levin M.H. (1993), Assumption-Based Planning: A planning tool for very uncertain times. MR-114-A, RAND, Santa Monica, California.
- ↑ National Research Council (2001), Theoretical Foundations for Decision Making in Engineering Design, Washington, DC: The National Academies Press. https://doi.org/10.17226/10566
- ↑ Kalra N., Hallegatte S., Lempert R.J., Brown C., Fozzard A., Gill S., et al. (2014), Agreeing on robust decisions: A new process of decision making under deep uncertainty, Policy Research Working Paper. World Bank
- ↑ Lempert R.J., Nakicenovic N., Sarewitz D., Schlesinger M. (2004), Characterizing climate change uncertainties for decision-makers — An editorial essay. Climatic Change, Volume 65, Issue 1–2, Page 1–9
- ↑ Helgeson C. (2018), Structuring decisions under deep uncertainty. Topoi, Page 1–13
- ↑ Dewar J.A. (2002), Assumption-based planning—A tool for reducing avoidable surprises, Cambridge: Cambridge University Press.
- ↑ Walker W.E., Haasnott M., Kwakkel J.H. (2013), Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty, Sustainability, Volume 5, Page 955-979, doi:10.3390/su5030955
- ↑ Walker, W.E., Rahman, S.A., Cave J. (2001), Adaptive policies, policy analysis, and policymaking. European Journal of Operational Research, Volume 128, Page 282–289
- ↑ Schoemaker P.J.H. (1993), Multiple scenario development: Its conceptual and behavioural foundation. Strategic Management Journal, Volume 14, Issue 3, Page 193–213
- ↑ Gong, M., Lempert, R. J., Parker, A. M., Mayer, L. A., Fischbach, J., Sisco, M., et al. (2017), Testing the scenario hypothesis: An experimental comparison of scenarios and forecasts for decision support in a complex decision environment. Environmental Modeling and Software, Volume 91, Page 135–155
- ↑ Groves D. G., Lempert R.J. (2007), A new analytic method for finding policy-relevant scenarios, Global Environmental Change, Volume 17, Page 73–85
- ↑ 21.0 21.1 Weaver C.P., Lempert R.J., Brown C., Hall J.A., Revell D., Sarewitz, D. (2013), Improving the contribution of climate model information to decision making: The value and demands of robust decision frameworks. WIREs Climate Change, Volume 4, Page 39–60
- ↑ Bhave A.G., Conway D., Dessai S., Stainforth D.A. (2016), Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world, Climate Risk Management, Volume 14, Pages 1-10, ISSN 2212-0963,
- ↑ Econadapt - Toolbox, Robust Decision Making, Retrieved February 16, (2021). https://econadapt-toolbox.eu/robust-decision-making
