Latest revision as of 11:55, 2 May 2023

[edit] Abstract

Robust Decision Making (RDM) is a computational framework integrating Decision Analysis, Assumption-Based Planning, Scenario Analysis, and Exploratory Modelling. This article critically reviews RDM, its principles, and applications in project management. The article suggests that RDM enables project managers to effectively address uncertainty, offering a powerful analytical framework.

[edit] Conceptualising Robust Decision Making at times of Uncertainty

[edit] Origins and Functions

Robust Decision Making (RDM) emerged in the 1980s, when analysts of the RAND Corporation, a California-based think tank affiliated with the U.S. Government, developed a framework to evaluate the effectiveness of nuclear weapon systems ^{[1] [2]}. Designed to mitigate the uncertainty and ambiguity experienced by U.S. Government officials involved in the planning and implementation of nuclear deterrence strategies, RDM included simulation techniques, sensitivity analysis, and real options analysis. In the 1990s and 2000s, RDM received increasing interest from private companies interested in exploring new project management techniques applicable to a wide range of industries, including construction, software development, and environmental management. Today, RDM is an established approach in project management, recognized for its ability to help project managers making well-informed and timely decisions under pressure and at times of uncertainty. According to former United States Secretary of Defence Donald Rumsfeld, there are different types of knowledge: known knowns, known unknowns, and unknown unknowns. Known knowns refer to things that we know for sure. Known unknowns refer to things that we know we do not know. However, the most challenging category is the unknown unknowns, which refers to things that we do not know we do not know ^{[3] [4]}. The decision-making process in situations affected by a great level of uncertainty is defined as decision making under deep uncertainty (DMDU) ^[2].

Robustness is a crucial aspect of effective DMDU ^{[5] [6]}. Conventional decision-analytic techniques for risk and decision analysis are designed to identify optimal strategies based on a characterization of uncertainty that follows the axioms of probability theory ^[7]. However, in scenarios where there is uncertainty about the system model or the distributions of its inputs, traditional decision-analytic approaches often utilize sensitivity analyses to assess the dependence of the optimum strategy on the specification of model and distributions ^[8]. While this approach may be suitable when the optimum strategy is relatively insensitive to these key assumptions, it can pose both conceptual and practical challenges when this is not the case. RDM is part of a new breed of computational, multi-scenario simulation approaches that integrates ideas from scenario-based planning into a quantitative framework ^{[9] [10] [6] [11]}. It inverts traditional sensitivity analysis by seeking optimization strategies which good performance is insensitive to the most significant uncertainties. Beginning with one or more system models that link optimization strategies to outcomes and a collection of several plausible probability distributions over the uncertain input parameters to these models, RDM describes uncertainty with various, plausible perspectives of the future ^[12]. RDM suggests robust strategies, identifies vulnerabilities, and proposes new or modified strategies.

[edit] Foundations of Robust Decision Making

RDM combines four crucial concepts - Decision Analysis, Assumption-Based Planning, scenarios, and Exploratory Modelling - to provide evidence-based analysis that informs choices. This approach empowers project managers to navigate uncertainty, assess vulnerabilities, and identify robust strategies that can withstand potential futures.

Decision Analysis (DA)

Decision Analysis is a discipline that involves the theory, methodology, and practice of designing and using decision aids to help people make better decisions by being explicit about their goals, using the best available evidence to understand potential consequences, considering trade-offs among alternatives, and following agreed-upon rules and norms to enhance the legitimacy of the decision-making process ^[2]. RDM exploits this approach by focusing specifically on finding trade-offs and describing vulnerabilities to create robust decisions based on stress testing of probable future routes. Both DA and RDM seek to improve the decision-making process by being clear about goals, utilizing the finest information available, carefully weighing trade-offs, and adhering to established standards and conventions to assure legitimacy for all parties involved. However, while DA seeks optimality of decisions ^[13], RDM seeks robustness assuming uncertainty as unescapable and probabilities as imprecise ^[14].

Assumption-Based Planning (ABP)

By expanding awareness of how and why things could fail, RDM encapsulates the ideas of stress testing and red teaming to lessen the harmful impacts of overconfidence in current plans and processes. The former idea puts a system through rigorous testing to establish its breaking points, whereas the latter forms an independent group of people (often external parties) to find ways to thwart an organization's plans ^[15]. This approach was first implemented in the so-called Assumption-Based Planning (ABP) framework. Starting with a written version of an organization's plans, ABP finds the explicit and implicit assumptions made during the formulation of that plan that, if inaccurate, would result in failure. These assumptions are identified by project managers, who then create backup plans and "hedging" strategies to be used in case of necessity ^[16].

Scenario Analysis (SA)

To deal with deep uncertainty, RDM builds upon the idea of SA ^[17]. Scenarios are defined as collections of potential future occurrences that illustrate various worldviews without explicitly assigning a relative likelihood score ^[18]. They are frequently envisioned in deliberative processes involving stakeholders and probabilities are often not considered. This is because, without considering probabilities, stakeholders are more prone to broaden the range of scenarios taken into consideration and consider their decisions from a larger variety of angles and perspectives, leading to including unexpected outcomes as well. In incorporating analytical techniques borrowed from SA, RDM declines the knowledge about the future into a selected range of potential situations—a technique that assists project managers in envisioning future risks and better navigating situation characterised by deep uncertainty.

Exploratory Modeling (EM)

In his study on predictive models and computer simulations, Bankes argued that Exploratory Modelling (EM) is one of the most appropriate tools that allows the integration of DA, ABP, and SA in RDM ^[19]. Without prioritizing one set of assumptions over another, EM factors a wide range of assumptions into a limited number of results. In other words, EM is strongly beneficial when a single model cannot be validated because of a lack of evidence, insufficient or conflicting ideas, or unknown futures. Therefore, by lowering the demands for analytic tractability on the models employed in the study, EM offers a quantitative framework for stress testing and scenario analysis and allows the exploration of futures and strategies. As EM favours no baseline scenario as an anchor point, it allows for genuinely global, large-N studies in support of the more qualitative methods ingrained in SA approaches.

[edit] Application

Theoretical framework

RDM is a learning process based on a process that the scholarship defines as "deliberation with analysis" ^[20]. The framework requires that the decision-making parties discuss their goals and alternatives, which are based on assessments provided by analysts who imagine scenarios and policy options. This is especially recommended in situations in which a variety of decision-makers operate in rapidly changing circumstances, and whose objectives may change because of their collaboration with others ^[20]. As illustrated in Figure 1, RDM methodology follows 5 major steps, described in the paragraphs below.

Figure 1: XLRM framework

Step1: Decision Framing.

The RDM process starts with a decision framing workshop in which stakeholders brainstorm and define the key factors in the analysis. These include project managers’ goals and criteria, the potential courses of action they may choose to accomplish those goals, the uncertainties that might impact the results, and the connections between actions, uncertainties, and goals. Once gathered, this information is put into a framework known as “XLRM” ^{[17] [21]}, where:

• “X” stands for exogenous variables (factors not under the control of the decision makers)
• “L” stands for policy levers (policies that affect the system to achieve goals)
• “R” stands for relationships (relevant variables needed to correctly evaluate and benchmark policies)
• “M” stands for measures of performance (metrics, not necessarily quantitative, given from stakeholders to evaluate policies)

The output is a set of potential robust strategies.

Step 2: Evaluate strategies.

According to the ABP approach, RDM exploits simulation models to assess the proposed strategies of Step 1 in each of many plausible paths into the future. This process of generating strategies may use a variety of techniques, spanning from optimization methods to public debate ^[22]. It is commonly observed, however, that strategy evaluation usually combines them all ^[23].

Step 3: Vulnerability analysis.

Data analytics and visualization techniques are then used to search for and describe vulnerabilities of the strategies under consideration. Specifically, statistical methods are used to find the critical variables that best distinguish futures in which these strategies succeed or fail. ^[24]. The output of this step is a multitude of scenarios which are then clustered based on the identified vulnerabilities.

Step 4: Trade-off analysis.

In this step, both analysts and decision makers use the scenarios to present and assess the trade-offs between strategies. To indicate the implications of selecting one strategy over another, it is often helpful illustrating these trade-offs. To do so, the effectiveness of one or more strategies can be plotted against the probability of the scenario the strategies belong to. This step is useful to give insights on how the future would look like if a strategy was chosen.

Step 5: New futures and strategies.

The results from Step 4 are necessary to identifying and appraising alternative solutions, allowing to focus on the most robust ones. Sometimes, the identification and appraisal process relies on experts’ opinions ^{[23] [17]}. In other cases, optimization approaches are preferred ^[25]. It is either when no more robust strategies can be generated, or when the already identified ones are considered sufficiently satisfactory that the procedure can be deemed as completed.

To appraise and quantify trade-offs between strategies, RDM exploits both absolute and relative performance indicators. Specifically, the formers are beneficial when specific objectives would like to be met (e.g., maximisation of profit). The latter are beneficial when decision-makers seek the evaluation of different strategies in different possible futures and search for the most robust ones.

[edit] Case study: Application to Water Planning Management

Introduction and background.

The Colorado River is the main source of water in southwestern United States, supplying 4.5 million acres of agriculture with irrigation as well as power and water to about 40 million people ^[26]. Four Upper Basin States (Colorado, New Mexico, Utah, and Wyoming) and three Lower Basin States (Arizona, California, and Nevada) each gets 15 million acre-feet of water under the terms of the 1922 Colorado River Compact. The system's dependability is being put under more and more pressure because of the deep supply uncertainty and rising demand. The Colorado River Basin Study was launched in 2010 by the seven Basin States and the US Bureau of Reclamation to:

• ensure a 10-years running water flow from the Upper to the Lower Basin with a minimum of 7.5 maf/year^[a] (Upper Basin reliability),
• maintain Lake Mead^[b] (situated at the border of Nevada and Arizona) at a minimum of 1000 feet of pool elevation (Lower Basin reliability).

The Bureau of Reclamation, which manages, develops, and protects water and related resources, evaluated DMDU methodologies in a pilot study, adopted them to assist ongoing planning, and utilized RDM to frame the vulnerability and adaptation evaluations for the Basin Study. The research results were used to specify a solid, flexible management plan.

Decision framing and current vulnerability analysis.

As illustrated in Figure 2, XLRM was used as the tool to evaluate a total of 23,508 different futures. To retrieve this figure, the Basin Study analysed a plethora of future hydrologic conditions in conjunction with six demand scenarios and two operating scenarios. Colorado River Simulation System (CRSS), a long-term planning tool for Reclamation, was used to assess the system's performance across a wide range of potential futures. These studies concentrated on two major goals: keeping Lake Mead's pool elevation above 1,000 feet and ensuring that the water flow from the Upper to Lower Basin reaches or surpasses 7.5 maf per year as measured over a period of ten years. In addition, CRSS employed alternative water management techniques such as desalination, wastewater reuse, municipal, industrial, and agricultural conservation.

Vulnerability analysis was done in the decision framing step. The current water management system was simulated across thousands of different scenarios. SA approaches were, then, used to identify significant vulnerabilities and CRSS to model Basin outcomes. If the long-term average streamflow falls below 15 maf and an eight-year drought occurs with average flows below 13 maf, the Lower Basin is at risk. The Basin Research also discovered a vulnerability for the Upper Basin characterized by streamflow traits that are only projected to occur in the future with declining supply.

Figure 2: XLRM framework for the Colorado River Basin study. Source: David G. Groves, Edmundo Molina-Perez, Evan Bloom and Jordan R. Fischbach, Robust Decision Making (RDM): Application to Water Planning and Climate.

Based on the above framework, the project team created portfolios of individual management choices that might either boost the supply or decrease demand for the Basin states. Stakeholders developed four portfolios, each with a unique set of investment possibilities. Future vulnerability management was represented by Portfolio B (Reliability Focus) and Portfolio C (Environmental Performance Focus). Portfolio D (Common Options) was established to contain just those choices in both Portfolios B and C, while Portfolio A (Inclusive) was defined to include all alternatives in either Portfolios B or C. Options were ranked in order of cost-effectiveness, which was calculated by dividing the average yearly yield by the total project cost. By modelling the investment choices that a basin manager would take under various simulated Basin conditions, CRSS modelled these portfolios as potential strategies.

Evaluate strategies.

Robustness was defined as the strategy that minimizes regret^[c] under a plethora of conceivable future scenarios. In this case, regret is defined as the extra total yearly supply (volume of water, in maf) required to keep Lake Mead at 1000 feet during the experiment. The more the supply, the more the regret. It is worth noting that regret is not eliminated. In fact, throughout the simulation, it was observed that some strategies pursued big investments in scenarios where annual precipitations are higher and little to no investments in the ones in which these are lower.

Trade-off analysis.

Each technique performs differently in terms of cost and dependability measure. In futures where the supply is lower it was observed that Portfolio A was the strategy with the highest likelihood to prevent water delivery vulnerability, at the expense of the highest cost. Portfolio D (a subset of A) had lower costs, but also lower probability to prevent vulnerabilities. It was concluded that Portfolios A and B were associated to lowest number of years with critical water levels, but the highest costs. Portfolio C had slightly more years with critical water levels, but lower costs.

New futures and strategies.

Probability thresholds that, with sufficient certainty, would indicate that a given vulnerability would be likely to occur, and that a consequence management action should be taken, were determined based on statistical analysis and the primary vulnerabilities found. This data was used to create an effective plan for the whole Basin that directs the investment of greater water supply yield and demand reductions. The sample paths (dashed lines) seen in Figure 3 illustrate one of many potential implementation routes. If the basin were to follow a trajectory that was consistent with the vulnerability “Below Historical Streamflow During Extreme Drought”, the example paths illustrate how basin managers would provide additional supply. The figure shows decisions taken from Basin managers up to 2030. It then highlights a decision point in 2030 where, by evaluating scenarios, it might be possible that the future will not be consistent with the same vulnerability. If conditions are consistent with the “Severe Declining Supply Scenario”, then the Basing managers should increase the net supply to more than 3.6 maf between 2031-2040. The same decision approach is taken for the following years.

Figure 3: Robust adaptive strategy for Colorado River.

[edit] Assessment and Conclusion

When used with project management frameworks, RDM can offer several advantages. First, RDM may help project managers make informed and data-driven decisions. It can assist project managers in identifying more reliable and adaptable plans by considering a variety of uncertainties and their possible impacts. Second, RDM can help project managers identify and assess various risks associated with a project, including both known and unknown risks. This can enable managers to develop contingency plans and other risk mitigation strategies to address potential issues. Finally, RDM can facilitate stakeholder engagement and participation in the decision-making process. By considering the perspectives and preferences of various stakeholders, RDM can help managers develop solutions that are more acceptable and feasible. However, there are also some limitations to using RDM within a project management framework. First, RDM can be resource-intensive, requiring significant data collection, analysis, and modelling. This can be particularly challenging for smaller projects or those with limited resources. Second, complexity and uncertainty can make it challenging to apply RDM effectively, particularly in cases where there are significant data gaps or limited information available. Third, models and simulations are only as good as the data and assumptions that underlie them. This can lead to errors or biases in the decision-making process.

This article explored the role of Robust Decision Making in project management, emphasizing its relevance for project managers operating at times of deep uncertainty. Although the relative paucity of scholarly works on the topic leaves important theoretical gaps to be addressed, the empirical evidence available confirms that RDM offers valuable practical applications in a wide range of industries. It enables data-driven decisions, facilitates risk identification and assessment, and improves stakeholder engagement.

As mentioned above, RDM can be resource-intensive. However, the benefits of implementing RDM, such as the identification of robust strategies, or mitigation of both known and unknown risks, far outweighs the costs. Project managers should possess knowledge of RDM and be proficient in its implementation to effectively navigate decision-making processes under condition of deep uncertainty.

[edit] Notes

[a] - Maf = million acre-feet. It is a non-SI used in reference to large-scale water sources.

[b] - In the Southwest of the United States, the Hoover Dam on the Colorado River creates the reservoir known as Lake Mead. It lies 39 kilometers east of Las Vegas in the states of Nevada and Arizona. In terms of water storage capacity, it is the biggest reservoir in the US. Nearly 20 million people and vast tracts of farmland are fed by Lake Mead's water supply, which also reaches parts of Mexico, Arizona, California, and Nevada.

[c] - The human emotional response of regret, which is frequently felt while making decisions under uncertainty, may be assessed as the amount of variance between a made option and the best choice, according to decision theory.

[edit] Annotated bibliography

The following is the annotated bibliography for this article:

• "Robustness and optimality as criteria for strategic decisions" by Rosenhead, Elton, and Gupta (1972) explores the use of robustness and optimality as decision criteria in the realm of decision-making under conditions of uncertainty. Additionally, the paper dives deeper into the importance of considering robustness, which ensures resilience to uncertainties, alongside traditional optimality in decision-making processes.
• "Climate change 2001: Mitigation" edited by Metz, Davidson, Swart, and Pan (2001) is a comprehensive contribution to the Third Assessment Report of the Intergovernmental Panel on Climate Change (IPCC). The book specifically focuses on strategies and actions for mitigating climate change, discussing policy options, technological advancements, and international cooperation to reduce greenhouse gas emissions.
• "Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis" by Morgan and Henrion (1990) is seen as a complete guide for effectively managing uncertainty in quantitative risk and policy analysis. The authors present methods and approaches to handle uncertainties, including probabilistic techniques and sensitivity analysis, specifically highlighting the importance of incorporating uncertainty considerations in decision-making processes.
• "Sensitivity Analysis" by Saltelli, Chan, and Scott (2000) provides a comprehensive overview of sensitivity analysis, a powerful tool for assessing the influence of uncertain input factors on model outputs. The book covers various techniques, including variance-based methods and local and global sensitivity measures, highlighting their applications in diverse fields such as engineering, environmental modelling, and policy analysis.
• "Why conventional tools for policy analysis are often inadequate for problems of global change" by Morgan, Kandlikar, Risebey, and Dowlatabadi (1999) critically examines the limitations of conventional policy analysis tools in addressing complex issues of global change. The paper stresses the need for more integrated, multidisciplinary approaches that account for uncertainties, non-linear dynamics, and long-term consequences when formulating effective policies in the face of global challenges.
• "Perspectives on Uncertainty and Risk" by van Asselt (2000) provides an insightful exploration of diverse perspectives on uncertainty and risk in decision-making processes. The book examines different conceptual frameworks and methods for understanding, characterizing, and managing uncertainty and risk, offering valuable insights for policymakers, analysts, and researchers grappling with complex decision problems affected by uncertainties.
• "Special report on emissions scenarios" by Nakicenovic et al. (2000) is a significant report produced by the Working Group III of the Intergovernmental Panel on Climate Change (IPCC). It presents a range of emissions scenarios, and this provides valuable information and projections to support climate change assessments and policy development. The report explores various socioeconomic and technological factors influencing greenhouse gas emissions, helping policymakers in understanding potential future trajectories and their implications.
• "A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios" by Lempert et al. (2006) presents a general analytic method for generating robust strategies and narrative scenarios. The paper introduces a structured approach to help decision-makers develop strategies that are resilient to uncertainties, incorporating robustness considerations into the decision-making process.
• The RAND Corporation's website provides information on robust decision making (RDM), a methodology that helps decision-makers analyse complex and uncertain problems. It offers tools and resources to support the application of RDM in various domains, including policy analysis, resource allocation, and strategic planning.
• "Robust Decision Making (RDM)" by Lempert (2019) discusses the concept of robust decision making and its application in addressing deep uncertainties in decision-making processes. The paper explores the key principles, methods, and case studies of RDM, highlighting its effectiveness in generating robust strategies under uncertain conditions.
• "Decision Making under Deep Uncertainty. From Theory to Practice" by Marchau et al. (2019) provides a comprehensive overview of decision-making approaches to deal with deep uncertainties. The book covers theoretical foundations, practical tools, and case studies, offering insights into managing decision problems characterized by high ambiguity and limited information.
• The February 12, 2002, Department of Defence News Briefing by Donald Rumsfeld discusses various aspects of national security and defence, including the challenges and uncertainties faced in the context of global events and conflicts.
• The Defense.gov News Transcript captures the February 12, 2002, Department of Defence News Briefing by Secretary Rumsfeld and General Myers, providing a detailed record of the discussions on defence-related matters.
• "Statistical Reasoning with Imprecise Probabilities" by Walley (1991) explores the use of imprecise probabilities in statistical reasoning. The book presents a comprehensive framework for handling uncertainty in statistical analysis, providing a broader perspective on decision-making and inference under uncertain conditions.
• "Assumption-Based Planning-A Planning Tool for Very Uncertain Times" by Dewar et al. (1993) introduces assumption-based planning as a decision-support tool suitable for highly uncertain environments. The publication describes the process of identifying critical assumptions and developing robust strategies that are resilient to a wide range of uncertainties.
• "Shaping the Next One Hundred Years: New Methods for Quantitative, Long-term Policy Analysis" by Lempert, Popper, and Bankes (2003) presents innovative methods for quantitative, long-term policy analysis. The book explores various modelling techniques and scenario-based approaches to support decision-making in complex, dynamic systems affected by long-term uncertainties.
• "The Gentle Art of Reperceiving-Scenarios: Uncharted Waters Ahead" (Part 1) by Wack (1985) discusses the concept of scenarios as a tool for reperceiving and exploring alternative futures. The article emphasizes the role of scenarios in expanding thinking and enabling organizations to navigate uncertain and ambiguous environments.
• "Exploratory Modeling for Policy Analysis" by Bankes (1993) introduces the concept of exploratory modelling as a technique for policy analysis under uncertain conditions. The paper stresses the importance of modelling and simulation to gain insights into complex systems and improve understanding of uncertainties to support robust decision-making.
• "Informing Decisions in a Changing Climate" by the National Research Council (2009) provides guidance on decision-making processes in the context of climate change. The report emphasizes the need to integrate scientific knowledge, uncertainty assessments, and stakeholder engagement to inform adaptive decision-making in a changing climate.
• "The Exploratory Modeling Workbench: An Open-Source Toolkit for Exploratory Modeling, Scenario Discovery, and (Multi-Objective) Robust Decision Making" by Kwakkel (2017) introduces the Exploratory Modeling Workbench (EMW), an open-source toolkit for exploratory modelling and decision analysis. This paper presents the features and capabilities of EMW, which supports scenario discovery, sensitivity analysis, and robust decision-making approaches, enabling stakeholders to explore and evaluate complex systems under deep uncertainty.
• "Robust Climate Policies under Uncertainty: A Comparison of Info-Gap and RDM Methods" by Hall et al. (2012) compares two different methods, Info-Gap and Robust Decision Making (RDM), for developing robust climate policies under conditions of uncertainties. This paper analyses the strengths and weaknesses of each method and provides insights into their application for climate policy analysis and decision-making.
• "Natural Gas and Israel's Energy Future: Near-term Decisions from a Strategic Perspective" by Popper et al. (2009) examines the role of natural gas in Israel's energy future and explores strategic decisions related to its utilization. The study provides a strategic perspective on energy choices, considering both economic, security, and environmental factors. Additionally, it offers insights into decision-making processes under conditions of deep uncertainty.
• "Scenarios that Illuminate Vulnerabilities and Robust Responses" by Lempert (2013) focuses on the use of scenarios to identify vulnerabilities and develop robust responses in the face of complex and uncertain challenges. This paper discusses the role of scenarios and their importance in discovering critical uncertainties. This helps decision-makers in informing adaptive decision-making processes.
• "A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios" by Lempert et al. (2006) introduces a general analytic method for generating robust strategies and narrative scenarios. The paper presents a structured approach that combines quantitative analysis with storytelling, enabling decision-makers to develop strategies that are robust to a range of uncertainties and convey their implications effectively.
• The "Colorado River Basin Water Supply and Demand Study" by the Bureau of Reclamation (2012) provides a comprehensive assessment of water supply and demand in the Colorado River Basin. The study examines future scenarios, analyses potential imbalances between supply and demand, and explores adaptive strategies to address water management challenges in the basin.

[edit] References

1. ↑ https://www.rand.org/pardee/methods-and-tools/robust-decision-making.html
2. ↑ ^{2.0 2.1 2.2} Lempert, R., J. (2019). Robust Decision Making (RDM), in Decision Making Under Deep Uncertainty — 2019, pp. 23-51
3. ↑ Donald Rumsfeld, Department of Defense News Briefing, February 12, 2002.
4. ↑ "Defense.gov News Transcript: DoD News Briefing – Secretary Rumsfeld and Gen. Myers, United States Department of Defense (defense.gov)". February 12, 2002. Archived from the original on March 20, 2018.
5. ↑ Rosenhead, M. J., M. Elton, S. K. Gupta. 1972. Robustness and optimality as criteria for strategic decisions.Oper. Res. Quart.23(4)413–430.
6. ↑ ^{6.0 6.1} Metz, B., O. Davidson, R. Swart, J. Pan, eds. 2001. Climate change 2001: Mitigation. Contribution of Working Group III to the Third Assessment [TAR] Report of the Intergovernmental Panel on Climate Change (IPCC), Cambridge University Press, Cambridge, UK.
7. ↑ Morgan, M. G., M. Henrion. 1990. Uncertainty: A Guide to Deal- ing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, Cambridge, UK.
8. ↑ Saltelli, A., K. Chan, E. M. Scott. 2000. Sensitivity Analysis. John Wiley & Sons, New York.
9. ↑ Morgan, M. G., M. Kandlikar, J. Risebey, H. Dowlatabadi. 1999. Why conventional tools for policy analysis are often inad- equate for problems of global change. Climatic Change 41 271–281.
10. ↑ van Asselt, M. B. A. 2000. Perspectives on Uncertainty and Risk. Kluwer Academic Publishers, Dordrecht, The Netherlands.
11. ↑ Nakicenovic, N., J. Alcamo, G. Davis, B. de Vries, J. Fenhann, S. Gaffin, K. Gregory, A. Grübler. 2000. Special report on emissions scenarios. Working Group III, Intergovernmental Panel on Climate Change (IPCC), Cambridge University Press, Cambridge, UK.
12. ↑ Lempert et al.: A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios Management Science 52(4), pp. 514–528, 2006 INFORMS
13. ↑ Morgan, M. G., & Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge, UK: Cambridge University Press.
14. ↑ Walley, P. (1991). Statistical reasoning with imprecise probabilities. London: Chapman and Hall.
15. ↑ Dewar, J. A., Builder, C. H., Hix, W. M., & Levin, M. H. (1993). Assumption-based planning—A planning tool for very uncertain times. Santa Monica, CA, RAND Corporation. https://www.rand.org/pubs/monograph_reports/MR114.html. Retrieved July 20, 2018.
16. ↑ Vincent A. W. J. Marchau, Warren E. Walker, Pieter J. T. M. Bloemen, Steven W. Popper (2019). Decision Making under Deep Uncertainty. From Theory to Practice
17. ↑ ^{17.0 17.1 17.2} Lempert, R. J., Popper, S. W., & Bankes, S. C. (2003). Shaping the Next One Hundred Years: New Methods for Quantitative, Long-term Policy Analysis. Santa Monica, CA, RAND Corporation, MR-1626-RPC.
18. ↑ Wack, P. (1985). The gentle art of reperceiving—scenarios: Uncharted waters ahead (part 1 of a two-part article). Harvard Business Review (September–October): 73–89.
19. ↑ Bankes, S. C. (1993). Exploratory modeling for policy analysis. Operations Research, 41(3), 435–449.
20. ↑ ^{20.0 20.1} National Research Council (NRC) (2009). Informing decisions in a changing climate. National Academies Press.
21. ↑ Jan H. Kwakkel, The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making, Environmental Modelling & Software, Volume 96, 2017, Pages 239-250, ISSN 1364-8152, https://doi.org/10.1016/j.envsoft.2017.06.054. (https://www.sciencedirect.com/science/article/pii/S1364815217301251)
22. ↑ Hall, J. M., Lempert, R. J., Keller, K., Hackbarth, A., Mijere, C., & McInerney, D. (2012). Robust Climate Policies under uncertainty: A comparison of Info-Gap and RDM methods. Risk Analysis, 32(10), 1657–1672.
23. ↑ ^{23.0 23.1} Popper, S. W., Berrebi, C., Griffin, J., Light, T., Min, E. Y., & Crane, K. (2009). Natural gas and Israel’s energy future: Near-term decisions from a strategic perspective. Santa Monica, CA, RAND Corporation, MG-927.
24. ↑ Lempert, R. J. (2013). Scenarios that illuminate vulnerabilities and robust responses. Climatic Change, 117, 627–646.
25. ↑ Lempert, R. J., Groves, D. G., Popper, S. W., & Bankes, S. C. (2006). A general, analytic method for generating robust strategies and narrative scenarios. Management Science, 52(4), 514–528.
26. ↑ Bureau of Reclamation. (2012). Colorado River Basin water supply and demand study: study report United States Bureau of Reclamation (Ed.). Retrieved July 11, 2018 from http://www.usbr.gov/ lc/region/programs/crbstudy/finalreport/studyrpt.html.