Revision as of 13:13, 4 February 2022 (view source) S144993 (Talk | contribs) (Robust decision making)		Latest revision as of 18:54, 22 March 2022 (view source) S144993 (Talk | contribs) (→‎Annotated Bibliography)
(87 intermediate revisions by one user not shown)
Line 1:		Line 1:
−	Robust decision making	+
		+	= Abstract/introduction =
		+
		+	Managing and planning projects or portfolios includes making big decision often based either on a very limited knowledge base, or an uncertain future - or both. Dealing with a great set of unknowns and uncertainties requires a skill set to make decisions making outcomes robust enough to cope unexpected futures and new disrupting knowledge. This is where '''Robust decision making''' proves its importance. Robust decision making (RDM) is a framework for decision making that take the uncertainty of different futures into consideration and helps the decision maker to reach his/her end goals by making decisions strong enough to fit a wider range of future scenarios. Rather than questioning the different future scenarios, the methodology of RDM focuses on what actions can be done at the present in order to improve future stands. Dealing with project, program and portfolio management, RDM is a vital methodology from the beginning where information can be limited, all the way to the end stage of a project where gained knowledge have the potential to disrupt previous made decisions.
		+
		+	This article will provide an all-around description of what RDM is and how the methodology works. Additionally, the article will address a short background description of RDM, why RDM is so useful as it is and where it differentiates from traditional risk management approaches. The theory behind traditional risk management will not be thoroughly gone through.
		+
		+	__TOC__
		+
		+	= History =
		+	As with many technologies and scientific approaches, RDM has its origin from military activities. In the early 1980's, when the Cold War was reaching its terminating, the RAND corporation was using simulation models designed to predict the potential outcome of different strategies. <ref name="DMUDU"/> At this stage of time, the methodology of RDM was not very well developed and RAND's predictions not very accurate. It was not until the early 1990's when the two RAND analysts Robert Lempert and Steven Popper further evolved the methodology framework in cooperation with RAND Computer Scientist Steve Bankes. The methods of RDM has been used ever since in an increasing manner on a wide range of different problematics of a very high degree of uncertainty including defence, flood risk management and climate change. <ref name="millennium project RDM"/>
		+
		+	= RDM VS traditional risk management =
		+	The use of a traditional risk management approach can prove very effective and often bring good results. These approaches are limited though, as they only work well when uncertainty is well addressed. The methodology of traditional risk assessment comes in a number of different variations, but generally share a distinct pattern known as "predict-then-act". This approach follows the method illustrated in picture 1[[File:Predict-then-act vs rdm.png|400px|thumb|right|Framework comparison between traditional risk assessment process and RDM <ref name="Predict-then-act VS RDM"/>]]. The approach first asks the question: "What will future conditions be?" and based in the information gained from this question, the next question: "What is the next near-term decision?" is asked. At the end, a final question in the form of a sensitivity analysis is asked: "How sensitive is the decision on the conditions?". However, the "predict-then-act" approach can prove valuable, it only holds its strength when the future is not changing and is not difficult to predict. In situations of severe uncertainty, using a traditional risk assessment approach often lead to an underestimation of uncertainties as an optimal decision otherwise would be hard to find. It is the search of the optimal solution that limits the "predict-then-act" methodology, as it does not account well enough for other outcomes.
		+	RDM addresses the tendency of narrow-mindedness that traditional risk assessment can have. RDM "flips" the order of step in the "predict-than-act". As visualized in picture 1, the approach that RDM takes, often called the bottom-up analysis, starts by looking at the proposed strategies. The next step is to address the vulnerabilities of these strategies and access whether or not they meet the set goals. The third and final step is to alter the initial strategies identified in step one with respect to the vulnerabilities identified in the second step. These steps are iterated over a number of times until a robust strategy is reached.
		+
		+	= Process of robust decision making: step-by-step =
		+	In this section, the different steps of the RDM process will be addressed. The section will give an insight in the process behind the methodology, and provide the background knowledge for the reader needed to perform simple step-by-step iterations of RDM. As RDM is made to solve very complicated problems with a high degree of uncertainty with potential outcomes counting hundreds if not thousands different futures, performing RDM in the scale it is made for, will require large simulation capacity. [[File:RDM steps.png|400px|thumb|right|Step-by-step process of RDM <ref name="DMUDU"/>]]
		+
		+	== Decision structuring ==
		+	The first step of the iterative RDM process is the decision structuring, a deliberation step often done in collaboration with stakeholders. In this step, stakeholders gain a common understanding of the challenge they are facing, what are the objectives and success criteria, what measures can be done to pursue the identifies objectives under the number of uncertainties the problem statement is underlying. <ref name="DMUDU"/> <ref name="RAND_RDM"/>
		+
		+	In this step, the "XLRM framework" is used in order to properly identify and organize all of the factors identified from the stakeholders analysis, making sure a common understanding of the problem is reached. The XLRM framework consists of four elements:
		+
		+	=== XLRM Framework ===
		+	The "XLRM framework" can be used to help properly identify and organize all of the factors identified from the stakeholders analysis, making sure a common understanding of the problem is reached. The XLRM framework consists of four elements: <ref name="RAND_RDM"/> [[File:XLRM Framework.png|300px|thumb|right|XLRM Framework <ref name="XLRM"/> ]]
		+
		+
		+	"'''X''': Uncertain factors": Here uncertain factors that may intervene with the ability to reach the goals set is listed and described.
		+
		+	"'''L''': Levers": The different "levers" of strategies policymakers can make and try to affect the outcomes of their strategies.
		+
		+	"'''R''': Relationship": Is the relationship between the policy "levers" and uncertainties identified and how these relationships can affect the outcome of different futures.
		+
		+	"'''M''': Metrics": Is the performance metrics or measures evaluating the outcomes and performances of strategies.
		+
		+	As illustrated in the picture to the right, the elements of uncertain factors, factors the stakeholder have no control of, and the identified policy levers collaborate to identify the relationships between the two, and from here, the metrics of the outcome of strategies can be assessed and potential measure to improve the outcome be defined. <ref name="XLRM"/>
		+
		+	== Strategy evaluation ==
		+	The second step of RDM is the step for seeking agreement on decisions between the stakeholders. In this step, the different strategies are run and evaluated on simulations, going through the many plausible futures. The simulations will give an insight in how the plausible futures in perform in comparison to a set measure of success. <ref name="RAND_RDM"/>
		+	The simulations are build up on top of the previous described XLRM framework. The simulations give evaluation of the strategies by analysing how the models, R, performs over the different strategies, L, and uncertainties, X, giving an output of performance of each strategy according to the metrics, M, defined. <ref name="XLRM"/> The advantage of running the simulations using the XLRM framework is that the model can be run as if it was a function:
		+
		+	<math>M = f(X,L)</math>
		+
		+	Looking at the evaluation of the strategies this way both help the decision makers to quantify the evaluations, but it also enables the methodology of RDM to interface with a number of modelling tools like Python, Excel and Vensim. <ref name="XLRM"/> The simulations will be able to generate a visualized graph pin pointing exactly which strategies will perform the best and with the best possibilities of reaching the set goals. Although the approach of simply counting up the number of strategies meeting the targeted goal is appealing, an overall assessment of the performance of the strategies should be conducted, leading to the next step of the process: vulnerability analysis.
		+
		+	== Vulnerability analysis ==
		+	In the third step of the RDM process, the vulnerability analysis, the strategies are evaluated on their uncertainties and under which condition the strategies will fail and does not meet the required goals. <ref name="XLRM"/> <ref name="RAND_RDM"/> Again, due to the degree of complexity and scale that RDM analysis typically is used for, the number of different outcomes dependent of strategies, uncertainties and policies will result in a number of outcomes not feasible for solving in a case to case approach. Therefore, a set of algorithms, typically statistical Scenario Discovery algorithms, are used in order to identify the most influential factors that pushes one future in a certain direction.<ref name="XLRM"/> <ref name="DMUDU"/> Having identified these factors, measures can be made in order to either try to eliminate them, or influence the outcome with a countermeasure.
		+
		+	== Trade-offs ==
		+	Coming from the vulnerability analysis having identified the vulnerabilities of the strategies, next to be done is an evaluating of trade-offs to make in order to pushed the outcomes of different strategies toward a more stable and certain future. The possibility of having a strategy that meets al goals and objectives are slim if not non existing. <ref name="RAND_RDM"/> The strategy will often have areas where it does not perform very well, and this is where the stakeholder will make trade-offs. An area where a trade-off would be made, could be the ability of keeping up with the set budget. Here the stakeholder might slacken boundaries of success by not meeting the budget, but meeting the goal of a successful project.
		+	As the trade-off analysis is conducted, the stakeholder might find a strategy that, by doing some trade-offs, will meet the requirement and result in a sufficiently acceptable futures outcome. In this stage, the stakeholders could end the RDM analysis. This highly depends on the stakeholders subjective view of the sufficiency of the strategy chosen and its ability to result in robust future outcome. <ref name="RAND_RDM"/>
		+
		+	== New futures and strategies ==
		+	Having completed the fourth step of the RDM analysis, the trade-off identification, the RDM analysis could as mentioned be done and a strategy be evaluated robust over a wide range of futures reaching the set goals. But, one might imagine, the previous steps in the analysis, the strategy evaluation and the vulnerability analysis could shine light on some new potential strategies resulting in potential future outcomes.
		+
		+
		+	= Human and machine collaboration =
		+	As emphasized through this article, RDM applies for complex decision making with a large number of possible direction to approach, with a high degree of uncertainty. <ref name="DMUDU"/> RDM can be applied on very small problems, by this only makes sense in an educational purpose, and cannot be expected to provide the same gained insight achieved when used on appropriately large and complex issues with the aid of software tools for support. When using RDM, have appropriate and efficient support in the form of software tools is key in order to achieve good analytic results. In RDM under deep uncertainty, simulation models are used for exploratory purposes and assists the decision maker in evaluating the different outcomes by asking the question: what if? - and then visualizes the different scenarios, their uncertainties and policy levers.
		+	Although the execution of a well conducted RDM analysis require analytical aids on the form of software tools, the analysis require the input of the stakeholders and decision makers. The software tools are not able to find a solution based on a set of variable on problems where RDM is used, as the problems suitable for RDM analysis underlie deep uncertainty and are not problems to which one finite solution can be given. The stakeholders and decision makers are as important a part in the analysis as the analytic tools, as the establishment of the strategies developed require a human mind. Additionally, through the steps of RDM, the human mind is essential for being able to identify necessary trade-offs needed to develop the strategy set at the end of the analysis. The collaboration between the decision makers and their identification of success criteria to be re-evaluated, as with the example of keeping up with the budget, in order to reach a robust decision is crucial.
		+
		+
		+	= Limitations of Robust decision making =
		+	When looking at areas of use for RDM, areas with a high degree of uncertainty, the biggest challenge, and a subject for limitation to the decision maker, is the analytic software tools needed. As this article also emphasizes, software tools are a key element when conducting a RDM analysis. Even though the decision maker are capable and well suitable for conduction the analysis with the methodology in mind, the software knowledge of decision maker may very well be limited or not existing. Therefore, RDM analysis most often will require an expert within analytical software tools. In relation to this, the conduction of simulations to most problems may be feasible on an average laptop or desktop computer, but on very large and complex problems these may proof inefficient for solving within a reasonable timeframe. In this case, either large-scale cluster computing or cloud based solutions will be necessary which will require additional funds. <ref name="DMUDU"/>
		+	Taking a different view at the analytical tools needed, these require data to contribute to the analysis - data that might not be accessible. An example of which this could be the case is when applying RDM in a climate change relation looking at change in a lest developed place on earth. <ref name="Barriers_opportunities"/> Usually, data on the climate development will be available from developed counties around earth, but this often does not apply to developing counties. These places simply has not had the required resources at the time, or present, to conduct and gather the data.
		+
		+	= Annotated Bibliography =
		+
		+	"'''Decision making under deep uncertainty'''" provides a good over all look in the struggles when dealing with deep uncertainty. Among other methodologies and approaches described in the book, it provides an insight in to the advantages of RDM and in what cases it proves its use. Additionally, the book addresses shortly some of the short comings of RDM. The book provides good explanatory examples showing the use of RDM.
		+
		+	"'''The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making'''" is a paper on dealing with deep uncertainty using RDM and particular the explanatory approach to RDM rather than the predictive approach associated with traditional risk management. The paper in particular gives an insight in the toolkit typically used for RDM, and the requirement of these tools to perform a good RDM analysis.
		+
		+	= References =
		+	<references>
		+
		+	<ref name="DMUDU"> Vincent A. W. J. Marchau, Warren E. Walker, Pieter J. T. M. Bloemen, Steven W. Popper, "Desicion Making under Deep Uncertainty" </ref>
		+
		+	<ref name="millennium project RDM"> Lembert, Robert; Popper, Steven; Banks, Steve; RAND, "ROBUST DECISIONMAKING (RDM)" </ref>
		+
		+	<ref name="RAND_RDM"> https://www.rand.org/pubs/tools/TL320/tool/robust-decision-making.html </ref>
		+
		+	<ref name="XLRM"> Kwakkel, Jan H.,"The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making" </ref>
		+
		+	<ref name="Predict-then-act VS RDM"> https://toolkit.climate.gov/image/3273 </ref>
		+
		+	<ref name="Barriers_opportunities"> Bhave, Ajay Gajanan; Conway, Declan; Dessai, Suraje; Stainforth, David A. (2016) "Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world" </ref>

---

Latest revision as of 18:54, 22 March 2022

[edit] Abstract/introduction

Managing and planning projects or portfolios includes making big decision often based either on a very limited knowledge base, or an uncertain future - or both. Dealing with a great set of unknowns and uncertainties requires a skill set to make decisions making outcomes robust enough to cope unexpected futures and new disrupting knowledge. This is where Robust decision making proves its importance. Robust decision making (RDM) is a framework for decision making that take the uncertainty of different futures into consideration and helps the decision maker to reach his/her end goals by making decisions strong enough to fit a wider range of future scenarios. Rather than questioning the different future scenarios, the methodology of RDM focuses on what actions can be done at the present in order to improve future stands. Dealing with project, program and portfolio management, RDM is a vital methodology from the beginning where information can be limited, all the way to the end stage of a project where gained knowledge have the potential to disrupt previous made decisions.

This article will provide an all-around description of what RDM is and how the methodology works. Additionally, the article will address a short background description of RDM, why RDM is so useful as it is and where it differentiates from traditional risk management approaches. The theory behind traditional risk management will not be thoroughly gone through.

Contents 1 Abstract/introduction 2 History 3 RDM VS traditional risk management 4 Process of robust decision making: step-by-step 4.1 Decision structuring 4.1.1 XLRM Framework 4.2 Strategy evaluation 4.3 Vulnerability analysis 4.4 Trade-offs 4.5 New futures and strategies 5 Human and machine collaboration 6 Limitations of Robust decision making 7 Annotated Bibliography 8 References

[edit] History

As with many technologies and scientific approaches, RDM has its origin from military activities. In the early 1980's, when the Cold War was reaching its terminating, the RAND corporation was using simulation models designed to predict the potential outcome of different strategies. ^[1] At this stage of time, the methodology of RDM was not very well developed and RAND's predictions not very accurate. It was not until the early 1990's when the two RAND analysts Robert Lempert and Steven Popper further evolved the methodology framework in cooperation with RAND Computer Scientist Steve Bankes. The methods of RDM has been used ever since in an increasing manner on a wide range of different problematics of a very high degree of uncertainty including defence, flood risk management and climate change. ^[2]

[edit] RDM VS traditional risk management

The use of a traditional risk management approach can prove very effective and often bring good results. These approaches are limited though, as they only work well when uncertainty is well addressed. The methodology of traditional risk assessment comes in a number of different variations, but generally share a distinct pattern known as "predict-then-act". This approach follows the method illustrated in picture 1. The approach first asks the question: "What will future conditions be?" and based in the information gained from this question, the next question: "What is the next near-term decision?" is asked. At the end, a final question in the form of a sensitivity analysis is asked: "How sensitive is the decision on the conditions?". However, the "predict-then-act" approach can prove valuable, it only holds its strength when the future is not changing and is not difficult to predict. In situations of severe uncertainty, using a traditional risk assessment approach often lead to an underestimation of uncertainties as an optimal decision otherwise would be hard to find. It is the search of the optimal solution that limits the "predict-then-act" methodology, as it does not account well enough for other outcomes.

RDM addresses the tendency of narrow-mindedness that traditional risk assessment can have. RDM "flips" the order of step in the "predict-than-act". As visualized in picture 1, the approach that RDM takes, often called the bottom-up analysis, starts by looking at the proposed strategies. The next step is to address the vulnerabilities of these strategies and access whether or not they meet the set goals. The third and final step is to alter the initial strategies identified in step one with respect to the vulnerabilities identified in the second step. These steps are iterated over a number of times until a robust strategy is reached.

[edit] Process of robust decision making: step-by-step

In this section, the different steps of the RDM process will be addressed. The section will give an insight in the process behind the methodology, and provide the background knowledge for the reader needed to perform simple step-by-step iterations of RDM. As RDM is made to solve very complicated problems with a high degree of uncertainty with potential outcomes counting hundreds if not thousands different futures, performing RDM in the scale it is made for, will require large simulation capacity.

[edit] Decision structuring

The first step of the iterative RDM process is the decision structuring, a deliberation step often done in collaboration with stakeholders. In this step, stakeholders gain a common understanding of the challenge they are facing, what are the objectives and success criteria, what measures can be done to pursue the identifies objectives under the number of uncertainties the problem statement is underlying. ^{[1] [4]}

In this step, the "XLRM framework" is used in order to properly identify and organize all of the factors identified from the stakeholders analysis, making sure a common understanding of the problem is reached. The XLRM framework consists of four elements:

[edit] XLRM Framework

The "XLRM framework" can be used to help properly identify and organize all of the factors identified from the stakeholders analysis, making sure a common understanding of the problem is reached. The XLRM framework consists of four elements: ^[4]

"X: Uncertain factors": Here uncertain factors that may intervene with the ability to reach the goals set is listed and described.

"L: Levers": The different "levers" of strategies policymakers can make and try to affect the outcomes of their strategies.

"R: Relationship": Is the relationship between the policy "levers" and uncertainties identified and how these relationships can affect the outcome of different futures.

"M: Metrics": Is the performance metrics or measures evaluating the outcomes and performances of strategies.

As illustrated in the picture to the right, the elements of uncertain factors, factors the stakeholder have no control of, and the identified policy levers collaborate to identify the relationships between the two, and from here, the metrics of the outcome of strategies can be assessed and potential measure to improve the outcome be defined. ^[5]

[edit] Strategy evaluation

The second step of RDM is the step for seeking agreement on decisions between the stakeholders. In this step, the different strategies are run and evaluated on simulations, going through the many plausible futures. The simulations will give an insight in how the plausible futures in perform in comparison to a set measure of success. ^[4] The simulations are build up on top of the previous described XLRM framework. The simulations give evaluation of the strategies by analysing how the models, R, performs over the different strategies, L, and uncertainties, X, giving an output of performance of each strategy according to the metrics, M, defined. ^[5] The advantage of running the simulations using the XLRM framework is that the model can be run as if it was a function:

Looking at the evaluation of the strategies this way both help the decision makers to quantify the evaluations, but it also enables the methodology of RDM to interface with a number of modelling tools like Python, Excel and Vensim. ^[5] The simulations will be able to generate a visualized graph pin pointing exactly which strategies will perform the best and with the best possibilities of reaching the set goals. Although the approach of simply counting up the number of strategies meeting the targeted goal is appealing, an overall assessment of the performance of the strategies should be conducted, leading to the next step of the process: vulnerability analysis.

[edit] Vulnerability analysis

In the third step of the RDM process, the vulnerability analysis, the strategies are evaluated on their uncertainties and under which condition the strategies will fail and does not meet the required goals. ^{[5] [4]} Again, due to the degree of complexity and scale that RDM analysis typically is used for, the number of different outcomes dependent of strategies, uncertainties and policies will result in a number of outcomes not feasible for solving in a case to case approach. Therefore, a set of algorithms, typically statistical Scenario Discovery algorithms, are used in order to identify the most influential factors that pushes one future in a certain direction.^{[5] [1]} Having identified these factors, measures can be made in order to either try to eliminate them, or influence the outcome with a countermeasure.

[edit] Trade-offs

Coming from the vulnerability analysis having identified the vulnerabilities of the strategies, next to be done is an evaluating of trade-offs to make in order to pushed the outcomes of different strategies toward a more stable and certain future. The possibility of having a strategy that meets al goals and objectives are slim if not non existing. ^[4] The strategy will often have areas where it does not perform very well, and this is where the stakeholder will make trade-offs. An area where a trade-off would be made, could be the ability of keeping up with the set budget. Here the stakeholder might slacken boundaries of success by not meeting the budget, but meeting the goal of a successful project. As the trade-off analysis is conducted, the stakeholder might find a strategy that, by doing some trade-offs, will meet the requirement and result in a sufficiently acceptable futures outcome. In this stage, the stakeholders could end the RDM analysis. This highly depends on the stakeholders subjective view of the sufficiency of the strategy chosen and its ability to result in robust future outcome. ^[4]

[edit] New futures and strategies

Having completed the fourth step of the RDM analysis, the trade-off identification, the RDM analysis could as mentioned be done and a strategy be evaluated robust over a wide range of futures reaching the set goals. But, one might imagine, the previous steps in the analysis, the strategy evaluation and the vulnerability analysis could shine light on some new potential strategies resulting in potential future outcomes.

[edit] Human and machine collaboration

As emphasized through this article, RDM applies for complex decision making with a large number of possible direction to approach, with a high degree of uncertainty. ^[1] RDM can be applied on very small problems, by this only makes sense in an educational purpose, and cannot be expected to provide the same gained insight achieved when used on appropriately large and complex issues with the aid of software tools for support. When using RDM, have appropriate and efficient support in the form of software tools is key in order to achieve good analytic results. In RDM under deep uncertainty, simulation models are used for exploratory purposes and assists the decision maker in evaluating the different outcomes by asking the question: what if? - and then visualizes the different scenarios, their uncertainties and policy levers. Although the execution of a well conducted RDM analysis require analytical aids on the form of software tools, the analysis require the input of the stakeholders and decision makers. The software tools are not able to find a solution based on a set of variable on problems where RDM is used, as the problems suitable for RDM analysis underlie deep uncertainty and are not problems to which one finite solution can be given. The stakeholders and decision makers are as important a part in the analysis as the analytic tools, as the establishment of the strategies developed require a human mind. Additionally, through the steps of RDM, the human mind is essential for being able to identify necessary trade-offs needed to develop the strategy set at the end of the analysis. The collaboration between the decision makers and their identification of success criteria to be re-evaluated, as with the example of keeping up with the budget, in order to reach a robust decision is crucial.

[edit] Limitations of Robust decision making

When looking at areas of use for RDM, areas with a high degree of uncertainty, the biggest challenge, and a subject for limitation to the decision maker, is the analytic software tools needed. As this article also emphasizes, software tools are a key element when conducting a RDM analysis. Even though the decision maker are capable and well suitable for conduction the analysis with the methodology in mind, the software knowledge of decision maker may very well be limited or not existing. Therefore, RDM analysis most often will require an expert within analytical software tools. In relation to this, the conduction of simulations to most problems may be feasible on an average laptop or desktop computer, but on very large and complex problems these may proof inefficient for solving within a reasonable timeframe. In this case, either large-scale cluster computing or cloud based solutions will be necessary which will require additional funds. ^[1] Taking a different view at the analytical tools needed, these require data to contribute to the analysis - data that might not be accessible. An example of which this could be the case is when applying RDM in a climate change relation looking at change in a lest developed place on earth. ^[6] Usually, data on the climate development will be available from developed counties around earth, but this often does not apply to developing counties. These places simply has not had the required resources at the time, or present, to conduct and gather the data.

[edit] Annotated Bibliography

"Decision making under deep uncertainty" provides a good over all look in the struggles when dealing with deep uncertainty. Among other methodologies and approaches described in the book, it provides an insight in to the advantages of RDM and in what cases it proves its use. Additionally, the book addresses shortly some of the short comings of RDM. The book provides good explanatory examples showing the use of RDM.

"The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making" is a paper on dealing with deep uncertainty using RDM and particular the explanatory approach to RDM rather than the predictive approach associated with traditional risk management. The paper in particular gives an insight in the toolkit typically used for RDM, and the requirement of these tools to perform a good RDM analysis.

[edit] References

1. ↑ ^{1.0 1.1 1.2 1.3 1.4 1.5} Vincent A. W. J. Marchau, Warren E. Walker, Pieter J. T. M. Bloemen, Steven W. Popper, "Desicion Making under Deep Uncertainty"
2. ↑ Lembert, Robert; Popper, Steven; Banks, Steve; RAND, "ROBUST DECISIONMAKING (RDM)"
3. ↑ https://toolkit.climate.gov/image/3273
4. ↑ ^{4.0 4.1 4.2 4.3 4.4 4.5} https://www.rand.org/pubs/tools/TL320/tool/robust-decision-making.html
5. ↑ ^{5.0 5.1 5.2 5.3 5.4 5.5} Kwakkel, Jan H.,"The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making"
6. ↑ Bhave, Ajay Gajanan; Conway, Declan; Dessai, Suraje; Stainforth, David A. (2016) "Barriers and opportunities for robust decision making approaches to support climate change adaptation in the developing world"