Robust Decision Making: better decisions under uncertainty
From apppm
(Difference between revisions)
Colombofabio (Talk | contribs) (→Foundations of Robust Decision Making) |
Colombofabio (Talk | contribs) |
||
| Line 3: | Line 3: | ||
| − | = | + | = Conceptualising Robust Decision Making at times of Uncertainty= |
| − | == | + | == Origins == |
| − | Robust Decision Making (RDM) emerged in the | + | Robust Decision Making (RDM) emerged in the 1980s, when analysts of the RAND Corporation, a California-based think tank affiliated with the U.S. Government, developed a framework to evaluate the effectiveness of nuclear weapon systems <ref name=”RAND corp”/> <ref name=”Lempert RDM”/>. Designed to mitigate the uncertainty and ambiguity experienced by U.S. Government officials involved in the planning and implementation of nuclear deterrence strategies, RDM included simulation techniques, sensitivity analysis, and real options analysis. In the 1990s and 2000s, RDM received increasing interest from private companies interested in exploring new project management techniques applicable to a wide range of industries, including construction, software development, and environmental management. Today, RDM is an established approach in project management, recognized for its ability to help project managers making well-informed and timely decisions under pressure ad at times of uncertainty. |
| + | == Literature review == | ||
| + | TO SHORTEN BY A LOT AND TO NARROW DOWN TO RDM | ||
| − | == | + | According to former United States Secretary of Defence Donald Rumsfeld, there are different types of knowledge: known knowns, known unknowns, and unknown unknowns. Known knowns refer to things that we know for sure. Known unknowns refer to things that we know we do not know. However, the most challenging category is the unknown unknowns, which refers to things that we do not know we do not know <ref name="Rumsfeld"/> <ref name="Defence">. Knight further elaborates on this concept and proposes a distinction between risk and uncertainty. The former indicates situations in which the unknown can be measured (through probabilities) and, therefore, controlled. The latter indicates situations in which the unknown can't be quantified and can't, therefore, be measured <ref name="Knight"/>. Based on Knight’s distinction, academics categorised the various levels of uncertainty in decision-making, ranging from complete certainty to total ignorance <ref name="Courtney"/> <ref name="Walker"/> <ref name="Lempert RDM"/>. These levels are categorized based on the knowledge possessed about various aspects of a problem, including the future state of the world, the model of the relevant system, the outcomes from the system, and the weights that the various stakeholders will put on the outcomes. The four intermediate levels of uncertainty are defined as Level 1, where historical data can be used as predictors of the future <ref name="Hillier"/>; Level 2, where probability and statistics can be used to solve problems; Level 3, where plausible future worlds are specified through scenario analysis; and Level 4, where the decision maker only knows that nothing can be known due to unpredictable events,lack of knowledge, or unavailability of data <ref name="Taleb"/> <ref name="Schwartz"/>. When dealing with issues distinguished by greater level of uncertainty (Level 4), a more sophisticated and in-depth data gathering is often unhelpful. The decision-making process in such situations is defined as ''decision making under deep uncertainty'' (DMDU) <ref name="Lempert RDM"/>. Instead of a "predict and act" paradigm, which attempts to anticipate potential future problems and act on that prediction, DMDU approaches are based on a "monitor and adapt" paradigm, which places more emphasis on efforts aimed at preparing for unknown occurrences and adjust accordingly <ref name="Walker"/>. In order to make decisions for unpredictable occurrences and long-term changes, this "monitor and adapt" paradigm "explicitly identifies the deep uncertainty surrounding decision making and underlines the necessity to take this deep uncertainty into consideration” (<ref name="DMUDU"/>, p. 11). This article explores RDM under uncertainty, an approach dwelling under the realm of DMDU methodologies. |
| − | + | ||
| − | + | ||
| − | According to | + | According to the "monitor and adapt" paradigm, RDM refers to a collection of ideas, procedures, and supportive technologies intended to rethink the function of quantitative models and data in guiding choices in situations affected by uncertainty. Models and data become tools for systematically exploring the consequences of assumptions, expanding the range of futures considered, creating innovative new responses to threats and opportunities, and sorting through a variety of scenarios, options, objectives, and problem framings to identify the most crucial trade-offs confronting decision makers. This contrasts with the traditional view of models as tools for prediction and the subsequent prescriptive ranking of decision options. This means that, rather than improving forecasts, models and data are used to facilitate decision makers in taking robust decisions <ref name="Popper"/>. As argued by Marchau et. al., robustness of decisions is, therefore, guaranteed by iterating several times the solution to a problem while straining the suggested decisions against a wide variety of potential scenarios. In doing so, RDM endure the decision-making process under deep uncertainty <ref name="Lempert RDM"/>. |
| − | + | Although scholars widely explored the practicalapplications of RDM in project management, the theoretical support of the application of this framework in project management practices remains largely unexplored. The remainder of the article will, therefore, concentrate on the fundamental principles of RDM, guide the reader through the methodology, give an illustration of how RDM has been successfully used in a large-scale project, and discuss benefits and limitation of the approach. | |
| − | |||
== Foundations of Robust Decision Making == | == Foundations of Robust Decision Making == | ||
| Line 25: | Line 24: | ||
'''Decision Analysis (DA)''' | '''Decision Analysis (DA)''' | ||
--- | --- | ||
| − | The discipline of DA provides a framework for creating and utilizing well-structured decision aids. RDM exploits this framework | + | The discipline of DA provides a framework for creating and utilizing well-structured decision aids. RDM exploits this framework by focusing specifically on finding trade-offs and describing vulnerabilities to create robust decisions based on stress testing of probable future routes. Both DA and RDM seek to improve the decision-making process by being clear about goals, utilizing the finest information available, carefully weighing trade-offs, and adhering to established standards and conventions to assure legitimacy for all parties involved. However, while DA seeks optimality through utility frameworks and assumptions <ref name="Morgan"/>, RDM seeks robustness assuming uncertainty as unescapable and probabilities as imprecise <ref name="Walley"/>, further highlighting trade-offs between plausible options. |
'''Assumption-Based Planning (ABP)''' | '''Assumption-Based Planning (ABP)''' | ||
--- | --- | ||
| − | By expanding awareness of how and why things could fail, RDM uses the ideas of stress testing and red teaming to lessen the harmful impacts of overconfidence in current plans and processes <ref name="Dewar"/>. This approach was first implemented in the so-called Assumption-Based Planning (ABP) framework. Starting with a written version of an organization's plans, ABP finds the explicit and implicit assumptions made during the formulation of that plan that, if | + | By expanding awareness of how and why things could fail, RDM uses the ideas of stress testing and red teaming to lessen the harmful impacts of overconfidence in current plans and processes <ref name="Dewar"/>. This approach was first implemented in the so-called Assumption-Based Planning (ABP) framework. Starting with a written version of an organization's plans, ABP finds the explicit and implicit assumptions made during the formulation of that plan that, if inaccurate, would result in failure. These assumptions are identified by decision-makers, who then create backup plans and "hedging" strategies to be used in case of necessity. ABP takes, then, into account "signposts", which refers to monitoring patterns and events to spot any faltering presumptions <ref name="DMUDU"/>. |
'''Scenario Analysis (SA)''' | '''Scenario Analysis (SA)''' | ||
--- | --- | ||
| − | In order to deal with deep uncertainty, RDM builds upon the idea of SA <ref name="Lempert 2003"/>. Scenarios are defined as collections of potential future occurrences that illustrate various worldviews without explicitly assigning a relative likelihood score <ref name="Wack 1985"/>. They are frequently | + | In order to deal with deep uncertainty, RDM builds upon the idea of SA <ref name="Lempert 2003"/>. Scenarios are defined as collections of potential future occurrences that illustrate various worldviews without explicitly assigning a relative likelihood score <ref name="Wack 1985"/>. They are frequently envisioned in deliberative processes involving stakeholders and not including probabilities of occurrence. This is done with the objective of broadening the range of scenarios taken into consideration and facilitate the interactions with a wide variety of futures to audiences. In incorporating analytical techniques borrowed from SA, RDM declines the knowledge about the future into a selected range of potential situations—a technique that assists decision-makers in envisioning future risks and better navigating strategic environments of deep uncertainty. |
'''Exploratory Modeling (EM)''' | '''Exploratory Modeling (EM)''' | ||
--- | --- | ||
| − | According to Bankes, | + | According to Bankes, Exploratory Modeling (EM) is one of the most appropriate tools that allow the integration of DA, ABP, and SA in RDM <ref name="Bankes 1993"/>. Without prioritizing one set of assumptions over another, EM factors a wide range of assumptions into a limited number of results. In other words,EM is strongly beneficial when a single model cannot be validated because of a lack of evidence, insufficient or conflicting ideas, or unknown futures. Therefore, by lowering the demands for analytic tractability on the models employed in the study, EM offers a quantitative framework for stress testing and scenario analysis and allows the exploration of futures and strategies. As EM favours no base case or one future as an anchor point, it allows for genuinely global, large-N studies in support of the more qualitative methods ingrained in SA approaches |
= Application = | = Application = | ||
'''Theoretical framework''' | '''Theoretical framework''' | ||
| − | RDM is a learning process based on the | + | RDM is a learning process based on a process that the scholarship defines as “deliberation with analysis”. The framework requires that the decision-making parties discuss their goals and alternatives, which are based on assessments provided by analysts , who imagine scenarios and policy options underpinned by their analysis of the quantitative data available. This is especially recommended in settings where there are a variety of decision-makers, who must make decisions in ever-changing environments, and whose objectives may change as a result of their collaboration with others. <ref name="NRC"/> As illustrated in Figure 1 RDM methodology follows 5 major steps, described in the paragraphs below. |
----> PICTURE HERE!!!!!! FIGURE 1 | ----> PICTURE HERE!!!!!! FIGURE 1 | ||
| − | |||
| − | |||
'''Step1: Decision Framing.''' | '''Step1: Decision Framing.''' | ||
| − | + | The RDM process starts with a decision framing workshop in which stakeholders brainstorm and define the key factors in the analysis. These include decision-makers’ goals and criteria, the potential courses of action they may choose to accomplish those goals, the uncertainties that might impact the link between actions and results, and the connections between actions, uncertainties, and goals. Once gathered, this information is put into a framework known as “XLRM” <ref name="Lempert 2003"/> <ref name="Jan H"/>, where: | |
| − | The RDM process starts with a decision framing workshop in which stakeholders brainstorm and define the key factors in the analysis. These include decision-makers’ goals and criteria, the potential courses of action they may choose to accomplish those goals, the uncertainties that might impact the link between actions and results, and the connections between actions, uncertainties, and goals. Once | + | |
| − | + | ||
* “X” stands for exogenous variables (factors not under the control of the decision makers) | * “X” stands for exogenous variables (factors not under the control of the decision makers) | ||
* “L” stands for policy levers (policies that affect the system to achieve goals) | * “L” stands for policy levers (policies that affect the system to achieve goals) | ||
* “R” stands for relationships (relevant variables needed to correctly evaluate and benchmark policies) | * “R” stands for relationships (relevant variables needed to correctly evaluate and benchmark policies) | ||
* “M” stands for measures of performance (metrics, not necessarily quantitative, given from stakeholders to evaluate policies) | * “M” stands for measures of performance (metrics, not necessarily quantitative, given from stakeholders to evaluate policies) | ||
| − | |||
The output is a set of potential robust strategies. | The output is a set of potential robust strategies. | ||
'''Step 2: Evaluate strategies.''' | '''Step 2: Evaluate strategies.''' | ||
| − | + | According to the ABP approach, RDM exploits simulation models to assess the proposed strategies of Step 1 in each of many plausible paths into the future. This process of generating strategies may use a variety of techniques, spanning from optimization methods to public debate <ref name="Hall J"/>. It is commonly observed, however, that strategy evaluation usually combines them all <ref name="Popper 2009"/>. | |
| − | According to the ABP approach, RDM | + | |
'''Step 3: Vulnerability analysis.''' | '''Step 3: Vulnerability analysis.''' | ||
| − | |||
Data analytics and visualization techniques are then used to search for and describe vulnerabilities of the strategies under consideration. Specifically, statistical methods are used to find the critical variables that best distinguish futures in which these strategies succeed or fail. <ref name="Lempert 2013"/>. The output of this step is a multitude of scenarios which are then clustered based on the identified vulnerabilities. | Data analytics and visualization techniques are then used to search for and describe vulnerabilities of the strategies under consideration. Specifically, statistical methods are used to find the critical variables that best distinguish futures in which these strategies succeed or fail. <ref name="Lempert 2013"/>. The output of this step is a multitude of scenarios which are then clustered based on the identified vulnerabilities. | ||
'''Step 4: Trade-off analysis.''' | '''Step 4: Trade-off analysis.''' | ||
| − | |||
The scenarios generated underlie the evaluation of trade-offs between strategies. This step is useful to give insights on how the future would look like if a strategy was chosen. | The scenarios generated underlie the evaluation of trade-offs between strategies. This step is useful to give insights on how the future would look like if a strategy was chosen. | ||
'''Step 5: New futures and strategies.''' | '''Step 5: New futures and strategies.''' | ||
| − | + | The result from Step 4 is necessary to identifying and appraising alternative solutions, allowing to focus on the most robust ones. Sometimes, the identification and appraisal process rely on experts’ opinions <ref name="Popper 2009"/> <ref name="Lempert 2003">. In other cases, optimization approaches are used instead <ref name="Lempert 2006">. It is either when no more robust strategies can be generated, or when the already identified ones are considered sufficiently satisfactory that the procedure ends. | |
| − | The result from Step 4 | + | |
To appraise and quantify trade-offs between strategies, RDM exploits both absolute and relative performance indicators. Specifically, the formers are beneficial when specific objectives would like to be met (e.g., maximisation of profit). The latter are beneficial when decision-makers seek the evaluation of different strategies in different possible futures and search for the most robust ones. | To appraise and quantify trade-offs between strategies, RDM exploits both absolute and relative performance indicators. Specifically, the formers are beneficial when specific objectives would like to be met (e.g., maximisation of profit). The latter are beneficial when decision-makers seek the evaluation of different strategies in different possible futures and search for the most robust ones. | ||
| − | |||
| Line 81: | Line 70: | ||
'''Introduction and background.''' | '''Introduction and background.''' | ||
--- | --- | ||
| − | The Colorado River is the main source of water in | + | The Colorado River is the main source of water in southwestern United States, supplying 4.5 million acres of agriculture with irrigation as well as power and water to about 40 million people <ref name="Reclamation 2012"/>. Four Upper Basin States (Colorado, New Mexico, Utah, and Wyoming) and three Lower Basin States (Arizona, California, and Nevada) each gets 15 million acre-feet of water under the terms of the 1922 Colorado River Compact. The system's dependability is being put under more and more pressure because of the deep supply uncertainty and rising demand. The Colorado River Basin Study was lunched in 2010 by the seven Basin States and the US Bureau of Reclamation to: |
| − | * ensure a 10-years running water flow from the Upper to the Lower Basin with a minimum of 7.5 maf/year (Upper Basin reliability), | + | * ensure a 10-years running water flow from the Upper to the Lower Basin with a minimum of 7.5 maf/year (Upper Basin reliability) (Maf = million acre-feet. It is a non-SI used in reference to large-scale water sources.), |
| − | * maintain Lake Mead | + | * maintain Lake Mead (situated at the border of Nevada and Arizona) at a minimum of 1000 feet of pool elevation (Lower Basin reliability). |
Reclamation evaluated DMDU methodologies in a pilot study, adopted them to assist ongoing planning, and utilized RDM to frame the vulnerability and adaptation evaluations for the Basin Study. The research results were used to specify a solid, flexible management plan. | Reclamation evaluated DMDU methodologies in a pilot study, adopted them to assist ongoing planning, and utilized RDM to frame the vulnerability and adaptation evaluations for the Basin Study. The research results were used to specify a solid, flexible management plan. | ||
| + | |||
'''Decision framing and current vulnerability analysis.''' | '''Decision framing and current vulnerability analysis.''' | ||
| Line 94: | Line 84: | ||
| − | |||
| + | -> PICTURE HERE!!!!!! FIGURE 2 | ||
Based on the above framework, the project team created portfolios of individual management choices that might either boost the supply or decrease demand for the Basin states. Stakeholders developed four portfolios, each with a unique set of investment possibilities. Future vulnerability management was represented by Portfolio B (Reliability Focus) and Portfolio C (Environmental Performance Focus). Portfolio D (Common Options) was established to contain just those choices in both Portfolios B and C, while Portfolio A (Inclusive) was defined to include all alternatives in either Portfolios B or C. Options were ranked in order of cost-effectiveness, which was calculated by dividing the average yearly yield by the total project cost. By modelling the investment choices that a basin manager would take under various simulated Basin conditions, CRSS modelled these portfolios as potential strategies. | Based on the above framework, the project team created portfolios of individual management choices that might either boost the supply or decrease demand for the Basin states. Stakeholders developed four portfolios, each with a unique set of investment possibilities. Future vulnerability management was represented by Portfolio B (Reliability Focus) and Portfolio C (Environmental Performance Focus). Portfolio D (Common Options) was established to contain just those choices in both Portfolios B and C, while Portfolio A (Inclusive) was defined to include all alternatives in either Portfolios B or C. Options were ranked in order of cost-effectiveness, which was calculated by dividing the average yearly yield by the total project cost. By modelling the investment choices that a basin manager would take under various simulated Basin conditions, CRSS modelled these portfolios as potential strategies. | ||
| Line 101: | Line 91: | ||
'''Evaluate strategies.''' | '''Evaluate strategies.''' | ||
--- | --- | ||
| − | Robustness was defined as the strategy that minimizes regret under a plethora of conceivable future scenarios. In this case, regret indicates the additional water supply needed to keep Lake Mead at | + | Robustness was defined as the strategy that minimizes regret under a plethora of conceivable future scenarios. In this case, regret indicates the additional water supply needed to keep Lake Mead at its minimum required level. The more the supply, the more the regret. It is worth noting that regret is not eliminated. In fact, throughout the simulation, it was observed that some strategies pursued big investments in the wettest futures and little to no investments in the driest ones. |
'''Trade-off analysis.''' | '''Trade-off analysis.''' | ||
--- | --- | ||
| − | Each technique performs differently in terms of cost and dependability measure. In futures where the supply is lower it was observed that Portfolio A was the strategy with the highest likelihood to prevent water delivery vulnerability, at the expense of the highest cost. Portfolio D (a subset of A) had lower costs, but also lower probability to prevent vulnerabilities. It was concluded that Portfolios A and B were associated to lowest number of | + | Each technique performs differently in terms of cost and dependability measure. In futures where the supply is lower it was observed that Portfolio A was the strategy with the highest likelihood to prevent water delivery vulnerability, at the expense of the highest cost. Portfolio D (a subset of A) had lower costs, but also lower probability to prevent vulnerabilities. It was concluded that Portfolios A and B were associated to lowest number of years with critical water levels, but the highest costs. Portfolio C had slightly more years with critical water levels, but lower costs. |
'''New futures and strategies.''' | '''New futures and strategies.''' | ||
--- | --- | ||
| − | Probability thresholds that, with sufficient certainty, would indicate that a given vulnerability would be likely to occur, and that a consequence management action should be taken, were determined based on statistical analysis and the primary vulnerabilities found. This data was | + | Probability thresholds that, with sufficient certainty, would indicate that a given vulnerability would be likely to occur, and that a consequence management action should be taken, were determined based on statistical analysis and the primary vulnerabilities found. This data was used to create an effective plan for the whole Basin that directs the investment of greater water supply yield and demand reductions. The sample paths (dashed lines) in the picture 3 illustrate one of many potential implementation routes. If the basin were to follow a trajectory that was consistent with the vulnerability “Below Historical Streamflow During Extreme Drought”, the example paths illustrate how basin managers would provide additional supply. The figure shows decisions taken from Basin managers up to 2030. It then highlights a decision point in 2030 where, by evaluating scenarios, it might be possible that the future will not be consistent with the same vulnerability. If conditions are consistent with the “Severe Declining Supply Scenario”, then the Basing managers should increase the net supply to more than 3.6 maf between 2031-2040. The same decision approach is taken for the following years. |
----> PICTURE HERE!!!!!! FIGURE 3 | ----> PICTURE HERE!!!!!! FIGURE 3 | ||
| − | == | + | == Assessment and Conclusion == |
| + | When used with project management frameworks, RDM can offer several advantages. First, RDM may help project managers make informed and data-driven decisions. It can assist managers in identifying more reliable and adaptable plans by considering a variety of uncertainties and their possible impacts. Second, RDM can help project managers identify and assess various risks associated with a project, including both known and unknown risks. This can enable managers to develop contingency plans and other risk mitigation strategies to address potential issues. Finally, RDM can facilitate stakeholder engagement and participation in the decision-making process. By considering the perspectives and preferences of various stakeholders, RDM can help managers develop solutions that are more acceptable and feasible. | ||
| + | However, there are also some limitations to using RDM in a project management framework. First, RDM can be resource-intensive, requiring significant data collection, analysis, and modelling. This can be particularly challenging for smaller projects or those with limited resources. Second, complexity and uncertainty can make it challenging to apply RDM effectively, particularly in cases where there are significant data gaps or limited information available. Third, models and simulations are only as good as the data and assumptions that underlie them. This can lead to errors or biases in the decision-making process. | ||
| − | + | = Conclusion = | |
| + | This article explored the realm of RDM with particular focus on deep uncertainty scenarios. A literature review pointed out that there is scarce theoretical background of RDM applied to project management and it was, therefore, done for the general RDM framework. Theoretical groundings of RDM were given, as well as the theoretical framework of the tool. This provides project managers with solid foundations to start with when thinking of implementing this framework. A practical example of how RDM was successfully implemented in a large-scale project was presented to guide managers through the application if the tool. Finally, benefits and limitations were discussed to highlight advantages and weaknesses of the approach. | ||
Revision as of 09:18, 5 April 2023
Contents |
Abstract
Robust Decision Making (RDM) involves a set of ideas, methods, and tools that employ computation to facilitate better decision-making when dealing with situations of significant uncertainty. It integrates Decision Analysis, Assumption-Based Planning, Scenario Analysis, and Exploratory Modelling to simulate multiple possible outcomes in the future, with the aim of identifying policy-relevant scenarios and robust adaptive strategies. These RDM analytic tools are frequently embedded in a decision support process referred to as "deliberation with analysis," which fosters learning and agreement among stakeholders [1]. This article provides a review of the current state of the art in RDM in project management, including the key principles and practices of RDM, such as the importance of data gathering and analysis, considering different options, and involving stakeholders. Furthermore, this article examines the benefits, challenges, and limitations of RDM in project management and provides insights into future directions for research in this area. Its aim is to provide project managers with a deeper understanding of the principles and practices of RDM, along with insights on and example of how to correctly implement RDM in project management. Ultimately, this article aims to contribute to the development of more effective and efficient approaches to project management and decision making by promoting the use of RDM in project management.
Conceptualising Robust Decision Making at times of Uncertainty
Origins
Robust Decision Making (RDM) emerged in the 1980s, when analysts of the RAND Corporation, a California-based think tank affiliated with the U.S. Government, developed a framework to evaluate the effectiveness of nuclear weapon systems Cite error: Invalid <ref> tag;
refs with no content must have a name Cite error: Invalid <ref> tag;
refs with no content must have a name. Designed to mitigate the uncertainty and ambiguity experienced by U.S. Government officials involved in the planning and implementation of nuclear deterrence strategies, RDM included simulation techniques, sensitivity analysis, and real options analysis. In the 1990s and 2000s, RDM received increasing interest from private companies interested in exploring new project management techniques applicable to a wide range of industries, including construction, software development, and environmental management. Today, RDM is an established approach in project management, recognized for its ability to help project managers making well-informed and timely decisions under pressure ad at times of uncertainty.
Literature review
TO SHORTEN BY A LOT AND TO NARROW DOWN TO RDM
According to former United States Secretary of Defence Donald Rumsfeld, there are different types of knowledge: known knowns, known unknowns, and unknown unknowns. Known knowns refer to things that we know for sure. Known unknowns refer to things that we know we do not know. However, the most challenging category is the unknown unknowns, which refers to things that we do not know we do not know [2] Cite error: Closing </ref> missing for <ref> tag
Cite error:
<ref> tags exist, but no <references/> tag was found
