Levels of uncertainties

Latest revision as of 15:27, 9 May 2023

WIKI ARTICLE

[edit] Abstract

This wiki article discusses uncertainties and risks and the different levels of uncertainty that exist. Uncertainty is defined as any departure from the ideal of complete determinism, and it has several facets, such as nature, levels, division of one's cognizance, location/source, difficulties in communication, and human values/subjectivity. The levels of uncertainty, or scales of uncertainty, have been proposed by various scholars and are a way of measuring the probability of uncertainty states within a given system. There are several levels of uncertainty, ranging from statistical uncertainty, which can be expressed in statistical terms, to total ignorance, where we do not know what we do not know. The article describes how managers can use the levels of uncertainties to classify uncertainties in projects, programs, and portfolios, and evaluate the predictive capabilities of the information given to them or set about gathering more information. Developing appropriate reactions is key in managing uncertainties and risks, and getting an understanding of the level of uncertainty is a starting point.

[edit] Introduction: Uncertainties and Risks

Levels of uncertainty is one of the facets of uncertainty. In the following section the basic framework of uncertainty is introduced.

Uncertainty is a multifaceted creature and has been generally defined as “Any departure from the unachievable ideal of complete determinism”. Many frameworks have been developed to explain uncertainty, with different aspects. Most frameworks base themselves on the work of Walker et. al. (2003) (original 3 parts demarcated with *), dealing with 3 facets of uncertainty. Additions to the frameworks have however been made over time (Bevan, 2022), and as such consist of:

1. Nature (epistemic/aleatory)*. Aleatory uncertainty is mainly about natural randomness, where epistemic is uncertainty in parameters. We will mainly be dealing with epistemic uncertainty in this article.
2. Level of uncertainty*. The level defines how certain we are on a spectrum between determinacy/risk to indeterminacy/ignorance.
3. Division of uncertainty by cognizance (or lack thereof). Different types of ignorance/surprises. (See Bevan (2022) for more information).
4. The location/source*. Here, we refer to Oehmen & Kwakkel (2023) where the location/source can be seen in management as organizational, technical or as requirement driven.
5. Difficulties in communication/forming consensus. This is seen as incommensurate values (i.e. not agreed upon values between stakeholders), and we will assume that projects, programs and portfolios will contain differences in these values that drive reasoning for projects. We will mainly deal with uncertainty under incommensurate values in this article. (Oehmen & Kwakkel 2023)
6. Incorporating uncertainties with human values/subjectivity, normativity. Incommensurate values, similar to point 5.

In this article, we will put our focus towards the levels of uncertainty.

[edit] Levels of Uncertainty – From Determinism to Indeterminacy

In the following section, a framework for the levels of uncertainty will be introduced, along with a guide on how to define the level of uncertainty a project, program or portfolio may face. This can be applied to activities, to decisions and to managerial practices, etc.

The levels of uncertainty is a way of describing the probability of uncertainties’ states within a given system (Bevan 2022). Usually, the levels are between determinacy and indeterminacy with intermediate states between the two outer cases (Walker et. al., 2003), but there have been changes to this original description over time, such as describing the levels as a magnitude of uncertainty, e.g. one’s ability to make judgements, qualitative uncertainty, e.g. expert opinions and ambiguities between people (Warmink et. al., 2010). The original distinction between levels of uncertainties was proposed by Knight (1921) between risk (measurable and quantifiable) and uncertainty (unmeasurable).

In more recent years, Walker et. al. (2003) developed one of the more used distinctions of the different levels of uncertainties (Bevan 2022), which gained the addition of qualitative uncertainty as proposed by Warmink et. al. (2010). The framework of level of uncertainty is however not completely agreed upon, as other distinctions of the levels are also prevalent (See Kwakkel et. al. (2010) and Oehmen & Kwakkel (2023)). The list below shows the different levels described by Walker et. al. (2003) and Warmink et. al. (2010) and moves from determinacy to total ignorance (the two ends of the spectrum):

Figure 1. Derived from Warmink (2010)

1. Statistical uncertainty. Any uncertainty that can be expressed in statistical terms, e.g. sampling error, measurement uncertainty with data, inaccuracy, etc. Statistical uncertainty is as close to determinism as uncertainty can get, and mainly affecting risk.
2. Scenario uncertainty. A scenario is a plausible description of how a system may act in the future, and act as an indication of a possibility. With a scenario uncertainty, mechanisms leading to specific outcomes are not well understood. It can manifest itself from e.g. different underlying assumptions from the modelers, uncertainty of driving forces in a given system, and uncertainty about the levels of changes.
3. Qualitative uncertainty – Qualitative uncertainty is defined as any uncertainty that cannot be expressed in terms of nominally measurable values. This level of uncertainty comprises opinions of experts, linguistic probabilities, and ambiguities between people.
4. Recognized ignorance. Uncertainty about underlying mechanisms and relationships. Scenarios are difficult to create due to this weak understanding of the system. Ignorance can be either reducible or irreducible (i.e., research can either reduce recognized ignorance or it will not be able to help). However, as Enserink et. al. (2013) writes: “… almost paradoxically, uncertainty is not merely a lack of knowledge, actually new knowledge can increase the uncertainty.”
5. Total ignorance. We do not know what we do not know, and surprises will arise.

Understanding the level of uncertainty within projects, programs or portfolios is important as it describes how well we understand the system that we are part of. Uncertainties are however often reduced, which can have negative consequences on the decisions and management processes. As Enserink et. al. (2013) writes:

“Uncertainty is something that makes decision-making difficult, and therefore it is in their [decision makers’] interest to see it reduced to probabilities and cost benefit ratios”. (Enserink et. al. 2013)

By reducing uncertainties like scenario uncertainty or qualitative uncertainty to statistical uncertainty, the predictions and knowledge of the behaviour in the project/program/portfolio system goes from being uncertain to perceived factual. Decisions made on changed levels of uncertainty are often poor. (Enserink et. al., 2013).

Choosing the level of uncertainty can be done following the questions proposed by Warmink et. al. (2010) as seen in figure 1.

[edit] Uncertainties in projects, program, and portfolio management

The body of work by the project management institute (PMI) offers several techniques, approaches and guidance for dealing with risk (The Standard for Project Management, 2021, The Standard for Program Management, 2017, The Standard for Portfolio Management, 2017). Uncertainty is however less specified. In the following we will quickly sum up the general approach to risk and uncertainty management covered in these standards along with a view from literature on uncertainty practices.

[edit] Projects

At the project level, uncertainty is expressed through the “risk threshold”. Here, uncertainties can blur probabilities and make expressions of risk near impossible to trust. Risk response strategies may be difficult to apply if uncertainties are at a high level. Applying levels of uncertainties to activities/decision highly influenced by uncertainties can aid to focus further exploration to make the uncertainty more transparent and make more guided decisions. Along with this, it is advised to implement adaptability and resiliency in projects, as uncertainties will have less of an impact this way. (The Standard for Project Management, 2021) When considering if an uncertainty should be pursued to be reduced/made transparent to make more guided decisions, it is very important to consider resource management aspects such as time, staff and money (Warmink et. al. 2010).

[edit] Programs

At the program level, much like at the project level, uncertainty is also expressed through “risk thresholds” that can be used for monitoring if program activities or projects are e.g., over schedule. Following this, appropriate measures can be taken (The Standard for Program Management, 2017). However, the interdependence between projects and the longer timeframe makes deeper uncertainties have a much bigger impact than in projects. Making programs resilient to uncertainties and changes (see section “Selection of appropriate analytical technique”) will increase the likelihood of realizing benefits from the program. As programs are, to a degree, projecting what will be deemed good for the organization, society or from an engineering standpoint, these are subject of uncertainty due to the time frame (Enserink et. al. 2013). Making plural and conditional guided choices based on the level of uncertainty faced in the program can be a determining factor of success (Stirling, 2010). For guidance on how to do this, see section “Application”.

[edit] Portfolios

Management of risk and uncertainties has the objective of meeting the value seeked through the portfolio while maintaining an agreed-upon level of confidence/risk in the portfolio. Uncertainty is greatest at the portfolio level as the impact of uncontrolled variables can have large implications for the organization. Risk perception is an important term here, as uncertainty can have a large impact on how risk is perceived and acted upon. It is therefore important to create an internal knowledge of uncertainty and the degree of risk that the organization is willing to embrace. (The Standard for Portfolio Management, 2017). Applying levels of uncertainties at the portfolio level may allow the decision maker to create more precise perceptions of the risks in the portfolio and make better decisions by knowing where the uncertainty arises from. Prediction is a key asset in portfolio management, and making transparent the level of uncertainty surrounding the portfolio will make decisions more robust. Further, the standards call for defining a risk threshold internally, but it is also adviced to embrace uncertainty, as the risk-thresholds are less likely to perceive reality under conditions of deep uncertainty (Enserink et. al., 2013, Stirling, 2010).

[edit] Application

The application of levels of uncertainty are within several domains. Below, this article will attempt to cover some of the essential aspects of the application of levels of uncertainties.

[edit] Selection of appropriate analytical technique

Selecting an appropriate analysis in uncertain situations is difficult, and often times a wrong approach can lead to faulty decisions. Based on the level of uncertainty project dimensions may be exposed to, the project manager can make better analysis of the parts of the system based on the level of uncertainty attained to that specific part. Stirling (2010) advices decision makers to move away from a pure risk driven decision making, as this will not allow for surprises to factor in. He also highlights that quantitative and qualitative methods when used to analyze purely risk are too ambiguous and will not present a whole picture of the consequences of actions. Here, the advice is to move beyond risk to uncertainty, ambiguity and ignorance, and use both quantitative and qualitative methods. By applying uncertainty levels to likelihood of events along with possibilities, it is possible to decide on an appropriate means of uncertainty analysis (Stirling, 2010) (See figure 2).

Figure 2. Adapted from Stirling 2010, “Keep it Complex”, originally published in Nature

The matrix can be used in the following way:

1. Map uncertainties about possibilities and probabilities onto matrix
2. Decide on quadrant
3. Choose at minimum 2 methods to make the uncertainty more transparent

Managers can equally apply the uncertainty, ambiguity and ignorance methods in projects, programs and portfolio management when there is not consensus internally over the framing of possible options, contexts, outcomes, benefits or harms. When this occurs, review the degree of uncertainty on the knowledge of outcomes and possibilities, and choose both quantitative and qualitative methods to review possible actions/decisions. If the degree of uncertainty is complex due to the system the project, program or portfolio is part of, consider using several methods from both uncertainty, ambiguity and ignorance. Managers should move away from singular guidance on decisions to plural and conditional guidance, which should follow the amount of uncertainty there is faced in the project. (Stirling, 2010)

Oehmen & Kwakkel (2023) goes into further detail about processes to perform under uncertainties are under incommensurate values, meaning that there is not a consensus on a definitive objective to improve upon.

Risk Approach

When uncertainties are at a risk level, it is especially important to consider the following three approaches:

1. Risk related public engagement – Public engagement can leverage collective knowledge, create buy-in and ownership, but can create anger and mistrust if not meeting quality standards and expectations.
2. Risk communication – Responsible risk communication must both address the subject at hand and improve discourse by using appropriate communication methods. For factors influencing this, see Oehmen & Kwakkel (2023) and guidance see section “Communicating Uncertainties”.
3. Social movement theory – A theory to explain when and why people move from being complacent to taking action.

Uncertainty approach

Approaches to robust uncertainty management practices generally follow 3 steps according to Oehmen & Kwakkel (2023):

1. Exploratory scenario thinking – explore multiple scenarios and their consequences to the system
2. Adaptive planning – plans are adaptable from the outset of the project, program or portfolio in response to future changes.
3. Decision aiding – Having multiple stakeholders agree requires facilitation and a learning process.

Ignorance approach

Addressing ignorance is especially important at the program and portfolio level, as future operating scenarios, technical limitations or similar may not be predictable. A main strategy for addressing ignorance is creating resilience (a systems capability of managing unforeseen changes) within the projects, programs and portfolios. There are two parts to resilience (as described by Oehmen & Kwakkel (2023)):

1. Socio-organizational resilience – This can be further deconstructed as; (1) Individual and team resilience; (2) Project and organisational resilience; and (3) Supply chain and industry resilience.
2. Technical/Engineering resilience – [The focus is on maintaining defined functions, avoiding discontinuities, and rapidly recovering functionality to a pre-disruption state.]

The Standard for Project Management (2021) guides managers on ways of increasing resilience. Examples of ways of this are (Taken from The Standard for Project Management (2021)):

• Short feedback loops
• Continuous learning and improvement
• Ability and willingness to anticipate multiple scenarios and prepare for multiple eventualities
• And more (See The Standard for Project Management (2021) for more examples)

Often in managing ignorance, the “precautionary principle” will also be used, which is similar to resilience but demands action. For more information on the risk, uncertainty and ignorance approaches, see Oehmen & Kwakkel (2023).

[edit] Classification of expressed uncertainty-descriptions

The level of uncertainty in reporting of project, program or portfolio is often not described thoroughly, or are not understood and the uncertainty level of a project, program or portfolio is interchanged between levels. This can lead to wrong interpretations of findings and analysis, leading further to poor decisions. When presented with uncertainties by project workers, managers, experts or similar, it can be very helpful to be able to evaluate if uncertainties and their levels are thoroughly discussed and considered. Failure to do so can have negative consequences on the project, program or portfolio management (Enserink et. al. 2013). Tenney et. al. (2006) proposes a framework for managers to do so:

With this categorization it is possible for the manager to set requirements for documentation when receiving information prior to making decisions. For example, a manager may require documentation prior to decisions to express uncertainties, at least through statements (such as probable, improbable, etc.) but not necessarily in the wording of the specific levels of uncertainty. This way, decisions are made more transparent and aware, and it is possible for a manager to set a baseline of uncertainty reflection (Tenney et. al. 2006). Managers can also require multiple perspectives to be highlighted (Enserink et. al., 2013).

[edit] Levels of uncertainty in future studies: What can be stated in practice

Almost all projects will use future studies to a degree, be it simulation, modelling, scenario-planning or similar. They are however often wrongly interpreted which can lead to poor management choices. Enserink (2013) summarizes good practice when considering future studies:

• Do not mix levels of uncertainty or reduce them inexplicably. This will lead to misinterpretations of the output and reduce uncertainties to risks. This may e.g., lead to scenarios to be interpreted as facts.
• Include users (managers or directors) in the production of the future studies in a co-design process. This way, miscommunication of uncertainty is reduced as they are part of defining it.
• In the creation of the future studies, involve non-experts.
• When managing uncertainty, managers should move away from a reduction mindset (i.e. ignoring or downgrading uncertainty levels) and towards embracing uncertainty. This could be done by making decisions transparent to the uncertainty prevailing them.
• The different perceptions on uncertainty should be outlined and acknowledged in models, predictions and projections. See section “Classification of expressed uncertainty-descriptions” for potential reporting requirements.

[edit] Communicating uncertainties

Managing uncertainties in projects, programs and portfolios is not just about the prediction quality, but also about the communication and presentation of uncertainties. Decision makers must be made aware of potential impacts hidden within the uncertainties before making decisions (Tenney et. al. 2006). When communicating uncertainties and their levels, use the following questions and prompts to guide your strategy (Adapted from van der Bles et. al. (2019)):

• What is the uncertainty about? (Underlying hypothesis, specific numbers, categorical facts, etc.). This is also the source of uncertainty. When describing the source, consider if there are multiple or if there are relations between the sources.
• Why is there uncertainty? (Natural variation, measuring difficulties, limited knowledge, disagreement between experts, etc.).
• Projects will have both uncertainties directly about specific aspects of the project (e.g. a probabilistic spread, a range of scenarios, etc.?) as well as the evidence of the quality of the levels of uncertainties (e.g. why have we categorized something as a qualitative uncertainty?). These should be communicated separately.
• Choose an expression that fits the level of uncertainty you have identified. This could be verbal, via media, written, or a mixture of these methods. Importantly is to consider your audience, your relationship to that audience (an vice versa), and the effect you want to impose through your communication.
• Keep the level of uncertainty separate from the magnitude of evidence about the specific aspects.
• Test the effect of your communication with your audience before presenting. Make sure that the target audience will be able to understand your descriptions and visualizations.

[edit] Limitations

• Having levels as rank-orderable is only a helpful metaphor, there are several inconsistencies within the idea. An example being that the “line” between indeterminacy and full determinacy is non-linear, although it may seem that way. The transition between levels are somewhat unclear, and as such should not be used as an indicator of knowledge within a project, program, or portfolio (Bevan, 2022).
• We may only be able to partially describe state-spaces, meaning that while we can attribute certain levels to certain parts of a system, other parts may have more indeterminate/determinate qualities that can affect the first described ones. This also means that we can feel a false sense of security in having some probabilitistic model of some events, leading to unexpected events with unknown probabilities having a much larger impact. (Bevan, 2022)
• While a transition towards plural and conditional guidance when making decisions as a manager is an improvement of singular advice, the deep complexity of uncertainty and group dynamics will still be factors that cannot be escaped. The transition will however make decisions and approaches to projects, programs and portfolios more accountable and explicit towards these factors of surprise. (Stirling, 2010)
• Considering levels of uncertainty in projects, programs and portfolios is not standard practice. Extending ones management practices to include this will however increase the transparency of activities and increase the understanding of the system that is managed within. It can also aid in realizing what “standard of proof” is necessary to make decisions and embracing uncertainty to a much larger extent. This is a key step in creating resilient projects, programs and portfolios. (Enserink et. al., 2013, The Standard for Project Management, 2021, Stirling, 2010).

[edit] References

1. Bevan, L. D. (2022). The ambiguities of uncertainty: A review of uncertainty frameworks relevant to the assessment of environmental change. Futures, 137, 102919. https://doi.org/10.1016/j.futures.2022.102919
2. Enserink, B., Kwakkel, J. H., & Veenman, S. (2013). Coping with uncertainty in climate policy making: (Mis)understanding scenario studies. Futures, 53, 1–12. https://doi.org/10.1016/j.futures.2013.09.006
3. Knight, F. H. (1922). Knight's Risk, Uncertainty and Profit, Risk, Uncerainty and Profit. Quarterly Journal of Economics, 36(4), 682. https://doi.org/10.2307/1884757
4. Kwakkel, J. H., Walker, W. E., & Marchau, V. A. W. J. (2010). Classifying and communicating uncertainties in model-based policy analysis. International Journal of Technology, Policy and Management, 10(4), 299–315. https://doi.org/10.1504/IJTPM.2010.036918
5. Oehmen, J., & Kwakkel, J. (2023). Risk, Uncertainty, and Ignorance in Engineering Systems Design. Handbook of Engineering Systems Design, 1–32. https://doi.org/10.1007/978-3-030-46054-9_10-2
6. The Standard for Portfolio Management — Fourth Edition, Standard for Portfolio Management. (2017). Standard for Portfolio Management — Fourth Edition. Project Management Institute.
7. The Standard for Program Management — Fourth Edition, Standard for Program Management. (2017). Standard for Program Management — Fourth Edition. Project Management Institute.
8. The standard for project management. (2021). A Guide To the Project Management Body of Knowledge (pmbok® Guide) – Seventh Edition and the Standard for Project Management (english) (pp. xxvi, 67, 274 Seiten (unknown). Project Management Institute, Inc.
9. Stirling, A. (2010). Keep it complex. Nature, 468(7327), 1029–1031. https://doi.org/10.1038/4681029a
10. Tennøy, A., Kværner, J., & Gjerstad, K. I. (2006). Uncertainty in environmental impact assessment predictions: The need for better communication and more transparency. Impact Assessment and Project Appraisal, 24(1), 45–56. https://doi.org/10.3152/147154606781765345
11. Van Der Bles, A. M., Van Der Linden, S., Freeman, A. L. J., Mitchell, J., Galvao, A. B., Zaval, L., & Spiegelhalter, D. J. (2019). Communicating uncertainty about facts, numbers and science. Royal Society Open Science, 6(5), 181870. https://doi.org/10.1098/rsos.181870
12. Walker, W. E., Harremoës, P., Romans, J., van der Sluus, J. P., van Asselt, M. B. A., Janssen, P., & Krauss, M. K. V. (2003). Defining uncertainty. A conceptual basis for uncertainty management in model-based decision support. Integrated Assessment, 4(1), 5–17. https://doi.org/10.1076/iaij.4.1.5.16466
13. Warmink, J. J., Janssen, J. A. E. B., Booij, M. J., & Krol, M. S. (2010). Identification and classification of uncertainties in the application of environmental models. Environmental Modelling and Software, 25(12), 1518–1527. https://doi.org/10.1016/j.envsoft.2010.04.011