Optimism bias, Strategic Misinterpretation and Reference Class Forecasting (RCF)

Abstract

Most major projects around the world are facing problems regarding planning and underestimating the costs of a project in the initial phases. These setbacks can cost an enormous sum of money and cause major setbacks of the planned schedule. The initial planning and make a reliable budget are some of the main organizational managerial skills for a project manager. .

But unfortunately, it is today seen that most projects will not fulfill the initial plans and will not meet the initial budget goals. It is logically a utopian idea of predicting the unpredictable (known unknowns) or even the things that are impossible to foresee. And that is why we, as project managers need tools and learn from the past and others.

The Danish professor Bent Flyvbjerg did research into cost overrun of major projects. By sub-dividing the broader aspect of this into the two topics: Optimism Bias and Strategic Misrepresentation Flyvbjerg explored and explained more about the cost and benefit shortfall of major projects. And through further investigation, Flyvbjerg came up with a possible solution: the use of Reference Class Forecasting. The Reference Class Forecasting approach provides a more general overview and “is beneficial for non-routine projects” .

Through reading this article, an explanation of the strategies and ideas mentioned above will be discussed. Furthermore, the idea of Reference Class Forecasting in larger projects will be covered as well as the studies' limitations. Examples of causal factors for cost overruns will be provided followed up by specific tools for project managers.

Word definitions

Optimism Bias

The term optimism bias was invented by the Nobel prize winner Daniel Kahneman describing the idea that most people consider themselves less likely to experience something unpleasant. This leads to the effect of overconfidence in personal judgments, leading to project managers misjudging the outcome of their decisions. Risks tend to be seen lower and own capabilities better although previous experience shows the opposite. This can result in project managers undertaking projects with too optimistic expectations about budget, risks and the project scope, which will most likely not be met. .

Strategic Misrepresentation

You have a great idea for a book, and you have found a publisher who is willing to pay. However, he needs to know more about the time perspective. "When can I have the first manuscript? Can you have it done by the end of next month?", he asks. You lower your eyes and gulp. "Of course, no problem" you answer. You have never managed that kind of timeline before, but you are aware that if you tell the truth, the publisher will not go with your idea. You do not feel like you just lied - and in official social terms, you did not, you performed the behavior of Strategic Misrepresentation and sorted out some of the truth. In a larger project, this Strategic Misrepresentation can result in underestimating costs, pre-determining and quite likely also overestimating the potential clients' benefits. Applying additional pressure and strain on individuals through manipulation, competing for scarce funds or jockeying for a position all qualify for the same over-arching category. . Note that Strategic Misrepresentation is a form of bias as well but is used more intentional, and more likely to be a form a technique. When project planners present their cases, they often brighten the numbers regarding the risk and the benefits of their projects. Regarding Flyvbjerg, they are deliberately deceiving the decision makers, since the projects that look the best on paper will be approved. One of the things that can make a project more vulnerable to Strategic Misinterpretation is the end date is a few years down the road . No one knows more about large-scale projects than professor Bent Flyvbjerg. Why are cost and schedule overruns so frequent? Because it is not certain that it is the best offer or project that wins. It is the project which looks best on the paper that wins.

Reference Class Forecasting

Go back to the previous example with the book. How would you decide how much time you actually needed? Reference Class Forecasting (RCF) could be a method to determine so. The problem can occur when to determine which class to place your project in. Would you compare with all books written? All books within your subject? Books at the same educational level as yours? This is the idea and challenge of RCF. RCF is a method of looking to future events by taking relatable situations and their previous outcomes. This approach aims to give a much less biased view on a specific event. The advantages and downsides of Reference Class Forecasting will be further clarified in this article.

Background

Risk contingency " is expected to be expended [and is] an amount added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and that experience shows will likely result, in aggregate, in additional costs". . Various methods are today used to quantify the risk contingency. Reference Class Forecasting is a possible approach, but as it is today the method “conventional contingency approach” is the most used. Estimating the costs of the budget is a different process depending on the stage of which the project is at. Different level of uncertainties is found at each stage of the project - often with the highest risks and uncertainties in the beginning. . As the project evolves estimation accuracies will increase. more information about the project's scope, design, and specifications become available as the project progresses through its lifecycle. This progression enables the estimation team to more accurately estimate the quantity and price of material and resources. As a result, generally more risk contingency is applied at the earlier stages than in later stages. With the use of Reference Class Forecasting, it would be possible to foresee these risk contingency more accurate in the initial phases of the project.

Project cost performance and causal factors for cost overruns

Cost overruns according to litterature

Author, Year	Sample size	Type of project	Mean [%]	Standard deviation [%]
Odeck, 2004	610	Road Infrastructure	7,88	29,20
Flyvbjerg, Holm, & Buhl, 2004	58	Rail	45	38
Flyvbjerg, Holm, & Buhl, 2004	33	Bridges and Tunnels	34	62
Flyvbjerg, Holm, & Buhl, 2004	167	Roads	20	30
Fouracre, Allport, & Thomson, 1990	21	Metro Projects	45	--

How RCF Works

Bent Flyvbjerg’s research identifies two basic areas, Optimistic and Strategic Misrepresentation, as major reasons for cost estimate errors. When optimistic and strategic mismanagement biases occur, the understatement of project costs can be scandalous. Flyvbjerg argues that RCF can be used for a more accurate measure of the outcome. RCF serves remarkably well to identify and restrict estimate bias. RCF is impersonal and removes moral and emotional issues. Mitigating the effect of optimistic and strategic biases is imperative and RCF can be a significant part of the solution. “Make RCF Mandatory” is the advice given by Flyvbjerg over a decade ago. RCF provides a foundation for more realistic contingency funding. RCF provides an opportunity to analyze project risk and to assign realistic budget and management reserves. The more people tied to the estimate, the greater the chance of avoiding bias estimating errors. Require contractors to be accountable to their estimate. Create incentives and penalties. Performance goals and responsibilities need to be clear, for example with the use of SMART goals. The project leader needs to have the power to fire contractors who fail to perform. Develop cooperative data banks that classify project by type, size, cost, and other criteria. Make these actual cost and schedule data available to be used to evaluate “inside” estimates of similar projects. Professional groups, such as engineers, lawyers, Project Management Institute, American Planning Association, and Forrester Research, could develop such data banks for other infrastructure types of projects. The six areas suggested for evaluation and improvement would vary depending on the project. .

Application

Tools for project managers

Use other biases to limit the effects

It is possible to use other biases to limit the effects. This bias is particularly important for decision-makers creating health or safety products, where the dangers of being overly optimistic can lead to dreadful outcomes. There are two researched ways of reducing the Optimism Bias by the use of other biases :

1. Make past bad events more easily retrievable from one’s memory (Availability Heuristic)
2. Highlight losses that are likely to occur because of these bad events (Loss Aversion)

Bringing negative events to our minds before we have the option to act can be a great technique to change people's behavior. The aim is to make the negative effects of a certain action clear to the individual and offer a clear. It is important to negate the potentially huge costs of the positivity bias when estimating the expected time to complete a task or project. When making negative events obvious will make for example managers less likely to engage in undesirable acts causing this bad event. The aim is to make the negative effects of a certain action clear to the individual and offer a clear, safer alternative. It could be argued that Reference Class Forecasting contributes to this way of minimizing the risk of optimism bias. By placing a project within a “class” every failure that frequently happens to these projects will occur.

Three-step approach

With use of RCF method it is suggested to follow a three-step approach. When using this approach the forecaster is asked to take an more “outside view” and his psychological biases are minimized. Nevertheless, the setting of the reference class is subjective and therefore prone to new biases, which will be discussed further in the section “Limitations”.

When increasing the “outside view” the accuracy of the forecast can be increased, which is the primarily aim of the RCF method. RCF collects differences from the initial budget and actual cost of similar completed projects. The collective empirical data will be collected and afterwards be arranged statistically to show the differences to illustrate the overrun percent. The three most significant benefits of this method are:

1. it is a real-world check on estimations.
2. more reliable budget based on previous experience.
3. gives accountability pride of place.

In order to minimize the optimism bias and produce a more realistic forecast, the RCF method requires the user to take an outside view comparing the project at hand to similar projects. The method comprises three steps

• Identifying relevant reference class.
• Establishing a probability distribution for the chosen class.
• Benchmarking the specific project to the reference class distribution.
  

  

  Figure 1, Required uplift as a function of maximum acceptable level of risk for cost overruns. Content:

The first step is to identify a reference class of similar, historical projects. This group of projects should be large enough to be statistically meaningful, but small enough to be comparable to the project at hand. The second step is then to create a probability distribution for certain variables of the projects like on-time delivery, cost overrun, or demand forecast mistakes. This can only be done with access to credible, empirical data of a statistically significant number of projects. Lastly, the specific project needs to be compared to the reference class in regard to the chosen variables for establishing the most probable outcome of it.

Uplift

In collaboration with the Danish consultancy COWI, Flyvbjerg developed guidelines for project planners to remediate their estimations. The application for a fixed link project will serve as an example. Figure 1 shows the required uplift of the initially estimated budget regarding the acceptable chance of a cost overrun. Such a tradeoff is the outcome of the second step of the RCF method. If a planner now wants to correct his estimations about budgeting, he can calculate the required uplift regarding a possible risk of a cost overrun. As an example, when the maximum acceptable chance for a cost overrun is set to be 20 %, the corresponding uplift of the budget is around 50 % in this case, illustrated in Figure 1. It has to be mentioned that the RCF method comprises simple looking steps on the first view. Nevertheless, the setting of the reference class is a difficult task. Gathering empirical data from the business sector the specific project is embedded in, selecting only comparable projects and creating the probability distribution are complex tasks.

Limitations of the Reference Class Forecast method

Like every other method, the RCF method is facing limitations. Although Reference Class Forecasting (RCF) can largely be quite an accurate and effective way to limit the bias in a situation, it is less likely that it ever will be possible to limit all the possible things that can go wrong and estimate a perfectly correct budget and time schedule.

A better illustration of this statement is an example of colossal-sized projects, like the Olympics. In terms of the management of such an event, with relation to RCF, the issue for an organization undertaking such as high-status event is that it is almost a certainty that it will the companies first time in charge of such a project. With never large projects never done before. There has not been an Olympic game that has met or been even close to its initial budget with an average of 51% cost overrun for the hosting country throughout the recent history of the event) . With this knowledge, although the Olympics can be considered a very niche event, it can be demonstrated that even with RCF in place, tasks with such significance make it an increasingly difficult task to refer to RCF as it is a first-time event for the country and organizations involved.

Flyvbjerg mentions that the host country has no idea of the actual position the country will be in at the time of the games. For Brazil in 2016, it was seen that the country was in a deep crisis, and things have not been that bad in a long time. Also, corruption had a huge impact on the country and the oil industry at the time. These unique events are impossible (or at least very tough) to incorporate in future Olympics. There seem to be three major issues limiting the application of the RCF method. These are:

• Probabilistic and the definition of frequentism
• The choice of reference class
• Acquisition of data and design of a database

Frequentism

The first one is more a philosophical problem and contains subjective perspectives as well. In the world of probabilistic frequentism defines the probability of an event as the measure of chance that a certain event will be the result of an experiment. The first problem arises by thinking about the total number of possible outcomes of the experiment. Is it possible for a person with limited knowledge to foresee every single, possible outcome of an experiment? Hájek argues that this is not the case. Reichenbach stated: "If we are asked to find the probability holding for an individual future event, we must first incorporate the case in a suitable reference class. An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class"" .

From this statement, it becomes clear that the chosen reference class for a project is highly important for the outcome and for the use and sake of the RCF method. Changing the reference class, will change the probability distribution and thus the whole outcome of the method. This may result than in return in false estimations or at least in lower accuracy than may be possible.

Subjectivity of reference classes

Choosing the right reference class is the main task of using the tool Reference Class Forecasting. When the forecaster is doing so, how certain can one be of choosing the right one, and not be biased? The aim of the RCF method is to overcome personal biases and rely on statistics. When using the RCF method is a certainty to use a reference class that is broad enough to be statistically meaningful and at the same time small enough to be comparable. Those requirements can seem very unclear and in the worst case leading to different, subjective interpretations. When is the class for example too small? How are projects comparable when we keep in mind that every project is created with its own particular aim and timeframe within its own environment. All in all, the setting of the reference class is exposed to a high risk of the subjectivity of the forecaster.

Design of a database

The last problem with the RCF in this article is about gathering the empirical data. In order to use the RCF method, a sufficient database of projects would be necessary. The main challenge for applying the RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information. Gathering data from finished projects that are similar enough to be comparable to the specific project at hand might be a pitfall. Acquiring this data might turn out to be a challenge. If not enough data is collected to be statistically meaningful the method cannot be applied.

These issues with the RCF method may not hold up to 100% for transportation infrastructure projects since they seem to be more alike than other kinds of projects and an accurate database has been created by Flyvbjerg to be accessed by project planners. But the core problems addressed still remain: When is a project really comparable and how to set the reference class?

Times change and the focus of the different societies changes. Would a 100-year-old project be relatable for anyone today? The cost of materials and workers change. Today, many of the countries in the western part of the world pay a lot of attention to climate change and greener solutions. These things cost as well and are again a change in the way we design and build our larger structures.

Annotated bibliography

Bent Flyvbjerg: ”Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice”

An exposition of the concepts of Optimism Bias and how to act to the potential consequences. Strategic Misrepresentation is in this article linked to cost overruns. Finally, Reference Class Forecasting is represented as a solution to the two forms of bias.

Daniel Kahnemann: 'Thinking fast and slow'

Kahnemann focuses on how cognitive biases, especially optimism bias, can affect the decision-making process. The book focuses on human behavior and is based on recent years of research. Throughout the book, Kahnemann set up real-life examples from for example the court and the judges as the decision makers. Liu, L., Wehbe, G., & Sisovic, J. (2010). The accuracy of hybrid estimating approaches? Case study of an Australian state road & traffic authority. Paper presented at PMI® Research Conference: Defining the Future of Project Management, Washington, DC. Newtown Square, PA: Project Management Institute. The article represents a new possible solution with the use of a hybrid approach. The hybrid will consist of parts from Reference Class Forecasting (RCF) and the conventional fixed contingency approach. The study was used to understand alternatives to RCF.

Clifford Gray: 'Revisit of Reference Class Forecasting (RCF): Estimating Costs of Infrastructure Projects'

The article represents the key managerial actions supporting the use of Reference Class Forecasting. C. Gray provides suggestions on how to implement Reference Class Forecasting.

Dan Benţa, Lucia Rusu, and Marius Podean: 'Successful Implemented Theories For Reference Class Forecasting in Industrial Field'

General issues regarding risk management are represented. Benta illustrates different theories regarding the Reference Class Forecasting e.g. “three-step approach”

Hans Reichenbach: “The theory of probability”

Reichenbach introduces the use of the theory behind probability. He explains the concepts of probability with the use of examples of calculations. Furthermore, the limitations of the theory of probability are presented.

The British Department for Transport, “Procedures for Dealing with Optimism Bias in Transport - Planning Guidance Document”

The paper investigates the potential of using a guide in future infrastructure projects. The theories leading to this idea are provided. Instructions are specific and explained for a specific reference project within the area of infrastructure. The use of the uplift method is explained, and real-world examples are given on how to use those uplifts to minimize the risk of optimism bias.

John Venn: “The logic of chance”

John Venn was thinking about the fundamental problems of the theory of probability. He introduced the main critics and arguments why the so-called frequentism has its flaws in the application in the real world outside of math. Since the RCF method is based on probability, this book provides a basis for showing the limitations of the RCF based on its very fundamentals - probability distribution

Bent Flyvbjerg: “Policy and planning for large-infrastructure projects: problems, causes, and cures”

In this paper, Flyvbjerg identifies the main reasons for false estimations in infrastructure projects and the resulting disadvantages for the projects. Having this explained, he continues by providing insights to possible cures for those false estimations.

Alan Hájek: “The Reference Class Problem is Your Problem Too”

Hájek takes a philosophical and mathematical perspective on probability and underlying assumptions. He presents the so-called reference class problems from various perspectives explaining the effects these problems have on the outcome of a probability calculation. He then gives examples for many different interpretations about probability, e.g. frequentism, logical or subjectivism and their view on the reference class problem.