Optimism bias, Strategic Misinterpretation and Reference Class Forecasting (RCF)

Developed by Mie Cuhre Anker

Abstract

Most major projects around the world are facing problems regarding planning and underestimating the costs of a project in the initial phases. These setbacks can cost an enormous sum of money and cause major setbacks of the planned schedule. The initial planning and make a reliable budget are some of the main organizational managerial skills for a project manager. .

But unfortunately, it is today seen that most projects will not fulfill the initial plans and will not meet the initial budget goals. It is logically a utopian idea of predicting the unpredictable (known unknowns) or even the things that are impossible to foresee. And that is why we, as project managers need tools and learn from the past and others.

The Danish professor Bent Flyvbjerg did research into the cost overrun of major projects. By sub-dividing, the broader aspect of this into the two topics: Optimism Bias and Strategic Misrepresentation Flyvbjerg explored and explained more about the cost and benefit shortfall of major projects. And through further investigation, Flyvbjerg came up with a possible solution: the use of Reference Class Forecasting. The Reference Class Forecasting approach provides a more general overview and “is beneficial for non-routine projects” .

Through reading this article, an explanation of the strategies and ideas mentioned above will be discussed. Furthermore, the idea of Reference Class Forecasting in larger projects will be covered as well as the studies' limitations. Examples of causal factors for cost overruns will be provided followed up by specific tools for project managers.

Word definitions

Optimism Bias

The term optimism bias was invented by the Nobel prize winner Daniel Kahneman describing the idea that most people consider themselves less likely to experience something unpleasant. This leads to the effect of overconfidence in personal judgments, leading to project managers misjudging the outcome of their decisions. Risks tend to be seen lower and own capabilities better although previous experience shows the opposite. This can result in project managers undertaking projects with too optimistic expectations about budget, risks and the project scope, which will most likely not be met. .

Strategic Misrepresentation

You have a great idea for a book, and you have found a publisher who is willing to pay. However, he needs to know more about the time perspective. "When can I have the first manuscript? Can you have it done by the end of next month?", he asks. You lower your eyes and gulp. "Of course, no problem" you answer. You have never managed that kind of timeline before, but you are aware that if you tell the truth, the publisher will not go with your idea. You do not feel like you just lied - and in official social terms, you did not, you performed the behavior of Strategic Misrepresentation and sorted out some of the truth. In a larger project, this Strategic Misrepresentation can result in underestimating costs, pre-determining and quite likely also overestimating the potential clients' benefits. Applying additional pressure and strain on individuals through manipulation, competing for scarce funds or jockeying for a position all qualify for the same over-arching category. . Note that Strategic Misrepresentation is a form of bias as well but is used more intentional, and more likely to be a form a technique. When project planners present their cases, they often brighten the numbers regarding the risk and the benefits of their projects. Regarding Flyvbjerg, they are deliberately deceiving the decision-makers, since the projects that look the best on paper will be approved. One of the things that can make a project more vulnerable to Strategic Misinterpretation is the end date is a few years down the road . No one knows more about large-scale projects than professor Bent Flyvbjerg. Why are cost and schedule overruns so frequent? Because it is not certain that it is the best offer or project that wins. It is the project which looks best on the paper that wins.

Reference Class Forecasting

Go back to the previous example with the book. How would you decide how much time you actually needed? Reference Class Forecasting (RCF) could be a method to determine so. The problem can occur when to determine which class to place your project in. Would you compare with all the books written? All books within your subject? Books at the same educational level as yours? This is the idea and challenge of RCF. RCF is a method of looking at future events by taking relatable situations and their previous outcomes. This approach aims to give a much less biased view on a specific event. The advantages and downsides of Reference Class Forecasting will be further clarified in this article.

Background

Risk contingency " is expected to be expended [and is] an amount added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and that experience shows will likely result, in aggregate, in additional costs". . Various methods are today used to quantify the risk contingency. Reference Class Forecasting is a possible approach, but as it is today the method “conventional contingency approach” is the most used. Estimating the costs of the budget is a different process depending on the stage of which the project is at. Different level of uncertainties is found at each stage of the project - often with the highest risks and uncertainties in the beginning. . As the project evolves estimation accuracies will increase. more information about the project's scope, design, and specifications become available as the project progresses through its lifecycle. This progression enables the estimation team to more accurately estimate the quantity and price of material and resources. As a result, generally more risk contingency is applied at the earlier stages than in later stages. With the use of Reference Class Forecasting, it would be possible to foresee these risk contingency more accurately in the initial phases of the project.

Project cost performance and causal factors for cost overruns

For a long time, it has been a focus for many professors, companies, and governments to reduce the problems with project cost overruns. Especially, large infrastructure projects continue to be overwhelmed by delays and large cost overruns. Studies on project cost performance showed around the globe frequent of cost overruns. The collective evidence from studies on project cost performance conducted throughout the world shows the recurring pervasiveness of cost overruns, which for example Flyvbjerg was a part of. The phenomenon is not limited to any geographic location, which makes it more crucial to find a solution . Table 1 compiles the key statistics by studies examining the cost performance of various larger infrastructure projects. In Table 1 it is clear that all the projects exceed the initial budget. The table is a part of the even bigger table found in the article The accuracy of hybrid estimating approaches? by Li Liu (2010) . Table 1 shows statistics from 889 projects, with cost overruns estimated to 7.88 % up to 45 %. These statistics show that the cost overrun for infrastructure projects is substantial and that the consistency of cost estimates is far from satisfactory. Flyvbjerg states that cost estimates for large infrastructure projects are “highly and systematically misleading”. His study found that 86% of all examined projects (258 in total) exceeded their estimated cost with an average overrun of between 20% and 45%.

Table 1, Cost overruns according to literature, Data used from

Author, Year	Sample size	Type of project	Mean [%]	Standard deviation [%]
Odeck, 2004	610	Road Infrastructure	7,88	29,20
Flyvbjerg, Holm, & Buhl, 2004	58	Rail	45	38
Flyvbjerg, Holm, & Buhl, 2004	33	Bridges and Tunnels	34	62
Flyvbjerg, Holm, & Buhl, 2004	167	Roads	20	30
Fouracre, Allport, & Thomson, 1990	21	Metro Projects	45	--

These cost overruns are so severe that it becomes highly important to look further into how to prevent it or minimize the effects. To solve a problem it is a necessity to first look more in-depth with the why? . The above studies have identified a number of factors contributing to cost overrun of larger projects:

1. Poorly defined scope or significant scope changes that were not accounted in the base estimate could lead to significant overruns.
2. Incorrect quantities and unit rates for material and labor quantities could produce inadequate bases estimate and subsequently lead to overruns.
3. The cost estimates of projects are sometimes so complex and unique that the ability to learn from past mistakes is limited.
4. The time length between estimation and project completion, price fluctuation in construction costs tends to increase the difficulty of estimating and drives estimates to become very unreliable.

When designing a project in the initial stages it is so important to pay attention to these potentially high-risk pitfalls. This leads to the how? .

How RCF Works

Bent Flyvbjerg’s research identifies two basic areas, Optimistic and Strategic Misrepresentation, as major reasons for cost estimate errors. When optimistic and strategic mismanagement biases occur, the understatement of project costs can be scandalous. Flyvbjerg argues that RCF can be used for a more accurate measure of the outcome. RCF serves remarkably well to identify and restrict estimate bias. RCF is impersonal and removes moral and emotional issues. Mitigating the effect of optimistic and strategic biases is imperative and RCF can be a significant part of the solution. “Make RCF Mandatory” is the advice given by Flyvbjerg over a decade ago. RCF provides a foundation for more realistic contingency funding. RCF provides an opportunity to analyze project risk and to assign realistic budget and management reserves. The more people tied to the estimate, the greater the chance of avoiding bias estimating errors. Require contractors to be accountable for their estimates. Create incentives and penalties. Performance goals and responsibilities need to be clear, for example with the use of SMART goals. The project leader needs to have the power to fire contractors who fail to perform. Develop cooperative data banks that classify project by type, size, cost, and other criteria. Make these actual cost and schedule data available to be used to evaluate “inside” estimates of similar projects. Professional groups, such as engineers, lawyers, Project Management Institute, American Planning Association, and Forrester Research, could develop such data banks for other infrastructure types of projects. The six areas suggested for evaluation and improvement would vary depending on the project. .

Application

Tools for project managers

Use other biases to limit the effects

It is possible to use other biases to limit the effects. This bias is particularly important for decision-makers creating health or safety products, where the dangers of being overly optimistic can lead to dreadful outcomes. There are two researched ways of reducing the Optimism Bias by the use of other biases :

1. Make past bad events more easily retrievable from one’s memory (Availability Heuristic)
2. Highlight losses that are likely to occur because of these bad events (Loss Aversion)

Bringing negative events to our minds before we have the option to act can be a great technique to change people's behavior. The aim is to make the negative effects of a certain action clear to the individual and offer a clear. It is important to negate the potentially huge costs of the positivity bias when estimating the expected time to complete a task or project. When making negative events obvious will make for example managers less likely to engage in undesirable acts causing this bad event. The aim is to make the negative effects of a certain action clear to the individual and offer a clear, safer alternative. It could be argued that Reference Class Forecasting contributes to this way of minimizing the risk of optimism bias. By placing a project within a “class” every failure that frequently happens to these projects will occur.

Three-step approach

With the use of the RCF method, it is suggested to follow a three-step approach. When using this approach the forecaster is asked to take a more “outside view” and his psychological biases are minimized. Nevertheless, the setting of the reference class is subjective and therefore prone to new biases, which will be discussed further in the section “Limitations”.

When increasing the “outside view” the accuracy of the forecast can be increased, which is the primary aim of the RCF method. RCF collects differences from the initial budget and actual cost of similar completed projects. The collective empirical data will be collected and afterward be arranged statistically to show the differences to illustrate overrun percent. The three most significant benefits of this method are:

1. it is a real-world check on estimations.
2. more reliable budget based on previous experience.
3. gives accountability pride of place.

In order to minimize the optimism bias and produce a more realistic forecast, the RCF method requires the user to take an outside view comparing the project at hand to similar projects. The method comprises three steps

• Identifying relevant reference class.
• Establishing a probability distribution for the chosen class.
• Benchmarking the specific project to the reference class distribution.
  

  

  Figure 1, Required uplift as a function of the maximum acceptable level of risk for cost overruns. Content and data provided by

The first step is to identify a reference class of similar, historical projects. This group of projects should be large enough to be statistically meaningful, but small enough to be comparable to the project at hand. The second step is then to create a probability distribution for certain variables of the projects like on-time delivery, cost overrun, or demand forecast mistakes. This can only be done with access to credible, empirical data of a statistically significant number of projects. Lastly, the specific project needs to be compared to the reference class in regard to the chosen variables for establishing the most probable outcome of it.

Uplift

In collaboration with the Danish consultancy COWI, Flyvbjerg developed guidelines for project planners to remediate their estimations. The application for a fixed link project will serve as an example. Figure 1 shows the required uplift of the initially estimated budget regarding the acceptable chance of a cost overrun. Such a tradeoff is the outcome of the second step of the RCF method. If a planner now wants to correct his estimations about budgeting, he can calculate the required uplift regarding a possible risk of a cost overrun. As an example, when the maximum acceptable chance for a cost overrun is set to be 20 %, the corresponding uplift of the budget is around 50 % in this case, illustrated in Figure 1. It has to be mentioned that the RCF method comprises simple-looking steps on the first view. Nevertheless, the setting of the reference class is a difficult task. Gathering empirical data from the business sector the specific project is embedded in, selecting only comparable projects and creating the probability distribution are complex tasks.

Limitations of the Reference Class Forecast method

Like every other method, the RCF method is facing limitations. Although Reference Class Forecasting (RCF) can largely be quite an accurate and effective way to limit the bias in a situation, it is less likely that it ever will be possible to limit all the possible things that can go wrong and estimate a perfectly correct budget and time schedule.

There has not been an Olympic game that has met or been even close to its initial budget with an average of 51% cost overrun for the hosting country throughout the recent history of the event) . With this knowledge, although the Olympics can be considered a very niche event, it can be demonstrated that even with RCF in place, tasks with such significance make it an increasingly difficult task to refer to RCF as it is a first-time event for the country and organizations involved. Flyvbjerg mentions that the host country has no idea of the actual position the country will be in at the time of the games. For Brazil in 2016, it was seen that the country was in a deep crisis, and things have not been that bad in a long time. Also, corruption had a huge impact on the country and the oil industry at the time. These unique events are impossible (or at least very tough) to incorporate in future Olympics. There seem to be three major issues limiting the application of the RCF method. These are:

• Probabilistic and the definition of frequentism
• The choice of reference class
• Acquisition of data and design of a database

Frequentism

The first one is more a philosophical problem and contains subjective perspectives as well. In the world of probabilistic frequentism defines the probability of an event as the measure of the chance that a certain event will be the result of an experiment. The first problem arises by thinking about the total number of possible outcomes of the experiment. Is it possible for a person with limited knowledge to foresee every single, possible outcome of an experiment? Hájek argues that this is not the case. Reichenbach stated: "If we are asked to find the probability holding for an individual future event, we must first incorporate the case in a suitable reference class. An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class"" . From this statement, it becomes clear that the chosen reference class for a project is highly important for the outcome and for the use and sake of the RCF method. Changing the reference class, will change the probability distribution and thus the whole outcome of the method. This may result in return in false estimations or at least in lower accuracy than may be possible.

Subjectivity of reference classes

Choosing the right reference class is the main task of using the tool Reference Class Forecasting. When the forecaster is doing so, how certain can one be of choosing the right one, and not be biased? The aim of the RCF method is to overcome personal biases and rely on statistics. When using the RCF method is a certainty to use a reference class that is broad enough to be statistically meaningful and at the same time small enough to be comparable. Those requirements can seem very unclear and in the worst case leading to different, subjective interpretations. When is the class for example too small? How are projects comparable when we keep in mind that every project is created with its own particular aim and timeframe within its own environment. All in all, the setting of the reference class is exposed to a high risk of the subjectivity of the forecaster.

Design of a database

The last problem with the RCF in this article is about gathering the empirical data. In order to use the RCF method, a sufficient database of projects would be necessary. The main challenge for applying the RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information. Gathering data from finished projects that are similar enough to be comparable to the specific project at hand might be a pitfall. Acquiring this data might turn out to be a challenge. If not enough data is collected to be statistically meaningful the method cannot be applied. These issues with the RCF method may not hold up to 100% for transportation infrastructure projects since they seem to be more alike than other kinds of projects and an accurate database has been created by Flyvbjerg to be accessed by project planners. But the core problems addressed still remain: When is a project really comparable and how to set the reference class?

Times change and the focus of the different societies changes. Would a 100-year-old project be relatable for anyone today? The cost of materials and workers change. Today, many of the countries in the western part of the world pay a lot of attention to climate change and greener solutions. These things cost as well and are again a change in the way we design and build our larger structures.

Annotated bibliography

Bent Flyvbjerg: ”Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice”

An exposition of the concepts of Optimism Bias and how to act to the potential consequences. Strategic Misrepresentation is in this article linked to cost overruns. Finally, Reference Class Forecasting is represented as a solution to the two forms of bias.

Daniel Kahnemann: 'Thinking fast and slow'

Kahnemann focuses on how cognitive biases, especially optimism bias, can affect the decision-making process. The book focuses on human behavior and is based on recent years of research. Throughout the book, Kahnemann set up real-life examples from for example the court and the judges as the decision-makers. Liu, L., Wehbe, G., & Sisovic, J. (2010). The accuracy of hybrid estimating approaches? Case study of an Australian state road & traffic authority. Paper presented at PMI® Research Conference: Defining the Future of Project Management, Washington, DC. Newtown Square, PA: Project Management Institute. The article represents a new possible solution with the use of a hybrid approach. The hybrid will consist of parts from Reference Class Forecasting (RCF) and the conventional fixed contingency approach. The study was used to understand alternatives to RCF.

Clifford Gray: 'Revisit of Reference Class Forecasting (RCF): Estimating Costs of Infrastructure Projects'

The article represents the key managerial actions supporting the use of Reference Class Forecasting. C. Gray provides suggestions on how to implement Reference Class Forecasting.

Dan Benţa, Lucia Rusu, and Marius Podean: 'Successful Implemented Theories For Reference Class Forecasting in Industrial Field'

General issues regarding risk management are represented. Benta illustrates different theories regarding the Reference Class Forecasting e.g. “three-step approach”

Hans Reichenbach: “The theory of probability”

Reichenbach introduces the use of the theory behind probability. He explains the concepts of probability with the use of examples of calculations. Furthermore, the limitations of the theory of probability are presented.

The British Department for Transport, “Procedures for Dealing with Optimism Bias in Transport - Planning Guidance Document”

The paper investigates the potential of using a guide in future infrastructure projects. The theories leading to this idea are provided. Instructions are specific and explained for a specific reference project within the area of infrastructure. The use of the uplift method is explained, and real-world examples are given on how to use those uplifts to minimize the risk of optimism bias.

John Venn: “The logic of chance”

Fundamental problems regarding the use of probability are introduced. John Venn is skeptical about the concept of frequentism applied to the real-world problem. If applying the "three-step approach" to Reference Class Forecasting, it will be based on this type of probabilistic. John Venn’s book provides the basics of the limitations that should be considered when using Reference Class Forecasting.

Bent Flyvbjerg: “Policy and planning for large infrastructure projects: problems, causes, and cures”

In this paper, Bent Flyvbjerg identifies false estimations in infrastructure projects as well as the concepts behind it. Disadvantages will, according to Flyvbjerg, occur to these projects. The explanation for the problems is followed up by providing insights to possible solutions for the false estimations.

Alan Hájek: “The Reference Class Problem is Your Problem Too”

With the use of a philosophical and mathematical perspective Hájek represents probabilistic and the inter-related assumptions. He presents problems of Reference Class Forecasting from various perspectives. Furthermore, the effects of these problems and their outcome of a probability calculation are given.