Reference class forecasting and the corresponding limitations

Developed by Georg Maximilian Ruven Schriewersmann

Many large scale projects face disproportionately high cost overruns and benefit shortfalls. These issues have been a major topic for researchers especially when it comes to large transportation infrastructure projects. Here, the Danish geographer Bent Flyvbjerg contributed immensely to the research of reasons for the cost overruns and benefit shortfalls and possible cures. He found that two main aspects lead to the mentioned problems of large infrastructure projects: Optimism bias and strategic misrepresentation. The suggested remedy for the first problem is the so called reference class forecasting (RCF) method, which forces the user to take an outside view and compare the project to others. The latter one requires changes in the power structure of the projects. The overall aim of the RCF method is to optimize forecasts in regards to projects variables like costs and benefits in order to avoid cost overruns and benefit shortfalls.

This article will first explain the background of the establishment of the RCF method. Then it will give an overview of the disadvantages of false forecasts followed by explanations for those false estimations. After that, the RCF method will be explained and an example for its application will be introduced. From there on, the limitations of the RCF method will be discussed and a possibility for overcoming them will be given. The article ends with a conclusion.

The main goal of this article is to present the method of RCF, illustrate and emphazise the limitations based on the underlying principles of this method and provide an extension to the RCF method in order to mitigate the limitations. Making the reader aware of the cognitive biases inherent in the RCF method, so he can mitigate them is another key point of the article. Due to restricted access rights, it is not possible for the author to present examplary calculations of the different steps of the RCF method.

Background

---

The research group on large infrastructure at Aalborg University, Denmark, led by Flyvbjerg, investigated megaprojects in the transportation infrastructure sector and their flaws. They divided the sector into rail, bridges and tunnels and road. The reason why infrastructure projects suit well as examples is their similarity. Building a road, a rail way or a bridge is often very similar and similarity is a premise for the RCF method. One of the most stunning findings is that nine out of ten projects have cost overruns and many projects have overestimations on traffic leading to benefit shortfalls. A good example for the disastrous effects such false estimations can have, is the Central Artery/Tunnel, also known as the Boston’s Big Dig. The budget overrun was 275% or 11 Billion USD when it opened. It can be inferred from these findings that there is a need for improvement in estimations, since cost overruns and benefit shortfalls are to be avoided.

Disadvantages of false estimations and their reasons

---

There seem to be four main categories of problems with the estimations, which are:

• Inefficient allocation of resources so that the wrong projects are subsidized and proceeded
• Delays resulting in problems with funding and re-approval
• Destabilization of policies, planning, implementation and operations through endless political debates and constant re-approval
• Growing size of projects so that cost overruns become tremendous and benefits shortfalls affect whole nations

These fours problems of false estimations are inherent in almost every project. Every organization, may it be a company, a NGO or a government, has to decide which projects will be undertaken how much resources are allocated. Choosing the wrong projects leads into a waste of resources of all kinds. Delays, cost overruns and such affect all stakeholders leading to disadvantages. Those problems start with false estimations in costs and benefits. Usually, costs are underestimated and benefits overestimated. The RCF method, as suggested by Flyvbjerg, can help to increase the accuracy of the estimations and therefore mitigating the effects of false ones. According to Flyvbjerg, there are two main reasons for such false estimations: The psychological optimism bias and the political, strategic misrepresentation.

Optimism bias

---

The term optimism bias was coined by the Nobel prize winner Daniel Kahnemann describing the idea that the most people see the world in a more positive light than it actually shines, their own characteristics more favorable than they really are and that their sought goals are more likely to be achieved than they truly are. This leads to the effect of overconfidence in personal judgements. Risks tend to be seen lower and own capabilities better than they are. Therefore, projects seem to be more feasible for their participants, who are then optimistically biased – the so called planner’s fallacy. This results in people undertaking projects with too optimistic expectations about budget and risk, which will most likely not be met. In this situation RCF can help to overcome this bias, since it forces the user to gather empirical data and comparing his own project to similar finished ones, which decreases the importance of the user’s inside view and internal judgement.

Strategic misrepresentation

---

Inherent in all business sectors are projects’ fights for scarce resources. Each company, NGO and government has to decide which projects will be undertaken and which not. Decision makers mainly base their judgement on a benefit-cost ratio and the immanent risk of the project. When project planners present their cases, they often brighten the numbers regarding the risk and the benefits of their projects. Regarding to Flyvbjerg, they are deliberately deceiving the decision makers, since the projects that look the best on paper will be approved. The planners seem to use the formula: Underestimated costs + overestimated benefits = project approval

In this case, where estimations are willingly manipulated to get a project approved, RCF is not an optimal solution. The incentives, power structures as well as responsibilities need to be altered in order to avoid such behavior. It is worth mentioning the strategic misrepresentation to see the full picture of reasons for false estimations, but since the focus is on the RCF method, the strategic misrepresentation will not be further pursued. For more information, see

RCF method and its application in infrastructure projects

---

In order to overcome the optimism bias and produce a more realistic forecast, the RCF method requires the user to take an outside view comparing the project at hand to similar other ones. The method basically comprises three steps:

• Identifying relevant reference class
• Establishing a probability distribution for the chosen class
• Benchmarking the specific project to the reference class distribution

The first step is to identify a reference class of similar, historical projects. This group of projects should be large enough to be statistically meaningful, but small enough to be comparable to the project at hand. The second step is then to create a probability distribution for certain variables of the projects like on-time delivery, cost overrun or demand forecast mistakes. This can only be done with access to credible, empirical data of a statistically significant number of projects. Lastly, the specific project needs to be compared to the reference class in regards to the chosen variables for establishing the most probable outcome of it. Flyvbjerg’s research led to a commonly used application of the RCF method namely the estimation’s uplift.

In collaboration with the Danish consultancy COWI, Flyvbjerg developed guidelines for project planners to remediate their estimations. The application for a rail infrastructure project will serve as an example. Figure 1 depicts the required uplift of the initially estimated budget in regards to the acceptable chance of a cost overrun. Such a tradeoff is the outcome of the second step of the RCF method. If a planner now wants to correct his estimations about budgeting, he can calculate the required uplift in regards to a possible risk of a cost overrun. When the acceptable chance for a cost overrun is set to be 10 % for example, the corresponding uplift of the budget is 68 % in this case. Those uplifts for budgets are only one example for the outcome of the RCF method. Likewise, reference classes can be established based on traffic forecasts. In those cases, a “downlift” might be possible as well, so that the tentative traffic forecast needs to be decreased in order to be more realistic. A real case application is presented in . It has to be mentioned that the RCF method comprises simple looking steps on the first view. Nevertheless, the setting of the reference class is a difficult task. Gathering empirical data from the business sector the specific project is embedded in, selecting only comparable projects and creating the probability distribution are complex tasks.

Limitations of the Reference Class Forecast method

---

As every other method, the RCF method is facing limitations. There seem to be three major issues limiting the application of the RCF method. These are:

• Definition of frequentism
• Subjectivity of reference classes
• Acquisition of data

The first one is more a philosophical problem, whereas the second one concerns the subjectivity of the method. The last one is about gathering the empirical data. Nevertheless, they go hand in hand.

Frequentism

---

The basic interpretation of probability in the school of frequentism defines the probability of an event as the measure of chance that a certain event will be the result of an experiment. That means that the probability assigned to a specific event A, is the actual number of ways that this event can happen divided by the total number of all possible outcomes of the experiment at hand. (See Formula 1) The probability of such an event can reach from 0 to 1, where 0 means that the event is occurring 0% of the times in an infinite number of repetitions and 1 correspondingly that it occurs 100 % of the times.

P(A)=(The number of ways event A can occur)/(The total number of possible outcomes)

The first problem arises by thinking about the total number of possible outcomes of the experiment. Is it possible for a person with limited knowledge to foresee every single, possible outcome of an experiment? Hájek argues that this is not the case.

Therefore, the range of possible outcomes needs to be defined by the operator. Taking the flipping of a coin as an example, the number of outcomes is infinite. Usually the operator sets the possible outcomes to the two obvious ones: heads or tails; but the coin can land on the rim as well as every other thinkable position. By scoping the possible outcomes, the probability is defined: For heads or tails the probability for each of them is 1/2, but if three outcomes are defined then it changes to 1/3. Scoping the possible outcomes is considered as setting the reference class and has direct impact on the probability distribution. Reichenbach brought this to a nutshell, when he stated: “If we are asked to find the probability holding for an individual future event, we must first incorporate the case in a suitable reference class. An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class.”

It can be inferred from this statement that setting the reference class in an adequate way is highly important for the RCF method. The method is based on the reference class and is therefore dependent on it. Changing the reference class, will change the probability distribution and thus the whole outcome of the method. This may result then in return again in false estimations or at least in lower accuracy than maybe possible.

Subjectivity of reference classes

---

This being said, it seems legit to claim that setting the reference class is a subjective task performed by the forecaster. It is the main idea of the RCF method to overcome personal biases and rely on statistics. But it seems that this claim is falsified in the face of the subjectivity of setting the reference class. The method asks for reference classes that are broad enough to be statistically meaningful, but small enough to be comparable. What does this exactly mean? Those requirements are vague leading to different, subjective interpretations. When is the class big enough? How are projects comparable when we keep in mind that every project is created with its own particular aim and timeframe within its own environment. All in all, the setting of the reference class is exposed to the subjectivity of the forecaster.

Acquisition of data

---

In the course of this, there seems to be another problem with the reference class regarding the data acquisition. Gathering ex post data about finished projects that are similar enough to be comparable to the specific project at hand might be a pitfall. Acquiring this data might turn out to be a challenge. If not enough data can be collected to be statistically meaningful, whatever this means, the method cannot be applied.

These issues with the RCF method may not hold up to 100% for transportation infrastructure projects, since they seem to be more alike than other kinds of projects and an accurate database has been created by Flyvbjerg to be accessed by project planners. But the core problems addressed still remain: When is a project really comparable and how to set the reference class?

A new approach – Combining RCF, Overconfidence Theory and Expert Judgements

---

Researchers at the Technical University of Denmark, DTU, have developed a framework to overcome the flaws of solely applying the RCF method. They combine the RCF method with the overconfidence theory to interpret expert judgements. What is new to this method is the use of scenarios elaborated by experts. The experts involved in the planning process of the project, e.g. design engineers and forecasters, are asked to provide assessments of Min/Max scenarios for the specific project about the cost of the project and the expected demand for the infrastructure, e.g. the number of tracks on a road or number of passengers per year. At this point the overconfidence theory becomes handy. The theory claims that people are, generally speaking, unaware of their capability to express the full, possible range of variations. According to (10), the statements given by the interviewees account for 60% of the full range of variations. This procedure explicitly asks for the inside view of the experts. After the assessment of the likely scenarios, the next step is to combine these scenarios with the probability distribution from the RCF method. The inside view of the experts is used to calibrate the outside, empirical data from the database. This leads to a case-specific estimation of the examined variables like costs or demand.

One may ask how this new approach helps to overcome the flaws discussed in “Limitations”. It seems like, it does not help to overcome the problem regarding data acquisition and nor the general issue of subjectivity of setting the reference class. What it does support is the adjustment of the estimations to the specific project. By this, it should help to increase the accuracy of the estimations, which is the basic idea of the RCF method. The historical data from the RCF method should be as comparable to the specific project, but every project is embedded in its own environment and faces its own problems based on individual circumstances. Why a project went wrong is not visible to the external analyzer using the data for the forecast of his own project’s outcome. A road might always be a road, but where this road is built, who builds it and the circumstances will most likely have an impact on the project’s results. By asking experts involved in the specific project, the individuality of the project is accounted for. By doing so, the likelihood of a more realistic forecast should be increased. Combing the empirical data from similar projects with expert judgements by those who are involved in the project, brings together the outside and inside view.

Conclusion

---

Reference class forecasting is a method aiming for an improved accuracy of estimations regarding projects’ costs and benefits. Since projects are inherent in almost every business sector and their growing size, it becomes and important aspect of management to provide adequate forecasts. The RCF method does so by following a three step plan where a reference class of similar projects is established at first. Then a probability distribution for certain variables is calculated and lastly, the project is benchmarked against the reference class and corrective actions for the estimations are taken. Hereby, the forecaster is asked to take an “outside view” and his psychological biases are bypassed. Nevertheless, the setting of the reference class is subjective and therefore susceptible to new biases. Furthermore, a pure empirical data analysis does not take the individual circumstances of the specific project into account. To overcome this problems, the RCF method, overconfidence theory and expert judgements can be combined. Here, experts create Min/Max scenarios and these are then combined with the probability distribution of the reference class. By this, the accuracy of the forecast can be increased, which is the overall aim of the RCF method.

Annotated Bibliography

---

• Department of Development and Planning - Aalborg University Bent Flyvbjerg
  • Description of the curriculum vitae of Bent Flyvbjerh and his contribtion to the management of infrastructure projects

• Flyvbjerg, Bent Policy and planning for large-infrastructure projects: problems,causes and cures
  • In this paper, Flyvbjerg indentifies the main reasons for false estimations in infrastructure projects and the resulting disadvantages for the projects. Having this explained, he continues by providing insights to possible cures for those false estimations.

• Kahnemann, Daniel Thinking, fast and slow
  • Kahnemann describes the fundamentals of the so called reductionist school, which explains human behavior by reducing the mental mechanisms involved in decision making down to cognitive biases and heuristics. He provides examples from everyday life to make the reader understand how those biases, especially the optimism bias, work and how they influence humans in their decision making process

• Flyvbjerg, Bent, Quality control and due diligence in project management: Getting decisions right by taking the outside view
  • This paper explores the planner's fallacy, namely the optimism bias, in the environment of infrastructure projects and how this knowledge could be used for quality control. He argues that false estimations are a often neglected problem of project management; thus it should be paid more attention to, since the effects they have are tremendous. The next step is a brief description of how the work of Kahnemann and Tversky helped to indentify the roots of those false estimations. He finishes the paper with explaining how quality control can be improved and explains an extended, 8-step version of the RCF method, namely due diligence.

• Benţa, Dan, Rusu, Lucia and Podean, Marius Ioan SUCCESSFUL IMPLEMENTED THEORIES FOR REFERENCE CLASS FORECASTING IN INDUSTRIAL FIELD
  • Benta et al. give an introduction to the general issues with risk management and a review of literature for explaining the roots of risk management. It furthermore comprises an overview of different reference class forecasting theories.

• Flyvbjerg, Bent and COWI Procedures for Dealing with Optimism Bias in Transportation Projects - Guidance Document
  • The British department of transportation hired Flyvbjerg and COWI to investigate possibilities for developing a framework guiding future infrastructure projects in the UK. The background of this investigation is briefly explained, i.e. theories and environment, followed by instructions for certain reference projects in infrastructure. The uplift method is introduced and real world examples are given how to use those uplifts to overcome the optimism bias.

• Venn, John The logic of chance
  • 140 years ago, the mathematician John Venn was thinking about the fundamental problems of the theory of probability. He introcuded the main critics and arguments why the so called frequentism has its flaws in the application in the real world outside of math. Since the RCF method is based on probability, this book provides a basis for showing the limitations of the RCF based on its very fundamentals - probability distribution

• Reichenbach, Hans The theory of probability
  • Reichenbach gives an introdcution to the theory of probability. He explains what probability is and where it comes from. Further calculation instructions are provided. In addition, Reichenbach illustrates the limilations and flaws of the theory of probability.

• Hájek, Alan The Reference Class Problem is Your Problem
  • Hájek takes an philosophical and mathematical perspective on probability and the underlying assumptions. He presents the so called reference class problems from various perspectives explaining the effects these problems have on the outcome of a probability calculation. He then gives examples for many different interpretations about probability, e.g. frequentism, logical or subjectivism and their view on the reference class problem.

• Leleur, Steen, et al. Combining Reference Class Forecasting with Overconfidence Theory for Better Risk Assessment of Transport Infrastructure Investments
  • The researchers at DTU give a brief introduction to the method of reference class forecasting and its limitations. This is followed by a suggestion of how to overcome the limitations and even improve the accuracy of estimations by combining the RCF with Overconfidence Theory and Expert Judgements. They back up their claims with a case study of the Nuuk airport.

---