Reference class forecasting and the corresponding limitations

1. Introduction

Many large scale projects face disproportionately high cost overruns and benefit shortfalls. These issues have been a major topic for researchers especially when it comes to large transportation infrastructure projects. Here, the Danish geographer Bent Flyvbjerg contributed immensely to the research of reasons for the cost overruns and benefit shortfalls and possible cure. (1) He found that two main aspects lead to the mentioned problems of large infrastructure projects: Optimism bias and strategic misrepresentation. The suggested remedy for the first problem is the so called reference class forecasting (RCF) method, which forces the user to take an outside view and compare the project to others. The latter one requires changes in the power structure of the projects. (2) Before the RCF method is explained in detail, the findings of Flyvbjerg’s research and the two main reasons for the mentioned problems found will be explained. (2)

1. Background

The research group on large infrastructure at Aalborg University, Denmark, led by Flyvbjerg, investigated megaprojects in the transportation infrastructure sector and their flaws. They divided the sector into rail, bridges and tunnels and road. They found that nine out of ten projects have cost overruns and many projects have overestimations on traffic leading to benefit shortfalls. (2) A good example for the disastrous effects such false estimations can have, is the Central Artery/Tunnel, also known as the Boston’s Big Dig. The budget overrun was 275% or 11 Billion USD when it opened. (2) It can be inferred from these findings that there is a need for improvement in estimations, since cost overruns and benefit shortfalls are to be avoided.

1. Disadvantages of false estimations and their reasons

There seem to be four main categories of problems with the estimations, which are:

• Inefficient allocation of resources so that the wrong projects are subsidized and proceeded
• Delays resulting in problems with funding and re-approval
• Destabilization of policies, planning, implementation and operations through endless political debates and constant re-approval
• Growing size of projects so that cost overruns become tremendous and benefits shortfalls affect whole nations (2)

These fours problems of false estimations are inherent in almost every project. Every organization, may it be a company, a NGO or a government, has to decide which projects will be undertaken how much resources are allocated. Choosing the wrong projects leads into a waste of resources of all kinds. Delays, cost overruns and such affect all stakeholders leading to disadvantages. Those problems start with false estimations in costs and benefits. Usually, costs are underestimated and benefits overestimated. The RCF method, as suggested by Flyvbjerg, can help to increase the accuracy of the estimations and therefore mitigating the effects of false ones. According to Flyvbjerg, there are two main reasons for such false estimations: The psychological optimism bias and the political, strategic misrepresentation.

1. 1. Optimism bias

The term optimism bias was coined by the Nobel prize winner Daniel Kahnemann describing the idea that the most people see the world in a more positive light than it actually shines, their own characteristics more favorable than they really are and that their sought goals are more likely to be achieved than they truly are. This leads to the effect of overconfidence in personal judgements. Risks tend to be seen lower and own capabilities better than they are. Therefore, projects seem to be more feasible for their participants, who are then optimistically biased – the so called planner’s fallacy. This results in people undertaking projects with too optimistic expectations about budget and risk, which will most likely not be met. (3) In this situation RCF can help to overcome this bias, since it forces the user to gather empirical data and comparing his own project to similar finished ones, which decreases the importance of the user’s inside view and internal judgement. (2)

1. 1. Strategic misrepresentation

Inherent in all business sectors are projects’ fights for scarce resources. Each company, NGO and government has to decide which projects will be undertaken and which not. Decision makers mainly base their judgement on a benefit-cost ratio and the immanent risk of the project. When project planners present their cases, they often brighten the numbers regarding the risk and the benefits of their projects. Regarding to Flyvbjerg, they are deliberately deceiving the decision makers, since the projects that look the best on paper will be approved. (2) The planners seem to use the formula: Underestimated costs + overestimated benefits = project approval (4) In this case, where estimations are willingly manipulated to get a project approved, RCF is not an optimal solution. The incentives, power structures as well as responsibilities need to be altered in order to avoid such behavior. For more information, see (2)

1. RCF method and its application in infrastructure projects

In order to overcome the optimism bias and produce a more realistic forecast, the RCF method requires the user to take an outside view comparing the project at hand to similar other ones. The method basically comprises three steps:

• Identifying relevant reference class
• Establishing a probability distribution for the chosen class
• Benchmarking the specific project to the reference class distribution (5)

The first step is to identify a reference class of similar, historical projects. This group of projects should be large enough to be statistically meaningful, but small enough to be comparable to the project at hand. The second step is then to create a probability distribution for certain variables of the projects like on-time delivery, cost overrun or demand forecast mistakes. This can only be done with access to credible, empirical data of a statistically significant number of projects. Lastly, the specific project needs to be compared to the reference class in regards to the chosen variables for establishing the most probable outcome of it. (5) Flyvbjerg’s research led to a commonly used application of the RCF method namely the estimation’s uplift. PICTURE HERE In collaboration with the Danish consultancy COWI, Flyvbjerg developed guidelines for project planners to remediate their estimations. The application for a rail infrastructure project will serve as an example. Figure 1 depicts the required uplift of the initially estimated budget in regards to the acceptable chance of a cost overrun. Such a tradeoff is the outcome of the second step of the RCF method. If a planner now wants to correct his estimations about budgeting, he can calculate the required uplift in regards to a possible risk of a cost overrun. When the acceptable chance for a cost overrun is set to be 10 % for example, the corresponding uplift of the budget is 68 % in this case. (6) Those uplifts for budgets are only one example for the outcome of the RCF method. Likewise, reference classes can be established based on traffic forecasts. In those cases, a “downlift” might be possible as well, so that the tentative traffic forecast needs to be decreased in order to be more realistic. (6) A real case application is presented in (6). It has to be mentioned that the RCF method comprises simple looking steps on the first view. Nevertheless, the setting of the reference class is a difficult task. Gathering empirical data from the business sector the specific project is embedded in, selecting only comparable projects and creating the probability distribution are complex tasks. (2)

1. Limitations of the Reference Class Forecast method

As every other method, the RCF method is facing limitations. There seem to be three major issues limiting the application of the RCF method. These are:

• Definition of frequentism
• Subjectivity of reference classes
• Acquisition of data

The first one is more a philosophical problem, whereas the second one concerns the subjectivity of the method. The last one is about gathering the empirical data. Nevertheless, they go hand in hand.

1. 1. Frequentism

The basic interpretation of probability in the school of frequentism defines the probability of an event as the measure of chance that a certain event will be the result of an experiment. That means that the probability assigned to a specific event A, is the actual number of ways that this event can happen divided by the total number of all possible outcomes of the experiment at hand. (See Formula 1) The probability of such an event can reach from 0 to 1, where 0 means that the event is occurring 0% of the times in an infinite number of repetitions and 1 correspondingly that it occurs 100 % of the times. (7) P(A)=(The number of ways event A can occur)/(The total number of possible outcomes) Formula 1 The first problem arises by thinking about the total number of possible outcomes of the experiment. Is it possible for a person with limited knowledge to foresee every single, possible outcome of an experiment? Hájek argues that this is not the case. (8) Therefore, the range of possible outcomes needs to be defined by the operator. Taking the flipping of a coin as an example, the number of outcomes is infinite. Usually the operator sets the possible outcomes to the two obvious ones: heads or tails; but the coin can land on the rim as well as every other thinkable position. By scoping the possible outcomes, the probability is defined: For heads or tails the probability for each of them is 1/2, but if three outcomes are defined then it changes to 1/3. Scoping the possible outcomes is considered as setting the reference class and has direct impact on the probability distribution. Reichenbach brought this to a nutshell, when he stated: “If we are asked to find the probability holding for an individual future event, we must first incorporate the case in a suitable reference class. An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class.” (8)

It can be inferred from this statement that setting the reference class in an adequate way is highly important for the RCF method. The method is based on the reference class and is therefore dependent on it. Changing the reference class, will change the probability distribution and thus the whole outcome of the method. This may result then in return again in false estimations or at least in lower accuracy than maybe possible. (2)

1. 1. Subjectivity of reference classes

This being said, it seems legit to claim that setting the reference class is a subjective task performed by the forecaster. It is the main idea of the RCF method to overcome personal biases and rely on statistics. But it seems that this claim is falsified in the face of the subjectivity of setting the reference class. The method asks for reference classes that are broad enough to be statistically meaningful, but small enough to be comparable. What does this exactly mean? Those requirements are vague leading to different, subjective interpretations. When is the class big enough? How are projects comparable when we keep in mind that every project is created with its own particular aim and timeframe within its own environment. All in all, the setting of the reference class is exposed to the subjectivity of the forecaster. (9)

1. 1. Acquisition of data

In the course of this, there seems to be another problem with the reference class regarding the data acquisition. Gathering ex post data about finished projects that are similar enough to be comparable to the specific project at hand might be a pitfall. Acquiring this data might turn out to be a challenge. If not enough data can be collected to be statistically meaningful, whatever this means, the method cannot be applied. (2)

These issues with the RCF method may not hold up to 100% for transportation infrastructure projects, since they seem to be more alike than other kinds of projects and an accurate database has been created by Flyvbjerg to be accessed by project planners. But the core problems addressed still remain: When is a project really comparable and how to set the reference class? (9)

1. A new approach – Combining RCF, Overconfidence Theory and Expert Judgements

Researchers at the Technical University of Denmark, DTU, have developed a framework to overcome the flaws of solely applying the RCF method. They combine the RCF method with the overconfidence theory to interpret expert judgements. What is new to this method is the use of scenarios elaborated by experts. The experts involved in the planning process of the project, e.g. design engineers and forecasters, are asked to provide assessments of Min/Max scenarios for the specific project about the cost of the project and the expected demand for the infrastructure, e.g. the number of tracks on a road or number of passengers per year. At this point the overconfidence theory becomes handy. The theory claims that people are, generally speaking, unaware of their capability to express the full, possible range of variations. According to (10), the statements given by the interviewees account for 60% of the full range of variations. This procedure explicitly asks for the inside view of the experts. After the assessment of the likely scenarios, the next step is to combine these scenarios with the probability distribution from the RCF method. The inside view of the experts is used to calibrate the outside, empirical data from the database. This leads to a case-specific estimation of the examined variables like costs or demand. (10) One may ask how this new approach helps to overcome the flaws discussed in “Limitations”. It seems like, it does not help to overcome the problem regarding data acquisition and nor the general issue of subjectivity of setting the reference class. What it does support is the adjustment of the estimations to the specific project. By this, it should help to increase the accuracy of the estimations, which is the basic idea of the RCF method. The historical data from the RCF method should be as comparable to the specific project, but every project is embedded in its own environment and faces its own problems based on individual circumstances. Why a project went wrong is not visible to the external analyzer using the data for the forecast of his own project’s outcome. A road might always be a road, but where this road is built, who builds it and the circumstances will most likely have an impact on the project’s results. By asking experts involved in the specific project, the individuality of the project is accounted for. By doing so, the likelihood of a more realistic forecast should be increased. Combing the empirical data from similar projects with expert judgements by those who are involved in the project, brings together the outside and inside view. (10)

1. Conclusion

Reference class forecasting is a method aiming for an improved accuracy of estimations regarding projects’ costs and benefits. Since projects are inherent in almost every business sector and their growing size, it becomes and important aspect of management to provide adequate forecasts. The RCF method does so by following a three step plan where a reference class of similar projects is established at first. Then a probability distribution for certain variables is calculated and lastly, the project is benchmarked against the reference class and corrective actions for the estimations are taken. Hereby, the forecaster is asked to take an “outside view” and his psychological biases are bypassed. Nevertheless, the setting of the reference class is subjective and therefore susceptible to new biases. Furthermore, a pure empirical data analysis does not take the individual circumstances of the specific project into account. To overcome this problems, the RCF method, overconfidence theory and expert judgements can be combined. Here, experts create Min/Max scenarios and these are then combined with the probability distribution of the reference class. By this, the accuracy of the forecast can be increased, which is the overall aim of the RCF method.

1. References

Reference class forecasting is a method aiming for an improved accuracy of estimations regarding projects’ costs and benefits. Since projects are inherent in almost every business sector and their growing size, it becomes and important aspect of management to provide adequate forecasts. The RCF method does so by following a three step plan where a reference class of similar projects is established at first. Then a probability distribution for certain variables is calculated and lastly, the project is benchmarked against the reference class and corrective actions for the estimations are taken. Hereby, the forecaster is asked to take an “outside view” and his psychological biases are bypassed. Nevertheless, the setting of the reference class is subjective and therefore susceptible to new biases. Furthermore, a pure empirical data analysis does not take the individual circumstances of the specific project into account. To overcome this problems, the RCF method, overconfidence theory and expert judgements can be combined. Here, experts create Min/Max scenarios and these are then combined with the probability distribution of the reference class. By this, the accuracy of the forecast can be increased, which is the overall aim of the RCF method.