Optimism bias, Strategic Misinterpretation and Reference Class Forecasting (RCF)

Revision as of 16:54, 1 March 2019

Abstract

Most major projects around the world are facing problems regarding planning and underestimating the costs of a project in the initial phases. These setbacks can cost an enormous sum of money and cause major setbacks of the planned schedule. The initial planning and make a reliable budget are some of the main organizational managerial skills for a project manager. ^[1].

But unfortunately, it is today seen that most projects will not fulfill the initial plans and will not meet the initial budget goals. It is logically a utopian idea of predicting the unpredictable (known unknowns) or even the things that are impossible to foresee. And that is why we, as project managers need tools and learn from the past and others.

The Danish professor Bent Flyvbjerg did research into cost overrun of major projects. By sub-dividing the broader aspect of this into the two topics: Optimism Bias and Strategic Misrepresentation Flyvbjerg explored and explained more about the cost and benefit shortfall of major projects. And through further investigation, Flyvbjerg came up with a possible solution: the use of Reference Class Forecasting ^[2]. The Reference Class Forecasting approach provides a more general overview and “is beneficial for non-routine projects” ^[3] .

Through reading this article, an explanation of the strategies and ideas mentioned above will be discussed. Furthermore, the idea of Reference Class Forecasting in larger projects will be covered as well as the studies' limitations. Examples of causal factors for cost overruns will be provided followed up by specific tools for project managers.

Word definitions

Optimism Bias

The term optimism bias was invented by the Nobel prize winner Daniel Kahneman describing the idea that most people consider themselves less likely to experience something unpleasant. This leads to the effect of overconfidence in personal judgements, leading to project managers misjudging the outcome of their decisions. Risks tend to be seen lower and own capabilities better although previous experience shows the opposite. This can result in project managers undertaking projects with too optimistic expectations about budget, risks and the project scope, which will most likely not be met. ^[4].

Strategic Misrepresentation

You have a great idea for a book, and you have found a publisher who is willing to pay. However, he needs to know more about the time perspective. "When can I have the first manuscript? Can you have it done by the end of next month?", he asks. You lower your eyes and gulp. "Of course, no problem" you answer. You have never managed that kind of timeline before, but you are aware that if you tell the truth, the publisher will not go with your idea. You do not feel like you just lied - and in official social terms, you did not, you performed the behavior of Strategic Misrepresentation and sorted out some of the truth. In a larger project, this Strategic Misrepresentation can result in underestimating costs, pre-determining and quite likely also overestimating the potential clients' benefits. Applying additional pressure and strain on individuals through manipulation, competing for scarce funds or jockeying for a position all qualify for the same over-arching category. ^[5]. Note that Strategic Misrepresentation is a form of bias as well but is used more intentional, and more likely to be a form a technique. When project planners present their cases, they often brighten the numbers regarding the risk and the benefits of their projects. Regarding Flyvbjerg, they are deliberately deceiving the decision makers, since the projects that look the best on paper will be approved. One of the things that can make a project more vulnerable to Strategic Misinterpretation is the end date is a few years down the road ^[6]. No one knows more about large-scale projects than professor Bent Flyvbjerg. Why are cost and schedule overruns so frequent? Because it is not certain that it is the best offer or project that wins. It is the project which looks best on the paper that wins.

Reference Class Forecasting

Go back to the previous example with the book. How would you decide how much time you actually needed? Reference Class Forecasting (RCF) could be a method to determine so. The problem can occur when to determine which class to place your project in. Would you compare with all books written? All books within your subject? Books at the same educational level as yours? This is the idea and challenge of RCF. RCF is a method of looking to future events by taking relatable situations and their previous outcomes. This approach aims to give a much less biased view on a specific event. The advantages and downsides of Reference Class Forecasting will be further clarified in this article.

Background

Risk contingency "is expected to be expended [and is] an amount added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and that experience shows will likely result, in aggregate, in additional costs" ^[7]. There are various methods that can be used to quantify the risk contingency, such as Reference Class Forecasting (RCF), the conventional contingency approach and risk-based estimating. This article focuses on the RCF. The cost estimates produced at different stages of the same project carries different levels of uncertainties and risks and thus different estimation accuracies. As a project progresses through its lifecycle, more information about the project's scope, design, and specifications become available, which enable the estimation team to more accurately estimate the quantity and price of material and resources. As a result, generally more risk contingency is applied at the earlier stages than in later stages.

Project cost performance and causal factors for cost overruns

How RCF Works

Bent Flyvbjerg’s research identifies two basic areas, Optimistic and Strategic Misrepresentation, as major reasons for cost estimate errors. When optimistic and strategic mismanagement biases occur, the understatement of project costs can be scandalous. Flyvbjerg argues that RCF can be used for more accurate measure of the outcome. RCF serves remarkably well to identify and restrict estimate bias. RCF is impersonal and removes moral and emotional issues. Mitigating the effect of optimistic and strategic biases is imperative and RCF can be a significant part of the solution. “Make RCF Mandatory” is the advice given by Flyvbjerg over a decade ago. RCF provides a foundation for more realistic contingency funding. RCF provides an opportunity to analyze project risk and to assign realistic budget and management reserves. The more people tied to the estimate, the greater the chance of avoiding bias estimating errors. Require contractors be accountable to their estimate. Create incentives and penalties. Performance goals and responsibilities need to be clear, for example with the use of SMART goals. The project leader needs to have power to fire contractors who fail to perform. Develop cooperative data banks that classify project by type, size, cost, and other criteria. Make these actual cost and schedule data available to be used to evaluate “inside” estimates of similar projects. Professional groups, such as engineers, lawyers, Project Management Institute, American Planning Association, and Forrester Research, could develop such data banks for other infrastructure types of projects. The six areas suggested for evaluation and improvement would vary depending on the project.

 [8].

Application

Tools for project managers

It is possible to use other biases to limit the effects. This bias is particularly important for decision-makers creating health or safety products, where the dangers of being overly optimistic can lead to dreadful outcomes. There are two researched ways of reducing the Optimism Bias:

1. Make past bad events more easily retrievable from one’s memory
2. Highlight losses that are likely to occur because of these bad events

Bringing negative events to our minds before we have the option to act can be a great technique to change people's behavior. The aim is to make the negative effects of a certain action clear to the individual and offer a clear. It is important to negate the potentially huge costs of the positivity bias when estimating the expected time to complete a task or project. As a project manager, remember to factor in a proportional optimism Bias multiplier into estimations given. Governments have this problem so consistently that there are even detailed documents outlining a 5-step approach of how to factor the bias into the planning of large projects. Unfortunately, past cost overruns in mega-projects have resulted in scandalous errors with projects finishing significantly higher than originally estimated budgets. The literature abounds with examples. In general, these studies agree that nine out of ten projects exceeded the budget. "Overruns of 50% are common; cost overruns over 50% are not uncommon" ^[9]. Flyvbjerg suggests the major causes for the differences in budget and actual are biased estimates of costs and benefits. Unfortunately, its adoption in large projects has been incredibly limited.

Figure 1, Required uplift as a function of maximum acceptable level of risk for cost overruns. Content: ^[10]

In collaboration with the Danish consultancy COWI, Flyvbjerg developed guidelines for project planners to remediate their estimations. The application for a fixed link project will serve as an example. Figure 1 shows the required uplift of the initially estimated budget regarding the acceptable chance of a cost overrun. Such a tradeoff is the outcome of the second step of the RCF method. If a planner now wants to correct his estimations about budgeting, he can calculate the required uplift regarding a possible risk of a cost overrun. As an example, when the maximum acceptable chance for a cost overrun is set to be 20 %, the corresponding uplift of the budget is around 50 % in this case, illustrated by Figure 1. It has to be mentioned that the RCF method comprises simple looking steps on the first view. Nevertheless, the setting of the reference class is a difficult task. Gathering empirical data from the business sector the specific project is embedded in, selecting only comparable projects and creating the probability distribution are complex tasks.

Use other biases to limit the effects

heehr e

Three-step approach

dfsdff

Uplift

COWI and Flyvbjerg

Limitations of the Reference Class Forecast method

Like every other method, the RCF method is facing limitations. Although Reference Class Forecasting (RCF) can largely be quite an accurate and effective way to limit the bias in a situation, it is less likely that it ever will be possible to limit all the possible things that can go wrong and estimate a perfectly correct budget and time schedule.

A better illustration of this statement is an example of colossal-sized projects, like the Olympics. In terms of the management of such an event, with relation to RCF, the issue for an organization undertaking such as high-status event is that it is almost a certainty that it will the companies first time in charge of such a project. With never large projects never done before. There has not been an Olympic game that has met or been even close to its initial budget with an average of 51% cost overrun for the hosting country throughout the recent history of the event) ^[11]. With this knowledge, although the Olympics can be considered a very niche event, it can be demonstrated that even with RCF in place, tasks with such significance make it an increasingly difficult task to refer to RCF as it is a first-time event for the country and organizations involved.

Flyvbjerg mentions that the host country has no idea of the actual position the country will be in at the time of the games. For Brazil in 2016, it was seen that the country was in a deep crisis, and things have not been that bad in a long time. Also, corruption had a huge impact on the country and the oil industry at the time. These unique events are impossible (or at least very tough) to incorporate in future Olympics. There seem to be three major issues limiting the application of the RCF method. These are:

• Probabilistic and the definition of frequentism
• The choice of reference class
• Acquisition of data and design of a database

Frequentism

The first one is more a philosophical problem and contains subjective perspectives as well. In the world of probabilistic frequentism defines the probability of an event as the measure of chance that a certain event will be the result of an experiment. ^[12] The first problem arises by thinking about the total number of possible outcomes of the experiment. Is it possible for a person with limited knowledge to foresee every single, possible outcome of an experiment? Hájek argues that this is not the case. ^[13] Reichenbach stated: If we are asked to find the probability holding for an individual future event, we must first incorporate the case in a suitable reference class. An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class ^[14]

From this statement it becomes clear that the chosen reference class for a project is highly important for the outcome and for the use and sake of the RCF method. Changing the reference class, will change the probability distribution and thus the whole outcome of the method. This may result then in return in false estimations or at least in lower accuracy than maybe possible. ^[15]

Subjectivity of reference classes

Choosing the right reference class is the main task of using the tool Reference Class Forecasting. When the forecaster is doing so, how certain can one be of choosing the right one, and not be biased? The aim of the RCF method is to overcome personal biases and rely on statistics. When using the RCF method is a certainty to use a reference class that is broad enough to be statistically meaningful and at the same time small enough to be comparable. Those requirements can seem very unclear and in the worst case leading to different, subjective interpretations. When is the class for example too small? How are projects comparable when we keep in mind that every project is created with its own particular aim and timeframe within its own environment. All in all, the setting of the reference class is exposed to a high risk of subjectivity of the forecaster. ^[16]

Design of a database

The last problem with the RCF in this article is about gathering the empirical data. In order to use the RCF method, a sufficient database of projects would be necessary. The main challenge for applying the RCF method is the accumulation of a sample of similar projects with large enough sample size and accurate cost information. Gathering data from finished projects that are similar enough to be comparable to the specific project at hand might be a pitfall. Acquiring this data might turn out to be a challenge. If not enough data is collected to be statistically meaningful the method cannot be applied. ^[17]

These issues with the RCF method may not hold up to 100% for transportation infrastructure projects since they seem to be more alike than other kinds of projects and an accurate database has been created by Flyvbjerg to be accessed by project planners. But the core problems addressed still remain: When is a project really comparable and how to set the reference class? ^[18]

Times change and the focus of the different societies changes. Would a 100-year-old project be relatable for anyone today? The cost of materials and workers change. Today, many of the countries in the western part of the world pay a lot of attention to climate change and greener solutions. These things cost as well and is again a change in the way we design and build our larger structures.

Annotated bibliography

Bib

References

1. ↑ PMI:Project Management Institute,Project Management: A guide to the Project Management Body of Knowledge (PMBOK guide), 6th Edition 2017 , Table 1-2.
2. ↑ Flyvbjerg, Bent, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, (European Planning Studies, 2008), 16. 3-21.
3. ↑ Flyvbjerg, B., From Nobel Prize to project management: getting risks right, (Paper presented at PMI® Research Conference: New Directions in Project Management, Montréal, Québec, Canada. Newtown Square, PA: Project Management Institute, 2006).
4. ↑ Kahnemann, Daniel, Thinking fast and slow, New York: Farrar, Straus and Giroux, 2013.
5. ↑ Flyvbjerg, Bent,Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice .
6. ↑ Hot Air - Meaning Ring, http://meaningring.com/2016/05/30/strategic-misrepresentation-by-rolf-dobelli/
7. ↑ Liu, L., Wehbe, G., & Sisovic, J. (2010). The accuracy of hybrid estimating approaches? Case study of an Australian state road & traffic authority. Paper presented at PMI® Research Conference: Defining the Future of Project Management, Washington, DC. Newtown Square, PA: Project Management Institute.
8. ↑ Gray, Clifford, Revisit of Reference Class Forecasting (RCF): Estimating Costs of Infrastructure Projects, PM World Journal, January 2018.
9. ↑ Clifford, Gray, Revisit of Reference Class Forecasting (RCF): Estimating Costs of Infrastructure Projects
10. ↑ The British Department for Transport, Procedures for Dealing with Optimism Bias in Transport. Planning Guidance Document, June 2004
11. ↑ Reichenbach, Hans The theory of probability, University of California Press, 1949
12. ↑ Venn, John The logic of chance, Macmillan and co, 1876..
13. ↑ Reichenbach, Hans The theory of probability, University of California Press, 1949.
14. ↑ Reichenbach, Hans The theory of probability, University of California Press, 1949.
15. ↑ Flyvbjerg, Bent Policy and planning for large-infrastructure projects: problems,causes and cures, Environment and Planning B: Planning and Design, 2005,Vol. 34, 578-597.
16. ↑ Hájek, Alan The Reference Class Problem is Your Problem Too, Synthese. 2006
17. ↑ Flyvbjerg, Bent Policy and planning for large-infrastructure projects: problems,causes and cures, Environment and Planning B: Planning and Design, 2005,Vol. 34, 578-597.
18. ↑ Hájek, Alan The Reference Class Problem is Your Problem Too, Synthese. 2006