Reference class forecasting

Revision as of 16:25, 23 February 2021

Abstract

The definition of project success according to the standard published by the project management institute is meeting customers' expectations without exceeding the desired requirement such as cost, duration, and scope. ^[1] However, executing projects on time following a planned framework and budget is a challenging aspect of project management. To overcome these challenges, the Danish professor Bent Flyvbjerg initiated a research to study the overrun cost of major projects. Flyvbjerg proposed a solution for these challenges which is called Reference Class Forecasting. The Reference class forecasting is a method that studies the overall view of certain projects by forecasting similar projects rather than focusing solely on the considered project. This method allows a project manager to avoid errors by basing the forecast on similar projects. It also assists to take decisions under uncertainties through assessing the risk of the planned project. ^[1] In this article, the RCFM method will be presented. That will be followed with clear guidance on how to use the method. Then its application and limitations.

Root causes of poor performance

Research made on a sample of 258 large projects executed over the last 7 decades shows that 90% of these projects exceeded the original budget. According to re-searcher, almost all projects do not deliver the intended promises such as on-time completion and staying within budget. ^[2]. The poor performance of a project is correlated to the managerial level such as project implementation and management methodology. ^[3] Project failures are usually covered up and overlooked. However, Flyvbjerg has identified two explanatory models for the poor performance of a project. ^[4] These models are:

• Psychological Explanations: It accounts for the benefit shortfalls that are caused by planning fallacy and optimism bias. Optimism bias is a term coined by Daniel Kahnemann means that people tend to see the world in a more positive light. Flyvbjerg states that human judgment is biased, as it tends to be more optimistic than realistic due to overconfidence which leads to underestimating cost, completion times, and risks of planned actions. Thus, planners who pursue initiative will most likely end up overrunning the estimated budget and duration ^[5]

• Strategic misrepresentation: This model accounts for unreliable planning and decision-making due to political pressure. When estimating the outcome of a project, planners tend consciously to overestimate the benefits and underestimate the cost to increase the likelihood of getting their projector plan approved. ^[5] Thus, when political and organizational pressure are high, planners or project managers non-intentionally tend to underestimate the project cost and overestimate the benefits. However, to gain approval or funding, they intentionally use the following formula: ^[4]

                              Project approval = Underestimated cost + Overestimated benefits [4]

Poor performance can also be caused from a technical point of view. For instance, unsuitable forecasting techniques, inadequate data, poor contract, etc. However, this aspect of poor performance is beyond the scope of this article.^[5] Consequently, understanding the technique Flyvbjerg proposed to encounter these aspects.

Big idea

One method that is being used in infrastructure projects is Reference Class forecasting. ^[3] The reference class forecast provided an external point of view and act as an enabler of better planning based on historical data of projects that have similar attributes. By doing so, the project managers can reduce bias that is caused due to assessing available information "inside views" and neglecting unknown unknowns or other considerations "outside views".^[5] RCFM is recommended by the American Planning Association which “encourages planners to use reference class forecasting in addition to traditional methods as a way to improve accuracy“. ^[5] The RCF attempts to fit a certain event into a probability of distribution of comparable class reference. Furthermore, this method of enhancing decision-making in light of un-certainties has proved to be effective. It allows for adjustments to be made in the original cost-benefit analysis (CBA) so the plan includes margin errors. ^[6]

Implementation

RCFM requires a large amount of work and should be implemented before initiating a project in order to get a unique opportunity on reflecting on the budget and planned duration. Implementing the RCFM requires a three-step approach, these steps are shown in figure 1 and explained thoroughly in the text below.

• Identify a reference class that has similar attributes to the project on hand. There is no role of thumbs when choosing a reference class. However, the reference class can not be narrow to get a reliable result if the categories were too small. The reference class can not be too wide either. Furthermore, the organization must also decide whether they want to create reference classes based on their projects within the programs or do they want to include reference classes from other organizations. ^[7]

• Determine a probability of distribution for the chosen reference classes. This requires trustful historical data from several projects within the reference class. The result of the probability distribution of the historical data will be used to estimate the level of uncertainty. From a statistical perspective, regression models are an essential tool in deriving the probability distribution

• Comparing the project on hand with the reference class distribution in order to determine the desired outcome such as budget and project duration.^[5]

Lastly, based on the degree of tolerance for the risk we decide what quantity of the optimism uplift would be needed to account for bias in cost and time estimates ^[6]

Limitation

RCF is not a new tool within the decision-making framework for mega infrastructure projects. However, its application to other fields is rare due to the lack of data on relevant reference classes or faulty information. The accuracy of the result generated by applying the RCF depends greatly on the sample size of the reference classes and the relevance of the reference class. Thus, if the reference classes were chosen poorly the forecasting result will give invalid results. Another factor to consider is having outliers in the identified samples. For instance, if the forecasting result falls within the insignificant region where the outlier stands, the RCF methods most likely will generate faulty predictions.

Last but not least, the RCF method is more convenient in cases where errors are due to non-random events such as human bias in decision making while uncertainty is present.

Annotated Bibliography

Flyvbjerg, B., Skamris Holm, M.K. and Buhl, S.L., 2004. What causes cost overrun in transport infrastructure projects?. Transport reviews. ^[2]

• An overview of statistical studies on 258 large projects
• Relationship between project cost escalation and sluggish project, big project.

Flyvbjerg, B., 2007. Policy and planning for large-infrastructure projects: problems, causes, cures. Environment and Planning B: planning and design. ^[5]

• Reasons of inaccuracy in forecasts of cost and benefits
• Reference class forecasting

Walczak, R. and Majchrzak, T., 2018. Implementation of the Reference Class Forecasting Method for Projects Implemented in a Chemical Industry Company. Acta Oeconomica Pragensia ^[3]

Awojobi, O. and Jenkins, G.P., 2016. Managing the cost overrun risks of hydroelectric dams: An application of reference class forecasting techniques. Renewable and Sustainable Energy Reviews ^[6]

Pindy Bhullar, 2018, De-risking the programme portfolio with reference class forecasting ^[7]

• An article about the poor performance of project and program and how can the RCFM be used

Flyvbjerg, B., 2013. Over budget, over time, over and over again: Managing major projects. ^[8]

References

1. ↑ ^{1.0 1.1} "Project Management Institute (PMI),2017, Guide to the Project Management Body of Knowledge (PMBOK® Guide) (6th Edition)"
2. ↑ ^{2.0 2.1} "Flyvbjerg, B., Skamris Holm, M.K. and Buhl, S.L., 2004. What causes cost overrun in transport infrastructure projects?. Transport reviews https://doi.org/10.1080/0144164032000080494a"
3. ↑ ^{3.0 3.1 3.2} "Walczak, R. and Majchrzak, T., 2018. Implementation of the Reference Class Forecasting Method for Projects Implemented in a Chemical Industry Company. Acta Oeconomica Pragensia. https://www.researchgate.net/publication/324337008_Implementation_of_the_Reference_Class_Forecasting_Method_for_Projects_Implemented_in_a_Chemical_Industry_Company"
4. ↑ Cite error: Invalid <ref> tag; no text was provided for refs named Flyvbjerg_2013
5. ↑ ^{5.0 5.1 5.2 5.3 5.4 5.5 5.6} " Flyvbjerg, B., 2007. Policy and planning for large-infrastructure projects: problems, causes, cures. Environment and Planning B: planning and design. http://documents1.worldbank.org/curated/en/968761468141298118/pdf/wps3781.pdf"
6. ↑ ^{6.0 6.1 6.2} "Awojobi, O. and Jenkins, G.P., 2016. Managing the cost overrun risks of hydroelectric dams: An application of reference class forecasting techniques. Renewable and Sustainable Energy Reviews. https://www.sciencedirect.com/science/article/pii/S1364032116301162/"
7. ↑ ^{7.0 7.1} "Pindy Bhullar, 2018, De-risking the programme portfolio with reference class forecasting. https://www.apm.org.uk/news/de-risking-the-programme-portfolio-with-reference-class-forecasting/"
8. ↑ "Flyvbjerg, Bent. "Over budget, over time, over and over again: Managing major projects." (2013): 321-344.https://www.researchgate.net/publication/235953357_Over_Budget_Over_Time_Over_and_Over_Again_Managing_Major_Projects"