Feasibility risk assessments of transport projects using Monte Carlo-simulations

Abstract

A common way of assessing the feasibility of an infrastructure projects (e.g., road or rail) is through a cost benefit analysis (CBA), resulting in an overview of the cost and benefits/ consumer surplus as the Net present value (NPV) or Benefit cost ratio (BCR) of the project. Values which can be compared to other projects/alternatives. The result of a CBA though relies on a single value where all assumptions, estimations and calculations are reduced to a single number. It can thus be difficult to describe the certainty of the result and due to the complexity of infrastructure projects a risk of cost overruns and overestimations of benefits are historically likely to occur . One way to challenge this problem is to perform a Feasibility risk assessment (FRA) moving from a single point result to an interval result.

One FRA technique trying to cope with some of the shortcomings of CBA are a stochastic approach based on Monte Carlo simulations using reference classes. Also called reference scenario forecasting (RSF). The idea of using reference classes is to gather all possible information regarding already implemented infrastructure investment projects, which thus can be used to provide a background for evaluation of future projects and identify probability distributions that can model the uncertainty of the project impacts. .

The result of the FRA using Monte Carlo simulation is a certainty graph and/ or certainty interval providing information about the certainty of achieving e.g., a BCR of at least one. That is potentially valuable information for decision makers describing in how many cases or how large the certainty is of obtaining a project that is feasible.

This article will describe how FRA can be implemented and support decision making within transport projects. Other areas where Monte Carlo simulation can be used is to determine the schedule of a project.

Reasoning behind and optimism bias

Decision support model towards the feasibility risk assesment

In order to get a social, environmental, and economical acceptance of an infrastructure project an assessment of the impacts is necessary. This includes travel time savings, emissions, accidents etc. as well as construction and maintenance cost summarised in a Cost Benefit analysis (CBA). The CBA has the advantage that it through the provided set of criteria sums up all aspects of the decision problem to one single monetary value, net present value and/or benefit cost ratio, and make the variety of projects comparable and consistent.

This though may also be considered a weakness as the methodology might be seen as a “black box” reducing all considerations and assumptions to one single value. Furthermore, the unit pricing principles and modelling of impacts involves uncertainties due to the unknowns of the future . Thus, the CBA only provides a single guesstimate on the feasibility of the project.

Megaprojects are inherently risky due to long planning horizons and complex interfaces and a systematic tendency to overstate some of the major impacts, such as user benefits and construction cost are demonstrated. This is also known as “optimism bias” where lessons learned are likely to be neglected with a with a common belief that the future will be better than the past and that ‘the same mistake won’t happen twice'. The traditional way of handling the above-described uncertainties is through a risk analysis. Here, the undesirable effects that the subsequent activities may impose are identified and quantified in a risk matrix and mitigation measures are determined. Each single project is thus handled as a unique project identifying what may go wrong within this context. The traditional Risk analysis of course gives some useful insights and provides the decision maker with a method by which he/her can gain information on the totality of any future outcome. By adding a quantitative risk analysis (QRA) it provides the possibility to differentiate the feature of risk information in terms of outcome by probability distributions and provide an indication of the degree of overall risk faced by the project, portfolio, or program. While the CBA is appealing to its adaptability and flexibility it often applies a ‘false sense of security’ where underlying model uncertainties are somehow forgotten. Complementing the CBA with a QRA in a “feasibility risk assessment” (FRA) thus provides additional knowledge on the viability of a project and handle the uncertainties present in the decision making by moving from a single point result in the CBA to an interval result with a corresponding probability. The result can further be used to estimate any contingency reserve.

Application

The structure of the QRA is similar to a deterministic single value rate of return model but instead each input variable (e.g., cost drivers) in the model is represented by a probability distribution function. By a random sampling method several scenarios (also called iterations) can be produced concerning different probability distributions and the outcome of the model thus reflects the probability of achieving a feasible project and at what extent. One technique, presented in the following, highly suitable for a QRA model is reference scenario forecasting using Monte Carlo simulations .

Monte Carlo simulation

Running a RSF as a Monte Carlo simulation is an “analytical technique used to show the combined effect of uncertainties in order to evaluate their potential impact on objectives” . Monte Carlo simulation is a stochastic simulation where random samples or ‘draws’ (numbers) of input variables are generated from a particular predefined distribution and forms the basis of hundreds or thousands “what-if” scenarios. In order to achieve a ‘stable’ certainty graph (see below) a good rule of a thumb is to run (at minimum) 2.000 Monte Carlo iterations .

Choosing distribution

When applying the QRA and the subsequent FRA a common mistake is to choose an inadequate distribution. In general, a parametric distribution (e.g., normal, gamma and beta) should be used, if the parameter is specified in literature or data while a non-parametric distribution (e.g., triangular or uniform) should be used if the parameter relies on judgment of the uncertainty by experts. Using a parametric distribution further care should be taken securing that:

• The chosen distribution applied to the problem is supported by theory.

• It has been proven beneficial to apply the chosen distribution through a general acceptance of the problem.

• The chosen distribution approximately fits the expert’s opinion being modelled and the required level of accuracy is not too high.

. Several research and case studies varying from air, rail to road projects have been made to find and validate suitable distributions describing different transport impacts as construction cost, travel time savings, accidents etc. Figure xx summarises the derived distributions found suitable for application on various transport parameters .

• Change figure and add further description

But in the end, it all comes down the level of knowledge about realised impacts of already implemented projects. And hence, it can be beneficial for an organisation to establish a project database for future reference class forecasting including information about estimated construction cost, estimated user benefits and other impacts as well as the actual construction cost and the realised benefits/impacts. In figure xx are presented examples of how a project database is used as reference scenarios deriving a suitable probability distribution. The data is based on a case study made by Flyvbjerg et al. 2003

• Add figure

Running the Monte Carlo simulation

While software Python or R studio are commonly used to run Monte Carlo simulation and subsequent FRA it requires some knowledge in programming. However, a Monte Carlo simulation can easily be run through build-in functions in Microsoft excel. The following will describe how random sampling of user benefits and construction cost as these are the major impacts of infrastructure projects. However, a further break down of e.g., user benefits could be made in order to do random sampling of e.g., time and travel distance savings, while externalities could be broken down into accidents and pollution. In excel a column for simulation number, Consumer surplus, Construction cost (etc) are defined. The first input needed for the simulations is the socio economical posts derived from the CBA as user-benefits, construction cost, scrap value, maintenance cost, and externalities. Next the parameters for the chosen distribution is needed. The parameter for this example is derived from the previous mentioned case study made Flyvbjerg et al. 2003 Simulating the consumer surplus a “Beta-Pert” distribution is chosen which needs parameters for the minimum, most likely, and maximum deviation of estimate.

A is calculated as: 1+Minimum

B is calculated as: 1+Most likely

C is calculated as: 1+Maximum

alpha= (4(b-a))/(c-a)+1

beta= (4(c-b))/(c-a)+1

The consumer surplus can now be simulated running the following excel function:

Beta.INV(Rand();alpha;beta;a;c)*CBA_consumersurplus

Simulating the construction cost an “Erlang” distribution is chosen which needs a shape and a scale parameter and the construction cost can now be simulated running the following excel function:

Gamma.INV(Rand();shape;scale)*CBA_construction_cost

Certainty graph

The result of running an FRA using Monte Carlo simulations is, as mentioned, a bunch of ‘What-if’ scenarios testing different combinations of construction cost and user benefits and the resulting cost benefit ratio. The simulated ‘scenarios’ can be summarised in an accumulated descending graph, or so-called ‘certainty graph’. The advantage of a certainty graph is that it provides the decision maker with a visual reference from which it is fairly easy to interpret the certainty of the project’s feasibility / or the risk of being unfeasible. The certainty graphs are derived by finding the BCR for a given probability (0,0 to 1,0). This is done by applying Excel’s percentile function to the array of simulated BCRs. From the certainty graph it can be seen that there is xx percentage chance of reaching a BCR of at least one

• Include Example of FRA outcome and how to interpret.

Limitations

When performing a feasibility risk assessment it is necessary to distinguish whether it’s a road or railway project. This is usually not an issue, however, some large infrastructure projects might include both. In this case it becomes slightly more difficult to perform the FRA as it is necessary to consider the construction cost and user benefits (and other costs/benefits) independently on the two modes in order to apply the right parameters when running the Monte Carlo simulation. Subsequently the costs and benefits simulated separately for each mode must hereafter be summarised in a “single” benefit and cost value respectively for every simulation run in order to evaluate the overall feasibility of the project. Hence, if its chosen to run e.g., 2000 Monte Carlo simulations for the project it is thus needed to run 2000 simulation for each mode and thereafter combine the outcomes. As described previously the parameters/distribution chosen relies on the reference classes. Hence, there must be similarities between the project at hand and the references. The current presented research does not provide a general guideline on how to define the level of similarities when choosing reference scenarios and it is thus up to the project/ risk manager to decide which scenarios to include. While it could be discussed whether e.g., a railway tunnel between two islands should be included as a reference scenario evaluating a new railway on land, an assessments of what reference scenarios to include also implies that a new data fit and definition of distribution parameters is needed which requires additional work. One of the greater disadvantage of the FRA is the requirement of realised benefits (and cost) of an already finished project. This requires a post analysis which a major task in itself. At the same time post analysis of transport projects are usually made on highly successful projects and projects of complete failure while those in between are somewhat forgotten. The FRA is a useful technique to supplement a cost benefit analysis providing decision makers with a better understanding of the sensitivity of the CBA and consequence of potential cost overrun and/or wrong estimated user benefits. While a project might have a high risk of not being feasible in a FRA it must be noted that assessments only relies on the results from a CBA. Thus, other criteria’s as e.g., urban development, Environmental impact, etc. should also be considered through a more extensive analysis combining a CBA with a multi criteria analysis before concluding on the overall feasibility of a project.

Other application areas

Project scheduling

• Optimism bias also applies to time planning.