Evaluation of project success

Project Evaluation is the focus on studying projects in order to evaluate what traits associated with the project. The main objective is twofold, where the first objective is evaluate the successful attributes of a project. The second objective can be expressed by Cobb's Paradox - "We know why projects fail; we know how to prevent their failure – so why do they still fail?". This is evaluation of traits that are undesired or byproducts of poor project management. This duality of project management is often described as dealing with a chaordic system , which implies there is both order and chaos in projects. The goal of the project manager is to attempt to use both these two warring forces to create a successful project.

The evaluation can not be done without a baseline of knowledge to base the evaluation off. The first objective is therefore to define how to evaluate both successful and failing traits of a project. This is done in a very general way, but will provide links to more in depth descriptions of most tools in this article. It is also important to point out that this is not just relevant to project management but also contributes toward certain important task in portfolio management that will be explored later.

Project success & failure

Introduction

To evaluate a project and the level of success or failure is not as straight forward as one would expect. Success and failure can be perceived substantially differently, depending on whether you are a project participant or a stakeholder in the project. For an internal actor of the project the success criteria is being able to meet deliverables on time and living up to certain Key Performance Indicators (KPI's) set by senior management. For external stakeholders success is very different. Here the value generated by the project is often more important. It is important to involve these stakeholders so their expectations are met. Even if a project is a success internally, it can be a failure if users of the output of the project do not feel the project aligns with their interests. The goal of this section is to highlight what are common ways of measuring success internally and externally. A later section will look at the drivers for failure.

Dimensions defined to evaluate project management success.

Achieving success

This first section focuses on how to evaluate project elements that contribute towards successfully executing a project.

Measuring project success

One of the traditional ways to self asses is by the Iron Triangle. The Iron triangle consist of three ground pillars and is used to set KPI's. Companies use this tool to self asses whether a project is living up to the estimated expectations.

1. Time
   All projects are limited by time, and deliveries must fall with in the expected time frame in order to be considered a success.
2. Cost
   Projects are often projected to have a certain cost and hence receive a budget based on this estimation of cost. Often competing projects bid for the same budget funding. This causes the estimators to lower their budget expectations, which in turn leads to lowering the quality of the project.
3. Quality
   The previous two legs focus on how the project is executed, while quality focuses on the output of the project. The higher the quality the higher the value generation of the project.

In the triangle it is impossible to run a project that produces maximum value in all three perspectives as they coexists as trade-offs between each element.

Major dimensions of project success, each containing a wide variety of complex elements.

The Iron triangle is however, a simplification of where success is derived from. In some cases more dimensions are added to the iron triangle to give a more complex but fulfilling perspective. Other suggestions from the literature council a different approach to success . It mentions three more complex but accurate pillars of project success.

1. Delivering Value
   The Iron Triangle falls into this category as the aspects of delivering value. This however also encapsulates things such as scope and how the project realizes the appropriate conditions for it to come to fruition.
2. Managing Risk
   Uncertainties is a big part of projects. The objective of a project is to try and form the future in a desired way, but often unexpected events occur that makes the future deviate from the initially chosen path. It is the project managers task to either reign the in the deviance, or choose to continue down this new path. This entire subject will be discussed later in the article.
3. Managing Relationships
   The last tenet is managing the stakeholders related to the project. The first step of doing that is defining who are the stakeholders. According to the ISO standard anyone that has interest in, or can affect, be affected, or perceive themselves to be affected are the stakeholders. This is a wide variety of stakeholders, and it is often important task to juggle. This task can however quickly eat up almost all of the project managers time. It is therefore often also important to acknowledge who are the most important stakeholders as they need the most attention. This will be discussed in the next section.

These principles of a successful project give a better perspective on how to achieve project success. The downside is a much higher level complexity, as can be seen by The Iron Triangle. It is just a tool within a category, and a wide variety of tools can be found in all of the categories. The focus should be on the value of both simple and complex tools. The tools such as The Iron Triangle provide the How, while the boarder and more complex tenets tends to the Why. The benefit of understanding the How and the Why is what create succesful projects. The next sections focus on detailing what is the focus of each section.

Stakeholder management

Undertaking projects means creating a temporary organization . Stakeholders in stakeholder management are defined as anyone that can be affected by the activities undertaken by this organization. It is therefore important to engage stakeholders in an appropriate fashion . The appropriate level of engagement means a need for the project manager to identify the level of engagement from each stakeholders. Some with high level of interest and power will be key players while other will only be observers and should only be monitored. If key players are misjudged and not consulted enough, they can cause troubles for any given project. The goal here is to soothe concerns and engage with actors that can benefit the project.

Risk management

Risk managementis a focus on the likelihood that a given project achieves it's objectives . This refers both to minimizing potential risks while capitalizing on potential opportunities . Each project risk identified also needs to be analyzed and have an action plan attached to it. To act upon identified risks and opportunities goes hand in hand with stakeholder management, as posed risk to the project easily can come from human causes. Other risks as environmental changes such as economic collapse or natural disasters also needs to be considered. Equally to stakeholder management it is often chosen to focus on the most likely risks and opportunities.

Summary of achieving success

Each of the previous mentioned subjects are all important factors in why a project is successful. It is not enough to master just one of the aspects as it is the joined effort of all them that creates a truly successful project. The need to manage all of them at the same time is one of the key drivers of complexity. Equally the explanation given in this article cannot convey each area in a complete fashion, as they each have multiple layers that should be explore further - this can be done in the Conceptbox . The tools provided there are all constructed so the project manager have multiple ways of approaching each of the subjects. The need for multiple tools stems from the need to contain the level of complexity, and each tool provides a different insight into how this can be done.

Another important point is also that project success is tied to stakeholders. Any project that meets time expectations and budget are not by necessarily a success if they fail to meet stakeholder expectations. Imagine a selling a project such as a house that was made just to be using cheap materials and delivered within the time frame of 3 months. Likely this kind of house would not be very successful in selling, "cheap" and "fast delivery" is not often attributes that people look for in houses. It is therefore important to understand what the customer wants, that this might be different from what you also want as a person.

Factors of project failure

The previous section provides a well rounded but simplistic approach of how to be successful, so why do we often still see failures happen? Projects fail because of a multitude of reasons such as overambition, not adapting to the new situation, unexpected events occurring, etc. These projects fail because of biases, miscommunication, and mismanaged risks, along with a slew of unexpected events. This section will cover what are some of the common mistakes made in project management. These mistakes are tightly tied to the successes as it is often because of no management or mismanagement of value, risk and stakeholder management that projects fail.

Cognitive Bias

One major reason why people mismanage their projects is because of some kind of bias. Biases comes in many different shades, but to have a bias generally means that you have some sort of blind side where your logic is flawed. Because of these flaws managers either undervalue, overvalue, or misinterpret data collected. The list of possible biases is very long and not all can be stated here but three common biases are stated here.

Survivorship Bias

Survivorship bias identifies that selection of data can also be impacted by our intuition. An often used example of this bias is that of air plane returning during the second world war. Here the allies wanted to enforce the airplanes and were screening each returning plane for bullet holes, they then recommended to enforce the areas where there were no bullet holes at all. This was done as planes hit in these areas were not returning and hence did not survive the selection process of flying over enemy territory. This is a great example of how to actually consider the survivorship bias and counter it, but in other cases this detail has not been attended to. In some surveys of companies, failed organizations have been dropped as they no longer exists, and this is also survivorship bias. It leads to misinterpretation of what is happening within that survey. The importance of choosing the right kind of data can impact your ability to make decisions based on complete or incomplete data.

Dunning-Kruger Effect & Imposter Syndrome

One of the bias that can in pact a project is the Dunning-Kruger Effect . This bias is found mostly in people that have little to no knowledge of a certain field of expertise, where the individual with no skills highly over estimate their ability within the field of expertise. People with a high level of knowledge, tends to have the opposite tendencies, where they develop the imposter syndrome and doubt their own ability. These two effects can in tandem ruin projects, as people who know nothing will forcefully exert their will on the project, while people with a high level of knowledge will back off as they fear they might be wrong. If the project manager does not identify who within their team processes true knowledge within a field, they can be led astray an focus on the wrong aspects of a project. The insight gained here, is that projects can fail if participants of the project are not also treated as stakeholders, and the project manager considers all of their concerns and not just the most vocal members.

Diagram to exemplify what the difference between effectiveness and efficiency. The major difference is whether or not a certain output contributes towards what ever the goal of the project is.

Suboptimization

As organizations often need to resolve issue within a certain timeframe, efficiency is often being measured. This is often done via the earlier mentioned KPI's, as they provide tangible and measurable goals for the members of the organization. Here it is important to understand the difference between efficiency versus effectiveness. Efficiency is mainly just the focus of doing something fast, whereas effectiveness is how much a certain task benefits goal completion. The issues related to KPI's and measuring is it puts a focus on "doing things right" - Peter Drucker. This is inherently not an issue, but rarely are the tasks that projects set out to solve simple, and this complexity makes it hard to define metrics that are well defined for what the project goals are. In other words it is hard to do the right things - Peter Drucker.

A common example here is that of call centers. They generally have a metrics for length of calls and of customer satisfaction. The goal here is for the worker to give good service over a short period of time. The challenge is knowing what is good customer service, whereas tracking the length of a call is very tangible and measurable objective. This can lead project managers to over stimulate their workers to have even shorter calls, which in turn leaves the customers unsatisfied. This creates an unintended effect of measuring, and is something to constantly needed to be managed.

Risk management

Risk management fails when risks were not managed or not within the control of the project manager. As risk management was also a part of the success section it is because one thing is to consider and plan for the risks, and another thing is them happen excatly as thought out by the project team. Even initially successful projects, that manage their risk optimally, can fall prey to unexpected circumstance, and that circumstance can cause the project to collapse. The effect of this could be that senior management might lose faith and start to squeeze the budget, while maintaining their required goals for the project. This effectively ensure that the project will fail, as the initial scope cannot be maintained because they fell behind due to the circumstance, or they simply fall behind due to lack of funding. Risk implies there is a outcome that has negative consequences for the project. When dealing with negative consequences it can vary quite abit the level of risk willingness of the project stakeholders. Naturally it is an important asset to manage, and even if the risk taken along fall out in favor of the project manager, the stakeholders might feel their risk tolerance have been exceeded and they no longer wish to participate in the project or percieve the solutions negatively because of this mistreatment.

Summary of factors for project failure

Given that some of the factors in this article can be described as key drivers of failure, an entire other article could outline drivers of failure in greater detail. Like project success it is very individual to each project what are the driving factors of failure. Controling the factors that produce failure can also be described as trying to control the chaos within the system that is project management. Going back to the idea of a chaordic system that allowed both chaos and order to be a part of a system, it becomes clear in this section what is actually meant by this concept. While the main focus of the success section focuses on planning and consideration of all the possible actions that can occur within a problem, the section on failures focuces on what happens within a system of chaos that is the present time we live in. The benefit of this section is hopefully an idea of why failures can occur. It is also to understand that projects that do fail should not just be disregarded, but be analysed in order to understand how to become better at discern what is actually impacting projects and how to deal with it.

The largest conflict with managing failures is found within time. Time is the limiting factor that drives the project manager to make choices without achieving perfect information . While planning is nice and what makes it easier to actually act fast, not everything can be considered even in successful projects. Often the analysis of preivous projects is put on hold because they are often viewed as concluded, and the organization moves on to other tasks that needs doing. The encouragement from this article is to learn to reflect on previous projects, as it could benefit projects in the future.

Successful failures & failing successes - limitations of project evaluation

An interesting observation found in the literature is project perception can change over time. The focus of this section is an exmaple of a project that failed initially but over time became success, and a project that initially were a success and then became a failure.

In this section a part of the objective is to show how to evaluate a project based on previous described measurements for failure and success. This will naturally be marked by some error since this article is not done by anyone involved with any of the projects, and based off information found online. This is however done deliberately, in order to show one of the main short comings of project evaluation, and will be the main focus in this topic.

A side note here is that all of the projects are construction based projects. This is done purely by accidents. It is however also an assumption that evaluation of construction projects have been measured in success for a longer period of time, than projects with no tangible deliverable such as projects focused on cultural change within an organization.

Millennium Dome - successful failure

The Millennium Dome was a project that began in 1997, and was finished in 1999. It was constructed on time, and with added funds from the government within budget. However the increase in budget also meant an increase in scope. At the opening of the dome Tony Blair said that the dome was triumph of confidence over cynicism . This is when things started to go sideways. The project was initially meant to be an attraction, but only managed to attract half of the projected customers expected. The dome was only open a year before it was closed based on the initial concept of The Millennium Dome being a celebration of changing into a new millennium. In the early 2000s the dome was only used for sparingly, and presented itself as a huge maintenance cost which marked it in the media landscape as a huge failure. This lasted until in the late 2000s it was redeveloped into the O2 arena.

This is an example of project delivering a result where the stakeholders has not been considered. The project was preformed as a top-down push which equally can be seen in the increase in funds. The objective of the project participants was clearly a focus on meeting the deadline and stay within budget. The How of the project was very well thought out and cannot be denied to have succeeded. The Why had been considered to a lesser extend, and it is unclear why they were so certain there would a larger interest in the Millennium Dome. Clearly they estimations done on The Millennium Dome was either biased expecting people to buy in on this idea of a mega attraction such as world expos , or focused on meeting ambitious demands set by the project managers and forgetting to consider stakeholders. Either way the outcome was that a project that initially found success turned into a failure over time because of unconsidered aspects.

Sidney Opera House - failing success

The Sidney Opera house was under construction from 1959 to 1973 which is a fairly long time for a construction to be built in. This was mostly due to the falling out between the Danish Architect Jørn Utzon and the government who approved and were the client of the project. The falling out was mainly due to Jørn's desire to be as perfect and close to his concept idea of how he wanted to create the opera house, while many of the politicians felt it trying that Utzon was unwilling to divert from the original plan. Another issue was due to the fact that the initial estimation of cost was very optimistic in order to get the approval in the first place, and as the cost started to outgrow expectations the government started to have doubts about the project. This split grew over time until Utzon decided to resign from the project all together. After the split it was given to another danish building contractor that finished the project.

Given that this project initially suffered through major crisis, and the government underway considered cutting the project completely, The Sidney Opera house is a major success today and a landmark that is often seen as the face of Sidney. This is the flipside story of The Millennium Dome. The interesting part of this tale is that the Sidney Opera House was handled in multiple steps instead creating an unsatisfying solutions for the stakeholders. While the project of creating the opera house can be conceived as project split into two section. The benefit of running the initial unsuccessful project was that the ideas generated and were tried and tested and shown to not work. This allowed the new project group to come up with other ideas that the client was satisfied by.

Limitations

In relation to the previous two section it is important to also state that this analysis is made as a bystander and the reason for the successes and failures related to both projects are more complex than stated here, and more in-depth analysis have been made. The are also done in retrospect and the risk and uncertainies that each projects faced are easier to discount because of this time perspective.

The time aspect is still very important and the single most major take away from this section, and can be stated as follows. Project success and failures are time relative. This is likely the single most difficult thing about evaluation of projects. It is understandable that it should create some unease. The fact that evaluation of projects should be done regularly and is not necessarily going to give the same outcome if done at two different times is interesting to say the least. It is however important to maintain that whatever knowledge is derived from reflection is not less valuable in either case. The main take away should be that there is always more to learn, and what is not an issue today can be an issue tomorrow. Even though projects are short lived and results created by them are often only considered by the project participants it should be evaluated nonetheless.

Application for program & portofolio management

Expectation of how to translate strategic goals into tangible project goals.

Reality of what often happens when executing projects.

While project evaluation intuitively should benefit project management, it is actually a better tool for program or portofolio management. The reason is that evaluation is not undertaken until after a project has been executed. The benefit is not directed towards projects as they are only very temporary constructs, and cease to exist after the set deliverables are produced as the projects output. Naturally some evaluation will take place post project, however these consideration cannot benefit the project anyway, but only contribute towards new upcoming projects. It does however benefit program and portofolio management. This is the case since it is both in the interest of program and portofolio managements, to consider whether the strategic objectives are aligned with the program or portofolio strategic management. Lynda Bourne suggest in her work The paradox of project control that the strategic objective formulated at the highest level of management often is lost through the middle management. The objective or the Why formulated at high levels is often lost in the translation to the How. Many of the issues related to project management is because the translation tools that are available, such as KPI's or the stakeholder analysis, do not align themselves hundred percent with the strategic objective and can lead to suboptimization or other negative effects. Evaluation as a tool comes into play to consider what should be done in order to get better alignment. Evaluation is in other words a tool used to smooth out the translation of strategic goals to strategic action. This is also why in large portofolios mutiple projects will be run with the same aim, so they can provide feedback to each other to generate better solutions.

Annotated biblography

Bourne, Lynda & Walker, Derek. (2005). "The paradox of project control" - This was the idea starter of the project. This focuses on this middle area where alot of things happen in middle management, where there is both planning and unexpected events. This whole idea of continously updating your expectations is exactly the focus of learning. The fundemental advantage of actually trying to learn is that you become better at recognizing certain mistakes or know how to take advantage of certain settings to create better projects.

Bourne, L. (2007). Avoiding the successful failure. Paper presented at PMI® Global Congress 2007—Asia Pacific, Hong Kong, People's Republic of China. Newtown Square, PA: Project Management Institute. - Initially when writing this article the focus was to find the ability to say whether or not a project is a success or not. As a new practitioner in the field of management science you learn a wide arrange of tools for your conceptbox, and you get shown examples of succesful projects and failures. It seemed however an often leftout topic was how to evaluate the project you just finished was less of a topic. It was only after speaking with Lynda Bourne and her pointing out that it could change over time. This article holds a huge thanks to her for making it even better and gaining another dimension.

Carlton, Darryl. (2017). "Competence versus Confidence in IT Project Leadership and its Impact on Project Outcomes". - One of the starting points in the failure section was the biases. I have heard of them in other courses but i needed to find an article that showed some prominent biases.

Eijnatten, Frans M.. (2004). Chaordic Systems Thinking: Some Suggestions for a Complexity Framework to Inform a Learning Organization. - During my writing i felt i often ran into contradicting elements in project management. You try and plan so you do not fail, but we often do. This idea of pushing order into chaos is also a major idea of why we do project, program and portofolio management. I found the concept of chaordic system intriguing because it presented a sitatuation that seemed closer to reality, as we plan via management but still need to deal with randomness and react fast in certain sitatuations. If you only plan and is unprepared or unwilling to take actions when things do not follow the laid out plan you are going to fail anyway.