Post-Implementation Review (PIR)

Author: Jakob Kehler, February 2022

The Post-Implementation Review (PIR) (also known as Business Review, Project Audit or Post-Project Review) is a methodology that can be applied to evaluate the success of a project after it has been undertaken based on

1. the key deliverables of the project,
2. the process itself going through all life cycles of a project and
3. the delivery of the business case.

In ISO 21502, the project life cycle is defined as a "defined set of phases from the start to the end of the project". This includes the initiation of the project up to all activities including the closure of the project. Pre- and post-project activities are not included in the project life cycle. Nevertheless, a project itself may have only minimal impact on the economy, environment and society in contrast to its deliverables. Taking this into account, a PIR is a useful methodology to review projects that are characterized by its temporary nature "within a broader framework of organizational objectives and given contexts and environments". .

This article will guide project managers on how to perform high quality PIRs by giving examples of several different industries, then elaborating its benefits and limitations.

The importance of a PIR is theoretically highly acknowledged whereas due to several reasons it is often not used in a practical setting. This is not only due to a missing general framework. In today´s fast changing world, a PIR sets the foundation for knowledge management as is successes and failures can be shared to guarantee an increasing success rate in upcoming projects. .

The role of the PIR in the project life cycle

Figure 1 Relationship between project life cycle, integrated project management practices and management practices for a project according to ISO21502:2020. Figure created by Jakob Kehler based on

According to ISO 21502:2020, a project life cycle is a “defined set of phases from the start to the end of a project […] [where] the number and names of a project´s phases upon the type of project being undertaken”. To define a phase, the following categories need to be considered:

• Risks
• Control factors
• Organizational & project governance
• Nature of characteristics of the project
• Other organizational & environmental factors

Figure 2 Proposed project life cycle phases for sustainable project life cycle management in the manufacturing according. Figure created by Jakob Kehler based on

Besides their temporary nature, projects are distinguished by their uniqueness. The output of a project is a service or product which is unique and differs from all the other products or services that have yet been created. Therefore, a project is never a mere replica of another project, even though they may be more or less similar. .

In accordance with ISO 21502:2020, the project life cycle itself must be defined by the project managers whereby each phase must have a clear start and end date. In contrast to the project lifecycle, management practices for the project cover activities taken before, during and after the project as seen in Figure 1. A clear start and end date may not be defined. “`Completing a project´ is not the same thing as ending the project management process.”

Traditionally, the success of a project can be defined by the three dimensions of the Iron Triangle covering the following dimensions: Cost, Quality and Time. However, due to the temporary characteristic of the project, the deliverable may not be explicitly of a temporary nature either. The project itself may only be a small part of the project´s deliverable itself. . For this reason, the PIR goes beyond these three factors and can add significant value. Further elements and activities that play an important role for the success of a project can be more easily identified .

For example, C. Labuschagne and A. C. Brent propose in their analysis on Sustainable Project Life Cycle Management a life cycle that consists of 6 phases. The initiation phase takes place before the pre-feasibility phase and is therefore not part of the project life cycle itself. However, the PIR is seen as an important part of the project scope to end the project with an overall assessment.

Purpose

The world is changing faster and faster and becoming more complex. In recent decades, investment patterns have changed and the scope of projects has increased across a wide range of sectors. To improve project management practices, the use of PIRs is strongly recommended by practitioners to gain insights that lead to better decisions in future projects. , The importance of decision-making procedures for more complex projects has increased due to their immediate impact on a company's performance. However, in common practice, the last step is often neglected when reflecting on projects. Therefore, the PIR is usually not part of the project's life cycle.

A fast-changing environment requires changes that do not exclude current project management methods. In addition, there is increasing pressure on companies to ensure sustainable development. It is related to the fact that companies might fear the increasing accountability on social, environmental and economic impact based on their activities. This applies especially for already implemented projects but also for currently performed projects.

Sustainable Development is defined as followed:

“[A]dopting business strategies and activities that meet the needs of the enterprise and its stakeholders today, while protecting, sustaining and enhancing the human and natural resources that will be needed in the future’’

A PIR tackles above mentioned issues, but also serves several other purposes:

1. Adaptation of the project management methods currently in use, as these do not yet take sufficient account of social and environmental aspects.
2. Creating a common and clear understanding of a project life cycle and its interactions.
3. Ensuring knowledge transfer which is often seen as a critical factor for good performance.
4. Increasing the probability of success in future projects.
5. Analyzing performance of executed project.
6. Receiving a better understanding of all categories of the Johari Window.

Impact

Projects affect many different areas inside and outside an organization. Therefore - depending on the structure - a PIR can have an impact on all the different areas associated with project. For example, it it may involve a change in organizational power relations, an update of project management methods or the promotion of more realistic project assessments.

Goal: Organizational & project-specific learnings

Overall, the goal of PIRs is to

1. share a common understanding of used project management methodologies and processes in the company,
2. analyze the project´s performance,
3. benefit from gained knowledge for future projects

to ensure that the effectiveness and therefore performance is maximized. Organizations are not only enabled to maximize the effectiveness during future projects, but also to attain post-implementation effectiveness. When considering the above points, it becomes clear that the scope of a PIR goes far beyond lessons learned, which are only a part of it. In addition, analyzing the project´s performance can serve the purpose of a control mechanism to compare the actual performance with the original expectations.

Status Quo

PIRs can be an efficient tool for evaluating the performance of a project. However, instead of a common understanding, there is currently only general guidance for their application. Originally, PIRs were used in investment and construction projects. Over time, it has been observed that they are being used in other different sectors (e.g. the service sector, pharmaceutical industry). Furthermore, it is important to understand that the quality and scope depend on following two factors:

1. the company performing the PIR and
2. the project itself.

A study regarding PIRs dealing with the expenditure analysis of construction projects has found out big differences in the quality of the reviews which can be attributed to the above-mentioned factors:

• Only <40% of the participants have calculated an Internal Rate of Return (IRR) by using actual results.
• Only <25% of the participants have calculated an actual NPV.
• In general, larger companies tend to have a more sophisticated approach on investment decisions.
• Smaller companies tend to introduce PIRs or improving its existing system.

An area where PIRs potentially have a big impact is within IT projects. Often the knowledge gained is lost or not fully utilized for future projects. An evaluation of several IT projects has shown that neglecting the importance of this gained expertise has led to avoidable costs. The need for and success of this knowledge transfer has been successfully observed, for example, in the implementation of Enterprise Resource Planning (ERP) systems in companies. A review in the post-implementation phase identified problems in organizational power relations. Thus changes ensured higher organizational performance and user-friendliness.

Guidance: How to create a high quality PIR

Although there is no general framework available, common patterns can be identified in different areas regarding the application of PIRs. In the following chapters, practical & hands-on tips and tricks are given.

When to apply

Generally, a post-implementation review is carried out when the project has stabilised after a certain project phase or its implementation. For some projects (e.g., IT) it can be more difficult to define a clear start and end state than for others (e.g., construction projects). This is due to the different nature of projects and the project management methods being applied (e.g. Waterfall vs. Scrum). Nevertheless, it can be helpful to clearly define the project deliverables (milestones), as well-defined deliverables are a decisive success factor for projects.

Watch Out! That must be considered!

When conducting a PIR, there are several factors that influence the quality of the review. First of all, project team members feel a stronger sense of accountability. To ensure satisfaction and reduce dissonance, both managers and project members should be introduced to the PIR methodology before the project starts. It must also be made clear that projects are never 100% smooth and that mistakes are allowed. However, the PIR tries to identify problems to ensure a learning curve for the next projects.

Further, it is essential to understand that the quality and structure of the document highly depends on the company and project itself. Therefore, the document cannot be used and recognized as a mere checklist. Thus, it cannot be neglected that any company can benefit from a PIR, the mere presence does not imply an increasing performance. It is widely discussed in the literature that the quality of the review itself has an important influence on the performance of the organisation.

To ensure high quality PIRs, some guidelines shall be followed:

• PIR as a Project Phase

  - Already when planning a project, the PIR should have its own defined phase.

  - Start and end date shall be determined at the beginning of the project but might be adapted later.

  - The PIR shall be conducted near the end of the project to ensure that the project, the processes and the results are well remembered.

• Time & Resources

  - Appropriate time and resources should be made available. The actions taken are time-consuming (e.g. interviews, workshops) and probably even cost-intensive activities. Therefore, time & resources dedicated to the PIR must be aligned with the scope & deliverables of the project.

• Independent Reviewer

  - It is strongly recommended that the reviewing responsible is a project independent person (e.g. external) who has experience with similar projects. ,

• Stakeholder Involvement

  - Include all important stakeholders (e.g., users of a system, PMs)

• Project - not peoples - review

  - The emphasis should be on the evaluation of the system and not on the individual performance. Thus, continuous cooperation can be achieved without creating fear of the consequences.

• Documentation

  - The results must be recorded in written form.

• Presentation of the review

  - Present the results in a way that all stakeholders are aware of them. Reports may not be sufficient.

• Taking Actions

  - Within a short period of time, a plan should be implemented to improve efficiency in certain key areas.

  - If applicable, the project management may take immediate action when the system (e.g., IT project) has a direct negative impact on customers.

What shall be included in the PIR

As of today, there is no general framework that can be applied to conduct high quality PIRs. One of the reasons is the high dependance on the company and the project itself. Nevertheless, throughout projects in different areas similar focus points and guidelines can be identified that shall be included in a high quality PIR.

In general, a PIR should always include a process- and product-oriented review. Additionally, different areas that are associated with a project (e.g., stakeholder management, reporting) need to be considered.

Table 1 Literature review of categories that shall be included in a PIR. Figure created by Jakob Kehler based on , , , , ,

It includes a review of: , , , , ,

• Organization
• Project management methodology
• Human resources
• Procurement
• Project scope & planning
• Driving principles for project development
• Benefits & costs
• Effectiveness of deliverables
• Learnings
• Relationships

Example: Contracts are part of procurement: A PIR between an Asian and European contractor has found out, that cultural difference leads to a different understanding of contracts. Firstly, not clearly defined contracts most likely lead to conflicts needing resolution during the project. Secondly, in the Asia contracts are more seen as a starting point for negotiation.

Examples

To better understand how a PIR can improve the performance of projects and organizations, three examples are given below.

IT projects

PIRs have become increasingly popular in the project life cycle of IT projects. Especially in the post-implementation phase of a system there are several factors that play an essential role in a system´s success. The introduction of Enterprise Resource Planning (ERP) systems is one of the most common IT projects in companies. Although the benefits of such a system are widely recognised in the literature, a high failure rate can be observed. This is due to the complexity of the system and the associated implementation challenges and obstacles. However, when PIRs were used, a higher success rate was observed because key areas were identified and immediate action was taken. In particular the a stage model, as it is often used in system development, has been benefited greatly from it. Additionally, the quality and effectiveness of the implementation process has a direct and significant impact on the quality of the PIR.

Construction Projects

Not only, but especially in production processes, the alignment of all areas with sustainable factors is more important than ever. To identify areas that should be optimised, a PIR can help to anchor sustainability in the organisation. In contrast to IT projects, other factors such as resources play an even more important role. These include possible indicators for environmental, social and economic sustainability. Possible main criteria can be:

• Economic sustainability

  - Financial health

  - Economic performance

  - Potential financial benefits

• Environmental sustainability

  - Air resources

  - Water resources

  - Land resources

  - Mineral resources

• Social sustainability

  - Internal human resources

  - External population

  - Stakeholder participation

  - Macro social performance

Finance Review

The Financial Accounting Foundation (FAF) in the US has determined to conduct a PIR dealing with the standard setting process of the Financial Accounting Standard Board (FASB) and the Governmental Accounting Standards Board (GASB). The generic objective was to evaluate the effectiveness of the implemented standard in accounting and identify weaknesses which then can be shared with future rulemaking. Thereby, three sub-objectives have been set by the FAF:

1. Assessment of whether the selected standard process meets its stated objective
2. Evaluate the costs of implementing the processes and ongoing compliance in relation to their benefits.,
3. Providing feedback to improve the standard-setting process

Overall, several objectives of the PIR were directly addressed.

Limitations & Benefits

Like any other instrument or methodology, a PIR has its limitations and advantages. These will be reviewed in the following.

Limitations

• Uncertain & complex world

  - It was found that companies that already carry out PIRs have difficulties due to the complexity and uncertainty of the environment and circumstances of the project.

• Qualitative and quantitative Factors

  - A PIR includes quantitative and qualitative measures that can be difficult to evaluate together. Problems often arise when qualitative information is expressed in numbers.

• Review technique

  - Yet, no generic framework for PIRs exist. In addition, technical difficulties may arise. These could include the collection and separation of relevant data in different systems or the tracking and measurement of payment flows. One way to obtain high quality PIRs is through process mining. With process mining, readily available data can be evaluated without the need to elaborate data collection methods.

• High dependence on company & project

  - PIRs can be carried out in many different ways. The quality, but also the acceptance, depends strongly on the company and the project itself. For example, if one of the most important values of a company is "constructive feedback", then PIRs will most likely be strongly supported. Nevertheless, resistance is to be expected from managers who might feel their authority undermined.

• Not the Answer to everything

  - PIRs help to identify the most important problems if they are carried out in an appropriate way. However, they are not the one solution that can save a company.

• Time & Resources

  - It is difficult to allocate the right amount of time and resources. In addition, managers try to avoid conducting reviews to save costs, as they can cost up to 10% of the price of, for example, an implemented system. In addition, managers may not see the link between completed and upcoming projects and the knowledge they could gain from the review. This may be related to the fact that there is more billable work to do. .

Benefits

• Serving variety of objectives

  - In the literature, the benefits of PIRs are generally recognised as they pursue several objectives. On the one hand, it identifies (past) strengths and weaknesses. On the other hand, it is also future-oriented. Furthermore, a distinction can be made between product, project-specific and more general (organizational) areas.

• Learnings for the future

  - Lesson Learned is one of the most important outcomes. These have proved particularly useful for junior project managers or similar projects if collected e.g. in a common data base.

  - Future investment decisions improve.

• Proven to be beneficial

  - A study has found that companies that have once conducted a PIR have found that the benefits outweigh the difficulties and costs. This turns out to be especially true in a competitive environment.

• Feedback function

  - Low performance appraisals can be identified and further action taken. The original expected performance can be compared with the actual performance. When IT systems are introduced, for example, it often turns out that users only need 50 % to 75 % of the functions to get the full benefit from the system.

• Proactive Approach

  - Conducting reviews is a proactive approach to identifying strengths, weaknesses, opportunities and threats, taking into account multiple areas. In this way, unpleasant truths can be uncovered that affect a company's performance.

• Considering the product life cycle

  - The product life cycle goes beyond the project life cycle, as products can also be the outcome of a project. A PIR can link the project with the product.

Annotated bibliography

Following, the titles of the references are given. The exact sources can be found in the chapter References.

• Post Implementation Reviews - A means of Applying Knowledge: Lessons from a failed project

This paper explains the characteristics of a PIR incl. its limitations and benefits. It focuses on the explanation of the gained knowledge based on a real case: a multi-national IT project that has failed.

• ISO 21502:2020

ISO standard that provides guidance and guidelines for project management.

• Sustainable Project Life Cycle Management: the need to integrate life cycles in the manufacturing sector

This paper analyses the different impacts on the manufacturing sector. It identifies the needs for change in product and project specific areas to guarantee more sustainability. A general framework and updated project life cycle is being proposed that takes the PIR into account to link the project and product life cycle.

• Post-Implementation Reviews - Making Sure That What You Delivered Actually Works

This website offers practical guidance for project managers on what should be considered in a PIR. Plausible and reasoned focus areas are given to make the project managers aware of the diversity on how to conduct a PIR.

• The Importance of Project Post-Implementation Reviews

This paper examines why project managers shall pay attention on conduction a PIR. Further, it provides guidance based on practical experience of what shall be included why. A clearly structured and good overview is given with well visualized quantitative date. It covers a) 9 main reasons on conducting a PIR, b) why PIRs represent an important step in developing cross-functional projects and c) practical experience.

• Quality of postimplementation review for enterprise resource planning systems

The article identifies critical factors based on IT projects (Enterprise Resource Planning Systems) that contribute to a high quality PIR. A study methodology has been used to compare the implementation of ERPs in two different organizations. A draft of a potential framework is being developed.

• A Methodology with Quality Tools to support Post Implementation Reviews

This paper identifies the importance of a PIR and critically examines the limitations that are associated with it. Eight practical techniques (qualitative & quantitative) and key elements for high quality PIRs are given in the context of software process improvement.

• Post-implementation reviews of investment: Evidence from a two-stage study

This paper summarizes the findings of a two-stage study. The study has been conducted with smaller and bigger companies in the UK in the two sectors financial services and distributive industries. Even though it is from 1993 and as there is no newer study that evaluates the PIR methodology across different companies, it provides interesting insights on how companies deal differently with the post-implementation stage of a project.

• Lessons learned through process thinking and review

This paper provides hands-on guidance on what to take into account when conducing a PIR. It includes e.g., the guidance for the critical facilitator role and key areas that shall be addressed in the review.

• ERP Post Implementation Review with Process Mining: A Case of Procurement Process

This article identifies critical factors that contribute to a high quality PIR based on IT projects (Enterprise Resource Planning Systems - ERP). Challenges of ERP implementation projects are being analyzed to explain the relatively high failure rate that has been observed in the past of such projects. Based on these findings, practical guidance and benefits of PIRs are given.

• Insights from the initial post-implementation review of FIN 48

This paper analyzes the PIR process of the initial conducted PIR of the FIN48. The Financial Accounting Foundation (FAF) determined that the a) Financial Accounting Standards Board (FASB) and b) the Governmental Accounting Standards Board (GASB) standard had to be reviewed in a formal way in 2009. To do so, the FAF selected the interpretation 48 (FIN 48) of the FASB. The idea behind this review was to monitor and address upcoming problems that may arise after the project has been finished to future rulemaking.