Post-Implementation Review (PIR)

Author: Jakob Kehler, February 2022

The Post-Implementation Review (PIR) (also known as Business Review, Project Audit or Post-Project Review) is a method to evaluate the success of a project and the effectiveness of the organization's procedures and processes after the project's implementation, based on

1. the key deliverables of the project,
2. the process itself covering all project´s life cycles and
3. the delivery of the business case.

In "ISO 21502", the project life cycle is defined as a "defined set of phases from the beginning to the end of the project". This includes the initiation of the project through to all activities including project closure. Pre- and post-project activities are not included in the project life cycle, even though the project may influence further life cycles, e.g. of a product. Nevertheless, a project itself may have only minimal impact on the economy, environment and society in contrast to its deliverables. Taking this into account, a PIR is a useful methodology to review projects that are characterized by its temporary nature "within a broader framework of organizational objectives and given contexts and environments".

This article demonstrates why a PIR methodology should be carried out and provides project managers with practical guidance on how to conduct high-quality reviews, citing several examples from different industries and then explaining the benefits and limitations of the methodology.

The importance of a PIR is highly recognized in theory, but for various reasons it is often not used in practice. This is not only due to the lack of a generic framework, but also often due to missing resources to conduct the review. In today's fast-changing world, a PIR lays the foundation for knowledge management, as discovered successes and failures can be shared to ensure a higher success rate in future projects. This methodology is most suitable for project management, not for programme or portfolio management.

The role of the PIR in the project life cycle

Figure 1 Relationship between project life cycle, integrated project management practices and management practices for a project according to ISO21502:2020. Figure created by Jakob Kehler based on

According to ISO 21502:2020, a project life cycle is a “defined set of phases from the start to the end of a project […] [where] the number and names of a project´s phases upon the type of project being undertaken”. To define a phase, the following categories need to be considered:

• Risks
• Control factors
• Organizational & project governance
• Nature of characteristics of the project
• Other organizational & environmental factors

Figure 2 Proposed project life cycle phases for sustainable project life cycle management in the manufacturing according. Figure created by Jakob Kehler based on

Besides their temporary nature, projects are distinguished by their uniqueness. The output of a project is a service or product which is unique and differs from all the other products or services that have yet been created. Therefore, a project is never a mere replica of another project, even though the process and deliverables may be more or less similar. .

In accordance with ISO 21502:2020, the project life cycle itself must be defined by the project managers whereby each phase must have a clear start and end date. In contrast to the project life cycle, management practices for the project cover activities taken before, during and after the project as seen in Figure 1. A clear start and end date may not be defined. “`Completing a project´ is not the same thing as ending the project management process.”

Traditionally, the success of a project can be defined by the three dimensions of the Iron Triangle covering the following dimensions: Costs, quality and time. However, due to the temporary characteristic of the project, the deliverable may not be explicitly of a temporary nature either. In other words, the project itself may only be a small part of the project´s deliverable itself. . For this reason, the PIR goes beyond these three factors and can add significant value. Further elements and activities that play an important role for the success of a project can be more easily identified.

For example, C. Labuschagne and A. C. Brent propose in their analysis on Sustainable Project Life Cycle Management a life cycle that consists of 6 phases. The initiation phase takes place before the pre-feasibility phase and is therefore not part of the project life cycle itself. However, the PIR is considered an important phase of the life cycle to complete the project with an overall assessment.

Purpose

The world is changing faster and faster and complexity is increasing. In recent decades, investment patterns have changed and the scope of projects has increased across a wide range of sectors. To improve project management practices, the use of PIRs is strongly recommended by practitioners to gain insights that lead to better decisions in future projects. , The importance of decision-making procedures for more complex projects has increased due to their immediate impact on a company's performance. However, in common practice, the review is often neglected. Therefore, the PIR is usually not part of the project's life cycle.

A fast-changing environment requires changes that do not exclude current project management methods. An increasing pressure is put on companies to ensure a sustainable development. Amongst others, it is due to the fact that companies might fear the increasing accountability on social, environmental and economic impact based on their activities. This applies not only to future projects, but also to projects that have already been implemented.

Definition of Sustainable Development:

“[A]dopting business strategies and activities that meet the needs of the enterprise and its stakeholders today, while protecting, sustaining and enhancing the human and natural resources that will be needed in the future.’’

A PIR tackles above mentioned issues, but also serves several other purposes:

1. Adapting project management methods currently in use, as these do not yet take sufficient account of social and environmental aspects.
2. Creating a common and clear understanding of a project life cycle and its interactions.
3. Ensuring knowledge transfer which is often seen as a critical factor for good performance.
4. Increasing the probability of success in future projects.
5. Analyzing performance of executed project.
6. Receiving a better understanding of knowns and unknowns (see Johari Window).

Impact

Projects affect the internal and external environment of an organization. Therefore - depending on the structure - a PIR can have an impact on all the different areas associated with project. For example, it it may involve a change in organizational power relations, an update of project management methods or the promotion of more realistic project assessments.

Goal: Organizational & project-specific learnings

Overall, the goal of PIRs is to

1. share a common understanding of used project management methodologies and processes in the company,
2. analyze the project´s performance and
3. to analyze the benefit from gained knowledge for future projects

to ensure that the organization´s effectiveness and performance is maximized. Organizations are not only enabled to maximize the effectiveness during future projects, but also to attain post-implementation effectiveness. When considering the above points, it becomes clear that the scope of a PIR goes far beyond lessons learned, which are only a part of it. In addition, analyzing the project´s performance can serve the purpose of a control mechanism to compare the actual performance with the original expectations.

Status Quo

PIRs can be an effective methodology for evaluating the performance of a project. However, instead of a common understanding, there are currently only general guidelines for its application. Originally, PIRs were used in investment and construction projects. Over time, it has been observed that they are being used in other different sectors (e.g. the service sector, pharmaceutical industry). Furthermore, it is important to understand that the quality and scope depend on following two factors:

1. the company performing the PIR and
2. the project itself.

A study regarding PIRs dealing with the expenditure analysis of construction projects has found out big differences in the quality of the reviews which can be attributed to the above-mentioned factors:

• Only <40% of the participants have calculated an Internal Rate of Return (IRR) by using actual results.
• Only <25% of the participants have calculated an actual NPV.
• In general, larger companies tend to have a more sophisticated approach on investment decisions.
• Smaller companies tend to introduce PIRs or improving its existing system.

An area where PIRs potentially have a big impact is within IT projects. Often the knowledge gained is lost or not fully utilized for future projects. An evaluation of several IT projects has shown that neglecting the importance of this gained expertise has led to avoidable costs. The need for and success of this knowledge transfer has been successfully observed, for example, in the implementation of Enterprise Resource Planning (ERP) systems in companies. A review in the post-implementation phase identified problems in organizational power relations. Thus changes ensured higher organizational performance and user-friendliness.

Guidance: How to create a high-quality PIR

Although there is no general framework, common patterns for the use of PIRs can be identified in different areas. In the following chapters, practical & hands-on tips and tricks are given.

When to apply

Generally, a post-implementation review is carried out when the project has stabilized after a certain project phase or its implementation. For some projects (e.g., IT) it can be more difficult to define a clear start and end state than for others (e.g., construction projects). This is due to the different nature of projects and the project management methods being applied (e.g. waterfall vs. scrum). Nevertheless, it can be helpful to clearly define the project deliverables (milestones), as well-defined deliverables are a decisive success factor for projects. Based on the research conducted, it must be said that a PIR is a methodology and not a specific tool. Therefore, the instruments for the various categories to be reviewed must be carefully selected to produce a high-quality PIR. Useful tools can be found here.

Watch Out! That must be considered!

When conducting a PIR, there are several factors that influence the quality of the review. First of all, project team members feel a stronger sense of accountability. To ensure satisfaction and reduce dissonance, both managers and project members should be introduced to the PIR methodology before the project starts. It must also be made clear that projects are never 100% smooth and that mistakes are allowed. However, the PIR tries to identify problems to ensure a learning curve for the next projects and increase performance.

Further, it is essential to understand that the quality and structure of the document highly depends on the company and project itself. Therefore, the document cannot be used and recognized as a mere checklist. Thus, it cannot be neglected that any company can benefit from a PIR. The mere presence does not imply an increasing performance. It is widely discussed in the literature that the quality of the review itself has an important influence on the performance of the organization.

To ensure high-quality PIRs, some guidelines shall be followed:

• PIR as a Project Phase

  - Already when planning a project, the PIR should have its own defined phase.

  - Start and end date shall be determined at the beginning of the project but might be adapted later.

  - The PIR shall be conducted near the end of the project to ensure that the project, the processes and the results are well remembered.

• Time & Resources

  - Appropriate time and resources should be made available. The actions taken are time-consuming (e.g. interviews, workshops) and probably even cost-intensive activities. Therefore, time & resources dedicated to the PIR must be aligned with the scope & deliverables of the project.

• Independent Reviewer

  - It is strongly recommended that the reviewing responsible is a project independent person (e.g. external) who has experience with similar projects. ,

• Stakeholder Involvement

  - Include all important stakeholders (e.g., users of a system, PMs)

• Project - not peoples - review

  - The emphasis should be on the evaluation of the system and not on the individual performance. Thus, continuous cooperation can be achieved without creating fear of the consequences.

• Documentation

  - The results must be recorded in written form.

• Presentation of the review

  - Present the results in a way that all stakeholders are aware of them. Reports may not be sufficient.

• Taking Actions

  - Within a short period of time, a plan should be implemented to improve efficiency in certain key areas.

  - If applicable, the project management may take immediate action when the system (e.g., IT project) has a direct negative impact on customers.

What shall be included

As of today, there is no general framework that can be applied to conduct high-quality PIRs. One of the reasons is the high dependance on the company and the project itself. Nevertheless, throughout projects in different areas similar focus points and guidelines can be identified that shall be included in a high-quality PIR.

In general, a PIR should always include a process- and product-oriented review. Additionally, different areas that are associated with a project (e.g., stakeholder management, reporting) need to be considered.

Table 1 Literature review of categories that shall be included in a PIR. Figure created by Jakob Kehler based on , , , , ,

It includes a review of: , , , , ,

• Organization
• Project management methodology
• Human resources
• Procurement
• Project scope & planning
• Driving principles for project development
• Benefits & costs
• Effectiveness of deliverables
• Learnings
• Relationships

However, based on the references it can be argued, that the scope of a PIR may not be limited to these categories. When considering the different standards (e.g. ISO or PMI), a project must consider both - internal and external environments. Therefore, the standards can provide further categories such as:

• Internal environment: Infrastructure, Resource availability or Employee capability.
• External environment: Marketplace conditions, regulatory environment or industry standards.

Example: Contracts are part of procurement: A PIR between an Asian and European contractor has found out, that cultural difference leads to a different understanding of contracts. Firstly, not clearly defined contracts most likely lead to conflicts needing resolution during the project. Secondly, in Asia contracts are more seen as a starting point for negotiation.

Examples

To better understand how a PIR can improve the performance of projects and organizations, three examples of different industries are given below.

IT projects

PIRs have become increasingly popular in the project life cycle of IT projects. Especially in the post-implementation phase of a system there are several factors that play an essential role in a system´s success. The introduction of Enterprise Resource Planning (ERP) systems is one of the most common IT projects in companies. Although the benefits of such a system are widely recognized in the literature, a high failure rate can be observed. This is due to the complexity of the system and the associated implementation challenges and obstacles. However, when PIRs were used, a higher success rate was observed because key areas were identified and immediate action was taken. In particular, a stage model, such as is often used in system development, has benefited greatly from this. Additionally, the quality and effectiveness of the implementation process has a direct and significant impact on the quality of the PIR.

Construction Projects

Not only, but especially in production processes, the alignment of all areas with sustainable factors is more important than ever. To identify areas that should be optimized, a PIR can help to anchor sustainability in the organization. In contrast to IT projects, other factors such as resources play an even more important role. These include possible indicators for environmental, social and economic sustainability such as:

• Economic sustainability

  - Financial health

  - Economic performance

  - Potential financial benefits

• Environmental sustainability

  - Air resources

  - Water resources

  - Land resources

  - Mineral resources

• Social sustainability

  - Internal human resources

  - External population

  - Stakeholder participation

  - Macro social performance

Finance Review

The Financial Accounting Foundation (FAF) in the US has determined to conduct a PIR dealing with the standard setting process of the Financial Accounting Standard Board (FASB) and the Governmental Accounting Standards Board (GASB). The generic objective was to evaluate the effectiveness of the implemented standard in accounting and identify weaknesses which then can be shared with future rulemaking. Thereby, three sub-objectives have been set by the FAF:

1. Assessment of whether the selected standard process meets its stated objective
2. Evaluate the costs of implementing the processes and ongoing compliance in relation to their benefits.
3. Providing feedback to improve the standard-setting process

Overall, several objectives of the PIR were directly addressed and recommendations were given to future rulemaking.

Limitations & Benefits

Like any other instrument or methodology, a PIR has its limitations and benefits. These will be reviewed in the following.

Limitations

• Uncertain & complex world

  - It was found that companies that already carry out PIRs have difficulties due to the complexity and uncertainty of the environment and circumstances of the project.

• Qualitative and quantitative Factors

  - A PIR includes quantitative and qualitative measures that can be difficult to evaluate together. Problems often arise when qualitative information is expressed in numbers.

• Review technique

  - Yet, no generic framework for PIRs exist. In addition, technical difficulties may arise. These could include the collection and separation of relevant data in different systems or the tracking and measurement of payment flows. PIR is a methodology. Different tools can be used to evaluate the different categories. For example, one way to obtain high-quality PIRs is through process mining. With process mining, readily available data can be evaluated without the need to elaborate data collection methods.

• High dependence on company & project

  - PIRs can be carried out in many different ways. The quality, but also the acceptance, depends strongly on the company and the project itself. For example, if one of the most important values of a company is "constructive feedback", then PIRs will most likely be strongly supported. Nevertheless, resistance is to be expected from managers who might feel their authority undermined.

• Not the answer to everything

  - PIRs help to identify the most important problems if they are carried out in an appropriate way. However, they are not the one solution that can save a company.

• Time & resources

  - It is difficult to allocate the right amount of time and resources. In addition, managers try to avoid conducting reviews to save costs, as they can cost up to 10% of the price of, for example, an implemented system. In addition, managers may not see the link between completed and upcoming projects and the knowledge they could gain from the review. This may be related to the fact that there is more billable work to do. .

Benefits

• Serving variety of objectives

  - In the literature, the benefits of PIRs are generally recognized as they pursue several objectives. On the one hand, it identifies (past) strengths and weaknesses. On the other hand, it is also future-oriented. Furthermore, a distinction can be made between product, project-specific and more general (organizational) areas.

• Learnings for the future

  - Lesson Learned is one of the most important outcomes. These have proved particularly useful for junior project managers or similar projects if collected e.g. in a common data base.

  - Future investment decisions improve.

• Proven to be beneficial

  - A study has found that companies that have once conducted a PIR have found that the benefits outweigh the difficulties and costs. This turns out to be especially true in a competitive environment.

• Feedback function

  - Low performance appraisals can be identified and further action taken. The original expected performance can be compared with the actual performance. When IT systems are introduced, for example, it often turns out that users only need 50% to 75% of the functions to get the full benefit from the system.

• Proactive approach

  - Conducting reviews is a proactive approach to identifying strengths, weaknesses, opportunities and threats, taking into account multiple areas. In this way, unpleasant truths can be uncovered that affect a company's performance.

• Taking the product life cycle into account

  - The product life cycle often goes far beyond the project life cycle, as products can be the outcome of a project. Therefore, a PIR can link the project with the product.

Annotated bibliography

Following, the titles of the most relevant references are given. The exact sources can be found in the chapter References.

• Post Implementation Reviews - A means of Applying Knowledge: Lessons from a failed project

This paper explains the characteristics of a PIR incl. its limitations and benefits. Firstly, it provides background information on IT projects and knowledge management in general. Then it explains the basic principles of a PIR (e.g., what to include and consider). The majority of the paper focuses on the analysis of a real case - a failed multi-national IT project – to provide insights on the methodology of a PIR and how it may be applied.

• Sustainable Project Life Cycle Management: the need to integrate life cycles in the manufacturing sector

This paper analyzes the various impacts of the rapidly changing and complex environment on the manufacturing sector. It highlights the need for changes in product and project specific areas based on the pressures placed on organizations to ensure greater sustainable development. A general framework for PIRs in the manufacturing sector and a general updated project life cycle are proposed. The new project life cycle considers PIRs to link the project and product life cycles and ensure greater performance in future projects.

• Post-Implementation Reviews - Making Sure That What You Delivered Actually Works

This website provides project managers with practical hands-on guidance on what should be considered in a PIR. Plausible focus areas are identified and justified to make project managers aware of the variety of ways to conduct a PIR. While not a scholarly treatise, the website provides practical guidance, making it a valuable resource as current literature tends to focus on specific advice for particular industries rather than general guidance.

• The Importance of Project Post-Implementation Reviews

This paper examines why project managers shall pay attention on conduction a PIR. Further, it provides guidance based on practical experience of what shall be included and why. A clearly structured and good overview is given with well visualized quantitative data. It covers a) 9 main reasons on why conducting a PIR, b) why PIRs represent an important step in developing cross-functional projects and c) practical experience by applying the methodology.

• Quality of postimplementation review for enterprise resource planning systems

This paper focuses on the post-implementation phase of an enterprise resource planning (ERS) system (IT project). It evaluates the review process and identifies critical factors that contribute to a high-quality reviews. A study methodology was used to compare the implementation of ERP systems in two different organizations. Using the PIR methodology, a potential framework for ensuring high effectiveness in ERP implementation is developed.

• A Methodology with Quality Tools to support Post Implementation Reviews

This paper highlights the importance of a PIR as a cost-effective tool for improving software processes. The underlying methodology of a PIR is explained and related to the tools that may be required to produce high-quality reviews. The limitations are critically examined. Several practical techniques (qualitative and quantitative) and key elements for high-quality PIRs in the context of software process improvement are also presented.

• Post-implementation reviews of investment: Evidence from a two-stage study

This paper summarizes the results of a two-stage study on the use of PIRs in different companies in the UK. Smaller and larger companies in the two sectors "financial services" and "trading companies" were investigated. The study offers interesting insights into the different ways in which companies deal with the post-implementation phase of a project, even though it dates back to 1993. Furthermore, there is no recent study that evaluates the PIR methodology for different companies which proves the relevance of this source.