Post-Implementation Review (PIR)

Author: Jakob Kehler, February 2022

The Post-Implementation Review (PIR) (also known as Business Review, Project Audit or Post-Project Review) is a methodology that can be applied to evaluate the success of a project after it has been undertaken based on

1. the key deliverables of the project,
2. the process itself going through all life cycles of a project and
3. the delivery of the business case.

In ISO 21502, the project life cycle is defined as a "defined set of phases from the start to the end of the project". This includes the initiation of the project up to all activities including the closure of the project. Pre- and post-project activities are not included in the project life cycle. Nevertheless, a project itself may have only minimal impact on the economy, environment and society in contrast to its deliverables. Taking this into account, a PIR is a useful methodology to review projects that are characterized by its temporary nature "within a broader framework of organizational objectives and given contexts and environments". .

This article will guide project managers on how to perform high quality PIRs by giving examples of several different industries, then elaborating its benefits and limitations.

The importance of a PIR is theoretically highly acknowledged whereas due to several reasons it is often not used in a practical setting. This is not only due to a missing general framework. In today´s fast changing world, a PIR sets the foundation for knowledge management as is successes and failures can be shared to guarantee an increasing success rate in upcoming projects. .

PIR as part of the project life cycle

Figure 1 Relationship between project life cycle, integrated project management practices and management practices for a project according to ISO21502:2020. Created by Jakob Kehler

According to ISO 21502:2020, a project life cycle is a “defined set of phases from the start to the end of a project […] [where] the number and names of a project´s phases upon the type of project being undertaken”. To define a phase, the following categories need to be considered when defining the project life cycle:

• Risks
• Control factors
• Organizational & project governance
• Nature of characteristics of the project
• Other organizational & environmental factors

Figure 2 Proposed project life cycle phases for sustainable project life cycle management in the manufacturing according to . Created by Jakob Kehler

Besides their temporary nature, projects are distinguished by their uniqueness. The output of a project is a service or product which is unique and differs from all the other products or services that have yet been created. Therefore, a project is never a mere replica of another project, even though they may be more or less similar. .

In accordance with ISO 21502:2020, the project life cycle itself must be defined by the project managers whereby each phase must have a clear start and end date. In contrast to the project lifecycle, management practices for the project cover activities taken before, during and after the project as seen in Figure 1. A clear start and end date may not be defined. “`Completing a project´ is not the same thing as ending the project management process.”

Traditionally, the success of a project can be defined by the three dimensions of the Iron Triangle covering the following dimensions: Cost, Quality and Time. However, due to the temporary characteristic of the project, the deliverable may not be explicitly of a temporary nature either. The project itself may only be a small part of the project´s deliverable itself. . For this reason, the PIR goes beyond these three factors and can add significant value. Further elements and activities that play an important role for the success of a project can be more easily identified .

For example, C. Labuschagne and A. C. Brent propose in their analysis on Sustainable Project Life Cycle Management a life cycle that consists of 6 phases with clear start and end. The initiation phase takes place before the pre-feasibility phase and is therefore not part of the project life cycle itself. However, the PIR is seen as an important part of the project scope to clearly end the project with an overall assessment.

Purpose: Why conducting a PIR?

The world is changing faster and faster and becoming more complex over time. In recent decades, investment patterns have changed and the scope of projects has increased across a wide range of sectors. Therefore, the use of PIRs is strongly recommended by practitioners to improve project management practices in the fast-changing and complex world and to gain insights that lead to better decisions in future projects. , The importance of decision-making procedures for more complex projects has increased due to their immediate impact on a company's performance. However, in common practice, the last step is often neglected when reflecting on projects. Therefore, the PIR is usually not part of the project's life cycle.

A fast-changing environment requires changes that do not exclude current project management methods. In addition, there is increasing pressure on companies to ensure sustainable development. It is related to the fact that companies might fear the increasing accountability on social, environmental and economic impact based on their activities. This applies especially for already implemented projects but also for currently performed projects.

Sustainable Development is defined as followed:

“[A]dopting business strategies and activities that meet the needs of the enterprise and its stakeholders today, while protecting, sustaining and enhancing the human and natural resources that will be needed in the future’’

A PIR tackles above mentioned issues, but also serves several other purposes:

1. Adaptation of the project management methods currently in use, as these do not yet take sufficient account of social and environmental aspects.
2. Creating a common and clear understanding of a project life cycle and its interactions.
3. Ensuring knowledge transfer which is often seen as a critical factor for good performance.
4. Increasing the probability of success in future projects.
5. Analyzing performance of executed project.
6. Receiving a better understanding of all categories of the Johari Window.

Impact

Projects affect many different areas inside and outside an organization. Therefore - depending on the structure - a PIR can have an impact on all the different areas associated with project. For example, it it may involve a change in organizational power relations, an update of project management methods or the promotion of more realistic project assessments.

Goal: Organizational & Project-specific learnings

Overall, in a generic point of view, the goal of PIRs is to share a common understanding of used project management methodologies and processes in the company and ensuring that their effectiveness is maximized. Organizations are not only enabled to maximize the effectiveness during future projects, but also to attain post-implementation effectiveness. It must be mentioned that the covered scope goes beyond a lessons learned which is just a part of it. Further, PIRs function as a control mechanism to compare the actual performance with the original expectations.

Status Quo

PIRs can be an efficient tool to evaluate the performance of a project. However, at the moment instead of a common understanding there is only general guidance on how to perform them. Originally, PIRs have been used in investment and construction projects. Over the time, it can be observed that their application can be found in several different industries nowadays (e.g., service sector, pharmaceutical industry). Further, the quality and the scope depend on two factors: The company performing the PIR and the project itself. For example, in the expenditure analysis of construction projects, a study has found out big differences in the quality of the reviews:

• Only <40% of the participants have calculated an Internal Rate of Return (IRR) by using actual results
• Only <25% of the participants have calculated an actual NPV
• In general, larger companies tend to have a more sophisticated approach on investment decisions
• Smaller companies are thinking of introducing PIRs or improving their existing system.

An area where PIRs tend to have a great impact is in IT projects. Often the gained knowledge is lost or not leveraged to its fullest potential for future projects. Disregarding the importance of this gained expertise has lead to avoidable expenses. For example, the need and success of this knowledge transfer has been successfully observed in the implementation of Enterprise Resource Planning (ERP) systems in companies. A review in the post-implementation phase identified issues in the organizational power relations and thus changes ensured higher organizational performance and user friendliness.

Guidance: How to create a high quality PIR

Even though no generic framework exists, common patterns can be identified throughout the application of PIRs in different areas. Following, practical hands-on tips & tricks will be given.

When to apply

Generally, a post-implementation review shall be carried out as the project has reached stabilization after a specific project phase or its implementation. In some projects (e.g., IT) it may be more difficult to define a clear start and end state than in others (e.g., construction). This is due to the different nature of the projects and its applied project management methodologies (e.g., waterfall vs. scrum). Nevertheless, it might help to clearly define project deliverables (Milestones) as well-defined deliverables are a decisive success factor for projects.

Watch Out! That need´s to be considered!

When performing a PIR, there are several factors that influence the quality of the review. Firstly, project team members feel more accountable. Hereby it must be considered that project team members will most likely have a stronger sense of accountability. To ensure satisfaction and reduce dissonance, managers as well as the project members shall be introduced to the PIR methodology before the start of the project. Further, it must be clearly stated that projects never run 100% smoothly and that mistakes are allowed. However, the PIR tries to identify issues to guarantee a learning curve for the next projects.

Further, it is essential to understand that the structure of the document highly depends on the company and project itself. Therefore, the document cannot be used and recognized as a mere checklist. Thus, it cannot be neglected that any company can benefit from a PIR, the mere presence does not imply an increasing performance. In several literature it has been discussed, that the quality of the review itself has an important impact on the organizations performance.

To ensure high quality PIRs, some guidelines shall be followed:

• PIR as a Project Phase

  - Already when planning a project, the PIR should have its own phase.

  - Start and end date shall be determined at the beginning of the project but might be adapted later.

  - The PIR shall be conducted close to the end of the project top ensure, that the project, processes and deliverables are well remembered.

• Time & Resources

  - Appropriate time and resources should be made available. The actions taken are time-consuming (e.g. interviews, workshops) and probably even cost-intensive activities. Therefore, time & resources dedicated to the PIR must be aligned with the scope & deliverables of the project.

• Independent Reviewer

  - It is highly recommended that the reviewer is a project independent person (e.g. external) that has experience with similar projects. ,

• Stakeholder Involvement

  - Include all important stakeholders (e.g., users of a system, PMs)

• Project - not peoples - review

  - the emphasis should be on the evaluation of the system and not on the individuals involved. Thus, continuous cooperation can be achieved without creating fear of the consequences.

• Documentation

  - The results must be recorded in written form.

• Presentation of the review

  - Present results that all stakeholders are made aware of them. Reports my not be sufficient.

• Taking Actions

  - Within a short period, a plan should be carried out to improve effectiveness in identified key areas.

  - If applicable, the project management may take immediate action when the system (e.g., IT project) has direct negative impact on customers.

What shall be included in the PIR

According to current state, there is no general framework that can be applied to conduct high quality PIRs. One of the reasons is that it is too much dependent on the company and the project itself. Nevertheless, it can be observed that throughout projects in different areas similar focus points and guidelines can be identified that shall be included in a high quality PIR.

In general, a PIR includes a process- and product-oriented review. Moreover, different areas need to be considered.

Table 1 Literature review of categories that shall be included in a PIR according to , , , , , . Created by Jakob Kehler

This includes a review of: , , , , ,

• Organization
• Project management methodology
• Human resources
• Procurement
• Project scope & planning
• Driving principles for project development
• Benefits & costs
• Effectiveness of deliverables
• Learnings
• Relationships

Giving an example for contracts being part of procurement: A PIR between an Asian and European contractor has found out, that cultural difference leads to a different understanding of contracts. Firstly, not clearly defined contracts most likely lead to conflicts needing resolution during the project. Secondly, in the Asian believe contracts are more seen as a starting point for negotiation.

Examples

To better understand on how a PIR can leverage the performance of projects and organizations, in the following there are three examples given.

IT projects

PIRs have become more and more popular in the project life cycle of IT projects. Especially in the post-implementation stage of a system´s development there are several factors that play an essential role in a system´s success. The implementation of Enterprise Resource Planning (ERP) systems turn out to be one of the most common IT projects in companies. Even though the benefits of such a system as widely acknowledged in the literature, a high failure rate has been observed. This is due to the complexity of the system its upcoming challenges and obstacles when it comes to the implementation. However, when using PIRs a higher success rate has been observed as key areas have been identified and immediate actions were taken. Especially in a stage model which is often used in system development, the quality and effectiveness of the implementation process has a direct and highly correlated impact on the quality of the PIR.

Construction Projects

Not only, but especially in manufacturing processes the alignment of all areas to sustainable factors is more crucial than ever. To identify areas that shall be optimized, a PIR can help to incorporate sustainability within the organization. In contrast to IT projects, other factors play a more important role. This includes possible indicators for environmental, social, and economic sustainability. Possible main criteria can be:

• Economic sustainability

  - Financial health

  - Economic performance

  - Potential financial benefits

• Environmental sustainability

  - Air resources

  - Water resources

  - Land resources

  - Mineral resources

• Social sustainability

  - Internal human resources

  - External population

  - Stakeholder participation

  - Macro social performance

Finance Review

The Financial Accounting Foundation (FAF) in the US has determined that a PIR is shall be conducted for the standard setting process of the Financial Accounting Standard Board (FASB) and the Governmental Accounting Standards Board (GASB). The generic objective was to evaluate the effectiveness of the implemented standard in accounting and identify weaknesses which then can be shared with future rulemaking. Thereby, three sub-objectives have been set by the FAF:

1. Evaluation whether standard process the selected standard fulfills its stated intended objective
2. Evaluation of processes implementation and continuing compliance costs related to its benefits,
3. Provision of feedback to improve standard-setting process

Overall, several PIR objectives have been directly addressed.

Limitations & Benefits

Like any other tools or methodologies, a PIR has its limitations and benefits. Following these are to be reviewed.

Limitations

• Uncertain & complex world

  - It has been observed that companies that are already performing PIRs face difficulties due to the complexity and uncertainty of the environment and circumstances regarding the project.

• Qualitative and quantitative Factors

  - A PIR includes quantitative and qualitative measures which together may be difficult to evaluate. Often problems occur when putting qualitative information into numbers.

• Review technique

  - Yet, no generic framework for PIRs exist. Additionally, technical difficulties may occur. This could include the collection and separation of relevant data across different systems or the tracking & measuring of cash flows. A way to achieve high quality PIRs is process mining. Process mining allows to evaluate easily available data avoiding sophisticated methods for data collection.

• Highly dependence on company & project

  - PIRs can be performed in many different manners. The quality but also the acceptance is highly dependent on the company and project itself. For example, if one of the key values of a company is “constructive feedback”, then it is most likely that PIRs will be strongly supported. Still, resistance is to be expected by managers that might feel undermined in their authority.

• Not the Answer to everything

  - PIRs help to identify key issues if conducted in a suitable way. However, it is not the one solution that can save a company.

• Time & Resources

  - It is difficult to allocate the right amount of time and resources. Additionally, managers try to prevent conducting the reviews to cut costs as it can cost up to 10% of e.g., an implemented system´s price tag. Further, the managers may not see the link between the completed and upcoming projects as well as the gain of knowledge they could extract of the review. This may be related to the fact that there is more billable work to do. .

Benefits

• Serving variety of objectives

  - In the literature the benefits of PIRs are widely acknowledged as it serves multiple objectives. On the one hand, it identifies strengths & weaknesses in the past. On the other hand, it is also future-oriented. Further, it can be differentiated between project specific and more generic (organizational) areas.

• Learnings for the future

  - Lesson learned is one of the key deliverables. These are proven to be useful especially for junior project managers or similar projects. Further it improves investment decisions.

• Proven to be beneficial

  - A study has found out that companies that once conducted a PIR, noticed the benefits that outweighed its difficulties and expenses. This turned out to be true especially in a highly competitive environment.

• Feedback function

  - Poorly performing assess can be identified and further actions taken. The initial expected performance can be compared with the actual one. For example, when implementing IT systems it can often be seen that users only need 50% to 75% of the features to gain full benefits of the system.

• Proactive Approach

  - Conducting reviews is a proactive approach on finding strengths, weaknesses, opportunities and threats taking several areas into account. This way ugly truths can be revealed that hinder a company´s performance.

• Considering the product life cycle

  - The product life cycle goes beyond a project life cycle as products can be outcomes of a project. A PIR may link both.

Annotated bibliography

This chapter is currently in progress and will be updated.