Metrics in Portfolio management

Revision as of 19:20, 9 May 2023

Abstract

According to the Project Management Institute (PMI), a metric is a "description of a project or product attribute and how to measure it" ^[1]. It is, indeed, a quantitative measure of performance or outcome that provides a basis for comparison, evaluation, or improvement.

Peter Drucker famously said, "If you cannot measure it, you cannot improve it". However, it is not enough just to measure. The ultimate purpose of metrics is, as well, to provide the right information to the right person at the right time, using the correct media and in a cost-effective manner ^[2]. Notably, building effective metrics is extremely relevant when it comes to decision making at all levels of a company, project or portfolio. Thereby, knowing specifically what to measure when it comes to portfolio management is a must to ensure the future of the organization.

The purpose of this article is to describe the type of metrics needed to evaluate Portfolio performance and outline their differences from the well-known metrics to measure Project performance. Firstly, this article will focus on introducing to general terms. Secondly, the relevant role of the portfolio management will be highlighted. Hereafter, what is needed to create an effective metric will take part in this essay. Finally, different useful metrics and techniques for portfolio management will be listed and explained in detail.

General terms

Firstly, to understand the topic under study, both the basics of Portfolio Management and Metrics need to be introduced.

Portfolio Managements

As the PMI states ^[1], a portfolio is a collection of projects, programs, subsidiary portfolios, and operations managed as a group to achieve strategic objectives. More importantly, those programs or projects within a portfolio may not be directly related or interdependent, they are linked to the organization’s strategic plan by means of the portfolio.

Figure 1: Portfolios, Programs, and Projects — High-Level View ^[3]

On the other hand, portfolio management is a crucial organization capability that enables them to effectively manage and balance their portfolio, align projects with their overarching strategy, and ensure the allocation of resources in order to maximize benefits ^[4]. The PMI ^[3] offers a broader definition and additionally, points out how portfolio management seeks for this alignment between the portfolio components and the organization strategy which are selecting the best portfolio components prioritizing the work, providing the needed resources, overseeing or working with portfolio component managers on their implementation, supervising proper transition into the operational environment, and enabling the achievement of portfolio value. Furthermore, Harvey Levine in his elaborated Project Portfolio Management Guide ^[5] mentions what portfolio management definitely is not: the error lies in assuming that it is primarily concerned with managing numerous individual projects, but this is not accurate. The real focus of portfolio management is on managing the collection of projects as a whole, with the aim of maximizing their overall contribution to the overall welfare and success of the enterprise.

On the other hand, when discussing the process of portfolio management, there are two key components or phases to consider:

1. Prioritization and selection of potential projects for the portfolio. This evaluation requires taking into account the opportunities (value and benefits) as well as the risks (which can alter the expected benefits)
2. Maintaining the pipeline: continuing, delaying, or terminating approved projects. Projects are periodically assessed based on the established criteria for selection and acceptable performance

Metrics

The simplest definition of metric is the one stated by Cambridge: "System for measuring something" ^[6]. Even though it turns out to be a poorly detailed definition, it lays the foundations for what a metric does. In fact, the core metric function is to measure something. What is being measured and why these are taking place are the components that make metrics relevant. Looking deeper, metrics are tools of quantitative evaluation that can be utilized for the purpose of monitoring and measuring performance and success. At business or company management level, metrics become even more relevant when they are added to a set of metrics to construct a dashboard that managers and analysts regularly consult to ensure that performance assessments and business strategies are maintained.

Metrics require a need and purpose and last but not least, might need both as reference points: target and baseline. On the one hand, it is necessary to specify what is intended to be measured in order to determine why this metric has to exist in the first place. On the other hand, targets need to be defined. If there is no point of comparison, what is going to be measured against? The metric would not have a raison d'être ^[2].

Roles and Responsibilities of a Portfolio Management Office

As the Portfolio Standards of PMI state ^[3], Portfolio managers have the responsibility for stating and implementing portfolio management. Among other contributions, the Portfolio Management Office (PMO) supports portfolio management by:

• Identifying, analyzing, coordinating, negotiating, monitoring, and controlling portfolio components; supporting component proposals and evaluations; facilitating prioritization; authorization; termination of components; and facilitating the allocation of resources in alignment with organizational strategy and objectives.
• Coordinating communication across portfolio components.
• Providing program and project progress information and metric reporting utilizing key performance indicators (KPIs) (e.g., expenditure, defects, resources) to the portfolio governance process.

The PMO office can provide recommendations on selecting, terminating, or initiating actions to maintain alignment with the organization's strategic objectives. Additionally, the portfolio management office contributes to the project or program management office with performance evaluations of portfolio components based on predefined metrics.

Remarkably, the Portfolio Management Office (PMO) generally establishes the metrics needed to validate the performance of the overall portfolio of projects. To be effective, Michael Wood ^[7] specifies that the PMO needs to define the success metrics that are going to be used to determine how well each project and the overall PMO organization is delivering value within and across the performance domain framework. The PMO might also want to weigh each Performance Domain (PD) in terms of how important is it in determining the PMO’s, and each project’s, success. Each metric should directly align with each domain’s intended outcomes to support efficient oversight.

According to Harold Kerzner ^[2], the PMO should address three crucial inquiries, including:

1. Whether or not the company is engaged in the proper projects, which entails determining if the projects align with strategic objectives and support strategic initiatives;
2. Whether or not the company is involved in a proper number of appropriate projects to maximize investment and shareholder value; and
3. Whether or not the company is executing the proper projects in the right manner, which involves assessing project completion time and cost.

Surprisingly, the term "proper" is present in all three questions and implies a sense of value.

Effective Metrics

Gold rules

Efficient use of time and resources is crucial in portfolio management, and progress measurement is no exception. Consequently, it is relevant for portfolio managers to point to the development of metrics that only measure relevant and useful insights for decision making. According to the PMI Standard ^[1] and George T. Doran ^[8], the SMART criteria is one of the golden rules to be followed in order to create effective metrics. Specifically, the acronym SMART stands for:

• Specific: measurements are specific as to what to measure. Define what is being measured
• Meaningful: metrics ought to be linked to the business case, baselines, or requirements. Evaluating project achievements that do not contribute to meeting objectives or enhancing performance are ineffective. Besides, metrics' meaningfulness is commonly related to their nature of being quantifiable or "measurable" against the baseline and stated requirements.
• Achievable: it is meaningless to establish unfeasible or out-of-reach targets. Targets defined need to be attainable given the people, technology, and environment considered.
• Relevant: the information provided by the measure should represent value and allow actionable information.
• Time-bound: metrics have to be associated to a specific time frame. Creating these boundaries allows to keep the measurements on track and accountable for.

Value based metrics vs. Traditional metrics

In 1996, Norton and Kaplan introduced the Balanced Scorecard (BSC) ^[9], a tool that already expanded beyond the traditional financial metrics to include non-financial metrics. The Balanced Scorecard framework aims to achieve desired outcomes aligned with a business' strategy. This includes creating value for present and future stakeholders, as well as leveraging internal capabilities and investments in people, systems, and procedures to improve future performance

Later in time, "Project Management Metrics, KPIs, and Dashboards" was presented by Harold Kerzner ^[2]. It highlights the key differences between traditional metrics and value-based metrics and explains how each type of metric can be useful. It is argued that traditional metrics are useful for tracking project performance and ensuring that it is completed within the constraints of the triple constraint model (time, cost, and scope) called Iron triangle. However, these metrics may not necessarily provide a comprehensive view of the projects' overall value to the organization.

On the other hand, value-based metrics are centered on the strategic objectives and intended benefits. They help to ensure that the projects are aligned with the organization's strategic objectives and that it generates the expected benefits. However, value-based metrics may be more difficult to measure and track than traditional metrics. The definition of value and how it is measured can differ depending on the industry, company, and the business characteristics. Different stakeholders may have their own ideas of what value means for them, such as job security, profitability, reputation, and the creation of intellectual property. It can be challenging to meet the needs and expectations of all stakeholders at the same time.

Notably, while traditional metrics have as primary audience the project manager and the team, value (or value reflective) metrics are more useful for the clients/stakeholders.

The importance of the value component in the definition of success cannot be overlooked. When management makes poor decisions managing project portfolios, companies end up working on the wrong projects. For example, the deliverable requested can be produced but either there is no market for the product or stakeholders may be displeased with management’s performance.

In any case, Kerzner suggests that the use of value-based metrics and traditional metrics in combination can provide a more accurate and comprehensive assessment of the project's performance. While the traditional metrics can be used to monitor the performance of the project and ensure that it is completed within the constraints of the triple constraint model, the value-based metrics can be used to measure the project's overall impact on the organization and assess its contribution to the organization's strategic objectives.

Regarding assessment and management at portfolio level, value-based metrics clearly gain greater importance than traditional ones: portfolio success has to be defined in terms of the value that was expected to be delivered.

Portfolio Performance Measurement Tools and Metrics

Figure 2: table comparing portfolio management metrics with project management metrics (own figure, based on Table 8-1 and Table 8-2 from "Project Management Metrics, KPIs, and Dashboards" ^[2])

There are countless possible indicators for measuring the performance of portfolios and certainly, new metrics are being created every day. This is because each enterprise, organization, and sector has different approaches to determining what is important to measure and how to measure progress toward short and long-term goals and objectives. With that said, the following section describes the objectives on which the portfolio management area relies to define its metrics and highlights some of the most popular and effective metrics according to the analyzed literature.

According to Robert G. Cooper ^[5], an effective portfolio management system should accomplish three primary goals that serve as a framework for defining metrics by determining the corresponding targets:

1. Ensure strategic alignment: the primary objective is to ensure that the final portfolio of projects accurately reflects the business's strategy by focusing on strategic alignment. This involves selecting and keeping those projects that are either in line with the strategy, support it, or are integral components of it. Additionally, the distribution of spending across various projects, areas, and markets must be directly linked to the business strategy.
2. Maximize the value of the portfolio: here the goal is to optimize the financial value of the portfolio for a given spending level by strategically allocating resources. This requires selecting projects that maximize their commercial value in relation to a specific business objective.
3. Seek the right balance of projects: it is important to achieve the desired balance of projects based on several factors, such as long-term vs. short-term projects or high-risk vs. low-risk projects across various markets, technologies, product categories, and project types.

From the previous three goals, the easiest and most transparent to measure is the second one: the financial value of the portfolio. Historically, the measurement system of companies has been financial. In fact, accounting has been called the language of business. There is evidence of financial accounting records that date back thousands of years. Reportedly, the set of well-known metrics employed to measure economic performance are almost standardized towards different companies and, most of them, have an intrinsic relationship with NPV (Net Present Value), ROI (Return on Investment) and EVA (Earned Value Analysis). At the same time, financial metrics are usually the main drivers of a performance management system.

Portfolio metrics

There are countless possible indicators for measuring the performance of portfolios and certainly, new metrics are being created every day. This is because each enterprise, organization, and sector has different approaches to determining what is important to measure and how to measure progress toward short and long-term goals and objectives. With that said, the following section describes some of the most popular and effective metrics according to the analyzed literature.

Average of EVA measurements

Earned Value Analysis (EVA) is an essential aspect of project management for performance reviews. It is a practical method for monitoring and assessing the performance of ongoing projects with the goal of identifying the ones that no longer align with the conditions on which approval was granted and considering their termination ^[5]. The four basic measurements related to EVA are:

• Budget of Completion (BAC) which is the assigned value to a task or any part of the project.
• Planned Value (PV) which is the weighted value of the task at a specific point in time during its planned execution.
• Earned Value (EV) stated as the percent complete of a task times the budget (BAC). It can be stated in monetary terms or time values.
• Actual Cost (AC) which is the actual cost of worked performed.

At portfolio level, what matters is to obtain the periodic information about the average of the cost performance index (CPI), schedule performance index (SPI), schedule variance (SV), and cost variance (CV) of the projects overseen by the portfolio ^[4].

Following, a brief explanation of these terms an how to calculate them is displayed below based on Harvey Levine's wrtitings ^[5]:

• CV (Cost Variance) = EV/AC
• CPI (Cost Performance Index) = EV/AC

Lower than 1 signifies subpar cost performance.

• SV (Schedule Variance) = EV/PV
• SPI (Schedule Performance Index) = EV/PV

Looking at the SPI, we can easily see that we are behind schedule.

Percentage of overdue portfolio components

As its name defines it, this metric measures the percentage of overdue projects out of the total number of them in a portfolio ^[4]. Its main purpose is to monitor the portfolio's progress since a large number of overdue projects could impede it and jeopardize the overall effort.

When defining this metric, some PMOs categorize overdue projects in order to analyze and solve the high-priority ones first. Targets for this metric should be established based on the nature of the business, client expectations, and the level of competition.

Percentage of portfolio components with missed milestones

A project is typically composed of several phases, starting from its conception and ending with product delivery or launch. Each phase contains several activities performed by multiple disciplines, resulting in a significant interim goal, or a milestone. These milestones represent crucial checkpoints in the project's lifecycle, with each having a deadline that must be met to maintain project compliance ^[5].

This metric evaluates the percentage of milestones missed during the development of the projects, out of the total number of project milestones ^[2]. In other words, it assesses the project's adherence to the original milestones schedule.

Regarding targets, organizations should aim to set them as low as possible while still being achievable so as to not lead to unrealistic goals.

Resource utilization and availability

The resources available within a portfolio can be categorized into four major categories ^[1]:

• Human capital, referring to the availability of human resources to support the portfolio.
• Financial capital, referring to the availability of funds to support the portfolio.
• Assets, referring to the availability of physical assets, such as machinery, office space, factory sites, etc.
• Intellectual capital, referring to the availability of patents, copyrights, etc.

The objective of managing resource utilization is to ensure that organizations optimally allocate resource capacity against resource requirements or demand based on the strategic goals of the organization and the composition of the portfolio. Since portfolio resources are always constrained, it is crucial to continuously review and balance supply and demand.

Remarkably, bottleneck resources are typically critical human ones. Therefore, it is relevant to measure their utilization. Overall resource utilization is a method used to measure the total utilization of a resource against its capacity or availability. It represents the cumulative time that employees spend on their respective activities. It can be calculated by comparing the total employees' available time against the total work assigned to them.

Stakeholder satisfaction

Stakeholder satisfaction is a metric used to assess how satisfied the stakeholders are with the portfolio based on different categories ^[4]. This metric is challenging to measure as it represents a subjective evaluation of performance. Typically, it is evaluated by asking stakeholders to rate their satisfaction on a numerical scale through surveys. To ensure the accuracy of the data, it's crucial to carefully design the survey questions and obtain a representative sample of stakeholders.

Risk indicators

Percentage of projects that fit organizational strategy

Other possible metrics

• Time spent approving a project.
• Percentage of protfolio components that have undergone scope reduction ^[2].
• Percentage of protfolio components that required rework ^[2].
• Percentage of protfolio components that are used to run the business, grow the business, and to innovate ^[2].
• Deviation of planned hours of work for projects ^[4].
• Average deviation of planned versus actual duration of components ^[4].
• Escalated risks and issues—their status and impact ^[4].
• Number of new components in the pipeline with their attributes and information ^[4].

Portfolio reporting tools

Kezner pg 265

Dashboards

Scorecards

Application

Case of study: Portfolio management system at Crompton Corporation

Limitations

It is relevant to note that advances in project metrics have been rapid, but advances in portfolio metrics have been slow because not all companies maintain a project management office (PMO) dedicated to portfolio management activities. This leads to changes in the role of the project manager, the metrics used, and the dashboard displays ^[2].

TO BE COMPLETED

Annotated Bibliography

Project Management Institute (2017). The Standard for Portfolio Management. PMBOOK 4th Edition.

Harold Kerzner (2017). Project Management Metrics, KPIs, and Dashboards. Wiley & Sons.

Harvey A. Levine (2005). Project Portfolio Management - A practical guide to selecting projects, managing portfolios, and maximizing benefits. Jossey-Bass.

References

1. ↑ ^{1.0 1.1 1.2 1.3} Project Management Institute (2021) <<Project Management: A guide to the Project Management Body of Knowledge and The Standard for Project Management>> PMBOOK guide 7th Edition
2. ↑ ^{2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9} Harold Kerzner (2017) <<Project Management Metrics, KPIs, and Dashboards>>, Wiley & Sons
3. ↑ ^{3.0 3.1 3.2} Project Management Institute (2017) <<The Standard for Portfolio Management>> 4th Edition
4. ↑ ^{4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7} G. Levin & J. Wyzalek <<Portfolio Management - A strategic approach>> Taylor & Francis Group
5. ↑ ^{5.0 5.1 5.2 5.3 5.4} Harvey A. Levine (2005) <<Project Portfolio Management - A practical guide to selecting projects, managing portfolios, and maximizing benefits>> Jossey-Bass
6. ↑ Cambridge dictonary, Cambridge University Press & Assessment 2023
7. ↑ Michael Wood (2021) <<Are You Effectively Measuring the Right Success Metrics?>> Project Management Institute
8. ↑ Doran, G.T. (1981) <<There's a S.M.A.R.T. way to write management's goals and objectives>>
9. ↑ Kaplan, R. S., & Norton, D. P. (1996) <<The balanced scorecard: Translating strategy into action>> Harvard Business School Press