Metrics in Portfolio management
(→Limitations) |
|||
| (72 intermediate revisions by one user not shown) | |||
| Line 1: | Line 1: | ||
==Abstract== | ==Abstract== | ||
| − | According to the Project Management Institute (PMI), a metric is a "description of a project or product attribute and how to measure it" <ref name=PMBOOKProject> Project Management Institute (2021) <<Project Management: A guide to the Project Management Body of Knowledge and The Standard for Project Management>> | + | According to the Project Management Institute (PMI), a metric is a "description of a project or product attribute and how to measure it" <ref name=PMBOOKProject> Project Management Institute (2021) <<Project Management: A guide to the Project Management Body of Knowledge and The Standard for Project Management>> PMBOOK guide 7th Edition </ref>. It is, indeed, a quantitative measure of performance or outcome that provides a basis for comparison, evaluation, or improvement. |
| − | Peter Drucker famously said, ''" | + | Peter Drucker famously said, ''"If you cannot measure it, you cannot improve it"''. However, it is not enough just to measure. The ultimate purpose of metrics is, as well, to provide the right information to the right person at the right time, using the correct media and in a cost-effective manner <ref name=Kerzner>Harold Kerzner (2017) <<Project Management Metrics, KPIs, and Dashboards>>, Wiley & Sons</ref>. Notably, building effective metrics is extremely relevant when it comes to decision making at all levels of a company, project or portfolio. Thereby, knowing specifically what to measure when it comes to portfolio management is a must to ensure the future of the organization. |
| − | The purpose of this article is to describe the type of metrics needed to evaluate Portfolio performance and outline their differences from the well-known metrics to measure Project performance. Firstly, this article will focus on introducing to general terms. Secondly, the relevant role of the portfolio management will be highlighted. Hereafter, what is needed to create an effective metric will take part in this essay. | + | The purpose of this article is to describe the type of metrics needed to evaluate Portfolio performance and outline their differences from the well-known metrics to measure Project performance. Firstly, this article will focus on introducing to general terms. Secondly, the relevant role of the portfolio management will be highlighted. Hereafter, what is needed to create an effective metric will take part in this essay. Finally, different useful metrics and techniques for portfolio management will be listed and explained in detail. |
==General terms== | ==General terms== | ||
| Line 13: | Line 13: | ||
===Portfolio Managements=== | ===Portfolio Managements=== | ||
| − | As the PMI states <ref name=PMBOOKProject></ref>, portfolio is a collection of projects, programs, subsidiary portfolios, and operations managed as a group to achieve strategic objectives. | + | As the PMI states <ref name=PMBOOKProject></ref>, a portfolio is a collection of projects, programs, subsidiary portfolios, and operations managed as a group to achieve strategic objectives. More importantly, those programs or projects within a portfolio may not be directly related or interdependent, they are linked to the organization’s strategic plan by means of the portfolio. |
| − | [[File:Screenshot 2023-02-19 233453.png| | + | [[File:Screenshot 2023-02-19 233453.png|thumb|500px|right|Figure 1: Portfolios, Programs, and Projects — High-Level View <ref name=PMBOOKPortfolio> Project Management Institute (2017) <<The Standard for Portfolio Management>> 4th Edition </ref> ]] |
| − | + | On the other hand, '''portfolio management''' is a crucial organization capability that enables them to effectively manage and balance their portfolio, align projects with their overarching strategy, and ensure the allocation of resources in order to maximize benefits <ref name=PMASA> G. Levin & J. Wyzalek (2015) <<Portfolio Management - A strategic approach>> Taylor & Francis Group </ref>. The PMI <ref name=PMBOOKPortfolio> </ref> offers a broader definition and additionally, points out how portfolio management seeks for this alignment between the portfolio components and the organization strategy which are selecting the best portfolio components prioritizing the work, providing the needed resources, overseeing or working with portfolio component managers on their implementation, supervising proper transition into the operational environment, and enabling the achievement of portfolio value. Furthermore, Harvey Levine in his elaborated Project Portfolio Management Guide <ref name=PPMPG> Harvey A. Levine (2005) <<Project Portfolio Management - A practical guide to selecting projects, managing portfolios, and maximizing benefits>> Jossey-Bass </ref> mentions what portfolio management definitely is not: the error lies in assuming that it is primarily concerned with managing numerous individual projects, but this is not accurate. The real focus of portfolio management is on managing the collection of projects as a whole, with the aim of maximizing their overall contribution to the overall welfare and success of the enterprise. | |
| − | + | On the other hand, when discussing the '''process''' of portfolio management, there are two key components or phases to consider: | |
| − | # | + | # Prioritization and selection of potential projects for the portfolio. This evaluation requires taking into account the opportunities (value and benefits) as well as the risks (which can alter the expected benefits) |
| − | + | # Maintaining the pipeline: continuing, delaying, or terminating approved projects. Projects are periodically assessed based on the established criteria for selection and acceptable performance | |
| − | # | + | |
| − | + | ||
| + | <!-- Although portfolio management is tailored for each organization, there are many common elements and approaches. Indeed, portfolio management aids in all decision-making processes across the full range of project portfolios through the following four steps <ref name=PMASA> </ref>: | ||
| + | # Gathering data from existing and proposed projects | ||
| + | # Arranging and categorizing the data | ||
| + | # Presenting the data to a carefully chosen decision-making team for comprehensive portfolio evaluations | ||
| + | # Establishing a framework for communicating and executing decisions --> | ||
===Metrics=== | ===Metrics=== | ||
| − | The simplest definition of '''metric''' is the one stated by Cambridge: " | + | The simplest definition of '''metric''' is the one stated by Cambridge: "System for measuring something" <ref name=Cambridge> Cambridge dictonary, Cambridge University Press & Assessment 2023</ref>. Even though it turns out to be a poorly detailed definition, it lays the foundations for what a metric does. In fact, the core metric function is to measure something. What is being measured and why these are taking place are the components that make metrics relevant. Looking deeper, metrics are tools of quantitative evaluation that can be utilized for the purpose of monitoring and measuring performance and success. At business or company management level, metrics become even more relevant when they are added to a set of metrics to construct a dashboard that managers and analysts regularly consult to ensure that performance assessments and business strategies are maintained. |
| − | Metrics require a need | + | Metrics require a need and purpose and last but not least, might need both as reference points: target and baseline. On the one hand, it is necessary to specify what is intended to be measured in order to determine why this metric has to exist in the first place. On the other hand, targets need to be defined. If there is no point of comparison, what is going to be measured against? The metric would not have a raison d'être <ref name=Kerzner></ref>. |
| + | ==Roles and Responsibilities of a Portfolio Management Office== | ||
| + | As the Portfolio Standards of PMI state <ref name=PMBOOKPortfolio> </ref>, Portfolio managers have the responsibility for stating and implementing portfolio management. Among other contributions, the Portfolio Management Office (PMO) supports portfolio management by: | ||
| + | * Identifying, analysing, coordinating, negotiating, monitoring, and controlling portfolio components; supporting component proposals and evaluations; facilitating prioritization; authorization; termination of components; and facilitating the allocation of resources in alignment with organizational strategy and objectives. | ||
| + | * Coordinating communication across portfolio components. | ||
| + | * Providing program and project progress information and metric reporting utilizing key performance indicators (KPIs) (e.g., expenditure, defects, resources) to the portfolio governance process. | ||
| − | + | The PMO office can provide recommendations on selecting, terminating, or initiating actions to maintain alignment with the organization's strategic objectives. Additionally, the portfolio management office contributes to the project or program management office with performance evaluations of portfolio components based on predefined metrics. | |
| − | + | Remarkably, the Portfolio Management Office (PMO) generally establishes the metrics needed to validate the performance of the overall portfolio of projects. To be effective, Michael Wood <ref name=PMReport> Michael Wood (2021) <<Are You Effectively Measuring the Right Success Metrics?>> Project Management Institute</ref> | |
| + | specifies that the PMO needs to define the success metrics that are going to be used to determine how well each project and the overall PMO organization is delivering value within and across the performance domain framework. The PMO might also want to weigh each Performance Domain (PD) in terms of how important is it in determining the PMO’s, and each project’s, success. Each metric should directly align with each domain’s intended outcomes to support efficient oversight. | ||
| − | + | According to Harold Kerzner <ref name=Kerzner></ref>, the PMO should address three crucial inquiries, including: | |
| + | # Whether or not the company is engaged in the proper projects, which entails determining if the projects align with strategic objectives and support strategic initiatives; | ||
| + | # Whether or not the company is involved in a proper number of appropriate projects to maximize investment and shareholder value; and | ||
| + | # Whether or not the company is executing the proper projects in the right manner, which involves assessing project completion time and cost. | ||
| + | Surprisingly, the term "proper" is present in all three questions and implies a sense of value. | ||
| − | + | ==Effective Metrics== | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | ==Effective | + | |
===Gold rules=== | ===Gold rules=== | ||
| − | Efficient use of time and resources is crucial in portfolio management, and progress measurement is no exception. Consequently, it is relevant for portfolio managers to point to the development of metrics that only measure relevant and useful insights for decision | + | Efficient use of time and resources is crucial in portfolio management, and progress measurement is no exception. Consequently, it is relevant for portfolio managers to point to the development of metrics that only measure relevant and useful insights for decision making. According to the PMI Standard <ref name=PMBOOKProject></ref> and George T. Doran <ref name=GTDoran> Doran, G.T. (1981) <<There's a S.M.A.R.T. way to write management's goals and objectives>> </ref>, the SMART criteria is one of the golden rules to be followed in order to create effective metrics. Specifically, the acronym SMART stands for: |
| − | * Specific: | + | * Specific: measurements are specific as to what to measure. Define what is being measured |
| − | * Meaningful: metrics ought to be linked to the business case, baselines, or requirements. Evaluating project | + | * Meaningful: metrics ought to be linked to the business case, baselines, or requirements. Evaluating project achievements that do not contribute to meeting objectives or enhancing performance are ineffective. Besides, metrics' meaningfulness is commonly related to their nature of being quantifiable or "measurable" against the baseline and stated requirements. |
* Achievable: it is meaningless to establish unfeasible or out-of-reach targets. Targets defined need to be attainable given the people, technology, and environment considered. | * Achievable: it is meaningless to establish unfeasible or out-of-reach targets. Targets defined need to be attainable given the people, technology, and environment considered. | ||
* Relevant: the information provided by the measure should represent value and allow actionable information. | * Relevant: the information provided by the measure should represent value and allow actionable information. | ||
| − | * Time-bound: metrics have to be associated to | + | * Time-bound: metrics have to be associated to a specific time frame. Creating these boundaries allows to keep the measurements on track and accountable for. |
===Value based metrics vs. Traditional metrics=== | ===Value based metrics vs. Traditional metrics=== | ||
| − | "Project Management Metrics, KPIs, and Dashboards" | + | [[File:PORTFOLIO metrics Kezner.png|thumb|500px|right|Figure 2: table comparing portfolio management metrics with project management metrics (own figure, based on Table 8-1 and Table 8-2 from "Project Management Metrics, KPIs, and Dashboards" <ref name=Kerzner> </ref>)]] |
| − | + | In 1996, Norton and Kaplan introduced the Balanced Scorecard (BSC) <ref name=BSC> Kaplan, R. S., & Norton, D. P. (1996) <<The balanced scorecard: Translating strategy into action>> Harvard Business School Press </ref>, a tool that already expanded beyond the ''traditional'' financial metrics to include non-financial metrics. The Balanced Scorecard framework aims to achieve desired outcomes aligned with a business' strategy. This includes creating ''value'' for present and future stakeholders, as well as leveraging internal capabilities and investments in people, systems, and procedures to improve future performance | |
| − | + | Later in time, "Project Management Metrics, KPIs, and Dashboards" was presented by Harold Kerzner <ref name=Kerzner> </ref>. It highlights the key differences between traditional metrics and value-based metrics and explains how each type of metric can be useful. It is argued that '''traditional metrics''' are useful for tracking project performance and ensuring that it is completed within the constraints of the triple constraint model (time, cost, and scope) called [[Iron triangle]]. However, these metrics may not necessarily provide a comprehensive view of the projects' overall value to the organization. | |
| − | The importance of the value component in the definition of success cannot be | + | On the other hand, '''value-based metrics''' are centred on the strategic objectives and intended benefits. They help to ensure that the projects are aligned with the organization's strategic objectives and that it generates the expected benefits. However, value-based metrics may be more difficult to measure and track than traditional metrics. The definition of ''value'' and how it is measured can differ depending on the industry, company, and the business characteristics. Different stakeholders may have their own ideas of what ''value'' means for them, such as job security, profitability, reputation, and the creation of intellectual property. It can be challenging to meet the needs and expectations of all stakeholders at the same time. |
| + | |||
| + | Notably, while traditional metrics have as primary audience the project manager and the team, value (or value reflective) metrics are more useful for the clients/stakeholders. | ||
| + | |||
| + | The importance of the value component in the definition of success cannot be overlooked. When management makes poor decisions managing project portfolios, companies end up working on the wrong projects. For example, the deliverable requested can be produced but either there is no market for the product or stakeholders may be displeased with management’s performance. | ||
In any case, Kerzner suggests that the use of value-based metrics and traditional metrics in combination can provide a more accurate and comprehensive assessment of the project's performance. While the traditional metrics can be used to monitor the performance of the project and ensure that it is completed within the constraints of the triple constraint model, the value-based metrics can be used to measure the project's overall impact on the organization and assess its contribution to the organization's strategic objectives. | In any case, Kerzner suggests that the use of value-based metrics and traditional metrics in combination can provide a more accurate and comprehensive assessment of the project's performance. While the traditional metrics can be used to monitor the performance of the project and ensure that it is completed within the constraints of the triple constraint model, the value-based metrics can be used to measure the project's overall impact on the organization and assess its contribution to the organization's strategic objectives. | ||
| − | Regarding | + | Regarding assessment and management at portfolio level, value-based metrics clearly gain greater importance than traditional ones: portfolio success has to be defined in terms of the value that was expected to be delivered. |
| − | + | ==Portfolio Performance Measurement Tools and Metrics== | |
| − | ==Portfolio | + | ===Portfolio metrics=== |
| − | + | According to Robert G. Cooper <ref name=PPMPG> </ref>, an effective portfolio management system should accomplish three primary goals that serve as a framework for defining metrics by determining the corresponding targets: | |
| − | + | # Ensure strategic alignment: the primary objective is to ensure that the final portfolio of projects accurately reflects the business's strategy by focusing on strategic alignment. This involves selecting and keeping those projects that are either in line with the strategy, support it, or are integral components of it. Additionally, the distribution of spending across various projects, areas, and markets must be directly linked to the business strategy. | |
| + | # Maximize the value of the portfolio: here the goal is to optimize the financial value of the portfolio for a given spending level by strategically allocating resources. This requires selecting projects that maximize their commercial value in relation to a specific business objective. | ||
| + | # Seek the right balance of projects: it is important to achieve the desired balance of projects based on several factors, such as long-term vs. short-term projects or high-risk vs. low-risk projects across various markets, technologies, product categories, and project types. | ||
| − | + | From the previous three goals, the easiest and most transparent to measure is the second one: the financial value of the portfolio. Historically, the measurement system of companies has been financial. In fact, accounting has been called the language of business. There is evidence of financial accounting records that date back thousands of years. Reportedly, the set of well-known metrics employed to measure economic performance are almost standardized towards different companies and, most of them, have an intrinsic relationship with NPV (Net Present Value) and ROI (Return on Investment). At the same time, financial metrics are usually the main drivers of a performance management system. | |
| − | + | There are countless possible indicators for measuring the performance of portfolios and certainly, new metrics are being created every day. This is because each enterprise, organization, and sector has different approaches to determining what is important to measure and how to measure progress toward short and long-term goals and objectives. With that said, the following section describes some of the most popular and effective metrics according to the analysed literature. | |
| + | |||
| + | ====Average of EVA measurements==== | ||
| + | |||
| + | [[File:Screenshot 2023-05-09 203314.png|thumb|500px|right|Figure 3: Planned Value, Actual Cost, and Earned Value <ref name=Anbari> Anbari, F. T. (2001) <<Applications and extensions of the earned value analysis method>> PA: Project Management Institute </ref> ]] | ||
| + | |||
| + | Earned Value Analysis (EVA) is an essential aspect of project management for performance reviews. It is a practical method for monitoring and assessing the performance of ongoing projects with the goal of identifying the ones that no longer align with the conditions on which approval was granted and considering their termination <ref name=PPMPG> </ref>. The four basic measurements related to EVA are <ref name=Anbari> </ref>: | ||
| + | * Budget of Completion (BAC): it is the total budget baseline for the activity, work package, or project. | ||
| + | * Planned Value (PV): it is the time-phased budget baseline. It is the approved budget for accomplishing the activity, work package, or project related to the schedule. It can be viewed as the value to be earned as a function of accomplishments up to a given point in time. | ||
| + | * Earned Value (EV): it is the earned value for the work completed to a point in time. Indeed, this is the percent complete of a task times the budget (BAC). | ||
| + | * Actual Cost (AC): it is the actual cost spent to accomplish an activity or project and to earn the related value up to a given point in time. | ||
| + | |||
| + | At portfolio level, what matters is to obtain the periodic information about the '''average of the cost performance index (CPI), schedule performance index (SPI), schedule variance (SV), and cost variance (CV)''' of the projects overseen by the portfolio <ref name=PMASA> </ref>. | ||
| + | |||
| + | Following, a brief explanation of these terms an how to calculate them is displayed below based on Frank Anbari's wrtitings <ref name=Anbari> </ref>. | ||
| + | |||
| + | The first set of queations are used to calculate the variances, generally based on cumulative data: | ||
| + | * The Cost Variance (CV) is a measure of the conformance of actual cost of work performed to the budget: ''CV = EV - AC'' | ||
| + | |||
| + | * The Schedule Variance (SV) is a measure of the conformance of actual progress to the schedule: ''SV = EV - PV'' | ||
| + | |||
| + | In the above equations, “0” indicates performance is on target. A negative value indicates poor performance. A positive value indicates good performance. | ||
| + | |||
| + | The following equations are used to calculate the performance indices: | ||
| + | |||
| + | * The Cost Performance Index is a measure of the conformance of actual cost of work performed to the budget: ''CPI = EV / AC'' | ||
| + | |||
| + | * The Schedule Performance Index is a measure of the conformance of actual progress to the schedule: ''SPI = EV / PV'' | ||
| + | |||
| + | In the above equations, “1” indicates performance is on target. More than “1” indicates good performance. Less than “1” indicates poor performance. Looking at the SPI, we can easily see that we are behind schedule. | ||
| + | |||
| + | |||
| + | ====Percentage of overdue portfolio components==== | ||
| + | |||
| + | As its name defines it, this metric measures the percentage of overdue projects out of the total number of them in a portfolio <ref name=PMASA> </ref>. Its main purpose is to monitor the portfolio's progress since a large number of overdue projects could impede it and jeopardize the overall effort. | ||
| + | |||
| + | When defining this metric, some PMOs categorize overdue projects in order to analyse and solve the high-priority ones first. Targets for this metric should be established based on the nature of the business, client expectations, and the level of competition. | ||
| + | |||
| + | ====Percentage of portfolio components with missed milestones==== | ||
| + | |||
| + | A project is typically composed of several phases, starting from its conception and ending with product delivery or launch. Each phase contains several activities performed by multiple disciplines, resulting in a significant interim goal, or a milestone. These milestones represent crucial checkpoints in the project's lifecycle, with each having a deadline that must be met to maintain project compliance <ref name=PPMPG> </ref>. | ||
| + | |||
| + | This metric evaluates the percentage of milestones missed during the development of the projects, out of the total number of project milestones <ref name=Kerzner> </ref>. In other words, it assesses the project's adherence to the original milestones schedule. | ||
| + | |||
| + | Regarding targets, organizations should aim to set them as low as possible while still being achievable so as to not lead to unrealistic goals. | ||
| + | |||
| + | ====Resource utilization and availability==== | ||
| + | |||
| + | The resources available within a portfolio can be categorized into four major categories <ref name=PMBOOKProject></ref>: | ||
| + | * Human capital, referring to the availability of human resources to support the portfolio. | ||
| + | * Financial capital, referring to the availability of funds to support the portfolio. | ||
| + | * Assets, referring to the availability of physical assets, such as machinery, office space, factory sites, etc. | ||
| + | * Intellectual capital, referring to the availability of patents, copyrights, etc. | ||
| + | |||
| + | The objective of managing resource utilization is to ensure that organizations optimally allocate resource capacity against resource requirements or demand based on the strategic goals of the organization and the composition of the portfolio. Since portfolio resources are always constrained, it is crucial to continuously review and balance supply and demand. | ||
| + | |||
| + | Remarkably, bottleneck resources are typically critical human ones. Therefore, it is relevant to measure their utilization. Overall resource utilization is a method used to measure the total utilization of a resource against its capacity or availability. It represents the cumulative time that employees spend on their respective activities. It can be calculated by comparing the total employees' available time against the total work assigned to them. | ||
| + | |||
| + | ====Stakeholder satisfaction==== | ||
| + | |||
| + | Stakeholder satisfaction is a metric used to assess how satisfied the stakeholders are with the portfolio based on different categories <ref name=PMASA> </ref>. This metric is challenging to measure as it represents a subjective evaluation of performance. Typically, it is evaluated by asking stakeholders to rate their satisfaction on a numerical scale through surveys. To ensure the accuracy of the data, it's crucial to carefully design the survey questions and obtain a representative sample of stakeholders. | ||
| + | |||
| + | ====Percentage of projects that fit organizational strategy==== | ||
| + | |||
| + | Undoubtedly, strategy alignment is a value metric par excellence. As a value metric, it is calculated as the result of different factor or attributes. Time, cost and profitability are generally attributes of every value metric and strategy alignment is not the exception <ref name=Kerzner> </ref>. Normally, other attributes included in this value metric are stakeholder satisfaction, level of innovation and market share growth. | ||
| + | The first step is to establish targets with thresholds for each attribute or component. Depending on how each component performs against its own target, it will receive specific value points. Then, each attribute is weighted according to its relevance to the metric being measured. Following, the weighting factors can be multiplied by the value points and added up to obtain the total value contribution of an specific project. Finally, this final value contribution is compared with the value reference previously defined as the minimum value contribution to determine that a project is align with the organization strategy. | ||
| + | |||
| + | The following table, show a brief example of results from this approach. If, for example, the value reference of strategy alignment is 2.0, the following project would meet and exceed its performance against this target. | ||
| + | |||
| + | {| class="wikitable" | ||
| + | |- style="font-weight:bold;" | ||
| + | ! style="vertical-align:bottom;" | VALUE COMPONENT | ||
| + | ! style="vertical-align:bottom;" | WEIGHTING FACTOR | ||
| + | ! style="vertical-align:bottom;" | VALUE MEASUREMENT | ||
| + | ! VALUE CONTRIBUTION | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | style="color:#242021;" | Time | ||
| + | | 10% | ||
| + | | 3 | ||
| + | | 0.3 | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | style="color:#242021;" | Cost | ||
| + | | 20% | ||
| + | | 2 | ||
| + | | 0.4 | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | style="color:#242021;" | Profitability | ||
| + | | 20% | ||
| + | | 4 | ||
| + | | 0.8 | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | style="color:#242021;" | Level of innovation | ||
| + | | 30% | ||
| + | | 2 | ||
| + | | 0.6 | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | style="color:#242021;" | Stakeholders' satisfaction | ||
| + | | 20% | ||
| + | | 3 | ||
| + | | 0.6 | ||
| + | |- style="vertical-align:bottom;" | ||
| + | | | ||
| + | | | ||
| + | | style="font-weight:bold;" | TOTAL | ||
| + | | 2.7 | ||
| + | |} | ||
| + | |||
| + | The aforementioned process can be repeated for every project in the portfolio. The percentage of projects that meet their target can then be calculated and reported as the ones algined with the organization strategy. | ||
| + | |||
| + | ====Other relevant metrics==== | ||
| + | |||
| + | * Time spent approving projects <ref name=Kerzner> </ref>. | ||
| + | * Percentage of portfolio components that have undergone scope reduction <ref name=Kerzner> </ref>. | ||
| + | * Percentage of portfolio components that required rework <ref name=Kerzner> </ref>. | ||
| + | * Percentage of portfolio components that are used to run the business, grow the business, and to innovate <ref name=Kerzner> </ref>. | ||
| + | * Deviation of planned hours of work for projects <ref name=PMASA> </ref>. | ||
| + | * Average deviation of planned versus actual duration of components <ref name=PMASA> </ref>. | ||
| + | * Status of escalated risks and issues <ref name=PMASA> </ref>. | ||
| + | * Number of new components in the pipeline with their attributes and information <ref name=PMASA> </ref>. | ||
| + | |||
| + | ===Portfolio reporting tools=== | ||
| + | |||
| + | [[File:Portfolio dashboard Strategic Approach.png|thumb|500px|right|Figure 4: sample dashboard that has high-level details of projects and | ||
| + | their health, in addition to the high-level information related to the project portfolio. <ref name=PMASA> </ref> ]] | ||
| + | |||
| + | Report and strategically visualize the performance of the portfolio through different tools is as important as the metric measurements themselves. To effectively communicate with stakeholders in a project or portfolio, it is important to recognize that each stakeholder may require different metrics and KPIs. However, showing them that critical data can be accessed with only 6 to 10 metrics or KPIs displayed on one screen can help them make informed decisions. For this specific purpose dashboards and scorecards are two effective data visualization solutions within a performance measurement system <ref name=Kerzner> </ref>. | ||
| + | |||
| + | ====Dashboards==== | ||
| + | |||
| + | Dashboards are used in operationally-oriented performance measurement systems to measure performance against targets and thresholds using real-time data. Specifically, they are communication tools for providing data to viewers. If the goal is to communicate the big picture quickly, perhaps in summary format, dashboards are the way to go. Dashboards often display performance visually, using charts or simple graphs. One key aspect of them is the availability of actions that a stakeholder could decide to take based on the available information <ref name=PMASA> </ref>. | ||
| + | |||
| + | ====Scorecards==== | ||
| + | |||
| + | Scorecards are used in strategically-oriented performance measurement systems to track progress towards achieving strategic goals and objectives by comparing performance against targets and thresholds. They are similar to performance charts used for tracking progress towards goals, but they are designed to provide monthly summaries of data for executives who track strategic and long-term objectives, or daily and weekly snapshots for managers overseeing the set of projects under the portfolio management. | ||
| + | Summarized data is presented in a visually appealing way, using charts and graphs to indicate performance status, trends, and variance from goals. Visual representation is preferred by higher-level organizational users. However, scorecards should also include textual commentary to interpret performance results, describe actions taken, and forecast future results <ref name=Kerzner></ref>. | ||
| + | |||
| + | <!-- Also consider EVA as metric since it provides an early warning system for schedule and cost overruns. The four basic measurements are BAC, PV, EV and AC and they are used to calculate the following metrics: | ||
| + | |||
| + | Following, the main financial metrics | ||
| + | |||
| + | Portfolio managers should decide what to prioritize between all the opportunities (projects) since the resources are not unlimited. As it was mentioned previously, the PMO pursues the strategic objectives of the company and success is defined in terms of value that is expected. Thereby, projects in portfolio have to be selected and monitorized that can maximize the value, based on the given constraints placed on the available resources. In order to do this, PMO are required to queantify value. | ||
| + | |||
| + | Certainly, since the definition of what value means depends on each organization, the metrics created to measure the alignment of the portfolio with these values do as well. However, | ||
| + | |||
| + | Specifically, Kezner <ref name=Kerzner></ref>, defined four main value categories (internal, financial, customer-related and future value) and their associated tracking metrics. These four categorias have a strong relationship with the performance perspectives considered in the Balanced Scorecard (BSC) <ref name=BSC> Kaplan, R. S., & Norton, D. P. (1996) <<The balanced scorecard: Translating strategy into action>> Harvard Business School Press </ref>. --> | ||
==Limitations== | ==Limitations== | ||
| − | It is relevant to note that advances in project metrics have been rapid, but advances in portfolio metrics have been slow because not all companies maintain a project management office (PMO) dedicated to portfolio management activities. This leads to changes in the role of the project manager, the metrics used, and the dashboard displays <ref name=Kerzner></ref>. | + | It is relevant to note that advances in project metrics have been rapid, but advances in portfolio metrics have been slow because not all companies maintain a project management office (PMO) dedicated to portfolio management activities. This leads to changes in the role of the project manager, the metrics used, and the dashboard displays <ref name=Kerzner></ref>. |
| + | |||
| + | On the other hand, the intangibility of value-based metrics presents barrier when defining them: | ||
| + | * Value is almost always based on subjective attributes of the person doing the measurement. | ||
| + | * The perceived value may be more of an art than a science. | ||
| + | * Limited models are available to perform measurement. | ||
| + | |||
| + | ==Annotated Bibliography== | ||
| + | |||
| + | '''Project Management Institute (2017). The Standard for Portfolio Management. PMBOOK 4th Edition.''' | ||
| + | |||
| + | '''Harold Kerzner (2017). Project Management Metrics, KPIs, and Dashboards. Wiley & Sons.''' | ||
| − | + | '''G. Levin & J. Wyzalek (2015). Portfolio Management - A strategic approach. Taylor & Francis Group''' | |
==References== | ==References== | ||
Latest revision as of 22:55, 9 May 2023
[edit] Abstract
According to the Project Management Institute (PMI), a metric is a "description of a project or product attribute and how to measure it" [1]. It is, indeed, a quantitative measure of performance or outcome that provides a basis for comparison, evaluation, or improvement.
Peter Drucker famously said, "If you cannot measure it, you cannot improve it". However, it is not enough just to measure. The ultimate purpose of metrics is, as well, to provide the right information to the right person at the right time, using the correct media and in a cost-effective manner [2]. Notably, building effective metrics is extremely relevant when it comes to decision making at all levels of a company, project or portfolio. Thereby, knowing specifically what to measure when it comes to portfolio management is a must to ensure the future of the organization.
The purpose of this article is to describe the type of metrics needed to evaluate Portfolio performance and outline their differences from the well-known metrics to measure Project performance. Firstly, this article will focus on introducing to general terms. Secondly, the relevant role of the portfolio management will be highlighted. Hereafter, what is needed to create an effective metric will take part in this essay. Finally, different useful metrics and techniques for portfolio management will be listed and explained in detail.
[edit] General terms
Firstly, to understand the topic under study, both the basics of Portfolio Management and Metrics need to be introduced.
[edit] Portfolio Managements
As the PMI states [1], a portfolio is a collection of projects, programs, subsidiary portfolios, and operations managed as a group to achieve strategic objectives. More importantly, those programs or projects within a portfolio may not be directly related or interdependent, they are linked to the organization’s strategic plan by means of the portfolio.
On the other hand, portfolio management is a crucial organization capability that enables them to effectively manage and balance their portfolio, align projects with their overarching strategy, and ensure the allocation of resources in order to maximize benefits [4]. The PMI [3] offers a broader definition and additionally, points out how portfolio management seeks for this alignment between the portfolio components and the organization strategy which are selecting the best portfolio components prioritizing the work, providing the needed resources, overseeing or working with portfolio component managers on their implementation, supervising proper transition into the operational environment, and enabling the achievement of portfolio value. Furthermore, Harvey Levine in his elaborated Project Portfolio Management Guide [5] mentions what portfolio management definitely is not: the error lies in assuming that it is primarily concerned with managing numerous individual projects, but this is not accurate. The real focus of portfolio management is on managing the collection of projects as a whole, with the aim of maximizing their overall contribution to the overall welfare and success of the enterprise.
On the other hand, when discussing the process of portfolio management, there are two key components or phases to consider:
- Prioritization and selection of potential projects for the portfolio. This evaluation requires taking into account the opportunities (value and benefits) as well as the risks (which can alter the expected benefits)
- Maintaining the pipeline: continuing, delaying, or terminating approved projects. Projects are periodically assessed based on the established criteria for selection and acceptable performance
[edit] Metrics
The simplest definition of metric is the one stated by Cambridge: "System for measuring something" [6]. Even though it turns out to be a poorly detailed definition, it lays the foundations for what a metric does. In fact, the core metric function is to measure something. What is being measured and why these are taking place are the components that make metrics relevant. Looking deeper, metrics are tools of quantitative evaluation that can be utilized for the purpose of monitoring and measuring performance and success. At business or company management level, metrics become even more relevant when they are added to a set of metrics to construct a dashboard that managers and analysts regularly consult to ensure that performance assessments and business strategies are maintained.
Metrics require a need and purpose and last but not least, might need both as reference points: target and baseline. On the one hand, it is necessary to specify what is intended to be measured in order to determine why this metric has to exist in the first place. On the other hand, targets need to be defined. If there is no point of comparison, what is going to be measured against? The metric would not have a raison d'être [2].
[edit] Roles and Responsibilities of a Portfolio Management Office
As the Portfolio Standards of PMI state [3], Portfolio managers have the responsibility for stating and implementing portfolio management. Among other contributions, the Portfolio Management Office (PMO) supports portfolio management by:
- Identifying, analysing, coordinating, negotiating, monitoring, and controlling portfolio components; supporting component proposals and evaluations; facilitating prioritization; authorization; termination of components; and facilitating the allocation of resources in alignment with organizational strategy and objectives.
- Coordinating communication across portfolio components.
- Providing program and project progress information and metric reporting utilizing key performance indicators (KPIs) (e.g., expenditure, defects, resources) to the portfolio governance process.
The PMO office can provide recommendations on selecting, terminating, or initiating actions to maintain alignment with the organization's strategic objectives. Additionally, the portfolio management office contributes to the project or program management office with performance evaluations of portfolio components based on predefined metrics.
Remarkably, the Portfolio Management Office (PMO) generally establishes the metrics needed to validate the performance of the overall portfolio of projects. To be effective, Michael Wood [7] specifies that the PMO needs to define the success metrics that are going to be used to determine how well each project and the overall PMO organization is delivering value within and across the performance domain framework. The PMO might also want to weigh each Performance Domain (PD) in terms of how important is it in determining the PMO’s, and each project’s, success. Each metric should directly align with each domain’s intended outcomes to support efficient oversight.
According to Harold Kerzner [2], the PMO should address three crucial inquiries, including:
- Whether or not the company is engaged in the proper projects, which entails determining if the projects align with strategic objectives and support strategic initiatives;
- Whether or not the company is involved in a proper number of appropriate projects to maximize investment and shareholder value; and
- Whether or not the company is executing the proper projects in the right manner, which involves assessing project completion time and cost.
Surprisingly, the term "proper" is present in all three questions and implies a sense of value.
[edit] Effective Metrics
[edit] Gold rules
Efficient use of time and resources is crucial in portfolio management, and progress measurement is no exception. Consequently, it is relevant for portfolio managers to point to the development of metrics that only measure relevant and useful insights for decision making. According to the PMI Standard [1] and George T. Doran [8], the SMART criteria is one of the golden rules to be followed in order to create effective metrics. Specifically, the acronym SMART stands for:
- Specific: measurements are specific as to what to measure. Define what is being measured
- Meaningful: metrics ought to be linked to the business case, baselines, or requirements. Evaluating project achievements that do not contribute to meeting objectives or enhancing performance are ineffective. Besides, metrics' meaningfulness is commonly related to their nature of being quantifiable or "measurable" against the baseline and stated requirements.
- Achievable: it is meaningless to establish unfeasible or out-of-reach targets. Targets defined need to be attainable given the people, technology, and environment considered.
- Relevant: the information provided by the measure should represent value and allow actionable information.
- Time-bound: metrics have to be associated to a specific time frame. Creating these boundaries allows to keep the measurements on track and accountable for.
[edit] Value based metrics vs. Traditional metrics
In 1996, Norton and Kaplan introduced the Balanced Scorecard (BSC) [9], a tool that already expanded beyond the traditional financial metrics to include non-financial metrics. The Balanced Scorecard framework aims to achieve desired outcomes aligned with a business' strategy. This includes creating value for present and future stakeholders, as well as leveraging internal capabilities and investments in people, systems, and procedures to improve future performance
Later in time, "Project Management Metrics, KPIs, and Dashboards" was presented by Harold Kerzner [2]. It highlights the key differences between traditional metrics and value-based metrics and explains how each type of metric can be useful. It is argued that traditional metrics are useful for tracking project performance and ensuring that it is completed within the constraints of the triple constraint model (time, cost, and scope) called Iron triangle. However, these metrics may not necessarily provide a comprehensive view of the projects' overall value to the organization.
On the other hand, value-based metrics are centred on the strategic objectives and intended benefits. They help to ensure that the projects are aligned with the organization's strategic objectives and that it generates the expected benefits. However, value-based metrics may be more difficult to measure and track than traditional metrics. The definition of value and how it is measured can differ depending on the industry, company, and the business characteristics. Different stakeholders may have their own ideas of what value means for them, such as job security, profitability, reputation, and the creation of intellectual property. It can be challenging to meet the needs and expectations of all stakeholders at the same time.
Notably, while traditional metrics have as primary audience the project manager and the team, value (or value reflective) metrics are more useful for the clients/stakeholders.
The importance of the value component in the definition of success cannot be overlooked. When management makes poor decisions managing project portfolios, companies end up working on the wrong projects. For example, the deliverable requested can be produced but either there is no market for the product or stakeholders may be displeased with management’s performance.
In any case, Kerzner suggests that the use of value-based metrics and traditional metrics in combination can provide a more accurate and comprehensive assessment of the project's performance. While the traditional metrics can be used to monitor the performance of the project and ensure that it is completed within the constraints of the triple constraint model, the value-based metrics can be used to measure the project's overall impact on the organization and assess its contribution to the organization's strategic objectives.
Regarding assessment and management at portfolio level, value-based metrics clearly gain greater importance than traditional ones: portfolio success has to be defined in terms of the value that was expected to be delivered.
[edit] Portfolio Performance Measurement Tools and Metrics
[edit] Portfolio metrics
According to Robert G. Cooper [5], an effective portfolio management system should accomplish three primary goals that serve as a framework for defining metrics by determining the corresponding targets:
- Ensure strategic alignment: the primary objective is to ensure that the final portfolio of projects accurately reflects the business's strategy by focusing on strategic alignment. This involves selecting and keeping those projects that are either in line with the strategy, support it, or are integral components of it. Additionally, the distribution of spending across various projects, areas, and markets must be directly linked to the business strategy.
- Maximize the value of the portfolio: here the goal is to optimize the financial value of the portfolio for a given spending level by strategically allocating resources. This requires selecting projects that maximize their commercial value in relation to a specific business objective.
- Seek the right balance of projects: it is important to achieve the desired balance of projects based on several factors, such as long-term vs. short-term projects or high-risk vs. low-risk projects across various markets, technologies, product categories, and project types.
From the previous three goals, the easiest and most transparent to measure is the second one: the financial value of the portfolio. Historically, the measurement system of companies has been financial. In fact, accounting has been called the language of business. There is evidence of financial accounting records that date back thousands of years. Reportedly, the set of well-known metrics employed to measure economic performance are almost standardized towards different companies and, most of them, have an intrinsic relationship with NPV (Net Present Value) and ROI (Return on Investment). At the same time, financial metrics are usually the main drivers of a performance management system.
There are countless possible indicators for measuring the performance of portfolios and certainly, new metrics are being created every day. This is because each enterprise, organization, and sector has different approaches to determining what is important to measure and how to measure progress toward short and long-term goals and objectives. With that said, the following section describes some of the most popular and effective metrics according to the analysed literature.
[edit] Average of EVA measurements
Earned Value Analysis (EVA) is an essential aspect of project management for performance reviews. It is a practical method for monitoring and assessing the performance of ongoing projects with the goal of identifying the ones that no longer align with the conditions on which approval was granted and considering their termination [5]. The four basic measurements related to EVA are [10]:
- Budget of Completion (BAC): it is the total budget baseline for the activity, work package, or project.
- Planned Value (PV): it is the time-phased budget baseline. It is the approved budget for accomplishing the activity, work package, or project related to the schedule. It can be viewed as the value to be earned as a function of accomplishments up to a given point in time.
- Earned Value (EV): it is the earned value for the work completed to a point in time. Indeed, this is the percent complete of a task times the budget (BAC).
- Actual Cost (AC): it is the actual cost spent to accomplish an activity or project and to earn the related value up to a given point in time.
At portfolio level, what matters is to obtain the periodic information about the average of the cost performance index (CPI), schedule performance index (SPI), schedule variance (SV), and cost variance (CV) of the projects overseen by the portfolio [4].
Following, a brief explanation of these terms an how to calculate them is displayed below based on Frank Anbari's wrtitings [10].
The first set of queations are used to calculate the variances, generally based on cumulative data:
- The Cost Variance (CV) is a measure of the conformance of actual cost of work performed to the budget: CV = EV - AC
- The Schedule Variance (SV) is a measure of the conformance of actual progress to the schedule: SV = EV - PV
In the above equations, “0” indicates performance is on target. A negative value indicates poor performance. A positive value indicates good performance.
The following equations are used to calculate the performance indices:
- The Cost Performance Index is a measure of the conformance of actual cost of work performed to the budget: CPI = EV / AC
- The Schedule Performance Index is a measure of the conformance of actual progress to the schedule: SPI = EV / PV
In the above equations, “1” indicates performance is on target. More than “1” indicates good performance. Less than “1” indicates poor performance. Looking at the SPI, we can easily see that we are behind schedule.
[edit] Percentage of overdue portfolio components
As its name defines it, this metric measures the percentage of overdue projects out of the total number of them in a portfolio [4]. Its main purpose is to monitor the portfolio's progress since a large number of overdue projects could impede it and jeopardize the overall effort.
When defining this metric, some PMOs categorize overdue projects in order to analyse and solve the high-priority ones first. Targets for this metric should be established based on the nature of the business, client expectations, and the level of competition.
[edit] Percentage of portfolio components with missed milestones
A project is typically composed of several phases, starting from its conception and ending with product delivery or launch. Each phase contains several activities performed by multiple disciplines, resulting in a significant interim goal, or a milestone. These milestones represent crucial checkpoints in the project's lifecycle, with each having a deadline that must be met to maintain project compliance [5].
This metric evaluates the percentage of milestones missed during the development of the projects, out of the total number of project milestones [2]. In other words, it assesses the project's adherence to the original milestones schedule.
Regarding targets, organizations should aim to set them as low as possible while still being achievable so as to not lead to unrealistic goals.
[edit] Resource utilization and availability
The resources available within a portfolio can be categorized into four major categories [1]:
- Human capital, referring to the availability of human resources to support the portfolio.
- Financial capital, referring to the availability of funds to support the portfolio.
- Assets, referring to the availability of physical assets, such as machinery, office space, factory sites, etc.
- Intellectual capital, referring to the availability of patents, copyrights, etc.
The objective of managing resource utilization is to ensure that organizations optimally allocate resource capacity against resource requirements or demand based on the strategic goals of the organization and the composition of the portfolio. Since portfolio resources are always constrained, it is crucial to continuously review and balance supply and demand.
Remarkably, bottleneck resources are typically critical human ones. Therefore, it is relevant to measure their utilization. Overall resource utilization is a method used to measure the total utilization of a resource against its capacity or availability. It represents the cumulative time that employees spend on their respective activities. It can be calculated by comparing the total employees' available time against the total work assigned to them.
[edit] Stakeholder satisfaction
Stakeholder satisfaction is a metric used to assess how satisfied the stakeholders are with the portfolio based on different categories [4]. This metric is challenging to measure as it represents a subjective evaluation of performance. Typically, it is evaluated by asking stakeholders to rate their satisfaction on a numerical scale through surveys. To ensure the accuracy of the data, it's crucial to carefully design the survey questions and obtain a representative sample of stakeholders.
[edit] Percentage of projects that fit organizational strategy
Undoubtedly, strategy alignment is a value metric par excellence. As a value metric, it is calculated as the result of different factor or attributes. Time, cost and profitability are generally attributes of every value metric and strategy alignment is not the exception [2]. Normally, other attributes included in this value metric are stakeholder satisfaction, level of innovation and market share growth. The first step is to establish targets with thresholds for each attribute or component. Depending on how each component performs against its own target, it will receive specific value points. Then, each attribute is weighted according to its relevance to the metric being measured. Following, the weighting factors can be multiplied by the value points and added up to obtain the total value contribution of an specific project. Finally, this final value contribution is compared with the value reference previously defined as the minimum value contribution to determine that a project is align with the organization strategy.
The following table, show a brief example of results from this approach. If, for example, the value reference of strategy alignment is 2.0, the following project would meet and exceed its performance against this target.
| VALUE COMPONENT | WEIGHTING FACTOR | VALUE MEASUREMENT | VALUE CONTRIBUTION |
|---|---|---|---|
| Time | 10% | 3 | 0.3 |
| Cost | 20% | 2 | 0.4 |
| Profitability | 20% | 4 | 0.8 |
| Level of innovation | 30% | 2 | 0.6 |
| Stakeholders' satisfaction | 20% | 3 | 0.6 |
| TOTAL | 2.7 |
The aforementioned process can be repeated for every project in the portfolio. The percentage of projects that meet their target can then be calculated and reported as the ones algined with the organization strategy.
[edit] Other relevant metrics
- Time spent approving projects [2].
- Percentage of portfolio components that have undergone scope reduction [2].
- Percentage of portfolio components that required rework [2].
- Percentage of portfolio components that are used to run the business, grow the business, and to innovate [2].
- Deviation of planned hours of work for projects [4].
- Average deviation of planned versus actual duration of components [4].
- Status of escalated risks and issues [4].
- Number of new components in the pipeline with their attributes and information [4].
[edit] Portfolio reporting tools
Report and strategically visualize the performance of the portfolio through different tools is as important as the metric measurements themselves. To effectively communicate with stakeholders in a project or portfolio, it is important to recognize that each stakeholder may require different metrics and KPIs. However, showing them that critical data can be accessed with only 6 to 10 metrics or KPIs displayed on one screen can help them make informed decisions. For this specific purpose dashboards and scorecards are two effective data visualization solutions within a performance measurement system [2].
[edit] Dashboards
Dashboards are used in operationally-oriented performance measurement systems to measure performance against targets and thresholds using real-time data. Specifically, they are communication tools for providing data to viewers. If the goal is to communicate the big picture quickly, perhaps in summary format, dashboards are the way to go. Dashboards often display performance visually, using charts or simple graphs. One key aspect of them is the availability of actions that a stakeholder could decide to take based on the available information [4].
[edit] Scorecards
Scorecards are used in strategically-oriented performance measurement systems to track progress towards achieving strategic goals and objectives by comparing performance against targets and thresholds. They are similar to performance charts used for tracking progress towards goals, but they are designed to provide monthly summaries of data for executives who track strategic and long-term objectives, or daily and weekly snapshots for managers overseeing the set of projects under the portfolio management. Summarized data is presented in a visually appealing way, using charts and graphs to indicate performance status, trends, and variance from goals. Visual representation is preferred by higher-level organizational users. However, scorecards should also include textual commentary to interpret performance results, describe actions taken, and forecast future results [2].
[edit] Limitations
It is relevant to note that advances in project metrics have been rapid, but advances in portfolio metrics have been slow because not all companies maintain a project management office (PMO) dedicated to portfolio management activities. This leads to changes in the role of the project manager, the metrics used, and the dashboard displays [2].
On the other hand, the intangibility of value-based metrics presents barrier when defining them:
- Value is almost always based on subjective attributes of the person doing the measurement.
- The perceived value may be more of an art than a science.
- Limited models are available to perform measurement.
[edit] Annotated Bibliography
Project Management Institute (2017). The Standard for Portfolio Management. PMBOOK 4th Edition.
Harold Kerzner (2017). Project Management Metrics, KPIs, and Dashboards. Wiley & Sons.
G. Levin & J. Wyzalek (2015). Portfolio Management - A strategic approach. Taylor & Francis Group
[edit] References
- ↑ 1.0 1.1 1.2 1.3 Project Management Institute (2021) <<Project Management: A guide to the Project Management Body of Knowledge and The Standard for Project Management>> PMBOOK guide 7th Edition
- ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 2.12 2.13 Harold Kerzner (2017) <<Project Management Metrics, KPIs, and Dashboards>>, Wiley & Sons
- ↑ 3.0 3.1 3.2 Project Management Institute (2017) <<The Standard for Portfolio Management>> 4th Edition
- ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 G. Levin & J. Wyzalek (2015) <<Portfolio Management - A strategic approach>> Taylor & Francis Group
- ↑ 5.0 5.1 5.2 5.3 Harvey A. Levine (2005) <<Project Portfolio Management - A practical guide to selecting projects, managing portfolios, and maximizing benefits>> Jossey-Bass
- ↑ Cambridge dictonary, Cambridge University Press & Assessment 2023
- ↑ Michael Wood (2021) <<Are You Effectively Measuring the Right Success Metrics?>> Project Management Institute
- ↑ Doran, G.T. (1981) <<There's a S.M.A.R.T. way to write management's goals and objectives>>
- ↑ Kaplan, R. S., & Norton, D. P. (1996) <<The balanced scorecard: Translating strategy into action>> Harvard Business School Press
- ↑ 10.0 10.1 10.2 Anbari, F. T. (2001) <<Applications and extensions of the earned value analysis method>> PA: Project Management Institute
