The Decision Matrix

Developed by Katrine Bager Olesen

Innovation within technology is moving faster than ever. Companies are forced to keep innovate new products or improve exsisting products. Due to strict time frames and many different competitors on the technological market, decision-making within product-development/innovation have to be credible and solid. However, in the decision making process in product-development/innovation, especially in the beginning of the innovation-funnel,[1] low knowledge and many uncertainties are dominating. By using the so-called ”Decision Matrix” (DM), knowledge can be increased and uncertainties can be decreased. Hence, in product-development/innovation the DM is extremely relevant and may create more knowledge and turn these uncertainties into measurable quantities and make a project tangible.

This wiki-article wishes to give a deeper insight and background description of the DM e.g. The Concept Screening Matrix and examine both the quantitative and qualitative approches (Stuart Pugh's approach).[ In addition the application procces will also be examined by focusing on innovation-projects including a ”Six Steps”-process description. Moreover, it will contain which parameters/criteria are key and appropriate in order to obtain an realistic and solid output when applying the DM, e.g. define the end-goal and stakeholders (customers), reflect on output and list the weight of parameters/criterion.

Challenges and difficulties when using the DM will also be examined focusing on how to chose the DM parameters/criteria and their weights and how to relate objectively to the outcome and how to use the DM as an iterative tool. Finally, suggestions are identified in order to accomodate and improve the challenges and difficulties and to identify for what the tool is applicable, for when the DM is applicable, how the DM is applicable and for who it is applicable and who can benefit from the DM.

The structure of the article is as follows:

- Concept desription of the tool, the DM (both quantitative and qualitative approaches) and description of a simplified example

- Application in innovation-projects and choice of approriate parameters/criteria when applying the DM

- Challenges when applying the DM

- Suggestions for improving the challenges of the DM

Introduction

The following wiki-article focus on the tool the Decision Matrix (DM). The DM is based on a set of different candidates and criteria in order to make the right decision. The DM is hence a very relevant tool for both Programs, Projects and Portfolios since decision making, no matter the complexity, is an inevitable factor in all perspectives of achieving an end-goal.

The DM may help both in everyday-problems like “What should we have for dinner?”, but the DM can also be conducted in more complex situations as a management tool. E.g. in innovation and development projects, where problems like “Which products should we improve?” and in bid-situations where many criteria are influencing the final candidate of the bid.

The article will elaborate on the concept of the DM, application of the DM in an innovation-project relation, challenges in relation to the DM and suggestions to improvements of the DM.

Big Idea

The basic concept

The Decision Matrix (DM) is a decision-making tool invented by Stuart Pugh in 1990 and hence the method is also known as the Pugh-Method. Initially the Pugh-Method was based upon qualitative assessment using only “+” and “-”. The DM has later on evolved to also use a quantitative assessment and hence a grading system. By using the DM a quick overview of different alternative solutions (candidates) is formed., [

The DM contains a list of candidates. Each of the candidates are judged and evaluated based on a row of different criteria. The candidates with the highest score is chosen as the final candidate and the decision is made. However, if two candidates score the same, a new iteration is performed. Moreover, if the external environment changes and influences the criteria and/or the candidates, a new DM needs to be formed. Hence, the DM should be perceived as an iterative tool. Table 1 shows the basic DM without any ranking:

Table 1. The basic Decision-Matrix, with four candidates and four criteria (this may vary relating to the complexity), a total sum column showing the best candidate and an additional comment column.

To elaborate how the DM works a very simplified example has been established in table 2. The example concerns a decision everyone has to make during their life and hence can relate to:

“What do I want to eat for breakfast?”

A list of optional candidates and criteria are listed in the DM. They are rated by using a quantitative grading system: 1-5, where 5 is the best.

Table 2. Example ”What do I want to eat for breakfast today?”

A lot of different questions may arise in relation to this example e.g. ”What if the price is more important to me than the taste”, “How do I select the different candidates?” etc. In the following section the process in choosing the final candidate is elaborated via Six Steps. Moreover, the rating is based on both objective criteria like the price but also on a subjective criterion like the taste. The challenge in using subjective criteria is elaborated in the section Challenges.

The Six Steps process

Whenever the DM is conducted, a specific process should be followed. The process consists of Six Steps. The different steps vary in complexity and hence in duration:

1. Prepare the matrix by setting up all optional candidates. Then identify which criteria the decision should be based upon. Compared to table 2 different types of breakfast are the candidates and the price, taste etc. are the criteria. This step is properly the most complex step. The candidates and criteria are the basic when applying the DM. The DM follows the concept that the output is a reflection of the input. So when the input is based on solid research the output will be reliable. Under the Application section an elaboration is explained, how to identify the most appropriate criteria.
2. Rating of the candidates can be done either by using a qualitative or quantitative approach. In the given example from table 2 ”What do I want to eat for breakfast today?” the quanititative approach is conducted. When many different candidates are occurring a qualitative approach is suitable. However, this is most suitable during the initial phase when you have to narrow down the candidates rather quickly. Moreover, a reference candidate can be chosen and the other candidates are then compared to the reference candidate and rated. This is also referred to as “The Concept Screening Matrix”. In this approach only "+", "0" or "–" are assigned. This gives a quick overview of potential candidates to pursue and hence is suitable in an early stage. From table 3, the usual breakfast is chosen as reference and the other candidates are rated compared to the usual breakfast. If there are more + than the reference then you should eat that breakfast candidate. The quantitative approach is based upon a scale like the one from table 2 and like the example, the candidate with the highest ranking is chosen. In some cases, a candidate who has a higher score for all criteria might occur and will naturally be chosen and decision process will end. This type of candidate is referred to as a “Dominant Candidate”.
3. Ranking each criterion can be assigned with a weight if they are of greater importance for the end-decision. From example, health is twice as important as the price and then health’s rating is multiplied with two.
4. Combine and improve candidates if this is possible and feasible.
5. Picking possible candidates (one or more) to pursue.
6. Reflect on the results and the process. The project team has to agree upon the final candidate, to achieve an optimal output.

Table 3. The Concept Screening Matrix where one candidate is chosen as reference and proceed with oatmeal and Skyr with fruit

Application

In this paragraph, the focus will be on application in innovation projects. The use of the DM is very applicable in the beginning of the innovation funnel where many uncertainties and low knowledge is occurring. It is thus necessary to identify who the customers are and key stakeholders and what are their needs. The reason for this is, as mentioned earlier, that output from the DM relies on the right set of criteria and those are based on these identified needs.

Different approaches to identify the needs are existing, however a general method also follow a Six Steps process in order to select the right criteria:

1. Define the scope of the candidates. This can be done by describing the stakeholders (both internal and external), their needs and goals, identify and define the given market and hence identify the benefit of a product without getting a specific product concept. Moreover, it is important to carefully setup assumptions in order to comply the project scope.
2. Gather raw data from customers. This is important in order to get a basic understanding of the customers. The raw data can be handled by using either:
   1. Interviews: Members from the innovation project discuss needs with the customer in their environment.
   2. Focus groups: A member from the innovation project facilitates a discussion with 8-12 customers.
   3. Observe an existing product in use: Members from the innovation project go out and observe an existing product in use. This can reveal what the customers’ needs really are.
3. Interpret the raw data. After having gathered the raw data it is important to clarify what the product has to do and hence use the identified needs as raw data. There are different approaches to express the needs:
   1. Ask "What does the product have to do?" instead "How does the product work?" By using this approach, technological solutions do not stand in the way and limit the product development.
   2. The level of needs should be on the same detail level as the raw data to secure information.
   3. Use positive phrasing when expressing the needs, however this can be quite challenging. E.g. “I spend a lot of money on breakfast” is very negative whereas “I like cheap food” is more positive but somehow more difficult to phrase.
   4. Use the need as attribute of the product.
   5. Avoid must and should since those words may contribute to limiting the development and innovation project.
4. Organize the needs into hierarchy by writing down needs on a i.e. a post-its, eliminate redundant statements, group the post-its in the needs, review and edit the statements
5. Establish the importance of the needs and compare to other needs.
6. Reflect on the results.

Once all the customers’ needs are gathered and defined and the criteria have been listed the innovation project team can proceed to the second step from Six Steps Process.

The need for applying the DM occurs when several options can be conducted for one end-goal. The DM is expedient when there is no clear option and an objective approach is needed. However, a dominant candidate may occur when rating has been done and this candidate is superior in all criteria . The appropriate phases to conduct the DM is when a project team needs to narrow down options. This is often the case in the early phases of an innovation project or when selecting and accommodate customers’ needs. Even though, this article focuses on innovation projects, the DM is applicable in other areas.

Applicable areas

In the construction industry a tendering process is carried out before a construction site is launched. During turnkey contracts different interested contractors give an offer on the construction. To decide on the final contractor the DM is very appropriate.

There are different criteria that influence the final choice of the contractor. Often architecture, time-frame, operation and maintenance and of course cost are the primary criteria which a contractor is evaluated upon. Moreover, these criteria are also weighted since operation and maintenance may be considered of higher influence on the final decision compared to architecture.

The building owner or its consultants are responsible for ranking, rating and evaluate each candidate based on each criterion. The contractor with the highest score then win the tender. As mentioned earlier it may be possible to combine two different candidates when they have been evaluated. This could result in more contractors then being responsible for subject areas. A new decision-making process could then be launched in order to decide if this solution is more feasible. Hence, new criteria for this DM could be based upon planning (which order should the work be done), more internal challenges with handling the logistic of workers etc. This shows that using the DM is an iterative process and that the DM is a vivid tool.

As it can be read from picture 1 the contract is awarded to the “most economically advantageous tender”, however the tendered prices and the technical solutions are both subcriteria which are weighted equally. The reasons could be that the prices and technical solutions often go hand in hand and that high quality reflects high cost.

Picture 1. A section how to evaluate in a tender process, taken BaneDanmark’ folder concerning “Information about tender, evaluation, negotiation and award”

During procurement of products it is often supplied from an external source. To decide the right suppliers, the DM could be applied. Moreover, if you are the supplier you could also apply the DM in order to decide where the best location is. The criteria used could then be based upon criteria like customers, price, transportation and costs as shown in table 4 below.

Table 4. Example of a simple DM when a production has to choose a new site location.

Due to no weight added to the criteria, it is not possible to determine which location is the most suitable, solely assessed on behalf of table 4. A new iteration should be performed.

Challenges

Even though the DM is a useful tool and gives a quick overview when many options are present, there are still challenges related to the tool.

As mentioned under Application, the DM is an iterative tool. This is especially the case when candidates are being combined or somehow improved. The iterative process also demands a new set of criteria or weights for each iteration. The reason for this is that the criteria and weights may change due to impacts from external environments. Moreover, the candidates should be checked thoroughly in order to see if it is minor details that caused a low score and if it could be fixed easily.

The overall purpose with the DM is to be as objective as possible so decisions are not based upon subjective interests. However, to keep the DM objective, is one of its main challenges.

By using the criteria selecting approach described under Application the risk for subjectivity should be decreased. However, there still might be a risk that the team will manipulate the stakeholders in order to conduct their own interest in the criteria selection even though the manipulation occurs unconsciously. Hence, a solution could be to use an external resource to identify the needs and interests of the relevant stakeholders and execute the six steps process from Application.

The main influences to having an objective and thorough DM is by having objective and clear criteria and have right weight conducted to the criteria. Hence if the input is invalid the output will reflect that. Four common traps during the identification of criteria are:

1. Incorrect criteria: this often happens whenever a team member or the entire team uses their personal interests instead of using the identified criteria from the internal and external stakeholders
2. Incomplete criteria: If all the relevant stakeholders have not been identified a lack of criteria will occur. Important perspectives and needs might be overlooked and hence the criteria will only focus on partial areas. A risk is also that the team/team member overlook important criteria deliberately to get their own interests carried out.
3. Inadequate criteria: This is caused due to lack in criteria definition and may result in confusion and multiple interpretation scenarios. [
4. Identical criteria: This is caused by two different criteria, which indirect indicates the same, but are weigthed differently. An example could be cost and quality as described in Applicable Areas. It is often desirable to have low cost product and high-quality products and both are the criteria are weighted high. However, these two often go hand in hand as good-quality products are often more expensive and, hence the criteria are not representative if they are the only criteria.

Choosing the right criteria is, as elaborated above, really challenging. If the wrong set of criteria are chosen, the DM will indeed has its limitation and the outcome will most likely result in poor decisions and projects will fail.

Improvements and recommendations

One of the key elements for a successful DM are the right criteria. However, as previously mentioned these are highly depended on internal and external stakeholders and their input to the criteria. It is also a fact that with complex problems the specified weights are highly recommended in order achieve the maximum of the DM. As for many cases, the recommended grading system should be performed by using a quantitative scale. This opens up for bigger differentiations during the decision-making-process.

As Pugh’s initial concept was to use a qualitative approach, only using “+” or “-“, this was not performed due to lack of knowledge to a quantitative grading. Pugh was convinced that the DM was to be used as a guideline and not as the resource actually taken decision. As he has stated: “The matrix does not make the decisions: it is simply a procedure for controlled convergence onto the best possible concept and is not composed for absolutes in the mathematical sense; the decisions remain with the user.”

One of the main challenges in conducting an objective DM, and hence take the most suitable decision, lies within the criteria setting. The challenge can be caused due to manipulation of stakeholders. The U.S. Small Business Administration states that often the wrong set of criteria is often when teams/companies get external advisors. To avoid manipulation in criteria it is advised by The U.S. Small Business Administration to always rely on own expertise and experience. As stated on their website: "Oftentimes, people have a tendency to place too much emphasis on what experts say. Remember, experts are only human and have their own set of biases and prejudices just like the rest of us."

Hence, it is key to listen to what the guts are telling you and one way could be to vizualise the outcome of each alternative canditate and afterwards make a critical evaluation. Hence a new iteration is performed. Another improvement could also be to make sure you have enough time to sit down and choose the right set of criteria and choose the right candidates. This might be costly however in the long run this will pay off knowing decisions were made on a solid basis.

The use of the DM can be used in different areas besides innnovation projects e.g. tendering process, procurement. However, the focus for this article has been on innovation projects. By using the DM during early stages innovation projects different candidats can be identified. The candidates may then be prioritized to different projects in order to succeed within a certain program.

As highlighted in the abstract innovation projects are highly dominated by many uncertainties. This condition is, of course, not desirable and hence the goal is to have as few uncertainties as possible. An approach is to provide more knowledge in the early process stage which in the abstract also is described as a threat in the beginning of a project since knowledge is often minimal on this stage. Providing more knowledge to a project can be secured through research, analysis of experiments and former experiences, and thereafter use the results to hopefully minimize the uncertainties by handle them as tangible risks. Hence the number of candidates may be narrowed down and the DM’s iterative approach may continue as knowledge will increase and new risks will be identified.

Annotated bibliography

For further elaboration it is encouraged to research the following sources:

Brooks, Chad (2014) http://www.businessnewsdaily.com/6162-decision-making.html

• The article is also formulating desicion-making tips and tools and explaning common mistakes. Both this and the previous focus on your own guts and states that importance in being open-minded towards other people' knowledge and use that in a constructive way. Both of the articles are good to get a feeling of the downsides and challenges to the DM and explains in an easy, however solid way.

Burge, Dr. Stuart (2009) The Systems Engineering Tool Box

• The paper is a complete to the Pugh-matrix. It clearly guides the reader through the different step of applying the matrix. Moreover it contains visualizations during the steps. The paper is clear but also presents nuanced perspectives in the application of the tool.

Conversation with a Construction Manager from BaneDanmark (12.06.2016)

• Unformal conversation with a construction Manager from BaneDanmark disussing the tender process for public construction projects with focus on the grading system during the tender and the process evolving that.

DTU Engineering Management (13.09.2016) Lecture 42457 Supply Chain Management

• Lecture in the course 42457 Supply Chain Management regarding strategic network design, Location-allocation problem and hence tools and methods to manage the problem.

Sørensen, Niels (2013) Information about tender, evaluation, negotiation and award

• An information folder which inform the tenderers of the tendering, evaluation, negotiation and award procedure. The folder contains general information and the general criteria selection for a tender process.

Ulrich, Karl T.; Eppinger, Steven D. (2013) Product Design and Development

• Description of perspectives on marketing, design and manufacturing where theory and practical cases are explained in order to elaborate and support the theory. The book is very thorough when describing the application scenarios of the tool. The book contains contains a lot of illustrations has illustration showing the different steps when applying. It takes in consideration cross-functional teams and hence reflects reality. It handles complex subjects and communicate them, so there are tangible.

The National Academies Press (2001)Theoretical Foundations for Decision Making in Engineering Design

• Excellent description of the methods, theories and tools during a decsion making process. It gives a broad overview to approaching decision making.

The U.S. Small Business Administration (unknown publish year) https://www.sba.gov/managing-business/leading-your-business/making-decisions

• The is a thorough examination of decision-making in a business. It examine both Tips for Wise Decision-Makingand Common Decision-Making Mistakes. It, precisely, highlights the common mistakes in an manageable way and makes a heavy topic more transparent and easy to understand.