The Delphi Technique in Project Management

Developed by Marco Ronzoni

According to the definition of ISO 21500:2012 a Project is: “A unique set of processes consisting of coordinated and controlled activities with start and end dates, performed to achieve project objectives. Achievement of the project objectives requires the provision of deliverables conforming to specific requirements”. The Delphi technique is used mostly for reaching consensus among experts, regarding the decision making process. Moreover, as a project manager is crucial to think about the impact that possible future events can have on your projects. This technique helps also in forecasting the future with some grade of certainty, estimating likelihood and outcomes. The iterative process consist of anonymous exchange of views, assumptions and prevision given to a facilitator, that will write a conclusive report. The group will read the report of the previous meeting, update their discussion and thoughts for the new report. The process continue in this way until the consensus is reached among all the participants. The anonymity assure the free expression of opinion and encourage openness. The process will start from a broad perspective than in the end will scope the core problems and questions to assure consensus; there are 3 phases to follow:

• Brainstorming: experts list the relevant factors and remove the duplicates.
• Narrowing down: differentiate in panels and factors selection.
• Ranking: rank the factors on own panel, calculate average, assess consensus.

This method can be reiterate many times based on the analysis and the ranking of different factors listed on panels; controlling every time the feedback (it can be done also online with virtual teams). It is used also to identify risks and opportunities, learning from mistakes, promoting a brainstorming session or creating a Work Breakdown Structure.

This article will have the following structure: the general idea will be presented, related to some definitions, steps to follow and different types of Delphi. Then applications, limitations and benefits will be explored (advantages and disadvantages), using a study case that will be presented, highlighting a real application of this procedure; finally there is the conclusion together with some strengths and weaknesses.

Big Idea

Brief history introduction

---

Delphi method was developed in 1953 by Olaf Helmer and Norman Dalkey of RAND Corporation as instrument to forecast the impact of technology on warfare. The name refers to the Delphi Oracle, a priestess at the Apollo temple in ancient Greece.

Definitions

---

According to Linstone and Turoff it is: "A method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem".

Following also the Web Dictionary IIASA it is a: “Technique to arrive at a group position regarding an issue under investigation. The Delphi method consists of a series of repeated interrogations, usually by means of questionnaires, of a group of individuals whose opinions or judgments are of interest. After the initial interrogation of each individual, each subsequent interrogation is accompanied by information regarding the preceding round of replies, usually presented anonymously. The individual is thus encouraged to reconsider and, if appropriate, to change his previous reply in light of the replies of other members of the group. After two or three rounds, the group position is determined by averaging”.

General Concept

---

It is an essential project management technique used to gather information, the goal is to reduce the amount of different responses and reach the consensus among the experts (stated by the studies of Broomfield and Humpris and also McKenna to be reached between 70% and 51%).

This method is recognized, adopted and is it still used nowadays. Treating uncertainty is a central issue in project management: “Project management is about reducing uncertainty step by step until it has been eliminated completely, for better or worse, and we achieved a particular result”, an entire literature about risk management can be found if interested in "How to do Projects" to have more explanations.

The Delphi method tries to break the stereotype that says: “When you get three experts together, you'll often end up with four different opinions.” Every project has his own characteristics and needs, consequently there is often not just one-size-fits all solution, we have to narrow down all parts to find the customized solution for the specific case. The participants in the process do not necessarily need to meet, sometimes they even do not know who are the other participants, only the facilitator knows everything and manages the information flow in order to avoid possible repercussions. The anonymity helps avoiding personal issues between member and the so-called “groupthink”. The phenomenon of groupthink can be really dangerous in an organisation, is marked by the consensus of opinion without thinking critically and evaluating other alternatives and consequences. The core point of this mistake is to do not break the status quo of the group, avoiding conflicts and creativity, this will reduce the pressure of the group toward conformity.

Figure 1: Example of Structure of Questionnaires ,,

Since are scheduled multiple rounds of questions in the panel, the thoughts of the groups are really investigated and studied, the communication has to be facilitated in every moment. This technique is used in the prediction of the future as Helmer , one of the inventors said: “The future is no longer viewed as unique, unforeseeable and inevitable; there are, instead, a multitude of possible futures, with associated probabilities that can be estimated and to some extend, manipulated”, one of the main aim with this method is the anticipation and recognition of needs.

According to Bishop, Williams and Helmer the main feature of consensus are as follow: (1) Anonymity, in order to avoid some dominant character and implemented using private questionnaires; (2) Iteration, the structure with rounds allows the participants to change their opinions along the process; (3) Controlled feedback, showing the different responses of the individuals inside the group; (4) Statistical group response, measuring and judging the summary of all the responses, having much more detailed information.

Respondents usually rate their agreement on a specific statement or sentence with a five point scale, for instance, (1) means “strongly disagree” and (5) “strongly agree”. In the end the topics with 4 as average (or higher) and with standard deviation below 1 are considered areas in which the agreement is found (the so-called: “items of consensus”). There is not the purpose to arrive always at the unanimity among all the experts, conversely a spread of opinions in the end is a common outcome (this can be considered as a limitation and risk).Typical evidences shows that the opinion is converging during the phases and the average response moves in the direction of the true reply; this is mainly due to three different factors explained by Dalkey: social pressure, re-thinking of the problem and movement of information during the feedback session. In the decision making process mainly three kind of information are playing a crucial role: knowledge, speculation and opinion (an entire set of data can be seen in the work of Dalkey ).

The main suggestion to success using this technique is searching for endorsement of an important person during the activity to sustain and keep high the involvement among all the participants. For example, calling the participants after the invitation or think about incentives for each round. Moreover, certain initial requirements are needed as: high level of effort among the participants, high skill level in writing the questionnaires and some time available to conduct the entire process (i.e. 6 weeks). Empirical studies made by Gordon and Helmer reveal that “Delphi Method is superior to conventional uses of groups in problem solving or forecasting”.

Steps

---

Generally, ten steps for the Delphi method are indicated by Dalkey ,:

• 1) Formation of a team on a subject
• 2) Selection of expert panels
• 3) Development of the first questionnaire
• 4) Testing the questionnaire
• 5) Transmission to the panel lists
• 6) Analysis of responses
• 7) Preparation of second round
• 8) Transmission of second round questionnaires to the panel lists
• 9) Analysis of the second round (iteration)
• 10) Preparation and presentation of report

Figure 2: Delphi Method Procedure Overview

Types of Delphi

---

Hanafin , in his literature review, explains some different types of Delphi:

• Classical Delphi: 5 features such as Anonymity, Iteration, Feedback, Statistical group answers and Response stability among experts on a specific issue.
• Policy Delphi: the aim is to generate policy using a structured dialogue, it is possible to have several divergent opinions from experts. The features are: Answer questions individually but together in the meeting, Iteration, Feedback, Polarized group response and conflicts.
• Decision Delphi: used to reach an high grade of involvement among decision makers on social issues, from different hierarchy positions. Here in this type, the names are mentioned, but the responses to the questionnaires are anonymous.

A modified Delphi method is called “The nominal group technique”. Bishop said that the main difference is that the meetings are done face-to-face, (starting with an open-ended question) aiming at a group discussion to reach consensus. Ideas and thoughts about the topic are shared and recorded, each person evaluate the ideas anonymously and individually votes for the best ones keeping the anonymity and finally, a report with the responses is prepared.

NGT is more structured than the Brainstorming technique, obtaining multiple inputs from experts on precise issues. Used mostly in educational contexts, this specific tool is helpful because researches show that individuals produce more ideas when working alone and they do it without sacrificing the quality. As it is for Delphi, the facilitator here has to be an expert in the field or a person with credibility among the participants. This technique it is useful when there are little evidences but some decisions needs to be made, or when there is the need to rank priorities during a group meeting.

Figure 3: Stages of NGT technique

Applications

The most common types of applications are when there is the need to explore an issue with a group of people, when you want to move a group towards consensus, when a particular problem does not permit the use of an analytical tool, when the participants are not able to meet physically or when they are too many. This method can be used for several reasons according to Linstone : helping the decision making process as a group, to gain inputs from experts, to define some needs, to identify results, to prioritize some causes, to suggest solutions, to define future goals. Moreover, other uses are: to anticipate outcomes, to plan and schedule, for problem solving, for data collection, to identify dimensions of a problem or to forecast.

Delphi Method can be applied with small or big group (hundreds of people), but the common way schedule nearly 20 people. It is useful especially when there are many experts to contact, meaning that the consensus is difficult to reach and when the experts are located in different parts of the world.

The first step is to select a group of experts based on the topic that has to be analyzed, later a questionnaire is given to all the participants with instructions for each topic and on how to write opinions, experience and personal researches. These questionnaires are then send to the facilitator that puts together all the comments and send a copy to each participant. At the end of every session all the questionnaire are send again to the facilitator who will decide if there is the need of another round or not, the iteration can be done as many time as necessary to achieve the final consensus.

Other common applications

---

Other common applications in projects, suggested and explained by Hanafin and Linstone are:

• In time management to estimate the duration of activities.
• In cost management to estimate the different costs.
• In risk management to identify the risks.
• In scope management to collect various requirements.
• Government planning.
• Business and industry.
• Culture, family and behavior.
• Education and training.
• Communications.
• International Security.

Limitations and Benefits

The limitations of this method are that the responses can take a lot of time in order to be expressed and elaborated. Also the rate of discussion can be low and sometimes the interaction is not straight as it is in a face to face meeting. Finally, it is possible that the information received from the experts are not going to create new value.

Unfortunately, some disadvantages are highlighted by Hanafin and Helmer :

• Facilitator´s fail (wrong expert or questions selection)
• Long time needed using mail or post
• Participants involvement is a requirement (quality of the responses usually decrease with the passing of time)
• Individual point of view
• Anonymity may lead to lack of responsibility/accountability
• Labour intensive
• Gupta and Clarke also said: ”Conceptual and methodological inadequacies, potential for sloppy execution, crudely designed questionnaires, poor choice of experts, unreliable result analysis, limited value of feedback and consensus, and instability of responses among consecutive Delphi rounds.”
• Little research in assessing the reliability of this method
• Risk of “groupthink” phenomenon

There are also some advantages and benefits that are crucial, explaining the success of this technique and why it is still used nowadays (collected partially by Hanafin and Helmer):

• Versatile technique, lot of different applications
• Group experts from different sectors/backgrounds
• No location limits, not so expensive
• Promote reflection, evaluation and involvement
• Recognition of the contribution of each participant
• Easy analysis of data
• Anonymity: free thoughts
• Snyder in his study said :”It´s adaptability to diverse data collection strategies, decreased peer pressure secondary to anonymity and the ease of condensing opinions of many and varied experts into a few precise statements”
• Overcome the problem of few individual dominating discussions

Study Case

“The Future of the academic library and the academic librarian. A Delphi study reloaded” this case is a nice example of the application of Delphi method in a project. In 1999 in Greece, it was done a study on a project to identify the role of the academic library and librarian, together with their use in the present and future scenario inside a bigger educational program. In 2005 this work was revised, due to the changing that are characterizing the world, especially in this case the continuous improvement of technology. These factors all together contributed to change the image of library and librarian through these years, so a new evaluation of this process was needed. The aims were to identify both the results in the management of the previous project goals/purposes and also focusing on the improvements to develop in order to maintain this project alive. It was done both to forecast new issues that can be crucial for the project and also to evaluate with use of experts (in this case panel made by librarians and insiders). It is clear the connection of a real case with the theory previously explained, comparing also the old statement made with the previous analysis and the new one.

Steps

---

The participants were 36 coming from 20 different countries (online meeting application) in 4 continents. The steps used in this specific case (accordingly with the theory) were:

• Formation of the panel of professionals
• Development and distribution of the first questionnaire
• Analysis of the first round responses
• Preparation and distribution of the second questionnaire
• Clarification of some points through a third round
• Final analysis and presentation of results

Rounds and Questionnaires

---

Mainly, because this method is time demanding, they did not have the opportunity to test it before effectively starting. This meant some misunderstandings in the beginning, but it went better and better with the passing of rounds. In the first one the questions were about the differences between 2005 (present) and 2015 (future), rating also the things decided in 1999 (past). In the second and third round the focus was on librarians and academic libraries among four categories: 1) Internet as a competitor, 2) Local vs. remote access, 3) Printed vs. electronic media and 4) Staff and user training. Examples of the questionnaires (rounds 1, 2, 3) can be found in the pictures below (for more detail check the references):

Figure 4: Part of the questionnaire round 1

Figure 5: Part of the questionnaire round 2 and 3

Example of First Round Results

---

To have an idea and a better understanding of this technique there are also some results (even if they are just a little part, for more detail check the references). For instance, during the first round the participants were asked to reply to four topics about present and future; using a ranking scale between 5 and 0, where 5 means “really important” and 0 “not important at all”; applied to this data mean and median were calculated. An example can be seen in the graphs below:

Figure 6: Data example of topic1

Figure 7: Data example of topic2

Comparison between present and future

---

For every topic some consequent observations and remarks were made and analyzed in order to improve in the future. Moreover, the comparison between the present and the future was underlined using the median values of the ranking (see below), together with relative observations and remarks:

Figure 8: Present and Future comparison1

Figure 9: Present and Future comparison2

The second and third round were done and analyzed in a similar way based on the four areas cited before, together with some open questions to have personal opinions from the experts and the final presentation of results.

Some main conclusions at the end of this study case were done:

• Library will still exist in the future (2015)
• Their main activities will change slightly compared with the actual state
• Libraries will need to improve in the implementation of IT and have more skilled employees
• Libraries will be used also for social scope and for studying
• Librarian will be a value added to the science found on internet

Other cases can be checked in the works of Giannarakis , of Hsu and Sandford and some real applications are also inside the works of Dalkey and Hanafin , if interested in having more detailed explanations of cases.

Conclusion

In conclusion, the structure of the Delphi technique gives time to the participants to think properly about their opinion and responses, being so able to contribute with full hand at the problem that has to be solved or in the discussion. Moreover, the iterative nature of this approach gives the opportunity to refine, narrow down and test the arguments of the members (being able also to tone down some possible hard statements). Also, there is the opportunity to aggregate different opinions from experts without the need to have a physical meeting; it is different also from other methods because it is flexible and there are incorporated qualitative and quantitative approaches in it.

Some strengths and weaknesses of Delphi are highlighted (related to the work done by Helmer and Bishop ):

Strengths:

• Less problems in group interaction
• Common thoughts periods allowed
• Listen also to minor views
• Both qualitative and quantitative data produced

Weaknesses:

• Risk of false consensus
• Problem of selection biases
• It require a strong/able facilitator

Annotated Bibliography

Brief explanation of some of the references used:

• Geraldi J., Thuesen C., Oehmen J., How to do Projects, Version 0.5

Book used in this course that explains Projects, Program and Portfolio management through a Scandinavian approach. Some definitions are taken from this work and it is used also as a starting point for the Delphi method discussion, in the toolbox about uncertainty. Particularly the chapter about uncertainty is really helpful underlying the importance of managing in a good way uncertainty in order to reach the predetermined goals and teaching how to deal with common mistakes.

• Hanafin S., Review of literature on the Delphi Technique, March 2004

Literature review of Delphi, with a complete overview of the topic having some good examples of advantages, disadvantages, the different types of Delphi (classical, policy and decision), critical assumptions that are the basis for this technique, the general scope and purpose using this method and consensus theory. Moreover, a practical case is presented in the end to highlight the utilization of Delphi in a real context.

• Feret B., Marcinek M., 2005, “The Future of the academic library and the academic librarian. A Delphi study reloaded”, Purdue University.

Case Study used as an example of the real life application of Delphi, took also some data and picture of their statistical results. In this study case it is investigated the role of the academic librarian and libraries in general in the future and how they will survive with the increasing power of internet and IT devices. It is relevant for this article because it is explained in deep detail how to use Delphi inside a project and how to learn from previous mistake and correct it for the decision that will affect the future.

• Helmer O., 1967, Analysis of the future: The Delphi method, The RAND Corporation, Santa Monica, California , Dalkey N.C., Delphi, 1967, The RAND Corporation, Santa Monica, California , Dalkey N.C., 1969, The delphi method: an experimental study of group opinion, prepared for united states air force project RAND

Pioneer contributions in this field (RAND corporation) about Delphi with case studies and data available. Helmer treat the broad concept of future, how to deal with forecasting future events in order to take the right decision for the company in a precise moment (cost and benefits approach). Dalkey express the concept of “Advice community” and the notion of opinion in his work as a pioneer in this sector, explaining for the first time the procedure and the characteristics of the method. Finally, group opinion and judgments are taken into account and explained inside the overall process of decision-making (Air-Force long-term assessment and forecast).

• Linstone, H.A. and Turoff, M. (eds.) (1975 and 2002 reviewed version) The Delphi Method Techniques and Applications.

Complete book on this method, used to underline some characteristics, uses, steps that has to be done, different variants of the standard method, common sector in which it is applicable and the philosophical and methodological foundations behind the concept. Also, some examples of communication techniques are presented and explained, together with a brief historical introduction of Delphi and his evolution throughout the time.

• Bishop C., Pitchforth E, Russell E., van Teijlingen E., Delphi method and nominal group techniques in family planning and reproductive health research

Theory overview on Delphi method explaining strengths, weaknesses, consensus methods and how much it is important to reach it in order to have a successful project and take the right decisions at the right time; finally, how to deal with individuals and their judgments in a group session. The alternative concept of Nominal group technique is also described, together with real life applications in health-related researches.