Revision as of 15:06, 27 March 2022

Witten by Kunyi Yang，February 2022

Abstract

The Wideband Delphi estimation method is a technique for estimating project workload based on participant consensus, mainly used in Management of Software development Projects. It is based on the Delphi method, famously developed by the RAND Corporation in the 1950s-1960s as a forecasting tool. The classic Delphi method is a method in which companies form a specialized forecasting organization, which includes several experts and corporate forecasting organizers, and follow a prescribed procedure to solicit experts' opinions or judgments about future markets back to back and then make forecasts. In the 1970s, Barry Boehm and John A. Farquhar proposed a wideband variant of the Delphi method. The term "wideband" is used because the wideband Delphi technique involves more interaction and more communication between participants than the Delphi method. It has since been adapted in many industries to estimate a wide range of tasks, from statistical data collection results to sales and marketing forecasts.^{[1] [2]} The main advantages of Wideband Delphi are simplicity of implementation, low cost, and no need for historical data. This paper will describe the main steps of Wideband Delphi and analyze its application in different areas, such as software development projects, presenting the role and advantages it plays in project management. This paper will also give practical examples with diagrams to help understand the process of its use.^[3] In addition, the paper will discuss what improvements can be made to Wideband Delphi within the existing framework and how it can be integrated with SCRUM(a framework for developing, delivering, and sustaining products in a complex environment^[4].） in Planning Poker.

Big idea

Introduction

Delphi method is a feedback anonymous correspondence method in essence. It originated from the prediction project conducted by Rand Corporation at the request of the US military during the Cold War. It assumes that the judgment of a group is more effective than that of an individual (usually an expert). The general process is to collect, summarize, and make statistics on the problems to be predicted after obtaining the opinions of experts, and then feedback anonymously to the experts. Opinions are solicited again, and then centralized again, and then feedback again until unanimous opinions are reached. Wideband Delphi was first proposed by Barry Boehm and John A. Farquhar in the 1970s and promoted by Boehm in his book Software Engineering Economics (1981)^[5][6].Unlike the traditional Delphi method, it encourages discussion among participants. When Boehm(1981) applied it to cost estimation, he found that facilitating discussion and wider channels of communication could produce more accurate results and contribute to experience sharing and team building. Compared with the Rand Delphi method of narrowband communication, this improved method is called the wideband Delphi method. Recent research in software project evaluation shows that evaluations that benefit from group discussions tend to be more accurate. ^[3]

The original Wideband Delphi steps were[6]:
1.The Coordinator provides each expert with a specification and an estimate sheet.
2.The coordinator convenes group meetings where experts discuss estimation issues with the coordinator and with each other.
3.Experts fill out forms anonymously.
4.The coordinator prepares and distributes summaries of estimates
5.The coordinator convenes group meetings with a particular focus on getting the experts to discuss points where their estimates vary widely
6.The expert filled out the form anonymously again and repeated steps 4 through 6 as needed.

It was later improved by Neil Potter and Mary Sakry of The Process Group to better embed The project management framework and came out the modern version.^[7]

Steps of Wideband Delphi

Figure 1: Steps of Wideband Delphi^[8]

As shown in the figure, Wideband Delphi requires the following process to proceed. The preparation phase is one-way, including the preparation of topics, materials, and ideas by the meeting stakeholders, followed by the conference-review section, which determines the final workload through multiple iterations.

Planning&Selection

- Selecting qualified teams is an important part of producing accurate estimates.

- Team members must be willing to assess each task honestly and should be willing to work with the rest of the team.

- Have an understanding of the organization's needs and past engineering projects. Knowledge of engineering projects in order to make educated estimates.

- The team should include representatives from all areas of the developer team: managers, developers, designers, architects, analysts, technical writers, contributors, etc.

- Facilitators，or called moderator，should be familiar with the Delphi process, but should not have anything to do with the Delphi results.

Kickoff Meeting

Provide vision, scope, and other documentation to members.

- The statement of objectives for the estimation meeting should be agreed upon by the project Manager and the facilitator and distributed to the team prior to the meeting.

- It should be no more than a few sentences describing the scope of work to be estimated. For example: generate estimates for phase 1 programming and testing of project A.

- The conference includes these activities:

1. The moderator explains the wideband Delphi method to any new evaluator.

2. If any team member has not read the vision and scope and supporting documentation, the facilitator will review it with the team.

3. The host reviews it with the team.

4. The reviewer reviews the goals of the evaluation meeting with the team and checks if team member has the knowledge to contribute.

5. The team discusses products under development and brainstorm hypotheses.

6. Generate a task list of 10-20 main tasks. These tasks represents the highest level of the work breakdown structure.

7. Agreed units of estimate (days, weeks, pages).

Individual Preparation

After initiating the meeting, the facilitator writes down the assumptions and tasks generated by the team and distributes them.The team member independently generates a set of preparation results, including:

Figure 2: Example Wideband Delphi estimation form ^[8]

1. Estimates for each task

2. Any additional tasks that should be included in WBS (Work Breakdown Structure)^[9]

3. Tasks the team missed in the launch meeting.

4. Any assumptions made by team members when creating the estimate.

- Any work related to project expenses (status meetings, reports, vacations, etc.) should not be taken into account. Should be added to the "Project Overhead Task" section.

- Potential delays should not be considered (e.g., some tasks can't start until after a certain date). Should be added to the "Calendar Wait time" section.

- Every estimate should be based on effort, not calendar time.

Estimation Session

- The moderator collects all estimates. Draw the estimated total on a line on the whiteboard and tabulate it.

- The estimator（one neither the moderator nor any team member） reads out clarifications and revisions to the list of tasks written on the estimator. Propose new or changed tasks, discovered hypotheses, or questions. No specific estimated time is discussed.

- The team resolved problems or disagreements. Since specific estimated times are not discussed, these disagreements are usually about the task itself and are often resolved by adding assumptions.

- Estimators modify their personal estimates by filling in the "Delta "column on the form.

Assemble Tasks

- The project manager works with the facilitator to gather all the results from the individual preparation and estimation meeting.

Figure 3: Typical Delphi task assembly form ^[8]

- The project manager removes excess content and resolves remaining estimate differences to produce a final to-do list, as well as estimates of effort - assumptions are summarized and added to the list. Update assumptions in Visio files and other files.

- The project manager should create spreadsheets listing the final estimates that everyone comes up with. A spreadsheet should indicate best and worst cases,

- Any tasks that differ significantly should be flagged for further discussion.

- The final task list should be in the same format as the individual preparation results.

Review Results

- Once the results are ready, the project manager holds a final meeting to review the estimates with the team. The objective of the meeting was to determine whether the results of the meeting were sufficient for further planning. The team should determine if the estimate makes sense and if the range is acceptable. They should also check the final to-do list to verify that it is complete.

- There may be one area that needs improvement. For example, a task may need to be broken down into sub-tasks. In this case, the team might agree to hold another estimation session to break down the tasks into their respective sub-tasks and estimate each sub-task. This is also a good way to handle any task where there is a big difference between the best and worst scenarios.

Application

Wideband Delphi has been used successfully in agile project management (especially in software development projects). Many of the principles of agile estimation are, in fact, derived from the expert-based group-consensus methods such as Planning Poker.

Planning Poker

Figure 4: Planning Poker ^[10]

Planning Poker is a consensus-based estimation technique used to estimate the amount of work or relative size described by users in Scrum.^[11] Planning Poker combines three estimation techniques: Wideband Delphi technique, analog estimation, and WBS estimation. In planning poker estimation technique, the estimation of user stories is obtained by playing planning poker. The entire Scrum team is involved and it produces quick but reliable estimates. Like Wideband Delphi, it iterates to improve the accuracy of estimates，uses a non-anonymous meeting approach, also to estimate working hours.

Plan poker is played with a pack of cards. When using the Fibonacci sequence, the card has the numbers - 1, 2, 3, 5, 8, 13, 21, 34, etc. These numbers represent "story points." Each estimator has a deck of cards. When one team member holds up the card, the number on the card should be large enough for all team members to see. One of the team members was chosen as the host. The moderator reads the description of the user story for which it is being estimated. If the assessors have any questions, the product owner will answer them.^[12]

Each estimator privately selects a card that represents his or her estimate. Cards are not displayed until all estimators have made their choices. At that point, all cards are flipped and held up at the same time so that all team members can see each estimate. Estimates are likely to change in the first round. The overestimators and underestimators explained the reasons for their estimates. It should be noted that all discussions are for understanding only and not personal. The moderator must ensure the same. The team can discuss the story and their estimates for a few more minutes. Facilitators can take notes during discussions, which will help develop specific stories. After the discussion, each estimator reestimated by selecting a card again. The cards are kept secret again until everyone has estimated them and then handed in at the same time. This process is repeated until the estimate converges to one that can be used for the story. The estimated number of rounds may vary from user story to user story.

Planning poker now becomes a popular method of for relative estimation of complex empirical work, such as engineering or software development.^[13] It encourages team members to think independently by asking them to reveal their score simultaneously with their peers. This avoids the cognitive bias known as anchoring^[14] and exposes situations where team members hold different information or perspectives on the work. Influential Lean Management theory categorises estimation as waste because it does not directly deliver value. Agile practitioners therefore try to reduce the time spent producing estimates, which have a poor history of inaccuracy anyway. Typically, conversations during the game will be focused on the perspectives that differ the most. This means that details that will not effect the estimate are less likely to be discussed and the conversation is much more efficient. Another reason this poker game process makes planning cheaper is that it takes relative complexity as the input, rather than a complex design document that may not be accurate. There is simply too much detail to be discovered down the road, and without an implementation in-progress much of that detail is not discoverable. Relative complexity however, is a tractable question. Only a broad outline of the solution is needed to determine this, and that outline can be produced in a fraction of the time, dramatically reducing the effort needed to produce an estimate.

Obviously, Wideband Delphi can be used in other areas as well，but there are few suitable cases to be found. An interesting case is that Indonesian scholars discussed the key requirements to be considered in the national cyber defense framework to prevent foreign surveillance after PRISM, and proposed effective mitigation control measures to safeguard and protect the national interests of the country. ^[15]There Wideband Delphi method was used to determine the analysis framework in conjunction with governance and legal remedies. They identified 25 mitigation control measures to address policy makers' priorities, which were divided into five deep defence elements. The difficulty in applying this method lies with the experts.In general, Wideband Delphi can be used in important situations where a large number of experts are involved and a full exchange of views is required. Of course, the characteristic of expert participation will also become the limitation of its application. First, this approach is not appropriate when one (especially the moderator) wants to build relationships and continuous communication among the participants, or when one wants to make decisions directly. Secondly, if the internal consistency and reliability of expert judgment are poor, the actual meaning cannot be inferred from the results. Lack of reliability can also result from inconsistent understanding of concepts by experts.

Advantage and disadvantage

The merits and demerits of this method can be evaluated from the following aspects. ^{[8] [3]} Then we can generally understand why Wideband Delphi is currently more likely to be used in areas where the process paradigm is more fixed but the content is more complex, such as software development.

Involvement of Experts

- Okay, but it's hard. Like other group estimation methods, Wideband Delphi requires extensive participation from multiple experts, who should represent a variety of expertise and perspectives on the software being developed. The experts remain anonymous throughout the process, so they are free to express their opinions and criticisms, carefully coordinated by the moderator. It asks experts to give reasons for their estimates and then debate those reasons; The validity of the results and the expertise of the participants depends largely on who attended the workshop. If the topic is too specialized, the barrier to expert participation can be high.

Data Requirement

- Okay, but not easy. The access method is simple but time-consuming. Wideband Delphi does not require any historical project data. But data can be difficult to quantify and time-consuming to obtain when large numbers of samples are needed.

Robustness

- Not easy. Expert judgment can be biased by irrelevant and misleading information. For example, a client's request for a project budget may cause an estimator (willing to meet the client) to favor that budget. But Wideband Delphi is designed to gather collective assessments from experts, and structured group consensus procedures help reduce bias without the risk that some voice or influential individual will distort the results.

Flexibility

- Good. Input from Wideband Delphi does not require any specific information and does not require any adjustments when applied to different situations and environments.

Complexity

- Good. Wideband Delphi is intuitive and easy to use.

Support level

- Good. Wideband Delphi is well documented. It doesn't require any sophisticated tool support -- beyond standard office software like word processors and computing spreadsheets.

Reusability

- Not good. Wideband Delphi does not provide an estimation model that can be reused in similar projects. Each time an estimate is required, the estimate procedure must be completely reapplied.

Ability to predict

- Okay, but hard to meet expectations. Human experts can consider (but implicitly) anomalous and unique project characteristics and their interactions. For example, experts can handle differences between past project experience and the new technology, architecture, or application involved in a new project. However, the human experts involved in the estimation process can vary widely in their individual estimates. Although the group estimation process should compensate for the influence of individual experts, group estimation can still be unstable. At the same time, estimates based on human judgment can be biased by a number of factors, such as the personal expertise, experience and preferences of human experts, as well as the influence of other experts involved in the estimation process and irrelevant and misleading information. For example, human estimators tend to underestimate tasks that are considered easy and overestimate tasks that are considered difficult.

Dealing with uncertainty

- Normal. Wideband Delphi can handle uncertain information (that is, the expertise of human experts). It can also provide information about the uncertainty of estimates, for example, experts provide their estimates in the form of three values: minimum, maximum, and most likely. However, this method does not provide any system technology to deal with uncertainty; This requires the use of additional analytical techniques, such as operations on simple probability distributions.

Range Estimation

- Not good and relatively small. The applicability of Wideband Delphi to estimating different types of activities is limited by the expertise of experts in the relevant domain. In practice, estimation is limited to project activities in which the participating human estimators have experiences.

Usability and Experience Acquisition

- Good. Wideband Delphi is suitable for any stage of software development. In addition, the general rule here applies to more accurate estimates performed later in the development lifecycle estimates because more is known about the project; Although Wideband Delphi has not been extensively studied, the group estimation method has been validated in numerous empirical studies in research and industrial Settings.

Prospect

Annotated bibliography

1. Trendowicz A., Jeffery R. (2014) Wideband Delphi. In: Software Project Effort Estimation. Springer, Cham. https://doi.org/10.1007/978-3-319-03629-8_12

Annotation: It provides a comprehensive introduction to the basic principles and usage of Wideband Delphi. Can be regarded as one of the best summaries of Wideband Delphi in existence.

2. B. W. Boehm, "Software Engineering Economics," in IEEE Transactions on Software Engineering, vol. SE-10, no. 1, pp. 4-21, Jan. 1984, doi: 10.1109/TSE.1984.5010193.https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5010193

Annotation: The authors investigate the field of software cost estimation, including the major estimation techniques available, the latest techniques in algorithmic cost models, and prominent research issues in software cost estimation. Including Wideband Delphi, as the creator of this method, he elaborated the basic idea of this method, for the needs and standard methods.

3. Valerdi R. 10.4. 2 convergence of expert opinion via the wideband delphi method: An application in cost estimation models[C]//Incose International Symposium. 2011, 21(1): 1246-1259.

Annotation：This paper not only reviews the origin and concept of Delphi method and Wideband Delphi method, but also uses cost estimation model to verify the effect of Wideband Delphi method in Expert Support.

References

1. ↑ Basili VR, Caldiera G, Rombach HD (1994b) The goal question metric approach.
2. ↑ In: Marciniak JJ (ed) Encyclopedia of software engineering, vol 1, 2nd edn. Wiley, New York, pp 528–532
3. ↑ ^{3.0 3.1 3.2} Valerdi R. 10.4. 2 convergence of expert opinion via the wideband delphi method: An application in cost estimation models[C]//Incose International Symposium. 2011, 21(1): 1246-1259.
4. ↑ Schwaber, Ken; Sutherland, Jeff (November 2017), The Scrum Guide: The Definitive Guide to Scrum: The Rules of the Game, retrieved May 13, 2020
5. ↑ Farquhar JA (1970) A preliminary inquiry into the software estimation process. The Rand Corporation, Santa Monica, CA, Technical Report RM-6271-PR
6. ↑ ^{6.0 6.1} Boehm BW (1981) Software engineering economics. Prentice-Hall, Englewood Cliffs, NJ
7. ↑ A. Stellman and J. Greene (2005), Applied software Project Management. 1st Edition. O’Reilly Media Inc.
8. ↑ ^{8.0 8.1 8.2 8.3} Trendowicz A., Jeffery R. (2014) Wideband Delphi. In: Software Project Effort Estimation. Springer, Cham. https://doi.org/10.1007/978-3-319-03629-8_12
9. ↑ Stefano Almanzi. Work Breakdown Structure.http://wiki.doing-projects.org/index.php/Work_Breakdown_Structure_(WBS)
10. ↑ Planning Poker. Mountaingoatsoftware.https://www.mountaingoatsoftware.com/agile/planning-poker/
11. ↑ Cohn, Mike (November 2005). "Agile Estimating and Planning". Mountain Goat Software. https://athena.ecs.csus.edu/~buckley/CSc131_files/Planning%20Poker.pdf
12. ↑ Kjetil Moløkken-Østvold, Nils Christian Haugen, Hans Christian Benestad, Using planning poker for combining expert estimates in software projects,Journal of Systems and Software,Volume 81, Issue 12,2008,Pages 2106-2117,ISSN 0164-1212,https://doi.org/10.1016/j.jss.2008.03.058.
13. ↑ Agile Stationery. Why is Planning Poker a popular estimation technique in agile and scrum teams? https://agilestationery.com/pages/planning-poker
14. ↑ Anchoring (cognitive bias). https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)
15. ↑ Y. Nugraha, I. Brown and A. S. Sastrosubroto, "An Adaptive Wideband Delphi Method to Study State Cyber-Defence Requirements," in IEEE Transactions on Emerging Topics in Computing, vol. 4, no. 1, pp. 47-59, Jan.-March 2016, doi: 10.1109/TETC.2015.2389661.