Applying the Delphi Method in Design Research

Design research is a core practice in design and when applying the design thinking process professionally. However, in some situations, we seek agreement from experts around one question, decision, or strategy while exploring problems before jumping to prototyping solutions. While there are several tools to explore problems from the user perspective, I would like to share with you the Delphi Method, which looks at the situation from the expert perspective. I used this method in my recent paper, Implementing Evidence-Based Design to Improve Adherence in Self-Administered Treatment Technology.

What is the Delphi Method?

The Delphi method is a mixed method aimed at building anonymised consensus between experts around one or more questions. It is an assessment tool that we can use to understand the design problem from the expertise point of view. Sometimes, seeing the issue from expertise and end-user perspectives is essential, especially policy making and strategic decisions. The method is named after the Delphi of the Oracles in Greek mythology who predicted the future.

The Delphi method has three main aspects: anonymity, consensus and iteration. The anonymity of the answers between the experts (panellists) ensures there are biases in the decision taken by each participant. The consensus means that the agreement between expertise in the study area provides inference evidence of the study’s outcome. The iteration nature of the method presents one of its main strengths, as the panellists can adjust their opinions because of the overall answers from the panel.

The Delphi method was developed by RAND Corporation in 1950, with two founders, Olaf Helmer and Norman Dalkey. The main focus was forecasting future technological capabilities, especially in military applications, but it was extended in the 1960s and 1970s to other fields such as economics, marketing, and business (What are Design Research Types and Applications?).

Types of the Delphi Studies

One of the advantages of the methodology is its flexibility, which led to the development of several types based on how and when the method is used. Some of these types include:

• Classic Delphi: This is the most common method, which consists of three or more rounds and can be administered by email.
• Modified Delphi: This method replaces the first postal (or email) questionnaire with face-to-face or focus groups, which makes the technique lose its anonymity but benefits from the discussion between the panellists. Another type similar to this method is called the real-time and technological Delphi, which aims to get immediate answers from the participants.
• Decision Delphi: This method aims to make decisions based on consensus.
• Policy Delphi: This method is used to agree on future policies.
• E-Delphi: This method differs from the classic Delphi only in that it uses a web-based questionnaire as the primary tool to collect answers from the panellists.
• Online Delphi: This method aims to build a discussion between panellists who must achieve consensus.
• Disaggregate Delphi: This method aims to agree on the uses of cluster analysis.

Using the Method in Design Research

Now, the question is when to use the Delphi methodology in the design process. First, the method can be used in the Discover stage to explore the design problem from an expert perspective. For example, we can use it to build consensus between nurses about the usability of a hospital’s sanitisation system and between interpreters about the governmental barriers that face their start-ups.

We can also use the Delphi method during the development stage to build consensus among the panellists on the prototype (Minimum Viable Product Examples and Applications), which is viable in practice. For example, we can explore appropriate sanitisation system design to apply in hospitals by asking nurses, clinicians, and patient advisory groups.

The Delphi Method Procedure

The method consists of several rounds in which the participants (panellists) answer the questions in the first round and agree on the findings in the following rounds. The number of rounds is determined by the extent to which the panellists agree. Let’s review the overview of the general steps of the Delphi Method (What is Kano Model Analysis? And How to Conduct it.).

Define the aim of the study.

The first point is to define the aim of the study, which can be exploring a design challenge, evaluating a prototype, defining a future impact of technology, defining a policy strategy, or defining the factors that affect a specific plan. The study aims to select the appropriate panel members and questions that will be asked during the rounds. To illustrate these steps, I will give an example of my research, which conducted a Delphi study to “identify the factors that affect adherence to self-administered treatment technology.”

Recruiting the Panellists

The panel is a group of experts with experience discussing the topic and its related areas. For example, when asking about self-administered health technology, the panel can include expertise from health tech designers and healthcare professionals. The method’s strength is underpinned by the expertise (of panellists) who contribute to making the consensus decision. Their area of expertise and experience with the topic help to build accurate predictions or decisions related to the design problem. 

The Delphi study has no specific number of members, which can vary from 15 to hundreds in extensive studies. As I highlighted earlier, recruitment depends on access to the right expertise in the area.

To recruit the participants, we need to define the inclusion criteria required of the panellists. For example, in my study, the inclusion criteria included professionals involved in designing treatment technology, healthcare professionals, and professors of psychology and behaviour.

Round One: Qualitative Inputs

The first round is a qualitative round that aims to collect data about the decision that needs to be taken. This round can be interviewed (online or face-to-face), open-ended questionnaires, focus groups or group interviews based on the design study you are conducting. If you would like to maintain the anonymity between the participants, the latter two methods are unsuitable. In my research, I chose to have a written interview where the participants answered the questions through a form because I wanted to maintain the anonymity between the participants, and the written questions were suitable for the participants as they could answer the questions when they got the chance. 

It is helpful to run a pilot study before sharing this round’s questions with all the participants. We run the interview with a smaller number of participants and ask them about the questions and if we need to improve them before rolling them out to all the participants, enhancing the questions that will reflect on the answers.

 Once the data from this round is collected, it is time to analyse the qualitative data. We can use thematic analysis to analyse the answers (especially in interviews and focus groups) to identify the decisions or themes that emerge from the conversation. My previous article provides a full guide to applying thematic analysis.

Round two: Quantitative Inputs

After collecting and analysing the data from the first round, we sent the findings to the participants through a questionnaire to collect their consensus on the conclusions generated. This round can be a simple online questionnaire to ask whether they agree. Based on the data collected and the type of consensus we seek from the study, I can run it as a rating or scale-based questionnaire.

If the answers are based on yes/no, then we can try to identify the agreement based on the overall answers. However, suppose the answer is based on rating questions ( such as ordinal or categorical). In that case, we can use values such as the standard deviation as a guide to evaluate the consensus between the panellists. 

Round Three: Iteration and Confirmation

In the third round, we shared the same questions from the second round with the participants, with one difference being that we anonymised the total answers for each question so they could see what other participants reached in total, review their answers, and revise them. 

After analysing the answers from this round, we review the level of agreement between the panellists and repeat the process in a fourth round until we reach the required consensus. In general, we need to set a saturation point, where we define the point at which we say whether there is a consensus. For example, we can determine the level of consensus as 90% agreement for yes/no questions. We can consider the standard deviation of one point above or below in the rating questions and mean. if the consensus can’t be reached then, we need to go back to review the main interview questions.

After completing the Delphi study, we can use its outcome to predict the design outcome or the application of policy or strategy. While the Delphi method can help us get feedback from the professional perspective, it is crucial to understand that input from the end user of the design or strategy is essential to building a holistic view of the design problem. 

Delphi Method Resources:

You can explore more about the Delphi method through the below reading resources:

• Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. Jessica Kingsley Publishers.
• Keeney, S., McKenna, H. A., & Hasson, F. (2011). The Delphi technique in nursing and health research. John Wiley & Sons.
• Turoff, M., & Linstone, H. A. (2002). The Delphi method-techniques and applications.

The Delphi method is a valuable tool for predicting outcomes and defining strategies based on the consensus of a group of experts with relevant experience to the design challenge we investigate. This makes it handy while conducting design research. The Delphi study is a practical approach to implement in design practice, mainly because of its scalability and flexibility to apply in different contexts. Another advantage of the method is that it combines both qualitative and quantitative methods, which gives it credibility in building evidence-based design.