ISHLT Delphi Methodology Guidance for Authors
Delphi methodology is a structured consensus-building process used to develop consensus among a panel of experts on topics where evidence is limited or evolving. The process starts with a predefined list of statements, recommendations, or questions that have already been developed through literature review, existing guidelines, or expert committee discussions, followed by iterative surveys or questionnaires interspersed with controlled feedback. Authors of ISHLT Consensus Documents who choose to use a Delphi process to develop consensus statements should use the modified Delphi methodology described below.
Panel Composition
The Writing Committee serves as the Delphi panel and participates in all rounds of voting. The panel members are therefore selected in accordance with the criteria put forward in the ISHLT Document Development Policies for the composition and selection of a writing committee. There is no prescribed Writing Committee/panel size. The number of panelists should be appropriate for the scope of the document and include individuals with the necessary expertise.
Development of Delphi Statements
Statements may be drafted by the full Writing Committee or a smaller subset of the Writing Committee (i.e., Section Leads). Project Leads should consider the potential for confirmation bias and make the determination on who should participate in this stage accordingly.
Statements are drafted based on a comprehensive review of the available scientific literature, existing guidelines, and expert knowledge. The evidence is organized into key topic areas, and draft statements are written to reflect the current evidence and address important clinical questions or areas where guidance is needed.
Each statement should be:
- Clear and concise.
- Focused on a single concept or recommendation.
- Supported by the available evidence whenever possible.
- Written so it can be easily evaluated by expert reviewers.
The drafted statements are to be reviewed and refined to ensure they are accurate, consistent, and free of duplication before they are circulated to the expert panel for review and voting as part of the modified Delphi process.
Creation of Surveys
Delphi surveys may be developed by the Project Leads using a survey tool of their choosing, or with the support of ISHLT Staff using an ISHLT-approved online survey platform (e.g., SurveyMonkey). Project Leads will provide staff with the list of statements to be voted on and will receive a link to the online survey. It is the responsibility of Project Leads to review the survey and let staff know if there are any changes to be made. Once finalized, it is the responsibility of the Project Leads to distribute the survey to the other members of the Writing Committee.
Delphi Voting
The Delphi voting process is conducted anonymously to promote independent judgment and minimize the influence of individual panel members. Each panelist reviews and rates each draft statement using a 5-point Likert scale, where 1 = strongly disagree and 5 = strongly agree. An additional option will be included for panelists to permit panelists to abstain from voting on specific statements for which they do not have the necessary expertise. Panelists are also encouraged to provide comments or suggested revisions for statements that require clarification or do not reflect their interpretation of the evidence.
The Delphi process will generally consist of up to three rounds of anonymous voting. Statements that achieve the predetermined consensus threshold during any round will be accepted. Statements that do not reach consensus will be revised based on panel feedback and redistributed for additional voting. After the third round, the Writing Committee will determine whether statements that fail to achieve consensus should be removed, substantially revised for future consideration, or included with an explanation of differing viewpoints, as appropriate. The Delphi process may conclude before three rounds if all statements achieve the predefined consensus threshold.
Although voting is anonymous, panelists must identify themselves when completing surveys developed by ISHLT Staff for the purposes of participant tracking. Responses are identifiable only to staff for tracking and are anonymized before review by the Project Leads and Writing Committee.
Defining Quorum and Consenus
Quorum is defined as at least 80% participation for each statement (i.e. 80% of panelists rating a statement using a 5-point Likert scale, or less than 20% of panelists abstaining rating a statement). Statements that do not achieve a Quorum during round one or two of voting may be discarded, or may be revised, as appropriate, and redistributed for additional voting. Statements that do not achieve 80% participation during the third and final round of voting should be discarded.
Consensus is defined as at least 75% of voting panelists rating a statement as "Agree" (4) or "Strongly Agree" (5) on the 5- point Likert scale. Panelists that abstain from voting are not be factored into the data analysis for that particular statement. Statements meeting this threshold are considered to have achieved consensus and are accepted for inclusion in the final document. Statements that do not achieve 75% agreement during round one or two of voting may be revised, as appropriate, and redistributed for additional voting. Statements that do not achieve 75% agreement during the third and final round of voting should be listed in Appendix 1 only, and not included in the body of the final Consensus Document.
Reporting of Methodology
The Methods section of the final consensus document should include a brief description of the Delphi process, incorporating the following elements:
- Methodology: State that a Delphi process was used to develop expert consensus.
- Expert Panel: Briefly describe who participated (e.g., multidisciplinary experts selected for their subject matter expertise).
- Statement Development: Explain that draft statements were developed by the Writing Committee based on a review of the available evidence.
- Voting Process: Describe that panelists anonymously rated each statement using a predefined Likert scale and provided comments for revisions.
- Definition of Consensus: State the predefined threshold for consensus.
- Statement Revision: Explain that statements not reaching consensus were revised based on panel feedback and redistributed for additional voting.
- Voting Rounds: Indicate the number of voting rounds conducted.
The final Consensus Document should report the methodology in accordance with the ACCORD (ACcurate COnsensus Reporting Document) reporting guideline.
Reporting of Consensus
Consensus Statements should be incorporated into the document as outlined in the ISHLT Consensus Document Template, which includes reporting of the percentage agreement achieved for each statement.
Appendix 1 of the final Consensus Document should be a summary of all consensus statements, developed and formatted in accordance with the ISHLT Consensus Document Template. Writers should create a table to collect all consensus statements made throughout the document. The table should list each section and subsection, a short (half a sentence) summary of each statement, and list the percentage agreement achieved for each statement.