How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development
File(s)s13063-023-07285-1.pdf (1.5 MB)
Published version
Author(s)
Webbe, James
Allin, Benjamin
Knight, Marian
Modi, Neena
Gale, Chris
Type
Journal Article
Abstract
Background: Core outcomes sets are increasingly used to define research outcomes that are most important for a condition. Different consensus methods are used in the
development of core outcomes sets; the most common is the Delphi process. Delphi methodology is increasingly standardised for core outcomes set development, but
uncertainties remain. We aimed to empirically test how the use of different summary statistics and consensus criteria impact Delphi process results.
Methods: Results from two unrelated child health Delphi processes were analysed. Outcomes were ranked by mean, median, or rate of exceedance, and then pairwise
comparisons were undertaken to analyse whether the rankings were similar. The correlation coefficient for each comparison was calculated, and Bland-Altman plots
produced. Youden’s index was used to assess how well the outcomes ranked highest by each summary statistic matched the final core outcomes sets. Consensus criteria identified in a review of published Delphi processes were applied to the results of the two child-health Delphi processes. The size of the consensus sets
produced by different criteria was compared, and Youden’s index was used to assess how well the outcomes that met different criteria matched the final core outcomes sets.
Results: Pairwise comparisons of different summary statistics produced similar correlation coefficients. Bland-Altman plots showed that comparisons involving ranked medians had wider variation in the ranking. No difference in Youden’s index for the summary statistics was found.
Different consensus criteria produced widely different sets of consensus outcomes (Range: 5-44 included outcomes). They also showed differing abilities to identify core outcomes (Youden’s index Range: 0.32-0.92). The choice of consensus criteria had a large impact on Delphi results.
Discussion: The use of different summary statistics is unlikely to affect how outcomes are ranked during a Delphi process: mean, median and rates of exceedance produce similar results. Different consensus criteria have a large impact on resultant consensus outcomes and potentially on subsequent core outcomes sets: our results confirm the importance of adhering to pre-specified consensus criteria.
development of core outcomes sets; the most common is the Delphi process. Delphi methodology is increasingly standardised for core outcomes set development, but
uncertainties remain. We aimed to empirically test how the use of different summary statistics and consensus criteria impact Delphi process results.
Methods: Results from two unrelated child health Delphi processes were analysed. Outcomes were ranked by mean, median, or rate of exceedance, and then pairwise
comparisons were undertaken to analyse whether the rankings were similar. The correlation coefficient for each comparison was calculated, and Bland-Altman plots
produced. Youden’s index was used to assess how well the outcomes ranked highest by each summary statistic matched the final core outcomes sets. Consensus criteria identified in a review of published Delphi processes were applied to the results of the two child-health Delphi processes. The size of the consensus sets
produced by different criteria was compared, and Youden’s index was used to assess how well the outcomes that met different criteria matched the final core outcomes sets.
Results: Pairwise comparisons of different summary statistics produced similar correlation coefficients. Bland-Altman plots showed that comparisons involving ranked medians had wider variation in the ranking. No difference in Youden’s index for the summary statistics was found.
Different consensus criteria produced widely different sets of consensus outcomes (Range: 5-44 included outcomes). They also showed differing abilities to identify core outcomes (Youden’s index Range: 0.32-0.92). The choice of consensus criteria had a large impact on Delphi results.
Discussion: The use of different summary statistics is unlikely to affect how outcomes are ranked during a Delphi process: mean, median and rates of exceedance produce similar results. Different consensus criteria have a large impact on resultant consensus outcomes and potentially on subsequent core outcomes sets: our results confirm the importance of adhering to pre-specified consensus criteria.
Date Issued
2023-05-22
Date Acceptance
2023-03-29
Citation
Trials, 2023, 24
ISSN
1745-6215
Publisher
BMC
Journal / Book Title
Trials
Volume
24
Copyright Statement
© The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativeco
mmons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativeco
mmons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
License URL
Identifier
https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-023-07285-1
Publication Status
Published
Article Number
345
Date Publish Online
2023-05-22