P>0.05 Is Good: The NORD-h Protocol for Several Hypothesis Analysis Based on Known Risks, Costs, and Benefits
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Statistical testing in medicine is a controversial and commonly misunderstood topic. Despite decades of efforts by renowned associations and international experts, fallacies such as nullism, the magnitude fallacy, and dichotomania are still widespread within clinical and epidemiological research. This can lead to serious health errors (e.g., misidentification of adverse reactions). In this regard, our work sheds light on another common interpretive and cognitive error: the fallacy of high significance, understood as the mistaken tendency to prioritize findings that lead to low p-values. Indeed, there are target hypotheses (e.g., a hazard ratio of 0.10) for which a high p-value is an optimal and desirable outcome. Accordingly, we propose a novel method that goes beyond mere null hypothesis testing by assessing the statistical surprise of the experimental result compared to the prediction of several target assumptions. Additionally, we formalize the concept of interval hypotheses based on prior information about costs, risks, and benefits for the stakeholders (NORD-h protocol). The incompatibility graph (or surprisal graph) is adopted in this context. Finally, we discuss the epistemic necessity for a descriptive, (quasi) unconditional approach in statistics, which is essential to draw valid conclusions about the consistency of data with all relevant possibilities, including study limitations. Given these considerations, this new protocol has the potential to significantly impact the production of reliable evidence in public health.
Abstract
Statistical testing in medicine is a controversial and commonly misunderstood topic. Despite decades of efforts by renowned associations and international experts, fallacies such as nullism, the magnitude fallacy, and dichotomania are still widespread within clinical and epidemiological research. This can lead to serious health errors (e.g., misidentification of adverse reactions). In this regard, our work sheds light on another common interpretive and cognitive error: the fallacy of high significance, understood as the mistaken tendency to prioritize findings that lead to low p-values. Indeed, there are target hypotheses (e.g., a hazard ratio of 0.10) for which a high p-value is an optimal and desirable outcome. Accordingly, we propose a novel method that goes beyond mere null hypothesis testing by assessing the statistical surprise of the experimental result compared to the prediction of several target assumptions. Additionally, we formalize the concept of interval hypotheses based on prior information about costs, risks, and benefits for the stakeholders (NORD-h protocol). The incompatibility graph (or surprisal graph) is adopted in this context. Finally, we discuss the epistemic necessity for a descriptive, (quasi) unconditional approach in statistics, which is essential to draw valid conclusions about the consistency of data with all relevant possibilities, including study limitations. Given these considerations, this new protocol has the potential to significantly impact the production of reliable evidence in public health.
참고문헌 (34)
- Greenland S, "Statistical tests, P values, confidence intervals, and power : a guide to misinterpretations", Eur J Epidemiol, 31(4), pp.337~350 (2016)[DOI]
- Rovetta A, "Statistical significance misuse in public health research: an investigation of the current situation and possible solutions", J Health Policy Outcomes Res (2014)[DOI]
- Wasserstein RL, "The ASA statement on p-values : context, process, and purpose", Am Stat, 70(2), pp.129~133 (2016)[DOI]
- Wasserstein RL, "Moving to a world beyond"p <0. 05"", Am Stat, 73(sup1), pp.1~19 (2019)[DOI]
- Amrhein V, "Scientists rise up against statistical significance", Nature, 567(7748), pp.305~307 (2019)[DOI]
- McShane BB, "Abandon statistical significance", Am Stat, 73(sup1), pp.235~245 (2019)[DOI]
- Gelman A, "The failure of null hypothesis significance testing when studying incremental changes, and what to do about it", Pers Soc Psychol Bull, 44(1), pp.16~23 (2018)[DOI]
- McShane BB, ""Statistical significance"and statistical reporting : moving beyond binary", J Mark, 88(3), pp.1~19 (2023)[DOI]
- Greenland S, "Divergence versus decision p-values : a distinction worth making in theory and keeping in practice : or, how divergence p-values measure evidence even when decision p-values do not", Scand J Stat, 50(1), pp.54~88 (2023)[DOI]
- Greenland S, "Valid p-values behave exactly as they should : some misleading criticisms of p-values and their resolution with s-values", Am Stat, 73(sup1), pp.106~114 (2019)[DOI]
- Rafi Z, "Semantic and cognitive tools to aid statistical science : replace confidence and significance by compatibility and surprise", BMC Med Res Methodol, 20(1), pp.244~ (2020)[DOI]
- Mansournia MA, "P-value, compatibility, and s-value", Glob Epidemiol, 4, pp.100085~ (2022)[DOI]
- Rovetta A, "S-values and surprisal intervals to replace p -values and confidence intervals", Revstat Stat J (2024)
- "American Statistical Association releases statement on statistical significance and p -values" (2016)[URL]
- Rovetta A, "How to read P-values and confidence intervals in public health studies", Saudi J Anaesth, 18(3), pp.459~460 (2024)[DOI]
- Amrhein V, "Inferential statistics as descriptive statistics : there is no replication crisis if we don’t expect replication", Am Stat, 73(sup1), pp.262~270 (2019)[DOI]
- Amrhein V, "Discuss practical importance of results based on interval estimates and p-value functions, not only on point estimates and null p-values", J Inf Technol, 37(3), pp.316~320 (2022)[DOI]
- Cole SR, "Surprise!", Am J Epidemiol, 190(2), pp.191~193 (2021)[DOI]
- Greenland S, "To curb research misreporting, replace significance and confidence by compatibility : a Preventive Medicine Golden Jubilee article", Prev Med, 164, pp.107127~ (2022)[DOI]
- Ting C, "Forcing a deterministic frame on probabilistic phenomena : a communication blind spot in media coverage of the"replication crisis"", Sci Commu, 46(5), pp.672~684 (2024)[DOI]
- Rubin M, ""Repeated sampling from the same population?"A critique of Neyman and Pearson’s responses to Fisher", Eur J Philo Sci, 10, pp.42~ (2020)[DOI]
- Rovetta A, "Multiple confidence intervals and surprisal intervals to avoid significance fallacy", Cureus, 16(1), pp.e51964~ (2024)[DOI]
- Rovetta A, "For a proper use of frequentist inferential statistics in public health", Glob Epidemiol, 8, pp.100151~ (2024)[DOI]
- Mansournia MA, "A checklist for statistical assessment of medical papers(the CHAMP statement) : explanation and elaboration", Br J Sports Med, 55(18), pp.1009~1017 (2021)[DOI]
- Greenland S, "Multiple comparisons controversies are about context and costs, not frequentism versus Bayesianism", Eur J Epidemiol, 34(9), pp.801~808 (2019)[DOI]
- National Academy of Sciences (US), "Responsible science: ensuring the integrity of the research process: volume I" (1992), National Academies Press (US)
- Good IJ, "Rational decisions", J R Stat Soc Ser B Stat Methodol, 14(1), pp.107~114 (1952)[DOI]
- Bann D, "Dialling back ‘impact’ claims: researchers should not be compelled to make policy claims based on single studies", Int J Epidemiol, 53(1), pp.dyad181~ (2024)[DOI]
- Greenland S, "Analysis goals, error-cost sensitivity, and analysis hacking : essential considerations in hypothesis testing and multiple comparisons", Paediatr Perinat Epidemiol, 35(1), pp.8~23 (2021)[DOI]
- Lash TL, "Good practices for quantitative bias analysis", Int J Epidemiol, 43(6), pp.1969~1985 (2014)[DOI]
- Mansournia MA, "Recommendations for accurate reporting in medical research statistics", Lancet, 403(10427), pp.611~612 (2024)[DOI]
- Poole C, "Beyond the confidence interval", Am J Public Health, 77(2), pp.195~199 (1987)[DOI]
- "Statistics is for statisticians" (2020)[URL]