Big little lies: a compendium and simulation of p-hacking strategies

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p-hacking. Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p-hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of 12 p-hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p-hacking at the level of strategies can provide a better understanding of the process of p-hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p-hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p-hacking in practice.

Automated summary

In various research fields, the credibility of scientific results has been compromised due to the prevalent use of questionable research practices. Among them, p-hacking is one of the most prominent practices, which refers to strategies used to manipulate non-significant hypothesis testing results to appear significant. However, there is currently a lack of comprehensive understanding of the different strategies involved in p-hacking, and the heterogeneity of these strategies is often overlooked in meta-scientific research. This study provides a compilation of 12 p-hacking strategies, identifies factors influencing their severity, and demonstrates their impact on false-positive rates through simulation studies. The findings suggest that investigating p-hacking at the strategy level enhances understanding and facilitates the development of effective countermeasures.