I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it.
At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.
I think you may be talking about "Mindless statistics" by Gigerenzer. He has some surveys about p-values and how radically wrong they are usually interpreted.
>At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.
P-values work great when they’re super low, experiments run at a human-scale frequency, and hypotheses are extremely precise in their predictions, e.g. some physics.
If you run an experiment a day and get p < 10^-9, your priors, your multiple hypothesis correction, even your interpretation of p-values approximately don’t matter. Running social sciences experiments with p < 0.05 threshold is where things get weird.
probably this article:
Hoekstra, R., Morey, R.D., Rouder, J.N. et al. Robust misinterpretation of confidence intervals. Psychon Bull Rev 21, 1157–1164 (2014). https://doi.org/10.3758/s13423-013-0572-3
another good article on misinterpretation of p-values and confidence intervals is:
Greenland, S., Senn, S.J., Rothman, K.J. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31, 337–350 (2016). https://doi.org/10.1007/s10654-016-0149-3
While I agree on the data point, it would kill so much research. It is bad that a lot of research validation basically comes down to "trust me guys", but with data being both very valuable and often times highly sensitive, it can be really difficult to just publish the data along with the research.
A decent compromise would be to at least require meta-data to sufficiently exclude some flaws. A different approach could be to have researchers document and publish the process of th research, similar to a git-repo with the main branch being completely off limits to history-rewriting.
At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.