True. But on the other hand, publishing ridiculous claims and incorrect results is a necessary part of science.

When we publish only results we know to be correct, because they agree with mainstream beliefs, we introduce a bias into the scientific process. In reality, if you publish 20 experiments with p=0.05 [1], 1 of them should be incorrect. If less than 1 in 20 of your papers isn't wrong (assuming p=0.05 is the gold standard), you are not doing science.

You can see a perfect illustration of this when people tried to reproduce Millikan's oil drop experiment. I'll quote Feynman: Millikan measured the charge on an electron...got an answer which we now know not to be quite right...It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of - this history - because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...

This is why I'm an advocate of accepting/rejecting scientific papers based solely on methodology, with referees being given no information about the conclusions and with authors being forbidden from post-hoc tweaks. You do your experiment, and if you disagree with Millikan/conclude that ESP exists, so be it. Everyone is allowed to be wrong 5% of the time.

[1] I'm wearing my frequentist hat for the purposes of this post. Even if you are a Bayesian, you should still publish, however.

And I suspect the ultimate reason it's not done this way... is that scientists in certain fields would publish a lot fewer papers, not slightly fewer but a lot fewer, if all the effects they were studying had to be real.

This would be simple to do by submitting ethics applications and an analysis plan to a trusted third party which would only release them once the author is in a position to publish, or at a pre-agreed cutoff (perhaps 2 years), whichever is the shorter (to avoid scooping). Perhaps I should set something up...

When some people find that their model doesn't quite fit, they make a more accurate model. Others make a less specific model. It's the difference between model parametrization and model selection.

So when we get a dubious result, we can either say "no result" or "possible result". The choice tends to depend on how the finding affects future research. Biology is more exploratory than confirmatory, so they go that way.

People occasionally mention p-value calibration and note, sadly, the damage caused by this reckless practice that allows false results to eke through the airtight, 300' tall walls of scientific publication. But there is value in being wrong. It's a part of science.

In a way, it's the MD's White Coat syndrome applied to PhDs. Something that is scientific and written in a journal is necessarily correct in public opinion instead of the rigorously considered opinion it really is. Both paper-reading public and the authors of some of those papers tend to believe this.

And to cover it from a Bayesian point of view, it's pretty vital to keep the culture such that the risk of publishing something incorrect doesn't to strongly dominate the decision to publish. You should be confident talking about your beliefs long before they distribute like deltas.

In reality it doesn't turn out this way because the results that get written and published tend to be biased in favour of novelty and demonstrating a relationship rather than the absence of one. How many similar experiments could have been terminated, never submitted or not published because they failed to show anything notable? This is one of reasons... 'Why Most Published Research Findings Are False' http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

That meta-study applied to medical studies and I think this genre would probably fair even worse when it came to long term replicability.

A million times yes. Also: no publication without the experiment's methodology and criteria for success having been registered prior to the experiment's commencement.

Instead, I should have focused on science journalists, who should be extra diligent when reporting these sorts of stories to point out the possibility that this is a false positive.

I agree, but we need to be really clear about what the claims are.

It could be that there is a reproducible 1% "mystery" effect that works from future to past, but only in experiments like this. In which case claim wouldn't be extraordinary, it'd just be something we can't understand.

Remember that he's still in the data gathering stage. If -- and it's a big if -- there is any kind of reproducible pattern that doesn't match known laws, that doesn't mean there is a claim. There is simply data that doesn't fit our current models.

That's why people who push frontiers have to be very, very careful about differentiating the data from the claims.

Lots of guys make lots of money with bogus TV shows and books on stuff like this, and it's a shame: many times there is something unusual in the data, but the claims jump far ahead of any reality. Fear of this effect has probably silenced a lot of little tiny pieces of data that wouldn't make sense -- it's simply too much trouble to have to keep explaining yourself. This might be one of the factors explaining Feynman's story of the Millikan oil drop experiment.

Erm, no, that would be pretty damn extraordinary. We know of nothing else in the universe that acts like this.

The idea that the effect could be the result of a programming error or a small amount of light leaking through/around the screen is completely plausible. Or it could be dumb luck as you suggest, but it's extraordinarily unlikely.

This type of attitude is infuriating - if a claim can survive the crucible of peer review then it we can be much, much more certain that it is true and correct. If humans were to posses a limited form of precognition that would be awesome - but before we can claim to posses a thing we must be sure that it is real.

Disease existed long before we figured out causes and targeted treatments; people possessed it, and people applied treatments with varying degrees of success and failure, including death. One of the more interesting inaccuracies of medical history is that of humors (random site - http://www.gallowglass.org/jadwiga/herbs/WomenMed.html). How we treated disease changed over time and is still changing as we learn new things.

My point? It is important to continue to apply rigid scientific study to all manners of phenomena, not only to validate its existence but also to figure out how to repeat or avoid said phenomena, depending on the need, the positives and the negatives of said phenomena. However, we should not turn a blind eye towards what people think they experience just because we have not yet come up with the right tool for measuring or the right study for identifying. There is always some reason behind the claim (even if the reason is "snake oil salesman").

Whatever is behind precognition (to take your example), people claim to experience it and always have made those claims. There is a certain burden of proof required, sure. How do you convince someone born deaf that there is a thing like sound that is experienced the way the hearing experience it? In the case of precognition, it tends to be self-validating (sometimes self-fulfilling), and yet it is still useful for precogs or those who believe in them, whether it is illusion or real, whether we have proven it concretely or not.

That bears a quick repeat... The information is somehow useful. These people who are shouting "open your mind" find their precognitive information useful; in their minds, challenges to this useful information are silly. In the name of understanding, the real focus should be on figuring out how that information is obtained. Is it psychic phenomena, a ghost whispering in the ear, great subconscious brain processing, or something else?

So if the response to an outright simple rejection is "open your minds", I think it is warranted. If the response is to indicate disagreement, however, I always thought that to be a useless response, as useless as the simple rejection.

Good scientists know there is much yet to be discovered.