Magnetic Healing and the Importance of Reproducibility, Then and Now

As a scientist, reproducibility is the little demon always lurking in your shadow.  At the helm of a series of often delicate experiments and complex analyses, you always wonder what will be discovered by those in your wake.  Did you carry out your study properly?  Were there any variables you didn't account for?  Above all, can an equally competent scientist independently reproduce your results?

In the past couple of years I’ve noticed increasing concern for reproducibility.  It’s an essential part of science, since it helps to cancel out the effects of human fallibility and conflicts of interest.  The more often a research finding is reproduced, the stronger a foundation it forms for future knowledge-building.

The problem is that most scientific journals do not publish straightforward reproducibility studies, with occasional high-profile exceptions -- for example, the apparently erroneous measurements of faster-than-light neutrinos mentioned in last week’s Nature podcast. Clearly, we need a place for reports like these to be routinely filed so that the value of any and every study can be more accurately judged.  Last week's Science podcast mentioned one such initiative in the field of psychology, called PsychFileDrawer.  I think every scientific field needs a repository like this, and the reproducibility studies should be linked to the original articles to minimize the time wasted following dead-end roads.  

As scientists, we need to be our own worst critics, for two main reasons:

1. People outside our own specialties are not trained to critique our work properly; and
2. When external audits discover problems, the whole of science suffers a blow to its reputation.

Of course, problems in scientific reproducibility are not a new phenomenon.  In his Compleated Autobiography, Ben Franklin mentions the case of a quack named Franz Mesmer who popularized a form of magnetic therapy based on his “theory” of animal magnetism:

We were much amiss’d in France for the past two years with the pretended new art of healing by what was called Magnetisme Animale.  The professor of this art, Mr. Mesmer, had in a short time made nearly twenty thousand Louis d’ors by teaching and practicing it.  He proposed the presence of a universal animal magnetic fluid that could heal the sick and prevent illness through a simple laying of hands, gestures, and signs… The remedy met with enough success to encourage the hopes of the ill; and even educated people, including doctors and surgeons, admitted to the school of magnetism.

One of Mesmer's devices, a magnetic tub called a baquet.  (Source) 

Franklin and others were curious and a little skeptical, so on orders of the King of France, he and a group of independent experimenters of the Academy of Science carried out their own study of Mesmer’s art of healing:

In our investigation, we tried to detect the presence of the magnetic fluid; but this fluid was imperceptible to all the senses.  The experiments we conducted on ourselves, including blindfolded subjects, caused us to reject it absolutely as a cure of illness.  We concluded that the system of magnetism did not cure anything; that both magnetism and his brilliant theory exist only in the imagination; and that a spectacle such as this seemed to transport us to the age and the reign of the fairies.

Unfortunately, Mesmer and his successors proceeded to swindle many credulous patients after this report was published.  Perhaps Mesmer earnestly believed in the technique.  Maybe his patients derived some comfort from the practice.  In any case, at least the study of the Academy of Science helped prevent magnetic healing from becoming an accepted mainstream medical treatment.

Non-reproducible science is bad science, and bad science is anti-science.  It's better to publish nothing at all than to publish an incorrect result.  As Franklin put it:

…one must not be indifferent to the ill-founded reign of false opinion: the sciences, which grow larger with the truth, have even more to gain by the suppression of an error.  An error is always a spoiled yeast which ferments and eventually corrupts the substance in which it is introduced.  But when this error leaves the empire of the sciences to spread among the multitude, and to divide and agitate minds when it presents a misleading way to heal the sick whom it discourages from looking elsewhere for help, a good government has an interest in destroying it.

Given the explosion of scientific inquiry in our era, we need a more concerted, systematic approach to finding the “spoiled yeast” and removing it before it spreads.  I think these reproducibility repositories are a step in the right direction.