#123 from R&D Innovator Volume 3, Number 10          October 1994

Even the Experts Can be Wrong
by Patrick J. Hannan

Mr. Hannan, a chemist, retired from the Naval Research Laboratory, lives in Bethesda, Maryland.

Readers of R&D Innovator won’t be shocked to hear that the experts can be wrong.  Certainly the public accepts the fallibility of TV weathermen, political pundits, and economists.  But aren't scientists expected to be a cut above the ordinary?  Perhaps, but I wouldn't bet the farm on experts, regardless of their station.

My special awareness of the problem dates back to 1960, when I was a chemist at the Naval Research Laboratory (NRL).  The first nuclear-powered submarine, Nautilus, had been launched in 1956.  Because the nuclear power plant allowed the submarine to remain submerged indefinitely, the limiting factor became the breathability of the air, and academic and industrial laboratories were bombarding the Office of Naval Research with proposals to use a mass culture of algae for purification.

The existing systems for absorbing the CO2 exhaled by the men and replenishing the O2 supply worked well, but a “back-to-nature” approach, using microscopic algae, merited investigation, and I was assigned to examine its feasibility.  Though ill-equipped for the assignment (I'd never studied algae), my ignorance became a multifaceted blessing, as I described in Chemtech (December, 1991). 

For example, I found that the new quartzline lamps—which the manufacturer said must be operated within 4° of horizontal—worked flawlessly in the vertical position.  The literature told me that the light intensity I found best was far too high for optimum algae growth.  And my experiments showed a very different relationship between cell numbers and oxygen production than predicted by expert consultants.

Similar experiences with this project made me especially sensitive to the fallibility of experts.  The same phenomenon is painfully obvious in scientific peer reviews, which editors of scientific journals use to evaluate papers for publication.  The use of reviewers who have proven expertise in the field (as least by the editor's standards) seems reasonable, but sometimes the reviewer lacks expertise in the subject, even though it would seem to be in the reviewer's field.  Sometimes the reviewer carries emotional baggage which diminishes objectivity. 

The words of Sir Gustav Nossal, a Nobel Prize winner, come to mind: 

“In most discoveries as they first emerge, there is a sufficient element of doubt and tentativeness to give scope to a stern critic; it is relatively easy for an intelligent man to pick holes in the incomplete but truly new discovery.  The overly critical, highly intelligent, but unoriginal scientist can become a negative force in research.”

To make the same point somewhat differently, remember that when the Wright brothers made history at Kitty Hawk, they weren't flying a Boeing 747.  No invention comes to light in its final form.

Expert’s Errors

I developed a more vivid awareness of reviewers' errors while examining the many volumes of Current Contents.  This weekly publication monitors the number of times particular articles are cited as references in the literature.  When an article achieves unusually high recognition, the author is asked to provide a short, informal account of the circumstances regarding the work.  I’m impressed with how many authors of frequently cited articles (indicating significance in the eyes of fellow scientists) had great difficulties with reviewers. 

Recently, Current Contents sampled highly cited articles, and more than 6 percent mentioned reviewer problems.  I particularly enjoyed the history of a paper by J. W. Paulley, published in British Medical Journal (2, 1562, 1960) and referenced more than 195 times.  Paulley’s recollection (Current Contents, Clinical Medicine, December 7, 1987) was:

“In a lighter vein, it may be a comfort for frustrated authors to know that the article was rejected by one journal and then by the one that eventually accepted it.  Fortunately for me, editors were then more independent of their expert assessors than they are today.  I pleaded to my editor that the fact that a cardiological advisor did not like my inclusion of aortic and coronary artery involvement ... because he had not personally seen it, or bothered to read about it, was insufficient reason for rejection.  I also reminded him the speed of a convoy was that of the slowest ship and that if assessors’ entrenched opinions were always to prevail, much deserving work would be lost.”

You’ve probably come across the word pulsars.  In the late 1960’s, astronomers did not agree about the origins of these rapidly pulsing radio sources.  In 1968, the Cambridge radio astronomy group reported observing four sources that produced short, regular radio pulses.  However, Thomas Gold of Cornell’s Department of Astronomy had envisioned such stellar objects 17 years earlier.  Gold wrote to the organizers of a 1969 conference asking for five minutes to discuss his theory, and was told:

“The suggestion was so outlandish that if this was admitted there would be no end to the number of other suggestions that would equally have to be allowed.”

Instead, Gold wrote a paper and sent it to Nature, which received it on May 20.  The editor, impressed, rushed the paper into print five days later.  What a contrast to the response of the “expert” conference organizers!

In 1968, reviewers for the New England Journal of Medicine rejected a paper on the prevention of symptoms from Wilson’s disease, because the authors had humanely disregarded a cardinal rule of medical ethics.  This rare and usually fatal buildup of copper in the body causes coloration of the cornea, atypical hepatitis, anemia, and psychiatric effects.  The authors, Irmin Sternlieb and I. Herbert Scheinberg, noted that Wilson's is particularly difficult to study and treat, since few patients exhibit symptoms for the first six years.

Sternlieb and Scheinberg suspected that two diagnostic tests could predict the disease, and that penicillamine was an effective therapy which worked by scavenging copper from the body.  They tested their theory, with one group of patients receiving penicillamine, and the other a placebo; became convinced they were correct; and decided that prolonging the placebo treatment would cause needless deaths.  In their wisdom, the Journal’s reviewing statisticians insisted that the paper was incomplete.  The authors revised it three times, but to no avail.  Finally, editor Franz Ingelfinger elected to print the paper and took the unusual step of writing an editorial to explain his action.  The editor’s judgment has been vindicated through decades of successful treatment of Wilson’s disease.

How Could it be Right?  I Didn't Think of it!

Since reviewers commonly reject significant articles, it would be well for all scientists (particularly young ones) to realize that their brightest ideas might be found wanting.  Why?  Probably because of the “We haven’t done it this way” syndrome.  The experts, you see, hadn’t thought of it—and they’re unlikely to be wrong! 

To explore this cynical opinion, I've discussed the issue with my contemporaries, who commonly remark that the root of the problem lies with reviewers' egos.  We couldn’t agree about the causes of this ego problem, or whether it’s growing more intense.  Arrogance and ignorance might form the underpinnings of this ego problem (the former being perhaps constant and the latter increasing).  

History indicates that significant advances are commonly under-appreciated.  Was the public interested when the typewriter was invented?  No.  What did Alexander Fleming do after 10 years of effort on penicillin?  He gave up.  If the medical community couldn’t even recognize the value of the first effective antibiotic, should we be surprised when breakthroughs of lesser importance--or greater subtlety--are ignored?

I assume that you, my reader, are an expert in some facet of science or technology.  Are you preventing good ideas from seeing the light?