The history of medical science is commonly taught that of a perceptive few discovering the true cause of a disease, and then being silenced by an unreceptive establishment. Ignaz Semmelweis is a commonly given case. He theorized that the difference between death rates in two maternal health clinics was doctors washing hands in one, but not the other. Perhaps physicians brought some pieces of cadaveric material into contact with their patients, and so he proposed they wash in a solution of chlorine lime. As we know now, hand washing kills or washes off the bacteria which cause the fever, and chlorine lime is a powerful disinfectant. Still, as Semmelweis could offer no theoretical reason it could work, doctors were resistant to adopt his treatment. Semmelweis later suffered a nervous breakdown, was committed to the asylum, and died as the result of a fight with guards. The medical establishment is cast as unthinking louts, resistant to truth and totally imbecilic.
I don’t think this narrative is fair to the establishment. Sure, they ended up being wrong. Would you have been able to tell? Would anyone? In the case of Semmelweis, the experimental evidence was probably sufficient to recommend the practice’s adoption, but even then his reasoning was completely wrong. He thought it was very specifically cadaveric material, and only recommended handwashing after trips to the morgue. His theory was not particularly different than a miasma theory of disease, which is also defensible as making predictions which lead to the reduction of disease.
Since you are conditioning the inclusion of scientists in your mind by them eventually being right, you’re ignoring the thousands of kooks who were seen as wrong then, and are very much wrong now. For every Semmelweis who, without theory, arrives at an improvement, there’s a crank claiming that some herb or mineral cures all ailments, or that you just have to take this genuine snake-oil to cure rheumatism or dropsy or consumption or vitiligo.
Democritus was the first person to think that the world was made of irreducible atomic bits. Now, did we really know enough then to know whether he was right? He was guessing — same as everyone, then. Can we really blame Aristotle for rejecting him? Heraclitus believed the fundamental stuff of the Universe was fire, Thales believed it water. Are those any more founded a guess than Democritus’s? Rejecting poorly reasoned thought leads to more right than wrong, and we would be more likely to come to bad conclusions than good, if just choosing between ideas at random.
Or take medical science. It turned out that the germ theory was correct — but were we really justified in thinking that for many years? The medical establishment largely dismissed homeopathy too — were homeopaths really more or less based in science than early theorizers of germ theory or anything? All pioneers have an element of madness to them, or else they would never have found the courage to explore new ideas. Most of the time, people’s hypotheses are wrong and we oughtn’t forget that.
So, it bugs me when people act as if scientific grant-writing is awful and terrible and does nothing. I am thinking in particular of an article published on the blog “Experimental History”, discussing ways of improving it. Of course there are things we could do to improve it. Since results are not guaranteeable ex ante, we need to be more tolerant of failure. We also need less specification about what scientists are going to find, and we probably have too much time wasted on filling out grants. But to pretend it does nothing is surely absurd.
There aren’t that many outright stupid grant applications, but we live in a world where outright stupid grant applications in the hard sciences would get detected and rejected, at considerable time cost to the applicant. Don’t assume this would last in a world where requirements are lower! By all means, we should undertake riskier projects as scientists. Do not pretend that there are easy and obvious ways to simply choose the good projects, and not choose the bad.