Log in

No account? Create an account


The Engineer

The Life and Times of Donald F. Simmons

Previous Entry Share Next Entry

Damn Lies and...

An essay in one of the latest issues of New Scientist I was reading today was about something that's always ticked me off to no end since my days in university, inadequate statistics in scientific studies.

Basically, science journals have been finding that many studies submitted to them have been performed with so few samples that it's impossible for them to say with any certainty whether an effect exists or not, in order to help cover up bad results, or just to let the authors grind out a paper quickly. The most glaring example in recent days was the first tests of the AIDS vaccine, which didn't work. The study suggested that they got some positive results among blacks and Asians that are encouraging, but a closer look reveals that only about 314 blacks were involved in the study, with four infections in the vaccine group and nine in the control group. That's such a small sample the numbers could easily come down to pure chance, and it's meaningless to say with any certainty that the vaccine shows promise.

This ticks me off because back in university, I always assumed going in that statistics courses would be about running experiments, how to figure out what your sample size should be, how to calculate the error bounds, etc. Instead, all we ever did was black balls and white balls in a bag type of problems. If I was interested in taking up gambling, what I did would help, but not if I wanted to do any serious research. I had to look up stuff like this on my own.

IMHO, part of the reason you get bad stats like this is that how to do proper scientific stats isn't part of the core curriculum, and it damn well should be. Bad studies waste time and money, and in medicine lives as well.

  • 1
From AJAY!:

You have to remember that reporting scientific research results has become more about beating others to the punch than being accuate. Remember the big cold fusion debate. When the results of the first experiment were announced (to the regular press, not peer-reviewed), everybody and their dog tried to reproduce them. We immediately got a glut of people agreeing with them, followed a month or so later by people saying it didn't work. Why the time gap? The experimentors who reproduced the results on their first try confirmed the experiment to the press right away and stopped there. Those who couldn't reproduce it kept working on it until they came to the conclusion they could not reproduce the original results.

The end result - cold fusion research based on those results have ceased. Well, except for in Japan, where they still pour money into it. Right now, I'd usually go "Wha...?", but remember, the best Japanese cars are now made in the US, so stuff like this just doesn't surprise me anymore.

  • 1