We live in an age of "junk" science where almost any industry or advocacy group can issue a "scientific" report they claim supports their own particular agenda. No matter how biased or frail the "study", may be, it can find a constituency among Congressional or state legislative committees, the media, or before cheering/screaming crowds who are true believers in whatever point the study claims to "prove."
This chapter will enable you to make your own decisions about the validity of "research" as presented in the media or by advocacy groups. You don't have to be a scientist to make good decisions, but you need to know what makes a valid study and what does not.
Junk science goes unchallenged, for the most part, because our increasingly technological society grows increasingly illiterate in science and technology. When citizens, public policy makers and members of the media are incapable of understanding and intelligently dealing with science, they all become easy pickings for junk science and technological demagoguery. The result is that important public policy decisions concerning science are frequently made based on emotions rather than fact.
Unfortunately, there are all too many ideologues ready to announce yet another "scientific" study that is, in reality, a thin veneer of credible-looking, pseudo-science wrapped around propaganda and hidden agendas.
Two classic examples of this are: (1) the tobacco industry's pack of nonsense and outright lies, presented to Americans for decades, insisting that tobacco didn't kill and (2) the environmental movement's over-reactive hype about Alar which turned out to be a non-danger, but nevertheless crippled apple farmers and discouraged Americans from eating one of nature's healthiest foods.
We all should be skeptical in viewing claims from either industry or advocacy groups. Both have a vested interest in distorting data, scientific or otherwise, to support their own agendas.
The media usually lacks the expertise to examine the scientific credibility of a study even if it had the time to devote to the task. So the result is that debates on public policy boil down to a "he said, she said" argument between two "scientific" positions that, to the scientifically unlearned, look mostly equal.
In these cases, the advocate group espousing the latest ideology -- that can stage a dramatic scene for television cameras in front of dying babies or other filmable scenes -- will win the battle. While it may make good theater, it makes for junk science and bad public policy making.
Good science can not be decided by public opinion polls or debate. Democracy is the best political system in the world, but a majority vote to repeal the law of gravity has no effect on the way that physics or biology or any other science really works.
Junk science wins so frequently because good science rarely offers the unequivocal results that make good dogma. And the quiet, reasoned, cautious voices of good scientists frequently drown beneath the shouts of the true believer's zeal.
This has been true of alcohol issues since Prohibitionists vandalized and destroyed taverns with axes to the present day when the anti-alcohol lobby often distorts facts to support their ideological position.
Health research is fraught with ethical dilemmas, the most important of which is when, how and if to experiment with human beings. Specific areas, such as new drug and medical treatment trials, have strict regulations and the results can be accurately judged.
But the study of lifestyle factors and how they relate to the health of millions of people is not so precise. This is especially true with chronic diseases like atherosclerosis and cancer. Because those diseases take so long to develop and not every person at risk contracts them, they are much more difficult to study. It is simply impractical -- not to mention unethical -- to put thousands of people in a laboratory in order to study their behavior and the resulting health outcomes.
The science of epidemiology studies these relationships. It tries to assess the "Relative Risks" of an individual developing a disease if he/she engages in certain conduct or possesses certain physical attributes.
Epidemiology is an observational science rather than an experimental. Ethics demand that, in most cases, scientists observe people and make conclusions based on those observations rather than perform experiments on them.
Epidemiology studies large populations (typically thousands or perhaps hundreds of thousands) and tries to draw "statistically valid" associations between the disease and certain risk factors. Thus a relative risk of 1 means the event is certain (the relative risk of death, for example) and a relative risk of 0 means the event cannot happen (living forever, for example). The closer the relative risk gets to 1, the more likely it is to happen.
But because epidemiology is a study of populations, it cannot -- with certainty -- predict that reducing relative risks will benefit a specific individual. Fastening your seat belt means you have decreased your relative risk of being injured in an automobile accident; it doesn't mean that you cannot be injured.
Epidemiology is less exact because many of its observations come from questioning people whose recall may be biased or incomplete. In addition, people who refuse to participate in the study may have some characteristic in common which biases the sample of participants and makes it unrepresentative of the population as a whole. Factors unaccounted for, including random statistical glitches, may bias the results.
As imperfect as epidemiology can be, techniques have been developed which increase accuracy and decrease bias. In addition, science has developed a number of "reality checks" that can be used to determine if studies -- epidemiological or otherwise -- are reliable.
You can unmask junk science yourself and determine the reliability of a scientific study by asking a few non-scientific questions.
Look for the name of a respected medical institution or university. Make sure the research was conducted at the institution rather than by someone who is (or was formerly) associated with it. Be cautious of "institutes" with unfamiliar but civic-minded names. Industry and advocacy groups have created hundreds of them to give junk science studies credibility and to hide particular agendas.
In medical research, look for M.D. or a Ph.D. in the relevant scientific discipline. Junk science medical studies often lack M.D.s. They often have an unusually large number of Ph.D.s who
have studied subjects peripherally related or totally unrelated to the science required by the study. A Ph.D. in sociology is rarely qualified to author an article on biochemistry.
Again, like "institutes", industry and advocacy groups have founded dubious "journals" to lend respectability to their junk science.
On the other hand, articles in journals like Lancet, Journal of the American Medical Association, Science, Nature, Cardiology and so forth have been extensively reviewed by panels of qualified scientists who examine them for accuracy and credibility. Beware that many of the advocacy journals have been similarly named with the intent to deliberately mislead.
The more people in a study, the more likely that the study will be statistically valid. Beware of studies with only a few dozen people. The results are more likely to be biased or to produce spurious or indirectly associated results.
A study on white males may, or may not, be valid for white or African American women. Beware of some hidden bias at work.
A spurious association is one which seems to be valid, but in fact, is unrelated. Sometimes these result from random fluctuations in the data used and can produce false results. These can happen to the best scientists with the best intentions. This is one good reason you should not make a decision based on one study. Spurious associations are not repeatable by other teams at other times.
In nineteenth century England, researchers found that cholera struck mostly in low-lying areas. They concluded the disease was carried by smelly contaminated air found in such places. But despite the impressive association that people living in low-lying areas got cholera (as compared to people living on mountain sides) scientists later discovered the disease was caused by bacteria in contaminated water.
Indirect and spurious associations can be eliminated by determining if the associations have a logical mechanism for action.
The search for a biological mechanism between sugar and heart disease turned up another indirect association back in the mid- 1960s. An early study turned up a link between sugar consumption and heart disease. But, as scientists looked for how sugar consumption affected the heart, they learned that cigarette smokers consumed far more sugar than non-smokers. It turned out that smoking was the real cause behind the increased heart disease and sugar was not the culprit, but was only indirectly linked.
Similarly, critics of some of the studies of the beneficial effects of moderate drinking have often charged that the beneficial effects must be due to something else. But the discovery that moderate consumption can increase "good" HDL cholesterol, and that increased HDL levels are related to lower heart attack risk, helps remove the effect from the indirect association category. Likewise, the discovery that moderate alcohol (like aspirin) decreases the tendency of blood to clot in the arteries offers another credible mechanism.
Valid decisions can be made only on data that has been confirmed in a wide variety of studies involving different populations, researchers and analytical methods. As this book has stated numerous times, a valid decision cannot be made on the basis of a single study. It must be made by looking at the overall body of research.
For example, not every study of alcohol and health has shown the cardio-protective effects of moderate consumption. In 1989, Dr. Keith Marton reviewed 22 scientific papers that evaluated the associations between alcohol and deaths from coronary artery disease. Of the 22 papers, 16 demonstrated the lowest mortality in moderate drinkers, five showed no difference and one of them focused on heavy drinkers only and was not relevant to moderate drinking. A closer examination of the five studies showing no difference indicated flaws that biased the studies toward their conclusions.
Since 1989, every major study (such as those by Klatsky at Kaiser Permanente Medical Center in Oakland, Calif., and by Rimm at the Harvard University School of Public Health) have demonstrated significantly lower death rates among moderate drinkers.
If more of a given factor produces more (or less) of a disease, the factor is said to be dose-related. Such a dose-related relationship is further evidence that the factor being studied is the cause.
In the case of alcohol and most pharmacueticals including antibiotics and aspirin, the dose relationships can be charted by J-shaped curves. In these cases none or little of the substance causes little or no effect; increasing the dosage increases the benefits. But at some point, the benefits begin to decrease and side effects appear. With most antibiotics and pharmacueticals, some vitamins and with alcohol, too much of a good thing becomes toxic.
Dose relationships give the lie to many advocacy positions which say that if high levels of a substance are proven harmful, then even extremely low levels of the substance are also harmful. For example, anti-alcohol groups say that since drinking five bottles of wine a day is unhealthy (no argument!) then drinking a quarter of a bottle a day is also unhealthy. This is simply untrue and scientifically invalid. In some cases, with some substances, it may be possible to extrapolate results like this, but not always and not with alcohol.
Junk science can often be disguised in papers filled with footnotes which seem to be valid articles. However, a closer examination often reveals that the article's conclusions are based on old, outdated or unreliable data sometimes taken out of context. At other times, the data is taken third-hand from other government agencies without regard for the accuracy with which it was collected.
Finally, one of the most common ways of making junk science appear to be valid is for a small group of people working in the same field to "swap attribution." In other words, if I write an article which is supported by a shred of data and a seat-of-the-pants opinion, and a friend quotes the article or uses it as a footnote, it gives my article legitimacy while making his article seem like it is quoting a valid scientific study. Next, I can write another article which cites this last one by my friend (and perhaps a few other friends who are doing the same thing). My friends and I write a lot of articles like this which means we get to list a lot of footnotes which give the illusion that any given article is based on a great deal of research.
Alcohol abuse pioneer and Yale professor Selden Bacon has accused people in the anti-alcohol movement of doing just this sort of attribution swapping to lend an air of scientific legitimacy to poorly conducted research which may be without basis at all. Many "journals" in the alcoholism field are filled with articles of this sort.
This is a far cry from respectable, credible scientific publications like the Journal of The American Medical Association and its companions which require rigorous research, original data -- all of which are peer-reviewed by medical and scientific experts.