Once upon a time - actually in the late 1980s and early 1990s - fears began to surface about a generation of "crack babies." These hapless infants, exposed to crack and cocaine in utero, would grow into out-of-control berserkers who would be the ruin of society. One magazine article at the time even described some of these victims: a three-month preemie baby with uncontrolled shakes; a three-year-old who functioned at the level of a four-month-old; and a five-year-old who sat stone-faced one minute and ran amok the next.

This firestorm was partially stoked by researchers who reported seriously reduced cognitive functioning and disorganized behavior in children of cocaine users. Rather than playing the way most toddlers do, these little ones scattered, batted, and poked haphazardly at their toys, and demonstrated little sustained play incorporating fantasy or exploration. They also showed poor attachment to their mothers, and were unable to use them as a secure base for exploration or as a source of comfort during stress. More fuel was added by scientists who speculated that cocaine affects neurotransmitters in the limbic system - a system associated with aggressive and violent behaviors.

In fact, the predicted epidemic never occurred. What happened? How could researchers and writers be so wrong? It is a simple question, but it raises all kinds of thorny, sticky issues, such as: What is the research that gives rise to these reports based on?

In fact, scientific research should be grounded in solid design and statistics - the hard numbers that support (or sometimes fail to support) the validity of the research hypothesis. What is the role of scientists in explaining research, and to what extent is it the responsibility of journalists to understand the statistics behind the conclusions? Whose job is it to explain? The scientist's? The journalist's?

What are the limitations of statistics? The samples we use as the basis for our statistics are just that - samples of the population we wish to talk about. And we have to be very careful about what samples we compare. In this case, experts speculated beyond their data, and the media let their statements go unexamined. Poor research design should have knocked many of these studies right out of consideration.

Researchers compared samples of youngsters that were not even close to equivalent. The research with the scariest findings compared babies exposed to cocaine prenatally (many of whom continued to live with their cocaine-abusing mothers) to babies who were simply born prematurely, without regard to the vastly different environments in which they were conceived, carried, and raised.

Then, of course, there was the problem of multiple drug use. If pregnant women used other drugs as well as cocaine, how can one attribute the effects on the babies solely to cocaine? Sophisticated statistical controls could have been used, but often weren't.

Then there is the whole question of how we define and measure outcomes. When we operationalize something, we take a complex notion and render it into something we can quantify. But we often don't know what this means in the "real world." Recently the New York Times published an article on Dean Ornish's program to reverse heart disease. His results, reported in the Journal of the American Medical Association (free registration required to view abstract) seem to indicate that such a reversal is indeed possible.

However, not everyone is buying the research. One criticism addressed the meaning of his outcome measure of width of coronary arteries. The average amount of stenosis of the artery was reduced, but what significance did this actually have for patients? Did they live longer, or have fewer heart attacks?

Ornish's other outcome measure, the number of cardiac events, lumped together heart attacks, coronary angioplasty, bypass, cardiac-related hospitalizations, and cardiac-related deaths. The experimental group had fewer of these events. Sound good? Well, a person can only die once, although he or she can be hospitalized many times. In that case, perhaps more "events" is better?

In other studies, researchers may look at the size of a lesion, or count hours in intensive care. But how do you provide meaning for these measures? Does a smaller lesion mean quicker healing? Does hours in intensive care mean fewer setbacks? Faster progress? Greater health?

At some point, even a significant difference may not be meaningful. If a cancer treatment results in a survival difference of two weeks, and that difference is statistically significant, does it make a meaningful difference? This treatment could be touted as a breakthrough, but if I have cancer and we aren't talking about a difference of months, many months, I'm probably not going to be impressed.

Finally, one of the most important and most ignored axioms in research is "correlation is not causation." Not only could A cause B, B could cause A, or both A and B could be a result of the unmeasured C. There probably is a significant correlation between the softness of road asphalt and the incidence of heat stroke. But would anyone assign causality for one to the other? Or would we decide that it is more likely that both result from the operation of a third variable? Yet time and again, there are reported linkages based not on experimentation but on correlation, as though the causal direction were clear and unequivocal.

Whose job is it to raise these questions, and to put research findings into perspective? The responsibility rests with both the journalist and the scientist. Readers depend on writers to provide not just information, but also its meaning and context - and its limitations.

In an informal and very unscientific survey of writers on the National Association of Science Writers email list, almost all of the 15 people who responded (a small, self-selected, and therefore not representative sample) report doing their own independent verification of findings - going back to the original publication, talking to people who know about statistics, looking at other research in the field.

But about one-third of these writers have had no formal exposure to statistics and are self-taught, having chosen to put in the time and effort to acquire the knowledge they need to do their job. How many less professional journalists simply don't take the trouble?

Scientists also have the obligation to make sure journalists do understand their findings, especially as many writers lack the highly technical background required to follow every step of the research. In the best of all possible worlds, this would happen.

But then, we don't live in the best of all possible worlds. Scientists complain about journalists who misunderstand or misrepresent their research. Journalists complain about scientists who won't speak to them, or hospitals and medical centers who refuse writers access to their doctors. The pressure to be in the media can lead some researchers to overgeneralize or overspeculate about their findings. Public relations departments may give in to hyperbole.

The unscientific sample of science writers cited above told of conferences in which scientists speculated very freely. The sample also described interviews with researchers who left out a few important details, or didn't know their colleagues' research as well as perhaps they should. There were also press releases that presented research in overly glowing terms.

The crack baby epidemic never happened. It was a fantasy fear that started with poor research design and inadequate statistics, grew with the speculation of researchers, and blossomed in an uncritical media. Stories like this will probably appear again. Despite the difficulties involved, we need to make the reporting of science research a joint effort. When we don't we undermine the credibility of everyone, writer and scientist alike.

Neala S. Schwartzberg is a research psychologist as well as a health writer.

Andrzej Krauze is an illustrator, poster maker, cartoonist, and painter who illustrates regularly for HMS Beagle, The Guardian, The Sunday Telegraph, Bookseller, and New Statesman.

Send us your comments and ideas for future articles.

Endlinks

Communicating Science News - the National Association of Science Writers' guide for public information officers, scientists, and physicians who deliver research information to journalists.

Do's and Don'ts for Interviewing with the Media - a brief article with several pointers for interview preparation. From the American Psychological Association.

Under the Microscope - a series of articles on physicians and the news media, including general aspects of science reporting. From the First Amendment Center.

Consequences of Prenatal Drug Exposure - a gateway to research and resources on prenatal drug exposure. From the National Institute on Drug Abuse.

Lies, Damned Lies, and Statistics: The Seven Deadly Sins - in Columbia University's Webzine 21stC, Steve Ross discusses some common statistical problems in media reporting.