Tobacco Documents Online

i! CHAPTER 1 Science is observation, experimentation, measurement, and all these involve numbers, whether we reporters pay attention to them or not. Statistics are used or misused even~by people who tell us, "I don't believe in statistics," then claim that all of us or most people or many do such and such. The question for reporters is, how should we not merely repeat such numbers, stated or implied, but also interpret them to deliver the best possible picture of reality? We can be better reporters if we understand how the best statisticians-the best figumrs-figure. And if we learn a few questions to help us separate the wheat from the chaff: I do not say that telling the truth-describing reality-will'~ then become easy, for we are constantly bombarded'witli sweep- ing claims in convincing wrappings, and the disputed subjects are endless. Medical and~ surgical treatments, radiation, pesti- cides„ nuclear power, the probability of environmental disasters, the side effects of inedicines-almost nothing seems settled. Like it or not, we must wade in. Whether we will' it or not,, we have in effect become part of' the regulatory apparatus. Dr. Peter Montague of Princeton University tells us, The environ- mental and toiric situation is so complex, we can't possibly have enough officials to monitor it. Reporters help officials decide where to focus their activity" "f,ournalists opened up" the Love Canal toxic waste issue by "independent investigation," according to Cornell University's I1r: Dorothy Nelkin. The extensive press coverage contributed to investigations that eventually forced the re-staffing of the En- vironmental Protection Agency and the creation of a national toxic waste disposal! program:"' That very coverage, however, may also have stampeded public officials into hasty, ill-conceived studies that left un- answered ~ the crucial question: Did the Love Canal wastes ac- tually cause birth defects and other physical problems?2 The very way we report a medical or environmental controversy can affect the outcome. If we ignore a bad situation,, the public may f I FACTS Af:D FIG suffer. If we v. "no danger," tI& experimental i false hope. It isnot , National Can refuse to con think "carcinc persons proba cancers are er most inforrnec related main] and very pos percent ofL aL carcinogens- foods, air, an When it issues, or wl- making the si state or unde of he 1J statisti~ at terpretationm evident; you negative]. A sterile is mon that apple pi We also the space or news di:recto ~ story yet." EN done. In a r major southc traction afren who worked numbers fro

suffer. If we write "danger; the public may quake. If we write "no danger," the public may be falsely teassured. If we paint an experimental rnedicali treatment too brigtitly, the public is given false hope. It is not just what we write, it is what we emphasize. A National Cancer Institute survey indicated that many persons refuse to consider healthy changes in life-style because they think "carcinogens are everywhere in the environment." Such persons probably have read or heard again and again that most cancers are environmentally related, although, in the opinion of most informed scientists, most fatal "environmental" cancers are rdated~ mainly to individual behavior, outstandingly smoking, and very possibly diet. By various estimates, perhaps 5 to 15 percent of all cancers are related to exposures to man-made carcinogens -chemicals we have inserted into the workplace, foods, air, and water.' When it comes to such emotionally charged and complex issues, or when it simply comes to nanning for page one or making the six o'clock news, the best among us sometimes over- state or understate. Philip Meyer, veteran reporter and' author of Rairion ,Journalirm, writes, 'Journalists who m.isinterprett statistical data usually tend to err in the direction of overin- terpretation.... The tzason for this professional bias is self- evident; you usually can't write a snappy lead upholding [Ithe negative]1 A story purporting to show that apple pie makes you sterile is morr interesting than one that says there is no evidence that apple pie changes your life' We also work fast, sometimes too fast, with severe limits on the space or tirne we may fill. We find it hard to tell editors or news directors, "I haven't had; enough time. I don't have the story yet:' Even, a long-term project or special may be hurriedly done. In a newsroom "long-term" may mean a few weeks. A majpr southern newspaper had to print a long, front-page re- traction after a series of front-page stories alleged that people who worked at or lived~ near a plutonium plant suffered in excess numbers from a blood; disease. "Our reporters obviously had

FACTS AND FIGURFS: WE CAN DO BETTER 7 not patently absurd, it may not be the ltad you would go for a year later" We reporters are also subjecr to human hope and human fean A new `cure" comes along, and we want to believe it. A new alarm is sounded, and:we too tremble.. Alarrns also make news. We too often obey a sardonic maxim: Bad news is good news; good news is no news. Ih: H. Jack Geiger, a respected former science writer andnow a profes- sor of medicine, says, I know I wrote stories in which I explained' or interpreted the results wtvngiy. I wrote stories that didn't have the dixlAuners I should have written. I wrote stories under competitive pressure, when it became clear later that I shouldti t have written them. I wrote stories when I hadn't asked-because I didn't know enough to ask-Was your study capable of getting the answers you wanted? Could' it be interpreted to say something else? Did you take into acmunt possible confounding factors?'

6 CHAPTER 1 confused statistics and'scientific data;' the editor admitted. "We did not ask enough questions."s We tend to oversimplify We may report„ "A study showed that black is white" or "So-and-so announced'~t}iar ...," when a study merely suggested that there was some evidence that such might be the case. We may slight or omit the fact that a scientist calls a result 'preliminary." As scientific unsophisticates, we may confuse a study that merely suggests a hypothesis that should be investigated-very frequently the case-with a study that presents strong and~ condusive evidence. We often omit essential perspective, context, or back- ground! Dr. Thomas Vogt of the Kaiser Permanente Center for Health~ Research tells of seeing, the headline `Heart Attacks From Lack of 'C' " and then, two months later, 'People Who Take Vitamin C Increase Their Chances of a Heart Attack"a Both stories were based on limited, and'i far from condusive, animal studies. Scientists who do poor studies or overstate their results deserve part of the blame. But bad~ science is no excuse for bad journalism. We tend to rely most on "authorities" who are either most quotable or quickly available or both, and'they often tend to be those who get most carried away with their sketchy and unconfirmed but "excfting" data-or have big, axes to grind, however lofty their motives. The cautious, unbiased scientist who ~ says, "Our results are incondusive" or "We don't have enough data yet to make any strong statement" or "I don't know" tends to be omitted or buried' someplace down in the story. We are influenced too by intense and growing competition to tell~ the story first and tell it most dramatically, I was once asked by a Harvard researcher, "Does competition, affect the way you present a story?' I thought and had to answer, "We have too almost overstate. We have to come as dose as we can within the boundaries of truth to a dramatic, compelling statement. A weak statement will go no place" Another reporter said4 'he fact is, you are going for the strong [lead and story]. And, while FACTS AND FIC not patently a' year later.'" We repor fear. A new "c new alarm is ~ Alarms a maximc Bad r Jack Geiger, a sor of inediar I know I wrote wrongly. I wroi written. I wrot dear later that hadnh asked-1 capable of getti say something factors?' How car.. N © N W C11 ~ IV ~ CA j

The Cerlainty of Uncertainty Too muah of the Kornce reporting in the press [blurs] what we'tc sure of and' what we're not very sure of and what is incandusive. The notion of tentative- ness tends to diop out of much reporting. -Lk. Harvey Brooks The only trouble with a sure thing is the uncertainty. -Author unknown THE first thing to understand about science is that it is almost always uncertain. A scientist, seeking to explain or trn- derstand something-be it the behavior of an atom or the effect of the toxic chemicals at a Love Canal-usually proposes a hypothesis, then seeks to test it by experiment or observation. If the evidence is strongly supportive, the hypothesis may then become a theory or at some point even a law, like the law of gravity. A theory may be so solid that it is generallyy accepted. Example: the theory that cigarette smoking causes lung cancer, for which almost any reasonable person would say the case has been proved, for all practical i purposes. The phrase "'for all prac- tical purposes" is important, for scientists, being practical peo- plh, musr often speak at two levels: the strictly scientific level and'the leveJ of ordinary reason that we require for daily guid- ance. Example: In June 1985, 16 forensic experts examined the bones that were supposedly those of the "Angel of Death," Dr. Josef Mengelt. Dr. Lowell Levine, deltgated by the Depart- ment of Justice, then said; 'he skeleton is that of Josef 0 THE CERTAIhM Mengele withi.r cos Segne of th with ~ the law o cians." Pushed important mati of the patdiolof findings.' (Iat, But all' ar tainty in almos bilit}• that such Widely bc wholly or part) say" reports L , Jnurnal of Madi, help the public with an eltmer a probable nat not certainty. V best opin,'^*t att furure." ) Exa.--,.e: mended'that M cal cancer. Th three years fo:. Statistics had matter is still c changed again Scientists a failing. Whe sionalliy' shows right, the scier ing. The publi have a hard sions. We all todaN, and ano show discussir

f Mengele within a reasonable scientific certainty;` and Dr. Mar- cos Segre of the Uivversity of Sao Paulo, explained, "We deal with the law of probabilities. We are scientists and not magi- cians" Pushed by reporters' questions- after all, this was an important matter, and what should the public believe? -several of the pathologists said they had "absoliitely no doubt" of their findings.` (Later evidence made the case even stronger.) But all any scientist can scientifically say -say with cer- tainty in almost any such case-is, there is a very strong proba- ? bility that such and such is true. : Widd'y believed theories or conclusions are often proved ~ wholly or partly wrong. 'When it comes to almost anything we * say;r reports Dn Arnold Relman, editor of the New Ergland s fournal of Medici'ru, 'you, the reporter, must realize-and~ must ; help the public undetstand-that we are almost always dealing ; with an dement of uncertainry. Most scientific information is of a probable nature, and we are only talking, about probabilities, not certainty. What we are concluding is the best we can do, our t best opinion at the moment, and~ things may be updated in the t~ future. Example: Until 1980 the American Cancer Society recom- mended that women have an annual Pap smear to detect cervi- cal cancer. The recommendation was then changed to every three years for many women, after two initial' examinations. Statistics had shown that this would be equally effective.j The matter is still controversial, and the recommendation has been changed again in the light of new knowledge. Scientists are often wrong. In science this is not necessarily a failing. When new evidence disproves an old~ theory, or occa- sionally shows that some little believed, even kooky notion is rigfit, the scientific method is doing what it should. It is work- ing. The public, and even some reporters and especially editors, have a hard time understanding these sometimes drastic revi- sions. We all hear the question, Why do they say one thing today and another thing tomorrow? I was once on a radio talk show discvssing unsettled medical controversies when a testy

10 c}ihPTE.R 2 listener phoned in to exdaim, ` They say is a damned liar!" 'hey" of course may be different theys who arrive at dif- ferent conclusions about inconclusive evidence in a thousand' areas: the role of fats and~ cholesterol in the diet, the effects of low-level radioactivity; the cause of' the extinction of dinosaurs. Why so much uncertainty? Science is always a continuing story. Nature is compltx, and almost all methods of observation and experiment are imperfect. "There are flaws in all studies," says Harvard's Dr. Marvin 2,r1en.' There may be weaknesses, often unavoidable ones, in the way a study is designed or con- ducted'. Observers are subject to human bias and error. Subjects fluctuate. Measurements fluctuate. Many studies are thus inconclusive, and virtually no single study proves anything. "Fundamentally" writes Dr. Thomas Vogt, "all scientific investigations require confirmation, and un- til it is forthcoming all results, no matter how sound they may seem s are preliminary.' Medicine, in particular, is full of disagreement and con- troversy. "No clinical trial is ever perfect" Harvard's Dr. John Bailar observes. Unlike new drugs, medical treatments and tests and surgicali operations need, not even be subjected to experi- mental studies before being applied. `Most treatments escape and will' continue to escape rigorous evaluation" Bailar says.s The reasons are many: lack of funds to mount enoughh trials; lack of enough patients at any one center to mount a meaningful trial; the expense and difficulty of doing multicenter trials; the swift evolution and obsolescence of medical tech- niques; the fact that, with the best of intentions,, medieal data- histories, physical examinations, interpretations of tests, descrip- tions of symptoms and discases-arz notoriously inexact and vary from physician to physician; and the serious ethical obsta- dts to trying a new procedure when an~ old'~ one is doing some good, or to experimenting on children, pregnant women, or the mentally ill! While all studies have flaws, some have more flaws than others. Study after study has found that many artides in the most prestigious medical journals are replete with shaky statis- THE CERTAINTY tics and lack of tients' complica: up. Papers pres reported by thc mere progress : tive results that or criticism or uncertain findi The upshc organization's care is based a . . Seemingly doctrines, perp out to be suppK be found." In genera possible benef. that only a ra cancer. Only r less dra-"- trc om}; o-" t_t~ is rich in, tren or statistically discarded. Occasiom sults: More of data that contr tical methods ascribing fraui inmindtheo competence tc So some tainry need n survive on, th policy, to govc basis of incom can do so. N C N W CA ~ N ~ m O

tics and lack of any explanation of such crucial matters as pa- tients' complications and the number of patients lost to follow- f up. Papers presented at medical meetings, many of them widely reported by the media, are even Itss reliable. Many papers are mere progress reports on incomplete studies. Some state tenta- tive results that later collapse. Some are given to draw comment or criticism or get others interested in a provocative but still uncertain~ finding.b The upshot, according to Dr. Gary Friedman of the Kaiser organization's Permanente Medical Group: "Much~ of health care is based on tenuous evidence and incomplete knowledge. . .. Seemingly authoritative statements and accepted~ medical doctrines, perpetuated through textbook and lectures, often turn out to be supported' by the most meager of evidence, if any can be found.' In: general, possible risks tend to be underestimated'd and possible benefits overestimated. For decades surgeons swore that only a radical mastectomy was the treatment for breast cancer. Only recently were clinical trials mounted to show that less drastic treatments seem equally effective. Prefrontal lobot- omy, overstrict bed rest, drugs by the c.arload-medical history is rich in treatments that were given for years without question or statistically rigorous study, only to be proved wrong and discarded. Occasionally, unscrupulous investigators falsify their re- sults. More often, they may wittingly or unwittingly play down data that contradict their theories, or they may search out statis- tical methodfr that give them the results they wanr. Before ascribing fraud, says Harvard's Dr. Frederick Mostelltr, " keep ~ in mind the old saying that most institutions have enough in- competence to explain almost any results" So some uncertainty almost always prevails. But uncer- tainty need not stand in the way of good sense. To live-to survive on this globe, to maintain our health, to set public policy, to govern ourselves -we almost always must act on the basis of incomplete or uncertain information. There is a way we can do so.

Somehow the wortdrous promise of t}ie earthl is thai ithert are things beautiful in it, things wondrous artd alluring, and by virtue of your trade, you want to underswtd them. -Mitchell Feigenbaum Corref/ Uaite+tity physiciu md'rmd&~ The great tragedy of Sciener-the slaying of a bwutifullhypothesis by an ugi~ fAct. -'ITwmas Henry Huxlev TO neporters, the worid is full of true believers, peddling their "truths." The sincerely misguided and the outright fakers are often highly convincing, also newsy. How can we tell the facts, or the probable facts, from the chaH? We can borrow from science. We can try to judge all possi- ble claims of fact by the same methods and rules of evidence that scientists use to derive some reasonable guidance in ~ scores of unsetded issues. As a start, we cam ask these questions: How do yvu knom? Have the cfaims ban subjeckd'to any studies or experiments? Were the studies acceptable ones, by general agraenunt ? For exam- ple: Were they without any substantial bias?' Have nsulls been fairly consirteni from study to study? Have du fmdtngs >•uvlted in a coruenrtu among others in the same frcld ? Do at luast the majority of infornrrd ' fxrsonr agrec?' Or should ux unlhhald judgnrn! until there is more euidence? Always: Are the cancGut'aru backed by beGictzbk stasistrcal aiderrce.P 12 THE SCIEh „rlC ' And mhat it t/u c be? Obviousty, rather than nur that reporters c There art usefiil' ones: T}' interpreting da a way of' extr-ac( of mathematic: Statistics c and inexpert si be difficult for t possible. Unce in~ almost all. There are "Edison had it author. 'It doe lt did not take ton's et~ -.nt t. centuny . .,'9( until'then hac Overwhe probability, 4 and the use c called the on many events: women, yet t before it bec: develbp hear some years r was to: womt The bes line (for exar a study is ac

® ® ® ® THE SCI&VIIFIC WAY 13 And uAet is tke degna of ccriainty or unartuiruy.~ How sure can you be? Obviously, much of statistics involves attitude or policy rather than numbers. And much, at least much of the statistics that reporters can most readily apply,, is good sense.. There are many definitions of statistics as a tool. A few useful ones: The science and art of gathering, analyzing, and interpreting data; a means of deciding whether an effect is real; a way of extracting information from a mass of raw data; a set of mathematical, processes derived from probability ttteory. Statistics can be manipulated by chaiiatans, seif-deluders, and inexpert statisticians. Deciding on the truth of a matter can be difficult for the best statisticians, andsometunes no decision is possible. Uhcertainry will ever rule in some situations and lurk in~ almost all. There are rare situations in which no statistics are needed.. "Edison had it easy," says Dr. Robert Hooke, a statistician and author. "It doesn't take statistics to see that a light has come on."' It did not take statistics to tell 29thrcentury physicians that Mor- tons ether anesthesia permitted painltss surgery or to tell 20th- century physicians that the first antibiotics cured infections that until then had' been highly fatal. Overwheltningly, however, the use of statistics, based on probability, is called the soundest method' of decision making, and the use of large numbers of cases, statistically analyzed, is called the only means for determining the unknown cause of many events. Birth control pills were tested on several hundred women, yet the pills had to be used for several years by millions before it became unequivocally dear that some women would develop heart attacks or strokes. The pills had to be used for some years more before it became dear that the greatest risk was to women who smoked and women over 35. The best statisticians, let alone practitioners on the firing line (for example, physicians), often have trouble deciding when a study is adequate or meaningfL Most of us cannot become N ® ® ® ® .,...-ow. *:. ® ®