Tobacco Documents Online

/SPECIAL NEWS REPORTi.w Epidemiology Faces Its Limits The search for subtle links between diet, lifestyle, or environmental factors and disease is an unending source of fear-but often yields little certainty The news about health risks comes thick and fast these days, and it seems al- most constitutionally con- tradictory. In January of last year, for instance, a Swedish study found a sig- nificant association be- tween residential radon k -~osure and lung cancer. ,anadian study did not. ree months later, it was pesticide residues. The Journal of the National Can- cer Institute published a study in April reporting- contrary to previous, less powerful studies-that the presence of DDT metabo- lites in the bloodstream seemed to have no effect on the risk of breast can- cer. In October, it was Anxiety epidemic. Protesting risks that may-or may not-be real. abortions and breast cancer. Maybe yes. Maybe no. In January of this year it was electromagnetic fields (EMF) from power lines. This time a study of electric utility workers in the United States suggested a pos- sible link between EMF and brain cancer but-contrary to a study a year ago in Canada and France-no link between EMF • ' leukemia. rhese are not isolated examples of the conflicting nature of epidemiologic studies; they're just the latest to hit the newspapers. Over the years, such studies have come up with a mind-numbing array of potential dis- ease-causing agents, from hair dyes (lym- phomas, myelomas, and leukemia) to coffee (pancreatic cancer and heart disease) to oral contraceptives and other hormone treat- ments (virtually every disorder known to woman). The pendulum swings back and forth, subjecting the public to an "epidemic of anxiety," as Lewis Thomas put it over a decade ago. Indeed, last July, the New England Journal of Medicine (NEJM) published an edi- torial by editors Marcia Angell and Jerome Kassirer asking the pithy question, "What Should the Public Believe?" Health-con- scious Americans, wrote Angell and Kas- sirer, "increasingly find themselves beset by contradictory advice. No sooner do they learn the results of one research study than they hear of one with the opposite message." Kassirer and Angell place responsibility on the press for its report- ing of epidemiology, and even on the public "for its unrealistic expectations" of what modem medical re- search can do for their health. But many epidemi- ologists interviewed by Sci- ence say the problem also lies with the very nature of epidemiologic studies-in particular those that try to isolate causes of noninfec- tious disease, known vari- ously as "observational" or "risk-factor" or "environ- mental" epidemiology. The predicament of these studies is a simple one: Over the past 50 years, epi- demiologists have succeeded in identifying the more con- spicuous determinants of noninfectious diseases-smoking, for in- stance, which can increase the risk of develop- ing lung cancer by as much as 3000%. Now they are left to search for subtler links be- tween diseases and environmental causes or lifestyles. And that leads to the Catch-22 of modem epidemiology. On the one hand, these subtle risks-say, the 30% increase in the risk of breast cancer from alco- hol consumption that some studies suggest-may affect such a large segment of the population that they have potentially huge impacts on public health. On the other, many epidemiologists con- cede that their studies are so plagued with biases, uncer- tainties, and methodologi- cal weaknesses that they may be inherently incapable of accurately discerning such weak associations. As Michael Thun, the director of analytic epidemiology for the American Cancer Soci- ety, puts it, "With epidemi- ology you can tell a little thing from a big thing. What's very hard to do is to tell a little thing from noth- ing at all." Agrees Ken , Rothman, editor of the journal Epidemiology: "We re pushing the edge bf what can be done with epidemiology." With epidemiology stretched to its limits or beyond, says Dimitrios Trichopoulos, head of the epidemiology department at the Harvard School of Public Health, studies will inevitably generate false positive and false negative results "with disturbing fre- quency." Most epidemiologists are aware of the problem, he adds, "and tend to avoid causal inferences on the basis of isolated studies or even groups of studies in the ab- sence of compelling biomedical evidence. However, exceptions do occur, and their fre- quency appears to be increasing." As Trichopoulos explains, "Objectively the problems are not more than they used to be, but the pressure is greater on the profession, and the number who practice it is greater." As a result, journals today are full of stud- ies suggesting that a little risk is not nothing at all. The findings are often touted in press releases by the journals that publish them or by the researchers' institutions, and news- papers and other media often report the claims uncritically (see box on p. 166). And so the anxiety pendulum swings at an ever more dizzying rate. "We are fast becoming a nuisance to society," says Trichopoulos. "People don't take us seriously anymore, and : "People don't;take :, us seriously.:. and when they; do .. `. e;mayuninten-~. '• tionally do more • ; -harm_tfian good." _ `' - ==Dimitrios - `"`Tiiehopoulos ' when they do take us seri- ously, we may unintention- ally do more harm than good." As a solution, epidemiolo- gists interviewed by Science could suggest only that the press become more skeptical of epidemiologic findings, that epidemiologists become more skeptical about their own findings-or both. An observational science What drives the epidem- iologic quest for risk factors is the strong circumstantial evidence that what we eat, drink, breathe, and so on are major factors in many devas- tating illnesses. Rates of heart disease, for example, have changed much faster over recent decades than can be explained by genetic changes, implicating dietary 164 SCIENCE • VOL. 269 • 14 JULY.1995

and environmental causes. And the fact that no single cancer affects every population at the same rate suggests that factors external to the human body cause 70% to 90% of all cancers. In other words, says Richard Peto, an Oxford University epidemiologist, "there are ways in which human beings can live whereby those cancers would not arise." Only a few of these environmental factors are known-cigarette smoke for lung cancer, for example, or sunlight for skin cancer- and epidemiology seems to provide the best shot at identifying the others. The most powerful tool for doing so is the randomized trial, which is the standard for studies of new drugs and other medical re- search: Assign subjects at random to test and control groups, alter the exposure of the test group to the suspected risk factor, and follow both groups to learn the outcome. Often, both the experimenters and the subjects are "blinded"-unaware who is in the test group and who is a control. But randomized trials would be prohibitively slow and expensive for most risk factors, because they can take years or decades to show an effect and hun- dreds of thousands of individuals may need to be followed to detect enough cases of the disease for the results to be significant. And randomly subjecting thousands of healthy people to pollutants or other possible car- cinogens raises obvious ethical problems. SCIENCE • VOL. 269 9 14 JULY 1995 Because the experimental approach is off- limits for much of epidemiology, researchers resort to observational approaches. In case- control studies, for example, they select a group of individuals afflicted with a particu- lar disorder, then identify a control group free of the disorder and compare the two, looking for differences in lifestyle, diet, or some environmental factor. Potentially more reliable, but also much more costly, are cohort studies, in which researchers take a large population-as many as 100,000-and question the subjects in detail about their habits and environment. They then follow the entire population for years or decades to see who gets sick and who doesn't, what dis- 165

166 ~...~.r. _; Press::;Coverage: Leavin• ~g O~t~ttie~ -3:.~~ .G.Yi:..r1C:~•l..~ts'-.w L. '.riy+'...r..r,~.; a-- `.'; ?`~~~ "7 ",~a + _...~,..i=oftheTor:Ange[esTmurs~~ec~iiomagn~t": ):$ia receiitl iathe~me' tsroii~i=tdadersc~tf~s'r- ~; '~[~i''range=of,p6teria~`aI`" ~~J'~:' ' ~ ':;Tlie£iststiiily;5f?23;OOOFreneh`and' .~,~.~ `ut~ir~eztraordinary.< ~"~~~tiig these puiative'hazards of snodein life;ate:'j~ orkeis; ~o"~und no _liiik:Eietwceri EMF- ~breast_iuipTatits; dioxin, atress; `astiestos; `alletgy drags;" .; ~tid 15:ti£~ie'-13 varieties of cancer in xlie stud}', ,the-exceptioiis, ' S:Y-:' . tvwg;m Otangc County, tulia°l ligation; sunscreen; ~`~wp r'are types~of'leukeuiis; haii awealc;arid'inconsiste'ntpqsitive 'EMP;rYe" #; pesticides; vase~tomy, Iiquor; working in restaurants, ;`associaao:iz, vitlTtahe Wall;§creet JournaI reported the y Yegetables;;taiy fat; delayed cliild-bearing; impun-:~.~~=stud:.lasE:sprin~nndet."`the lieadlme~~vlagneuc~Fields Luiked:to "'f ~ _"'~~ ~~; ea~aiid,les~ianism: This litany;of fear was accotnpariied' `Irepkemiii,":, v;..•,_:..::: t gy publislied la%'ialtlioiigli slioitie ; eries of.repoits on iiierary ilabits: ~' ' '~: Eaily this'~ear, iie liirerscim Jounwl af Epuieriiiolo tyles tliat.aiay reduce cancerrisk. PaCrallel.coverage ap ~;, ~the,aecond study, on `i39;OQ0; workers at:five U.S.' utilities.; It = Yother riesvspapeis and.magazines and.on television To ound no associatiori between exposure.to EMF and 17 of 18 zypes ienfi'sts .''ttiough;; rkiemedia. would:do :well to ceirb'.'its ~=ofccaricer,:;including't~ie'Ieukemiasariked•.to:EMF: by"the:fiist . '^ a_ .t`. .. , .. . .. . . ... , , - . fo%~ii~ news~ w ~ - . - =~ :'= -- ='Y ud •The:'sole'•eacceptions~avei~e'eye~and brain•cancers=condi= ~ ,~: Y nbleiii;~inany`tesearchers say; is that y'oariialists nften :-s:=fions that tiad shown noaiiik to EMF in.the firststudy. Ye`t°the' ~;. ist_aiid.th'e-`cbii-texi:of tlie research: 'Because of..r.lieriita=>':ri~ieadlirie.of the Wali Street Joicrnal article thatrePoited the second acto'"rtpidemiolo~y; most individual stddies:canno['' Between-:EMF, :Brain Cancer .is:Suggested=by.:• . ioritauve findings°(see • main text):`;."Articles. ` uIi~ ;t -r~..•S•tudy at-5 Utit iiies ~ Says Jerry Bishop, who wrote one•of t1ie4Wall - . IniiinaLs are'often misconsuued by thelaypte5s~treeeJourriir7 aracles,-eople arenot:interested CnMtiqhatdiseases `' Iiad;' tfiari;t3iey really are„'~saYs Lai?Y Freedman,~a~~~aiis .1CfarEo=Tdcesn'tcatise, but whatit miglitcause:»:X~Ve'v~. argument witli scientrsts, many New Stady Links Abortions and Increas® in Breast Cancar Risk +r {~ 3zoccoli pre-::. ~-times over _the past fewlears . ~~...,.., -... a. •....~ ~ ... . , .r . ... ,,. _,.. .... - ~• s,. _~io" revents ~Zirries~ P. ~;.. ~ . +" ' . •.~. ... r iovided another exain le"' ;wlieii`: it ~.... . ~'. P. r . terature Biir epide= _ „ reported. on'a `stirdy'_:iii;the=Joumal of ~ s~a~'e~tscaiid:very~rvell .' 'Magnetic Fields Linked to Leukemia~ -.rhe: Nadonal Cancer Tnsucuie: tINCI).• ; 1ies~are far• om a,e M.,~ c,°av, scUav; from the Fred HutcliinsoitCancerRe`:; „--,..-•, _,,, ~arar~ `orw.-+ irunu.rr.~ ai k : ..._ .~.a~. . F.>= to sn~ E~ ~.search Genter iii SeatElexliatsu e's' ..r . }~~~~~~vhcn a°bodp .'_._ t~ c.~ ~r tne c~ -.. n ..r.. .. ~ ~ ~., , ` ed' induc'ed;aabortio inciease ' - rmariyrmanystudies n;;migl?t, li' io ~lioiil3reailygerseri te'risk tif:'bieast;cancet.by:50% ~ tha~ 40<pr'e- vu>~t~e PublicadyTce" Between EMF, Brain Cancer though the:aiticle,notea ` ~~_. ~ ~' `§' ;;:vious stiidies' of aliortioti.'aiiea ~ing surve~s`of tlie:iiig~ ts Suggested by Study at 5 Utilitiest _ , e2~d ~ieis say, thenme- ~ ,, r:, ~~,~ ~ cancer hac~found no such Eozre~atto~, [3ut InkWrv•F~nwiecf Wark i e inrsolaa FmbNoL:ukmuaRuk. r+-~~ tlie headliiie zead `~Te~Stud Ltn~Cs ~_oK.•~Y., Unlike Wrr R.yurti, XIm,= W.. ..~ rm~, x 1^...rrt Co "ura~`~e3=by~:Press:'iel~eases~u~, w„« : ~ ' ~ ~= x Cancer 'Risk:' ' Inevirabry;~pu~iiic~t .` ,. `=- ~~~.., _ . : . :' c.:..,u` ' y . .... izr.rs ri~stitutioiis> Bbi evec ~ "~`~ tentiori:,wasidirecfedrto;as _nslk ~~~ .;r:.;:~ ~e~,an,epideniiolog~stat~Iie~iuVe~ iinlikely to bereal: :~ ='~ =~•;~ r ~.-~ •. r ~ ~'~" ^e; tlie~eAresult~is= If tliere is: . blaine. e.; fo~su clg over- - .; tCiVC~tr :age;'argues'I:~wrence ATfirian;` ,-...iave a serions message ~y~ntliof ; ~_ . elarge;numbet sezn of the Tiines article; iniiZh-of iC'be _1~s~`temfro'riL"Tirr ~magazm s_ Aai ~~ t..;. :_-.> to scieiitific - jouma , ls 'Th~•tsent Y-,- ~ 't~•se ~•ecen~earr:hx 1b~ telease to'utin~ that study as: if. it wer"e the.~iggest~hmg _ ~ ae ~ori ha``I don'callhetelhgh~'ti~~ eentrrteee sysrre tm,n lrs tastas ~ ef iaste:goo~: iaa " noneao studres and probably had httlc inean~ng~" ~. ~-. , i ik`inbreafiarf once a n°~tmar''is= view„_epidemiologists who= coinplarn~~ ` ~ ~iho~ ~~ -:co~'Qerag` ~mpaied=with wome~x~are_tryirig to have it botli ways "5cienasts~up ~ o~~attthe~iisk _reducnon= Is~noaller y_want us not to go outsrde tlie scientific process,-butwau : ve a eared in a eer-reviewe~s rofesstonal ~ _ ts~~stst'th`ui~-raii-6eieliably:detec~3;un ha PP P P we. dy-~tocdid itgoint out tliacthe°study=appaim.iilg~~~~uriials"~'~Uhen `do that,, they aPParently: complain•that=we~ ; ~Q~pariisli woineri reported seyecaLweek~~eailie~ats~idit.'rga oiiEside..the scientific process and say..thatra:publislied'.d allourriaIofCancer-isinconfliccwidi~nangvthe~r~~~:tepiircis~ieaninglesa.° .;" 'a ' r ~'~ ~"~ aua~aIists do,overemphasize individual studies;=.liu~t~ey`.aie- ' , t6at die>`aryfats may raisc cazh_ :. . ce%Although ~theoverallfa~reasrran ` rriivit`ed:io :do that'[by 'medical jouiiialsj, .ngrs ~Ross , ~ l~iiye o1 inaypose Iess caidiovascular?irsWeiss's colleagues arthe Univcrsrfy of Washing- 0 •. tfac,~fewepidemiologists •wouldiruespiet=tli sorne of the press releases that journals and uriiver- -,_. ~vqmeit3liould' t~essent~ Lajoumalists.;Itsa wonder sometimes tliat:tlie~ieport= nce~aY~~~ _- _ ~d lChe Cn' _ °news;' ne e%an ~ aeevssr ,; , r~•. -arls 1Glan roclivity .for wspap anr;~ ~ sesniC~ a sea :p~iiegativt c]ataF~ t i¢a~~'t """` =`'es~;1~ u die co arirhor, wuhdvfmk~: Plumirier~of Noah'~~ 00 ~.~_,~ W /~~ SCIENCE • VOL. 269 • 14 JULY 1995 ~

° a K•-~ ~ SPECIAL NEwS REPORT eases they suffer from, and what factors might be different between them. Either way, risk- factor epidemiology is "a much duller scal- pel" than randomized trials, says Scott Zeger, a biostatistician at the Johns Hopkins School of Mental and Public Health. What blunts its edge are systematic errors, known in the lingo as biases and confound- ing factors. "Bias and confounders are the plague upon the house of epidemiology," says Philip Cole, chair of epidemiology at the University of Alabama. They represent any- thing that might lead an epidemiologic study to come up with the wrong answer, to postu- late the existence of a causal association that does not exist or vice versa. Confounding factors are the hidden vari- ables in the populations being studied, which can easily generate an association that may be real but is not what the epidemiologist 6 ks it is. A ubiquitous example is cigarette ,cing, which can confound any study looking, for instance, at the effects of alcohol on cancer. "It just so happens," explains Trichopoulos, "that people who drink also tend to smoke," boosting their risk of cancer. As a result, epidemiologists face the possibil- ity that any apparent cancer-alcohol link may be spurious. Smoking may also have confounded a study Trichopoulos himself co-authored linking coffee-drinking and pancreatic cancer-a finding that has not been replicated. The study, published over a decade ago, corrected for smoking, which often accompanies heavy coffee drinking- but only for smoking during the 5 years be- fore the cancer was diagnosed. Trichopoulos now says that he and his colleagues might have done better to ask about smoking habits a full 20 years before diagnosis. Biases are problems within study designs themselves. The process of choosing an ap- riate population of controls in a case- trol study, for instance, can easily lead to an apparent difference between cases and controls that has nothing to do with what caused the disease. "It's often not even theo- retically clear who the right comparison group is," says Harvard epidemiologist Walter Willett. "And sometimes, even ifyou can design the study so that you have the theoretically correct comparison group, you usually don't get everybody willing to partici- pate, and the people who do participate in your study will be different from the people who don't, often in health-related ways." For example, Charles Poole of Boston University has spent several years analyzing the results and methodology of a 1988 study of EMF and cancer, which found that expo- sure to relatively high EMF from power lines appeared to increase the risk of leukemia and brain cancer in children. David Savitz of the University of North Carolina, the study's author, selected controls for that study with a common technique known as random digit dialing: Researchers take the phone numbers of their cases and randomly change the last four digits until they find a suitable control. Random digit dialing, however, seems to cre- ate "a pronounced bias toward the control group being deficient in persons of very low socioeconomic status," says Poole. Poor people, it seems, are either less likely to be home during the day to answer the phone, less likely to want to take part in a study, or less likely to have an answering machine and call the researchers back. Indeed, the North Carolina researchers reported that their data showed that the risk of leukemia and brain cancer rises not just with exposure to EMF but also with higher levels of breast-feeding, ma- temal smoking, and traffic density, all of which are mar- kers for poverty. This sug- gests, says Poole, that the study group was poorer than the controls, and that some pov- erty-associated factor other than EMF could have re- sulted in the apparent in- crease in cancer risk. None- theless, the study is still cited as supporting the hypothesis that EMF causes childhood cancer, although even Savitz concedes that the random dig- it dialing problem is "a legiti- mate source of uncertainty." Even when such biases can be identified, their mag- sure can be measured reliably, a subtle asso- ciation may be credible-as it is in the case of early childbirth and a lower risk of breast cancer. The reason -is that both cause and effect can be measured with some certainty, says Harvard epidemiologist Jamie Robins. "It's easy to know which people got breast cancer, and it's easy to know at what age they had kids," he says, adding that virtually every study on the subject comes to the same con- clusion: Early childbirth reduces the risk by about 30%. But epidemiologists are quick to list risk factors for which accurate exposure measure- ments are virtually impossible. Joe Fraumeni, director of the epidemiology and biostatis- tics program at the National Cancer Institute (NCI), points to radon: "When you're studying smoking," he says, "that's easy. Just count the number of cigarettes and du- ration and packs per day. But something like radon, how do you measure exposure, particularly biologically rel- evant exposure that has taken place in the past?" Equally uncertain are those risk factors recorded only in human memory, such as consumption of coffee or di- etary fat. Ross Prentice of the University of Wash- ington notes, for example, that underweight individu- "We're pushing the edge of what can be done with epidemiology." -Ken Rothman nitude-and sometimes even their direc- tion-can be nearly impossible to assess. David Thomas, for example, an epidemiolo- gist at the Fred Hutchinson Cancer Research Center in Seattle, points to studies analyzing the effect of Breast Self-Examination (BSE) on breast cancer mortality rates, which, he says, have yielded some "modest suggestion that there might be a beneficial effect" from BSE. "You have to ask what motivates a woman to practice BSE," says Thomas. "Maybe she has a strong family history of breast cancer. If so, she's more likely to get breast cancer. That would be an obvious bias," which could make BSE look less useful than it is. "Or maybe a woman with a strong family history of breast cancer would be afraid to practice BSE. You have no way of predicting the direction of the bias. So it would be very difficult to interpret your re- sults. You have to go to a randomized study to get a reliable answer." Tricks of memory Of all the biases that plague the epidemio- logic study of risk factors, the most pernicious is the difficulty of assessing exposure to a particular risk factor. Rothman, for instance, calls it "a towering obstacle." When expo- SCIENCE • VOL. 269 • 14 JULY 1995 als tend to overreport fat intake on question- naires or in interviews and obese subjects tend to underreport it. Such recall bias is known to be especially strong, as Willett points out, among patients diagnosed with the disease in question or among their next of kin. In studies of a pos- sible relationship between fat intake and breast cancer, for instance, says Willett, "people may recall their past intake of far differently if they have just been diagnosed with breast cancer than if you pluck them out of a random sample, call them up out of the blue over the phone, and ask them what their past diet was." Recall bias, for instance, apparently ac- counts for the conflicting findings about oral contraceptive use and breast cancer. Many studies have looked for this association over the years, both case-control studies and co- hort studies. Trichopoulos notes that case- control studies have tended to show an asso- ciation between oral contraceptives and breast cancer, while cohort studies have not. Epidemiologists who have done cohort stud- ies say the problem is in case-control studies, which are thrown off by recall bias-women who are diagnosed with breast cancer are more likely to give complete information 167

about contraceptive use than women who don't. Those who did case-control studies say the bias is in the cohort studies. Cohort stud- ies have to rely on impersonal questionnaires because they are so much larger than case- control studies, and women are less likely to give complete and honest information than they are in the more intimate interviews pos- sible in case-control studies. "The point," says Trichopoulos, "is which do we believe." It's not just the subjects of studies who are prone to bias; epidemiologic studies can be plagued by interviewer bias as well. The in- terviewers are rarely blinded to cases and controls, after all, and questionnaires, the traditional measuring instrument of epide- miology, are neither peer-reviewed nor pub- lished with the eventual papers. "In the labo- ratory," as Yale University clinical epidemi- ologist Alvin Feinstein puts it, "you have all ds of procedures for calibrating equip- nt and standardizing measurement proce- dures. In epidemiology ... it's all immensely prey to both the vicissitudes of human memory and the biases of the interview." Salvation from statistics? With confounders, biases, and measurement errors virtually inevitable, many epidemiolo- gists interviewed by Science say that risk-fac- tor epidemiology is increasingly straying be- yond the limits of the possible no matter how carefully the studies are done. "I have trouble imagining a system involving a human habit over a prolonged period of time that could give you reliable estimates of [risk] increases that are of the order of tens of percent," says Harvard epidemiologist Alex Walker. Even the sophisticated statistical techniques that have entered epidemiologic research over the past 20 years-tools for teasing out subtle effects, calculating the theo- cal effect of biases, cor- cting for possible con- founders, and so on-can't compensate for the limita- tions of the data, says biostat- istician Norman Breslow of the University of Washing- ton, Seattle. "In the past 30 years," he says, "the methodology has changed a lot. Today people are doing much more in the way of mathematical modeling of the results of their study, fitting of regres- sion equations, regression analysis. But the question re- mains: What is the funda- mental quality of the data, and to what extent are there biases in the data that cannot be controlled by statistical analysis? One of the dangers of having all these fancy mathematical techniques is people will think they have been able to control for things that are inherently not controllable." Breslow adds that epide- miologists will commonly re- port that they have unveiled a possible causal association between a risk factor and a disease because the associa- tion is "statistically signifi- cant," meaning that the error bars-the limits of a 95% confidence interval--do not include the null result, which is the absence of an effect. But, as Breslow explains, such statistical "confidence" means considerably less than it seems to. The calculation of confi- dence limits only takes into consideration random varia- tion in the data. It ignores the systematic errors, the bi- "Authors and investigators are worried that there's a bias against negative studies." -Marcia Angell ases and confounders, that will almost in- variably overwhelm the statistical variation. University of California, Los Angeles (UCLA) epidemiologist Sander Greenland says most of his colleagues fail to understand this simple point. "What people want to do when they see a 95% confidence [interval]," he says, "is say `I bet there's a 95% chance the true value is in there.' Even if they deny it, you see them behaving and discussing their study result as though that's exactly what it means. There are certain conditions under which it's not far from the truth, but those conditions are generally not satisfied in an epidemiologic study." "People [may] - think they have been able to control for things that are inherently not controilabie." -Norman Breslow What to believe? So what does it take to make a study worth taking seriously? Over the years, epidemiologists have of- fered up a variety of crite- ria, the most important of which are a very strong asso- ciation between disease and risk factor and a highly plausible biological mecha- nism. The epidemiologists interviewed by Science say they prefer to see both be- fore believing the latest study, or even the latest group of studies. Many re- spected epidemiologists have published erroneous results in the past and say it is so easy to be fooled that it is almost impossible to believe less-than-stunning results. Sir Richard Doll of Ox- ford University, who once co-authored a study errone- ously suggesting that women who took the anti-hyperten- sion medication reserpine had up to a fourfold increase in their risk of breast cancer, suggests that no single epi- demiologic study is persua- sive by itself unless the lower limit of its 95% confidence level falls above a threefold increased risk. Other re- searchers, such as Harvard's Trichopoulos, opt for a four- fold risk increase as the lower limit. Trichopoulos's ill-fated paper on coffee con- sumption and pancreatic cancer had reported a 2.5- fold increased risk. "As a general rule of thumb," says Angell of the New England Journal, "we are looking for a relative risk of three or more [before accept- ing a paper for publication], particularly if it is biologically implausible or if it's a brand- new finding." Robert Temple, director of drug evaluation at the Food and Drug Adminis- tration, puts it bluntly: "My basic rule is if the relative risk isn't at least three or four, forget it." But as John Bailar, an epidemiologist at McGill University and former statistical consultant for the NEJM, points out, there is no reliable way of identifying the dividing line. "If you see a 10-fold relative risk and it's replicated and it's a good study with biologi- cal backup, like we have with cigarettes and lung cancer, you can draw a strong infer- ence," he says. "If it's a 1.5 relative risk, and it's only one study and even a very good one, you scratch your chin and say maybe." Some epidemiologists say that an asso- ciation with an increased risk of tens of per- cent might be believed if it shows up consis- tently in many different studies. That's the rationale for meta-analysis-a technique for combining many ambiguous studies to see whether they tend in the same direction (Science, 3 August 1990, p. 476). But when Science asked epidemiologists to identify weak associations that are now considered convincing because they show up repeatedly, opinions were divided-consistently. Take the question of alcohol and breast cancer. More than 50 studies have been done, and more than 30 have reported that women who drink alcohol have a 50% in- creased risk of breast cancer. Willett, whose Nurse's Health Study was among those that showed a positive association, calls it "highly probable" that alcohol increases the risk of breast cancer. Among other compelling fac- tors, he says, the finding has been "repro- duced in many countries with many investi- gators controlling for lots of confounding 168 SCIENCE • VOL. 269 • 14 JULY 1995

variables, and the association keeps coming up." But Greenland isn't so sure. "I'd bet right now there isn't a consensus. I do know just from talking to people that some hold it's a risk factor and others deny it." Another Boston-based epidemiologist, who prefers to remain anonymous, says nobody is con- vinced of the breast cancer-alcohol connec- tion "except Walt Willett.". Another example is long-term oral con- traceptive use and breast cancer, a link that has been studied for a quarter of a century. Thomas of the Fred Hutchinson Cancer Re- search Center says he did a meta-analysis in 1991 and found a dozen studies showing a believable association in younger women who were long-rime users of oral contracep- tives. "The bottom line," he says, "is it's taken us over 20 years of studies before some •1r, istency starts to emerge. Now it's fairly there's a modest risk." But Noel Weiss of the University of Washington says he did a similar review of the data that left him unconvinced. "We don't know yet," he says. "There is a small increased risk associated [with oral contraceptive use], but what that represents is unclear." Mary Charleson, a Cornell Medical Center epidemiologist, calls the association "questionable." Marcia Angell calls it "still controversial." Consistency has a catch, after all, explains David Sackett of Oxford University: It is per- suasive only if the studies use different archi- tectures, methodologies, and subject groups and still come up with the same results. If the studies have the same design and "if there's an inherent bias," he explains, "it wouldn't make any difference how many times it's rep- licated. Bias times 12 is still bias." What's more, the epidemiologists interviewed by Sri-nce point out that an apparently consis- body of published reports showing a sitive association between a risk factor and a disease may leave out other, negative find- ings that never saw the light of day. "Authors and investigators are worried that there's a bias against negative studies," and that they will not be able to get them published in the better journals, if at all, says Angell of the NEJM. "And so they'll try very hard to convert what is essentially a negative study into a positive study by hanging on to very, very small risks or seizing on one posi- tive aspect of a study that is by and large negative." Or, as one National Institute of Environmental Health Sciences researcher puts it, asking for anonymity, "Investigators who find an effect get support, and investiga- tors who don't find an effect don't get support. When times are tough it becomes extremely difficult for investigators to be objective." When asked why they so willingly publish inconclusive research, epidemiologists say they have an obligation to make the data public and justify the years of work. They also argue that if the link is real, the public health effect may be so dramatic that it would be irresponsible not to publish it. The Univer- sity of North Carolina's Savitz, for instance, who recently claimed a possible link between EMF exposure and a tens of percent increase in the risk of breast cancer, says: "This is minute. ... But you could make an argument that even if this evidence is 1000-fold less than for [an EMF-leukemia link], it is still more important, because the disease is 1000- fold more prevalent." One of the more pervasive arguments for publishing weak effects, Rothman adds, is that any real effect may be stronger than the reported one. Any mismeasurement of expo- sure, so the argument goes, will only serve to reduce the observed size of the association. Once researchers learn how to measure ex- posure correctly, in other words, the actual association will turn out to be bigger-and thus more criti- cal to public health. That was the case in studies of steel- workers and lung cancer de- cades ago, says Robins. Early studies saw only a weak asso- ciation, but once researchers homed in on coke-oven workers, the group most ex- posed to the carcinogens, the relative risk shot up. None of the epidemiologists who spoke to Science could recall any more recent parallels, however. An unholy alliance There would be few draw- backs to publishing weak, uncertain associations if epi- demiologists operated in a vacuum, wrote Brian Mac- Mahon, professor emeritus of epidemiology at Harvard, in an April 1994 editorial in the ~ ~ b s ~ "The sin comes in believing a causal ' hypothesis is true because your study came up with a positive result." -Sander Greenland Journal of the National Cancer Institute. But they do not, he said. "And, however cau- tiously the investigator may report his con- clusions and stress the need for further evalu- ation," he added, "much of the press will pay little heed to such cautions. ... By the time the information reaches the public mind, via print or screen, the tentative suggestion is likely to be interpreted as a fact." This is what one epidemiologist calls the "unholy alliance" between epidemiology, the journals, and the lay press. The first one or two papers about a suspected association "spring into the general public consciousness in way that does not happen in any other field of scientific endeavor," says Harvard's Walker. And once a possible link is in the public eye, it can be virtually impossible to discredit. As far as scientists were concerned, for instance, a 1981 epidemiologic study put to rest a suggestion that saccharine can cause SCIENCE • VOL. 269 • 14 JULY 1995 'SPECIAL NEWS REPORT bladder cancer-one of the few cases in which epidemiology had managed to put an end to a suspected association. Yet 14 years later, television advertisements for Nutra- Sweet, which contains the artificial sweet- eneraspartame, still tout it as the sweetener that does not have saccharine. Epidemiologists themselves are at a loss as to how to curb the "anxiety of the week" syndrome. Many, like Rothman, simply ar- gue that risk factor epidemiology is a young science that will take time to mature. Others, like Robins, suggest that barring a major breakthrough in the methodological tools of epidemiology, maturity will be hard to come by. The pressures to publish inconclusive re-, suits and the eagerness of the press to publi- cize them, he and others say, mean that the anxiety pendulum, like Foucault's, will con- tinue to swing indefinitely .(see box on p. 165). The FDA's Temple does make one positive sugges- tion: Although risk-factor epidemiology will never be as sharp a tool as random- ized clinical trials, epidemi- ologists could still benefit by adopting some of the scien- tific practices of those stud- ies. "The great thing about a clinical control trial," he says, "is that, within limits, you don't have to believe anybody or trust anybody. The planning for a clinical control trial is prospective; they've written the protocol before they've done the study, and any deviation that you introduce later is completely visible." While agencies like the NCI do in- sist on seeing study proto- cols in risk-factor epidemi- ology prospectively, this is still not standard procedure throughout the field. Without it, says Temple, "you always wonder how many ways they cut the data. It's very hard to be reassured, because there are no rules for doing it." In the meantime, UCLA's Greenland has one piece of advice to offer what he calls his "most sensible, level-headed, estimable col- leagues." Remember, he says, "there is noth- ing sinful about going out and getting evi- dence, like asking people how much do you drink and checking breast cancer records. There's nothing sinful about seeing if that evidence correlates. There's nothing sinful about checking for confounding variables. The sin comes in believing a causal hypoth- esis is true because your study came up with a positive result, or believing the opposite be- cause your study was negative." -Gary Taubes 169