Tobacco Documents Online

® News~ Numbers A GUIDE 'IC7 REPORTING STATISTICALCLAIMS AND CONTROVERSIES IN HEALTH AND OTHER FIELDS Victor Cohn SENIOR WRRER AND COLUMNiST, FORMER SCIENCE EDf[DR. Wachingfan Flui FOREMP-D sv Frederick Mosteller ROGER G,LEE PROFESSOR EMERTMS OF MATHEMATICAL STA77STdCS, Hmrmd (Iniurney A1~-oject of the CentEr for Health Communication Harvard School of Public Halth Iff IOWA STATE UNIVERSITY PRESS / AMES ® 8 m ® ® ®

A Note tc ® 1989 Victor Cohn. All rights neserved Compoeed by Iowa State LJnivreisry Pness Printed in the United States of Americs No pan of this book may be reproduced in any form or by any ekcvonie or merlianical means, inrliiding information aorage and reuieval rystems, without written pennission from the publisher, except for brief passages quoted in a nview. First edition, 1989 Library of Congress Cataiogin;-in-Publication Data Cohn, Victoq 1919- News & numbers. 'A project of the Center for Health Communication, Harvud School of Public Health.' 1. Public health-Statiaia. 2: Environmental bealth-Statistics. 3. Vital .utiatio. I. Harvard School of' Public Health. Center for Health Communiea- uon:, II. Tide. III. Tide: News and numbers. RA407.Cb4 1989 362.1'021 88-6807 ISBN 0-8138-1442-1 ISBN 0-8138-1437-3 (pblc.) THE ruTe fied. They aF stated or iinF porting, busii This gui language of : about t},- ^na on sor.( P project of the health and th ciples and mz used by inqu: a scientific re environment weighandc shows how tJ N 0 N W U1 ~ W

A Note to Readers THE rules of statistics are the rules of good thinking, codi- fied. They apply to any kind of reporting in which numbers- stated or implied-are involved: political reporting, science re- porting, business, economics, sports, or whatever: This guide is an attempt to explain the role, logic, and language of statistics, so we reporters can ask better questions about the many alleged facts or findings that rest, or should rest, on some credible numbers. Because this manual began as a project of the Harvard School of Public Health, the reporting of health and the environment is the major example. But the prin- ciples and many of the suggested "questions for reporters" can be used by inquiring reporters in any field. They can help you read a scientific report or listen to the conflicting claims of politicians, environmentalists, physicians, scientists, or almost anyone and weigh and explain them. And the final chapter specifically shows how these principles apply in all areas. ! VICTOR COHN N 0 N Ca t!1 ~ N >~A tiP CA 0 0

~._ ~_--- - ~. Contents hQRF:WL)FCI) KY F':rdentk r4fnatel4., ix ACKNCXNLEDC:Iv1F.P1"I?5; xi 1. Facts and Figures-We Can Do Better, 3 2. The Certainty of Uncertainty, 8 3. The Scientific Way, 12 Probability, 14 'Power' and Numbers, 20 Bias and Confounden, 24 Vuiability, 30 4. Studies, Goo&and Bad; 35 Experiments versus Seductive Anecdotes, 37 Clinical Trials, 38: What Makes a Study Honest' 40 Epidcmiolcgy: Hippocrates to AIDS, 43 S. Questions Reporters Can Ask, 48 6. 'I'ests and Testing, 64 Drugs and Drug Triali, 68 Animals as Models for Us. 72 7. Vital Statistics: The Numbers of Life and Health, 74 Crude Rates versus Rates Tham Compare, 76 OtAer Ways to Compaee, 78 Rcponing Hoapita! Death Rate, 79 Cancer Rates and Cancer'Cutei , 86 The linportant Quetioru about Cuxer, 88 Shifes, Drifts, and Blip, 96 vii ® ® ® w a 0 ® 0

viii CON'1't:N'tS 8. The Statistics of Environment and Risk, 98 Who's Bdleva6le? 1U7' Qlleft1oT/5 ttl Ask, 108 Evaluating Envnnnmemal Huards, 116 Advice from kcponers, 121. 9. The Statistics of Politics, Economics, and' Democraey 126 The State of'the Nation's Sutittia, 146 The Bonom Lne„ 151 w r•1 t:R t-: '1 v r.EA R N moR r: A Bibleogapliy and orheT Sourcu, 153'. NO'1'F.S, 157 GLOSSARY/iNAEX, 165 Foreword REPORIT science to the accvracy.A1th! stories, the bic( presents speci2 misleading mt sistent," and 'y sults that ane layTnen' r definitio.__ -a~ftc siderable differ Science h such as biostat have been imp ertheless, they permanent for Victor Cc ual to'help all wants to give t facts or mystif Cohn's bo Science Policy Research and that faculty m have been able

/ < ® N Foreword REPORTERS play an essential role in communicating science to the public. In common with scientists, they desire accuracy Although health and medicine provide many exciting stories, the biostatistScs that scientists must use in their studies presents speciaI problems for reporters. It gives uncommon and misleading meanings to common~ words like "significant," "con- sistent," and "power." Mathematical statistics often produces re- sults that are disturbingly counterintuiti've, at least at first, to laymen and scientists alike. In vital statistics and epidemiology,, definitions often seem arbitrary, and slight changes make con- siderable differences in the findings. Science writers often take short courses in special topics such as biostatistics. I have taught in some of these courses and have been impressed by the seriousness of the participants. Nev- ertheless, they need some of this material in an accessible and! permanent form. Victor Cohn~ of the Y1rashington Post has prepared this man- ual to help all reporters cut through these statistical tangles. He wants to give them a guide to the ways that statistics can darify facts or mystify the reader. Cohn's book grew out of the Media Project of our Health Science Policy Working Group of the Division of Health Policy Researeh and Education at Harvard University. I am pleased that faculty members of the Harvard School' of Public Healtlt have been able to help him produce this book as a visiting fellow ® ix 0 ® t ®

x FOREWURD in 1978 and 1984 and as a contributor to the Health Science Policy Working Group. Through the Media Project, with the help of Jay Winstens we have also examined sources of pressures on the science writer.' In the future we wanr to use what we have learned through many discussions with science writers to advise scien, tists on their role in the media. By such e$brts, including this book, and by many similar efforts in this and other fields, scientists and writers may gradu- ally upgrade the whole communication system, scientific and journalistic: Thus we may clear the communication channel between science and~ the public. FREDSRICK MOSTELr _FR AcknowlE MY mai has been Ur. tus of mathe partments of Harvard Sch energy, and! } for the fact • approach rat: statcrr ~r Tt,, prc tions and by ing„ which p journalists, b Ididmt School of Pu Center for IF guide, and : Thomas A. l were Dts. C Kaiser orgar: and Peter Iv writings 116 Cass Pete.r,o Runkle, no : I aLso o, .

i i R ~ MY main mentor and guide in the preparetion of tlus book r has been Dr. Frederick Mosteller, Roger I. Lee professor emeri- s tus of mathematical statistics and former chairman of the de- s pamnents of Biostatistics and' Health Policy and'Managemenr, 2 Harvard School of Public Health. He gave so fully of his time, ; energy, and knowledge that he should be listed as coauthor but for the fact that I sometimes used a journalist's freewheding ~ approach rather than a statistician's rigor. This makes any mis- ;~ statements mine. The project was supported~ by the Russell Sage Founda- tion, and by the Council for the Advancement of Science Writ- ing, which pointed the way in holding seminars on statistics for journalists„ induding the first of its kind in 1964. I did much of the work as a visiting fellow at the Harvard~ School of Public Health, where Dt: Jay Winsten, director of the Center for Health Communication, was another indispensable guide, and Drs. John Bailar III, Nan Laird, Philip Lavin, Thomas A. Louis, and Marvin Zelen were valuable helpers. As were Dts. Gary D. Friedman and T homas M. Vogt of the Kaiser organizations, Michael Greenberg of Rutgers UniNersity, n. and Peter Montague of Princeton University (on all of whose Q writings I leaned); Lewis Cope of the Minneapolis Star Tribune; r, w", Cass Peterson of the Washington Post; and my daughter, Deborah ci Runkle, no mean statistician. ~J1 I also owe thanks to Harvard's Drs. Peter Braun, Harvey ~ ~ ©

Fineberg, Howarr] Frazier, Howatd' Hiatt, William Hsaio, Hetb Sherman, and William Stason. And to Drs. Stuart A. Bessler, Syntex Corporation; H. Jack Geiger, City University of New York; Nicole Schupf Geiger, Manhattanville College; Charjes Moertel, Mayo Clinic; Arnold Reltnan, New Eregland fourrusl ofil?edr<inc• Eugene Robin, Stanford Universiry;and Sid- ney Wolfe, Public Citizen Health Research Group. Also Katherine W2llman, Council of Professional Associations on Federali Statistics; Howard~ L. Lewis, American Heart Associa- tion; Philip Meyer, University of North Carolina; Mildred~ Spencer Sanes; Earl Ubell, WCBS-TV, New York City; and Philip Hilts, Cristine Russell, and Barry Sussman, Washington Po.rt. I am indebted to my editors at the Washington Post, particu- larly Abigail 1 Trafford, Ben Cason, Carol Krucoff, Len Downie, and Howard Simons for their understanding and support. The work was also aided by the Andrew W. Mellon Foun- dation. The American Cancer Society, American Heart Asso- ciation, Commonwealth Fund„ Gannett Foundation, Henry J. Kaiser Family Foundation, Mayo Medical Resources, Milbank Memorial Funds Pew Charitable Trusts, Philip ~ L. Graham Fund, Russell Sage Foundation, and John~ Cowles, Jr., have contributed to this manual's initial distribution.

a Facts and Figures - We Can Do Better Facts and Figures! Put 'em Down!'. -Chules Dick'ens (in 77r Chbnc) There are lies, there arr damtud lies, and thete axe statistics. -Duraeli Almost everyone has heard that 'tigures don't 6e, but liars can figute' We need statistics, but Uars give them a bad name, eo to be abk to tell the liars from the statisticiasu is crucial. , ® -Dr. Robert Hooke I I E journalists like to think we deal mainly in facts and ideas, but much of what we report is based on numbers. Politics comes down to votes. Budgets and dollais dominate government. The economy, business, employment, sport.s-all dtmand numbers. 'Ihe environment, pollutants, toxic chemicals. Again, we see counts and measurements and, most likely, widely varying estimates, some careful, some questionably high or low: An environmentalist says a nuclear power plant or toxic waste dump will cause so many cases of cancer. An industry spokes- man denies it. What are their numbers? Where did they get them? How valid are they? A doctor reports a promising, even exciting new treatment. Is the claim justified or based on a biased or unrepresentative sample? Or too few patients to justify any claim? Science, medi- cine, technology, the weather, intelligence- all are statistical. IN