Tobacco Documents Online

NEGATIVE STUDIES IN THE LITERATURE: SUMMARY OF SELECTED DISCUSSIONS OF NEGATIVE STUDIES IN MEDICAL PUBLICATIONS I include this section to document my statements on pages 4 and 72 about negative studies. Over the past 25 years in academic medicine, I have had the opportunity to serve as an editorial consultant and literature reviewer for several medical and scientific journals. I have also served on different editorial boards and in different editorial functions for editorial boards, including the editing of two recent journal issues on tobacco and health. I have also served at different times on various study sections for the National Institutes of Health, and on various review committees for the American Heart Association, the American Lung Association, The American Thoracic Society, and other professional organizations where I have reviewed for those institutions and organizations a large number of research grant applications, as well as scientific programs. The purpose of this portion of my statement is to comment on negative studies (or the lack thereof) in the medical literature. At the time of the preparation of this statement, there were in the National Library of Medicine over 6,300,000 published articles, exclusive of abstracts, that are accessible for computerized searches through MEDLINE or MEDLARS. Approximately - 140 -

3500 journals are reviewed and entered into this literature data base now on a monthly basis, and the computerized data base end retrieval network are updated daily. These articles have been accumulated only from 1966 to the present. They are all indexed by key words and key phrases. These key words and key phrases are in the titles of the articles, in the abstract of the articles, or in a list of key words and phrases developed by the authors, by the journals, or by the indexing organizations at the National Library of Medicine. Searching these data bases, with the help of the National Library of Medicine, for "negative studies" is quite revealing. Simply stated, there are extremely few negative studies published. In general, less than one-half of one percent (less than 0.5%) of all retrospective studies and of all prospective studies published are truly negative. To the degree that these are focused on tobacco smoking and environmental tobacco smoking, the percentages are slightly better, at 1.1 percent and 0.9 percent, respectively. Thus, in general, about 1 out of every 200 studies published overall is negative and, for tobacco and for environmental tobacco smoke, only about 1 out of every 100 articles published, or slightly less, is negative. As an editorial reviewer, I am very well aware that negative studies are not oublished. As a research grant reviewer, I am very much aware that requests for grant renewals are usuallv=not funded if the initial study funded generated ~ ~ C!I L^, .G - 141 - .i CJ1

neaative results. Simply stated, publication and research funding are contingent essentially on positive results. Although negative study results, meta-analyses, and publication bias are inextricably-linked, a review of statements about negative study results only reveals some important conclusions. The near unpublishability of negative study results is a serious and-widespread problem. It has received careful attention by editors and scholars in the late 1980s. Begg and Berlin published recently the most comprehensive analysis of the problem: Their article was followed by an extensive discussion by international experts (Begg and Berlin, 1988). The causes of bias against negative study results appear to be competition for journal space and the belief among the authors as well as editors that negative results are not as "interesting" and are not "important," and therefore are harder to interpret than "positive" findings. Most important is the refusal of many journal editors to publish clinical trial results that are not statistically significant. Proposed-solutions to the problem include the provision of incentives to editors to publish negative results, the creation of an international registry of clinical trials, and the abandonment of statistical significance as the ultimate qualification for- publication of otherwise important data. Consider the following quotations. ~ a7 t11 - 142 - ~ 0~

Extent of the Problem It is widely believed that reports of negative studies are less likely to be published than those of positive studies, and some data have been put forward to support this belief (Angel,1989). Publication bias, the phenomenon in which studies with positive results are more likely to be published than studies with negative results, is a -serious problem in the interpretation of scientific research (Begg-and Berlin, 1988). On the basis of no quantitative data, I suspect that most negative studies of important matters are somehow reported-and that a broader definition of publication would reduce concerns about publication bias (Professor John C. Bailar of McGill University in a discussion following Begg and Berlin, 1988). So far, no satisfactory procedures have been developed to assure that well-executed negative studies can be readily published. Few definite data are available regarding the impact of this bias, and evidence is largely anecdotal. Nonetheless,-this issue is likely to be of increasing importance in relation to both experimental and epidemiological investigation due to growing public needs for balanced control of potential hazards in the environment (Dr. John Higgins of Georgetown University in the discussion following Begg and Berlin, 1988). Bailer and Simon have both expressed the view that the deleterious effects of publication bias may be over-emphasized on the grounds that negative studies are less important than positive studies and that sophisticated observers presumably have a built-in process of discounting exaggerated claims. We believe that the problem is - sufficiently serious to require remedial action (Begg and Berlin, 1988). Negative trial outcome was stated as the reason for rejection of 7 of 23 submitted papers (30%) or 7 of 13 papers (54%) for which peer review was complete...Sixty-four unpublished trials (31%) went unreported because of the negative results. - 143 -

Lack of interest in pursuing the study any further was cited as the reason for nonpublications of 27 (13%) of the trials (Dickerson, 1987). Sterling was probably the first to emphasize that the tendency to publish positive results and reject negative findings is a serious problem. He reviewed all articles published in four journals in one year (1955 or 1966) and found that 97% of the articles provided data for rejection of the null hypothesis...From results of the present study, it seems likely that bias against the publication of "negative" results does exist. The approximate size of the problem can only be roughly estimated from the data... (Dickerson, 1987). A problem in evaluating different therapies from a review of clinical trials is that_the published clinical trial literature may be biased in favor of positive or promising results. Causes of the Problem It is probably true that authors are sometimes reluctant to acknowledge negative results, partly because they assume they will not be published and perhaps also because of an intellectual or even financial commitment to positive results. I doubt, however, whether such bias on the part of researchers is very important...It is more likely that authors do not submit their works for publication because they have become aware of serious limitations to it, and this may well be the case more often with negative studies than with positive ones (Angel, 1989). ...the interpretation of a negative study is more complicated than that of a positive study (Angel, 1989). ..' many negative reports deal with hypotheses that few experts seriously entertain.(Angel, 1989). ~ ~ C!1 - 144 - C^ ~ GD

What Causes Publication Bias? At the editorial stage there is generally a need to prune the submitted manuscripts as a result of lack of space in the journal (Begg and Berlin, 1988). ...it seems certain that articles quoting statistical significance have an increased chance of acceptance (Begg and Berlin, 1988). - To some extent the perception that editors are concerned that results be conclusive may induce a corresponding response in authors which magnifies this source of bias... (Begg and Berlin, 1988). The potentially distinguishing features (of bias) are presence or absence of randomization, sample size, exploratory versus confirming studies, protocol definition, the nature of the journal, calendar time and source of funding (Begg and Berlin, 1988). ...a highly visible publication can have a dramatic and prolonged impact on medical practice even if the results are demonstrated to be unreliable (Begg and Berlin, 1988). ..* it is surprising that more suppression of 'negative' results appears to be by authors than by editors (Coursol and Wagner, 1986; Dickerson et al., 1987)," cited by Douglas G. Altman of the Imperial Research Fund, London in discussion following Begg and Berlin, 1988). ..e the better designed a study is the more negative results there will be." Dr. JM Bland of St.George's Hospital, London, (in discussion following Begg and Berlin, 1968). Negative results are rarely as important as positive results..." Professor John C. Bailar of McGill University (cited in discussion following Begg and Berlin, 1988). For trails that remain unpublished because of 'negative results,' it was primarily the authors not the editors, who iaade the decision not to go to press." Kay Dickerson of Johns Hopkins - 145 -

University (cited in discussion following Begg and Berlin, 1988). (negative studies] may not be accepted or even submitted for publication unless they deal with a topical issue. Non-publication arises for several reasons, including the view that 'negative' results are 'uninteresting' and that limited journal space could be used more effectively for positive or 'interesting' reports." Dr. John Higginson of Georgetown University (cited in discussion following Begg and Berlin, 1988). Part of the bias is caused by the failure of potential authors to submit reports on statistically non-significant studies and by the reduced likelihood that journals would accept such reports." Thomas A. Louis the University of Minnesota (cited in discussion following Begg and Berlin, 1988). Editors who require a statistically significant finding before a paper may be considered for publication have elevated the significance test to a level of importance that it does not deserve." Professor Steven Schwartz of Queensland University (cited in discussion following Begg and Berlin, 1988). Negative 'exploratory' studies are generally less important than positive studies in providing leads for the discovery of improved treatments." Dr. Richard Simon of the National Cancer Institute (cited in discussion following Begg and Berlin, 1988). The journals are still the primary mode of communication, but they are not suited to the task at hand, since they are motivated by individual incentives, rather than being responsive to a global scientific strategy (Begg and Berlin, 1988). - Since investigators, sponsoring pharmaceutical companies, and journal editors may all be less interested in publishing unpromising 'negative' results than in publishing exciting 'positive' results, overviews based only on published results may be materially biased (Collins, 1987). - 146 -

Negative trials, defined as those showing no statistically significant difference between the test and control treatment or showing superiority of the control treatment, may be submitted for publication less often or rejected more frequently because the authors or editors do not consider the findings worthy of publication (Dickerson, 1987). Proposed Solutions to the Problem "... we have no policy against publishing well- done negative studies. Indeed, we feel a particular obligation to publish a negative study when it contradicts an earlier study we have published and is of a similar or superior quality (Angel, 1989). An additional goal is to encourage investigators to be diligent about publishing relevant negative studies. There is a clear problem of incentives here, and we have no good ideas on this score, although Higginson (1987) has suggested that journals might be provided with financial - incentives to make space for short format reports of negative studies, although the source of such funding is not obvious. Maxwell (1981) has suggested that-negative studies might be published by title only, rather than not at all (Begg and Berlin, 1988). Rather than finding ways to encourage authors to publish negative results we should abandon the concept of positive and negative studies. Statistical significance should cease to be the main way of assessing study results and thus should never be used as a basis for publish- ability." Douglas G. Altman of the Imperial Cancer Research Fund, London (cited in discussion following Begg and Berlin, 1988). ...because a single statistically significant result may be meaningless while a particular sequence of non-significant results may be quite important, replications are mandatory. The publication of programmatic research should be the prime aim of researchers and journal editors whether or not the results are statistically significant. If prepublication replication were the rule, and if researchers were required to - 147 -

justify their sample size on the basis of the expected effect size then prepublication bias would be greatly diminished--it may even disappear entirely." Professor Steven Schwartz of Queensland University (cited in discussion following Begg and Berlin, 1988). ...it is desirable that acceptance for publication be based on methods rather than results .... (Begg and Berlin, 1988). Ultimately the answer lies in registers of undertaken and planned as well as published RCTs. In the meantime we must rely on investigators and editors to recognize the seriousness of the problem and publish well-collected data no matter what the outcome (Chalmers, 1987). These examples illustrate an approach to reviewing the clinical trial literature, which is free from publication bias, and demonstrate the value and importance of an international registry of all clinical trials (Simes, 1986). Meta-Analyses Meta-analysis is a potentially very important tool to analyze a number of randomized trails that are by themselves too small or too undersized to answer the questions posed. The opportunities for the introduction bias, however, are quite significant. This is especially true if there are no negative studies or,- as in the case generally, negative studies comprise only one-half of one percent or so. When meta-analyses are undertaken, the selection of papers must be blinded, the extraction of data must be blinded, and the making of duplicate determinations must be blinded (Chalmers et al, 1987). If these factors are not so blinded, that must be acknowledged and they - 148 -

must be controlled in some other way. Without exception, this has not been done in those studies employing meta-analyses that initially appear to "strengthen" the positive effects of environmental tobacco smoke on a biological outcome. SUlQKARY Negative studies are generally not published. Negative studies are generally not funded in research grant renewal requests.