Tobacco Documents Online

: Meta-Analysis in Epidemiology, with Special Reference to Studies of the Association Between Exposure to Environmental Tobacco Smoke and Lung Cancer: A Critique'. Joseph L. Fleiss Columbia University New York, NY USA and Alan J. Gross Medical University of South Carolina Charleston, SC USA 1. This research was supported by a grant from The Tobacco Institute, Washington, D.C. USA July 6,1990

Abstract Meta-analysis, a set of statistical tools for combining and integrating the results of independent studies of a given scientific issue, can be useful when the stringent conditions under which such integration is valid are met. In this report we point out the difficulties in obtaining sound meta-analyses of either controlled clinical trials or epidemiological studies. We demonstrate that hastily or improperly designed meta-analyses can lead to results that may not be scientifically valid. We note that much care is typically taken when meta-analysis is applied to the results of clinical trials. The Food and Drug Administration, for example, requires strict adherence to the principles we discuss in this paper before it allows a drug's sponsor to use a meta-analysis of separate clinical studies in support of a New Drug Application. Such care does not always carry over to epidemiological studies, as demonstrated by the 1986 report of the National Research Council concerning the reported association between exposure to environmental tobacco smoke and the risk of lung cancer. On the basis of a meta-analysis of 13 studies, ten of which were retrospective and the remaining three prospective in nature, the Council concluded that nonsmokers who are exposed to environmental tobacco smoke are at greater risk of acquiring lung cancer than nonsmokers not so exposed. In our opinion, this conclusion in unwarranted given the poor quality of the studies on which it is based. - 2

I. Introduction A working definition of meta-analysis is given by Huque 1: "...the term 'meta- analysis' refers to a statistical analysis which combines or integrates the results of several independent clinical trials, considered by the analyst to be 'combinable'." As indicated by this characterization of meta-analysis, its key application is to be found in the analysis and synthesis of data from clinical trials. The question then remains, can meta-analytic techniques be applied in the analysis of other kinds of data such as those that arise in cohort and case-control studies found in epidemiology? The answer to this question is a guarded "yes." The criteria for reaching this affirmative answer are now considered. In applications of meta-analysis to clinical trials, the following questions, among many others that must be addressed, arise. . Are all published studies to be included in the meta-analysis, or only the "good" ones? . Are all studies to be included in the meta-analysis, or only the published ones? . When there are heterogeneous studies, how may they be included in a meta-analysis, or should they be included at all? . Within each study, should all subjects in a treatment group be considered in a meta-analysis, or only those subjects who were compliant with the treatment? The same question applies to subjects in the control group. 3

Similar issues are of concern in epidemiological studies . However, when case-control studies are under consideration, the issue of "intention-to-treat," which is the final question just listed with regard to meta-analysis in clinical trials, is not of direct concern. Instead, the following additional question needs to be addressed in a meta-analysis of case-control epidemiological studies. Has proper control or adjustment been made for the biases that frequently occur in epidemiological studies, such as sociodemographic or clinical differences between study populations, misclassification of subjects with regard to case-control status and to levels of exposure, factors other than the level of exposure that may affect whether -"°:;ubject is a case or control (i.e. confounding variables), and the publication bias/file drawer phenomenon wherein studies that fail to show a positive association tend not to be published and are thus not candidates for inclusion in the meta- analysis? Meta-analysis was first applied to the study of psychotherapy and to the study of educational interventions (see Hedges and Olkinz, for example), and is now widely used to provide overviews of randomized controlled clinical trials. It is also applied in the synthesis of data from epidemiological case-control studies, but, as will be covered in Section 3, with uncertain theoretical justification. Among the principal uses of a properly performed meta-analysis are: . To increase statistical power for important endpoints and subgroups. . To resolve controversy when studies disagree. . To improve estimates of effect size. 4

. To answer new questions that were not previously posed in the individual studies. How meta-analysis is applied in both randomized clinical trials and epidemiological case-control studies are the topics of sections II and III of this paper. Special emphasis will be placed on epidemiological studies of the reported association between exposure to environmental tobacco smoke and lung cancer. II. Applications of Meta-Analysis to Clinical Trials Before reviewing and criticizing the application of meta-analysis to epidemiological studies, it is worth while to review and critique its application to a methodologically stronger kind of study, the randomized controlled clinical trial. We will identify a number of areas of uncertainty and controversy concerning such applications, present the points on which consensus seems to exist, and then use the results of this review as a template for our critique of meta-analyses of studies in epidemiology. Analyze all published studies or only the "good"ones? In their review of published meta-analyses, Sacks et al.3 found that nearly 309'0 of them combined results from both -randomized and nonrandomized studies. If there is unanimity among meta-analytic methodologists on any issue, however, it is on the requirement that only randomized clinical trials be included in a meta-analysis.*' 8 These experts take it as axiomatic that the potential for bias in the nonrandomized assignment of patients to treatment groups is too great for the results of such studies to be trusted. There exist examples of nonrandomized studies that are, in other respects, superior in quality to randomized studies, but the concern about the quality of nonrandomized studies in general is a valid one. 5

The principle that meta-analyses be restricted to randomized studies is by-and- large appropriate. Having agreed on the criterion of nonrandomized treatment assignment for excluding a study from a meta-analysis, the experts disagree on other possible criteria for excluding studies (absence of double-blinding, efficacy rather than intention-to-treat analysis, study is out of date, etc.), and indeed disagree on whether any randomized trial, no matter how poorly designed, should ever be excluded. ''9 Hedges, for example, points out that it appears to be standard procedure in meta-analyses in the physical sciences to delete experiments that are deemed to be flawed.10 H.J. Eysenck, a British psychologist and philosopher of science, labels as "mega-silliness" the practice of including methodologically inadequate research in a meta-analysis.ll Chalmers and his colleagues have developed rigorous, reproducible and unbiased methods for measuring the quality of a study.12 One may use the derived measurements to decide whether to accept or reject the study for a meta-analysis, to determine in an informal way whether a study's quality and its estimate of treatment effect are correlated, 13 or, more formally, to weight studies differentially according to their measured qualities (such a suggestion was recently made by Jenicek14). One way to carry out this latter strategy is to modify the meaning of the weight to be assigned to a study's estimated treatment effect, e, in the weighted average E = Ewe/Ew. In most applications of ineta-analysis, w is roughly proportional to the total number of patients in the study. If q is the study's value on the measure of 6

quality, perhaps scaled to vary from 0 to 1, the proposed overall measure of the effect of treatment is E = E (qw)e/E(qw). A large study whose quality is good will receive all or nearly all of the weight it is entitled to, whereas a study of comparable size but measurably poorer quality will have its contribution to the weighted average correspondingly reduced. We are not aware of any meta-analyses in which the measures of quality have actually been formally incorporated into the analysis. The proposed method is novel, and worth a critical evaluation. A.; lyze all studies or only the published ones? When planning a meta- analysis, the investigator must decide whether to analyze only those studies that had been published or to analyze as well studies that had not been published. Apparently, no consensus has yet been reached concerning which strategy to adopt. Yusuf"I recommends that the results of both published and unpublished controlled trials be considered for analysis; he and his colleagues were successful in locating several unpublished studies in an influential meta-analysis of the effects of beta blockers after myocardial infarction.15 Abstracts of papers presented at meetings and abstracts of master's theses and doctoral dissertations were perused in the pioneering meta-analyses of studies evaluating the efficacy of psychotherapy. 16 There are two main reasons for including unpublished studies in a meta- analysis. One is to overcome "publication bias," the acknowledged tendency of reviewers to recommend against and of editors to decide against publishing studies that failed to show an effect of treatment. The other is to overcome the bias 7

due to the related "file drawer phenomenon," the ter dency on the part of the - author not even to bother submitting for publication an article that fails to show an effect.l' If either of these two sources of bias operates, then a meta-analysis only of results reported in published articles will tend to overstate the degree of statistical significance of the treatment's effect, and to overestimate that effect. An alternative to the difficult task of searching for unpublished studies is to attempt to undo this bias by applying statistical adjustments to the data. 111,19 These statistical procedures are still too new for them to have been theoretically and empirically evaluated. Thus, effectively, the publication bias/file drawer issue remains a serious problem in performing a meta-analysis. Chalmers et a14 have warned that anything less than a systematic and rigorous attempt at obtaining the results of unpublished studies may reduce bias but at the cost of unduly increasing imprecision. The careful and thorough search for unpublished studies is an expensive and time consuming endeavor, but no validated alternative is currently available. Analyze only homogeneous studies? Some statistical reviewers at the U.S. Food and Drug Administration have strongly criticized the pooling of results from controlled clinical trials in which there is heterogeneity of treatment effect -- that is, sizable differences exist between studies in their estimates of the effect of treatment -- and have suggested that it is valid to combine results only from studies in which the estimates are sufficiently close one to another.z0,21 Stein, in fact, denigrated as a mere "computational exercise" the meta-analysis of studies in which the estimated treatment effects were heterogeneous.21 Sacks et al. refer to this criterion as combinabilitv.3 8

Fairly straightforward statistical methods exist to test the hypothesis of heterogeneity of both continuous measurements 22 and categorical data23 (see the statistical appendix), although not all FDA reviewers are in agreement as to how strict the statistical criteria should be for deciding that a set of studies are not combinable.' Furthermore, it is not clear that these reviewers would accept as evidence for efficacy the finding of a statistically significant pooled effect even if the meta-analysis were restricted to studies that were combinable. Some justification may be granted to the tough stand taken by these FDA reviewers, as they are responsible for interpreting and applying regulations handed down to them. In settings other than regulat. -y ones, however, it is not obvious that the criterion of combinability must always be satisfied before a meta- analysis may be applied. DeMets, for example, questions the meaning that attaches to the overall results of a meta-analysis when there is heterogeneity across studies.24 Others, however, suggest that it is precisely when studies differ with respect to the magnitude and perhaps even the direction of treatment effect that the formal methods of meta-analysis are needed to summarize in an unbiased manner all of the information available to date.6.25 With respect to the possibility that the effect of a treatment is strongly positive in one study and strongly negative in another, Peto states that "(this) situation... would be unusual, although certainly not impossible (p.233)."s The frequency with which such a aualitative interaction occurs may be greater than he and others (including the two authors of this paper) have believed. A recent randomized controlled trial of the post-infarction effect of a calcium channel blocker, for example, found this very kind of interaction.26 9

Whether and how to carry out a meta-analysis in the presence of heterogeneous effects are still unanswered questions. There appears to be only one point on which there is agreement: It is invalid to delete from the set of studies to be meta-analyzed those whose results are in the "wrong direction," for the opportunity for bias in identifying the "deviant" studies is too great. Furthermore, one would be left drawing the inane conclusion that "in those studies in which the treatment effect was positive, the overall effect of treatment was positive." Intention-to-treat versus efI`icacv an vsis: The controversy that exists concerning the appropriate samples of patients to be analyzed within a single trial carries over into the realm of meta-analysis. According to the intention-to-treat principle, patients are to be analyzed within the treatment groups they were randomly assigned to, no matter how much or how little treatment they actually received.27 In an efficacy analysis, on the other hand, only data from compliant patients are analyzed.2,8 Sacks et al. found, in their review, that nine of 19 meta-analyses that considered this issue restricted their analyses to studies that employed the intention-to-treat principle, and nine analyzed data from either kind of study.3 Only one meta-analysis restricted attention to studies in which efficacy analyses were performed. Because efficacy analyses tend to produce overestimates of a treatment's effect, and intention-to-treat analyses tend to produce underestimates, caution suggests that, when sufficient information is provided to ascertain which approach was used, only studies that analyzed data from the more conservative intention-to-treat perspective be included in a meta-analysis. When studies that performed efficacy analyses are included in a meta-analysis, 10