Tobacco Documents Online

Evaluating the Quality of Articles Published in Journal Supplements Compared With the Quality of Those Published in the Parent Journal Paula A. Rochon, MD, MPH, FRCPC; Jerry H. Gurwitz, MD; C. Mark Cheung, MD, FRCPC; Jason A. Hayes; Thomas C. Chalmers, MD Objectives.-To determine the relationship between the quality of articles and whether they were published in a supplement or in the parent journal. Data Sources and Study Selection.-AII randomized control trials of drug therapies in adults published in the American Joumal of Cardiology, the American Journal of Medicine, and the American Heart Journal from January 1990 and ob- tained in November 1992 by means of a MEDLINE search. A total of 318 abstracts appeared to meet our inclusion criteria, and these articles were obtained and re- viewed in further detail. An additional 76 were excluded. Data Extraction.-Three reviewers who were "blinCa-i" and thus unaware of supplement status independently assessed the quality of each of the remaining 242 articles according to a standard quality scoring system. Data Synthesis.-Overall, 67 (27.7%) of the articles were published in journal supplements. Article quality scores ranged from 4.2% to 87.5%, with a mean (±SD) score of 37.2%±13.1%. Quality scores were lower in articles published in joumai supplements than in those publisl:ed in the parent joumal (t(240]=2.61, Ft=.01). The mean quality score for articles published in journal supplements was 33.6%f12.8°/a compared with a score of 38.5°ia±13.1 % for articles published in the parent journal. Supplement articles included in their final analysis a smaller proportion of the pa- tients initially randomized (t[75]=2.8, P=.007). Conclusion.-Our findings suggest that randomized control trials published in journal supplements are generally of inferior quality compared with articles published in the parent journal. The review process surrounding the publication of journal supplements should be consistent with that of the parent journal. (JAMA. 1994;272:1f)8-113) From the Baycrest Centre for Geriatric Care (Ors Rochon and Cheung and Mr Hayes) and Mount Sinai Hospital, Division of Geriatric Medicine, Department of Me,iic;rs (Drs Rochon and Cheung), and Department of Preventive Medicine and Biostatistics (Dr Roctan), University of Toronto (Ontario); Program for the Analy- s s of Clinical Strategies, Gerontology Division. Depart- men2 of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Mass, and the Brockton/West Roxbury (Mass) Veterans Affairs Medi- cal Center (Dr Gurwitz); and MetaWorks Inc, West lebanon. NH (Dr Chalmers). Presented in part at the Second International Con- gress on Peer Review in Biomedical Publ cation, Chi- cago. III, September 9, 1993. Reprint requests to Baycrest Centre for Geriatric Care, Un'rversity of Toronto, 3560 Bathurst St, North Yo: k. Ontar o, Canada M6A 2E1 (or Rochon). 108 JA,'v1A, July 13, 1994-Vol 272, No. 2 permission of the copyrfg CONCERN has been raised about sci- entific articles published in journal supplements." Supplements are fre- quently underwritten by pharmaceuti- cal manufacturer.q ar.d are less likely to undergo the peer revieti-nrocess.',1-5 De- spite these concerns, articles published in journal supplements are not easily or clearly differentiated from peer-re- viewed articles. Supplement articles are listed in databases, such as MEDLINE, housed in medical libraries, included in meta-anal;,ses, referenced in reviews, and distributed as reprints. They are UMI Article Clearinghouseh~owner.oFuurtherhreproductiontis prohibited, also routinely cited in promotional lit- erature distributed by pharmaceutical manufacturers. To evaluate the quality ofarticles pub- lished in journal supplements, we re- viewed all randomized control trials of drug therapies published between 1990 and 1992 in three journals that fre- quently publish supplements': the Am,erican Heart Journal, the Ameri- can Journal of Cardiology, and the American Journal of Medieine. Our ob- jective was to rigorously assess the qual- ity of scientific articles published in jour- nal supplements compared with those published in the parent journal,by means of a standard quality scoring system. MATERIALS AND METHODS All randomized control trials of drug therapies listed in MEDLINE from January 1990 and obtained in Novem- ber 1992 in the American Journal of Cardiology, the American Journal of Medicine, and the American Heart Jour- nal were identified by the following MEDLINE search strategy: CLINI- CAL TRIALS or PROSPECTIVE STUDIES orexplode RESEARCH DE- SIGN (includes DOUBLE-BLIND METHOD; RANDOM ALLOCATION) or (text root word) random (with any ending, eg, randomized, randomly) and HUMAN and ADULT. This searc * iden- tified a total of 504 articles. The abstracts for these 504 articles were reviewed by one of us (P.A.R.). All 318 articles that appeared to be of pos- sible relevance were retrieved and in• dependently reviewed in detail by threE of us (C.M.C., J.H.G., and P.A.R.). WE Journal Supplement Quality-Rochon et ai

I included all randomized control trials that reported the results of drug trials in adults, excluding drug washout stud- ies and secondary analyses of previously published studies. After a detailed re- view of the articles, an additional 76 were excluded for the following reasons: nonrandomized control trial (n=41), sec- ondary analysis (n=20), drug washout (n=7), non-drug related (n=4), pediatric study population (n=2), and unobtain- able (n=2). The remaining 242 articles comprised our study sample. Supplement status was determined by verifying whether articles in the study sample were published in a free- standing supplement issue or in the par- ent journal issue. Characteristics of Articles In the Sample Information was obtained on (1) the drug therapies evaluated, (2) source of acknowledged financial support, (3) the number of patients randomized in the trials, and (4) the frequency with which the reference list included articles pub- lished in journal supplements. Financial support was defined as one of the following four mutually exclusive categories: (1) pharmaceutical manufac- turer, (2) government or foundation, (3) both pharmaceutical manufacturer and government or foundation, and (4) no support acknowledged. Pharmaceutical manufacturer support included (1) ac- knowledged grant support from the pharmaceutical company identified ei- ther on the article or on the cover sheet of the supplement issue; (2) an employee of the manufacturer listed as an author, (3) drug supplied by the manufacturer, and (4) other types of pharmaceutical company support (eg, coordinating the study, conducting laboratory analyses, etc). Each article was assessed for the dis- crepancy between the number of sub- ;,ects randomized and the number sub- sequently analyzed in the study. Quality Scoring Three trained reviewers (C.M.C., J. H.G., and P.A.R.), "blinded" to supple- ment status, independently collected in- formation related to quality for each of the 242 articles. Formal assessment of the article quality was based on a modi- fied version of a quality scoring sys- tem" that evaluated the study protocol and the data analysis as outlined below. The questions selected from this quality scoring system to generate the quality score were those deemed to be most applicable to clinical drug trials. The quality scores as,signed were percent- ages (total score/tutal possible score) be- cause not all items were always appli- JAMA. July 13, 1994-Vol 272, No. 2 cable to the study under review. The total quality score assigned for each ar- ticle was the average of the scores of the respective reviewers. Control Appearance and/or Regimen (3, Same; 0, DifferentJUnstated).-Full credit can be attained only when the appearance of the control regimen is identical to the treatment. In trials in- volving parenteral medication, this should include vehic.: s and mode of ad- ministration as well as physical appear- ance of the drug. In any case, the dosage regimens must be identical for credit. This item is scored "not applica~le" when both pa,i?nts and observers are not blinded. Randomization Blinding: Was it Blind? (10, Yes; 5, Partial; 0, No/Un- known.- The crux of this issue lies in determining whether the investigators could discern which treatment was next in line. Blinding of rar:domization is en- sured by assignment via telephone com- munication, preferably by an individual not involved in the actual treatment, or with indistinguishable treatments ran- domly precoded by the pharmacy. Com- puter programs are available that re- quire insertion of the name and identi- fication number of the patient before treatment assignment is determined, re- corded, and revealed. Use of patient chart numbers, birth dates, dates of ad- mission, coin flips, or alternate (odd- even) allocation are all examples of un- blinded, quasi-randomization methods and should receive no credit. It is nec- essary to reaffirm the imperfections of open tables of random numbers or sealed envelopes (regardless of opacity). In gen- eral, if it is impossible to predict the next treatment in line, full credit shcald be given. If there is a small chance of the next treatment being predicted (eg, sealed envelopes), partial credit should be given. Mention of random numbers and coin flips could receive half credit if they are used by someone other than the person enrolling study participants. No credit for randomization blinding is given if pharmacy-supplied tablets are labeled A and B and any leak is possible. Patients Blinded (8, Complete; 4, Partial; 0, None).-The confusion on this issue lies in determining whether a trial is blinded merely by its statement, re- gardless of contradictory evidence. Tech- niques of blinding must be fully reported; there should be little chance of patients guessing which treatment they are get- ting. In the specific case where different dosages of the same drug are given with- out "dummies" to make up the differ- ence, no credit can be given. When side effects clearly reveal the patient's group, the study should be considered partially blinded. cit.gle-blindedness, unless oth- erwise specified, should be assumed to refer to patiezzts' blinding. A statement that the trial was double blinded with- out supporting evidence receives par- tial points. Observers Blinded to Treatment (8, Complete; 4, Partial; 0, None).-The same conditions for patients blinded to treatment also apply to observers. A study is classified by column as to whether or not all observers could have been blinded, some only, or none. Then these questions are answered as to whether complete, partial, or no blind- ing was actually employed. A stand-alone statement that a study was double blin d, without any details allowing judgment about the thoroughness, will receive only four points each for this question and the preceding one. Observers Blinded to Results (1G, Yes; 5, Partial; 0, No/Unknown).-Full credit is given solely when there is a separate data-monitoring person(s). Par- tial credit is given when less satisfac- tory attempts are made to blind the ob- servers as to the results. A score of "partial" is also given when a coordi- nating or data-monitoring committee is mentioned but there is no specific state- ment about individual centers or observ- ers being blinded to ongoing results. No credit is given when no effort is made to blind the observers as to the results. (Note: This item refers only to the on- going results of the study, not to any individual patient's outcome.) Previous Estimate of Numbers (3, Yes; 0, No/Unknown).-There should be evidence that the numbers of patients required were previous:y estimated. An excellent article lists in its "Methods" section the expected rate of control out- comes, the largest improvement of clini- cal interest that should not be missed, the chosen levels of error (a and (3), and the numbers anticipated. Testing Compliance (3, Adequate;1.5, Fair, 0, Inadequate).-This item is scored "not applicable" for in-hosp~~al studies except when a test of compli- ance (eg consulted nurses' notes) is spe- cifically mentioned, in which case full credit is given. In addition, a score of "not applicable" is given for trails that use parenteral medications or surgical procedures. Full credit is given for out- patient studies that use pill counts or other similar tests of compliance. A score of "fair" reflects a flawed attempt to assess compliance (eg, asking the pa- tients if they took their medication).111ea- surements of biologic equivalence will, in some cases, satisfy this item. Results of Randomization on Pre- treatment Variables and Inclusion in Analysis (3, Adequate; 1.5, Fair, 0, In- Joumal Supplement Quality-Rochon et at 109

adequate).-Analyses done to assess the baseline comparability of the study groups maygice the interpreter of the results of the trial reasons for being cau- tious or even suspicious. Thus, the issue here is whether the prognostic variables have been analyzed well and whether the author(s) considered their impact on the results. Either statistical analysis or a statement that the groups are too similar to require statistical analysis is necessary for a score of adequate. Major End Points (4, Test and P; 1, P, No Test;1, Test, No P; 0, Neither).- When significance is reported, it should be given so that the reader can make the actual calculations. Both the test statis- tic (where appplicable) and the observed P values should be stated. If the ob- served P level is present but not the test statistic, or the test statistic is given without the P level, the reader may have trouble verifying the statistical conclu- sions. The absence of both is unaccept- able. Post-0 Estimate (Nega.ive Trials Only)(3, Adequate; 1.5, Fair, 0, Inad- equate; 9, Not Applicable).-If the dif- ference between the compared treat- ments is not statistically significant, then a discussion of the type II error and its probability should be included in the "Re- sults" or the "Discussion" section of the trial. Actually estimating a posterior 0 fora clinically interesting difference gets a score of "adequate," as does a com- ment about the confidence intervals around the difference. Mentioning the problem and admitting a necessity for more patients ("problem needs further study to be sure a difference is not being missed") gets a score of "fair." Trials in which all differences measured are sta- tistically significant warrant a score of "not applicable." Confidence Limits (3, Yes; 1.5, Par- tial; 0, No).-Confidence limits should be provided for differences in the pro- portions, rates, or means used as trial end points. Reporting SD or SE of end points by group gains 1.5 points. Statistical Analyses (4, Excellent; 2, Good;1, Fair, 0, Poor).-This is an over- all assessment of the statistical methods that are used in the trial. When all ap- propriate analyses have been performed and clearly and precisely reported, a score of "excellent" should be given. If many, but not all, analyses have been performed, or not all results of these analyses are reported, ie, the test sta- tistic or the name of the test (where no test statistic exists) and the P value, a score of "good" is appropriate. A score of "fair" should be given when more se- rious deficiencies exist. (Note that the answer to this question is highly sub- jective.) Withdrawals After Randomization (3, None; 1.5, Listed With Reason; 0, Listed Without Reason No List/Un- known or>15%).-Withdrawals are de- fined as patients who were randomized but did not receive the allocated treat- ment or complete the specified obser- vation period, or who were removed from the study during the analysis period. If investigators specify the numbers of pa- tients who withdrew from the trial and the reasons why, this would represent an adequate "list " If the numbers are listed without the reasons, no points are given. If there are no withdrawals, and it is so stated in the article, this item would be scored as "none." "No list/un- known" means that no information is given. If more that 15~'0 of the random- ized patients withdraw from a study or are removed, the results are suspect and no points are given. Side Effects Discussion (3, Adequate; 1.5, Fair; 0, Inadequate).-To obtain a score of "adequate," side effects should be reported and discussed, and adequate statistical analyses should be carried ot;o. Comparison of percentages with a sta- tistical test of significance and the ob- served probability should be done if the sample size warrants it. The name of the test (and test statistic where applicable) and the P value must be recorded. A score of "fair" reflects an incomplete re- port or discussion. When no attempt is made to deal with side effects beyond simply listing them, or not listing them at all, a score of "inadequate" should be given. Statistical Analyses Differences in continuous variables be- tween articles published in a supplement issue and those published in the parent journal were assessed by t tests. Sepa- rate variance estimates were used where appropriate, and in such cases Satter- thwaite's approximation to the numbers of df was useV We used xg tests to assess differences in categorical vari- ables between articles published as supplements and those published in the parent journal. To assess the consistency of scoring between the three reviewers, we performed a principal component analysis that took into account all three quality scores simultaneously. Analyses were performed with SPSS 4.0 for the Macintosh9 and SAS System for Win- dows 3.10.10 RESULTS Characteristics of Articles Of the 242 articles in our sample, 133 (55.0%) were published in the Ameri- can Journal of Cardiology, 68 (28.1%) in the American Journal of dledicine, and 41 (16.9%) in the American Heart Journal. Overall, 67 (27.7%) of the ar- ticles were published in journal supple- ments, with the American Jourual of Medicine publishing the highest percent- age of articles in supplements (Fig 1). Most of the therapies evaluated in these articles were cardiovascular (177 [73.1%]) and antibiotic (24 [9.9%]). Oth- ers included gastrointestinal, oncologic, and respiratory. The number of patients randomized in the trials ranged from six to 8245, with a median of 50. Support was acknowledged from a pharmaceutical manufacturer in 142 (58.7%), from a government or founda- tion source in 16 (6.6%), and from both a pharmaceutical manufacturer and a government or foundation source in 25 (10.3%); no source of funding was ac- knowledged in 59 (24.4%) of the articles. The sources of financial support acknowl- edged differed between supplement ar- ticles and those published in the parent journal (x3=50.1, df=3, P<.0001) (Fig 2). No supplement article received funding solely from a government or foundation source. All of the supplement articles in our sample were published in a supple- ment that was sponsored by a pharma- ceutical manufacturer. In 63 (94.0;"0) of the 67 supplement articles in our sample, only support from a pharmaceutical manufacturer was acknowledged. In the remaining four supplement articles, sup- port was also acknowledged from a gov- ernment or foundation source. In 39 (58.2%) of the 67 articles published in journal supplements, the acknowledg- ment of pharmaceutical manufacturer support was referred to only in the in- troduction to the supplement issue, with no mention of such support in the article itself. A systematic examination of the ref- erences listed in each article in the study sample indicated that at least one ar- ,,icle published in a journal supplement was refererced in 165 (68.2%) of the ar- ticles reviewed. Articles published in supplements were more frequently ref- erenced in supplement articles in the sample than in nonsupplement articles. Supplement articles comprised 17.2%±16.1% (mean±SD) of the refer- ences listed in articles published in jour- nal supplements compared with 8.1%±9.8% of those referenced in non- supplement articles (t[85]=4.4, P<.001). No statistically significant difference was found between the number of sub- jects randomized in trials publish2d in either supplement issues or the parent jourra.l. On average, articles published in journal supplements randomized 280.0t1001.5 patients (mean±SD) in their trials compared with 148.0±664.2 in the parent journal (t[89]=1.0, P=.32). Articles 110 JAMA, July 13, 1q-34-Vol 272, No. 2 Journal Supplement Quality-Rochon et al

Journal Fig 1.-Number of articses among those included in the study sarnple (r>=24. that were published in jour- nal su~s (dark bars) cornpared wiih the num- ber that were published 'n the parent jaunal (I'ight bars) for each of the three ioumals evafuated. published in journal supplements in- cluded in their final analysis a smaller proportion of patients initially random- ized. Supplement articles included 85.0%±15.8% of patients initially ran- domized, compared with 91.3%±11.0% of patients included in thi final analysis by articles published in the parent jour- nal (t[75]=2.8, P=.007). Overall Quality Score Article quality scores ranged from 42% to 87.5% of a possible 100%, with a mean score of 372%t13.1%. On aver- age, articles published in the Amerzcan Journal of Cardiology received a score of 38.1qc±13.9%, those published in the Amzrican Journal of hiedicine received a score of 35:I~1c.±13.0%, and articles pub- lished in tl:e American Heart Journal received a score of 37.4%±10.7%. Qual- ity scores were lower in articles pub- lished in journal supplements than in those published in the parent journal (t[240]=2.61, P=.01). The mean quality score for articles published in journal supplements was 33.6%±12.8% com- pared with a score of 38.5%±13.1% for articles published in the parent journal. A principal-component analysis sug- gested that the quality scores assigned by the three reviewers were consistent. The first principal component explained 76% of the total variation between the quality scores obtained by the review- ers, and these scores contributed evenly to this first principal component. Summary of tndividual ftems Used to Measure Quality Articles published in journal supple- ments and those published in the parent journal were compared on each of the 14 items used to calculate the overall qual- ity score. The average scores obtained for each of the items used to assess the quality of the articles are summarized in Pharma- Govemment/ Both No ceutical foundat'ion Support Support Type I Fig 2.-Percentage of articles acknowledging each type of support according to whether they were pub- tished in a journal supplement (dark bars) or thQ par- ent journal (light bars). Pharmaceutical support in- dudes those articles published in a journal supplement sponsored by a pharmaceutical manu- facturerwith no supportacknowiedged on the article. the Table. In 12 of the 14 items that were used to evaluate quality, supple- ment zrticles received a lower score than articles published in the parent journal. For two of the quality score items, one that assessed whether the randomiza- tion process was blind and another that evaluated whether the description of the drug side effect profile was adequate, articles published in supplements re- ceived higher scores, although these dif- ferences were not statistically signifi- cant. Six of the 14 quality scoring items that we used were found to differentiate clearly the quality of articles published in supplements compared with the par- ent journal. Articles published in jour- nal supplements were less likely to pro- vide information on the testing of com- pliance (t[240]=4.5, P<.001), provided fewer details on the results of preran- domization (t[96]=3.0, P=.003), provided less information about the use of major end points (t[94]=2.7, P=.008), were less likely to include a post-(3 estimate when applicable (t[78]=2.3, P=.022), were judged to be poorer in overall presentation of the statistical analysis (t[240]=2.7, P=.007), and dealt with the withdrawal of subjects from the trials in a less satisfactory man- ner (t[240]=3.75, P<,001). COMMENT Quality Issues Our data suggest that scientific ar- ticles on randomized control trials ofdrug therapies published in journal supple- ments are generally of inferior quality compared with articles published in the parent journal. To our knowledge, this is among the first stu<lies" to evaluate formally the quality of supplement ar- tictes and to make such a compatison. Quality was e%aluated bysever•al mea- sures. Using a standardized qual ity scor- ing system, we objectively score<I the article quality. The quality scot-ingsys- tem method has been described in its original forme and adapted forms-,•'=" and has been -ipplied i.a its original or adapted form to assess the quality of articles included in meta-analyses of clinical topics'1=' and for other research applications.'x~`,=7 Overall, the quality scoreo chat were ot-ained for both the parent and the supplement journals in our sample were low compared with those in previous reports. Reitman et aln _.,und that quality scores improved over time. The mean (±SD) quality score for articles published in the 1980s was 43%±18% in a study of more than 400 articles describing th.•: results of ran- domized control trials. In examining the individual items used to determine the overall quality score, our results showed generally lower scores for articles published in supple- ments than for articles published in the parent journal. In the assessment of the study protocol, articles publishe-l in jour- nal supplernents provided less informa- tion to ensure that adequate steps had been taken to assess compliance with the study medication. Supplement ar- ticles provided less satisfactory details about the baseline characteristics of the groups of patients being studied before being randomized to their treatment. In the data analysis, articles published in journal supplements were less likely to provid- complete information on major end points, and, when a negative trial was reported, a post-(3 estimate or dis- cussion was less often provided. When the reviewers were asked about their global rating of the statistical analysis, supplement articles received, on aver- age, lower scores. Finally, data on pa- tients who withdrew from the study were less likely to be handled in a sat- isfactory manner. Another measure used to assess ar- ticle quality was an evaluation of the discrepancy between the number of pa- tients randomized and subsequently ana- lyzed in each trial. It is generally ac- cepted, according to the "intention to treat" principle, that all patients initially entering a study should be included in the final analyses. Deviations from this policy lead to concerns about the quality of the study results.6" We found that, on average, there is a greater discrep- ancy between the number of patients randomized and the number of patients analyzed in articles published in journal supplements than in parent journal ar- ticles. Because withdrawals could influ- ence the final composition of the various JAtvV~ Ju`y 13, 19?4-Vd 272. No. 2 Journal Supplement Quahty-Rochon et al 111

Scores of Individual items Used in the Calcutation of the Overall Quality Score Quality Measure' Supplement Consinsus Score (ltean:SO) Parent Journal Consensus Scores (MeansSD) tTest, P Study protocol Control appearancet 1.3x1.2 1.6s1.2 t(240)=1.8, .07 Blinding Randomization 1 Q:3.2 1.5z2.9 t(240)=0.8..41 Patients 3.9z2.5 4.4:.2.5 t(240)=1.4,.16 Observer 4.0•.2.4 4.0±2.6 t{240)-0.07, .95 Observer to resultst 0.5±1.5 0.89±2.0 tj158)=1.6, .11 Previous estimate number 0.33x0.9 0.35±0.9 t(240)=0,19, .85 Testing compliance 0,8m12 1.6±1.3 t(240)=4.5, <.001 Results of prerandomizationt 2.0s0.9 2.4±0.7 t(96)=3.0, .003 Data anatysis Major end points 2.5z0.8 2.8±0.6 t(94)=2.7, .008 Post•P estimate (negative) 0.1 x0.2 0.3±0.7 t(78)=2.3, .02 Confidpnce limits 0.5±0.7 0.7±0.8 t(240)=1.6, .1: 3 Statistical analyses 1.8±0.7 2.0±0.6 t(240)=2.7, .007 Withdrawalst 1.1±0.6 1.5±0.7 t(240)=3.7,<.001 Side effects discussion 1.910.9 1.7±1.0 t(24o)=1.41, .158 'See the'Materiats and Mettrods'seCion for descriptron of possible points available for each item used to assess quality and definitions used. tWeightings used on these mea.sures dMered slightly from those described in the original quality sroring system. treatment groups and undermine the benefit of randomization,r, this measure was believed to be important. Consistent with a previous reports our results suggest that articles pub- lished in jotlrnal supplements frequently do not acknowledge association with a pharmaceutical manufacturer in the ar- ticle itself. Previous work21 suggested that there may be a relationship between manufacturer sponsorship and findings of drug efficacy and toxicity studies. In- formation regarding sponsorship is therefore potentially relevant in the com- prehensive assessment of the literature regarding specific therapeutic agents. Because individual articles are fre- quently distributed as reprints, photo- copied, or read outside of the context of the entire journal issue, it is important that the sources of financial support be acknowledged in each article. Informa- tion provided only on the cover sheet or in the introduction to the supplement issue may not always be available to those reading selected reprints. Why Supplements Are of Lesser Quality One reason that articles published as supplements are of less quality than the parent journal may be a difference in the peer review process. A report of published symposiums dealing with therapy for angina found that few ar- ticles published in this form underwent the traditional peer review process s While supplement articles may in fact undergo peer review, this review pro- cess is not consistently stated in the three journals we reviewed.' Some jour- nals have already taken steps to review their policy on journal supplements with the goal to improve their quality.~ This is an important step given our findings that supplement articles are frequently referenced in articles, as well as a re- port that they are frequently referenced in drug advertisements 30 Because of the fairly restrictive cri- teria used in the selection of articles inc.:uded in the study sample, the qual- ity score results that we obtained are not necessarily representative of the overall quality of the individual jour- nals. We included only randomized con- trol trials that compared two drug thera- pies in our sample and excluded those that did not strictly fit with this defini- tion. For example, if a trial appeared to be a randomized control trial but did not specifically use the word "randomized" in the text, then it was excluded. Epi- demiologic studies were also excluded from evaluation. What Does This Mean in Clinical Practice? Clinicians reading the medical litera- ture should be aware that jourrlal supple- ments are not always peer reviewed and should use this information to guide their interpretation of the results presented in these articles and to read supplement articles in a more critical manner. Supplement articles must be clearly dis- tinguished from those published in regu- lar journal issues. We found that no con- sistent method was used to identify supplement articles either in the MED- LINE search or in the actual article. This lack of uniformity would make it difficult for clinicians to differentiate supplement articles from nonsupplement articles. Since supplements carry the name of an established journal, general 112 JAMA, July 13. 193- 4Voi 272. No. 2 readers may incorrectly assume that ar- ticles published in supplements undergo the same editorial and peer review pro- cess as do articles published in the par- ent journal. In our sample, all of the supplement articles were published in pharmaceu- tical manufacturer-sponsored journal supplements. The funding source may influence whether or not a particular study is published and thus lead to a form of publication bia0 R'Ianufacturer- sponsored publications may tend to fa- vor the manufacturer-sponsored drug.'-'x Including non-peer-reviewed and manufacturer-sponsored supple- ment articles in meta-analyses may bias the results. The findings of carefully con- structed meta-analyses are increasingly being relied on to guide medical prac- tice. Given the lesser quality of supple- ment articles, the routine inclusion of supplement articles in meta-analyses needs to be reconsidered. Under some circumstances, journal supplements can play an important edu- cational role for the medical community. A special issue that focuses on a single topic may provide a handy and usePil reference for clinicians. One example is the "Third ACCP Consensus Confer- ence on Antithrombotic Therapy" pub- lished in October 1992 as a supplement issue to Chest. However, this type of supplement issue did not cont3in origi- nal articles describing randomized con- trol trials of drug therapies and there- fore is quite different from our study sample. Conclusion We demonstrated, by using a series of measures of quality, that articles pub- lished as journal supplements are of lesser quality than articles published in the parent journal. Given our findings and the general concern about journal supplements, the role of these publica- tions in the medical literature should be questioned and reevaluated. We recommend two major strategies to deal with journal supplements in the medical literature. The first is that they should be clearly and consistently iden- tified as being different from traditional peer-reviewed articles in databases, such as MEDLINE, as well as in individual articles. We also strongly suggest that all journal supplements be subjected to the same rigorous peer-review process employed by the parent journal. This study was supported in part by grant R01 HS-05936 and by Clinical Investigator Award KOS AG00510 from the National Institute on Aging (Dr Gurwftz). Mr Hayes was supported by the :1iati and Rosalyn Gordon Summer Scholarship Fund through Baycrest Centre for Geria.:ic Care. Journal Supplement QuaGty-Rochon et al 2448252385

t We are indebted to Geoffrey Litner and Jason Litner for data management, Bruce Iiupelnick for his assistance with various aspects of the project, Jialcotm A. Binns, statistician, for reN•ie«ir,g the manuscript, and.Michael Gordon, \iD, FRCPC, for his editorial comments and ongoing support. References 1. Bero LA, Galbraith A, Rennie D. The pubiic:t- tion of sponsor-d symposiums in medical journals. N Engl J .lled. 1992;32'i:11355-11-10. ". Soffer A. Hazards in publication of procE~eciing-s. Arch Intent Med. 19S?;1-t2~207-1-2076. 3. Kessler DA. Drug promotion and scientific ex- change: the role of the clinical im•estigator. \'„ngl J .llcd• 1991;325•201 -203. 4. Finucane T. Drug company-sponsored sympo- si:u pros and cons. Am J 31ed. 19,47;S'.i:811.-81'L S. Jlassie BM, Rothenberg D. Publication of spon- sored symposiums in medical journals. N Engl J .t f ed. 1993;328:1196-1197. 6. Chalmers TC, Smith HJ, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Controlled Clin Trials. 1981;231-19. 7. Antczak AA, Tang J, Chalmers TC. Quality as- se«ment of tandomized control trials in dental re- search, l: methods.JPeriodont Res.1986;21305:31d. & Snedecor G1V, Cochran WG. Statistical Meth- ods. Sth ed. Ames: Iowa State University Press; 1989;1. 9. SPSS Inc. SPSS Inc Base System User's Guide. Chicago, I1L• SPSS Inc;1990:520• 10. SAS Institute Inc. SAS/STAT User's Cuide, Release 6.03 Edition. Cary, NC: SAS Institute Inc; 198,S:1:1029. 11. Gozsche PC. Methodology and overt and hid- den bias in reports of 196 double-blind tiials of nonsteroidal antiinflammatory drugs in rheuma- toid arhritis. Controlled Clin Trials. 19ti9;1(1:31- 56. 12. Chalmers l, Adarns Ji, Dickelsitt K, et al. A cuhort study of summary repott_ of controlled tri- als. J:1.1f.-L 191+0;:.'Q3:1-t01-1-305. 13. Detsky AS, Naylor CD, O'Rourke K, 1lcGeer AJ, L:Abbe KA. Incorlwr3ting variations in the quality of individual randomized trials into meta- analysis. J Clin Epidenriul. 1tIt12;d5•':.9o-ZG;i. Ia. L'Abbe KA, Detsky AS, O'Rourke K. Meta- analy,is in clinical research. Ann Intem .lfcd. 1937; 10 7 --'12-i-2:?:3• 15. Lam W. Sacks HS, Sze PC, Chalmer: TC.1Ieta- anaiy:is of randomized controlled trials of nicotine che%cing gum. Lancet. 1987•?•27-30. 16. Goldstein P, Berrier J, Rosen S, Sacks HS, Chalmers TC. A meta-analysis of randomized con- trol trials ofprogestational agents in pregnancy. Br J O&tet Gytreeol. 1989;9G•26.i-274. 17. Sze PC, Reitman D, Pincus.M, Sacks HS,Chalm- ers TC. Antiplatelet agents in the secondary pre- vention of stroke: meta-anah•sis of the randomized control trials. Stroke. 1988;19:-136-1-12 18. Sacks HS, Ancona-Berk A, Berrier J, Nagal- ingarn R, Chalmers TC. Dipyridamole in the treat- ment ofangina pectoris: a meta-analysis. Cliu Plrar- ntacol Ther. 1938;4o:G10-610. 19. Liffierati A, Himel HN, Chalmers TC. A quality asse__~ment ofrandomized control trials ofprimary treatment of breast cancer. J Cliia Oucoi. 1986;9: 9-t2-951. 20. Baum i<IL, Anish DS, Chalmers TC, Sacks HS, Smith HJ, Fagerstrom RHI. A survey of clinical trials of antibiotic prophylaxis in colon surgery: evidence against further use of no-treatment con- trols. N Engl J .S1ed. 1981;30b:795-799• 21. Rochon PA, Gurwitz JH, Simms RW, et al. A study of manufacturer supported trials of nonste- roidal anti-?nflammatory drugs in the treatment of arthritis. Arch Intern .lfcd. 199-1;1o-1:157-1G3. 22. Naylor CD, 0'Rourke K, Det: ky AS, HakerJl'. Parenter.nl nutrition with brmnched-chain amino ar- ids in hepatic encephalopathv: a meta-analyci<. Gas- t ro e u t erologtt. 1959;97:1033-10-12. 23. R'ong DKH, Choung AM, O'Rourke K, Naylor CD, Detsky AS, Heathcote J. Effect of alpha-in- terferon treatment in patients with hepatitis B e antigen-positive chronic hepatitis B: a mc•ta-analy- sis. A nt Ir tent .lfcd. 1993;119:312-3tr.i. 2-1. Detsky AS, Baker JP, O'Rourke K, Goel V. Perioperative parenteral nutrition: a meta-analv- sis. Ann Intem •1fed. 1987;107:195-203. 25. Antczak AA, Tang J, Chalmers TC. Quality assessment of randomized control trials in dental research, 1L• results: periodontal research. J Pcri- odo 1t Res. 19S6;21~315-321. 26. Emerson JD, Burdick E, Hoaglin DC, Mos- teller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Caaa- trolled Cliit Trials. 1990;11:339-352. 27. Reitman D, Sacks HS, Chalmers TC. Techniral quality assessment of randomized control trials (RCTs). Controlled Cliu Trials. 1937;42a`2. 28. Goz:;che PC. Bias in double-blind trials. Dau Afed Bull. 1990;37:329-33G. 29. Cavalli F. Our policy regarding supplements of sponsored symposia. Ann Oncol. 1993;-1:99. 30. Finucane TE. Publication of sponsore 3 sympo- siums in medical journals. N Eugl J,ltcd. 1993,328: 1197. 31. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAS1A. 1990; 263:1385-13S9. 32. Davidson RA. Source of funding and outcome of clinical trials. J Geu Intem Med. 1956;1:1ai-158. JAP1;4, July 13, 1994--Vol 272, No. 2 Journal Supplement Quality--Rocnon et al 113