Tobacco Documents Online

THE ROLE OF EXPERT JUDGMENT IN RISK ANALYSIS 1. USE OF EXPERT JUDGMENT IN RISK ANALYSIS c. a. There is no Question That Expert Judgment is Needed; The Only Question is Whether to Use It Explicitly b. It Requires No More Use of "Art" Than Any Other Field Expert Judgment May Not Provide the Solution; However, It Will Often Show You Where and How to Seek It d. Properly Applied, Expert Judgment Can Assist In: Identifying Decision Alternatives; Validating Analytical Models; Assessing Influencing Parameters Note: False Assumptions Relative to A Parameter, Such as Its Degree of Independence, Can Significantly Affect The Final Results e. Expert Judgment Should Reflect Current Knowledge and Quantify Attendant Uncertainties; While the Additional Information Provided Will Help Reduce Ignorance, It Will Not Necessarily Reduce Uncertainty f. Although Expensive, Its Use "Up Front" Can be Very Cost-Effective in Terms of the Avoidance of Delays and Associated Costs at Later Stages 1

2. HOW MUCH DATA ARE SUFFICIENT a. Expert Judgment is not a Substitute for Data When It can be Readily Obtained b. Expert Judgment Should Follow Only After an Agreed Upon Level of Effort to Perform Additional Scientific Studies, and to Analyze and Evaluate the Resulting Data, Has Been Met 3. OBSERVATIONS ON THE USE OF EXPERT JUDGMENT a. It Is Well Established as a Tool in Risk Analysis b. Its Use is Mandatory Where Data are Inadequate; However, It Should Not Be Used as a Replacement for "Hard" Data c. Proper Application of Expert Judgment Requires Care, Planning, Documentation, and Resources. Protocols for Elicitation Should Be Systematic, Visible, and Easily Understood d. It Should Not Be Conducted as Simply a Poll of Experts; It Should Include Explicit Articulation of the Principles Reasoning, and Data on Which the Judgments are Based e. Its Formal Solicitation Can and Should Clearly Explicate the Uncertainties for the Decisionmakers; At the Same Time, It Should be Recognized That Such Judgments Represent Only "Snapshots" in Time of Prevelant Opinion; They Are Not a Means for "Revealing" Truths f. The Methodology Used Can Significantly Influence the Result of the Process; Key Factors Include: Identification and Selection of Issues; Identification, Selection, and Use of Experts and Solicitors; The Biases of the Expert and Elicitors; The Methods for Aggregating the Results g. An Important Issue is Estimating the Magnitude of the Uncertainty, Especially Where Data Are Limited *,h. In All Cases, the Selected Approach (or Conclusion) Should be Justified; The Most Conservative Approach Should Not Necessarily Be Chosen Simply Because the Data Are Inadequate f 2

4. OBSERVATIONS FROM PAST USE OF EXPERT PANELS a. b. c. d. e. There Is Always a Lack of Sufficient Time for Adequate Iteration With the Experts on Their Results The Definition of Issues is Always Critical; This Involves a Balance Between the Freedom of the Expert and Overdefining and Overconstructing the Issues Anonymity Should be Avoided; Knowledge of the Sources of Specific Opinions Improves The Defensibility of the Results Each Expert Should be Encouraged to Analyze the Questions Indepth Prior to the Panel Being Convened A Full Range of Weighting Strategies Should be Evaluated in Aggregating the Results 5. QUESTIONS TO BE CONSIDERED IN THE USE OF EXPERT JUDGMENT a. How Should the Experts Be Identified and Selected? b, What Are The Preferred Methods for Using Formal Expert Judgment? How are the Mechanics of the Elicitations to be Carried Out? c. How Are the Pertinent Issues and Questions to be Selected? d. What Are The Pitfalls Associated With the Use of Expert Judgment? For Example, How Can Biases (Institutional and Motivational) Be Evaluated and Reduced? -r e. Is Dependency Among Experts (Mutual Influence) Desirable? If so, What Are the Appropriate Ways to Encourage It? If not, How Can It Be Minimized or Eliminated? f. Are There Appropriate Protocols for Weighting the Judgments of Experts and for Weighting Alternative Models and Their Results? If so, What Are the Criteria for Assigning Weighting Factors? g• How Should Judgments Be Aggregated? What Are The Criteria for Choosing an Aggregation Method? h. What is the Appropriate Level of Documentation for the Elicitation? i. What Steps Should be Taken to Control the Potential Influence of Normative Experts on the Outcome? 3

~ 6. QUESTIONS ON ELICITATIONS FROM SINGLE EXPERTS a. Lack of Benefit of Interaction with Peers b. Assumptions Underlying Judgments May Not Be Articulated c. Expert May be More Subject to Motivational and Cognitive Biases d. Quantification of Uncertainty Will Generally be Less e. Results May be Less Defensible; A Single Expert (Versus a Panel) May Have Less Credibility With Outside Groups 7. SELECTION AND ORGANIZATION OF PANEL MEMBERS . They Can Be Selected on the Basis of: Their Publications; Their Relevant Experience; Their Experimental Knowledge;, Their Degree of Cooperation; and The Input From Outside Groups (These Can Be Solicited by Mail or Through an RFP) b. Should They Be Organized Horizontally of Vertically? Vertical Interaction is Helpful; Horizontal May Make Interfaces Difficult (They Must Interact to Exchange Information at Each "Nodal" Point) 8. TRAINING OF PANEL MEMBERS a. Is Required for Essentially All Panel Members b. Should Be Designed to Sensitize the Experts on How Their Information Will be Used and How It Could Be Abused c. Should Recognize That Motivational Biases (Hidden Agenda) Are Less Responsive Than Cognitive Biases 4

9. PITFALLS AND ADVANTAGES OF THE PANEL APPROACH a. Dominant Personalities May Heavily Influence the Results: Avoid by Having Each Member Formally Document His/Her Opinion and Rationale; Experts Can Also Be Required to Construct Probability Distributions for Their Positions The Greatest Sources of Bias Are Unstated Assumptions b. To Avoid Criticism, It is Often Necessary to Use "Experts" With Relatively Shallow-Knowledge of the Subject: This May Require Training That Can Introduce Biases c. To Overcome Biases, Panels Can be Made Up of Experts With Diverse Perspectives: While This Can Remove Some of the Biases, It May "Fix" the Results 10. BENEFITS OF THE PANEL APPROACH a. All Members Can Be Provided the Same Information b. Approach or Thoughts of Each Will Be Transparent for Other Members to Review c. Positions Will Change As a Result of New Information Provided and Group Dynamics 5

11. STEPS IN THE PANEL PROCESS a. The Issues Must be Identified and Selected b. The Experts Must be Identified and Selected c. The Issues Must be Discussed and Refined d. The Experts Must be Trained for Elicitation e. The Information Must be Elicited f. The Results Must be Analyzed and Aggregated, and Disagreements Resolved h. The Results Must Be Documented and Communicated to the Needed Parties 12. GUIDANCE FOR GOOD PANEL OPERATIONS a. Ambiguity Can Be Minimized by Making the Questions Specific b. If Final Conclusions Are to be Aggregated, They Must Be in the Same Format c. If an Expert Has Difficulty Quantifying an Answer, Decompose the Question for Which He/She Can Construct Answers 6

13. GUIDANCE ON PANEL METHODOLOGY a. The Methodology Must Allow Different Experts to Share Each Others' Information so that Each Expert Can Base His/Her Ultimate Judgment on a Common Set of Information b. The Methodology Must Provide for Significant Inter- actions Among the Experts, So That They Can Share Different Perspectives and Approaches c. The Methodology Must Allow The Experts to Define or Redefine the Questions Being Asked, So That They Can Answer the Questions in Their Terms d. The Methodology Must Lead to the Conduct of the Proceedings in a Manner so as to Motivate the Experts e. The Methodology Must Provide Structured Ways in Which "Biases" (Different Perspectives) Can Be Identified, Brought to the Surface, and Subjected to Scrutiny f. The Methodology Must Allow Sufficient Time so That the Experts Can Perform Their Own Research and Analysis; Usually This Calls for Two Iterations in the Elicitation g• The Methodology Must Recognize That the Questions Being Addressed Do Not Have One "Correct" Answer, or Even "One" Correct Approach h. The Methodology Must Provide Full Documentation of What Happened, What Factors Underlie the Judgments of Each Expert, and Why; This Means That Each Expert Must be Comfortable With the Way in Which His/Her Judgment is Documented ~ 7

14. AGGREGATION OF CONCLUSIONS OF INDIVIDUAL PANEL MEMBERS a. The Opinions of "Mainstream" Experts Can be Requested to Evaluate the Expertise of Individual Panel Members b. If a Rationale Exists to Reject the Opinion of One or Any Be More Panel Members, This Can Be Done; However, Quick Ploy to Remove Dissenting Opinion Must Avoided c. Oftentimes It is Important to Retain the Component Contributions of Individual Panel Members, Even if Only One Aggregated Result is Needed in the Particular Application d. Forced Consensus Compromises the Defensibility of the Results 15. DOCUMENTATION OF THE ELICITATION a. It Should Include Unambiguous Specification of - the Events for Which Probabilities Were Assessed b. The Assessed Probabilities for Those Events Should be Clearly Expressed c. The Data, Reasoning, Models, and Calculations Used in the Assessments Should be Documented, As Well As How and Why They Were Used d. The Documentation Should Also Include a Rational Basis for Eliminating and/or Not Documenting Any Information That Was Considered to be "Obvious" 8

16. CONCLUSIONS a. Expert Judgment is a Necessary Part of Risk Analysis b. Expert Judgment Must be Applied in Both an Explicit and Implicit Manner, With a Reasonable Division Between the Two c. The Final Product Should Include Detailed Information on the Principles, Rationale, and Data On Which the Conclusions are Based d. Where There are Major Weaknesses in the Experimental Data, the "Strengths" of Expert Judgment Can Be Extremely Helpful e. Explicit Expert Judgment is Primarily Designed to Aid in Decision Making; It is Not There,Solely to Provide a "Warm Fuzzy Feeling" 17. OTHER CONCLUSIONS a. Where the Reasoning Process Can Be Understood (and Accepted by Other Reasonable People), A Decision Based on Expert Judgment Should Stand Up in the Courts b. Although the Process of Solicitation of the Experts is Important, the Legal System Will Be More Interested in the Facts and Rationale for the Decision c. The Primary Purpose of Expert Judgment is to Provide a Foundation on Which the Decisionmakers Can Act d. Expert Judgment is Always Subject to Change and Should be Interpreted Accordingly Reference: Bonano, E. J., Hora, S. C., Keeney, R. L., and von Winterfeldt, D., "Elicitation and Use of Expert Judgment in Performance Assessment for High-Level Radioactive Waste Repositories," Report NUREG/CR- 5411, U.S. Nuclear Regulatory Commission, Washington, DC (May, 1990). 9