Statistics and Policy Decisions: Issues in Statistical Analyses
DOI:
https://doi.org/10.6000/1929-6029.2015.04.02.1Keywords:
Policy decisions, Statistical Significance, Practical or Policy Significance, Methodological Errors, Lead/IQ Association.Abstract
When national policy decisions are to be guided by the results of statistical analyses, it is important, to avoid being misled to look beyond the authors’ conclusions and first to assess the study design, measurement and analytic methods, in order to decide whether a study’s conclusions rest on a solid foundation. In particular, observational studies must be carefully and critically evaluated. Using a study widely cited concerning the effects of low-level lead exposure and IQ, we illustrate several methodological errors, long known but often ignored. The goal is not to settle the controversies about the effect of lead on IQ, nor to disparage observational studies, for they are the foundation of all studies done to guide policy, but to encourage additional care in the use of such studies to address policy questions.
References
Crump KS, Van Landingham C, Bowers TS, Dexter C, Chyandalia JK. A statistical reevaluation of the data used in the Lanphear et al. (2005) pooled-analysis that related low levels of blood lead to intellectual deficits in children. Critical Reviews in Toxicology 2013; 45(9): 785-99. http://dx.doi.org/10.3109/10408444.2013.832726 DOI: https://doi.org/10.3109/10408444.2013.832726
Pirkle JL, Brody DJ, Gunter EW, Kramer RA, Paschal DC, Flegal KM, et al. The Decline in Blood Lead Levels in the United States: The National Health and Nutrition Examinations Surveys (NHANES) Journal of the American Medical Association 1994; 272(4): 284-91. http://dx.doi.org/10.1001/jama.1994.03520040046039 DOI: https://doi.org/10.1001/jama.1994.03520040046039
Lanphear BP, Hornung R, Khoury J, Yolton K, Baghurst P, Bellinger DC, et al. Low-Level Environmental Lead Exposure and Chidlren's Intellectual Function: An International Pooled Analysis. Environmental Health Perspectives 2005; 113(7): 894-9. http://dx.doi.org/10.1289/ehp.7688 DOI: https://doi.org/10.1289/ehp.7688
Schulz KF, Altman DG, Moher D, Consort_Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. British Medical Journal 2010; 340: 698-702. http://dx.doi.org/10.1136/bmj.c332 DOI: https://doi.org/10.1016/j.ijsu.2010.09.006
Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJW. Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement. Journal of the American Medical Association 2006; 295(10): 1152-60. http://dx.doi.org/10.1001/jama.295.10.1152 DOI: https://doi.org/10.1001/jama.295.10.1152
Altman DG, Schulz KF, Hoher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine 2001; 134(8): 663-94. http://dx.doi.org/10.7326/0003-4819-134-8-200104170-00012 DOI: https://doi.org/10.7326/0003-4819-134-8-200104170-00012
Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. Journal of the American Medical Association 1999; 276: 637-9. http://dx.doi.org/10.1001/jama.1996.03540080059030 DOI: https://doi.org/10.1001/jama.1996.03540080059030
Rennie D. How to report randomized controlled trials: The CONSORT Statement. Journal of the American Medical Association 1996; 276(8): 649. http://dx.doi.org/10.1001/jama.1996.03540080071033 DOI: https://doi.org/10.1001/jama.1996.03540080071033
Moses LE, Mosteller F. Experimentation: Just do it! 1995.
Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association 2005; 294(2): 218-28. http://dx.doi.org/10.1001/jama.294.2.218 DOI: https://doi.org/10.1001/jama.294.2.218
Keyfitz N, editor. Why forecasts fail and policies are often frustrated. Oxford: Clarendon Press; 1997.
Lanphear BP, Hornung R, Khoury J, Dietrich KN, D.A. C-S, Canfiled RL. The Conundrum of unmeasured confounding: "can some of the detrimental neurodevelopmental effects attributed to lead be due to pesticides? by Brian Gulson". Sci Total Environ 2008; 396(2-3): 196-200. http://dx.doi.org/10.1016/j.scitotenv.2008.01.039 DOI: https://doi.org/10.1016/j.scitotenv.2008.01.039
Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to Terms with the Terms of Risk. Archives of General Psychiatry 1997; 54: 337-43. http://dx.doi.org/10.1001/archpsyc.1997.01830160065009 DOI: https://doi.org/10.1001/archpsyc.1997.01830160065009
Kraemer HC, Stice E, Kazdin A, Kupfer D. How do risk factors work together to produce an outcome? Mediators, Moderators, Independent, Overlapping and Proxy Risk Factors. The American Journal of Psychiatry 2001; 158: 848-56. http://dx.doi.org/10.1176/appi.ajp.158.6.848 DOI: https://doi.org/10.1176/appi.ajp.158.6.848
Smith MA, Grant LD, Sors AI. Lead Exposure and Child Development Dordrecht/Boston/London Kluwer Academic Publishers 1989. DOI: https://doi.org/10.1007/978-94-009-0847-5
Shannon M, Graef JW. Lead Intoxication in Children with Pervasive Developmnetal Disorders Clinical Toxicology 1996; 34(2): 177-81. DOI: https://doi.org/10.3109/15563659609013767
Pocock SJ, Smith M, Baghurst P. Environmental lead and children's intelligence: a systematic review of the epidemiological evidence. . British Journal of Medicine 1994; 309: 11898-1197. http://dx.doi.org/10.1136/bmj.309.6963.1189 DOI: https://doi.org/10.1136/bmj.309.6963.1189
Thacker SB, Hoffman DA, Smith J, Steinberg K, Zack M. Effect of Low-level Body Burdens of Lead on the Mental Development of children: Limitations of Meta-analysis in a Review of Longitudinal Data. Archives of Environmental Health 1992; 47(5): 336-46. http://dx.doi.org/10.1080/00039896.1992.9938372 DOI: https://doi.org/10.1080/00039896.1992.9938372
Wortman PM. Judging Research Quality. In: Cooper H, Hedges LV, editors. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994. p. 97-109.
Hall JA, Tickle-Degnen L, Rosenthal R, Mosteller F. Hypotheses and Problems in Research Synthesis. In: Cooper H, Hedges LV, editors. The Handbook of Research Synthesis. New York: Russel Sage Foundation; 1994. p. 17-28.
Simpson EH. The Interpretation of Interaction in Contingency Tables. Journal Of The Royal Statistical Society, SerB 1951; 13: 238-41. DOI: https://doi.org/10.1111/j.2517-6161.1951.tb00088.x
Samuels ML. Simpson's Paradox and Related Phenomena. Journal of the American Statistical Association 1951; 88: 81-8. DOI: https://doi.org/10.1080/01621459.1993.10594297
Kraemer HC. Individual and Ecological Correlation in a General Context: Investigation of Testosterone and Orgasmic Frequency in the Human Male. Behavioral Science 1978; 23: 67-72. http://dx.doi.org/10.1002/bs.3830230203 DOI: https://doi.org/10.1002/bs.3830230203
Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
Box GEP, Draper NR. Empirical Model-Building and Response Surfaces. New York, NY: John Wiley & Sons, Inc.; 1986.
Nickerson RS. Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods 2000; 5(2): 241-301. http://dx.doi.org/10.1037/1082-989X.5.2.241 DOI: https://doi.org/10.1037/1082-989X.5.2.241
Krantz DH. The null hypothesis testing controversy in psychology. Journal of the American Statistical Association 1999; 44(448): 1372-81. http://dx.doi.org/10.1080/01621459.1999.10473888 DOI: https://doi.org/10.1080/01621459.1999.10473888
Thompson B. Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review 1999; 11: 157-69. http://dx.doi.org/10.1023/A:1022028509820 DOI: https://doi.org/10.1023/A:1022028509820
Wilkinson L, The_Task_Force_on_Statistical_Inference. Statistical Methods in Psychology Journals: Guidelines and Explanations. American Psychologist 1999; 54: 594-604. http://dx.doi.org/10.1037/0003-066X.54.8.594 DOI: https://doi.org/10.1037/0003-066X.54.8.594
Shrout PE. Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science 1997; 8(1): 1-2. http://dx.doi.org/10.1111/j.1467-9280.1997.tb00533.x DOI: https://doi.org/10.1111/j.1467-9280.1997.tb00533.x
Hunter JE. Needed: A ban on the significance test. Psychological Science 1997; 8(1): 3-7. http://dx.doi.org/10.1111/j.1467-9280.1997.tb00534.x DOI: https://doi.org/10.1111/j.1467-9280.1997.tb00534.x
Trafimow D, Marks M. Editorial. Basic and Applied Social Psychology 2015; 37: 1-3. http://dx.doi.org/10.1080/01973533.2015.1012991 DOI: https://doi.org/10.1080/01973533.2015.1012991
Trafimow D. Editorial. Basic and Applied Social Psychology 2014; 36(1): 1-2. http://dx.doi.org/10.1080/01973533.2014.865505 DOI: https://doi.org/10.1080/01973533.2014.865505
Rutter M. Epidemiological methods to tackle causal questions. International Journal of Epidemiology 2009; 38: 3-6. http://dx.doi.org/10.1093/ije/dyn253 DOI: https://doi.org/10.1093/ije/dyn253
Tesla N. BrainyQuote.com 2015 [February 26,2015 ]. Available from: http://www.brainyquote/quotes/authros/n/ nikola_tesla.html
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2015 Helena Chmura Kraemer
This work is licensed under a Creative Commons Attribution 4.0 International License.
Policy for Journals/Articles with Open Access
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
Policy for Journals / Manuscript with Paid Access
Authors who publish with this journal agree to the following terms:
- Publisher retain copyright .
- Authors are permitted and encouraged to post links to their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work .
