Bayesian Model Averaging for Selection of a Risk Prediction Model for Death within Thirty Days of Discharge: The SILVER-AMI Study

Terrence E. Murphy; Sui W. Tsang; Linda S. Leo-Summers; Mary Geda; Dae H. Kim; Esther Oh; Heather G. Allore; John Dodson; Alexandra M. Hajduk; Thomas M. Gill; Sarwat I. Chaudhry

doi:10.6000/1929-6029.2019.08.01

Authors

Terrence E. Murphy Yale University School of Medicine, New Haven, CT, USA
Sui W. Tsang Yale University School of Medicine, New Haven, CT, USA
Linda S. Leo-Summers Yale University School of Medicine, New Haven, CT, USA
Mary Geda Yale University School of Medicine, New Haven, CT, USA
Dae H. Kim Harvard University School of Medicine, Boston, MA, USA
Esther Oh Johns Hopkins University School of Medicine, Baltimore, MD, USA
Heather G. Allore Yale University School of Medicine, New Haven, CT, USA
John Dodson New York University School of Medicine, New York, NY, USA
Alexandra M. Hajduk Yale University School of Medicine, New Haven, CT, USA
Thomas M. Gill Yale University School of Medicine, New Haven, CT, USA
Sarwat I. Chaudhry Yale University School of Medicine, New Haven, CT, USA

DOI:

https://doi.org/10.6000/1929-6029.2019.08.01

Keywords:

Risk prediction, AMI, Bayesian model averaging, AIC, BIC, backward-selection.

Abstract

We describe a selection process for a multivariable risk prediction model of death within 30 days of hospital discharge in the SILVER-AMI study. This large, multi-site observational study included observational data from 2000 persons 75 years and older hospitalized for acute myocardial infarction (AMI) from 94 community and academic hospitals across the United States and featured a large number of candidate variables from demographic, cardiac, and geriatric domains, whose missing values were multiply imputed prior to model selection. Our objective was to demonstrate that Bayesian Model Averaging (BMA) represents a viable model selection approach in this context. BMA was compared to three other backward-selection approaches: Akaike information criterion, Bayesian information criterion, and traditional p-value. Traditional backward-selection was used to choose 20 candidate variables from the initial, larger pool of five imputations. Models were subsequently chosen from those candidates using the four approaches on each of 10 imputations. With average posterior effect probability ≥ 50% as the selection criterion, BMA chose the most parsimonious model with four variables, with average C statistic of 78%, good calibration, optimism of 1.3%, and heuristic shrinkage of 0.93. These findings illustrate the utility and flexibility of using BMA for selecting a multivariable risk prediction model from many candidates over multiply imputed datasets.

References

Burnham K, Anderson D. Multimodel inference: understanding AIC and BIC in Model Selection. Sociological Methods & Research 2004; 33(2): 261-304. https://doi.org/10.1177/0049124104268644 DOI: https://doi.org/10.1177/0049124104268644

Hurvich CM, Tsai C-L. The impact of model selection on inference in linear regression. The American Statistician 1990; 44(3): 214-7 DOI: https://doi.org/10.1080/00031305.1990.10475722

Posada D, Buckley TR. Model selection and model averaging in phylogenetics: advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests. Syst Biol 2004; 53(5): 793-808. https://doi.org/10.1080/10635150490522304 DOI: https://doi.org/10.1080/10635150490522304

Harrell FE Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15(4): 361-87. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4 DOI: https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

Raftery AE, Richardson S. Model selection for generalized linear models via GLIB, with application to epidemiology. In Bayesian Biostatistics (Berry DA, Stangl DK, Eds.), New York: Marcel Dekker 1996; pp. 321-354.

Hoeting JA, Raftery AE, Volinsky CT. Bayesian model averaging: a tutorial. Statistical Science 1999; 14(4): 382-417. DOI: https://doi.org/10.1214/ss/1009212519

Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control 1974; AC-19(6): 716-23. https://doi.org/10.1109/TAC.1974.1100705 DOI: https://doi.org/10.1109/TAC.1974.1100705

Schwarz G. Estimating the dimension of a model. Annals of Statistics 1978; 6(2): 461-4. https://doi.org/10.1214/aos/1176344136 DOI: https://doi.org/10.1214/aos/1176344136

Goodman S. A dirty dozen: twelve p-value misconceptions. Semin Hematol 2008; 45(3): 135-40. https://doi.org/10.1053/j.seminhematol.2008.04.003 DOI: https://doi.org/10.1053/j.seminhematol.2008.04.003

Wasserstein RL, NLL. The ASA's statement on p-values: context, process, and purpose. Am Stat 2016; 70: 129-33. https://doi.org/10.1080/00031305.2016.1154108 DOI: https://doi.org/10.1080/00031305.2016.1154108

Dodson JA, Geda M, Krumholz HM, Lorenze N, Murphy TE, Allore HG, et al. Design and rationale of the comprehensive evaluation of risk factors in older patients with AMI (SILVER-AMI) study. BMC Health Serv Res 2014; 14: 506.PMCPMC4239317. DOI: https://doi.org/10.1186/s12913-014-0506-4

Raftery A, Hoeting JA, Volinsky C, Painter I, Yeung KY. R package 'BMA' 2015.

White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011; 30(4): 377-99. https://doi.org/10.1002/sim.4067 DOI: https://doi.org/10.1002/sim.4067

Madigan D, Raftery A. Model selection and accounting for model uncertainty in graphical models using Occam's window. Journal of the American Statistical Association 1994; 89: 1535-1546. https://doi.org/10.1080/01621459.1994.10476894 DOI: https://doi.org/10.1080/01621459.1994.10476894

Hanley JA, BJ M. The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 1982; 143(1): 29-36. https://doi.org/10.1148/radiology.143.1.7063747 DOI: https://doi.org/10.1148/radiology.143.1.7063747

Hosmer DW, S L. Applied Logistic Regression. New York: Wiley 2013. DOI: https://doi.org/10.1002/9781118548387

van Houwelingen JC, le Cessie S. Predictive value of statistical models. Stat Med 1990; 8: 1303-25. https://doi.org/10.1002/sim.4780091109 DOI: https://doi.org/10.1002/sim.4780091109

Rubin DB. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons Inc., New York 1987.

http://dx.doi.org/10.1002/9780470316696 DOI: https://doi.org/10.1002/9780470316696

Newell MC, Henry JT, Henry TD, Duval S, Browning JA, Christiansen EC, et al. Impact of age on treatment and outcomes in ST-elevation myocardial infarction. Am Heart J 2011; 161(4): 664-72. https://doi.org/10.1016/j.ahj.2010.12.018 DOI: https://doi.org/10.1016/j.ahj.2010.12.018

Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49(12): 1373-9. https://doi.org/10.1016/S0895-4356(96)00236-3 DOI: https://doi.org/10.1016/S0895-4356(96)00236-3

Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (pages 87-89): Springer 2008; p. 500.