Comparison of Bayesian and Frequentist Methods in Estimating the Net Reclassification and Integrated Discrimination Improvement Indices for Evaluation of Prediction Models: Tehran Lipid and Glucose Study

Introduction: The Frequency-based method is commonly used to estimate the Net Reclassification Improvement (NRI)- and Integrated Discrimination Improvement (IDI) indices. These indices measure the magnitude of the performance of statistical models when a new biomarker is added. This method has poor performance in some cases, especially in small samples. In this study, the performance of two Bayesian and Frequentist methods were evaluated and compared for the diabetes prediction model in pre-diabetic women. Materials and Methods: A total of 734 pre-diabetic women aged≥20 years, participated in the first and second phases of the Tehran Lipid and Glucose Study (TLGS) were enrolled in this research and the logistic regression model was used with variables of the Atherosclerosis Risk in Communities (ARIC) in order to estimate the probability of diabetes. Predictors of diabetes in the ARIC study were waist circumference, hypertension, height, weight, age, family history of diabetes, smoking and pulse rate. The Frequentist and Bayesian methods were used to estimate the NRI and IDI. In addition these methods were compared in a small sample size selected randomly from the primary sample. Statistical analyses were performed using R- software version 3.1.3. Results: Estimates of the NRI and IDI indices by the Bayesian and Frequentist methods provided almost similar results. Adding new markers of wrist size, history of macrosomia, and history of preeclampsia to the ARIC model did not show significant effect on the improvement indices. In the small sample size, IDI and NRI showed better performance in the Bayesian method compared to the frequentist method. Conclusion: The Bayesian method had a more reliable performance in comparison with the frequency-based method, especially in small samples.