The validity of oral assessment (viva) that assesses specific and unique competencies in a post-graduate psychiatry examination

Background Studies have criticized oral assessments for having poor validity and reliability. There is limited research on structured oral assessments that assess specific competencies. Aims To evaluate the validity of the oral assessment component of a postgraduate psychiatry examination. Methods A retrospective analysis of the examination scores of 154 candidates from 12 postgraduate psychiatry examinations conducted during an 8 year period was carried out. Concurrent and construct validity was examined by correlating marks at the viva with the marks of theory, clinical long case and clinical short case components of the candidates. Separate multiple regression analyses were conducted to predict the scores of the different components of the examination. Results Repeated measure ANOVA showed there was no significant difference in the means between the different components of the examination (F=0.49, p=0.486). The viva was a sensitive method of assessment (79/88 =89.7%) but the specificity was low (36/66 =54.5%). Positive predictive value was 72.5% and the negative predictive value was 80.0%. Logistic regression analysis showed that the odds of passing the viva and passing the examination compared with passing the viva and failing the exam was 10.53 (95% CI 4.5424.47). There was a statistically significant, moderately high correlation between viva and theory components (r=.50 p<0.001). Multiple regression models showed that viva marks were a predictor of performance at the clinical short case and theory components but not the clinical long case. Conclusion Viva had good sensitivity and positive and negative predictive values. Instead of discontinuing the use of oral assessments, ways should be identified to improve the reliability and validity of the oral assessment.