SyCo: A Probabilistic Machine Learning Method for Classifying Chief Complaints into Symptom and Syndrome Categories

OBJECTIVE Design, build and evaluate a symptom-based probabilistic chief complaint classifier for the Real-time Outbreak and Disease Surveillance System (RODS). BACKGROUND Scientists have utilized many chief complaint (CC) classification techniques in biosurveillance including keyword search, weighted keyword search, and naïve Bayes. These techniques may utilize CC-to-syndrome or CC-to-symptom-to-syndrome classification approaches. In the former approach, we classify a CC directly into syndrome categories. In the latter approach, we first classify a CC into symptom categories. Then, we use a syndrome definition, a combination of one or more symptoms, to determine whether or not a chief complaint belongs in a particular syndrome category. One approach to CC-to-symptom-to-syndrome classification uses manually weighted keyword search and Boolean operations to build syndrome classifiers. A limitation to this approach is that it does not address uncertainty in the data and the system is manually parameterized. A CC-tosymptom-to-syndrome approach that is both probabilistic and utilizes machine learning addresses these limitations. METHODS We constructed SyCo — a CC-to-symptom-to-syndrome probabilistic chief complaint classifier. SyCo learns a Naïve Bayes model of the relationship between words and symptoms given a training set of labeled chief complaints. To perform a classification, SyCo first computes the posterior probability of each symptom using the odds formulation of Bayes rule. SyCo can compute the posterior probability traditionally or in single word mode. When single word mode is enabled SyCo will only use the likelihood ratio of the word (given a symptom) that maximizes the posterior probability. Finally, SyCo uses the posterior probabilities from the first step to compute the posterior probability of a syndrome given a chief complaint. A syndrome is defined as any combination of symptoms and Boolean operations. SyCo supports the operations AND, OR, and NOT by using the rules of conjunction, disjunction and negation of independent events. For example, P(A) AND P(B) = P(A) x P(B). A board certified infectious disease physician [JD] read 16718 chief complaints and indicated the presence or absence of seventeen symptoms for each chief complaint. We measured the performance of SyCo when classifying seventeen individual symptoms and three syndromes with and without the single word mode using leave-one-out cross validation. We measured the area under the curve (AUC) of the resultant receiver operator characteristic (ROC) curves and established 90% confidence intervals using 100 iterations of non-parametric bootstrapping. RESULTS The area under the curve without and with the single word assumption ranged from 0.785 to 0.9918 and 0.7442 to 0.9916, respectively. The single word mode improved performance significantly in 7 out of 20 cases and degraded performance in 2 out of the 20 cases. CONCLUSION SyCo is a symptom-based probabilistic chief complaint classifier that has excellent discriminatory ability for classifying chief complaints into symptom categories and syndromes. We have made SyCo available in RODS Version 4.2. ACKNOWLEDGEMENTS This research was supported in part by DARPA/Mellon/Pitt grant F30602-01-2-0550, NSF grant IIS-0325581 and PA Department of Health grant ME-01-737