Online Cost-Sensitive Learning for Efficient Interactive Classification

A lot of practical machine learning applications deal with interactive classification problems where trained classifiers are used to help humans find positive examples that are of interest to them. Typically, these classifiers label a large number of test examples and present the humans with a ranked list to review. The humans involved in this process are often expensive domain experts with limited time. We present an online cost-sensitive learning approach (more-like-this) that focuses on reducing the time it takes for the experts to review and label examples in interactive machine learning systems. We target the scenario where a batch classifier has been trained for a given classification task and optimize the interaction between the classifier and the domain experts who are consuming the results of this classifier. The goal is to make these experts more efficient and effective in performing their task as well as eventually improving the classifier over time. We validate our approach by applying it to the problem of detecting errors in health insurance claims and show significant reduction in labeling time while increasing the overall performance of the system.