Optimizing User Experience throu Language Understanding

We describe a case study depicting two different strategies for creating a spoken language understanding (SLU) module for a complex telecommunications natural language application. During Phase 1 we created an Annotation Guide that defines semantic call-types and uses these call-types to train the SLU. In Phase 2 we separated the process of creating the Annotation Guide from the training of the SLU. We designed the Annotation Guide with granular and hierarchical call-types that could be combined easily. After the data was labeled, we followed an iterative process of building SLUs with different configurations of the Annotation Guide call-types until we found the SLU that performed the best, given the application requirements. To evaluate the performance of the SLU, we relied on the overall F-Measure traditionally used for technical evaluations. In addition, we considered the F-Measures for the individual call-types. The goal was to have the largest number of application-specific call-types with F-Measures of 70.0 and higher. The results indicated that the Phase 2 SLU far surpassed the performance of the Phase 1 SLU.