Fertility Models for Statistical Natural Language Understanding

Several recent efforts in statistical natural language understanding (NLU) have focused on generating clumps of English words from semantic meaning concepts (Miller et al., 1995; Levin and Pieraccini, 1995; Epstein et al., 1996; Epstein, 1996). This paper extends the IBM Machine Translation Group's concept of fertility (Brown et al., 1993) to the generation of clumps for natural language understanding. The basic underlying intuition is that a single concept may be expressed in English as many disjoint clump of words. We present two fertility models which attempt to capture this phenomenon. The first is a Poisson model which leads to appealing computational simplicity. The second is a general nonparametric fertility model. The general model's parameters are bootstrapped from the Poisson model and updated by the EM algorithm. These fertility models can be used to impose clump fertility structure on top of preexisting clump generation models. Here, we present results for adding fertility structure to unigram, bigram, and headword clump generation models on ARPA's Air Travel Information Service (ATIS] domain. 1 I n t r o d u c t i o n The goal of a natural language understanding (NLU) system is to interpret a user's request and respond with an appropriate action. We view this interpretation as translation from a natural language expression, E, into an equivalent expression, F, in an unambigous formal language. Typically, this formal language will be hand-crafted to enhance performance on some task-specific domain. A statistical NLU system translates a request E as the most likely formal expression ~' according to a probability model p, = are maxp(F[E) --are maxp(F, E). o v e r a l l F o v e r a l l F We have previously built a fully automatic statistical NLU system (Epstein et al., 1996) based on the source-channel factorization of the joint distribution p ( f , E) p ( f , E) = p( f )p(ZlF ). This factorization, which has proven effective in speech recognition (Bahl, Jelinek, and Mercer, 1983), partitions the joint probability into an a priori intention model p(F), and a translation model p(E[F) which models how a user might phrase a request F in English. For the ATIS task, our formal language is a minor variant of the NL-Parse (Hemphill, Godfrey, and Doddington, 1990) used by ARPA to annotate the ATIS corpus. An example of a formal and natural language pair is: • F : List flights from New Orleans to Memphis flying on Monday departing early_morning • E: do you have any flights going to Memphis leaving New Orleans early Monday morning Here, the evidence for the formal language concept 'early_morning' resides in the two disjoint clumps of English 'early' and 'morning'. In this paper, we introduce the notion of concept fertility into our translation models p(EIF ) to capture this effect and the more general linguistic phenomenon of embedded clauses. Basically, this entails augmenting the translation model with terms of the form p(nlf), where n is the number of clumps generated by the formal language word f . The resulting model can be trained automatically from a bilingual corpus of English and formal language sentence pairs. Other attempts at statistical NLU systems have used various meaning representations such as concepts in the AT&T system (Levin and Pieraccini, 1995) or initial semantic structure in the BBN system (Miller et al., 1995). Both of these systems require significant rule-based transformations to produce disambiguated interpretations which are then