Session 3: Natural Language Evaluation

The session on Natural Language Evaluation focused on methods for evaluating text understanding systems. Beginning with the first Message Understanding Conference (MUCK-l) in 1987, there has been increasing focus on how to measure and evaluate text understanding systems. The MUCK-1 conference required developers to port their system to a common domain of Navy intelligence messages; MUCK-2 (May 1989) developed the first sconng system for evaluation based on template fill (also on a domain of Navy messages). MUCK-3 (to take place in late 1990 and early 1991) will refine that process and include an automated sconng procedure to grade quality of template fill. Meanwhile, a second evaluation effort has started, related to the MURASAKI multi-lingual text understanding project. It also uses the notion of evaluating systems doing template fill.