1998 Hub-4 Information Extraction Evaluation

This paper documents the Information Extraction Named-Entity Evaluation (IE-NE), one of the new spokes added to the DARPA-sponsored 1998 Hub-4 Broadcast News Evaluation. This paper discusses the information extraction task as posed for the 1998 Broadcast News Evaluation. This paper reviews the evaluation metrics, the scoring process, and the test corpus that was used for the evaluation. Finally, this paper reviews the results of the first running of a Hub-4 IE-NE Evaluation. The Baseline IE-NE evaluation, in which BBN’s IdentiFinder was run on the primary system transcripts submitted for the Hub-4 Broadcast News evaluation, found that the transcripts generated by LIMSI’s automatic speech recognition system produced the "highest" F-measure score (82.39). In the Quasi IE-NE evaluation, where sites ran their own NEtaggers on a set of three baseline recognizer transcripts, the SRI developed tagger achieved the highest F-measure score for baseline recognizers 1 & 3, while the BBN developed tagger achieved the highest score for baseline recognizer 2. In the Full IE-NE evaluation, where sites implemented their own NE-tagger on the their own automatic speech recognizer transcripts, BBN achieved the highest overall F-measure score of 82.22.