NIST and NFI-TNO evaluations of automatic speaker recognition

In the past years, several text-independent speaker recognition evaluation campaigns have taken place. This paper reports on results of the NIST evaluation of 2004 and the NFI-TNO forensic speaker recognition evaluation held in 2003, and reflects on the history of the evaluation campaigns. The effects of speech duration, training handsets, transmission type, and gender mix show expected behaviour on the DET curves. New results on the influence of language show an interesting dependence of the DET curves on the accent of speakers. We also report on a number of statistical analysis techniques that have recently been introduced in the speaker recognition community, as well as a new application of the analysis of deviance analysis. These techniques are used to determine that the two evaluations held in 2003, by NIST and NFITNO, are of statistically different difficulty to the speaker recognition systems. 2005 Published by Elsevier Ltd. 0885-2308/$ see front matter 2005 Published by Elsevier Ltd. doi:10.1016/j.csl.2005.07.001 * Corresponding author. Tel.: +31 346 356 235; Fax: +31 346 353 977. E-mail addresses: [email protected] (D.A. van Leeuwen), [email protected] (A.F. Martin), [email protected] (M.A. Przybocki), [email protected] (J.S. Bouten). D.A. van Leeuwen et al. / Computer Speech and Language 20 (2006) 128–158 129