End-to-End page-Level assessment of handwritten text recognition

The evaluation of Handwritten Text Recognition (HTR) systems has traditionally used metrics based on the edit distance between HTR and ground truth (GT) transcripts, at both character word levels. This is very adequate when experimental protocol assumes that GT text lines are same, which allows distances to be independently computed each given line. Driven by recent advances in pattern recognition, increasingly face end-to-end page-level transcription a document, where precision locating different their corresponding reading order (RO) play key role. In such case, standard do not take into account inconsistencies might appear. this paper, problem evaluating page level introduced detail. We analyse convenience using two-fold evaluation, accuracy RO goodness considered separately. Different alternatives proposed, analysed empirically compared through partially simulated real, full experiments. Results support validity proposed approach. An important conclusion an can adequately achieved just two simple well-known metrics: Word Error Rate (WER), takes sequentiality account, here re-formulated Bag Words (bWER), ignores order. While latter directly accurately assess intrinsic recognition errors, difference gracefully correlates with Normalised Spearman's Foot Rule Distance (NSFD), metric explicitly measures errors associated layout analysis flaws.