Using BLSTM for Interpretation of 2D Languages - Case of Handwritten Mathematical Expressions

In this work, we study how to extend the capability of BLSTM networks to process data which are not only text strings but graphical two-dimensional languages such as handwritten mathematical expressions. The proposed solution aims at transforming the mathematical expression description into a sequence including at the same time symbol labels and relationship labels, so that classical supervised sequence labeling with recurrent neural networks can be applied. For simple one-dimensional (1-D) expression, we use the Right label to segment one symbol from the next one, as with the standard blank label for regular text. For genuine twodimensional (2-D) expressions, we introduce additional specific labels assigned to each of the different possible spatial relationships that exist between sub-expressions. As a result, BLSTM network is able to perform at the same time the symbol recognition task and the segmentation task, which is a new perspective for the mathematical expression domain. MOTS-CLÉS : Reconnaissance d’expressions mathématiques, écriture manuscrite, réseau récurrent, BLSTM.