Looking for Low-proficiency Sentences in ELL Writing

Determining whether an author is writing in their native language (L1) or a second language (L2) is a problem that lies at the intersection of four traditional NLP tasks: native language identification, similar language identification, detecting translationese, and grammatical error correction. In general, the goal of the language learner is to improve their proficiency until their writing is indistinguishable from that of a native speaker. By being able to automatically and reliably determine whether a section of text looks like L1 or L2 text, areas of writing that still need improvement can be brought to the learner’s attention. Additionally, the state of the art for correcting grammatical errors involves using machine translation to translate from errorful text to corrected text[1] and there is interesting work being done in generating new training examples by using machine translation to go from error-free text to errorful text.[2] Both approaches could be enhanced by a system that can tell how close sections of the translation are to an L1 or L2 target. I present Deep Filter, a convolutional neural network that uses a deep network as the convolution, for determining the probability that an entire essay was written by an English Language Learner (ELL), using the document-level label of whether the writer’s L1 was English. I then use the unpooled activations from the convolutional filter to provide insight into the probability that sections of the text were written by a non-native writer. The model is able to learn to differentiate native from non-native writing, and can identify both low-proficiency sections of the essays as well as other idiosyncracies of non-native English writers.