Real or Fake Text?: Investigating Human Ability to Detect Boundaries between Human-Written and Machine-Generated Text

As text generated by large language models proliferates, it becomes vital to understand how humans engage with such text, and whether or not they are able detect when the reading did originate a human writer. Prior work on detection of focuses case where an entire passage is either human-written machine-generated. In this paper, we study more realistic setting begins as transitions being state-of-the-art neural models. We show that, while annotators often struggle at task, there substantial variance in annotator skill that given proper incentives, can improve task over time. Furthermore, conduct detailed comparison analyze variety variables (model size, decoding strategy, fine-tuning, prompt genre, etc.) affect performance. Finally, collect error annotations from our participants use them certain textual genres influence make different types errors sentence-level features correlate highly selection. release RoFT dataset: collection 21,000 paired classifications encourage future evaluation text.