The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval

On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity self-attention mechanism, however, such difficulties long documents. Recent works dealing with this issue include truncating documents, in which case one loses potential relevant information, segmenting them into several passages, may lead miss some high computational when number passages is large, or modifying mechanism make it sparser as sparse-attention at risk again missing information. We follow here slightly different approach first selects key blocks document by local query-block pre-ranking, then few are aggregated form short that can be processed model BERT. Experiments conducted on standard Information Retrieval datasets demonstrate effectiveness proposed approach.