Isotropic Representation Can Improve Dense Retrieval

The latest Dense Retrieval (DR) models typically encode queries and documents using BERT subsequently apply a cosine similarity-based scoring to determine the relevance. representations, however, are known follow an anisotropic distribution of narrow cone shape such can be undesirable for relevance estimation. In this work, we first show that representations in DR also distribution. We adopt unsupervised post-processing methods Normalizing Flow whitening cope with problem, develop token-wise method addition sequence-wise method. proposed effectively enhance isotropy thereby improving performance as ColBERT RepBERT. To examine potential isotropic representation robustness models, investigate out-of-distribution tasks where test dataset differs from training dataset. results certainly achieve generally improved (The code is available at https://github.com/SNU-DRL/IsotropicIR.git ).