Mining Relations Among Cross-Frame Affinities for Video Semantic Segmentation

The essence of video semantic segmentation (VSS) is how to leverage temporal information for prediction. Previous efforts are mainly devoted developing new techniques calculate the cross-frame affinities such as optical flow and attention. Instead, this paper contributes from a different angle by mining relations among affinities, upon which better aggregation could be achieved. We explore in two aspects: single-scale intrinsic correlations multi-scale relations. Inspired traditional feature processing, we propose Single-scale Affinity Refinement (SAR) Multi-scale Aggregation (MAA). To make it feasible execute MAA, Selective Token Masking (STM) strategy select subset consistent reference tokens scales when calculating also improves efficiency our method. At last, strengthened SAR MAA adopted adaptively aggregating information. Our experiments demonstrate that proposed method performs favorably against state-of-the-art VSS methods. code publicly available at https://github.com/GuoleiSun/VSS-MRCFA .