Head-Free Lightweight Semantic Segmentation with Linear Transformer

Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by overall structure has long ignored, which hinders their applications resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for segmentation, named Adaptive Frequency Transformer (AFFormer). AFFormer adopts parallel to leverage prototype representations as specific learnable local descriptions replaces decoder and preserves rich image semantics high-resolution features. Although removing compresses most of computation, accuracy is still limited low resources. Therefore, employ heterogeneous operators (CNN vision Transformer) pixel embedding further save costs. Moreover, it very difficult linearize complexity from perspective spatial domain. Due fact that sensitive frequency information, construct learning block with adaptive filter O(n) replace standard self attention O(n^2). Extensive experiments widely adopted datasets demonstrate achieves superior while retaining only 3M parameters. On ADE20K dataset, 41.8 mIoU 4.6 GFLOPs, 4.4 higher than Segformer, 45% less GFLOPs. Cityscapes 78.7 34.4 2.5 Segformer 72.5% Code available at https://github.com/dongbo811/AFFormer.