Learning Compact Features via In-Training Representation Alignment

Deep neural networks (DNNs) for supervised learning can be viewed as a pipeline of the feature extractor (i.e., last hidden layer) and linear classifier output that are trained jointly with stochastic gradient descent (SGD) on loss function (e.g., cross-entropy). In each epoch, true is estimated using mini-batch sampled from training set model parameters then updated gradients. Although latter provides an unbiased estimation former, they subject to substantial variances derived size number mini-batches, leading noisy jumpy updates. To stabilize such undesirable variance in estimating gradients, we propose In-Training Representation Alignment (ITRA) explicitly aligns distributions two different mini-batches matching SGD process. We also provide rigorous analysis desirable effects representation learning: (1) extracting compact representation; (2) reducing over-adaption via adaptively weighting mechanism; (3) accommodating multi-modalities. Finally, conduct large-scale experiments both image text classifications demonstrate its superior performance strong baselines.