Motion and appearance based Multi-Task Learning network for autonomous driving

Autonomous driving has various visual perception tasks such as object detection, 1 motion detection, depth estimation and flow estimation. Multi-task learning (MTL) 2 has been successfully used for jointly estimating some of these tasks. Previous 3 work was focused on utilizing appearance cues. In this paper, we address the gap 4 of incorporating motion cues in a multi-task learning system. We propose a novel 5 two-stream architecture for joint learning of object detection, road segmentation 6 and motion segmentation. We designed three different versions of our network to 7 establish systematic comparison. We show that the joint training of tasks signifi8 cantly improves accuracy compared to training them independently even with a 9 relatively smaller amount of annotated samples for motion segmentation. To enable 10 joint training, we extended KITTI object detection dataset to include moving/static 11 annotations of the vehicles. An extension of this new dataset named KITTI MOD 12 is made publicly available via the official KITTI benchmark website . Our baseline 13 network outperforms MPNet which is a state of the art for single stream CNN-based 14 motion detection. The proposed two-stream architecture improves the mAP score 15 by 21.5% in KITTI MOD. We also evaluated our algorithm on the non-automotive 16 DAVIS dataset and obtained accuracy close to the state-of-the-art performance. 17 The proposed network runs at 8 fps on a Titan X GPU using a two-stream VGG16 18 encoder. Demonstration of the work is provided in. 19