Deep Learning with Limited Numerical Precision
نویسندگان
چکیده
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.
منابع مشابه
ADaPTION: Toolbox and Benchmark for Training Convolutional Neural Networks with Reduced Numerical Precision Weights and Activation
Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) are useful for many practical tasks in machine learning. Synaptic weights, as well as neuron activation functions within the deep network are typically stored with high-precision formats, e.g. 32 bit floating point. However, since storage capacity is limited and each memory access consumes power, both storage capacity and memo...
متن کاملAnalytical Guarantees on Numerical Precision of Deep Neural Networks
The acclaimed successes of neural networks often overshadow their tremendous complexity. We focus on numerical precision a key parameter defining the complexity of neural networks. First, we present theoretical bounds on the accuracy in presence of limited precision. Interestingly, these bounds can be computed via the back-propagation algorithm. Hence, by combining our theoretical analysis and ...
متن کاملFlexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present th...
متن کاملOvercoming Challenges in Fixed Point Training of Deep Convolutional Networks
It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the wellaccepted solutions facilitating the training of low precision fixed p...
متن کاملExperimental and Numerical Study of Preform Design in Multi Stage Deep Drawing of High Strength Thin Steel Sheet
In this paper, experimental results of a deep drawing process to produce a cylinder of high strength steel with a spherical head were compared with it’s simulation results and three proposal design types. Meanwhile, the amount of limiting draw ratio in some stages was determined. Accuracy and precision of the results of a finite element software to predicting the multi stage deep drawing proces...
متن کامل