How important are activation functions in regression and classification? A survey, performance comparison, and future directions

Inspired by biological neurons, the activation functions play an essential part in learning process of any artificial neural network (ANN) commonly used many real-world problems. Various have been proposed literature for classification as well regression tasks. In this work, we survey that employed past current state-of-the-art. particular, present various developments over years and advantages disadvantages or limitations these functions. We also discuss classical (fixed) functions, including rectifier units, adaptive addition to discussing taxonomy based on characterization, a applications is presented. To end, systematic comparison fixed performed datasets such MNIST, CIFAR-10, CIFAR-100. recent years, physics-informed machine framework has emerged solving problems related scientific computations. For purpose, requirements framework. Furthermore, comparisons are made among different using libraries TensorFlow, PyTorch, JAX. Our findings show rectified linear unit (ReLU) its variants, which currently state-of-the-art problems, do not work frameworks due stringent requirement existence derivatives, whereas other hyperbolic tangent, swish, sine give better performance, with superior results achieved especially multiscale