AbstractWell-known activation functions like ReLU or Leaky are non-differentiable at the origin. Over years, many smooth approximations of have been proposed using various smoothing techniques. We propose new a function by convolving it with approximate identities. In particular, we present and show that they outperform several well-known in datasets models. call this Smooth Activation Unit (SA...

We discuss a modification of the chained Rosenbrock function introduced by Nesterov. This function rN is a polynomial of degree four defined for x ∈ R. Its only stationary point is the global minimizer x∗ = (1, 1, . . . , 1) with optimal value zero. A point x in the box B := {x | −1 ≤ xi ≤ 1 for 1 ≤ i ≤ n} with rN (x) = 1 is given such that there is a continuous descent path within B that start...