Fast Rates with Unbounded Losses

We present new excess risk bounds for randomized and deterministic estimators for general unbounded loss functions including log loss and squared loss. Our bounds are expressed in terms of the information complexity and hold under the recently introduced v-central condition, allowing for high-probability bounds, and its weakening, the v-pseudoprobability convexity condition, allowing for bounds in expectation even under heavy-tailed distributions. The parameter v determines the achievable rate and is akin to the exponent in the Tsybakov margin condition and the Bernstein condition for bounded losses, which the v-conditions generalize; favorable v in combination with small information complexity leads to Õ(1/n) rates. While these fast rate conditions control the lower tail of the excess loss, the upper tail is controlled by a new type of witness-of-badness condition which allows us to connect the excess risk to a generalized Rényi divergence, generalizing previous results connecting Hellinger distance to KL divergence.