Shape-constrained Estimation of Value Functions

We present a fully nonparametric method to estimate the value function, via simulation, in the context of expected infinite-horizon discounted rewards for Markov chains. Estimating such value functions plays an important role in approximate dynamic programming. We incorporate “soft information” into the estimation algorithm, such as knowledge of convexity, monotonicity, or Lipchitz constants. In the presence of such information, a nonparametric estimator for the value function can be computed that is provably consistent as the simulated time horizon tends to infinity. As an application, we implement our method on price tolling agreement contracts in energy markets. Corresponding author. Department of Management Science & Engineering, Stanford University, email: [email protected], web: www.stanford.edu/∼mousavi. Department of Management Science & Engineering, Stanford University, email: [email protected], web: www.stanford.edu/∼glynn.