We consider the Bellman residual minimization approach for solving discounted Markov decision problems, where we assume that a generative model of the dynamics and rewards is available. At each policy iteration step, an approximation of the value function for the current policy is obtained by minimizing an empirical Bellman residual defined on a set of n states drawn i.i.d. from a distribution ...

We study numerical approximations for the payoff function of the stochastic optimal stopping and control problem. It is known that the payoff function of the optimal stopping and control problem corresponds to the solution of a normalized Bellman PDE. The principal aim of this thesis is to study the rate at which finite difference approximations, derived from the normalized Bellman PDE, converg...

In the spring of 2003, I began to wonder about the history of the Bellman-Ford algorithm [CLRS01] for finding shortest paths in a graph. In particular, I was interested in understanding when Bellman’s name and Ford’s name became jointly associated with the algorithm. To research these questions, I made trips to the BBN Technologies and MIT libraries, did Web searches, and sent emails to univers...

We study the Bellman equation for undiscounted exit time optimal control problems for nonlinear systems and nonnegative instantaneous costs using the dynamic programming approach. Using viscosity solution theory, we prove a uniqueness theorem that characterizes the value functions for these problems as the unique viscosity solutions of the corresponding Bellman equations that satisfy appropriat...

We propose a novel stochastic extension of timed automata, i.e. Markovian Timed Automata (MTA). We study the problem of optimizing the reachability probabilities in this model. Two variants are considered, namely, the timebounded and unbounded reachability. In each case, we propose Bellman equations to characterize the probability. For the former, we provide two approaches to solve the Bellman ...

In this article, an efficient algorithm for an optimal decision strategy approximation is introduced. The proposed approximation of the Bellman equation is based on HDMR technique. This non-parametric function approximation is used not only to reduce memory demands necessary to store Bellman function, but also to allow its fast approximate minimization. On that account, a clear connection betwe...

In this paper a hybrid implementation for Bellman-Ford to solve shortest path problems is proposed using OpenCL. Here first parallel implementation for Bellman-Ford for single source shortest path (SSSP) problem and all pair shortest path (APSP) are analyzed on CPU and GPU and based on this analysis work is divided among CPU and GPU and hybrid implementation is done. As proper resource utilizat...

1 Wald, A., Statistical Decision Functions (New York: John Wiley & Sons, 1950). 2 Wald, A., Sequential Analysis (New York: John Wiley & Sons, 1947). Bussgang, J. J., and D. Middleton, "Optimum sequential detection of signals in noise," IRE Trans. on Inform. Theory, IT-1, 5 (December) 1955. 4Middleton, D., An Introduction to Statistical Communication Theory (New York: McGrawHill, 1960), Chapter ...

Consider the single-source cheapest paths problem in a digraph with negative edge costs allowed. A hybrid of Bellman-Ford and Dijkstra algorithms is suggested, improving the running time bound upon Bellman-Ford for graphs with a sparse distribution of negative cost edges. The algorithm iterates Dijkstra several times without reinitializing values d(v) at vertices. At most k+2 executions of Dijk...