This paper takes an empirical approach to evaluating three model-based reinforcementlearning methods. All methods intend to speed the learning process by mixing exploitation of learned knowledge with exploration of possibly promising alternatives. We consider -greedy exploration, which is computationally cheap and popular, but unfocused in its exploration effort; R-Max exploration, a simplifica...