One-Counter Markov Decision Processes

Warning

This publication doesn't include Institute of Computer Science. It includes Faculty of Informatics. Official publication website can be found on muni.cz.
Authors

BRÁZDIL Tomáš BROŽEK Václav ETESSAMI Kousha KUČERA Antonín WOJTCZAK Dominik

Year of publication 2010
Type Article in Proceedings
Conference Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms
MU Faculty or unit

Faculty of Informatics

Citation
Web Link to SODA 2010 electronic proceedings
Field Informatics
Keywords Markov decision proces; probability; one counter MDP; reachability; termination
Description We study One-Counter Markov Decision Processes (OC-MDPs), which extend finite-state MDPs with an unbounded counter. The counter can be incremented, decremented, or not changed during each state transition. Basic objectives for OC-MDPs include ``termination'' (Does the OC-MDP reach counter 0?) and ``limit'' questions (Is the limsup value infinity?). We may ask what is the optimal probability of such objectives, or ask for the existence and synthesis of optimal strategies. We show that several quantitative and almost-sure limit problems can be answered in polynomial time, and that almost-sure termination problems (without selection of desired terminal states) can also be answered in polynomial time. On the other hand, we show that the almost-sure termination problem with selected terminal states is PSPACE-hard and we provide an exponential time algorithm for this problem. We also characterize classes of strategies that suffice for optimality in several of these settings.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.

More info