Title: A Bayesian Method to uncertain MDPs by Markov set-chains

Masayuki Horiguchi, Kanagawa University, Japan, horiguchi@kanagawa-u.ac.jp Masami Yasuda, Chiba University, Japan, yasuda@math.s.chiba-u.ac.jp Masami Kurano, Chiba University, emeritus, Japan, kurano@seaple.ne.jp Abstract: We are concerned with Bayesian inference of transition law in MDPs. Applying De Robertis/Hartigan’s result(1981, Ann. Math) of Bayesian inference on intervals of measure for MDPs, it is formulated as uncertain MDPs with intervals of prior measures. We characterize the uncertain model as a controlled Markov set-chain model with intervals of transition matrices. Here we adapted the notion of Markov se-chain by Hartfiel(1998, Lec. Note in Math).

Firstly a model construction to MDP is given by a set of interval matrices. The true transition matrix is estimated by an interval matrix and then uncertain MDPs of Bayesian approach is defined by a controlled Markov set-chain model. The sufficient and necessary condition that a policy is Pareto-optimal by an optimality inclusion relation. By using Bayesian inference on intervals of measure for uncertain MDPs, interval matrices of posterior measures are given. Some convergence property for uncertain MDPs of Bayesian approach is discussed and also a numerical experiment is shown.