Belief Dependent Rewards

[josh0tt]

Are there any existing examples of a POMDPs.jl problem setup with belief-dependent rewards (rho-POMDP)? Currently, I have setup my problem as a belief MDP so that my reward function can make use of the belief.

[johannes-fischer]

I have worked with belief-dependent rewards in POMDPs.jl here https://github.com/johannes-fischer/icml2020_ipft, but it is not up to date with the current POMDPs.jl version. I was using a particle belief and a belief MDP wrapper to compute the belief-based reward from a particle belief.

Today, I would probably do it differently and introduce abstract type ρPOMDP{S, A, O, B} end. Downside is that then all methods in POMDPs.jl that are defined for MDP/POMDP are not available for the new type and would have to be reimplemented. I am not sure, if it is feasible or advisable to make ρPOMDP a subtype of MDP{B,A} or POMDP{S,A,O}, which would probably allow to reuse some methods without copy&paste-ing code for ρPOMDP. But declaring it as MDP{B,A} could be problematic since the transition should still be for states, not only for beliefs, and POMDP{S,A,O} might be undesired since a POMDP by definition has a state-based reward.

[zsunberg]

Right now I am keeping the true state around only to determine the true performance during the evaluation (printout the true reward for taking an action), but I guess this is unnecessary since during the tree search / rollouts only the belief is used.

I don't think you should do this. Instead, I'd recommend something like the following:

m = YourPOMDP()
up = DiscreteUpdater(m) # or particle filter or whatever you want
belief_reward(b, a) = sum(pdf(b, s)*reward(m, s, a) for s in support(b)) # this is the mean reward, but you could use any reward.
bm = BeliefMDP(m, up, belief_reward) # you would write this, similar to GenerativeBeliefMDP
planner = solve(MCTS.DPWSolver(), bm) # IMPORTANT: the planner only knows about bm, not m

# evaluate with this. IMPORTANT: we are simulating with the original POMDP, but the planner is using the belief MDP
mean(simulate(RolloutSImulator(), m, planner, up) for _ in 1:100)

[josh0tt]

This looks much more structured and helpful than the work arounds I'm using right now. The idea is that BeliefMDP would use GenerativeBeliefMDP but just define its own custom reward? I'm not sure I follow the last point though. Why simulate with the original POMDP instead of the belief MDP?

[zsunberg]

The idea is that BeliefMDP would use GenerativeBeliefMDP but just define its own custom reward?

Not exactly. BeliefMDP would be similar to GenerativeBeliefMDP, but it would not use GenerativeBeliefMDP (except, perhaps a few methods could be defined for Union{BeliefMDP,GenerativeBeliefMDP}).

I'm not sure I follow the last point though. Why simulate with the original POMDP instead of the belief MDP?

If you simulate with the belief MDP, the result will be the accumulated reward with the information gathering bonus; IIUC you want to evaluate how much reward in the original POMDP you get when using the belief-space planner.

[josh0tt]

to follow up on this, I created a BeliefMDP structure for a project I was working on here which is largely based off of GenerativeBeliefMDP. Right now, I've isolated it just for that project but it could be brought into POMDPs.jl later on.

[zsunberg]

Thanks @josh0tt , please do remember to circle back and see if you can contribute it if you find it useful!