How to constitute POMDP with continuous action space?

[tyoshidais]

How to constitute POMDP with continuous action space?

I have looked at samples on POMDPGallery.jl and POMDPModels.jl.
However, they seem to be designed to have a discrete action space.

[tyoshidais]

I found that VDPTag2.jl has a continuous action space.

However, its source code is so complicated for beginners like me.
Are there some simple examples having a continuous action space?

Thank you.

[WhiffleFish]

Hi @tyoshidais. Currently for POMDPs.jl there is no single standardized way of defining a continuous action space. How you define the space depends on the solver you intend to use.

For VDPTag you linked is intended for continuous action space compatible tree search solvers such as ParticleFilterTrees, or POMCPOW. The only thing these solvers need from the continuous action space is to be able to sample from it via rand. So, you can do it the same way if your intention is to use one of those solvers.

As simple example, here's something that gives you a continuous space of numbers between 0 and 10:

struct MyContinuousActionSpace end
POMDPs.actions(::MyPOMDP) = MyContinuousActionSpace()
Base.rand(rng, ::MyContinuousActionSpace) = rand(rng)*10

[zsunberg]

A small note on what @WhiffleFish mentioned. You may actually want to implement

Base.rand(rng::AbstractRNG, ::Random.SamplerTrivial{MyContinuousActionSpace}) = rand(rng)*10

as explained here: https://docs.julialang.org/en/v1/stdlib/Random/#A-simple-sampler-without-pre-computed-data

[tyoshidais]

Thank you for the advices.

I implemented the example @WhiffleFish indicated, and its works correctly on the POMCPOW.

Are most of the implemented methods incompatible for continuous action spaces?

[zsunberg]

Are most of the implemented methods incompatible for continuous action spaces?

On the POMDPs.jl readme, there is a table for all the solvers indicating whether they can work on continuous action spaces.

You might also want to check this out: https://github.com/michaelhlim/VOOTreeSearch.jl/tree/main

[zsunberg]

It looks like only POMCPOW and ParticleFilterTrees work on continuous action spaces based on the readme. Continuous action POMDPs still need more research, and solving them with methods other than reinforcement learning is still at the edge of or beyond the state of the art.

[zsunberg]

You can also use other packages such as IntervalSets and DomainSets, i.e. for a continuous action space from 0 to 10 with a QuickPOMDP, you could use

using IntervalSets

m = QuickPOMDP(
    actions = 0.0..10.0
)

DomainSets should give you multi-dimensional options