action space dependent on state function

[danortega2014]

So I have several continuous variables which affect whether or not certain actions are available. I have 15 discrete actions. Below is my action state dependent function which I believe is the problem as I get the error 'ERROR: MethodError: no method matching actions(::firmgame)' and everything worked prior to trying to implement this new function. My thought was to assign my actions as nothing until they meet certain state requirements, however I don't think I implemented it right. The return statement is meant to rid the vector of actions who are still labeled as nothing. If anyone could help that would be greatly appreciated!

action function:

function POMDPs.actions(m::firmgame, s::Tuple{Float64,Float64,Float64,Float64,Float64,Float64,Float64}) 
    nomprice, wealth, debt, capital, interest, labor, inventory, consumption, timestep  = s
    a1 = nothing
    a2 = nothing 
    a3 = nothing 
    a4 = nothing 
    a5 = nothing
    a6 = nothing
    a7 = nothing 
    a8 = nothing 
    a9 = nothing 
    a10 = nothing 
    a11 = nothing
    a12 = nothing 
    a13 = nothing 
    a14 = nothing 
    a15 = nothing  
    while inventory >= 1.0 
        a1 =:np2 
    end 
    
    while inventory >= 2.0
        a2 =:np1 
    end

    while inventory >= 3.0
        a3 =:p0 
    end

    while inventory >= 4.0
        a4 =:p1 
    end 

    while inventory >= 5.0
        a5 =:p2
    end

    while  wealth >= (50.0 + debt*interest/6.0)
        a6 = :smallinvest 
    end

    while wealth >= (100.0 + debt*interest/6.0) 
        a7 == :largeinvest
    end

    while wealth >= 0.0 && debt <= 1000
        a8 = :smallborrow 
        a9 = :largeborrow 
    end


    while debt >=  50 && wealth >= (50.0 + debt*interest/6.0)
        a10 = :smallpayback
    end

    while debt >= 100.0  && wealth >= (100.0 + debt*interest/6.0)
        a11 = :largepayback 
    end

    while labor >= 12.0  
        a12 =:smallcraft 
    end


    while labor >= 24.0  
        a13 =:largecraft 
    end


    while wealth >= (50.0+.10*wealth) + debt*(1.0+interest/6.0)
        a14 =:smallconsume
    end

    while wealth >= (100.0 +.2*wealth) + debt*(1.0+interest/6.0)
        a15 =:largeconsume 
    end

    m = [a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11,a12,a13,a14,a15]
    return (x->x!=nothing, m)
end

[lassepe]

Hello @danortega2014,

I suspect that you are using a solver that does not support state-dependent actions and thus internally calls POMDPs.action(m). Unfortunately, not all solvers support state dependent actions: e.g. DiscreteValueIteration.jl wouldn't work, while MCTS.jl and related solvers should work.

Beyond that, there is a small error in your implementation: the expression return (x->x!=nothing, m) returns a tuple of a function and the array of actions. If you want to filter the list of actions by the function you can use filter(!isnothing, m) instead.

---

@zsunberg given that this question has popped up a couple of times, it may be a good idea to add a column to the table in the README or the docs that shows which solver supports this. Currently, the only way to find that out seems to be by looking at the code.

[danortega2014]

Here's all the code if that helps: https://github.com/danortega2014/test2/blob/main/README.md

[lassepe]

Yes, this is mainly a representation problem in your case: In deep Q earning you train a value network that predicts the value of each of the actions given the current observation or state (depending on whether you solve an POMDP or MDP). In any case, the network needs to have a fixed output size (at least in the current implementation) and so you cannot have a different number of actions for different states (because the Q-value-network is shared across all states).

So if you want to use DeepQLearning you would have to modify the solver code in order support state dependent actions. A hot fix could be to also implement POMDPs.actions(m) for your problem that returns all possible actions (also potentially invalid ones) and then filter out illegal ones on the fly in DeepQLearning._action(policy, o) and DeepQLearning._actionvalues. A more elegant fix would be to modify the Q-value-network to take a representation of the action candidate as an input so that you can predict the value only for valid actions; and this could be done efficiently by passing all valid action encodings in as a batch.

Alternatively, you can of course also "soft constraint" illegal actions by giving a large negative reward for them (but that is not very satisfying).

[danortega2014]

Thank you for your prompt and well-written response. I am currently doing the last option which has caused training time to extremely long, I would like to try and implement the hotfix option you mentioned. How do you filter out illegal ones on the fly in DeepQlearning.action(policy, s)/._actionvalues? I'm a bit confused on how you would do that? Again, I really appreciate the help!

[lassepe]

I don't have a complete answer for you since I have not written that code but the main things that would need to happen for that would be to the following:

DeepQLearning._action computes the value of each action given the current observation or state. It then indexes into the action map to select the action with the highest value. Rather than finding the argmax (i.e. the index of the array) that has the highest value, you only consider those actions that are valid. Qualitatively, you would use some more efficient version of something like this:

mask = [is_valid(m, o, a) for a in policy.action_map]
vals = policy.qnetwork(obatch)[mask]
return policy.action_map[mask][argmax(valid_vals)]

Where your problem would have to implement is_valid(m, o, a) (or you compute it from the set difference between POMDPs.actions(m) and POMDPs.actions(m, o) but that may be too slow for large problem).

A similar pattern should also work for _value and _actionvalues. Though in that case I would not simply filter them out but rather return -Inf for illegal actions.

[zsunberg]

Thanks for answering this, @lassepe !

A couple of notes: 1. If you're dealing with performance issues, make sure to profile to see what is taking the time! 2. CommonRLInterface.jl has masking in its interface, so it may be useful to use that.