Create MOMDPs package based on DDNStructure functionality

[zsunberg]

This is a continuation of https://github.com/JuliaPOMDP/POMDPs.jl/issues/282 . Please see that issue for background.

[zsunberg]

I looked at the PomdpX file format, and there are a few issues that we will have to think about:

1. It requires knowledge of whether a variable is a "StateVar", so we have to think about how to encode that
2. Each "StateVar" corresponds to two DDN vertices: "vnamePrev", and "vnameCurr", so we need to think about how to link those.
3. Each "StateVar", "ObsVar", and "ActionVar" has its own associated space - the state, action, and observation spaces are the cartesian products of the spaces for the individual variables.

[afansi]

@zsunberg Giving the constraints that we are imposing ourselves ( [1] the MOMDP should look like a POMDP from the outside so that vanilla POMDP solvers can work without knowing anything about MOMDPs, and [2] It should be straightforward to define a MOMDP explicitly or with a generative model), I think one simplest way to go is to replicate the exact functionality of the POMDP while enforcing the constraints, especially those related to the DBN structure, of the MOMDP.

We could go generic and allow that a given user defines any number of components as part of the state while deciding which ones are completely visible or not. However, this may require a lot changes.

On the other hand, we can try to mimic the DBN of the MOMDP by imposing 2 such components (one completely visible and the other not). I agree that we may need an abstract class StateNode as a base class for the state :s and :sp which can then be extended to support the state of an MOMDP which thus contain 2 components (let say :x and :y). To be fully compliant with POMDP, we should keep transition and observation functions. While the observation function need to be defined as usual like in the case of POMDP, the transition function could be predefined internally and would depend on two functions that need to be defined by the user: obs_state_transition and hid_state_transition (I confess, I am not good at choosing names :) ). As expected, the obs_state_transition will focus on the observable part of the state (here, :x) and will return a distribution probability on the possible values of :x. Its signature would be something like obs_state_transition(problem::MOMDP, state, action). On the other hand the hid_state_transition will focus on the hidden part of the state (here, :y) and will return a distribution probability on the possible values of :y. Its signature would be something like hid_state_transition(problem::MOMDP, state, action, obs_part). Finally, transition function would be implemented as (python snippet):

def transition(state, action):
   final_prob = []
   final_states = []
   obs_part_distrib = obs_state_transition(state, action)
   for x,p in obs_part_distrib:
       hid_part_distrib = hid_state_transition(state, action, x)
       for y, q in hid_part_distrib:
           final_prob.append(p*q) 
           final_states.append(StateNode(x,y))
   return sparsecat(final_prob, final_states)

This is for the compliance with POMDP. But we also need to think about the specificity of MOMDPs and provides interfaces that MOMDPs solvers will leverage. One of such example is the belief. As some part of the state is observable, we only need the belief on the hidden part. to make sure we are using all the info regarding the hidden part of the state from the transition function, we may need to have a function like the following one which aggregate the uncertainties regarding the hidden part from the observed part:

def hidden_transition_aggregation(state, action):
   final_prob = Dict()
   obs_part_distrib = obs_state_transition(state, action)
   for x,p in obs_part_distrib:
       hid_part_distrib = hid_state_transition(state, action, x)
       for y, q in hid_part_distrib:
           if not y in final_prob:
                final_prob[y] = q*p
           else:
                final_prob[y] = final_prob[y] + q*p
   return sparsecat(final_prob.values(), final_prob.keys())

More thoughts to come later!

[zsunberg]

Hi @afansi , thanks for the thoughts! I think I have an idea of how to do it with fewer unique / long names. Will respond tomorrow.

[zsunberg]

Hi everyone, sorry for the delays. After thinking about this some more, I think problem definitions for MOMDPs will actually look somewhat different from POMDPs, but will still be compatible. The new definitions will be very focused on the Variables/DDNs. In the long run, POMDPs.jl definitions themselves may start to look more like this, but for now, we can experiment here. This is heavily inspired by the POMDPX file format.

I haven't had time to work out the exact syntax, but here is the conceptual idea. A MOMDP definition will include

• A list of state variable names ((:x, :y) by default)
• A list of pure observation (i.e. information not stored in the state) variable names ((:z) by default)
• Metadata for all variables: Types, conditional dependence
• Spaces for each variable (perhaps specified with space(::Variable{:x}, ::MyMOMDP)
• Conditional distributions for each variable (perhaps specified with distribution(::Variable{:x}, ::MyMOMDP, inputs...)

The MOMDPs package will contain code that can package this into something that looks like a POMDP (i.e. with a DDN that contains :s, :o, and :sp) by taking the cross product of the spaces, integrating the probability distributions, etc.

I'm not even sure if MOMDP should actually be a subtype of POMDP, or whether we should instead supply a convert(::Type{POMDP}, ::MOMDP) function instead.

[zsunberg]

btw I don't yet know how python will fit into this :)

Also, what is a simple MOMDP that we can prototype with? I guess the simple 3-state rock sample https://bigbird.comp.nus.edu.sg/pmwiki/farm/appl/index.php?n=Main.PomdpXDocumentation would work well