Skip to content

MDPs and POMDPs in Julia - An interface for defining, solving, and simulating fully and partially observable Markov decision processes on discrete and continuous spaces.

License

Notifications You must be signed in to change notification settings

dykim07/POMDPs.jl

 
 

Repository files navigation

POMDPs

Linux Mac OS X Windows
Build Status Build Status Build Status

Docs Dev-Docs Gitter Slack

This package provides a core interface for working with Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). The POMDPTools package acts as a "standard library" for the POMDPs.jl interface, providing implementations of commonly-used components such as policies, belief updaters, distributions, and simulators.

Our goal is to provide a common programming vocabulary for:

  1. Expressing problems as MDPs and POMDPs.
  2. Writing solver software.
  3. Running simulations efficiently.

POMDPs.jl integrates with other ecosystems:

For a detailed introduction, check out our Julia Academy course! For help, please post in GitHub Discussions tab. We welcome contributions from anyone! See CONTRIBUTING.md for information about contributing.

Installation

POMDPs.jl and associated solver packages can be installed using Julia's package manager. For example, to install POMDPs.jl and the QMDP solver package, type the following in the Julia REPL:

using Pkg; Pkg.add("POMDPs"); Pkg.add("QMDP")

Quick Start

To run a simple simulation of the classic Tiger POMDP using a policy created by the QMDP solver, you can use the following code (note that POMDPs.jl is not limited to discrete problems with explicitly-defined distributions like this):

using POMDPs, QuickPOMDPs, POMDPModelTools, POMDPSimulators, QMDP

m = QuickPOMDP(
    states = ["left", "right"],
    actions = ["left", "right", "listen"],
    observations = ["left", "right"],
    initialstate = Uniform(["left", "right"]),
    discount = 0.95,

    transition = function (s, a)
        if a == "listen"
            return Deterministic(s) # tiger stays behind the same door
        else # a door is opened
            return Uniform(["left", "right"]) # reset
        end
    end,

    observation = function (s, a, sp)
        if a == "listen"
            if sp == "left"
                return SparseCat(["left", "right"], [0.85, 0.15]) # sparse categorical distribution
            else
                return SparseCat(["right", "left"], [0.85, 0.15])
            end
        else
            return Uniform(["left", "right"])
        end
    end,

    reward = function (s, a)
        if a == "listen"
            return -1.0
        elseif s == a # the tiger was found
            return -100.0
        else # the tiger was escaped
            return 10.0
        end
    end
)

solver = QMDPSolver()
policy = solve(solver, m)

rsum = 0.0
for (s,b,a,o,r) in stepthrough(m, policy, "s,b,a,o,r", max_steps=10)
    println("s: $s, b: $([s=>pdf(b,s) for s in states(m)]), a: $a, o: $o")
    global rsum += r
end
println("Undiscounted reward was $rsum.")

For more examples with visualization see the documentation below and POMDPGallery.jl.

Documentation and Tutorials

In addition to the above-mentioned Julia Academy course, detailed documentation can be found here.

Docs Docs

Several tutorials are hosted in the POMDPExamples repository.

Supported Packages

Many packages use the POMDPs.jl interface, including MDP and POMDP solvers, support tools, and extensions to the POMDPs.jl interface. POMDPs.jl and all packages in the JuliaPOMDP project are fully supported on Linux. OSX and Windows are supported for all native solvers*, and most non-native solvers should work, but may require additional configuration.

Tools:

POMDPs.jl itself contains only the core interface for communicating about problem definitions; these packages contain implementations of commonly-used components:

Package Build Coverage
POMDPTools (hosted in this repository) Build Status
ParticleFilters Build Status codecov.io

Implemented Models:

Many models have been implemented using the POMDPs.jl interface for various projects. This list contains a few commonly used models:

Package Build Coverage
POMDPModels CI Coverage Status
LaserTag CI Coverage Status
RockSample CI Coverage Status
DroneSurveillance Build status codecov
ContinuumWorld CI Coverage Status
VDPTag Build Status
Roomba Localization CI

MDP solvers:

Package Build/Coverage Online/
Offline
Continuous
States - Actions
Rating3
Value Iteration Build Status
Coverage Status
Offline N-N ★★★★★
Local Approximation Value Iteration Build Status
Coverage Status
Offline Y-N ★★
Global Approximation Value Iteration Build Status
Coverage Status
Offline Y-N ★★
Monte Carlo Tree Search Build Status
Coverage Status
Online Y (DPW)-Y (DPW) ★★★★

POMDP solvers:

Package Build/Coverage Online/
Offline
Continuous
States-Actions-Observations
Rating3
QMDP (suboptimal) Build Status
Coverage Status
Offline N-N-N ★★★★★
FIB (suboptimal) Build Status
Coverage Status
Offline N-N-N ★★
BeliefGridValueIteration Build Status
codecov
Offline N-N-N ★★
SARSOP* Build Status
Coverage Status
Offline N-N-N ★★★★
NativeSARSOP Build Status
Coverage Status
Offline N-N-N ★★★★
ParticleFilterTrees (SparsePFT, PFT-DPW) Build Status
codecov
Online Y-Y2-Y ★★★
BasicPOMCP Build Status
Coverage Status
Online Y-N-N1 ★★★★
ARDESPOT Build Status
Coverage Status
Online Y-N-N1 ★★★★
AdaOPS Build Status
codecov.io
Online Y-N-Y ★★★★
MCVI Build Status
Coverage Status
Offline Y-N-Y ★★
POMDPSolve* Build Status
Coverage Status
Offline N-N-N ★★
IncrementalPruning Build Status
Coverage Status
Offline N-N-N ★★★
POMCPOW Build Status
Coverage Status
Online Y-Y2-Y ★★★
AEMS Build Status
Coverage Status
Online N-N-N ★★
PointBasedValueIteration Build status
Coverage Status
Offline N-N-N ★★

1: Will run, but will not converge to optimal solution

2: Will run, but convergence to optimal solution is not proven, and it will likely not work well on multidimensional action spaces

Reinforcement Learning:

Package Build/Coverage Continuous
States
Continuous
Actions
Rating3
TabularTDLearning Build Status
Coverage Status
N N ★★
DeepQLearning Build Status
Coverage Status
Y1 N ★★★

1: For POMDPs, it will use the observation instead of the state as input to the policy.

3 Subjective rating; File an issue if you believe one should be changed

  • ★★★★★: Reliably Computes solution for every problem.
  • ★★★★: Works well for most problems. May require some configuration, or not support every edge of interface.
  • ★★★: May work well, but could require difficult or significant configuration.
  • ★★: Not recently used (unknown condition). May not conform to interface exactly, or may have package compatibility issues
  • ★: Not known to run

Performance Benchmarks:

Package
DESPOT

*These packages require non-Julia dependencies

Citing POMDPs

If POMDPs is useful in your research and you would like to acknowledge it, please cite this paper:

@article{egorov2017pomdps,
  author  = {Maxim Egorov and Zachary N. Sunberg and Edward Balaban and Tim A. Wheeler and Jayesh K. Gupta and Mykel J. Kochenderfer},
  title   = {{POMDP}s.jl: A Framework for Sequential Decision Making under Uncertainty},
  journal = {Journal of Machine Learning Research},
  year    = {2017},
  volume  = {18},
  number  = {26},
  pages   = {1-5},
  url     = {http://jmlr.org/papers/v18/16-300.html}
}

About

MDPs and POMDPs in Julia - An interface for defining, solving, and simulating fully and partially observable Markov decision processes on discrete and continuous spaces.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Julia 100.0%