Using MuJoCo in Julia?

[tyler-ingebrand]

Hello,

I've noticed there is a file for gym environments and MuJoCo in ReinforcementLearningEnvironments.jl/3rd_party. However, the source file gym.jl is not included in the module. When including it on my own, it throws errors at various points.

Is MuJoCo supported by ReinforcementLearning.jl? If so, is there a special process to activate gym environments? If not, are there plans to do so in the future?

Thanks,
Tyler

[findmyway]

Hi Tyler,

MuJoCo used to be supported through PyCall.jl. You need to have it installed in Python first. But I haven't tested for a long time.

Another approach is to use https://github.com/Lyceum/MuJoCo.jl directly. It's on the todo list in the next release. #614

[findmyway]

I've never seen such errors before. Could you also share which version of Julia and the version of RL.jl you are testing?

[tyler-ingebrand]

Julia 1.7.2
[158674fc] ReinforcementLearning v0.10.0
[de1b191a] ReinforcementLearningCore v0.8.11
[25e41dd2] ReinforcementLearningEnvironments v0.6.12

[438e738f] PyCall v1.93.1
[d330b81b] PyPlot v2.10.0

[tyler-ingebrand]

Whoops! It is throwing an error because the definition of GymEnv is found in the structs file.This updated code fixes the bug, although I still get an error when trying to load an environment listed in the file:

using ReinforcementLearning
using PyCall
using Random
using IntervalSets
include("/home/undergrad/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/structs.jl") # had to do this because it is not included otherwise!
include("/home/undergrad/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/gym.jl") # had to do this because it is not included otherwise!

env = GymEnv("gym.envs.mujoco.ant_v3")
reset!(env)

for _ in 1:1000
    state = state(env)
    action = map(x -> rand(x), action_space(env))
    env(action)
    render(env)

LoadError: Gym environment gym.envs.mujoco.ant_v3 not found.

Run `ReinforcementLearningEnvironments.list_gym_env_names()` to find supported environments.

Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] GymEnv(name::String; seed::Nothing)
   @ Main ~/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/gym.jl:14
 [3] GymEnv(name::String)
   @ Main ~/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/gym.jl:4
 [4] top-level scope
   @ ~/TylerIngebrand/AlgebraicMDP/test.jl:8
 [5] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [6] top-level scope
   @ REPL[1]:1
in expression starting at /home/undergrad/TylerIngebrand/AlgebraicMDP/test.jl:8

caused by: PyError ($(Expr(:escape, :(ccall(#= /home/undergrad/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'gym.error.NameNotFound'>
NameNotFound("Environment gym.envs.mujoco.ant_v3 doesn't exist. ",)
  File "/home/undergrad/.local/lib/python3.6/site-packages/gym/envs/registration.py", line 578, in make
    _check_version_exists(ns, name, version)
  File "/home/undergrad/.local/lib/python3.6/site-packages/gym/envs/registration.py", line 209, in _check_version_exists
    _check_name_exists(ns, name)
  File "/home/undergrad/.local/lib/python3.6/site-packages/gym/envs/registration.py", line 188, in _check_name_exists
    f"Environment {name} doesn't exist{namespace_msg}. {suggestion_msg}"

Stacktrace:
  [1] pyerr_check
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:62 [inlined]
  [2] pyerr_check
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:66 [inlined]
  [3] _handle_error(msg::String)
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/exception.jl:83
  [4] macro expansion
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:97 [inlined]
  [5] #107
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ ./c.jl:458 [inlined]
  [7] __pycall!
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{String}, nargs::Int64, kw::Ptr{Nothing})
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:29
  [9] _pycall!
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:11 [inlined]
 [10] #_#114
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:86 [inlined]
 [11] (::PyObject)(args::String)
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:86
 [12] GymEnv(name::String; seed::Nothing)
    @ Main ~/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/gym.jl:12
 [13] GymEnv(name::String)
    @ Main ~/.julia/packages/ReinforcementLearningEnvironments/Pzsys/src/environments/3rd_party/gym.jl:4
 [14] top-level scope
    @ ~/TylerIngebrand/AlgebraicMDP/test.jl:8
 [15] include(fname::String)
    @ Base.MainInclude ./client.jl:451
 [16] top-level scope
    @ REPL[1]:1

Still a bit weird I have to include the files manually. Structs is included into reinforcementLearning.jl, but structs does not include any of the other 3rd_party code, so gym is not actually usable.

[findmyway]

Other code under 3rd_party are loaded dynamically.

ReinforcementLearning.jl/src/ReinforcementLearningEnvironments/src/ReinforcementLearningEnvironments.jl

Lines 17 to 34 in 77b3376

	# dynamic loading environments
	function __init__()
	@require ArcadeLearningEnvironment = "b7f77d8d-088d-5e02-8ac0-89aab2acc977" include(
	"environments/3rd_party/atari.jl",
	)
	@require PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0" include(
	"environments/3rd_party/gym.jl",
	)
	@require OpenSpiel = "ceb70bd2-fe3f-44f0-b81f-41608acaf2f2" include(
	"environments/3rd_party/open_spiel.jl",
	)
	@require SnakeGames = "34dccd9f-48d6-4445-aa0f-8c2e373b5429" include(
	"environments/3rd_party/snake.jl",
	)
	@require OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed" include(
	"environments/3rd_party/AcrobotEnv.jl",
	)
	@require Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" include("plots.jl")

It's quite strange that you still need to manually include those files.

LoadError: Gym environment gym.envs.mujoco.ant_v3 not found.

This means that you haven't installed mujoco correctly through PyCall. You may need to make sure mujoco is installed correctly in the python environment specified in PyCall.jl

[findmyway]

I just did a quick test on the Mac with a fresh new environment of the packages you listed above, and at least I can load CartPole-v2 successfully.

(test) pkg> st
      Status `~/workspace/work/test/Project.toml`
  [438e738f] PyCall v1.93.1
  [158674fc] ReinforcementLearning v0.10.0

julia> using ReinforcementLearning

julia> using PyCall

julia> env = GymEnv("CartPole-v1")
# TimeLimit

## Traits

| Trait Type        |                  Value |
|:----------------- | ----------------------:|
| NumAgentStyle     |          SingleAgent() |
| DynamicStyle      |           Sequential() |
| InformationStyle  | ImperfectInformation() |
| ChanceStyle       |           Stochastic() |
| RewardStyle       |           StepReward() |
| UtilityStyle      |           GeneralSum() |
| ActionStyle       |     MinimalActionSet() |
| StateStyle        |     Observation{Any}() |
| DefaultStateStyle |     Observation{Any}() |

## Is Environment Terminated?

No

## State Space

`Space{Vector{IntervalSets.ClosedInterval{Float32}}}(IntervalSets.ClosedInterval{Float32}[-4.8..4.8, -3.4028235e38..3.4028235e38, -0.41887903..0.41887903, -3.4028235e38..3.4028235e38])`

## Action Space

`ZeroTo(1)`

## Current State

Float32[0.035740428, -0.046641443, 0.022965545, -0.032434437]

[tyler-ingebrand]

You are correct, CartPolve-v1 works, and my MuJoCo was installed wrong. Sorry about that.

Also, the files are actually loaded here: https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/blob/c67a604336923ee9120d75dee74456759860fcda/src/ReinforcementLearningEnvironments/src/ReinforcementLearningEnvironments.jl

So I was wrong about that too. Thanks for the help!!!!!