2

I am building an environment in the maintained fork of gym: Gymnasium by Farama. In my gym environment, I state that the action_space = gym.spaces.Discrete(5) and the observation_space = gym.spaces.MultiBinary(25). Running the environment with the agent-environment loop suggested on the Gym Basic Usage website runs with no problems: I registered the environment and it is simply callable by gym.make().

However, I want to now train a reinforcement learning agent on this environment. Now I have come across Stable Baselines3, which makes a DQN agent implementation fairly easy. However, it does seem to support the new Gymnasium. Namely:

import gymnasium as gym
from stable_baselines3.ppo.policies import MlpPolicy
from stable_baselines3 import DQN

env = gym.make("myEnv")
model = DQN(MlpPolicy, env, verbose=1)

Yes I know, "myEnv" is not reproducable, but the environment itself is too large (along with the structure of the file system), but that is not the point of this question

This code produces an error:

AssertionError: The algorithm only supports (<class 'gym.spaces.discrete.Discrete',) as action spaces but Discrete(5) was provided

My question is the following: does Stable Baselines3 support Gymnasium?

I have tried to instead use gym.spaces in order to define the action_space and observation_space, such that

from gym.spaces import Discrete, MultiBinary
action_space = Discrete(5)
observation_space = MultiBinary(25)

but along with this, I have to rewrite a large portion of the environment to support the old gym package. I wonder whether there is a better solution than that.

Lexpj
  • 921
  • 2
  • 6
  • 15

2 Answers2

3

I was a bit confused by the other answer here as I was sure I'd seen gymnasium in the Stable Baselines 3 docs somewhere. Sure enough, it's even in the most basic "getting started" example: https://stable-baselines3.readthedocs.io/en/master/guide/quickstart.html.

However, I just tried running exactly this code and received the same error as the OP. Replacing gymnasium with gym 0.21 in the same example works without a problem.

Edit: The "getting started" example using gymnasium works with stable_baselines3 version 2.0.0a1 and above.

42bsk
  • 76
  • 1
  • 10
  • I see that I forgot to denote that in my OP, but I have been using gym >= 0.26. I guess there are some inconsistances between 0.21 and 0.26 if we are talking about stable baselines 3. I remember switching to 0.26 since the interchange between rendering in the agent-environment loop and gym.env are inconsistent between these versions. Good to denote though, that it apparently does work with 0.21! – Lexpj May 05 '23 at 08:46
  • 1
    Not sure if you saw my edit, but your code should work with gymnasium as well if you upgrade to version 2.0.0a1 or later. – 42bsk May 05 '23 at 14:38
  • Right! I marked your thread as answer – Lexpj May 06 '23 at 16:46
  • 1
    For the record, I think the other answer was correct when it was posted :) – 42bsk May 06 '23 at 18:47
1

does Stable Baselines3 support Gymnasium?

If you look into setup.py, you will see that a master branch as well as a PyPI release are both coupled with gym 0.21.

However, there is a branch with a support for Gymnasium. I haven't tested it yet, I think it is going to be merged into master in several weeks, then I will migrate myself. So far this is the current state of the art.

gehirndienst
  • 424
  • 2
  • 13
  • Thank you for your answer! Do you possibly know an estimate of the date? Like, do you know if several weeks is around a month or do we look at three months? – Lexpj Mar 28 '23 at 09:23
  • 1
    taking into account that the current state of the corresponding PR is "draft" -- I wouldn't expect it to be merged earlier than this summer – gehirndienst Mar 29 '23 at 14:55