[Bug Report] gymnasium.utils.play for gymnasium-robotics environments

[haidertom]

Describe the bug

I am trying to use the function gymnasium.utils.play.play with mujoco-based environments. However, I get this error.

gymnasium/utils/play.py:137: RuntimeWarning: invalid value encountered in divide
arr = 255.0 * (arr - arr_min) / (arr_max - arr_min)

It appears that after initializing the pygame display, (https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/utils/play.py#L65)

the mujoco rendering (https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/envs/mujoco/mujoco_rendering.py#L232) returns all zeros.

Any ideas how to resolve this?

Code example

import gymnasium as gym
import numpy as np
from gymnasium.utils.play import play

env = gym.make("FetchPickAndPlace-v2", render_mode="rgb_array")
print(env.action_space)
play(
    env,
    keys_to_action={
        "a": np.array([1, 0, 0, 0], dtype=np.float32),
    },
    noop=np.array([0, 0, 0, 0], dtype=np.float32),
)

System info

Ubuntu 22.04.3 LTS
python 3.10.13
gymnasium==0.28.1
gymnasium-robotics==1.2.2
mujoco==2.3.3

Additional context

No response

Checklist

• I have checked that there is no similar issue in the repo

[Kallinteris-Andreas]

Edit: I am an idiot 🥇 , I confused the repositories, reopening issue

Spoiler warning

From what I can tell, this also fails with gymnasium environments, so it is not an issue with `gymnasium_robotics`, you should report it to `gymnasium`, ```py import gymnasium as gym import numpy as np from gymnasium.utils.play import play

env = gym.make("InvertedPendulum-v5", render_mode="rgb_array")
print(env.action_space)
play(
env,
keys_to_action={
"a": np.array([1], dtype=np.float32),
},
noop=np.array([0], dtype=np.float32),
)

 I have no idea how `play()` works, and I can not help you
 </details>


https://github.com/Farama-Foundation/Gymnasium/blob/72cfbc204beca309579681b1201990a3d706e070/gymnasium/utils/play.py#L126-L146

[Kallinteris-Andreas]

Also fails for gymnasium.mujoco environments, issue is independent of gymnasium-robotics environment

import gymnasium as gym
import numpy as np
from gymnasium.utils.play import play

env = gym.make("InvertedPendulum-v5", render_mode="rgb_array")
print(env.action_space)
play(
    env,
    keys_to_action={
        "a": np.array([1], dtype=np.float32),
    },
    noop=np.array([0], dtype=np.float32),
)

[pseudo-rnd-thoughts]

Very strange, I dont get an issue with MacOS
@Kallinteris-Andresa what is the error that you get to "fail"?

For the warning, I don't think we should be normalising the arr, if the dtype is uint8
Therefore, I would propose we add a dtype check, assert isinstance(arr, np.ndarray) and arr.dtype == np.uint8
At the same time, we should add a check on the keys_to_action provided

    assert keys_to_action is not None

    # validate the `keys_to_action` set provided
    assert isinstance(keys_to_action, dict)
    for key, action in keys_to_action.items():
        if isinstance(key, tuple):
            assert len(key) > 0
            assert all(isinstance(k, (str, int)) for k in key)
        else:
            assert isinstance(key, (str, int))

        assert action in env.action_space

[Kallinteris-Andreas]

@pseudo-rnd-thoughts do the rendered window work? and do you get any warnings printed on the terminal?

$ py test2.py      
Box(-3.0, 3.0, (1,), float32)
/home/master-andreas/gym/cartpole/Gymnasium/gymnasium/utils/play.py:137: RuntimeWarning: invalid value encountered in divide
  arr = 255.0 * (arr - arr_min) / (arr_max - arr_min)
/home/master-andreas/.local/lib/python3.11/site-packages/pygame/surfarray.py:123: RuntimeWarning: invalid value encountered in cast
  array = array.round(0).astype(numpy_uint32)

[pseudo-rnd-thoughts]

do the rendered window work? and do you get any warnings printed on the terminal?

Yes and no

Might be because I'm on mac, I can also interact with the environment and its works

[elexunix]

I have the same issue:

import numpy as np
import gym
from gymnasium.utils.play import play

env = gym.make('InvertedPendulum-v4', render_mode='rgb_array')
env.reset()
keys_to_action = {
  (): np.array([0], dtype=np.float32),
  (ord('a'),): np.array([1], dtype=np.float32)
}
play(env, zoom=2, fps=40, keys_to_action=keys_to_action)

For BreakoutNoFrameskip-v4 (without custom keybindings, as they are present there), it works

OS: Ubuntu 22.04
Version of Python: 3.10.12
gym==0.26.2
gymnasium==0.29.1

The game window for this environment opens, but it is completely black, and the terminal shows that there were same errors with division etc, as described here by other users. For BreakoutNoFrameskip-v4, there are no errors and the game is playable

[Bpoole908]

I have a similar issue, except even after fixing the normalization issue, pygame still displays a black screen only for mujoco tasks (renders nothing even though the image received is displayed fine if displayed via matplotlib). Has there been any updates to this issue?

[pseudo-rnd-thoughts]

I have a similar issue, except even after fixing the normalization issue, pygame still displays a black screen only for mujoco tasks (renders nothing even though the image received is displayed fine if displayed via matplotlib). Has there been any updates to this issue?

Could you include a script to test with? Also, what OS are you using?

[Bpoole908]

Could you include a script to test with? Also, what OS are you using?

OS: Ubuntu 20.04.6 LTS

Yes, below I have attached a test script along with a conda environment. The test script proves, on my setup, the original issue and the issue of not being able to render to PyGame screen even if the original issue is avoided. I left some comments within the render_test.py script itself describing the behaviors when changing variables.

From what I can tell, as soon as you call env.render() using a Mujoco environment, it breaks the screen being displayed by PyGame. Maybe this relates to Mujoco using OpenGL for rendering? But ideally, I'd like to be able to capture the rgb image from
Mujoco and display it via PyGame, so I can utilize some of my existing PyGame code for capturing human demonstrations.

After digging around more, I believe my issues are related to gym issue and this Gymnasium merge.

mujoco-pygame-render.zip

[pseudo-rnd-thoughts]

Can you produce a minimally working example such that I don't need to download anything?

[Bpoole908]

Env

name: mujoco_pygame
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.9
  - pip
  - pip:
    - gymnasium[mujoco]
    - pygame>=2.1.0
    - numpy
    - matplotlib

Code

import numpy as np
import gymnasium as gym
import pygame
import matplotlib.pyplot as plt

# NOTE: The default behavior is for the pygame screen to display a red screen (never updated
#       with the image or green background) while matplotlib displays correct mujoco images.

info_str = ""
# Steps to take in environment
steps = 100
info_str += f"steps={steps}"
# Determines if first/init env.render() method is called (i.e., outside the step loop)
# NOTE: If enabled, pygame screen will remain red but project an empty image over the screen.
#       Additionally, matplotlib will display a black image as well as image values are all 0.
env_init_render = False
info_str += f"\nenv_init_render={env_init_render}"
# Determines if ALL env.render() method is called
# NOTE: If disabled, pygame screen will display red and then green correctly but no 
#       image will be disabled as env.render() is not called
env_render = True
info_str += f"\nenv_render={env_render}"
# Enable image rendering via pygame
use_pygame = False
info_str += f"\nuse_pygame={use_pygame}"
# Enable image rendering via matplotlib
use_mat = True
info_str += f"\nuse_mat={use_mat}"

print(info_str)

def pygame_init():
    pygame.init()
    pygame.display.init()
    display = (800, 600)
    screen = pygame.display.set_mode(display)
    return screen

env = gym.make("Ant-v4", render_mode="rgb_array")
env.reset()

if env_init_render and env_render:
    print("Running initial env.render()")
    env.render()
    
if use_pygame:
    screen = pygame_init()

if use_mat:
    fig = plt.figure()
    viewer = fig.add_subplot(111)

for i in range(steps):
    state_tuple = env.step(env.action_space.sample())
    
    # Render pygame screen green on first step to check when Pygame display is working
    # If it renders black or stays red, then screen is not being updated correctly
    # Thus, the first step should render a red screen followed by a green screen or the actual images
    if use_pygame:
        if i==0:
            screen.fill((255, 0, 0))
            pygame.display.flip() 
            pygame.time.wait(1000)
        else:       
            screen.fill((0, 255, 0))
            
    if env_render:      
        img = env.render()
        
        if i == steps//2:
            print(f"Max pixel: {np.max(img)}")
        
        if use_pygame:
            pyg_img = pygame.surfarray.make_surface(img)
            screen.blit(pyg_img, (0, 0))
        if use_mat:
            viewer.clear() 
            viewer.imshow(img)
            plt.pause(.01)
            fig.canvas.draw()
    
    if use_pygame:
        pygame.display.flip()

[pseudo-rnd-thoughts]

@Bpoole908 Can you test with gymnasium.utils.play rather than your own custom implementation as it is not possible to know if it is Gymnasium's, Pygame's or Mujoco's problem

[why-does-ie-still-exist]

I have a similar issue, I am trying to render a custom mujoco environment using the dm_robotics package and a camera sensor, and I am trying to use the play() method. I get a black screen on pygame, but if I stick a cv.imshow in my render() I can see the frames are being rendered fine. Similarly, I can follow the debugger into the play() method and see that the rendered image returned is in an appropriate range. Maybe this is some subtle incompatibility between pygame and mujoco, and it's not a gymnasium issue?

[pseudo-rnd-thoughts]

Could you provide a script to test with?

[Bpoole908]

I have a similar issue, I am trying to render a custom mujoco environment using the dm_robotics package and a camera sensor, and I am trying to use the play() method. I get a black screen on pygame, but if I stick a cv.imshow in my render() I can see the frames are being rendered fine. Similarly, I can follow the debugger into the play() method and see that the rendered image returned is in an appropriate rage. Maybe this is some subtle incompatibility between pygame and mujoco, and it's not a gymnasium issue?

I agree with this analysis after diving deeper into the issue. It seems to be some incompatibility with screen allocation between PyGame and Mujoco. I was hoping PyGame with SDL 2 might alleviate the issue but no luck as of yet.

@Bpoole908 Can you test with gymnasium.utils.play rather than your own custom implementation as it is not possible to know if it is Gymnasium's, Pygame's or Mujoco's problem

I have tested this with the gymnasium.utils.play and the same issue arises. Since both my script and play utilize PyGame, it is likely a PyGame/Mujoco conflict?

[pseudo-rnd-thoughts]

It is likely a mujoco and pygame issue but without someone doing a deep dive we can't be certain.
Annoyingly this means that we might not be able to anything in Gymnasium until either project updates / fixes their implementation