Performance for RL training of humanoid

[ucacaxm]

Hi theres,

I know the benchmark presented here
https://github.com/zhouxian/genesis-speed-benchmark?tab=readme-ov-file

I used ml-agent a lot (PhysX+Unity). I trained an humanoid with million of iterations
in several hours with several envs in parallel (about 40h for 100 millions of iteration with
16 envs in parallel on a 5 years old computer, i7, RTX3060).

I was impressed by the learning time of Brax with Mujoco MJX on the humanoid : few minutes.

The GPU parallelization seems a good direction, then I decided to move my code to Genesis
which seems promising in the long term.

With code inspired from go2flip I am trying to train an humanoid on simple moves (with rsl_rl).
For 10000 iterations with batch of 1024 the ETA: 39786.1s => 11h
The performance seems slow ... since 10000 iterations is not a lot.

Did I miss something in the configuration?
size of the batch ?

Do you know any repo with RL and humanoid ?

Thanks,

[jgillick]

Can you post the code for your go2flip inspired humanoid environment, and the accompanying training configuration? I personally haven't seen any examples of genesis humanoid robots, but I would assume the performance would not be too much slower than a quadruped robot.

Things I've found that can affect performance:

• Number of individual geoms/bodies in mujoco XML -- if possible, merge bodies that have rigid joints.
• Number of collision pairs
  • If you need collisions, you can reduce the maximum number of collision pairs (example)
  • If possible, disable self_collsiion in RigidOptions, or disable collisions altogether.
• Rough terrain can slow things down (this is probably related to the number of faces + collision detection)
• Rendering
  • Rendering a camera sensor can slow down performance
  • Be sure to reduce the number of environments being visualized even if the viewer is disabled (example)
• Complex reward functions.

[jgillick]

Also be sure you're on the latest version of Genesis. There have been a number of important fixes since 0.3.3. Version 0.3.4 was released today and should have the latest and greatest.

[ucacaxm]

Thanks for your answers.

Everything is a draft in my code: I just wanted to try and measure the perf. I attached it.
loco.zip

I moved to rsl_rl version 3.1.0. My humanoid is just a trunk and 2 legs.
The humanoid learns to stay up right easily.
I just changed a bit the reward to force him to lift his left leg a little.
It does not learn yet, the reward was written quickly just to test the performance.

If you need collisions, you can reduce the maximum number of collision pairs

30

If possible, disable self_collsiion in RigidOptions, or disable collisions altogether.

I have to check that.

Rough terrain can slow things down

My ground is flat

Rendering a camera sensor can slow down performance

I do not use this.

Be sure to reduce the number of environments being visualized even if the viewer is disabled ([example]

Yes I have this vis_options=gs.options.VisOptions(rendered_envs_idx=list(range(1)))

Also be sure you're on the latest version of Genesis. There have been a number of important fixes since 0.3.3.
Version 0.3.4 was released today and should have the latest and greatest.

Thanks, I have to try that, also.

[duburcqa]

Be sure to reduce the number of environments being visualized even if the viewer is disabled

I think this particular issue has been fixed. This viewer (and rasterising context in particular) should not be updated anymore if there is no camera and the interactive viewer is disable.

[duburcqa]

Everything is a draft in my code: I just wanted to try and measure the perf. I attached it.

I will have a look if I have time. Thank you for the report.

[jgillick]

@ucacaxm I quickly took a look at your code and noticed that when I tried to open your model in the Mujoco Viewer, it gave me the following error: "mass and inertia of moving bodies must be larger than mjMINVAL". So it's possible that there could be something wrong with the configuration of your model that might be putting genesis in a weird place.

I couldn't stop thinking about your question about existing Genesis humanoid examples, so I created one using the Berkeley humanoid robot. You can check it out here: https://github.com/jgillick/genesis-forge/tree/main/examples/berkeley_humanoid
On my NVIDIA 5070ti It performed similarly to the quadruped environments, and was able to teach the robot to walk in ~15 minutes.

Give it a try:

• Clone the genesis-forge repo: https://github.com/jgillick/genesis-forge.git
• Install it:
  • cd genesis-forge
  • pip install .
• Run the example:
  • cd examples/berkeley_humanoid
  • python ./train.py

I'm curious how this works for you. For me, for some reason, it runs well on my NVIDIA machine, but it doesn't work at all on my mac. (for reference, all the other examples work fine on my mac). I haven't looked too deeply into that yet.

[ucacaxm]

Thanks a lot, I'll have a look.

[ucacaxm]

Yes your code is learning and walking at the end (40min on my old i7-12700K / RTX3060, 4 seconds by iteration).

Thanks, I will draw inspiration from it.
I have to change my habits with num_env at 4096, there's no need to iterate much.

Also, you're right for many link the "inertial" section was missing.
I have now to improve my reward funtion (I knew that before posting).

Many thanks

[andrevic1]

Hi,

At the end were you able to draw a comparison with Mujoco MJX in terms of performances on this specific task?

Thanks!

[jgillick]

Great, I'm glad this helps. Good luck!