Adds camera to G1 Steering Wheel environment #3549

jaybdub · 2025-09-24T22:21:01Z

Description

This PR adds the following changes

Adds support for rendering images to Locomanipulation SDG envs
Renames DisjointNav -> LocomanipulationSDG in docs and code

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (existing functionality will not work without user modification)
Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

yanchangNvidia · 2025-10-02T21:25:49Z

Please add how you tested these changes.

yanchangNvidia · 2025-10-02T21:53:50Z

@ooctipus to review.

jaybdub · 2025-10-03T01:57:00Z

@yanchangNvidia -- These changes were tested by

Running the datageneration pipeline with sample data
Plotting the output trajectory
Viewing images from dataset

yanchangNvidia

Overall looks great. Thank you for the changes. Left a few comments.

yanchangNvidia · 2025-10-03T02:34:53Z

docs/source/overview/imitation-learning/teleop_imitation.rst

+you may convert the generated dataset to LeRobot format as expected by GR00T N1.5, and then run the finetuning script provided
+in the GR00T N1.5 repository.  An example closed-loop policy rollout is shown in the video below:
+
+.. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/locomanipulation_sdg_disjoint_nav_groot_policy_4x.gif


Could we change this gif name to "locomanipulation_sdg_groot_policy_4x.gif"?

We can but we need to ask for these images to be re-uploaded, I don't have direct access to do this.

Got it. Who can help us to get these images re-uploaded?

docs/source/overview/imitation-learning/teleop_imitation.rst

yanchangNvidia · 2025-10-03T02:38:19Z

scripts/imitation_learning/locomanipulation_sdg/README.md

+## Usage
+
+To run the locomanipulation SDG replay script execute the following command.
+


Do we need another step of running manipulation trajectory augmentation before the navigation trajectory augmentation?

It is not needed, but desired. The script is generic, it can use the teleop dataset directly, or instead use the manipulation augmented trajectory.

Got it. It would be great to add this information as a part of the documentation so that users know that they can use have another step to do the manipulation augmented trajectory before this step to scale up the data.

scripts/imitation_learning/locomanipulation_sdg/README.md

...asks/isaaclab_tasks/manager_based/locomanipulation/pick_place/locomanipulation_g1_env_cfg.py

scripts/imitation_learning/locomanipulation_sdg/README.md

source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/envs/__init__.py

docs/source/overview/imitation-learning/teleop_imitation.rst

Mayankm96 · 2025-10-10T06:05:06Z

...asks/isaaclab_tasks/manager_based/locomanipulation/pick_place/locomanipulation_g1_env_cfg.py

        spawn=sim_utils.DomeLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
    )

+    robot_pov_cam = CameraCfg(


FYI this is not the tiled camera class. For that, you need to use TiledCamera.

@Mayankm96 Great catch. @jaybdub Will this affect the data generation pipeline throughput without TiledCamera?

I accidentally named the USD path "tiled_camera" based on an older version where I used TiledCamera.

However, I updated this to use the Camera class to emulate the existing manipulation environments:

IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/pick_place/nutpour_gr1t2_base_env_cfg.py

Line 161 in 2d80bd8

robot_pov_cam = CameraCfg(

The code is intended, so I've updated the USD path so the name is "camera" to make it less confusing.

Mayankm96

I assume that others have checked that the code works correctly and all the documentation is correct as well. Left a minor note on TiledCamera vs Camera.

…rding

michaellin6

Changes look good to me. Need to address Mayank's comment on TiledCamera, but besides that it LGTM

# Description #3549 introduced some linter issues. fixing them here with the formatter. ## Type of change  - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there

jaybdub added 3 commits September 24, 2025 14:42

add camera to g1 locomanipulation env

a714181

run format

72dcbd8

update docs and readme

f72371d

jaybdub requested review from Mayankm96, jtigue-bdai and kellyguo11 as code owners September 24, 2025 22:21

github-actions bot added documentation Improvements or additions to documentation isaac-mimic Related to Isaac Mimic team labels Sep 24, 2025

jaybdub added 2 commits September 25, 2025 12:58

add gr00t N1.5 note with GIF placeholder

f9a9931

add gif of locomanipulation policy rollout

9541cdb

jaybdub changed the title ~~add camera to g1 locomanipulation env~~ add camera to g1 locomanipulation env and update docs with G1.5 training note Oct 2, 2025

rename disjoint nav -> locomanipulation SDG

7bb37eb

yanchangNvidia reviewed Oct 3, 2025

View reviewed changes

jaybdub added 2 commits October 3, 2025 15:31

address PR feedback

bcb835a

fix hdf5 output path

aafe10c

Mayankm96 reviewed Oct 7, 2025

View reviewed changes

scripts/imitation_learning/locomanipulation_sdg/README.md Outdated Show resolved Hide resolved

Mayankm96 reviewed Oct 7, 2025

View reviewed changes

source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/envs/__init__.py Outdated Show resolved Hide resolved

Mayankm96 reviewed Oct 7, 2025

View reviewed changes

docs/source/overview/imitation-learning/teleop_imitation.rst Outdated Show resolved Hide resolved

jaybdub added 2 commits October 7, 2025 10:49

Address PR comments

320119a

rename task

7c07e97

Mayankm96 changed the title ~~add camera to g1 locomanipulation env and update docs with G1.5 training note~~ Adds camera to G1 Steering Wheel environment Oct 10, 2025

Mayankm96 reviewed Oct 10, 2025

View reviewed changes

Mayankm96 approved these changes Oct 10, 2025

View reviewed changes

Mayankm96 added enhancement New feature or request and removed documentation Improvements or additions to documentation labels Oct 10, 2025

Merge branch 'release/2.3.0' into disjoint-navigation-add-camera-reco…

434a21b

…rding

github-actions bot added the documentation Improvements or additions to documentation label Oct 13, 2025

yanchangNvidia approved these changes Oct 14, 2025

View reviewed changes

michaellin6 approved these changes Oct 14, 2025

View reviewed changes

rename camera usd path

c610593

kellyguo11 merged commit 3de863a into isaac-sim:release/2.3.0 Oct 14, 2025
1 of 2 checks passed

kellyguo11 mentioned this pull request Oct 14, 2025

Fixes code formatting for locomanipulation PR #3710

Merged

7 tasks

		## Usage

		To run the locomanipulation SDG replay script execute the following command.

Adds camera to G1 Steering Wheel environment #3549

Adds camera to G1 Steering Wheel environment #3549

Uh oh!

Conversation

jaybdub commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Screenshots

Checklist

Uh oh!

yanchangNvidia commented Oct 2, 2025

Uh oh!

yanchangNvidia commented Oct 2, 2025

Uh oh!

jaybdub commented Oct 3, 2025

Uh oh!

yanchangNvidia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yanchangNvidia Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mayankm96 left a comment

Choose a reason for hiding this comment

Uh oh!

michaellin6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jaybdub commented Sep 24, 2025 •

edited

Loading

yanchangNvidia Oct 14, 2025 •

edited

Loading