-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
When running Ray directly from tuner.py
, Ray is not correctly initialized within invoke_tuning_run()
. Particularly, there are two problems:
- The following lines, which are supposed to initialize Ray, are actually never used, as
util.get_gpu_node_resources()
initializes Ray before.
IsaacLab/scripts/reinforcement_learning/ray/tuner.py
Lines 224 to 229 in 3a0db9d
if not ray.is_initialized(): | |
ray.init( | |
address=args.ray_address, | |
log_to_driver=True, | |
num_gpus=len(resources), | |
) |
util.get_gpu_node_resources()
initializes Ray within, withlog_to_driver = False
, which is wrong as it prevents the subprocess output from reaching the main process (e.g., the debug lines within theTrainable
step()
function are gone). The original behaviour of Ray was the other way around.
Steps to reproduce
Run Ray from tuner.py according to the tutorial documentation, you won't see the debug lines within the Trainable
step()
function.
System Info
- Commit: 3a0db9d
- Isaac Sim Version: 5.0
- OS: Ubuntu 22.04
- GPU: 4090 RTX
- CUDA: 12.2
- GPU Driver: 535.129.03
Additional context
Previously, I had prepared #3350, which was merged to main, but this PR was not the correct way. I will create a new PR to fix the issue.
Checklist
- I have checked that there is no similar issue in the repo (required)
- I have checked that the issue is not in running Isaac Sim itself and is related to the repo
Acceptance Criteria
Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.
- Ray is initialized correctly for
invoke_tuning_run()
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working