Skip to content

[Bug Report] Wrong ray initialization within invoke_tuning_run() #3532

@ozhanozen

Description

@ozhanozen

Describe the bug

When running Ray directly from tuner.py, Ray is not correctly initialized within invoke_tuning_run(). Particularly, there are two problems:

  1. The following lines, which are supposed to initialize Ray, are actually never used, as util.get_gpu_node_resources() initializes Ray before.

if not ray.is_initialized():
ray.init(
address=args.ray_address,
log_to_driver=True,
num_gpus=len(resources),
)

  1. util.get_gpu_node_resources() initializes Ray within, with log_to_driver = False, which is wrong as it prevents the subprocess output from reaching the main process (e.g., the debug lines within the Trainable step() function are gone). The original behaviour of Ray was the other way around.

Steps to reproduce

Run Ray from tuner.py according to the tutorial documentation, you won't see the debug lines within the Trainable step() function.

System Info

  • Commit: 3a0db9d
  • Isaac Sim Version: 5.0
  • OS: Ubuntu 22.04
  • GPU: 4090 RTX
  • CUDA: 12.2
  • GPU Driver: 535.129.03

Additional context

Previously, I had prepared #3350, which was merged to main, but this PR was not the correct way. I will create a new PR to fix the issue.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

  • Ray is initialized correctly for invoke_tuning_run()

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions