Offline RL ?

[artbelyaev0]

I have deep dived through your paper and don't understand how CtRL-SIm was trained, but may be i just skipped some thoughts :)
I have question:

Is it correct, that you pretrained CtRL-SIm model and then fine tuned it by PPO in online rl ? (sampling trajectories in gpudrive by using several updates policy)

If answer is no, what you have used for old policy in PPO ?

[RLuke22]

Hello,

I think there may be some confusion regarding the behaviour model (CtRL-Sim; trained via Offline RL) and the planner (RL agent trained via PPO in GPUDrive).

The behaviour model controls the surrounding actors in the simulation environment (the blue/purple/green agents in the video in the README), whereas the PPO-trained RL agent controls the planner (the red agent). GPUDrive is used to train the PPO planner.

We followed the same training recipe for CtRL-Sim as the CtRL-Sim paper, with a few minor modifications outlined in Section C.2 of the Scenario Dreamer Appendix. We did not do any online finetuning of CtRL-Sim.

Hopefully this clears up the confusion!

Best,
Luke