How to modify the advantage computation in GRPOTrainer

[Tuziking]

I’m looking to customize the advantage computation used in the DAPO algorithm. Do I need to subclass the full GRPOTrainer to do this, or is it sufficient to overwrite the logic in _generate_and_score_completions, since that method appears to handle the advantage calculation?

[Tuziking]

I think it would be good to decouple advantage computation from the reward function and expose an interface for it. This might spark more interest in algorithmic modifications within the community and make it easier for people to implement their own ideas. I'm currently studying the source code, and I welcome anyone who’d like to discuss it with me.