Best model selection uses test set instead of validation set

[meaten]

Thanks for your great work!

I noticed that the training code seems to select the best model using the test set, since the dataset class only uses a boolean flag training (True for training data, False for test data).

For a fair evaluation, it would be better to use a separate validation set for model selection instead of relying on the test set. This ensures that the reported test performance truly reflects generalization, without being indirectly optimized during training.

[meaten]

Note: SDD and NBA datasets do not provide a validation set, but a validation split should be used for ETH/UCY datasets.

[felix-yuxiang]

Yes, we are aware of that. You are correct about the validation procedure. To ensure a fair comparison, it is necessary to reproduce all previous works and use only the validation score as the criterion for hyperparameter tuning. However, several prior studies appear to have ignored the validation sets altogether, based on the preprocessed ETH/UCY datasets they released. A further complication is that some baseline methods did not release their code, while others did but used different train/test splits for the ETH/UCY datasets. It is a call to the community to develop a standardized benchmark, akin to the Waymo Motion Prediction Challenge, to facilitate fair comparison among approaches.