Whether the feature attribution method can be applied on training data?

[Weiting-Gao]

Thanks for your great work! Can I ask a general and quick question? Is it reasonable to make explanations on training instances?

To give an example, model f is trained on dataset X and tested on dataset Y. When I debug a model f, can I use the explanation method to explain the training data, to know which features model f focuses on?
(Usually, the explanation method is applied on test/validate data. According to your paper, the explanation method is also applied on validation set.)

Thank you!