Awaiting multi-node, MoE, Megatron, and more

[chujiezheng]

Thanks for your great work and generous releasing. Can’t wait to see the integration of multi-node training, MoE, Megatron, and more features!

[rizar]

@chujiezheng multi-node training already works, we just need to write the readme.

You get MoE training by using an MoE model from HF/transformers.

As for Megatron, for that we'd need help. It's not on the roadmap so far. I'd like to chat to Megatron expert to assess the difficulty.

[icwhite]

@rizar, could you provide instructions on how to run with multi-node?