Relationship between DS, TP,CP and EP

[githubsgi]

There are following are two checks on EP . The first one does not include EP - is that intentional ? The second one implies that EP shares the same GPU's as cptpds/m .

1. assert dp_replicate * dp_shard * cp * tp * pp == self.world_size

2. # EP would borrow all cp and tp and some dp_shard degree assert ep % (cp * tp) == 0 and (dp_shard * cp * tp) % ep == 0

With TP=2 CP=2 DS=2 , EP=4 or 8 satisfies the second assert. When this scenario is applied to a TransformerBlock ( e.g. attention and ffn) , what gets applied to what part of the TransformerBlock ?

One tangentially related issue is 1977 .