Yet another suggestion from @hariharan-devarajan

Currently, the tags for UCX's tag-matching communication are generated with:

tag = (producer_rank << 32) | consumer_rank

The benefit of this approach is it makes use of information from Flux to create a tag unique to the producer and consumer applications. However, the downside is that multiple communications between the same producer/consumer pair will not use a unique tag, which introduces potential issues with UCX tag matching.

Instead of the current approach, we could maintain a 32-bit counter (or possibly larger) in the module. Using this counter, the module could generate tags using:

tag = (producer_rank << 32) | counter

This tag will be unique for a particular transfer that is outgoing from a particular producer module. As a result, all transfers will have a unique tag until the counter overflows.