Shared memory backend support

[termi-official]

Great to see the package evolving step by step.

I would like to use the library in two related variants.

1. Shared memory partitioned matrices without MPI
2. Shared memory partitioned matrices with MPI

The first variant is directly related to #188 , but also very interesting for threaded software (e.g. threaded assembly variants in FEM). The latter should be helpful in maximizing on-node performance for MPI parallel software by using threads on each node in a cluster.

If I understand the idea correctly, then making 1 work should be straight-forward. I just start from the DebugArray implementation and make a custom type e.g. ThreadedArray where the dispatches are correctly implemented, correct? If that is the way to go, is there some easy way to compose this new ThreadedArray with MPIArray or should I already start differently?

[fverdugo]

Hi @termi-official

(1) implementing a "ThreadedArray" and an "MPIThreadedArray" is theoretically possible, but I am not a fan of it. You can always spawn several MPI ranks per compute node and you will achieve pretty much the same. Maybe (or maybe not) there is a marginal performance benefit in the communications among intra-node parts, but does it pay the pain?

I am however most concerned about this other aspect: You want to use shared memory algorithms (not distributed algorithms) to really benefit from threads. Say you want to compute the sum of two vectors of type Vector{Float64}. For me, it is quite overkill to partition these two vectors into PVectors and then process the parts in parallel using threads. It is more straightforward using the original vectors directly in a shared-memory parallel sum.

(2) What we can do however, is to use standard multi-threading to speed up local computations. E.g., if you want to sum two PVectors, you need to sum sequential vectors at each part. You can use threads to sum these sequential vectors. This is pretty much what we want to do with GPUs, but it also makes sense with threads.

I am not sure if your comment referred to (1) or (2).

[termi-official]

Thanks for the quick response on that matter Francesc. Right now I am really more concerned with (1). We have for some of our algorithms (space-time adaptivity) significant benefits in a threaded environment over MPI. Using threads allows us to reduce the communication volume and simplify some of the algorithms related to the mesh data structure, as you also point out. For this reason I wanted to implement the ThreadedArray idea first to have hard measured numbers on the benefits beyond the theoretical pen-and-paper examples. If we also see significant benefits in these measurement, then I would also like to explore the latter data structure. Does this make sense?

And just to make sure we are on the same page, I am not asking you too implement it. I will do the implementation myself, but I wanted to see if you guys are interested in having it upstreamed later and if it is possible to use the existing infrastructure here in first place and if I might miss something which would hinder implementing it or if I should wait until you are done with restructuring the package.

[fverdugo]

Hi @termi-official

If there is an obvious benefit I would be happy to consider the threaded array. But it should be a very clear improvement to Justify to maintain another back-end.

My point is that this threaded array is not going to be a real MPI+Threads implementation. For me, MPI+Threads means combining distributed and shared memory algorithms. The threaded array as described above in (1) is just another way of running a distributed algorithm.

[termi-official]

My point is that this threaded array is not going to be a real MPI+Threads implementation.

Sorry for not being fully clear here. I agree with you. I would indeed also just consider MPI+Threads if we can see significant improvements with using threads, as MPI+Threads will open the box of pandorra regarding load balancing for our problems.