Benchmarks

[szaghi]

Here we can discuss the benchmarks analysis.

It could be useful to compare with the performance of other non abstract implementations.

Architecture	Comparison with other non ACP	Conclusion
Serial
Parallel shared memory
Parallel distributed memory

[szaghi]

I have just added the first OpenMP test (v0.0.8) it seems to scale reasonably!

not all parallelizable parts are actually parallelized yet

More detailed analysis will come soon!

[szaghi]

I have made some progress with OpenMP benchmarks. You can find a script for it under the paper sub-directory. Into the paper I placed some figure for strong and weak scaling that I also upload here. It seem that at 8 cores my workstation finishes the fuel...

strong scaling

weak scaling

With my workstation (Intel Xeon with 12 physical cores) the performance are not exiting, but the test has been done very quickly and it could be some huge mistakes. For example the "size" of the test could be meaningless (maybe it is to small and I spent more time to create the OpenMP threads than for the actual computations). I am almost sure that the MPI test will perform better. However, I need some help from HPC guru ( @francescosalvadore, @muellermichel, all other than me...): the tricky point with OpenMP are the operator overloading, e.g. see the multiply. To take advantage of the automatic lhs-reallocation I have to split the operators implementation with something like a serial section and threaded one:

  ! serial
  select type(opr)
  class is(euler_1D_openmp)
    opr = lhs
  endselect
  ! parallel
  !$OMP PARALLEL DEFAULT(NONE) PRIVATE(i) SHARED(lhs, rhs, opr)
  select type(opr)
  class is(euler_1D_openmp)
    !$OMP DO
    do i=1, lhs%Ni
      opr%U(:, i) = lhs%U(:, i) * rhs
    enddo
  endselect
  !$OMP END PARALLEL

This is due to the fact that the first "serial" section is parallelized it self, it being the assignment operator where I placed something like:

    !$OMP PARALLEL DEFAULT(NONE) PRIVATE(i) SHARED(lhs, rhs)
  select type(opr)
  class is(euler_1D_openmp)
    !$OMP DO
    do i=1, lhs%Ni
      lhs%U(:, i) = rhs%U(:, i)
    enddo
  endselect
  !$OMP END PARALLEL

Because operators overloading is ubiquitous in FOODiE, could my bad OpenMP implementation be the reason of these bad performance? Obviously, we must ensure that the benchmark has sense before answer this question, but I think that your experience can already discover the murder...

The complete OpenMP test is here. I will very appreciate your opinions on it. How you implement it?

See you soon.

[szaghi]

Hi all,
just a remark: I realized that the weak scaling is not computed right. The Euler test has a variable time resolution that increases as the space resolution increases accordingly to the CFL condition.... thus in the weak scaling when the size increases the number of steps performed increases as well! I have to modify the main program fixing the total number of time steps independently on resolution, I am sorry.

For the moment consider only the strong scaling, that indead is not so bad up to 8 cores.

[milancurcic]

@szaghi Thank you more making progress with this. I have been having some thoughts about the usefulness of parallel benchmarks for FOODiE. As you state in the manuscript draft, FOODiE is unaware of the parallelism implemented in the user's external program. Other than just proof of concept which is useful, what is the purpose then of parallel benchmarks? The parallelism is never implemented in the same dimension as integration dimension, and is not part of the library itself, so these results do not tell me much about FOODiE.

What do you think?

[szaghi]

@milancurcic you are right, my intention is not clear.

The parallel benchmarks are aimed to prove that the ADT and ACP do not decrease or affect the performance when used into a parallel framework. Most of the critics about OOP and abstraction in the Fortran community are focused on the performance issue: I would like to prove that Damian's theory is right, if used carefully high level of abstraction does not decrease performance.

As you said, FOODiE is unaware of parallelism, but I want to prove that it can be used into HPC codes without much concern.

I would like that these benchmarks (MPI will come soon, but I would like to try hybrid-fortran and coarray) say to you about FOODiE Milan trust me, you can safe use me, I will not devaste your parallel scaling

[milancurcic]

👍

If you are going parallel, I'd recommend getting started with coarray Fortran (CAF) early on and please, please, please don't go down the MPI path. There are so many reasons to choose CAF over MPI in a new project and I fear I'm not going to have time to write them all. If you guys ever think about setting up occasional teleconferences via Skype or Google Hangout, I would be glad to join and could probably provide guidance more efficiently in that forum than online.

I only have a moment and will try to type what I can fast.

First, I'm sure you know the saying, "MPI is the assembly language of parallel programming." I used to interpret that as a joke made at MPI's expense. Then I mentioned that joke to none other than Bill Gropp who led the development of MPI since the early days more than 20 years ago and his response was (paraphrasing): "That's what we intended! We were originally targeting compilers, not applications. The problem is that it took so long for parallel languages to come along that MPI ended up in the hands of application developers." Well, CAF is now the most widely installed parallel language. Anyone who has a recent release of the GCC, Intel, or Cray compiler has CAF in their hands. There is no longer a good reason to write assembly language -- not even for performance which I'll mention next.

On the OpenCoarrays project, we now have data showing CAF outperforming a code with raw MPI in the source even when OpenCoarrays wraps MPI. See, for example, the first recent article at http://www.opencoarrays.org/publications. Why is this the case? I'm glad you asked! :) It's because one way to do a fair comparison is to compare CAF to the MPI feature set used by every application developer with whom I've had this discussion. The OpenCoarrays CAF implementation exploits the one-sided "put" and "get" communication offered in MPI-3; whereas every application developer I've met is still using MPI's older two-sided send/receive communication. The one-sided communication often outperforms the two-sided communication on hardware with support for remote-direct memory access (RDMA), which includes Infiniband and the proprietary Cray/Intel interconnects. Based on the aforementioned article, it apparently also outperforms MPI in shared memory and when exploiting a GPU.

I'd say it might even be a good idea to start down the CAF path even before adding OpenMP to your code. I say this partly because CAF can work in either shared or distributed memory, whereas OpenMP only works in memory that is at least logically shared (though on some architectures, it might be physically distributed). I would add OpenMP in a secondary stage of performance tuning.

Also, I'd love to hear your about your experience if you take a look at DO CONCURRENT. It can replace some some of the simpler loop-level OpenMP directives, but it also can be used in conjunction with OpenMP. Either way, I'd start with fully exploiting Fortran's own capabilities before adding directives that are external to the language. It will greatly aid the portability and clarity and generality of your code -- partly because it will offload some decisions to the compiler the the compiler might be better qualified to make. For example, sometimes the compiler might decide not to multithread the loop if there is a better technology such as loop unrolling or vectorization that might be better suited for the particular loop. With OpenMP, you're forcing one technology on the compiler (multithreading) to the exclusion of other options.

Parallelization using CAF is one area where I'd love to help although I'd still probably help in more of an advisory mode than in a coding mode for the near future.

[szaghi]

@sourceryinstitute ok, you have convinced me! I will start with CAF. However, I have never done it before... I need help. Can you give me some references (books, papers, examples)? I will surely start with opencoarray, but I have not yet finc the time to study it: are there (in opencoarray) references to learn CAF?

Thank you very much for your help!

[szaghi]

The OpenCoarrays CAF implementation exploits the one-sided "put" and "get" communication offered in MPI-3; whereas every application developer I've met is still using MPI's older two-sided send/receive communication.

Very very interesting! I am one of the addicted to the simple send/receive pattern (the most simple approach for my cases). I am very excited to learn and try opencoarrays!

I'd say it might even be a good idea to start down the CAF path even before adding OpenMP to your code. I say this partly because CAF can work in either shared or distributed memory, whereas OpenMP only works in memory that is at least logically shared (though on some architectures, it might be physically distributed). I would add OpenMP in a secondary stage of performance tuning.

Indeed I planned to do a comprhensive tests suite for parallel benchmarks:

• serial, non abstract, without overloaded operators, ptocedural version (almost completed, but not yet tested and uploaded on github);
• OpenMP FOODiE aware (completed in few minutes... it was difficult for you stopping me in time :-), tested and uploaded);
• CAF FOODiE aware (I must learn CAF and OpenCoarrays before try to implement it, the timeline is not clear);
• MPI FOODiE aware, now freezed until CAF OpenCoarrays will be completed;
• Cuda/hybrid-fortran FOODiE aware, delayed in the near future;

See you tomorrow.

Of the two application developers I've talked to who investigated switching from two-sided send/receive to one-sided put/get, one concluded that it would require more time than he could invest and the other attempted to make the switch but found it too complicated. This is the real value of OpenCoarrays: we do the puts and gets for you and save you a lot of the hassle. You get to just use a syntax and semantics that feels like a very natural extension of prior versions of Fortran.

I have a collection of tutorial videos online at http://www.sourceryinstitute.org/videos.html. The total amount of content is 1 hour and is intended more as a rapid overview that takes the viewer from the basic concepts up to writing an object-oriented PDE solver using a "functional programming style", which is how I more often refer to the Abstract Calculus pattern now. I do that because more people are familiar with functional programming than with patterns and far more than are familiar with the Abstract Calculus pattern as described in my book.

The aforementioned videos pre-date OpenCoarrays and therefore use the Intel compiler. I would like to produce a new video focused on how to use OpenCoarrays, but it requires a significant investment of time. The previous videos were produced and edited by a film professional when I was working at Stanford full-time last year so it will be challenging to match the quality.

Also, the previous videos move through the material quite fast so they are intended more as a teaser to show enough capability to get people interested in studying further. Most modern Fortran books include some coarray material and my book has a small amount of coarray material in Chapter 12. I also frequently teach tutorials at conferences and other locations. In case you're attending the SC15 supercomputing conference next month, I'll co-teach a full-day tutorial there that will focus primarily on CAF: http://sc15.supercomputing.org/schedule/event_detail?evid=tut103.

[szaghi]

Of the two application developers I've talked to who investigated switching from two-sided send/receive to one-sided put/get, one concluded that it would require more time than he could invest and the other attempted to make the switch but found it too complicated. This is the real value of OpenCoarrays: we do the puts and gets for you and save you a lot of the hassle. You get to just use a syntax and semantics that feels like a very natural extension of prior versions of Fortran.

I agree, it is a great plus of OpenCoarrays to make easy this kind of coversion: I never tried to move from send/receive because I feel it is a hard passage which requires a lot of time. I cannot wait for the tomorrow lunch break for start learning CAF and OpenCoarrays!

I have a collection of tutorial videos online at http://www.sourceryinstitute.org/videos.html. The total amount of content is 1 hour and is intended more as a rapid overview that takes the viewer from the basic concepts up to writing an object-oriented PDE solver using a "functional programming style", which is how I more often refer to the Abstract Calculus pattern now. I do that because more people are familiar with functional programming than with patterns and far more than are familiar with the Abstract Calculus pattern as described in my book.

Wonderful! I will try to watch them when my family does not watch me!

Unfortunately, my institute does not allow me to attend conferences that are not focused on hydrodynamics... thus not I cannot attend your conference :-(

[szaghi]

I just read my first CAF example... it is quantum leap with respect MPI! It is extremely beatiful!

I can write my first observation: even in the worst case that for some special obscure circumstances some MPI implementations are faster than an ipothetical equivalent CAF (that from Damian's experience I guess it is a very irrealistic scenario), the consiseness, clearness Fortranish-style of CAF syntax overcomes such lacks. Modern fortran is fantastic!