LFRic on Monsoon3

[dennissergeev]

Reposting my question (#403) here as it seems more appropriate: I would like to run LFRic Apps on Monsoon3 but the problem is that the relevant environment is not set up.

I tried loading the modules listed in suite_config_ex1a.cylc but they don't seem to available (neither lfric-cray, nor lfric-gnu, nor shumlib).

For example, when I try module spider shumlib, I get
Lmod has detected the following error: Unable to find: "shumlib".

🆘 @yaswant @t00sa @mo-mahammedk

[jennyhickson]

Hi Dennis,

The software stacks are all there. You'll need to do module use /projects/metoff/spackadmin.mon/ngms in order to access the various LFRic modules.

Let us know if that doesn't do it for you.

Thanks

[yaswant]

or add MODULEPATH=${MODULEPATH:+$MODULEPATH:}/projects/metoff/spackadmin.mon/ngms to your ~/.bashrc

[dennissergeev]

Ah so this is the key step! Thanks @jennyhickson and @yaswant!

Is there a plan to create a separate site configuration for Monsoon3 in LFRic's rose-stem?

[dennissergeev]

I'm now getting this error when I try to submit the job:

[STDERR] qsub: Unauthorized Request

Any ideas what I need to change?

[dennissergeev]

Sorry, it was a silly question. I should have changed the queue name to collabshared.

[dennissergeev]

There seem to be a problem with shumlib on Monsoon3. While the module loads ok and LFRic Apps vn2.2 compiles ok, linking fails with the following error:

/usr/bin/ld: cannot find -lshum: No such file or directory

The module is indeed loaded:

Currently Loaded Modules

Currently Loaded Modules:
  1) mo-certs/2025-05-28
  2) mo-craylibs-milan/1.0
  3) userenv/1.2.1.1
  4) moose-client/1.1.0
  5) craype-x86-milan
  6) libfabric/1.15.2.0
  7) craype-network-ofi
  8) xpmem/2.6.2-2.5_2.40__gd067c3f.shasta
  9) cray-pals/1.2.13
 10) cray-pmi/6.1.11
 11) perftools-base/23.05.0
 12) cpe/23.05
 13) craype/2.7.21
 14) cray-dsmml/0.2.2
 15) cray-libsci/23.05.1.4
 16) PrgEnv-gnu/8.4.0
 17) gcc/12.2.0
 18) cray-mpich/8.1.26
 19) gcc-runtime/12.2.0-4eol2av
 20) gftl/1.14.0-cyccj7n
 21) m4/1.4.18-gjke3uy
 22) gftl-shared/1.9.0-srnw4hu
 23) fargparse/1.8.0-tlmnxt4
 24) pkg-config/0.29.2-2ucarv3
 25) zlib-ng/2.2.1-julfaek
 26) hdf5/1.14.4-3-rt7n6zv
 27) netcdf-c/4.8.1-ii2z7uf
 28) netcdf-fortran/4.6.1-fedn5ui
 29) nccmp/1.9.1.0-qbnzcgm
 30) python/3.11.5-chisvem
 31) pfunit/4.10.0-2ejgntf
 32) python-venv/1.0-7fp4fcn
 33) py-markupsafe/2.1.3-vmzky4j
 34) py-jinja2/3.1.4-jdq3jlw
 35) py-configparser/5.2.0-cfrz22b
 36) py-zipp/3.17.0-mg5jxub
 37) py-importlib-metadata/7.0.1-sx5itgw
 38) git/2.35.3-2zihzjo
 39) py-packaging/24.1-rhw5mer
 40) py-setuptools/69.2.0-vi4ycea
 41) py-tomli/2.0.1-6s5io2v
 42) py-typing-extensions/4.12.2-mt4mawx
 43) py-setuptools-scm/8.0.4-wknrct5
 44) py-fparser/0.2.0-pqbzmly
 45) py-graphviz/0.13.2-kg4jbls
 46) py-attrs/23.1.0-mypn23r
 47) py-pyrsistent/0.19.3-jgl5ja3
 48) py-jsonschema/4.17.3-afrbk4h
 49) py-pyparsing/3.1.2-luol674
 50) py-mpmath/1.3.0-ldwltcy
 51) py-sympy/1.13.0-sr43zsb
 52) py-termcolor/1.1.0-albef2n
 53) py-psyclone/3.1.0-sw5elul
 54) rose-picker/2.0.0-lbc6gyf
 55) xios/2701-tnwhmlz
 56) yaxt/0.10.0-naxdf2i
 57) lfric-gnu/12.2.0/1.0.1
 58) shumlib/2023.08.1-uvap4hj

but linking fails (both using a local command-line build and rose-stem build).

As far as I understand it, the problem is that shumlib is missing from /projects/metoff/spackadmin.mon/releases/2025.5.2/install/ - could this be fixed please?

Any help will be greatly appreciated...

[t00sa]

This should now be working. I've corrected the path in the shumlib module to point to the correct location on monsoon, so if you ensure that you use the $FFLAGS and $LDFLAGS environment variables with the compiler - something the LFRic build system does for you - you should be able to to link with the library.

[dennissergeev]

Much appreciated @t00sa! It works now 🎉

[dennissergeev]

Reposting the issue with Cray compilers (#416) here - has anyone looked into that by any chance?

[dennissergeev]

Hi again,

Just using this thread to report a recent issue with compiling LFRic on Monsoon3. It used to work before the software update but now I get this error:

/projects/metoff/spackadmin.mon/releases/2025.5.2/install/bin/sbang: line 102: /projects/metoff/spackadmin.mon/releases/2025.5.2/install/linux-sles15-zen3/gcc-12.2.0/python-venv-1.0-7fp4fcnpvbrtl6hxxg2kajdo2p7i66es/bin/python3: No such file or directory

This looks like a broken symlink issue. Could you fix this please? Or should I update module-load commands in flow.cylc of my suite?

Cheers,
Denis

[dennissergeev]

@t00sa perhaps you could have a look at this please? (in addition to #416 😃)

[t00sa]

If you're happy to use the slightly older version of Cray MPICH (8.1.26) which has a few known bugs, you can change your prescript to pick up the updated software environment as follows. Change the module directory to the following location:

{% set ex1a_base = 'module use /projects/metoff/spackadmin.mon/releases/2025.10.1/ngms' %}

Then update the modules in the GNU and Cray sections to:

• module load lfric-gnu/12.2.0/3.0- || true
• module load lfric-cray/15.0.0/3.0- || true

If you want to try MPICH 8.1.27, all you need to do is change the final module name to 3.0+. You may also be able to clean up some of the other prescript settings - see the changes in trac ticket 1030 for details.

[dennissergeev]

Thanks very much @t00sa. I can confirm the GNU option works (Cray still crashes, #416).

One small question: do I need this line in my pre-script:
ulimit -s unlimited -c unlimited?

[t00sa]

I think these are still necessary. The -s option removes the stack size limit, which has caused failures on some platforms, and the -c option removes the size limit on the core files generated in the event of a crash.

[dennissergeev]

many thanks @t00sa!

[iboutle]

I've tried upgrading this suite (u-dr662) to @git_migration (i.e. vn3.0 under fcm), but whilst it works fine on our internal systems, it doesn't work on Monsoon. I presume this is the software stack being out-of-date with the head of trunk again - is it possible to get the latest internal software stack installed on Monsoon please?

It fails with Psyclone errors, and I think the Psyclone versions used are different (3.2 vs 3.2.2).

Many thanks!

[t00sa]

I've updated monsoon with the latest software stack, so if you update the module search path as follows:

module use /projects/metoff/spackadmin.mon/releases/2025.12.18/ngms

you should be able to load the lfric-cray/15.0.0/vn3.0+ module in the same way as on the internal systems. I've also fixed a CCE/XIOS bug which might have broken interactive builds on some of the login nodes.