Skip to content

Commit f62ed31

Browse files
committed
Minor corrections to the Lmod section.
1 parent 36934df commit f62ed31

File tree

1 file changed

+30
-25
lines changed

1 file changed

+30
-25
lines changed

docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md

Lines changed: 30 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -437,10 +437,10 @@ search for those too.
437437
buildtools/21.12 (LUMI/21.12 partition/D)
438438
buildtools/21.12 (LUMI/21.12 partition/C)
439439
buildtools/21.12 (CrayEnv)
440-
```
441-
(output abbreviated and slightly reformatted for readability). This tells that
442-
``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there
443-
are 5 different ways to make that package available.
440+
```
441+
(output abbreviated and slightly reformatted for readability). This tells that
442+
``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there
443+
are 5 different ways to make that package available.
444444

445445
??? Bug "Restrictions with older Lmod versions"
446446
At the time of development of this tutorial, Cray is still using the pretty old
@@ -553,7 +553,9 @@ and the module file can also detect in which mode it is executing.
553553
Modes include "load", "unload" but also "spider". E.g., when the mode is "load", the
554554
``setenv`` function will set an environment variable to the indicated value while in
555555
"unload" mode that environment variable will be unset, and in "spider" mode the
556-
environment variable is left untouched. The working of ``prepend_path``, a function
556+
environment variable is left untouched.
557+
558+
The working of ``prepend_path``, a function
557559
that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is
558560
possible to work with reference counts), but in its most basic mode, ``prepend_path``
559561
will add a given directory to a given PATH-style environment variable (or move it to
@@ -563,6 +565,7 @@ be generated should the directory that is used as the argument not be part of th
563565
in that PATH-style variable). When the mode is "spider", the function has special behaviour
564566
if it is used to change the ``MODULEPATH``. It will then note the change and add that
565567
directory to the list of directories that has to be searched for module files.
568+
566569
This makes ``module spider`` a very expensive command as it may have to traverse a lot
567570
of directories and has to execute all module files in there. Therefore Lmod will build
568571
a so-called spider cache which can be pre-built in the system for certain directories
@@ -598,22 +601,25 @@ whatis("Description: memory usage tester")
598601
It is not all that important to include all those lines in a module file, but some of
599602
those lines get a special treatment from Lmod. The line starting with ``Description``
600603
is used by ``module spider`` to provide some brief information about the module if it
601-
is not totally resolved. This comes with a limitation though: It is not show for each
604+
is not totally resolved. This comes with a limitation though: It is not shown for each
602605
version of the module, so ideally all "GROMACS" modules should contain the same
603606
description line and use other lines to provide further information about what
604607
distinguished a particular version.
605608
Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be
606609
used to add decoration to the spider level 1 output.
607-
All in all the ``whatis`` function if often overlooked in Lmod-based module functionx
610+
All in all the ``whatis`` function is often overlooked in Lmod-based module functionx
608611
but it is a very useful function to include in the proper way in module files.
612+
The EasyBuild support for the ``whatis`` lines is also far from ideal. It will autogenerate
613+
certain lines from information specified in the EasyBuild recipes, but it also allows to
614+
specify ``whatis`` lines yourself via a parameter in the EasyBuild recipes. However,
615+
as soon as you specify the parameter, it will no longer auto-generate the other lines.
609616
610617
A third function that provides information to the search commands is ``extensions``.
611618
It can be used to list up the extensions supported by the module. The argument list
612619
may seem strange as it takes only a single argument, a string of comma-separated ``extension/version``
613620
elements, but that is because the number of arguments to a function is limited in
614621
Lua and that limit can actually be met easily by modules for Python, Perl or R packages.
615622
616-
617623
---
618624
619625
## Some warnings about writing modulefiles
@@ -631,13 +637,13 @@ and basing actions of modulefiles on their position in the hierarchy.
631637
632638
One case where passing information between modules through environment variables will
633639
go wrong is when that environment variable is subsequently used to compute a directory
634-
name that should be added to a PATH-like variable. Assume we have two versions of
640+
name that is then added to a PATH-like variable. Assume we have two versions of
635641
a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then
636642
sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``.
637643
Next we have a module ``MyPythonPackage`` that makes a number of Python packages available
638644
for both Python modules. However, as some Python packages have to be installed separately
639645
for each Python version, it does so by adding a directory to the environment variable
640-
``PYTHON_PATH`` that contains the version which it gets by using the Lua function
646+
``PYTHONPATH`` that contains the version which it gets by using the Lua function
641647
``os.getenv`` to request the value of ``PYTHON_API_VERSION``.
642648
643649
One problem becomes clear in the following scenario:
@@ -648,24 +654,24 @@ module load MyPython/3.6.10
648654
```
649655
The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API``
650656
with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory
651-
for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10``
657+
for the packages for version 2.7 to ``PYTHONPATH``. The ``module load MyPython/3.6.10``
652658
command will trigger two operations because of the *"one name rule"*: First it will
653659
automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and
654660
next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``.
655-
However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point
661+
However, ``MyPythonPackage`` is not reloaded so the ``PYTHONPATH`` variable will now point
656662
to the wrong directory. One would be tempted to think that the easy fix for the user would
657663
be to reload ``MyPythonPackage/1.0``:
658664
``` bash
659665
module load MyPythonPackage/1.0
660666
```
661667
Because of the *"one name rule"* this will again trigger an unload followed by a load
662668
of the module. The problem is in the unload. One would expect that first unloading
663-
``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it
669+
``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHONPATH`` but it
664670
will not. Lmod does not remember that last time it loaded ``MyPythonPackage`` it added
665671
the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the
666672
modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value
667673
``3.6``, it will try to remove the directory for version ``3.6`` which is not in the
668-
``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH``
674+
``PYTHONPATH``. The subsequent load will then add the 3.6 directory to ``PYTHONPATH``
669675
so the environment variable now contains both directories.
670676
671677
In this simple case, a ``module purge`` after the first two ``module load`` commands would
@@ -692,9 +698,9 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
692698
``` lua
693699
LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() )
694700
LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') )
695-
prepend_path( 'PYTHON_PATH', 'someroot/python' ..
701+
prepend_path( 'PYTHONPATH', 'someroot/python' ..
696702
( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' )
697-
LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') )
703+
LmodMessage( 'PYTHONPATH = ' .. ( os.getenv( 'PYTHONPATH' ) or '') )
698704
```
699705
??? Note "Solution with a hierarchy"
700706
The better way in Lmod to implement the above scenario would be in a module hierarchy.
@@ -731,9 +737,9 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
731737
LmodMessage( 'Detected Python API version from environment: ' ..
732738
( os.getenv( 'PYTHON_API_VERSION' ) or '' ) )
733739
734-
prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' )
740+
prepend_path( 'PYTHONPATH', 'someroot/python' .. python_api_version .. '/packages' )
735741
736-
LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') )
742+
LmodMessage( 'PYTHONPATH = ' .. (os.getenv( 'PYTHONPATH' ) or '') )
737743
```
738744
739745
Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory
@@ -758,13 +764,13 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
758764
``module avail`` will show the ``MyPythonPackage/1.0`` module.
759765

760766
The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for
761-
determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire
767+
determining the right directory to add to ``PYTHONPATH``. The fragile way is to enquire
762768
the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``.
763769
The more robust way is to use the Lmod introspection function ``myFileName()`` which returns
764770
the full path and file name of the module file that is executing, and extracting the version
765771
from the path with a pattern matching function. In this particular situation both computed
766772
values are the same so both would have worked to correctly add
767-
``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``.
773+
``somedir/python2.7/packages`` to the front of ``PYTHONPATH``.
768774

769775
The next command, ``module load MyPython/3.6.10`` triggers a chain of events.
770776

@@ -777,11 +783,11 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
777783
the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from
778784
``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading
779785
that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we
780-
computed the name of the directory to add to ``PYTHON_PATH`` using the value of that
786+
computed the name of the directory to add to ``PYTHONPATH`` using the value of that
781787
environment variable, the module would have failed to compute the correct directory name
782-
to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable
788+
to remove so ``prepend_path`` would have left the ``PYTHONPATH`` environment variable
783789
untouched. However, by computing that value from the directory of the modulefile, we get
784-
the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``.
790+
the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHONPATH``.
785791
Lmod will also remember that the module was only unloaded due to a change in the
786792
``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers
787793
the module as deactivated but not as unloaded.
@@ -794,7 +800,7 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
794800
of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is
795801
now available. Hence it will now automatically load that module from the
796802
``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects
797-
that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``.
803+
that ``somedir/python3.6/package`` should be added to ``PYTHONPATH``.
798804

799805
Hence at the end of the cycle we have again a correctly configured environment with no
800806
trace of the ``2.7`` version that was loaded initially and with no action required from
@@ -804,7 +810,6 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``.
804810
This idea is used on LUMI to implement the various versions of the software stack with
805811
for each software stack also optimised binaries for each of the node types.
806812

807-
808813
---
809814

810815
## Further reading

0 commit comments

Comments
 (0)