Post‑processing fails in pptransform: Unknown dataset_type 0

[mo-leaprevost]

Hi,

I’m running an ensemble (suite u-dv704) and am having issues with the post‑processing:
main_pp.py atmos is failing consistently for all ensemble members during the pptransform stage.

Earlier months (roughly up to ~1 year before the failure point) processed successfully, although I sometimes had to manually re‑trigger post‑processing tasks. From December onwards, however, every ensemble member fails in the same way.

The error I see is repeated Rose date warnings such as:
[WARN] rose date requires length=5 input date array - adding 0: [2017, 12, 1, 0]
[WARN] rose date requires length=5 input date array - adding 0: [2016, 12, 1, 0]

The job then crashes during the mean creation step with:
ValueError: Unknown dataset_type 0, supported types are dict_keys([1, 2, 3, 4, 5]), coming from mule.load_umfile inside atmos_transform.py.

For context (not sure if this is related): from around September onwards, the pptransform and archive tasks appeared to run successfully, but the corresponding output does not appear on MASS.

I’d be grateful for any advice on what might cause this issue, or where best to look to diagnose it further.

Thanks very much,
Léa

[r-sharp]

Léa - I'm going to take a look at this with you.
It will probably be easiest if we move to Teams though.

However, for anyone who's ended up here by searching for a problem similar to theirs, some of the intitial things I'm going to be looking for are :
The full task output (.out and .err files). The suite output will also hopefully detail the operations run on the specific files involved - can the error be replicated on the command line ? What are the files ? what was the command ?
For diagnosis, it will probably help (me) to have copies of the files and details of the command so I can firstly easily and repeatably replicate the error and secondly quickly test theories and attempted workarounds/fixes.

In this particular question I feel there are 2 distinct issues.
1 the transform unsing Mule isn't working
2 there was a period where it appeared to be working, but the output is not in the archive.
I will be concentrating on 1 and my guess is that it stopped working some time ago, but was incorrectly reporting 'sucess' to cylc as the most likely reason output is not in the archive.

[Pierre-siddall]

Hi Lea,

As discussed offline this issue seemed to be caused by no memory being allocated in the [[HPC]] block (between lines 50-64) of the meto_ex.cylc file which was then being inherited to setup other environments within the file. This resulted in no memory being allocated to the job when the ensemble was run and subsequently the job complained with an out of memory error. This was resolved by adding the following under the directives of the [[HPC]] class with the required amount of memory:

[[[directives]]]
     -l mem=10GB