Intermittent ConcurrentModificationException error

[muffato]

Hello,

We're getting this intermittent error

java.util.ConcurrentModificationException: null
        at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
        at java.base/java.util.ArrayList$Itr.next(ArrayList.java:967)
        at nextflow.Session.callIgniters(Session.groovy:545)
        at nextflow.Session.fireDataflowNetwork(Session.groovy:539)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:250)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:139)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:379)
        at nextflow.cli.Launcher.run(Launcher.groovy:513)
        at nextflow.cli.Launcher.main(Launcher.groovy:673)

The error is intermittent, and while we're bumping into this error more regularly these days, we also have hundreds of successful runs. When it happens, it's very early on, before any job is even submitted. As far as I can tell, Nextflow hasn't even finished parsing all the files / preparing everything.

Here are the logs of two failes runs:
ilEpiCruc3.nextflow.log
ilGodZava5.nextflow.log

Compared to a successful run, some [main] DEBUG nextflow.processor.TaskProcessor - Starting process lines are missing.

There are similarities with #675. We do define a channel by concatenating channels made from remote (https) paths

    Channel.empty()
        .concat( Channel.fromPath(params.blastn).map { tuple(["type": "blastn"], it) } )
        .concat( Channel.fromPath(params.blastx).map { tuple(["type": "blastx"], it) } )
        .concat( Channel.fromPath(params.blastp).map { tuple(["type": "blastp"], it) } )
        .concat( params.precomputed_busco ? Channel.fromPath(params.precomputed_busco).map { tuple([ "type": "precomputed_busco"], it ) } : Channel.empty() )
        .concat( params.busco ? Channel.fromPath(params.busco).map { tuple([ "type": "busco"], it ) } : Channel.empty() )
        .concat( Channel.fromPath(params.taxdump).map { tuple(["type": "taxdump"], it) } )
        .set { ch_databases }

Can it be related ?

Nextflow and system info:

Version: 25.04.6 build 5954
Created: 01-07-2025 11:27 UTC (12:27 BST)
System: Linux 5.15.0-141-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 17.0.15+6-LTS
Encoding: UTF-8 (UTF-8)
Process: 2880498@tol-1-2-1 [10.160.8.36]
CPUs: 2 - Mem: 376.5 GB (139 GB) - Swap: 19.5 GB (19.4 GB)

[muffato]

ping @DLBPointon who's been facing the issue and has provided the log files

[bentsherman]

The co-mod exception happens when Nextflow starts the pipeline execution by invoking the "igniters". These igniters are created by channel factories.

But I don't understand how this could happen, because the script is completely evaluated before the igniters are called.

Maybe if you called a channel factory within an operator closure or process? That would create a race condition, though I suspect it would cause other errors too

[DLBPointon]

Hi Ben,

Just scanned through the pipeline, doesn't look like any Channels are getting made in closures, nothing comes even close from what I can tell.

The pipeline in question is https://github.com/sanger-tol/blobtoolkit/tree/0.8.0 in case you'd rather look your self.

[muffato]

Hi @bentsherman . What about this line ? https://github.com/sanger-tol/blobtoolkit/blob/0.8.0/subworkflows/local/input_check.nf#L170

 ch_parsed_busco_filtered = ch_parsed_busco_filtered.ifEmpty { Channel.value([]) }

[bentsherman]

Yeah, that could be it. You don't need to wrap the empty list as a value channel

[jfy133]

We are also getting similar errors on nf-core/createtaxdb,

e.g. https://github.com/nf-core/createtaxdb/actions/runs/18185953128/job/51770241561?pr=111

 > ERROR ~ Error executing process > 'NFCORE_CREATETAXDB:CREATETAXDB:PREPROCESSING:FIND_CONCATENATE_AA (1)'
    > 
    > Caused by:
    >   java.util.ConcurrentModificationException
    > 
    > 
    > 
    >  -- Check '/home/runner/_work/createtaxdb/createtaxdb/~/tests/b51a56addccd32d7ddca4165b28568dd/meta/nextflow.log' file for details

I unfortunately don't yet have a log file from running it locally as it is very intermittent on my laptop (I've had once, but disregarded at the time as a one off), but happens more regularly on GitHub actions.

We only started getting these error in the last months or so. What is odd, is that our case we've not changed anything in the pipeline prior the FIND_CONCATENATE_AA module in the PR above...

I can't see myself for anything obviously like the suggestions above (the empty list, or a channel within an operator)... at least around or prior the affected module:

https://github.com/nf-core/createtaxdb/blob/07f49813f34d648a9bcd54f0a546af5eca545e06/subworkflows/local/preprocessing/main.nf#L147-L151

Do you have any other suggestions how to debug/identify the course of the issue in this case??

[Midnighter]

I asked Claude if it can identify any concurrency issues and the only point where I'm not 100% sure if it's an issue is the use of the following pattern regarding mixing version channels. However, this is used in all nf-core pipelines and I've not seen a problem before. I always wondered how the re-assignment of ch_versions is handled, though.

Claude Sonnet 4 output

Both blocks mix versions into the same channel:

ch_versions = ch_versions.mix(UNPIGZ_DNA.out.versions.first())
ch_versions = ch_versions.mix(UNPIGZ_AA.out.versions.first())

Issue: If processes execute concurrently, version information could be lost or corrupted.

[bentsherman]

Yeah that's a false positive, those mix operations will be handled correctly by the DAG