Improve SLURM executor efficiency by supporting the SLURM Apache Kafka job completion plugin

[tcutts]

New feature

Nextflow can be quite aggressive with job schedulers. While there are good features now to rate limit the polling of the job scheduler, that does slow things down, and still doesn't really solve the problem of repeated polling of the scheduler for job status.

SLURM now supports a kafka plugin (https://slurm.schedmd.com/jobcomp_kafka.html).

Use case

Polling of SLURM by Nextflow could be almost eliminated, and delays due to polling intervals eliminated, by updating the SLURM executor to support the kafka plugin.

Suggested implementation

An event-driven executor like this is probably a significant re-write, so I realise in practice it might be simpler to implement it as a completely separate executor, rather than an option to the existing slurm executor.

I haven't looked at the code, but if the AWS Batch executor already uses EventBridge or something like that to notice job completion in AWS Batch, that mechanism could be used as a model, since the concept is the same.

[bentsherman]

How exactly does the event push work? How would the SLURM scheduler notify the nextflow process that a job is done?

[tcutts]

The implementation on the server side is a plugin which sends job events to a topic on a kafka server (which the admin maintains outside of SLURM): https://slurm.schedmd.com/jobcomp_kafka.html

Nextflow would then need to subscribe to events on that kafka topic, in much the same way that it would if it was using SQS or Event Bridge with AWS Batch.

[bentsherman]

I see. A new executor would probably make the most sense then, at least for the first iteration

@jagedn this project sounds right up your alley 😄

[jagedn]

we started a PoC long time ago....

https://github.com/nextflow-io/nf-kafka

[tcutts]

That project looks very interesting in that, if I understand it correctly, it allows Kafka events to be a source of input, and can therefore be very useful as a way to trigger nextflow processing as data arrives.

But that doesn't solve the problem my users (and many other SLURM/LSF administrators have) which is to stop nextflow beating the heck out of the scheduler, while still maintaining maximum workflow performance. The current workaround measures - reducing submission rate, keeping the queue of jobs small etc, all have distinct downsides.

At the same time I don't want the users to have to re-write their workflows. Today, they can run their workflows on SLURM or on AWS Batch just by changing the executor. With the plugin, the whole workflow needs rearchitecting, and you'd have horrible things like different kafka topics for different sections of the workflow to form queues between real workflow steps. That would get very messy very quickly.

I agree @bentsherman that what we would really like is an executor so that the workflows don't need to change.

[jagedn]

Yes, of course you're right. I've pointed this (proof of concept) project as an example as nextflow can work with Kafka
A new slurm executor is the best approach. Sadly I've not idea of slurm nor financial resources to afford this task

[bentsherman]

Fair enough. Thanks for sharing, I do remember you starting that plugin.

If anyone is interested, you can use nf-kafka as an example of how to integrate with Kafka in the Nextflow runtime and use it in a custom executor based on the SLURM executor (rather than a custom operator as in nf-kafka)

[KevinSayers]

@tcutts Another option could be to use the Flux scheduler within a Slurm job. The main Slurm scheduler then isn't constantly polled but instead a user specific Flux scheduler. You then would just have to switch the executor to flux, have Flux installed/built, and the pipeline could stay the same. Just throwing out another idea.

[tcutts]

Nice idea. I wasn’t aware of Flux. I can see some advantages. However, I can also see a disadvantage; if I’m submitting directly to SLURM, then the allocation of tasks to nodes can nicely elastically scale in and out as the workflow runs, nicely shared with other SLURM users. If we’re nesting Flex inside SLURM, then we’re effectively running a large MPI job, which is (a) hard to allocate if we make it large, and (b) inelastic in size, which means if there are steps in the workflow which are barriers, we’re going to end up with a lot of inefficiently used resources in the allocation, which is probably a worse problem than the initial disease we’re trying to cure. Thanks for the pointer to Flux, though, it looks like it could be interesting for some use cases. Tim From: Kevin Sayers ***@***.***> Date: Tuesday, 16 September 2025 at 19:57 To: nextflow-io/nextflow ***@***.***> Cc: Cutts, Tim ***@***.***>, Mention ***@***.***> Subject: Re: [nextflow-io/nextflow] Improve SLURM executor efficiency by supporting the SLURM Apache Kafka job completion plugin (Issue #6371) [https://avatars.githubusercontent.com/u/10041334?s=20&v=4]KevinSayers left a comment (nextflow-io/nextflow#6371)<#6371 (comment)> @tcutts<https://github.com/tcutts> Another option could be to use the Flux scheduler within a Slurm job. The main Slurm scheduler then isn't constantly polled but instead a user specific Flux scheduler. You then would just have to switch the executor to flux, have Flux installed/built, and the pipeline could stay the same. Just throwing out another idea. — Reply to this email directly, view it on GitHub<#6371 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AASQ7CSWW2STFY6HLJO77HL3TBMQJAVCNFSM6AAAAACFE47LMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOJZHE3DSOBZGA>. You are receiving this because you were mentioned.

…

________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>