[FEA] Consolidate Docker images to a single tag per CUDA major version

[bdice]

Proposal: Consolidate Docker images to a single tag per CUDA major version

• Goal: Stop producing per-minor CUDA 12 images (cuda12.0, cuda12.8, cuda12.9, …) and publish a single, rolling cuda12 image that always tracks the latest supported 12.x release.
• Rationale: Minor releases in CUDA 12 are intended to be compatible within the major line. Our main constraint for customers is the NVIDIA driver version, not runtime/library compatibility. Maintaining per-minor images adds build/CI complexity, documentation churn, and support overhead without clear customer benefit.

Customer impact

• Benefits

  • Simpler choices: One image per major version reduces confusion and “which minor do I pick?” questions. RAPIDS supports minor version compatibility, so the choice of CUDA minor version is less important.
  • Less documentation churn: Eliminates repeated RSNs and doc updates for “drop 12.x, add 12.y”. Examples:
    • https://docs.rapids.ai/notices/rsn0051/
    • https://docs.rapids.ai/notices/rsn0042/
    • https://docs.rapids.ai/notices/rsn0039/
    • https://docs.rapids.ai/notices/rsn0035/
  • Faster adoption: Customers automatically get bug fixes and feature updates in the latest minor of the desired CUDA major line.

• Potential downsides and mitigations

  • Minimum driver version bumps: When a new CUDA minor version is available, the driver support could change based on the requirements of the latest nvidia/cuda images. However, this only reflects the support lifecycle of the CUDA drivers themselves. Users who need long term support should be adopting LTS driver branches already.
  • Reproducibility concerns: Changing CUDA minor versions could introduce defects from the CUDA Toolkit itself.
    • Mitigation: If such a bug were identified, we would be able to revert to an older CUDA Toolkit version without needing to rename the container tag.

Is the minor version significant?

• Summary: Within a major version, CUDA is designed for application binary compatibility; the main operational constraint is the GPU driver meeting or exceeding the minimum required for that CUDA minor. Because of Minor Version Compatibility, this is a non-issue for RAPIDS. This supports the reliability of a rolling cuda12 tag.
• References:
  • NVIDIA CUDA Compatibility Guide for CUDA Toolkits
  • NVIDIA Data Center Drivers Documentation
  • CUDA Toolkit Release Notes

Implementation plan

• Tagging and image strategy

  • Publish a single rolling tag per CUDA major and Python version:
    • <RAPIDS_VERSION>-cuda<CUDA_MAJOR_VERSION>-py<PYTHON_VERSION>: e.g. 25.10-cuda12-py3.13

• CI/build matrix changes

  • Remove older CUDA 12.x releases from the build/test matrix; keep only the latest 12.x.
  • Update the CUDA minor versions used as part of build-planning migrations to new minor versions
    • Should we consider similar changes in ci-imgs?

• Documentation and comms

  • Update README and contributing docs to use major versions
  • Replace per-minor RSNs with a single note in regular release notes when cuda12 advances to a new minor. Clearly state minimum driver version (typically no changes will be expected except when LTS branches end their support cycle) and/or any notable changes in RAPIDS related to the CUDA minor version upgrade.

• Testing

  • Confirm all existing test suites pass on the consolidated cuda12 image. Based on preliminary tests in Test earliest driver with CUDA 12.9 #780, this shouldn't be an issue.

Benefits

• Ops: Fewer CI jobs/build minutes; fewer cache entries; reduced image storage and tag count.
• Docs/Support: Fewer RSNs and docs PRs for CUDA minor versions; reduced customer questions related to “which minor should I use?”
• Adoption: Majority of pulls switch from cuda12.x to cuda12 within one release cycle.

Rollout

1. 25.10: Publish cuda12 alongside existing cuda12.x for one release; publish only cuda13 (no cuda13.0); announce deprecation of CUDA minor tags.
2. 25.12: stop building cuda12.0 and cuda12.9

Tasks

• Update build scripts to retag cuda12.9 builds as cuda12
• Update docs: README, compatibility matrix, release notes template, migration guide.
• When adding CUDA 13, only create cuda13 images
• Announce deprecation of cuda12.x tags with an RSN

[jameslamb]

Thanks for the excellent write-up! I'm convinced, let's do this.

Documentation and comms

If the tag structure is changing, we'll also need significant updates to https://docs.rapids.ai/deployment/stable/

These images are used heavily there, in examples for using RAPIDS on various cloud service providers' services. Most of those uses flow through this configuration:

https://github.com/rapidsai/deployment/blob/4322d89f6b19bd3ab1311265c5ffd158db03a3e8/source/conf.py#L31-L32

https://github.com/rapidsai/deployment/blob/4322d89f6b19bd3ab1311265c5ffd158db03a3e8/source/conf.py#L44-L45

cc @jacobtomlinson @ncclementi @jayavenkatesh19

[jacobtomlinson]

This may be a naive question but I was under the impression that minor version compatibility was introduced part way through CUDA 12, but maybe I dreamt this. Can I just check my understanding that a package built with CUDA 12.9 and containing CTK libraries for 12.9 will work happily on a host with a driver that only supports CUDA 12.0?

Other than that I'd be very happy to simplify the images. I just had to bump 12.8 to 12.9 and it was painless. So bumping to 12 would be just as straightforward. rapidsai/deployment#585

[bdice]

I was under the impression that minor version compatibility was introduced part way through CUDA 12, but maybe I dreamt this.

Minor Version Compatibility has existed since CUDA 11, it was introduced in a patch release of driver 450 (which originally came with CUDA 11.0 but was updated for MVC with the release of CUDA 11.2 -- please feel free to check my facts, this is partly from memory).

Can I just check my understanding that a package built with CUDA 12.9 and containing CTK libraries for 12.9 will work happily on a host with a driver that only supports CUDA 12.0?

I tested RAPIDS Docker images built with CUDA 12.9 on driver 535, which is the oldest driver we have in CI. (535 is also the oldest driver that is still supported, since it is an LTS branch.) See CI runs from #780, specifically these three jobs (1, 2, 3).

[jacobtomlinson]

We might need to update the install doc then as it says we support CUDA 525+.

It looks to me like 535 is CUDA 12.2, which makes me think there is even less value in having a CUDA 12.0 image.

[bdice]

We have been gradually softening the language to “requires CUDA 12 with a supported driver” and not calling out specific versions. People probably do like to see a number in the reference docs if they need to check their version. We could change that to 535. It just becomes something we have to keep in sync with the driver lifecycle…