Estimate economic benefits of moving runners from c5d to c7i or c7a

[malfet]

Two generations of CPU hardware should bring some much needed build speed improvements, wouldn't they?

[huydhn]

This is a good BE opportunity to reduce CI cost, we need an owner to driver this change. Maybe @zxiiro or @seemethere could take a stab

• Make the change
• Experiment with the new runner
• c7i vs c7a
• Win!

[zxiiro]

I've done a little bit of research in this area previously when I was investigating Intel cost usage. In general this would increase costs as newer hardware generally costs more than older hardware. However AMD hardware is about 10% cheaper than Intel hardware sometimes.

For example: 2xlarge CPU instance

c5.2xlarge: $0.34 / hr
c5a.2xlarge: $0.308 / hr

vs

c7i.2xlarge: $0.357 / hr
c7a.2xlarge: $0.4106 / hr

So if reducing costs is our goal than moving to c5a.2xlarge would cut costs by about 10%. However moving to c7i would be roughly 5% more expensive. If jobs run faster with the newer hardware we could shave off some of that extra costs but we'd have to test and see if it's meaningfully faster enough to offset the costs.

Edit: I'd be willing to take this on and try comparing c7i / c7a against some jobs that use c5 systems currently and see if there's any meaningful speed up from utilizing the different hardware.

[zxiiro]

I'm going to start running a few different build jobs using the c5 / c7i / c7a and comparing the build times I get from them to see if there's anything obvious that's quick to see from the different hardware.

[zxiiro]

@malfet c5d mentioned in the subject means SSD systems. So for those specific instances I don't see an SSD equivalent system in the c7i series. Not sure if jobs using c5d specifically need SSDs or not but I can check for those too.

Edit: A quick search find .github/workflows -type f | xargs grep 'c5d' does not find any workflows using c5d systems in pytorch/pytorch currently. Maybe that specific instance type is unused currently.

[zxiiro]

I made a spreadsheet [1] and did some really simple testing runs with each of the instance types, ran 3 times just in case anomalies showed up. I think in general moving the c7i is the right move as the 5% increase in price justifies the 15% - 20% decrease in time spent running the builds, we should see some savings here based on this.

The AMD c7a is 20% more expensive but most of the decrease in time is similar to the Intel c7i in most cases not justifying the increase in price. However, I think there is a few cases where moving to c7a might be a worthy move. Specifically these jobs:

• linux-jammy-py3.10-gcc11-pch (~34.5% decreased time)
• linux-jammy-py3.10-clang12-onnx (~24% decreased time)
• linux-jammy-py3.13-clang12 (~23.5% decreased time)
• linux-jammy-cuda12.8-cudnn9-py3.10-clang12 (~21% decreased time)
• linux-jammy-xpu-n-py3.9 (~35.5% decreased time)

[1] https://docs.google.com/spreadsheets/d/12QAkIFIUuioUeBLHfe8tn4J1sesyV3qoXPXUbxNWo78/edit?usp=sharing

[zxiiro]

We discussed this on the CI call yesterday. General consensus that moving to c7i is a good idea as a first step. The c7a optimization can be additionally performed as a separate task which we could investigate at a later time.

2 suggestions from Eli & Jean.

• Create new runner types that include c7i in the name and migrate the workflows rather than updating existing runners eg. linux.2xlarge
• Test each workflow one at a time when we migrate rather than deploying all at once to confirm that we in fact do get increase cost savings as there maybe jobs types where we don't see increase in time spent.

I will look into this and likely deploy to pull.yml first as I've already tested the build jobs here.

[zxiiro]

There's quite a few c5 runners and workflows that use them so I'm going to try to break them down into smaller tasks so that we can tackle them one at a time. For the larger runners like 9xlarge, 12xlarge, 24xlarge, it would be interesting also to see if we can reduce them to a lower size machine considering the increased performance of c7i systems.

I plan to prioritize Pull and Inductor workflows as I believe those will give us the biggest improvements.

Here's a list of all the c5 runners that currently exist:

c5.large: linux.large
c5.2xlarge: linux.2xlarge
c5.4xlarge: linux.4xlarge
c5.9xlarge: linux.9xlarge.ephemeral
c5.12xlarge: linux.12xlarge, linux.12xlarge.ephemeral
c5.24xlarge: linux.24xlarge, linux.24xlarge.ephemeral
c5d.4xlarge: windows.4xlarge, windows.4xlarge.nonephemeral

Workflows that use linux.large:

find .github/workflows -type f | xargs grep 'linux.large' | awk '{ print $1 }' | sort | uniq
.github/workflows/_bazel-build-test.yml:
.github/workflows/docker-release.yml:
.github/workflows/lint.yml:
.github/workflows/pull.yml:
.github/workflows/stale.yml:

Workflows that use linux.2xlarge:

find .github/workflows -type f | xargs grep 'linux.2xlarge' | awk '{ print $1 }' | sort | uniq
.github/workflows/_docs.yml:
.github/workflows/_link_check.yml:
.github/workflows/_linux-build.yml:
.github/workflows/build-magma-linux.yml:
.github/workflows/build-magma-rocm-linux.yml:
.github/workflows/create_release.yml:
.github/workflows/docker-release.yml:
.github/workflows/lint-autoformat.yml:
.github/workflows/lint.yml:
.github/workflows/nightly.yml:
.github/workflows/pull.yml:
.github/workflows/riscv64.yml:
.github/workflows/slow.yml:
.github/workflows/target_determination.yml:
.github/workflows/trunk.yml:
.github/workflows/upload-test-stats-while-running.yml:

Workflows that use linux.4xlarge:

find .github/workflows -type f | xargs grep 'linux.4xlarge' | awk '{ print $1 }' | sort | uniq
.github/workflows/_binary-test-linux.yml:
.github/workflows/build-triton-wheel.yml:
.github/workflows/generated-linux-binary-libtorch-nightly.yml:
.github/workflows/generated-linux-binary-manywheel-nightly.yml:
.github/workflows/llm_td_retrieval.yml:
.github/workflows/periodic.yml:
.github/workflows/pull.yml:
.github/workflows/slow.yml:
.github/workflows/trunk.yml:

Workflows that use linux.9xlarge:

find .github/workflows -type f | xargs grep 'linux.9xlarge' | awk '{ print $1 }' | sort | uniq
.github/workflows/build-almalinux-images.yml:
.github/workflows/build-libtorch-images.yml:
.github/workflows/build-manywheel-images.yml:

Workflows that use linux.12xlarge:

find .github/workflows -type f | xargs grep 'linux.12xlarge' | awk '{ print $1 }' | sort | uniq
.github/workflows/_binary-build-linux.yml:
.github/workflows/_docs.yml:
.github/workflows/b200-symm-mem.yml:
.github/workflows/build-vllm-wheel.yml:
.github/workflows/docker-builds.yml:
.github/workflows/h100-distributed.yml:
.github/workflows/inductor-perf-test-b200.yml:
.github/workflows/inductor-perf-test-nightly-h100.yml:
.github/workflows/inductor-perf-test-nightly.yml:
.github/workflows/inductor-unittest.yml:
.github/workflows/operator_benchmark.yml:
.github/workflows/operator_microbenchmark.yml:
.github/workflows/test-b200.yml:
.github/workflows/test-h100.yml:
.github/workflows/xpu.yml:

Workflows that use linux.24xlarge:

find .github/workflows -type f | xargs grep 'linux.24xlarge' | awk '{ print $1 }' | sort | uniq
.github/workflows/inductor-perf-test-nightly-x86-zen.yml:
.github/workflows/vllm.yml: