Update workflows to gcc-toolset-13 and fix OpenBLAS build #1746

jayhawk-commits · 2025-10-09T19:04:15Z

Update workflows to use new manylinux docker image produced by Update manylinux containers to use gcc-toolset-13 #1745
For Linux, turn on DYNAMIC_ARCH for OpenBLAS build to resolve OpenBLAS with -DDYNAMIC_ARCH=ON produces illegal ELF files #83.
DYNAMIC_ARCH support does not exist for MSVC and multi-arch support for Windows seems shoddy overall. Reference 1 and Reference 2
Remove usage of f2c version of an older lapack as it is considered fallback option. Reference
Patch amd-llvm compiler to be able to find stdc++ from gcc-toolset-13 during link time.

jayhawk-commits · 2025-10-10T19:34:25Z

Build is past third-party deps, so signs are promising. Windows job is failing at environment setup steps due to infra issues.

jayhawk-commits · 2025-10-10T23:36:13Z

The previously failing tests for loading the OpenBLAS library from turning DYNAMIC_ARCH on pass in the CI run.

HereThereBeDragons

the failure of rocm sanitycheck on linux seem to be a problem how the docker container is started and what group access it has.

INFO     test_rocm_sanity:test_rocm_sanity.py:24 ++ Run [None]$ /__w/TheRock/TheRock/build/bin/rocminfo
ERROR    test_rocm_sanity:test_rocm_sanity.py:29 Command failed!
ERROR    test_rocm_sanity:test_rocm_sanity.py:30 command stdout:
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 ROCk module version 6.12.12 is loaded
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Unable to open /dev/kfd read-write: Permission denied
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Failed to get user name to check for video group membership

HereThereBeDragons · 2025-10-13T08:04:28Z

third-party/host-blas/CMakeLists.txt

-        -DC_LAPACK=ON
+        -DTARGET=CORE2
+        -DDYNAMIC_ARCH=ON
+        -DDYNAMIC_OLDER=ON


do we need dynamic_older?

from the changelog:

Version 0.3.1 01-Jul-2018 ... x86_64: retired some older targets of DYNAMIC_ARCH builds to a new option DYNAMIC_OLDER, this affects PENRYN,DUNNINGTON,OPTERON,OPTERON_SSE3,BOBCAT,ATOM and NANO (which will still be supported via the slower PRESCOTT kernels when this option is not set)

I won't comment specifically about need, but I picked for breadth of CPU coverage for potential users of TheRock.

MSVC seems to have issues with small matrix optimizations required to support older CPUs, so Windows build compatibility might dictate how far back the build flags can go.

Maybe we can compile this with [amd]clang instead of msvc?

This comment block has me concerned about trying that. https://github.com/ROCm/TheRock/blob/main/core/CMakeLists.txt#L2C1-L11C10

I'll see if removing DYNAMIC_OLDER fixes the issue. If not, I will try HASWELL which should be a minimum baseline for x86_64 CPUs from Intel and AMD from the past decade. I think I remember one build recipe picking that one.

OK, looks like MSVC's limitation with DYNAMIC_ARCH is known OpenMathLib/OpenBLAS#2826 (comment)

My latest set of build flags, which limits DYNAMIC_ARCH to Linux, while still leveraging a TARGET that should support almost all AMD and Intel CPUs from the past decade is able to compile on both our Linux and Windows workflows now.

third-party/host-blas/CMakeLists.txt

jayhawk-commits · 2025-10-14T13:51:39Z

the failure of rocm sanitycheck on linux seem to be a problem how the docker container is started and what group access it has.

INFO     test_rocm_sanity:test_rocm_sanity.py:24 ++ Run [None]$ /__w/TheRock/TheRock/build/bin/rocminfo
ERROR    test_rocm_sanity:test_rocm_sanity.py:29 Command failed!
ERROR    test_rocm_sanity:test_rocm_sanity.py:30 command stdout:
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 ROCk module version 6.12.12 is loaded
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Unable to open /dev/kfd read-write: Permission denied
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Failed to get user name to check for video group membership

I believe there were issues late last week where it was not picking pods with GPUs.

jayhawk-commits · 2025-10-14T13:56:48Z

the failure of rocm sanitycheck on linux seem to be a problem how the docker container is started and what group access it has.

INFO     test_rocm_sanity:test_rocm_sanity.py:24 ++ Run [None]$ /__w/TheRock/TheRock/build/bin/rocminfo
ERROR    test_rocm_sanity:test_rocm_sanity.py:29 Command failed!
ERROR    test_rocm_sanity:test_rocm_sanity.py:30 command stdout:
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 ROCk module version 6.12.12 is loaded
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Unable to open /dev/kfd read-write: Permission denied
ERROR    test_rocm_sanity:test_rocm_sanity.py:32 Failed to get user name to check for video group membership

I believe there were issues late last week where it was not picking pods with GPUs.

Sanity test failures did not occur just now, with some of the infra issues fixed, when I just re-ran the failures.

jayhawk-commits · 2025-10-14T15:10:23Z

Seeing error below with Windows builds with the new flags:

2025-10-14T14:21:39.8009716Z [therock-host-blas configure] CMake Error in B:/build/third-party/host-blas/source/kernel/CMakeLists.txt:
2025-10-14T14:21:39.8009916Z [therock-host-blas configure]   MSVC_RUNTIME_LIBRARY value 'MultiThreadedDLL' not known for this ASM
2025-10-14T14:21:39.8009989Z [therock-host-blas configure]   compiler.

third-party/host-blas/CMakeLists.txt

- Addresses #1877 - #1746 introduced an issue in the pytorch workflows, which aren't included in pre-submit CI, so the issue was missed then. - Switching back to `C_LAPACK` for now as quickest solution. - `CMAKE_CROSSCOMPILING` is not needed for our current use case. # Test Plan - Trigger the pytorch workflows to verify fortran issue is not present.

- Update workflows to use new manylinux docker image produced by #1745 - For Linux, turn on `DYNAMIC_ARCH` for OpenBLAS build to resolve #83. - `DYNAMIC_ARCH` support does not exist for MSVC and multi-arch support for Windows seems shoddy overall. [Reference 1](OpenMathLib/OpenBLAS#2826 (comment)) and [Reference 2](OpenMathLib/OpenBLAS#3985) - Remove usage of f2c version of an older lapack as it is considered fallback option. [Reference](OpenMathLib/OpenBLAS#4318 (comment)) - Patch amd-llvm compiler to be able to find stdc++ from gcc-toolset-13 during link time.

- Addresses #1877 - #1746 introduced an issue in the pytorch workflows, which aren't included in pre-submit CI, so the issue was missed then. - Switching back to `C_LAPACK` for now as quickest solution. - `CMAKE_CROSSCOMPILING` is not needed for our current use case. # Test Plan - Trigger the pytorch workflows to verify fortran issue is not present.

Point to staging docker build for comprehensive CI/CD testing

7fe51cf

github-project-automation bot moved this to TODO in TheRock Triage Oct 9, 2025

github-project-automation bot added this to TheRock Triage Oct 9, 2025

jayhawk-commits and others added 4 commits October 9, 2025 23:51

Add gcc-toolset-13 prefix detection

3371906

Reference SHA of docker with gcc 13

87906a4

Merge branch 'main' into users/jayhawk-commits/update-gcc-toolset-13

a891f1f

Follow Fedora OpenBLAS build flags

6f27f7c

jayhawk-commits changed the title ~~[DO NOT MERGE] Test gcc-toolset-13 update~~ Update workflows to gcc-toolset-13 and fix OpenBLAS build Oct 10, 2025

Change ending of new patch to match other patches

b27bbc2

jayhawk-commits mentioned this pull request Oct 10, 2025

OpenBLAS: Set CMAKE_CROSSCOMPILING #1700

Closed

jayhawk-commits marked this pull request as ready for review October 10, 2025 23:36

jayhawk-commits requested review from HereThereBeDragons, ScottTodd and marbre October 10, 2025 23:36

HereThereBeDragons reviewed Oct 13, 2025

View reviewed changes

This was referenced Oct 14, 2025

Update manylinux containers to use gcc-toolset-13 #1745

Merged

[Documentation]: GCC toolset is now 13 but documented as 12 #1779

Closed

jayhawk-commits requested a review from HereThereBeDragons October 14, 2025 13:57

jayhawk-commits added 6 commits October 14, 2025 11:52

Unset MultithreadedDLL flag for OpenBLAS MSVC build

4f350d1

Update CMakeLists.txt

9dd409d

Set HASWELL baseline

52d5c72

Fixes to build on MSVC

23fe22f

Forgot to remove amd-llvm ref

18ee303

CMAKE_MSVC_RUNTIME_LIBRARY workaround not needed anymore.

855fc71

marbre reviewed Oct 15, 2025

View reviewed changes

third-party/host-blas/CMakeLists.txt Show resolved Hide resolved

jayhawk-commits requested a review from marbre October 15, 2025 16:48

marbre approved these changes Oct 17, 2025

View reviewed changes

jayhawk-commits merged commit f8df463 into main Oct 17, 2025
79 of 81 checks passed

jayhawk-commits deleted the users/jayhawk-commits/update-gcc-toolset-13 branch October 17, 2025 18:38

github-project-automation bot moved this from TODO to Done in TheRock Triage Oct 17, 2025

This was referenced Oct 17, 2025

Update manylinux build docker to gcc-toolset-13 #1738

Closed

[Issue]: rocBLAS via OpenBLAS seg-faulting during unit tests #1605

Open

OpenBLAS flag updates to address #1877 #1914

Merged

Update workflows to gcc-toolset-13 and fix OpenBLAS build #1746

Update workflows to gcc-toolset-13 and fix OpenBLAS build #1746

Uh oh!

Conversation

jayhawk-commits commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jayhawk-commits commented Oct 10, 2025

Uh oh!

jayhawk-commits commented Oct 10, 2025

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

jayhawk-commits Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jayhawk-commits Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jayhawk-commits Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jayhawk-commits Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jayhawk-commits Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jayhawk-commits commented Oct 14, 2025

Uh oh!

jayhawk-commits commented Oct 14, 2025

Uh oh!

jayhawk-commits commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jayhawk-commits commented Oct 9, 2025 •

edited

Loading