[SYCL RTC] Use in-memory libcxx/libc headers for SPIR-V targets by aelovikov-intel · Pull Request #20502 · intel/llvm

aelovikov-intel · 2025-10-29T16:58:12Z

While resulting in extra restrictions to ensure not ABI-violations when
passing data between host and device, that makes distribution of SYCL
applications using SYCL RTC much easier, so we're willing to pay that
price. Kernel compiler extension is being updated accordingly.

Ideally, we'd like to use the same STL implementation as the host code,
but we cannot re-distribute GNU libstdc++/libc or MS C++/C
headers (including Windows SDK) due to legal restrictions, so we embed
LLVM's libcxx/libc headers instead.

The way SYCL (non-RTC) works is that we use the same host C++/C
headers for device compilation and ignore the actual
libc.so/libstdc++.so as we provide known entry points ourselves (be it
libclc/libdevice/etc.). As such, for SYCL RTC, we don't actually need to
build those runtimes for a gpu target either, and can just use the
headers they provide. However, libc implementation of
entrypoints/headers on Windows is very incomplete, but we don't really
need that. All we need is declarations of C Standard Library/POSIX
functions and there is a way to pass some extra defines/configs to make
it generate that. As such, we avoid LLVM_ENABLE_RUNTIMES and instead
configure libcxx/libc as extra projects manually, passing needed option
and avoid full builds and installation for the regular host toolchain.o

Another caveat is that we have to disable system include paths (to
exclude regular libc headers as having two libc implementations is
causing all sorts of issues). As such, this approach isn't working for
CUDA/HIP targets that need corresponding SDKs. Previous behavior of
requiring/using system C++ toolchain is retained for those targets.

aelovikov-intel · 2025-11-04T15:45:19Z

libc/include/wchar.yaml

      - type: size_t
      - type: size_t
-      - type: mbstate_t
+      - type: mbstate_t *__restrict


This and the one below are part of the upstream llvm/llvm-project#164666.

aelovikov-intel · 2025-11-14T17:46:52Z

I've changed the approach to enable new mode by default whenever possible. Also merged latest origin/sycl as some minor changes were implemented in other PRs.

This is a significant change from the last reviewed state, so please re-review. CMake logic was updated a lot too.

clang/include/clang/Options/Options.td

sarnex · 2025-11-14T21:50:11Z

sycl-jit/jit-compiler/lib/resource-includes/include/sycl-rtc-standalone/__external_threading

+#define _LIBCPP_MUTEX_INITIALIZER 0
+
+using __libcpp_recursive_mutex_t = int;
+


I have a feeling I asked this before on another PR but are these files basically to support stuff not supported by LLVM's libc/cxx?

The short answer something didn't work when I used libcxx in (default) mode that uses pthreads to implement <thread>. GPU doesn't really support anything, so all we need is declarations and it's just easier to do this. I see that there is libc/include/pthread.yml, so maybe it would be possible to make that work, but there would be no benefit.

sycl-jit/jit-compiler/lib/rtc/DeviceCompilation.cpp

sarnex · 2025-11-14T21:55:12Z

sycl-jit/jit-compiler/lib/rtc/DeviceCompilation.cpp

+      // includes search path as those files aren't provide anywhere else.
+      AddInc("include/sycl-rtc-standalone/");
+#if !defined(_WIN32)
+      // On Windows `LIBCXX_GENERATED_INCLUDE_TARGET_DIR` is off and thus we


is this option always off for windows and always on for linux? or can the user set it or something?

Don't know. Probably off by default but can be set:

llvm/llvm/CMakeLists.txt

Lines 1011 to 1017 in ccac4e0

if(CMAKE_SYSTEM_NAME MATCHES "BSD|Linux|OS390|AIX")

set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)

else()

set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)

endif()

set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default} CACHE BOOL

"Enable per-target runtimes directory")

However, we configure libcxx ourselves and we don't pass many passthrough CMake variables, so I'd think it will be off reliably. Also, if somehow it changes, <__config_site> won't be found and our test-e2e/KernelCompiler tests would fail letting us know about it.

sycl-jit/jit-compiler/lib/rtc/DeviceCompilation.cpp

sarnex · 2025-11-14T22:12:58Z

sycl-jit/jit-compiler/utils/generate.py

+            os.path.realpath(
+                os.path.join(
+                    os.path.dirname(os.path.realpath(__file__)),
+                    "../lib/resource-includes/",


what do we get from using the relative path

I think os.path.relpath at line 39 didn't work for me without this. From the docs:

Return a relative filepath to path either from the current directory or from an optional start directory. This is a path computation: the filesystem is not accessed to confirm the existence or nature of path or start.

so my recollection is probably correct here.

sarnex · 2025-11-14T22:14:47Z

sycl-jit/jit-compiler/CMakeLists.txt

+endif()
+
+# Couldn't pass -DLLVM_ENABLE_RUNTIMES=<libc;libcxx;...> through CMAKE_ARGS
+# below because semicolon is used as a separate for CMAKE_ARGS itself.


Not sure I understand the limitation we're hiting

llvm/llvm/cmake/modules/LLVMExternalProjectUtils.cmake

Lines 68 to 73 in ccac4e0

function(llvm_ExternalProject_Add name source_dir)

cmake_parse_arguments(ARG

"ENABLE_FORTRAN;USE_TOOLCHAIN;EXCLUDE_FROM_ALL;NO_INSTALL;ALWAYS_CLEAN"

"SOURCE_DIR;FOLDER"

"CMAKE_ARGS;TOOLCHAIN_TOOLS;RUNTIME_LIBRARIES;DEPENDS;EXTRA_TARGETS;PASSTHROUGH_PREFIXES;STRIP_TOOL;TARGET_TRIPLE"

${ARGN})

would see ARGN as ...;-DVAR0=VAL0;-DLLVM_ENABLE_RUNTIMES=libc;libcxx;-DVAR1=VAL1 resulting in LLVM_ENABLE_RUNTIMES parsed as just libc because libcxx would become an extra argument instead of an extra value for LLVM_ENABLE_RUNTIMES. I tried to play with different escaping/double-quotes/etc., but couldn't make it work.

sycl/doc/extensions/experimental/sycl_ext_oneapi_kernel_compiler.asciidoc

sarnex · 2025-11-14T22:17:57Z

sycl/test-e2e/KernelCompiler/in_memory_only.cpp

+  // Without this we see stack overflows on Win, but for some reason only in
+  // `--sycl-rtc-in-memory-fs-only` mode when it should really be failing
+  // earlier.
+  opts.push_back("-fconstexpr-depth=128");


We might want to make sure we investigate this later to see it's not possible to hit it in the real use case

I don't think we can do much. And the fix (or workaround?) would be to either build user app with bigger stack or pass this option from inside the app to the SYCL RTC.

bader · 2025-11-18T02:10:09Z

libc/include/CMakeLists.txt

+  add_custom_target(generate-libc-headers
+                    DEPENDS libc-headers)


Let's commit libc and libcxx changes to https://github.com/llvm/llvm-project/ as well. It would be ideal if we merge them to https://github.com/llvm/llvm-project/ before merging this PR to gather the feedback from the LLVM's libc and libcxx communities.

I don't have an in-tree use-case to make upstream PR.

I don't have an in-tree use-case to make upstream PR.

Could you please check with libc/libcxx maintainers if such use case is required?

I don't feel comfortable asking to contribute something that I myself believe shouldn't go upstream. For bugfixes I intended to do so, only to find later that those were fixed in the trunk, so it's not that I refuse to work with them.

@bader , ping

@bader , ping

@aelovikov-intel, do you have a question for me?

No, I have an answer to your previous question. I don't think those changes can/should be upstreamed. If it's your stance that somebody has to go to community and get either an approval or rejection there before moving this PR forward, then we need to find another owner for this task.

aelovikov-intel · 2025-11-20T16:17:42Z

Ping everybody except @sarnex (who has approved already).

sycl/doc/extensions/experimental/sycl_ext_oneapi_kernel_compiler.asciidoc

sycl/test-e2e/KernelCompiler/sycl_host_device_data_layout.cpp

gmlueck

Looks good!

aelovikov-intel · 2025-11-24T22:08:54Z

@intel/llvm-reviewers-runtime, Chris is on vacation, can somebody else take a look?

aelovikov-intel temporarily deployed to WindowsCILock October 29, 2025 16:58 — with GitHub Actions Inactive

aelovikov-intel had a problem deploying to WindowsCILock October 29, 2025 17:29 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from f54abd9 to c9bdf63 Compare October 29, 2025 18:54

aelovikov-intel had a problem deploying to WindowsCILock October 29, 2025 18:54 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from c9bdf63 to 350e8f2 Compare October 29, 2025 19:02

aelovikov-intel had a problem deploying to WindowsCILock October 29, 2025 19:02 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 350e8f2 to 5c6f8b1 Compare October 29, 2025 19:41

aelovikov-intel had a problem deploying to WindowsCILock October 29, 2025 19:41 — with GitHub Actions Failure

aelovikov-intel had a problem deploying to WindowsCILock October 29, 2025 20:21 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 5c6f8b1 to 9d0562d Compare October 30, 2025 00:07

aelovikov-intel had a problem deploying to WindowsCILock October 30, 2025 00:08 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 9d0562d to 2299dcb Compare October 30, 2025 14:35

aelovikov-intel had a problem deploying to WindowsCILock October 30, 2025 14:35 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 2299dcb to 9032697 Compare October 30, 2025 15:15

aelovikov-intel had a problem deploying to WindowsCILock October 30, 2025 15:15 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 9032697 to 846f1c3 Compare October 30, 2025 17:01

aelovikov-intel temporarily deployed to WindowsCILock October 30, 2025 17:01 — with GitHub Actions Inactive

aelovikov-intel had a problem deploying to WindowsCILock October 30, 2025 17:33 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 846f1c3 to 99376a1 Compare November 3, 2025 15:49

aelovikov-intel had a problem deploying to WindowsCILock November 3, 2025 15:49 — with GitHub Actions Failure

aelovikov-intel had a problem deploying to WindowsCILock November 3, 2025 16:25 — with GitHub Actions Failure

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 99376a1 to 5506074 Compare November 3, 2025 21:42

aelovikov-intel temporarily deployed to WindowsCILock November 3, 2025 21:43 — with GitHub Actions Inactive

aelovikov-intel temporarily deployed to WindowsCILock November 3, 2025 22:19 — with GitHub Actions Inactive

aelovikov-intel commented Nov 4, 2025

View reviewed changes

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from 5506074 to 6fffeb8 Compare November 4, 2025 17:15

aelovikov-intel had a problem deploying to WindowsCILock November 7, 2025 16:09 — with GitHub Actions Failure

aelovikov-intel had a problem deploying to WindowsCILock November 7, 2025 16:59 — with GitHub Actions Error

aelovikov-intel force-pushed the sycl-rtc-runtimes-headers branch from f2edb85 to 289ae37 Compare November 7, 2025 17:01

aelovikov-intel had a problem deploying to WindowsCILock November 7, 2025 17:02 — with GitHub Actions Error

aelovikov-intel added 3 commits November 7, 2025 09:12

[WIP] Flip the default for libcxx/libc usage

649561a

Merge remote-tracking branch 'origin/sycl' into HEAD

99affbb

Better CMake integration

f7a3ed5

sarnex reviewed Nov 14, 2025

View reviewed changes

clang/include/clang/Options/Options.td Show resolved Hide resolved

sarnex reviewed Nov 14, 2025

View reviewed changes

Fix typos/add comments (thanks to Nick)

da5ee80

bader reviewed Nov 18, 2025

View reviewed changes

mdtoguchi approved these changes Nov 20, 2025

View reviewed changes

srividya-sundaram approved these changes Nov 20, 2025

View reviewed changes

elizabethandrews approved these changes Nov 20, 2025

View reviewed changes

gmlueck reviewed Nov 21, 2025

View reviewed changes

sycl/doc/extensions/experimental/sycl_ext_oneapi_kernel_compiler.asciidoc Outdated Show resolved Hide resolved

sycl/doc/extensions/experimental/sycl_ext_oneapi_kernel_compiler.asciidoc Outdated Show resolved Hide resolved

aelovikov-intel added 3 commits November 21, 2025 09:50

Merge remote-tracking branch 'origin/sycl' into HEAD

6a6770c

Update extension

fab3b51

Add layout test

d76c4df

gmlueck reviewed Nov 24, 2025

View reviewed changes

Partially address review feedback

3729836

gmlueck approved these changes Nov 24, 2025

View reviewed changes

clang-format

5f88bca

aelovikov-intel added 3 commits November 24, 2025 15:40

fp64 is optional

1067a08

Merge remote-tracking branch 'origin/sycl' into HEAD

4e0b4ba

clang-format

8a39ac6

cperkinsintel approved these changes Dec 1, 2025

View reviewed changes

		#define _LIBCPP_MUTEX_INITIALIZER 0

		using __libcpp_recursive_mutex_t = int;

	if(CMAKE_SYSTEM_NAME MATCHES "BSD\|Linux\|OS390\|AIX")
	set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)
	else()
	set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
	endif()
	set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default} CACHE BOOL
	"Enable per-target runtimes directory")

	function(llvm_ExternalProject_Add name source_dir)
	cmake_parse_arguments(ARG
	"ENABLE_FORTRAN;USE_TOOLCHAIN;EXCLUDE_FROM_ALL;NO_INSTALL;ALWAYS_CLEAN"
	"SOURCE_DIR;FOLDER"
	"CMAKE_ARGS;TOOLCHAIN_TOOLS;RUNTIME_LIBRARIES;DEPENDS;EXTRA_TARGETS;PASSTHROUGH_PREFIXES;STRIP_TOOL;TARGET_TRIPLE"
	${ARGN})

		add_custom_target(generate-libc-headers
		DEPENDS libc-headers)

Conversation

aelovikov-intel commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aelovikov-intel commented Nov 14, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aelovikov-intel commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmlueck left a comment

Choose a reason for hiding this comment

Uh oh!

aelovikov-intel commented Nov 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

aelovikov-intel commented Oct 29, 2025 •

edited

Loading