[BUG]: 0.24.0 CUDA detection bug

[BifrostTitan]

Description

CPU will always load. When using fallback set to false and skipcheck() it will crash and throw the native api exception. Forcing CUDA does not work.

Temp Solution:
Use NextCoder model instead of my Qwen3 model and use version 0.23.0 of llamasharp with the cuda backend.
Llama loads up and shows backend_ptrs.size() = 2. This normally shows 1 when using 0.24.0.

Problem comes from 0.24.0 specifically.

Strange part:
I have 0.24.0 CUDA working on a MAUI application that is available on the Microsoft store so I know 0.24.0 works for CUDA but it uses a custom library that doesnt seem to work for the console/API server application.

Changing the targeted type from Any CPU to x64 is required to find the backend .dlls as well so x86 does not work correctly.

Reproduction Steps

Attempt to use Qwen3 model with latest llamasharp cuda backend.
Create a .NET 9 console application or API server and reference the latest CUDA backend.

Load the weights and look at where its assigned.

Environment & Configuration

• Operating system: Windows 11
• .NET runtime version: .NET 9
• LLamaSharp version: 0.24.0
• CUDA version (if you are using cuda backend): 12.6 & 12.9
• CPU & GPU device: RTX 3090

Known Workarounds

No response

[Crelex]

I have a fix for this if you want to pull down my branch it has it included.

https://github.com/Crelex/LlamaSharp

[BifrostTitan]

Paths and version are working correctly when I debug. This issue comes from the DLLs themselves. On Windows it fails to load ggml-base.dll and llama.dll but it loads ggml-cuda.dll successfully. Path is correct since it loads 2 out of the 4 libraries but 2 always fail. When running 0.23.0 all the DLLs load successfully.

System.Runtime.InteropServices.NativeLibrary.TryLoad(fullPath, out var handle) returns false

#if NET6_0_OR_GREATER
            // Try to load a DLL from the path.
            // Returns null if nothing is loaded.
            static IntPtr TryLoad(string path, IEnumerable<string> searchDirectories, NativeLogConfig.LLamaLogCallback? logCallback)
            {
                var fullPath = TryFindPath(path, searchDirectories);
                Log($"Found full path file '{fullPath}' for relative path '{path}'", LLamaLogLevel.Debug, logCallback);
                bool exists = File.Exists(fullPath);
                Log($"Path Exist:{exists}", LLamaLogLevel.Debug, logCallback);
                if (NativeLibrary.TryLoad(fullPath, out var handle))
                {
                    Log($"Successfully loaded '{fullPath}'", LLamaLogLevel.Info, logCallback);
                    return handle;
                }

                Log($"Failed Loading '{fullPath}'", LLamaLogLevel.Info, logCallback);
                return IntPtr.Zero;
            }
#endif
        }

[m0nsky]

Hi @BifrostTitan , can you use a dependency viewer like this to check out why ggml-base.dll and llama.dll aren't loading for you?

[TehWardy]

I'm hitting this too ... i've been fighting this all day.
I even took to building llama.cpp and then using NativeLibraryConfig to point at the new build but the bindings are somehow no longer aligned so it just explodes on me.

I've also noticed the latest cuda version is v13 ... so after installing both 13 and 12 I still got no luck.
When I reference the cuda 12 backend it pulls in the right backend files into the build output folder then always falls back to CPU.
There seems to be no way to override it even when explicitly pointing at it with NativeLibraryConfig settings.

Note:
Seems to still be present in 0.20.0 to 0.25.0 ...
I figure if I go back much further I think i'll be back beyond the date the gPU code was written.
As far as I can tell ... this just doesn't work unless I've missed someting?

[martindevans]

I even took to building llama.cpp and then using NativeLibraryConfig to point at the new build but the bindings are somehow no longer aligned so it just explodes on me.

Any given LLamaSharp version corresponds to exactly one llama.cpp version, there's no stability between versions of llama.cpp! You can build your own binaries, but you must use exactly the right commit. Check the table at the bottom of the readme for the exact versions.

If you clone this repo and run the examples, do you see the same behaviour with failing to use CUDA?