Homebrew package unable to find the file "ggml-metal.metal"

[mislav]

Hi, reporting something that is not necessarily a bug with this codebase, but a packaging problem. I'm not familiar with this project enough to determine that, so I'm leaving it here:

I was trying out this project by installing it via Homebrew:

brew install whisper-cpp

It seems to successfully transcribe audio, but it's very slow on an M1 MacBook. The output suggests that it wasn't able to use Metal backend:

whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1
ggml_metal_init: picking default device: Apple M1
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSURL=ggml-metal.metal -- file:///Users/mislav/Projects/whisper/, NSUnderlyingError=0x60000014c360 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed

The homebrew formula does include this file during installation and its location is share/whisper-cpp/ggml-metal.metal under its installation prefix: https://github.com/Homebrew/homebrew-core/blob/418ec8d5274aa16e99bdbb96dfc9c88f01a3b0ac/Formula/w/whisper-cpp.rb#L29

But I'm guessing at runtime whisper-cpp does not discover this file at this location. It can be helped by setting an environment variable:

GGML_METAL_PATH_RESOURCES="$(brew --prefix whisper-cpp)/share/whisper-cpp" whisper-cpp ...

Now it uses Metal and it seems noticeably faster. I think that ideally, setting GGML_METAL_PATH_RESOURCES should not be necessary as that the metal file should have been auto-discovered, but this might be a problem with how whisper-cpp is packaged within Homebrew. It that seems to be the case, please close this issue with a suggestion of a better location for the file, and I will go ahead and submit a PR to Homebrew. Thanks!

[ggerganov]

The homebrew formula is a bit outdated. It should be updated in a similar way to llama.cpp.rb. When this is done, the Metal library will be embedded in the binary and the env path would not be required. I just haven't had the time to look into this. If you are interested in giving it a try, please go ahead and feel free to ping me with any questions.

[ggerganov]

Ah and it will likely require updating whisper.cpp to the latest v1.7.3. I've currently release a pre-release version: https://github.com/ggerganov/whisper.cpp/releases/tag/v1.7.3-pre and planning to push the official v1.7.3 release soon.

[mislav]

Thanks for chiming in. After whisper.cpp v1.7.3 is out I can try improving the Homebrew formula for whisper-cpp.

BTW, did I open this issue in the wrong repo? I was meaning to open it in https://github.com/ggerganov/whisper.cpp; please feel free to move it there. Thank you!

[ggerganov]

It's ok to have the issue here - no problem.

https://github.com/ggerganov/whisper.cpp/releases/tag/v1.7.3 has just been released.

[mislav]

Here is the Homebrew PR to bump the version: Homebrew/homebrew-core#201851

It installs the "main" example as the whisper-cpp binary. This is the full directory tree of the resulting install:

$ tree /opt/homebrew/Cellar/whisper-cpp/*    
/opt/homebrew/Cellar/whisper-cpp/1.7.3
├── AUTHORS
├── INSTALL_RECEIPT.json
├── LICENSE
├── README.md
├── bin
│   └── whisper-cpp
├── ggml
│   └── src
│       ├── ggml-blas
│       │   └── libggml-blas.dylib
│       ├── ggml-metal
│       │   └── libggml-metal.dylib
│       ├── libggml-base.dylib
│       ├── libggml-cpu.dylib
│       └── libggml.dylib
├── sbom.spdx.json
├── src
│   ├── libwhisper.1.7.3.dylib
│   ├── libwhisper.1.dylib -> libwhisper.1.7.3.dylib
│   └── libwhisper.dylib -> libwhisper.1.dylib
└── test
    ├── for-tests-ggml-tiny.bin
    └── jfk.wav

To figure out the right build incantation was very much trial-and-error on my side, so please correct me if I'm doing something obviously wrong. Notice that the installation now produces dylib files, whereas the build system for v1.5.4 (a simple make invocation, it seems) did not produce any. The compiled main binary depends on these dylibs at runtime. However, the Metal support is successfully embedded, like you promised, so an external ggml-metal.metal file isn't needed anymore. Finally, I keep for-tests-ggml-tiny.bin and jfk.wav there merely for the brew test whisper-cpp functionality. This all seems to work, so I've submitted it to Homebrew in this format.

---

P.S. here is some feedback about the build system from someone who has zero experience with cmake, but who has a solid grasp of the ./configure && make && make install combo from other projects.

(BTW, everything that follows is in the context of the https://github.com/ggerganov/whisper.cpp repo.)

On a fresh download of this project's source code, I've tried this (simplified for brevity):

cmake -B "$BUILD_DIR" -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" [args...]
cmake --build "$BUILD_DIR" --target main
cmake --install "$BUILD_DIR"

What I expected was that the main binary that I specifically asked for with --target main would be a part of the final installation. But that wasn't the case, as this was the resulting INSTALL_DIR:

/tmp/whisper-install
├── bin
│   └── ggml-metal.metal
├── include
│   ├── ggml-alloc.h
│   ├── ggml-backend.h
│   ├── ggml-blas.h
│   ├── ggml-cann.h
│   ├── ggml-cpu.h
│   ├── ggml-cuda.h
│   ├── ggml-kompute.h
│   ├── ggml-metal.h
│   ├── ggml-opt.h
│   ├── ggml-rpc.h
│   ├── ggml-sycl.h
│   ├── ggml-vulkan.h
│   ├── ggml.h
│   └── whisper.h
└── lib
    ├── cmake
    │   └── whisper
    │       ├── whisper-config.cmake
    │       └── whisper-version.cmake
    ├── libggml-base.dylib
    ├── libggml-blas.dylib
    ├── libggml-cpu.dylib
    ├── libggml-metal.dylib
    ├── libggml.dylib
    ├── libwhisper.1.7.3.dylib
    ├── libwhisper.1.dylib -> libwhisper.1.7.3.dylib
    ├── libwhisper.dylib -> libwhisper.1.dylib
    └── pkgconfig
        └── whisper.pc

7 directories, 26 files

It looks like the product of the install command are header files and dylibs, which is fine for other programs to use when linking to the whisper library, but I'm mainly interested in the main executable (which I now understand is only an example program for the libwhisper library).

The main program is still in the build directory, but is configured to load dylibs not from INSTALL_DIR, but from BUILD_DIR:

$ DYLD_PRINT_LIBRARIES=1 DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 DYLD_PRINT_RPATHS=1 /tmp/whisper-build/bin/main 2>&1 | grep whisper-build 
dyld[30327]: <E1FE1DB8-B103-34C9-BB31-795823E3D463> /private/tmp/whisper-build/bin/main
dyld[30327]: <176162D3-B7E0-3B15-903C-E64C517C2547> /private/tmp/whisper-build/src/libwhisper.1.7.3.dylib
dyld[30327]: <17593FD3-96D1-36AC-8438-BE1EF34278F1> /private/tmp/whisper-build/ggml/src/libggml.dylib
dyld[30327]: <F743383C-518A-3DF5-9D3A-6B6916A8DBD2> /private/tmp/whisper-build/ggml/src/libggml-cpu.dylib
dyld[30327]: <1D570838-3F60-3400-BCC6-CA8D135BEC4A> /private/tmp/whisper-build/ggml/src/ggml-blas/libggml-blas.dylib
dyld[30327]: <E0996A4F-E71C-3072-8E52-E5E66D2F1D07> /private/tmp/whisper-build/ggml/src/ggml-metal/libggml-metal.dylib
dyld[30327]: <47234C84-317E-3276-9871-427F0A46CF61> /private/tmp/whisper-build/ggml/src/libggml-base.dylib
usage: /tmp/whisper-build/bin/main [options] file0.wav file1.wav ...

I find this to be a problem, because if I move main to a desired location and then rm -rf $BUILD_DIR as usual part of post-installation cleanup, main cannot find required libraries anymore. I would expect that it searches for them in INSTALL_DIR instead. Could this be an oversight?

And finally, a small feature request: I wonder if my system already has libggml dylibs, the build system for this repo could be instructed to link to those at runtime instead of building fresh ones? For example, I can use Homebrew to install llama.cpp:

$ brew install llama.cpp && ls -l /opt/homebrew/lib/libggml*
...
lrwxr-xr-x  1 mislav  admin  47 Dec 20 01:17 /opt/homebrew/lib/libggml-base.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-base.dylib
lrwxr-xr-x  1 mislav  admin  47 Dec 20 01:17 /opt/homebrew/lib/libggml-blas.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-blas.dylib
lrwxr-xr-x  1 mislav  admin  46 Dec 20 01:17 /opt/homebrew/lib/libggml-cpu.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-cpu.dylib
lrwxr-xr-x  1 mislav  admin  48 Dec 20 01:17 /opt/homebrew/lib/libggml-metal.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-metal.dylib
lrwxr-xr-x  1 mislav  admin  42 Dec 20 01:17 /opt/homebrew/lib/libggml.dylib -> ../Cellar/llama.cpp/4367/lib/libggml.dylib

This gives me libggml*.dylib that are symlinked into /opt/homebrew/lib and presumeably available for other programs to link to. I imagine that at build time, the future whisper-cpp Homebrew formula could be made to depend on llama.cpp and then reuse these libraries instead of having to build them? I could not figure out how to reuse existing libggml*.dylib with the current build system.

Please forgive me if I'm not making sense; I just have a basic understanding of dylibs in general, which is that they're needed at runtime and that they are discovered in a list of locations that is dictated by the program's embedded configuration (and possibly some environment variables).

And finally, thank you for this awesome project. It certainly solved my problem (a personal tool to grep podcasts I've listened to) and I hope this can help you improve the build system for everyone who comes next. 💫

[ggerganov]

Thank you for the very detailed explanation! I will attempt to address the install and link problems in ggml-org/whisper.cpp#2648.

Regarding the shared libggml - not sure we can do it at this point because llama.cpp and whisper.cpp often depend on a different version of the library. Also having one package depend on the other does not seem ideal. Maybe in the future ggml can become a separate package that installs the library and the other packages (such as whisper.cpp and llama.cpp) depend on it.

[ggerganov]

Btw, I only now realize that the libggml binaries from the llama.cpp package are symlinked into /opt/homebrew/lib. I wonder if this is a good idea due to the explanation in my previous comment.

[mislav]

Btw, I only now realize that the libggml binaries from the llama.cpp package are symlinked into /opt/homebrew/lib. I wonder if this is a good idea due to the explanation in my previous comment.

Ah, I thought that was intentional for the llama.cpp Homebrew formula. The way Homebrew works is that if the install block of the formula places any headers into #{prefix}/include or libraries in #{prefix}/lib, those will automatically get symlinked into /opt/homebrew. I don't know if there is a way to disable that apart from marking a formula as "keg only", in which case nothing ever gets symlinked, not even the binaries in bin. If you don't want libraries from llama.cpp to get exposed, I think the best way would be for the installer to place them in a place different than #{prefix}/lib.

[ggerganov]

Yeah, so as far as I understand, the correct way to distribute both the llama.cpp and whisper.cpp packages on Homebrew would be to have a common package ggml that would deploy the shared libggml* binaries to be used by both projects (and potentially others in the future). At the moment, there would be a clash of the libggml binaries from the 2 projects and I am not sure if Homebrew has some mechanism to resolve this.

So I am thinking that there are 2 possible solutions:

• We make a ggml Homebrew package
• Find some way in which both llama.cpp and whisper.cpp ship their own versions of libggml and link to those respectively

In the long term, the first option is what we should aim for, but I am not sure if we are ready for it at the moment. Maybe there is some way to make the second option work somehow?

I will cc @slaren as he has very good knowledge about the build system and might have some additional insights on the topic.

[slaren]

I am looking into packaging now, I am just not sure yet what's the best way to handle it. Generally I think we should have separate packages for the libraries that other developers can also use, but that will probably require at least us to start versioning ggml, which I am not sure that would be compatible with the way we currently have a duplicate of ggml in every repository.

Some notes that may fix the immediate issues:

• Running cmake --install should clear the dynamic library search path
• You can also change the search path by setting CMAKE_INSTALL_RPATH
• If there is a single executable, which I believe is the case for the whisper.cpp brew package, then a static build should also work. This can be done by disabling BUILD_SHARED_LIBS. In the long term we should avoid this in favor of separate packages for the libraries and backends, but it should solve the immediate problem.

[mislav]

• If there is a single executable, which I believe is the case for the whisper.cpp brew package, then a static build should also work. This can be done by disabling BUILD_SHARED_LIBS

@slaren Thanks for that tip; I was able to simplify the Homebrew formula using that flag. https://github.com/mislav/homebrew-core/blob/e4dbcb99e2ce4ab6cbd167126578afe2fa795b9f/Formula/w/whisper-cpp.rb#L28-L42

It was apparently needed too, because with shared libs, the Homebrew formula seemed have been failing on Linux due to a dylib not being found: https://github.com/Homebrew/homebrew-core/actions/runs/12422801076/job/34685263836. I've only ever tested the formula on macOS, so I'm not sure what is different on Linux. By doing BUILD_SHARED_LIBS=OFF, I was able to avoid having to debug the Linux failure and just end up with a single self-contained binary. 👍

[ggerganov]

I pushed a v1.7.4-pre-0 tag that renames the binaries for consistency with llama.cpp. For example, the main binary now is called whisper-cli.

I also fixed the install rule for the binary so it should appear in the bin folder.

You can also change the search path by setting CMAKE_INSTALL_RPATH

@mislav What if we keep the shared libraries build ON and try to set the RPATH relative to the binaries, such as: ../lib. Not sure if this is correct, but my idea is to try to make the whisper-cli first look in the local lib folder for libggml and libwhisper. This way it would not matter which version of libggml is symlinked in /opt/homebrew/lib.

[mislav]

Update: Homebrew v1.7.3 formula has been merged and the original issue in this thread fixed, so as far as I'm concerned, this can be closed and followup work tracked in separate issues. 🎉

What if we keep the shared libraries build ON and try to set the RPATH relative to the binaries, such as: ../lib

I'm not familiar with RPATH at all, but it sounds like that might work. However, now that we have a self-contained binary in Homebrew, what incentive would we have to break out dylibs into separate files anymore? As previously concluded in this thread, if both llama.cpp and whisper-cpp tried to publish libggml dylibs, Homebrew would deem these formulae incompatible with each other due to filename clash.

But in any case, feel free to share suggestions here and if I can help apply them to future versions of the Homebrew formula, I will do my best. Note that am not a machine learning developer of any sort, so I am not a target audience for libggml libraries and cannot weigh in on how they should be used. However, I support the previously expressed idea that in the future where libggml is more "stable" and is versioned on its own, both llama.ccp and whisper-cpp could be made to depend on it externally.

I pushed a v1.7.4-pre-0 tag that renames the binaries for consistency with llama.cpp. For example, the main binary now is called whisper-cli.

Great, thank you! BTW, in future Homebrew formula upgrades for this, should the Homebrew-installed binary be continued to be called whisper-cpp (to match the formula name) or renamed to whisper-cli (to match what the installer generates)?

[ggerganov]

However, now that we have a self-contained binary in Homebrew, what incentive would we have to break out dylibs into separate files anymore?

The static builds are not ideal because ggml has a mechanism to load a dynamic backend library that is optimized for the specific user CPU that is going to run whisper.cpp. This would allow to take advantage of all the CPU features and get the optimal performance and we won't be able to use this mechanism with BUILD_SHARED_LIBS=OFF.

However, I support the previously expressed idea that in the future where libggml is more "stable" and is versioned on its own, both llama.ccp and whisper-cpp could be made to depend on it externally.

Yes, this seems like the ultimate solution.

BTW, in future Homebrew formula upgrades for this, should the Homebrew-installed binary be continued to be called whisper-cpp (to match the formula name) or renamed to whisper-cli (to match what the installer generates)?

For consistency with the llama.cpp package where we have llama-cli, llama-server, etc, it would be better to call it whisper-cli.