feat(macOS): Capture audio on macOS using Tap API#4209
feat(macOS): Capture audio on macOS using Tap API#4209ThomVanL wants to merge 39 commits intoLizardByte:masterfrom
Conversation
ThomVanL
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Bundle ReportChanges will increase total bundle size by 121 bytes (0.01%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sunshine-esmAssets Changed:
Files in
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4209 +/- ##
==========================================
- Coverage 12.09% 12.09% -0.01%
==========================================
Files 87 87
Lines 17612 17613 +1
Branches 8097 8097
==========================================
Hits 2131 2131
- Misses 14579 14580 +1
Partials 902 902
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Thanks for the PR. I have built this on my M1 Pro MBP and have been trying to get it to work, but haven't had much success so far. I am not able to get any sound to be captured and/or sent. So far I am only testing with the simplest use case of audio playing out of my MBP speakers. I edited the code to make the tap and aggregate non-private, so I could try to view the tap/aggregate with Apple's sample app I can see the tap, but not the aggregate. I'm not sure what's wrong. Can you detail your testing process? I do seem to be getting OK log entries and since I'm running from iTerm, all my permissions seem to be in order (iTerm has access to many things). I had to make the following changes to get it to build: plus a fix in our input.cpp that breaks the latest Xcode 16.4, I'm surprised if you didn't run into this one. I am running Xcode 16.4 clang-1700.0.13.5). |
|
Posting this here for reference in case it's needed for permissions of unit tests. We used to need something like this for macports, but it was never necessary for homebrew. Sunshine/.github/workflows/CI.yml Lines 841 to 898 in b662b8e |
|
Hey @andygrundman, thanks for taking the time to test the PR. Sorry to hear it’s not working correctly. Here’s my setup on an M4 MBP and what I did. clang --version
Apple clang version 17.0.0 (clang-1700.0.13.5)
Target: arm64-apple-darwin24.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/binI made the changes in VS Code and followed the steps in mkdir build
cmake -B build -G Ninja -S .
ninja -C buildFor testing I did the following steps:
So at one point I'm running three streams simultaneously and the audio plays through the devices. My audio_sink = Steam Streaming Speakers
macos_system_wide_audio_tap = true
origin_web_ui_allowed = pc
stream_audio = true
wan_encryption_mode = 2With I’m familiar with the sample app! It might be the case that the aggregate device settings are probably still marked as private. You’ll need to flip those to Sunshine/src/platform/macos/av_audio.mm Line 604 in 6404705 Sunshine/src/platform/macos/av_audio.mm Line 649 in 6404705 Then the tap will show up in Apple's sample app's UI. 
And the aggregate device shows up as well. 
But even with those values flipped to |
|
Thanks for the detailed info. I forgot my log only had Info level, when using Debug my log does look exactly like yours. I feel like I must just have a permission issue, maybe using iTerm as the permission "owner" isn't correct. Do you see both of these items? What process names are they using? I only get a System Audio item when recording a test file in AudioTapSample. Here's basically what my Screen & System Audio Recording settings look like: If this is the issue, I wonder how Sunshine can detect that it doesn't actually have permission, hmm. |
|
You're right, because I ran into a similar issue with VS Code. Granting it "Screen & System Audio Recording" wasn’t enough; I had to explicitly allow "System Audio Recording Only." From what I found online, the VS Code app bundle itself might be causing the problem. I also tried running 
I did not even notice the little privacy notice at the top until just now, thanks for that. Here's what it's like on my end. 
When I launched Edit: just to be clear, my dev loop consists of launching sunshine builds through the VS Code integrated terminal. |
ThomVanL
force-pushed
the
users/thomasvanlaere/feat-macos-ca-taps
branch
from
d75fde0 to
db3d2df
Compare
…e/feat-macos-ca-taps
|
|
Hi @ReenigneArcher and @andygrundman, happy new year! 🎉 Also, my apologies for the delay in regards to this PR. I've removed some of the unit tests that turned out to be integration tests in disguise (particularly the TCC ones). After poking around for best practices on running integration tests in a CI/CD pipeline with TCC, I mostly found workarounds that felt a little hacky, so I decided to strip those integration tests out and stick with pure unit tests instead. Not sure if that sits right with you both, but feel free to let me know if it needs to be changed. |
This comment was marked as outdated.
This comment was marked as outdated.
|
This is looking pretty good, thanks! What I did was create a branch from yours, cherry-pick in some helper things (my build script, input fixes, my previous compile fix (how does this build for you in Tahoe?), and moved a bunch of noisy logging from debug to verbose). With all that out of the way, the main commit in my branch has a bunch of improvements for you to consider. Here's the items in that commit: https://github.com/andygrundman/Sunshine/commits/andyg.feat-macos-ca-taps-review/ I'm not married to these changes, so feel free to counter any of them. I'm not sure what the right buffer size should be, but I felt like 4096 * channels seemed small and just as arbitrary. The public tap thing is neat to look at, especially check out the Apple utility HALLab which exposes all the internals of audio devices. I worry that public taps may not get cleaned up in some situations such as a crash, although maybe private taps get automatically cleaned up. Using an env var for devs is a bit ugly but there's no other way to view the aggregate device. Re: resampling: do we need to have so much or any resampling code? Doesn't CoreAudio handle a lot of this behind the scenes? I couldn't get the system tap to ever need resampling (I think because it's not using a specific device), and when using BlackHole 16ch sink directly, the resampling code doesn't seem to be called anyway. Even if I run BlackHole at 192k or something, it seems to work fine. I did a trace in Activity Monitor and you can find the bit where the OS is resampling from 192k -> 48k if you search for This all works quite well for stereo. For surround, I guess we need to think about it some more. I just stumbled across a project called SoundPusher that creates a virtual 5.1 device and does on-demand AC3 encoding. I want to study it some more. https://codeberg.org/q-p/SoundPusher I found it via https://www.maven.de/2025/04/coreaudio-taps-for-dummies/ by the way. Edit: I think I just realized the conversion code is mostly there for multichannel right? I honestly have not even tested surround because I think that still requires a virtual audio device like what SoundPusher includes, or BlackHole. I'm not sure it works with Tap. This PR should probably just focus on making stereo taps work in the most user-friendly way possible. |
|
Another edit: I had a few moments of panic while testing this, you know the feeling: the audio is skipping and glitching out real bad, and your first thought is there's some bug in the realtime buffer code. Nope, it's all just AWDL. This code basically becomes unusable on wifi devices that don't have N1 chips. The iPad I have been using as a client still had 6ghz 6E mode enabled which is a death sentence for Moonlight. My N1-equipped iPhone Air streams it perfectly. Thanks Apple. |
cmake/dependencies/common.cmake
Outdated
| if(OPENSSL_FOUND) | ||
| include_directories(SYSTEM ${OPENSSL_INCLUDE_DIR}) | ||
| endif() |
There was a problem hiding this comment.
This shouldn't be necessary?
cmake/dependencies/macos.cmake
Outdated
| include_directories(/opt/homebrew/opt/opus/include) | ||
| link_directories(/opt/homebrew/opt/opus/lib) |
There was a problem hiding this comment.
These paths depend on the architecture of homebrew, and can't be hardcoded.
There was a problem hiding this comment.
I've removed it and managed to sort out my dev environment's quirks. :)
docs/getting_started.md
Outdated
| Sunshine supports native system audio capture on macOS 14.0 (Sonoma) and newer via Apple’s Audio Tap API. | ||
| To use it, simply leave the **Audio Sink** setting blank. | ||
|
|
||
| If you are running macOS 13 (Ventura) or earlier—or if you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. |
There was a problem hiding this comment.
| If you are running macOS 13 (Ventura) or earlier—or if you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. | |
| If you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. |
macos-13 isn't supported anymore
packaging/sunshine.rb
Outdated
| Sunshine now supports system audio capture natively on macOS 14.0 (Sonoma) and later, | ||
| using the built-in Core Audio Tap API. | ||
|
|
||
| On macOS 13 or earlier, or if you prefer a virtual loopback device, | ||
| you can still use "Soundflower" or "BlackHole" for system audio capture. | ||
|
|
||
| Gamepads are not currently supported on macOS. |
There was a problem hiding this comment.
| Sunshine now supports system audio capture natively on macOS 14.0 (Sonoma) and later, | |
| using the built-in Core Audio Tap API. | |
| On macOS 13 or earlier, or if you prefer a virtual loopback device, | |
| you can still use "Soundflower" or "BlackHole" for system audio capture. | |
| Gamepads are not currently supported on macOS. | |
| Gamepads are not currently supported on macOS. |
Since we don't need to worry about macos-13 now, and audio will work out of the box... we can remove the comment about audio altogether.
There was a problem hiding this comment.
Done, I've removed that bit of info.
src_assets/macos/assets/Info.plist
Outdated
| @@ -8,5 +8,8 @@ | |||
| <string>Sunshine</string> | |||
| <key>NSMicrophoneUsageDescription</key> | |||
| <string>This app requires access to your microphone to stream audio.</string> | |||
There was a problem hiding this comment.
Maybe this comment should be updated?
There was a problem hiding this comment.
I've updated it with something similar to what we use in en.json.
.gitignore
Outdated
| venv/ | ||
|
|
||
| # Caches | ||
| .cache/ |
There was a problem hiding this comment.
What, specifically, is generating this directory?
There was a problem hiding this comment.
This was due to one of the VS Code extensions I have installed, mainly clangd. I removed this entry and now keep it in a global .gitignore file in my home directory!
| @@ -0,0 +1,388 @@ | |||
| /** | |||
| * @file tests/unit/platform/test_macos_av_audio.mm | |||
There was a problem hiding this comment.
| * @file tests/unit/platform/test_macos_av_audio.mm | |
| * @file tests/unit/platform/macos/test_av_audio.mm |
Can you rename the file to be in a macos specific folder. We also have a folder for windows there now as well.
| TEST_F(AVAudioTest, ObjectLifecycle) { | ||
| AVAudio *avAudio = [[AVAudio alloc] init]; | ||
| EXPECT_NE(avAudio, nil); // Should create successfully | ||
| [avAudio release]; // Should not crash |
There was a problem hiding this comment.
If this can crash, I think it should be wrapped in a try/catch. If we hit the catch, then it should trigger GTEST to fail.
Same for anywhere else it could crash. This is because when a test crashes it crashes the whole test run.
There was a problem hiding this comment.
I've added some try/catch blocks and combined it with gtest's FAIL() but I have a feeling we can fine-tune the messages a bit more.
| "audio_sink": "Audio Sink", | ||
| "audio_sink_desc_linux": "The name of the audio sink used for Audio Loopback. If you do not specify this variable, pulseaudio will select the default monitor device. You can find the name of the audio sink using either command:", | ||
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Sunshine can only access microphones on macOS due to system limitations. To stream system audio using Soundflower or BlackHole.", | ||
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture (requires macOS 14.0 or later, using the Core Audio Tap API). Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", |
There was a problem hiding this comment.
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture (requires macOS 14.0 or later, using the Core Audio Tap API). Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", | |
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture. Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", |
ReenigneArcher
added
roadmap
Suggested improvements to Audio Tap code: - Refactor microphone.mm sample() to memcpy the buffer directly instead of copying twice via std::vector, and optimize partial read case. - Move initializeAudioBuffer before anything can use the buffer. - Change size of ring buffer to 30ms (6 * 240 * channels * sizeof(float)). - Avoid confusion in converter setup between float and int sample rate. - Add helper for logging CoreAudio error codes in FourCC text mode. Remove redundant logging, and always log the exact API call that is failing. - Add optional way to create the aggregate in public mode so it's visible in Audio MIDI Setup/HALLab. Set SUNSHINE_PUBLIC_AUDIO_TAP=1. - Use non-deprecated AVCaptureDeviceType constants. - Use serial dispatch queue for audio sampling. - Update InitializeAudioBuffer test with buffer size verification. - Retain [aggregateProperties release] after AudioHardwareCreateAggregateDevice to avoid a memory leak under MRC (mutableCopy returns a +1 retained object).
…rdByte#4209 - Alphabetize FIND_LIBRARY and SUNSHINE_EXTERNAL_LIBRARIES lists in CMake. - Remove unnecessary include_directories(SYSTEM ${OPENSSL_INCLUDE_DIR}). - Remove hardcoded /opt/homebrew/opt/opus/ paths from cmake/dependencies/macos.cmake. - Remove outdated macOS 13 references from docs and packaging caveats. - Simplify audio_sink_desc_macos locale string by removing version/API details. - Update NSMicrophoneUsageDescription to reflect external device usage. - Remove .cache/ from .gitignore. (originated from the clangd extension in VS Code) - Move test file to tests/unit/platform/macos/test_av_audio.mm. - Wrap Objective-C tests in @try/@catch to prevent crashes from aborting he entire test run.
…ere/feat-macos-ca-taps # Conflicts: # src/platform/macos/microphone.mm
There was a problem hiding this comment.
Pull request overview
Adds native system-wide audio capture on macOS (14.0+) via Apple’s Audio Tap API, integrating it into the existing cross-platform audio capture interface and exposing it through configuration/UI strings and documentation.
Changes:
- Introduces a new Objective-C++
AVAudioimplementation supporting both microphone capture and Core Audio system taps, including format conversion + buffering. - Extends the platform
audio_control_t::microphoneAPI with ahost_audio_enabledflag and wires it through the audio pipeline. - Adds macOS-specific unit tests and updates build/config/docs/UI strings to reflect the new capture option.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/platform/macos/test_av_audio.mm | New macOS unit tests for AVAudio (microphone + system tap paths). |
| tests/CMakeLists.txt | Includes .mm test sources when building on macOS. |
| src_assets/macos/assets/Info.plist | Updates macOS permission usage strings for microphone/system audio/screen capture. |
| src_assets/common/assets/web/public/assets/locale/en.json | Updates macOS audio sink description to mention built-in system audio capture. |
| src/platform/windows/audio.cpp | Updates microphone(...) override signature to include host_audio_enabled. |
| src/platform/macos/microphone.mm | Uses system tap when sink is blank; propagates host_audio_enabled. |
| src/platform/macos/coreaudio_helpers.h | Adds helper to stringify Core Audio OSStatus values (FourCC-friendly). |
| src/platform/macos/av_audio.mm | New Objective-C++ implementation of dual-path macOS audio capture + system tap IOProc. |
| src/platform/macos/av_audio.m | Removes old Objective-C implementation (replaced by .mm). |
| src/platform/macos/av_audio.h | Expands API/docs and adds system tap support declarations + shared buffer/semaphore. |
| src/platform/linux/audio.cpp | Updates microphone(...) override signature to include host_audio_enabled. |
| src/platform/common.h | Extends audio_control_t::microphone virtual interface with host_audio_enabled. |
| src/config.h | Adds inline documentation to config::audio_t fields. |
| src/audio.cpp | Passes HOST_AUDIO flag through to control->microphone(...). |
| packaging/sunshine.rb | Removes outdated macOS limitation note about system audio capture. |
| docs/getting_started.md | Updates macOS docs to describe native system audio capture on macOS 14+. |
| cmake/dependencies/macos.cmake | Adds CoreAudio/AudioToolbox/AudioUnit libraries for macOS builds. |
| cmake/compile_definitions/unix.cmake | Adjusts asset-dir prefixing logic for absolute paths/dev builds. |
| cmake/compile_definitions/macos.cmake | Links new macOS audio libs and switches av_audio source to .mm. |
| AudioBufferList audioBufferList; | ||
| CMBlockBufferRef blockBuffer; | ||
|
|
||
| CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer); | ||
|
|
||
| // NSAssert(audioBufferList.mNumberBuffers == 1, @"Expected interleaved PCM format but buffer contained %u streams", audioBufferList.mNumberBuffers); | ||
|
|
||
| // this is safe, because an interleaved PCM stream has exactly one buffer, | ||
| // and we don't want to do sanity checks in a performance critical exec path | ||
| AudioBuffer audioBuffer = audioBufferList.mBuffers[0]; | ||
|
|
||
| TPCircularBufferProduceBytes(&self->audioSampleBuffer, audioBuffer.mData, audioBuffer.mDataByteSize); | ||
| dispatch_semaphore_signal(self->audioSemaphore); | ||
| } |
There was a problem hiding this comment.
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer returns a retained blockBuffer, but it is never released. This will leak memory in the audio callback path; ensure blockBuffer is released (e.g., CFRelease) after producing bytes into the circular buffer.
src/platform/macos/av_audio.mm
Outdated
| UInt32 aggregateDeviceSampleRate = 48000; // Default fallback | ||
| UInt32 aggregateDeviceChannels = 2; // Default fallback | ||
|
|
||
| // Get actual sample rate from the aggregate device | ||
| // XXX Do we need this, won't it always be 48000? We set it above in createAggregateDeviceWithTapDescription. | ||
| UInt32 sampleRateQuerySize = sizeof(Float64); | ||
| Float64 var = 0.0; | ||
| OSStatus sampleRateStatus = [self getDeviceProperty:self->aggregateDeviceID | ||
| selector:kAudioDevicePropertyNominalSampleRate | ||
| scope:kAudioObjectPropertyScopeGlobal | ||
| element:kAudioObjectPropertyElementMain | ||
| size:&sampleRateQuerySize | ||
| data:&var]; | ||
|
|
||
| if (sampleRateStatus != noErr) { | ||
| BOOST_LOG(error) << "Failed to get device sample rate, using default 48kHz: " << ca::Status(sampleRateStatus); | ||
| } | ||
|
|
||
| // Get actual channel count from the device's input stream configuration | ||
| AudioObjectPropertyAddress streamConfigAddr = { | ||
| .mSelector = kAudioDevicePropertyStreamConfiguration, | ||
| .mScope = kAudioDevicePropertyScopeInput, | ||
| .mElement = kAudioObjectPropertyElementMain | ||
| }; | ||
|
|
||
| UInt32 streamConfigSize = 0; | ||
| OSStatus streamConfigSizeStatus = AudioObjectGetPropertyDataSize(self->aggregateDeviceID, &streamConfigAddr, 0, NULL, &streamConfigSize); | ||
|
|
||
| if (streamConfigSizeStatus == noErr && streamConfigSize > 0) { | ||
| AudioBufferList *streamConfig = (AudioBufferList *) malloc(streamConfigSize); | ||
| if (streamConfig) { | ||
| OSStatus streamConfigStatus = AudioObjectGetPropertyData(self->aggregateDeviceID, &streamConfigAddr, 0, NULL, &streamConfigSize, streamConfig); | ||
| if (streamConfigStatus == noErr && streamConfig->mNumberBuffers > 0) { | ||
| aggregateDeviceChannels = streamConfig->mBuffers[0].mNumberChannels; | ||
| BOOST_LOG(debug) << "Device reports "sv << aggregateDeviceChannels << " input channels"sv; | ||
| } else { | ||
| BOOST_LOG(warning) << "Failed to get stream configuration, using default 2 channels: "sv << streamConfigStatus; | ||
| } | ||
| free(streamConfig); | ||
| } | ||
| } else { | ||
| BOOST_LOG(warning) << "Failed to get stream configuration size, using default 2 channels: "sv << streamConfigSizeStatus; | ||
| } | ||
|
|
||
| BOOST_LOG(debug) << "Device properties - Sample Rate: "sv << aggregateDeviceSampleRate << "Hz, Channels: "sv << aggregateDeviceChannels; | ||
|
|
There was a problem hiding this comment.
The queried nominal sample rate is stored in var, but aggregateDeviceSampleRate is never updated from it, so the device sample rate is effectively always treated as 48kHz. Update aggregateDeviceSampleRate from the query result when it succeeds to avoid incorrect needsConversion decisions and misleading logs.
There was a problem hiding this comment.
Made some changes to this code as part of commit 3831795.
src/platform/macos/av_audio.mm
Outdated
| NSMutableArray *result = [[NSMutableArray alloc] init]; | ||
|
|
||
| for (AVCaptureDevice *device in [AVAudio microphones]) { | ||
| [result addObject:[device localizedName]]; | ||
| } | ||
|
|
||
| BOOST_LOG(info) << "Found "sv << [result count] << " microphones"sv; | ||
| return result; | ||
| } |
There was a problem hiding this comment.
microphoneNames allocates result with init and returns it without autoreleasing/releasing, which leaks under manual retain/release. Return an autoreleased object (e.g., use a convenience constructor) or autorelease result before returning.
There was a problem hiding this comment.
Swapped the result to NSMutableArray *result = [NSMutableArray array].
| BOOST_LOG(debug) << "Successfully added audio input to capture session"sv; | ||
| } else { | ||
| BOOST_LOG(error) << "Cannot add audio input to capture session"sv; | ||
| [audioInput dealloc]; |
There was a problem hiding this comment.
Calling [audioInput dealloc] directly is unsafe/undefined (objects should not receive dealloc outside their own deallocation path). Replace this with correct ownership handling (typically release if you own it, or simply return after letting the autorelease pool handle it).
| [audioInput dealloc]; | |
| [audioInput release]; |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ictionary literal
…reads fail. Handle a Core Audio edge case where getDeviceProperty can leave aggregateDeviceSampleRate invalid (observed as 0) and avoid using hardcoded fallback formats. Default aggregate sample rate/channels to clientSampleRate/clientChannels, validate non-positive sample-rate reads, and log explicit fallback behavior when property queries fail.
|
Description
This PR adds system-wide audio tap support for macOS. The implementation introduces:
Additional changes
cmakefiles for compatibility with Homebrew-based setup.opensslandopusare found automatically. This replaces the need to run manuallncommands, but I’m not sure if this is the best long-term approach. Feedback welcome.cmake/compile_definitions/unix.cmaketo ensureSUNSHINE_ASSETS_DIRresolves correctly.src_assets/macos/assets/Info.plistto prepare for required macOS permission prompts.macos_system_wide_audio_tapconfig option to theaudio_tstruct.Testing
Notes
setupMicrophonefunctionality as it is also a viable option!macos_system_wide_audio_tapsetting.If the AI involvement is a blocker for merging, no worries. I can totally understand! This was also a learning project for me and I had a lot of fun building it.
Screenshot
Web UI – Audio/Video Configuration
New option for enabling system-wide audio recording on macOS. Disables the audio sink option when checked.
macOS Permission Prompt
System permission request when Sunshine first tries to access system audio:
System Settings – Screen & System Audio Recording
macOS privacy settings showing Sunshine/Terminal access to "System Audio Recording Only":
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage