feat(linux): Add Vulkan video encoder support to FFmpeg build

[neatnoise]

Description

Add Vulkan video encoding support to FFmpeg builds for use with Sunshine's new Vulkan encoder. Requires vulkan-headers and vulkan-icd-loader packages to be installed (or similar on another distros).

Changes:

• Add BUILD_FFMPEG_VULKAN CMake option (default: ON)
• Add --enable-vulkan flag to FFmpeg configure
• Enable Vulkan encoders: h264_vulkan, hevc_vulkan, av1_vulkan

This is a dependency for the Sunshine Vulkan encoder PR which adds zero-copy DMA-BUF/EGL encoding on Linux KMS.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[neatnoise]

@psyke83 @ReenigneArcher added dependencies, updated the docs

[ReenigneArcher]

@neatnoise thanks for updating it! There must still be something missing on Linux as the build is failing to find vulkan. It does find it on FreeBSD though.

Edit: actually it worked on FreeBSD amd64, but failed for aarch64.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ReenigneArcher]

the vulkan package in 22.04 is too old according to https://ubuntuhandbook.org/index.php/2025/09/compile-ffmpeg-8-0-in-ubuntu-with-nvidia-gpu-acceleration/

[ReenigneArcher]

Fixed in #606

Please test/review if vulkan is properly compiled and would work in the Sunshine PR. You can get artifacts from https://github.com/LizardByte/build-deps/actions/runs/21850415771?pr=606