Benchmarking against other quantized kernels

[ncylich]

Summary

• Adds a unified matmul benchmark suite (tests/bench/) that compares Cactus INT8/INT4 kernels against 6 other inference frameworks (GGML, MLX, MLC-LLM, LiteRT, ONNX Runtime, ExecuTorch) on identical 1024x1024 workloads
• Moves third-party dependencies from third_party/ to ../third_party/ (outside the repo root) to keep the repository clean
• Adds missing PARAKEET model type, num_mel_bins, encoder_hidden_act, pad_token_id, and hann_periodic fields to engine config to fix build on this branch

Benchmark architecture

Each backend implements a single run_kernel(M, weights, activations, act_int8, act_scales, output, reference) function pointer. When output/reference are null (the hot path), the kernel runs with zero capture overhead. When non-null (called once for accuracy), the kernel writes fp32 output and an optional dequantized reference for NRMSE checking.

The timed loop cycles through 64 distinct weight matrices per iteration to force L2 cache misses on every call, matching real inference where each transformer layer has unique weights. On the M4 Pro (16 MB L2), 64 matrices x ~1 MB each = 64 MB, exceeding both L2 and SLC.

Two matmul sizes are tested: 1x1024x1024 (GEMV, single-token decode) and 1024x1024x1024 (GEMM, batched/prefill).

Backends (11 total)

Framework	Backends	Build flag
Cactus	cactus_int8, cactus_int4	always on
GGML	ggml_q4_0, ggml_q8_0, ggml_q4_0_graph, ggml_q8_0_graph	-DWITH_GGML=ON
MLX	mlx_q4_cpu, mlx_q8_cpu, mlx_q4_gpu, mlx_q8_gpu	-DWITH_MLX=ON
MLC-LLM	mlc_int4, mlc_int8	-DWITH_MLC=ON
LiteRT	litert_neon, ruy, litert_4bit_neon	-DWITH_LITERT=ON
ONNX Runtime	onnxrt_int8, onnxrt_int4	-DWITH_ONNXRT=ON
ExecuTorch	executorch_int8, executorch_int4	-DWITH_EXECUTORCH=ON

Key results (M4 Pro, 14 cores)

GEMV (M=1): Cactus INT8 at 32.5 us is 2nd only to LiteRT NEON (23.6 us), beating GGML (57 us), MLX (140 us), and ONNX Runtime (587 us). Cactus INT4 at 33.6 us is 2nd to LiteRT 4bit (10.8 us).

GEMM (M=1024): Cactus INT8 at 828 us (2593 GOPS) is 2nd to MLX (606 us, using Accelerate/AMX), beating Ruy (1348 us), ONNX Runtime (1616 us), and GGML (3269 us). Cactus INT4 at 1192 us is again 2nd to MLX (595 us).

Changes

• tests/bench/ — new benchmark suite (matmul_bench.cpp, bench_driver, bench_common, 7 backend files, README with full results)
• tests/CMakeLists.txt — build rules for benchmark executable and optional framework backends, updated third_party paths to ../../third_party/
• cactus/engine/engine.h — added PARAKEET enum value, num_mel_bins, encoder_hidden_act, pad_token_id, hann_periodic fields (fixes build)

[Copilot]

Pull request overview

This PR adds comprehensive benchmarking infrastructure to compare Cactus's quantized matrix multiplication kernels (INT4/INT8) against other popular inference frameworks including GGML, LiteRT (TensorFlow Lite), MLX, MLC-LLM, ONNX Runtime, and ExecuTorch/XNNPACK. The benchmark suite measures performance and accuracy on standard matrix sizes (1x1024x1024 for GEMV and 1024x1024x1024 for GEMM), enabling objective comparisons of CPU-based quantized inference performance.

Changes:

• Added modular benchmark driver infrastructure with backend registration system
• Implemented 7+ backend adapters for external frameworks with optional CMake integration
• Added shell script for automated framework detection and build configuration

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
tests/run_benchmark.sh	Shell script for building and running benchmarks with automatic framework detection
tests/bench/matmul_bench.cpp	Main benchmark entry point that orchestrates test execution
tests/bench/bench_driver.{h,cpp}	Core benchmark driver with backend registry and execution logic
tests/bench/bench_common.{h,cpp}	Shared quantization utilities and accuracy checking functions
tests/bench/backend_*.cpp	Backend implementations for Cactus, GGML, LiteRT, MLX, MLC, ONNX Runtime, and ExecuTorch
tests/bench/README.md	Comprehensive documentation with setup instructions and benchmark results
tests/CMakeLists.txt	CMake configuration with conditional framework integration
cactus/engine/engine.h	Model configuration updates (unrelated to benchmarking)

Comments suppressed due to low confidence (1)

tests/run_benchmark.sh:83

• The executable path references test_matmul_bench but the CMakeLists.txt defines the executable as matmul_bench (line 241). This will cause the script to fail when attempting to run the benchmark. The path should be "$BUILD_DIR/matmul_bench".

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The documentation mentions backend names ruy_mc and ruy_1c that are not registered in backend_litert.cpp. The actual registered backends are litert_neon, ruy, and litert_4bit_neon. The documentation should be updated to reflect the actual backend names.

Suggested change

	Fetches FlatBuffers + TFLite deps on first build (requires network). Enables `litert_neon`, `ruy_mc`, `ruy_1c`, and `litert_4bit_neon` backends.
	Fetches FlatBuffers + TFLite deps on first build (requires network). Enables `litert_neon`, `ruy`, and `litert_4bit_neon` backends.

[Copilot]

The Quick Start instructions reference a "cactus build" command which is not a standard command in the repository. The run_benchmark.sh script sources "./setup" instead (line 24). The documentation should be consistent with the actual build process, or explain what "cactus build" refers to.

Suggested change

	cactus build
	./setup

[Copilot]

The addition of preemphasis and hann_periodic fields to SpectrogramConfig appears unrelated to the benchmarking PR. These audio processing configuration changes should be in a separate PR.

[Copilot]

Missing validation: The function assumes K is evenly divisible by kGroupSize (32). If K is not a multiple of kGroupSize, the function will access out-of-bounds memory or produce incorrect results. The code should either validate this precondition or handle partial groups.

[Copilot]

Missing validation: The function assumes interleaved.size() is evenly divisible by 32. If the size is not a multiple of 32, the loop will access out-of-bounds memory when i + 16 + j exceeds the vector size. The code should either validate this precondition or handle the remainder properly.

[Copilot]

Missing directory validation: The LITERT_DIR path is constructed without checking if it exists. If the directory doesn't exist, the build will fail with cryptic errors. Consider adding a check like: if(NOT EXISTS "$${LITERT_DIR}") message(FATAL_ERROR "LiteRT directory not found at $${LITERT_DIR}") endif().

Suggested change

	set(LITERT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../third_party/litert)
	set(LITERT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../third_party/litert)
	if(NOT EXISTS "${LITERT_DIR}")
	message(FATAL_ERROR "LiteRT directory not found at ${LITERT_DIR}")
	endif()

[Copilot]

Missing error handling: The posix_memalign function can fail (return non-zero), but the code doesn't check the return value. If allocation fails, raw will remain nullptr, and w->prepacked will be set to nullptr. This will likely cause a crash when passed to the Prepack function on line 176. The code should check the return value and handle allocation failures appropriately.

[Copilot]

Missing error handling: The posix_memalign function can fail (return non-zero), but the code doesn't check the return value. If allocation fails, raw will remain nullptr, and w->ref_prepacked will be set to nullptr. This will likely cause a crash when passed to ReferencePrepack on line 185. The code should check the return value and handle allocation failures appropriately.

[Copilot]

The addition of num_mel_bins to the Config struct appears unrelated to the benchmarking infrastructure described in the PR title and description. This change should be in a separate PR focused on model configuration updates.

[Copilot]

The documentation mentions backend names ggml_q4_0_graph and ggml_q8_0_graph that are commented out in the actual code (backend_ggml.cpp lines 292-299). These backends are not actually registered or available, which contradicts the documentation. Either remove these from the documentation or uncomment the registration code.

Suggested change

	Builds GGML from source. Enables `ggml_q4_0`, `ggml_q8_0`, `ggml_q4_0_graph`, and `ggml_q8_0_graph` backends.
	Builds GGML from source. Enables `ggml_q4_0` and `ggml_q8_0` backends.