Base64: performance tuning #9215
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Remove unused ReadSeek trait and rely solely on Read for inputs - Stream stdin directly instead of buffering into memory - Simplify fast decode/encode capability detection to be format-only - Always use padded base64 alphabet for consistent SIMD-based handling
- Replace explicit lifetime parameter with anonymous lifetime to reduce noise - Keeps semantics unchanged while aligning with modern Rust idioms
- Introduce READ_BUFFER_MAX to limit read buffer size during encoding - Clamp buffer size to avoid excessively large allocations while preserving behavior
- Add missing semicolons on USimpleError returns for consistent style - Reformat encode_to_vec_deque call to improve readability without logic changes
- Add "hellohello" to ignored words to prevent false positives - Keeps spell-checker from flagging known test data in base_common.rs
Contributor
|
i wonder how we could check that in the CI to make sure we don't regress |
|
GNU testsuite comparison: |
Contributor
|
sorry, small conflict |
- Clarify trait alias comment for ReadSeek in English - Refine padding detection doc to describe scanning including trailing whitespace
CodSpeed Performance ReportMerging #9215 will degrade performances by 5.77%Comparing Summary
Benchmarks breakdown
Footnotes
|
|
GNU testsuite comparison: |
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Reworked base_common streaming so base64/basenc no longer slurp stdin into RAM; both encoder/decoder now read from any Read stream with bounded buffers, keep line-wrapping support, and reuse the SIMD wrapper.
Tightened the Base64SimdWrapper integration and added regression tests that cover streaming encode/decode behaviors plus chunked readers.
Fixed Clippy’s elidable lifetime warning and ensured Base58 handling no longer triggers gigantic buffer allocations.
Testing
cargo test -p uu_base32
cargo test -p uu_base64
cargo test -p uu_basenc
/usr/bin/time -l sh -c "cat benchmarks/bench_500mb.bin | /usr/bin/base64 > /dev/null" → max RSS 1.85 MB
/usr/bin/time -l sh -c "cat benchmarks/bench_500mb.bin | ./target/release/base64 > /dev/null" → max RSS 2.38 MB
/usr/bin/time -l sh -c "cat benchmarks/bench_500mb.b64 | /usr/bin/base64 -D > /dev/null" → max RSS 1.85 MB
/usr/bin/time -l sh -c "cat benchmarks/bench_500mb.b64 | ./target/release/base64 -d > /dev/null" → max RSS 2.33 MB