Chunking Algorithms #336
Draft
Chunking Algorithms #336
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sluongng
force-pushed
the
sluongng/chunking-algo
branch
from
456e902 to
b25c8e4
Compare
mostynb
reviewed
Jul 12, 2025
sluongng
force-pushed
the
sluongng/chunking-algo
branch
from
b25c8e4 to
27b0d6c
Compare
sluongng
force-pushed
the
sluongng/chunking-algo
branch
from
27b0d6c to
39bbe03
Compare
Introduce ChunkingFunction which enum is a set of known chunking algorithms that the server can recommend to the client. Provide FastCDC_2020 as the first explicit chunking algorithm. The server advertise these through a new chunking_configuration field in CacheCapabilities message. There, the server may set the chunking functions that it supports as well as the relevant configuration parameters for that chunking algorithm.
sluongng
force-pushed
the
sluongng/chunking-algo
branch
from
39bbe03 to
19f1152
Compare
tjgq
requested changes
Jul 17, 2025
Comment on lines
+1980
to
+1984
| // The chunking function that the client prefers to use. | ||
| // | ||
| // The server MAY use a different chunking function. The client MUST check | ||
| // the chunking function used in the response. | ||
| ChunkingFunction.Value chunking_function = 4; |
There was a problem hiding this comment.
- Is this field intended to be mandatory or optional (with the latter giving the server complete leeway in choosing a function)? If optional, can we document it?
- When the field is present, should it be required to match one of the functions declared in the server capabilities?
There was a problem hiding this comment.
Ah, it defaults to UNKNOWN if unset, because this is proto3. (But perhaps it's still worth spelling it out - up to you.)
Comment on lines
+2036
to
+2037
| // The chunking function that the client used to split the blob. | ||
| ChunkingFunction.Value chunking_function = 5; |
There was a problem hiding this comment.
Why is this necessary? Isn't the result of a splice completely independent from the function originally used to do the splitting?
It also imposes a requirement that the chunks must have necessarily originated from a split operation, which seems counter to the spirit of the original proposal (that split and splice are independent optimizations).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on #282
Introduce ChunkingFunction which enum is a set of known chunking
algorithms that the server can recommend to the client.
Provide FastCDC_2020 as the first explicit chunking algorithm.
The server advertises these through a new chunking_configuration field in
CacheCapabilities message. There, the server may set the chunking
functions that it supports as well as the relevant configuration
parameters for that chunking algorithm.
I recommend reading https://joshleeb.com/posts/fastcdc.html to understand more about the available FastCDC configuration parameters.