[GLUTEN-11605][VL] Write per-block column statistics in shuffle writer#11769
Open
acvictor wants to merge 2 commits intoapache:mainfrom
Open
[GLUTEN-11605][VL] Write per-block column statistics in shuffle writer#11769acvictor wants to merge 2 commits intoapache:mainfrom
acvictor wants to merge 2 commits intoapache:mainfrom
Conversation
acvictor
force-pushed
the
acvictor/writerChanges
branch
6 times, most recently
from
cb073fd to
19b8d5a
Compare
acvictor
force-pushed
the
acvictor/writerChanges
branch
from
19b8d5a to
50e0444
Compare
Contributor
Author
|
@marin-ma @zhztheplayer this is ready for review. I will push one more commit to again disable by default. |
marin-ma
reviewed
Mar 19, 2026
| } | ||
| // Check each bit — return early on first null found. | ||
| for (uint32_t i = 0; i < numRows; ++i) { | ||
| if (!arrow::bit_util::GetBit(validityBuffer->data(), i)) { |
Contributor
There was a problem hiding this comment.
Perhaps check each bytes by comparing with 0xff can be faster
| // Reuse the dynamic filter config to also enable block statistics collection, | ||
| // since stats are only useful when dynamic filter pushdown is active. | ||
| const auto& confMap = ctx->getConfMap(); | ||
| auto it = confMap.find("spark.gluten.sql.columnar.backend.velox.valueStream.dynamicFilter.enabled"); |
Contributor
There was a problem hiding this comment.
Other option values are passing through function args. Can you add a new arg enableBlockStatistics?
| mergedStats.merge(*append->blockStats_); | ||
| result->setBlockStats(std::move(mergedStats)); | ||
| } else if (source->hasBlockStats()) { | ||
| result->setBlockStats(*source->blockStats_); |
Contributor
There was a problem hiding this comment.
If only source or append has blockStats, should we either discard it or compute for the missing side and merge them?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
This PR adds per-block column statistics (min/max/hasNull) to the shuffle writer pipeline as a prerequisite for block-level pruning using dynamic filters at the shuffle reader. When
spark.gluten.sql.columnar.backend.velox.valueStream.dynamicFilter.enabledis true, the shuffle writer computes per-column min/max statistics from raw Arrow buffers duringevictBuffers()and serializes them as akStatisticsPayloadblock before each non-dictionary payload in the output file. This mirrors how parquet row group statistics enable predicate pushdown.How was this patch tested?
Added new tests and also ran the CI with config set to true.
Was this patch authored or co-authored using generative AI tooling?
No
Related issue: #11605