fix: Optimize memory on rolling groups in ApplyExpr
#24709
Merged
+130
−15
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #22725
This PR introduces a lazy iterator over groups
iter_groups_lazy(), which helps to avoid memory explosion when groups are overlapping (rolling).The initial PR has the minimum code to fix the issue reported above. More work is needed: (i) add
GroupsType::Idximplementation, (ii) provide an indexed parallel iterator for multi-threading , (iii) leverage the iterators in other expressions and/or implementation paths that may suffer from the same memory explosion pattern when aggregating fromNotAggregatedtoAggregatedListon rolling groups.Micro-benchmark results (
/usr/bin/time -von debug build on the MRE as documented in the issue, single-threaded).Memory reduced from 1.6 GB to 132 MB.
For the original issue: from 11.8 GB to 0.36 GB.
Ping @coastalwhite