Skip to content

branch-3.1: [Improve](segment) Put Segment footer into index page cache #56382 #56459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 28, 2025
Merged

branch-3.1: [Improve](segment) Put Segment footer into index page cache #56382 #56459

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions be/src/olap/rowset/segment_v2/segment.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,13 +196,16 @@ Status Segment::_open(OlapReaderStatistics* stats) {
_num_rows = footer_pb_shared->num_rows();

// An estimated memory usage of a segment
_meta_mem_usage += footer_pb_shared->ByteSizeLong();
// Footer is seperated to StoragePageCache so we don't need to add it to _meta_mem_usage
// _meta_mem_usage += footer_pb_shared->ByteSizeLong();
if (_pk_index_meta != nullptr) {
_meta_mem_usage += _pk_index_meta->ByteSizeLong();
}

_meta_mem_usage += sizeof(*this);
_meta_mem_usage += _tablet_schema->num_columns() * config::estimated_mem_per_column_reader;
_meta_mem_usage += std::min(static_cast<int>(_tablet_schema->num_columns()),
config::max_segment_partial_column_cache_size) *
config::estimated_mem_per_column_reader;

// 1024 comes from SegmentWriterOptions
_meta_mem_usage += (_num_rows + 1023) / 1024 * (36 + 4);
Expand Down Expand Up @@ -1017,11 +1020,16 @@ Status Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& footer_pb,

PageCacheHandle cache_handle;

// Put segment footer into index page cache.
// Rationale:
// - Footer is metadata (small, parsed with indexes), not data page payload.
// - Using PageTypePB::INDEX_PAGE keeps it under the same eviction policy/shards
// as other index/metadata pages and avoids competing with DATA_PAGE budget.
if (!segment_footer_cache->lookup(cache_key, &cache_handle,
segment_v2::PageTypePB::DATA_PAGE)) {
segment_v2::PageTypePB::INDEX_PAGE)) {
RETURN_IF_ERROR(_parse_footer(footer_pb_shared, stats));
segment_footer_cache->insert(cache_key, footer_pb_shared, footer_pb_shared->ByteSizeLong(),
&cache_handle, segment_v2::PageTypePB::DATA_PAGE);
&cache_handle, segment_v2::PageTypePB::INDEX_PAGE);
} else {
VLOG_DEBUG << fmt::format("Segment footer of {}:{}:{} is found in cache",
_file_reader->path().native(), _file_reader->size(),
Expand Down
Loading