Skip to content

branch-3.1: [Improve](segment) Put Segment footer into index page cache #56382 #56459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 28, 2025
Merged

branch-3.1: [Improve](segment) Put Segment footer into index page cache #56382 #56459

merged 1 commit into from
Sep 28, 2025

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #56382

@eldenmoon
[Improve](segment) Put Segment footer into index page cache (#56382)
6e560e1 
This commit modified the caching logic for the Segment footer.

Previously, the Segment footer was cached in the `DATA_PAGE` cache. This
was incorrect as the footer is metadata, not data. This commit changes
it to use the more appropriate `INDEX_PAGE` cache.

Additionally, the calculation for `_meta_mem_usage` has been adjusted
for better accuracy. The footer size is no longer included in the
calculation since it's now managed by the `StoragePageCache`. The memory
estimation for column readers is also updated to use
`config::max_segment_partial_column_cache_size` for a more realistic
value.
@github-actions github-actions bot requested a review from morrySnow as a code owner September 25, 2025 08:00
@Thearas
Copy link
Contributor

Thearas commented Sep 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 25, 2025
@Thearas
Copy link
Contributor

Thearas commented Sep 25, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32923 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e560e197bae5067740cbfeb19dd371c2581291b, data reload: false

------ Round 1 ----------------------------------
q1	17593	5511	5429	5429
q2	2026	392	281	281
q3	11812	1220	734	734
q4	10266	879	462	462
q5	8627	2412	2161	2161
q6	182	163	133	133
q7	901	758	610	610
q8	9326	1417	1208	1208
q9	5143	4902	4970	4902
q10	6747	2304	1819	1819
q11	465	284	273	273
q12	334	350	208	208
q13	17789	3617	3019	3019
q14	234	229	215	215
q15	530	474	463	463
q16	420	438	374	374
q17	600	849	375	375
q18	6780	6451	6393	6393
q19	1654	979	567	567
q20	345	343	209	209
q21	3078	2248	2095	2095
q22	1051	1013	993	993
Total cold run time: 105903 ms
Total hot run time: 32923 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5481	5462	5403	5403
q2	234	334	240	240
q3	2248	2661	2366	2366
q4	1394	1809	1408	1408
q5	4374	4945	4979	4945
q6	174	161	126	126
q7	2087	1965	1822	1822
q8	2610	2845	2717	2717
q9	7296	7238	7223	7223
q10	3061	3318	2705	2705
q11	575	495	494	494
q12	645	755	629	629
q13	3423	3785	3149	3149
q14	274	298	275	275
q15	514	473	456	456
q16	454	510	436	436
q17	1227	1750	1272	1272
q18	7635	7499	7380	7380
q19	803	1135	1075	1075
q20	2022	2054	1917	1917
q21	5326	5023	4516	4516
q22	1093	1082	971	971
Total cold run time: 52950 ms
Total hot run time: 51525 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193548 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e560e197bae5067740cbfeb19dd371c2581291b, data reload: false

query1	957	410	404	404
query2	6251	1936	1826	1826
query3	8681	198	194	194
query4	33407	23631	23910	23631
query5	4349	646	473	473
query6	298	205	192	192
query7	4209	500	341	341
query8	314	257	246	246
query9	9300	2623	2601	2601
query10	491	331	260	260
query11	18450	15959	15243	15243
query12	163	116	105	105
query13	1621	558	421	421
query14	10244	7843	6831	6831
query15	248	198	192	192
query16	7992	689	543	543
query17	1625	763	610	610
query18	2205	414	326	326
query19	233	180	165	165
query20	128	121	125	121
query21	207	128	105	105
query22	4448	4681	4361	4361
query23	35246	34712	34240	34240
query24	7158	2736	2786	2736
query25	498	478	413	413
query26	734	287	174	174
query27	1931	529	359	359
query28	5224	2240	2196	2196
query29	679	588	469	469
query30	245	190	159	159
query31	972	937	861	861
query32	96	60	62	60
query33	506	368	315	315
query34	764	888	534	534
query35	823	819	756	756
query36	1042	1096	994	994
query37	117	102	82	82
query38	4101	4119	4002	4002
query39	1545	1468	1501	1468
query40	199	125	110	110
query41	50	50	48	48
query42	123	116	116	116
query43	501	526	499	499
query44	1329	831	838	831
query45	194	183	172	172
query46	921	1088	700	700
query47	1972	2005	1893	1893
query48	426	438	350	350
query49	724	527	400	400
query50	705	699	439	439
query51	7443	7383	7343	7343
query52	110	103	95	95
query53	239	263	200	200
query54	565	573	497	497
query55	80	91	84	84
query56	272	290	258	258
query57	1276	1246	1249	1246
query58	249	222	217	217
query59	3009	3199	3000	3000
query60	308	318	287	287
query61	154	150	155	150
query62	795	750	751	750
query63	245	204	208	204
query64	3586	1114	767	767
query65	3386	3352	3336	3336
query66	801	430	333	333
query67	16261	15786	15657	15657
query68	7050	850	553	553
query69	497	321	298	298
query70	1214	1132	1121	1121
query71	372	303	280	280
query72	5701	3760	3843	3760
query73	651	767	360	360
query74	10316	9163	9291	9163
query75	3212	3189	2726	2726
query76	2989	1199	781	781
query77	498	381	292	292
query78	10468	10480	9572	9572
query79	3842	852	596	596
query80	747	535	457	457
query81	500	260	228	228
query82	581	116	90	90
query83	160	163	142	142
query84	236	97	79	79
query85	778	358	293	293
query86	398	320	309	309
query87	4297	4352	4267	4267
query88	5335	2433	2427	2427
query89	410	341	298	298
query90	1824	192	192	192
query91	133	141	109	109
query92	65	56	52	52
query93	2724	939	561	561
query94	705	425	308	308
query95	365	284	275	275
query96	501	619	288	288
query97	3190	3237	3176	3176
query98	224	214	196	196
query99	1377	1424	1308	1308
Total cold run time: 294785 ms
Total hot run time: 193548 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e560e197bae5067740cbfeb19dd371c2581291b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.50	0.51	0.52
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.51	0.49
query10	0.56	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.61	0.60
query14	0.78	0.79	0.81
query15	0.85	0.84	0.82
query16	0.41	0.41	0.37
query17	1.08	1.06	1.07
query18	0.25	0.22	0.24
query19	1.98	1.90	1.79
query20	0.02	0.01	0.02
query21	15.36	0.92	0.58
query22	0.75	0.86	0.53
query23	15.19	1.39	0.50
query24	3.43	0.48	1.47
query25	0.28	0.06	0.14
query26	0.24	0.15	0.14
query27	0.06	0.05	0.04
query28	13.61	1.00	0.44
query29	12.56	3.90	3.21
query30	0.25	0.09	0.07
query31	2.81	0.58	0.38
query32	3.23	0.54	0.47
query33	3.01	3.04	3.04
query34	16.46	5.20	4.55
query35	4.59	4.49	4.58
query36	0.63	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.01 s
Total hot run time: 28.22 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.57% (12784/28053)
Line Coverage 36.41% (114045/313248)
Region Coverage 34.02% (65203/191640)
Branch Coverage 31.05% (34216/110202)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.57% (21127/27590)
Line Coverage 69.90% (218271/312282)
Region Coverage 67.88% (130615/192417)
Branch Coverage 61.43% (68014/110716)

@morrySnow morrySnow merged commit f97020e into branch-3.1 Sep 28, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-56382-branch-3.1 branch September 28, 2025 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @eldenmoon