Skip to content

Conversation

liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Sep 24, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This pull request refactors the Doris FE loading framework to improve support for broker load tasks with multiple file groups. The changes ensure that all file groups share a single destination tuple and descriptor table, and that filters and scan contexts are handled correctly for each file group. Additionally, new regression tests are added to verify multi-filegroup broker load functionality.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01 liaoxin01 marked this pull request as draft September 24, 2025 02:56
@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8857f842347b7d9c2c1b7c16ff0d17f5e2d2f50f, data reload: false

query1	0.05	0.04	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.26	0.27	0.25
query6	1.17	0.63	0.65
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.64	0.53	0.54
query10	0.58	0.58	0.58
query11	0.17	0.11	0.11
query12	0.16	0.13	0.12
query13	0.63	0.63	0.62
query14	1.04	1.04	1.02
query15	0.87	0.84	0.88
query16	0.41	0.41	0.40
query17	1.06	1.03	1.04
query18	0.21	0.20	0.20
query19	1.98	1.83	1.81
query20	0.01	0.02	0.01
query21	15.49	0.93	0.57
query22	0.77	1.12	0.73
query23	14.89	1.39	0.65
query24	7.05	1.49	1.12
query25	0.53	0.15	0.25
query26	0.59	0.16	0.14
query27	0.06	0.06	0.06
query28	9.87	1.36	0.94
query29	12.56	3.88	3.22
query30	0.29	0.14	0.11
query31	2.84	0.58	0.41
query32	3.24	0.56	0.48
query33	3.08	3.14	3.08
query34	16.26	5.46	4.85
query35	4.92	4.89	4.82
query36	0.73	0.52	0.49
query37	0.10	0.07	0.07
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.17	0.16	0.16
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.97 s
Total hot run time: 30.74 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 96.00% (48/50) 🎉
Increment coverage report
Complete coverage report

@liaoxin01
Copy link
Contributor Author

run buildall

@liaoxin01 liaoxin01 marked this pull request as ready for review September 25, 2025 15:02
@doris-robot
Copy link

ClickBench: Total hot run time: 30.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c039e63da20840f7cb6f531fcfde0d7b96ec90b, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.06
query3	0.26	0.08	0.08
query4	1.61	0.13	0.12
query5	0.27	0.26	0.26
query6	1.20	0.66	0.65
query7	0.04	0.03	0.03
query8	0.07	0.04	0.05
query9	0.62	0.54	0.53
query10	0.59	0.58	0.57
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.62	0.63	0.62
query14	1.04	1.06	1.04
query15	0.88	0.89	0.88
query16	0.40	0.39	0.40
query17	1.03	1.09	1.05
query18	0.21	0.20	0.20
query19	1.93	1.83	1.86
query20	0.01	0.01	0.02
query21	15.43	0.93	0.58
query22	0.75	1.26	0.67
query23	14.89	1.40	0.61
query24	7.42	0.97	0.36
query25	0.28	0.11	0.08
query26	0.59	0.17	0.13
query27	0.07	0.05	0.05
query28	8.68	1.41	0.92
query29	12.59	3.94	3.30
query30	0.29	0.14	0.11
query31	2.83	0.60	0.39
query32	3.23	0.55	0.50
query33	3.01	3.18	3.17
query34	16.16	5.45	4.84
query35	4.94	4.92	4.87
query36	0.70	0.52	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.04
query43	0.04	0.04	0.04
Total cold run time: 103.65 s
Total hot run time: 30.1 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 96.72% (59/61) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 96.72% (59/61) 🎉
Increment coverage report
Complete coverage report

@liaoxin01
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 96.72% (59/61) 🎉
Increment coverage report
Complete coverage report

Copy link
Contributor

@starocean999 starocean999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 27, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit c1f28b7 into apache:master Sep 27, 2025
31 of 32 checks passed
@liaoxin01 liaoxin01 deleted the fix_multi_group branch September 27, 2025 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/4.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants