Skip to content

branch-3.1: [fix](load) fix s3 load check failed when no source file found #56319 #56376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: branch-3.1
Choose a base branch
from
Open

branch-3.1: [fix](load) fix s3 load check failed when no source file found #56319 #56376

wants to merge 1 commit into from
+76 −0

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #56319

@liaoxin01
[fix](load) fix s3 load check failed when no source file found (#56319)
4afd5fe 
Related PR: #55498

Problem Summary:

When importing a file via S3 load, if the file does not exist on S3, the
load will report an error: label has already been used.

2025-09-22 23:00:57,177 WARN (pending-load-task-scheduler-pool-0|483)
[LoadTask.exec():110] LOAD_JOB=1758553246283, error_msg={Unexpected
failed to execute load task}
java.lang.IllegalStateException: null
at
com.google.common.base.Preconditions.checkState(Preconditions.java:499)
~[guava-33.2.1-jre.jar:?]
at
org.apache.doris.nereids.load.NereidsLoadingTaskPlanner.plan(NereidsLoadingTaskPlanner.java:146)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.load.loadv2.LoadLoadingTask.init(LoadLoadingTask.java:138)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.load.loadv2.BrokerLoadJob.createTask(BrokerLoadJob.java:274)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.load.loadv2.BrokerLoadJob.createLoadingTask(BrokerLoadJob.java:311)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.load.loadv2.BrokerLoadJob.onPendingTaskFinished(BrokerLoadJob.java:204)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.load.loadv2.BrokerLoadJob.onTaskFinished(BrokerLoadJob.java:163)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.load.loadv2.LoadTask.exec(LoadTask.java:102)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.task.MasterTask.run(MasterTask.java:31)
~[doris-fe.jar:1.2-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
@github-actions github-actions bot requested a review from morrySnow as a code owner September 24, 2025 03:58
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 24, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32616 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4afd5feafd9e6f80792af4b736eedac2dd733a66, data reload: false

------ Round 1 ----------------------------------
q1	17608	5434	5380	5380
q2	2031	413	292	292
q3	11888	1230	738	738
q4	10558	860	459	459
q5	9754	2369	2092	2092
q6	203	164	131	131
q7	906	746	616	616
q8	9347	1428	1152	1152
q9	5235	4900	4867	4867
q10	6756	2246	1831	1831
q11	483	273	264	264
q12	330	350	214	214
q13	17781	3562	3010	3010
q14	222	227	218	218
q15	525	488	455	455
q16	428	413	379	379
q17	626	847	375	375
q18	6806	6567	6457	6457
q19	1275	962	537	537
q20	330	342	214	214
q21	2743	2163	1974	1974
q22	1032	1008	961	961
Total cold run time: 106867 ms
Total hot run time: 32616 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5515	5488	5464	5464
q2	229	323	237	237
q3	2224	2576	2301	2301
q4	1415	1830	1365	1365
q5	4419	4941	4996	4941
q6	172	164	126	126
q7	2076	1952	1802	1802
q8	2713	2908	2734	2734
q9	7385	7349	7314	7314
q10	3102	3299	2740	2740
q11	580	523	483	483
q12	672	767	640	640
q13	3401	3771	3180	3180
q14	290	284	268	268
q15	521	476	468	468
q16	430	479	438	438
q17	1217	1732	1263	1263
q18	7727	7441	7394	7394
q19	783	1084	1143	1084
q20	1993	2029	1895	1895
q21	5296	4793	4594	4594
q22	1081	1065	1003	1003
Total cold run time: 53241 ms
Total hot run time: 51734 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192379 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4afd5feafd9e6f80792af4b736eedac2dd733a66, data reload: false

query1	962	400	401	400
query2	6240	1923	1841	1841
query3	8689	199	192	192
query4	33660	23779	23639	23639
query5	3847	594	446	446
query6	285	197	177	177
query7	4203	485	326	326
query8	303	243	230	230
query9	9338	2577	2563	2563
query10	482	343	244	244
query11	18116	15500	15218	15218
query12	180	110	103	103
query13	1567	558	430	430
query14	9856	6514	7183	6514
query15	254	190	176	176
query16	8125	595	512	512
query17	1587	775	595	595
query18	2175	414	327	327
query19	247	196	164	164
query20	122	121	123	121
query21	207	137	108	108
query22	4726	4662	4635	4635
query23	35280	34369	34040	34040
query24	7291	2704	2697	2697
query25	551	504	434	434
query26	814	286	181	181
query27	2058	496	368	368
query28	5135	2253	2217	2217
query29	713	595	455	455
query30	259	201	163	163
query31	1031	933	854	854
query32	103	60	64	60
query33	490	359	320	320
query34	765	895	526	526
query35	789	824	714	714
query36	996	1055	989	989
query37	108	96	70	70
query38	4074	4041	3970	3970
query39	1547	1461	1491	1461
query40	226	123	111	111
query41	50	48	46	46
query42	119	109	106	106
query43	507	511	496	496
query44	1317	841	833	833
query45	180	180	176	176
query46	883	1043	676	676
query47	1963	1995	1935	1935
query48	426	447	349	349
query49	728	508	398	398
query50	682	697	428	428
query51	7378	7336	7238	7238
query52	105	106	91	91
query53	231	262	191	191
query54	546	551	469	469
query55	83	83	80	80
query56	279	284	271	271
query57	1299	1289	1223	1223
query58	239	221	217	217
query59	3053	3132	3047	3047
query60	290	293	264	264
query61	116	116	108	108
query62	847	760	710	710
query63	233	191	206	191
query64	3811	1006	648	648
query65	3362	3320	3276	3276
query66	810	428	320	320
query67	16732	15829	15833	15829
query68	7711	833	558	558
query69	501	302	269	269
query70	1205	1104	1140	1104
query71	443	291	270	270
query72	5166	3749	3805	3749
query73	643	742	355	355
query74	10132	9528	8971	8971
query75	3926	3153	2678	2678
query76	3540	1167	767	767
query77	756	373	283	283
query78	10294	10516	9637	9637
query79	3922	888	588	588
query80	766	522	430	430
query81	491	262	220	220
query82	342	118	90	90
query83	176	163	141	141
query84	291	100	77	77
query85	738	381	306	306
query86	344	314	286	286
query87	4400	4309	4285	4285
query88	3226	2420	2421	2420
query89	428	343	292	292
query90	2031	189	188	188
query91	133	143	110	110
query92	66	56	54	54
query93	2165	899	556	556
query94	643	416	270	270
query95	353	284	274	274
query96	505	611	284	284
query97	3185	3323	3159	3159
query98	217	210	199	199
query99	1523	1438	1302	1302
Total cold run time: 293932 ms
Total hot run time: 192379 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4afd5feafd9e6f80792af4b736eedac2dd733a66, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.52	0.51	0.51
query6	1.13	0.72	0.75
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.58	0.51	0.50
query10	0.55	0.54	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.10
query13	0.63	0.60	0.59
query14	0.78	0.79	0.80
query15	0.84	0.83	0.82
query16	0.39	0.38	0.37
query17	0.96	1.04	1.00
query18	0.23	0.23	0.23
query19	1.94	1.87	1.80
query20	0.01	0.01	0.01
query21	15.39	0.94	0.58
query22	0.74	0.82	0.67
query23	15.04	1.41	0.60
query24	3.34	1.18	2.27
query25	0.13	0.07	0.05
query26	0.26	0.14	0.13
query27	0.05	0.05	0.04
query28	14.50	0.94	0.43
query29	12.57	3.86	3.26
query30	0.25	0.10	0.06
query31	2.81	0.60	0.37
query32	3.23	0.53	0.47
query33	3.08	2.98	3.05
query34	16.82	5.13	4.50
query35	4.60	4.56	4.58
query36	0.64	0.49	0.47
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.78 s
Total hot run time: 28.99 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hello-stephen @doris-robot @dataroaring @liaoxin01