HADOOP-19740. S3A: add explicit "sdk" and "ec2" regions for region resolution. #8058

steveloughran · 2025-11-05T21:22:56Z

Pull out region resolution to new class RegionResolution for isolation and testing through TestRegionResolution.
Add new "sdk" and "ec2" mappings to hand down to SDK
Test in ITestS3AEndpointRegion to verify that either the SDK fails to resolve with an SDK message or it does resolve to something and the test assert fails instead.

How was this patch tested?

New tests.

running regression test with configured endpoint.
planning to run a test with a bucket set to "sdk" to see that the sdk handles it through my .aws/config setting.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2025-11-05T23:07:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 10s		trunk passed
+1 💚	compile	0m 45s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 46s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 35s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	1m 25s	/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html	hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings.
+1 💚	shadedclient	26m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 23s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 23 new + 16 unchanged - 4 fixed = 39 total (was 20)
+1 💚	mvnsite	0m 43s		the patch passed
-1 ❌	javadoc	0m 30s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 17 new + 928 unchanged - 0 fixed = 945 total (was 928)
-1 ❌	javadoc	0m 28s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 15 new + 850 unchanged - 0 fixed = 865 total (was 850)
+1 💚	spotbugs	1m 21s		the patch passed
+1 💚	shadedclient	26m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 18s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		103m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/1/artifact/out/Dockerfile
GITHUB PR	#8058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux be91d7e026c0 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ec5ae6b`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/1/testReport/
Max. process+thread count	613 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/1/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2025-11-06T14:43:45Z

running full test suites with java17 and the test region set to sdk. as usual, it's that forked ITestS3ACommitterMRJob playing up, specifically with "null" somehow ending up as the region/endpoint.

[ERROR] org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute -- Time elapsed: 9.958 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
Job job_1762435181334_0003 failed in state FAILED with cause Job setup failed : java.net.UnknownHostException: getFileStatus on s3a://stevel-london/test/ITestS3ACommitterMRJob-execute-magic/_SUCCESS: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.:    software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.: stevel-london.s3.null.amazonaws.com
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.apache.hadoop.fs.s3a.impl.ErrorTranslation.wrapWithInnerIOE(ErrorTranslation.java:237)
        at org.apache.hadoop.fs.s3a.impl.ErrorTranslation.maybeExtractIOException(ErrorTranslation.java:207)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:212)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4061)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3971)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteWithoutCloseCheck(S3AFileSystem.java:3570)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3542)
        at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:467)
        at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:611)
        at org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.setupJob(MagicS3GuardCommitter.java:106)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59)
        at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:181)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
        at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:7029)
        at software.amazon.awssdk.services.s3.DelegatingS3Client.lambda$headObject$56(DelegatingS3Client.java:5678)
        at software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient.invokeOperation(S3CrossRegionSyncClient.java:67)
        at software.amazon.awssdk.services.s3.DelegatingS3Client.headObject(DelegatingS3Client.java:5678)
        at org.apache.hadoop.fs.s3a.impl.S3AStoreImpl.lambda$headObject$3(S3AStoreImpl.java:688)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.impl.S3AStoreImpl.headObject(S3AStoreImpl.java:675)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:3048)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:3028)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4043)
        ... 11 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: stevel-london.s3.null.amazonaws.com
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper2.setLastException(RetryableStageHelper2.java:226)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:65)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        ... 31 more
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: stevel-london.s3.null.amazonaws.com: nodename nor servname provided, or not known
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: stevel-london.s3.null.amazonaws.com
Caused by: java.net.UnknownHostException: stevel-london.s3.null.amazonaws.com
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at software.amazon.awssdk.thirdparty.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
        at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
        ... 43 more

.
Consult logs under /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/yarn-2025-11-06-13.19.38.98/yarn-166883501
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
        at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:311)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

…solution. * Pull out region resolution to new class RegionResolution for isolation and testing through TestRegionResolution. * Add new "sdk" and "ec2" mappings to hand down to SDK * Test in ITestS3AEndpointRegion to verify that either the SDK fails to resolve with an SDK message or it does resolve to *something* and the test assert fails instead.

IF the region == ec2 then we don't handle off to the SDK chain, instead only use the bit of the SDK for metadata retrieval. Nominally private/unstable, but cloudstore has used it in the past... If it is removed after an SDK update, we will have to handle it.

hadoop-yetus · 2025-11-06T19:42:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	3m 34s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	compile	0m 54s	/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 24s	/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-0 ⚠️	checkstyle	0m 22s	/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 24s	/branch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in trunk failed.
-1 ❌	javadoc	0m 24s	/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 24s	/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	spotbugs	0m 25s	/branch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in trunk failed.
+1 💚	shadedclient	2m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 24s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	0m 23s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	0m 23s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 15s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	0m 15s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 21s	/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 23s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javadoc	0m 23s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 23s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	spotbugs	0m 24s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	shadedclient	4m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 24s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 16s		ASF License check generated no output?
		16m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/2/artifact/out/Dockerfile
GITHUB PR	#8058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 72c378f32934 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6f662bf`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/2/testReport/
Max. process+thread count	79 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8058/2/console
versions	git=2.25.1 maven=3.9.11
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

github-actions bot added trunk TOOLS AWS labels Nov 5, 2025

steveloughran added 3 commits November 6, 2025 19:24

HADOOP-19740. flip the if() clause in client factory

6f662bf

steveloughran force-pushed the s3/HADOOP-19740-regions branch from ec5ae6b to 6f662bf Compare November 6, 2025 19:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19740. S3A: add explicit "sdk" and "ec2" regions for region resolution. #8058

HADOOP-19740. S3A: add explicit "sdk" and "ec2" regions for region resolution. #8058

Uh oh!

steveloughran commented Nov 5, 2025

Uh oh!

hadoop-yetus commented Nov 5, 2025

Uh oh!

steveloughran commented Nov 6, 2025

Uh oh!

hadoop-yetus commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HADOOP-19740. S3A: add explicit "sdk" and "ec2" regions for region resolution. #8058

Are you sure you want to change the base?

HADOOP-19740. S3A: add explicit "sdk" and "ec2" regions for region resolution. #8058

Uh oh!

Conversation

steveloughran commented Nov 5, 2025

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Nov 5, 2025

Uh oh!

steveloughran commented Nov 6, 2025

Uh oh!

hadoop-yetus commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants