OOM and request cancellation when reading large objects

[michaelpaliy]

Problem:

When attempting to read a large file (~1GB) from S3 using the v3 Encryption SDK, the request fails with a hidden OutOfMemoryError. In v2 SDK, the same operation works without issues.

Stack trace:

java.io.IOException: Request cancelled
	at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:63)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
	at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
	at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
	at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1324)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:856)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$createExecutionFuture$3(NettyRequestExecutor.java:135)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:840)

The exception reported is misleading (FutureCancelledException), but after debugging I found it’s masking an actual OOM:

software.amazon.awssdk.http.nio.netty.internal.FutureCancelledException: java.lang.OutOfMemoryError: Java heap space

Steps to reproduce:

S3EncryptionClient s3EncryptionClient = S3EncryptionClient.builder()
        .enableDelayedAuthenticationMode(true)
        .kmsKeyId("XXX")
        .build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
        .bucket("XXX")
        .key("XXX")
        .build();
try (BufferedReader stream = new BufferedReader(new InputStreamReader(
        s3EncryptionClient.getObject(getObjectRequest, ResponseTransformer.toInputStream())))) {
    String line;
    while ((line = stream.readLine()) != null) {
        System.out.println("Read line: " + line);
        // Process the line
    }
}

I am using the latest version 3.4.0. The file size being tested is approximately 1GB. The runtime environment is Java 17.

[lucasmcdonald3]

Hi @michaelpaliy --

I wasn't able to reproduce the issue on my end -- v2 and v3 used comparable amounts of memory in my testing.

Could you profile your memory usage and let us know what you're seeing? We'd be happy to dig into this with more information.

Thanks,
Lucas

[michaelpaliy]

Hi @lucasmcdonald3 , Thanks for the reply!

Changing the crypto provider to Bouncy Castle fixed the issue for me, as it's done in the tests: S3EncryptionClientStreamTest:
https://github.com/aws/amazon-s3-encryption-client-java/blob/main/src/test/java/software/amazon/encryption/s3/S3EncryptionClientStreamTest.java#L422

It's important to note that the exception thrown before applying this fix doesn’t really point to the actual cause, and the OutOfMemoryError seems to be swallowed, which makes debugging quite tricky.