Unable to use 2g files as the content for an UploadPartRequest when performing multipart uploads

[docrozza]

Describe the bug

Unable to perform very large multipart uploads as the largest size for an individual part is Integer.MAX_VALUE.

The docs say the current limits for a part should be 5g and I could uploads parts of at least 2g via the CLI - just not via the java SDK

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The multipart upload to work for large parts

Current Behavior

Fails with the error

Exception in thread "AwsEventLoop 6" java.lang.NullPointerException: Cannot invoke "java.nio.ByteBuffer.hasRemaining()" because "out" is null
	at software.amazon.awssdk.utils.async.ByteBufferStoringSubscriber.transferTo(ByteBufferStoringSubscriber.java:100)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtRequestBodyStreamAdapter.sendRequestBody(S3CrtRequestBodyStreamAdapter.java:47)
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure. (SDK Attempt Count: 1)
	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure. (SDK Attempt Count: 1)
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
	... 25 more

When debug was enabled on the CRT client the logs had an entry with error code 2064 but little else

Reproduction Steps

Create 2 files via

mkfile -n 2147483647 file2g_1
mkfile -n 2g file2g

Then pass these paths (and their file size of (long) Integer.MAX_VALUE and Integer.MAX_VALUE + 1L) to the following bit of code

    private void reproducer(String bucket, String s3Key, Path filePath, long fileSize) throws Exception {
        try (var client = S3AsyncClient.crtBuilder().build()) {
            var uploadId = client.createMultipartUpload(CreateMultipartUploadRequest.builder()
                    .bucket(bucket)
                    .key(s3Key)
                    .build())
                    .get()
                    .uploadId();

            var partRequest = UploadPartRequest.builder()
                    .bucket(bucket)
                    .key(s3Key)
                    .uploadId(uploadId)
                    .partNumber(1)
                    .contentLength(fileSize)
                    .build();

            var eTag = client.uploadPart(partRequest, fromFile(filePath)).get().eTag();
            var parts = List.of(CompletedPart.builder()
                    .partNumber(1)
                    .eTag(eTag)
                    .build());

            client.completeMultipartUpload(CompleteMultipartUploadRequest.builder()
                    .bucket(bucket)
                    .key(s3Key)
                    .uploadId(uploadId)
                    .multipartUpload(CompletedMultipartUpload.builder()
                            .parts(parts)
                            .build())
                    .build())
                    .get();
        }
    }

It works OK for file2g_1 but fails on file2g.

Possible Solution

No response

Additional Information/Context

I tried via the input stream method of AsyncRequestBody and that failed too

I had a suspicion it might be related to the checksum calculations following some step-debugging but changing the algorithms had no effect sadly

I also tried with a file of size 4g to check it wasn't a boundary issue on Integer.MAX but that failed as well.

AWS Java SDK version used

2.41.4

JDK version used

21

Operating System and version

Mac and Linux (EC2)

[RanVaknin]

Hi @docrozza,

Thanks for the info. I looked a bit into it. From what I can tell, the issue is happening on the CRT layer, and not the SDK itself.
Specifically, in this line the JNI code calls NewDirectByteBuffer(env, ptr, capacity) which calls Java's ByteBuffer API. Java has a 2GB limitation on byteBuffers because they are backed by arrays which use int for array indexing.

When NewDirectByteBuffer is passed a capacity > 2GB it returns null, resulting in the NPE that you see.

I'm going to reach out to the CRT team in hopes that they can provide some more guidance on this, however I'm not sure if this can be easily solved.

Thanks,
Ran~