How to limit/abort the multipart upload part requests after failed upload?

[jigneshkumarparekh]

Hello Everyone,

I'm trying to upload a large file (>5GB) using AWS SDK for Javascript v3 - Multipart Upload.
While upload is in progress, if any error happens like sessionToken expires, it throws 400 Bad Request. Even though the upload of a chunk failed, it keeps triggering request for other parts and keeps on trying for all the remaining parts. I'm calling abort in case of any error but the thing is catch block doesn't get executed until SDK is done trying for all the parts.

Is there a way I can tell SDK to stop trying let's say after few call failed?

The reason I'm asking is: I'm displaying progress bar on the screen and in case of any error, I want to stop displaying the progress bar.

Here is the code I'm trying:

import {
  AbortMultipartUploadCommandOutput,
  CompleteMultipartUploadCommandOutput,
  S3,
  Tag
} from '@aws-sdk/client-s3';
import { Progress, Upload } from '@aws-sdk/lib-storage';

...

const s3Client = new S3({
  region: config.region,
  credentials: {
    accessKeyId: config.accessKeyId,
    secretAccessKey: config.secretAccessKey,
    sessionToken: config.sessionToken
  }
});

const uploadReq = new Upload({
  client: s3Client,
  params: {
    Bucket: <bucketName>,
    Key: <key>,
    Body: <file_body>
  },
  tags: [], // optional tags
  queueSize: 4, // optional concurrency configuration
  partSize: 1024 * 1024 * 10, // (10MB) - optional size of each part, in bytes. e.g. 1024 * 1024 * 10
  leavePartsOnError: false // optional manually handle dropped parts
});

const uploadReq$ = from(uploadReq.done()).pipe(
  catchError(() => {
    uploadReq.abort();
    return of(null);
  })
);

Here is the screenshot:

As you can see ^, first few calls succeed and then failure happens. But instead of stop trying, it continue to try uploading for all the remaining parts and then only go execute catch block.

I tried configuring maxAttempts or retryMode but no luck.

Am I doing something wrong here? Please advice.

Many thanks in advance!

[ajredniwja]

I think you should be able to use something like: http error event
https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-a-response-event-handler.html

Discussion on retry behavior of parts on error: #2311

[jigneshkumarparekh]

Thanks @ajredniwja for the guidance. Let me try that out.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.