How to Prevent Hidden S3 Charges from Incomplete Multipart Uploads

Overview

When a large file upload to Amazon S3 fails partway through — whether due to a network interruption, a timeout, or a server restart — the parts that were already uploaded remain in your S3 bucket as an incomplete multipart upload. These parts are billed as standard S3 storage, but they are completely invisible in the S3 console, aws s3 ls, and the SFTP Gateway admin UI. They only appear through a specific API call that most people never run.

Left unchecked, incomplete multipart uploads can silently accumulate and inflate your S3 storage bill. For example, a single failed 100 GiB upload can leave behind ~80 GiB of orphaned parts — roughly $1.84/month in us-east-1 — with no indication in the console that the storage is being used.

The fix is a one-time S3 lifecycle rule that automatically cleans up incomplete uploads older than 7 days.

Azure and GCP customers: This issue is specific to Amazon S3. Azure Blob Storage and Google Cloud Storage automatically discard incomplete uploads after 7 days with no configuration needed. No action is required on your part.

How incomplete multipart uploads happen

When SFTP Gateway uploads a file to S3, it uses S3's multipart upload API for any file larger than 8 MB (the default threshold). The file is split into parts, each part is uploaded individually, and then a final "complete" call assembles them into the final object. Since most real-world file transfers exceed 8 MB, multipart uploads are used for the majority of uploads.

SFTP Gateway does handle failures gracefully — if it detects an error during upload, it aborts the multipart upload and cleans up the parts. However, there are situations where the abort cannot run:

The EC2 instance is terminated or restarted during an upload
The JVM crashes or is killed mid-upload
A network disconnection happens at the wrong moment, bypassing the error handler

In these cases, the already-uploaded parts remain in the bucket as orphaned storage.

These orphaned parts:

Are billed at the same rate as regular S3 objects
Do not appear in the S3 console object listing, aws s3 ls, or the SFTP Gateway admin UI
Persist indefinitely unless explicitly aborted or cleaned up by a lifecycle rule
Cannot be downloaded — they are not usable as files

This is not an SFTP Gateway-specific issue. Any application that uses S3 multipart uploads (including the AWS CLI, SDKs, and other transfer tools) can create orphaned parts if an upload fails. Adding a lifecycle rule is considered standard S3 hygiene.

How to check for incomplete multipart uploads

Run this command to check a specific bucket:

aws s3api list-multipart-uploads --bucket YOUR-BUCKET-NAME

If there are incomplete uploads, you will see output like:

{
    "Uploads": [
        {
            "Key": "user1/large-file.zip",
            "UploadId": "exampleId123...",
            "Initiated": "2026-01-15T10:30:00+00:00",
            "StorageClass": "STANDARD"
        }
    ]
}

If the output contains no Uploads array, the bucket is clean.

To check the total size of parts for a specific incomplete upload:

aws s3api list-parts \
  --bucket YOUR-BUCKET-NAME \
  --key "user1/large-file.zip" \
  --upload-id "exampleId123..."

The fix: add a lifecycle rule

Add an S3 lifecycle rule that automatically aborts incomplete multipart uploads older than 7 days. This is safe — a legitimate SFTP upload completes in minutes to hours, not days. Only uploads that have been stuck for more than 7 days — which are almost certainly abandoned — will be aborted.

Once the rule is in place, it also retroactively cleans up any existing orphaned uploads.

Option 1: AWS Console

Open the S3 console and select your bucket.
Go to the Management tab.
Click Create lifecycle rule.
Configure the rule:
- Rule name: abort-incomplete-multipart-uploads
- Rule scope: Apply to all objects in the bucket
- Check Delete expired object delete markers or incomplete multipart uploads
- Check Delete incomplete multipart uploads
- Enter 7 for the number of days
Click Create rule.

Option 2: AWS CLI

aws s3api put-bucket-lifecycle-configuration \
  --bucket YOUR-BUCKET-NAME \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "abort-incomplete-multipart-uploads",
        "Status": "Enabled",
        "Filter": {},
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 7
        }
      }
    ]
  }'

Important: If your bucket already has lifecycle rules, the command above will replace them. To add the rule without removing existing rules, first retrieve the current configuration, add the new rule, and put it back:

# 1. Save existing rules
aws s3api get-bucket-lifecycle-configuration \
  --bucket YOUR-BUCKET-NAME > lifecycle.json

# 2. Add the new rule (requires jq)
jq '.Rules += [{
  "ID": "abort-incomplete-multipart-uploads",
  "Status": "Enabled",
  "Filter": {},
  "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
}]' lifecycle.json > lifecycle-updated.json

# 3. Apply the updated configuration
aws s3api put-bucket-lifecycle-configuration \
  --bucket YOUR-BUCKET-NAME \
  --lifecycle-configuration file://lifecycle-updated.json

Option 3: Terraform

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id     = "abort-incomplete-multipart-uploads"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

Option 4: CloudFormation

BucketLifecycleConfiguration:
  Type: AWS::S3::Bucket
  Properties:
    LifecycleConfiguration:
      Rules:
        - Id: abort-incomplete-multipart-uploads
          Status: Enabled
          AbortIncompleteMultipartUpload:
            DaysAfterInitiation: 7

How to clean up existing incomplete uploads

The lifecycle rule is retroactive — once applied, S3 will automatically abort any existing incomplete uploads that are older than the specified number of days. S3 runs lifecycle rules once per day, so it may take up to 24–48 hours for existing uploads to be cleaned up.

If you need to clean up a specific incomplete upload immediately, you can abort it manually:

aws s3api abort-multipart-upload \
  --bucket YOUR-BUCKET-NAME \
  --key "user1/large-file.zip" \
  --upload-id "exampleId123..."

To abort all incomplete uploads in a bucket:

aws s3api list-multipart-uploads --bucket YOUR-BUCKET-NAME \
  --query 'Uploads[].{Key:Key,UploadId:UploadId}' --output json | \
  jq -r '.[] | "\(.Key)\t\(.UploadId)"' | \
  while IFS=$'\t' read -r key upload_id; do
    echo "Aborting: $key"
    aws s3api abort-multipart-upload \
      --bucket YOUR-BUCKET-NAME \
      --key "$key" \
      --upload-id "$upload_id"
  done

Verifying the rule is in place

To confirm your bucket has the lifecycle rule:

aws s3api get-bucket-lifecycle-configuration --bucket YOUR-BUCKET-NAME

You should see AbortIncompleteMultipartUpload with DaysAfterInitiation in the output.

FAQ

Will this rule delete files that are currently being uploaded?

No. The 7-day threshold is deliberately conservative. A normal SFTP upload completes in minutes to hours, not days. Only uploads that have been stuck for more than 7 days — which are almost certainly abandoned — will be aborted.

Does this apply to Azure Blob Storage or Google Cloud Storage?

No. Azure automatically discards uncommitted block blobs after 7 days, and Google Cloud Storage automatically expires incomplete resumable uploads after 7 days. No configuration is needed for either provider. This lifecycle rule is only necessary for Amazon S3.

How much could this save?

S3 Standard storage costs $0.023/GiB/month in us-east-1. A single failed 100 GiB upload can strand ~80 GiB of parts ($1.84/month). Multiple retries multiply the waste. The cost is modest per upload but accumulates indefinitely since the parts never expire on their own.

Is this an SFTP Gateway issue?

SFTP Gateway properly aborts multipart uploads when it detects a failure. However, if the server crashes, is terminated, or loses network connectivity mid-upload, the abort cannot run and parts are left behind. This is a general S3 behavior — any tool that uses multipart uploads can leave behind orphaned parts. Adding this lifecycle rule is standard S3 best practice regardless of which tools you use.

SFTP Gateway Support

3.0