Understanding S3 Storage Events from SFTP Gateway File Uploads

Overview

When uploading files through SFTP Gateway to Amazon S3, you may notice multiple storage events triggered for a single file upload. This is expected behavior due to how the SFTP protocol translates file operations to S3 API calls.

This article explains why these events occur and provides solutions for filtering them in your AWS Lambda functions or automation workflows.

How Many Events Are Generated?

Each file upload through SFTP Gateway generates 2-3 S3 events:

1. Initial file creation with metadata - A small file (0 bytes) is created with timestamp metadata (mtime, ctime, atime, md5).

2. File data upload - The file is overwritten with the actual file content.

3. Metadata update - File metadata is updated after the upload completes (timestamps are updated if preservation is enabled).

Why Does This Happen?

This behavior is inherent to how SFTP operations are translated to S3 APIs:

1. CREATE FILE command - When an SFTP client initiates an upload, it sends a "create file" command, which SFTP Gateway translates into creating a small object in S3 with initial metadata.

2. PUT FILE command - After the entire file data is transferred, the file is overwritten with the actual content.

3. SET ATTRIBUTES command - After the upload completes, SFTP Gateway updates metadata and optionally updates timestamps if preservation is enabled.

Note: Different SFTP clients may behave slightly differently, which can affect the exact timing of events. This is normal and depends on the client's implementation of the SFTP protocol.

S3 Event Details

For a single file upload to S3, you'll see:

Event Type	Object Size	Description
PutObject	0-36 bytes	Initial file creation with metadata
PutObject	Actual file size	File content uploaded
CopyObject	0 bytes	Metadata update (upload complete)

Example from CloudWatch Logs:

1. 2025-12-10T22:09:18.503Z  PutObject   uploads/Thorn_Tech.pdf  36 bytes
2. 2025-12-10T22:09:21.282Z  PutObject   uploads/Thorn_Tech.pdf  937756 bytes
3. 2025-12-10T22:09:23.355Z  CopyObject  uploads/Thorn_Tech.pdf  0 bytes

How to Handle Multiple Events in AWS Lambda

If these duplicate events are causing issues in your Lambda functions, here are the recommended approaches:

Solution 1: Filter by File Size (Recommended)

The simplest and most reliable solution is to ignore events for very small files (< 1 KB), as the initial creation event will always be minimal in size.

Python Example:

import json

def lambda_handler(event, context):
    for record in event['Records']:
        # Get the object size from the S3 event
        object_size = record['s3']['object']['size']
        object_key = record['s3']['object']['key']
        bucket_name = record['s3']['bucket']['name']
        
        # Ignore small files (initial creation event)
        if object_size < 1024:  # Less than 1 KB
            print(f"Ignoring small file event: {object_key} ({object_size} bytes)")
            continue
        
        # Process the actual file upload
        print(f"Processing file: {object_key} ({object_size} bytes)")
        process_file(bucket_name, object_key)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Processing complete')
    }

def process_file(bucket, key):
    # Your file processing logic here
    print(f"Processing s3://{bucket}/{key}")

Why this works:

• The initial file creation is always 0-36 bytes (shown as 0 bytes in S3 console)
• Actual file uploads will be larger than 1 KB in most cases
• Simple, fast, and doesn't require additional API calls
• Works reliably across all SFTP clients

Solution 2: Add a Delay Before Processing

For simple workflows with files under 25 MB, adding a brief delay before processing can ensure you're working with the final version of the file.

Python Example:

import time
import json

def lambda_handler(event, context):
    for record in event['Records']:
        object_key = record['s3']['object']['key']
        bucket_name = record['s3']['bucket']['name']
        
        # Wait 15 seconds to ensure all events have completed
        time.sleep(15)
        
        # Process the file
        print(f"Processing file: {object_key}")
        process_file(bucket_name, object_key)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Processing complete')
    }

def process_file(bucket, key):
    # Your file processing logic here
    print(f"Processing s3://{bucket}/{key}")

Note: This approach works well for smaller files (under 25 MB) where the upload completes quickly. For larger files or high-throughput systems, use Solution 1 instead.

Recommended Approach

For most use cases: Use Solution 1 (filter by file size) - it's simple, reliable, efficient, and doesn't require additional S3 API calls or delays.

For simple, low-traffic workflows: Use Solution 2 (delay approach) - only suitable for small files and scenarios where a 15-second delay is acceptable.

Viewing Events in CloudWatch Logs

To observe these events for debugging:

Step 1: Enable CloudTrail Data Events

1. Go to CloudTrail Console → Trails
2. Create a new trail or edit an existing one
3. Under Data events, select S3
4. Choose Write-only events (or All if you want read events too)
5. Specify your S3 bucket or select all buckets

Step 2: Enable CloudWatch Logs Integration

1. In your trail settings, click Edit next to CloudWatch Logs
2. Select Enabled
3. Create a new log group (e.g., /aws/cloudtrail/sftpgw-events)
4. CloudTrail will create an IAM role automatically

Step 3: Query Events in CloudWatch Logs Insights

1. Go to CloudWatch → Log groups → Select your log group
2. Click Logs Insights
3. Use this query to see S3 events:

fields @timestamp, eventName, requestParameters.key, additionalEventData.bytesTransferredIn
| filter eventSource = "s3.amazonaws.com"
| filter eventName in ["PutObject", "CopyObject"]
| filter requestParameters.bucketName = "your-bucket-name"
| sort @timestamp asc

To filter for a specific file:

fields @timestamp, eventName, requestParameters.key, additionalEventData.bytesTransferredIn
| filter eventSource = "s3.amazonaws.com"
| filter eventName in ["PutObject", "CopyObject"]
| filter requestParameters.key = "uploads/your-file.pdf"
| sort @timestamp asc

Lambda Function Configuration Tips

Event Notification Setup

1. Go to your S3 bucket → Properties → Event notifications
2. Create a notification configuration:
   • Event types: Select s3:ObjectCreated:* or specific events like s3:ObjectCreated:Put and s3:ObjectCreated:Copy
   • Destination: Choose your Lambda function
   • Prefix/Suffix filters: Optional, to limit which objects trigger the function (e.g., prefix: uploads/)

Lambda Timeout Considerations

Set appropriate timeouts based on your solution:

• Solution 1 (file size filter): 30-60 seconds is usually sufficient
• Solution 2 (delay approach): Minimum 30 seconds (15s delay + processing time)

Configure via Lambda console or AWS CLI:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --timeout 60

Permissions Required

Your Lambda execution role needs these permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:HeadObject"
            ],
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Additional Notes

• This behavior is expected and cannot be disabled, as it's how the SFTP protocol translates to S3 APIs.
• The extra events do not increase storage costs (only one object exists in S3).
• Timestamp preservation can be disabled in your SFTP client settings if you don't need original timestamps, but the three-event pattern will still occur.
• Different SFTP clients may produce slightly different event patterns, but the core behavior remains the same.
• SFTP Gateway sets custom metadata on objects including mtime, ctime, atime, and md5 for file verification.

Need Help?

If you're still experiencing issues with duplicate events or need assistance implementing these filtering strategies, please contact our support team at support@thorntech.com.