SFTP Gateway Support

3.0

SFTP Gateway 2.0 Allow Users to Delete from Downloads

Note: This page applies to SFTP Gateway version 2.x. Visit Here for documentation on version 3.x.

By default, downloads directories are read-only because the S3 bucket is intended to be the source of truth.

Unlike uploads, which are transferred to S3 bucket in near real-time, the downloads directory is synced with the contents of the S3 bucket by the s3sync process every few minutes. If a file is deleted from the local server, it will reappear the next time the s3sync process runs.

In order to allow users to delete from the downloads directory, there are two options:

  • Modify the s3sync process to make the local directory the source of truth

  • Listen for IN_DELETE Event and run a custom script to delete from S3 bucket

Make download directory the source of truth

The s3sync command uses the /opt/sftpgw/sync.d/username.sync file to map source S3 bucket folders to target server directories.

Each entry in the username.sync file is a single line and consists of 3 segments separated by semicolons.

  • Segment 1, is the local target directory that the S3 bucket content will be synced to.

  • Segment 2, is the user for which the sync operation will apply to.

  • Segment 3, is the S3 bucket source folder that the content is synced from.

If you swap segment 1 with segment 3, the server will become the source of truth. The files will still be stored on S3 bucket, but to add, modify, or delete the files those operations need to be done on the server.

You can then change the permissions of the downloads directory to allow the user read-write access. To do this run the command:

sudo chown username:username /home/username/home/username/downloads/

Listen for IN_DELETE event

If you want to allow users to delete files from their downloads directory, you could add a custom incrontab for the user to monitor their downloads directory for an IN_DELETE event. This could then call a custom script that will delete the file from S3 bucket if it is deleted from the server. Once the file is deleted from S3 bucket, it will no longer get synced back to the server.

To add an incrontab for the user, do the following:

Create a custom user incrontab with the command:

sudo vim /etc/incron.d/username.download.sftpgw.incron

This will open the incrontab in vim [1], then add this line:

/home/username/home/username/downloads IN_DELETE /opt/sftpgw/deletefroms3.sh $# username

The deletefroms3.sh script does not exist currently, but can easily be created to call the AWS cli to delete the file in the S3 bucket.

Create this script with the command sudo vim /opt/sftpgw/deletefroms3.sh[1], and add the following script:

#!/bin/bash
  
LOG_FILE="/var/log/sftpgw/deletefroms3.log"

function sftpgwprop {
        local prefix="${1}="
        local str=$(grep "${1}" /opt/sftpgw/sftpgateway.properties 2>/dev/null)
        echo ${str#${prefix}}
}

# This pulls the default S3 bucket from the sftpgateway.properties file. 
# If you have set a custom S3 bucket for a user then you may have to pull the 
# bucket from the user.properties file or hardcode it here. 
bucketname="$(sftpgwprop 'sftpgateway.bucketname')"
region="$(sftpgwprop 'sftpgateway.region')"

file=$1
user=$2

aws s3 --region "$region" rm "s3://${bucketname}/${user}/downloads/${file}" >> $LOG_FILE 2>&1

# end of script

You will have to make deletefroms3.sh executable with the command:

sudo chmod +x /opt/sftpgw/deletefroms3.sh

You will then have to change the permissions of the downloads directory to allow the user to delete items from it. To do this run the command:

sudo chown username:username /home/username/home/username/downloads/

Finally, you will have to add the s3:DeleteObject action to the IAM role attached to the EC2 instance

Some things to consider with either of these approaches:

  • You may want to turn on versioning in your S3 bucket, so that if a user deletes an item from the directory you will still be able to access it from S3. For more information on S3 versioning please see this [AWS documentation][versioning].

  • You will have to do this for each user you wish to give this functionality to.

  • You may need to add the sse.sync.option encryption option property to the user's /home/username/.sftpgateway/user.properties file if you are synching files from the server to the bucket.

  • You will also have to change the ownership of each user's downloads directory to allow them to write and delete files from it. To do this run the command sudo chown username:username /home/username/home/username/downloads

Queue IN_DELETE events in Task Spooler

If your SFTP users are deleting many files at once (e.g. 1,000 files at a time), this will crash the server. This is because each file deletion creates a new thread containing the AWS CLI. And hundreds of these threads could easily consume all server resources.

To fix this, you will need to do the following:

(1) Rename your deletefroms3.sh script:

mv /opt/sftpgw/deletefroms3.sh /opt/sftpgw/deletefroms3

Your incrontab entries are calling the script /opt/sftpgw/deletefroms3.sh.

You don't want to call this script directly, so you rename the script to something slightly different.

Instead, you'll call a wrapper script, which you'll create in the next step.

(2) Create a wrapper script:

touch /opt/sftpgw/deletefroms3.sh
chmod +x /opt/sftpgw/deletefroms3.sh

deletefroms3.sh is now your wrapper script. At the moment it's empty, so you'll fill in the contents in the next step.

(3) Paste in the following contents into /opt/sftpgw/deletefroms3.sh:

#!/bin/sh

(/usr/local/bin/ts -n bash -c '${0} ${1+"$@"}' /opt/sftpgw/deletefroms3 "$@") &

This wrapper script calls /opt/sftpgw/deletefroms3 while forwarding its parameters (i.e. "$@").

The key point is that the wrapper script calls deletefroms3 indirectly, by queueing it in task spooler (ts).

The syntax is a bit difficult to reason about, but here's an explanation of the syntax:

  • ( ... ) &: Runs the command within the parentheses in a background sub-thread.
  • /usr/local/bin/ts: This is the task spooler command.
  • -n bash: Specifies the scripting language.
  • -c /opt/sftpgw/deletefroms3 "$@": The command is a string that is passed into Bash.
  • -c '${0} ${1+"$@"}: Parses the command string. ${0} represents /opt/sftpgw/deletefroms3. ${1+"$@"} puts everything afterward into an array.

  1. If you are unfamiliar with the text editor vim, here is a good resource to get you started - https://learnxinyminutes.com/docs/vim/ ↩ ↩

← SFTP Gateway 2.0 Multiple SSH Keys per UserAWS Enable Fips Mode →