Allow Users to Delete from Downloads

By default, downloads directories are read only because the S3 bucket is intended to be the source of truth.

Unlike uploads, which are transferred to S3 in near real-time, the downloads directory is synced with the contents of S3 by the s3sync process every few minutes. If a file is deleted from the local server, it will reappear the next time the s3sync process runs.

In order to allow users to delete from the downloads directory, there are two options:

• Modify the s3sync process to make the local directory the source of truth

• Listen for IN_DELETE Event and run a custom script to delete from S3

Make download directory the source of truth

The s3sync command uses the /home/ec2-user/.sftpgateway/sftpgateway.sync file to map source S3 folders to target server directories.

Each entry in the sftpgateway.sync file is a single line and consists of 3 segments separated by semicolons.

• Segment 1, is the local target directory that the S3 content will be synced to.

• Segment 2, is the user for which the sync operation will apply to.

• Segment 3, is the S3 source folder that the content is synced from.

If you swap segment 1 with segment 3, the server will become the source of truth. The files will still be stored on S3, but to add, modify, or delete the files those operations need to be done on the server.

You can then change the permissions of the downloads directory to allow the user read/write access. To do this run the command:

sudo chown username:username /home/username/home/username/downloads/

Listen for IN_DELETE event

If you want to allow users to delete files from their downloads directory, you could add a custom entry to the root incrontab for their downloads directory to monitor for an IN_DELETE event. This could then call a custom script that will delete the file from S3 if it is deleted from the server. Once the file is deleted from S3, it will no longer get synced back to the server. To add an incrontab entry, do the following:

Edit the root incrontab with the command:

sudo incrontab -e

This will open the incrontab in vim^[1], then add this line:

/home/username/home/username/downloads IN_DELETE /opt/sftpgw/deletefroms3.sh $# username

The deletefroms3.sh script does not exist currently, but can easily be created to call the AWS S3 cli to delete the file in the S3 bucket.

Create this script with the command sudo vim /opt/sftpgw/deletefroms3.sh^[1], and add the following script:

 #!/bin/bash

 LOG_FILE="/var/log/movetos3/movetos3.log"

 file=$1

 user=$2

 aws s3 rm s3://sftpgateway-bucketname/$user/downloads/$file >> $LOG_FILE 2>&1

 # end of script

Note: Remember to change the sftpgateway-bucketname to reflect your S3 bucket.

You will have to make deletefroms3.sh executable with the command:

sudo chmod +x /opt/sftpgw/deletefroms3.sh

You will then have to change the permissions of the downloads directory to allow the user to delete items from it. To do this run the command:

sudo chown username:username /home/username/home/username/downloads/

Some things to consider with either of these approaches:

• You may want to turn on versioning in your S3 bucket, so that if a user deletes an item from the directory you will still be able to access it from S3. For more information on S3 versioning please see this AWS documentation.

• You will have to do this for each user you wish to give this functionality to.

• You may need to add the sse.sync.option encryption option property to the user's /home/username/.sftpgateway/user.propteries file if you are synching files from the server to the bucket.

• You will have to add the s3:DeleteObject action to the IAM role attached to the EC2 instance

• You will also have to change the ownership of each user's downloads directory to allow them to write and delete files from it. To do this run the command sudo chown username:username /home/username/home/username/downloads

Queue IN_DELETE events in Task Spooler

If your SFTP users are deleting many files at once (e.g. 1,000 files at a time), this will crash the server. This is because each file deletion creates a new thread containing the AWS CLI. And hundreds of these threads could easily consume all server resources.

To fix this, you will need to do the following:

(1) Rename your deletefroms3.sh script:

mv /opt/sftpgw/deletefroms3.sh /opt/sftpgw/deletefroms3

Your incrontab entries are calling the script /opt/sftpgw/deletefroms3.sh.

You don't want to call this script directly, so you rename the script to something slightly different.

Instead, you'll call a wrapper script, which you'll create in the next step.

(2) Create a wrapper script:

touch /opt/sftpgw/deletefroms3.sh
chmod +x /opt/sftpgw/deletefroms3.sh

deletefroms3.sh is now your wrapper script. At the moment it's empty, so you'll fill in the contents in the next step.

(3) Paste in the following contents into /opt/sftpgw/deletefroms3.sh:

#!/bin/sh

(/usr/local/bin/ts -n bash -c '${0} ${1+"$@"}' /opt/sftpgw/deletefroms3 "$@") &

This wrapper script calls /opt/sftpgw/deletefroms3 while forwarding its parameters (i.e. "$@").

The key point is that the wrapper script calls deletefroms3 indirectly, by queueing it in task spooler (ts).

The syntax is a bit difficult to reason about, but here's an explanation of the syntax:

• ( ... ) &: Runs the command within the parentheses in a background sub-thread.
• /usr/local/bin/ts: This is the task spooler command.
• -n bash: Specifies the scripting language.
• -c /opt/sftpgw/deletefroms3 "$@": The command is a string that is passed into Bash.
• -c '${0} ${1+"$@"}: Parses the command string. ${0} represents /opt/sftpgw/deletefroms3. ${1+"$@"} puts everything afterward into an array.

---

1. If you are unfamiliar with the text editor vim, here is a good resource to get you started - Learn X in Y Minutes - Vim ↩ ↩