We have developed the following process for recovering SFTP Gateway users and configurations. This will be useful in the following cases:
- A region outage requires you to do a cold failover to another region
- You need to migrate or replicate to another region
- An SFTP Gateway instance is accidentally deleted
- You are upgrading from a single SFTP Gateway instance to a high availability stack
Note: The backup and recovery process captures the credentials for both password and SSH key based authentication.
Make sure you test this process thoroughly in your own environment.
Disclaimer: This backup and recovery strategy is intended for standard installations of SFTP Gateway. If you have customized your SFTP Gateway installation(s), please contact us at firstname.lastname@example.org to see if this solution will work for you.
Creating a backup
The SFTP Gateway backup script is a Python script that you install on your SFTP Gateway instance. The script backs up the following data to a single flat YAML file (and later gets compressed as a
- user properties
- user passwords/keys
- global SFTP Gateway properties
This backup artifact can be imported into a new SFTP Gateway stack to restore settings and configuration.
sftpgw-backup.py script will create a backup file named
sftpgw-backup-YYYY-MM-DD-HH-MM-SS.tar.gz. By default, this file is saved in the
/home/ec2-user/backups/ directory (which gets created if it doesn't already exist).
To run the backup script:
- SSH to the server as the
- Download the backup script
cd /home/ec2-user/ wget https://s3.amazonaws.com/thorntech-public-documents/sftpgateway/backup-and-recovery/sftpgw-backup.py
- Run the backup script
sudo python sftpgw-backup.py
This will create a backup file
sftpgw-YYYY-MM-DD-HH-MM-SS.tar.gz. This will later be used to recover SFTP Gateway users and configurations.
(Optional) Use the
--destination-dir parameter to save the backup artifact to a specific location.
sudo python sftpgw-backup.py --destination-dir /path/to/backup/location/
This parameter will create the directory if it doesn't exist. The script will fail if the destination is a file.
Once a backup artifact has been created, you can copy it to S3 or some other external location. This can be done using a "backup" SFTP user and saving the file to that user's uploads directory.
To copy the backup artifact using SFTP Gateway:
- Create a "backup" user with the web interface, CLI, or API
- You can point this user's uploads to a specific S3 bucket
- Run the backup script with the backup user's
/home/backup-user/home/backup-user/uploadsdirectory as the destination
sudo python sftpgw-backup.py --destination-dir /home/backup/home/backup/uploads
This will upload the backup file to S3 as soon as it is created. Storing the backup artifact on S3 will improve its availability should something happen to the SFTP Gateway instance.
Alternatively, you can copy the backup artifact to an external location manually, or in an automated fashion.
A scheduled backup can help improve business continuity since your user settings will be up to date as far as the most recent backup.
To schedule a backup, you can create a cron job to run the backup script.
Here is an example for running the backup at 11:00 PM every Friday:
- Open the root crontab for editing. This will open the crontab in vim1
sudo crontab -e
- Enter the following as a new line
0 23 * * Fri PATH=$PATH:/usr/local/bin; python /home/ec2-user/sftpgw-backup.py --destination-dir /home/ec2-user/backups >> /var/log/sftpgw/backup.log 2>&1
This is a breakdown of the syntax:
0 23 * * Fri- Time schedule that runs script at 23:00 server time every Friday.
/usr/local/binto the working path for this cronjob.
python /home/ec2-user/sftpgw-backup.py --destination-dir /home/ec2-user/backups- Runs the script and writes the backup files to
>> /var/log/sftpgw/backup.log 2>&1- Directs all output to a specific log.
Here is a good resource for building cronjob schedules - https://crontab.guru/
To recover SFTP Gateway users and configurations:
- Connect to the server as the
ec2-userover SSH (https://help.thorntech.com/help/log-into-the-ec2-instance)
- Download the recovery script:
cd /home/ec2-user/ wget https://s3.amazonaws.com/thorntech-public-documents/sftpgateway/backup-and-recovery/sftpgw-recovery.py
- Copy the
sftpgw-backup-YYYY-MM-DD-HH-MM-SS.yml.tar.gzfile to the server. For example, if the backup file is stored in S3, you can use the
aws s3 cpcommand:
aws s3 cp s3://bucketname/backups/sftpgw-backup-YYYY-MM-DD-HH-MM-SS.tar.gz /home/ec2-user/backup_files/
- Run the python recovery script
sudo python sftpgw-recovery.py sftpgw-YYYY-MM-DD-HH-MM-SS.tar.gz
Your new instance will be in the same state as the original instance, including your users and SFTP Gateway configuration.
1: If you are unfamiliar with the text editor vim, here is a good resource to get you started - https://learnxinyminutes.com/docs/vim/