Enable Multi-threaded Support

Version 1.003.3 of SFTP Gateway supports configurable multi-threading for uploading files to S3 in parallel.

In previous versions of SFTP Gateway, it was recommended that the server configuration disable the multi-threading capabilities to avoid overloading the system's memory. In the new version, we recommend enabling the multi-threading capabilities and setting a fixed number of worker threads to alleviate the load on the system's memory .

The number of workers is set to 5 by default, which functions well on instances as small as a 1GB t2.micro instance. On larger instances, the number of workers can be increased, around 5 slots per every 2 GB of memory is a good rule of thumb for setting the number of workers.

To configure SFTP Gateway to use multi-threading:

1. Modify the /home/ec2-user/.sftpgateway/sftpgateway.properties file to set sftpgateway.singlethread=no
2. Set the desired number of workers by running the command: ts -S n where n is the number of workers.
3. Edit the TS_SLOTS value in /usr/local/bin/movetos3.sh to ensure the configured number of workers is preserved through reboots.

Large sets of files

In version 1.003.3, SFTP Gateway can handle more files uploaded to the server at one time. The number of files the server can handle, now depends on the memory size of the instance. To calculate the number of files your instance can handle at one time, take the memory size in KB and divide it by 256. For example, a 1GB instance can handle about 3,900 files uploaded to it at a time.