SFTP Gateway Support

3.0

Python Post Configuration

Overview

When migrating from SFTP Gateway v2 to v3, you may notice some differences between how SFTP directories are configured. If your goal is to have a seamless transition, you may end up performing post-configuration changes on the v3 pre-production server to mimic the v2 experience.

On top of this, the custom container and upload paths on v2 do not get converted to Cloud Connections on v3 during the import process. While this feature exists on AWS, it does not exist on our Azure product. So you would need to create these Cloud Connections manually.

To address these gaps in the v2 to v3 migration, we have created a Python script to automate these post-configuration steps. You can run this Python script as-is, or you can further modify it to meet specific use cases. The end result of running the Python script is to have a v3 environment that is as similar to v2 as possible. The goal is that when you cutover from v2 to v3, you want to have minimal disruption to existing business processes.

Problem statement

Cloud Connections are not imported

On SFTP Gateway v2, you may have configured SFTP users to point to a custom container and upload path.

v2 configuration

For example, in this screenshot the SFTP user robtest points to the default container named rob-v2 (which is based on the VM name). The upload path defaults to /robtest/uploads/.

But let's say you have an SFTP user brycetest that points to a custom container and upload path. In this case, it's brycecontainer/subfolderA/subfolderB/.

If you go through the import process on v3, you will see the expected configuration for the robtest user:

robtest on v3

If we compare the paths between v2 and v3, they are quite similar:

robtest:

v2: robertsa/rob-v2/robtest/uploads/
v3: robertsa/rob-v2/users/robtest/uploads/

Note: The additional users folder in the v3 path can be easily bypassed by mapping the users Folder to the default Cloud Connection.

But if we look at the imported brycetest user, you will see an unexpected result:

brycetest on v3

If we compare the paths between v2 and v3, they are quite different:

brycetest:

v2: robertsa/brycecontainer/subfolderA/subfolderB/
v3: robertsa/rob-v2/users/brycetest/uploads/

The problem is that the v3 import process on our Azure product is not creating a cloud connection for brycecontainer/subfolderA/subfolderB/. So brycetest ends up pointing to the default cloud connection on v3.

So what is needed is a way to automatically create a new Cloud Connection for each v2 user that is configured with a custom container and upload path. Also, the imported SFTP user should have their uploads pointing to this Cloud Connection.

Uploads directory should be write-only

Another gap when migrating from v2 to v3 is the permissions on the uploads folder.

On v2, existing SFTP users are accustomed to having an uploads directory. When they drop a file, the file is swept away as it gets moved to the corresponding Blob storage location for this user.

On v3, SFTP users have read-write access to everything in their chroot directory, including the uploads folder. These new freedoms may not be desired behavior. For example, SFTP users may be startled to see all their previously uploaded files and assume something is wrong. Or, SFTP users may start deleting files. Since they are performing these actions directly against Blob storage, you could end up in a data loss situation.

So what is needed on v3 is a write-only permission applied to the uploads directory. This requires creating the uploads folder under the user's chroot directory, and then overriding the permissions to write-only.

Chroot directory should be read-only

Another gap when migrating from v2 to v3 is the permissions on the chroot directory.

On v2, OpenSSH had a limitation where the chroot directory could not be writable by anyone other than the root user (for security reasons). As a result, we had to create uploads, downloads, and local folders on users' behalf.

On v3, SFTP users have read/write access to their chroot directory, as well as everything underneath. While this access is a welcome change, the behavior does differ from v2. So you may want to perpetuate the v2 behavior as a form of "baby gates" to prevent users from accidentally deleting their entire uploads folder.

So what is needed on v3 is to override the default read-write permissions on the chroot directory to read-only.

What does the Python script do?

The Python script in this article performs post-configuration fixes to address each of the problems mentioned above.

So, you would import your yaml backup from v2 into a fresh v3 server. Then, you would run the Python script. The end result is that your v3 server will be configured to behave much like the v2 server.

This is what the Python script does:

  • Reads the yaml backup file, and then iterates over every SFTP user on v3
  • Locks down the chroot directory to read-only permissions
  • Creates an uploads folder on the v3 Folders tab
  • Locks down the uploads folder to write-only permissions
  • If there's a custom container and upload path for the v2 user, create a Cloud Connection on v3, and then map the uploads folder to it

Running the Python script

In a web browser, navigate to our source code repository for the Python script: https://bitbucket.org/thorntechnologies/sftp-gateway-python-scripts/src/main/

In the top right, click the Clone button for instructions on how to clone the repository using git.

Alternatively, you can click on the three dots to the right of the Clone button, and choose Download repository.

Once you have downloaded the repository to your local workstation, edit the config.yml file:

yaml_file_path: v2-backup.yml
client_id: ABCDEFGHIJKLMOP
client_secret: 0123456789ABCDEFG
admin_username: admin
admin_password:
api_url: https://ip-address/backend

The Python script will read the contents of this config.yml file, so make sure the information is updated.

  • yaml_file_path: Points to a sample v2-backup.yml file. Point this to your actual yml backup from v2.
  • client_id: The security.client.id property from /opt/sftpgw/application.properties. You will need to SSH into the v3 VM to get this information.
  • client_secret: The security.client.secret property from application.properties
  • admin_username: The web admin username you created when first setting up v3
  • admin_password: The web admin password
  • api_url: The URL to the API backend. Replace ip-address with the public IP of your VM

This is how to run the Python script:

python3 v2-v3-post-configuration.py

Your output should look something like this:

python3 v2-v3-post-configuration.py

Processing user: brycetest
User brycetest 'uploads' folder is mapped to cloud connection brycetest-subfolderA-subfolderB

Processing user: robtest
User robtest missing 'container' or 'uploadPath', skipping...

Summary Report:
- Created Cloud Connections: ['brycetest-subfolderA-subfolderB']
- Users Skipped Due to User Not Found in SFTPGW: []

Done!

To test whether everything is configured correctly, try logging in as an SFTP user, and note the behavior when you cd to the uploads folder and try dropping a file. The behavior should be similar to v2, and the file should end up in the same location as it did with v2.

Running the test script

The repository also includes a test script named test_configuration.py. This is how to run it:

python3 test_configuration.py

You should see the following output:

$ python3 test_configuration.py

-------------
username::  brycetest
Passed: chroot directory is read-only
Passed: uploads directory is write-only
Passed: uploads resolved path on v2 matches v3
-------------
username::  robtest
Passed: chroot directory is read-only
Passed: uploads directory is write-only
**************************************************
Failed: uploads resolved paths do not match
v2 resolved uploads path: https://sftpgatewaytest.blob.core.windows.net/rob-v2/robtest/uploads
v3 resolved uploads path: https://sftpgatewaytest.blob.core.windows.net/rob-v2/users/robtest/uploads
**************************************************

The Python test script can be used to test whether your post-configuration is correct for every SFTP user. While you can manually spot-check whether settings are correct for one or two users, you ideally want to know for sure that every setting on every SFTP user is correct. This is to help make sure the transition to v3 is as smooth as possible.

These are the tests performed by the script:

  • The chroot directory should be read-only
  • The uploads directory should exist, and be write-only
  • The uploads directory on v3 should point to the same location it pointed to on v2

Feel free to modify or extend the test_configuration.py script to cover your specific use case.

← Deploy a v3 serverNext →