How to transfer files between Multiple Cloud Connections

Overview

This article covers how to move files between AWS S3, Azure Blob Storage and Google Cloud Storage, via SFTP.

To achieve this, you will need an SFTP Gateway v3 server (which can be on either AWS or Azure). Then, using Cloud Connections, you point SFTP Gateway to:

1. An S3 bucket
2. A Blob Storage Container
3. A Google Cloud Storage bucket

Once this is done, you mount a Folder to each of these Cloud Connections. And then you point the SFTP user's Home Directory to a Folder that is a parent to the three previously mentioned Folder mappings.

When the SFTP user logs in, they will see their Folder mappings. And then you can simply drag a file from S3 to Blob to GCS, or whichever order you want.

Before you begin

You will need SFTP Gateway v3 deployed in your Azure subscription. You can accomplish this from AWS, but this article uses an Azure VM.

Next, make sure that your Default Cloud Connection is configured properly. Here's an article with instructions on how to do this.

Note: It doesn't matter where Default Cloud Connection points to. It just needs to work, so the SFTP user has a file system to work with.

Create a Cloud Connection to AWS S3

First, create a Cloud Connection to AWS S3. Refer to this article in our AWS documentation for instructions on how to do this.

In this example, the Cloud Connection has the following properties:

• Connection Name: AWS Cloud Connection
• S3 URL: s3://blob-to-s3-example
• Region: us-east-1

Create a Cloud Connection to Azure Blob Storage

Next, create a Cloud Connection that points to Azure Blob Storage. Refer to this article for instructions.

In this example, the Cloud Connection has the following properties:

• Connection Name: Azure Cloud Connection
• Storage Account: bryce
• Container: blob-to-s3-example
• Unique credentials: This is the Connection String of the Storage Account access key

Create a Cloud Connection to Google Cloud Storage

Finally, create a Cloud Connection that points to Google Cloud Storage. Refer to this article for instructions on how to set up a Google Cloud Connection.

In this example, the Cloud Connection has the following properties:

• Connection Name: GCP Cloud Connection
• Container Resource URL: gs://winterwonderland
• Cloud Connection Credentials: This is a Service Account JSON Key File

Map Folders to the Cloud Connections

In order to make use of Cloud Connections, you will need to map Folders to them.

Click on the Folders tab. The default Folder structure looks like this:

/
|--Users/  <-- SFTP home directories are created in here by default when creating a user

Create a new folder named multi-cloud, and set it to inherited. Your folder structure should now look like this:

/
|--multi-cloud/
|--Users/  

Under the /multi-cloud Folder, create a new Folder named aws-folder that points to the AWS Cloud Connection.

Likewise, create a new Folder named azure-folder that points to the Azure Cloud Connection.

Finally, create a new Folder named google-folder that points to the Google Cloud Connection.

The structure of the folder system should look like this:

/
|--multi-cloud/
   |--aws-folder/    <-- maps to "AWS Cloud Connection"
   |--azure-folder/  <-- maps to "Azure Cloud Connection"
   |--google-folder/ <-- maps to "Google Cloud Connection"
|--Users/

Set the SFTP user's Home Directory

Next, create a new SFTP user.

Under Home Directory, choose Select home directory.

Select the /multi-cloud Folder.

The SFTP user's Home Directory points to the /multi-cloud Folder, which is the parent of the three other Folders pointing to the AWS, Azure and Google Cloud Connections.

This means that this SFTP user can access all three clouds.

Transferring files

Finally, log in as the SFTP user via FileZilla.

You should see a chroot directory "/".

Underneath that, you should see a subfolder for each Cloud Connection.

To transfer a file, simply drag and drop it from one folder to the other.