SFTP Gateway Support

3.0

Alibaba Diagnostic Page

Overview

The high level steps to configure the Diagnostic page are as follows:

  1. Create a Logging project and LogStore
  2. Create a Machine Group
  3. Create Logtail Configurations for Application & Audit Logs
  4. Create RAM user for Log Service permissions
  5. Fill out Cloud Log Connection Info in SFTP Gateway.

Configure Alibaba Logging

Create Project and Logstore

In the Alibaba console, navigate to the Log Service, which is under the Storage Data Services.

Once in the Log Service, under Import Data, select RegEx - Text Log, also known as Regular Expression - Text Log.

Create a new Project and make sure it is in the same region as your SFTP Gateway instance.

Additionally, you'll also need to create a new Logstore for your project.

Once you've configured the Project & Logstore, click Next.

Create Machine Group

Under the Machine Group Settings, select Create Machine Group.

Check the box next to your SFTP Gateway instance and scroll until you see the IP Address values.

Copy the Private IP address value for your instance and under the Configure Machine Group, paste this value into the IP Addresses field.

After checking the box next to your instance and configuring the machine group, click OK.

After adding your instance to the Machine Group, verify that the Heartbeat is OK.

Note: It may take a couple of minutes for the result to reach OK.

Once you've confirmed the Heartbeat is working, click Next.

Create Logtail Config

Under the Logtail config, set the Log Path to use these values:

  1. /opt/sftpgw/log
  2. application-*.log

Turn Singleline off and paste in this value for the Log Sample:

2023-12-20T03:57:04.579+08:00  INFO 881 --- [httpclient-dispatch-1] darabonba.core.TeaModel                  : [com.aliyun.sdk.service.oss20190517.models.ListObjectsV2ResponseBody.isTruncated] There are some cast events happening. expect: java.lang.Boolean, but: java.lang.String, value: false.
2023-12-20T03:57:08.457+08:00 ERROR 881 --- [http-nio-8080-exec-2] .b.s.c.AlibabaCloudConnectionServiceImpl : Bucket failed to create sftpgw-shanghai

java.util.concurrent.ExecutionException: com.aliyun.sdk.gateway.oss.exception.OSSServerException: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. (Status Code: 409, Code: BucketAlreadyExists, Request ID: 6581F594B3A78A3133544E13)
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at com.sftpgateway.backend.service.cloudconnection.AlibabaCloudConnectionServiceImpl.createStorageLocation(AlibabaCloudConnectionServiceImpl.java:82)

For the Regex to Match First Line, set this to Manual and use this Regex value:

^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}(?:Z|[+-]\d{2}:\d{2}))\s*(\w+)\s*(\d*)\s*---\s*\[\s*([a-zA-Z0-9\._-]+)?\]\s*([^\s]+)\s*:\s*([^\n]+)$

Move the slider to enable Extract Field and set the RegEx as the same value you used previously, then click Validate. Next, input these values for the Extracted Content Key field:

  1. timestamp

  2. log_level

  3. pid

  4. thread

  5. source

  6. message

On the Configure Query and Analysis section, click Next at the bottom of the page.

You have now completed the steps to enable logging for the Application logs on the intance.

The next step would be to create a new Logstore for the Audit logs. So, navigate back to your logging project and under the Logstores section, click the + icon. This will open a new popup window to your right where you can configure your logstore.

In my example, I am just specifying the name of the logstore and clicking OK to create it.

After creating your logstore, you will be prompted to use the Data Import Wizard.

Click on the button to navigate to the Quick Data Import page and click Integrate Now for RegEx - Text Log.

For the RegEx - Text Log configuration, select the same Machine Group you used previously.

Under the Logtail config, set the Log Path to use these values:

  1. /opt/sftpgw/log
  2. sftp-audit-*.log

Follow the same steps you did on the previous Logtail config, except the Log Sample will be different.

Turn Singleline off and paste in this value for the Log Sample:

2023-12-20T04:27:08.293+08:00  WARN 881 --- [pool-7-thread-1] sftp-audit                               : Cannot get username from security context for auditing
2023-12-20T04:27:08.520+08:00  INFO 881 --- [pool-6-thread-1] sftp-audit                               : USERAUTH_STARTED   REMOTE_ADDRESS: 64.98.240.105   USERNAME: dan   AUTH_METHOD: publickey
2023-12-20T04:27:08.524+08:00  INFO 881 --- [pool-7-thread-1] sftp-audit                               : USERAUTH_SUCCESS   REMOTE_ADDRESS: 64.98.240.105   USERNAME: dan   AUTH_METHOD: publickey
2023-12-20T04:27:08.524+08:00  INFO 881 --- [pool-7-thread-1] sftp-audit                               : AUTHENTICATION_COMPLETE    REMOTE_ADDRESS: 64.98.240.105   USERNAME: dan   AUTH_METHODS: [publickey]
2023-12-20T04:27:09.009+08:00  INFO 881 --- [pool-7-thread-1] sftp-audit                               : SFTP_SESSION_STARTED   REMOTE_ADDRESS: 64.98.240.105   USERNAME: dan
2023-12-20T04:27:10.893+08:00  INFO 881 --- [pool-6-thread-1] sftp-audit                               : SFTP_DIR   REMOTE_ADDRESS: 64.98.240.105   USERNAME: dan   FILE_NAME: /    BYTES_TRANSFERRED: 479

For the Regex to Match First Line, set this to Manual and use this Regex value:

^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}(?:Z|[+-]\d{2}:\d{2}))\s*(\w+)\s*(\d*)\s*---\s*\[\s*([a-zA-Z0-9\._-]+)?\]\s*([^\s]+)\s*:\s*([^\n]+)$

Move the slider to enable Extract Field and set the RegEx as the same value you used previously, then click Validate. Next, input these values for the Extracted Content Key field:

  1. timestamp

  2. log_level

  3. pid

  4. thread

  5. source

  6. message

On the Configure Query and Analysis section, click Next at the bottom of the page.

You have now completed the steps to enable logging for the Audit logs on the intance.

Access Key & Secret

The final step is to retrieve the values needed for the Cloud Log Connection Info in SFTP Gateway.

You should be able to retrieve the Region, Project & Logstore values from the Log Service. However, to retrieve the Access Key Id & Access Secret, you will need to go to the Resource Access Management service.

Once inside the RAM service, navigate to the Users tab on the left side-bar menu and click Create User.

Configure a name (log-service for example) and give this user only OpenAPI Access, then click OK.

Once your user has created, make sure to copy AccessKey ID & AccessKey Secret values, otherwise you will need to create a new Access Key & Secret pair later.

Edit your user and under the Permissions tab, give this user the AliyunLogFullAccess policy.

Finally, paste the AccessKey ID & AccessKey Secret values you copied previously into the Cloud Log Connection Info in SFTP Gateway.

← Previous