Setting up Log streaming to CloudWatch

Overview

You can deploy SFTP Gateway as a stand-alone EC2 instance, as an alternative to CloudFormation. This approach will deploy most of the same resources which are created via CloudFormation, with a few important exceptions such as the IAM role, and log streaming to CloudWatch.

This article covers setting up log streaming to CloudWatch, as this isn't setup by default when deploying via EC2 versus the CloudFormation deployment.

SFTP Gateway Permissions

Since we're working with CloudWatch, make sure you have the necessary permissions on the IAM role for your SFTP Gateway deployment.

An IAM policy we recommend is CloudWatchAgentServerPolicy, which has the following permissions:

{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Sid" : "CWACloudWatchServerPermissions",
      "Effect" : "Allow",
      "Action" : [
        "cloudwatch:PutMetricData",
        "ec2:DescribeVolumes",
        "ec2:DescribeTags",
        "logs:PutLogEvents",
        "logs:PutRetentionPolicy",
        "logs:DescribeLogStreams",
        "logs:DescribeLogGroups",
        "logs:CreateLogStream",
        "logs:CreateLogGroup",
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords",
        "xray:GetSamplingRules",
        "xray:GetSamplingTargets",
        "xray:GetSamplingStatisticSummaries"
      ],
      "Resource" : "*"
    },
    {
      "Sid" : "CWASSMServerPermissions",
      "Effect" : "Allow",
      "Action" : [
        "ssm:GetParameter"
      ],
      "Resource" : "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
    }
  ]
}

Once you've granted your instance IAM permissions to CloudWatch, we can configure the CloudWatch agent on the server to stream the SFTP Gateway logs.

Access your SFTP Gateway deployment

To configure the CloudWatch logging on the instance, we will need SSH access. For more info about how to SSH into SFTP Gateway check out this article.

SSH into your SFTP Gateway instance on the default SSH port of port 2222:

ssh -i private.key ec2-user@<ip-address> -p 2222

Once inside your instance, navigate to the CloudWatch agent directory:

cd /opt/aws/amazon-cloudwatch-agent/

Setting up CloudWatch

Since we haven't set anything up yet, we need to create a new configuration file, which we can do with this command:

sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Paste this content into the config file:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/sftpgw/cloudinit.log",
            "log_group_name": "sftpgw-{instance_id}",
            "log_stream_name": "cloudinit-{instance_id}",
            "timestamp_format": "%b %d %H:%M:%S"
          },
          {
            "file_path": "/opt/sftpgw/log/application*.log",
            "log_group_name": "sftpgw-{instance_id}",
            "log_stream_name": "application-{instance_id}",
            "timestamp_format": "%b %d %H:%M:%S"
          },
          {
            "file_path": "/opt/sftpgw/log/sftp-audit*.log",
            "log_group_name": "sftpgw-{instance_id}",
            "log_stream_name": "audit-{instance_id}",
            "timestamp_format": "%b %d %H:%M:%S"
          }
        ]
      }
    }
  }
}

Next, we're going to stop the CloudWatch agent, apply our configuration file, and start the agent so our changes take effect:

sudo systemctl stop amazon-cloudwatch-agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
 
sudo systemctl start amazon-cloudwatch-agent

Now, you should be successfully streaming your server application & audit logs to CloudWatch.

You can confirm this by tailing the CloudWatch agent logs:

sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log