Azure Troubleshooting Stuck Files

Sometimes there are files stuck in a user's uploads folder. There are a few things you can do to troubleshoot this.

Check the movetos3 log for activity

First, you can check to see if any files are currently being uploaded to Blob:

tail -f /var/log/sftpgw/movetos3.log

Hit Enter a few times, and see if there is any new activity.

Touching files to trigger an upload

If there is no new activity in the movetos3.log, you can touch one of the files in the user's uploads folder.

cd /home/user/home/user/uploads/
touch existing-file.txt

Make sure that the existing file is not currently being uploaded via SFTP, as this would result in certain file corruption. A good way to check for this is to look at the timestamp of the file. (Note: running the touch command will update this timestamp.)

Check the incron service

If files are still not uploading, it's possible that the incron service is down. You can check this by running the command:

systemctl status incrond

If the service is down, you can restart the service:

systemctl restart incrond

Count the number of processes

It's possible that the server is struggling with a large backlog of files. To check for this, run the command:

ps -ef | grep movetos3 | wc -l

This will count the number of processes containing the command movetos3. To give a little background, when a file is ready to be uploaded to S3, a movetos3 command is queued in the ts spooler.

If there are a lot of processes and the server is locked up, you can reboot the VM to clear these processes.

Uploading a backlog of stuck files

If there are a lot of stuck files in a user's uploads directory, you can touch all the files to trigger them to upload:

cd /home/user/home/user/uploads/
find . -mtime +1 -type f -exec touch {} \;

As mentioned earlier, touching a file in the middle of SFTP transfer will result in file corruption. So the above command will only touch files that are older than a day.

Again, touching a file will update the timestamp, so you will not be able to use this approach again until the next day.

Files stuck in a specific folder

Incron works by watching for file events in a specific folder. If there is a tree of subfolders, there needs to be an entry in the user's incrontab for each folder. If a subfolder is missing, files in that subfolder will not upload, even after using the touch command.

To check for this, list the stuck files:

cd /home/robtest/home/robtest/uploads/
find . -type f

If the stuck files tend to be clustered within a few subfolders, you might want to check the user's incrontab:

/etc/incron.d/robtest.sftpgw.incron

To update the user's incrontab file, run the following command:

sudo usersetup robtest