Liquibase Lock Error

Overview

Although this is rare, you may encounter an issue where the web admin portal is stuck on the loading screen. And if you examine the application logs, you may see the following error:

2022-04-26 15:47:24.284  INFO 267325 --- [main] liquibase.lockservice : Waiting for changelog lock....

This is the result of a Liquibase lock preventing the Java service from starting properly.

To resolve the issue, you will need to connect to the database and manually delete the lock.

Background

In version 3.3.0, we introduced an HA feature. Along with HA, we had to also introduce a data base locking mechanism (Liquibase) to prevent multiple instances from writing conflicting changes to the database.

If the Java service halts while it is in the process of writing to the database, it will not have a chance to clear its database lock. As a result, the database lock remains indefinitely, preventing Java from initializing.

This is what the error looks like, when inspecting the log file /opt/sftpgw/log/application.log:

2022-04-26 15:47:24.284  INFO 267325 --- [main] liquibase.lockservice                    : Waiting for changelog lock....

Resolving the Error

To fix this issue, you will need to connect to the Postgresql database, and manually delete the lock.

SSH into your VM, and run this command to connect to the Postgresql database:

sudo -i -u postgres psql

You may encounter this error message when connecting to the database:

This account is currently not available.

To resolve the issue run this command:

sudo chsh -s /bin/bash postgres

Now, rerun the command to connect to the Postgresql database.

Once connected to Postgres, run the following database commands:

\c sftpgw;
DROP TABLE DATABASECHANGELOGLOCK;

This connects to the sftpgw database.

Then, it drops the entire DATABASECHANGELOGLOCK table.

Hit Ctrl-D a few times to disconnect from the database and log out.

For security purposes, we recommend locking the database after you have exited:

sudo chsh -s /sbin/nologin postgres

At this point, you should be able to restart Java with this command:

sudo su
service sftpgw-admin-api restart

Run netstat -nltp to see if Java is listening on port 22. (Or run the command ss -aunp if you're on Ubuntu.)