OS Patching

TLDR

What: A Thorn Technologies-maintained script that applies operating system security updates to your StorageLink VM safely

Why: A running service keeps using the old copy of every library an update replaces, and a new kernel only takes effect after a reboot. Ubuntu (Azure, Google Cloud) installs security updates but never restarts StorageLink, and Amazon Linux 2023 (AWS) installs none on its own. The script installs the updates, restarts what they replaced, and reboots when needed.

Before You Run It: Export a backup file (Settings → Backup & Recovery → Export Backup File) and snapshot the disk

Run: curl -sSO https://thorntech-products.s3.amazonaws.com/storagelink/os-patching-script/thorntech-os-patch.sh, then sudo bash ./thorntech-os-patch.sh

Reboot: Automatic when the updates require it, after a two-minute warning you can cancel; add --no-reboot to report instead

Schedule: Add a monthly root cron entry (on AWS, install cronie first)

Product: StorageLink by Thorn Technologies — cloud storage gateway for secure file sharing

Overview

Keeping the operating system patched is part of running StorageLink securely, and installing updates is only half of it. A running process keeps using the old copy of every executable and library that an update replaces, so a patched Java runtime or OpenSSL does not protect StorageLink until StorageLink restarts, and a new kernel does not take effect until the server reboots.

This matters on every cloud:

• Azure and Google Cloud (Ubuntu). Current StorageLink images keep Ubuntu's unattended-upgrades enabled, so security updates are installed automatically. Unattended-upgrades does not restart StorageLink or reboot the server, so the Java runtime and libraries StorageLink is actually running, and the kernel, stay at the versions that were in place when StorageLink last started — even though a package scan of the disk reports them as patched.
• AWS (Amazon Linux 2023). Amazon Linux 2023 does not install updates on its own. Each instance is also locked to the repository version it launched with, so a plain dnf upgrade never finds newer security updates.

The ThornTech OS Patching script handles both: it installs the pending updates, restarts the StorageLink services that are still running on replaced files, and reboots the server when the updates require it.

Which operating system your VM runs depends on the cloud and the StorageLink version you deployed:

Cloud	Operating system	StorageLink versions	Standard support ends
AWS	Amazon Linux 2023	1.1.1 and later	June 2029
Azure, Google Cloud	Ubuntu 24.04	1.1.5 and later	April 2029

AWS images before 1.1.1 run Amazon Linux 2, and Azure and Google Cloud images before 1.1.5 run an earlier Ubuntu LTS release. The script supports Ubuntu 20.04, 22.04 and 24.04 and Amazon Linux 2023; it refuses to run on anything else, including Amazon Linux 2. An in-place upgrade of StorageLink does not change the operating system; moving to a newer operating system means deploying the current StorageLink image and importing your backup file, as described in the Upgrade Process.

The script:

• Checks the appliance is healthy before patching — the StorageLink application and its health endpoint, the nginx web server, the local database, and admin SSH — and refuses to run if it is not.
• Preserves your local configuration files, including a customized SSH port, instead of overwriting edited configs with package defaults. If an update changes the effective SSH port anyway, the run fails loudly before the change can lock you out at the next reboot.
• Restarts what the update replaced. Any StorageLink service still running on replaced files is restarted in dependency order — the database, then StorageLink, then nginx — so the patched code is what actually runs. Services that did not use a replaced file are left alone.
• Verifies the appliance is still healthy afterwards and reports loudly if anything is wrong.
• Reports exactly what changed: package counts and the services it restarted in the run summary, and a full list of upgraded, installed, and removed packages in the log.
• Finishes the job. If the updates require a reboot, it says so, waits two minutes so you can cancel with sudo shutdown -c, then reboots. StorageLink starts on its own afterwards. Add --no-reboot if reboots must go through your own change window.

Before you begin

Step 1: Export a backup file. In the web admin interface, open Settings, scroll to Backup & Recovery, and click Export Backup File. Keep the downloaded YAML file somewhere other than this VM. If patching leaves the server unrecoverable, that file restores your users, folders, cloud connections, identity providers, and settings onto a fresh deployment through Settings → Backup & Recovery → Import (the full procedure is in Upgrade Process for StorageLink).

Step 2: Snapshot the disk. A snapshot of the VM's disk is the one-step rollback if the operating system itself is left unbootable.

AWS

In the EC2 console, open the instance, go to the Storage tab, and click the Volume ID of the root device. Select the volume, then choose Actions → Create snapshot.

Azure

In the Azure portal, open the VM, go to Disks, select the OS disk, and click Create snapshot.

Google

In the Google Cloud console, go to Compute Engine → Snapshots, click Create snapshot, and choose the VM's boot disk as the Source disk.

Step 3: Connect over SSH. Admin SSH is on port 22 unless you changed it. See SSH into StorageLink for the full steps on each cloud.

AWS

ssh -i private.key ec2-user@<your-vm-ip>

Azure

ssh -i private.key azureuser@<your-vm-ip>

Google

ssh -i private.key ubuntu@<your-vm-ip>

Download and run

Run it during a maintenance window

When an update replaces a library StorageLink uses — the Java runtime, OpenSSL, or the C library, for example — the script restarts StorageLink, and it reboots the server if the updates require it. Either way, web sessions and transfers in progress drop, and StorageLink is unavailable for a few minutes.

The appliance mounts /tmp as no-exec, so download the script to your home directory and run it with sudo bash (rather than executing the file directly):

cd ~
curl -sSO https://thorntech-products.s3.amazonaws.com/storagelink/os-patching-script/thorntech-os-patch.sh
sudo bash ./thorntech-os-patch.sh

The script logs everything to /var/log/thorntech-os-patch.log and to the system log.

Preview without changing anything

To see the pending updates and confirm the appliance is healthy without patching, use --check. It changes nothing, so if an earlier update left a reboot pending, it only reports that:

sudo bash ./thorntech-os-patch.sh --check

Scheduling monthly patching

Expect a short interruption on every scheduled run

A run that updates a library StorageLink uses restarts StorageLink, and a kernel update reboots the server. Either way, web sessions and transfers in progress drop and StorageLink is unavailable for a few minutes. Pick a cron time inside a maintenance window when users are idle, give each instance of a high-availability deployment its own time so they never restart together, and use --no-reboot if reboots must be approved separately.

A scheduled run has nobody to export a backup first. Export a backup file whenever you change the configuration, and let your cloud take regular disk snapshots ahead of the patch window — for example with Amazon Data Lifecycle Manager on AWS, Azure Backup on Azure, or a snapshot schedule on Google Cloud.

Install the script once and add a root cron entry. This example patches at 02:00 on the second Tuesday of each month.

AWS

Amazon Linux 2023 does not include cron, so install it first:

sudo dnf install -y cronie
sudo systemctl enable --now crond
sudo cp thorntech-os-patch.sh /usr/local/bin/
sudo crontab -e

Azure

sudo cp thorntech-os-patch.sh /usr/local/bin/
sudo crontab -e

Google

sudo cp thorntech-os-patch.sh /usr/local/bin/
sudo crontab -e

Add the line:

0 2 8-14 * * [ "$(date +\%u)" = "2" ] && bash /usr/local/bin/thorntech-os-patch.sh

A scheduled run reboots the server on its own when the updates require it, after the warning delay (two minutes by default; change it with --reboot-delay=MIN). If your change process needs reboots approved separately, add --no-reboot to the cron line and the run will report the pending reboot instead.

High-availability deployments

Run the script on every instance, at a different time on each, so the load balancer always has a healthy instance to send users to. Instances that an Auto Scaling group, scale set, or managed instance group launches later start from the original image, unpatched, so patch them once they are in service.

What the script reports

Every run ends with one of these outcomes (also reflected in the exit code):

Result	Meaning
PATCH SUCCESS	Updates applied and the appliance verified healthy afterwards. The summary shows how many packages were upgraded, installed, and removed, and which services were restarted; the full package list is in the PACKAGE CHANGES section of the log.
PATCH REBOOT SCHEDULED	Patching succeeded and a reboot is needed to finish (usually a kernel update). The server reboots after the warning delay, during which new SSH logins are refused by the system; cancel with sudo shutdown -c if you must. StorageLink starts on its own after the reboot.
PATCH ACTION REQUIRED	Same situation, but the run was started with --no-reboot. Reboot in your change window with sudo reboot, or sudo shutdown -r +15 to give logged-in users notice.
PRE health failed	The appliance was already unhealthy, so nothing was patched. The log lists each failed check.
POST health failed	Patching completed but a post-patch check failed — investigate before rebooting.

The summary's Service line records which StorageLink services the run restarted because they were still running on files the update replaced — for example restarted postgresql@18-main.service, swiftgateway, nginx. When nothing StorageLink runs was affected, it reads no restart needed.

On Ubuntu, the summary's Sources line lists any package repository the run had to skip. That happens when a third-party source stops publishing an index — for example apt.postgresql.org, which provides PostgreSQL on the StorageLink Ubuntu images, once an Ubuntu release reaches end of life. The Ubuntu sources still refresh and the run continues without that repository. The run fails only if an Ubuntu archive itself cannot be refreshed.

note

On Amazon Linux 2023, each run moves the instance to the newest Amazon Linux 2023 repository version (dnf upgrade --releasever=latest), which is how Amazon delivers security updates for it. Amazon Linux never replaces a configuration file you edited; it saves the package's version alongside it as a .rpmnew file instead.

Need help?

If a patch run reports a health failure, or you would like help scheduling patching across a fleet, contact our support team at support@thorntech.com. Include the revision shown at the top of the run output and the end of /var/log/thorntech-os-patch.log.

Optional: email notifications

No mail server is needed. Every run writes its result to /var/log/thorntech-os-patch.log and to syslog (tag thorntech-os-patch), which is what most monitoring hooks into. If you also want an email per run, install and configure a mail transfer agent on the appliance (for example mailutils with a configured relay; none is installed by default), then pass --mail-to:

sudo bash ./thorntech-os-patch.sh --mail-to=ops@example.com

Add the same flag to the cron entry above to receive the monthly result by email.

Script contents

The full script is reproduced below so you can review exactly what it does before running it on your appliance. This copy matches the script published on September 28, 2026 (revision 2026-09-28, shown in the run header). The download URL always serves the current version, so use the curl command in Download and run rather than copying from this page.

#!/bin/bash
#
# ============================================================================
#  ThornTech OS Patching Script — StorageLink edition
# ============================================================================
#
#  Applies operating system updates to a StorageLink appliance safely:
#  it verifies the appliance is healthy before touching anything, patches
#  with settings that preserve your local configuration files (a customized
#  SSH port included), restarts any StorageLink service still running on
#  files the updates replaced, verifies the appliance is still healthy
#  afterwards, and reboots the server when the updates require it (after a
#  warning delay; see --no-reboot).
#
#  Supported: Amazon Linux 2023 (StorageLink on AWS) and Ubuntu
#  20.04/22.04/24.04 (StorageLink on Azure and Google Cloud).
#
#  USAGE (note: /tmp is mounted noexec on the appliance — download to your
#  home directory and invoke with 'sudo bash' rather than executing the file
#  directly):
#
#      cd ~
#      curl -sSO https://thorntech-products.s3.amazonaws.com/storagelink/os-patching-script/thorntech-os-patch.sh
#      sudo bash ./thorntech-os-patch.sh
#
#  Options:
#      --check              Report pending updates and appliance health only;
#                           changes nothing.
#      --no-reboot          Report a required reboot instead of performing it
#                           (default: reboot automatically when required, after
#                           the warning delay below). StorageLink starts on its
#                           own after a reboot.
#      --reboot-delay=MIN   Whole minutes between the warning and the automatic
#                           reboot (default 2; 0 reboots immediately). Cancel a
#                           pending reboot with: sudo shutdown -c
#      --mail-to=ADDR       Optional: also email each run's result to ADDR.
#                           Needs a working 'mail' command and MTA on the
#                           appliance (none is installed by default). Results
#                           always go to the log and syslog regardless.
#
#  Environment variables MAIL_TO, MAIL_FROM, AUTO_REBOOT (yes/no) and
#  REBOOT_DELAY_MINUTES provide defaults; command-line flags override them.
#
#  Scheduled use — install once, then add a root crontab entry
#  (sudo crontab -e). Example: 02:00 on the second Tuesday of each month:
#
#      sudo cp thorntech-os-patch.sh /usr/local/bin/
#      0 2 8-14 * * [ "$(date +\%u)" = "2" ] && bash /usr/local/bin/thorntech-os-patch.sh
#
#  A third-party package source that no longer resolves (for example
#  apt.postgresql.org after an Ubuntu release reaches end of life) is skipped
#  with a warning; the run fails only if an Ubuntu archive itself cannot be
#  refreshed. Skipped sources are listed in the run summary.
#
#  Every run appends to /var/log/thorntech-os-patch.log (rotated once past
#  5 MB) and writes result lines to syslog (tag: thorntech-os-patch). The log
#  includes a PACKAGE CHANGES section listing every package the run upgraded,
#  installed, or removed; the run summary shows the counts.
#
#  Exit codes:
#      0   success (the server may be about to reboot if updates required it)
#      1   another run is already in progress
#      2   unsupported operating system
#      3   pre-patch health check failed — nothing was changed
#      4   patching succeeded but the post-patch health check failed
#      5   a reboot was required but scheduling it failed — reboot manually
#      64  usage error (unknown or invalid option)
#      77  not running as root
#      100 the package manager reported an error
#
# ============================================================================

set -u
set -o pipefail

# Cron runs with a minimal PATH that lacks /usr/sbin (where sshd, nginx,
# ss and shutdown live). Interactive sudo masks this; cron does not.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

SCRIPT_REVISION="2026-09-28"   # date this copy was published; quote it when contacting support
PRODUCT_NAME="StorageLink"

# ----------------------------------------------------------------------------
# Appliance facts (StorageLink edition — do not change)
#
#   - The StorageLink application is the 'swiftgateway' Java service. It
#     listens on 127.0.0.1:8080 and reports its health, including its
#     database connection, at /actuator/health.
#   - nginx serves the web interface on ports 80 and 443.
#   - Single-instance deployments run PostgreSQL on the VM (port 5432).
#     High-availability deployments use an external database, so the local
#     database is checked only when a local PostgreSQL service is enabled.
#   - Admin SSH is OpenSSH on port 22 unless you changed it. Nothing else
#     competes for that port (unlike SFTP Gateway), so no port pin is needed.
#     Instead the run records the effective SSH port(s) before patching and
#     fails loudly if an update changed them.
#   - The StorageLink Ubuntu images do not ship needrestart, and Amazon Linux
#     has no equivalent, so the script restarts any StorageLink service that
#     is still running on files the update replaced.
# ----------------------------------------------------------------------------
APP_SERVICE=swiftgateway
APP_HEALTH_URL="http://127.0.0.1:8080/actuator/health"
WEB_PORT=443
DB_PORT=5432

# ----------------------------------------------------------------------------
# Configuration (environment variables provide defaults; flags override)
# ----------------------------------------------------------------------------
MAIL_TO="${MAIL_TO:-}"
MAIL_FROM="${MAIL_FROM:-thorntech-os-patch@$(hostname -f 2>/dev/null || hostname)}"
AUTO_REBOOT="${AUTO_REBOOT:-yes}"
REBOOT_DELAY_MINUTES="${REBOOT_DELAY_MINUTES:-2}"
CHECK_ONLY="no"

LOG_FILE="/var/log/thorntech-os-patch.log"
LOCK_FILE="/var/run/thorntech-os-patch.lock"

usage() {
  # Print the header block (from line 2 down to the first non-comment line).
  awk 'NR==1{next} !/^#/{exit} {sub(/^# ?/,""); print}' "$0"
}

for arg in "$@"; do
  case "$arg" in
    --check)            CHECK_ONLY="yes" ;;
    --mail-to=*)        MAIL_TO="${arg#*=}" ;;
    --no-reboot)        AUTO_REBOOT="no" ;;
    --auto-reboot)      AUTO_REBOOT="yes" ;;   # accepted for older cron entries; now the default
    --reboot-delay=*)   REBOOT_DELAY_MINUTES="${arg#*=}" ;;
    --help|-h)          usage; exit 0 ;;
    *)
      echo "Unknown option: $arg (try --help)" >&2
      exit 64
      ;;
  esac
done

case "$REBOOT_DELAY_MINUTES" in
  ''|*[!0-9]*)
    echo "--reboot-delay must be a whole number of minutes (got: '$REBOOT_DELAY_MINUTES')" >&2
    exit 64
    ;;
esac

if [ "$(id -u)" -ne 0 ]; then
  echo "This script must run as root: sudo bash $0" >&2
  exit 77
fi

HOSTNAME_SHORT="$(hostname)"
TIMESTAMP="$(date '+%Y-%m-%d %H:%M:%S %Z')"

# Rotate the log once, past 5 MB, so unattended monthly use never fills a disk.
if [ -f "$LOG_FILE" ] && [ "$(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)" -gt 5242880 ]; then
  mv -f "$LOG_FILE" "$LOG_FILE.1"
fi
touch "$LOG_FILE"
chmod 640 "$LOG_FILE"
chown root:root "$LOG_FILE"

# Package inventory snapshots (before/after) drive the PACKAGE CHANGES report.
PKG_BEFORE="$(mktemp)"
PKG_AFTER="$(mktemp)"
PKG_DIFF="$(mktemp)"
PKG_SUMMARY="not measured"
PKG_UPGRADED=0
PKG_INSTALLED=0
PKG_REMOVED=0
SERVICE_ACTION="not evaluated (nothing was patched)"
APT_SOURCE_WARNINGS=""
APT_REFRESHED="no"
SSH_PORTS_BEFORE=""

# Log everything while keeping output visible when run interactively; the
# cleanup trap waits for tee so the tail of the log is complete on exit.
exec > >(tee -a "$LOG_FILE") 2>&1
TEE_PID=$!
cleanup() {
  rm -f "$PKG_BEFORE" "$PKG_AFTER" "$PKG_DIFF"
  exec 1>&- 2>&-
  [ -n "${TEE_PID:-}" ] && wait "$TEE_PID" 2>/dev/null
}
trap cleanup EXIT

send_mail() {
  local subject="$1" body="$2"
  # Always leave a machine-findable trace, regardless of mail delivery.
  echo "[NOTIFY] $subject"
  logger -t thorntech-os-patch "$subject" 2>/dev/null || true
  if [ -n "$MAIL_TO" ] && command -v mail >/dev/null 2>&1; then
    if ! echo -e "$body" | mail -s "$subject" -r "$MAIL_FROM" "$MAIL_TO"; then
      echo "[NOTIFY] mail delivery to $MAIL_TO failed; see $LOG_FILE"
      logger -t thorntech-os-patch "mail delivery to $MAIL_TO failed" 2>/dev/null || true
    fi
  fi
}

print_header() {
  echo "========================================"
  echo "$1"
  echo "========================================"
}

# ----------------------------------------------------------------------------
# OS detection
# ----------------------------------------------------------------------------
OS_FAMILY=""
SSH_SERVICE=""
if [ -r /etc/os-release ]; then
  . /etc/os-release
  case "${ID:-}" in
    ubuntu)
      OS_FAMILY="ubuntu"
      SSH_SERVICE="ssh"
      ;;
    amzn)
      if [ "${VERSION_ID:-}" = "2023" ]; then
        OS_FAMILY="al2023"
        SSH_SERVICE="sshd"
      fi
      ;;
  esac
fi

# ----------------------------------------------------------------------------
# Package inventory — snapshot the installed set before and after patching so
# the run can report exactly what changed, independent of the package
# manager's own output format. One line per package: "<name>\t<version>".
# ----------------------------------------------------------------------------
snapshot_packages() {
  case "$OS_FAMILY" in
    ubuntu)
      dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${Version}\n' 2>/dev/null \
        | awk -F'\t' '$1 ~ /^ii/ {print $2 "\t" $3}' | sort > "$1"
      ;;
    al2023)
      rpm -qa --qf '%{NAME}.%{ARCH}\t%{EVR}\n' 2>/dev/null | sort > "$1"
      ;;
  esac
}

# Writes "<U|I|R>\t<name>\t<detail>" lines to $PKG_DIFF and sets the counters.
compute_package_changes() {
  awk -F'\t' '
    NR==FNR { b[$1] = (b[$1] == "" ? $2 : b[$1] "," $2); next }
            { a[$1] = (a[$1] == "" ? $2 : a[$1] "," $2) }
    END {
      for (n in a) if (!(n in b)) print "I\t" n "\t" a[n];
      for (n in b) if (!(n in a)) print "R\t" n "\t" b[n];
      for (n in a) if ((n in b) && a[n] != b[n]) print "U\t" n "\t" b[n] " -> " a[n];
    }' "$PKG_BEFORE" "$PKG_AFTER" | sort -k1,1 -k2,2 > "$PKG_DIFF"
  PKG_UPGRADED=$(grep -c '^U' "$PKG_DIFF" || true)
  PKG_INSTALLED=$(grep -c '^I' "$PKG_DIFF" || true)
  PKG_REMOVED=$(grep -c '^R' "$PKG_DIFF" || true)
  PKG_SUMMARY="$PKG_UPGRADED upgraded, $PKG_INSTALLED newly installed, $PKG_REMOVED removed"
}

print_package_changes() {
  print_header "PACKAGE CHANGES ($PKG_SUMMARY)"
  if [ ! -s "$PKG_DIFF" ]; then
    echo "No packages were changed."
    return
  fi
  local section name detail
  for section in U I R; do
    grep -q "^$section" "$PKG_DIFF" || continue
    case "$section" in
      U) echo "Upgraded:" ;;
      I) echo "Newly installed:" ;;
      R) echo "Removed:" ;;
    esac
    while IFS=$'\t' read -r _ name detail; do
      printf '  %s  %s\n' "$name" "$detail"
    done < <(grep "^$section" "$PKG_DIFF")
  done
}

# ----------------------------------------------------------------------------
# apt-get update wrapper. A dead third-party repository (common once an Ubuntu
# release passes end of life, e.g. apt.postgresql.org dropping its suite) makes
# apt-get update exit non-zero even though every Ubuntu index refreshed fine.
# Skip such sources with a warning; fail only if an Ubuntu archive itself could
# not be refreshed, or the failure cannot be attributed to a source.
# ----------------------------------------------------------------------------
apt_update() {
  local out rc failed ubuntu_failed
  out=$(mktemp)
  apt-get "$@" update >"$out" 2>&1; rc=$?
  cat "$out"
  if [ "$rc" -ne 0 ]; then
    failed=$(grep -E "^(E|W): .*(Release file|Failed to fetch|no longer|not signed|NO_PUBKEY)" "$out" | grep -oE "https?://[^ ']+( [^ ']+)?" | sed -E 's#/(InRelease|Release)(\.gpg)?$##' | sort -u)
    ubuntu_failed=$(printf '%s\n' "$failed" | grep -E 'ubuntu\.com' || true)
    if [ -z "$failed" ] || [ -n "$ubuntu_failed" ]; then
      rm -f "$out"; return "$rc"
    fi
    APT_SOURCE_WARNINGS=$(printf '%s\n' "$failed" | sed -E 's#^https?://##' | paste -sd';' | sed 's/;/; /g')
    echo "WARNING: skipping package source(s) that failed to refresh: $APT_SOURCE_WARNINGS"
    echo "         Updates from those sources are not applied this run; all Ubuntu sources refreshed normally."
  fi
  rm -f "$out"
  return 0
}

# ----------------------------------------------------------------------------
# Appliance health check — called before patching (to refuse to patch an
# already-broken box) and after (to catch breakage loudly).
#
# The POST phase retries: the StorageLink application takes a minute or two
# to start after a restart, so a single-shot probe right after patching can
# catch it mid-start.
# ----------------------------------------------------------------------------

# Effective sshd port(s), resolved from sshd_config and its drop-ins. Empty
# when sshd cannot evaluate its configuration.
sshd_ports() {
  # 'sshd -T' fatally exits ("Missing privilege separation directory") when
  # /run/sshd is absent, which is exactly the state an openssh-server upgrade
  # leaves behind until ssh.service next starts. Recreating it is the same
  # idempotent step the service itself performs, so do it before probing.
  [ -d /run/sshd ] || install -d -m 0755 /run/sshd 2>/dev/null || true
  sshd -T 2>/dev/null | awk '/^port /{print $2}' | sort -un | paste -sd,
}

# True when this VM runs its own PostgreSQL server (single-instance
# deployments). High-availability deployments use an external database.
local_db_expected() {
  systemctl list-unit-files --type=service --state=enabled --no-legend 'postgresql*' 2>/dev/null | grep -q .
}

appliance_problems() {
  local problems="" health ports port

  # StorageLink application and its health endpoint. The endpoint reports
  # DOWN when the application cannot reach its database.
  systemctl is-active --quiet "$APP_SERVICE" || problems+="- $APP_SERVICE (the StorageLink application) is not active\n"
  health=$(curl -s --max-time 5 "$APP_HEALTH_URL" 2>/dev/null)
  printf '%s' "$health" | grep -q '"status":"UP"' || problems+="- StorageLink is not reporting healthy at $APP_HEALTH_URL (response: ${health:-none})\n"

  # Web front end.
  systemctl is-active --quiet nginx || problems+="- nginx is not active\n"
  nginx -t >/dev/null 2>&1 || problems+="- nginx config fails validation (nginx -t)\n"
  ss -tln 2>/dev/null | grep -q ":$WEB_PORT " || problems+="- nothing is listening on port $WEB_PORT (StorageLink web interface down)\n"

  # Local database (single-instance deployments only).
  if local_db_expected; then
    ss -tln 2>/dev/null | grep -q ":$DB_PORT " || problems+="- the local PostgreSQL database is not listening on port $DB_PORT\n"
  fi

  # Admin SSH. No service-state check: Ubuntu 24.04 socket-activates sshd, so
  # ssh.service is legitimately inactive until the first connection after a
  # reboot. A listener on every configured port is the real signal, and
  # 'sshd -T' failing means the next restart would leave admins locked out.
  ports=$(sshd_ports)
  if [ -z "$ports" ]; then
    problems+="- sshd configuration fails validation (sshd -T); admin SSH would not start after a restart or reboot\n"
  else
    for port in ${ports//,/ }; do
      ss -tln 2>/dev/null | grep -q ":$port " || problems+="- nothing is listening on SSH port $port (admin SSH down)\n"
    done
    if [ -n "$SSH_PORTS_BEFORE" ] && [ "$ports" != "$SSH_PORTS_BEFORE" ]; then
      problems+="- the SSH port changed during patching (was $SSH_PORTS_BEFORE, now $ports); after the next restart or reboot, admin SSH will only answer on the new port\n"
    fi
  fi

  printf '%s' "$problems"
}

check_appliance_health() {
  local phase="$1" attempts="$2" delay="$3" problems="" i
  for (( i=1; i<=attempts; i++ )); do
    problems="$(appliance_problems)"
    [ -z "$problems" ] && break
    [ "$i" -lt "$attempts" ] && sleep "$delay"
  done

  if [ -n "$problems" ]; then
    echo "$PRODUCT_NAME HEALTH CHECK FAILED ($phase, after $attempts attempt(s)):"
    echo -e "$problems"
    send_mail "[PATCH $phase HEALTH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"$PRODUCT_NAME health check failed ($phase patching):

$(echo -e "$problems")
Host: $HOSTNAME_SHORT
Log : $LOG_FILE

If the application is not healthy: sudo systemctl status $APP_SERVICE
and sudo journalctl -u $APP_SERVICE -n 100 show why.

If the SSH port changed: /etc/ssh/sshd_config was likely replaced during
patching. Restore your Port line in /etc/ssh/sshd_config, then restart SSH
(Ubuntu 24.04: systemctl daemon-reload && systemctl restart ssh.socket;
otherwise: systemctl restart $SSH_SERVICE)."
    return 1
  fi
  echo "$PRODUCT_NAME health check OK ($phase)"
  return 0
}

# ----------------------------------------------------------------------------
# Reboot-required detection
# ----------------------------------------------------------------------------
reboot_required() {
  case "$OS_FAMILY" in
    ubuntu)
      [ -f /var/run/reboot-required ]
      ;;
    al2023)
      if command -v needs-restarting >/dev/null 2>&1; then
        ! needs-restarting -r >/dev/null 2>&1
      else
        # Fallback: a newer kernel is installed than the one running.
        local newest running
        newest=$(rpm -q kernel --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' 2>/dev/null | grep -v 'not installed' | sort -V | tail -1)
        running=$(uname -r)
        [ -n "$newest" ] && [ "$newest" != "$running" ]
      fi
      ;;
  esac
}

# ----------------------------------------------------------------------------
# Service restart after patching
#
# A running process keeps using the old copy of every executable and library
# an update replaced until it restarts, so a patched Java runtime or OpenSSL
# only takes effect once StorageLink restarts. The script finds out directly:
# a service whose main process still maps a replaced (deleted) file under
# /usr or /lib is restarted. (/opt is deliberately excluded: packages never
# update it, and the application maps a self-deleting JNA helper from
# /opt/swiftgw/.cache that would otherwise trigger a restart on every run.)
# Order matters: the local database first,
# so the application reconnects to the new server, then the application,
# then nginx. Package scripts often restart nginx and PostgreSQL themselves,
# in which case they are no longer on replaced files and are left alone.
# Runs before the POST health check, which then verifies the restart.
# ----------------------------------------------------------------------------
uses_replaced_files() {
  local pid
  pid=$(systemctl show -p MainPID --value "$1" 2>/dev/null)
  [ -n "$pid" ] && [ "$pid" -gt 0 ] 2>/dev/null || return 1
  grep -qE ' /(usr|lib|lib64|bin|sbin)/[^ ]* \(deleted\)$' "/proc/$pid/maps" 2>/dev/null
}

# systemd unit of the local PostgreSQL server (postgresql@18-main.service on
# Ubuntu, for example), found from the process listening on the database port.
local_db_unit() {
  local pid
  pid=$(ss -tlnpH "sport = :$DB_PORT" 2>/dev/null | grep -oE 'pid=[0-9]+' | head -1 | cut -d= -f2)
  [ -n "$pid" ] && ps -o unit= -p "$pid" 2>/dev/null | awk 'NF && $1 != "-" {print $1; exit}'
}

restart_services_if_needed() {
  local units="" unit restarted="" failed=""
  local_db_expected && units="$(local_db_unit)"
  units="$units $APP_SERVICE nginx"
  for unit in $units; do
    uses_replaced_files "$unit" || continue
    echo "$unit is still running on files the update replaced; restarting it."
    if systemctl restart "$unit"; then
      restarted+="${restarted:+, }$unit"
    else
      failed+="${failed:+, }$unit"
      echo "WARNING: systemctl restart $unit failed (the POST health check will report the state)."
    fi
  done
  if [ -n "$restarted$failed" ]; then
    SERVICE_ACTION="restarted ${restarted:-nothing} (running on files the update replaced)${failed:+; restart FAILED for $failed}"
  else
    SERVICE_ACTION="no restart needed (no StorageLink service was running on replaced files)"
  fi
}

# ----------------------------------------------------------------------------
# Run
# ----------------------------------------------------------------------------
echo
print_header "THORNTECH OS PATCH RUN START (revision $SCRIPT_REVISION)"
echo "Host        : $HOSTNAME_SHORT"
echo "Product     : $PRODUCT_NAME"
echo "OS          : ${PRETTY_NAME:-unknown}"
echo "Mode        : $([ "$CHECK_ONLY" = "yes" ] && echo 'CHECK ONLY (no changes)' || echo 'PATCH')"
echo "Time        : $TIMESTAMP"
if [ "$CHECK_ONLY" != "yes" ]; then
  if [ "$AUTO_REBOOT" = "yes" ]; then
    echo "Reboot      : automatic if the updates require it, after a $REBOOT_DELAY_MINUTES-minute warning (--no-reboot to report instead)"
  else
    echo "Reboot      : NOT automatic (--no-reboot); a required reboot will be reported for you to perform"
  fi
fi
echo

if [ -z "$OS_FAMILY" ]; then
  echo "Unsupported operating system (${PRETTY_NAME:-unknown})."
  echo "This script supports Amazon Linux 2023 and Ubuntu 20.04/22.04/24.04."
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT][UNSUPPORTED-OS]" \
    "Patch run aborted: unsupported OS (${PRETTY_NAME:-unknown}). Log: $LOG_FILE"
  exit 2
fi

# Prevent concurrent runs. flock is atomic and the kernel releases it when the
# process exits however it dies, so no stale-lock handling is needed (the lock
# file itself remaining on disk is harmless).
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
  echo "Another patch run is already in progress. Exiting."
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT][LOCK]" \
    "Patch run skipped: another run is already in progress."
  exit 1
fi

RC=0
REBOOT_REQUIRED="no"
FAIL_REASON=""

# Refuse to patch a box that is already unhealthy. A freshly booted appliance
# can still be starting services, so allow a few settle attempts.
if ! check_appliance_health "PRE" 6 10; then
  RC=3
  FAIL_REASON="Pre-patch $PRODUCT_NAME health check failed"
fi

# Remember the effective SSH port(s) so the POST check can prove patching did
# not change them.
SSH_PORTS_BEFORE="$(sshd_ports)"
echo "SSH port(s) : ${SSH_PORTS_BEFORE:-unknown}"

# --check mode: report pending updates and stop (never mutates anything).
if [ "$CHECK_ONLY" = "yes" ]; then
  print_header "PENDING UPDATES (check mode)"
  case "$OS_FAMILY" in
    ubuntu)
      if apt_update -o DPkg::Lock::Timeout=600 -qq; then
        apt-get -s -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade 2>/dev/null \
          | grep -E '^(Inst|The following)' || echo "System is up to date."
      else
        echo "WARNING: apt-get update failed for an Ubuntu package source; pending-update list unavailable."
      fi
      ;;
    al2023)
      # check-update exits 100 when updates are available, 0 when none, 1 on error.
      dnf -q --releasever=latest check-update; cu_rc=$?
      case "$cu_rc" in
        0)   echo "System is up to date." ;;
        100) ;;
        *)   echo "WARNING: dnf check-update failed (exit $cu_rc); pending-update list unavailable." ;;
      esac
      ;;
  esac
  reboot_required && echo "NOTE: a reboot is already pending from an earlier update. A PATCH run will perform it after the warning delay, or run: sudo reboot"
  print_header "CHECK COMPLETE (no changes made)"
  exit "$RC"
fi

# Record the installed package set before anything is upgraded.
[ "$RC" -eq 0 ] && snapshot_packages "$PKG_BEFORE"

# ----------------------------------------------------------------------------
# Patch — Ubuntu (apt)
#
# The apt options are load-bearing:
#   - DPkg::Lock::Timeout waits for unattended-upgrades (which runs by default
#     on these appliances) instead of failing on the dpkg lock.
#   - force-confdef/confold keep locally-modified config files even under an
#     org-wide confnew policy, and prevent interactive conffile prompts from
#     killing the run.
#   - UCF_FORCE_CONFFOLD does the same for files managed by ucf rather than
#     dpkg — most importantly /etc/ssh/sshd_config, so a customized SSH port
#     survives an openssh-server update.
# ----------------------------------------------------------------------------
if [ "$RC" -eq 0 ] && [ "$OS_FAMILY" = "ubuntu" ]; then
  export DEBIAN_FRONTEND=noninteractive
  export UCF_FORCE_CONFFOLD=1
  APT_OPTS=(-y -o DPkg::Lock::Timeout=600
            -o Dpkg::Options::=--force-confdef
            -o Dpkg::Options::=--force-confold)

  if apt_update "${APT_OPTS[@]}"; then APT_REFRESHED="yes"; else RC=100; FAIL_REASON="apt-get update failed for an Ubuntu package source (see log)"; fi
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" upgrade || { RC=100; FAIL_REASON="apt-get upgrade failed"; }; }
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" dist-upgrade || { RC=100; FAIL_REASON="apt-get dist-upgrade failed"; }; }
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" autoremove || true; }

# ----------------------------------------------------------------------------
# Patch — Amazon Linux 2023 (dnf)
#
# rpm marks sshd_config %config(noreplace), so dnf writes .rpmnew files rather
# than replacing edited configs — the apt confnew hazard does not exist here.
# dnf waits on a busy RPM lock by itself; no timeout option is needed.
#
# --releasever=latest is load-bearing: AL2023 locks each instance to the
# repository snapshot it launched with, so a plain 'dnf upgrade' reports
# "nothing to do" forever — security updates only arrive by moving to the
# newest release snapshot.
# ----------------------------------------------------------------------------
elif [ "$RC" -eq 0 ] && [ "$OS_FAMILY" = "al2023" ]; then
  dnf -y upgrade --releasever=latest --refresh || { RC=100; FAIL_REASON="dnf upgrade failed"; }
fi

# What changed? Reported even if the package manager failed part-way, so the
# log shows exactly how far the run got.
if [ -s "$PKG_BEFORE" ]; then
  snapshot_packages "$PKG_AFTER"
  compute_package_changes
  print_package_changes
fi

if [ "$RC" -eq 0 ]; then
  restart_services_if_needed
fi

if [ "$RC" -eq 0 ] && reboot_required; then
  REBOOT_REQUIRED="yes"
fi

# Catch anything patching broke, while we can still alert about it. Retries
# ride out the application's start-up after the restart above.
if [ "$RC" -eq 0 ]; then
  if ! check_appliance_health "POST" 18 10; then
    RC=4
    FAIL_REASON="Post-patch $PRODUCT_NAME health check failed (see health mail/log)"
  fi
fi

print_header "THORNTECH OS PATCH RUN END"
echo "Result      : $([ "$RC" -eq 0 ] && echo 'PATCH SUCCESS' || echo "PATCH FAILED (${FAIL_REASON:-exit $RC})")"
echo "Packages    : $PKG_SUMMARY"
echo "Service     : $SERVICE_ACTION"
if [ "$OS_FAMILY" = "ubuntu" ]; then
  if [ "$APT_REFRESHED" = "yes" ]; then
    echo "Sources     : ${APT_SOURCE_WARNINGS:+WARNING — skipped: }${APT_SOURCE_WARNINGS:-all package sources refreshed}"
  else
    echo "Sources     : not refreshed"
  fi
fi
if [ "$REBOOT_REQUIRED" = "yes" ]; then
  [ "$AUTO_REBOOT" = "yes" ] && echo "Reboot      : REQUIRED — rebooting in $REBOOT_DELAY_MINUTES minute(s)" || echo "Reboot      : REQUIRED — not performed (--no-reboot); reboot manually"
else
  echo "Reboot      : not required"
fi
echo "Full list   : $LOG_FILE (PACKAGE CHANGES section)"
echo

# ----------------------------------------------------------------------------
# Notifications and reboot handling
# ----------------------------------------------------------------------------
if [ "$RC" -ne 0 ]; then
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"PATCH FAILED

Host: $HOSTNAME_SHORT
Reason: $FAIL_REASON
Packages: $PKG_SUMMARY

Log: $LOG_FILE"
  exit "$RC"
fi

send_mail "[PATCH SUCCESS][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"PATCH SUCCESS

Host: $HOSTNAME_SHORT
Packages: $PKG_SUMMARY
Service: $SERVICE_ACTION
Sources: ${APT_SOURCE_WARNINGS:+skipped: }${APT_SOURCE_WARNINGS:-all refreshed}
Reboot required: $REBOOT_REQUIRED
Log: $LOG_FILE (full package list in the PACKAGE CHANGES section)"

if [ "$REBOOT_REQUIRED" = "yes" ]; then
  if [ "$AUTO_REBOOT" = "yes" ]; then
    echo
    echo "*** REBOOT REQUIRED to finish applying the updates. ***"
    echo "*** This server will reboot in $REBOOT_DELAY_MINUTES minute(s). Web sessions and transfers in progress will drop and"
    echo "*** $PRODUCT_NAME will be unavailable for a few minutes, then start on its own. To cancel: sudo shutdown -c"
    echo
    if shutdown -r "+$REBOOT_DELAY_MINUTES" "ThornTech OS patch: rebooting to finish applying updates (cancel with: shutdown -c)"; then
      send_mail "[PATCH REBOOT SCHEDULED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"Patching succeeded and a reboot is required to finish applying updates.
Rebooting in $REBOOT_DELAY_MINUTES minute(s); $PRODUCT_NAME starts on its own afterwards.

Host: $HOSTNAME_SHORT
Packages: $PKG_SUMMARY
Log : $LOG_FILE"
    else
      send_mail "[PATCH REBOOT SCHEDULING FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"Patching succeeded and a reboot is required, but scheduling the automatic
reboot FAILED. Reboot this host manually: sudo reboot

Host: $HOSTNAME_SHORT
Log : $LOG_FILE"
      exit 5
    fi
  else
    echo "Reboot required but NOT performed (--no-reboot). To finish applying the updates, run one of:"
    echo "  sudo reboot                               # now"
    echo "  sudo shutdown -r +15 \"OS patch reboot\"   # in 15 minutes, with a warning to logged-in users"
    echo "$PRODUCT_NAME starts on its own after the reboot; no service restart is needed."
    send_mail "[PATCH ACTION REQUIRED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"REBOOT REQUIRED (manual — the run was started with --no-reboot)

Host: $HOSTNAME_SHORT
Time: $TIMESTAMP
Packages: $PKG_SUMMARY

The system requires a reboot to finish applying updates. In your change
window, run one of:

  sudo reboot
  sudo shutdown -r +15 \"OS patch reboot\"

$PRODUCT_NAME starts on its own after the reboot; no service restart is needed.

Log: $LOG_FILE"
  fi
fi

exit 0