Patching the Operating System

TLDR - Quick Summary

What: A ThornTech-maintained script that applies OS security updates to your SFTP Gateway Compute Engine instance safely.

Why: SFTP Gateway runs its SFTP service on port 22 and moves OpenSSH admin access to port 2222. A routine OS update that replaces sshd_config can revert SSH to port 22 and knock the appliance offline. This script prevents that.

Before you run it: export a backup file (Settings > Backup & Recovery > Export) and snapshot the disk, so the configuration and the server can both be restored if patching goes wrong.

Run: wget https://thorntech-products.s3.amazonaws.com/sftpgateway/os-patching-script/thorntech-os-patch.sh then sudo bash ./thorntech-os-patch.sh

Reboot: performed automatically when the updates require it, after a two-minute warning you can cancel; add --no-reboot to report instead.

Schedule: add it to root's crontab to patch on a monthly change window.

Overview

Keeping the operating system patched is part of running SFTP Gateway securely, but the appliance's port layout makes a generic update risky: the SFTP service (Java) owns port 22 and OpenSSH admin access lives on port 2222. An OS update that replaces /etc/ssh/sshd_config with the package default puts SSH back on port 22, where it collides with the SFTP service, and the appliance becomes unreachable after the next restart.

The ThornTech OS Patching script applies OS updates while guarding against that failure. On Google Cloud, SFTP Gateway runs on Ubuntu, which the script patches with apt.

Which Ubuntu release your appliance runs depends on the SFTP Gateway version you deployed:

Ubuntu release	SFTP Gateway versions	Ubuntu standard support ends
20.04	3.3.2 and earlier	April 2025 (ended; security updates only with Ubuntu Pro)
22.04	3.3.3 to 3.7.3	April 2027
24.04	3.7.4 and later	April 2029

An in-place upgrade of SFTP Gateway does not change the operating system; moving to a newer Ubuntu release means deploying the current SFTP Gateway image and importing your backup file.

The script:

• Checks the appliance is healthy before patching and refuses to run if it is not.
• Pins the admin SSH port with a drop-in that survives any package replacing sshd_config.
• Preserves your local configuration files instead of overwriting edited configs with package defaults.
• Verifies the appliance is still healthy afterwards and reports loudly if anything is wrong.
• Reports exactly what changed: package counts in the run summary, and a full list of upgraded, installed, and removed packages in the log.
• Finishes the job. If the updates require a reboot, it says so, waits two minutes so you can cancel with sudo shutdown -c, then reboots. SFTP Gateway starts on its own afterwards. Add --no-reboot if reboots must go through your own change window.

Before you begin

Step 1: Export a backup file. In the web admin interface open Settings, scroll to Backup & Recovery, click Export, and choose Export Backup File. Keep the downloaded YAML file somewhere other than this Compute Engine instance. If patching leaves the server unrecoverable, that file restores your users, folders, cloud connections, and settings onto a fresh deployment through Settings > Backup & Recovery > Import (the full procedure is in Upgrading SFTP Gateway). Automated Backup shows how to schedule these exports.

Step 2: Snapshot the disk. Create a snapshot of the boot disk (Compute Engine > Snapshots > Create snapshot, or gcloud compute disks snapshot). That is the one-step rollback if the operating system itself is left unbootable.

Step 3: Connect over SSH. Connect to your Compute Engine instance over SSH (admin SSH is on port 2222):

gcloud compute ssh <your-instance-name> --zone <zone> -- -p 2222

Download and run

Run it during a maintenance window: the run may restart the SFTP Gateway service and will reboot the server if the updates require it, which drops active transfers for a minute or two.

The appliance mounts /tmp as no-exec, so download the script to your home directory and run it with sudo bash (rather than executing the file directly):

cd ~
wget https://thorntech-products.s3.amazonaws.com/sftpgateway/os-patching-script/thorntech-os-patch.sh
sudo bash ./thorntech-os-patch.sh

The script logs everything to /var/log/thorntech-os-patch.log and to the system log.

Preview without changing anything

To see the pending updates and confirm the appliance is healthy without patching, use --check. It changes nothing, so if an earlier update left a reboot pending it only reports that:

sudo bash ./thorntech-os-patch.sh --check

Scheduling monthly patching

⚠️ Expect a short interruption on every scheduled run. Updating a library the SFTP Gateway service depends on restarts the service, and a kernel update reboots the server. Either way, active SFTP transfers drop and the appliance is unavailable for a minute or two. Pick a cron time inside a maintenance window when clients are idle, give each instance of an HA deployment its own time so they never restart together, and use --no-reboot if reboots must be approved separately.

A scheduled run has nobody to export a backup first, so keep the Automated Backup job running and schedule it ahead of the patch window (for example backups at 01:00, patching at 02:00).

Install the script once and add a root cron entry. This example patches at 02:00 on the second Tuesday of each month:

sudo cp thorntech-os-patch.sh /usr/local/bin/
sudo crontab -e

Add the line:

0 2 8-14 * * [ "$(date +\%u)" = "2" ] && bash /usr/local/bin/thorntech-os-patch.sh

A scheduled run reboots the server on its own when the updates require it, after the warning delay (two minutes by default; change it with --reboot-delay=MIN). If your change process needs reboots approved separately, add --no-reboot to the cron line and the run will report the pending reboot instead. In a high-availability deployment, give each instance a different cron time so they never reboot together.

What the script reports

Every run ends with one of these outcomes (also reflected in the exit code):

Result	Meaning
PATCH SUCCESS	Updates applied and the appliance verified healthy afterwards. The summary shows how many packages were upgraded, installed, and removed; the full list is in the PACKAGE CHANGES section of the log.
PATCH REBOOT SCHEDULED	Patching succeeded and a reboot is needed to finish (usually a kernel update). The server reboots after the warning delay, during which new SSH logins are refused by the system; cancel with sudo shutdown -c if you must. SFTP Gateway starts on its own after the reboot.
PATCH ACTION REQUIRED	Same situation, but the run was started with --no-reboot. Reboot in your change window with sudo reboot, or sudo shutdown -r +15 to give users notice.
PRE health failed	The appliance was already unhealthy, so nothing was patched.
POST health failed	Patching completed but a post-patch check failed — investigate before rebooting.

The summary's Sources line lists any package repository the run had to skip. That happens when a third-party source no longer publishes an index, for example apt.postgresql.org once an Ubuntu release reaches end of life; the Ubuntu sources still refresh and the run continues without that repository. To stop the warning, disable the dead source (sudo mv /etc/apt/sources.list.d/pgdg.list /etc/apt/sources.list.d/pgdg.list.disabled) or point it at apt-archive.postgresql.org. The run fails only if an Ubuntu archive itself cannot be refreshed.

Note: Ubuntu 20.04 (SFTP Gateway 3.3.2 and earlier) reached end of standard support in April 2025 and no longer receives security updates without Ubuntu Pro. Patching still applies whatever Canonical last published, but plan an upgrade to a current SFTP Gateway release.

Note: On Ubuntu, needrestart runs as part of the update and restarts any service whose libraries were upgraded, including the SFTP Gateway service, so you do not need to restart it by hand. The script waits for those restarts before running its post-patch health check, and the Service line in the summary records what was restarted. The 24.04 images ship without needrestart, so there the script restarts the service itself when a runtime library changed; the Service line records which happened.

Need help?

If a patch run reports a health failure, or you would like help scheduling patching across a fleet, contact our support team at support@thorntech.com.

Optional: email notifications

No mail server is needed. Every run writes its result to /var/log/thorntech-os-patch.log and to syslog (tag thorntech-os-patch), which is what most monitoring hooks into. If you also want an email per run, install and configure a mail transfer agent on the appliance (for example mailutils with a configured relay; none is installed by default), then pass --mail-to:

sudo bash ./thorntech-os-patch.sh --mail-to=ops@example.com

Add the same flag to the cron entry above to receive the monthly result by email.

Script contents

The full script is reproduced below so you can review exactly what it does before running it on your appliance. This copy matches the script published on September 22, 2026 (revision 2026-09-22, shown in the run header). The download URL always serves the current version, so use the curl command in Download and run rather than copying from this page.

#!/bin/bash
#
# ============================================================================
#  ThornTech OS Patching Script — SFTP Gateway edition
# ============================================================================
#
#  Applies operating system updates to an SFTP Gateway appliance safely:
#  it verifies the appliance is healthy before touching anything, pins the
#  admin SSH port so a package update can never knock the appliance offline,
#  patches with settings that preserve your local configuration files,
#  verifies the appliance is still healthy afterwards, and reboots the server
#  when the updates require it (after a warning delay; see --no-reboot).
#
#  Supported: Ubuntu 20.04/22.04/24.04 and Amazon Linux 2023
#  (covers SFTP Gateway 3.6.0+ images on AWS, Azure, and Google Cloud).
#
#  USAGE (note: /tmp is mounted noexec on the appliance — invoke with
#  'sudo bash' rather than executing the file directly):
#
#      wget https://thorntech-products.s3.amazonaws.com/sftpgateway/os-patching-script/thorntech-os-patch.sh
#      sudo bash ./thorntech-os-patch.sh
#
#  Options:
#      --check              Report pending updates and appliance health only;
#                           changes nothing.
#      --no-reboot          Report a required reboot instead of performing it
#                           (default: reboot automatically when required, after
#                           the warning delay below). SFTP Gateway starts on its
#                           own after a reboot.
#      --reboot-delay=MIN   Whole minutes between the warning and the automatic
#                           reboot (default 2; 0 reboots immediately). Cancel a
#                           pending reboot with: sudo shutdown -c
#      --mail-to=ADDR       Optional: also email each run's result to ADDR.
#                           Needs a working 'mail' command and MTA on the
#                           appliance (none is installed by default). Results
#                           always go to the log and syslog regardless.
#
#  Environment variables MAIL_TO, MAIL_FROM, AUTO_REBOOT (yes/no) and
#  REBOOT_DELAY_MINUTES provide defaults; command-line flags override them.
#
#  Scheduled use — install once, then add a root crontab entry
#  (sudo crontab -e). Example: 02:00 on the second Tuesday of each month:
#
#      sudo cp thorntech-os-patch.sh /usr/local/bin/
#      0 2 8-14 * * [ "$(date +\%u)" = "2" ] && bash /usr/local/bin/thorntech-os-patch.sh
#
#  A third-party package source that no longer resolves (for example
#  apt.postgresql.org after an Ubuntu release reaches end of life) is skipped
#  with a warning; the run fails only if an Ubuntu archive itself cannot be
#  refreshed. Skipped sources are listed in the run summary.
#
#  Every run appends to /var/log/thorntech-os-patch.log (rotated once past
#  5 MB) and writes result lines to syslog (tag: thorntech-os-patch). The log
#  includes a PACKAGE CHANGES section listing every package the run upgraded,
#  installed, or removed; the run summary shows the counts.
#
#  Exit codes:
#      0   success (the server may be about to reboot if updates required it)
#      1   another run is already in progress
#      2   unsupported operating system
#      3   pre-patch health check failed — nothing was changed
#      4   patching succeeded but the post-patch health check failed
#      5   a reboot was required but scheduling it failed — reboot manually
#      64  usage error (unknown or invalid option)
#      77  not running as root
#      100 the package manager reported an error
#
# ============================================================================

set -u
set -o pipefail

# Cron runs with a minimal PATH that lacks /usr/sbin (where sshd, nginx,
# ss and shutdown live). Interactive sudo masks this; cron does not.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

SCRIPT_REVISION="2026-09-22"   # date this copy was published; quote it when contacting support
PRODUCT_NAME="SFTP Gateway"

# ----------------------------------------------------------------------------
# Appliance facts (SFTP Gateway edition — do not change)
#
#   - The SFTP service (java) owns port 22. OpenSSH admin access is on 2222.
#   - If /etc/ssh/sshd_config is ever replaced by a package default, sshd
#     falls back to port 22, collides with the SFTP service, and the appliance
#     becomes unreachable for both SFTP users and admins after the next
#     service restart or reboot. The port pin drop-in below makes that
#     impossible regardless of package-manager configuration policy.
# ----------------------------------------------------------------------------
SSH_ADMIN_PORT=2222
SSH_PIN_FILE=/etc/ssh/sshd_config.d/10-sftpgw-port.conf
APP_SERVICE=sftpgw-admin-api

# Packages whose upgrade means the running SFTP Gateway service (and the nginx
# front end) are still using old code until restarted. Ubuntu's needrestart
# handles this automatically; where it is absent (Amazon Linux 2023, and
# Ubuntu images without needrestart) the script restarts the services itself.
SERVICE_LIB_PATTERN='^(java|.*corretto|.*openjdk|openssl|libssl|glibc|libc6|zlib|nss|libnss|ca-certificates|libcurl|curl|libgcc|libstdc\+\+|krb5|libkrb5|gnutls|nginx)'

# ----------------------------------------------------------------------------
# Configuration (environment variables provide defaults; flags override)
# ----------------------------------------------------------------------------
MAIL_TO="${MAIL_TO:-}"
MAIL_FROM="${MAIL_FROM:-thorntech-os-patch@$(hostname -f 2>/dev/null || hostname)}"
AUTO_REBOOT="${AUTO_REBOOT:-yes}"
REBOOT_DELAY_MINUTES="${REBOOT_DELAY_MINUTES:-2}"
CHECK_ONLY="no"

LOG_FILE="/var/log/thorntech-os-patch.log"
LOCK_FILE="/var/run/thorntech-os-patch.lock"

usage() {
  # Print the header block (from line 2 down to the first non-comment line).
  awk 'NR==1{next} !/^#/{exit} {sub(/^# ?/,""); print}' "$0"
}

for arg in "$@"; do
  case "$arg" in
    --check)            CHECK_ONLY="yes" ;;
    --mail-to=*)        MAIL_TO="${arg#*=}" ;;
    --no-reboot)        AUTO_REBOOT="no" ;;
    --auto-reboot)      AUTO_REBOOT="yes" ;;   # accepted for older cron entries; now the default
    --reboot-delay=*)   REBOOT_DELAY_MINUTES="${arg#*=}" ;;
    --help|-h)          usage; exit 0 ;;
    *)
      echo "Unknown option: $arg (try --help)" >&2
      exit 64
      ;;
  esac
done

case "$REBOOT_DELAY_MINUTES" in
  ''|*[!0-9]*)
    echo "--reboot-delay must be a whole number of minutes (got: '$REBOOT_DELAY_MINUTES')" >&2
    exit 64
    ;;
esac

if [ "$(id -u)" -ne 0 ]; then
  echo "This script must run as root: sudo bash $0" >&2
  exit 77
fi

HOSTNAME_SHORT="$(hostname)"
TIMESTAMP="$(date '+%Y-%m-%d %H:%M:%S %Z')"

# Rotate the log once, past 5 MB, so unattended monthly use never fills a disk.
if [ -f "$LOG_FILE" ] && [ "$(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)" -gt 5242880 ]; then
  mv -f "$LOG_FILE" "$LOG_FILE.1"
fi
touch "$LOG_FILE"
chmod 640 "$LOG_FILE"
chown root:root "$LOG_FILE"

# Package inventory snapshots (before/after) drive the PACKAGE CHANGES report.
PKG_BEFORE="$(mktemp)"
PKG_AFTER="$(mktemp)"
PKG_DIFF="$(mktemp)"
PKG_SUMMARY="not measured"
PKG_UPGRADED=0
PKG_INSTALLED=0
PKG_REMOVED=0
SERVICE_ACTION="no restart needed"
APT_SOURCE_WARNINGS=""

# Log everything while keeping output visible when run interactively; the
# cleanup trap waits for tee so the tail of the log is complete on exit.
exec > >(tee -a "$LOG_FILE") 2>&1
TEE_PID=$!
cleanup() {
  rm -f "$PKG_BEFORE" "$PKG_AFTER" "$PKG_DIFF"
  exec 1>&- 2>&-
  [ -n "${TEE_PID:-}" ] && wait "$TEE_PID" 2>/dev/null
}
trap cleanup EXIT

send_mail() {
  local subject="$1" body="$2"
  # Always leave a machine-findable trace, regardless of mail delivery.
  echo "[NOTIFY] $subject"
  logger -t thorntech-os-patch "$subject" 2>/dev/null || true
  if [ -n "$MAIL_TO" ] && command -v mail >/dev/null 2>&1; then
    if ! echo -e "$body" | mail -s "$subject" -r "$MAIL_FROM" "$MAIL_TO"; then
      echo "[NOTIFY] mail delivery to $MAIL_TO failed; see $LOG_FILE"
      logger -t thorntech-os-patch "mail delivery to $MAIL_TO failed" 2>/dev/null || true
    fi
  fi
}

print_header() {
  echo "========================================"
  echo "$1"
  echo "========================================"
}

# ----------------------------------------------------------------------------
# OS detection
# ----------------------------------------------------------------------------
OS_FAMILY=""
SSH_SERVICE=""
if [ -r /etc/os-release ]; then
  . /etc/os-release
  case "${ID:-}" in
    ubuntu)
      OS_FAMILY="ubuntu"
      SSH_SERVICE="ssh"
      ;;
    amzn)
      if [ "${VERSION_ID:-}" = "2023" ]; then
        OS_FAMILY="al2023"
        SSH_SERVICE="sshd"
      fi
      ;;
  esac
fi

# ----------------------------------------------------------------------------
# Package inventory — snapshot the installed set before and after patching so
# the run can report exactly what changed, independent of the package
# manager's own output format. One line per package: "<name>\t<version>".
# ----------------------------------------------------------------------------
snapshot_packages() {
  case "$OS_FAMILY" in
    ubuntu)
      dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${Version}\n' 2>/dev/null \
        | awk -F'\t' '$1 ~ /^ii/ {print $2 "\t" $3}' | sort > "$1"
      ;;
    al2023)
      rpm -qa --qf '%{NAME}.%{ARCH}\t%{EVR}\n' 2>/dev/null | sort > "$1"
      ;;
  esac
}

# Writes "<U|I|R>\t<name>\t<detail>" lines to $PKG_DIFF and sets the counters.
compute_package_changes() {
  awk -F'\t' '
    NR==FNR { b[$1] = (b[$1] == "" ? $2 : b[$1] "," $2); next }
            { a[$1] = (a[$1] == "" ? $2 : a[$1] "," $2) }
    END {
      for (n in a) if (!(n in b)) print "I\t" n "\t" a[n];
      for (n in b) if (!(n in a)) print "R\t" n "\t" b[n];
      for (n in a) if ((n in b) && a[n] != b[n]) print "U\t" n "\t" b[n] " -> " a[n];
    }' "$PKG_BEFORE" "$PKG_AFTER" | sort -k1,1 -k2,2 > "$PKG_DIFF"
  PKG_UPGRADED=$(grep -c '^U' "$PKG_DIFF" || true)
  PKG_INSTALLED=$(grep -c '^I' "$PKG_DIFF" || true)
  PKG_REMOVED=$(grep -c '^R' "$PKG_DIFF" || true)
  PKG_SUMMARY="$PKG_UPGRADED upgraded, $PKG_INSTALLED newly installed, $PKG_REMOVED removed"
}

print_package_changes() {
  print_header "PACKAGE CHANGES ($PKG_SUMMARY)"
  if [ ! -s "$PKG_DIFF" ]; then
    echo "No packages were changed."
    return
  fi
  local section name detail
  for section in U I R; do
    grep -q "^$section" "$PKG_DIFF" || continue
    case "$section" in
      U) echo "Upgraded:" ;;
      I) echo "Newly installed:" ;;
      R) echo "Removed:" ;;
    esac
    while IFS=$'\t' read -r _ name detail; do
      printf '  %s  %s\n' "$name" "$detail"
    done < <(grep "^$section" "$PKG_DIFF")
  done
}

# Names of packages upgraded or installed this run (one per line).
changed_package_names() {
  awk -F'\t' '$1 == "U" || $1 == "I" {print $2}' "$PKG_DIFF"
}

# ----------------------------------------------------------------------------
# apt-get update wrapper. A dead third-party repository (common once an Ubuntu
# release passes end of life, e.g. apt.postgresql.org dropping its suite) makes
# apt-get update exit non-zero even though every Ubuntu index refreshed fine.
# Skip such sources with a warning; fail only if an Ubuntu archive itself could
# not be refreshed, or the failure cannot be attributed to a source.
# ----------------------------------------------------------------------------
apt_update() {
  local out rc failed ubuntu_failed
  out=$(mktemp)
  apt-get "$@" update >"$out" 2>&1; rc=$?
  cat "$out"
  if [ "$rc" -ne 0 ]; then
    failed=$(grep -E "^(E|W): .*(Release file|Failed to fetch|no longer|not signed|NO_PUBKEY)" "$out" | grep -oE "https?://[^ ']+( [^ ']+)?" | sed -E 's#/(InRelease|Release)(\.gpg)?$##' | sort -u)
    ubuntu_failed=$(printf '%s\n' "$failed" | grep -E 'ubuntu\.com' || true)
    if [ -z "$failed" ] || [ -n "$ubuntu_failed" ]; then
      rm -f "$out"; return "$rc"
    fi
    APT_SOURCE_WARNINGS=$(printf '%s\n' "$failed" | sed -E 's#^https?://##' | paste -sd';' | sed 's/;/; /g')
    echo "WARNING: skipping package source(s) that failed to refresh: $APT_SOURCE_WARNINGS"
    echo "         Updates from those sources are not applied this run; all Ubuntu sources refreshed normally."
  fi
  rm -f "$out"
  return 0
}

# ----------------------------------------------------------------------------
# Appliance health check — called before patching (to refuse to patch an
# already-broken box) and after (to catch breakage loudly).
#
# The POST phase retries: on Ubuntu, needrestart auto-restarts services whose
# libraries were upgraded (including the java SFTP service), so a single-shot
# probe right after apt finishes can catch a service mid-restart.
# ----------------------------------------------------------------------------
appliance_problems() {
  local problems=""
  local sshd_ports

  # 'sshd -T' fatally exits ("Missing privilege separation directory") when
  # /run/sshd is absent, which is exactly the state an openssh-server upgrade
  # leaves behind until ssh.service next starts. Recreating it is the same
  # idempotent step the service itself performs, so do it before probing.
  [ -d /run/sshd ] || install -d -m 0755 /run/sshd 2>/dev/null || true

  sshd_ports=$(sshd -T 2>/dev/null | awk '/^port /{print $2}' | sort -u | paste -sd,)
  if [ -n "$sshd_ports" ]; then
    # sshd -T resolved the effective config (main + Include drop-ins): this is
    # the authoritative future-reboot guard.
    [ "$sshd_ports" = "$SSH_ADMIN_PORT" ] || problems+="- sshd config port is '$sshd_ports' (expected $SSH_ADMIN_PORT). If this reaches a restart/reboot, sshd will collide with the SFTP service on port 22 and the appliance will be unreachable.\n"
  else
    # sshd -T could not run (e.g. mid-upgrade). Don't treat "can't determine
    # the config port" as the brick condition — the runtime 2222-listener
    # check below is the real safety signal. Note it and move on.
    echo "NOTE: sshd -T could not be evaluated this pass (likely mid-upgrade); relying on the live $SSH_ADMIN_PORT listener check."
  fi
  ss -tlnp 2>/dev/null | grep -q ":$SSH_ADMIN_PORT " || problems+="- nothing is listening on $SSH_ADMIN_PORT (admin SSH down)\n"
  ss -tlnp 2>/dev/null | grep ":22 " | grep -q java || problems+="- SFTP service (java) is not listening on port 22\n"
  systemctl is-active --quiet "$APP_SERVICE" || problems+="- $APP_SERVICE is not active\n"
  systemctl is-active --quiet nginx || problems+="- nginx is not active\n"
  nginx -t >/dev/null 2>&1 || problems+="- nginx config fails validation (nginx -t)\n"
  printf '%s' "$problems"
}

check_appliance_health() {
  local phase="$1" attempts="$2" delay="$3" problems="" i
  for (( i=1; i<=attempts; i++ )); do
    problems="$(appliance_problems)"
    [ -z "$problems" ] && break
    [ "$i" -lt "$attempts" ] && sleep "$delay"
  done

  if [ -n "$problems" ]; then
    echo "$PRODUCT_NAME HEALTH CHECK FAILED ($phase, after $attempts attempt(s)):"
    echo -e "$problems"
    send_mail "[PATCH $phase HEALTH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"$PRODUCT_NAME health check failed ($phase patching):

$(echo -e "$problems")
Host: $HOSTNAME_SHORT
Log : $LOG_FILE

If 'sshd config port' is wrong: /etc/ssh/sshd_config was likely replaced
during patching. Restore 'Port $SSH_ADMIN_PORT' in /etc/ssh/sshd_config
(the safety drop-in $SSH_PIN_FILE should also be present),
then: systemctl restart $SSH_SERVICE"
    return 1
  fi
  echo "$PRODUCT_NAME health check OK ($phase)"
  return 0
}

# ----------------------------------------------------------------------------
# Reboot-required detection
# ----------------------------------------------------------------------------
reboot_required() {
  case "$OS_FAMILY" in
    ubuntu)
      [ -f /var/run/reboot-required ]
      ;;
    al2023)
      if command -v needs-restarting >/dev/null 2>&1; then
        ! needs-restarting -r >/dev/null 2>&1
      else
        # Fallback: a newer kernel is installed than the one running.
        local newest running
        newest=$(rpm -q kernel --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' 2>/dev/null | grep -v 'not installed' | sort -V | tail -1)
        running=$(uname -r)
        [ -n "$newest" ] && [ "$newest" != "$running" ]
      fi
      ;;
  esac
}

# ----------------------------------------------------------------------------
# Service restart after patching
#
# On Ubuntu, needrestart (part of the default server image) restarts every
# service whose libraries were upgraded, the java SFTP service included, as
# part of the apt run. Amazon Linux has no equivalent, and older Ubuntu images
# may lack needrestart, so in those cases the script restarts the SFTP Gateway
# service (and nginx) itself when a library they depend on was upgraded.
# Runs before the POST health check, which then verifies the restart.
# ----------------------------------------------------------------------------
restart_services_if_needed() {
  local hits count examples
  hits=$(changed_package_names | grep -E "$SERVICE_LIB_PATTERN" | sed 's/:.*//' | sort -u)
  if [ -z "$hits" ]; then
    SERVICE_ACTION="no restart needed (no runtime libraries changed)"
    return
  fi
  count=$(printf '%s\n' "$hits" | grep -c .)
  examples=$(printf '%s\n' "$hits" | head -3 | paste -sd, | sed 's/,/, /g')
  [ "$count" -gt 3 ] && examples="$examples, ..."
  if [ "$OS_FAMILY" = "ubuntu" ] && command -v needrestart >/dev/null 2>&1; then
    SERVICE_ACTION="restarted by needrestart during the apt run ($count runtime libraries changed: $examples)"
    return
  fi
  echo "Runtime libraries changed ($count: $examples); restarting $APP_SERVICE and nginx."
  systemctl restart "$APP_SERVICE" || echo "WARNING: systemctl restart $APP_SERVICE failed (the POST health check will report the state)."
  systemctl try-restart nginx 2>/dev/null || true
  SERVICE_ACTION="restarted $APP_SERVICE and nginx ($count runtime libraries changed: $examples)"
}

# ----------------------------------------------------------------------------
# Run
# ----------------------------------------------------------------------------
echo
print_header "THORNTECH OS PATCH RUN START (revision $SCRIPT_REVISION)"
echo "Host        : $HOSTNAME_SHORT"
echo "Product     : $PRODUCT_NAME"
echo "OS          : ${PRETTY_NAME:-unknown}"
echo "Mode        : $([ "$CHECK_ONLY" = "yes" ] && echo 'CHECK ONLY (no changes)' || echo 'PATCH')"
echo "Time        : $TIMESTAMP"
if [ "$CHECK_ONLY" != "yes" ]; then
  if [ "$AUTO_REBOOT" = "yes" ]; then
    echo "Reboot      : automatic if the updates require it, after a $REBOOT_DELAY_MINUTES-minute warning (--no-reboot to report instead)"
  else
    echo "Reboot      : NOT automatic (--no-reboot); a required reboot will be reported for you to perform"
  fi
fi
echo

if [ -z "$OS_FAMILY" ]; then
  echo "Unsupported operating system (${PRETTY_NAME:-unknown})."
  echo "This script supports Ubuntu 20.04/22.04/24.04 and Amazon Linux 2023."
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT][UNSUPPORTED-OS]" \
    "Patch run aborted: unsupported OS (${PRETTY_NAME:-unknown}). Log: $LOG_FILE"
  exit 2
fi

# Prevent concurrent runs. flock is atomic and the kernel releases it when the
# process exits however it dies, so no stale-lock handling is needed (the lock
# file itself remaining on disk is harmless).
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
  echo "Another patch run is already in progress. Exiting."
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT][LOCK]" \
    "Patch run skipped: another run is already in progress."
  exit 1
fi

RC=0
REBOOT_REQUIRED="no"
FAIL_REASON=""

# Refuse to patch a box that is already unhealthy. A freshly booted appliance
# can still be starting services, so allow a few settle attempts.
if ! check_appliance_health "PRE" 3 10; then
  RC=3
  FAIL_REASON="Pre-patch $PRODUCT_NAME health check failed"
fi

# --check mode: report pending updates and stop (never mutates anything).
if [ "$CHECK_ONLY" = "yes" ]; then
  print_header "PENDING UPDATES (check mode)"
  case "$OS_FAMILY" in
    ubuntu)
      if apt_update -o DPkg::Lock::Timeout=600 -qq; then
        apt-get -s -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade 2>/dev/null \
          | grep -E '^(Inst|The following)' || echo "System is up to date."
      else
        echo "WARNING: apt-get update failed for an Ubuntu package source; pending-update list unavailable."
      fi
      ;;
    al2023)
      dnf -q --releasever=latest check-update || true
      ;;
  esac
  reboot_required && echo "NOTE: a reboot is already pending from an earlier update. A PATCH run will perform it after the warning delay, or run: sudo reboot"
  print_header "CHECK COMPLETE (no changes made)"
  exit "$RC"
fi

# Safety net: keep the admin SSH port pinned in a drop-in that survives a
# package replacing /etc/ssh/sshd_config. Harmless if already present.
# Installed only after the PRE gate so exit 3 leaves the box untouched.
#
# Some appliance images (notably AL2023) ship an sshd_config with no Include
# directive, which would leave the drop-in directory inert — so the Include
# is ensured too, inserted as the first line (Includes must precede any
# Match block). sshd -t guards the edit; on failure everything is reverted
# and patching continues without the pin.
if [ "$RC" -eq 0 ]; then
  mkdir -p /etc/ssh/sshd_config.d
  if [ ! -f "$SSH_PIN_FILE" ]; then
    printf 'Port %s\n' "$SSH_ADMIN_PORT" > "$SSH_PIN_FILE"
    chmod 644 "$SSH_PIN_FILE"
    echo "Installed $SSH_PIN_FILE (Port $SSH_ADMIN_PORT safety pin)"
  fi
  if ! grep -iEq '^[[:space:]]*include[[:space:]]+.*sshd_config\.d' /etc/ssh/sshd_config; then
    cp -a /etc/ssh/sshd_config "/etc/ssh/sshd_config.thorntech-preinclude"
    sed -i '1i Include /etc/ssh/sshd_config.d/*.conf' /etc/ssh/sshd_config
    if sshd -t >/dev/null 2>&1; then
      echo "Added 'Include /etc/ssh/sshd_config.d/*.conf' to /etc/ssh/sshd_config (was missing; required for the port pin to take effect)"
      rm -f /etc/ssh/sshd_config.thorntech-preinclude
    else
      mv -f /etc/ssh/sshd_config.thorntech-preinclude /etc/ssh/sshd_config
      rm -f "$SSH_PIN_FILE"
      echo "WARNING: could not enable the sshd port-pin safety net (sshd -t rejected the Include directive). Continuing without it."
    fi
  fi
fi

# Record the installed package set before anything is upgraded.
[ "$RC" -eq 0 ] && snapshot_packages "$PKG_BEFORE"

# ----------------------------------------------------------------------------
# Patch — Ubuntu (apt)
#
# The apt options are load-bearing:
#   - DPkg::Lock::Timeout waits for unattended-upgrades (which runs by default
#     on these appliances) instead of failing on the dpkg lock.
#   - force-confdef/confold keep locally-modified config files (most
#     importantly /etc/ssh/sshd_config) even under an org-wide confnew policy,
#     and prevent interactive conffile prompts from killing the run.
# ----------------------------------------------------------------------------
if [ "$RC" -eq 0 ] && [ "$OS_FAMILY" = "ubuntu" ]; then
  export DEBIAN_FRONTEND=noninteractive
  APT_OPTS=(-y -o DPkg::Lock::Timeout=600
            -o Dpkg::Options::=--force-confdef
            -o Dpkg::Options::=--force-confold)

  apt_update "${APT_OPTS[@]}" || { RC=100; FAIL_REASON="apt-get update failed for an Ubuntu package source (see log)"; }
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" upgrade || { RC=100; FAIL_REASON="apt-get upgrade failed"; }; }
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" dist-upgrade || { RC=100; FAIL_REASON="apt-get dist-upgrade failed"; }; }
  [ "$RC" -eq 0 ] && { apt-get "${APT_OPTS[@]}" autoremove || true; }

# ----------------------------------------------------------------------------
# Patch — Amazon Linux 2023 (dnf)
#
# rpm marks sshd_config %config(noreplace), so dnf writes .rpmnew files rather
# than replacing edited configs — the apt confnew hazard does not exist here.
# The port pin drop-in is still installed as a structural guarantee.
# dnf waits on a busy RPM lock by itself; no timeout option is needed.
#
# --releasever=latest is load-bearing: AL2023 locks each instance to the
# repository snapshot it launched with, so a plain 'dnf upgrade' reports
# "nothing to do" forever — security updates only arrive by moving to the
# newest release snapshot.
# ----------------------------------------------------------------------------
elif [ "$RC" -eq 0 ] && [ "$OS_FAMILY" = "al2023" ]; then
  dnf -y upgrade --releasever=latest --refresh || { RC=100; FAIL_REASON="dnf upgrade failed"; }
fi

# What changed? Reported even if the package manager failed part-way, so the
# log shows exactly how far the run got.
if [ -s "$PKG_BEFORE" ]; then
  snapshot_packages "$PKG_AFTER"
  compute_package_changes
  print_package_changes
fi

if [ "$RC" -eq 0 ]; then
  restart_services_if_needed
fi

if [ "$RC" -eq 0 ] && reboot_required; then
  REBOOT_REQUIRED="yes"
fi

# Catch anything patching broke, while we can still alert about it. Retries
# ride out needrestart's automatic service restarts on Ubuntu (and the
# script's own restart above) — the java service takes a while to come back.
if [ "$RC" -eq 0 ]; then
  if ! check_appliance_health "POST" 12 10; then
    RC=4
    FAIL_REASON="Post-patch $PRODUCT_NAME health check failed (see health mail/log)"
  fi
fi

print_header "THORNTECH OS PATCH RUN END"
echo "Result      : $([ "$RC" -eq 0 ] && echo 'PATCH SUCCESS' || echo "PATCH FAILED (${FAIL_REASON:-exit $RC})")"
echo "Packages    : $PKG_SUMMARY"
echo "Service     : $SERVICE_ACTION"
echo "Sources     : ${APT_SOURCE_WARNINGS:+WARNING — skipped: }${APT_SOURCE_WARNINGS:-all package sources refreshed}"
if [ "$REBOOT_REQUIRED" = "yes" ]; then
  [ "$AUTO_REBOOT" = "yes" ] && echo "Reboot      : REQUIRED — rebooting in $REBOOT_DELAY_MINUTES minute(s)" || echo "Reboot      : REQUIRED — not performed (--no-reboot); reboot manually"
else
  echo "Reboot      : not required"
fi
echo "Full list   : $LOG_FILE (PACKAGE CHANGES section)"
echo

# ----------------------------------------------------------------------------
# Notifications and reboot handling
# ----------------------------------------------------------------------------
if [ "$RC" -ne 0 ]; then
  send_mail "[PATCH FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"PATCH FAILED

Host: $HOSTNAME_SHORT
Reason: $FAIL_REASON
Packages: $PKG_SUMMARY

Log: $LOG_FILE"
  exit "$RC"
fi

send_mail "[PATCH SUCCESS][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"PATCH SUCCESS

Host: $HOSTNAME_SHORT
Packages: $PKG_SUMMARY
Service: $SERVICE_ACTION
Sources: ${APT_SOURCE_WARNINGS:+skipped: }${APT_SOURCE_WARNINGS:-all refreshed}
Reboot required: $REBOOT_REQUIRED
Log: $LOG_FILE (full package list in the PACKAGE CHANGES section)"

if [ "$REBOOT_REQUIRED" = "yes" ]; then
  if [ "$AUTO_REBOOT" = "yes" ]; then
    echo
    echo "*** REBOOT REQUIRED to finish applying the updates. ***"
    echo "*** This server will reboot in $REBOOT_DELAY_MINUTES minute(s). SFTP sessions will drop and $PRODUCT_NAME will be"
    echo "*** unavailable for a minute or two, then start on its own. To cancel: sudo shutdown -c"
    echo
    if shutdown -r "+$REBOOT_DELAY_MINUTES" "ThornTech OS patch: rebooting to finish applying updates (cancel with: shutdown -c)"; then
      send_mail "[PATCH REBOOT SCHEDULED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"Patching succeeded and a reboot is required to finish applying updates.
Rebooting in $REBOOT_DELAY_MINUTES minute(s); $PRODUCT_NAME starts on its own afterwards.

Host: $HOSTNAME_SHORT
Packages: $PKG_SUMMARY
Log : $LOG_FILE"
    else
      send_mail "[PATCH REBOOT SCHEDULING FAILED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"Patching succeeded and a reboot is required, but scheduling the automatic
reboot FAILED. Reboot this host manually: sudo reboot

Host: $HOSTNAME_SHORT
Log : $LOG_FILE"
      exit 5
    fi
  else
    echo "Reboot required but NOT performed (--no-reboot). To finish applying the updates, run one of:"
    echo "  sudo reboot                               # now"
    echo "  sudo shutdown -r +15 \"OS patch reboot\"   # in 15 minutes, with a warning to logged-in users"
    echo "$PRODUCT_NAME starts on its own after the reboot; no service restart is needed."
    send_mail "[PATCH ACTION REQUIRED][$PRODUCT_NAME][$HOSTNAME_SHORT]" \
"REBOOT REQUIRED (manual — the run was started with --no-reboot)

Host: $HOSTNAME_SHORT
Time: $TIMESTAMP
Packages: $PKG_SUMMARY

The system requires a reboot to finish applying updates. In your change
window, run one of:

  sudo reboot
  sudo shutdown -r +15 \"OS patch reboot\"

$PRODUCT_NAME starts on its own after the reboot; no service restart is needed.

Log: $LOG_FILE"
  fi
fi

exit 0