Imported from alrcatraz/astra-sre (skills/astra-sre-synapse-maintenance/SKILL.md). Install upstream with npx skills add alrcatraz/astra-sre --skill astra-sre-synapse-maintenance. Copyright stays with the author.

Synapse Server Maintenance

1. When to Use

When performing Synapse/Matrix server maintenance — data migration, pre-upgrade backup, worker tuning, and rescue-mode recovery.

2. Pitfalls

(No known pitfalls for this skill.)

3. Use Cases

• Move Synapse's PostgreSQL data + media store to a separate data disk
• Fix "data directory must not be owned by root" / permission errors after migration
• Plan and execute Synapse version upgrades (apt-based Debian installations)
• Create pre-upgrade full backups of config + database + media

4. SSH Security Hardening (Initial Setup)

When setting up a new Synapse server, harden SSH access before anything else.

Generate SSH key and add to server

ssh-keygen -t ed25519 -C "hermes-agent@$(hostname)" -f ~/.ssh/id_ed25519 -N ""
ssh root@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
ssh-copy-id -i ~/.ssh/id_ed25519.pub root@server

Change SSH port (CRITICAL ORDER: firewall FIRST)

Add the firewall rule BEFORE changing the SSH config:

# 1. Check UFW status
ufw status verbose

# 2. Add new port to allow list FIRST
ufw allow 2222

# 3. Dual-port safety pattern — keep port 22 as fallback
sed -i '/^#\?Port 22/a\Port 2222' /etc/ssh/sshd_config
grep '^Port ' /etc/ssh/sshd_config  # Verify: Port 22 then Port 2222
sshd -t && systemctl restart sshd

# 4. Test new port BEFORE closing current session
ssh -p 2222 root@server "hostname"

Why dual-port: Keep port 22 alongside the new port. Even if the UFW rule for 2222 is lost, the server stays reachable on port 22.

Pitfall — UFW lockout: Changing SSHD port without adding the matching UFW allow rule = permanent lockout after reboot. UFW loads rules before SSHD starts.

Pitfall — MaxStartups rate limiting: Rapid SSH connections (common during automated repair) trigger MaxStartups throttling. Both ports refuse all connections for ~2 minutes. Wait and retry; use ControlMaster to batch operations.

Disable password auth (after confirming key auth works)

sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

5. Architecture Overview

Nginx (port 443/8448) → Synapse (port 8008 localhost) → PostgreSQL (port 5432)
                                                      → Media Store (local filesystem)

Synapse runs under systemd as matrix-synapse.service with optional worker processes (matrix-synapse-generic-worker-*).

6. Related Skills

• autonomous-ai-agents/hermes-agent → references/matrix-gateway-setup.md — Hermes Matrix platform configuration, E2EE cross-signing troubleshooting

7. Absorbed Skills

The following standalone skills have been consolidated into this umbrella — their content was already present in or is a subset of the main skill:

8. Reference Files

• references/pre-upgrade-audit-example.md — Full real-world example of an 8-service pre-upgrade audit (source VPS → target VPS, 3 overlay networks, rescue-mode recovery)
• references/filesystem-selection-guide.md — Filesystem comparison (ext4 / xfs / btrfs / zfs) for a Synapse/PostgreSQL data disk on a small HDD-backed VPS
• references/sliding-sync-performance-diagnostics.md — SQL queries, access-log analysis, and tuning for low-resource servers
• references/postgresql-ssd-tuning.md — SSD vs HDD detection, 5-parameter tuning
• references/sync-intermittent-failure-diagnosis.md — Diagnosis guide for "desktop Element spins but phone gets push notifications": interpret Cloudflare 502 + SSH "exec request failed on channel 0" as VPS memory exhaustion/OOM, with triage steps, remote API diagnosis, memory tuning for 4GB VPS, and post-reboot OOM cascade recovery (shared_buffers, random_page_cost, effective_io_concurrency, work_mem, maintenance_work_mem), ALTER SYSTEM quoting trap, restart impact on Synapse
• references/common-missing-configs.md — User directory, rate limiting, media cleanup, push notifications, and Sliding Sync configuration examples
• references/bot-account-management.md — Bot password reset, access token generation, device key cleanup, and registration_shared_secret HMAC details
• references/state-explosion-compression.md — Synapse state_groups_state explosion diagnosis (27:1 ratio to events), compression via admin_compress_state, and PostgreSQL tuning for HDD-backed servers

9. Scripts

• scripts/purge-remote-media.sh — Weekly cron script to purge remote media cache older than 90 days

10. Data Migration: PostgreSQL + Media to Separate Disk

Prerequisites

• Identify disk layout: lsblk, df -h
• Data disk should be mounted (e.g. /www) with correct fstab entry
• Synapse version: dpkg -l matrix-synapse-py3

Step 1: Create directories

mkdir -p /www/matrix/postgresql /www/matrix/media_store

Step 2: Backup and stop services

systemctl stop matrix-synapse postgresql

# Backups at original locations
cp -a /var/lib/postgresql/15/main /var/lib/postgresql/15/main.bak
cp -a /etc/matrix-synapse/media_store /etc/matrix-synapse/media_store.bak
cp /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/homeserver.yaml.bak

Step 3: Copy data to new disk

cp -a /var/lib/postgresql/15/main/* /www/matrix/postgresql/
cp -a /etc/matrix-synapse/media_store/* /www/matrix/media_store/

Step 4: Fix permissions

Critical! Every directory level must allow the owner user to traverse:

chown -R postgres:postgres /www/matrix/postgresql/
chown -R matrix-synapse:matrix-synapse /www/matrix/media_store/
chmod 0700 /www/matrix/postgresql/
chmod 0751 /www/matrix/       # must be traversable by both postgres AND matrix-synapse

Step 5: Update configs

PostgreSQL: Update data_directory in /etc/postgresql/15/main/postgresql.conf:

sed -i "s|^data_directory = '/var/lib/postgresql/15/main'|data_directory = '/www/matrix/postgresql'|" /etc/postgresql/15/main/postgresql.conf

Synapse: Update media_store_path in /etc/matrix-synapse/homeserver.yaml:

sed -i 's|media_store_path: /etc/matrix-synapse/media_store|media_store_path: /www/matrix/media_store|' /etc/matrix-synapse/homeserver.yaml

Step 6: Start services and verify

# For PostgreSQL: must use pg_ctl directly (systemd startup may fail due to Debian pg_ctlcluster checks)
su - postgres -c '/usr/lib/postgresql/15/bin/pg_ctl start -D /www/matrix/postgresql -l /var/log/postgresql/pg-start.log -w'

# Verify
pg_isready
su - postgres -c 'psql -d matrixdb -c "SELECT count(*) FROM access_tokens;"'

# Start Synapse
systemctl start matrix-synapse
systemctl is-active matrix-synapse

Step 7: Clean up old backups (after verification)

rm -rf /var/lib/postgresql/15/main /var/lib/postgresql/15/main.bak
rm -rf /etc/matrix-synapse/media_store /etc/matrix-synapse/media_store.bak

11. Pre-Upgrade Full Backup

Critical rule: Never set the backup target to the same machine being upgraded. If the upgrade resets the system or data disk, the backup goes with it. Use a separate machine in a different region/availability zone — see Backup target evaluation below.

Phase 1 — Service discovery (before touching any backup)

Servers almost always host more than Synapse: Docker containers, VPN control consoles, reverse proxies, cron jobs, databases inside containers. All need backing up. Do not assume Synapse is alone.

echo '=== ALL LISTENING PORTS ==='
ss -tlnp | sort -n -t: -k2

echo '=== ALL DOCKER CONTAINERS ==='
docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'

echo '=== ALL SYSTEMD SERVICES (non-default) ==='
systemctl list-units --type=service --state=running --no-pager | head -40

echo '=== CRONTAB ==='
crontab -l 2>/dev/null

echo '=== NGINX CONFIG SITES ==='
grep -r 'server_name' /etc/nginx/conf.d/ 2>/dev/null | grep -v '#'

echo '=== DISK LAYOUT ==='
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE

From the port list, identify every service. Common co-located services to watch for:

Record the results — you'll use them as the backup checklist.

Phase 2 — Backup target evaluation

Check every candidate destination's available space before proposing it:

echo '=== CANDIDATE: remote VPS ==='
ssh root@<target-ip> 'df -h /'

echo '=== CANDIDATE: local storage ==='
# Adapt per access method
ssh root@<target-ip> 'df -h /'

echo '=== CANDIDATE: Local mini PC ==='
df -h /

Selection criteria:

• ❌ Same machine — worthless (loses data with the upgrade)
• ❌ Same cloud provider, same region — correlated failure risk
• ❌ Home LAN machine via HK→China link — large encrypted transfers may trigger ISP monitoring; use only as last resort
• ✅ Different provider, different region, server-to-server — zero ISP risk, no correlated failure

Preference order: cross-region VPS > object storage (S3/B2) > local storage > same-machine local copy (accept as temporary staging only).

Phase 3 — Create backup archives

Back up in order of criticality. Keep each archive separate for targeted restore.

#!/bin/bash
# pre-upgrade-backup.sh — run on the source server
# Requires: ssh key to remote target already configured

BACKUP_TARGET=/www/matrix/backup-pre-upgrade
mkdir -p "$BACKUP_TARGET"

# 1. PostgreSQL SQL dump (cleanest restore path, works across PG versions)
su - postgres -c 'pg_dump -d matrixdb --no-owner --no-acl | gzip' \
  > "$BACKUP_TARGET/matrixdb-$(date +%Y%m%d_%H%M%S).sql.gz"

# 2. Raw PostgreSQL data directory (alternative restore path, saves replay time)
#    Only if you have space. The SQL dump above is sufficient.
# tar czf "$BACKUP_TARGET/postgresql-data-$(date +%Y%m%d_%H%M%S).tar.gz" /www/matrix/postgresql/

# 3. Synapse config + workers + signing key
#    ⚠ signing key is CRITICAL — without it, federation peers reject the server
tar czf "$BACKUP_TARGET/synapse-config-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /etc/matrix-synapse/ \
  /etc/systemd/system/matrix-synapse*.service \
  /etc/default/matrix-synapse

# 4. Media store + media archive (user uploads — as important as the database)
tar czf "$BACKUP_TARGET/media-store-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /www/matrix/media_store/ \
  /www/matrix/media_archive/

# 5. Nginx configs + SSL certs (all proxied domains)
tar czf "$BACKUP_TARGET/nginx-config-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /etc/nginx/
tar czf "$BACKUP_TARGET/ssl-certs-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /etc/letsencrypt/

# 6. System configs (essential for restoring to a clean OS)
tar czf "$BACKUP_TARGET/system-configs-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /etc/fstab \
  /etc/hostname \
  /etc/hosts \
  /etc/ufw/ \
  /etc/ssh/sshd_config \
  /etc/resolv.conf \
  /etc/apt/sources.list.d/

# 7. Docker service configs + their internal databases
tar czf "$BACKUP_TARGET/docker-services-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /root/docker-zerotier-controller/ \
  /root/ztnet/ \
  /root/data/docker_data/

# 8. Docker-internal databases (clean dumps for each)
for container in ztnet-db; do
  docker exec "$container" pg_dump -U postgres --all --no-owner --no-acl 2>/dev/null | \
    gzip > "$BACKUP_TARGET/${container}-dump-$(date +%Y%m%d_%H%M%S).sql.gz"
done

# 9. EasyTier control console SQLite database
tar czf "$BACKUP_TARGET/easytier-data-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /opt/easytier/web/et.db* \
  /etc/systemd/system/easytier*.service

# 10. Custom scripts + crontab
tar czf "$BACKUP_TARGET/custom-scripts-$(date +%Y%m%d_%H%M%S).tar.gz" \
  /root/*.sh \
  /var/spool/cron/crontabs/root

Phase 4 — Bandwidth measurement

Before starting a long transfer, measure the actual throughput between source and target.

# Create a small test file (5MB is enough)
dd if=/dev/zero of=/tmp/speedtest.dat bs=1M count=5

# Time the SCP transfer
time scp -o StrictHostKeyChecking=no /tmp/speedtest.dat root@<target-ip>:/tmp/

# Calculate: 5 MB / elapsed_seconds = MB/s
# Example: 5 MB / 12.6 s ≈ 0.4 MB/s (3.2 Mbps)

rm -f /tmp/speedtest.dat
ssh root@<target-ip> 'rm -f /tmp/speedtest.dat'

Use the result to decide transfer strategy:

• < 10 min → foreground
• 10–60 min → background with nohup + status file (see Phase 6)
• > 60 min → chunk the backup, or use rsync --partial for resume

Phase 5 — Local archive verification

Before transferring, verify every archive is valid:

for f in /var/backups/synapse/*.tar.gz; do
  tar tzf "$f" > /dev/null 2>&1 && echo "✅ $f" || echo "❌ CORRUPT: $f"
done
for f in /var/backups/synapse/*.sql.gz; do
  gunzip -t "$f" > /dev/null 2>&1 && echo "✅ $f" || echo "❌ CORRUPT: $f"
done

Phase 6 — Transfer

Fast link (rsync available): Use --partial for auto-resume:

rsync -avzP --partial /var/backups/synapse/ root@<target-ip>:/root/backup/

Slow link, rsync unavailable: Background SCP with status tracking:

nohup bash -c '
  scp -o StrictHostKeyChecking=no /var/backups/synapse/full-backup-*.tar.gz \
    root@<target-ip>:/root/backup/ && \
  echo DONE > /tmp/backup_status.txt || \
  echo FAIL > /tmp/backup_status.txt
' > /tmp/scp_backup.log 2>&1 &

# Track progress
ssh root@<target-ip> 'ls -lh /root/backup/full-backup-*.tar.gz'
ps -o etime= -p <PID>

Phase 7 — Remote integrity verification (MANDATORY)

A transfer that completes silently may still be corrupt. Always verify checksums after transfer.

# Verify a single file
SRC=$(md5sum /var/backups/synapse/matrixdb-*.sql.gz | awk '{print $1}')
DST=$(ssh root@<target-ip> 'md5sum /root/backup/matrixdb-*.sql.gz' | awk '{print $1}')
[ "$SRC" = "$DST" ] && echo "✅ DB dump intact" || echo "❌ MISMATCH"

# Verify all small files in batch
for f in synapse-config nginx-config docker-ztnet; do
  SRC=$(md5sum /var/backups/synapse/${f}-*.tar.gz 2>/dev/null | awk '{print $1}')
  DST=$(ssh root@<target-ip> "md5sum /root/backup/${f}-*.tar.gz 2>/dev/null" | awk '{print $1}')
  [ "$SRC" = "$DST" ] && echo "✅ $f" || echo "❌ $f"
done

Phase 8 — Restore verification checklist

After transfer, confirm on the remote that every service can be rebuilt from the backup:

If any item on this checklist could not be verified, the backup is incomplete — fix the gap before proceeding with the upgrade.

12. Post-Upgrade Recovery (Rescue Mode)

When a partition/filesystem change during upgrading prevents normal boot, use the provider's rescue/recovery mode. Key recovery sequence:

Phase 1 — Access the rescue system

SSH to the rescue system using the provided temporary password. Disk device names may differ from normal boot:

# Normal boot:         Rescue mode:
# /dev/vda = system    /dev/vdb = system  (mount to /mnt/root)
# /dev/vdb = data      /dev/vdc = data    (mount to /mnt/data)

mount /dev/vdb1 /mnt/root      # original system disk
mount /dev/vdc1 /mnt/data       # original data disk

Phase 2 — Fix filesystem issues

If the partition was resized but not formatted, do so now:

mkfs.xfs -f /dev/vdc1                       # new filesystem
VDB_UUID=$(blkid -s UUID -o value /dev/vdc1)
sed -i "s|/dev/vdb1[[:space:]].*/www.*|UUID=$VDB_UUID  /www    xfs    defaults    0 0|" /mnt/root/etc/fstab

Phase 3 — Prevent SSH lockout

Disable UFW and add a fallback SSH port:

sed -i 's/ENABLED=yes/ENABLED=no/' /mnt/root/etc/ufw/ufw.conf
sed -i '/^Port 2222/c\Port 22\nPort 2222' /mnt/root/etc/ssh/sshd_config

Phase 4 — Restore data

Rescue mode has no overlay-network connectivity (EasyTier/ZeroTier/Tailscale). Use jump hosts or public-IP paths to pull the backup. See the multi-hop relay pattern in Pitfalls below.

Phase 5 — Boot and verify

Exit rescue mode → normal boot → verify all services:

systemctl enable postgresql@15-main          # ensure PG auto-starts
systemctl start matrix-synapse
systemctl is-active matrix-synapse nginx postgresql@15-main

Pitfalls

1. Backing up to the same machine — if the upgrade reformats the data disk, the backup vanishes. Always pick a different machine.
2. Missing the signing key — the .signing.key file is tiny (~60 bytes) but without it, Matrix federation treats the server as a different identity. This is the single most critical file after the database itself.
3. Skipping Docker-internal databases — containers like ZTNet run their own PostgreSQL/MariaDB. The compose files are useless without the data inside them. Dump each container DB separately.
4. Not checking capacity before proposing targets — always run df -h on the candidate before committing to it. A machine with 5GB free cannot receive a 1.3GB backup comfortably.
5. Forgetting VPN control-console data — EasyTier, ZeroTier web UIs store network topology and user registrations in SQLite. If the server is the only place this data lives, losing it means rebuilding the VPN mesh.
6. Not exporting UFW rules when server has custom iptables — ufw status numbered shows active rules but the raw files live in /etc/ufw/. Back them up as part of system-configs.
7. Backing up deprecated/unused services — scan docker ps -a for stale containers with no running process. If a Docker image exists but its port isn't listening and no container runs, verify whether it's still needed before backing it up. Including it wastes space and complicates restore.
8. Assuming only one network path — servers often have EasyTier, ZeroTier, Tailscale, or WireGuard simultaneously. Test each path's latency and jitter before picking one. The first route that works may be the most unstable.
9. Skipping bandwidth measurement — a 1.1GB file at 0.4 MB/s takes 46 minutes, not "a few minutes". Measure before telling the user an estimate.
10. Trusting transfer success without MD5 — scp can report success even when the file is truncated (e.g., remote disk full mid-write). Always compare checksums between source and destination for every file.

A real-world example of this workflow (source VPS → target VPS, 3 overlay networks, 8 services, 1 deprecated service removed, plus full rescue-mode recovery when the filesystem change went wrong) is documented in references/pre-upgrade-audit-example.md.

11. /tmp is tmpfs on modern distros — backing up to /tmp is equivalent to backing up to RAM. The data disappears on reboot. Always use a persistent filesystem (e.g. /var/backups/, the data disk's backup directory, or directly SCP to the remote target). If you must use /tmp as a staging area, verify the data survives a sync + reboot cycle.

12. Rescue mode = no overlay networks — EasyTier, ZeroTier, Tailscale, and WireGuard are unavailable during rescue-mode boots. Only public-IP connectivity works. Plan for this when the backup target only has an overlay-network address (e.g. 10.x.x.x). Have a fallback path through a jump host that IS publicly reachable, or ensure a direct public-IP route exists to the backup target.

13. nginx upstream localhost may resolve to ::1 while Synapse workers bind 127.0.0.1 — Synapse workers listen on IPv4 only (bind_addresses: ['127.0.0.1']), but nginx upstream blocks use server localhost:8081 / proxy_pass http://localhost:8081. If the system resolver returns ::1 before 127.0.0.1, nginx connects to [::1]:8081 → Connection refused even though the worker is alive. Symptom: intermittent "sync slow" on mobile clients — the client hits Connection-refused, retries with pos=0 full resync, and waits 30s per attempt.

    Two fix approaches (pick based on context):

    A) Change nginx upstream to explicit IPv4 (zero-downtime, minimal impact):

    - server localhost:8081;
    + server 127.0.0.1:8081;

    Check ALL upstream blocks in upstream.conf AND all proxy_pass directives in worker.conf. Then nginx -t && nginx -s reload.

    B) Add ::1 to worker bind_addresses (root cause fix — makes workers accept both families):

    # In each worker YAML under /etc/matrix-synapse/workers/:
    #   bind_addresses: ['127.0.0.1']  →  bind_addresses: ['127.0.0.1', '::1']
    # Then restart the workers:
    for s in matrix-synapse-worker1 matrix-synapse-worker2 matrix-synapse-worker3; do
      systemctl restart "$s"
    done

    Diagnostic confirmation (either approach):

    journalctl -u nginx --since "5 min ago" | grep "connect() failed (111: Connection refused)"

    Should drop to zero after the fix takes effect (allow ~30s for nginx resolver cache expiry).

14. Multi-hop data relay — when source and destination cannot reach each other (no overlay, no public IP, or firewall blocking), use an intermediate machine that can reach both. Pattern:

    # On relay machine (must have scp/ssh access to both sides):
    sshpass scp from=source file.tar.gz relay:/tmp/
    ssh relay "sshpass scp /tmp/file.tar.gz dest:/tmp/"

    This is slower than direct transfer but works when all other paths are blocked. Pre-measure bandwidth on each leg so the user can decide whether to proceed.

13. Upgrade Synapse (apt-based)

Check available versions

apt-cache policy matrix-synapse-py3

Review breaking changes

https://element-hq.github.io/synapse/latest/upgrade.html

Key checks before upgrading:

• Python version (1.118 requires 3.10+)
• PostgreSQL version
• Deprecated config options (worker_replication_*, etc.)
• Room list publication rules changes
• Authenticated media defaults

Execute upgrade

apt update && apt upgrade matrix-synapse-py3 -y

Synapse automatically runs database migrations on startup. Direct jump between versions is supported (official docs: "you do not need to perform an upgrade to each individual version that was missed").

Pitfall: After this fix, also verify that the @ unit starts before Synapse in the boot order. If Synapse starts first and can't connect to the DB, it may need a manual restart: systemctl restart matrix-synapse.

SSH MaxStartups rate limiting (all connections refused after rapid retries)

When making many rapid SSH connections (e.g. during debugging, backup verification, or auto-repair scripts), SSHD's MaxStartups setting kicks in and refuses all new connections on every port — including the fallback port. Symptoms:

ssh: Connection closed by <host> port 2222
ssh: Connection closed by <host> port 22

Both ports report "Connection closed" immediately, but the server IS up (ping works). This is not a UFW or SSHD failure — it's MaxStartups rate limiting.

Recovery: Wait ~2 minutes for existing unauthenticated connection slots to time out. The default MaxStartups 10:30:100 means up to 10 connections pass freely, then 30% of further connections are dropped, then all new connections after ~100 are dropped. After the 120s SSH handshake timeout, connections resume normally.

Prevention: Use SSH ControlMaster (ControlMaster auto; ControlPath ~/.ssh/cm-%r@%h:%p; ControlPersist 10m in ~/.ssh/config) or batch multiple operations into fewer SSH calls instead of many discrete connections.

Multi-hop data relay (rescue mode without overlay networks)

When a server is in rescue mode, overlay networks (EasyTier, ZeroTier, Tailscale, WireGuard) are unavailable. If the backup target only has an overlay-network address:

Pattern: Source (rescue) ← relay jump host ← backup target

1. Ensure the relay host can reach both source (via public IP) and target (via whichever path works)
2. Transfer from target → relay: sshpass scp target:/path/file relay:/tmp/
3. From relay → rescue: sshpass scp relay:/tmp/file rescue:/tmp/
4. Optionally verify with MD5 after transfer completes

This is slower than direct transfer (each leg may have different bandwidth) but works when no direct path exists.

Synapse fails with "IncorrectDatabaseSetup: collation ... Should be 'C'"

When restoring a PostgreSQL dump into a freshly initialized database cluster, Synapse reports IncorrectDatabaseSetup: Database has incorrect collation of 'en_US.UTF-8'. Should be 'C'.

Quick fix — add allow_unsafe_locale: true to the Synapse database config (safe for personal/small servers):

# Under the `args:` block in homeserver.yaml:
database:
  args:
    password: ...
    allow_unsafe_locale: true    # ← add this

Proper fix — re-initialize the database cluster explicitly with the C locale before restoring:

su - postgres -c "/usr/lib/postgresql/15/bin/initdb \
  -D /www/matrix/postgresql --locale=C --encoding=UTF8"

"data directory must not be owned by root"

The data directory or the symlink to it is owned by root. Use chown to set postgres:postgres.

"could not access configuration file" / "conf.d not found"

Debian's PostgreSQL stores config in /etc/postgresql/15/main/, not in the data directory. Create symlinks:

ln -s /etc/postgresql/15/main/postgresql.conf /www/matrix/postgresql/postgresql.conf
ln -s /etc/postgresql/15/main/pg_hba.conf /www/matrix/postgresql/pg_hba.conf
mkdir -p /www/matrix/postgresql/conf.d

"has invalid permissions"

PostgreSQL requires data directory permissions 0700 or 0750:

chmod 0700 /www/matrix/postgresql/

Synapse can't connect to PostgreSQL

Ensure /www/matrix/ is traversable (0751) by both matrix-synapse and postgres users.

Use namei -l /www/matrix/postgresql/ to check every directory level's permissions.

SSH MaxStartups rate limiting (all connections refused after rapid retries)

When making many rapid SSH connections (e.g. during debugging, backup verification, or auto-repair scripts), SSHD's MaxStartups setting kicks in and refuses all new connections on every port — including the fallback port. Symptoms:

ssh: Connection closed by <host> port 2222
ssh: Connection closed by <host> port 22

Both ports report "Connection closed" immediately, but the server IS up (ping works). This is not a UFW or SSHD failure — it's MaxStartups rate limiting.

Recovery: Wait ~2 minutes for existing unauthenticated connection slots to time out. The default MaxStartups 10:30:100 means:

• Up to 10 connections: no restriction
• 11-100 connections: 30% of new connections are randomly dropped
• Beyond 100: all new connections are dropped

Each dropped connection consumes a slot until the SSH handshake timeout (~120s default). After the wait, connections resume normally.

Prevention: Use a single persistent SSH control socket instead of many discrete connections:

# ~/.ssh/config
Host target-server
  HostName <target-vps-ip>
  Port 2222
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 10m

Or batch operations into fewer SSH calls. Each Hermes terminal() call opens a new connection — avoid running dozens of quick commands when a single compound command would do.

E2EE cross-signing fails after database restore

After restoring the Synapse database from a SQL dump, the Hermes Gateway's E2EE cross-signing may fail with recovery key verification failed or Invalid signature for <device>.

Root cause: The recovery key in .env was generated for a previous cross-signing session. After the database restore, the old cross-signing keys in the DB were either stale or deleted, leaving the recovery key with nothing to recover.

Fix — full wipe+rebuild procedure:

# 1. Delete ALL bot crypto keys from Synapse DB (via psql on the server)
#    Target ALL six tables that may hold stale keys:
DELETE FROM e2e_cross_signing_keys WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_cross_signing_signatures WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_one_time_keys WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_fallback_keys WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_device_keys WHERE user_id = '@bot:your-domain';
DELETE FROM access_tokens WHERE user_id = '@bot:your-domain';

# 2. Also delete the _json variant tables (these hold the raw uploaded key data)
DELETE FROM e2e_one_time_keys_json WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_fallback_keys_json WHERE user_id = '@bot:your-domain';
DELETE FROM e2e_device_keys_json WHERE user_id = '@bot:your-domain';

# 3. CRITICAL — use a NEVER-BEFORE-SEEN device ID
#    Any previously used device ID (even if deleted from DB) may have
#    orphaned one-time keys cached by the server. A fresh ID guarantees
#    no OTK conflicts. Append the current Unix timestamp.
NEW_DEVICE="hermes-bot-v$(date +%s)"

# 4. Generate a new access token bound to the fresh device ID
python3 -c "
import urllib.request, json
data = json.dumps({
    'type': 'm.login.password',
    'user': '@bot:your-domain',
    'password': 'the-bot-password',
    'device_id': '$NEW_DEVICE',
    'initial_device_display_name': 'Hermes Gateway (machine-name)'
}).encode()
req = urllib.request.Request('http://127.0.0.1:8008/_matrix/client/v3/login',
    data=data, headers={'Content-Type': 'application/json'})
r = json.loads(urllib.request.urlopen(req).read())
open('/tmp/new_token', 'w').write(r['access_token'] + '\n')
open('/tmp/new_device', 'w').write(r['device_id'] + '\n')
print(r['device_id'])
"

# 5. Comment out MATRIX_RECOVERY_KEY in .env (so gateway generates fresh keys)
sed -i 's/^MATRIX_RECOVERY_KEY/# MATRIX_RECOVERY_KEY/' ~/.env

# 6. Read token from server and write to local .env
#    Copy /tmp/new_token from server, then:
hermes config set MATRIX_ACCESS_TOKEN 'the-new-token'
hermes config set MATRIX_DEVICE_ID '$NEW_DEVICE'

# 7. Clear local crypto store and restart gateway
rm -f ~/.config/gateway/store/crypto.db*
systemctl --user restart hermes-gateway

After fresh gateway start: User must log into bot account via Element Web (desktop) → Settings → Security & Privacy → set up Security Phrase → generate Recovery Key → verify the gateway device. This is the only reliable method when UIA blocks code-side cross-signing bootstrap. Save the new recovery key to .env.

Encrypted messages from before the restore cannot be decrypted (old Megolm sessions are gone).

systemd postgresql fails after migration

The pg_ctlcluster wrapper may fail; use direct pg_ctl start -D as the postgres user instead. This is a Debian packaging quirk — the running instance will be tracked correctly despite systemd showing failed.

Worker Reduction for Memory-Constrained Servers (4GB or less)

When running Synapse with multiple workers on a small VPS (4GB RAM), each generic worker consumes ~130MB RSS. Reducing workers saves memory:

Step 1: Identify current worker layout

# List worker config files
ls /etc/matrix-synapse/workers/

# Check which systemd units exist
systemctl list-units --type=service | grep synapse

# Check nginx upstream configuration
cat /etc/nginx/snippets/upstream.conf

A typical layout has 3+ workers:

Step 2: Redirect traffic from removed worker

Edit upstream.conf (or wherever your upstream blocks are):

# Change generic_worker_lc to point at worker2 instead of worker3
sed -i 's/server localhost:8083;/server localhost:8082;/' /etc/nginx/snippets/upstream.conf

# Verify
grep 'server localhost:808' /etc/nginx/snippets/upstream.conf
# Expected output shows 8081, 8082, 8082 (no 8083)

Step 3: Test nginx config and reload

nginx -t && nginx -s reload

Step 4: Stop and disable the removed worker

systemctl stop matrix-synapse-worker3
systemctl disable matrix-synapse-worker3

Step 5: Verify

systemctl is-active matrix-synapse-worker1 matrix-synapse-worker2
# Both should show "active"
systemctl is-active matrix-synapse-worker3
# Should show "inactive"

Expected memory saving: ~130MB per worker removed. For a personal server with 1-2 users, 2 generic workers + 1 federation sender is sufficient.

Pitfall: The worker config files (.yaml under /etc/matrix-synapse/workers/) remain on disk even when the worker is disabled — this is harmless and allows easy re-enablement. Do NOT delete them unless you're permanently removing the worker.

CRITICAL: Disabling a worker via systemctl is NOT sufficient. You MUST also update homeserver.yaml to remove the worker from instance_map and stream_writers.events. Otherwise, Synapse continues to route events to the disabled worker's port, causing ALL encrypted sends to hang:

# Fix: use Python to edit YAML properly (sed will break indentation)
python3 -c "
import yaml
with open('/etc/matrix-synapse/homeserver.yaml') as f:
    config = yaml.safe_load(f)
config['instance_map'].pop('generic_worker3', None)
config['stream_writers']['events'] = [
    w for w in config['stream_writers']['events'] if w != 'generic_worker3'
]
with open('/etc/matrix-synapse/homeserver.yaml', 'w') as f:
    yaml.dump(config, f, default_flow_style=False, sort_keys=False)
"
systemctl restart matrix-synapse-worker1 matrix-synapse-worker2

Diagnosis: Compare systemctl list-units --type=service --state=running | grep synapse against grep -E 'generic_worker[0-9]:' homeserver.yaml. If a worker is in YAML but not running, that is the root cause of all encrypted send hangs.

14. zram swap setup (modern alternative to disk swapfile)

After upgrading server RAM, replace the traditional disk swapfile with zram (compressed RAM swap) for much faster performance. On Debian 12:

# Install
apt install systemd-zram-generator

# Configure — 2GB zstd compression, high priority
cat > /etc/systemd/zram-generator.conf << 'EOF'
[zram0]
zram-size = 2048
compression-algorithm = zstd
swap-priority = 100
EOF

# Enable
systemctl daemon-reload
systemctl start systemd-zram-setup@zram0

# Verify
zramctl        # shows algorithm, disk size, compression ratio
swapon --show  # /dev/zram0 should appear
free -h        # shows RAM + zram

# Optionally disable old disk swap
swapoff /swapfile && sed -i "/swapfile/d" /etc/fstab

Why zram: On a RAM-constrained server with zram (zstd), the effective swap capacity is substantially larger than the allocated zram size (2:1 to 3:1 compression) entirely in RAM. This is orders of magnitude faster than disk swap, which on a cloud HDD adds 5−10ms latency per page fault.

15. zram + swappiness Companion Tuning

After setting up zram, tune the kernel to use it proactively — zram is RAM-speed, so higher swappiness is safe and prevents OOM.

cat > /etc/sysctl.d/99-zram.conf << 'EOF'
vm.swappiness=150
vm.vfs_cache_pressure=200
EOF
sysctl -p /etc/sysctl.d/99-zram.conf

Why high swappiness with zram: Unlike disk swap, zram compresses pages in RAM. Reading a zram page is ~10µs (same as RAM), not 5-10ms (disk). Proactively swapping cold pages to zram frees physical RAM for hot cache — a net win for performance.

Verify

sysctl vm.swappiness vm.vfs_cache_pressure

16. Synapse Memory Cache Tuning

Synapse's default cache autotune (autotune_memory_cache_ratio: 0.5) allows it to consume up to 50% of available memory for caches. On small VPS (4GB or less), this combines poorly with multiple workers, PostgreSQL, and system services — leading to OOM and SSH lockout (exec request failed on channel 0).

When to tune

• VPS with ≤4GB RAM running Synapse + workers + PostgreSQL
• Repeated "Dropped 0 items from caches" in Synapse logs (cache full but cannot evict)
• SSH authentication succeeds but commands fail with exec request failed on channel 0 (OOM)
• Cloudflare 502 errors suggesting upstream timeouts (nginx → Synapse)

Tuning parameters

Add to homeserver.yaml (before the rate-limiting section):

# Memory cache tuning (4GB server)
caches:
  autotune_memory_cache_ratio: 0.3
  cache_factor: 0.5

Insert before the rc_* rate limiting section using sed:

# Backup first
cp /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/homeserver.yaml.bak.$(date +%Y%m%d_%H%M%S)

# Find the rate-limiting line and insert before it
LINE=$(grep -n '^rc_message:' /etc/matrix-synapse/homeserver.yaml | cut -d: -f1)
head -n $((LINE-1)) /etc/matrix-synapse/homeserver.yaml > /tmp/homeserver_new.yaml
printf '\n# 内存缓存调优（4GB 服务器）\ncaches:\n  autotune_memory_cache_ratio: 0.3\n  cache_factor: 0.5\n\n' >> /tmp/homeserver_new.yaml
tail -n +$LINE /etc/matrix-synapse/homeserver.yaml >> /tmp/homeserver_new.yaml
mv /tmp/homeserver_new.yaml /etc/matrix-synapse/homeserver.yaml

Verify

grep -A4 'caches:' /etc/matrix-synapse/homeserver.yaml

Restart Synapse to apply: systemctl restart matrix-synapse

17. Systemd Memory Limits (Prevent OOM Lockout)

Without limits, Synapse can consume all available RAM and prevent SSH from forking — even though SSH authentication succeeds, the server cannot execute commands (symptom: exec request failed on channel 0). A systemd drop-in prevents this.

Create memory limit drop-in

mkdir -p /etc/systemd/system/matrix-synapse.service.d
cat > /etc/systemd/system/matrix-synapse.service.d/memory-limit.conf << 'EOF'
[Service]
MemoryHigh=2.5G
MemoryMax=3G
EOF
systemctl daemon-reload
systemctl restart matrix-synapse

On a typical VPS with zram, this ensures:

• Synapse capped at 3G → leaves headroom for PG, Docker, TS, next-server
• SSH stays responsive even under heavy load

Verify

systemctl show matrix-synapse -p MemoryCurrent -p MemoryHigh -p MemoryMax
systemctl is-active matrix-synapse
curl -s --max-time 5 -o /dev/null -w "Matrix API: HTTP %{http_code}\n" \
  "https://matrix.your.domain/_matrix/client/versions"

PostgreSQL auto-start after reboot

On Debian 12, the postgresql.service unit is an empty wrapper that runs /bin/true. The actual service is postgresql@15-main.

# Check current auto-start state
systemctl is-enabled postgresql@15-main
# → shows "enabled-runtime" (only this boot) or "disabled"

# Set permanent auto-start
systemctl enable postgresql@15-main

# Verify
systemctl is-enabled postgresql@15-main
# → should show "enabled"

Without this fix, PostgreSQL (and therefore Synapse) will not start after a server reboot, even though postgresql.service shows active (exited) — it only asserts that the @ unit was called, not that it was enabled.

Pitfall: After this fix, also verify that the @ unit starts before Synapse in the boot order. If Synapse starts first and can't connect to the DB, it may need a manual restart: systemctl restart matrix-synapse.