Matrix Homeserver (Synapse)

Project: Synapse Host: synapse / 10.69.70.30 (LXC 730, DMZ / VLAN 70) OS: Debian 12 (LXC, unprivileged, keyctl=1) Server name: matrix.helix9.org Public URL: https://matrix.helix9.org Element URL: https://matrix.helix9.org/ Source role: roles/synapse/ in the Ansible repo

Overview

Self-hosted Matrix homeserver running as three podman Quadlet containers (postgres + synapse + element-web). TLS, federation, and .well-known delegation are handled by Traefik on 10.69.20.40 — there is no in-stack Caddy. External traffic enters via the VPS edge router (vyos-edge, 152.53.173.192) and reaches Traefik over the IPsec VPN.

Application service bridges (mautrix-meta, mautrix-discord, hookshot) connect on the local loopback at 127.0.0.1:8008. See Matrix Bridges.

Architecture

Internet
   │
   │ 443 / 8448
   ▼
vyos-edge (152.53.173.192)   ── NAT dst → 10.69.20.40 ──┐
   │                                                    │
   │ IPsec VPN (10.255.255.0/30)                        │
   ▼                                                    │
vyos-fw (home router)                                   │
   │                                                    │
   ▼                                                    │
┌─────────────── traefik (10.69.20.40) ──────────────┐  │
│  entrypoints:                                       │ │
│    websecure  :443  (matrix client + Element)       │◄┘
│    matrix-federation :8448                          │
│                                                     │
│  TLS: Let's Encrypt DNS-01 (Cloudflare)             │
│                                                     │
│  Routers (Host=matrix.helix9.org):                  │
│    /_matrix/*, /_synapse/*  → synapse:8008          │
│    /.well-known/matrix/*    → element-web:8088      │
│    /                        → element-web:8088      │
│    (federation entrypoint)  → synapse:8008          │
└─────────────────────────────────────────────────────┘
                  │                          │
                  ▼                          ▼
┌──────────── synapse (10.69.70.30, host net) ──────────┐
│  synapse-db.service     postgres:16-alpine  :5432     │
│  synapse.service        matrixdotorg/synapse :8008    │
│  element-web.service    vectorim/element-web :8088    │
│                                                       │
│  Bridges (loopback to 8008):                          │
│    mautrix-discord.service                            │
│    mautrix-meta.service                               │
│    hookshot  (separate LXC, see hookshot.md)          │
└───────────────────────────────────────────────────────┘

Containers (Quadlets)

Unit	Image	Listens	Notes
`synapse-db.service`	`docker.io/library/postgres:16-alpine`	`:5432`	Volume `/opt/matrix/postgres-data`; healthcheck `pg_isready`
`synapse.service`	`docker.io/matrixdotorg/synapse:latest`	`:8008`, `:8448` (host net, federation listener unused — Traefik does it)	Volume `/opt/matrix/synapse-data`; `Requires=synapse-db.service`
`element-web.service`	`docker.io/vectorim/element-web:latest`	`:8088`	Mounts `element-config.json` + `element-nginx.conf` (as nginx template); serves Element SPA + `.well-known/matrix/{server,client}`

All three are root podman containers managed via Quadlet files in /etc/containers/systemd/. Boot start: systemd enables the generated .service units (AutoUpdate=registry keeps images current).

Element-web nginx config

The element-web container's stock nginx is replaced by a small template that:

serves the Element SPA at /
returns the two .well-known/matrix/* JSON responses inline

The image's entrypoint runs envsubst against /etc/nginx/templates/default.conf.template, so the file is mounted at that path (not directly into conf.d/) to avoid the read-only-filesystem error.

Traefik configuration

Defined in roles/traefik/templates/services.yml.j2:

Entrypoint matrix-federation: ":8448" (added in host_vars/traefik/vars.yml)
Routers:
- matrix-synapse — Host + PathPrefix(/_matrix, /_synapse) → service synapse
- matrix-federation — Host on entrypoint matrix-federation → service synapse
- matrix-element — Host catch-all → service element-web
Services:
- synapse → http://10.69.70.30:8008
- element-web → http://10.69.70.30:8088
TLS via existing letsencrypt resolver (Cloudflare DNS-01).

Firewall (vyos-fw)

Direction	Policy	Rule	Detail
VPN → SERVERS	`VPN-SERVERS` 30	accept	VPS → `10.69.20.40:443,8448`
SERVERS → DMZ	`SERVERS-SCAN` 250	accept	Traefik → `10.69.70.30:8008,8088`

The previous VPN-DMZ rules 120/130/140 (direct VPS → synapse :443/:8448/:80) were removed.

VPS edge NAT (vyos-edge)

Rule	Public port	Translation
20	443	`10.69.20.40` (Traefik — matrix, openclaw, …)
21	8448	`10.69.20.40` (Traefik — matrix federation)
80	80	`10.69.20.40` (Traefik — HTTP→HTTPS redirect)

Old rules 22 (:80 → synapse) and 125 (disabled :443 → traefik) deleted. DNS-01 ACME removes the need for synapse to be reachable on port 80.

Ansible

Deploy:

ansible-playbook playbooks/synapse.yml --ask-vault-pass

Secrets (inventory/host_vars/synapse/vault.yml):

vault_synapse_pg_password — postgres password (matches existing data dir)

Defaults (roles/synapse/defaults/main.yml) cover image tags, paths, and the server name.

Manual operations

Task	Command
Tail synapse	`ssh synapse 'journalctl -fu synapse'`
Tail postgres	`ssh synapse 'journalctl -fu synapse-db'`
Restart stack	`ssh synapse 'systemctl restart synapse-db synapse element-web'`
Federation check	https://federationtester.matrix.org/#matrix.helix9.org
Client API probe	`curl https://matrix.helix9.org/_matrix/client/versions`
Well-known probe	`curl https://matrix.helix9.org/.well-known/matrix/server`

Troubleshooting

`Bad Gateway` on `/` or `/.well-known/matrix/*`

Element-web not running. Common cause: image entrypoint runs envsubst and tries to write /etc/nginx/conf.d/default.conf. Make sure the nginx config is mounted at /etc/nginx/templates/default.conf.template, not directly into conf.d/.

ssh synapse 'journalctl -u element-web -n 30 --no-pager'

Federation tester fails

Check :8448 reachable: curl -k https://matrix.helix9.org:8448/_matrix/federation/v1/version
Confirm VyOS edge rule 21 (:8448 → 10.69.20.40) and home-fw VPN-SERVERS rule 30 cover :8448
Confirm Traefik has the matrix-federation entrypoint and a router on it

Cert renewal failing

DNS-01 via Cloudflare. Check CF_DNS_API_TOKEN in the Traefik environment and journalctl -u traefik -n 100 for ACME errors. Cert lives in /etc/traefik/acme/acme.json.

Synapse won't start on boot

Containers must have systemd units (Quadlets) or podman-restart.service enabled. Pre-Quadlet, this stack was started by podman-compose with restart=always but no podman-restart.service → no boot start. The current Quadlet-based units are enabled by the role.

Matrix Voice/Video (Element Call) — LiveKit SFU + lk-jwt-service backend
Matrix Bridges — mautrix-meta, mautrix-discord
Hookshot — webhook bridge appservice
Traefik — reverse proxy + TLS
Home Router — firewall rules

Overview​

Architecture​

Containers (Quadlets)​

Element-web nginx config​

Traefik configuration​

Firewall (vyos-fw)​

VPS edge NAT (vyos-edge)​

Ansible​

Manual operations​

Troubleshooting​

Bad Gateway on / or /.well-known/matrix/*​

Federation tester fails​

Cert renewal failing​

Synapse won't start on boot​

Related​