IPsec Site-to-Site VPN — home ↔ VPS

Route-based IKEv2 IPsec tunnel between the home router (vyos-fw, PPPoE WAN) and the VPS edge (vyos-edge, static 159.195.87.143). Uses VTI for clean route-based forwarding and x509 ECDSA certificate authentication.

Topology

Role	Hostname	WAN	VTI IP
Home	`vyos-fw`	PPPoE on `pppoe0` (dyn)	`10.255.255.1/30`
VPS	`vyos-edge`	`eth0` `159.195.87.143`	`10.255.255.2/30`

VTI interface: vti0 on both ends, MTU 1400
Crypto: IKEv2, AES-GCM-256, DH group 19 (ECP256), SHA-256, lifetime 3600s
Auth: x509 ECDSA P-256 (custom CA IPSEC-CA-V2)
IDs: bare DNS form (home.helix9.org / vps.helix9.org) — matches SAN, avoids CN= lookup collisions

Initial Setup

1. Generate the CA (home only)

The CA lives on the home router. Pick a CN that does not match either peer CN — collisions break strongSwan's pubkey lookup.

configure
run generate pki ca install IPSEC-CA-V2

Prompts:

key type: ec, bits 256
country: DE
CN: IPSEC-CA-V2 ← do not reuse a peer CN
valid days: 3650
encrypt private key: N

commit
save

2. Generate peer certs (home only)

Both certs are signed by the CA. Home cert installs locally; VPS cert is exported.

Home cert (install):

run generate pki certificate sign IPSEC-CA-V2 install home-v2

existing CSR: N
ec / 256, DE
CN: home.helix9.org
SAN: dns:home.helix9.org
days: 1000
type: server
encrypt: N

commit
save

VPS cert (export — do NOT install):

run generate pki certificate sign IPSEC-CA-V2

Same answers but CN vps.helix9.org, SAN dns:vps.helix9.org, install: N.

The output dumps cert PEM + private key PEM. Save both to files (e.g. vps.crt, vps.key). The private key never touches disk on the home router — copy and clear terminal scrollback after.

Export the CA public cert:

run show pki ca IPSEC-CA-V2 pem

Save as ca.pem.

3. Import on VPS

VyOS PKI takes single-line base64 (no headers, no newlines). Use the helper script:

#!/usr/bin/env bash
# pem2vyos.sh
set -euo pipefail
CA_PEM="${1:?ca pem path required}"
CRT_PEM="${2:?cert pem path required}"
KEY_PEM="${3:?key pem path required}"
CA_NAME="${4:-IPSEC-CA-V2}"
CERT_NAME="${5:-vps-v2}"

strip() { grep -v -- '-----' "$1" | tr -d '\n\r '; }

cat <<EOF
configure
set pki ca ${CA_NAME} certificate '$(strip "$CA_PEM")'
set pki certificate ${CERT_NAME} certificate '$(strip "$CRT_PEM")'
set pki certificate ${CERT_NAME} private key '$(strip "$KEY_PEM")'
commit
save
exit
EOF

Run on a workstation, paste output into the VPS CLI:

./pem2vyos.sh ca.pem vps.crt vps.key

4. IPsec configuration

Crypto groups (both sides — names differ per host):

# home: ESP-VPS / IKE-VPS
# vps:  ESP-HOME / IKE-HOME
set vpn ipsec esp-group <name> proposal 1 encryption aes256gcm128
set vpn ipsec esp-group <name> proposal 1 hash sha256
set vpn ipsec esp-group <name> pfs dh-group19
set vpn ipsec esp-group <name> lifetime 3600

set vpn ipsec ike-group <name> proposal 1 dh-group 19
set vpn ipsec ike-group <name> proposal 1 encryption aes256gcm128
set vpn ipsec ike-group <name> proposal 1 hash sha256
set vpn ipsec ike-group <name> key-exchange ikev2
set vpn ipsec ike-group <name> lifetime 3600
set vpn ipsec ike-group <name> dead-peer-detection action restart
set vpn ipsec ike-group <name> dead-peer-detection interval 15
set vpn ipsec ike-group <name> dead-peer-detection timeout 60

Peer auth — home:

set vpn ipsec site-to-site peer vps authentication mode x509
set vpn ipsec site-to-site peer vps authentication x509 ca-certificate IPSEC-CA-V2
set vpn ipsec site-to-site peer vps authentication x509 certificate home-v2
set vpn ipsec site-to-site peer vps authentication local-id 'home.helix9.org'
set vpn ipsec site-to-site peer vps authentication remote-id 'vps.helix9.org'
set vpn ipsec site-to-site peer vps connection-type initiate
set vpn ipsec site-to-site peer vps remote-address 159.195.87.143
set vpn ipsec site-to-site peer vps local-address any
set vpn ipsec site-to-site peer vps ike-group IKE-VPS
set vpn ipsec site-to-site peer vps vti bind vti0
set vpn ipsec site-to-site peer vps vti esp-group ESP-VPS
set vpn ipsec interface pppoe0

Peer auth — VPS: mirror, swap names + connection-type trap, remote-address any, local-address 159.195.87.143, interface eth0.

VTI + MTU (both ends):

set interfaces vti vti0 address 10.255.255.1/30   # vps: .2/30
set interfaces vti vti0 mtu 1400
set interfaces vti vti0 description 'IPsec tunnel'

Static route on VPS (to reach home LAN):

set protocols static route 10.69.0.0/16 interface vti0

Firewall — home WAN-LOCAL must accept ESP + IKE explicitly (ESP is stateless, conntrack won't help):

set firewall ipv4 name WAN-LOCAL rule 30 action accept
set firewall ipv4 name WAN-LOCAL rule 30 protocol esp
set firewall ipv4 name WAN-LOCAL rule 30 source address 159.195.87.143
set firewall ipv4 name WAN-LOCAL rule 31 action accept
set firewall ipv4 name WAN-LOCAL rule 31 protocol udp
set firewall ipv4 name WAN-LOCAL rule 31 destination port 500,4500
set firewall ipv4 name WAN-LOCAL rule 31 source address 159.195.87.143

5. Verify

run show vpn ipsec sa
run show vpn ipsec connections

Expect up state, proposal AES_GCM_16_256, byte counters climbing on traffic.

Certificate Renewal

Peer certs expire ~1000 days after issue (check exact dates with run show pki certificate). CA expires 5 years after issue. Renewals reuse the existing CA — only peer certs are reissued.

Renewing peer certs (CA still valid)

Use the same V2/V3 versioning trick to avoid still in use errors during cleanup. The tunnel drops only briefly when the second side's commit lands.

1. On home — generate new home cert + new VPS cert:

configure
run generate pki certificate sign IPSEC-CA-V2 install home-v3
# CN: home.helix9.org, SAN dns:home.helix9.org, days 1000, server, no encrypt
commit
save

run generate pki certificate sign IPSEC-CA-V2
# CN: vps.helix9.org, SAN dns:vps.helix9.org, install: N
# copy cert + key PEMs

2. Import new VPS cert on VPS, swap reference, commit:

configure
set pki certificate vps-v3 certificate '<b64>'
set pki certificate vps-v3 private key '<b64>'
set vpn ipsec site-to-site peer home authentication x509 certificate vps-v3
commit
save

3. Swap home reference:

configure
set vpn ipsec site-to-site peer vps authentication x509 certificate home-v3
commit
save

4. Verify tunnel up, then delete old:

run show vpn ipsec sa

# home
configure
delete pki certificate home-v2
commit
save

# vps
configure
delete pki certificate vps-v2
commit
save

Renewing the CA

When the CA approaches expiry, repeat the full Initial Setup steps 1–3 with new names (IPSEC-CA-V3, home-v4, vps-v4), then for each peer:

set vpn ipsec site-to-site peer <peer> authentication x509 ca-certificate IPSEC-CA-V3
set vpn ipsec site-to-site peer <peer> authentication x509 certificate <new-cert>
delete vpn ipsec site-to-site peer <peer> authentication x509 ca-certificate IPSEC-CA-V2
commit
save

Then delete old PKI on both ends.

ca-certificate is a multi-value leaf — set adds, doesn't replace. Always delete the old reference explicitly before deleting the old CA, or pki commit will fail with still in use.

Troubleshooting

Symptom	Likely cause	Fix
SA up, 0 bytes, ESP not reaching peer	WAN firewall drops proto 50	Add ESP accept on `WAN-LOCAL` from peer IP
SA up, 0 bytes, ping replies missing	LAN host firewall drops VPN-source ICMP	Allow `10.255.255.0/30` on host
`AUTH_FAILED` / `no trusted ECDSA public key found for 'CN=...'`	CA CN matches peer CN — strongSwan lookup collision	Regenerate CA with unique CN, switch IDs to bare DNS form
Commit fails: `Embedded PKI ... still in use`	Old cert/CA referenced by IPsec	`delete vpn ipsec ... ca-certificate <old>` first
`commit` fails: `Missing CA certificate on specified PKI CA`	Empty/malformed base64 in `set pki ca`	Re-strip PEM (`grep -v -- '-----' \| tr -d '\n\r '`), reapply

Useful diagnostic commands

# show installed PKI
run show pki ca
run show pki certificate

# strongSwan view (live)
sudo swanctl --list-conns
sudo swanctl --list-certs
sudo journalctl -u strongswan -f --no-pager

# kernel xfrm state (proof packets are encrypting)
sudo ip xfrm policy
sudo ip xfrm state
sudo ip route get <lan-host>

Security Notes

Private keys never leave the router that uses them. The VPS key is generated on home only because VyOS lacks a CSR-from-VPS workflow in current versions; copy via secure channel and clear scrollback.
CA private key lives on the home router. For higher assurance, generate the CA offline, sign certs offline, and only import the public CA cert to routers (run generate pki ca without install, then keep the key off-router).
Rotate peer certs annually even if validity is longer.
Monitor expiry: run show pki certificate lists Expiry per cert.

Topology​

Initial Setup​

1. Generate the CA (home only)​

2. Generate peer certs (home only)​

3. Import on VPS​

4. IPsec configuration​

5. Verify​

Certificate Renewal​

Renewing peer certs (CA still valid)​

Renewing the CA​

Troubleshooting​

Useful diagnostic commands​

Security Notes​