Creating and Managing a New Host

This guide walks through the full lifecycle of adding a new LXC container to the homelab — from provisioning in Proxmox via Terraform to full OS and service configuration via Ansible.

---

How It Works — The Big Picture

Every host goes through two layers:

┌─────────────────────────────────────────────────────────┐
│  Layer 1 — Infrastructure (Terraform)                   │
│                                                         │
│  Creates the LXC container in Proxmox:                  │
│  CPU, RAM, disk, network (VLAN, IP), SSH key inject     │
└────────────────────────┬────────────────────────────────┘
                         │ container exists, SSH accessible
┌────────────────────────▼────────────────────────────────┐
│  Layer 2 — Configuration (Ansible)                      │
│                                                         │
│  Configures the OS inside the container:                │
│  users, SSH hardening, packages, security, monitoring   │
│  + deploys the service (if applicable)                  │
└─────────────────────────────────────────────────────────┘

The Ansible inventory is the single source of truth. It drives both layers — Terraform reads from it to know what to create, and Ansible reads from it to know what to configure.

---

Step 1 — Plan the Host

Before touching any file, decide:

Decision	Convention
Zone / VLAN	Which network does it belong to? See VyOS zone table below
IP address	Pick a free static IP in the zone's subnet
VMID	Encodes zone + IP: VLAN 20, IP .76 → VMID 276
Hostname	Short, lowercase, no underscores

Zone reference

Zone	VLAN	Subnet	Use for
MGMT	10	10.69.10.x	Infrastructure, management services
SERVERS	20	10.69.20.x	Self-hosted services
HOMELAB	30	10.69.30.x	Home automation, experimenting
DMZ	70	10.69.70.x	Public-facing / community services

---

Step 2 — Add to Inventory

Open inventory/hosts.yml and add the host under the correct zone group:

servers:
  hosts:
    myservice:
      ansible_host: 10.69.20.77
      vmid: 277

This one entry is enough for both Terraform and Ansible to know the host exists.

---

Step 3 — Create host_vars

Create inventory/host_vars/myservice/vars.yml. The lxc_* keys tell Terraform what resources to allocate. Everything else is Ansible config for the service.

---
# Service-specific Ansible vars go here (if any)
# myservice_version: "latest"

# Terraform LXC config — this is what makes Terraform manage this host
lxc_description: "My service — short description"
lxc_tags: ["servers", "myservice"]
lxc_cores: 2
lxc_memory: 1024   # MB
lxc_swap: 256      # MB
lxc_disk: 10       # GB

If the service needs secrets (passwords, API keys), create a separate vault.yml alongside it:

ansible-vault create inventory/host_vars/myservice/vault.yml

---

Step 4 — Provision with Terraform

On pve02:

cd ~/ansible && git pull
cd terraform/proxmox

# Review what will be created
terraform plan -var-file="secrets.tfvars"

# Create the container
terraform apply -var-file="secrets.tfvars"

Terraform will:

• Create the LXC container in Proxmox with the specs from host_vars
• Assign the IP and VLAN from hosts.yml + group_vars/{zone}.yml
• Inject your SSH public key (from secrets.tfvars) so Ansible can connect immediately

---

Step 5 — Bootstrap with Ansible

The bootstrap playbook sets up the baseline so all subsequent playbooks can run without a password:

cd ~/ansible
ansible-playbook playbooks/bootstrap.yml -l myservice -k

The -k flag prompts for the root password (set by Proxmox at container creation). After bootstrap, SSH key auth is in place and -k is never needed again.

What bootstrap does:

• Installs Python (required by Ansible)
• Creates the marko admin user with sudo
• Deploys SSH authorized keys for both root and marko
• Configures /etc/hosts and hostname

---

Step 6 — Apply Full Configuration

ansible-playbook playbooks/site.yml -l myservice

This applies the full role stack to the host. What runs depends on which groups the host belongs to.

---

Step 7 — Deploy the Service (if applicable)

If the host runs a specific service managed by a dedicated playbook:

ansible-playbook playbooks/myservice.yml

If the service is new and has no playbook yet, you need to create the role first — see Creating a Service Role below.

---

How Ansible Manages a Host

The Role Stack

Every managed host gets a baseline set of roles applied in order:

common      → timezone, hostname, locale, base packages
ssh         → hardened sshd_config, deploy SSH authorized keys
users       → create marko user, configure sudo
packages    → unattended-upgrades (security patches)
security    → fail2ban; UFW (DMZ zone only)
monitoring  → node_exporter on port 9100 (when enable_monitoring: true)

These run on every host every time site.yml runs. They are idempotent — running them again on an already-configured host changes nothing.

Service-specific roles (podman, uptime-kuma, onedev, etc.) run only when their playbook is invoked.

How Variables Flow

Ansible merges variables from multiple sources, in order of precedence (lowest → highest):

group_vars/all/vars.yml          → applies to every host
group_vars/{zone}.yml            → applies to all hosts in a zone (e.g. servers.yml)
group_vars/lxc.yml               → applies to all LXC containers
host_vars/{host}/vars.yml        → applies to this host only
host_vars/{host}/vault.yml       → encrypted secrets for this host

Example: vlan_id and vlan_gateway come from group_vars/servers.yml and are available to every role running on a SERVERS host — you never need to repeat them in host_vars.

How Secrets Work

Sensitive values (passwords, tokens, API keys) live in vault.yml files encrypted with Ansible Vault. They are never stored in plain text and never committed unencrypted.

Convention: vault variables are prefixed with vault_ and aliased in vars.yml:

# vault.yml (encrypted)
vault_myservice_password: "supersecret"

# vars.yml (plain text, references vault)
myservice_password: "{{ vault_myservice_password }}"

The vault password lives in ~/.ansible_vault_pass on pve02. Ansible picks it up automatically via ansible.cfg.

How Services Are Deployed (Podman + Quadlet)

Services run as rootful Podman containers managed by systemd quadlets. A quadlet is a .container unit file that systemd reads and converts into a full service — no Docker Compose, no custom scripts.

/etc/containers/systemd/myservice.container
        │
        │  read by systemd-generator at boot
        ▼
myservice.service   ←→   podman run ...

The .container file is a Jinja2 template in the role (roles/myservice/templates/myservice.container.j2) and deployed by Ansible. Key fields:

[Container]
Image=docker.io/org/image:{{ myservice_version }}
AutoUpdate=registry          # podman auto-update handles image updates
Volume={{ myservice_data_dir }}:/data
PublishPort={{ myservice_port }}:8080

[Service]
Restart=on-failure

AutoUpdate=registry means the running image is compared to the registry on a schedule. When podman auto-update runs (via its systemd timer), it pulls a newer image and restarts the service automatically — no playbook needed for routine updates.

---

Creating a Service Role

If the service is new, create the role structure:

roles/myservice/
├── defaults/main.yml      # default variable values
├── tasks/main.yml         # what to do
├── handlers/main.yml      # restart/reload triggers
└── templates/
    └── myservice.container.j2   # quadlet unit file

defaults/main.yml — sensible defaults, overridable in host_vars:

---
myservice_version: "latest"
myservice_port: 8080
myservice_data_dir: /srv/myservice
myservice_quadlet_dir: /etc/containers/systemd

tasks/main.yml:

---
- name: Create myservice data directory
  ansible.builtin.file:
    path: "{{ myservice_data_dir }}"
    state: directory
    owner: root
    group: root
    mode: "0755"

- name: Deploy myservice quadlet container file
  ansible.builtin.template:
    src: myservice.container.j2
    dest: "{{ myservice_quadlet_dir }}/myservice.container"
    owner: root
    group: root
    mode: "0644"
  notify: reload systemd

- name: Unmask myservice service
  ansible.builtin.systemd:
    name: myservice.service
    masked: false
  failed_when: false

- name: Reload systemd to pick up quadlet changes
  ansible.builtin.systemd:
    daemon_reload: true

- name: Enable and start myservice
  ansible.builtin.systemd:
    name: myservice.service
    enabled: true
    state: started

handlers/main.yml:

---
- name: reload systemd
  ansible.builtin.systemd:
    daemon_reload: true
  listen: reload systemd

- name: restart myservice
  ansible.builtin.systemd:
    name: myservice.service
    state: restarted
    daemon_reload: true
  listen: restart myservice

templates/myservice.container.j2:

# {{ ansible_managed }}
[Unit]
Description=My Service
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/org/image:{{ myservice_version }}
AutoUpdate=registry
ContainerName=myservice

Volume={{ myservice_data_dir }}:/data

PublishPort={{ myservice_port }}:8080

[Service]
ExecStartPre=-/usr/sbin/nft flush chain inet netavark nv_2f259bab_10_88_0_0_nm16_dnat
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

Then create a playbook playbooks/myservice.yml:

---
- name: Deploy myservice
  hosts: myservice
  become: true
  roles:
    - podman
    - myservice

Note on the nft line: The ExecStartPre flushes a stale netavark DNAT chain that Podman leaves behind on published ports after a restart. The - prefix means failure is ignored (the chain won't exist on first boot).

---

Day-to-Day Operations

Check if a host is correctly configured

ansible-playbook playbooks/site.yml -l myservice --check --diff

Re-apply config after a change

# Edit the relevant vars or role, commit, push, then on pve02:
cd ~/ansible && git pull
ansible-playbook playbooks/site.yml -l myservice
# Or just the service playbook if only the service changed:
ansible-playbook playbooks/myservice.yml

Update container images manually

# On the host itself:
podman auto-update
# Or via Ansible on all hosts:
ansible all -m command -a "podman auto-update" -l lxc

Check service status

# On the host:
systemctl status myservice.service
journalctl -u myservice.service -f

Update all systems (OS packages)

ansible-playbook playbooks/update.yml