# Runbook: Nginx reverse proxy

## Purpose

Operate and extend the standard homelab Nginx reverse proxy for LAN/Tailscale-only services.

This runbook documents the current MVP reverse-proxy LXC.

Related spec:

- `tickets/artifacts/2026-06-06-nginx-reverse-proxy-and-tls-standard/02-reverse-proxy-implementation-spec.md`

## Preconditions

- Reverse proxy host/LXC exists and is documented in `infra/proxmox-registry.yaml` and `systems/inventory.md`.
- Current host: CTID 105 `reverse-proxy` at `192.168.0.137`.
- Nginx is installed.
- Service DNS names resolve to the reverse proxy IP, or testing uses `curl --resolve` / temporary hosts entries.
- Backend service direct IP/port is known and reachable from the proxy.
- TLS certificate/key exists on the proxy host; private keys are not stored in git.

## Safety Notes

- Default access boundary is LAN + Tailscale only.
- Do not add public port forwarding without explicit approval and threat review.
- Do not commit TLS private keys, API tokens, admin credentials, or DNS provider credentials.
- Keep direct backend access available during first migration of a service.
- For high-value services such as Vaultwarden, test with a lower-risk service first.

## Standard Paths

Expected paths on the reverse proxy host:

```text
/etc/nginx/sites-available/<service>.conf
/etc/nginx/sites-enabled/<service>.conf -> ../sites-available/<service>.conf
/etc/nginx/snippets/             # optional shared proxy snippets
/etc/nginx/tls/                  # cert/key storage; private keys root-readable only
/var/log/nginx/                  # access/error logs
```

## Add a Proxied Service

1. Confirm backend is reachable from the proxy:

   ```sh
   curl -fsS http://<backend-ip>:<backend-port>/ >/dev/null
   ```

2. Confirm desired service name and DNS target:

   ```text
   <service>.dropcutstud.io -> <reverse-proxy-ip>
   ```

3. Create Nginx site config:

   ```nginx
   server {
       listen 80;
       server_name <service>.dropcutstud.io;
       return 301 https://$host$request_uri;
   }

   server {
       listen 443 ssl http2;
       server_name <service>.dropcutstud.io;

       ssl_certificate /etc/nginx/tls/<service>.crt;
       ssl_certificate_key /etc/nginx/tls/<service>.key;

       location / {
           proxy_pass http://<backend-ip>:<backend-port>;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto https;
       }
   }
   ```

4. Enable site:

   ```sh
   ln -s /etc/nginx/sites-available/<service>.conf /etc/nginx/sites-enabled/<service>.conf
   nginx -t
   systemctl reload nginx
   ```

5. Verify locally/on LAN:

   ```sh
   curl -k --resolve <service>.dropcutstud.io:443:<reverse-proxy-ip> https://<service>.dropcutstud.io/ -I
   ```

6. Update DNS service so clients resolve the service name to the proxy IP.

7. Verify from a trusted client browser.

8. Update docs and change log.

## SearxNG MVP Route

Current first test route:

```text
search.dropcutstud.io -> reverse proxy 192.168.0.137 -> http://192.168.0.133:8080
```

Current certificate: temporary self-signed leaf certificate stored on CT 105 under `/etc/nginx/tls/`; not a final trust standard.

Fallback/direct backend:

```text
http://192.168.0.133:8080/
```

## Verification

Proxy host:

```sh
systemctl is-active nginx
nginx -t
ss -ltnp | grep -E ':(80|443)\b'
```

Service route:

```sh
curl -fsS http://<backend-ip>:<backend-port>/ >/dev/null
curl -k --resolve <service>.dropcutstud.io:443:<reverse-proxy-ip> https://<service>.dropcutstud.io/ -I

# Current SearXNG MVP route:
curl -k --resolve search.dropcutstud.io:443:192.168.0.137 https://search.dropcutstud.io/ -I
```

Client:

- Browser loads service over HTTPS.
- Expected certificate behavior matches current cert strategy.
- Backend direct URL still works during migration window.

## Backups / Restore

- Backup class: `standard`
- Backup status: Nginx config backup configured and verified on 2026-06-07; TLS private-key backup remains pending an approved encrypted path
- Backup scope: `/etc/nginx/nginx.conf`, `/etc/nginx/conf.d/`, `/etc/nginx/sites-available/`, `/etc/nginx/sites-enabled/`, and `/etc/nginx/snippets/` when present
- Excluded from current backup: `/etc/nginx/tls/` private keys/certificates
- Guest-local destination: `/var/backups/reverse-proxy/`
- Off-guest destination: `/home/piagent/backups/reverse-proxy/` on Nimrod LXC 104
- Encryption: not currently required for Nginx route config alone; required before backing up TLS private keys, DNS provider credentials, or other secrets
- Schedule: manual only during bootstrap
- Retention: manual/no pruning until the broader backup destination/retention policy is selected
- Restore test required: yes for standard class; full isolated restore/rebuild test is still pending

Backup script source of truth in this repo: `ansible/files/reverse-proxy-backup`.

Install/update the script with Ansible from Nimrod:

```sh
ansible reverse-proxy -i ansible/inventories/homelab/hosts.yml \
  -m copy \
  -a 'src=ansible/files/reverse-proxy-backup dest=/usr/local/sbin/reverse-proxy-backup owner=root group=root mode=0755'
```

Manual backup command on `reverse-proxy`:

```sh
sudo /usr/local/sbin/reverse-proxy-backup
```

Verification on `reverse-proxy`:

```sh
cd /var/backups/reverse-proxy
sudo sha256sum -c <latest>.sha256
sudo nginx -t
systemctl is-active nginx
```

Off-guest copy from Nimrod:

```sh
mkdir -p /home/piagent/backups/reverse-proxy
scp reverse-proxy:/var/backups/reverse-proxy/<artifact>.tar.gz /home/piagent/backups/reverse-proxy/
scp reverse-proxy:/var/backups/reverse-proxy/<artifact>.sha256 /home/piagent/backups/reverse-proxy/
cd /home/piagent/backups/reverse-proxy && sha256sum -c <artifact>.sha256
```

Restore/rebuild outline:

1. Provision or choose an isolated test target with Nginx installed.
2. Extract the backup artifact into a temporary directory.
3. Restore the Nginx config paths to `/etc/nginx/` on the target.
4. Provide replacement test TLS certs/keys or restore production TLS material only from an approved encrypted secret/backup path.
5. Run `sudo nginx -t`.
6. Start/reload Nginx on the target.
7. Verify representative routes such as `search.dropcutstud.io` and `dashboard.dropcutstud.io` against test or production backends as appropriate.
8. Do not overwrite production proxy config during restore testing.

## Rollback / Recovery

For one service route:

1. Disable site:

   ```sh
   rm /etc/nginx/sites-enabled/<service>.conf
   nginx -t
   systemctl reload nginx
   ```

2. Remove/undo DNS record or hosts entry.
3. Use direct backend URL.
4. Preserve config/certs for review unless user approves deletion.
5. Log rollback.

For proxy host failure:

1. Use direct backend IP/port URLs.
2. Stop proxy host if it causes disruption.
3. Revert to last known-good snapshot if one exists.
4. Destroy proxy LXC only if approved and no needed files remain.

## Related Files / Systems

- Ticket: `tickets/active/2026-06-06-nginx-reverse-proxy-and-tls-standard.md`
- DNS ticket: `tickets/active/2026-06-06-internal-dns-and-tailscale-naming.md`
- Design artifact: `tickets/artifacts/2026-06-06-internal-dns-and-tailscale-naming/01-dns-tailscale-tls-design.md`
- Implementation spec: `tickets/artifacts/2026-06-06-nginx-reverse-proxy-and-tls-standard/02-reverse-proxy-implementation-spec.md`
- Registry: `infra/proxmox-registry.yaml`
- Change log: `docs/server-change-log.md`