# Runbook: Homelab SearxNG

## Purpose

Operate the private SearxNG search service used as the user's default search engine, a search backend for internal webapps, and the controlled search provider for future safe web research tooling.

## Service Summary

- Proxmox host: `buntbox01` / `192.168.0.88`
- Guest: LXC `103` / `searxng`
- Guest IP: `192.168.0.133` via DHCP
- Intended DNS name: `search.dropcutstud.io`
- Current URL until DNS/TLS is finalized: `http://192.168.0.133:8080/`
- Intended access policy: LAN and Tailscale only
- Runtime: Docker Compose under `/opt/searxng`
- Services: `searxng` plus `valkey`
- Snapshot: `post-searxng-initial`

## Preconditions

- SSH access to Proxmox as `piagent` using `.ssh/piagent_homelab`.
- `piagent` has passwordless sudo on `buntbox01`.
- LXC `103` is running.

## Safety Notes

- Do not expose this service publicly without a separate security review.
- Search snippets are still untrusted external content.
- For normal human/browser and internal webapp use, this service is a general metasearch engine.
- For Pi/assistant use, treat it as search metadata only; it does not make arbitrary fetched pages safe for LLM context.
- DNS currently needs follow-up: public `search.dropcutstud.io` does not point at the internal service.

## Common Commands

Check guest status:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 'sudo pct status 103'
```

Check containers:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct exec 103 -- bash -lc "cd /opt/searxng && docker-compose ps"'
```

Restart SearxNG:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct exec 103 -- systemctl restart searxng-compose.service'
```

View logs:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct exec 103 -- bash -lc "cd /opt/searxng && docker-compose logs --tail=100 searxng"'
```

Verify locally from the LAN/container environment:

```sh
curl -I http://192.168.0.133:8080/
curl 'http://192.168.0.133:8080/search?q=pi%20coding%20agent&format=json'
```

## Update Procedure

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct exec 103 -- bash -lc "cd /opt/searxng && docker-compose pull && docker-compose up -d"'
```

## Backups / Restore

- Backup class: `standard`
- Backup status: sanitized config backup configured and verified on 2026-06-07; restore/rebuild test pending
- Backup scope: `/opt/searxng/docker-compose.yml`, sanitized `/opt/searxng/searxng/settings.yml`, and `/etc/systemd/system/searxng-compose.service`
- Backup destination: `/home/piagent/backups/searxng/` on Nimrod LXC 104
- Encryption: bypassed for SearXNG by user direction; the live `secret_key` is not backed up and must be regenerated on restore
- Schedule: manual only during bootstrap
- Retention: follow `docs/service-backup-standard.md`; pruning is not yet automated
- Restore test required: yes for standard class; full isolated restore/rebuild test is still pending

Safety note: SearXNG `settings.yml` contains a service `secret_key`. SearXNG backups intentionally replace that value with `__REGENERATE_ON_RESTORE__` instead of using age identity or copying the live secret. If the key is exposed in logs or terminal output, rotate it and restart SearXNG.

Backup script source of truth in this repo: `ansible/files/searxng-sanitized-backup-from-nimrod`.

The script runs from Nimrod, streams selected files out of CT 103 through the Proxmox host over SSH, and writes a sanitized `.tar.gz` artifact plus checksum to Nimrod backup staging. It does not use Vaultwarden backup recipients, age identity, or private key material.

Manual backup command from Nimrod:

```sh
ansible/files/searxng-sanitized-backup-from-nimrod
```

Verification from Nimrod:

```sh
cd /home/piagent/backups/searxng
sha256sum -c <artifact>.sha256
ls -lh <artifact>.tar.gz <artifact>.sha256
tar -xOzf <artifact>.tar.gz opt/searxng/searxng/settings.yml | grep 'secret_key: "__REGENERATE_ON_RESTORE__"'
curl -sS -o /dev/null -w 'direct=%{http_code}\n' http://192.168.0.133:8080/
curl -k --resolve search.dropcutstud.io:443:192.168.0.137 \
  -sS -o /dev/null -w 'proxy=%{http_code}\n' https://search.dropcutstud.io/
```

Restore/rebuild outline:

1. Provision or choose an isolated test target with Docker Compose available.
2. Extract the sanitized backup artifact on the isolated target.
3. Restore `docker-compose.yml`, `settings.yml`, and `searxng-compose.service` to their expected paths.
4. Generate a fresh SearXNG `secret_key` on the target and replace `__REGENERATE_ON_RESTORE__` before starting the service.
5. Start the service with systemd or Docker Compose.
6. Verify the direct HTTP endpoint and a sample query.
7. Do not overwrite production SearXNG config during restore testing.

Do not change Vaultwarden critical backup encryption as part of SearXNG backup changes.

## Rollback / Recovery

A post-deployment snapshot exists:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct rollback 103 post-searxng-initial'
```

To stop the service without destroying the guest:

```sh
ssh -i .ssh/piagent_homelab piagent@192.168.0.88 \
  'sudo pct exec 103 -- systemctl stop searxng-compose.service'
```

To remove the LXC entirely, confirm with the user first, then stop and destroy LXC `103` from Proxmox.

## Follow-Up Needed

- Configure `search.dropcutstud.io` split-horizon/internal DNS to resolve to `192.168.0.133` for LAN users.
- Configure Tailscale access, either through existing subnet routing/internal DNS or by adding a Tailscale-specific DNS path.
- Decide whether to add internal TLS/reverse proxy.
- Test browser default-search integration once DNS/name is stable.
- Define expected integration mode for webapps: browser-style URL, JSON API, or both.