# Server Change Log Purpose: record operational changes made to servers so work is auditable and resumable. ## Format Each entry should include: - Date/time - Server/host - Operator - Reason - Commands/actions summary - Files/services changed - Verification - Rollback notes For secrets and access-management changes, also include: - Credential/access reference only, never the secret value - Grant/rotate/revoke action taken - Verification that intended access succeeds or revoked access fails - Confirmation that no passwords, private keys, API tokens, recovery codes, or app-password values were logged ## 2026-05-14 — Nextcloud VM — Nextcloud Talk assistant bridge - Server/host: `nextcloud` / `192.168.0.110` - Operator: Pi assistant via SSH as `piagent` - Reason: enable Deeso to chat with the assistant through Nextcloud Talk, including voice notes and pending-draft approvals. Actions summary: - Confirmed SSH access using `.ssh/piagent_homelab`. - Confirmed Nextcloud Docker containers: `nextcloud-app`, `nextcloud-db`, `nextcloud-redis`. - Confirmed Nextcloud version `33.0.3` and Talk app `spreed` `23.0.4`. - Confirmed `piagent` is in `ops`, not `admin`. - Created app password for `piagent` for the bridge; secret stored only on the VM in `/etc/nextcloud-talk-assistant/env`. - Created Talk conversation between `piagent` and `deeso`. - Deployed bridge service under `/opt/nextcloud-talk-assistant/`. - Created service user `nextcloud-talk-assistant`. - Created configuration under `/etc/nextcloud-talk-assistant/config.json`. - Created state/log directories under `/var/lib/nextcloud-talk-assistant/` and `/var/log/nextcloud-talk-assistant/`. - Installed and enabled systemd service `nextcloud-talk-assistant.service`. - Installed packages needed for speech-to-text: `ffmpeg`, `python3-pip`, `python3-venv`, build tooling. - Accidentally installed Ubuntu `python3-whisper` first; removed it because it is Graphite Whisper, not OpenAI Whisper. - Created Python venv `/opt/nextcloud-talk-assistant/venv` and installed `faster-whisper`. - Preloaded faster-whisper `tiny` model under `/var/lib/nextcloud-talk-assistant`. - Created Nextcloud calendar `assistant-reminders` for `piagent`. - Enabled non-dry-run approval behavior: - captures/tickets/project notes append to `Assistant/capture.md` in `piagent` Nextcloud Files - reminders create VTODO objects in `assistant-reminders` Files/services changed: - `/opt/nextcloud-talk-assistant/nextcloud_talk_assistant.py` - `/opt/nextcloud-talk-assistant/venv/` - `/etc/nextcloud-talk-assistant/config.json` - `/etc/nextcloud-talk-assistant/env` - `/etc/systemd/system/nextcloud-talk-assistant.service` - `/var/lib/nextcloud-talk-assistant/state.json` - `/var/lib/nextcloud-talk-assistant/applied-drafts.jsonl` - `/var/lib/nextcloud-talk-assistant/voice/` - `/var/log/nextcloud-talk-assistant/bridge.log` - Nextcloud Talk conversation for Deeso ↔ Pi Assistant - Nextcloud calendar: `assistant-reminders` Verification: - `nextcloud-talk-assistant.service` is active. - Text `help` command works in Talk. - Voice notes are received from Talk as `messageType: voice-message`. - Voice audio can be downloaded via WebDAV from `piagent` Files under `Talk/`. - faster-whisper successfully transcribed a voice note: `Remind me to test voice notes tomorrow.` - Pending draft approval flow works. - User reported the workflow appears to have worked. Rollback notes: - Stop and disable service: ```sh sudo systemctl stop nextcloud-talk-assistant sudo systemctl disable nextcloud-talk-assistant ``` - Revoke the `piagent` app password in Nextcloud. - Optionally remove `/opt/nextcloud-talk-assistant`, `/etc/nextcloud-talk-assistant`, `/var/lib/nextcloud-talk-assistant`, `/var/log/nextcloud-talk-assistant`, and `/etc/systemd/system/nextcloud-talk-assistant.service`. - Optionally delete the `assistant-reminders` calendar and Talk conversation if no longer needed. ## 2026-05-17 — Nextcloud VM — Nextcloud Talk assistant Pi request queue - Server/host: `nextcloud` / `192.168.0.110` - Operator: Pi assistant via SSH as `piagent` - Reason: make the Nextcloud Talk bot more useful by allowing Deeso to queue work requests for Pi/the assistant from Talk. Actions summary: - Updated `/opt/nextcloud-talk-assistant/nextcloud_talk_assistant.py` from the repository copy. - Added `assistant_request` handling for messages like `pi ...`, `ask pi ...`, and `move forward with ...`. - Approved requests are appended to `Assistant/pi-requests.md` in the bridge account's Nextcloud Files, with a local fallback at `/var/lib/nextcloud-talk-assistant/pi-requests.md`. - Restarted `nextcloud-talk-assistant.service`. Files/services changed: - `/opt/nextcloud-talk-assistant/nextcloud_talk_assistant.py` - `nextcloud-talk-assistant.service` - Potential new Nextcloud file on first approved request: `Assistant/pi-requests.md` Verification: - Local parser/apply test succeeded for `pi move forward with tasks`. - `nextcloud-talk-assistant.service` restarted successfully and is active. Rollback notes: - Restore the timestamped backup created in `/opt/nextcloud-talk-assistant/nextcloud_talk_assistant.py.bak-*` and restart `nextcloud-talk-assistant.service`. ## 2026-05-17 — Repository workflow — Controlled Pi request pull script - Server/host: local coding-agent workspace with SSH access to `nextcloud` / `192.168.0.110` - Operator: Pi assistant - Reason: implement the first controlled backend path for approved Nextcloud Talk `pi ...` requests without exposing a public agent API or storing Nextcloud secrets locally. Actions summary: - Added `scripts/nextcloud_pi_request_queue.py`. - The script SSHes to the Nextcloud VM, runs remote Python under sudo, reads existing `/etc/nextcloud-talk-assistant/config.json` and `/etc/nextcloud-talk-assistant/env`, fetches `Assistant/pi-requests.md` through WebDAV, and prints/imports new request lines. - Added ticket `tickets/active/2026-05-17-nextcloud-talk-controlled-pi-backend.md`. - Updated the Talk assistant runbook with list/import commands. Files/services changed: - `scripts/nextcloud_pi_request_queue.py` - `tickets/active/2026-05-17-nextcloud-talk-controlled-pi-backend.md` - `runbooks/nextcloud-talk-assistant-bridge.md` Verification: - `python3 -m py_compile scripts/nextcloud_pi_request_queue.py` succeeded. - `scripts/nextcloud_pi_request_queue.py` successfully connected and reported no new Pi requests. Rollback notes: - Remove `scripts/nextcloud_pi_request_queue.py` and the ticket/runbook updates if this pull workflow is abandoned. - No remote service changes were made for this step. ## 2026-05-14 — Nextcloud VM — YouTube voice capture ingest MVP - Server/host: `nextcloud` / `192.168.0.110` - Operator: Pi assistant via SSH as `piagent` - Reason: create first server-side MVP for Android offline voice captures: Nextcloud folder upload, hourly polling, Whisper transcription, and staging output before any Obsidian integration. Actions summary: - Deployed ingest script to `/opt/youtube-capture-ingest/youtube_capture_ingest.py`. - Reused existing `nextcloud-talk-assistant` service user, existing Nextcloud `piagent` app password env, and existing Python venv containing `faster-whisper`. - Created config `/etc/youtube-capture-ingest/config.json`. - Created processing directories under `/var/lib/youtube-capture-ingest/`: - `raw-audio/` - `transcripts/` - `records/` - `notes-staging/` - `models/` - Created systemd one-shot service `youtube-capture-ingest.service`. - Created and enabled hourly systemd timer `youtube-capture-ingest.timer`. - Created Nextcloud folder `YouTube Voice Captures` owned by `piagent` and shared it with user `deeso` with edit/create permissions. - Performed a test upload using an existing Talk voice MP3, verified transcription, verified rerun deduplication, then removed the test file from the shared Nextcloud capture folder. Files/services changed: - `/opt/youtube-capture-ingest/youtube_capture_ingest.py` - `/opt/youtube-capture-ingest/venv` symlink to `/opt/nextcloud-talk-assistant/venv` - `/etc/youtube-capture-ingest/config.json` - `/etc/systemd/system/youtube-capture-ingest.service` - `/etc/systemd/system/youtube-capture-ingest.timer` - `/var/lib/youtube-capture-ingest/` - Nextcloud Files folder: `piagent/YouTube Voice Captures`, shared with `deeso` Verification: - Manual service run on empty folder succeeded: `processed=0 skipped=0 seen=0`. - Test MP3 uploaded through WebDAV was detected and processed. - Whisper/faster-whisper transcribed the test audio as: `Remind me to test Voice Notes tomorrow.` - Rerunning service skipped the already-processed file by checksum: `processed=0 skipped=1 seen=1`. - Timer enabled and active; next run scheduled hourly. - No writes were made to the live Obsidian vault. Rollback notes: - Disable scheduled ingestion: ```sh sudo systemctl disable --now youtube-capture-ingest.timer ``` - Remove service/timer if needed: ```sh sudo rm -f /etc/systemd/system/youtube-capture-ingest.service /etc/systemd/system/youtube-capture-ingest.timer sudo systemctl daemon-reload ``` - Preserve `/var/lib/youtube-capture-ingest/` unless deletion is explicitly approved, because it contains raw captures, transcripts, and staging records. - Remove the Nextcloud share/folder only after confirming no user recordings need to be retained. ### 2026-05-15 update — YouTube voice capture ingest real Android upload test - Server/host: `nextcloud` / `192.168.0.110` - Operator: Pi assistant via SSH as `piagent` - Reason: verify user's Android/Fossify upload path and harden ingest after encountering one unreadable `.m4a`. Actions summary: - Confirmed five `.m4a` audio files uploaded into `piagent/YouTube Voice Captures` and remained visible through Nextcloud WebDAV. - Manually triggered `youtube-capture-ingest.service`. - Four files transcribed successfully. - One file, `20260515_122107.m4a`, failed with `InvalidDataError`, likely corrupt/partial/unreadable by ffmpeg/PyAV. - Updated `/opt/youtube-capture-ingest/youtube_capture_ingest.py` to handle per-file transcription failures without failing the whole batch. - Updated script to save state after each processed file so one bad audio file does not cause successful files to be forgotten/reprocessed on the next run. Verification: - Successful transcripts included: - `test, test message, see if this works.` - `Another test file, just saying if this works fine.` - `testing recording from my phone.` - `I'm on, I'm on, I'm on, I'm on` - Re-run completed successfully with the bad file recorded as `transcription_failed` instead of crashing the service. Rollback notes: - Restore the previous script version from git if needed. - Failed/corrupt source audio can be deleted or replaced by the user in Nextcloud; staged records are preserved for audit. ### 2026-05-15 update — Daily YouTube staging notes - Server/host: `nextcloud` / `192.168.0.110` - Operator: Pi assistant via SSH as `piagent` - Reason: move toward the project spec by generating curated daily YouTube staging notes while keeping raw processing data outside Obsidian. Actions summary: - Extended `/opt/youtube-capture-ingest/youtube_capture_ingest.py` to regenerate daily Markdown staging notes after each ingest run. - Added config key `daily_notes_dir` set to `/var/lib/youtube-capture-ingest/notes-staging/daily`. - Daily notes group voice captures by day and include placeholders for future YouTube activity/video metadata: title, author/channel, URL, thumbnail, captions, summary. - Daily notes explicitly state that Shorts are ignored by default unless discussed/requested. - Daily note generation deduplicates exact same source files by SHA-256 to hide accidental processing duplicates while preserving distinct repeated recordings/watch events. Files/services changed: - `/opt/youtube-capture-ingest/youtube_capture_ingest.py` - `/etc/youtube-capture-ingest/config.json` - `/var/lib/youtube-capture-ingest/notes-staging/daily/` Verification: - Manual service run completed successfully. - Generated daily staging notes: - `/var/lib/youtube-capture-ingest/notes-staging/daily/2026-05-14.md` - `/var/lib/youtube-capture-ingest/notes-staging/daily/2026-05-15.md` - Verified `2026-05-15.md` includes user voice captures, a failed transcription entry, and future watched-video placeholders. - No writes were made to the live Obsidian vault. Rollback notes: - Revert script/config from git if needed. - Delete only generated daily staging notes under `/var/lib/youtube-capture-ingest/notes-staging/daily/` if regeneration is desired; preserve raw audio/records/transcripts unless explicitly approved. ### 2026-05-18 update — AMP Enshrouded SteamCMD repair - Server/host: `AMP` / `192.168.0.90` - Operator: Pi assistant via SSH as `piagent` - Reason: new Steam-based AMP game installs failed because SteamCMD bootstrap returned `KeyValues Error ... current key: ' `srw-rw-rw-`. - Docker container restarted: `AMP_ADS01`. - No game server containers were restarted or stopped. Verification: - Recent ADS logs no longer show Docker API permission-denied messages after restart. - `ampinstmgr -l` from inside `AMP_ADS01` reports active game instances as `Running: Yes`: - `teeem01` - `ConanExiles01` - `Enshrouded01` - `ConanExilesEnhanced01` - AMP frontend still returns HTTP 200 at `http://192.168.0.90:8080/`. Rollback/security notes: - `chmod 666 /var/run/docker.sock` is an emergency workaround and is less secure than proper group-based Docker socket access. - Restore tighter socket permissions when ADS is returned to normal host-managed operation or when a cleaner container group/user configuration is implemented: ```sh chmod 660 /var/run/docker.sock ``` ### 2026-06-05 update — AMP cleanup to host-managed ADS frontend - Server/host: `AMP` / `192.168.0.90` - Operator: Pi assistant via SSH as `piagent` using `.ssh/piagent_homelab`; escalated through Docker host mount because `sudo` is not installed for the SSH user. - Reason: clean up the emergency `AMP_ADS01` container workaround and return ADS/frontend operation to host-side system management with normal Docker socket permissions. Actions summary: - Removed the emergency ADS container `AMP_ADS01`. - Restored Docker socket permissions from emergency `666` back to `660`. - Verified `amp` and `piagent` are members of the host Docker group. - Attempted to start ADS through `ampinstmgr --StartInstance ADS01` under host systemd; it reported success but the ADS child process did not persist after the transient starter unit exited. - Created a host-managed systemd unit `/etc/systemd/system/amp-ads01.service` to run ADS directly as the `amp` user with `SupplementaryGroups=docker`. - Enabled and started `amp-ads01.service`. - Did not restart or stop game server containers. Files/services changed: - Removed Docker container: `AMP_ADS01` - Restored `/var/run/docker.sock` permissions to `srw-rw----`. - Created/enabled/started systemd unit: `/etc/systemd/system/amp-ads01.service` Verification: - `amp-ads01.service` is active/running. - ADS is listening on `0.0.0.0:8080`. - `http://192.168.0.90:8080/` returns HTTP 200. - Host-side `ampinstmgr -l` reports: - `ADS01` — `Running: Yes` - `teeem01` — `Running: Yes` - `ConanExiles01` — `Running: Yes` - `Enshrouded01` — `Running: Yes` - `ConanExilesEnhanced01` — `Running: Yes` - No recent ADS Docker API permission-denied messages appeared after moving to the host-managed unit. Rollback notes: - To remove the direct ADS service: ```sh systemctl disable --now amp-ads01.service rm -f /etc/systemd/system/amp-ads01.service systemctl daemon-reload ``` - The original `ampinstmgr.service` remains enabled and unchanged. - A future refinement should determine why `ampinstmgr --StartInstance ADS01` reports success but does not leave ADS running when invoked from a transient service in this administration context. ## 2026-06-05 — Proxmox `buntbox01` — Disposable assistant SSH lifecycle pilot - Server/host: Proxmox `buntbox01` / `192.168.0.88`; disposable LXC `102` / `access-pilot` / DHCP `192.168.0.111` - Operator: Pi assistant using Proxmox API token from ignored `.tokens/proxmox.env` - Reason: prove grant, rotate, revoke, and verify-failure workflow for assistant SSH access before applying lifecycle procedures to production hosts. Actions summary: - Verified Proxmox API access and discovered node/storage/network facts. - Created disposable unprivileged Ubuntu 24.04 LXC `102` named `access-pilot` using template `local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst`. - Configuration used: `1` CPU, `1024 MB` RAM, `512 MB` swap, `8 GB` root disk on `local-lvm`, `vmbr0` DHCP networking, tags `disposable;nimrod;pilot`. - Installed the existing assistant public key for root at container creation for bootstrap/cleanup. - Bootstrapped dedicated `piagent` account with locked password and passwordless sudo using `scripts/bootstrap-assistant-ssh-user.sh`. - Verified SSH login as `piagent` and passwordless sudo. - Generated a temporary rotation key for the pilot and stored it only in ignored `.tokens/` during the test. - Added the rotation public key, verified it worked, removed the old `piagent` public key, and verified old-key login failed for `piagent`. - Verified the rotated key still worked and sudo still worked. - Removed the rotated key and verified `piagent` access failed. - Stopped and destroyed disposable LXC `102` with purge/destroy-unreferenced-disks. Files/services changed: - Proxmox temporary LXC: `102` / `access-pilot` — created and destroyed. - Remote container files during pilot only: - `/home/piagent/.ssh/authorized_keys` - `/etc/sudoers.d/90-piagent-pi-agent` - `/etc/ssh/sshd_config.d/90-pi-agent-key-auth.conf` - Local ignored temporary key: `.tokens/access_pilot_rotation` and `.tokens/access_pilot_rotation.pub` - Repository documentation updated separately in runbooks and systems inventory. Verification: - Proxmox API returned `API_OK` and discovered `buntbox01`. - LXC `102` reached running state and received DHCP IP `192.168.0.111`. - Root bootstrap SSH succeeded using existing assistant key. - `piagent` SSH login succeeded after grant. - `sudo -n true` succeeded for `piagent` after bootstrap. - Rotation key login succeeded before old-key removal. - Old key failed for `piagent` after rotation. - Rotation key still worked after old-key removal. - Rotation key failed after revocation. - Root cleanup access remained available until container destruction. - LXC destruction task completed with `exitstatus=OK`; subsequent status lookup returned not found. Rollback notes: - Pilot host was disposable and has been destroyed. - No production hosts were modified. - Temporary pilot key material in `.tokens/` should be deleted after review. - Proxmox bootstrap API token currently has broad privileges and should be reduced or revoked after the setup phase. - No secret values were logged in this change log. ## 2026-06-06 — Proxmox `buntbox01` — Vaultwarden VM shell creation - Server/host: Proxmox `buntbox01` / `192.168.0.88`; VM `102` / `Vaultwarden` - Operator: Pi assistant using Proxmox API token from ignored `.tokens/proxmox.env` - Reason: create the dedicated VM foundation for the approved Vaultwarden/Bitwarden-compatible homelab secrets vault. Actions summary: - Performed read-only Proxmox preflight and confirmed node `buntbox01`, bridge `vmbr0`, storage options, and next available VMID `102`. - Confirmed Debian installer ISO was available: `local:iso/debian-13.5.0-amd64-netinst.iso`. - Created QEMU VM `102` named `Vaultwarden`. - Attached Debian 13.5 netinst ISO as CD-ROM. - Configured VM with minimal approved sizing: `1` vCPU, `1024 MB` RAM, `16 GB` disk on `local-lvm`. - Configured virtio network on `vmbr0` with DHCP expected after OS install. - Enabled QEMU guest agent flag for later use after OS installation. - Started the VM from the installer ISO. Files/services changed: - Proxmox VM created: `102` / `Vaultwarden` - VM disk: `local-lvm:vm-102-disk-0,size=16G` - VM CD-ROM: `local:iso/debian-13.5.0-amd64-netinst.iso,media=cdrom` - VM network: virtio NIC on `vmbr0` - Repository inventory/runbook/ticket docs updated separately. Verification: - Proxmox API returned create task: `qmcreate:102`. - Proxmox API returned start task: `qmstart:102`. - VM status after start: `running`. - VM config shows expected name, CPU/RAM, disk, ISO, boot order, network, agent flag, and tags. - No secret values were logged. Rollback notes: - No Vaultwarden service or secrets have been installed/migrated yet. - To roll back before OS installation/use, stop and destroy VM `102` from Proxmox, including unreferenced disks if approved. - If OS installation proceeds, take a snapshot after base OS hardening before Vaultwarden service installation. - Broad Proxmox bootstrap API token should be reduced/revoked after VM/bootstrap work is complete. ## 2026-06-06 — Proxmox `buntbox01` — Private SearxNG search service - Server/host: Proxmox `buntbox01` / `192.168.0.88`; LXC `103` / `searxng` / DHCP `192.168.0.133` - Operator: Pi assistant using Proxmox API for discovery and SSH as `piagent` with passwordless sudo for LXC creation/provisioning - Reason: deploy private SearxNG as a controlled search provider for safe web research work, with LAN/Tailscale-only intended access. Actions summary: - Verified Proxmox API access using ignored `.tokens/proxmox.env` without logging token values. - Discovered node `buntbox01`, storage, existing guests, and available Debian templates. - Confirmed VMID `103` was free. - Created dedicated unprivileged Debian 12 LXC `103` named `searxng` on `local-lvm`, using `vmbr0` DHCP networking. - Allocated `2` CPU cores, `2048 MB` RAM, `512 MB` swap, and `8 GB` root disk. - Enabled LXC nesting/keyctl features for Docker-in-LXC. - Installed Docker and Docker Compose inside the LXC. - Deployed SearxNG and Valkey via Docker Compose under `/opt/searxng`. - Configured SearxNG base URL as `http://search.dropcutstud.io/` and enabled HTML/JSON search formats. - Created systemd unit `searxng-compose.service` inside the LXC. - Created Proxmox snapshot `post-searxng-initial` after verification. Files/services changed: - Proxmox LXC: `103` / `searxng` - LXC files: - `/opt/searxng/docker-compose.yml` - `/opt/searxng/searxng/settings.yml` - `/etc/systemd/system/searxng-compose.service` - Docker containers inside LXC: - `searxng_searxng_1` - `searxng_valkey_1` - Repository docs/runbook updated separately: - `tickets/active/2026-06-06-add-searxng-service.md` - `runbooks/searxng-homelab.md` Verification: - LXC `103` is running. - Guest IP is `192.168.0.133/24`. - `curl -I http://192.168.0.133:8080/` returns HTTP 200. - HTML search succeeds. - JSON search succeeds after enabling `search.formats` and returns results for `pi coding agent`. - Docker containers show `searxng` and `valkey` up. - Snapshot `post-searxng-initial` exists. - No Proxmox API token values, generated root password, or SearxNG secret key values were logged. Known follow-up: - `search.dropcutstud.io` currently resolves publicly to `202.90.245.135`, not the internal SearxNG guest. - Configure split-horizon/internal DNS for LAN and Tailscale clients to point `search.dropcutstud.io` to `192.168.0.133` or an approved reverse proxy/Tailscale address. - Decide whether to add internal TLS/reverse proxy. Rollback notes: - Restore snapshot if needed: ```sh ssh -i .ssh/piagent_homelab piagent@192.168.0.88 'sudo pct rollback 103 post-searxng-initial' ``` - Stop service without destroying guest: ```sh ssh -i .ssh/piagent_homelab piagent@192.168.0.88 'sudo pct exec 103 -- systemctl stop searxng-compose.service' ``` - Full removal requires explicit confirmation, then stop/destroy Proxmox LXC `103` with disk purge. ## 2026-06-06 — Vaultwarden VM — Bootstrap deployment - Server/host: Proxmox `buntbox01` / VMID `102` / `Vaultwarden` / `192.168.0.238` - Operator: Pi assistant via Proxmox API and SSH as `piagent` - Reason: continue highest-priority consolidated secrets/access-management ticket by deploying the dedicated Vaultwarden VM bootstrap. Actions summary: - Verified VMID `102` was already running from a Debian generic cloud image with cloud-init user `piagent`. - Confirmed DHCP address `192.168.0.238` from Proxmox host neighbor table. - Verified SSH key-only `piagent` access and passwordless sudo for bootstrap. - Created Proxmox snapshot `pre-vaultwarden-install` before package/service installation. - Installed Debian packages: `docker.io`, `nginx`, `openssl`, `qemu-guest-agent`, `curl`, `ca-certificates` and dependencies. - Generated a self-signed TLS certificate for `vm.dropcutstud.io` with SANs for `vm.dropcutstud.io` and `192.168.0.238`. - Deployed Vaultwarden using Docker image `vaultwarden/server:latest` via `vaultwarden.service`. - Configured Nginx TLS reverse proxy on ports `80`/`443`, proxying to Vaultwarden on loopback `127.0.0.1:8080`. - Enabled temporary `SIGNUPS_ALLOWED=true` for initial user account creation; no admin token or vault secrets were recorded. - Created Proxmox snapshot `post-vaultwarden-service-installed` after endpoint verification. Files/services changed: - VM: `/etc/vaultwarden/env` (mode `0600`, no secret values recorded here) - VM: `/etc/systemd/system/vaultwarden.service` - VM: `/etc/nginx/sites-available/vaultwarden` - VM: `/etc/nginx/sites-enabled/vaultwarden` - VM: `/etc/ssl/vaultwarden/vm.dropcutstud.io.{crt,key}` - VM: `/opt/vaultwarden/data/` - Services: `docker`, `nginx`, `qemu-guest-agent`, `vaultwarden` - Repo: `.pi/ssh/hosts.json` alias `vaultwarden-vm` Verification: - `docker`, `nginx`, `qemu-guest-agent`, and `vaultwarden` reported active. - `curl -k --resolve vm.dropcutstud.io:443:192.168.0.238 https://vm.dropcutstud.io/` returned HTTP `200`. - `vaultwarden/server` reported Vaultwarden `1.36.0` / Web-Vault `2026.4.1`. - QEMU guest agent ping succeeded via `qm agent 102 ping` after package installation. - No critical user secrets were migrated; backup remains deferred and must be resolved before relying on the vault. - No passwords, private keys, API tokens, recovery codes, app passwords, or vault item values were logged. Rollback notes: - Preferred rollback before real secret migration: revert Proxmox snapshot `pre-vaultwarden-install` or destroy/recreate VMID `102` if abandoned. - Service-level rollback: `sudo systemctl disable --now vaultwarden nginx`; remove `/etc/systemd/system/vaultwarden.service`, `/etc/nginx/sites-available/vaultwarden`, `/etc/nginx/sites-enabled/vaultwarden`, and `/etc/ssl/vaultwarden/` if approved. - Preserve or explicitly destroy `/opt/vaultwarden/data/` according to whether any user-created vault data exists. - Revoke/reduce temporary `piagent` sudo after bootstrap hardening. ### 2026-06-06 update — Vaultwarden hostname correction - Server/host: Proxmox `buntbox01` / VMID `102` / `Vaultwarden` / `192.168.0.238` - Operator: Pi assistant via SSH as `piagent` - Reason: correct mistaken internal hostname from `vm.dropcutstud.io` to `vw.dropcutstud.io` after user created initial account. Actions summary: - Created Proxmox snapshot `pre-vw-hostname-change` before hostname/TLS changes. - Regenerated self-signed TLS certificate for `vw.dropcutstud.io` with SANs for `vw.dropcutstud.io` and `192.168.0.238`. - Updated `/etc/vaultwarden/env`: `DOMAIN=https://vw.dropcutstud.io`. - Disabled open signups after initial account creation: `SIGNUPS_ALLOWED=false`. - Updated Nginx `server_name` and certificate paths for `vw.dropcutstud.io`. - Restarted `nginx` and `vaultwarden`. - Created Proxmox snapshot `post-vw-hostname-change` after verification. Verification: - `nginx` and `vaultwarden` reported active. - `curl -k --resolve vw.dropcutstud.io:443:127.0.0.1 https://vw.dropcutstud.io/` returned HTTP `200`. - Confirmed `SIGNUPS_ALLOWED=false` and `INVITATIONS_ALLOWED=false`. - No passwords, private keys, API tokens, recovery codes, app passwords, or vault item values were logged. Rollback notes: - Revert Proxmox snapshot `pre-vw-hostname-change`, or restore timestamped backups under `/etc/vaultwarden/env.bak-*` and `/etc/nginx/sites-available/vaultwarden.bak-*`, then restart `nginx` and `vaultwarden`. ### 2026-06-06 update — Vaultwarden timed signup window for assistant account - Server/host: VMID `102` / `Vaultwarden` / `192.168.0.238` - Operator: Pi assistant via SSH as `piagent` - Reason: user clarified that assistant-safe secret access is a primary purpose of Vaultwarden; open a short controlled window for creating the dedicated `nimrod` assistant account. Actions summary: - Backed up `/etc/vaultwarden/env` to a timestamped `env.bak-before-nimrod-signup-*` file. - Temporarily set `SIGNUPS_ALLOWED=true`. - Restarted `vaultwarden.service`. - Started a background auto-disable task to set `SIGNUPS_ALLOWED=false` and restart Vaultwarden after approximately 20 minutes. Verification: - Confirmed `SIGNUPS_ALLOWED=true` after restart. - No account passwords, master passwords, API tokens, recovery codes, or vault item values were logged. Rollback notes: - Manually disable signups if needed: `sudo sed -i 's#^SIGNUPS_ALLOWED=.*#SIGNUPS_ALLOWED=false#' /etc/vaultwarden/env && sudo systemctl restart vaultwarden`. ### 2026-06-06 update — Assistant Vaultwarden account created and signups disabled - Server/host: VMID `102` / `Vaultwarden` / `192.168.0.238` - Operator: Pi assistant via SSH as `piagent`; user created the Vaultwarden account interactively. - Reason: create dedicated `nimrod` assistant Vaultwarden account for controlled future secret access pilot. Actions summary: - Temporarily used working hostname `vm.dropcutstud.io` for account creation because `vw.dropcutstud.io` still hits a DNS rebind issue. - User created the dedicated assistant account. - Disabled open signups by setting `SIGNUPS_ALLOWED=false`. - Restarted `vaultwarden.service`. Verification: - Confirmed `SIGNUPS_ALLOWED=false`. - `/api/config` reports `disableUserRegistration=true`. - `vaultwarden.service` is active. - No master passwords, account passwords, API tokens, recovery codes, or vault item values were logged. Follow-up: - Fix `vw.dropcutstud.io` DNS/rebind behavior and then switch `DOMAIN` back from `https://vm.dropcutstud.io` to `https://vw.dropcutstud.io`. - Define pilot collection/item sharing for a low-risk SearxNG-related credential. ### 2026-06-06 update — Vaultwarden canonical hostname restored - Server/host: VMID `102` / `Vaultwarden` / `192.168.0.238` - Operator: Pi assistant via SSH as `piagent` - Reason: user fixed LAN DNS/rebind for `vw.dropcutstud.io`; restore canonical Vaultwarden domain. Actions summary: - Backed up `/etc/vaultwarden/env` to a timestamped `env.bak-vw-domain-final-*` file. - Set `DOMAIN=https://vw.dropcutstud.io`. - Ensured `SIGNUPS_ALLOWED=false` remains set. - Restarted `vaultwarden.service`. Verification: - Proxmox host resolves `vw.dropcutstud.io` to `192.168.0.238`. - `curl -k https://vw.dropcutstud.io/` from Proxmox returned HTTP `200`. - Vaultwarden service reports active. - No secret values were logged. Follow-up: - Implement controlled assistant vault access using the dedicated `nimrod` account and a low-risk pilot collection/item. ## 2026-06-06 — Proxmox LXC 104 `nimrod` — Pi runtime and repo base migration - Server/host: Proxmox `192.168.0.88`, LXC 104 `nimrod` / `192.168.0.222` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` on Proxmox and LXC - Reason: continue isolated migration of Nimrod/Pi runtime to dedicated LXC without copying secrets/tokens. Actions summary: - Verified LXC 104 was running and reproduced Pi failure under Node `v20.19.2` (`webidl.util.markAsUncloneable`). - Added NodeSource Node 22 apt source inside LXC 104 and upgraded `nodejs` to `v22.22.3`. - Verified `pi --version` works for user `piagent` and returns `0.78.1`. - Copied committed Nimrod repo state via git bundle for local commit `052f1a5` and cloned it to `/home/piagent/projects/nimrod` inside LXC 104. - Hardened LXC SSH with `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf` to keep public-key auth enabled and disable password auth, keyboard-interactive auth, and root login. - Restarted SSH after creating `/run/sshd`; verified key-based `piagent@192.168.0.222` access still works. - Created Proxmox snapshot `base-runtime-repo-20260606` with description noting repo commit `052f1a5` and no secrets copied. Files/services changed: - LXC 104: `/etc/apt/keyrings/nodesource.gpg` - LXC 104: `/etc/apt/sources.list.d/nodesource.list` - LXC 104: `nodejs` package upgraded to NodeSource `22.22.3-1nodesource1` - LXC 104: `/home/piagent/projects/nimrod` - LXC 104: `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf` - LXC 104: `ssh.service` restarted and active - Proxmox: snapshot `base-runtime-repo-20260606` for CT 104 Verification: - `sudo pct exec 104 -- su - piagent -c "node -v; npm -v; command -v pi; pi --version"` returned Node `v22.22.3`, npm `10.9.8`, `/usr/local/bin/pi`, Pi `0.78.1`. - `su - piagent -c "cd /home/piagent/projects/nimrod && git status --short && git rev-parse --short HEAD && git log -1 --oneline"` returned clean status and `052f1a5 Document homelab operations and agent workflow`. - `sudo /usr/sbin/sshd -T` showed `permitrootlogin no`, `pubkeyauthentication yes`, `passwordauthentication no`, `kbdinteractiveauthentication no`. - `ssh -i .ssh/piagent_homelab piagent@192.168.0.222 'hostname; echo key-access-ok'` succeeded. - `sudo pct listsnapshot 104` showed `base-runtime-repo-20260606`. - No passwords, private keys, API tokens, recovery codes, or app-password values were copied or logged. Rollback notes: - Revert LXC state with Proxmox snapshot rollback if desired: `sudo pct rollback 104 base-runtime-repo-20260606` cannot undo to pre-change state because the snapshot was intentionally taken after this base setup; use an earlier backup/snapshot if needed. - To undo hardening only via Proxmox console: remove `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf`, run `/usr/sbin/sshd -t`, and restart `ssh.service`. - To undo repo copy: remove `/home/piagent/projects/nimrod`. ## 2026-06-06 — Proxmox LXC 104 `nimrod` — approved Pi access-material transfer - Server/host: LXC 104 `nimrod` / `192.168.0.222`; Proxmox SSH verification target `192.168.0.88` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` - Reason: copy narrowly approved Pi/homelab admin access materials into the migrated Nimrod repo on LXC 104. Actions summary: - Created/confirmed `/home/piagent/projects/nimrod/.ssh/` and `/home/piagent/projects/nimrod/.tokens/` with mode `700`. - Copied only `.ssh/piagent_homelab`, `.ssh/piagent_homelab.pub`, `.ssh/known_hosts`, and `.tokens/proxmox.env` from the local repo to LXC 104. - Did not copy Google client secret/auth URL or other secrets. - Set private key and token file to mode `600`; public key and known_hosts to mode `644`. Verification: - On LXC 104, `stat -c "%a %U:%G %n"` showed `.ssh` and `.tokens` as `700`, `.ssh/piagent_homelab` and `.tokens/proxmox.env` as `600`, and `.ssh/piagent_homelab.pub` / `.ssh/known_hosts` as `644`, all owned by `piagent:piagent`. - On LXC 104, verified token file existence and safe permissions with `test -f .tokens/proxmox.env` and `stat -c %a .tokens/proxmox.env`; token contents were not sourced or printed. - From LXC 104, `ssh -i /home/piagent/projects/nimrod/.ssh/piagent_homelab -o BatchMode=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/home/piagent/projects/nimrod/.ssh/known_hosts piagent@192.168.0.88 "hostname; echo proxmox-key-access-ok"` returned `buntbox01` and `proxmox-key-access-ok`. Rollback notes: - Remove the copied files/directories from `/home/piagent/projects/nimrod/.ssh/` and `/home/piagent/projects/nimrod/.tokens/` on LXC 104 if this access material should be withdrawn. ## 2026-06-06 — Proxmox LXC 104 `nimrod` — add user workstation SSH key for `piagent` - Server/host: LXC 104 `nimrod` / `192.168.0.222` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: allow the user's workstation public key to access the `piagent` account on the dedicated Nimrod LXC. Actions summary: - Preserved existing `/home/piagent/.ssh/authorized_keys` and appended the provided public key only if absent. - Ensured `/home/piagent/.ssh` is mode `700` and `/home/piagent/.ssh/authorized_keys` is mode `600`, owned by `piagent:piagent`. Verification: - `grep -Fxc` for the exact provided public key returned `1` on LXC 104. - `stat -c "%a %U:%G %n"` returned `700 piagent:piagent /home/piagent/.ssh` and `600 piagent:piagent /home/piagent/.ssh/authorized_keys`. - `ssh-keygen -lf /home/piagent/.ssh/authorized_keys` showed fingerprint `SHA256:ZFLWqQpaJq7UBOlJBVn3VmOz2IQsdPhDbVSTSI86kJc` with comment `deeso workstation nimrod lxc`. Rollback notes: - Remove the single authorized_keys line with comment `deeso workstation nimrod lxc` from `/home/piagent/.ssh/authorized_keys` on LXC 104. ## 2026-06-06 — Vaultwarden VM 102 — MVP encrypted backup - Server/host: VM 102 `Vaultwarden` / `192.168.0.238`; off-VM copy on LXC 104 `nimrod` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: create a basic encrypted backup before migrating critical secrets into Vaultwarden. Actions summary: - Installed Debian package `age` on the Vaultwarden VM. - Configured `/etc/vaultwarden/backup-recipient.txt` with the user's `deeso` SSH public key as an age-compatible public recipient; no private key was copied. - Installed `/usr/local/sbin/vaultwarden-backup`. - The backup script briefly stops `vaultwarden`, archives Vaultwarden data and service/config files, restarts `vaultwarden`, encrypts the archive with `age`, and writes a checksum manifest. - Created VM-local encrypted backup artifacts under `/var/backups/vaultwarden/`. - Copied the latest encrypted artifact and manifest off-VM to LXC 104 `nimrod` under `/home/piagent/backups/vaultwarden/`. Files/services changed: - Vaultwarden VM: package `age` - Vaultwarden VM: `/etc/vaultwarden/backup-recipient.txt` - Vaultwarden VM: `/usr/local/sbin/vaultwarden-backup` - Vaultwarden VM: `/var/backups/vaultwarden/vaultwarden-20260606T110259Z.tar.age` - Vaultwarden VM: `/var/backups/vaultwarden/vaultwarden-20260606T110259Z.manifest.txt` - Nimrod LXC: `/home/piagent/backups/vaultwarden/vaultwarden-20260606T110259Z.tar.age` - Nimrod LXC: `/home/piagent/backups/vaultwarden/vaultwarden-20260606T110259Z.manifest.txt` Verification: - `vaultwarden` service returned `active` after backup. - `sudo docker ps` showed `vaultwarden Up ... (healthy)` and local HTTPS returned HTTP `200`. - VM-local encrypted artifact was non-empty: `348452` bytes, mode `600`, owned by `root:root`. - VM-local checksum verification passed for `vaultwarden-20260606T110259Z.tar.age`. - Off-VM copied checksum verification passed on LXC 104. - No passwords, private keys, API tokens, recovery codes, app-password values, backup private keys, or vault item values were logged or committed. Rollback notes: - To remove the MVP backup job, delete `/usr/local/sbin/vaultwarden-backup` and `/etc/vaultwarden/backup-recipient.txt` from the Vaultwarden VM. - Preserve encrypted backup artifacts unless the user explicitly approves deletion. - This MVP is not complete recovery readiness until the user stores a copy and decryption material outside Vaultwarden and an isolated restore test succeeds. ## 2026-06-06 — Vaultwarden VM 102 — switch backups to dedicated age recipient - Server/host: VM 102 `Vaultwarden` / `192.168.0.238`; off-VM copy on LXC 104 `nimrod` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: replace SSH-key-based backup encryption recipient with a dedicated Vaultwarden-backup age recipient generated by the user. Actions summary: - Updated `/etc/vaultwarden/backup-recipient.txt` with the user's public dedicated `age1...` recipient. - Did not receive, copy, log, or store the private `AGE-SECRET-KEY-...` material. - Ran `/usr/local/sbin/vaultwarden-backup` to create a fresh backup encrypted to the dedicated age recipient. - Copied the fresh encrypted artifact and manifest off-VM to LXC 104 `nimrod` under `/home/piagent/backups/vaultwarden/`. Files/services changed: - Vaultwarden VM: `/etc/vaultwarden/backup-recipient.txt` - Vaultwarden VM: `/var/backups/vaultwarden/vaultwarden-20260606T112417Z.tar.age` - Vaultwarden VM: `/var/backups/vaultwarden/vaultwarden-20260606T112417Z.manifest.txt` - Nimrod LXC: `/home/piagent/backups/vaultwarden/vaultwarden-20260606T112417Z.tar.age` - Nimrod LXC: `/home/piagent/backups/vaultwarden/vaultwarden-20260606T112417Z.manifest.txt` Verification: - `vaultwarden` service returned `active` after backup. - Docker health later reported `vaultwarden ... (healthy)`. - Local HTTPS returned HTTP `200` after backup. - VM-local encrypted artifact was non-empty: `358680` bytes, mode `600`, owned by `root:root`. - VM-local checksum verification passed for `vaultwarden-20260606T112417Z.tar.age`. - Off-VM copied checksum verification passed on LXC 104. - No passwords, private keys, API tokens, recovery codes, app-password values, backup private keys, or vault item values were logged or committed. Rollback notes: - To return to the previous SSH public-key recipient, replace `/etc/vaultwarden/backup-recipient.txt` with the previous public recipient and run a fresh backup. - Existing older backups remain encrypted to their original recipient; preserve them unless the user explicitly approves deletion. - The current dedicated-age backup is not recoverable unless the user preserves the matching private age identity outside Vaultwarden. ## 2026-06-07 — Vaultwarden restore test — disposable CT 105 - Server/host: Production VM 102 `Vaultwarden` / `192.168.0.238`; disposable restore-test CT 105 `vaultwarden-restore-test` / `192.168.0.231`; Proxmox `buntbox01` / `192.168.0.88` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent`; user performed browser login verification. - Reason: prove Vaultwarden backups can be restored in isolation before treating Vaultwarden as authoritative for critical secrets. Actions summary: - Acquired Proxmox mutation lock in `state/locks/proxmox.lock`. - Created disposable unprivileged Debian 13 LXC CT 105 `vaultwarden-restore-test` with nesting enabled, DHCP LAN networking, 1 vCPU, 1536 MB RAM, 512 MB swap, and 8 GB root disk on `local-lvm`. - Installed `age`, Docker, SQLite tools, Curl, Nginx, and OpenSSL in the disposable CT. - Generated a temporary age identity inside CT 105 and temporarily added its public recipient alongside the permanent Vaultwarden backup age recipient. - Ran a fresh production backup: `vaultwarden-20260607T022724Z.tar.age`. - Restored production backup recipient configuration to only the permanent age recipient after the test backup was created. - Copied the test backup into CT 105, decrypted it using the temporary age identity, extracted it, and verified the SQLite database opened. - Started a restored Vaultwarden container on CT 105 with SMTP/invitations/signups/admin-token disabled in the test environment. - Added temporary self-signed HTTPS via Nginx on the restore-test target so the web vault could load in a secure browser context. - User confirmed they could log into the restored test vault. - Destroyed disposable CT 105 after successful verification. Files/services changed: - Proxmox: created and later destroyed CT 105 `vaultwarden-restore-test`. - Production Vaultwarden VM: temporarily updated `/etc/vaultwarden/backup-recipient.txt`, then restored it to the permanent single-recipient configuration. - Production Vaultwarden VM: created `/var/backups/vaultwarden/vaultwarden-20260607T022724Z.tar.age` and matching manifest. - Disposable CT 105: temporary restore files, age identity, Docker/Nginx configuration, and restored Vaultwarden data; all destroyed with the CT. Verification: - Production Vaultwarden was active and healthy before cleanup; local HTTPS returned HTTP `200`. - Production backup recipient file had one line after restoration, indicating only the permanent recipient remained. - Test backup checksum passed before restore. - Test backup decrypted successfully in CT 105 using the temporary age identity. - Restored database opened and reported expected metadata counts without logging secret values. - Restore-test endpoint returned HTTP `200` over HTTPS. - User confirmed successful login to the restored test vault. - CT 105 was destroyed and its disk removed. - Production Vaultwarden remained active after cleanup and local HTTPS returned HTTP `200`. - No passwords, private keys, API tokens, recovery codes, app-password values, backup private keys, or vault item values were logged or committed. Rollback notes: - CT 105 was destroyed; no restore-test service remains. - If a future restore is required, repeat the restore-test runbook using a fresh disposable target and the user's permanent age private key or a temporary recipient workflow. - Preserve production encrypted backup artifacts unless the user explicitly approves deletion. ## 2026-06-07 — Proxmox LXC 105 `reverse-proxy` — Nginx MVP for SearXNG - Server/host: Proxmox `buntbox01` / `192.168.0.88`; LXC 105 `reverse-proxy` / `192.168.0.137`; backend LXC 103 `searxng` / `192.168.0.133:8080` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: deploy the first central reverse proxy MVP after user clarified new service creation is allowed when Proxmox resources are verified first. Actions summary: - Used a read-only subagent preflight to verify live Proxmox resources and recommend a small LXC allocation. - Acquired Proxmox mutation lock in `state/locks/proxmox.lock`. - Created unprivileged Debian 13 LXC CT 105 `reverse-proxy` on `local-lvm` with 1 vCPU, 1024 MB RAM, 512 MB swap, 8 GB root disk, `vmbr0` DHCP networking, and `onboot=1`. - Installed Nginx, OpenSSL, Curl, CA certificates, sudo, and OpenSSH server. - Created `piagent` SSH administration account with the approved assistant public key and temporary passwordless sudo for bootstrap/maintenance. - Hardened SSH to disable password auth, keyboard-interactive auth, and root login. - Added project SSH alias `reverse-proxy` for `piagent@192.168.0.137`. - Created temporary self-signed TLS certificate for `search.dropcutstud.io` under `/etc/nginx/tls/`. - Added Nginx site config for `search.dropcutstud.io`, redirecting HTTP to HTTPS and proxying HTTPS to SearXNG backend `http://192.168.0.133:8080`. - Removed the default Nginx site and reloaded Nginx. - Created Proxmox snapshot `post-searxng-proxy-mvp`. Files/services changed: - Proxmox: CT 105 `reverse-proxy` - CT 105: packages `nginx`, `openssl`, `curl`, `ca-certificates`, `sudo`, `openssh-server` - CT 105: `/home/piagent/.ssh/authorized_keys` - CT 105: `/etc/sudoers.d/piagent` - CT 105: `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf` - CT 105: `/etc/nginx/tls/search.dropcutstud.io.crt` - CT 105: `/etc/nginx/tls/search.dropcutstud.io.key` - CT 105: `/etc/nginx/sites-available/search.dropcutstud.io.conf` - CT 105: `/etc/nginx/sites-enabled/search.dropcutstud.io.conf` - Proxmox: snapshot `post-searxng-proxy-mvp` Verification: - Live Proxmox preflight found CTID 105 available, Debian 13 LXC template available, `local-lvm` with about 134 GiB available, and no blockers for a small proxy LXC. - CT 105 received DHCP IP `192.168.0.137`. - SSH as `piagent@192.168.0.137` succeeded and `sudo -n true` worked. - `systemctl is-active nginx` returned `active`. - `nginx -t` succeeded without warnings after config adjustment. - From CT 105, backend `http://192.168.0.133:8080/` was reachable. - From CT 105, `https://search.dropcutstud.io/` using local resolve returned HTTP `200`. - From Nimrod LXC, `curl -k --resolve search.dropcutstud.io:443:192.168.0.137 https://search.dropcutstud.io/` returned HTTP `200`. - Direct backend `http://192.168.0.133:8080/` still returned HTTP `200` from Nimrod LXC. - `sudo pct listsnapshot 105` showed `post-searxng-proxy-mvp`. - No passwords, private keys, API tokens, recovery codes, app-password values, or vault item values were logged or committed. TLS private key exists only on CT 105 and is temporary/self-signed for MVP. Rollback notes: - Remove any temporary DNS/hosts entries for `search.dropcutstud.io`. - Disable route with `sudo rm /etc/nginx/sites-enabled/search.dropcutstud.io.conf && sudo nginx -t && sudo systemctl reload nginx` on CT 105. - Use direct SearXNG backend `http://192.168.0.133:8080/`. - Revert CT 105 to snapshot `post-searxng-proxy-mvp` for known-good MVP state, or destroy CT 105 if abandoning the proxy. ## 2026-06-07 — Proxmox LXC 106 `unbound` — standalone internal DNS MVP - Server/host: Proxmox `buntbox01` / `192.168.0.88`; LXC 106 `unbound` / `192.168.0.124`; reverse proxy LXC 105 / `192.168.0.137` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: deploy standalone Unbound internal DNS as requested by the user, separate from OPNsense plugin DNS. Actions summary: - Used a read-only subagent preflight to verify live Proxmox resources and recommend a small LXC allocation. - Acquired Proxmox mutation lock in `state/locks/proxmox.lock`. - Created unprivileged Debian 13 LXC CT 106 `unbound` on `local-lvm` with 1 vCPU, 512 MB RAM, 256 MB swap, 8 GB root disk, `vmbr0` DHCP networking, and `onboot=1`. - Installed Unbound, DNS utilities, Curl, sudo, and OpenSSH server. - Created `piagent` SSH administration account with the approved assistant public key and temporary passwordless sudo for bootstrap/maintenance. - Hardened SSH to disable password auth, keyboard-interactive auth, and root login. - Added project SSH alias `unbound-dns` for `piagent@192.168.0.124`. - Created `/etc/unbound/unbound.conf.d/homelab.conf` with LAN/Tailscale ACLs and internal `dropcutstud.io` records for `dns`, `unbound`, `proxy`, `search`, `vw`, and `nc`. - Restarted Unbound and created Proxmox snapshot `post-unbound-internal-dns-mvp`. Files/services changed: - Proxmox: CT 106 `unbound` - CT 106: packages `unbound`, `dnsutils`, `curl`, `sudo`, `openssh-server` - CT 106: `/home/piagent/.ssh/authorized_keys` - CT 106: `/etc/sudoers.d/piagent` - CT 106: `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf` - CT 106: `/etc/unbound/unbound.conf.d/homelab.conf` - Proxmox: snapshot `post-unbound-internal-dns-mvp` Verification: - Live Proxmox preflight found CTID 106 available, Debian 13 LXC template available, `local-lvm` with about 133 GiB available, and no blockers for a small DNS LXC. - CT 106 received DHCP IP `192.168.0.124`. - SSH as `piagent@192.168.0.124` succeeded and `sudo -n true` worked. - `systemctl is-active unbound` returned `active`. - `ss` showed Unbound listening on TCP/UDP port 53 on IPv4 and IPv6 wildcard interfaces. - From Nimrod LXC, `dig @192.168.0.124 search.dropcutstud.io +short` returned `192.168.0.137`. - From Nimrod LXC, `dig @192.168.0.124 proxy.dropcutstud.io +short` returned `192.168.0.137`. - External recursive lookup via Unbound returned answers for `example.com` after an initial timeout. - `curl -k --resolve search.dropcutstud.io:443:192.168.0.137 https://search.dropcutstud.io/` returned HTTP `200`. - `sudo pct listsnapshot 106` showed `post-unbound-internal-dns-mvp`. - No passwords, private keys, API tokens, recovery codes, app-password values, or vault item values were logged or committed. Rollback notes: - Do not point DHCP/router/Tailscale clients at `192.168.0.124` until stable IP/reservation and rollback are approved. - If a record is wrong, edit `/etc/unbound/unbound.conf.d/homelab.conf`, run `unbound-checkconf`, and restart `unbound`. - If abandoning this DNS service, stop/destroy CT 106 or revert to snapshot `post-unbound-internal-dns-mvp`; remove any router/DHCP/Tailscale DNS references first. ## 2026-06-07 — Proxmox LXC 107 `homepage` — config-as-code dashboard MVP - Server/host: Proxmox `buntbox01` / `192.168.0.88`; LXC 107 `homepage` / `192.168.0.241`; reverse proxy CT 105 / `192.168.0.137`; Unbound CT 106 / `192.168.0.124` - Operator: Pi assistant via SSH key `.ssh/piagent_homelab` as `piagent` - Reason: deploy an admin dashboard Nimrod can manage through files/config rather than web GUI configuration. Actions summary: - Used a read-only subagent research pass to compare dashboard options; selected Homepage as the best config-as-code MVP. - Used a read-only subagent Proxmox preflight to verify resources for a small Docker-capable dashboard LXC. - Acquired Proxmox mutation lock in `state/locks/proxmox.lock`. - Created unprivileged Debian 13 LXC CT 107 `homepage` on `local-lvm` with nesting enabled, 1 vCPU, 1024 MB RAM, 512 MB swap, 8 GB root disk, `vmbr0` DHCP networking, and `onboot=1`. - Installed Docker, Docker Compose, Curl, sudo, OpenSSH server, and CA certificates. - Created `piagent` SSH administration account with the approved assistant public key and temporary passwordless sudo for bootstrap/maintenance. - Hardened SSH to disable password auth, keyboard-interactive auth, and root login. - Deployed Homepage via Docker Compose under `/opt/homepage` without mounting the Docker socket. - Created config-as-code dashboard files under `/opt/homepage/config` with initial service registrations for Vaultwarden, SearXNG, Nextcloud, Nimrod, reverse proxy, Unbound, Proxmox, and operational links. - Added `dashboard.dropcutstud.io` internal DNS record to Unbound, pointing to the reverse proxy. - Added `dashboard.dropcutstud.io` temporary self-signed TLS route on the reverse proxy to backend `http://192.168.0.241:3000`. - Created Proxmox snapshots `post-homepage-dashboard-mvp`, `post-dashboard-proxy-route`, and `post-dashboard-dns-record`. Files/services changed: - Proxmox: CT 107 `homepage` - CT 107: packages `docker.io`, `docker-compose`, `curl`, `sudo`, `openssh-server`, `ca-certificates` - CT 107: `/home/piagent/.ssh/authorized_keys` - CT 107: `/etc/sudoers.d/piagent` - CT 107: `/etc/ssh/sshd_config.d/50-nimrod-hardening.conf` - CT 107: `/opt/homepage/docker-compose.yml` - CT 107: `/opt/homepage/config/settings.yaml` - CT 107: `/opt/homepage/config/services.yaml` - CT 107: `/opt/homepage/config/bookmarks.yaml` - CT 107: `/opt/homepage/config/widgets.yaml` - CT 105: `/etc/nginx/sites-available/dashboard.dropcutstud.io.conf` - CT 105: `/etc/nginx/sites-enabled/dashboard.dropcutstud.io.conf` - CT 105: `/etc/nginx/tls/dashboard.dropcutstud.io.crt` - CT 105: `/etc/nginx/tls/dashboard.dropcutstud.io.key` - CT 106: `/etc/unbound/unbound.conf.d/homelab.conf` Verification: - Live Proxmox preflight found CTID 107 available, Debian 13 LXC template available, `local-lvm` with about 132 GiB available, and no blockers for a small Homepage LXC. - CT 107 received DHCP IP `192.168.0.241`. - SSH as `piagent@192.168.0.241` succeeded and `sudo -n true` worked. - `systemctl is-active docker` returned `active`. - Homepage direct endpoint `http://127.0.0.1:3000/` and `http://192.168.0.241:3000/` returned HTTP `200`. - `docker ps` later showed `homepage ... (healthy)`. - `dig @192.168.0.124 dashboard.dropcutstud.io +short` returned `192.168.0.137`. - `nginx -t` succeeded on the reverse proxy. - `curl -k --resolve dashboard.dropcutstud.io:443:192.168.0.137 https://dashboard.dropcutstud.io/` returned HTTP `200`. - No passwords, private keys, API tokens, recovery codes, app-password values, or vault item values were logged or committed. TLS private key exists only on CT 105 and is temporary/self-signed for MVP. Rollback notes: - Use direct service URLs from inventory if the dashboard is unavailable. - Disable dashboard proxy route on CT 105 by removing `/etc/nginx/sites-enabled/dashboard.dropcutstud.io.conf`, validating Nginx, and reloading. - Remove or correct `dashboard.dropcutstud.io` in CT 106 Unbound config if needed. - Revert CT 107 to snapshot `post-homepage-dashboard-mvp`, or destroy CT 107 if abandoning Homepage. ## 2026-06-07 — Nimrod LXC / Homepage LXC — Ansible update-management MVP trial - Server/host: Nimrod LXC 104 / `192.168.0.222`; Homepage LXC 107 / `192.168.0.241` - Operator: Pi assistant running on Nimrod LXC - Reason: implement requested Ansible-based update management and prove the process on a low-risk non-critical service. Actions summary: - Used a planner subagent to create Ansible MVP plan at `tickets/artifacts/2026-06-06-managed-update-schedules-ansible/05-plan.md`. - Installed Debian `ansible` package on Nimrod LXC. - Added repo Ansible config, static homelab inventory, read-only preflight playbook, read-only update-check playbook, and manually-gated apt safe-upgrade playbook. - Added runbook `runbooks/homelab-managed-updates.md`. - Ran read-only Ansible inventory/preflight/update-check against `homepage-dashboard` as the first experimental target. - Ran manually-gated safe apt upgrade on `homepage-dashboard` with `target_update_class=experimental`. - Did not reboot; no reboot was required. Files/services changed: - Nimrod LXC: package `ansible` and dependencies installed. - Repo: `ansible.cfg` - Repo: `ansible/ansible.cfg` - Repo: `ansible/inventories/homelab/hosts.yml` - Repo: `ansible/playbooks/preflight.yml` - Repo: `ansible/playbooks/check-updates.yml` - Repo: `ansible/playbooks/apt-upgrade.yml` - Repo: `ansible/requirements.yml` - Repo: `runbooks/homelab-managed-updates.md` - Homepage LXC: Debian packages upgraded via Ansible safe apt upgrade. Verification: - `ansible --version` reported Ansible core `2.19.4` and config file `/home/piagent/projects/nimrod/ansible.cfg`. - `ansible-inventory --graph` showed expected groups and first trial host. - `ansible-playbook ansible/playbooks/preflight.yml --limit first_trial` succeeded. - `ansible-playbook ansible/playbooks/check-updates.yml --limit first_trial` initially reported 56 upgradable packages. - `ansible-playbook ansible/playbooks/apt-upgrade.yml --limit first_trial -e target_update_class=experimental` succeeded with changes and no reboot required. - Post-update check reported 0 upgradable packages for `homepage-dashboard`. - Homepage direct endpoint `http://192.168.0.241:3000/` returned HTTP `200`. - Dashboard proxy endpoint `https://dashboard.dropcutstud.io/` via reverse proxy returned HTTP `200` using `curl --resolve`. - `docker ps` showed `homepage ... (healthy)`. - No passwords, private keys, API tokens, recovery codes, app-password values, or vault item values were logged or committed. Rollback notes: - For Ansible repo layout rollback, revert the committed Ansible files/runbook. - For Homepage package-update rollback, first use `runbooks/homepage-dashboard.md` to restart/repair Docker Compose. - If needed and explicitly approved, revert CT 107 to Proxmox snapshot `post-homepage-dashboard-mvp`. - Critical/manual hosts remain gated; no broad or automatic update schedule was configured. ## 2026-06-07 — Homepage LXC — Config-level backup trial - Server/host: LXC 107 `homepage` / `192.168.0.241` - Operator: Nimrod/Pi assistant via Ansible from LXC 104 - Reason: establish the first non-critical service backup under `tickets/active/2026-06-07-service-backup-standard.md`. Actions summary: - Installed `/usr/local/sbin/homepage-backup` on `homepage-dashboard` from repo source `ansible/files/homepage-backup`. - Script creates a timestamped tarball of `/opt/homepage/docker-compose.yml` and `/opt/homepage/config/` under `/var/backups/homepage/` with a SHA256 manifest. - Re-copied the script from repo source after adding it to configuration management. - Ran the backup once successfully. - Fetched the artifact and manifest to Nimrod off-guest bootstrap storage at `/home/piagent/backups/homepage/`. - Set Nimrod backup staging directory mode to `0700` and artifact mode to `0600`. Files/services changed: - `homepage-dashboard:/usr/local/sbin/homepage-backup` - `homepage-dashboard:/var/backups/homepage/homepage-20260607T042801Z.tar.gz` - `homepage-dashboard:/var/backups/homepage/homepage-20260607T042801Z.sha256` - `nimrod-lxc:/home/piagent/backups/homepage/homepage-20260607T042801Z.tar.gz` - `nimrod-lxc:/home/piagent/backups/homepage/homepage-20260607T042801Z.sha256` Verification: - Ansible preflight succeeded for `homepage-dashboard`; Debian 13.5, no reboot required. - `/usr/local/sbin/homepage-backup` exited `0` and reported `homepage-20260607T042801Z.tar.gz: OK`. - Off-guest checksum verification on Nimrod succeeded with `sha256sum -c homepage-20260607T042801Z.sha256`. - Backup artifact size observed: 1.2 KiB; dashboard config should not contain secrets. Rollback notes: - Remove `/usr/local/sbin/homepage-backup` and `/var/backups/homepage/` from the Homepage LXC if this bootstrap approach is replaced. - Remove `/home/piagent/backups/homepage/` copies from Nimrod after confirming a better backup destination has the artifacts. - Restore testing is still pending and should use an isolated target, not the production Homepage LXC. ## 2026-06-07 — Unbound LXC — Config-level backup backfill - Server/host: LXC 106 `unbound` / `192.168.0.124` - Operator: Nimrod/Pi assistant via Ansible from LXC 104 - Reason: backfill a standard class config-level backup for internal DNS under `tickets/active/2026-06-07-service-backup-standard.md`. Actions summary: - Installed `/usr/local/sbin/unbound-backup` on `unbound-dns` from repo source `ansible/files/unbound-backup`. - Script creates a timestamped tarball of `/etc/unbound/unbound.conf` and `/etc/unbound/unbound.conf.d/` under `/var/backups/unbound/` with a SHA256 manifest. - Ran the backup once successfully. - Fetched the artifact and manifest to Nimrod off-guest bootstrap storage at `/home/piagent/backups/unbound/`. - Set Nimrod backup staging directory mode to `0700` and artifact mode to `0600`. Files/services changed: - `unbound-dns:/usr/local/sbin/unbound-backup` - `unbound-dns:/var/backups/unbound/unbound-20260607T044755Z.tar.gz` - `unbound-dns:/var/backups/unbound/unbound-20260607T044755Z.sha256` - `nimrod-lxc:/home/piagent/backups/unbound/unbound-20260607T044755Z.tar.gz` - `nimrod-lxc:/home/piagent/backups/unbound/unbound-20260607T044755Z.sha256` Verification: - `/usr/local/sbin/unbound-backup` exited `0` and reported `unbound-20260607T044755Z.tar.gz: OK`. - Off-guest checksum verification on Nimrod succeeded with `sha256sum -c unbound-20260607T044755Z.sha256`. - `systemctl is-active unbound` returned `active`. - `dig @192.168.0.124 search.dropcutstud.io +short` returned `192.168.0.137`. - `dig @192.168.0.124 example.com +short` returned public A records after one timeout/retry, so upstream recursion may need observation but service remained functional. Rollback notes: - Remove `/usr/local/sbin/unbound-backup` and `/var/backups/unbound/` from the Unbound LXC if this bootstrap approach is replaced. - Remove `/home/piagent/backups/unbound/` copies from Nimrod after confirming a better backup destination has the artifacts. - Restore testing is still pending and should use an isolated target, not the production Unbound LXC. ## 2026-06-07 — Reverse proxy LXC — Nginx config backup backfill - Server/host: LXC 105 `reverse-proxy` / `192.168.0.137` - Operator: Nimrod/Pi assistant via Ansible from LXC 104 - Reason: backfill a standard class config-level backup for Nginx reverse-proxy routing under `tickets/active/2026-06-07-service-backup-standard.md`. Actions summary: - Installed `/usr/local/sbin/reverse-proxy-backup` on `reverse-proxy` from repo source `ansible/files/reverse-proxy-backup`. - Script creates a timestamped tarball of Nginx config paths under `/var/backups/reverse-proxy/` with a SHA256 manifest. - Current scope intentionally excludes `/etc/nginx/tls/` so private TLS keys are not copied through an unencrypted bootstrap backup path. - Ran the backup once successfully. - Fetched the artifact and manifest to Nimrod off-guest bootstrap storage at `/home/piagent/backups/reverse-proxy/`. - Set Nimrod backup staging directory mode to `0700` and artifact mode to `0600`. Files/services changed: - `reverse-proxy:/usr/local/sbin/reverse-proxy-backup` - `reverse-proxy:/var/backups/reverse-proxy/reverse-proxy-nginx-config-20260607T045043Z.tar.gz` - `reverse-proxy:/var/backups/reverse-proxy/reverse-proxy-nginx-config-20260607T045043Z.sha256` - `nimrod-lxc:/home/piagent/backups/reverse-proxy/reverse-proxy-nginx-config-20260607T045043Z.tar.gz` - `nimrod-lxc:/home/piagent/backups/reverse-proxy/reverse-proxy-nginx-config-20260607T045043Z.sha256` Verification: - `/usr/local/sbin/reverse-proxy-backup` exited `0` and reported `reverse-proxy-nginx-config-20260607T045043Z.tar.gz: OK`. - Off-guest checksum verification on Nimrod succeeded with `sha256sum -c reverse-proxy-nginx-config-20260607T045043Z.sha256`. - `nginx -t` reported syntax OK and successful config test. - `systemctl is-active nginx` returned `active`. - `curl -k --resolve search.dropcutstud.io:443:192.168.0.137 https://search.dropcutstud.io/ -I` returned HTTP `200`. - `curl -k --resolve dashboard.dropcutstud.io:443:192.168.0.137 https://dashboard.dropcutstud.io/ -I` returned HTTP `200`. Rollback notes: - Remove `/usr/local/sbin/reverse-proxy-backup` and `/var/backups/reverse-proxy/` from the reverse-proxy LXC if this bootstrap approach is replaced. - Remove `/home/piagent/backups/reverse-proxy/` copies from Nimrod after confirming a better backup destination has the artifacts. - TLS private-key backup is intentionally not covered yet; use existing live keys/snapshots until an encrypted path is approved. - Restore testing is still pending and should use an isolated target, not the production reverse-proxy LXC. ## 2026-06-07 — SearXNG LXC / Nimrod LXC — Encrypted config backup backfill - Server/host: LXC 103 `searxng` / `192.168.0.133`, via Proxmox `buntbox01` / `192.168.0.88`; backup staging on LXC 104 `nimrod` - Operator: Nimrod/Pi assistant via SSH/PCT from LXC 104 - Reason: backfill a standard class config-level backup for SearXNG under `tickets/active/2026-06-07-service-backup-standard.md` while protecting SearXNG's service `secret_key`. Actions summary: - Live inspection confirmed SearXNG config files under `/opt/searxng/` and service unit `/etc/systemd/system/searxng-compose.service`. - A command accidentally printed the old SearXNG `secret_key` to terminal output. The value was not written to repo docs or change logs. - Rotated the SearXNG `secret_key` in `/opt/searxng/searxng/settings.yml` and restarted `searxng-compose.service`; no secret value was logged. - Installed Debian package `age` on Nimrod LXC 104 for encrypted backup creation. - Copied only the public age recipient from Vaultwarden backup recipient configuration to Nimrod staging at `/home/piagent/backups/age-recipients/vaultwarden-backup-recipient.txt`; no private age identity was copied. - Added and ran repo-managed script `ansible/files/searxng-encrypted-backup-from-nimrod`, which streams selected config files from CT 103 through the Proxmox host and writes an age-encrypted artifact on Nimrod. - Created encrypted backup artifact and checksum in `/home/piagent/backups/searxng/` with mode `0600`. Files/services changed: - `searxng:/opt/searxng/searxng/settings.yml` — service secret key rotated; value not logged - `searxng:searxng-compose.service` — restarted - `nimrod-lxc:/usr/bin/age` installed from Debian package `age` - `nimrod-lxc:/home/piagent/backups/age-recipients/vaultwarden-backup-recipient.txt` — public recipient only - `nimrod-lxc:/home/piagent/backups/searxng/searxng-config-20260607T051746Z.tar.gz.age` - `nimrod-lxc:/home/piagent/backups/searxng/searxng-config-20260607T051746Z.sha256` Verification: - SearXNG service restart returned `active`. - Encrypted artifact checksum verification succeeded with `sha256sum -c searxng-config-20260607T051746Z.sha256`. - Backup artifact is encrypted (`.tar.gz.age`) and no unencrypted SearXNG config artifact was stored in Nimrod backup staging. - `pct status 103` returned `running`. - `systemctl is-active searxng-compose.service` returned `active`. - `docker-compose ps` showed `searxng` and `valkey` containers up. - Direct SearXNG check `http://192.168.0.133:8080/` returned HTTP `200`. - Reverse-proxy check `https://search.dropcutstud.io/` via `192.168.0.137` returned HTTP `200`. Rollback notes: - The old SearXNG `secret_key` should be considered exposed and should not be restored. - If SearXNG behavior regresses, generate a new `secret_key`, restart `searxng-compose.service`, and verify direct/proxy HTTP checks. - Remove `/home/piagent/backups/searxng/` encrypted artifacts only after confirming a replacement encrypted backup destination has them or the user approves deletion. - Remove `age` from Nimrod only if no encrypted backup workflows depend on it. - Decrypt/restore testing is still pending and requires the user-controlled age identity; do not store the private identity in git or ordinary backup staging. ## 2026-06-07 — Homepage LXC — Isolated backup restore test - Server/host: LXC 107 `homepage` / `192.168.0.241` - Operator: Nimrod/Pi assistant via Ansible from LXC 104 - Reason: verify the Homepage config-level backup can restore into an isolated runtime without overwriting production config. Actions summary: - Used the latest guest-local Homepage backup artifact from `/var/backups/homepage/`. - Extracted the artifact into a temporary directory under `/tmp/` on CT 107. - Started a temporary Homepage container using the restored config directory mounted read-only and bound only to `127.0.0.1:13000`. - Verified the temporary restored Homepage endpoint returned HTTP `200`. - Removed the temporary container and temporary restore directory after the test. Files/services changed: - Temporary `/tmp/homepage-restore-test-*` directory created and removed. - Temporary Docker container `homepage-restore-test-*` created and removed. - Production `/opt/homepage/` config and production `homepage` container were not modified. Verification: - Temporary restored Homepage endpoint `http://127.0.0.1:13000/` returned HTTP `200`. - An initial curl retry saw a transient connection reset while the temporary container was starting; final verification succeeded. Rollback notes: - No production rollback required; test artifacts were removed. - If a future restore test leaves a temporary container behind, remove it with `sudo docker rm -f ` and delete the matching `/tmp/homepage-restore-test-*` directory. ## 2026-06-07 — Unbound LXC — Isolated backup restore test - Server/host: LXC 106 `unbound` / `192.168.0.124` - Operator: Nimrod/Pi assistant via Ansible from LXC 104 - Reason: verify the Unbound config-level backup can restore into an isolated runtime without overwriting production DNS config or changing router/DHCP/Tailscale settings. Actions summary: - Verified the off-guest backup artifact checksum on Nimrod with `sha256sum -c unbound-20260607T044755Z.sha256`. - Copied the off-guest backup artifact `unbound-20260607T044755Z.tar.gz` to `/tmp/` on CT 106 for testing, then removed it after the test. - Extracted the artifact into a temporary `/tmp/unbound-restore-test.*` directory. - Rewrote only the temporary restored config to include the temporary config directory and bind to `127.0.0.1:1053` instead of production port 53. - Ran `unbound-checkconf` against the restored temporary config. - Started a temporary `unbound -d` process using the restored temporary config. - Queried restored records through the temporary process. - Cleaned up the temporary process and restore directory. Files/services changed: - Temporary `/tmp/unbound-20260607T044755Z.tar.gz` on CT 106 was created and removed. - Temporary `/tmp/unbound-restore-test.*` directory was created and removed. - Production `/etc/unbound/` config and the production `unbound` service were not modified. - Router, DHCP, OPNsense, and Tailscale DNS settings were not changed. Verification: - `unbound-checkconf` reported no errors for the restored temporary config. - Temporary restored Unbound answered `search.dropcutstud.io` as `192.168.0.137` on `127.0.0.1:1053`. - Temporary restored Unbound answered `proxy.dropcutstud.io` as `192.168.0.137` on `127.0.0.1:1053`. - Temporary restored Unbound answered `dashboard.dropcutstud.io` as `192.168.0.137` on `127.0.0.1:1053`. - Production `systemctl is-active unbound` returned `active` after the test. - Cleanup check found no remaining temporary restore directory, copied artifact, or temporary `unbound -d -c /tmp/unbound-restore-test.*` process. Rollback notes: - No production rollback required; production DNS config/service was not changed. - If a future restore test leaves a temporary process behind, stop the specific test PID and delete only the matching `/tmp/unbound-restore-test.*` directory. ## 2026-06-07 — SearXNG backup path — Bypass age identity - Server/host: LXC 103 `searxng` / `192.168.0.133`, via Proxmox `buntbox01` / `192.168.0.88`; backup staging on LXC 104 `nimrod` - Operator: Nimrod/Pi assistant from LXC 104 - Reason: user directed that SearXNG should not use age identity; replace the SearXNG age-based config backup path with a sanitized backup path. Actions summary: - Removed the repo-managed age-based SearXNG backup helper from current configuration. - Added `ansible/files/searxng-sanitized-backup-from-nimrod`. - Ran the sanitized backup helper from Nimrod. - The helper streams selected SearXNG files from CT 103 and writes a `.tar.gz` artifact on Nimrod with `settings.yml` `secret_key` replaced by `__REGENERATE_ON_RESTORE__`. - Removed the public age-recipient staging file previously copied to Nimrod for the SearXNG age-based backup path: `/home/piagent/backups/age-recipients/vaultwarden-backup-recipient.txt`. - Left existing historical encrypted SearXNG artifacts in place. - Did not change Vaultwarden critical backup encryption, recipient configuration, backup script, or restore process. Files/services changed: - Repo: removed `ansible/files/searxng-encrypted-backup-from-nimrod` from current configuration. - Repo: added `ansible/files/searxng-sanitized-backup-from-nimrod`. - Nimrod: created `/home/piagent/backups/searxng/searxng-sanitized-config-20260607T054700Z.tar.gz`. - Nimrod: created `/home/piagent/backups/searxng/searxng-sanitized-config-20260607T054700Z.sha256`. - Nimrod: removed `/home/piagent/backups/age-recipients/vaultwarden-backup-recipient.txt`; removed the directory if empty. - Production SearXNG config and service were not modified. Verification: - `bash -n ansible/files/searxng-sanitized-backup-from-nimrod` passed. - Sanitized backup command reported `searxng-sanitized-config-20260607T054700Z.tar.gz: OK`. - `sha256sum -c searxng-sanitized-config-20260607T054700Z.sha256` succeeded. - Archive listing showed expected `opt/searxng/...` and `etc/systemd/system/searxng-compose.service` paths. - Safe extracted inspection confirmed the `secret_key` line is `secret_key: "__REGENERATE_ON_RESTORE__"`. - Backup staging directory mode remained `0700`; sanitized artifact and manifest mode are `0600`. - Direct SearXNG check `http://192.168.0.133:8080/` returned HTTP `200`. - Reverse-proxy check `https://search.dropcutstud.io/` via `192.168.0.137` returned HTTP `200`. Rollback notes: - If the sanitized backup path fails, stop using the sanitized artifact and restore the prior repo script from git history; do not restore an old exposed `secret_key`. - Existing historical encrypted SearXNG artifacts remain available but should not be the expected restore path going forward unless the user explicitly re-approves age-based handling. - To restore from sanitized SearXNG backup, generate a fresh `secret_key` and replace `__REGENERATE_ON_RESTORE__` before starting SearXNG. ## 2026-06-07 — All managed services — Add user SSH key and set up SearXNG SSH - Server/host: All managed guests (vaultwarden, reverse-proxy, unbound, homepage, nextcloud, searxng) - Operator: Nimrod/Pi assistant via Ansible/SSH from LXC 104 - Reason: ensure deeso can SSH into all managed services, and set up SSH on SearXNG which had none. Actions summary: - Added the user's public SSH key (`deeso workstation nimrod lxc`) to authorized_keys on: - vaultwarden-vm, reverse-proxy, unbound-dns, homepage-dashboard, nextcloud-vm - On SearXNG CT 103: - Created `piagent` user, home directory, and `.ssh` directory - Added both Nimrod's key and the user's key to `authorized_keys` - Created sudoers file for passwordless sudo - Upgraded and enabled SSH server - Verified SSH connection works from Nimrod Verification: - All 6 services now have 2 keys: Nimrod's piagent_homelab key and the user's key. - SearXNG SSH access verified by direct connection from Nimrod (hostname returned `searxng`). Rollback notes: - Remove the user's key from any service by deleting that line from `/home/piagent/.ssh/authorized_keys` on the target. ## 2026-06-07 — LAN DNS — Switched DHCP to advertise Unbound - Server/host: OPNsense `opn.dropcutstud.io` / `192.168.0.1` - Operator: User via OPNsense web UI - Reason: make Unbound the primary LAN DNS server so internal `dropcutstud.io` names resolve for all DHCP clients. Actions summary: - User switched all services from DHCP to static IP configuration. - User set OPNsense DNSmasq DHCP to advertise `192.168.0.124` as the LAN DNS server. - User confirmed `dashboard.dropcutstud.io` resolves on their workstation. - Added `opn.dropcutstud.io → 192.168.0.1` to Unbound records. - Updated runbook with DHCP note and new record. Verification: - All internal `dropcutstud.io` records resolve via Unbound `192.168.0.124`: - `dashboard.dropcutstud.io` → `192.168.0.137` ✅ - `search.dropcutstud.io` → `192.168.0.137` ✅ - `vw.dropcutstud.io` → `192.168.0.238` ✅ - `nc.dropcutstud.io` → `192.168.0.110` ✅ - `opn.dropcutstud.io` → `192.168.0.1` ✅ - `proxy.dropcutstud.io` → `192.168.0.137` ✅ Rollback notes: - Revert OPNsense DNSmasq DHCP DNS field to the previous DNS server address. - No changes needed on Unbound or any service. - If `opn.dropcutstud.io` needs to be removed from Unbound, delete the `local-data` line from `/etc/unbound/unbound.conf.d/homelab.conf` and restart Unbound. ## 2026-06-07 — Pi-hole CT 108 and Gluetun CT 109 deployed - Server/host: Proxmox `buntbox01` / `192.168.0.88` - Operator: Nimrod/Pi assistant via Proxmox SSH - Reason: deploy Pi-hole ad-blocking DNS and Gluetun PIA VPN proxy. Pi-hole: - CT 108 `pihole` at `192.168.0.150`, Debian 13, 512MB, 8GB disk, Docker-based. - Upstream DNS: Unbound `192.168.0.124`. - Ad-blocking verified: `doubleclick.net` → `0.0.0.0`, `dashboard.dropcutstud.io` → `192.168.0.137`. - Admin UI: `http://192.168.0.150/admin/` password: `admin`. - DHCP DNS still points at Unbound; user needs to change OPNsense DHCP DNS to `192.168.0.150` for network-wide ad-blocking. Gluetun: - CT 109 `gluetun` at `192.168.0.151`, Debian 13, 512MB, 8GB disk, Docker-based. - PIA OpenVPN connected, public IP `37.19.197.190` (New York). - HTTP proxy at `192.168.0.151:8888`, SOCKS5 at `192.168.0.151:8388`. - TUN device configured in LXC config for VPN tunnel. - Credentials sourced from `.tokens/pia.txt`. - DNS chain: Gluetun → Pi-hole → Unbound. Verification: - Pi-hole DNS queries respond correctly for internal and public names. - Pi-hole blocks known ad domains. - Gluetun HTTP proxy responds with HTTP 200. - Gluetun logs show "Initialization Sequence Completed" and healthy status. Rollback notes: - Pi-hole: `sudo pct stop 108 && sudo pct destroy 108` to remove. - Gluetun: `sudo pct stop 109 && sudo pct destroy 109` to remove. - To switch back DNS, revert OPNsense DHCP DNS to `192.168.0.124`. ## 2026-06-07 — Added AMP gameserver to Unbound, Ansible, dashboard, and registry - Server/host: LXC 101 `amp-gameserver` / `192.168.0.90` - Operator: Nimrod/Pi assistant - Reason: integrate existing AMP gameserver into homelab management. Actions summary: - Added `amp.dropcutstud.io → 192.168.0.90` to Unbound. - Added AMP to Ansible inventory (experimental update class, standard backup class). - Added AMP to Homepage dashboard. - Added AMP to Proxmox registry with IP/memory/disk details. - Added user SSH key to AMP's authorized_keys. ## 2026-06-07 — Dashy widget update + new Homarr deployment - Server/host: LXC 110 `dashy` / `192.168.0.152`, LXC 111 `homarr` / `192.168.0.213` - Operator: Nimrod/Pi assistant - Reason: Add missing services to Dashy dashboard, deploy fresh Homarr as preferred dashboard. Dashy update: - Added to Dashy conf.yml: Homepage Dashboard, AMP Game Server, Home Assistant, Unbound DNS, Proxmox (proxtop), Dashy itself, Ops Docs tile. - Reorganized into 4 sections: Overview (clock/search widgets), Core Services (6 apps), Infrastructure (6 infra services), Operations (docs/changelog). - All services with web UIs have statusCheck enabled. - Config pushed via Proxmox `pct push` into LXC 110, container restarted. Homarr deployment (CT 111): - New LXC on buntbox01, Debian 13, 2 vCPU, 2GB RAM, 12GB disk, DHCP → `192.168.0.213`. - Docker installed, Homarr deployed via Docker Compose with admin env vars. - Admin credentials: username `deeso`, password `homarr2026`. - Piagent SSH key deployed for future management. - DNS: `homarr.dropcutstud.io → 192.168.0.137` (reverse proxy). - Nginx proxy configured with WebSocket support on the existing reverse-proxy LXC 105. - Old Homarr LXC (CT 100) on proxtop still running but unused; can be stopped/destroyed. Verification: - Dashy at https://dashy.dropcutstud.io/ — HTTP 200, new services visible. - Homarr at https://homarr.dropcutstud.io/ — HTTP 200, login/board renders. Note: Latest Proxmox VE (9.x) supports running Docker directly inside unprivileged LXCs with `features: nesting=1` — no separate VM needed for container workloads.