technical/architecture-overview.md

@@ -0,0 +1,6 @@
+# Architecture Overview
+
+The Nest is a high-availability Kubernetes cluster running [k3s](https://k3s.io/) on three mirrored Raspberry Pi 5 nodes. The architecture prioritizes resilience, security, and ease of maintenance for both the admin and end users.
+
+## Topology
+

technical/backups-and-recovery.md

@@ -0,0 +1,114 @@
+# Backups & Recovery
+
+The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up.
+
+## Backup Architecture
+
+
+## Longhorn Recurring Backups
+
+Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`.
+
+### Schedule
+
+| Group | Task | Cron | Retain |
+|-------|------|------|--------|
+| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots |
+
+### What Gets Backed Up
+
+Any PVC with the recurring labels is included. This covers:
+
+- **Application config:** Vault data, Longhorn system volumes
+- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
+- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values)
+
+### What Does NOT Get Backed Up
+
+- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups.
+- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).
+
+### Configuration
+
+Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`:
+
+### Backup Destination
+
+Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`.
+
+## Nextcloud User Data Backup
+
+Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace.
+
+### Schedule
+
+- **Frequency:** Daily at 02:00 EST
+- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped.
+
+### Retention Policy
+
+| Period | Keep |
+|--------|------|
+| Daily | 7 days |
+| Weekly | 4 weeks |
+| Monthly | 12 months |
+
+After retention enforcement, old snapshots are pruned automatically.
+
+### How It Works
+
+1. The CronJob spins up a Restic container (`restic/restic:0.16.4`).
+2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`.
+3. If the B2 repository doesn't exist yet, Restic initializes it on first run.
+4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output.
+5. Retention is enforced and old snapshots pruned.
+
+### Configuration
+
+Defined in `services/nextcloud/nextcloud-backups.yaml`:
+
+### Secrets
+
+Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`):
+
+- `B2_ACCOUNT_ID` — Backblaze application key ID
+- `B2_ACCOUNT_KEY` — Backblaze application key secret
+- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`)
+- `RESTIC_PASSWORD` — Encryption password for Restic snapshots
+
+## Recovery Procedures
+
+### Recovering a Longhorn Volume Snapshot
+
+1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`.
+2. Create a new volume from the snapshot (or detach and reattach the original).
+3. Update PVC references if needed.
+
+### Restoring Nextcloud User Data
+
+1. Spin up a temporary pod with the Restic image and secrets.
+2. Run:
+
+   ```bash
+   restic restore latest --target /restore --repo $RESTIC_REPOSITORY
+   ```
+
+3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`).
+
+## Monitoring & Alerts
+
+Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:
+
+- Failed Longhorn backup jobs
+- Restic CronJob failures (Nextcloud backups)
+- Storage capacity warnings on Longhorn volumes
+
+## Related Documentation
+
+- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes
+- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster
+- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider
+
+---
+
+*TODO: Add a diagram showing the full backup data flow with timestamps.*

technical/networking.md

@@ -0,0 +1,64 @@
+# Networking
+
+The Nest's networking stack handles external ingress (from the internet), internal routing (between services), and load balancing for services that require public IP addresses.
+
+## External Ingress: HAProxy on pfSense
+
+All traffic from the internet enters through an [HAProxy](https://www.haproxy.org/) instance running on the pfSense router. HAProxy performs:
+
+- **TLS termination** — Handles SSL certificates for all `*.mynest.love` domains
+- **Hostname-based routing** — Forwards requests to the appropriate internal service based on the `Host` header
+- **Health checks** — Monitors backend availability and fails over if a node is down
+
+HAProxy routes traffic to MetalLB-assigned IPs within the cluster, which then forward to Traefik ingress controllers.
+
+## Internal Ingress: Traefik
+
+[Traefik](https://traefik.io/) is the Kubernetes-native ingress controller. It receives traffic from HAProxy and routes it to the correct service based on:
+
+- **Host headers** (e.g., `beak.mynest.love` → Jellyfin)
+- **Path prefixes** (e.g., `/_matrix/...` → Synapse, `/sfu/get` → LiveKit JWT service)
+- **Middleware annotations** for headers, redirects, and TLS enforcement
+
+Traefik is configured via Kubernetes CRDs (`Ingress`, `Middleware`) defined in YAML files under `services/*/`.
+
+## Load Balancing: MetalLB
+
+[MetalLB](https://metallb.io/) provides Layer 2 load balancing for services that need a public IP (e.g., Matrix LiveKit's UDP ports, Forgejo SSH).
+
+### Configuration
+
+- **IP Pool:** `192.168.1.150`–`192.168.1.200` (defined in `metallb/metallb.yaml`)
+- **Advertisement Protocol:** L2 (ARP) — suitable for home networks without BGP support
+
+### Services Using MetalLB
+
+| Service | IP Assignment | Purpose |
+|---------|--------------|---------|
+| LiveKit (Matrix RTC) | `192.168.1.160` | WebRTC media relay (UDP/TCP) |
+| Forgejo SSH | Load-balanced | Git over SSH (port 22) |
+
+## DNS & Hostname Resolution
+
+- **External:** The `*.mynest.love` domain points to the pfSense router's public IP. HAProxy handles virtual hosting.
+- **Internal:** Services resolve via Kubernetes DNS (`<service>.<namespace>.svc.cluster.local`). External hostnames (e.g., `robin.home.nest`) are resolved via the home network's DNS server.
+
+## TLS & Certificates
+
+All external-facing services use TLS certificates issued by [Let's Encrypt](https://letsencrypt.org/) via cert-manager:
+
+- **Cluster Issuer:** `letsencrypt-prod` (defined in `cert-manager/letsencrypt-cluster-issuer.yaml`)
+- **Challenge Type:** HTTP-01 (validated through Traefik)
+- **Auto-renewal:** Handled automatically by cert-manager 30 days before expiry
+
+Internal-only services (e.g., `*.k3s.home.nest`) may use self-signed certificates or no TLS, depending on the service configuration.
+
+## Network Policies
+
+The cluster does not enforce strict network policies between namespaces by default. Services communicate freely within the cluster via Kubernetes DNS and ClusterIP services. For enhanced security, consider implementing [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) in future iterations.
+
+## Related Documentation
+
+- [MetalLB](metallb.md) — Detailed MetalLB configuration
+- [Security & Identity](security-and-identity.md) — TLS termination and authentication flow
+- [Architecture Overview](architecture-overview.md) — High-level data flow

technical/security-and-identity.md

@@ -0,0 +1,57 @@
+# Security & Identity
+
+The Nest's security model is built around three pillars: **identity management** (Keycloak), **secrets management** (Vault), and **network security** (TLS, MetalLB, Traefik middleware).
+
+## Identity: Keycloak
+
+[Keycloak](https://keycloak.org/) is the central identity provider for all user-facing services. It handles authentication, authorization, and single sign-on (SSO) via OpenID Connect (OIDC).
+
+### Configuration
+
+- **Namespace:** `security`
+- **Image:** `quay.io/keycloak/keycloak:latest`
+- **Hostname:** `sso.mynest.love`
+- **Database:** PostgreSQL (`keycloak-postgres.security.svc.cluster.local`)
+- **Proxy Headers:** `xforwarded` (required behind HAProxy)
+
+### Authentication Flow
+
+1. User visits a service (e.g., Jellyfin).
+2. Service redirects to Keycloak's OIDC authorization endpoint.
+3. User authenticates via **WebAuthn passkey** (passwordless).
+4. Keycloak issues an ID token and access token.
+5. Service validates the token and grants access based on group membership.
+
+### Passkeys (WebAuthn)
+
+- All authentication is **passwordless** after initial setup.
+- Users enroll passkeys on their devices (smartphone, security key, etc.).
+- Passkeys are stored in Keycloak's `webauthn` credential store.
+- Account recovery requires admin intervention + email verification.
+
+### Group-Based Authorization
+
+Access to services is controlled by Keycloak groups:
+
+| Group | Services Unlocked |
+|-------|-------------------|
+| (default) | Matrix, Vaultwarden, Forgejo, Documentation |
+| `media-users` | Jellyfin, Sonarr, Radarr, Bazarr |
+| `music-users` | DroppedNeedle, slskd |
+| `books-users` | Calibre-Web, Shelfmark, Audiobookshelf |
+| `storage-users` | Nextcloud |
+| `ttrpg-users` | Foundry VTT |
+
+Groups are managed by an admin. Users request access via the Matrix bot (`@access-bot:starling.mynest.love`).
+
+### Related Documentation
+
+- [Keycloak (Technical)](keycloak-technical.md) — Deep dive into Keycloak configuration
+- [Matrix Architecture](matrix-architecture.md) — How MAS integrates with Keycloak
+
+## Secrets Management: HashiCorp Vault
+
+[HashiCorp Vault](https://www.vaultproject.io/) manages secrets for all services. It is deployed as a 3-node Raft cluster with transit sealing via [Keykeeper](vault-and-keykeeper.md).
+
+### Architecture
+