diff --git a/technical/architecture-overview.md b/technical/architecture-overview.md new file mode 100644 index 0000000..8f5565e --- /dev/null +++ b/technical/architecture-overview.md @@ -0,0 +1,6 @@ +# Architecture Overview + +The Nest is a high-availability Kubernetes cluster running [k3s](https://k3s.io/) on three mirrored Raspberry Pi 5 nodes. The architecture prioritizes resilience, security, and ease of maintenance for both the admin and end users. + +## Topology + diff --git a/technical/backups-and-recovery.md b/technical/backups-and-recovery.md new file mode 100644 index 0000000..730b9a1 --- /dev/null +++ b/technical/backups-and-recovery.md @@ -0,0 +1,114 @@ +# Backups & Recovery + +The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up. + +## Backup Architecture + + +## Longhorn Recurring Backups + +Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`. + +### Schedule + +| Group | Task | Cron | Retain | +|-------|------|------|--------| +| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots | + +### What Gets Backed Up + +Any PVC with the recurring labels is included. This covers: + +- **Application config:** Vault data, Longhorn system volumes +- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data +- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values) + +### What Does NOT Get Backed Up + +- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups. +- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files). + +### Configuration + +Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`: + +### Backup Destination + +Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`. + +## Nextcloud User Data Backup + +Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace. + +### Schedule + +- **Frequency:** Daily at 02:00 EST +- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped. + +### Retention Policy + +| Period | Keep | +|--------|------| +| Daily | 7 days | +| Weekly | 4 weeks | +| Monthly | 12 months | + +After retention enforcement, old snapshots are pruned automatically. + +### How It Works + +1. The CronJob spins up a Restic container (`restic/restic:0.16.4`). +2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`. +3. If the B2 repository doesn't exist yet, Restic initializes it on first run. +4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output. +5. Retention is enforced and old snapshots pruned. + +### Configuration + +Defined in `services/nextcloud/nextcloud-backups.yaml`: + +### Secrets + +Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`): + +- `B2_ACCOUNT_ID` — Backblaze application key ID +- `B2_ACCOUNT_KEY` — Backblaze application key secret +- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`) +- `RESTIC_PASSWORD` — Encryption password for Restic snapshots + +## Recovery Procedures + +### Recovering a Longhorn Volume Snapshot + +1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`. +2. Create a new volume from the snapshot (or detach and reattach the original). +3. Update PVC references if needed. + +### Restoring Nextcloud User Data + +1. Spin up a temporary pod with the Restic image and secrets. +2. Run: + + ```bash + restic restore latest --target /restore --repo $RESTIC_REPOSITORY + ``` + +3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`). + +## Monitoring & Alerts + +Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for: + +- Failed Longhorn backup jobs +- Restic CronJob failures (Nextcloud backups) +- Storage capacity warnings on Longhorn volumes + +## Related Documentation + +- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes +- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster +- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider + +--- + +*TODO: Add a diagram showing the full backup data flow with timestamps.* diff --git a/technical/networking.md b/technical/networking.md new file mode 100644 index 0000000..c711dfe --- /dev/null +++ b/technical/networking.md @@ -0,0 +1,64 @@ +# Networking + +The Nest's networking stack handles external ingress (from the internet), internal routing (between services), and load balancing for services that require public IP addresses. + +## External Ingress: HAProxy on pfSense + +All traffic from the internet enters through an [HAProxy](https://www.haproxy.org/) instance running on the pfSense router. HAProxy performs: + +- **TLS termination** — Handles SSL certificates for all `*.mynest.love` domains +- **Hostname-based routing** — Forwards requests to the appropriate internal service based on the `Host` header +- **Health checks** — Monitors backend availability and fails over if a node is down + +HAProxy routes traffic to MetalLB-assigned IPs within the cluster, which then forward to Traefik ingress controllers. + +## Internal Ingress: Traefik + +[Traefik](https://traefik.io/) is the Kubernetes-native ingress controller. It receives traffic from HAProxy and routes it to the correct service based on: + +- **Host headers** (e.g., `beak.mynest.love` → Jellyfin) +- **Path prefixes** (e.g., `/_matrix/...` → Synapse, `/sfu/get` → LiveKit JWT service) +- **Middleware annotations** for headers, redirects, and TLS enforcement + +Traefik is configured via Kubernetes CRDs (`Ingress`, `Middleware`) defined in YAML files under `services/*/`. + +## Load Balancing: MetalLB + +[MetalLB](https://metallb.io/) provides Layer 2 load balancing for services that need a public IP (e.g., Matrix LiveKit's UDP ports, Forgejo SSH). + +### Configuration + +- **IP Pool:** `192.168.1.150`–`192.168.1.200` (defined in `metallb/metallb.yaml`) +- **Advertisement Protocol:** L2 (ARP) — suitable for home networks without BGP support + +### Services Using MetalLB + +| Service | IP Assignment | Purpose | +|---------|--------------|---------| +| LiveKit (Matrix RTC) | `192.168.1.160` | WebRTC media relay (UDP/TCP) | +| Forgejo SSH | Load-balanced | Git over SSH (port 22) | + +## DNS & Hostname Resolution + +- **External:** The `*.mynest.love` domain points to the pfSense router's public IP. HAProxy handles virtual hosting. +- **Internal:** Services resolve via Kubernetes DNS (`..svc.cluster.local`). External hostnames (e.g., `robin.home.nest`) are resolved via the home network's DNS server. + +## TLS & Certificates + +All external-facing services use TLS certificates issued by [Let's Encrypt](https://letsencrypt.org/) via cert-manager: + +- **Cluster Issuer:** `letsencrypt-prod` (defined in `cert-manager/letsencrypt-cluster-issuer.yaml`) +- **Challenge Type:** HTTP-01 (validated through Traefik) +- **Auto-renewal:** Handled automatically by cert-manager 30 days before expiry + +Internal-only services (e.g., `*.k3s.home.nest`) may use self-signed certificates or no TLS, depending on the service configuration. + +## Network Policies + +The cluster does not enforce strict network policies between namespaces by default. Services communicate freely within the cluster via Kubernetes DNS and ClusterIP services. For enhanced security, consider implementing [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) in future iterations. + +## Related Documentation + +- [MetalLB](metallb.md) — Detailed MetalLB configuration +- [Security & Identity](security-and-identity.md) — TLS termination and authentication flow +- [Architecture Overview](architecture-overview.md) — High-level data flow diff --git a/technical/security-and-identity.md b/technical/security-and-identity.md new file mode 100644 index 0000000..1d8b2e4 --- /dev/null +++ b/technical/security-and-identity.md @@ -0,0 +1,57 @@ +# Security & Identity + +The Nest's security model is built around three pillars: **identity management** (Keycloak), **secrets management** (Vault), and **network security** (TLS, MetalLB, Traefik middleware). + +## Identity: Keycloak + +[Keycloak](https://keycloak.org/) is the central identity provider for all user-facing services. It handles authentication, authorization, and single sign-on (SSO) via OpenID Connect (OIDC). + +### Configuration + +- **Namespace:** `security` +- **Image:** `quay.io/keycloak/keycloak:latest` +- **Hostname:** `sso.mynest.love` +- **Database:** PostgreSQL (`keycloak-postgres.security.svc.cluster.local`) +- **Proxy Headers:** `xforwarded` (required behind HAProxy) + +### Authentication Flow + +1. User visits a service (e.g., Jellyfin). +2. Service redirects to Keycloak's OIDC authorization endpoint. +3. User authenticates via **WebAuthn passkey** (passwordless). +4. Keycloak issues an ID token and access token. +5. Service validates the token and grants access based on group membership. + +### Passkeys (WebAuthn) + +- All authentication is **passwordless** after initial setup. +- Users enroll passkeys on their devices (smartphone, security key, etc.). +- Passkeys are stored in Keycloak's `webauthn` credential store. +- Account recovery requires admin intervention + email verification. + +### Group-Based Authorization + +Access to services is controlled by Keycloak groups: + +| Group | Services Unlocked | +|-------|-------------------| +| (default) | Matrix, Vaultwarden, Forgejo, Documentation | +| `media-users` | Jellyfin, Sonarr, Radarr, Bazarr | +| `music-users` | DroppedNeedle, slskd | +| `books-users` | Calibre-Web, Shelfmark, Audiobookshelf | +| `storage-users` | Nextcloud | +| `ttrpg-users` | Foundry VTT | + +Groups are managed by an admin. Users request access via the Matrix bot (`@access-bot:starling.mynest.love`). + +### Related Documentation + +- [Keycloak (Technical)](keycloak-technical.md) — Deep dive into Keycloak configuration +- [Matrix Architecture](matrix-architecture.md) — How MAS integrates with Keycloak + +## Secrets Management: HashiCorp Vault + +[HashiCorp Vault](https://www.vaultproject.io/) manages secrets for all services. It is deployed as a 3-node Raft cluster with transit sealing via [Keykeeper](vault-and-keykeeper.md). + +### Architecture +