From 069c7f0be45b2f4129f0209b476fdc51b69765b4 Mon Sep 17 00:00:00 2001 From: Astra Logical Date: Sun, 26 Jul 2026 17:48:52 -0400 Subject: [PATCH] Cleanup --- context.txt | 29 -------- technical/architecture-overview.md | 6 -- technical/backups-and-recovery.md | 114 ----------------------------- technical/networking.md | 64 ---------------- technical/security-and-identity.md | 57 --------------- 5 files changed, 270 deletions(-) delete mode 100644 context.txt delete mode 100644 technical/architecture-overview.md delete mode 100644 technical/backups-and-recovery.md delete mode 100644 technical/networking.md delete mode 100644 technical/security-and-identity.md diff --git a/context.txt b/context.txt deleted file mode 100644 index c109abb..0000000 --- a/context.txt +++ /dev/null @@ -1,29 +0,0 @@ -This git repo contains all of the config for a k3s cluster called "The Nest". The Nest is my Homelab setup running off of 3 mirrored rpi5s with attached longhorn and etcd storage, enabling a highly resiliant HA setup. I also have a custom domain, "mynest.love", which routes to an HAProxy instance running on my pfSense router that serves as ingress into the Nest. - -For the last few months I have been preparring The Nest so that I can share it with my friends and family. One of the things I would like to do is provide documentation on what's available in the Nest, how to access it, and how to use it. To that end, I would like you to write documentation for each service you come across in this repo. The documentation should broadly be split into two categories: user-facing and technical. User-facing documentation should contain explinations of services that are accessible to end-users over the mynest.love domain. User-facing documentation is intended for non-technical audience and should be focused on general information and how-tos. An example of a service user-facing documentation should be written foris "DroppedNeedle" (services/droppedneedle). Technical documentation should include information on backend services, architecture, data, networking, and security. Technical documentation is intended for a technical audience. An example of a service technical documentation should be written for is metallb. Being a user-facing service does not preclude a service from having technical documentation in addition to its user-facing documentation if that topic is complex enough to warrent a technical writeup. - -Documentation should be written to the "documentation" folder. -I would like the documentation to be organized by function/topic. For example, services dealing with media should be grouped together and perhaps further broken down by media type (music, TV/Movies, books, etc.). -The documentation should be easy to navigate and search, with links where appropriate. Those links may be to other documents or resources on the internet (ex. GitHub pages, Wikipedia, or homepages). -The documentation should not be overly verbose. Do a second pass on all documents written optomizing for conciseness and clarity. -The documentation should be written in in standard Markdown format so that it can be easily edited and converted into a webpage. -The documentation should be consistent in format, layout, and tone across services, though that format and tone may differ between user-facing and technical documentation. -The documentation should be consistent in the facts. Cross-reference your own work to avoid hallucinations. - -As you are writing the documentation consider the following additional context where appropriate: -- Backups of longhorn storage are conducted daily at 1am -- There are no backups of media storage. -- Nextcloud backups are conducted daily at 2am -- Non-critical services are auto-updated daily at 3am (or according to their deployment's keel annotations). Users may experience service disruptions during this time. -- K3S is updated via system-upgrade-controller -- Nest host OSes are updated weekly on Sundays at 5am. Users may experience more serious service disruption at this time, however, updates/restarts are performed in a rolling fashion to minimize this. -- Access to all user-facing services is governed by Keycloak group. -- Users are granted access to the Matrix homeserver, Vaultwarden, forgejo, and the documentation you are about to write by default. All other services must be requested. Access to the default resources can be revoked. -- Requests for service access are made via Matrix bot. Users are to send a message to @access-bot:starling.mynest.love, using the !groups command to list the keycloak groups available, and the !request groupname command to request access. An admin will approve or deny their request. -- The Keycloak instance is set up such that it only accepts passkeys (i.e. passwordless). Other than an initial first time password, users will never be able to use a password to sign in. They must use a password manager that supports passkeys (such as the Vaultwarden instance I provide for them). -- Users must provide a valid email for the purposes of initial access and account recovery. Some downstream services may use the email address to send notifications if the user opts into them. I have no interest in selling people's info. The email stays secure. Otherwise, the only other info needed is a username in order to enroll. -- If you encounter an issue, join the "Admin" space and ask questions in the "Ask and Admin" room. -- At the time of writing, I am the only admin and developer. It may take some time to resolve issues. - -As you are writing, if you require additional context or have questions, stop and ask me. -I intend to go back through the documentation to add screenshots and additional how-tos. As you are writing, if you recognize a good place for this, leave message starting with "TODO" so I can search it out and add it when you are done. diff --git a/technical/architecture-overview.md b/technical/architecture-overview.md deleted file mode 100644 index 8f5565e..0000000 --- a/technical/architecture-overview.md +++ /dev/null @@ -1,6 +0,0 @@ -# Architecture Overview - -The Nest is a high-availability Kubernetes cluster running [k3s](https://k3s.io/) on three mirrored Raspberry Pi 5 nodes. The architecture prioritizes resilience, security, and ease of maintenance for both the admin and end users. - -## Topology - diff --git a/technical/backups-and-recovery.md b/technical/backups-and-recovery.md deleted file mode 100644 index 730b9a1..0000000 --- a/technical/backups-and-recovery.md +++ /dev/null @@ -1,114 +0,0 @@ -# Backups & Recovery - -The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up. - -## Backup Architecture - - -## Longhorn Recurring Backups - -Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`. - -### Schedule - -| Group | Task | Cron | Retain | -|-------|------|------|--------| -| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots | - -### What Gets Backed Up - -Any PVC with the recurring labels is included. This covers: - -- **Application config:** Vault data, Longhorn system volumes -- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data -- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values) - -### What Does NOT Get Backed Up - -- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups. -- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files). - -### Configuration - -Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`: - -### Backup Destination - -Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`. - -## Nextcloud User Data Backup - -Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace. - -### Schedule - -- **Frequency:** Daily at 02:00 EST -- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped. - -### Retention Policy - -| Period | Keep | -|--------|------| -| Daily | 7 days | -| Weekly | 4 weeks | -| Monthly | 12 months | - -After retention enforcement, old snapshots are pruned automatically. - -### How It Works - -1. The CronJob spins up a Restic container (`restic/restic:0.16.4`). -2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`. -3. If the B2 repository doesn't exist yet, Restic initializes it on first run. -4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output. -5. Retention is enforced and old snapshots pruned. - -### Configuration - -Defined in `services/nextcloud/nextcloud-backups.yaml`: - -### Secrets - -Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`): - -- `B2_ACCOUNT_ID` — Backblaze application key ID -- `B2_ACCOUNT_KEY` — Backblaze application key secret -- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`) -- `RESTIC_PASSWORD` — Encryption password for Restic snapshots - -## Recovery Procedures - -### Recovering a Longhorn Volume Snapshot - -1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`. -2. Create a new volume from the snapshot (or detach and reattach the original). -3. Update PVC references if needed. - -### Restoring Nextcloud User Data - -1. Spin up a temporary pod with the Restic image and secrets. -2. Run: - - ```bash - restic restore latest --target /restore --repo $RESTIC_REPOSITORY - ``` - -3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`). - -## Monitoring & Alerts - -Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for: - -- Failed Longhorn backup jobs -- Restic CronJob failures (Nextcloud backups) -- Storage capacity warnings on Longhorn volumes - -## Related Documentation - -- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes -- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster -- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider - ---- - -*TODO: Add a diagram showing the full backup data flow with timestamps.* diff --git a/technical/networking.md b/technical/networking.md deleted file mode 100644 index c711dfe..0000000 --- a/technical/networking.md +++ /dev/null @@ -1,64 +0,0 @@ -# Networking - -The Nest's networking stack handles external ingress (from the internet), internal routing (between services), and load balancing for services that require public IP addresses. - -## External Ingress: HAProxy on pfSense - -All traffic from the internet enters through an [HAProxy](https://www.haproxy.org/) instance running on the pfSense router. HAProxy performs: - -- **TLS termination** — Handles SSL certificates for all `*.mynest.love` domains -- **Hostname-based routing** — Forwards requests to the appropriate internal service based on the `Host` header -- **Health checks** — Monitors backend availability and fails over if a node is down - -HAProxy routes traffic to MetalLB-assigned IPs within the cluster, which then forward to Traefik ingress controllers. - -## Internal Ingress: Traefik - -[Traefik](https://traefik.io/) is the Kubernetes-native ingress controller. It receives traffic from HAProxy and routes it to the correct service based on: - -- **Host headers** (e.g., `beak.mynest.love` → Jellyfin) -- **Path prefixes** (e.g., `/_matrix/...` → Synapse, `/sfu/get` → LiveKit JWT service) -- **Middleware annotations** for headers, redirects, and TLS enforcement - -Traefik is configured via Kubernetes CRDs (`Ingress`, `Middleware`) defined in YAML files under `services/*/`. - -## Load Balancing: MetalLB - -[MetalLB](https://metallb.io/) provides Layer 2 load balancing for services that need a public IP (e.g., Matrix LiveKit's UDP ports, Forgejo SSH). - -### Configuration - -- **IP Pool:** `192.168.1.150`–`192.168.1.200` (defined in `metallb/metallb.yaml`) -- **Advertisement Protocol:** L2 (ARP) — suitable for home networks without BGP support - -### Services Using MetalLB - -| Service | IP Assignment | Purpose | -|---------|--------------|---------| -| LiveKit (Matrix RTC) | `192.168.1.160` | WebRTC media relay (UDP/TCP) | -| Forgejo SSH | Load-balanced | Git over SSH (port 22) | - -## DNS & Hostname Resolution - -- **External:** The `*.mynest.love` domain points to the pfSense router's public IP. HAProxy handles virtual hosting. -- **Internal:** Services resolve via Kubernetes DNS (`..svc.cluster.local`). External hostnames (e.g., `robin.home.nest`) are resolved via the home network's DNS server. - -## TLS & Certificates - -All external-facing services use TLS certificates issued by [Let's Encrypt](https://letsencrypt.org/) via cert-manager: - -- **Cluster Issuer:** `letsencrypt-prod` (defined in `cert-manager/letsencrypt-cluster-issuer.yaml`) -- **Challenge Type:** HTTP-01 (validated through Traefik) -- **Auto-renewal:** Handled automatically by cert-manager 30 days before expiry - -Internal-only services (e.g., `*.k3s.home.nest`) may use self-signed certificates or no TLS, depending on the service configuration. - -## Network Policies - -The cluster does not enforce strict network policies between namespaces by default. Services communicate freely within the cluster via Kubernetes DNS and ClusterIP services. For enhanced security, consider implementing [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) in future iterations. - -## Related Documentation - -- [MetalLB](metallb.md) — Detailed MetalLB configuration -- [Security & Identity](security-and-identity.md) — TLS termination and authentication flow -- [Architecture Overview](architecture-overview.md) — High-level data flow diff --git a/technical/security-and-identity.md b/technical/security-and-identity.md deleted file mode 100644 index 1d8b2e4..0000000 --- a/technical/security-and-identity.md +++ /dev/null @@ -1,57 +0,0 @@ -# Security & Identity - -The Nest's security model is built around three pillars: **identity management** (Keycloak), **secrets management** (Vault), and **network security** (TLS, MetalLB, Traefik middleware). - -## Identity: Keycloak - -[Keycloak](https://keycloak.org/) is the central identity provider for all user-facing services. It handles authentication, authorization, and single sign-on (SSO) via OpenID Connect (OIDC). - -### Configuration - -- **Namespace:** `security` -- **Image:** `quay.io/keycloak/keycloak:latest` -- **Hostname:** `sso.mynest.love` -- **Database:** PostgreSQL (`keycloak-postgres.security.svc.cluster.local`) -- **Proxy Headers:** `xforwarded` (required behind HAProxy) - -### Authentication Flow - -1. User visits a service (e.g., Jellyfin). -2. Service redirects to Keycloak's OIDC authorization endpoint. -3. User authenticates via **WebAuthn passkey** (passwordless). -4. Keycloak issues an ID token and access token. -5. Service validates the token and grants access based on group membership. - -### Passkeys (WebAuthn) - -- All authentication is **passwordless** after initial setup. -- Users enroll passkeys on their devices (smartphone, security key, etc.). -- Passkeys are stored in Keycloak's `webauthn` credential store. -- Account recovery requires admin intervention + email verification. - -### Group-Based Authorization - -Access to services is controlled by Keycloak groups: - -| Group | Services Unlocked | -|-------|-------------------| -| (default) | Matrix, Vaultwarden, Forgejo, Documentation | -| `media-users` | Jellyfin, Sonarr, Radarr, Bazarr | -| `music-users` | DroppedNeedle, slskd | -| `books-users` | Calibre-Web, Shelfmark, Audiobookshelf | -| `storage-users` | Nextcloud | -| `ttrpg-users` | Foundry VTT | - -Groups are managed by an admin. Users request access via the Matrix bot (`@access-bot:starling.mynest.love`). - -### Related Documentation - -- [Keycloak (Technical)](keycloak-technical.md) — Deep dive into Keycloak configuration -- [Matrix Architecture](matrix-architecture.md) — How MAS integrates with Keycloak - -## Secrets Management: HashiCorp Vault - -[HashiCorp Vault](https://www.vaultproject.io/) manages secrets for all services. It is deployed as a 3-node Raft cluster with transit sealing via [Keykeeper](vault-and-keykeeper.md). - -### Architecture -