From 5a9712c9a5782974c13dbb0637cf61f6032fcdfc Mon Sep 17 00:00:00 2001 From: KnightArtorias Date: Fri, 24 Jul 2026 08:50:38 -0400 Subject: [PATCH] Backup and recovery docs --- documentation/README.md | 6 +- .../technical/backups-and-recovery.md | 133 ++++++++++++++++++ .../media-and-entertainment/audiobookshelf.md | 6 +- 3 files changed, 137 insertions(+), 8 deletions(-) create mode 100644 documentation/technical/backups-and-recovery.md diff --git a/documentation/README.md b/documentation/README.md index c50f232..4afeb89 100644 --- a/documentation/README.md +++ b/documentation/README.md @@ -1,6 +1,6 @@ # The Nest — Documentation -Welcome to the documentation for **The Nest**, a home lab Kubernetes cluster running on three mirrored Raspberry Pi 5s. This documentation is split into two audiences: +Welcome to the documentation for **The Nest**: one bird's homelab, shared from me to you! This documentation is split into two audiences: ## For Users (Accessible via `mynest.love`) @@ -16,12 +16,12 @@ These guides explain what each service does, how to reach it, and how to get sta - **Media Request** - [Seerr](user-facing/media-request/seerr.md) — Request movies, TV shows, and anime for Jellyfin - [Shelfmark](user-facing/media-request/shelfmark.md) — Request ebooks and audiobooks for Calibre and Audiobookshelf - - [DroppedNeedle](user-facing/media-and-entertainment/droppedneedle.md) — Discover and request music from Usenet, Soulseek, and your local library + - [DroppedNeedle](user-facing/media-and-entertainment/droppedneedle.md) — Discover and request music - **Productivity & Creation** - [Forgejo](user-facing/productivity-and-creation/forgejo.md) — Git hosting and code collaboration - [Foundry VTT](user-facing/productivity-and-creation/foundry-vtt.md) — Virtual tabletop for TTRPGs - **Storage & Files** - - [Nextcloud](user-facing/storage-and-files/nextcloud.md) — File sync, sharing, and cloud storage + - [Nextcloud](user-facing/storage-and-files/nextcloud.md) — File sync, sharing, cloud storage, office suite, and more! - **Communication & Security** - [Matrix](user-facing/communication-and-security/matrix.md) — Chat, voice/video calls, and federation - [Vaultwarden](user-facing/communication-and-security/vaultwarden.md) — Password manager (Bitwarden-compatible) diff --git a/documentation/technical/backups-and-recovery.md b/documentation/technical/backups-and-recovery.md new file mode 100644 index 0000000..0beda4b --- /dev/null +++ b/documentation/technical/backups-and-recovery.md @@ -0,0 +1,133 @@ +# Backups & Recovery + +The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up. + +## Backup Architecture + +┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐ + + +│ Longhorn │────▶│ Recurring Jobs │────▶│ Backblaze B2 │ + +│ Volumes │ │ (weekly) │ │ (offsite) │ + + +└─────────────┘ └──────────────────┘ └─────────────────┘ + + +┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐ + + +│ Nextcloud │────▶│ Restic CronJob │────▶│ Backblaze B2 │ + +│ User Data │ │ (daily) │ │ (offsite) │ + +└─────────────┘ └──────────────────┘ └─────────────────┘ + +## Longhorn Recurring Backups + +Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`. + +### Schedule + +| Group | Task | Cron | Retain | +|-------|------|------|--------| +| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots | + +### What Gets Backed Up + +Any PVC with the recurring labels is included. This covers: + +- **Application config:** Vault data, Longhorn system volumes +- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data +- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values) + +### What Does NOT Get Backed Up + +- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups. +- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files). + +### Configuration + +Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`: + +### Backup Destination + +Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`. + +## Nextcloud User Data Backup + +Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace. + +### Schedule + +- **Frequency:** Daily at 02:00 EST +- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped. + +### Retention Policy + +| Period | Keep | +|--------|------| +| Daily | 7 days | +| Weekly | 4 weeks | +| Monthly | 12 months | + +After retention enforcement, old snapshots are pruned automatically. + +### How It Works + +1. The CronJob spins up a Restic container (`restic/restic:0.16.4`). +2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`. +3. If the B2 repository doesn't exist yet, Restic initializes it on first run. +4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output. +5. Retention is enforced and old snapshots pruned. + +### Configuration + +Defined in `services/nextcloud/nextcloud-backups.yaml`: + +### Secrets + +Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`): + +- `B2_ACCOUNT_ID` — Backblaze application key ID +- `B2_ACCOUNT_KEY` — Backblaze application key secret +- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`) +- `RESTIC_PASSWORD` — Encryption password for Restic snapshots + +## Recovery Procedures + +### Recovering a Longhorn Volume Snapshot + +1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`. +2. Create a new volume from the snapshot (or detach and reattach the original). +3. Update PVC references if needed. + +### Restoring Nextcloud User Data + +1. Spin up a temporary pod with the Restic image and secrets. +2. Run: + + ```bash + restic restore latest --target /restore --repo $RESTIC_REPOSITORY + ``` + +3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`). + +## Monitoring & Alerts + +Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for: + +- Failed Longhorn backup jobs +- Restic CronJob failures (Nextcloud backups) +- Storage capacity warnings on Longhorn volumes + +## Related Documentation + +- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes +- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster +- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider + +--- + +*TODO: Add a diagram showing the full backup data flow with timestamps.* diff --git a/documentation/user-facing/media-and-entertainment/audiobookshelf.md b/documentation/user-facing/media-and-entertainment/audiobookshelf.md index ee61d2d..ea2c7c4 100644 --- a/documentation/user-facing/media-and-entertainment/audiobookshelf.md +++ b/documentation/user-facing/media-and-entertainment/audiobookshelf.md @@ -5,13 +5,11 @@ ## Access - **Web UI:** -- **Internal (home network):** ## Features - Organize audiobooks by author, series, or custom tags - Track progress across devices — pick up where you left off -- Podcast subscription and auto-download - Background playback and sleep timer (mobile apps) - Support for MP3, M4B, OPUS, and other common formats @@ -19,8 +17,7 @@ 1. Open in your browser or mobile app. 2. Click **Sign In** and choose **Sign in with The Nest**. -3. Add your first audiobook folder (if you have local files) or wait for admin-provided content. -4. Start listening! +3. Start listening! ## Mobile Apps @@ -30,7 +27,6 @@ ## Tips - **Syncing progress:** Make sure you're signed into the same account on all devices for seamless cross-device sync. -- **Podcast subscriptions:** Use the web UI to subscribe to RSS feeds — new episodes download automatically. - **Bookmarks:** You can set bookmarks on any audiobook chapter via the mobile app or web player. ## Where Does the Content Come From?