nest/k3s
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Backup and recovery docs

Browse source
This commit is contained in:
KnightArtorias 2026-07-24 08:50:38 -04:00
parent d3e8ce1978
commit 5a9712c9a5
3 changed files with 137 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
documentation/README.md
View file

@ -1,6 +1,6 @@
# The Nest — Documentation # The Nest — Documentation
Welcome to the documentation for **The Nest**, a home lab Kubernetes cluster running on three mirrored Raspberry Pi 5s. This documentation is split into two audiences: Welcome to the documentation for **The Nest**: one bird's homelab, shared from me to you! This documentation is split into two audiences:
## For Users (Accessible via `mynest.love`) ## For Users (Accessible via `mynest.love`)
@ -16,12 +16,12 @@ These guides explain what each service does, how to reach it, and how to get sta
- **Media Request** - **Media Request**
- [Seerr](user-facing/media-request/seerr.md) — Request movies, TV shows, and anime for Jellyfin - [Seerr](user-facing/media-request/seerr.md) — Request movies, TV shows, and anime for Jellyfin
- [Shelfmark](user-facing/media-request/shelfmark.md) — Request ebooks and audiobooks for Calibre and Audiobookshelf - [Shelfmark](user-facing/media-request/shelfmark.md) — Request ebooks and audiobooks for Calibre and Audiobookshelf
- [DroppedNeedle](user-facing/media-and-entertainment/droppedneedle.md) — Discover and request music from Usenet, Soulseek, and your local library - [DroppedNeedle](user-facing/media-and-entertainment/droppedneedle.md) — Discover and request music
- **Productivity & Creation** - **Productivity & Creation**
- [Forgejo](user-facing/productivity-and-creation/forgejo.md) — Git hosting and code collaboration - [Forgejo](user-facing/productivity-and-creation/forgejo.md) — Git hosting and code collaboration
- [Foundry VTT](user-facing/productivity-and-creation/foundry-vtt.md) — Virtual tabletop for TTRPGs - [Foundry VTT](user-facing/productivity-and-creation/foundry-vtt.md) — Virtual tabletop for TTRPGs
- **Storage & Files** - **Storage & Files**
- [Nextcloud](user-facing/storage-and-files/nextcloud.md) — File sync, sharing, and cloud storage - [Nextcloud](user-facing/storage-and-files/nextcloud.md) — File sync, sharing, cloud storage, office suite, and more!
- **Communication & Security** - **Communication & Security**
- [Matrix](user-facing/communication-and-security/matrix.md) — Chat, voice/video calls, and federation - [Matrix](user-facing/communication-and-security/matrix.md) — Chat, voice/video calls, and federation
- [Vaultwarden](user-facing/communication-and-security/vaultwarden.md) — Password manager (Bitwarden-compatible) - [Vaultwarden](user-facing/communication-and-security/vaultwarden.md) — Password manager (Bitwarden-compatible)

133
documentation/technical/backups-and-recovery.md Normal file
View file

@ -0,0 +1,133 @@
# Backups & Recovery
The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up.
## Backup Architecture
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Longhorn │────▶│ Recurring Jobs │────▶│ Backblaze B2 │
│ Volumes │ │ (weekly) │ │ (offsite) │
└─────────────┘ └──────────────────┘ └─────────────────┘
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Nextcloud │────▶│ Restic CronJob │────▶│ Backblaze B2 │
│ User Data │ │ (daily) │ │ (offsite) │
└─────────────┘ └──────────────────┘ └─────────────────┘
## Longhorn Recurring Backups
Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`.
### Schedule
| Group | Task | Cron | Retain |
|-------|------|------|--------|
| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots |
### What Gets Backed Up
Any PVC with the recurring labels is included. This covers:
- **Application config:** Vault data, Longhorn system volumes
- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values)
### What Does NOT Get Backed Up
- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups.
- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).
### Configuration
Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`:
### Backup Destination
Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`.
## Nextcloud User Data Backup
Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace.
### Schedule
- **Frequency:** Daily at 02:00 EST
- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped.
### Retention Policy
| Period | Keep |
|--------|------|
| Daily | 7 days |
| Weekly | 4 weeks |
| Monthly | 12 months |
After retention enforcement, old snapshots are pruned automatically.
### How It Works
1. The CronJob spins up a Restic container (`restic/restic:0.16.4`).
2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`.
3. If the B2 repository doesn't exist yet, Restic initializes it on first run.
4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output.
5. Retention is enforced and old snapshots pruned.
### Configuration
Defined in `services/nextcloud/nextcloud-backups.yaml`:
### Secrets
Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`):
- `B2_ACCOUNT_ID` — Backblaze application key ID
- `B2_ACCOUNT_KEY` — Backblaze application key secret
- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`)
- `RESTIC_PASSWORD` — Encryption password for Restic snapshots
## Recovery Procedures
### Recovering a Longhorn Volume Snapshot
1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`.
2. Create a new volume from the snapshot (or detach and reattach the original).
3. Update PVC references if needed.
### Restoring Nextcloud User Data
1. Spin up a temporary pod with the Restic image and secrets.
2. Run:
```bash
restic restore latest --target /restore --repo $RESTIC_REPOSITORY
```
3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`).
## Monitoring & Alerts
Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:
- Failed Longhorn backup jobs
- Restic CronJob failures (Nextcloud backups)
- Storage capacity warnings on Longhorn volumes
## Related Documentation
- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes
- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster
- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider
---
*TODO: Add a diagram showing the full backup data flow with timestamps.*

6
documentation/user-facing/media-and-entertainment/audiobookshelf.md
View file

@ -5,13 +5,11 @@
## Access ## Access
- **Web UI:** <https://lyrebird.mynest.love> - **Web UI:** <https://lyrebird.mynest.love>
- **Internal (home network):** <http://audiobookshelf.k3s.home.nest>
## Features ## Features
- Organize audiobooks by author, series, or custom tags - Organize audiobooks by author, series, or custom tags
- Track progress across devices — pick up where you left off - Track progress across devices — pick up where you left off
- Podcast subscription and auto-download
- Background playback and sleep timer (mobile apps) - Background playback and sleep timer (mobile apps)
- Support for MP3, M4B, OPUS, and other common formats - Support for MP3, M4B, OPUS, and other common formats
@ -19,8 +17,7 @@
1. Open <https://lyrebird.mynest.love> in your browser or mobile app. 1. Open <https://lyrebird.mynest.love> in your browser or mobile app.
2. Click **Sign In** and choose **Sign in with The Nest**. 2. Click **Sign In** and choose **Sign in with The Nest**.
3. Add your first audiobook folder (if you have local files) or wait for admin-provided content. 3. Start listening!
4. Start listening!
## Mobile Apps ## Mobile Apps
@ -30,7 +27,6 @@
## Tips ## Tips
- **Syncing progress:** Make sure you're signed into the same account on all devices for seamless cross-device sync. - **Syncing progress:** Make sure you're signed into the same account on all devices for seamless cross-device sync.
- **Podcast subscriptions:** Use the web UI to subscribe to RSS feeds — new episodes download automatically.
- **Bookmarks:** You can set bookmarks on any audiobook chapter via the mobile app or web player. - **Bookmarks:** You can set bookmarks on any audiobook chapter via the mobile app or web player.
## Where Does the Content Come From? ## Where Does the Content Come From?