4.2 KiB
Backups & Recovery
The Nest uses a two-tier backup strategy: Longhorn volume snapshots for application configuration data (backed up to Backblaze B2 weekly), and Restic for Nextcloud user data (backed up daily). Media storage is not backed up.
Backup Architecture
Longhorn Recurring Backups
Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with recurring-job.longhorn.io/source: "enabled" and grouped under app-config-group.
Schedule
| Group | Task | Cron | Retain |
|---|---|---|---|
app-config-group |
backup | 0 4 * * 0 (Sundays, 04:00 EST) |
4 snapshots |
What Gets Backed Up
Any PVC with the recurring labels is included. This covers:
- Application config: Vault data, Longhorn system volumes
- Service state: Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
- Monitoring: Prometheus and Grafana PVCs (labeled in HelmChart values)
What Does NOT Get Backed Up
- Media storage (
/yarr/media,/yarr/downloads) — NFS shares are not part of Longhorn and have no backups. - Transcoding scratch space — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).
Configuration
Longhorn recurring jobs are defined in longhorn/weekly-backups.yaml:
Backup Destination
Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (longhorn/backblaze-secrets.yaml). The S3-compatible endpoint is configured through AWS_ENDPOINTS.
Nextcloud User Data Backup
Nextcloud user data (files, contacts, calendars) is backed up separately using Restic to Backblaze B2. This runs as a Kubernetes CronJob in the nextcloud namespace.
Schedule
- Frequency: Daily at 02:00 EST
- Concurrency policy:
Forbid— if a previous run is still active, the new job is skipped.
Retention Policy
| Period | Keep |
|---|---|
| Daily | 7 days |
| Weekly | 4 weeks |
| Monthly | 12 months |
After retention enforcement, old snapshots are pruned automatically.
How It Works
- The CronJob spins up a Restic container (
restic/restic:0.16.4). - Secrets (B2 credentials, repository URL, password) are injected from Vault via
ExternalSecret. - If the B2 repository doesn't exist yet, Restic initializes it on first run.
- The entire Nextcloud user data volume (
/data) is backed up with verbose output. - Retention is enforced and old snapshots pruned.
Configuration
Defined in services/nextcloud/nextcloud-backups.yaml:
Secrets
Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (services/nextcloud/nextcloud-backups.yaml):
B2_ACCOUNT_ID— Backblaze application key IDB2_ACCOUNT_KEY— Backblaze application key secretRESTIC_REPOSITORY— B2 bucket path (e.g.,b2:bucket-name:path)RESTIC_PASSWORD— Encryption password for Restic snapshots
Recovery Procedures
Recovering a Longhorn Volume Snapshot
- Identify the snapshot you want to restore from in the Longhorn UI or via
longhornctl. - Create a new volume from the snapshot (or detach and reattach the original).
- Update PVC references if needed.
Restoring Nextcloud User Data
-
Spin up a temporary pod with the Restic image and secrets.
-
Run:
restic restore latest --target /restore --repo $RESTIC_REPOSITORY -
Copy restored files back to the Nextcloud NFS share (
robin.home.nest:/nextcloud).
Monitoring & Alerts
Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:
- Failed Longhorn backup jobs
- Restic CronJob failures (Nextcloud backups)
- Storage capacity warnings on Longhorn volumes
Related Documentation
- Longhorn Storage — Volume management and storage classes
- External Secrets Operator — How Vault secrets are synced into the cluster
- Backblaze B2 — Offsite object storage provider
TODO: Add a diagram showing the full backup data flow with timestamps.