6.7 KiB
Backups & Recovery
The Nest uses a two-tier backup strategy: Longhorn volume snapshots for application configuration data (backed up to Backblaze B2 weekly), and Restic for Nextcloud user data (backed up daily). Media storage is not backed up.
Backup Architecture
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Longhorn │────▶│ Recurring Jobs │────▶│ Backblaze B2 │
│ Volumes │ │ (weekly) │ │ (offsite) │
└─────────────┘ └──────────────────┘ └─────────────────┘
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Nextcloud │────▶│ Restic CronJob │────▶│ Backblaze B2 │
│ User Data │ │ (daily) │ │ (offsite) │
└─────────────┘ └──────────────────┘ └─────────────────┘
Longhorn Recurring Backups
Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with recurring-job.longhorn.io/source: "enabled" and grouped under app-config-group.
Schedule
| Group | Task | Cron | Retain |
|---|---|---|---|
app-config-group |
backup | 0 4 * * 0 (Sundays, 04:00 EST) |
4 snapshots |
What Gets Backed Up
Any PVC with the recurring labels is included. This covers:
- Application config: Vault data, Longhorn system volumes
- Service state: Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
- Monitoring: Prometheus and Grafana PVCs (labeled in HelmChart values)
What Does NOT Get Backed Up
- Media storage (
/yarr/media,/yarr/downloads) — NFS shares are not part of Longhorn and have no backups. - Transcoding scratch space — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).
Configuration
Longhorn recurring jobs are defined in longhorn/weekly-backups.yaml:
Backup Destination
Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (longhorn/backblaze-secrets.yaml). The S3-compatible endpoint is configured through AWS_ENDPOINTS.
Nextcloud User Data Backup
Nextcloud user data (files, contacts, calendars) is backed up separately using Restic to Backblaze B2. This runs as a Kubernetes CronJob in the nextcloud namespace.
Schedule
- Frequency: Daily at 02:00 EST
- Concurrency policy:
Forbid— if a previous run is still active, the new job is skipped.
Retention Policy
| Period | Keep |
|---|---|
| Daily | 7 days |
| Weekly | 4 weeks |
| Monthly | 12 months |
After retention enforcement, old snapshots are pruned automatically.
How It Works
- The CronJob spins up a Restic container (
restic/restic:0.16.4). - Secrets (B2 credentials, repository URL, password) are injected from Vault via
ExternalSecret. - If the B2 repository doesn't exist yet, Restic initializes it on first run.
- The entire Nextcloud user data volume (
/data) is backed up with verbose output. - Retention is enforced and old snapshots pruned.
Configuration
Defined in services/nextcloud/nextcloud-backups.yaml:
Secrets
Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (services/nextcloud/nextcloud-backups.yaml):
B2_ACCOUNT_ID— Backblaze application key IDB2_ACCOUNT_KEY— Backblaze application key secretRESTIC_REPOSITORY— B2 bucket path (e.g.,b2:bucket-name:path)RESTIC_PASSWORD— Encryption password for Restic snapshots
Recovery Procedures
Recovering a Longhorn Volume Snapshot
- Identify the snapshot you want to restore from in the Longhorn UI or via
longhornctl. - Create a new volume from the snapshot (or detach and reattach the original).
- Update PVC references if needed.
Restoring Nextcloud User Data
-
Spin up a temporary pod with the Restic image and secrets.
-
Run:
restic restore latest --target /restore --repo $RESTIC_REPOSITORY -
Copy restored files back to the Nextcloud NFS share (
robin.home.nest:/nextcloud).
Monitoring & Alerts
Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:
- Failed Longhorn backup jobs
- Restic CronJob failures (Nextcloud backups)
- Storage capacity warnings on Longhorn volumes
Related Documentation
- Longhorn Storage — Volume management and storage classes
- External Secrets Operator — How Vault secrets are synced into the cluster
- Backblaze B2 — Offsite object storage provider
TODO: Add a diagram showing the full backup data flow with timestamps.