134 lines
6.7 KiB
Markdown
134 lines
6.7 KiB
Markdown
|
# Backups & Recovery
|
||
|
|
|
||
|
|
The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up.
|
||
|
|
|
||
|
|
## Backup Architecture
|
||
|
|
|
||
|
|
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||
|
|
|
||
|
|
|
||
|
|
│ Longhorn │────▶│ Recurring Jobs │────▶│ Backblaze B2 │
|
||
|
|
|
||
|
|
│ Volumes │ │ (weekly) │ │ (offsite) │
|
||
|
|
|
||
|
|
|
||
|
|
└─────────────┘ └──────────────────┘ └─────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||
|
|
|
||
|
|
|
||
|
|
│ Nextcloud │────▶│ Restic CronJob │────▶│ Backblaze B2 │
|
||
|
|
|
||
|
|
│ User Data │ │ (daily) │ │ (offsite) │
|
||
|
|
|
||
|
|
└─────────────┘ └──────────────────┘ └─────────────────┘
|
||
|
|
|
||
|
|
## Longhorn Recurring Backups
|
||
|
|
|
||
|
|
Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`.
|
||
|
|
|
||
|
|
### Schedule
|
||
|
|
|
||
|
|
| Group | Task | Cron | Retain |
|
||
|
|
|-------|------|------|--------|
|
||
|
|
| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots |
|
||
|
|
|
||
|
|
### What Gets Backed Up
|
||
|
|
|
||
|
|
Any PVC with the recurring labels is included. This covers:
|
||
|
|
|
||
|
|
- **Application config:** Vault data, Longhorn system volumes
|
||
|
|
- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
|
||
|
|
- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values)
|
||
|
|
|
||
|
|
### What Does NOT Get Backed Up
|
||
|
|
|
||
|
|
- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups.
|
||
|
|
- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).
|
||
|
|
|
||
|
|
### Configuration
|
||
|
|
|
||
|
|
Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`:
|
||
|
|
|
||
|
|
### Backup Destination
|
||
|
|
|
||
|
|
Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`.
|
||
|
|
|
||
|
|
## Nextcloud User Data Backup
|
||
|
|
|
||
|
|
Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace.
|
||
|
|
|
||
|
|
### Schedule
|
||
|
|
|
||
|
|
- **Frequency:** Daily at 02:00 EST
|
||
|
|
- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped.
|
||
|
|
|
||
|
|
### Retention Policy
|
||
|
|
|
||
|
|
| Period | Keep |
|
||
|
|
|--------|------|
|
||
|
|
| Daily | 7 days |
|
||
|
|
| Weekly | 4 weeks |
|
||
|
|
| Monthly | 12 months |
|
||
|
|
|
||
|
|
After retention enforcement, old snapshots are pruned automatically.
|
||
|
|
|
||
|
|
### How It Works
|
||
|
|
|
||
|
|
1. The CronJob spins up a Restic container (`restic/restic:0.16.4`).
|
||
|
|
2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`.
|
||
|
|
3. If the B2 repository doesn't exist yet, Restic initializes it on first run.
|
||
|
|
4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output.
|
||
|
|
5. Retention is enforced and old snapshots pruned.
|
||
|
|
|
||
|
|
### Configuration
|
||
|
|
|
||
|
|
Defined in `services/nextcloud/nextcloud-backups.yaml`:
|
||
|
|
|
||
|
|
### Secrets
|
||
|
|
|
||
|
|
Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`):
|
||
|
|
|
||
|
|
- `B2_ACCOUNT_ID` — Backblaze application key ID
|
||
|
|
- `B2_ACCOUNT_KEY` — Backblaze application key secret
|
||
|
|
- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`)
|
||
|
|
- `RESTIC_PASSWORD` — Encryption password for Restic snapshots
|
||
|
|
|
||
|
|
## Recovery Procedures
|
||
|
|
|
||
|
|
### Recovering a Longhorn Volume Snapshot
|
||
|
|
|
||
|
|
1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`.
|
||
|
|
2. Create a new volume from the snapshot (or detach and reattach the original).
|
||
|
|
3. Update PVC references if needed.
|
||
|
|
|
||
|
|
### Restoring Nextcloud User Data
|
||
|
|
|
||
|
|
1. Spin up a temporary pod with the Restic image and secrets.
|
||
|
|
2. Run:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
restic restore latest --target /restore --repo $RESTIC_REPOSITORY
|
||
|
|
```
|
||
|
|
|
||
|
|
3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`).
|
||
|
|
|
||
|
|
## Monitoring & Alerts
|
||
|
|
|
||
|
|
Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:
|
||
|
|
|
||
|
|
- Failed Longhorn backup jobs
|
||
|
|
- Restic CronJob failures (Nextcloud backups)
|
||
|
|
- Storage capacity warnings on Longhorn volumes
|
||
|
|
|
||
|
|
## Related Documentation
|
||
|
|
|
||
|
|
- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes
|
||
|
|
- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster
|
||
|
|
- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*TODO: Add a diagram showing the full backup data flow with timestamps.*
|