k3s/documentation/technical/backups-and-recovery.md

# Backups & Recovery

The Nest uses a two-tier backup strategy: **Longhorn volume snapshots** for application configuration data (backed up to Backblaze B2 weekly), and **Restic** for Nextcloud user data (backed up daily). Media storage is not backed up.

## Backup Architecture


## Longhorn Recurring Backups

Longhorn's built-in recurring job system handles snapshots of all PVCs labeled with `recurring-job.longhorn.io/source: "enabled"` and grouped under `app-config-group`.

### Schedule

| Group | Task | Cron | Retain |
|-------|------|------|--------|
| `app-config-group` | backup | `0 4 * * 0` (Sundays, 04:00 EST) | 4 snapshots |

### What Gets Backed Up

Any PVC with the recurring labels is included. This covers:

- **Application config:** Vault data, Longhorn system volumes
- **Service state:** Seerr cache, DroppedNeedle config/cache, Shelfmark config, slskd app data, audiobookshelf metadata/config, sabnzbd config, sonarr/radarr/bazarr configs, nextcloud DB, keykeeper data
- **Monitoring:** Prometheus and Grafana PVCs (labeled in HelmChart values)

### What Does NOT Get Backed Up

- **Media storage** (`/yarr/media`, `/yarr/downloads`) — NFS shares are not part of Longhorn and have no backups.
- **Transcoding scratch space** — DroppedNeedle's transcode PVC is excluded (it regenerates from source FLAC files).

### Configuration

Longhorn recurring jobs are defined in `longhorn/weekly-backups.yaml`:

### Backup Destination

Snapshots are pushed to Backblaze B2 using credentials stored in Vault and synced via the External Secrets Operator (`longhorn/backblaze-secrets.yaml`). The S3-compatible endpoint is configured through `AWS_ENDPOINTS`.

## Nextcloud User Data Backup

Nextcloud user data (files, contacts, calendars) is backed up separately using **Restic** to Backblaze B2. This runs as a Kubernetes CronJob in the `nextcloud` namespace.

### Schedule

- **Frequency:** Daily at 02:00 EST
- **Concurrency policy:** `Forbid` — if a previous run is still active, the new job is skipped.

### Retention Policy

| Period | Keep |
|--------|------|
| Daily | 7 days |
| Weekly | 4 weeks |
| Monthly | 12 months |

After retention enforcement, old snapshots are pruned automatically.

### How It Works

1. The CronJob spins up a Restic container (`restic/restic:0.16.4`).
2. Secrets (B2 credentials, repository URL, password) are injected from Vault via `ExternalSecret`.
3. If the B2 repository doesn't exist yet, Restic initializes it on first run.
4. The entire Nextcloud user data volume (`/data`) is backed up with verbose output.
5. Retention is enforced and old snapshots pruned.

### Configuration

Defined in `services/nextcloud/nextcloud-backups.yaml`:

### Secrets

Backblaze credentials are stored in Vault and synced into the cluster via ExternalSecrets (`services/nextcloud/nextcloud-backups.yaml`):

- `B2_ACCOUNT_ID` — Backblaze application key ID
- `B2_ACCOUNT_KEY` — Backblaze application key secret
- `RESTIC_REPOSITORY` — B2 bucket path (e.g., `b2:bucket-name:path`)
- `RESTIC_PASSWORD` — Encryption password for Restic snapshots

## Recovery Procedures

### Recovering a Longhorn Volume Snapshot

1. Identify the snapshot you want to restore from in the Longhorn UI or via `longhornctl`.
2. Create a new volume from the snapshot (or detach and reattach the original).
3. Update PVC references if needed.

### Restoring Nextcloud User Data

1. Spin up a temporary pod with the Restic image and secrets.
2. Run:

   ```bash
   restic restore latest --target /restore --repo $RESTIC_REPOSITORY
   ```

3. Copy restored files back to the Nextcloud NFS share (`robin.home.nest:/nextcloud`).

## Monitoring & Alerts

Backup health is monitored by the Prometheus stack. Alertmanager sends notifications to Discord for:

- Failed Longhorn backup jobs
- Restic CronJob failures (Nextcloud backups)
- Storage capacity warnings on Longhorn volumes

## Related Documentation

- [Longhorn Storage](longhorn-storage.md) — Volume management and storage classes
- [External Secrets Operator](external-secrets-operator.md) — How Vault secrets are synced into the cluster
- [Backblaze B2](https://www.backblaze.com/cloud-storage) — Offsite object storage provider

---

*TODO: Add a diagram showing the full backup data flow with timestamps.*