Snapshot backup host, prune and compact system backups #216

Open

fabianhauser wants to merge 5 commits from backup-snapshots into main

fabianhauser commented

2026-11-02 14:32:39 +00:00

Owner

Backup clients keep full borg access and now prune (within=14d, weekly=4, monthly=6, yearly=-1, as lindberg data-local) and compact after each backup. As a layer the clients can't reach, the backup host keeps weekly read-only snapper snapshots of the backup root for 31 days.

PeriodicJobNotSucceeded now compares the last success with the job timer's gap between its last and next trigger (+6h) instead of a fixed 30h. Prometheus retention is raised to 31d to back the lookback.

Migration on cyprianspitz (before deploy)

These commands produce the same layout disko creates (/backup and /backup/.snapshots as plain btrfs subvolume create below the top-level volume). Run them while no client backup is running (clients start at 07:06):

pgrep -u borg -a borg                         # must print nothing
btrfs subvolume show /mnt/data/backup         # expected to fail: still a plain directory
mv /mnt/data/backup /mnt/data/backup.old
btrfs subvolume create /mnt/data/backup
btrfs subvolume create /mnt/data/backup/.snapshots
cp -a --reflink=always /mnt/data/backup.old/. /mnt/data/backup/
rm -rf /mnt/data/backup.old

After deploy, tmpfiles sets .snapshots to 0750 root:root.

Checklist

migration ran before deploy; btrfs subvolume list /mnt/data shows backup and backup/.snapshots
systemctl list-timers snapper-timeline shows the next run; after it, snapper -c backup list shows a snapshot and btrfs property get -ts /mnt/data/backup/.snapshots/<n>/snapshot ro is ro=true
client borgbackup-job-system-* runs succeed with prune + compact
systemd_job_next_execution_gap_seconds is visible in Prometheus for all periodic jobs, and no spurious PeriodicJobNotSucceeded fires

Backup clients keep full borg access and now prune (`within=14d, weekly=4, monthly=6, yearly=-1`, as lindberg `data-local`) and compact after each backup. As a layer the clients can't reach, the backup host keeps weekly read-only snapper snapshots of the backup root for 31 days. `PeriodicJobNotSucceeded` now compares the last success with the job timer's gap between its last and next trigger (+6h) instead of a fixed 30h. Prometheus retention is raised to 31d to back the lookback. ## Migration on cyprianspitz (before deploy) These commands produce the same layout disko creates (`/backup` and `/backup/.snapshots` as plain `btrfs subvolume create` below the top-level volume). Run them while no client backup is running (clients start at 07:06): ```bash pgrep -u borg -a borg # must print nothing btrfs subvolume show /mnt/data/backup # expected to fail: still a plain directory mv /mnt/data/backup /mnt/data/backup.old btrfs subvolume create /mnt/data/backup btrfs subvolume create /mnt/data/backup/.snapshots cp -a --reflink=always /mnt/data/backup.old/. /mnt/data/backup/ rm -rf /mnt/data/backup.old ``` After deploy, tmpfiles sets `.snapshots` to `0750 root:root`. ## Checklist - [ ] migration ran before deploy; `btrfs subvolume list /mnt/data` shows `backup` and `backup/.snapshots` - [ ] `systemctl list-timers snapper-timeline` shows the next run; after it, `snapper -c backup list` shows a snapshot and `btrfs property get -ts /mnt/data/backup/.snapshots/<n>/snapshot ro` is `ro=true` - [ ] client `borgbackup-job-system-*` runs succeed with prune + compact - [ ] `systemd_job_next_execution_gap_seconds` is visible in Prometheus for all periodic jobs, and no spurious `PeriodicJobNotSucceeded` fires

fabianhauser added 4 commits

2026-11-02 14:32:39 +00:00

Alert on periodic jobs by their schedule gap d5ebddf318

Store cyprianspitz backups in a btrfs subvolume 239dac5b7a

Snapshot backup repos weekly with snapper 4236017860

Prune and compact system backups

CI / build (push) Successful in 13m26s

Details

CI / deploy (docs-ops.qo.is) (push) Has been skipped

Details

CI / deploy (system-physical) (push) Has been skipped

Details