qo.is/infrastructure
Watch
3
0
Fork
You've already forked infrastructure
 0
Code Issues 14 Pull requests 4 Projects 1 Activity Actions

Alert on periodic jobs without recent successful exit #195

Merged
fabianhauser merged 3 commits from backup-job-success into main 2026-10-07 18:36:17 +00:00
Conversation 0 Commits 3 Files changed 7 +103 -1
fabianhauser commented 2026-10-04 10:27:07 +00:00
Owner

Alternative to #186, for #178: alerts when a borg or postgres backup service hasn't exited successfully within 30h, based on systemd's own exit state.

  • After deploy, systemd_job_success is reported for every borg/postgres backup unit on all hosts
  • PeriodicJobNotSucceeded shows up in Grafana alerting and isn't firing
  • Remove the manual Grafana backup alerts from #178
  • Close #186
Alternative to #186, for #178: alerts when a borg or postgres backup service hasn't exited successfully within 30h, based on systemd's own exit state. - [ ] After deploy, `systemd_job_success` is reported for every borg/postgres backup unit on all hosts - [ ] `PeriodicJobNotSucceeded` shows up in Grafana alerting and isn't firing - [ ] Remove the manual Grafana backup alerts from #178 - [ ] Close #186
Alert on periodic jobs without recent successful exit
All checks were successful
CI / build (push) Successful in 5m1s
Details
CI / deploy (docs-ops.qo.is) (push) Has been skipped
Details
CI / deploy (system-physical) (push) Has been skipped
Details
CI / deploy (system-vm) (push) Has been skipped
Details
CI / deploy-ci (push) Has been skipped
Details
1acd2d007b
Move periodic job alert rule to alertmanager module
All checks were successful
CI / build (push) Successful in 4m51s
Details
CI / deploy (docs-ops.qo.is) (push) Has been skipped
Details
CI / deploy (system-physical) (push) Has been skipped
Details
CI / deploy (system-vm) (push) Has been skipped
Details
CI / deploy-ci (push) Has been skipped
Details
919017b784
fabianhauser deleted branch backup-job-success 2026-10-07 18:36:17 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
urgent
This issue should be tackled soon

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
urgent
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
qo.is/infrastructure!195
No description provided.