qo.is/infrastructure
Watch
3
0
Fork
You've already forked infrastructure
 0
Code Issues 16 Pull requests 5 Projects 1 Activity Actions

Observability: updates, filesystem errors and backups #178

New issue
Open
opened 2026-08-23 11:00:13 +00:00 by fabianhauser · 12 comments
fabianhauser commented 2026-08-23 11:00:13 +00:00
Owner
Copy link

We are currently missing o11y and alerts for some stuff:

  • Filesystem integrity: we should collect btrfs metrics, and have alerts, so we know that our filesystems are fine.

Updates

  • make sure there has been a successfull main CI run of this repo at least once every 48h
  • maybe we can check that our nixos /etc/lsb-release DISTRIB_RELEASE is <1y e.g. once a week?

Backups

  • backups: the relevant service has run once in last 30h and exited without error
    • postgresBackup on hosts with postgres
    • borg backups (note that some hosts might have multiple, see qois.backup module.
  • maybe we could have more metrics on this, e.g. borg list has backup with recent date
  • Maybe we could introduce some verification, e.g. for verification (borg compact && borg check monthly?)
We are currently missing o11y and alerts for some stuff: - [ ] Filesystem integrity: we should collect `btrfs` metrics, and have alerts, so we know that our filesystems are fine. ## Updates - [ ] make sure there has been a successfull main CI run of this repo at least once every 48h - [ ] maybe we can check that our nixos `/etc/lsb-release` `DISTRIB_RELEASE` is <1y e.g. once a week? ## Backups - [ ] backups: the relevant service has run once in last 30h and exited without error - [ ] postgresBackup on hosts with postgres - [x] borg backups (note that some hosts might have multiple, see `qois.backup` module. - [ ] maybe we could have more metrics on this, e.g. `borg list` has backup with recent date - [ ] Maybe we could introduce some verification, e.g. for verification (`borg compact && borg check` monthly?)
fabianhauser changed title from Observability for updates and backups to Observability for updates, filesystem errors and backups 2026-08-23 11:49:06 +00:00
fabianhauser changed title from Observability for updates, filesystem errors and backups to Observability: updates, filesystem errors and backups 2026-08-23 13:38:55 +00:00
fabianhauser commented 2026-08-29 14:27:34 +00:00
Author
Owner
Copy link

Key topics:

  • prometheus metrics
    • telegraf systemd metrics
    • how to query prometheus metrics
    • probably we get an exit status and a timestamp when that happend or changed
  • alertmanager alerts
    • srvos as reference
Key topics: * prometheus metrics * telegraf systemd metrics * how to query prometheus metrics * probably we get an exit status and a timestamp when that happend or changed * alertmanager alerts * srvos as reference
das-g commented 2026-08-29 14:29:20 +00:00
Owner
Copy link

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/systemd_units/README.md sieht nützlich aus.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/systemd_units/README.md sieht nützlich aus.
das-g commented 2026-08-29 15:08:36 +00:00
Owner
Copy link

OK, die richtige Metrik(en) hab' ich glaub' gefunden: systemd_units_sub_code mit name​~=​.*borg.*

Aber wie sind die Infos dort zu interpretieren? Offenbar sind alle betreffenden Units dead.

OK, die richtige Metrik(en) hab' ich glaub' gefunden: [`systemd_units_sub_code` mit `name`​`~=`​`.*borg.*`](https://monitoring.qo.is/a/grafana-metricsdrilldown-app/drilldown?from=now-2d&to=now&timezone=browser&var-metrics_filters=name%7C%3D~%7C.%2Aborg.%2A&var-filters=name%7C%3D~%7C.%2Aborg.%2A&var-labelsWingman=%28none%29&layout=grid&filters-rule=&filters-prefix=&filters-suffix=&search_txt=systemd_units_sub&filter-firing-alerts=&var-metrics-reducer-sort-by=default&filters-recent=&var-ds=PBFA97CFB590B2093&var-other_metric_filters=&metric=systemd_units_sub_code&actionView=results&var-groupby=$__all&breakdownLayout=grid&breakdownSearchText=&from-2=now-1h&to-2=now&timezone-2=browser) Aber wie sind die Infos dort zu interpretieren? Offenbar sind alle betreffenden Units `dead`.
das-g commented 2026-08-29 15:33:04 +00:00
Owner
Copy link

Ah, systemd_units_status_errno ergibt wohl mehr Sinn, da der "Value" dort wohl der Exit-Code ist.

Hab' damit nun mal einen Alert erstellt: https://monitoring.qo.is/alerting/grafana/cfwnmnkzh83cwc/view

apiVersion: 1
groups:
    - orgId: 1
      name: Check Backups
      folder: Backups
      interval: 1h
      rules:
        - uid: cfwnmnkzh83cwc
          title: borgbackup systemd_units exit code
          condition: B
          data:
            - refId: systemd_units_status_errno-avg
              relativeTimeRange:
                from: 600
                to: 0
              datasourceUid: PBFA97CFB590B2093
              model:
                datasource:
                    type: prometheus
                    uid: PBFA97CFB590B2093
                editorMode: code
                exemplar: false
                expr: systemd_units_status_errno{ name=~"borgbackup-.*"}
                format: time_series
                fromExploreMetrics: false
                instant: true
                interval: ""
                intervalMs: 15000
                legendFormat: __auto
                maxDataPoints: 500
                range: false
                refId: systemd_units_status_errno-avg
            - refId: B
              queryType: expression
              datasourceUid: __expr__
              model:
                conditions:
                    - evaluator:
                        params:
                            - 0
                        type: ne
                      operator:
                        type: and
                      query:
                        params:
                            - B
                      reducer:
                        params: []
                        type: last
                      type: query
                datasource:
                    type: __expr__
                    uid: __expr__
                expression: systemd_units_status_errno-avg
                intervalMs: 1000
                maxDataPoints: 43200
                refId: B
                type: threshold
          noDataState: NoData
          execErrState: Error
          annotations: {}
          labels: {}
          isPaused: false
          notification_settings:
            receiver: grafana-default-email

@fabianhauser, kannst du mal draufgucken, ob der so sinnvoll parametrisiert ist?

(Dieser Alert meldet nur, wenn ein non-zero Exit-Code von einem Backup-Prozess beobachtet wird. Ob die Backup-Services in den letzten 30 Tagen gelaufen sind, prüft er nicht.)

Ah, `systemd_units_status_errno` ergibt wohl mehr Sinn, da der "Value" dort wohl der Exit-Code ist. Hab' damit nun mal einen Alert erstellt: https://monitoring.qo.is/alerting/grafana/cfwnmnkzh83cwc/view <details> ```yaml apiVersion: 1 groups: - orgId: 1 name: Check Backups folder: Backups interval: 1h rules: - uid: cfwnmnkzh83cwc title: borgbackup systemd_units exit code condition: B data: - refId: systemd_units_status_errno-avg relativeTimeRange: from: 600 to: 0 datasourceUid: PBFA97CFB590B2093 model: datasource: type: prometheus uid: PBFA97CFB590B2093 editorMode: code exemplar: false expr: systemd_units_status_errno{ name=~"borgbackup-.*"} format: time_series fromExploreMetrics: false instant: true interval: "" intervalMs: 15000 legendFormat: __auto maxDataPoints: 500 range: false refId: systemd_units_status_errno-avg - refId: B queryType: expression datasourceUid: __expr__ model: conditions: - evaluator: params: - 0 type: ne operator: type: and query: params: - B reducer: params: [] type: last type: query datasource: type: __expr__ uid: __expr__ expression: systemd_units_status_errno-avg intervalMs: 1000 maxDataPoints: 43200 refId: B type: threshold noDataState: NoData execErrState: Error annotations: {} labels: {} isPaused: false notification_settings: receiver: grafana-default-email ``` </details> @fabianhauser, kannst du mal draufgucken, ob der so sinnvoll parametrisiert ist? (Dieser Alert meldet nur, wenn ein non-zero Exit-Code von einem Backup-Prozess beobachtet wird. Ob die Backup-Services in den letzten 30 Tagen gelaufen sind, prüft er nicht.)
👍 1
das-g commented 2026-08-29 16:42:21 +00:00
Owner
Copy link

(Dieser Alert meldet nur, wenn ein non-zero Exit-Code von einem Backup-Prozess beobachtet wird. Ob die Backup-Services in den letzten 30 Tagen gelaufen sind, prüft er nicht.)

Um das zu lösen:

Zusätzlicher Alert, der feuert, wenn borgbackup-job-data-local.service in den letzten 30 Tagen nicht gelaufen ist (oder der entsprechende Exit-Code nicht eingesammelt wurde): https://monitoring.qo.is/alerting/grafana/afwnrnhti5ukgf/view

apiVersion: 1
groups:
    - orgId: 1
      name: Check Backups
      folder: Backups
      interval: 1h
      rules:
        - uid: cfwnmnkzh83cwc
          title: borgbackup systemd_units exit code
          condition: B
          data:
            - refId: systemd_units_status_errno-avg
              relativeTimeRange:
                from: 600
                to: 0
              datasourceUid: PBFA97CFB590B2093
              model:
                datasource:
                    type: prometheus
                    uid: PBFA97CFB590B2093
                editorMode: code
                exemplar: false
                expr: systemd_units_status_errno{ name=~"borgbackup-.*"}
                format: time_series
                fromExploreMetrics: false
                instant: true
                interval: ""
                intervalMs: 15000
                legendFormat: __auto
                maxDataPoints: 500
                range: false
                refId: systemd_units_status_errno-avg
            - refId: B
              queryType: expression
              datasourceUid: __expr__
              model:
                conditions:
                    - evaluator:
                        params:
                            - 0
                        type: ne
                      operator:
                        type: and
                      query:
                        params:
                            - B
                      reducer:
                        params: []
                        type: last
                      type: query
                datasource:
                    type: __expr__
                    uid: __expr__
                expression: systemd_units_status_errno-avg
                intervalMs: 1000
                maxDataPoints: 43200
                refId: B
                type: threshold
          noDataState: NoData
          execErrState: Error
          isPaused: false
          notification_settings:
            receiver: grafana-default-email
        - uid: afwnrnhti5ukgf
          title: Did borgbackup-job-data-local run in the last 30 days?
          condition: C
          data:
            - refId: A
              relativeTimeRange:
                from: 2592000
                to: 0
              datasourceUid: PBFA97CFB590B2093
              model:
                editorMode: builder
                expr: count(systemd_units_status_errno{name="borgbackup-job-data-local.service"})
                instant: true
                intervalMs: 1000
                legendFormat: __auto
                maxDataPoints: 43200
                range: false
                refId: A
            - refId: C
              queryType: expression
              datasourceUid: __expr__
              model:
                conditions:
                    - evaluator:
                        params:
                            - 1
                        type: lt
                      operator:
                        type: and
                      query:
                        params:
                            - C
                      reducer:
                        params: []
                        type: last
                      type: query
                datasource:
                    type: __expr__
                    uid: __expr__
                expression: A
                intervalMs: 1000
                maxDataPoints: 43200
                refId: C
                type: threshold
          noDataState: Alerting
          execErrState: Error
          annotations: {}
          labels: {}
          isPaused: false
          notification_settings:
            receiver: grafana-default-email

Kannst du dir auch den ansehen, @fabianhauser? Falls das so passt, können wir das für die anderen borgbackup-.*-Services duplizieren:

  • borgbackup-job-system-cyprianspitz.service
  • borgbackup-job-data-local.service
  • borgbackup-repo-system-calanda.service
  • borgbackup-repo-system-cyprianspitz.service
  • borgbackup-repo-system-lindberg-build.service
  • borgbackup-repo-system-lindberg-nextcloud.service
  • borgbackup-repo-system-lindberg-webapps.service
  • borgbackup-repo-system-lindberg.service

(Diese Checkliste wurde anhand der von dieser Query zurückgegebenen Namen erstellt. To Do:

  • in Nix-Option qois.backup gucken, ob das alle sind

)

> (Dieser Alert meldet nur, wenn ein non-zero Exit-Code von einem Backup-Prozess beobachtet wird. Ob die Backup-Services in den letzten 30 Tagen gelaufen sind, prüft er nicht.) Um das zu lösen: Zusätzlicher Alert, der feuert, wenn `borgbackup-job-data-local.service` in den letzten 30 Tagen nicht gelaufen ist (oder der entsprechende Exit-Code nicht eingesammelt wurde): https://monitoring.qo.is/alerting/grafana/afwnrnhti5ukgf/view <details> ```yaml apiVersion: 1 groups: - orgId: 1 name: Check Backups folder: Backups interval: 1h rules: - uid: cfwnmnkzh83cwc title: borgbackup systemd_units exit code condition: B data: - refId: systemd_units_status_errno-avg relativeTimeRange: from: 600 to: 0 datasourceUid: PBFA97CFB590B2093 model: datasource: type: prometheus uid: PBFA97CFB590B2093 editorMode: code exemplar: false expr: systemd_units_status_errno{ name=~"borgbackup-.*"} format: time_series fromExploreMetrics: false instant: true interval: "" intervalMs: 15000 legendFormat: __auto maxDataPoints: 500 range: false refId: systemd_units_status_errno-avg - refId: B queryType: expression datasourceUid: __expr__ model: conditions: - evaluator: params: - 0 type: ne operator: type: and query: params: - B reducer: params: [] type: last type: query datasource: type: __expr__ uid: __expr__ expression: systemd_units_status_errno-avg intervalMs: 1000 maxDataPoints: 43200 refId: B type: threshold noDataState: NoData execErrState: Error isPaused: false notification_settings: receiver: grafana-default-email - uid: afwnrnhti5ukgf title: Did borgbackup-job-data-local run in the last 30 days? condition: C data: - refId: A relativeTimeRange: from: 2592000 to: 0 datasourceUid: PBFA97CFB590B2093 model: editorMode: builder expr: count(systemd_units_status_errno{name="borgbackup-job-data-local.service"}) instant: true intervalMs: 1000 legendFormat: __auto maxDataPoints: 43200 range: false refId: A - refId: C queryType: expression datasourceUid: __expr__ model: conditions: - evaluator: params: - 1 type: lt operator: type: and query: params: - C reducer: params: [] type: last type: query datasource: type: __expr__ uid: __expr__ expression: A intervalMs: 1000 maxDataPoints: 43200 refId: C type: threshold noDataState: Alerting execErrState: Error annotations: {} labels: {} isPaused: false notification_settings: receiver: grafana-default-email ``` </details> Kannst du dir auch den ansehen, @fabianhauser? Falls das so passt, können wir das für die anderen `borgbackup-.*`-Services duplizieren: - [x] borgbackup-job-system-cyprianspitz.service - [x] borgbackup-job-data-local.service - [x] borgbackup-repo-system-calanda.service - [x] borgbackup-repo-system-cyprianspitz.service - [x] borgbackup-repo-system-lindberg-build.service - [x] borgbackup-repo-system-lindberg-nextcloud.service - [x] borgbackup-repo-system-lindberg-webapps.service - [x] borgbackup-repo-system-lindberg.service (Diese Checkliste wurde anhand der von [dieser Query](https://monitoring.qo.is/explore?schemaVersion=1&panes=%7B%22boq%22:%7B%22datasource%22:%22PBFA97CFB590B2093%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22group%20by%28name%29%20%28systemd_units_status_errno%7Bname%3D~%5C%22borgbackup-.%2A%5C%22%7D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22PBFA97CFB590B2093%22%7D,%22editorMode%22:%22builder%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22-3600000%22,%22to%22:%22now%22%7D,%22compact%22:false%7D%7D&orgId=1) zurückgegebenen Namen erstellt. To Do: - [ ] in Nix-Option `qois.backup` gucken, ob das alle sind )
fabianhauser commented 2026-08-29 18:50:04 +00:00
Author
Owner
Copy link

@das-g nice! Ich habe die alerts noch etwas angepasst - jetzt feuern sie in einem 49h zeitraum, für alle borgbackup-job.* services :)

Evtl. kannst du den alert in code umsetzen, und dann auch noch einen für den postgres backup job hinzufügen?

@das-g nice! Ich habe die alerts noch etwas angepasst - jetzt feuern sie in einem 49h zeitraum, für alle `borgbackup-job.*` services :) Evtl. kannst du den alert in code umsetzen, und dann auch noch einen für den postgres backup job hinzufügen?
das-g commented 2026-08-29 19:17:36 +00:00
Owner
Copy link

Export nach den in #178 (comment) genannten Anpassungen von Fabian:

apiVersion: 1
groups:
    - orgId: 1
      name: Check Backups
      folder: Backups
      interval: 1h
      rules:
        - uid: cfwnmnkzh83cwc
          title: borgbackup systemd_units exit code
          condition: B
          data:
            - refId: systemd_units_status_errno-avg
              relativeTimeRange:
                from: 176400
                to: 0
              datasourceUid: PBFA97CFB590B2093
              model:
                datasource:
                    type: prometheus
                    uid: PBFA97CFB590B2093
                editorMode: code
                exemplar: false
                expr: sum_over_time(systemd_units_status_errno{name=~"borgbackup-job.*"}[49h])
                format: time_series
                fromExploreMetrics: false
                instant: true
                interval: ""
                intervalMs: 2.88e+07
                legendFormat: __auto
                maxDataPoints: 500
                range: false
                refId: systemd_units_status_errno-avg
            - refId: B
              queryType: expression
              datasourceUid: __expr__
              model:
                conditions:
                    - evaluator:
                        params:
                            - 0
                        type: ne
                      operator:
                        type: and
                      query:
                        params:
                            - B
                      reducer:
                        params: []
                        type: last
                      type: query
                datasource:
                    type: __expr__
                    uid: __expr__
                expression: systemd_units_status_errno-avg
                intervalMs: 1000
                maxDataPoints: 43200
                refId: B
                type: threshold
          noDataState: NoData
          execErrState: Error
          isPaused: false
          notification_settings:
            receiver: grafana-default-email
        - uid: afwnrnhti5ukgf
          title: Did borgbackup-job run in the last 26h?
          condition: C
          data:
            - refId: A
              relativeTimeRange:
                from: 172800
                to: 0
              datasourceUid: PBFA97CFB590B2093
              model:
                editorMode: code
                expr: changes(systemd_units_active_enter_timestamp_us{name=~"borgbackup-job.*"}[48h])
                instant: true
                intervalMs: 4.32e+07
                legendFormat: __auto
                maxDataPoints: 43200
                range: false
                refId: A
            - refId: C
              queryType: expression
              datasourceUid: __expr__
              model:
                conditions:
                    - evaluator:
                        params:
                            - 1
                        type: lt
                      operator:
                        type: and
                      query:
                        params:
                            - C
                      reducer:
                        params: []
                        type: last
                      type: query
                datasource:
                    type: __expr__
                    uid: __expr__
                expression: A
                intervalMs: 1000
                maxDataPoints: 43200
                refId: C
                type: threshold
          noDataState: Alerting
          execErrState: Error
          annotations: {}
          labels: {}
          isPaused: false
          notification_settings:
            receiver: grafana-default-email
Export nach den in https://git.qo.is/qo.is/infrastructure/issues/178#issuecomment-1628 genannten Anpassungen von Fabian: <details> ```yaml apiVersion: 1 groups: - orgId: 1 name: Check Backups folder: Backups interval: 1h rules: - uid: cfwnmnkzh83cwc title: borgbackup systemd_units exit code condition: B data: - refId: systemd_units_status_errno-avg relativeTimeRange: from: 176400 to: 0 datasourceUid: PBFA97CFB590B2093 model: datasource: type: prometheus uid: PBFA97CFB590B2093 editorMode: code exemplar: false expr: sum_over_time(systemd_units_status_errno{name=~"borgbackup-job.*"}[49h]) format: time_series fromExploreMetrics: false instant: true interval: "" intervalMs: 2.88e+07 legendFormat: __auto maxDataPoints: 500 range: false refId: systemd_units_status_errno-avg - refId: B queryType: expression datasourceUid: __expr__ model: conditions: - evaluator: params: - 0 type: ne operator: type: and query: params: - B reducer: params: [] type: last type: query datasource: type: __expr__ uid: __expr__ expression: systemd_units_status_errno-avg intervalMs: 1000 maxDataPoints: 43200 refId: B type: threshold noDataState: NoData execErrState: Error isPaused: false notification_settings: receiver: grafana-default-email - uid: afwnrnhti5ukgf title: Did borgbackup-job run in the last 26h? condition: C data: - refId: A relativeTimeRange: from: 172800 to: 0 datasourceUid: PBFA97CFB590B2093 model: editorMode: code expr: changes(systemd_units_active_enter_timestamp_us{name=~"borgbackup-job.*"}[48h]) instant: true intervalMs: 4.32e+07 legendFormat: __auto maxDataPoints: 43200 range: false refId: A - refId: C queryType: expression datasourceUid: __expr__ model: conditions: - evaluator: params: - 1 type: lt operator: type: and query: params: - C reducer: params: [] type: last type: query datasource: type: __expr__ uid: __expr__ expression: A intervalMs: 1000 maxDataPoints: 43200 refId: C type: threshold noDataState: Alerting execErrState: Error annotations: {} labels: {} isPaused: false notification_settings: receiver: grafana-default-email ``` </details>
fabianhauser commented 2026-08-30 10:34:18 +00:00
Author
Owner
Copy link

Danke @das-g ! Magst du einen PR erstellen um diese in code aufzunehmen? Siehe srvos - glaub wir können den Alert auch auf die Metrik und den Text reduzieren.

Danke @das-g ! Magst du einen PR erstellen um diese in code aufzunehmen? Siehe [srvos](https://github.com/nix-community/srvos/blob/main/nixos/roles/prometheus/default-alerts.nix) - glaub wir können den Alert auch auf die Metrik und den Text reduzieren.
das-g commented 2026-08-30 11:22:08 +00:00
Owner
Copy link

github.com/nix-community/srvos@a00436b148/nixos/roles/prometheus/default-alerts.nix (L54-L69) sieht ja auch interessant aus.

https://github.com/nix-community/srvos/blob/a00436b148ec1c2f54f60e3973af197d7c7d8009/nixos/roles/prometheus/default-alerts.nix#L54-L69 sieht ja auch interessant aus.
das-g commented 2026-08-30 12:03:48 +00:00
Owner
Copy link

@das-g wrote in #178 (comment):

expr: sum_over_time(systemd_units_status_errno{name=~"borgbackup-job.*"}[49h])

Ist sum_over_time hier nicht kontraproduktiv, @fabianhauser? Wenn einmal Exit-Code -1 und einmal Exit-Code 1 auftritt, würde sich das in einer Summe ja "ausgleichen".

@das-g wrote in https://git.qo.is/qo.is/infrastructure/issues/178#issuecomment-1631: > expr: sum_over_time(systemd_units_status_errno{name=~"borgbackup-job.*"}[49h]) Ist `sum_over_time` hier nicht kontraproduktiv, @fabianhauser? Wenn einmal Exit-Code `-1` und einmal Exit-Code `1` auftritt, würde sich das in einer Summe ja "ausgleichen".
fabianhauser commented 2026-08-30 13:51:16 +00:00
Author
Owner
Copy link

Hmm, ja vielleicht. Evtl. besser count in dem fall...

Hmm, ja vielleicht. Evtl. besser `count` in dem fall...
das-g commented 2026-08-30 14:14:24 +00:00
Owner
Copy link

Braucht's überhaupt eine Aggregierung? Ich vermute ohne eine würden wir über jeden fehlgeschlagenen Backup-Job alertet, was ja auch OK (und evtl. sogar gewünscht) wäre, oder?

systemd_units_status_errno{name=~"borgbackup-job.*"} != 0

scheint eine gültige Query zu sein.

Braucht's überhaupt eine Aggregierung? Ich vermute ohne eine würden wir über jeden fehlgeschlagenen Backup-Job alertet, was ja auch OK (und evtl. sogar gewünscht) wäre, oder? ``` systemd_units_status_errno{name=~"borgbackup-job.*"} != 0 ``` [scheint](https://monitoring.qo.is/explore?schemaVersion=1&panes=%7B%22e8w%22:%7B%22datasource%22:%22PBFA97CFB590B2093%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22systemd_units_status_errno%7Bname%3D~%5C%22borgbackup-job.%2A%5C%22%7D%20%21%3D%200%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22PBFA97CFB590B2093%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22,%22exemplar%22:false%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D,%22compact%22:false%7D%7D&orgId=1) eine gültige Query zu sein.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/lock-file-maintenance
monitor-backup-jobs
update-sops-rekey
kanidm-identity-provider
cyprianspitz-lanzaboote
storage-metrics-plan
nginx-access-log-loki
jellyfin
microvm-services
microvm-experiment
test-cloud
18-caral
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
urgent
This issue should be tackled soon

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
urgent
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
2026 Autumn Hackathon
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
qo.is/infrastructure#178
No description provided.