qo.is/infrastructure
Watch
3
0
Fork
You've already forked infrastructure
 0
Code Issues 16 Pull requests 5 Projects 1 Activity Actions

Fixes from log file analysis #166

New issue
Open
opened 2026-08-20 21:20:37 +00:00 by fabianhauser · 0 comments
fabianhauser commented 2026-08-20 21:20:37 +00:00
Owner
Copy link

I ran some analysis on our new monitoring/logs and already fixed some things. Below are the remaining findings - some may not be very relevant (or even incorrect - copy-paste from AI output).

Log Research Report

Source: Grafana Loki (P8E80F9AEF21F6940), last 7 days as of 2026-08-20.
Links are Grafana Explore deeplinks.

Checklist below tracks resolution status. Each item is tagged
[actionable] (has a concrete fix we can plan/implement),
[investigate] (needs root-causing before a fix is clear), or
[noise] (expected/non-actionable, informational only).

General Issues (multiple hosts)

  • tailscaled/headscale reconnect flapping [investigate]:
    lindberg and cyprianspitz tailscaled repeatedly lose control-plane
    poll (PollNetMap: unexpected EOF, ~every 40–70s continuously);
    cyprianspitz nginx (fronting vpn.qo.is) shows matching SSL_read() failed crit errors proxying /ts2021 (99.8% of its nginx log volume,
    3,647 lines/7d); headscale.service on cyprianspitz logs 42,766
    reconnect-churn lines/7d for node.id 7/8 (lindberg/cyprianspitz). One
    coherent flapping-VPN-control-plane story across 3 services/2 hosts —
    worth root-causing (MTU? relay/DERP path?).

    • Action: investigate MTU along the path and DERP/relay behavior for
      node.id 7/8; compare against a healthy node's polling interval.
    • Received error: PollNetMap: unexpected EOF

  • SSH brute-force/scanning traffic [investigate]: on
    internet-facing calanda (20,352 "Invalid user" attempts/7d) and
    lindberg (30,713/7d) — expected background noise for exposed SSH, but
    volume is large enough to warrant a fail2ban/rate-limit review.

    • Action: check whether fail2ban (or equivalent) is currently configured
      on calanda/lindberg; if not, evaluate adding it or tightening SSH
      exposure (e.g. VPN-only access, port change, key-only + rate limiting).

cyprianspitz

  • headscale.service invalid hostname rejection [actionable]:
    13× WRN Rejecting invalid hostname update from hostinfo for node.id=13,
    hostname "suurmunamägi" (non-ASCII/uppercase fails headscale's
    validation) — client-side misconfig.
    • Action: rename that client's Tailscale hostname to an ASCII-safe value.
  • smartd Raw_Read_Error_Rate change [investigate]: single
    SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 79 to 80
    on /dev/sdb — no failed self-tests, worth watching, not yet actionable.
  • fwupd-refresh.service bursty cadence [investigate]: running
    ~every 25 min (bursty) instead of the expected daily cadence.
    • Checked 2026-08-20: live systemctl list-timers/status on
      cyprianspitz shows a normal ~1.5h gap between the last two runs, next
      trigger over an hour out, and no custom OnCalendar override in this
      repo — doesn't reproduce the reported bursty pattern right now. Needs
      a fresh Loki query to see whether the ~25min bursts correlate with
      reboots/restarts before any config change is warranted.

lindberg-nextcloud

  • Postgres unique-constraint violation in Nextcloud filecache
    [investigate]: 33× duplicate key value violates unique constraint "oc_filecache_extended_pkey" over 7d, different fileids/times — looks
    like a concurrency bug in upload tracking, not one-off.
    • Action: check installed Nextcloud version against upstream bug
      tracker for this constraint name; consider filing/tracking upstream if
      it's a known race.
  • Nextcloud News app feed parse errors [actionable]: 327× Error while parsing feed over 7d (e.g. https://www.nginx.com/feed/ — no
    parser can handle stream; https://www.inside-it.ch/atom.xml — HTTP
    404), each with a full PHP stack trace — broken/stale RSS subscriptions.
    • Action: remove or fix the offending feed subscriptions in the News app
      (user-level cleanup, not infra code).

lindberg-webapps

  • nginx 404 scanning from 10.250.0.2 [investigate]: ~50,750
    [error] open() ... failed (2: No such file or directory) over 7d —
    automated WordPress/CMS vuln scanning against fabianhauser.ch (all
    404s, not exploitable, but worth rate-limiting/blocking). See the
    general "mystery internal client 10.250.0.2" item above — same source IP,
    should be investigated together.
    • Action: identify 10.250.0.2 first (see general issues); once known,
      decide whether to rate-limit/block at nginx or fix the source if it's
      an internal misconfigured scanner.
  • Postgres invalid uuid literal [actionable]: single ERROR: invalid input syntax for type uuid: "${Oe.uuid}" — client sent a literal
    unrendered template placeholder.
    • Action: find and fix the calling app/frontend template that isn't
      interpolating ${Oe.uuid} before sending the request.
  • forgejo Libravatar bot email lookup failure [noise]: 59×
    LibravatarService.FromEmail(email=gitea-actions[bot]@noreply.gitea.io): error mail: missing '@' or angle-addr — CI bot's synthetic email fails
    avatar lookup on every actions run (cosmetic).
  • vaultwarden dotfile-probing scanner traffic [noise]: 27×
    [WARN] Parameter guard 'p: PathBuf' is forwarding: BadStart('.') in
    tight bursts — dotfile-probing scanner traffic (.env, .git), same
    actor family as the WordPress scanning above.
  • nginx "upstream buffered to temp file" warning noise [noise]:
    3,678 of 3,692 total nginx [warn] lines (99.6%) for large git.qo.is
    git-upload-pack transfers — expected, but drowns out real nginx
    warnings.

lindberg-build

  • libpod-conmon-*.scope cgroups warning [noise]: Failed to open cgroups file: .../memory.events on lifecycle events (6× in sample
    window) — recurring low-signal podman/conmon warning.
I ran some analysis on our new monitoring/logs and already fixed some things. Below are the remaining findings - some may not be very relevant (or even incorrect - copy-paste from AI output). --- # Log Research Report Source: Grafana Loki (`P8E80F9AEF21F6940`), last 7 days as of 2026-08-20. Links are Grafana Explore deeplinks. Checklist below tracks resolution status. Each item is tagged **[actionable]** (has a concrete fix we can plan/implement), **[investigate]** (needs root-causing before a fix is clear), or **[noise]** (expected/non-actionable, informational only). ## General Issues (multiple hosts) - [ ] **tailscaled/headscale reconnect flapping** `[investigate]`: `lindberg` and `cyprianspitz` tailscaled repeatedly lose control-plane poll (`PollNetMap: unexpected EOF`, ~every 40–70s continuously); `cyprianspitz` nginx (fronting `vpn.qo.is`) shows matching `SSL_read() failed` crit errors proxying `/ts2021` (99.8% of its nginx log volume, 3,647 lines/7d); `headscale.service` on `cyprianspitz` logs 42,766 reconnect-churn lines/7d for node.id 7/8 (lindberg/cyprianspitz). One coherent flapping-VPN-control-plane story across 3 services/2 hosts — worth root-causing (MTU? relay/DERP path?). - Action: investigate MTU along the path and DERP/relay behavior for node.id 7/8; compare against a healthy node's polling interval. - `Received error: PollNetMap: unexpected EOF` - [ ] **SSH brute-force/scanning traffic** `[investigate]`: on internet-facing `calanda` (20,352 "Invalid user" attempts/7d) and `lindberg` (30,713/7d) — expected background noise for exposed SSH, but volume is large enough to warrant a fail2ban/rate-limit review. - Action: check whether fail2ban (or equivalent) is currently configured on calanda/lindberg; if not, evaluate adding it or tightening SSH exposure (e.g. VPN-only access, port change, key-only + rate limiting). ## cyprianspitz - [ ] **`headscale.service` invalid hostname rejection** `[actionable]`: 13× `WRN Rejecting invalid hostname update from hostinfo` for node.id=13, hostname `"suurmunamägi"` (non-ASCII/uppercase fails headscale's validation) — client-side misconfig. - Action: rename that client's Tailscale hostname to an ASCII-safe value. - [ ] **`smartd` Raw_Read_Error_Rate change** `[investigate]`: single `SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 79 to 80` on `/dev/sdb` — no failed self-tests, worth watching, not yet actionable. - [ ] **`fwupd-refresh.service` bursty cadence** `[investigate]`: running ~every 25 min (bursty) instead of the expected daily cadence. - Checked 2026-08-20: live `systemctl list-timers`/`status` on cyprianspitz shows a normal ~1.5h gap between the last two runs, next trigger over an hour out, and no custom `OnCalendar` override in this repo — doesn't reproduce the reported bursty pattern right now. Needs a fresh Loki query to see whether the ~25min bursts correlate with reboots/restarts before any config change is warranted. ## lindberg-nextcloud - [ ] **Postgres unique-constraint violation in Nextcloud filecache** `[investigate]`: 33× `duplicate key value violates unique constraint "oc_filecache_extended_pkey"` over 7d, different fileids/times — looks like a concurrency bug in upload tracking, not one-off. - Action: check installed Nextcloud version against upstream bug tracker for this constraint name; consider filing/tracking upstream if it's a known race. - [ ] **Nextcloud News app feed parse errors** `[actionable]`: 327× `Error while parsing feed` over 7d (e.g. `https://www.nginx.com/feed/` — no parser can handle stream; `https://www.inside-it.ch/atom.xml` — HTTP 404), each with a full PHP stack trace — broken/stale RSS subscriptions. - Action: remove or fix the offending feed subscriptions in the News app (user-level cleanup, not infra code). ## lindberg-webapps - [ ] **nginx 404 scanning from `10.250.0.2`** `[investigate]`: ~50,750 `[error] open() ... failed (2: No such file or directory)` over 7d — automated WordPress/CMS vuln scanning against `fabianhauser.ch` (all 404s, not exploitable, but worth rate-limiting/blocking). See the general "mystery internal client 10.250.0.2" item above — same source IP, should be investigated together. - Action: identify `10.250.0.2` first (see general issues); once known, decide whether to rate-limit/block at nginx or fix the source if it's an internal misconfigured scanner. - [ ] **Postgres invalid uuid literal** `[actionable]`: single `ERROR: invalid input syntax for type uuid: "${Oe.uuid}"` — client sent a literal unrendered template placeholder. - Action: find and fix the calling app/frontend template that isn't interpolating `${Oe.uuid}` before sending the request. - [ ] **forgejo Libravatar bot email lookup failure** `[noise]`: 59× `LibravatarService.FromEmail(email=gitea-actions[bot]@noreply.gitea.io): error mail: missing '@' or angle-addr` — CI bot's synthetic email fails avatar lookup on every actions run (cosmetic). - [ ] **vaultwarden dotfile-probing scanner traffic** `[noise]`: 27× `[WARN] Parameter guard 'p: PathBuf' is forwarding: BadStart('.')` in tight bursts — dotfile-probing scanner traffic (`.env`, `.git`), same actor family as the WordPress scanning above. - [ ] **nginx "upstream buffered to temp file" warning noise** `[noise]`: 3,678 of 3,692 total nginx `[warn]` lines (99.6%) for large `git.qo.is` `git-upload-pack` transfers — expected, but drowns out real nginx warnings. ## lindberg-build - [ ] **`libpod-conmon-*.scope` cgroups warning** `[noise]`: `Failed to open cgroups file: .../memory.events` on lifecycle events (6× in sample window) — recurring low-signal podman/conmon warning.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/lock-file-maintenance
monitor-backup-jobs
update-sops-rekey
kanidm-identity-provider
cyprianspitz-lanzaboote
storage-metrics-plan
nginx-access-log-loki
jellyfin
microvm-services
microvm-experiment
test-cloud
18-caral
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
urgent
This issue should be tackled soon

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
urgent
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
qo.is/infrastructure#166
No description provided.