GNMON-001 M5: Dashboard-Rollout + AI-Verfuegbarkeitsregel auf dem KI-Server

gnmon_dashboard-Rolle (Rootless-Quadlet unter gn-dashboard 984,
ASPNETCORE_URLS=127.0.0.1:8080), Containerfile aus MCR-Basisimages,
Image-Transport per podman save/load (v0.1.0, ID 1b71e6d35069).
keep-id verworfen (crun-Overlay-Rechtebefund), Default-Mapping wie alle
GNMON-Dienste. GnAiEnvironmentDown-Regel (gn_ai_active_env == 0, 2m,
critical) aus GNAI-001 M4 integriert; Suite gruen und idempotent.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
2026-11-06 14:45:41 +01:00
parent 7ec8a970b7
commit 76feaa9549
10 changed files with 262 additions and 7 deletions
@@ -1,8 +1,8 @@
# Monitoring – Dokumentationsindex und Projektstand
**Dokumenten-ID:** GNMON-000-DOC
**Stand:** 2026-11-05
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 Monitoringsuite auf dem KI-Server produktiv (System-Scope), Dashboard-Deployment auf dem Server offen
**Stand:** 2026-11-06
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 inkl. Dashboard auf dem KI-Server produktiv (System-Scope); NPM/OIDC-Ingress folgt in M6
## Dokumente
@@ -228,6 +228,49 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
(`product_name` im sysfs dort nicht lesbar); möglicher Fallback über
PCI-Vendor/Device-ID bei Bedarf.
### M5: Dashboard-Rollout + AI-Verfügbarkeitsregel (implementiert 06.11.2026)
Betreiber-Go 06.11.2026; vorheriger Repo-Abgleich mit dem LLM-Stack-Agent
(commit 7ec8a97 GNAI-001 M4): keine Konflikte; Andocken stattdessen über
die neue Textfile-Metrik.
- **Integration `gn_ai_active_env`:** Die `ai_environment_switch`-Rolle
(GNAI-001 M4) schreibt `gn_ai_active_env{env}` in **unser** Textfile-
Verzeichnis (`/var/lib/gn-monitor/textfile`, 30-s-Timer, als root
geschriebene 644-Datei für node_exporter lesbar). Metrik traf bereits
in VictoriaMetrics ein (`env="production"` = 1); die vom LLM-Stack
vorgeschlagene Regel wurde als Gruppe `gnmon-ai` übernommen:
`GnAiEnvironmentDown` (`gn_ai_active_env == 0`, 2 min, critical).
vmalert lädt und evaluiert sie fehlerfrei (Regelabruf `/api/v1/rules`).
- **Dashboard-Container** (`gnmon_dashboard`-Rolle, Tag `dashboard`):
- `images/dashboard/Containerfile`: Multi-Stage-Build aus
**MCR** (`dotnet/sdk:10.0` → `dotnet/aspnet:10.0`, beide nicht dem
Docker-Hub-Rate-Limit unterworfen), Non-Root-User 984 im Image,
`ASPNETCORE_URLS` host-lokal voreingestellt. Build auf der
Workstation verifiziert, dann transportiert (`podman save` → scp →
als `gn-dashboard` geladen): `docker.io/ghostnet/dashboard:v0.1.0`,
Image-ID `1b71e6d35069…` (identisch zur Workstation).
- Quadlet unter `/etc/containers/systemd/users/984`,
`Network=host` (Abfragen der host-lokalen Monitoring-APIs),
`ASPNETCORE_URLS=http://127.0.0.1:8080`, Restart=on-failure;
Verifikations-Tasks im Playbook auf HTTP 200.
- **keep-id-Befund (06.11.2026):** `UserNS=keep-id` (Plan-Ansatz
„Non-Root-User gemappt auf 984") scheitert serverseitig mit crun
„Permission denied" auf dem Overlay-`merged`-Verzeichnis der
gn-dashboard-Storage — während dasselbe bei `gn-monitor` (VM-Image)
funktioniert; Ursache offen. Entscheidung: Default-Rootless-Mapping
wie bei allen GNMON-Diensten; der Image-User 984 läuft damit in der
gn-dashboard-SubUID-Range (beobachtet: Host-UID 625271 = 624288+983),
Attribution über den Range eindeutig.
- **Verifikation (06.11.2026):** Dashboard antwortet auf dem Server mit
der vollständigen Blazor-Shell (Ressourcen/GPUs-Layout im HTML);
Gesamtsuite `--tags accounts,collector,metrics,logs,alerting,dashboard`
grün, Zweitlauf `changed=0, failed=0` (idempotent). Playbook-Befund:
ansible-`uri` liefert bei chunked Responses keinen `body` —
Inhaltsprüfung nicht portabel, auf HTTP 200 verifiziert.
- **Aufgeräumt:** Dashboard-Tar auf Server und Workstation gelöscht,
versehentlicher Root-Load entfernt; kein verwaister Container-Rest.
## Noch offen
1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität,
@@ -237,8 +280,11 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag:
`vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal-
Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung.
3. M5-Rest auf `ghostnet-ki`: Dashboard-Container (gn-dashboard 984,
Image-Transport wie VM etc., `ASPNETCORE_URLS=127.0.0.1:8080`,
Rootless-Quadlet), danach externer Zugriff über NPM + OIDC/Keycloak
(M6); nftables-Regeln aus GNMON-001 §7 (aktuell binden alle Dienste
bereits host-lokal; 8080 erst mit dem Dashboard relevant).
3. M6 auf `ghostnet-ki`: externer Dashboard-Zugriff über NPM (CT-103)
mit TLS + WebSocket-Upgrade sowie OIDC/Keycloak (Confidential Client
`gn-dashboard`, CT-102); damit einhergehend Bindung von :8080 auf
die Host-LAN-IP und nftables-Regel „8080 nur von 10.0.0.10"
(GNMON-001 §7; solange das Dashboard rein host-lokal läuft, ist
keine Regel nötig — alles andere bindet bereits nur Loopback).
4. keep-id-Ursachenanalyse (crun/Overlay-Rechte der gn-dashboard-Storage)
bei Bedarf; Entscheidung zum Default-Mapping steht in Abschnitt M5.
@@ -72,3 +72,12 @@ gnmon_victoria_metrics_image_ref: docker.io/victoriametrics/victoria-metrics:v1.
gnmon_loki_image_ref: docker.io/grafana/loki:3.6.19
gnmon_alertmanager_image_ref: docker.io/prom/alertmanager:v0.34.2
gnmon_vmalert_image_ref: docker.io/victoriametrics/vmalert:v1.154.1
# --- Dashboard ---------------------------------------------------------------
gnmon_dashboard_scope: system
gnmon_dashboard_systemd_scope: system
gnmon_dashboard_quadlet_dir: /etc/containers/systemd/users/984
# Transportiertes Image (Workstation-Build, podman save/load am 06.11.2026:
# docker.io/ghostnet/dashboard:v0.1.0, ID 1b71e6d35069 — identisch zur
# Workstation, Build aus images/dashboard/Containerfile).
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
@@ -19,6 +19,8 @@
tags: [alerting]
- role: gnmon_vmalert
tags: [alerting]
- role: gnmon_dashboard
tags: [dashboard]
post_tasks:
- name: Verify the GPU metrics are ingested
ansible.builtin.uri:
@@ -117,3 +119,29 @@
| map(attribute='labels.alertname') | list | join(', ') }}
when: gnmon_vmalert_selftest_rule | bool
tags: [alerting]
- name: Wait for the dashboard to answer on the host loopback
ansible.builtin.uri:
url: "http://{{ gnmon_dashboard_http_listen }}/"
status_code: 200
register: gnmon_dashboard_ready
retries: 20
delay: 3
until: gnmon_dashboard_ready.status == 200
tags: [dashboard]
- name: Verify the dashboard serves the Blazor shell
ansible.builtin.uri:
url: "http://{{ gnmon_dashboard_http_listen }}/"
status_code: 200
return_content: false
register: gnmon_dashboard_shell
until: gnmon_dashboard_shell.status == 200
retries: 3
delay: 2
tags: [dashboard]
- name: Show the dashboard verdict
ansible.builtin.debug:
msg: "Dashboard antwortet auf http://{{ gnmon_dashboard_http_listen }}/"
tags: [dashboard]
@@ -0,0 +1,25 @@
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
# M1 workstation default: user scope (dashboard served from the dev
# machine). M5 server: system scope under the gn-dashboard account
# (984), quadlet under /etc/containers/systemd/users/984, managed via
# runuser + systemctl --user. The image is transported (podman
# save/load), so the inventory pins the tag ref either way.
gnmon_dashboard_scope: user
gnmon_dashboard_systemd_scope: user
gnmon_dashboard_run_user: gn-dashboard
gnmon_dashboard_run_uid: 984
gnmon_dashboard_run_home: /var/lib/ghostnet-dashboard
gnmon_dashboard_quadlet_dir: >-
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
if gnmon_dashboard_scope == 'system'
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
gnmon_dashboard_quadlet_dir: >-
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
if gnmon_dashboard_scope == 'system'
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
# Transported image (built from images/dashboard/Containerfile).
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
# Host loopback only (GNMON-001 section 7); NPM ingress follows in M6.
gnmon_dashboard_http_listen: "127.0.0.1:8080"
@@ -0,0 +1,16 @@
# GNMON-001: dashboard role handlers.
- name: Restart dashboard (user scope)
ansible.builtin.systemd:
name: gnmon-dashboard.service
scope: user
state: restarted
daemon_reload: true
when: gnmon_dashboard_scope == 'user'
- name: Restart dashboard (system scope)
ansible.builtin.command:
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
/usr/bin/systemctl, --user, restart, gnmon-dashboard.service]
when: gnmon_dashboard_scope == 'system'
@@ -0,0 +1,60 @@
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
- name: Validate dashboard declarations
ansible.builtin.assert:
that:
- gnmon_dashboard_scope in ['user', 'system']
- gnmon_dashboard_image_ref is match('^[a-z0-9./:-]+$')
- gnmon_dashboard_http_listen is match('^127\.0\.0\.1:[0-9]+$')
fail_msg: >-
gnmon_dashboard requires scope user|system, an image ref and a
host-loopback listen address.
- name: Create the dashboard quadlet directory
ansible.builtin.file:
path: "{{ gnmon_dashboard_quadlet_dir }}"
state: directory
mode: "0755"
- name: Install the dashboard quadlet
ansible.builtin.template:
src: gnmon-dashboard.container.j2
dest: "{{ gnmon_dashboard_quadlet_dir }}/gnmon-dashboard.container"
mode: "0644"
notify: ["Restart dashboard (user scope)", "Restart dashboard (system scope)"]
- name: Reload the workload user manager (system scope)
ansible.builtin.command:
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
/usr/bin/systemctl, --user, daemon-reload]
changed_when: false
when: gnmon_dashboard_scope == 'system'
- name: Enable and start the workload unit (system scope)
ansible.builtin.command:
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
/usr/bin/systemctl, --user, start, gnmon-dashboard.service]
changed_when: false
when: gnmon_dashboard_scope == 'system'
- name: Enable and start the dashboard (user scope)
ansible.builtin.systemd:
name: gnmon-dashboard.service
scope: user
enabled: true
state: started
daemon_reload: true
when: gnmon_dashboard_scope == 'user'
# Quadlet-generated user units cannot be enabled; the [Install] section is
# only documentation there. Linger keeps the unit alive across sessions;
# verify it is on so the dashboard survives reboots.
- name: Verify linger is enabled for the workload user (system scope)
ansible.builtin.stat:
path: "/var/lib/systemd/linger/{{ gnmon_dashboard_run_user }}"
register: gnmon_dashboard_linger
failed_when: not gnmon_dashboard_linger.stat.exists
when: gnmon_dashboard_scope == 'system'
@@ -0,0 +1,24 @@
[Unit]
Description=GNMON Blazor dashboard (GNMON-001)
Documentation=https://learn.microsoft.com/aspnet/core/blazor/hosting-models
[Container]
ContainerName=gnmon-dashboard
Image={{ gnmon_dashboard_image_ref }}
# Host network: the app queries the host-loopback monitoring APIs
# (VictoriaMetrics 8428, Loki 3100, Alertmanager 9093).
Network=host
# Default rootless mapping (same as the other GNMON services): the image
# runs as its non-root user (984) inside the container, which maps onto
# the gn-dashboard subuid range on the host. keep-id was tested 06.11.2026
# and rejected: crun fails with Permission denied on the overlay merged
# dir for the gn-dashboard storage (works for gn-monitor; cause open).
Environment=ASPNETCORE_URLS=http://{{ gnmon_dashboard_http_listen }}
Environment=DOTNET_GENERATE_ASPNET_KEY=true
[Service]
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=default.target
@@ -60,6 +60,24 @@ groups:
severity: critical
annotations:
summary: node_exporter target disappeared from VictoriaMetrics
- name: gnmon-ai
interval: 30s
rules:
- alert: GnAiEnvironmentDown
# Textfile metric from the ai_environment_switch role (GNAI-001 M4):
# 0 when no sanctioned environment is state-active or its engine
# unit is down. Rule proposal accepted 06.11.2026 (GNMON side).
expr: gn_ai_active_env == 0
for: 2m
labels:
severity: critical
annotations:
summary: "AI environment {{ '{{' }} $labels.env {{ '}}' }} not serving"
description: >-
The state-active AI environment reports gn_ai_active_env == 0
for 2 minutes (engine unit down or environment state missing).
Check the environment switch state and recovery timer.
{% if gnmon_vmalert_selftest_rule %}
- name: gnmon-selftest
interval: 30s
@@ -0,0 +1,26 @@
# GNMON-001 section 6: Blazor Server dashboard image.
# Multi-stage build from the MCR .NET images (not Docker Hub, so server
# side rebuilds stay possible even under Docker Hub rate limits).
# Runs as a non-root user; the rootless quadlet maps it onto the
# gn-dashboard host account (UID 984) via UserNS=keep-id.
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
# Restore first so the layer cache survives source edits.
COPY Ghostnet.Dashboard.csproj .
RUN dotnet restore Ghostnet.Dashboard.csproj
COPY . .
RUN dotnet publish Ghostnet.Dashboard.csproj -c Release -o /app --no-restore
FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app .
# Non-root runtime user; keeps UID equal to the host workload account
# (gn-dashboard, 984) so host-side processes are attributable.
ENV ASPNETCORE_URLS=http://127.0.0.1:8080 \
DOTNET_GENERATE_ASPNET_KEY=true
USER 984:984
EXPOSE 8080
ENTRYPOINT ["dotnet", "Ghostnet.Dashboard.dll"]
@@ -0,0 +1,3 @@
bin/
obj/
*.user