GNMON-001 M5: Dashboard-Rollout + AI-Verfuegbarkeitsregel auf dem KI-Server
gnmon_dashboard-Rolle (Rootless-Quadlet unter gn-dashboard 984, ASPNETCORE_URLS=127.0.0.1:8080), Containerfile aus MCR-Basisimages, Image-Transport per podman save/load (v0.1.0, ID 1b71e6d35069). keep-id verworfen (crun-Overlay-Rechtebefund), Default-Mapping wie alle GNMON-Dienste. GnAiEnvironmentDown-Regel (gn_ai_active_env == 0, 2m, critical) aus GNAI-001 M4 integriert; Suite gruen und idempotent. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
# Monitoring – Dokumentationsindex und Projektstand
|
||||
|
||||
**Dokumenten-ID:** GNMON-000-DOC
|
||||
**Stand:** 2026-11-05
|
||||
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 Monitoringsuite auf dem KI-Server produktiv (System-Scope), Dashboard-Deployment auf dem Server offen
|
||||
**Stand:** 2026-11-06
|
||||
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 inkl. Dashboard auf dem KI-Server produktiv (System-Scope); NPM/OIDC-Ingress folgt in M6
|
||||
|
||||
## Dokumente
|
||||
|
||||
@@ -228,6 +228,49 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
|
||||
(`product_name` im sysfs dort nicht lesbar); möglicher Fallback über
|
||||
PCI-Vendor/Device-ID bei Bedarf.
|
||||
|
||||
### M5: Dashboard-Rollout + AI-Verfügbarkeitsregel (implementiert 06.11.2026)
|
||||
|
||||
Betreiber-Go 06.11.2026; vorheriger Repo-Abgleich mit dem LLM-Stack-Agent
|
||||
(commit 7ec8a97 GNAI-001 M4): keine Konflikte; Andocken stattdessen über
|
||||
die neue Textfile-Metrik.
|
||||
|
||||
- **Integration `gn_ai_active_env`:** Die `ai_environment_switch`-Rolle
|
||||
(GNAI-001 M4) schreibt `gn_ai_active_env{env}` in **unser** Textfile-
|
||||
Verzeichnis (`/var/lib/gn-monitor/textfile`, 30-s-Timer, als root
|
||||
geschriebene 644-Datei für node_exporter lesbar). Metrik traf bereits
|
||||
in VictoriaMetrics ein (`env="production"` = 1); die vom LLM-Stack
|
||||
vorgeschlagene Regel wurde als Gruppe `gnmon-ai` übernommen:
|
||||
`GnAiEnvironmentDown` (`gn_ai_active_env == 0`, 2 min, critical).
|
||||
vmalert lädt und evaluiert sie fehlerfrei (Regelabruf `/api/v1/rules`).
|
||||
- **Dashboard-Container** (`gnmon_dashboard`-Rolle, Tag `dashboard`):
|
||||
- `images/dashboard/Containerfile`: Multi-Stage-Build aus
|
||||
**MCR** (`dotnet/sdk:10.0` → `dotnet/aspnet:10.0`, beide nicht dem
|
||||
Docker-Hub-Rate-Limit unterworfen), Non-Root-User 984 im Image,
|
||||
`ASPNETCORE_URLS` host-lokal voreingestellt. Build auf der
|
||||
Workstation verifiziert, dann transportiert (`podman save` → scp →
|
||||
als `gn-dashboard` geladen): `docker.io/ghostnet/dashboard:v0.1.0`,
|
||||
Image-ID `1b71e6d35069…` (identisch zur Workstation).
|
||||
- Quadlet unter `/etc/containers/systemd/users/984`,
|
||||
`Network=host` (Abfragen der host-lokalen Monitoring-APIs),
|
||||
`ASPNETCORE_URLS=http://127.0.0.1:8080`, Restart=on-failure;
|
||||
Verifikations-Tasks im Playbook auf HTTP 200.
|
||||
- **keep-id-Befund (06.11.2026):** `UserNS=keep-id` (Plan-Ansatz
|
||||
„Non-Root-User gemappt auf 984") scheitert serverseitig mit crun
|
||||
„Permission denied" auf dem Overlay-`merged`-Verzeichnis der
|
||||
gn-dashboard-Storage — während dasselbe bei `gn-monitor` (VM-Image)
|
||||
funktioniert; Ursache offen. Entscheidung: Default-Rootless-Mapping
|
||||
wie bei allen GNMON-Diensten; der Image-User 984 läuft damit in der
|
||||
gn-dashboard-SubUID-Range (beobachtet: Host-UID 625271 = 624288+983),
|
||||
Attribution über den Range eindeutig.
|
||||
- **Verifikation (06.11.2026):** Dashboard antwortet auf dem Server mit
|
||||
der vollständigen Blazor-Shell (Ressourcen/GPUs-Layout im HTML);
|
||||
Gesamtsuite `--tags accounts,collector,metrics,logs,alerting,dashboard`
|
||||
grün, Zweitlauf `changed=0, failed=0` (idempotent). Playbook-Befund:
|
||||
ansible-`uri` liefert bei chunked Responses keinen `body` —
|
||||
Inhaltsprüfung nicht portabel, auf HTTP 200 verifiziert.
|
||||
- **Aufgeräumt:** Dashboard-Tar auf Server und Workstation gelöscht,
|
||||
versehentlicher Root-Load entfernt; kein verwaister Container-Rest.
|
||||
|
||||
## Noch offen
|
||||
|
||||
1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität,
|
||||
@@ -237,8 +280,11 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
|
||||
keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag:
|
||||
`vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal-
|
||||
Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung.
|
||||
3. M5-Rest auf `ghostnet-ki`: Dashboard-Container (gn-dashboard 984,
|
||||
Image-Transport wie VM etc., `ASPNETCORE_URLS=127.0.0.1:8080`,
|
||||
Rootless-Quadlet), danach externer Zugriff über NPM + OIDC/Keycloak
|
||||
(M6); nftables-Regeln aus GNMON-001 §7 (aktuell binden alle Dienste
|
||||
bereits host-lokal; 8080 erst mit dem Dashboard relevant).
|
||||
3. M6 auf `ghostnet-ki`: externer Dashboard-Zugriff über NPM (CT-103)
|
||||
mit TLS + WebSocket-Upgrade sowie OIDC/Keycloak (Confidential Client
|
||||
`gn-dashboard`, CT-102); damit einhergehend Bindung von :8080 auf
|
||||
die Host-LAN-IP und nftables-Regel „8080 nur von 10.0.0.10"
|
||||
(GNMON-001 §7; solange das Dashboard rein host-lokal läuft, ist
|
||||
keine Regel nötig — alles andere bindet bereits nur Loopback).
|
||||
4. keep-id-Ursachenanalyse (crun/Overlay-Rechte der gn-dashboard-Storage)
|
||||
bei Bedarf; Entscheidung zum Default-Mapping steht in Abschnitt M5.
|
||||
|
||||
@@ -72,3 +72,12 @@ gnmon_victoria_metrics_image_ref: docker.io/victoriametrics/victoria-metrics:v1.
|
||||
gnmon_loki_image_ref: docker.io/grafana/loki:3.6.19
|
||||
gnmon_alertmanager_image_ref: docker.io/prom/alertmanager:v0.34.2
|
||||
gnmon_vmalert_image_ref: docker.io/victoriametrics/vmalert:v1.154.1
|
||||
|
||||
# --- Dashboard ---------------------------------------------------------------
|
||||
gnmon_dashboard_scope: system
|
||||
gnmon_dashboard_systemd_scope: system
|
||||
gnmon_dashboard_quadlet_dir: /etc/containers/systemd/users/984
|
||||
# Transportiertes Image (Workstation-Build, podman save/load am 06.11.2026:
|
||||
# docker.io/ghostnet/dashboard:v0.1.0, ID 1b71e6d35069 — identisch zur
|
||||
# Workstation, Build aus images/dashboard/Containerfile).
|
||||
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
|
||||
|
||||
@@ -19,6 +19,8 @@
|
||||
tags: [alerting]
|
||||
- role: gnmon_vmalert
|
||||
tags: [alerting]
|
||||
- role: gnmon_dashboard
|
||||
tags: [dashboard]
|
||||
post_tasks:
|
||||
- name: Verify the GPU metrics are ingested
|
||||
ansible.builtin.uri:
|
||||
@@ -117,3 +119,29 @@
|
||||
| map(attribute='labels.alertname') | list | join(', ') }}
|
||||
when: gnmon_vmalert_selftest_rule | bool
|
||||
tags: [alerting]
|
||||
|
||||
- name: Wait for the dashboard to answer on the host loopback
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ gnmon_dashboard_http_listen }}/"
|
||||
status_code: 200
|
||||
register: gnmon_dashboard_ready
|
||||
retries: 20
|
||||
delay: 3
|
||||
until: gnmon_dashboard_ready.status == 200
|
||||
tags: [dashboard]
|
||||
|
||||
- name: Verify the dashboard serves the Blazor shell
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ gnmon_dashboard_http_listen }}/"
|
||||
status_code: 200
|
||||
return_content: false
|
||||
register: gnmon_dashboard_shell
|
||||
until: gnmon_dashboard_shell.status == 200
|
||||
retries: 3
|
||||
delay: 2
|
||||
tags: [dashboard]
|
||||
|
||||
- name: Show the dashboard verdict
|
||||
ansible.builtin.debug:
|
||||
msg: "Dashboard antwortet auf http://{{ gnmon_dashboard_http_listen }}/"
|
||||
tags: [dashboard]
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
|
||||
# M1 workstation default: user scope (dashboard served from the dev
|
||||
# machine). M5 server: system scope under the gn-dashboard account
|
||||
# (984), quadlet under /etc/containers/systemd/users/984, managed via
|
||||
# runuser + systemctl --user. The image is transported (podman
|
||||
# save/load), so the inventory pins the tag ref either way.
|
||||
gnmon_dashboard_scope: user
|
||||
gnmon_dashboard_systemd_scope: user
|
||||
gnmon_dashboard_run_user: gn-dashboard
|
||||
gnmon_dashboard_run_uid: 984
|
||||
gnmon_dashboard_run_home: /var/lib/ghostnet-dashboard
|
||||
gnmon_dashboard_quadlet_dir: >-
|
||||
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
|
||||
if gnmon_dashboard_scope == 'system'
|
||||
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
|
||||
gnmon_dashboard_quadlet_dir: >-
|
||||
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
|
||||
if gnmon_dashboard_scope == 'system'
|
||||
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
|
||||
|
||||
# Transported image (built from images/dashboard/Containerfile).
|
||||
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
|
||||
|
||||
# Host loopback only (GNMON-001 section 7); NPM ingress follows in M6.
|
||||
gnmon_dashboard_http_listen: "127.0.0.1:8080"
|
||||
@@ -0,0 +1,16 @@
|
||||
# GNMON-001: dashboard role handlers.
|
||||
- name: Restart dashboard (user scope)
|
||||
ansible.builtin.systemd:
|
||||
name: gnmon-dashboard.service
|
||||
scope: user
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
when: gnmon_dashboard_scope == 'user'
|
||||
|
||||
- name: Restart dashboard (system scope)
|
||||
ansible.builtin.command:
|
||||
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||
/usr/bin/systemctl, --user, restart, gnmon-dashboard.service]
|
||||
when: gnmon_dashboard_scope == 'system'
|
||||
@@ -0,0 +1,60 @@
|
||||
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
|
||||
- name: Validate dashboard declarations
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- gnmon_dashboard_scope in ['user', 'system']
|
||||
- gnmon_dashboard_image_ref is match('^[a-z0-9./:-]+$')
|
||||
- gnmon_dashboard_http_listen is match('^127\.0\.0\.1:[0-9]+$')
|
||||
fail_msg: >-
|
||||
gnmon_dashboard requires scope user|system, an image ref and a
|
||||
host-loopback listen address.
|
||||
|
||||
- name: Create the dashboard quadlet directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ gnmon_dashboard_quadlet_dir }}"
|
||||
state: directory
|
||||
mode: "0755"
|
||||
|
||||
- name: Install the dashboard quadlet
|
||||
ansible.builtin.template:
|
||||
src: gnmon-dashboard.container.j2
|
||||
dest: "{{ gnmon_dashboard_quadlet_dir }}/gnmon-dashboard.container"
|
||||
mode: "0644"
|
||||
notify: ["Restart dashboard (user scope)", "Restart dashboard (system scope)"]
|
||||
|
||||
- name: Reload the workload user manager (system scope)
|
||||
ansible.builtin.command:
|
||||
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||
/usr/bin/systemctl, --user, daemon-reload]
|
||||
changed_when: false
|
||||
when: gnmon_dashboard_scope == 'system'
|
||||
|
||||
- name: Enable and start the workload unit (system scope)
|
||||
ansible.builtin.command:
|
||||
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||
/usr/bin/systemctl, --user, start, gnmon-dashboard.service]
|
||||
changed_when: false
|
||||
when: gnmon_dashboard_scope == 'system'
|
||||
|
||||
- name: Enable and start the dashboard (user scope)
|
||||
ansible.builtin.systemd:
|
||||
name: gnmon-dashboard.service
|
||||
scope: user
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
when: gnmon_dashboard_scope == 'user'
|
||||
|
||||
# Quadlet-generated user units cannot be enabled; the [Install] section is
|
||||
# only documentation there. Linger keeps the unit alive across sessions;
|
||||
# verify it is on so the dashboard survives reboots.
|
||||
- name: Verify linger is enabled for the workload user (system scope)
|
||||
ansible.builtin.stat:
|
||||
path: "/var/lib/systemd/linger/{{ gnmon_dashboard_run_user }}"
|
||||
register: gnmon_dashboard_linger
|
||||
failed_when: not gnmon_dashboard_linger.stat.exists
|
||||
when: gnmon_dashboard_scope == 'system'
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
[Unit]
|
||||
Description=GNMON Blazor dashboard (GNMON-001)
|
||||
Documentation=https://learn.microsoft.com/aspnet/core/blazor/hosting-models
|
||||
|
||||
[Container]
|
||||
ContainerName=gnmon-dashboard
|
||||
Image={{ gnmon_dashboard_image_ref }}
|
||||
# Host network: the app queries the host-loopback monitoring APIs
|
||||
# (VictoriaMetrics 8428, Loki 3100, Alertmanager 9093).
|
||||
Network=host
|
||||
# Default rootless mapping (same as the other GNMON services): the image
|
||||
# runs as its non-root user (984) inside the container, which maps onto
|
||||
# the gn-dashboard subuid range on the host. keep-id was tested 06.11.2026
|
||||
# and rejected: crun fails with Permission denied on the overlay merged
|
||||
# dir for the gn-dashboard storage (works for gn-monitor; cause open).
|
||||
Environment=ASPNETCORE_URLS=http://{{ gnmon_dashboard_http_listen }}
|
||||
Environment=DOTNET_GENERATE_ASPNET_KEY=true
|
||||
|
||||
[Service]
|
||||
Restart=on-failure
|
||||
RestartSec=2s
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -60,6 +60,24 @@ groups:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: node_exporter target disappeared from VictoriaMetrics
|
||||
|
||||
- name: gnmon-ai
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: GnAiEnvironmentDown
|
||||
# Textfile metric from the ai_environment_switch role (GNAI-001 M4):
|
||||
# 0 when no sanctioned environment is state-active or its engine
|
||||
# unit is down. Rule proposal accepted 06.11.2026 (GNMON side).
|
||||
expr: gn_ai_active_env == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "AI environment {{ '{{' }} $labels.env {{ '}}' }} not serving"
|
||||
description: >-
|
||||
The state-active AI environment reports gn_ai_active_env == 0
|
||||
for 2 minutes (engine unit down or environment state missing).
|
||||
Check the environment switch state and recovery timer.
|
||||
{% if gnmon_vmalert_selftest_rule %}
|
||||
- name: gnmon-selftest
|
||||
interval: 30s
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# GNMON-001 section 6: Blazor Server dashboard image.
|
||||
# Multi-stage build from the MCR .NET images (not Docker Hub, so server
|
||||
# side rebuilds stay possible even under Docker Hub rate limits).
|
||||
# Runs as a non-root user; the rootless quadlet maps it onto the
|
||||
# gn-dashboard host account (UID 984) via UserNS=keep-id.
|
||||
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Restore first so the layer cache survives source edits.
|
||||
COPY Ghostnet.Dashboard.csproj .
|
||||
RUN dotnet restore Ghostnet.Dashboard.csproj
|
||||
|
||||
COPY . .
|
||||
RUN dotnet publish Ghostnet.Dashboard.csproj -c Release -o /app --no-restore
|
||||
|
||||
FROM mcr.microsoft.com/dotnet/aspnet:10.0
|
||||
WORKDIR /app
|
||||
COPY --from=build /app .
|
||||
|
||||
# Non-root runtime user; keeps UID equal to the host workload account
|
||||
# (gn-dashboard, 984) so host-side processes are attributable.
|
||||
ENV ASPNETCORE_URLS=http://127.0.0.1:8080 \
|
||||
DOTNET_GENERATE_ASPNET_KEY=true
|
||||
USER 984:984
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["dotnet", "Ghostnet.Dashboard.dll"]
|
||||
@@ -0,0 +1,3 @@
|
||||
bin/
|
||||
obj/
|
||||
*.user
|
||||
Reference in New Issue
Block a user