GNMON-001 M5: Dashboard-Rollout + AI-Verfuegbarkeitsregel auf dem KI-Server
gnmon_dashboard-Rolle (Rootless-Quadlet unter gn-dashboard 984, ASPNETCORE_URLS=127.0.0.1:8080), Containerfile aus MCR-Basisimages, Image-Transport per podman save/load (v0.1.0, ID 1b71e6d35069). keep-id verworfen (crun-Overlay-Rechtebefund), Default-Mapping wie alle GNMON-Dienste. GnAiEnvironmentDown-Regel (gn_ai_active_env == 0, 2m, critical) aus GNAI-001 M4 integriert; Suite gruen und idempotent. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
@@ -1,8 +1,8 @@
|
|||||||
# Monitoring – Dokumentationsindex und Projektstand
|
# Monitoring – Dokumentationsindex und Projektstand
|
||||||
|
|
||||||
**Dokumenten-ID:** GNMON-000-DOC
|
**Dokumenten-ID:** GNMON-000-DOC
|
||||||
**Stand:** 2026-11-05
|
**Stand:** 2026-11-06
|
||||||
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 Monitoringsuite auf dem KI-Server produktiv (System-Scope), Dashboard-Deployment auf dem Server offen
|
**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 inkl. Dashboard auf dem KI-Server produktiv (System-Scope); NPM/OIDC-Ingress folgt in M6
|
||||||
|
|
||||||
## Dokumente
|
## Dokumente
|
||||||
|
|
||||||
@@ -228,6 +228,49 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
|
|||||||
(`product_name` im sysfs dort nicht lesbar); möglicher Fallback über
|
(`product_name` im sysfs dort nicht lesbar); möglicher Fallback über
|
||||||
PCI-Vendor/Device-ID bei Bedarf.
|
PCI-Vendor/Device-ID bei Bedarf.
|
||||||
|
|
||||||
|
### M5: Dashboard-Rollout + AI-Verfügbarkeitsregel (implementiert 06.11.2026)
|
||||||
|
|
||||||
|
Betreiber-Go 06.11.2026; vorheriger Repo-Abgleich mit dem LLM-Stack-Agent
|
||||||
|
(commit 7ec8a97 GNAI-001 M4): keine Konflikte; Andocken stattdessen über
|
||||||
|
die neue Textfile-Metrik.
|
||||||
|
|
||||||
|
- **Integration `gn_ai_active_env`:** Die `ai_environment_switch`-Rolle
|
||||||
|
(GNAI-001 M4) schreibt `gn_ai_active_env{env}` in **unser** Textfile-
|
||||||
|
Verzeichnis (`/var/lib/gn-monitor/textfile`, 30-s-Timer, als root
|
||||||
|
geschriebene 644-Datei für node_exporter lesbar). Metrik traf bereits
|
||||||
|
in VictoriaMetrics ein (`env="production"` = 1); die vom LLM-Stack
|
||||||
|
vorgeschlagene Regel wurde als Gruppe `gnmon-ai` übernommen:
|
||||||
|
`GnAiEnvironmentDown` (`gn_ai_active_env == 0`, 2 min, critical).
|
||||||
|
vmalert lädt und evaluiert sie fehlerfrei (Regelabruf `/api/v1/rules`).
|
||||||
|
- **Dashboard-Container** (`gnmon_dashboard`-Rolle, Tag `dashboard`):
|
||||||
|
- `images/dashboard/Containerfile`: Multi-Stage-Build aus
|
||||||
|
**MCR** (`dotnet/sdk:10.0` → `dotnet/aspnet:10.0`, beide nicht dem
|
||||||
|
Docker-Hub-Rate-Limit unterworfen), Non-Root-User 984 im Image,
|
||||||
|
`ASPNETCORE_URLS` host-lokal voreingestellt. Build auf der
|
||||||
|
Workstation verifiziert, dann transportiert (`podman save` → scp →
|
||||||
|
als `gn-dashboard` geladen): `docker.io/ghostnet/dashboard:v0.1.0`,
|
||||||
|
Image-ID `1b71e6d35069…` (identisch zur Workstation).
|
||||||
|
- Quadlet unter `/etc/containers/systemd/users/984`,
|
||||||
|
`Network=host` (Abfragen der host-lokalen Monitoring-APIs),
|
||||||
|
`ASPNETCORE_URLS=http://127.0.0.1:8080`, Restart=on-failure;
|
||||||
|
Verifikations-Tasks im Playbook auf HTTP 200.
|
||||||
|
- **keep-id-Befund (06.11.2026):** `UserNS=keep-id` (Plan-Ansatz
|
||||||
|
„Non-Root-User gemappt auf 984") scheitert serverseitig mit crun
|
||||||
|
„Permission denied" auf dem Overlay-`merged`-Verzeichnis der
|
||||||
|
gn-dashboard-Storage — während dasselbe bei `gn-monitor` (VM-Image)
|
||||||
|
funktioniert; Ursache offen. Entscheidung: Default-Rootless-Mapping
|
||||||
|
wie bei allen GNMON-Diensten; der Image-User 984 läuft damit in der
|
||||||
|
gn-dashboard-SubUID-Range (beobachtet: Host-UID 625271 = 624288+983),
|
||||||
|
Attribution über den Range eindeutig.
|
||||||
|
- **Verifikation (06.11.2026):** Dashboard antwortet auf dem Server mit
|
||||||
|
der vollständigen Blazor-Shell (Ressourcen/GPUs-Layout im HTML);
|
||||||
|
Gesamtsuite `--tags accounts,collector,metrics,logs,alerting,dashboard`
|
||||||
|
grün, Zweitlauf `changed=0, failed=0` (idempotent). Playbook-Befund:
|
||||||
|
ansible-`uri` liefert bei chunked Responses keinen `body` —
|
||||||
|
Inhaltsprüfung nicht portabel, auf HTTP 200 verifiziert.
|
||||||
|
- **Aufgeräumt:** Dashboard-Tar auf Server und Workstation gelöscht,
|
||||||
|
versehentlicher Root-Load entfernt; kein verwaister Container-Rest.
|
||||||
|
|
||||||
## Noch offen
|
## Noch offen
|
||||||
|
|
||||||
1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität,
|
1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität,
|
||||||
@@ -237,8 +280,11 @@ Verifikation von `gn-dashboard` (984, vorbestehend).
|
|||||||
keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag:
|
keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag:
|
||||||
`vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal-
|
`vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal-
|
||||||
Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung.
|
Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung.
|
||||||
3. M5-Rest auf `ghostnet-ki`: Dashboard-Container (gn-dashboard 984,
|
3. M6 auf `ghostnet-ki`: externer Dashboard-Zugriff über NPM (CT-103)
|
||||||
Image-Transport wie VM etc., `ASPNETCORE_URLS=127.0.0.1:8080`,
|
mit TLS + WebSocket-Upgrade sowie OIDC/Keycloak (Confidential Client
|
||||||
Rootless-Quadlet), danach externer Zugriff über NPM + OIDC/Keycloak
|
`gn-dashboard`, CT-102); damit einhergehend Bindung von :8080 auf
|
||||||
(M6); nftables-Regeln aus GNMON-001 §7 (aktuell binden alle Dienste
|
die Host-LAN-IP und nftables-Regel „8080 nur von 10.0.0.10"
|
||||||
bereits host-lokal; 8080 erst mit dem Dashboard relevant).
|
(GNMON-001 §7; solange das Dashboard rein host-lokal läuft, ist
|
||||||
|
keine Regel nötig — alles andere bindet bereits nur Loopback).
|
||||||
|
4. keep-id-Ursachenanalyse (crun/Overlay-Rechte der gn-dashboard-Storage)
|
||||||
|
bei Bedarf; Entscheidung zum Default-Mapping steht in Abschnitt M5.
|
||||||
|
|||||||
@@ -72,3 +72,12 @@ gnmon_victoria_metrics_image_ref: docker.io/victoriametrics/victoria-metrics:v1.
|
|||||||
gnmon_loki_image_ref: docker.io/grafana/loki:3.6.19
|
gnmon_loki_image_ref: docker.io/grafana/loki:3.6.19
|
||||||
gnmon_alertmanager_image_ref: docker.io/prom/alertmanager:v0.34.2
|
gnmon_alertmanager_image_ref: docker.io/prom/alertmanager:v0.34.2
|
||||||
gnmon_vmalert_image_ref: docker.io/victoriametrics/vmalert:v1.154.1
|
gnmon_vmalert_image_ref: docker.io/victoriametrics/vmalert:v1.154.1
|
||||||
|
|
||||||
|
# --- Dashboard ---------------------------------------------------------------
|
||||||
|
gnmon_dashboard_scope: system
|
||||||
|
gnmon_dashboard_systemd_scope: system
|
||||||
|
gnmon_dashboard_quadlet_dir: /etc/containers/systemd/users/984
|
||||||
|
# Transportiertes Image (Workstation-Build, podman save/load am 06.11.2026:
|
||||||
|
# docker.io/ghostnet/dashboard:v0.1.0, ID 1b71e6d35069 — identisch zur
|
||||||
|
# Workstation, Build aus images/dashboard/Containerfile).
|
||||||
|
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
|
||||||
|
|||||||
@@ -19,6 +19,8 @@
|
|||||||
tags: [alerting]
|
tags: [alerting]
|
||||||
- role: gnmon_vmalert
|
- role: gnmon_vmalert
|
||||||
tags: [alerting]
|
tags: [alerting]
|
||||||
|
- role: gnmon_dashboard
|
||||||
|
tags: [dashboard]
|
||||||
post_tasks:
|
post_tasks:
|
||||||
- name: Verify the GPU metrics are ingested
|
- name: Verify the GPU metrics are ingested
|
||||||
ansible.builtin.uri:
|
ansible.builtin.uri:
|
||||||
@@ -117,3 +119,29 @@
|
|||||||
| map(attribute='labels.alertname') | list | join(', ') }}
|
| map(attribute='labels.alertname') | list | join(', ') }}
|
||||||
when: gnmon_vmalert_selftest_rule | bool
|
when: gnmon_vmalert_selftest_rule | bool
|
||||||
tags: [alerting]
|
tags: [alerting]
|
||||||
|
|
||||||
|
- name: Wait for the dashboard to answer on the host loopback
|
||||||
|
ansible.builtin.uri:
|
||||||
|
url: "http://{{ gnmon_dashboard_http_listen }}/"
|
||||||
|
status_code: 200
|
||||||
|
register: gnmon_dashboard_ready
|
||||||
|
retries: 20
|
||||||
|
delay: 3
|
||||||
|
until: gnmon_dashboard_ready.status == 200
|
||||||
|
tags: [dashboard]
|
||||||
|
|
||||||
|
- name: Verify the dashboard serves the Blazor shell
|
||||||
|
ansible.builtin.uri:
|
||||||
|
url: "http://{{ gnmon_dashboard_http_listen }}/"
|
||||||
|
status_code: 200
|
||||||
|
return_content: false
|
||||||
|
register: gnmon_dashboard_shell
|
||||||
|
until: gnmon_dashboard_shell.status == 200
|
||||||
|
retries: 3
|
||||||
|
delay: 2
|
||||||
|
tags: [dashboard]
|
||||||
|
|
||||||
|
- name: Show the dashboard verdict
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: "Dashboard antwortet auf http://{{ gnmon_dashboard_http_listen }}/"
|
||||||
|
tags: [dashboard]
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
|
||||||
|
# M1 workstation default: user scope (dashboard served from the dev
|
||||||
|
# machine). M5 server: system scope under the gn-dashboard account
|
||||||
|
# (984), quadlet under /etc/containers/systemd/users/984, managed via
|
||||||
|
# runuser + systemctl --user. The image is transported (podman
|
||||||
|
# save/load), so the inventory pins the tag ref either way.
|
||||||
|
gnmon_dashboard_scope: user
|
||||||
|
gnmon_dashboard_systemd_scope: user
|
||||||
|
gnmon_dashboard_run_user: gn-dashboard
|
||||||
|
gnmon_dashboard_run_uid: 984
|
||||||
|
gnmon_dashboard_run_home: /var/lib/ghostnet-dashboard
|
||||||
|
gnmon_dashboard_quadlet_dir: >-
|
||||||
|
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
|
||||||
|
if gnmon_dashboard_scope == 'system'
|
||||||
|
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
|
||||||
|
gnmon_dashboard_quadlet_dir: >-
|
||||||
|
{{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid
|
||||||
|
if gnmon_dashboard_scope == 'system'
|
||||||
|
else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }}
|
||||||
|
|
||||||
|
# Transported image (built from images/dashboard/Containerfile).
|
||||||
|
gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0
|
||||||
|
|
||||||
|
# Host loopback only (GNMON-001 section 7); NPM ingress follows in M6.
|
||||||
|
gnmon_dashboard_http_listen: "127.0.0.1:8080"
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# GNMON-001: dashboard role handlers.
|
||||||
|
- name: Restart dashboard (user scope)
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: gnmon-dashboard.service
|
||||||
|
scope: user
|
||||||
|
state: restarted
|
||||||
|
daemon_reload: true
|
||||||
|
when: gnmon_dashboard_scope == 'user'
|
||||||
|
|
||||||
|
- name: Restart dashboard (system scope)
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||||
|
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||||
|
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||||
|
/usr/bin/systemctl, --user, restart, gnmon-dashboard.service]
|
||||||
|
when: gnmon_dashboard_scope == 'system'
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# GNMON-001 section 6: Blazor dashboard rootless quadlet.
|
||||||
|
- name: Validate dashboard declarations
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- gnmon_dashboard_scope in ['user', 'system']
|
||||||
|
- gnmon_dashboard_image_ref is match('^[a-z0-9./:-]+$')
|
||||||
|
- gnmon_dashboard_http_listen is match('^127\.0\.0\.1:[0-9]+$')
|
||||||
|
fail_msg: >-
|
||||||
|
gnmon_dashboard requires scope user|system, an image ref and a
|
||||||
|
host-loopback listen address.
|
||||||
|
|
||||||
|
- name: Create the dashboard quadlet directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ gnmon_dashboard_quadlet_dir }}"
|
||||||
|
state: directory
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Install the dashboard quadlet
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnmon-dashboard.container.j2
|
||||||
|
dest: "{{ gnmon_dashboard_quadlet_dir }}/gnmon-dashboard.container"
|
||||||
|
mode: "0644"
|
||||||
|
notify: ["Restart dashboard (user scope)", "Restart dashboard (system scope)"]
|
||||||
|
|
||||||
|
- name: Reload the workload user manager (system scope)
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||||
|
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||||
|
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||||
|
/usr/bin/systemctl, --user, daemon-reload]
|
||||||
|
changed_when: false
|
||||||
|
when: gnmon_dashboard_scope == 'system'
|
||||||
|
|
||||||
|
- name: Enable and start the workload unit (system scope)
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --,
|
||||||
|
/usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}",
|
||||||
|
"XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}",
|
||||||
|
/usr/bin/systemctl, --user, start, gnmon-dashboard.service]
|
||||||
|
changed_when: false
|
||||||
|
when: gnmon_dashboard_scope == 'system'
|
||||||
|
|
||||||
|
- name: Enable and start the dashboard (user scope)
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: gnmon-dashboard.service
|
||||||
|
scope: user
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
|
when: gnmon_dashboard_scope == 'user'
|
||||||
|
|
||||||
|
# Quadlet-generated user units cannot be enabled; the [Install] section is
|
||||||
|
# only documentation there. Linger keeps the unit alive across sessions;
|
||||||
|
# verify it is on so the dashboard survives reboots.
|
||||||
|
- name: Verify linger is enabled for the workload user (system scope)
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "/var/lib/systemd/linger/{{ gnmon_dashboard_run_user }}"
|
||||||
|
register: gnmon_dashboard_linger
|
||||||
|
failed_when: not gnmon_dashboard_linger.stat.exists
|
||||||
|
when: gnmon_dashboard_scope == 'system'
|
||||||
+24
@@ -0,0 +1,24 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=GNMON Blazor dashboard (GNMON-001)
|
||||||
|
Documentation=https://learn.microsoft.com/aspnet/core/blazor/hosting-models
|
||||||
|
|
||||||
|
[Container]
|
||||||
|
ContainerName=gnmon-dashboard
|
||||||
|
Image={{ gnmon_dashboard_image_ref }}
|
||||||
|
# Host network: the app queries the host-loopback monitoring APIs
|
||||||
|
# (VictoriaMetrics 8428, Loki 3100, Alertmanager 9093).
|
||||||
|
Network=host
|
||||||
|
# Default rootless mapping (same as the other GNMON services): the image
|
||||||
|
# runs as its non-root user (984) inside the container, which maps onto
|
||||||
|
# the gn-dashboard subuid range on the host. keep-id was tested 06.11.2026
|
||||||
|
# and rejected: crun fails with Permission denied on the overlay merged
|
||||||
|
# dir for the gn-dashboard storage (works for gn-monitor; cause open).
|
||||||
|
Environment=ASPNETCORE_URLS=http://{{ gnmon_dashboard_http_listen }}
|
||||||
|
Environment=DOTNET_GENERATE_ASPNET_KEY=true
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=2s
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -60,6 +60,24 @@ groups:
|
|||||||
severity: critical
|
severity: critical
|
||||||
annotations:
|
annotations:
|
||||||
summary: node_exporter target disappeared from VictoriaMetrics
|
summary: node_exporter target disappeared from VictoriaMetrics
|
||||||
|
|
||||||
|
- name: gnmon-ai
|
||||||
|
interval: 30s
|
||||||
|
rules:
|
||||||
|
- alert: GnAiEnvironmentDown
|
||||||
|
# Textfile metric from the ai_environment_switch role (GNAI-001 M4):
|
||||||
|
# 0 when no sanctioned environment is state-active or its engine
|
||||||
|
# unit is down. Rule proposal accepted 06.11.2026 (GNMON side).
|
||||||
|
expr: gn_ai_active_env == 0
|
||||||
|
for: 2m
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
annotations:
|
||||||
|
summary: "AI environment {{ '{{' }} $labels.env {{ '}}' }} not serving"
|
||||||
|
description: >-
|
||||||
|
The state-active AI environment reports gn_ai_active_env == 0
|
||||||
|
for 2 minutes (engine unit down or environment state missing).
|
||||||
|
Check the environment switch state and recovery timer.
|
||||||
{% if gnmon_vmalert_selftest_rule %}
|
{% if gnmon_vmalert_selftest_rule %}
|
||||||
- name: gnmon-selftest
|
- name: gnmon-selftest
|
||||||
interval: 30s
|
interval: 30s
|
||||||
|
|||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# GNMON-001 section 6: Blazor Server dashboard image.
|
||||||
|
# Multi-stage build from the MCR .NET images (not Docker Hub, so server
|
||||||
|
# side rebuilds stay possible even under Docker Hub rate limits).
|
||||||
|
# Runs as a non-root user; the rootless quadlet maps it onto the
|
||||||
|
# gn-dashboard host account (UID 984) via UserNS=keep-id.
|
||||||
|
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
# Restore first so the layer cache survives source edits.
|
||||||
|
COPY Ghostnet.Dashboard.csproj .
|
||||||
|
RUN dotnet restore Ghostnet.Dashboard.csproj
|
||||||
|
|
||||||
|
COPY . .
|
||||||
|
RUN dotnet publish Ghostnet.Dashboard.csproj -c Release -o /app --no-restore
|
||||||
|
|
||||||
|
FROM mcr.microsoft.com/dotnet/aspnet:10.0
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=build /app .
|
||||||
|
|
||||||
|
# Non-root runtime user; keeps UID equal to the host workload account
|
||||||
|
# (gn-dashboard, 984) so host-side processes are attributable.
|
||||||
|
ENV ASPNETCORE_URLS=http://127.0.0.1:8080 \
|
||||||
|
DOTNET_GENERATE_ASPNET_KEY=true
|
||||||
|
USER 984:984
|
||||||
|
EXPOSE 8080
|
||||||
|
ENTRYPOINT ["dotnet", "Ghostnet.Dashboard.dll"]
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
bin/
|
||||||
|
obj/
|
||||||
|
*.user
|
||||||
Reference in New Issue
Block a user