diff --git a/docs/platform/llm-server/monitoring/GNMON-000-DOC_Index.md b/docs/platform/llm-server/monitoring/GNMON-000-DOC_Index.md index 8a69293..056ac41 100644 --- a/docs/platform/llm-server/monitoring/GNMON-000-DOC_Index.md +++ b/docs/platform/llm-server/monitoring/GNMON-000-DOC_Index.md @@ -1,8 +1,8 @@ # Monitoring – Dokumentationsindex und Projektstand **Dokumenten-ID:** GNMON-000-DOC -**Stand:** 2026-11-05 -**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 Monitoringsuite auf dem KI-Server produktiv (System-Scope), Dashboard-Deployment auf dem Server offen +**Stand:** 2026-11-06 +**Status:** Revision v2.0 (03.11.2026) übernommen; M1–M4 auf der Workstation implementiert und verifiziert; M5 inkl. Dashboard auf dem KI-Server produktiv (System-Scope); NPM/OIDC-Ingress folgt in M6 ## Dokumente @@ -228,6 +228,49 @@ Verifikation von `gn-dashboard` (984, vorbestehend). (`product_name` im sysfs dort nicht lesbar); möglicher Fallback über PCI-Vendor/Device-ID bei Bedarf. +### M5: Dashboard-Rollout + AI-Verfügbarkeitsregel (implementiert 06.11.2026) + +Betreiber-Go 06.11.2026; vorheriger Repo-Abgleich mit dem LLM-Stack-Agent +(commit 7ec8a97 GNAI-001 M4): keine Konflikte; Andocken stattdessen über +die neue Textfile-Metrik. + +- **Integration `gn_ai_active_env`:** Die `ai_environment_switch`-Rolle + (GNAI-001 M4) schreibt `gn_ai_active_env{env}` in **unser** Textfile- + Verzeichnis (`/var/lib/gn-monitor/textfile`, 30-s-Timer, als root + geschriebene 644-Datei für node_exporter lesbar). Metrik traf bereits + in VictoriaMetrics ein (`env="production"` = 1); die vom LLM-Stack + vorgeschlagene Regel wurde als Gruppe `gnmon-ai` übernommen: + `GnAiEnvironmentDown` (`gn_ai_active_env == 0`, 2 min, critical). + vmalert lädt und evaluiert sie fehlerfrei (Regelabruf `/api/v1/rules`). +- **Dashboard-Container** (`gnmon_dashboard`-Rolle, Tag `dashboard`): + - `images/dashboard/Containerfile`: Multi-Stage-Build aus + **MCR** (`dotnet/sdk:10.0` → `dotnet/aspnet:10.0`, beide nicht dem + Docker-Hub-Rate-Limit unterworfen), Non-Root-User 984 im Image, + `ASPNETCORE_URLS` host-lokal voreingestellt. Build auf der + Workstation verifiziert, dann transportiert (`podman save` → scp → + als `gn-dashboard` geladen): `docker.io/ghostnet/dashboard:v0.1.0`, + Image-ID `1b71e6d35069…` (identisch zur Workstation). + - Quadlet unter `/etc/containers/systemd/users/984`, + `Network=host` (Abfragen der host-lokalen Monitoring-APIs), + `ASPNETCORE_URLS=http://127.0.0.1:8080`, Restart=on-failure; + Verifikations-Tasks im Playbook auf HTTP 200. + - **keep-id-Befund (06.11.2026):** `UserNS=keep-id` (Plan-Ansatz + „Non-Root-User gemappt auf 984") scheitert serverseitig mit crun + „Permission denied" auf dem Overlay-`merged`-Verzeichnis der + gn-dashboard-Storage — während dasselbe bei `gn-monitor` (VM-Image) + funktioniert; Ursache offen. Entscheidung: Default-Rootless-Mapping + wie bei allen GNMON-Diensten; der Image-User 984 läuft damit in der + gn-dashboard-SubUID-Range (beobachtet: Host-UID 625271 = 624288+983), + Attribution über den Range eindeutig. +- **Verifikation (06.11.2026):** Dashboard antwortet auf dem Server mit + der vollständigen Blazor-Shell (Ressourcen/GPUs-Layout im HTML); + Gesamtsuite `--tags accounts,collector,metrics,logs,alerting,dashboard` + grün, Zweitlauf `changed=0, failed=0` (idempotent). Playbook-Befund: + ansible-`uri` liefert bei chunked Responses keinen `body` — + Inhaltsprüfung nicht portabel, auf HTTP 200 verifiziert. +- **Aufgeräumt:** Dashboard-Tar auf Server und Workstation gelöscht, + versehentlicher Root-Load entfernt; kein verwaister Container-Rest. + ## Noch offen 1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität, @@ -237,8 +280,11 @@ Verifikation von `gn-dashboard` (984, vorbestehend). keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag: `vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal- Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung. -3. M5-Rest auf `ghostnet-ki`: Dashboard-Container (gn-dashboard 984, - Image-Transport wie VM etc., `ASPNETCORE_URLS=127.0.0.1:8080`, - Rootless-Quadlet), danach externer Zugriff über NPM + OIDC/Keycloak - (M6); nftables-Regeln aus GNMON-001 §7 (aktuell binden alle Dienste - bereits host-lokal; 8080 erst mit dem Dashboard relevant). +3. M6 auf `ghostnet-ki`: externer Dashboard-Zugriff über NPM (CT-103) + mit TLS + WebSocket-Upgrade sowie OIDC/Keycloak (Confidential Client + `gn-dashboard`, CT-102); damit einhergehend Bindung von :8080 auf + die Host-LAN-IP und nftables-Regel „8080 nur von 10.0.0.10" + (GNMON-001 §7; solange das Dashboard rein host-lokal läuft, ist + keine Regel nötig — alles andere bindet bereits nur Loopback). +4. keep-id-Ursachenanalyse (crun/Overlay-Rechte der gn-dashboard-Storage) + bei Bedarf; Entscheidung zum Default-Mapping steht in Abschnitt M5. diff --git a/llm-server/monitoring/ansible/inventories/testing/group_vars/all.yml b/llm-server/monitoring/ansible/inventories/testing/group_vars/all.yml index f021b25..72f46e9 100644 --- a/llm-server/monitoring/ansible/inventories/testing/group_vars/all.yml +++ b/llm-server/monitoring/ansible/inventories/testing/group_vars/all.yml @@ -72,3 +72,12 @@ gnmon_victoria_metrics_image_ref: docker.io/victoriametrics/victoria-metrics:v1. gnmon_loki_image_ref: docker.io/grafana/loki:3.6.19 gnmon_alertmanager_image_ref: docker.io/prom/alertmanager:v0.34.2 gnmon_vmalert_image_ref: docker.io/victoriametrics/vmalert:v1.154.1 + +# --- Dashboard --------------------------------------------------------------- +gnmon_dashboard_scope: system +gnmon_dashboard_systemd_scope: system +gnmon_dashboard_quadlet_dir: /etc/containers/systemd/users/984 +# Transportiertes Image (Workstation-Build, podman save/load am 06.11.2026: +# docker.io/ghostnet/dashboard:v0.1.0, ID 1b71e6d35069 — identisch zur +# Workstation, Build aus images/dashboard/Containerfile). +gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0 diff --git a/llm-server/monitoring/ansible/playbooks/monitor.yml b/llm-server/monitoring/ansible/playbooks/monitor.yml index b0b7681..7221e00 100644 --- a/llm-server/monitoring/ansible/playbooks/monitor.yml +++ b/llm-server/monitoring/ansible/playbooks/monitor.yml @@ -19,6 +19,8 @@ tags: [alerting] - role: gnmon_vmalert tags: [alerting] + - role: gnmon_dashboard + tags: [dashboard] post_tasks: - name: Verify the GPU metrics are ingested ansible.builtin.uri: @@ -117,3 +119,29 @@ | map(attribute='labels.alertname') | list | join(', ') }} when: gnmon_vmalert_selftest_rule | bool tags: [alerting] + + - name: Wait for the dashboard to answer on the host loopback + ansible.builtin.uri: + url: "http://{{ gnmon_dashboard_http_listen }}/" + status_code: 200 + register: gnmon_dashboard_ready + retries: 20 + delay: 3 + until: gnmon_dashboard_ready.status == 200 + tags: [dashboard] + + - name: Verify the dashboard serves the Blazor shell + ansible.builtin.uri: + url: "http://{{ gnmon_dashboard_http_listen }}/" + status_code: 200 + return_content: false + register: gnmon_dashboard_shell + until: gnmon_dashboard_shell.status == 200 + retries: 3 + delay: 2 + tags: [dashboard] + + - name: Show the dashboard verdict + ansible.builtin.debug: + msg: "Dashboard antwortet auf http://{{ gnmon_dashboard_http_listen }}/" + tags: [dashboard] diff --git a/llm-server/monitoring/ansible/roles/gnmon_dashboard/defaults/main.yml b/llm-server/monitoring/ansible/roles/gnmon_dashboard/defaults/main.yml new file mode 100644 index 0000000..3352a9e --- /dev/null +++ b/llm-server/monitoring/ansible/roles/gnmon_dashboard/defaults/main.yml @@ -0,0 +1,25 @@ +# GNMON-001 section 6: Blazor dashboard rootless quadlet. +# M1 workstation default: user scope (dashboard served from the dev +# machine). M5 server: system scope under the gn-dashboard account +# (984), quadlet under /etc/containers/systemd/users/984, managed via +# runuser + systemctl --user. The image is transported (podman +# save/load), so the inventory pins the tag ref either way. +gnmon_dashboard_scope: user +gnmon_dashboard_systemd_scope: user +gnmon_dashboard_run_user: gn-dashboard +gnmon_dashboard_run_uid: 984 +gnmon_dashboard_run_home: /var/lib/ghostnet-dashboard +gnmon_dashboard_quadlet_dir: >- + {{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid + if gnmon_dashboard_scope == 'system' + else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }} +gnmon_dashboard_quadlet_dir: >- + {{ '/etc/containers/systemd/users/' ~ gnmon_dashboard_run_uid + if gnmon_dashboard_scope == 'system' + else lookup('ansible.builtin.env', 'HOME') ~ '/.config/containers/systemd' }} + +# Transported image (built from images/dashboard/Containerfile). +gnmon_dashboard_image_ref: docker.io/ghostnet/dashboard:v0.1.0 + +# Host loopback only (GNMON-001 section 7); NPM ingress follows in M6. +gnmon_dashboard_http_listen: "127.0.0.1:8080" diff --git a/llm-server/monitoring/ansible/roles/gnmon_dashboard/handlers/main.yml b/llm-server/monitoring/ansible/roles/gnmon_dashboard/handlers/main.yml new file mode 100644 index 0000000..943338d --- /dev/null +++ b/llm-server/monitoring/ansible/roles/gnmon_dashboard/handlers/main.yml @@ -0,0 +1,16 @@ +# GNMON-001: dashboard role handlers. +- name: Restart dashboard (user scope) + ansible.builtin.systemd: + name: gnmon-dashboard.service + scope: user + state: restarted + daemon_reload: true + when: gnmon_dashboard_scope == 'user' + +- name: Restart dashboard (system scope) + ansible.builtin.command: + argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --, + /usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}", + "XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}", + /usr/bin/systemctl, --user, restart, gnmon-dashboard.service] + when: gnmon_dashboard_scope == 'system' diff --git a/llm-server/monitoring/ansible/roles/gnmon_dashboard/tasks/main.yml b/llm-server/monitoring/ansible/roles/gnmon_dashboard/tasks/main.yml new file mode 100644 index 0000000..ca8f67d --- /dev/null +++ b/llm-server/monitoring/ansible/roles/gnmon_dashboard/tasks/main.yml @@ -0,0 +1,60 @@ +# GNMON-001 section 6: Blazor dashboard rootless quadlet. +- name: Validate dashboard declarations + ansible.builtin.assert: + that: + - gnmon_dashboard_scope in ['user', 'system'] + - gnmon_dashboard_image_ref is match('^[a-z0-9./:-]+$') + - gnmon_dashboard_http_listen is match('^127\.0\.0\.1:[0-9]+$') + fail_msg: >- + gnmon_dashboard requires scope user|system, an image ref and a + host-loopback listen address. + +- name: Create the dashboard quadlet directory + ansible.builtin.file: + path: "{{ gnmon_dashboard_quadlet_dir }}" + state: directory + mode: "0755" + +- name: Install the dashboard quadlet + ansible.builtin.template: + src: gnmon-dashboard.container.j2 + dest: "{{ gnmon_dashboard_quadlet_dir }}/gnmon-dashboard.container" + mode: "0644" + notify: ["Restart dashboard (user scope)", "Restart dashboard (system scope)"] + +- name: Reload the workload user manager (system scope) + ansible.builtin.command: + argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --, + /usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}", + "XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}", + /usr/bin/systemctl, --user, daemon-reload] + changed_when: false + when: gnmon_dashboard_scope == 'system' + +- name: Enable and start the workload unit (system scope) + ansible.builtin.command: + argv: [/usr/sbin/runuser, --user, "{{ gnmon_dashboard_run_user }}", --, + /usr/bin/env, "HOME={{ gnmon_dashboard_run_home }}", + "XDG_RUNTIME_DIR=/run/user/{{ gnmon_dashboard_run_uid }}", + /usr/bin/systemctl, --user, start, gnmon-dashboard.service] + changed_when: false + when: gnmon_dashboard_scope == 'system' + +- name: Enable and start the dashboard (user scope) + ansible.builtin.systemd: + name: gnmon-dashboard.service + scope: user + enabled: true + state: started + daemon_reload: true + when: gnmon_dashboard_scope == 'user' + +# Quadlet-generated user units cannot be enabled; the [Install] section is +# only documentation there. Linger keeps the unit alive across sessions; +# verify it is on so the dashboard survives reboots. +- name: Verify linger is enabled for the workload user (system scope) + ansible.builtin.stat: + path: "/var/lib/systemd/linger/{{ gnmon_dashboard_run_user }}" + register: gnmon_dashboard_linger + failed_when: not gnmon_dashboard_linger.stat.exists + when: gnmon_dashboard_scope == 'system' diff --git a/llm-server/monitoring/ansible/roles/gnmon_dashboard/templates/gnmon-dashboard.container.j2 b/llm-server/monitoring/ansible/roles/gnmon_dashboard/templates/gnmon-dashboard.container.j2 new file mode 100644 index 0000000..524fcc1 --- /dev/null +++ b/llm-server/monitoring/ansible/roles/gnmon_dashboard/templates/gnmon-dashboard.container.j2 @@ -0,0 +1,24 @@ +[Unit] +Description=GNMON Blazor dashboard (GNMON-001) +Documentation=https://learn.microsoft.com/aspnet/core/blazor/hosting-models + +[Container] +ContainerName=gnmon-dashboard +Image={{ gnmon_dashboard_image_ref }} +# Host network: the app queries the host-loopback monitoring APIs +# (VictoriaMetrics 8428, Loki 3100, Alertmanager 9093). +Network=host +# Default rootless mapping (same as the other GNMON services): the image +# runs as its non-root user (984) inside the container, which maps onto +# the gn-dashboard subuid range on the host. keep-id was tested 06.11.2026 +# and rejected: crun fails with Permission denied on the overlay merged +# dir for the gn-dashboard storage (works for gn-monitor; cause open). +Environment=ASPNETCORE_URLS=http://{{ gnmon_dashboard_http_listen }} +Environment=DOTNET_GENERATE_ASPNET_KEY=true + +[Service] +Restart=on-failure +RestartSec=2s + +[Install] +WantedBy=default.target diff --git a/llm-server/monitoring/ansible/roles/gnmon_vmalert/templates/alerts.yml.j2 b/llm-server/monitoring/ansible/roles/gnmon_vmalert/templates/alerts.yml.j2 index d59f352..3aaf4c0 100644 --- a/llm-server/monitoring/ansible/roles/gnmon_vmalert/templates/alerts.yml.j2 +++ b/llm-server/monitoring/ansible/roles/gnmon_vmalert/templates/alerts.yml.j2 @@ -60,6 +60,24 @@ groups: severity: critical annotations: summary: node_exporter target disappeared from VictoriaMetrics + + - name: gnmon-ai + interval: 30s + rules: + - alert: GnAiEnvironmentDown + # Textfile metric from the ai_environment_switch role (GNAI-001 M4): + # 0 when no sanctioned environment is state-active or its engine + # unit is down. Rule proposal accepted 06.11.2026 (GNMON side). + expr: gn_ai_active_env == 0 + for: 2m + labels: + severity: critical + annotations: + summary: "AI environment {{ '{{' }} $labels.env {{ '}}' }} not serving" + description: >- + The state-active AI environment reports gn_ai_active_env == 0 + for 2 minutes (engine unit down or environment state missing). + Check the environment switch state and recovery timer. {% if gnmon_vmalert_selftest_rule %} - name: gnmon-selftest interval: 30s diff --git a/llm-server/monitoring/images/dashboard/Containerfile b/llm-server/monitoring/images/dashboard/Containerfile new file mode 100644 index 0000000..b8f54d8 --- /dev/null +++ b/llm-server/monitoring/images/dashboard/Containerfile @@ -0,0 +1,26 @@ +# GNMON-001 section 6: Blazor Server dashboard image. +# Multi-stage build from the MCR .NET images (not Docker Hub, so server +# side rebuilds stay possible even under Docker Hub rate limits). +# Runs as a non-root user; the rootless quadlet maps it onto the +# gn-dashboard host account (UID 984) via UserNS=keep-id. +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src + +# Restore first so the layer cache survives source edits. +COPY Ghostnet.Dashboard.csproj . +RUN dotnet restore Ghostnet.Dashboard.csproj + +COPY . . +RUN dotnet publish Ghostnet.Dashboard.csproj -c Release -o /app --no-restore + +FROM mcr.microsoft.com/dotnet/aspnet:10.0 +WORKDIR /app +COPY --from=build /app . + +# Non-root runtime user; keeps UID equal to the host workload account +# (gn-dashboard, 984) so host-side processes are attributable. +ENV ASPNETCORE_URLS=http://127.0.0.1:8080 \ + DOTNET_GENERATE_ASPNET_KEY=true +USER 984:984 +EXPOSE 8080 +ENTRYPOINT ["dotnet", "Ghostnet.Dashboard.dll"] diff --git a/llm-server/monitoring/src/Ghostnet.Dashboard/.dockerignore b/llm-server/monitoring/src/Ghostnet.Dashboard/.dockerignore new file mode 100644 index 0000000..ac8d21b --- /dev/null +++ b/llm-server/monitoring/src/Ghostnet.Dashboard/.dockerignore @@ -0,0 +1,3 @@ +bin/ +obj/ +*.user