Files
infrastructure/llm-server/monitoring/ansible/playbooks/monitor.yml
T
spektr fd8e1a5559 GNMON-001 M6 abgenommen: Dashboard oeffentlich via mon.ghostnet.selfhost.eu (Keycloak-OIDC)
Kette: Browser -> https://mon.ghostnet.selfhost.eu (NPM CT-103 ID 14, LE-TLS,
Force SSL, WS-Support, X-Forwarded-Proto/Host + 32k-Puffer) -> 10.0.0.90:8080
(nftables gnmon_firewall: nur von 10.0.0.10) -> Blazor v0.3.2 -> Keycloak
Realm Ghost.net, Client gnai-dashboard, Rollen gnai-dashboard-user/-admin.

Neu/geaendert:
- Rolle gnmon_firewall (nftables auf ghostnet-ki, Policy ACCEPT, 8080 nur
  von NPM, Monitoring-Ports non-loopback-drop), Tag firewall in monitor.yml
- Dashboard-Bind 10.0.0.90:8080 (Assertion: LAN-Bind nur mit Firewall-Tag);
  Image v0.3.2; Playbook-Verifikation akzeptiert 200/302 mit Forwarded-Headern
- App: ClientId gnai-dashboard, Auth:AllowedRoles konfigurierbar,
  KnownNetworks/KnownProxies per .Clear() (leere Initializer = Loopback-Default),
  GetClaimsFromUserInfoEndpoint=true (realm-roles-Mapper-Flags im Realm unset),
  SaveTokens=true (Logout: id_token_hint), Cookie AccessDeniedPath
  /access-denied + Page (sonst Redirect-Loop), csproj PrunePackageData=false +
  Containerfile ohne --no-restore (sonst fehlt blazor.web.js im Image ->
  Blazor-Circuit startet nicht)
- KeycloakRealmRolesClaimTransformation auf ClaimTypes.Role + [gnmon-auth]-Log
- CT-100: Adoption 09.11. (pct exec), Split-DNS mon -> 10.0.0.10 live;
  Befunde gefixt: dns_pihole_hosts_records in allen 3 Inventaren gespiegelt,
  FTLCONF_dns_hosts als eine ';'-gejointe Environment-Zeile (systemd: letzter
  Key gewinnt); gnmon_alloy doppelte Key korrigiert
- Doku: GNMON-000 M6 abgenommen (alle Befunde), GNMON-001 v2.2, GNAI-000
  Monitoring/DNS-Zeilen, GNAI-006 v3.1 (Proxy-Host 14), GNAI-012 Stand

E2E 09.11.2026: unauth -> 302 mit PAR request_uri; Direktzugriff 10.0.0.90:8080
gedroppt; blazor.web.js 200 text/javascript; Login ghost -> Dashboard 200 mit
Live-Daten, Unterseiten + WebSocket ok; testuser ohne Rollen -> saubere 403;
/logout ok; monitor.yml gesamt ok=84 changed=0. Server-Image-Store aufgeraeumt
(v0.3.2 + Rollback v0.2.0/v0.1.0).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-09 19:34:50 +01:00

160 lines
5.9 KiB
YAML

---
# GNMON-001 v2.0: host monitoring deployment.
# M1 (workstation): metrics path (collector + node_exporter + VictoriaMetrics).
# M2 (workstation): log path (Loki + Alloy). Dashboard (M3) follows.
- name: GNMON metrics and log paths
hosts: all
roles:
- role: gnmon_accounts
tags: [accounts, never]
- role: gnmon_firewall
tags: [firewall]
- role: gnmon_collector
tags: [collector]
- role: gnmon_victoria_metrics
tags: [metrics]
- role: gnmon_loki
tags: [logs]
- role: gnmon_alloy
tags: [logs]
- role: gnmon_alertmanager
tags: [alerting]
- role: gnmon_vmalert
tags: [alerting]
- role: gnmon_dashboard
tags: [dashboard]
post_tasks:
- name: Verify the GPU metrics are ingested
ansible.builtin.uri:
url: "http://127.0.0.1:{{ gnmon_victoria_metrics_port }}/api/v1/query?query=gn_gpu_edge_temp_celsius"
status_code: 200
register: gnmon_query_result
retries: 15
delay: 4
until: >-
gnmon_query_result.json is defined
and gnmon_query_result.json.data.result | length > 0
tags: [metrics]
- name: Show the ingested GPU temperature sample
ansible.builtin.debug:
msg: "gn_gpu_edge_temp_celsius = {{ gnmon_query_result.json.data.result[0].value[1] }} °C"
tags: [metrics]
- name: Seed the development log source with a fresh line
ansible.builtin.shell: |
set -e
dev_log="{{ lookup('ansible.builtin.env', 'HOME') }}/.local/state/gnmon/dev-test.log"
mkdir -p "$(dirname "$dev_log")"
echo "gnmon-m2-verification $(date -u +%%Y-%%m-%%dT%%H:%%M:%%SZ)" >> "$dev_log"
args:
executable: /bin/bash
changed_when: false
when: gnmon_alloy_file_targets | length > 0
tags: [logs]
- name: Seed the journal with a verification line
# Journal-based rebuild of the log-path verify: in the server (system)
# scope Alloy has no file targets, only the journal. Both scopes label
# the message via SYSLOG_IDENTIFIER (JSON pipeline).
ansible.builtin.command:
argv:
- /usr/bin/logger
- -t
- gnmon-verify
- "gnmon-m2-verification {{ ansible_date_time['iso8601'] }}"
changed_when: false
tags: [logs]
- name: Verify the log line arrives in Loki
ansible.builtin.uri:
url: >-
http://127.0.0.1:{{ gnmon_loki_port }}/loki/api/v1/query_range?query={syslog_identifier="gnmon-verify"}&limit=5&start={{
(now_epoch | int - 300) * 1000000000 }}&end={{ (now_epoch | int + 60) * 1000000000 }}
status_code: 200
vars:
now_epoch: "{{ ansible_facts['date_time']['epoch'] }}"
register: gnmon_loki_query
retries: 10
delay: 3
until: >-
gnmon_loki_query.json is defined
and gnmon_loki_query.json.data.result | length > 0
tags: [logs]
- name: Show the ingested log sample
ansible.builtin.debug:
msg: "Loki-Ergebnisstreams: {{ gnmon_loki_query.json.data.result | length }}"
tags: [logs]
- name: Wait until vmalert has evaluated its first cycle
ansible.builtin.uri:
url: "http://{{ gnmon_vmalert_http_listen }}/-/healthy"
status_code: 200
register: gnmon_vmalert_health
retries: 12
delay: 5
until: gnmon_vmalert_health.status == 200
tags: [alerting]
- name: Verify the self-test alert reaches Alertmanager (workstation)
ansible.builtin.uri:
url: "http://{{ gnmon_alertmanager_http_listen }}/api/v2/alerts"
status_code: 200
register: gnmon_am_alerts
retries: 15
delay: 8
until: >-
gnmon_am_alerts.json is defined
and gnmon_am_alerts.json | selectattr('labels.alertname', 'defined')
| selectattr('labels.alertname', 'equalto', 'GnmonSelfTest')
| list | length > 0
when: gnmon_vmalert_selftest_rule | bool
tags: [alerting]
- name: Verify Alertmanager answers (server, no self-test rule)
ansible.builtin.uri:
url: "http://{{ gnmon_alertmanager_http_listen }}/api/v2/alerts"
status_code: 200
when: not (gnmon_vmalert_selftest_rule | bool)
tags: [alerting]
- name: Show the alerting path verdict
ansible.builtin.debug:
msg: >-
vmalert -> Alertmanager funktioniert:
{{ gnmon_am_alerts.json | selectattr('labels.alertname', 'defined')
| map(attribute='labels.alertname') | list | join(', ') }}
when: gnmon_vmalert_selftest_rule | bool
tags: [alerting]
# M6: with Auth enabled, "/" answers 302 (OIDC challenge via PAR toward
# Keycloak) instead of 200; without auth it serves the Blazor shell (200).
# The forwarded headers simulate the NPM ingress so the OIDC redirect
# URI resolves to the external host registered in Keycloak.
- name: Wait for the dashboard to answer
ansible.builtin.uri:
url: "http://{{ gnmon_dashboard_http_listen }}/"
status_code: [200, 302]
headers:
X-Forwarded-Proto: https
X-Forwarded-Host: mon.ghostnet.selfhost.eu
register: gnmon_dashboard_ready
retries: 20
delay: 3
until: gnmon_dashboard_ready.status in [200, 302]
tags: [dashboard]
- name: Verify the dashboard answers (200 shell or 302 OIDC challenge)
ansible.builtin.uri:
url: "http://{{ gnmon_dashboard_http_listen }}/"
status_code: [200, 302]
headers:
X-Forwarded-Proto: https
X-Forwarded-Host: mon.ghostnet.selfhost.eu
return_content: false
register: gnmon_dashboard_shell
until: gnmon_dashboard_shell.status in [200, 302]
retries: 3
delay: 2
tags: [dashboard]
- name: Show the dashboard verdict
ansible.builtin.debug:
msg: "Dashboard antwortet auf http://{{ gnmon_dashboard_http_listen }}/"
tags: [dashboard]