GNMON-001 M2: Logpfad Loki + Alloy auf der Workstation implementiert

gnmon_loki: rootless Quadlet (3.6.19 digest-gepinnt, TSDB v13, 30d
Retention, User=0 wegen Rootless-UID-Mapping). gnmon_alloy: Host-Binary
v1.21.3 (SHA256SUMS-verifiziert) mit Journald- und Datei-Quellen,
loki.relabel-Label-Regeln. Verifikation: Dev-Logzeile in Loki
nachweisbar, Idempotenz changed=0. Offen: Journald-Ingestion braucht
Gruppenmitgliedschaft adm+systemd-journal (Operator-Schritt).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
2026-11-04 06:25:18 +01:00
parent aa8de57c62
commit 0a2a8b6863
12 changed files with 403 additions and 4 deletions
@@ -43,6 +43,32 @@ User-Scope-systemd) im Verzeichnis `llm-server/monitoring/`:
- Verifikation: PromQL-Abfrage liefert `gn_gpu_edge_temp_celsius` (51 °C
auf der Workstation-APU); Idempotenzlauf `changed=0, failed=0`.
### M2: Logpfad (implementiert 04.11.2026, Workstation-Entwicklung)
- `gnmon_loki`-Rolle: rootless Quadlet, `Network=host`, `User=0`
(Rootless-Mapping; das Upstream-Image läuft intern als UID 10001,
die im User-Namespace nicht gemappt ist), Loki **3.6.19** (Digest
`sha256:0f9cbc340cbe33ee498e654075034336962057db072f57b37f253e14d476443a`,
verifizierter Pull 04.11.2026), TSDB-Schema v13, Filesystem-Store,
Retention 30 d mit Compactor, Health-Check im Playbook.
- `gnmon_alloy`-Rolle: Alloy **v1.21.3** als Host-Binary (GNMON-001 §3;
im rootless Container wäre die Journal-Gruppenzuordnung gid 981 nicht
mappbar), User-Unit, SHA256SUMS-verifizierter Download, Journald-Quelle
mit `loki.relabel`-Regeln (unit/syslog_identifier/priority als Labels)
plus Datei-Quellen (Workstation: `dev-test.log`).
- River-Syntax-Befunde (Alloy 1.21): Listenelemente benötigen
Trailing-Commas; `relabel_rules` ist ein Attribut, das den `rules`-Export
einer `loki.relabel`-Komponente nimmt; Rule-Attribute sind
`source_labels`/`target_label` (Promtail-Stil).
- Verifikation: Dev-Logzeile erscheint in Loki
(`query_range` mit `source="dev-test"`), Idempotenzlauf
`changed=0, failed=0`.
- **Offener Operator-Schritt (Workstation):** `sudo usermod -aG
adm,systemd-journal ghost` — die Journald-Quelle läuft ohne Fehler,
sammelt aber ohne Gruppenmitgliedschaft keine Einträge (dokumentiertes
Alloy-Verhalten). Danach Journald-Ingestion in einer neuen Session
verifizieren.
## Noch offen
1. Punkte 1–6 aus GNMON-001 v2.0 Abschnitt 9 (Metrics-Account-Identität,
@@ -52,8 +78,8 @@ User-Scope-systemd) im Verzeichnis `llm-server/monitoring/`:
keinen Schwellwert-Alarm (Motivation des Vorfalls). Vorschlag:
`vmalert`-Regeln gegen VictoriaMetrics mit Notification-Kanal; Kanal-
Auswahl (Mail/ntfy/Matrix) ist eine Betreiberentscheidung.
3. M2 (Loki + Alloy Log-Pipeline) und M3 (Blazor-Dashboard) auf der
Workstation.
3. M3 (Blazor-Dashboard) und M4 (Alerting: vmalert + Alertmanager, E-Mail
an SMTP — Betreiberentscheidung 04.11.2026) auf der Workstation.
4. M5 Server-Rollout auf `ghostnet-ki` nach Behebung des GPU0-Kühldefekts
(GNAI-010 10.10): System-Scope-Rollen, Konten 983/984, nftables,
Abnahmekriterien aus GNMON-001 Abschnitt 10.
@@ -1,8 +1,8 @@
---
# GNMON-001 v2.0: host monitoring deployment.
# M1 (workstation): metrics path (collector + node_exporter + VictoriaMetrics).
# Tags: `metrics` for the M1 path; log path (M2) and dashboard (M3) follow.
- name: GNMON metrics path
# M2 (workstation): log path (Loki + Alloy). Dashboard (M3) follows.
- name: GNMON metrics and log paths
hosts: all
connection: local
roles:
@@ -10,6 +10,10 @@
tags: [collector]
- role: gnmon_victoria_metrics
tags: [metrics]
- role: gnmon_loki
tags: [logs]
- role: gnmon_alloy
tags: [logs]
post_tasks:
- name: Verify the GPU metrics are ingested
ansible.builtin.uri:
@@ -26,3 +30,32 @@
ansible.builtin.debug:
msg: "gn_gpu_edge_temp_celsius = {{ gnmon_query_result.json.data.result[0].value[1] }} °C"
tags: [metrics]
- name: Seed the development log source with a fresh line
ansible.builtin.shell: |
set -e
dev_log="{{ lookup('ansible.builtin.env', 'HOME') }}/.local/state/gnmon/dev-test.log"
mkdir -p "$(dirname "$dev_log")"
echo "gnmon-m2-verification $(date -u +%%Y-%%m-%%dT%%H:%%M:%%SZ)" >> "$dev_log"
args:
executable: /bin/bash
changed_when: false
tags: [logs]
- name: Verify the log line arrives in Loki
ansible.builtin.uri:
url: >-
http://127.0.0.1:{{ gnmon_loki_port }}/loki/api/v1/query_range?query={source="dev-test"}&limit=5&start={{
(now_epoch | int - 300) * 1000000000 }}&end={{ (now_epoch | int + 60) * 1000000000 }}
status_code: 200
vars:
now_epoch: "{{ ansible_facts['date_time']['epoch'] }}"
register: gnmon_loki_query
retries: 10
delay: 3
until: >-
gnmon_loki_query.json is defined
and gnmon_loki_query.json.data.result | length > 0
tags: [logs]
- name: Show the ingested log sample
ansible.builtin.debug:
msg: "Loki-Ergebnisstreams: {{ gnmon_loki_query.json.data.result | length }}"
tags: [logs]
@@ -0,0 +1,28 @@
# GNMON-001 v2.0 section 3: Grafana Alloy as host-side log shipper.
#
# Per the plan, Alloy runs as a host binary (not containerized) under the
# metrics account on the server (groups: systemd-journal + gn-media).
# M2 workstation scope: user unit under the current user; the journald
# source requires membership in the `systemd-journal` group (operator
# step on the workstation; documented in GNMON-000).
gnmon_alloy_version: "1.21.3"
gnmon_alloy_url: "https://github.com/grafana/alloy/releases/download"
gnmon_alloy_user_home: "{{ lookup('ansible.builtin.env', 'HOME') }}"
gnmon_alloy_binary_dir: "{{ gnmon_alloy_user_home }}/.local/bin"
gnmon_alloy_config_dir: "{{ gnmon_alloy_user_home }}/.config/gnmon"
gnmon_alloy_data_dir: "{{ gnmon_alloy_user_home }}/.local/share/gnmon/alloy-data"
gnmon_alloy_cache_dir: "{{ gnmon_alloy_user_home }}/.cache/gnmon"
gnmon_alloy_http_listen: "127.0.0.1:12345"
gnmon_alloy_loki_push_url: "http://127.0.0.1:3100/loki/api/v1/push"
# journald source (needs systemd-journal group membership).
gnmon_alloy_journal_enabled: true
gnmon_alloy_journal_max_age: "12h"
# Additional plain-file log sources (workstation development only).
gnmon_alloy_file_targets:
- path: "{{ gnmon_alloy_user_home }}/.local/state/gnmon/dev-test.log"
source: dev-test
@@ -0,0 +1,7 @@
# GNMON-001: Alloy role handlers (user scope, M2).
- name: Restart Alloy
ansible.builtin.systemd:
name: gnmon-alloy.service
scope: user
state: restarted
daemon_reload: true
@@ -0,0 +1,91 @@
# GNMON-001 v2.0 section 5: Grafana Alloy host binary (M2: user scope).
- name: Validate Alloy declarations
ansible.builtin.assert:
that:
- gnmon_alloy_user_home is match('^/')
- gnmon_alloy_version is match('^[0-9]+\\.[0-9]+\\.[0-9]+$')
fail_msg: Alloy declarations are incomplete or ambiguous.
- name: Create Alloy directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- "{{ gnmon_alloy_config_dir }}"
- "{{ gnmon_alloy_data_dir }}"
- "{{ gnmon_alloy_cache_dir }}"
- name: Fetch the Alloy release checksums
ansible.builtin.get_url:
url: "{{ gnmon_alloy_url }}/v{{ gnmon_alloy_version }}/SHA256SUMS"
dest: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}.SHA256SUMS"
mode: "0644"
- name: Record the pinned Alloy zip checksum
ansible.builtin.set_fact:
gnmon_alloy_zip_name: "alloy-linux-amd64.zip"
gnmon_alloy_zip_checksum: "{{ lookup('ansible.builtin.file',
gnmon_alloy_cache_dir ~ '/alloy-' ~ gnmon_alloy_version ~ '.SHA256SUMS')
| regex_findall('^([0-9a-f]{64})\\s+alloy-linux-amd64\\.zip$', multiline=True) | first }}"
- name: Download Alloy v{{ gnmon_alloy_version }}
ansible.builtin.get_url:
url: "{{ gnmon_alloy_url }}/v{{ gnmon_alloy_version }}/{{ gnmon_alloy_zip_name }}"
dest: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}-linux-amd64.zip"
checksum: "sha256:{{ gnmon_alloy_zip_checksum }}"
mode: "0644"
- name: Create the Alloy extract directory
ansible.builtin.file:
path: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}"
state: directory
mode: "0755"
- name: Extract Alloy
ansible.builtin.unarchive:
src: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}-linux-amd64.zip"
dest: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}"
remote_src: true
creates: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}/alloy-linux-amd64"
- name: Install the Alloy binary
ansible.builtin.copy:
src: "{{ gnmon_alloy_cache_dir }}/alloy-{{ gnmon_alloy_version }}/alloy-linux-amd64"
dest: "{{ gnmon_alloy_binary_dir }}/alloy"
mode: "0755"
notify: Restart Alloy
- name: Install the Alloy configuration
ansible.builtin.template:
src: alloy-config.alloy.j2
dest: "{{ gnmon_alloy_config_dir }}/alloy-config.alloy"
mode: "0644"
notify: Restart Alloy
- name: Install the Alloy service unit
ansible.builtin.template:
src: gnmon-alloy.service.j2
dest: "{{ gnmon_alloy_user_home }}/.config/systemd/user/gnmon-alloy.service"
mode: "0644"
notify: Restart Alloy
- name: Enable and start Alloy
ansible.builtin.systemd:
name: gnmon-alloy.service
scope: user
enabled: true
state: started
daemon_reload: true
- name: Flush handlers so config changes are applied before the readiness check
ansible.builtin.meta: flush_handlers
- name: Wait for the Alloy readiness endpoint
ansible.builtin.uri:
url: "http://{{ gnmon_alloy_http_listen }}/-/ready"
status_code: 200
register: gnmon_alloy_ready
retries: 30
delay: 2
until: gnmon_alloy_ready.status == 200
@@ -0,0 +1,55 @@
// GNMON-001 section 5: log pipeline.
// journald -> Loki (system + container logs), plus optional plain-file
// sources (workstation development). The write target is Loki's push API.
loki.write "gnmon" {
endpoint {
url = "{{ gnmon_alloy_loki_push_url }}"
}
}
{% if gnmon_alloy_journal_enabled %}
// Relabel journal fields into Loki labels before writing
// (documented pattern: pass the loki.relabel rules export to the
// journal source; internal __journal_* labels are dropped otherwise).
loki.relabel "gnmon_journal" {
forward_to = [loki.write.gnmon.receiver]
rule {
action = "replace"
source_labels = ["__journal_unit"]
target_label = "unit"
}
rule {
action = "replace"
source_labels = ["__journal_syslog_identifier"]
target_label = "syslog_identifier"
}
rule {
action = "replace"
source_labels = ["__journal_priority_keyword"]
target_label = "priority"
}
}
loki.source.journal "gnmon" {
max_age = "{{ gnmon_alloy_journal_max_age }}"
forward_to = [loki.relabel.gnmon_journal.receiver]
relabel_rules = loki.relabel.gnmon_journal.rules
}
{% endif %}
{% if gnmon_alloy_file_targets | length > 0 %}
local.file_match "gnmon_files" {
path_targets = [
{% for target in gnmon_alloy_file_targets %}
{ __path__ = "{{ target.path }}", source = "{{ target.source }}" },
{% endfor %}
]
}
loki.source.file "gnmon_files" {
targets = local.file_match.gnmon_files.targets
forward_to = [loki.write.gnmon.receiver]
}
{% endif %}
@@ -0,0 +1,11 @@
[Unit]
Description=GNMON Grafana Alloy log shipper (GNMON-001)
Documentation=https://grafana.com/docs/alloy/
[Service]
ExecStart={{ gnmon_alloy_binary_dir }}/alloy run --server.http.listen-addr={{ gnmon_alloy_http_listen }} --storage.path={{ gnmon_alloy_data_dir }} {{ gnmon_alloy_config_dir }}/alloy-config.alloy
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=default.target
@@ -0,0 +1,20 @@
# GNMON-001 v2.0 section 5: Loki as rootless quadlet (log aggregation,
# filesystem store, no S3). M2 scope: user-scope on the workstation.
gnmon_loki_image: docker.io/grafana/loki
gnmon_loki_image_tag: "3.6.19"
# Digest of the verified pull on the workstation (2026-11-04, M2).
gnmon_loki_image_digest: 0f9cbc340cbe33ee498e654075034336962057db072f57b37f253e14d476443a
gnmon_loki_user_home: "{{ lookup('ansible.builtin.env', 'HOME') }}"
gnmon_loki_quadlet_dir: "{{ gnmon_loki_user_home }}/.config/containers/systemd"
gnmon_loki_config_dir: "{{ gnmon_loki_user_home }}/.config/gnmon"
gnmon_loki_data_dir: "{{ gnmon_loki_user_home }}/.local/share/gnmon/loki-data"
gnmon_loki_port: 3100
gnmon_loki_retention_days: 30
gnmon_loki_image_ref: >-
{{ (gnmon_loki_image_digest is not none)
| ternary(gnmon_loki_image ~ '@sha256:' ~ gnmon_loki_image_digest,
gnmon_loki_image ~ ':' ~ gnmon_loki_image_tag) }}
@@ -0,0 +1,7 @@
# GNMON-001: Loki role handlers (user scope, M2).
- name: Restart Loki
ansible.builtin.systemd:
name: gnmon-loki.service
scope: user
state: restarted
daemon_reload: true
@@ -0,0 +1,63 @@
# GNMON-001 v2.0 section 5: Loki rootless quadlet (M2: user scope).
- name: Validate Loki declarations
ansible.builtin.assert:
that:
- gnmon_loki_user_home is match('^/')
- gnmon_loki_retention_days | int >= 1
fail_msg: Loki declarations are incomplete or ambiguous.
- name: Create Loki directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- "{{ gnmon_loki_quadlet_dir }}"
- "{{ gnmon_loki_config_dir }}"
- "{{ gnmon_loki_data_dir }}"
- "{{ gnmon_loki_data_dir }}/chunks"
- "{{ gnmon_loki_data_dir }}/rules"
- name: Install the Loki configuration
ansible.builtin.template:
src: loki-config.yaml.j2
dest: "{{ gnmon_loki_config_dir }}/loki-config.yaml"
mode: "0644"
notify: Restart Loki
- name: Install the Loki quadlet
ansible.builtin.template:
src: gnmon-loki.container.j2
dest: "{{ gnmon_loki_quadlet_dir }}/gnmon-loki.container"
mode: "0644"
notify: Restart Loki
- name: Enable and start Loki
ansible.builtin.systemd:
name: gnmon-loki.service
scope: user
enabled: true
state: started
daemon_reload: true
- name: Record the running image digest
ansible.builtin.command:
cmd: "podman inspect --format '{{ '{{' }}.ImageDigest{{ '}}' }}' gnmon-loki"
changed_when: false
register: gnmon_loki_inspect
failed_when: false
- name: Show the running image digest (digest pin)
ansible.builtin.debug:
msg: >-
gnmon-loki runs {{ gnmon_loki_image }}:{{ gnmon_loki_image_tag }}
(digest {{ gnmon_loki_inspect.stdout | default('n/a') }})
- name: Wait for the Loki readiness endpoint
ansible.builtin.uri:
url: "http://127.0.0.1:{{ gnmon_loki_port }}/ready"
status_code: 200
register: gnmon_loki_ready
retries: 30
delay: 2
until: gnmon_loki_ready.status == 200
@@ -0,0 +1,22 @@
[Unit]
Description=GNMON Loki (GNMON-001)
Documentation=https://grafana.com/docs/loki/
[Container]
ContainerName=gnmon-loki
Image={{ gnmon_loki_image_ref }}
# Rootless quadlet; API stays on the host loopback interface.
Network=host
# The upstream image defaults to uid 10001, which is not mapped in a
# rootless container. Run as the mapped root instead (= host user); the
# same pattern the VictoriaMetrics quadlet uses.
User=0
Exec=-config.file=/etc/loki/loki-config.yaml
Volume={{ gnmon_loki_config_dir }}/loki-config.yaml:/etc/loki/loki-config.yaml:ro,Z
Volume={{ gnmon_loki_data_dir }}:/loki:Z
[Service]
Restart=on-failure
[Install]
WantedBy=default.target
@@ -0,0 +1,36 @@
# GNMON-001 section 5: Loki single-instance configuration.
# Filesystem store (no S3), TSDB schema v13, 30d retention via compactor.
auth_enabled: false
server:
http_listen_port: {{ gnmon_loki_port }}
grpc_listen_port: 9096
common:
instance_addr: 127.0.0.1
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: "2026-01-01"
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: {{ gnmon_loki_retention_days * 24 }}h
compactor:
retention_enabled: true
delete_request_store: filesystem