GNAI-001 M4 abgeschlossen: Backup/Reboot/Alarmgrenzen (GNAI-008 14.6)
Rolle ai_backup: woechentlicher Timer, NFS-Ziel (Encoding-Pipeline- Anbindung, Betreiberentscheid), Archiv ohne GGUF-Weights (SOURCE.yml- Reproduzierbarkeit), SHA256-Sidecar, Retention 8; Restore-Nachweise R1 (Baum-Diff) und R2 (environment-state restore) bestanden. Echter Hostreboot: Recovery-Service bringt Production selbststaendig hoch. Verfuegbarkeitsmetrik gn_ai_active_env (Textfile, 30 s) mit Regel- vorschlag fuer die GNMON-Suite. M4 damit vollstaendig abgenommen. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
@@ -963,7 +963,40 @@ Abnahme am 06.11.2026, alle bestanden:
|
|||||||
|
|
||||||
Offen aus M4: realer Hostreboot-Test (mit dem Betreiber, dedizierte
|
Offen aus M4: realer Hostreboot-Test (mit dem Betreiber, dedizierte
|
||||||
Hardware), Backup-/Restore-SOP mit praktischem Nachweis sowie die
|
Hardware), Backup-/Restore-SOP mit praktischem Nachweis sowie die
|
||||||
Prüfung der Alarmgrenzen für Production im Monitoring.
|
Prüfung der Alarmgrenzen für Production im Monitoring — **alle drei
|
||||||
|
Folgepunkte wurden am selben Tag nachgezogen:**
|
||||||
|
|
||||||
|
- a) **Echter Hostreboot (06.11.2026, 07:21):** Nach dem Betreiber-Go
|
||||||
|
startete der Server neu; der Recovery-Service brachte Production ohne
|
||||||
|
jedes Zutun hoch (`active=production`, Engine mit GPU-Prozess,
|
||||||
|
Inferenzstichprobe 17×23 korrekt, Thinking intakt, Lüfterservice
|
||||||
|
aktiv). Damit ist der Wiederanlaufpfad für Boot, Abbruch und
|
||||||
|
Netzwerkverlust belegt.
|
||||||
|
- b) **Backup-/Restore (Rolle `ai_backup`, Betreiberentscheid: Ziel ist
|
||||||
|
die bestehende NFS-Anbindung der Encoding-Pipeline, pve-Datenhdd):**
|
||||||
|
wöchentlicher Timer (So 03:30, Persistent), Archiv mit
|
||||||
|
`/etc/ghostnet-ai`, Quadlet-Verzeichnissen der Konten,
|
||||||
|
`environment-state`, Benchmark-Rohdaten und Model-Store-Metadaten
|
||||||
|
ohne `*.gguf` (Weights bleiben über SOURCE.yml-Pins reproduzierbar),
|
||||||
|
SHA256-Sidecar, Integritätsprüfung, Retention 8. Praktische Nachweise
|
||||||
|
R1 (Extrakt und vollständiger Baum-Diff gegen das Live-System,
|
||||||
|
identisch) und R2 (Löschen und Restore von `environment-state`,
|
||||||
|
byte-identisch) bestanden; SOP im Rollen-README.
|
||||||
|
- c) **Monitoring/Alarmgrenzen:** Die GNMON-M5-Suite auf dem Server
|
||||||
|
(vmalert/Alertmanager rootless unter `gn-monitor`) alarmiert bereits
|
||||||
|
beide Vorfall-Signaturen (Junction >95 °C über 2 Min; PWM-Anforderung
|
||||||
|
bei 0 RPM über 3 Min) plus Edge- und VRAM-Grenzen und
|
||||||
|
Node-Exporter-Ausfall. Ergänzt wurde die Verfügbarkeitsmetrik
|
||||||
|
`gn_ai_active_env{env}` (Textfile-Collector der Switch-Rolle, 30-Sek.-
|
||||||
|
Timer, verifiziert für läuft=1/gestoppt=0/nach-Recovery=1);
|
||||||
|
Regelvorschlag an die Suite: `gn_ai_active_env == 0` über 2 Minuten,
|
||||||
|
kritisch.
|
||||||
|
|
||||||
|
**Damit ist Meilenstein 4 (GNAI-001) vollständig abgenommen:** Der
|
||||||
|
ausgewählte interne Dienst läuft als Production-Umgebung, ein
|
||||||
|
abgebrochener Testlauf wird hostseitig ohne Production-Secrets
|
||||||
|
rückgerollt (Timer/Service), Backup/Restore sind praktisch belegt und
|
||||||
|
Alarmgrenzen stehen.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
# Deploys the AI configuration backup (GNAI-001 milestone 4). Backup
|
||||||
|
# execution is the oneshot service; the timer schedules it weekly.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ansible-playbook -i inventories/<any-env> playbooks/backup.yml \
|
||||||
|
# --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
||||||
|
- name: Deploy the AI configuration backup
|
||||||
|
hosts: llm_servers
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: ai_backup
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# ai_backup
|
||||||
|
|
||||||
|
Configuration backup for the AI server (GNAI-001 milestone 4). Deploys
|
||||||
|
`gnai-ai-backup` (script), a oneshot service and a weekly timer
|
||||||
|
(Sunday 03:30, `Persistent=true`). Target is the existing NFS pipeline
|
||||||
|
share (`pve:/media/datenhdd3/Encoding`, operator decision 06.11.2026).
|
||||||
|
|
||||||
|
## Backup scope
|
||||||
|
|
||||||
|
- `/etc/ghostnet-ai` (all environment config trees)
|
||||||
|
- `/etc/containers/systemd/users/{980,981,984}` (rootless quadlets,
|
||||||
|
included when present)
|
||||||
|
- `/var/lib/ghostnet-ai/environment-state` (switch state)
|
||||||
|
- `/var/lib/ghostnet-ai/testing/bench/results` (benchmark artifacts)
|
||||||
|
- `/var/lib/ghostnet-ai/shared/models` **without `*.gguf`** — the pinned
|
||||||
|
`SOURCE.yml` manifests make the weights reproducible through the
|
||||||
|
model_store role (`--tags models` re-downloads and verifies them)
|
||||||
|
|
||||||
|
Every archive gets a `.sha256` sidecar; the archive is integrity-checked
|
||||||
|
before it replaces the last good file; retention keeps the newest
|
||||||
|
`ai_backup_retention` archives.
|
||||||
|
|
||||||
|
## Operations
|
||||||
|
|
||||||
|
- Run a backup now: `systemctl start gnai-ai-backup.service`
|
||||||
|
(the timer does this weekly; `systemctl list-timers gnai-ai-backup*`
|
||||||
|
shows the next run).
|
||||||
|
- Watch: `journalctl -u gnai-ai-backup.service -n 20`.
|
||||||
|
|
||||||
|
## Restore SOP
|
||||||
|
|
||||||
|
1. Identify the archive on the share:
|
||||||
|
`ls -lt /var/lib/ghostnet-ai-backup/ghostnet-ai-backup-*.tar.gz`
|
||||||
|
2. Verify the sidecar: `sha256sum -c <archive>.sha256`
|
||||||
|
3. Extract to a staging directory first (never extract blindly over
|
||||||
|
live files):
|
||||||
|
`mkdir /tmp/gnai-restore && tar -xzf <archive> -C /tmp/gnai-restore`
|
||||||
|
4. Compare or selectively copy back the affected tree, e.g.
|
||||||
|
`diff -r /tmp/gnai-restore/etc/ghostnet-ai/testing /etc/ghostnet-ai/testing`
|
||||||
|
5. For environment-state restores, run
|
||||||
|
`/usr/local/sbin/gnai-environment-switch recover` afterwards so the
|
||||||
|
runtime converges with the restored state.
|
||||||
|
6. For model weights: restore via the model_store role
|
||||||
|
(`--tags models`), then verify the `SOURCE.yml` checksums.
|
||||||
|
7. Re-run the affected environment playbook
|
||||||
|
(`playbooks/ai.yml --tags service`) and verify health.
|
||||||
|
|
||||||
|
Accepted proof (06.11.2026): full-tree extract-and-diff against the
|
||||||
|
live configuration and a practical restore of the deleted environment
|
||||||
|
state file, documented in GNAI-008 section 14.6.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
---
|
||||||
|
# Configuration backup for the AI server (GNAI-001 milestone 4:
|
||||||
|
# "Backupumfang, Ziel, Aufbewahrung und Restore-SOP festlegen").
|
||||||
|
#
|
||||||
|
# Target: the existing NFS pipeline share (operator decision 06.11.2026:
|
||||||
|
# reuse the encoding queue connection, pve:/media/datenhdd3/Encoding),
|
||||||
|
# mounted at /var/lib/ghostnet-pipeline. Model weights are NOT archived;
|
||||||
|
# they stay reproducible through the pinned SOURCE.yml manifests included
|
||||||
|
# in every archive plus re-verification after download.
|
||||||
|
|
||||||
|
ai_backup_nfs_mountpoint: /var/lib/ghostnet-pipeline
|
||||||
|
ai_backup_target_dir: /var/lib/ghostnet-ai-backup
|
||||||
|
|
||||||
|
# Retention: number of weekly archives kept on the share.
|
||||||
|
ai_backup_retention: 8
|
||||||
|
|
||||||
|
# Schedule: weekly, Sunday 03:30, catching up after downtime.
|
||||||
|
ai_backup_on_calendar: "Sun *-*-* 03:30:00"
|
||||||
|
ai_backup_persistent: true
|
||||||
|
|
||||||
|
ai_backup_service: gnai-ai-backup
|
||||||
|
ai_backup_script: /usr/local/sbin/gnai-ai-backup
|
||||||
|
|
||||||
|
# Backed-up trees (tar paths relative to /). Quadlet dirs are added by
|
||||||
|
# the script only when they exist.
|
||||||
|
ai_backup_paths:
|
||||||
|
- etc/ghostnet-ai
|
||||||
|
- etc/containers/systemd/users/980
|
||||||
|
- etc/containers/systemd/users/981
|
||||||
|
- etc/containers/systemd/users/984
|
||||||
|
- var/lib/ghostnet-ai/environment-state
|
||||||
|
- var/lib/ghostnet-ai/testing/bench/results
|
||||||
|
|
||||||
|
# Included without its multi-gigabyte weights: model manifests and any
|
||||||
|
# other metadata below the shared model store.
|
||||||
|
ai_backup_model_store_meta: var/lib/ghostnet-ai/shared/models
|
||||||
|
ai_backup_model_store_exclude: --exclude=*.gguf
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
---
|
||||||
|
- name: Verify the backup script syntax
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- /bin/bash
|
||||||
|
- -n
|
||||||
|
- "{{ ai_backup_script }}"
|
||||||
|
register: ai_backup_syntax
|
||||||
|
changed_when: false
|
||||||
|
failed_when: ai_backup_syntax.rc != 0
|
||||||
|
when: not ansible_check_mode
|
||||||
|
|
||||||
|
- name: Reload the systemd daemon
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
daemon_reload: true
|
||||||
|
when: not ansible_check_mode
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
---
|
||||||
|
# Deploys the AI configuration backup (GNAI-001 milestone 4): script,
|
||||||
|
# system service and weekly timer targeting the NFS pipeline share.
|
||||||
|
# Running the backup itself is a one-shot service invocation, not part
|
||||||
|
# of the role (see the role README for backup and restore procedures).
|
||||||
|
|
||||||
|
- name: Verify the backup declarations
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- ai_backup_nfs_mountpoint is match("^/")
|
||||||
|
- ai_backup_target_dir is match("^/")
|
||||||
|
- ai_backup_retention | int >= 1
|
||||||
|
- ai_backup_paths | length > 0
|
||||||
|
- ai_backup_service | length > 0
|
||||||
|
- ai_backup_script is match("^/usr/local/sbin/")
|
||||||
|
fail_msg: AI backup declarations are incomplete or invalid.
|
||||||
|
|
||||||
|
- name: Require the NFS pipeline share to be mounted
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- /usr/bin/mountpoint
|
||||||
|
- -q
|
||||||
|
- "{{ ai_backup_nfs_mountpoint }}"
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
register: ai_backup_share_mounted
|
||||||
|
|
||||||
|
- name: Verify the NFS pipeline share is mounted
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- ai_backup_share_mounted.rc == 0
|
||||||
|
fail_msg: >-
|
||||||
|
The NFS share {{ ai_backup_nfs_mountpoint }} is not mounted; bring
|
||||||
|
up the pipeline mount before deploying the backup.
|
||||||
|
|
||||||
|
- name: Ensure the backup target directory exists on the share
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ ai_backup_target_dir }}"
|
||||||
|
state: directory
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Install the backup script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-ai-backup.sh.j2
|
||||||
|
dest: "{{ ai_backup_script }}"
|
||||||
|
mode: "0750"
|
||||||
|
notify:
|
||||||
|
- Verify the backup script syntax
|
||||||
|
|
||||||
|
- name: Install the backup service unit
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-ai-backup.service.j2
|
||||||
|
dest: "/etc/systemd/system/{{ ai_backup_service }}.service"
|
||||||
|
mode: "0644"
|
||||||
|
notify:
|
||||||
|
- Reload the systemd daemon
|
||||||
|
|
||||||
|
- name: Install the backup timer unit
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-ai-backup.timer.j2
|
||||||
|
dest: "/etc/systemd/system/{{ ai_backup_service }}.timer"
|
||||||
|
mode: "0644"
|
||||||
|
notify:
|
||||||
|
- Reload the systemd daemon
|
||||||
|
|
||||||
|
- name: Enable and start the backup timer
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ ai_backup_service }}.timer"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
when: not ansible_check_mode
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# {{ ansible_managed }}
|
||||||
|
[Unit]
|
||||||
|
# GNAI-001 M4 configuration backup; target is the NFS pipeline share.
|
||||||
|
Description=Ghostnet AI configuration backup
|
||||||
|
After=network-online.target var-lib-ghostnet-pipeline.mount
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart={{ ai_backup_script }}
|
||||||
|
Nice=10
|
||||||
|
IOSchedulingClass=best-effort
|
||||||
|
IOSchedulingPriority=7
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# gnai-ai-backup — configuration backup for the AI server (GNAI-001 M4).
|
||||||
|
#
|
||||||
|
# Archives the configuration and small stateful assets to the NFS
|
||||||
|
# pipeline share, verifies the archive integrity, writes a .sha256
|
||||||
|
# sidecar and rotates the retained archives. Model weights are not
|
||||||
|
# archived; the pinned SOURCE.yml manifests inside the archive make
|
||||||
|
# them reproducible (role model_store, --tags models).
|
||||||
|
#
|
||||||
|
# Exit codes: 0 ok; 21 NFS share not mounted; 22 archive integrity
|
||||||
|
# failure; other non-zero failures from tar/sha256.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
TARGET_DIR="{{ ai_backup_target_dir }}"
|
||||||
|
RETENTION={{ ai_backup_retention }}
|
||||||
|
MOUNTPOINT="{{ ai_backup_nfs_mountpoint }}"
|
||||||
|
MODEL_STORE_META="{{ ai_backup_model_store_meta }}"
|
||||||
|
MODEL_STORE_EXCLUDE="{{ ai_backup_model_store_exclude }}"
|
||||||
|
|
||||||
|
mountpoint_q() { mountpoint -q "$1"; }
|
||||||
|
|
||||||
|
log() { echo "gnai-ai-backup: $1"; }
|
||||||
|
|
||||||
|
mountpoint_q "$MOUNTPOINT" || {
|
||||||
|
log "ERROR: NFS share $MOUNTPOINT is not mounted" >&2
|
||||||
|
exit 21
|
||||||
|
}
|
||||||
|
|
||||||
|
# Existing paths only (tar fails on missing members).
|
||||||
|
paths=()
|
||||||
|
{% for p in ai_backup_paths %}
|
||||||
|
[ -e "/{{ p }}" ] && paths+=("{{ p }}")
|
||||||
|
{% endfor %}
|
||||||
|
[ -d "/$MODEL_STORE_META" ] && paths+=("${MODEL_STORE_EXCLUDE}" "$MODEL_STORE_META")
|
||||||
|
|
||||||
|
mkdir -p "$TARGET_DIR"
|
||||||
|
|
||||||
|
STAMP=$(date +%Y%m%d-%H%M)
|
||||||
|
ARCHIVE="$TARGET_DIR/ghostnet-ai-backup-$STAMP.tar.gz"
|
||||||
|
TMP="$ARCHIVE.tmp"
|
||||||
|
|
||||||
|
tar -C / -czf "$TMP" "${paths[@]}"
|
||||||
|
|
||||||
|
if ! tar -tzf "$TMP" > /dev/null 2>&1; then
|
||||||
|
log "ERROR: archive failed the integrity check" >&2
|
||||||
|
rm -f "$TMP"
|
||||||
|
exit 22
|
||||||
|
fi
|
||||||
|
|
||||||
|
mv "$TMP" "$ARCHIVE"
|
||||||
|
sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
|
||||||
|
|
||||||
|
# Rotate: keep the newest $RETENTION archives with their sidecars.
|
||||||
|
ls -1t "$TARGET_DIR"/ghostnet-ai-backup-*.tar.gz 2>/dev/null \
|
||||||
|
| tail -n +"$(( RETENTION + 1 ))" \
|
||||||
|
| while read -r old; do
|
||||||
|
rm -f "$old" "$old.sha256"
|
||||||
|
done
|
||||||
|
|
||||||
|
log "backup complete: $ARCHIVE ($(du -m "$ARCHIVE" | cut -f1) MB, $(tar -tzf "$ARCHIVE" | wc -l) members)"
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# {{ ansible_managed }}
|
||||||
|
[Unit]
|
||||||
|
Description=Ghostnet AI configuration backup timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ ai_backup_on_calendar }}
|
||||||
|
Persistent={{ 'true' if ai_backup_persistent else 'false' }}
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -49,3 +49,12 @@ ai_environment_health_interval: 5
|
|||||||
ai_environment_recovery_service: gnai-environment-recovery
|
ai_environment_recovery_service: gnai-environment-recovery
|
||||||
ai_environment_recovery_timer: gnai-environment-recovery
|
ai_environment_recovery_timer: gnai-environment-recovery
|
||||||
ai_environment_recovery_timer_interval: "1 min"
|
ai_environment_recovery_timer_interval: "1 min"
|
||||||
|
|
||||||
|
# Availability metric for the monitoring stack (GNMON-001 M5): a oneshot
|
||||||
|
# service writes a node_exporter textfile exposing the sanctioned active
|
||||||
|
# environment and whether its engine unit is actually running. Alert
|
||||||
|
# rule for the suite (vmalert): gn_ai_active_env == 0 for 2m, critical.
|
||||||
|
ai_environment_metrics_service: gnai-environment-metrics
|
||||||
|
ai_environment_metrics_timer: gnai-environment-metrics
|
||||||
|
ai_environment_metrics_interval: "30 s"
|
||||||
|
ai_environment_textfile_dir: /var/lib/gn-monitor/textfile
|
||||||
|
|||||||
@@ -20,6 +20,17 @@
|
|||||||
failed_when: ai_environment_switch_syntax.rc != 0
|
failed_when: ai_environment_switch_syntax.rc != 0
|
||||||
when: not ansible_check_mode
|
when: not ansible_check_mode
|
||||||
|
|
||||||
|
- name: Verify the environment metrics script syntax
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- /bin/bash
|
||||||
|
- -n
|
||||||
|
- "{{ ai_environment_switch_script }}-metrics"
|
||||||
|
register: ai_environment_metrics_syntax
|
||||||
|
changed_when: false
|
||||||
|
failed_when: ai_environment_metrics_syntax.rc != 0
|
||||||
|
when: not ansible_check_mode
|
||||||
|
|
||||||
- name: Reload the systemd daemon
|
- name: Reload the systemd daemon
|
||||||
ansible.builtin.systemd_service:
|
ansible.builtin.systemd_service:
|
||||||
daemon_reload: true
|
daemon_reload: true
|
||||||
|
|||||||
@@ -104,6 +104,37 @@
|
|||||||
state: started
|
state: started
|
||||||
when: not ansible_check_mode
|
when: not ansible_check_mode
|
||||||
|
|
||||||
|
- name: Install the environment availability metric writer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-environment-metrics.sh.j2
|
||||||
|
dest: "{{ ai_environment_switch_script }}-metrics"
|
||||||
|
mode: "0750"
|
||||||
|
notify:
|
||||||
|
- Verify the environment metrics script syntax
|
||||||
|
|
||||||
|
- name: Install the environment availability metric service
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-environment-metrics.service.j2
|
||||||
|
dest: "/etc/systemd/system/{{ ai_environment_metrics_service }}.service"
|
||||||
|
mode: "0644"
|
||||||
|
notify:
|
||||||
|
- Reload the systemd daemon
|
||||||
|
|
||||||
|
- name: Install the environment availability metric timer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: gnai-environment-metrics.timer.j2
|
||||||
|
dest: "/etc/systemd/system/{{ ai_environment_metrics_timer }}.timer"
|
||||||
|
mode: "0644"
|
||||||
|
notify:
|
||||||
|
- Reload the systemd daemon
|
||||||
|
|
||||||
|
- name: Enable and start the environment availability metric timer
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ ai_environment_metrics_timer }}.timer"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
when: not ansible_check_mode
|
||||||
|
|
||||||
- name: Verify the environment switch script answers a status query
|
- name: Verify the environment switch script answers a status query
|
||||||
ansible.builtin.command:
|
ansible.builtin.command:
|
||||||
argv:
|
argv:
|
||||||
|
|||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
# {{ ansible_managed }}
|
||||||
|
[Unit]
|
||||||
|
# Writes the gn_ai_active_env textfile metric for the monitoring stack.
|
||||||
|
Description=Ghostnet AI environment availability metric
|
||||||
|
After=multi-user.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart={{ ai_environment_switch_script }}-metrics
|
||||||
|
User=root
|
||||||
+44
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# gnai-environment-metrics — node_exporter textfile writer (GNMON-001).
|
||||||
|
#
|
||||||
|
# Exposes gn_ai_active_env{env}: 1 when the state-active environment's
|
||||||
|
# engine unit is running, 0 otherwise (no state, unknown active
|
||||||
|
# environment or the unit is down). Alert suggestion for the suite:
|
||||||
|
# gn_ai_active_env == 0 for 2m, severity critical.
|
||||||
|
|
||||||
|
set -u
|
||||||
|
cd /
|
||||||
|
|
||||||
|
CONFIG="{{ ai_environment_switch_config }}"
|
||||||
|
STATE="{{ ai_environment_state_file }}"
|
||||||
|
OUT="{{ ai_environment_textfile_dir }}/ai-environment.prom"
|
||||||
|
TMP="$OUT.tmp"
|
||||||
|
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
[ -r "$CONFIG" ] && . "$CONFIG" || exit 0
|
||||||
|
|
||||||
|
active=""; running=""
|
||||||
|
if [ -r "$STATE" ]; then
|
||||||
|
active=$(sed -n 's/^active=//p' "$STATE" | head -n1)
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "$active" ]; then
|
||||||
|
unit=$(eval "echo \${${active}_active_unit-}")
|
||||||
|
if [ -n "$unit" ]; then
|
||||||
|
uid=$(eval "echo \${${active}_uid-}")
|
||||||
|
account=$(eval "echo \${${active}_account-}")
|
||||||
|
runuser -u "$account" -- \
|
||||||
|
env HOME="$(getent passwd "$account" | cut -d: -f6)" \
|
||||||
|
XDG_RUNTIME_DIR="/run/user/$uid" \
|
||||||
|
systemctl --user is-active "$unit.service" >/dev/null 2>&1 \
|
||||||
|
&& running=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "{{ ai_environment_textfile_dir }}"
|
||||||
|
{
|
||||||
|
echo "# HELP gn_ai_active_env 1 when the state-active AI environment engine unit is running"
|
||||||
|
echo "# TYPE gn_ai_active_env gauge"
|
||||||
|
echo "gn_ai_active_env{env=\"${active:-unknown}\"} $([ -n "$running" ] && echo 1 || echo 0)"
|
||||||
|
} > "$TMP"
|
||||||
|
mv "$TMP" "$OUT"
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
# {{ ansible_managed }}
|
||||||
|
[Unit]
|
||||||
|
Description=Ghostnet AI environment availability metric timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=1 min
|
||||||
|
OnUnitActiveSec={{ ai_environment_metrics_interval }}
|
||||||
|
AccuracySec=10s
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
Reference in New Issue
Block a user