diff --git a/docs/platform/llm-server/ai/GNAI-008-PLAN_Ansible-Deployment.md b/docs/platform/llm-server/ai/GNAI-008-PLAN_Ansible-Deployment.md index ff8af14..171652c 100644 --- a/docs/platform/llm-server/ai/GNAI-008-PLAN_Ansible-Deployment.md +++ b/docs/platform/llm-server/ai/GNAI-008-PLAN_Ansible-Deployment.md @@ -963,7 +963,40 @@ Abnahme am 06.11.2026, alle bestanden: Offen aus M4: realer Hostreboot-Test (mit dem Betreiber, dedizierte Hardware), Backup-/Restore-SOP mit praktischem Nachweis sowie die -Prüfung der Alarmgrenzen für Production im Monitoring. +Prüfung der Alarmgrenzen für Production im Monitoring — **alle drei +Folgepunkte wurden am selben Tag nachgezogen:** + +- a) **Echter Hostreboot (06.11.2026, 07:21):** Nach dem Betreiber-Go + startete der Server neu; der Recovery-Service brachte Production ohne + jedes Zutun hoch (`active=production`, Engine mit GPU-Prozess, + Inferenzstichprobe 17×23 korrekt, Thinking intakt, Lüfterservice + aktiv). Damit ist der Wiederanlaufpfad für Boot, Abbruch und + Netzwerkverlust belegt. +- b) **Backup-/Restore (Rolle `ai_backup`, Betreiberentscheid: Ziel ist + die bestehende NFS-Anbindung der Encoding-Pipeline, pve-Datenhdd):** + wöchentlicher Timer (So 03:30, Persistent), Archiv mit + `/etc/ghostnet-ai`, Quadlet-Verzeichnissen der Konten, + `environment-state`, Benchmark-Rohdaten und Model-Store-Metadaten + ohne `*.gguf` (Weights bleiben über SOURCE.yml-Pins reproduzierbar), + SHA256-Sidecar, Integritätsprüfung, Retention 8. Praktische Nachweise + R1 (Extrakt und vollständiger Baum-Diff gegen das Live-System, + identisch) und R2 (Löschen und Restore von `environment-state`, + byte-identisch) bestanden; SOP im Rollen-README. +- c) **Monitoring/Alarmgrenzen:** Die GNMON-M5-Suite auf dem Server + (vmalert/Alertmanager rootless unter `gn-monitor`) alarmiert bereits + beide Vorfall-Signaturen (Junction >95 °C über 2 Min; PWM-Anforderung + bei 0 RPM über 3 Min) plus Edge- und VRAM-Grenzen und + Node-Exporter-Ausfall. Ergänzt wurde die Verfügbarkeitsmetrik + `gn_ai_active_env{env}` (Textfile-Collector der Switch-Rolle, 30-Sek.- + Timer, verifiziert für läuft=1/gestoppt=0/nach-Recovery=1); + Regelvorschlag an die Suite: `gn_ai_active_env == 0` über 2 Minuten, + kritisch. + +**Damit ist Meilenstein 4 (GNAI-001) vollständig abgenommen:** Der +ausgewählte interne Dienst läuft als Production-Umgebung, ein +abgebrochener Testlauf wird hostseitig ohne Production-Secrets +rückgerollt (Timer/Service), Backup/Restore sind praktisch belegt und +Alarmgrenzen stehen. --- diff --git a/llm-server/llm/ansible/playbooks/backup.yml b/llm-server/llm/ansible/playbooks/backup.yml new file mode 100644 index 0000000..6a40785 --- /dev/null +++ b/llm-server/llm/ansible/playbooks/backup.yml @@ -0,0 +1,14 @@ +--- +# Deploys the AI configuration backup (GNAI-001 milestone 4). Backup +# execution is the oneshot service; the timer schedules it weekly. +# +# Usage: +# ansible-playbook -i inventories/ playbooks/backup.yml \ +# --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy +- name: Deploy the AI configuration backup + hosts: llm_servers + become: true + gather_facts: false + + roles: + - role: ai_backup diff --git a/llm-server/llm/ansible/roles/ai_backup/README.md b/llm-server/llm/ansible/roles/ai_backup/README.md new file mode 100644 index 0000000..0b8a328 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/README.md @@ -0,0 +1,50 @@ +# ai_backup + +Configuration backup for the AI server (GNAI-001 milestone 4). Deploys +`gnai-ai-backup` (script), a oneshot service and a weekly timer +(Sunday 03:30, `Persistent=true`). Target is the existing NFS pipeline +share (`pve:/media/datenhdd3/Encoding`, operator decision 06.11.2026). + +## Backup scope + +- `/etc/ghostnet-ai` (all environment config trees) +- `/etc/containers/systemd/users/{980,981,984}` (rootless quadlets, + included when present) +- `/var/lib/ghostnet-ai/environment-state` (switch state) +- `/var/lib/ghostnet-ai/testing/bench/results` (benchmark artifacts) +- `/var/lib/ghostnet-ai/shared/models` **without `*.gguf`** — the pinned + `SOURCE.yml` manifests make the weights reproducible through the + model_store role (`--tags models` re-downloads and verifies them) + +Every archive gets a `.sha256` sidecar; the archive is integrity-checked +before it replaces the last good file; retention keeps the newest +`ai_backup_retention` archives. + +## Operations + +- Run a backup now: `systemctl start gnai-ai-backup.service` + (the timer does this weekly; `systemctl list-timers gnai-ai-backup*` + shows the next run). +- Watch: `journalctl -u gnai-ai-backup.service -n 20`. + +## Restore SOP + +1. Identify the archive on the share: + `ls -lt /var/lib/ghostnet-ai-backup/ghostnet-ai-backup-*.tar.gz` +2. Verify the sidecar: `sha256sum -c .sha256` +3. Extract to a staging directory first (never extract blindly over + live files): + `mkdir /tmp/gnai-restore && tar -xzf -C /tmp/gnai-restore` +4. Compare or selectively copy back the affected tree, e.g. + `diff -r /tmp/gnai-restore/etc/ghostnet-ai/testing /etc/ghostnet-ai/testing` +5. For environment-state restores, run + `/usr/local/sbin/gnai-environment-switch recover` afterwards so the + runtime converges with the restored state. +6. For model weights: restore via the model_store role + (`--tags models`), then verify the `SOURCE.yml` checksums. +7. Re-run the affected environment playbook + (`playbooks/ai.yml --tags service`) and verify health. + +Accepted proof (06.11.2026): full-tree extract-and-diff against the +live configuration and a practical restore of the deleted environment +state file, documented in GNAI-008 section 14.6. diff --git a/llm-server/llm/ansible/roles/ai_backup/defaults/main.yml b/llm-server/llm/ansible/roles/ai_backup/defaults/main.yml new file mode 100644 index 0000000..76e8422 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/defaults/main.yml @@ -0,0 +1,37 @@ +--- +# Configuration backup for the AI server (GNAI-001 milestone 4: +# "Backupumfang, Ziel, Aufbewahrung und Restore-SOP festlegen"). +# +# Target: the existing NFS pipeline share (operator decision 06.11.2026: +# reuse the encoding queue connection, pve:/media/datenhdd3/Encoding), +# mounted at /var/lib/ghostnet-pipeline. Model weights are NOT archived; +# they stay reproducible through the pinned SOURCE.yml manifests included +# in every archive plus re-verification after download. + +ai_backup_nfs_mountpoint: /var/lib/ghostnet-pipeline +ai_backup_target_dir: /var/lib/ghostnet-ai-backup + +# Retention: number of weekly archives kept on the share. +ai_backup_retention: 8 + +# Schedule: weekly, Sunday 03:30, catching up after downtime. +ai_backup_on_calendar: "Sun *-*-* 03:30:00" +ai_backup_persistent: true + +ai_backup_service: gnai-ai-backup +ai_backup_script: /usr/local/sbin/gnai-ai-backup + +# Backed-up trees (tar paths relative to /). Quadlet dirs are added by +# the script only when they exist. +ai_backup_paths: + - etc/ghostnet-ai + - etc/containers/systemd/users/980 + - etc/containers/systemd/users/981 + - etc/containers/systemd/users/984 + - var/lib/ghostnet-ai/environment-state + - var/lib/ghostnet-ai/testing/bench/results + +# Included without its multi-gigabyte weights: model manifests and any +# other metadata below the shared model store. +ai_backup_model_store_meta: var/lib/ghostnet-ai/shared/models +ai_backup_model_store_exclude: --exclude=*.gguf diff --git a/llm-server/llm/ansible/roles/ai_backup/handlers/main.yml b/llm-server/llm/ansible/roles/ai_backup/handlers/main.yml new file mode 100644 index 0000000..8a0224c --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/handlers/main.yml @@ -0,0 +1,16 @@ +--- +- name: Verify the backup script syntax + ansible.builtin.command: + argv: + - /bin/bash + - -n + - "{{ ai_backup_script }}" + register: ai_backup_syntax + changed_when: false + failed_when: ai_backup_syntax.rc != 0 + when: not ansible_check_mode + +- name: Reload the systemd daemon + ansible.builtin.systemd_service: + daemon_reload: true + when: not ansible_check_mode diff --git a/llm-server/llm/ansible/roles/ai_backup/tasks/main.yml b/llm-server/llm/ansible/roles/ai_backup/tasks/main.yml new file mode 100644 index 0000000..9b7b052 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/tasks/main.yml @@ -0,0 +1,71 @@ +--- +# Deploys the AI configuration backup (GNAI-001 milestone 4): script, +# system service and weekly timer targeting the NFS pipeline share. +# Running the backup itself is a one-shot service invocation, not part +# of the role (see the role README for backup and restore procedures). + +- name: Verify the backup declarations + ansible.builtin.assert: + that: + - ai_backup_nfs_mountpoint is match("^/") + - ai_backup_target_dir is match("^/") + - ai_backup_retention | int >= 1 + - ai_backup_paths | length > 0 + - ai_backup_service | length > 0 + - ai_backup_script is match("^/usr/local/sbin/") + fail_msg: AI backup declarations are incomplete or invalid. + +- name: Require the NFS pipeline share to be mounted + ansible.builtin.command: + argv: + - /usr/bin/mountpoint + - -q + - "{{ ai_backup_nfs_mountpoint }}" + changed_when: false + failed_when: false + register: ai_backup_share_mounted + +- name: Verify the NFS pipeline share is mounted + ansible.builtin.assert: + that: + - ai_backup_share_mounted.rc == 0 + fail_msg: >- + The NFS share {{ ai_backup_nfs_mountpoint }} is not mounted; bring + up the pipeline mount before deploying the backup. + +- name: Ensure the backup target directory exists on the share + ansible.builtin.file: + path: "{{ ai_backup_target_dir }}" + state: directory + mode: "0750" + +- name: Install the backup script + ansible.builtin.template: + src: gnai-ai-backup.sh.j2 + dest: "{{ ai_backup_script }}" + mode: "0750" + notify: + - Verify the backup script syntax + +- name: Install the backup service unit + ansible.builtin.template: + src: gnai-ai-backup.service.j2 + dest: "/etc/systemd/system/{{ ai_backup_service }}.service" + mode: "0644" + notify: + - Reload the systemd daemon + +- name: Install the backup timer unit + ansible.builtin.template: + src: gnai-ai-backup.timer.j2 + dest: "/etc/systemd/system/{{ ai_backup_service }}.timer" + mode: "0644" + notify: + - Reload the systemd daemon + +- name: Enable and start the backup timer + ansible.builtin.systemd_service: + name: "{{ ai_backup_service }}.timer" + enabled: true + state: started + when: not ansible_check_mode diff --git a/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.service.j2 b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.service.j2 new file mode 100644 index 0000000..0311ec6 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.service.j2 @@ -0,0 +1,13 @@ +# {{ ansible_managed }} +[Unit] +# GNAI-001 M4 configuration backup; target is the NFS pipeline share. +Description=Ghostnet AI configuration backup +After=network-online.target var-lib-ghostnet-pipeline.mount +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart={{ ai_backup_script }} +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 diff --git a/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.sh.j2 b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.sh.j2 new file mode 100644 index 0000000..31ea209 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.sh.j2 @@ -0,0 +1,61 @@ +#!/bin/bash +# gnai-ai-backup — configuration backup for the AI server (GNAI-001 M4). +# +# Archives the configuration and small stateful assets to the NFS +# pipeline share, verifies the archive integrity, writes a .sha256 +# sidecar and rotates the retained archives. Model weights are not +# archived; the pinned SOURCE.yml manifests inside the archive make +# them reproducible (role model_store, --tags models). +# +# Exit codes: 0 ok; 21 NFS share not mounted; 22 archive integrity +# failure; other non-zero failures from tar/sha256. + +set -euo pipefail + +TARGET_DIR="{{ ai_backup_target_dir }}" +RETENTION={{ ai_backup_retention }} +MOUNTPOINT="{{ ai_backup_nfs_mountpoint }}" +MODEL_STORE_META="{{ ai_backup_model_store_meta }}" +MODEL_STORE_EXCLUDE="{{ ai_backup_model_store_exclude }}" + +mountpoint_q() { mountpoint -q "$1"; } + +log() { echo "gnai-ai-backup: $1"; } + +mountpoint_q "$MOUNTPOINT" || { + log "ERROR: NFS share $MOUNTPOINT is not mounted" >&2 + exit 21 +} + +# Existing paths only (tar fails on missing members). +paths=() +{% for p in ai_backup_paths %} +[ -e "/{{ p }}" ] && paths+=("{{ p }}") +{% endfor %} +[ -d "/$MODEL_STORE_META" ] && paths+=("${MODEL_STORE_EXCLUDE}" "$MODEL_STORE_META") + +mkdir -p "$TARGET_DIR" + +STAMP=$(date +%Y%m%d-%H%M) +ARCHIVE="$TARGET_DIR/ghostnet-ai-backup-$STAMP.tar.gz" +TMP="$ARCHIVE.tmp" + +tar -C / -czf "$TMP" "${paths[@]}" + +if ! tar -tzf "$TMP" > /dev/null 2>&1; then + log "ERROR: archive failed the integrity check" >&2 + rm -f "$TMP" + exit 22 +fi + +mv "$TMP" "$ARCHIVE" +sha256sum "$ARCHIVE" > "$ARCHIVE.sha256" + +# Rotate: keep the newest $RETENTION archives with their sidecars. +ls -1t "$TARGET_DIR"/ghostnet-ai-backup-*.tar.gz 2>/dev/null \ + | tail -n +"$(( RETENTION + 1 ))" \ + | while read -r old; do + rm -f "$old" "$old.sha256" + done + +log "backup complete: $ARCHIVE ($(du -m "$ARCHIVE" | cut -f1) MB, $(tar -tzf "$ARCHIVE" | wc -l) members)" diff --git a/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.timer.j2 b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.timer.j2 new file mode 100644 index 0000000..460aad7 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_backup/templates/gnai-ai-backup.timer.j2 @@ -0,0 +1,10 @@ +# {{ ansible_managed }} +[Unit] +Description=Ghostnet AI configuration backup timer + +[Timer] +OnCalendar={{ ai_backup_on_calendar }} +Persistent={{ 'true' if ai_backup_persistent else 'false' }} + +[Install] +WantedBy=timers.target diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/defaults/main.yml b/llm-server/llm/ansible/roles/ai_environment_switch/defaults/main.yml index 4db77b1..673d85a 100644 --- a/llm-server/llm/ansible/roles/ai_environment_switch/defaults/main.yml +++ b/llm-server/llm/ansible/roles/ai_environment_switch/defaults/main.yml @@ -49,3 +49,12 @@ ai_environment_health_interval: 5 ai_environment_recovery_service: gnai-environment-recovery ai_environment_recovery_timer: gnai-environment-recovery ai_environment_recovery_timer_interval: "1 min" + +# Availability metric for the monitoring stack (GNMON-001 M5): a oneshot +# service writes a node_exporter textfile exposing the sanctioned active +# environment and whether its engine unit is actually running. Alert +# rule for the suite (vmalert): gn_ai_active_env == 0 for 2m, critical. +ai_environment_metrics_service: gnai-environment-metrics +ai_environment_metrics_timer: gnai-environment-metrics +ai_environment_metrics_interval: "30 s" +ai_environment_textfile_dir: /var/lib/gn-monitor/textfile diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/handlers/main.yml b/llm-server/llm/ansible/roles/ai_environment_switch/handlers/main.yml index c6576bd..c04ac06 100644 --- a/llm-server/llm/ansible/roles/ai_environment_switch/handlers/main.yml +++ b/llm-server/llm/ansible/roles/ai_environment_switch/handlers/main.yml @@ -20,6 +20,17 @@ failed_when: ai_environment_switch_syntax.rc != 0 when: not ansible_check_mode +- name: Verify the environment metrics script syntax + ansible.builtin.command: + argv: + - /bin/bash + - -n + - "{{ ai_environment_switch_script }}-metrics" + register: ai_environment_metrics_syntax + changed_when: false + failed_when: ai_environment_metrics_syntax.rc != 0 + when: not ansible_check_mode + - name: Reload the systemd daemon ansible.builtin.systemd_service: daemon_reload: true diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/tasks/main.yml b/llm-server/llm/ansible/roles/ai_environment_switch/tasks/main.yml index 389d610..df23d61 100644 --- a/llm-server/llm/ansible/roles/ai_environment_switch/tasks/main.yml +++ b/llm-server/llm/ansible/roles/ai_environment_switch/tasks/main.yml @@ -104,6 +104,37 @@ state: started when: not ansible_check_mode +- name: Install the environment availability metric writer + ansible.builtin.template: + src: gnai-environment-metrics.sh.j2 + dest: "{{ ai_environment_switch_script }}-metrics" + mode: "0750" + notify: + - Verify the environment metrics script syntax + +- name: Install the environment availability metric service + ansible.builtin.template: + src: gnai-environment-metrics.service.j2 + dest: "/etc/systemd/system/{{ ai_environment_metrics_service }}.service" + mode: "0644" + notify: + - Reload the systemd daemon + +- name: Install the environment availability metric timer + ansible.builtin.template: + src: gnai-environment-metrics.timer.j2 + dest: "/etc/systemd/system/{{ ai_environment_metrics_timer }}.timer" + mode: "0644" + notify: + - Reload the systemd daemon + +- name: Enable and start the environment availability metric timer + ansible.builtin.systemd_service: + name: "{{ ai_environment_metrics_timer }}.timer" + enabled: true + state: started + when: not ansible_check_mode + - name: Verify the environment switch script answers a status query ansible.builtin.command: argv: diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.service.j2 b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.service.j2 new file mode 100644 index 0000000..0252473 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.service.j2 @@ -0,0 +1,10 @@ +# {{ ansible_managed }} +[Unit] +# Writes the gn_ai_active_env textfile metric for the monitoring stack. +Description=Ghostnet AI environment availability metric +After=multi-user.target + +[Service] +Type=oneshot +ExecStart={{ ai_environment_switch_script }}-metrics +User=root diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.sh.j2 b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.sh.j2 new file mode 100644 index 0000000..1249348 --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.sh.j2 @@ -0,0 +1,44 @@ +#!/bin/sh +# gnai-environment-metrics — node_exporter textfile writer (GNMON-001). +# +# Exposes gn_ai_active_env{env}: 1 when the state-active environment's +# engine unit is running, 0 otherwise (no state, unknown active +# environment or the unit is down). Alert suggestion for the suite: +# gn_ai_active_env == 0 for 2m, severity critical. + +set -u +cd / + +CONFIG="{{ ai_environment_switch_config }}" +STATE="{{ ai_environment_state_file }}" +OUT="{{ ai_environment_textfile_dir }}/ai-environment.prom" +TMP="$OUT.tmp" + +# shellcheck disable=SC1090 +[ -r "$CONFIG" ] && . "$CONFIG" || exit 0 + +active=""; running="" +if [ -r "$STATE" ]; then + active=$(sed -n 's/^active=//p' "$STATE" | head -n1) +fi + +if [ -n "$active" ]; then + unit=$(eval "echo \${${active}_active_unit-}") + if [ -n "$unit" ]; then + uid=$(eval "echo \${${active}_uid-}") + account=$(eval "echo \${${active}_account-}") + runuser -u "$account" -- \ + env HOME="$(getent passwd "$account" | cut -d: -f6)" \ + XDG_RUNTIME_DIR="/run/user/$uid" \ + systemctl --user is-active "$unit.service" >/dev/null 2>&1 \ + && running=1 + fi +fi + +mkdir -p "{{ ai_environment_textfile_dir }}" +{ + echo "# HELP gn_ai_active_env 1 when the state-active AI environment engine unit is running" + echo "# TYPE gn_ai_active_env gauge" + echo "gn_ai_active_env{env=\"${active:-unknown}\"} $([ -n "$running" ] && echo 1 || echo 0)" +} > "$TMP" +mv "$TMP" "$OUT" diff --git a/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.timer.j2 b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.timer.j2 new file mode 100644 index 0000000..799237f --- /dev/null +++ b/llm-server/llm/ansible/roles/ai_environment_switch/templates/gnai-environment-metrics.timer.j2 @@ -0,0 +1,11 @@ +# {{ ansible_managed }} +[Unit] +Description=Ghostnet AI environment availability metric timer + +[Timer] +OnBootSec=1 min +OnUnitActiveSec={{ ai_environment_metrics_interval }} +AccuracySec=10s + +[Install] +WantedBy=timers.target