GNAI-001 M4 abgeschlossen: Backup/Reboot/Alarmgrenzen (GNAI-008 14.6)

Rolle ai_backup: woechentlicher Timer, NFS-Ziel (Encoding-Pipeline-
Anbindung, Betreiberentscheid), Archiv ohne GGUF-Weights (SOURCE.yml-
Reproduzierbarkeit), SHA256-Sidecar, Retention 8; Restore-Nachweise
R1 (Baum-Diff) und R2 (environment-state restore) bestanden. Echter
Hostreboot: Recovery-Service bringt Production selbststaendig hoch.
Verfuegbarkeitsmetrik gn_ai_active_env (Textfile, 30 s) mit Regel-
vorschlag fuer die GNMON-Suite. M4 damit vollstaendig abgenommen.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
2026-11-06 07:34:52 +01:00
parent 1c9011a90b
commit 7ec8a970b7
15 changed files with 422 additions and 1 deletions
@@ -963,7 +963,40 @@ Abnahme am 06.11.2026, alle bestanden:
Offen aus M4: realer Hostreboot-Test (mit dem Betreiber, dedizierte
Hardware), Backup-/Restore-SOP mit praktischem Nachweis sowie die
Prüfung der Alarmgrenzen für Production im Monitoring.
Prüfung der Alarmgrenzen für Production im Monitoring — **alle drei
Folgepunkte wurden am selben Tag nachgezogen:**
- a) **Echter Hostreboot (06.11.2026, 07:21):** Nach dem Betreiber-Go
startete der Server neu; der Recovery-Service brachte Production ohne
jedes Zutun hoch (`active=production`, Engine mit GPU-Prozess,
Inferenzstichprobe 17×23 korrekt, Thinking intakt, Lüfterservice
aktiv). Damit ist der Wiederanlaufpfad für Boot, Abbruch und
Netzwerkverlust belegt.
- b) **Backup-/Restore (Rolle `ai_backup`, Betreiberentscheid: Ziel ist
die bestehende NFS-Anbindung der Encoding-Pipeline, pve-Datenhdd):**
wöchentlicher Timer (So 03:30, Persistent), Archiv mit
`/etc/ghostnet-ai`, Quadlet-Verzeichnissen der Konten,
`environment-state`, Benchmark-Rohdaten und Model-Store-Metadaten
ohne `*.gguf` (Weights bleiben über SOURCE.yml-Pins reproduzierbar),
SHA256-Sidecar, Integritätsprüfung, Retention 8. Praktische Nachweise
R1 (Extrakt und vollständiger Baum-Diff gegen das Live-System,
identisch) und R2 (Löschen und Restore von `environment-state`,
byte-identisch) bestanden; SOP im Rollen-README.
- c) **Monitoring/Alarmgrenzen:** Die GNMON-M5-Suite auf dem Server
(vmalert/Alertmanager rootless unter `gn-monitor`) alarmiert bereits
beide Vorfall-Signaturen (Junction >95 °C über 2 Min; PWM-Anforderung
bei 0 RPM über 3 Min) plus Edge- und VRAM-Grenzen und
Node-Exporter-Ausfall. Ergänzt wurde die Verfügbarkeitsmetrik
`gn_ai_active_env{env}` (Textfile-Collector der Switch-Rolle, 30-Sek.-
Timer, verifiziert für läuft=1/gestoppt=0/nach-Recovery=1);
Regelvorschlag an die Suite: `gn_ai_active_env == 0` über 2 Minuten,
kritisch.
**Damit ist Meilenstein 4 (GNAI-001) vollständig abgenommen:** Der
ausgewählte interne Dienst läuft als Production-Umgebung, ein
abgebrochener Testlauf wird hostseitig ohne Production-Secrets
rückgerollt (Timer/Service), Backup/Restore sind praktisch belegt und
Alarmgrenzen stehen.
---
@@ -0,0 +1,14 @@
---
# Deploys the AI configuration backup (GNAI-001 milestone 4). Backup
# execution is the oneshot service; the timer schedules it weekly.
#
# Usage:
# ansible-playbook -i inventories/<any-env> playbooks/backup.yml \
# --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
- name: Deploy the AI configuration backup
hosts: llm_servers
become: true
gather_facts: false
roles:
- role: ai_backup
@@ -0,0 +1,50 @@
# ai_backup
Configuration backup for the AI server (GNAI-001 milestone 4). Deploys
`gnai-ai-backup` (script), a oneshot service and a weekly timer
(Sunday 03:30, `Persistent=true`). Target is the existing NFS pipeline
share (`pve:/media/datenhdd3/Encoding`, operator decision 06.11.2026).
## Backup scope
- `/etc/ghostnet-ai` (all environment config trees)
- `/etc/containers/systemd/users/{980,981,984}` (rootless quadlets,
included when present)
- `/var/lib/ghostnet-ai/environment-state` (switch state)
- `/var/lib/ghostnet-ai/testing/bench/results` (benchmark artifacts)
- `/var/lib/ghostnet-ai/shared/models` **without `*.gguf`** — the pinned
`SOURCE.yml` manifests make the weights reproducible through the
model_store role (`--tags models` re-downloads and verifies them)
Every archive gets a `.sha256` sidecar; the archive is integrity-checked
before it replaces the last good file; retention keeps the newest
`ai_backup_retention` archives.
## Operations
- Run a backup now: `systemctl start gnai-ai-backup.service`
(the timer does this weekly; `systemctl list-timers gnai-ai-backup*`
shows the next run).
- Watch: `journalctl -u gnai-ai-backup.service -n 20`.
## Restore SOP
1. Identify the archive on the share:
`ls -lt /var/lib/ghostnet-ai-backup/ghostnet-ai-backup-*.tar.gz`
2. Verify the sidecar: `sha256sum -c <archive>.sha256`
3. Extract to a staging directory first (never extract blindly over
live files):
`mkdir /tmp/gnai-restore && tar -xzf <archive> -C /tmp/gnai-restore`
4. Compare or selectively copy back the affected tree, e.g.
`diff -r /tmp/gnai-restore/etc/ghostnet-ai/testing /etc/ghostnet-ai/testing`
5. For environment-state restores, run
`/usr/local/sbin/gnai-environment-switch recover` afterwards so the
runtime converges with the restored state.
6. For model weights: restore via the model_store role
(`--tags models`), then verify the `SOURCE.yml` checksums.
7. Re-run the affected environment playbook
(`playbooks/ai.yml --tags service`) and verify health.
Accepted proof (06.11.2026): full-tree extract-and-diff against the
live configuration and a practical restore of the deleted environment
state file, documented in GNAI-008 section 14.6.
@@ -0,0 +1,37 @@
---
# Configuration backup for the AI server (GNAI-001 milestone 4:
# "Backupumfang, Ziel, Aufbewahrung und Restore-SOP festlegen").
#
# Target: the existing NFS pipeline share (operator decision 06.11.2026:
# reuse the encoding queue connection, pve:/media/datenhdd3/Encoding),
# mounted at /var/lib/ghostnet-pipeline. Model weights are NOT archived;
# they stay reproducible through the pinned SOURCE.yml manifests included
# in every archive plus re-verification after download.
ai_backup_nfs_mountpoint: /var/lib/ghostnet-pipeline
ai_backup_target_dir: /var/lib/ghostnet-ai-backup
# Retention: number of weekly archives kept on the share.
ai_backup_retention: 8
# Schedule: weekly, Sunday 03:30, catching up after downtime.
ai_backup_on_calendar: "Sun *-*-* 03:30:00"
ai_backup_persistent: true
ai_backup_service: gnai-ai-backup
ai_backup_script: /usr/local/sbin/gnai-ai-backup
# Backed-up trees (tar paths relative to /). Quadlet dirs are added by
# the script only when they exist.
ai_backup_paths:
- etc/ghostnet-ai
- etc/containers/systemd/users/980
- etc/containers/systemd/users/981
- etc/containers/systemd/users/984
- var/lib/ghostnet-ai/environment-state
- var/lib/ghostnet-ai/testing/bench/results
# Included without its multi-gigabyte weights: model manifests and any
# other metadata below the shared model store.
ai_backup_model_store_meta: var/lib/ghostnet-ai/shared/models
ai_backup_model_store_exclude: --exclude=*.gguf
@@ -0,0 +1,16 @@
---
- name: Verify the backup script syntax
ansible.builtin.command:
argv:
- /bin/bash
- -n
- "{{ ai_backup_script }}"
register: ai_backup_syntax
changed_when: false
failed_when: ai_backup_syntax.rc != 0
when: not ansible_check_mode
- name: Reload the systemd daemon
ansible.builtin.systemd_service:
daemon_reload: true
when: not ansible_check_mode
@@ -0,0 +1,71 @@
---
# Deploys the AI configuration backup (GNAI-001 milestone 4): script,
# system service and weekly timer targeting the NFS pipeline share.
# Running the backup itself is a one-shot service invocation, not part
# of the role (see the role README for backup and restore procedures).
- name: Verify the backup declarations
ansible.builtin.assert:
that:
- ai_backup_nfs_mountpoint is match("^/")
- ai_backup_target_dir is match("^/")
- ai_backup_retention | int >= 1
- ai_backup_paths | length > 0
- ai_backup_service | length > 0
- ai_backup_script is match("^/usr/local/sbin/")
fail_msg: AI backup declarations are incomplete or invalid.
- name: Require the NFS pipeline share to be mounted
ansible.builtin.command:
argv:
- /usr/bin/mountpoint
- -q
- "{{ ai_backup_nfs_mountpoint }}"
changed_when: false
failed_when: false
register: ai_backup_share_mounted
- name: Verify the NFS pipeline share is mounted
ansible.builtin.assert:
that:
- ai_backup_share_mounted.rc == 0
fail_msg: >-
The NFS share {{ ai_backup_nfs_mountpoint }} is not mounted; bring
up the pipeline mount before deploying the backup.
- name: Ensure the backup target directory exists on the share
ansible.builtin.file:
path: "{{ ai_backup_target_dir }}"
state: directory
mode: "0750"
- name: Install the backup script
ansible.builtin.template:
src: gnai-ai-backup.sh.j2
dest: "{{ ai_backup_script }}"
mode: "0750"
notify:
- Verify the backup script syntax
- name: Install the backup service unit
ansible.builtin.template:
src: gnai-ai-backup.service.j2
dest: "/etc/systemd/system/{{ ai_backup_service }}.service"
mode: "0644"
notify:
- Reload the systemd daemon
- name: Install the backup timer unit
ansible.builtin.template:
src: gnai-ai-backup.timer.j2
dest: "/etc/systemd/system/{{ ai_backup_service }}.timer"
mode: "0644"
notify:
- Reload the systemd daemon
- name: Enable and start the backup timer
ansible.builtin.systemd_service:
name: "{{ ai_backup_service }}.timer"
enabled: true
state: started
when: not ansible_check_mode
@@ -0,0 +1,13 @@
# {{ ansible_managed }}
[Unit]
# GNAI-001 M4 configuration backup; target is the NFS pipeline share.
Description=Ghostnet AI configuration backup
After=network-online.target var-lib-ghostnet-pipeline.mount
Wants=network-online.target
[Service]
Type=oneshot
ExecStart={{ ai_backup_script }}
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
@@ -0,0 +1,61 @@
#!/bin/bash
# gnai-ai-backup — configuration backup for the AI server (GNAI-001 M4).
#
# Archives the configuration and small stateful assets to the NFS
# pipeline share, verifies the archive integrity, writes a .sha256
# sidecar and rotates the retained archives. Model weights are not
# archived; the pinned SOURCE.yml manifests inside the archive make
# them reproducible (role model_store, --tags models).
#
# Exit codes: 0 ok; 21 NFS share not mounted; 22 archive integrity
# failure; other non-zero failures from tar/sha256.
set -euo pipefail
TARGET_DIR="{{ ai_backup_target_dir }}"
RETENTION={{ ai_backup_retention }}
MOUNTPOINT="{{ ai_backup_nfs_mountpoint }}"
MODEL_STORE_META="{{ ai_backup_model_store_meta }}"
MODEL_STORE_EXCLUDE="{{ ai_backup_model_store_exclude }}"
mountpoint_q() { mountpoint -q "$1"; }
log() { echo "gnai-ai-backup: $1"; }
mountpoint_q "$MOUNTPOINT" || {
log "ERROR: NFS share $MOUNTPOINT is not mounted" >&2
exit 21
}
# Existing paths only (tar fails on missing members).
paths=()
{% for p in ai_backup_paths %}
[ -e "/{{ p }}" ] && paths+=("{{ p }}")
{% endfor %}
[ -d "/$MODEL_STORE_META" ] && paths+=("${MODEL_STORE_EXCLUDE}" "$MODEL_STORE_META")
mkdir -p "$TARGET_DIR"
STAMP=$(date +%Y%m%d-%H%M)
ARCHIVE="$TARGET_DIR/ghostnet-ai-backup-$STAMP.tar.gz"
TMP="$ARCHIVE.tmp"
tar -C / -czf "$TMP" "${paths[@]}"
if ! tar -tzf "$TMP" > /dev/null 2>&1; then
log "ERROR: archive failed the integrity check" >&2
rm -f "$TMP"
exit 22
fi
mv "$TMP" "$ARCHIVE"
sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
# Rotate: keep the newest $RETENTION archives with their sidecars.
ls -1t "$TARGET_DIR"/ghostnet-ai-backup-*.tar.gz 2>/dev/null \
| tail -n +"$(( RETENTION + 1 ))" \
| while read -r old; do
rm -f "$old" "$old.sha256"
done
log "backup complete: $ARCHIVE ($(du -m "$ARCHIVE" | cut -f1) MB, $(tar -tzf "$ARCHIVE" | wc -l) members)"
@@ -0,0 +1,10 @@
# {{ ansible_managed }}
[Unit]
Description=Ghostnet AI configuration backup timer
[Timer]
OnCalendar={{ ai_backup_on_calendar }}
Persistent={{ 'true' if ai_backup_persistent else 'false' }}
[Install]
WantedBy=timers.target
@@ -49,3 +49,12 @@ ai_environment_health_interval: 5
ai_environment_recovery_service: gnai-environment-recovery
ai_environment_recovery_timer: gnai-environment-recovery
ai_environment_recovery_timer_interval: "1 min"
# Availability metric for the monitoring stack (GNMON-001 M5): a oneshot
# service writes a node_exporter textfile exposing the sanctioned active
# environment and whether its engine unit is actually running. Alert
# rule for the suite (vmalert): gn_ai_active_env == 0 for 2m, critical.
ai_environment_metrics_service: gnai-environment-metrics
ai_environment_metrics_timer: gnai-environment-metrics
ai_environment_metrics_interval: "30 s"
ai_environment_textfile_dir: /var/lib/gn-monitor/textfile
@@ -20,6 +20,17 @@
failed_when: ai_environment_switch_syntax.rc != 0
when: not ansible_check_mode
- name: Verify the environment metrics script syntax
ansible.builtin.command:
argv:
- /bin/bash
- -n
- "{{ ai_environment_switch_script }}-metrics"
register: ai_environment_metrics_syntax
changed_when: false
failed_when: ai_environment_metrics_syntax.rc != 0
when: not ansible_check_mode
- name: Reload the systemd daemon
ansible.builtin.systemd_service:
daemon_reload: true
@@ -104,6 +104,37 @@
state: started
when: not ansible_check_mode
- name: Install the environment availability metric writer
ansible.builtin.template:
src: gnai-environment-metrics.sh.j2
dest: "{{ ai_environment_switch_script }}-metrics"
mode: "0750"
notify:
- Verify the environment metrics script syntax
- name: Install the environment availability metric service
ansible.builtin.template:
src: gnai-environment-metrics.service.j2
dest: "/etc/systemd/system/{{ ai_environment_metrics_service }}.service"
mode: "0644"
notify:
- Reload the systemd daemon
- name: Install the environment availability metric timer
ansible.builtin.template:
src: gnai-environment-metrics.timer.j2
dest: "/etc/systemd/system/{{ ai_environment_metrics_timer }}.timer"
mode: "0644"
notify:
- Reload the systemd daemon
- name: Enable and start the environment availability metric timer
ansible.builtin.systemd_service:
name: "{{ ai_environment_metrics_timer }}.timer"
enabled: true
state: started
when: not ansible_check_mode
- name: Verify the environment switch script answers a status query
ansible.builtin.command:
argv:
@@ -0,0 +1,10 @@
# {{ ansible_managed }}
[Unit]
# Writes the gn_ai_active_env textfile metric for the monitoring stack.
Description=Ghostnet AI environment availability metric
After=multi-user.target
[Service]
Type=oneshot
ExecStart={{ ai_environment_switch_script }}-metrics
User=root
@@ -0,0 +1,44 @@
#!/bin/sh
# gnai-environment-metrics — node_exporter textfile writer (GNMON-001).
#
# Exposes gn_ai_active_env{env}: 1 when the state-active environment's
# engine unit is running, 0 otherwise (no state, unknown active
# environment or the unit is down). Alert suggestion for the suite:
# gn_ai_active_env == 0 for 2m, severity critical.
set -u
cd /
CONFIG="{{ ai_environment_switch_config }}"
STATE="{{ ai_environment_state_file }}"
OUT="{{ ai_environment_textfile_dir }}/ai-environment.prom"
TMP="$OUT.tmp"
# shellcheck disable=SC1090
[ -r "$CONFIG" ] && . "$CONFIG" || exit 0
active=""; running=""
if [ -r "$STATE" ]; then
active=$(sed -n 's/^active=//p' "$STATE" | head -n1)
fi
if [ -n "$active" ]; then
unit=$(eval "echo \${${active}_active_unit-}")
if [ -n "$unit" ]; then
uid=$(eval "echo \${${active}_uid-}")
account=$(eval "echo \${${active}_account-}")
runuser -u "$account" -- \
env HOME="$(getent passwd "$account" | cut -d: -f6)" \
XDG_RUNTIME_DIR="/run/user/$uid" \
systemctl --user is-active "$unit.service" >/dev/null 2>&1 \
&& running=1
fi
fi
mkdir -p "{{ ai_environment_textfile_dir }}"
{
echo "# HELP gn_ai_active_env 1 when the state-active AI environment engine unit is running"
echo "# TYPE gn_ai_active_env gauge"
echo "gn_ai_active_env{env=\"${active:-unknown}\"} $([ -n "$running" ] && echo 1 || echo 0)"
} > "$TMP"
mv "$TMP" "$OUT"
@@ -0,0 +1,11 @@
# {{ ansible_managed }}
[Unit]
Description=Ghostnet AI environment availability metric timer
[Timer]
OnBootSec=1 min
OnUnitActiveSec={{ ai_environment_metrics_interval }}
AccuracySec=10s
[Install]
WantedBy=timers.target