GNAI-001 M4 abgeschlossen: Backup/Reboot/Alarmgrenzen (GNAI-008 14.6)
Rolle ai_backup: woechentlicher Timer, NFS-Ziel (Encoding-Pipeline- Anbindung, Betreiberentscheid), Archiv ohne GGUF-Weights (SOURCE.yml- Reproduzierbarkeit), SHA256-Sidecar, Retention 8; Restore-Nachweise R1 (Baum-Diff) und R2 (environment-state restore) bestanden. Echter Hostreboot: Recovery-Service bringt Production selbststaendig hoch. Verfuegbarkeitsmetrik gn_ai_active_env (Textfile, 30 s) mit Regel- vorschlag fuer die GNMON-Suite. M4 damit vollstaendig abgenommen. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
---
|
||||
# Deploys the AI configuration backup (GNAI-001 milestone 4). Backup
|
||||
# execution is the oneshot service; the timer schedules it weekly.
|
||||
#
|
||||
# Usage:
|
||||
# ansible-playbook -i inventories/<any-env> playbooks/backup.yml \
|
||||
# --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
||||
- name: Deploy the AI configuration backup
|
||||
hosts: llm_servers
|
||||
become: true
|
||||
gather_facts: false
|
||||
|
||||
roles:
|
||||
- role: ai_backup
|
||||
@@ -0,0 +1,50 @@
|
||||
# ai_backup
|
||||
|
||||
Configuration backup for the AI server (GNAI-001 milestone 4). Deploys
|
||||
`gnai-ai-backup` (script), a oneshot service and a weekly timer
|
||||
(Sunday 03:30, `Persistent=true`). Target is the existing NFS pipeline
|
||||
share (`pve:/media/datenhdd3/Encoding`, operator decision 06.11.2026).
|
||||
|
||||
## Backup scope
|
||||
|
||||
- `/etc/ghostnet-ai` (all environment config trees)
|
||||
- `/etc/containers/systemd/users/{980,981,984}` (rootless quadlets,
|
||||
included when present)
|
||||
- `/var/lib/ghostnet-ai/environment-state` (switch state)
|
||||
- `/var/lib/ghostnet-ai/testing/bench/results` (benchmark artifacts)
|
||||
- `/var/lib/ghostnet-ai/shared/models` **without `*.gguf`** — the pinned
|
||||
`SOURCE.yml` manifests make the weights reproducible through the
|
||||
model_store role (`--tags models` re-downloads and verifies them)
|
||||
|
||||
Every archive gets a `.sha256` sidecar; the archive is integrity-checked
|
||||
before it replaces the last good file; retention keeps the newest
|
||||
`ai_backup_retention` archives.
|
||||
|
||||
## Operations
|
||||
|
||||
- Run a backup now: `systemctl start gnai-ai-backup.service`
|
||||
(the timer does this weekly; `systemctl list-timers gnai-ai-backup*`
|
||||
shows the next run).
|
||||
- Watch: `journalctl -u gnai-ai-backup.service -n 20`.
|
||||
|
||||
## Restore SOP
|
||||
|
||||
1. Identify the archive on the share:
|
||||
`ls -lt /var/lib/ghostnet-ai-backup/ghostnet-ai-backup-*.tar.gz`
|
||||
2. Verify the sidecar: `sha256sum -c <archive>.sha256`
|
||||
3. Extract to a staging directory first (never extract blindly over
|
||||
live files):
|
||||
`mkdir /tmp/gnai-restore && tar -xzf <archive> -C /tmp/gnai-restore`
|
||||
4. Compare or selectively copy back the affected tree, e.g.
|
||||
`diff -r /tmp/gnai-restore/etc/ghostnet-ai/testing /etc/ghostnet-ai/testing`
|
||||
5. For environment-state restores, run
|
||||
`/usr/local/sbin/gnai-environment-switch recover` afterwards so the
|
||||
runtime converges with the restored state.
|
||||
6. For model weights: restore via the model_store role
|
||||
(`--tags models`), then verify the `SOURCE.yml` checksums.
|
||||
7. Re-run the affected environment playbook
|
||||
(`playbooks/ai.yml --tags service`) and verify health.
|
||||
|
||||
Accepted proof (06.11.2026): full-tree extract-and-diff against the
|
||||
live configuration and a practical restore of the deleted environment
|
||||
state file, documented in GNAI-008 section 14.6.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
# Configuration backup for the AI server (GNAI-001 milestone 4:
|
||||
# "Backupumfang, Ziel, Aufbewahrung und Restore-SOP festlegen").
|
||||
#
|
||||
# Target: the existing NFS pipeline share (operator decision 06.11.2026:
|
||||
# reuse the encoding queue connection, pve:/media/datenhdd3/Encoding),
|
||||
# mounted at /var/lib/ghostnet-pipeline. Model weights are NOT archived;
|
||||
# they stay reproducible through the pinned SOURCE.yml manifests included
|
||||
# in every archive plus re-verification after download.
|
||||
|
||||
ai_backup_nfs_mountpoint: /var/lib/ghostnet-pipeline
|
||||
ai_backup_target_dir: /var/lib/ghostnet-ai-backup
|
||||
|
||||
# Retention: number of weekly archives kept on the share.
|
||||
ai_backup_retention: 8
|
||||
|
||||
# Schedule: weekly, Sunday 03:30, catching up after downtime.
|
||||
ai_backup_on_calendar: "Sun *-*-* 03:30:00"
|
||||
ai_backup_persistent: true
|
||||
|
||||
ai_backup_service: gnai-ai-backup
|
||||
ai_backup_script: /usr/local/sbin/gnai-ai-backup
|
||||
|
||||
# Backed-up trees (tar paths relative to /). Quadlet dirs are added by
|
||||
# the script only when they exist.
|
||||
ai_backup_paths:
|
||||
- etc/ghostnet-ai
|
||||
- etc/containers/systemd/users/980
|
||||
- etc/containers/systemd/users/981
|
||||
- etc/containers/systemd/users/984
|
||||
- var/lib/ghostnet-ai/environment-state
|
||||
- var/lib/ghostnet-ai/testing/bench/results
|
||||
|
||||
# Included without its multi-gigabyte weights: model manifests and any
|
||||
# other metadata below the shared model store.
|
||||
ai_backup_model_store_meta: var/lib/ghostnet-ai/shared/models
|
||||
ai_backup_model_store_exclude: --exclude=*.gguf
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
- name: Verify the backup script syntax
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /bin/bash
|
||||
- -n
|
||||
- "{{ ai_backup_script }}"
|
||||
register: ai_backup_syntax
|
||||
changed_when: false
|
||||
failed_when: ai_backup_syntax.rc != 0
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Reload the systemd daemon
|
||||
ansible.builtin.systemd_service:
|
||||
daemon_reload: true
|
||||
when: not ansible_check_mode
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
# Deploys the AI configuration backup (GNAI-001 milestone 4): script,
|
||||
# system service and weekly timer targeting the NFS pipeline share.
|
||||
# Running the backup itself is a one-shot service invocation, not part
|
||||
# of the role (see the role README for backup and restore procedures).
|
||||
|
||||
- name: Verify the backup declarations
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ai_backup_nfs_mountpoint is match("^/")
|
||||
- ai_backup_target_dir is match("^/")
|
||||
- ai_backup_retention | int >= 1
|
||||
- ai_backup_paths | length > 0
|
||||
- ai_backup_service | length > 0
|
||||
- ai_backup_script is match("^/usr/local/sbin/")
|
||||
fail_msg: AI backup declarations are incomplete or invalid.
|
||||
|
||||
- name: Require the NFS pipeline share to be mounted
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/mountpoint
|
||||
- -q
|
||||
- "{{ ai_backup_nfs_mountpoint }}"
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
register: ai_backup_share_mounted
|
||||
|
||||
- name: Verify the NFS pipeline share is mounted
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ai_backup_share_mounted.rc == 0
|
||||
fail_msg: >-
|
||||
The NFS share {{ ai_backup_nfs_mountpoint }} is not mounted; bring
|
||||
up the pipeline mount before deploying the backup.
|
||||
|
||||
- name: Ensure the backup target directory exists on the share
|
||||
ansible.builtin.file:
|
||||
path: "{{ ai_backup_target_dir }}"
|
||||
state: directory
|
||||
mode: "0750"
|
||||
|
||||
- name: Install the backup script
|
||||
ansible.builtin.template:
|
||||
src: gnai-ai-backup.sh.j2
|
||||
dest: "{{ ai_backup_script }}"
|
||||
mode: "0750"
|
||||
notify:
|
||||
- Verify the backup script syntax
|
||||
|
||||
- name: Install the backup service unit
|
||||
ansible.builtin.template:
|
||||
src: gnai-ai-backup.service.j2
|
||||
dest: "/etc/systemd/system/{{ ai_backup_service }}.service"
|
||||
mode: "0644"
|
||||
notify:
|
||||
- Reload the systemd daemon
|
||||
|
||||
- name: Install the backup timer unit
|
||||
ansible.builtin.template:
|
||||
src: gnai-ai-backup.timer.j2
|
||||
dest: "/etc/systemd/system/{{ ai_backup_service }}.timer"
|
||||
mode: "0644"
|
||||
notify:
|
||||
- Reload the systemd daemon
|
||||
|
||||
- name: Enable and start the backup timer
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ ai_backup_service }}.timer"
|
||||
enabled: true
|
||||
state: started
|
||||
when: not ansible_check_mode
|
||||
@@ -0,0 +1,13 @@
|
||||
# {{ ansible_managed }}
|
||||
[Unit]
|
||||
# GNAI-001 M4 configuration backup; target is the NFS pipeline share.
|
||||
Description=Ghostnet AI configuration backup
|
||||
After=network-online.target var-lib-ghostnet-pipeline.mount
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart={{ ai_backup_script }}
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/bin/bash
|
||||
# gnai-ai-backup — configuration backup for the AI server (GNAI-001 M4).
|
||||
#
|
||||
# Archives the configuration and small stateful assets to the NFS
|
||||
# pipeline share, verifies the archive integrity, writes a .sha256
|
||||
# sidecar and rotates the retained archives. Model weights are not
|
||||
# archived; the pinned SOURCE.yml manifests inside the archive make
|
||||
# them reproducible (role model_store, --tags models).
|
||||
#
|
||||
# Exit codes: 0 ok; 21 NFS share not mounted; 22 archive integrity
|
||||
# failure; other non-zero failures from tar/sha256.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_DIR="{{ ai_backup_target_dir }}"
|
||||
RETENTION={{ ai_backup_retention }}
|
||||
MOUNTPOINT="{{ ai_backup_nfs_mountpoint }}"
|
||||
MODEL_STORE_META="{{ ai_backup_model_store_meta }}"
|
||||
MODEL_STORE_EXCLUDE="{{ ai_backup_model_store_exclude }}"
|
||||
|
||||
mountpoint_q() { mountpoint -q "$1"; }
|
||||
|
||||
log() { echo "gnai-ai-backup: $1"; }
|
||||
|
||||
mountpoint_q "$MOUNTPOINT" || {
|
||||
log "ERROR: NFS share $MOUNTPOINT is not mounted" >&2
|
||||
exit 21
|
||||
}
|
||||
|
||||
# Existing paths only (tar fails on missing members).
|
||||
paths=()
|
||||
{% for p in ai_backup_paths %}
|
||||
[ -e "/{{ p }}" ] && paths+=("{{ p }}")
|
||||
{% endfor %}
|
||||
[ -d "/$MODEL_STORE_META" ] && paths+=("${MODEL_STORE_EXCLUDE}" "$MODEL_STORE_META")
|
||||
|
||||
mkdir -p "$TARGET_DIR"
|
||||
|
||||
STAMP=$(date +%Y%m%d-%H%M)
|
||||
ARCHIVE="$TARGET_DIR/ghostnet-ai-backup-$STAMP.tar.gz"
|
||||
TMP="$ARCHIVE.tmp"
|
||||
|
||||
tar -C / -czf "$TMP" "${paths[@]}"
|
||||
|
||||
if ! tar -tzf "$TMP" > /dev/null 2>&1; then
|
||||
log "ERROR: archive failed the integrity check" >&2
|
||||
rm -f "$TMP"
|
||||
exit 22
|
||||
fi
|
||||
|
||||
mv "$TMP" "$ARCHIVE"
|
||||
sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
|
||||
|
||||
# Rotate: keep the newest $RETENTION archives with their sidecars.
|
||||
ls -1t "$TARGET_DIR"/ghostnet-ai-backup-*.tar.gz 2>/dev/null \
|
||||
| tail -n +"$(( RETENTION + 1 ))" \
|
||||
| while read -r old; do
|
||||
rm -f "$old" "$old.sha256"
|
||||
done
|
||||
|
||||
log "backup complete: $ARCHIVE ($(du -m "$ARCHIVE" | cut -f1) MB, $(tar -tzf "$ARCHIVE" | wc -l) members)"
|
||||
@@ -0,0 +1,10 @@
|
||||
# {{ ansible_managed }}
|
||||
[Unit]
|
||||
Description=Ghostnet AI configuration backup timer
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ ai_backup_on_calendar }}
|
||||
Persistent={{ 'true' if ai_backup_persistent else 'false' }}
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -49,3 +49,12 @@ ai_environment_health_interval: 5
|
||||
ai_environment_recovery_service: gnai-environment-recovery
|
||||
ai_environment_recovery_timer: gnai-environment-recovery
|
||||
ai_environment_recovery_timer_interval: "1 min"
|
||||
|
||||
# Availability metric for the monitoring stack (GNMON-001 M5): a oneshot
|
||||
# service writes a node_exporter textfile exposing the sanctioned active
|
||||
# environment and whether its engine unit is actually running. Alert
|
||||
# rule for the suite (vmalert): gn_ai_active_env == 0 for 2m, critical.
|
||||
ai_environment_metrics_service: gnai-environment-metrics
|
||||
ai_environment_metrics_timer: gnai-environment-metrics
|
||||
ai_environment_metrics_interval: "30 s"
|
||||
ai_environment_textfile_dir: /var/lib/gn-monitor/textfile
|
||||
|
||||
@@ -20,6 +20,17 @@
|
||||
failed_when: ai_environment_switch_syntax.rc != 0
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Verify the environment metrics script syntax
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /bin/bash
|
||||
- -n
|
||||
- "{{ ai_environment_switch_script }}-metrics"
|
||||
register: ai_environment_metrics_syntax
|
||||
changed_when: false
|
||||
failed_when: ai_environment_metrics_syntax.rc != 0
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Reload the systemd daemon
|
||||
ansible.builtin.systemd_service:
|
||||
daemon_reload: true
|
||||
|
||||
@@ -104,6 +104,37 @@
|
||||
state: started
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Install the environment availability metric writer
|
||||
ansible.builtin.template:
|
||||
src: gnai-environment-metrics.sh.j2
|
||||
dest: "{{ ai_environment_switch_script }}-metrics"
|
||||
mode: "0750"
|
||||
notify:
|
||||
- Verify the environment metrics script syntax
|
||||
|
||||
- name: Install the environment availability metric service
|
||||
ansible.builtin.template:
|
||||
src: gnai-environment-metrics.service.j2
|
||||
dest: "/etc/systemd/system/{{ ai_environment_metrics_service }}.service"
|
||||
mode: "0644"
|
||||
notify:
|
||||
- Reload the systemd daemon
|
||||
|
||||
- name: Install the environment availability metric timer
|
||||
ansible.builtin.template:
|
||||
src: gnai-environment-metrics.timer.j2
|
||||
dest: "/etc/systemd/system/{{ ai_environment_metrics_timer }}.timer"
|
||||
mode: "0644"
|
||||
notify:
|
||||
- Reload the systemd daemon
|
||||
|
||||
- name: Enable and start the environment availability metric timer
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ ai_environment_metrics_timer }}.timer"
|
||||
enabled: true
|
||||
state: started
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Verify the environment switch script answers a status query
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
# {{ ansible_managed }}
|
||||
[Unit]
|
||||
# Writes the gn_ai_active_env textfile metric for the monitoring stack.
|
||||
Description=Ghostnet AI environment availability metric
|
||||
After=multi-user.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart={{ ai_environment_switch_script }}-metrics
|
||||
User=root
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
#!/bin/sh
|
||||
# gnai-environment-metrics — node_exporter textfile writer (GNMON-001).
|
||||
#
|
||||
# Exposes gn_ai_active_env{env}: 1 when the state-active environment's
|
||||
# engine unit is running, 0 otherwise (no state, unknown active
|
||||
# environment or the unit is down). Alert suggestion for the suite:
|
||||
# gn_ai_active_env == 0 for 2m, severity critical.
|
||||
|
||||
set -u
|
||||
cd /
|
||||
|
||||
CONFIG="{{ ai_environment_switch_config }}"
|
||||
STATE="{{ ai_environment_state_file }}"
|
||||
OUT="{{ ai_environment_textfile_dir }}/ai-environment.prom"
|
||||
TMP="$OUT.tmp"
|
||||
|
||||
# shellcheck disable=SC1090
|
||||
[ -r "$CONFIG" ] && . "$CONFIG" || exit 0
|
||||
|
||||
active=""; running=""
|
||||
if [ -r "$STATE" ]; then
|
||||
active=$(sed -n 's/^active=//p' "$STATE" | head -n1)
|
||||
fi
|
||||
|
||||
if [ -n "$active" ]; then
|
||||
unit=$(eval "echo \${${active}_active_unit-}")
|
||||
if [ -n "$unit" ]; then
|
||||
uid=$(eval "echo \${${active}_uid-}")
|
||||
account=$(eval "echo \${${active}_account-}")
|
||||
runuser -u "$account" -- \
|
||||
env HOME="$(getent passwd "$account" | cut -d: -f6)" \
|
||||
XDG_RUNTIME_DIR="/run/user/$uid" \
|
||||
systemctl --user is-active "$unit.service" >/dev/null 2>&1 \
|
||||
&& running=1
|
||||
fi
|
||||
fi
|
||||
|
||||
mkdir -p "{{ ai_environment_textfile_dir }}"
|
||||
{
|
||||
echo "# HELP gn_ai_active_env 1 when the state-active AI environment engine unit is running"
|
||||
echo "# TYPE gn_ai_active_env gauge"
|
||||
echo "gn_ai_active_env{env=\"${active:-unknown}\"} $([ -n "$running" ] && echo 1 || echo 0)"
|
||||
} > "$TMP"
|
||||
mv "$TMP" "$OUT"
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
# {{ ansible_managed }}
|
||||
[Unit]
|
||||
Description=Ghostnet AI environment availability metric timer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=1 min
|
||||
OnUnitActiveSec={{ ai_environment_metrics_interval }}
|
||||
AccuracySec=10s
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user