GNAI-008/M4: Exklusive Umschaltmechanik ai_environment_switch (7.3)

Rolle deployt das privilegierte Host-Skript gnai-environment-switch
(flock-Lock, Stop-vor-Start, GPU-Freigabenachweis je UID, Health-Warte,
atomare Zustandsdatei), Environment-Mapping, Recovery-Service und
Recovery-Timer (1 Min): Boot-Wiederanlauf und Rueckfall auf Production
nach abgelaufenem Wartungsfenster unabhaengig von CI. Non-Fallback-
Wechsel verlangen ein Wartungsfenster. Ausfuehrung ueber
playbooks/switch-environment.yml.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
2026-11-06 06:51:16 +01:00
parent 4653fb573a
commit 6664344b44
9 changed files with 607 additions and 0 deletions
@@ -0,0 +1,63 @@
---
# Performs the exclusive AI environment switch (GNAI-008 section 7.3).
#
# Usage:
# ansible-playbook -i inventories/<any-env> playbooks/switch-environment.yml \
# -e ai_environment_target=production \
# --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
#
# Non-fallback targets require a maintenance window in minutes:
# -e ai_environment_target=testing -e ai_environment_window_minutes=180
#
# The inventory only provides the connection; the environment mapping
# lives in the ai_environment_switch role defaults. The recovery service
# and timer return to the fallback automatically once a window expired,
# independently of this playbook or any CI runner.
- name: Perform the exclusive AI environment switch
hosts: llm_servers
become: true
gather_facts: false
roles:
- role: ai_environment_switch
tasks:
- name: Require a switch target
ansible.builtin.assert:
that:
- ai_environment_target is defined
- ai_environment_target in ai_environment_definitions
fail_msg: >-
Set -e ai_environment_target=production|testing (must be a known
environment).
- name: Require a maintenance window for non-fallback targets
ansible.builtin.assert:
that:
- ai_environment_target == ai_environment_fallback
or ai_environment_window_minutes is defined
fail_msg: >-
Switching to {{ ai_environment_target | default('<unset>') }}
requires -e ai_environment_window_minutes=<minutes>.
- name: Flush role handlers before switching
ansible.builtin.meta: flush_handlers
- name: Execute the environment switch
ansible.builtin.command:
argv: >-
{{ [ai_environment_switch_script, 'switch-to', ai_environment_target]
+ ([ai_environment_window_minutes | string]
if (ai_environment_window_minutes is defined
and ai_environment_window_minutes | string | length > 0)
else []) }}
register: ai_environment_switch_result
changed_when: >-
'switched to' in ai_environment_switch_result.stdout
or 'already active and running' in ai_environment_switch_result.stdout
failed_when: ai_environment_switch_result.rc != 0
- name: Report the switch result
ansible.builtin.debug:
msg: "{{ ai_environment_switch_result.stdout_lines
+ ai_environment_switch_result.stderr_lines }}"
@@ -0,0 +1,25 @@
# ai_environment_switch
Exclusive switch between the AI environments (GNAI-008 section 7.3,
GNAI-001 milestone 4). Deploys:
- `gnai-environment-switch` (privileged host script): flock-locked
transitions with stop-before-start, GPU release proof (`/dev/kfd`,
render nodes per environment UID), health wait and atomic state file
(`/var/lib/ghostnet-ai/environment-state`).
- `/etc/ghostnet-ai/environment-switch.conf`: the environment mapping
(accounts, UIDs, unit prefixes, active units) from `defaults/main.yml`.
- `gnai-environment-recovery.service` + `.timer`: idempotent
reconciliation — after host boots it brings the state-active
environment back up; every minute it returns to the fallback
(`production`) once a maintenance window has expired. This covers
runner aborts and lost CI without production secrets in any CI path.
Subcommands of the script: `status`, `switch-to <env> [window-minutes]`
and `recover`. Non-fallback switches require a maintenance window and
are validated; the window end is recorded in the state file.
The switch itself is executed via `playbooks/switch-environment.yml`
(`-e ai_environment_target=… [-e ai_environment_window_minutes=…]`).
Unit names and accounts must stay in sync with the per-environment
`inference_engine` group_vars.
@@ -0,0 +1,51 @@
---
# Exclusive AI environment switch (GNAI-008 section 7.3).
#
# A privileged host script performs the switch (lock, stop-before-start,
# GPU release proof, timeouts, recovery); this role deploys the script,
# its environment mapping, the recovery service and the recovery timer.
# The state file is the queryable status source for external callers
# (fleet tooling, future CI).
ai_environment_switch_script: /usr/local/sbin/gnai-environment-switch
ai_environment_switch_config: /etc/ghostnet-ai/environment-switch.conf
ai_environment_state_file: /var/lib/ghostnet-ai/environment-state
ai_environment_state_dir: /var/lib/ghostnet-ai
# Environment mapping (accounts, unit prefixes and the desired active unit
# per environment as of the latest deployment). Unit prefixes follow the
# inference_engine group_vars of each environment.
ai_environment_definitions:
production:
account: gnai-production
uid: 980
unit_prefix: gnai-production-llama
active_unit: gnai-production-llama-swift15
testing:
account: gnai-testing
uid: 981
unit_prefix: gnai-testing-llama
active_unit: gnai-testing-llama-swift15
# Which environment is the stable fallback (GNAI-008 section 7.3): the
# recovery service starts it after host boots and the recovery timer
# returns to it once a maintenance window has expired.
ai_environment_fallback: production
# Timeouts for one switch operation, in seconds.
ai_environment_stop_timeout: 180
ai_environment_gpu_release_timeout: 60
ai_environment_start_timeout: 240
ai_environment_health_timeout: 300
ai_environment_health_interval: 5
# Maintenance windows: a switch to a non-fallback environment records a
# window end timestamp in the state file; the recovery timer returns to
# the fallback automatically once the window has expired. The window
# length is provided per invocation by the caller (playbook/CI), not
# defaulted here.
# Recovery service/timer (M4 recovery: host boot and aborted runs).
ai_environment_recovery_service: gnai-environment-recovery
ai_environment_recovery_timer: gnai-environment-recovery
ai_environment_recovery_timer_interval: "1 min"
@@ -0,0 +1,35 @@
---
- name: Verify the environment switch configuration
ansible.builtin.command:
argv:
- /bin/bash
- -n
- "{{ ai_environment_switch_config }}"
changed_when: false
failed_when: false
when: not ansible_check_mode
- name: Verify the environment script syntax
ansible.builtin.command:
argv:
- /bin/bash
- -n
- "{{ ai_environment_switch_script }}"
register: ai_environment_switch_syntax
changed_when: false
failed_when: ai_environment_switch_syntax.rc != 0
when: not ansible_check_mode
- name: Reload the systemd daemon
ansible.builtin.systemd_service:
daemon_reload: true
when: not ansible_check_mode
- name: Restart the environment recovery units
ansible.builtin.systemd_service:
name: "{{ item }}"
state: restarted
loop:
- "{{ ai_environment_recovery_service }}.service"
- "{{ ai_environment_recovery_timer }}.timer"
when: not ansible_check_mode
@@ -0,0 +1,101 @@
---
# Deploys the exclusive AI environment switch (GNAI-008 section 7.3):
# the privileged host script, its environment mapping, the recovery
# service and the recovery timer. The switch itself is executed through
# playbooks/switch-environment.yml.
- name: Verify the environment switch declarations
ansible.builtin.assert:
that:
- ai_environment_fallback in ai_environment_definitions
- ai_environment_definitions | length >= 2
- >-
ai_environment_definitions.values() | map(attribute='account') | unique
| list | length == ai_environment_definitions | length
- >-
ai_environment_definitions.values() | map(attribute='uid') | unique
| list | length == ai_environment_definitions | length
- >-
ai_environment_definitions.values() | map(attribute='unit_prefix')
| unique | list | length == ai_environment_definitions | length
- ai_environment_gpu_release_timeout | int > 0
- ai_environment_start_timeout | int > 0
- ai_environment_health_timeout | int > 0
- ai_environment_health_interval | int > 0
fail_msg: AI environment switch declarations are incomplete or ambiguous.
- name: Verify every environment account exists on the host
ansible.builtin.getent:
database: passwd
key: "{{ item.value.account }}"
loop: "{{ ai_environment_definitions | dict2items }}"
loop_control:
label: "{{ item.value.account }}"
- name: Verify the account UID matches the declaration
ansible.builtin.assert:
that:
- getent_passwd[item.value.account][1] | int == item.value.uid | int
fail_msg: >-
UID mismatch for {{ item.value.account }}: declared {{ item.value.uid }},
host reports {{ getent_passwd[item.value.account][1] }}.
loop: "{{ ai_environment_definitions | dict2items }}"
loop_control:
label: "{{ item.value.account }}"
- name: Ensure the host configuration directory exists
ansible.builtin.file:
path: "{{ ai_environment_switch_config | dirname }}"
state: directory
mode: "0755"
- name: Install the environment switch configuration
ansible.builtin.template:
src: environment-switch.conf.j2
dest: "{{ ai_environment_switch_config }}"
mode: "0644"
notify:
- Verify the environment switch configuration
- name: Install the environment switch script
ansible.builtin.template:
src: gnai-environment-switch.sh.j2
dest: "{{ ai_environment_switch_script }}"
mode: "0750"
notify:
- Verify the environment script syntax
- name: Install the environment recovery service
ansible.builtin.template:
src: gnai-environment-recovery.service.j2
dest: "/etc/systemd/system/{{ ai_environment_recovery_service }}.service"
mode: "0644"
notify:
- Reload the systemd daemon
- Restart the environment recovery units
- name: Install the environment recovery timer
ansible.builtin.template:
src: gnai-environment-recovery.timer.j2
dest: "/etc/systemd/system/{{ ai_environment_recovery_timer }}.timer"
mode: "0644"
notify:
- Reload the systemd daemon
- Restart the environment recovery units
- name: Enable and start the environment recovery timer
ansible.builtin.systemd_service:
name: "{{ ai_environment_recovery_timer }}.timer"
enabled: true
state: started
when: not ansible_check_mode
- name: Verify the environment switch script answers a status query
ansible.builtin.command:
argv:
- "{{ ai_environment_switch_script }}"
- status
register: ai_environment_switch_status
changed_when: false
failed_when: ai_environment_switch_status.rc != 0
when: not ansible_check_mode
@@ -0,0 +1,16 @@
# {{ ansible_managed }}
# Environment mapping for gnai-environment-switch (GNAI-008 7.3).
# Deployed by the ai_environment_switch role; do not edit on the host.
ENVIRONMENTS="{% for env in ai_environment_definitions.keys() | sort %}{{ env }}{% if not loop.last %} {% endif %}{% endfor %}"
FALLBACK="{{ ai_environment_fallback | default('production') }}"
TIMEOUT_GPU_RELEASE={{ ai_environment_gpu_release_timeout }}
TIMEOUT_START_AND_HEALTH={{ [ai_environment_start_timeout, ai_environment_health_timeout] | max }}
HEALTH_INTERVAL={{ ai_environment_health_interval }}
{% for env, definition in ai_environment_definitions.items() %}
{{ env }}_account={{ definition.account }}
{{ env }}_uid={{ definition.uid }}
{{ env }}_unit_prefix={{ definition.unit_prefix }}
{{ env }}_active_unit={{ definition.active_unit }}
{% endfor %}
@@ -0,0 +1,12 @@
# {{ ansible_managed }}
[Unit]
# M4 recovery path (GNAI-008 7.3): reconciles the AI environment state
# after host boots and aborted runs without any CI involvement. The
# timer {{ ai_environment_recovery_timer }}.timer calls the same
# operation every minute (expired windows return to the fallback).
Description=Ghostnet AI environment recovery
After=multi-user.target
[Service]
Type=oneshot
ExecStart={{ ai_environment_switch_script }} recover
@@ -0,0 +1,13 @@
# {{ ansible_managed }}
[Unit]
# Periodic reconciliation of the AI environment state; see the
# gnai-environment-recovery service unit for the operation.
Description=Ghostnet AI environment recovery timer
[Timer]
OnBootSec=2 min
OnUnitActiveSec={{ ai_environment_recovery_timer_interval }}
AccuracySec=30s
[Install]
WantedBy=timers.target
@@ -0,0 +1,291 @@
#!/bin/bash
# gnai-environment-switch — exclusive AI environment switch
# (GNAI-008 section 7.3).
#
# Performs the ordered environment transition: takes an exclusive lock,
# stops the running environment (stop-before-start), proves the GPU
# release, starts the target environment's active unit and waits for its
# health endpoint. Every non-fallback target records a maintenance window
# end in the state file; the recovery service and timer return to the
# fallback once that window has expired.
#
# Subcommands:
# status print the state file and the actually
# running units per environment
# switch-to <env> [minutes] switch to <env>; [minutes] records a
# maintenance window (required and validated
# for non-fallback targets)
# recover idempotent reconciliation: start the state
# active environment when it is down; return
# to the fallback once its window expired
#
# Exit codes: 0 ok; 1 usage error; 2 unknown environment or config;
# 10 switch already in progress (locked); 11 stop or GPU release timeout;
# 12 start or health timeout.
set -u
cd /
CONFIG="{{ ai_environment_switch_config }}"
STATE="{{ ai_environment_state_file }}"
LOCK_DIR=/run/ghostnet-ai
LOCK="$LOCK_DIR/environment-switch.lock"
# shellcheck source=/dev/null
[ -r "$CONFIG" ] || { echo "config missing: $CONFIG" >&2; exit 2; }
# shellcheck disable=SC1090
. "$CONFIG"
: "${ENVIRONMENTS:?}" "${FALLBACK:?}" "${TIMEOUT_GPU_RELEASE:?}" \
"${TIMEOUT_START_AND_HEALTH:?}" "${HEALTH_INTERVAL:?}"
env_value() {
# $1 = environment, $2 = key; prints <env>_<key> from the config
local name="${1}_${2}"
echo "${!name-}"
}
epoch() { date +%s; }
write_state() {
# $1.. = key=value pairs (atomic write)
local tmp="$STATE.tmp"
: > "$tmp"
printf '# managed by gnai-environment-switch\n' >> "$tmp"
for kv in "$@"; do printf '%s\n' "$kv" >> "$tmp"; done
mv "$tmp" "$STATE"
}
state_get() {
# $1 = key; prints its value or empty
sed -n "s/^$1=//p" "$STATE" 2>/dev/null | head -n1
}
as_env() {
# $1 = environment, rest = command; runs as the environment account
local account uid home
account=$(env_value "$1" account)
uid=$(env_value "$1" uid)
home=$(getent passwd "$account" | cut -d: -f6)
runuser -u "$account" -- \
env HOME="$home" XDG_RUNTIME_DIR="/run/user/$uid" "${@:2}"
}
running_units() {
# $1 = environment; prints the running engine unit names
local status
status=$(as_env "$1" systemctl --user is-active \
"$(env_value "$1" active_unit).service" 2>/dev/null) || status=inactive
if [ "$status" = "active" ]; then
echo "$(env_value "$1" active_unit).service"
fi
}
gpu_holders() {
# $1 = uid; prints PIDs of that uid holding /dev/kfd or a render node
local pid fd target
for pid in /proc/[0-9]*; do
[ "$(stat -Lc %u "$pid" 2>/dev/null)" = "$1" ] || continue
for fd in "$pid"/fd/*; do
target=$(readlink "$fd" 2>/dev/null) || continue
case "$target" in
/dev/kfd|/dev/dri/*) echo "${pid##*/}"; break ;;
esac
done
done | sort -u
}
wait_gpu_release() {
# $1 = uid, $2 = timeout seconds
local waited=0
while [ "$waited" -lt "$2" ]; do
[ -z "$(gpu_holders "$1")" ] && return 0
sleep 5
waited=$((waited + 5))
done
echo "GPU release timeout for uid $1 (holders: $(gpu_holders "$1" | tr '\n' ' '))" >&2
return 11
}
wait_health() {
# $1 = environment, $2 = timeout seconds, $3 = interval seconds
local waited=0
while [ "$waited" -lt "$2" ]; do
if as_env "$1" podman exec "$(env_value "$1" active_unit)" \
sh -c "curl -fsS -o /dev/null http://127.0.0.1:8080/health" \
>/dev/null 2>&1; then
return 0
fi
sleep "$3"
waited=$((waited + $3))
done
return 12
}
lock_or_exit() {
mkdir -p "$LOCK_DIR"
exec 9>"$LOCK"
flock -n 9 || { echo "environment switch already in progress" >&2; exit 10; }
}
stop_environment() {
# $1 = environment; ordered stop of every running engine unit
local unit
for unit in $(as_env "$1" systemctl --user list-units \
"$(env_value "$1" unit_prefix)-*.service" --no-legend --plain \
2>/dev/null | awk '{print $1}'); do
as_env "$1" systemctl --user stop "$unit" 2>/dev/null
done
sleep 2
}
start_active_unit() {
# $1 = environment
as_env "$1" systemctl --user start "$(env_value "$1" active_unit).service"
}
switch_to() {
# $1 = target environment, $2 = window minutes (optional)
local target="$1" window_minutes="$2" other uid
local active unit refresh_window=""
case " $ENVIRONMENTS " in *" $target "*) ;; *)
echo "unknown environment: $target" >&2; exit 2 ;;
esac
lock_or_exit
if [ "$target" != "$FALLBACK" ]; then
case "$window_minutes" in
''|*[!0-9]*)
echo "refusing non-fallback switch without a maintenance window" >&2
exit 1 ;;
esac
refresh_window=yes
fi
unit=$(env_value "$target" active_unit)
[ -n "$unit" ] || { echo "no active unit configured for $target" >&2; exit 2; }
active=$(state_get active)
if [ "$active" = "$target" ] && [ -n "$(running_units "$target")" ]; then
# Refresh the maintenance window when the target already runs.
if [ -n "$refresh_window" ]; then
write_state "active=$target" "active_unit=$unit" \
"active_since=$(epoch)" \
"window_until=$(( $(epoch) + window_minutes * 60 ))"
fi
echo "environment $target already active and running (idempotent)"
return 0
fi
# Stop every OTHER environment (stop-before-start), prove GPU release.
for other in $ENVIRONMENTS; do
[ "$other" = "$target" ] && continue
if [ -n "$(running_units "$other")" ]; then
uid=$(env_value "$other" uid)
stop_environment "$other"
wait_gpu_release "$uid" "$TIMEOUT_GPU_RELEASE" || exit 11
fi
done
start_active_unit "$target"
if ! wait_health "$target" "$TIMEOUT_START_AND_HEALTH" "$HEALTH_INTERVAL"; then
echo "target $target failed to become healthy" >&2
if [ "$target" != "$FALLBACK" ]; then
echo "rolling back to $FALLBACK" >&2
stop_environment "$target" 2>/dev/null || true
wait_gpu_release "$(env_value "$target" uid)" \
"$TIMEOUT_GPU_RELEASE" 2>/dev/null || true
start_active_unit "$FALLBACK" 2>/dev/null || true
wait_health "$FALLBACK" "$TIMEOUT_START_AND_HEALTH" \
"$HEALTH_INTERVAL" 2>/dev/null || true
fi
exit 12
fi
if [ -n "$refresh_window" ]; then
write_state "active=$target" "active_unit=$unit" \
"active_since=$(epoch)" \
"window_until=$(( $(epoch) + window_minutes * 60 ))"
else
write_state "active=$target" "active_unit=$unit" \
"active_since=$(epoch)" "window_until="
fi
echo "switched to $target (unit $unit)"
return 0
}
cmd_status() {
local env uid units
echo "state file: $STATE"
if [ -r "$STATE" ]; then sed 's/^/ /' "$STATE"; else echo " (unset)"; fi
for env in $ENVIRONMENTS; do
units=$(running_units "$env")
uid=$(env_value "$env" uid)
echo "runtime: $env uid=$uid units=${units:-none} gpu=$(gpu_holders "$uid" | tr '\n' ' ')"
done
return 0
}
cmd_recover() {
local active window_until now other
lock_or_exit
if [ ! -r "$STATE" ]; then
# No state yet: the fallback is the defined stable state.
if [ -z "$(running_units "$FALLBACK")" ]; then
start_active_unit "$FALLBACK" || return 0
fi
write_state "active=$FALLBACK" \
"active_unit=$(env_value "$FALLBACK" active_unit)" \
"active_since=$(epoch)" "window_until="
return 0
fi
active=$(state_get active)
window_until=$(state_get window_until)
now=$(epoch)
if [ "$active" != "$FALLBACK" ]; then
if [ -z "$window_until" ] || [ "$now" -ge "$window_until" ]; then
echo "maintenance window over, returning to $FALLBACK"
switch_to "$FALLBACK" "" </dev/null || true
return 0
fi
# Inside a valid window: keep the active environment up.
if [ -z "$(running_units "$active")" ]; then
start_active_unit "$active" || true
fi
return 0
fi
# Active is the fallback: ensure it runs, enforce exclusivity.
if [ -z "$(running_units "$FALLBACK")" ]; then
for other in $ENVIRONMENTS; do
[ "$other" = "$FALLBACK" ] && continue
if [ -n "$(running_units "$other")" ]; then
echo "unsanctioned $other running while $FALLBACK should be active; enforcing" >&2
stop_environment "$other"
wait_gpu_release "$(env_value "$other" uid)" \
"$TIMEOUT_GPU_RELEASE" || true
fi
done
start_active_unit "$FALLBACK" || true
fi
return 0
}
main() {
local cmd="${1:-}"; shift || true
case "$cmd" in
status) cmd_status "$@" ;;
switch-to) switch_to "$@" ;;
recover) cmd_recover "$@" ;;
*) echo "usage: $0 {status|switch-to <env> [window-minutes]|recover}" >&2
exit 1 ;;
esac
}
main "$@"