LLM-Server: Benchmark-Harness (lm-eval) aufgesetzt und per Smoke-Test verifiziert

Neue Rolle benchmark (GNAI-009 5.1/5.2): Egress-Netzwerk-Quadlet
gnai-testing-bench (dokumentierte Ausnahme nur fuer Datensatz-Downloads),
persistentes results/- und HF-Cache-Verzeichnis, gepinntes Image
gnai-bench:lm-eval-0.4.13 (python:3.13-slim per Digest, lm-eval[api]-Pin,
bewusst ohne Torch), Runner-Skript mit Network-Unit-Absicherung.

Live-Funde aus dem Smoke-Test (2 AIME-Samples gegen Swift 1.5):
- lokale Chat-Completions brauchen --apply_chat_template (Server wendet
  sein --jinja-Template an, kein lokaler Tokenizer noetig)
- Request-Timeout 1800 s statt lm-eval-Default 300 s (xhigh-Thinking bei
  ~18 t/s)
- Quadlet-Netzwerke materialisieren erst mit gestarteter Unit; Rolle und
  Runner starten sie explizit
- Ergebnis: exact_match 1.0, Per-Sample-Logs in results/; Tokenzaehlung
  erfolgt nachtraglich exakt ueber die /tokenize-Endpunkte der Engine

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
2026-11-02 05:58:58 +01:00
parent 6384f72314
commit c6cabd98ef
8 changed files with 311 additions and 0 deletions
@@ -13,3 +13,9 @@ inference_engine_network_name: gnai-testing-llama
# accepted separately (GNAI-010 section 10.3). 64k bring-ups passed for all
# three profiles (10.5, 10.8, 10.9); null runs the 128k target context.
inference_engine_ctx_size_override: null
# Benchmark harness (GNAI-009 5.1); reuses the workload identity.
benchmark_user: gnai-testing
benchmark_uid: 981
benchmark_user_home: /var/lib/ghostnet-ai/testing
benchmark_engine_network_name: gnai-testing-llama
@@ -0,0 +1,5 @@
- name: Prepare the benchmark harness of one environment
hosts: llm_servers
become: true
roles:
- role: benchmark
@@ -0,0 +1,55 @@
# benchmark
Prepares the lm-evaluation-harness benchmark environment for one KI
environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's
rootless store so the bench container can attach to the internal engine
network.
## What it creates
- `gnai-testing-bench.network` quadlet: egress network, deliberately not
`Internal=true`. Its ONLY purpose is benchmark dataset downloads by the
pinned bench container (documented exception, GNAI-009 5.1); the engine
network stays internal.
- `/var/lib/ghostnet-ai/testing/bench/` with `results/` and `hf/` (persistent
HF dataset cache) owned by the workload account.
- `Containerfile` + locally built, pinned image
`gnai-bench:lm-eval-0.4.13` (python:3.13-slim by digest; the build needs
host egress and takes several minutes the first time because of the
lm-eval dependency chain).
- `run-bench.sh` runner.
## Usage
```bash
# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24
# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval
```
The runner attaches the container to both the internal engine network
(`http://llama-cpp:8080`) and the bench network, writes lm-eval results
with `--log_samples` under `results/<profile>-<timestamp>/` and reuses the
persistent HF cache.
## Token accounting
llama-server reports `usage.completion_tokens` including reasoning tokens
(no separate `reasoning_tokens` field; verified 02.11.2026). Completion
tokens are therefore the exact generated-token count per sample and form
the basis of the thinking-efficiency metric (score per median generated
tokens, GNAI-009 5.2).
## Notes
- The engine profile under test is whatever is currently active; switch
profiles with the `inference_engine` role first (`-e
inference_engine_active_profile=...`).
- The server-side sampling/reasoning configuration comes from the profile
quadlet; the harness deliberately sends no generation overrides.
@@ -0,0 +1,36 @@
---
# Benchmark harness for the LLM evaluation (GNAI-009 sections 5.1/5.2).
# Runs the pinned lm-evaluation-harness against the running engine over the
# internal podman network; a dedicated egress network exists ONLY for the
# benchmark dataset downloads (documented exception, GNAI-009 5.1).
#
# The harness reuses the environment identity from the group_vars
# (inference_engine_user/uid/home/network) and runs in the workload
# account's rootless store so it can attach to the engine network.
benchmark_python_image: docker.io/library/python:3.13-slim@sha256:2d44738ebf4ce5f63b9d9ea871e9c7cfdc0426f7b0c97cac915fd88ded1d86bb
benchmark_lm_eval_version: "0.4.13"
benchmark_image_tag: "gnai-bench:lm-eval-0.4.13"
benchmark_image_force_rebuild: false
benchmark_bench_network_name: gnai-testing-bench
benchmark_quadlet_dir: /etc/containers/systemd/users/981
benchmark_base_dir: /var/lib/ghostnet-ai/testing/bench
benchmark_results_dir: "{{ benchmark_base_dir }}/results"
benchmark_hf_cache_dir: "{{ benchmark_base_dir }}/hf"
# Engine endpoint inside the shared podman network.
benchmark_engine_url: http://llama-cpp:8080/v1
# lm-eval sample/token accounting: llama-server reports completion_tokens
# including reasoning (no separate reasoning_tokens field, verified
# 02.11.2026). completion_tokens is therefore the exact per-task generated
# token count and the basis of the thinking-efficiency metric
# (GNAI-009 section 5.2).
benchmark_task_args:
model: dummy-model-name
base_url: "{{ benchmark_engine_url }}/chat/completions"
num_concurrent: 1
max_retries: 2
tokenized_requests: false
# xhigh thinking on this hardware needs well over the 300 s default.
timeout: 1800
@@ -0,0 +1,151 @@
---
# Prepares the benchmark harness (GNAI-009 5.1/5.2): egress network quadlet,
# persistent result/cache directories, pinned lm-eval image and the runner
# script. Benchmark runs themselves are started manually via the script.
- name: Validate benchmark harness declarations
ansible.builtin.assert:
that:
- benchmark_user | length > 0
- benchmark_uid | int > 0
- benchmark_user_home is match("^/")
- benchmark_python_image is match("^docker\\.io/library/python(:[A-Za-z0-9._-]+)?@sha256:[0-9a-f]{64}$")
- benchmark_lm_eval_version is match("^[0-9]+\\.[0-9]+\\.[0-9]+$")
- benchmark_bench_network_name | length > 0
- benchmark_engine_network_name | length > 0
fail_msg: Benchmark harness declarations are incomplete or ambiguous.
tags:
- bench
- name: Ensure the bench base and result directories exist
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ benchmark_user }}"
group: "{{ benchmark_user }}"
mode: "0750"
loop:
- "{{ benchmark_base_dir }}"
- "{{ benchmark_results_dir }}"
- "{{ benchmark_hf_cache_dir }}"
tags:
- bench
- name: Render the benchmark egress network quadlet
# Deliberately NOT Internal=true: this network exists only for the
# benchmark dataset downloads (GNAI-009 5.1). The engine network stays
# internal; the bench container is the only consumer of this network.
ansible.builtin.template:
src: bench.network.j2
dest: "{{ benchmark_quadlet_dir }}/{{ benchmark_bench_network_name }}.network"
owner: root
group: root
mode: "0644"
tags:
- bench
- name: Render the benchmark Containerfile
ansible.builtin.template:
src: bench.Containerfile.j2
dest: "{{ benchmark_base_dir }}/Containerfile"
owner: "{{ benchmark_user }}"
group: "{{ benchmark_user }}"
mode: "0640"
tags:
- bench
- name: Render the benchmark runner script
ansible.builtin.template:
src: run-bench.sh.j2
dest: "{{ benchmark_base_dir }}/run-bench.sh"
owner: "{{ benchmark_user }}"
group: "{{ benchmark_user }}"
mode: "0750"
tags:
- bench
- name: Reload the workload user manager for the bench network
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ benchmark_user }}"
- --
- /usr/bin/env
- "HOME={{ benchmark_user_home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }}"
- /usr/bin/systemctl
- --user
- daemon-reload
chdir: "{{ benchmark_user_home }}"
changed_when: false
when: not ansible_check_mode
tags:
- bench
- name: Start the bench network unit
# Quadlet networks only materialize when their generated unit runs; the
# bench container is started with plain `podman run` and has no systemd
# dependency, so the unit must be started explicitly.
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ benchmark_user }}"
- --
- /usr/bin/env
- "HOME={{ benchmark_user_home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }}"
- /usr/bin/systemctl
- --user
- start
- "{{ benchmark_bench_network_name }}-network.service"
chdir: "{{ benchmark_user_home }}"
changed_when: false
when: not ansible_check_mode
tags:
- bench
- name: Check whether the pinned bench image is already built
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ benchmark_user }}"
- --
- /usr/bin/env
- "HOME={{ benchmark_user_home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }}"
- /usr/bin/podman
- image
- exists
- "{{ benchmark_image_tag }}"
chdir: "{{ benchmark_user_home }}"
changed_when: false
failed_when: benchmark_image_present.rc not in [0, 1]
check_mode: false
register: benchmark_image_present
tags:
- bench
- name: Build the pinned bench image
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ benchmark_user }}"
- --
- /usr/bin/env
- "HOME={{ benchmark_user_home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }}"
- /usr/bin/podman
- build
- -t
- "{{ benchmark_image_tag }}"
- "{{ benchmark_base_dir }}"
chdir: "{{ benchmark_user_home }}"
when:
- not ansible_check_mode
- benchmark_image_force_rebuild or benchmark_image_present.rc != 0
register: benchmark_image_build
tags:
- bench
@@ -0,0 +1,12 @@
# Benchmark image for the KI evaluation harness (GNAI-009 5.1).
# Base image and lm-eval version are pinned in the benchmark role defaults.
FROM docker.io/library/python:3.13-slim@sha256:2d44738ebf4ce5f63b9d9ea871e9c7cfdc0426f7b0c97cac915fd88ded1d86bb
ENV PIP_NO_CACHE_DIR=1 \
HF_HOME=/hf \
PYTHONUNBUFFERED=1
RUN pip install --no-cache-dir "lm-eval[api]=={{ benchmark_lm_eval_version }}"
WORKDIR /work
ENTRYPOINT ["lm_eval"]
@@ -0,0 +1,6 @@
[Network]
# Benchmark egress network (GNAI-009 section 5.1): exists ONLY for benchmark
# dataset downloads by the pinned lm-eval container. Deliberately not
# Internal. The engine network remains internal.
NetworkName={{ benchmark_bench_network_name }}
Driver=bridge
@@ -0,0 +1,40 @@
#!/bin/bash
# Benchmark runner for one engine profile (GNAI-009 5.1/5.2).
# Usage: run-bench.sh <profile-label> <limit> <task> [<task> ...]
# profile-label: e.g. swift-1.5-qwen3.8-27b-q6k (used for the result dir)
# limit: max samples per task (e.g. 2 for a smoke test, empty for all)
#
# Runs lm-eval inside the pinned bench image, attached to BOTH the internal
# engine network (to reach http://llama-cpp:8080) and the bench network
# (egress, only for benchmark dataset downloads). Results and the HF
# dataset cache persist under the workload home.
set -euo pipefail
PROFILE=$1; shift
LIMIT=$1; shift
TS=$(date +%Y-%m-%d-%H%M)
OUT=/results/"$PROFILE"-"$TS"
# Ensure the bench network unit is up (quadlet networks materialize only
# when their unit runs; this container has no systemd dependency).
/usr/sbin/runuser --user {{ benchmark_user }} -- \
/usr/bin/env HOME={{ benchmark_user_home }} XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }} \
/usr/bin/systemctl --user start {{ benchmark_bench_network_name }}-network.service
ARGS=(--model local-chat-completions
--model_args "model={{ benchmark_task_args.model }},base_url={{ benchmark_task_args.base_url }},num_concurrent={{ benchmark_task_args.num_concurrent }},max_retries={{ benchmark_task_args.max_retries }},tokenized_requests={{ benchmark_task_args.tokenized_requests }},timeout={{ benchmark_task_args.timeout }}"
--apply_chat_template
--tasks "$*"
--log_samples
--output_path "$OUT")
if [ -n "$LIMIT" ]; then
ARGS+=(--limit "$LIMIT")
fi
exec /usr/sbin/runuser --user {{ benchmark_user }} -- \
/usr/bin/env HOME={{ benchmark_user_home }} XDG_RUNTIME_DIR=/run/user/{{ benchmark_uid }} \
/usr/bin/podman run --rm \
--name gnai-bench-run \
--network {{ benchmark_engine_network_name }} \
--network {{ benchmark_bench_network_name }} \
-v {{ benchmark_results_dir }}:/results:Z \
-v {{ benchmark_hf_cache_dir }}:/hf:Z \
{{ benchmark_image_tag }} "${ARGS[@]}"