Neue Rolle benchmark (GNAI-009 5.1/5.2): Egress-Netzwerk-Quadlet gnai-testing-bench (dokumentierte Ausnahme nur fuer Datensatz-Downloads), persistentes results/- und HF-Cache-Verzeichnis, gepinntes Image gnai-bench:lm-eval-0.4.13 (python:3.13-slim per Digest, lm-eval[api]-Pin, bewusst ohne Torch), Runner-Skript mit Network-Unit-Absicherung. Live-Funde aus dem Smoke-Test (2 AIME-Samples gegen Swift 1.5): - lokale Chat-Completions brauchen --apply_chat_template (Server wendet sein --jinja-Template an, kein lokaler Tokenizer noetig) - Request-Timeout 1800 s statt lm-eval-Default 300 s (xhigh-Thinking bei ~18 t/s) - Quadlet-Netzwerke materialisieren erst mit gestarteter Unit; Rolle und Runner starten sie explizit - Ergebnis: exact_match 1.0, Per-Sample-Logs in results/; Tokenzaehlung erfolgt nachtraglich exakt ueber die /tokenize-Endpunkte der Engine Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2.2 KiB
benchmark
Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.
What it creates
gnai-testing-bench.networkquadlet: egress network, deliberately notInternal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal./var/lib/ghostnet-ai/testing/bench/withresults/andhf/(persistent HF dataset cache) owned by the workload account.Containerfile+ locally built, pinned imagegnai-bench:lm-eval-0.4.13(python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).run-bench.shrunner.
Usage
# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24
# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval
The runner attaches the container to both the internal engine network
(http://llama-cpp:8080) and the bench network, writes lm-eval results
with --log_samples under results/<profile>-<timestamp>/ and reuses the
persistent HF cache.
Token accounting
llama-server reports usage.completion_tokens including reasoning tokens
(no separate reasoning_tokens field; verified 02.11.2026). Completion
tokens are therefore the exact generated-token count per sample and form
the basis of the thinking-efficiency metric (score per median generated
tokens, GNAI-009 5.2).
Notes
- The engine profile under test is whatever is currently active; switch
profiles with the
inference_enginerole first (-e inference_engine_active_profile=...). - The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.