eval-efficiency.py/.sh in der benchmark-Rolle: liest die Per-Sample-Logs von lm-eval, zaehlt die exakt generierten Tokens je Antwort ueber den /tokenize-Endpunkt der Engine (Qwen-Tokenizer, keine Schaetzung), zieht den Score und aggregiert Score, Median/Mean/Max-Tokens sowie die Effizienzkennzahl Score je Median-Token. Smoke-Test auf den AIME-2-Ergebnissen: Score 1.0, Median 645 generierte Tokens je Aufgabe. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
benchmark
Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.
What it creates
gnai-testing-bench.networkquadlet: egress network, deliberately notInternal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal./var/lib/ghostnet-ai/testing/bench/withresults/andhf/(persistent HF dataset cache) owned by the workload account.Containerfile+ locally built, pinned imagegnai-bench:lm-eval-0.4.13(python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).run-bench.shrunner.
Usage
# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24
# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval
The runner attaches the container to both the internal engine network
(http://llama-cpp:8080) and the bench network, writes lm-eval results
with --log_samples under results/<profile>-<timestamp>/ and reuses the
persistent HF cache.
Token accounting
llama-server reports usage.completion_tokens including reasoning tokens
(no separate reasoning_tokens field; verified 02.11.2026). Completion
tokens are therefore the exact generated-token count per sample and form
the basis of the thinking-efficiency metric (score per median generated
tokens, GNAI-009 5.2).
Notes
- The engine profile under test is whatever is currently active; switch
profiles with the
inference_enginerole first (-e inference_engine_active_profile=...). - The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.