Files
infrastructure/llm-server/llm/ansible/roles/benchmark
spektr 89e42ecd9b LLM-Server: Effizienz-Auswertung fuer Benchmark-Ergebnisse (GNAI-009 5.2)
eval-efficiency.py/.sh in der benchmark-Rolle: liest die Per-Sample-Logs
von lm-eval, zaehlt die exakt generierten Tokens je Antwort ueber den
/tokenize-Endpunkt der Engine (Qwen-Tokenizer, keine Schaetzung), zieht
den Score und aggregiert Score, Median/Mean/Max-Tokens sowie die
Effizienzkennzahl Score je Median-Token. Smoke-Test auf den AIME-2-Ergebnissen:
Score 1.0, Median 645 generierte Tokens je Aufgabe.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-02 06:44:44 +01:00
..

benchmark

Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.

What it creates

  • gnai-testing-bench.network quadlet: egress network, deliberately not Internal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal.
  • /var/lib/ghostnet-ai/testing/bench/ with results/ and hf/ (persistent HF dataset cache) owned by the workload account.
  • Containerfile + locally built, pinned image gnai-bench:lm-eval-0.4.13 (python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).
  • run-bench.sh runner.

Usage

# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24

# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval

The runner attaches the container to both the internal engine network (http://llama-cpp:8080) and the bench network, writes lm-eval results with --log_samples under results/<profile>-<timestamp>/ and reuses the persistent HF cache.

Token accounting

llama-server reports usage.completion_tokens including reasoning tokens (no separate reasoning_tokens field; verified 02.11.2026). Completion tokens are therefore the exact generated-token count per sample and form the basis of the thinking-efficiency metric (score per median generated tokens, GNAI-009 5.2).

Notes

  • The engine profile under test is whatever is currently active; switch profiles with the inference_engine role first (-e inference_engine_active_profile=...).
  • The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.