Files
infrastructure/llm-server/llm/ansible/roles/benchmark/README.md
T
spektr c6cabd98ef LLM-Server: Benchmark-Harness (lm-eval) aufgesetzt und per Smoke-Test verifiziert
Neue Rolle benchmark (GNAI-009 5.1/5.2): Egress-Netzwerk-Quadlet
gnai-testing-bench (dokumentierte Ausnahme nur fuer Datensatz-Downloads),
persistentes results/- und HF-Cache-Verzeichnis, gepinntes Image
gnai-bench:lm-eval-0.4.13 (python:3.13-slim per Digest, lm-eval[api]-Pin,
bewusst ohne Torch), Runner-Skript mit Network-Unit-Absicherung.

Live-Funde aus dem Smoke-Test (2 AIME-Samples gegen Swift 1.5):
- lokale Chat-Completions brauchen --apply_chat_template (Server wendet
  sein --jinja-Template an, kein lokaler Tokenizer noetig)
- Request-Timeout 1800 s statt lm-eval-Default 300 s (xhigh-Thinking bei
  ~18 t/s)
- Quadlet-Netzwerke materialisieren erst mit gestarteter Unit; Rolle und
  Runner starten sie explizit
- Ergebnis: exact_match 1.0, Per-Sample-Logs in results/; Tokenzaehlung
  erfolgt nachtraglich exakt ueber die /tokenize-Endpunkte der Engine

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-02 05:58:58 +01:00

2.2 KiB

benchmark

Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.

What it creates

  • gnai-testing-bench.network quadlet: egress network, deliberately not Internal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal.
  • /var/lib/ghostnet-ai/testing/bench/ with results/ and hf/ (persistent HF dataset cache) owned by the workload account.
  • Containerfile + locally built, pinned image gnai-bench:lm-eval-0.4.13 (python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).
  • run-bench.sh runner.

Usage

# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24

# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval

The runner attaches the container to both the internal engine network (http://llama-cpp:8080) and the bench network, writes lm-eval results with --log_samples under results/<profile>-<timestamp>/ and reuses the persistent HF cache.

Token accounting

llama-server reports usage.completion_tokens including reasoning tokens (no separate reasoning_tokens field; verified 02.11.2026). Completion tokens are therefore the exact generated-token count per sample and form the basis of the thinking-efficiency metric (score per median generated tokens, GNAI-009 5.2).

Notes

  • The engine profile under test is whatever is currently active; switch profiles with the inference_engine role first (-e inference_engine_active_profile=...).
  • The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.