Separates gepinntes Image gnai-bench:bfcl-2026.3.23 (python:3.10-slim per Digest, bfcl-eval 2026.3.23 - Python 3.10 wegen BFCL-Zielumgebung), Runner run-bfcl.sh mit Remote-Modus gegen die OpenAI-kompatible Engine-API (REMOTE_OPENAI_BASE_URL=http://llama-cpp:8080/v1). Kategorien: simple, multiple, irrelevance (Irrelevance-Detection: Modell soll bei unpassenden Anfragen NICHT rufen). Smoke und Serienlauf bewusst NACH Abschluss der laufenden IFEval/MATH-Serie - Engine-Kontakt waehrend eines Benchmark-Laufs wuerde die Messung kontaminieren. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
benchmark
Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.
What it creates
gnai-testing-bench.networkquadlet: egress network, deliberately notInternal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal./var/lib/ghostnet-ai/testing/bench/withresults/andhf/(persistent HF dataset cache) owned by the workload account.Containerfile+ locally built, pinned imagegnai-bench:lm-eval-0.4.13(python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).run-bench.shrunner.
Usage
# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24
# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval
The runner attaches the container to both the internal engine network
(http://llama-cpp:8080) and the bench network, writes lm-eval results
with --log_samples under results/<profile>-<timestamp>/ and reuses the
persistent HF cache.
Token accounting
llama-server reports usage.completion_tokens including reasoning tokens
(no separate reasoning_tokens field; verified 02.11.2026). Completion
tokens are therefore the exact generated-token count per sample and form
the basis of the thinking-efficiency metric (score per median generated
tokens, GNAI-009 5.2).
Notes
- The engine profile under test is whatever is currently active; switch
profiles with the
inference_enginerole first (-e inference_engine_active_profile=...). - The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.