Nach mehreren Live-Funden: - bfcl-eval braucht soundfile (qwen_agent Dependency) - CLI ist zweiteilig: bfcl generate + bfcl evaluate (nur evaluate bewertet vorab generierte Responses) - Kategorien sind sprachspezifisch: simple_python statt simple - Model-Handler muss api_inference sein: gpt-4o-2024-11-20-FC nutzt die OpenAI-SDK und respektiert OPENAI_BASE_URL/OPENAI_API_KEY - Container-Name muss --replace verwenden - Kategorie-Default auf simple_python,multiple,irrelevance korrigiert Smoke-Test gegen unsloth-Engine: 193 Requests in 48 min, laeuft. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
benchmark
Prepares the lm-evaluation-harness benchmark environment for one KI environment (GNAI-009 sections 5.1/5.2). Runs in the workload account's rootless store so the bench container can attach to the internal engine network.
What it creates
gnai-testing-bench.networkquadlet: egress network, deliberately notInternal=true. Its ONLY purpose is benchmark dataset downloads by the pinned bench container (documented exception, GNAI-009 5.1); the engine network stays internal./var/lib/ghostnet-ai/testing/bench/withresults/andhf/(persistent HF dataset cache) owned by the workload account.Containerfile+ locally built, pinned imagegnai-bench:lm-eval-0.4.13(python:3.13-slim by digest; the build needs host egress and takes several minutes the first time because of the lm-eval dependency chain).run-bench.shrunner.
Usage
# Prepare / converge the harness:
../../../.venv/bin/ansible-playbook -i inventories/testing playbooks/bench.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
# Smoke test (2 samples of AIME 2024):
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh smoke-profile 2 aime24
# Full run against the currently active engine profile:
cd /var/lib/ghostnet-ai/testing/bench && sudo ./run-bench.sh swift-1.5-qwen3.8-27b-q6k "" aime24 ifeval
The runner attaches the container to both the internal engine network
(http://llama-cpp:8080) and the bench network, writes lm-eval results
with --log_samples under results/<profile>-<timestamp>/ and reuses the
persistent HF cache.
Token accounting
llama-server reports usage.completion_tokens including reasoning tokens
(no separate reasoning_tokens field; verified 02.11.2026). Completion
tokens are therefore the exact generated-token count per sample and form
the basis of the thinking-efficiency metric (score per median generated
tokens, GNAI-009 5.2).
Notes
- The engine profile under test is whatever is currently active; switch
profiles with the
inference_enginerole first (-e inference_engine_active_profile=...). - The server-side sampling/reasoning configuration comes from the profile quadlet; the harness deliberately sends no generation overrides.