Files
infrastructure/llm-server/llm/ansible/roles/inference_engine
spektr 8480d53349
ci-static / secret-scan (push) Successful in 21s
ci-static / yamllint (push) Successful in 31s
ci-static / ansible-syntax (push) Successful in 27s
ci-static / markdown-links (push) Successful in 20s
ci-static / secret-scan (pull_request) Successful in 21s
ci-static / yamllint (pull_request) Successful in 44s
ci-static / ansible-syntax (pull_request) Successful in 1m3s
ci-static / markdown-links (pull_request) Successful in 23s
llama.cpp v1.0.0: Image-Pin auf server-rocm-v1.0.0-b12359-b1b770d heben (GNAI-010 0.20)
- inference_engine + rocm_container_probe: Digest sha256:4bbbd2e3... (amd64-Manifest
  sha256:abb28972...), Build b12359, Revision b1b770d90..., OCI-Version
  v1.0.0-b12359-b1b770d. v1.0.0 (Release 13.11.2026) loest ueber nightly-tag.txt
  auf b12359 auf; moving tag server-rocm zeigt auf denselben Digest. Semver-Policy
  upstream ab v1.0.0 — passt zur Digest-Pinning-Konvention.
- Kompatibilitaet gegen b12359 llama-server --help geprueft (Image lokal gepullt):
  alle 26 Exec=-Flags des Quadlets existieren; einzige entfernte Option
  --defrag-thold (#30565) wird nicht verwendet. ENTRYPOINT /app/llama-server
  unveraendert. Server-Haertung #32113/#32115/#32098 greift nicht (kein
  server-seitiges --tools/MCP, keine privaten Media-URLs, OpenWebUI ist
  Non-Browser-Client). libllama.so.1/36 entfernte Symbole betreffen nur native
  Bindings — wir sind HTTP-only.
- Neue Optionen bewertet (GNAI-010 0.20): FP8-KV f8_e4m3 (auf gfx1030 emuliert,
  Benchmark vor Adoption, Default q8_0), --moe-cache-cpu-share (nur MoE),
  --sleep-idle-seconds (kontraproduktiv), --api-key (optional), die Opt-out-Flags
  --allow-unauthenticated-tools/--media-allow-private-urls bleiben aus.
- Doku: GNAI-010 v0.20 + Abschnitt 3.1, GNAI-008 v0.23 + Probe-Abschnitt,
  Host-README Pin-Block (alter Pin als historisch markiert).
- Quadlet Exec bleibt unveraendert. Rollout auf 10.0.0.90 steht noch aus.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-14 15:04:14 +01:00
..

inference_engine

Deploys the inference engine for one KI environment as rootless Podman Quadlets (GNAI-010 sections 5 and 6; first profile: llama.cpp). Runs after model_store in the same play and reuses model_store_root.

Variables

  • inference_engine_image_ref – immutable image digest (ghcr.io/...@sha256:...).
  • inference_engine_user / inference_engine_uid / inference_engine_user_home – workload account created by the host workload_accounts role.
  • inference_engine_unit_prefix – unit and container prefix of the environment (Testing: gnai-testing-llama).
  • inference_engine_network_name / inference_engine_network_alias – internal-only Quadlet network (Internal=true), alias llama-cpp for the later API policy layer.
  • inference_engine_profiles – map of model profiles; each needs unit_suffix, model_directory, model_file, alias, ctx_size, n_predict, reasoning_effort, sampling values and an mtp block.
  • inference_engine_active_profile – the single profile to start.
  • inference_engine_ctx_size_override – when not null, replaces every profile's ctx_size (Testing bring-up: 65536; set to null after the 128k acceptance, GNAI-008 section 7.5).

Behaviour

  • Validates declarations, requires /dev/kfd plus exactly two render nodes, and requires every profile's model artifact (run --tags models first).
  • Pulls the pinned image by digest into the environment's rootless store only when missing.
  • Renders <network>.network and one .container per profile into /etc/containers/systemd/users/<uid>/; units are never enabled for default.target.
  • Units mutually exclude each other via Conflicts=/Before= (GNAI-010 section 5.3).
  • Exercises the Quadlet generator with --dryrun before any reload; quadlet changes notify a daemon-reload of the workload user manager. Services are never restarted automatically.
  • Switch/start (only with the service tag, never in check mode): stop any other active engine unit, start the selected unit, wait for /health, and verify the served alias via /v1/models. The HTTP check uses whichever of curl/wget exists in the pinned image (GNAI-010 section 8).

Tags

  • service – everything in this role. Model provisioning belongs to the models tag of the model_store role.

Logs

conmon writes the container output to the system journal (journald driver) with the syslog identifier set to the container name. Read as root:

journalctl _COMM=conmon -t gnai-testing-llama-swift --no-pager

podman logs returns empty for the workload account (journal file permissions); acceptance protocols are collected as root (GNAI-010 section 8.1).

Notes

  • There is no PublishPort=; the engine is reachable only inside the environment network.
  • The role does not start the engine in check mode and never downloads models.