infrastructure/llm-server/llm/ansible/roles/inference_engine

inference_engine

Deploys the inference engine for one KI environment as rootless Podman Quadlets (GNAI-010 sections 5 and 6; first profile: llama.cpp). Runs after model_store in the same play and reuses model_store_root.

Variables

• inference_engine_image_ref – immutable image digest (ghcr.io/...@sha256:...).
• inference_engine_user / inference_engine_uid / inference_engine_user_home – workload account created by the host workload_accounts role.
• inference_engine_unit_prefix – unit and container prefix of the environment (Testing: gnai-testing-llama).
• inference_engine_network_name / inference_engine_network_alias – internal-only Quadlet network (Internal=true), alias llama-cpp for the later API policy layer.
• inference_engine_profiles – map of model profiles; each needs unit_suffix, model_directory, model_file, alias, ctx_size, n_predict, reasoning_effort, sampling values and an mtp block.
• inference_engine_active_profile – the single profile to start.
• inference_engine_ctx_size_override – when not null, replaces every profile's ctx_size (Testing bring-up: 65536; set to null after the 128k acceptance, GNAI-008 section 7.5).

Behaviour

• Validates declarations, requires /dev/kfd plus exactly two render nodes, and requires every profile's model artifact (run --tags models first).
• Pulls the pinned image by digest into the environment's rootless store only when missing.
• Renders <network>.network and one .container per profile into /etc/containers/systemd/users/<uid>/; units are never enabled for default.target.
• Units mutually exclude each other via Conflicts=/Before= (GNAI-010 section 5.3).
• Exercises the Quadlet generator with --dryrun before any reload; quadlet changes notify a daemon-reload of the workload user manager. Services are never restarted automatically.
• Switch/start (only with the service tag, never in check mode): stop any other active engine unit, start the selected unit, wait for /health, and verify the served alias via /v1/models. The HTTP check uses whichever of curl/wget exists in the pinned image (GNAI-010 section 8).

Tags

• service – everything in this role. Model provisioning belongs to the models tag of the model_store role.

Logs

conmon writes the container output to the system journal (journald driver) with the syslog identifier set to the container name. Read as root:

journalctl _COMM=conmon -t gnai-testing-llama-swift --no-pager

podman logs returns empty for the workload account (journal file permissions); acceptance protocols are collected as root (GNAI-010 section 8.1).

Notes

• There is no PublishPort=; the engine is reachable only inside the environment network.
• The role does not start the engine in check mode and never downloads models.