infrastructure/llm-server/ansible/README.md

LLM server Ansible control project

This directory contains the reproducible control project for ghostnet-ki. The read-only playbooks/preflight.yml verifies the documented host identity and hardware. The shared playbooks/host.yml applies the host baseline, Podman runtime, workload identities and narrowly scoped AMD device access through the dedicated deployment account.

Control-node setup

Use Python 3.12 through 3.14 and create a virtual environment at the repository root:

python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -r shared/ansible/requirements-control.txt

Load the private key for the read-only ghost-agent account into an SSH agent. Do not store private keys or passphrases in this repository:

ssh-add /path/to/ghost-agent-private-key

The workstation must already trust the verified SSH host key for 10.0.0.90. An unexpected host-key change must be checked through a trusted console before continuing.

Read-only preflight

Run the commands from the host project directory:

cd llm-server/ansible
../../.venv/bin/ansible-inventory \
  --inventory inventories/testing/hosts.yml \
  --graph
../../.venv/bin/ansible-playbook \
  --inventory inventories/testing/hosts.yml \
  playbooks/preflight.yml \
  --syntax-check
../../.venv/bin/ansible-playbook \
  --inventory inventories/testing/hosts.yml \
  playbooks/preflight.yml

The final recap must report changed=0 and failed=0. The playbook verifies:

• hostname ghostnet-ki;
• Debian major version 13;
• default IPv4 address 10.0.0.90;
• remote Python availability; and
• two AMD Navi 21 PCI devices with ID 1002:73bf.

The inventory deliberately contains no private-key path. Each operator supplies their authorized key through ssh-agent or a local Ansible override.

One-time deployment-account bootstrap

Status 2026-09-19: The initial bootstrap has been applied to ghostnet-ki, and the separate verification playbook confirmed SSH access and non-interactive effective UID 0 with changed=0 and failed=0. Run the bootstrap again only for an intentional repair, rotation or idempotence check.

The bootstrap creates the dedicated ansible-deploy account. Its password is locked, its committed public key is restricted to the LAN and WireGuard source networks, and /etc/sudoers.d/90-ansible-deploy is validated with visudo before installation. The account receives NOPASSWD root escalation because the planned host, package, systemd and container management cannot be expressed as a small stable command allowlist.

The matching private key stays outside this repository. On the workstation on which the initial key was created, it is stored as:

~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

The automation key has no interactive passphrase so a future runner can use it non-interactively. It is protected by filesystem mode 0600, SSH source-network restrictions and the dedicated account. Compromise of this private key would nevertheless grant root-equivalent deployment access; rotate it when moving the deployment identity to the runner or after any suspected disclosure.

Load the existing personal administrator key into an SSH agent, then run the bootstrap interactively from llm-server/ansible/. Ansible will ask for the personal administrator's existing sudo password:

ssh-add /path/to/personal-ghost-private-key
../../.venv/bin/ansible-playbook \
  --inventory inventories/bootstrap/hosts.yml \
  playbooks/bootstrap-deploy-account.yml \
  --ask-become-pass

Do not send the password through chat, store it in Git, or place it in a shell history entry. The bootstrap is idempotent; a second run should report no changes.

After a successful bootstrap, verify the new account independently:

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/verify-deploy-access.yml \
  --extra-vars ansible_user=ansible-deploy \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

The verification must report changed=0 and failed=0. It confirms both the unprivileged SSH identity and non-interactive escalation to effective UID 0.

Shared host baseline

The host baseline is common to all workloads. It verifies the target identity and at least 100 GiB free on the root filesystem, then manages these root-owned parent and shared-data roots:

/etc/ghostnet-ai
/var/lib/ghostnet-ai
/var/lib/ghostnet-ai/shared

It does not install software or start services. Use the dedicated host inventory and supply the private deployment key outside the repository:

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/host.yml \
  --syntax-check

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/host.yml \
  --check --diff \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/host.yml \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

Run the final command a second time. The second recap must report changed=0 and failed=0; otherwise the role has not demonstrated convergence.

Podman runtime

The shared host play also installs the Podman runtime from Debian 13. The role requires systemd and cgroup v2, prepares /etc/containers/systemd, verifies the overlay storage driver and exercises the installed Quadlet generator without creating a workload.

The role explicitly keeps the Podman API socket and automatic image-update timer disabled. It pulls no image. Apply and check it through the same playbooks/host.yml commands shown above. The first run may install packages; an immediate second run must report changed=0.

Workload accounts

The shared host play creates three locked service identities with deterministic UID/GID and subordinate-ID ranges:

Account	UID/GID	Purpose	Rootless Podman graph root
gnai-production	980	stable AI Production	/var/lib/ghostnet-ai/production/.local/share/containers/storage
gnai-testing	981	temporary AI Testing	/var/lib/ghostnet-ai/testing/.local/share/containers/storage
gn-encoding	982	scheduled CPU video encoding	/var/lib/ghostnet-encoding/.local/share/containers/storage

Each account has a persistent systemd user manager and a separate, administrator-managed directory below /etc/containers/systemd/users/<UID>. Configuration is root-owned and group-readable; workload data belongs to its service account. The two AI identities have render as their explicit supplementary-group baseline; the encoding identity has no GPU group. The role creates no workload containers.

AMD GPU access

The amd_gpu_access role verifies exactly two PCI devices 1002:73bf using the amdgpu kernel driver. It maps their sysfs identities to the corresponding DRM render nodes, checks /dev/kfd, derives the device groups and confirms read/write access as both AI workload identities.

On the current host, the verified devices are /dev/kfd, /dev/dri/renderD128 and /dev/dri/renderD129; all use group render. gnai-production and gnai-testing can access them, while gn-encoding remains outside that group. The final convergence run reported ok=55 changed=0 unreachable=0 failed=0.

This proves host-level device permissions. The separate container probe below verifies the next layer without creating a permanent service.

Rootless ROCm container probe

The dedicated verification playbook uses gnai-testing and an immutable amd64 manifest of the official llama.cpp ROCm image:

version:  b11058
revision: f072b103714dfa1eee531f80b24512faf38e3dd2
digest:   sha256:b429b8be10d3d0353a09c98f4ab840fe516f779858a28f9fd4e5bdf10bc63cb4

It passes /dev/kfd and both discovered render nodes with --group-add=keep-groups. rocminfo must report exactly two gfx1030 agents, and llama-server --list-devices must expose exactly two ROCm devices. No architecture override is set.

Run the probe from llm-server/ansible/:

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/verify-rocm-container.yml \
  --syntax-check

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/verify-rocm-container.yml \
  --check \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

../../.venv/bin/ansible-playbook \
  --inventory inventories/host/hosts.yml \
  playbooks/verify-rocm-container.yml \
  --private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy

The verified live run on 2026-09-20 reported ok=33 changed=0 unreachable=0 failed=0. The ephemeral probe containers are removed; the pinned image remains in the separate Testing image store for the next deployment step. This proves ROCm discovery and the llama.cpp backend, but not model loading, multi-GPU splitting or inference.