ci-static / secret-scan (push) Successful in 21s
ci-static / yamllint (push) Successful in 31s
ci-static / ansible-syntax (push) Successful in 27s
ci-static / markdown-links (push) Successful in 20s
ci-static / secret-scan (pull_request) Successful in 21s
ci-static / yamllint (pull_request) Successful in 44s
ci-static / ansible-syntax (pull_request) Successful in 1m3s
ci-static / markdown-links (pull_request) Successful in 23s
- inference_engine + rocm_container_probe: Digest sha256:4bbbd2e3... (amd64-Manifest sha256:abb28972...), Build b12359, Revision b1b770d90..., OCI-Version v1.0.0-b12359-b1b770d. v1.0.0 (Release 13.11.2026) loest ueber nightly-tag.txt auf b12359 auf; moving tag server-rocm zeigt auf denselben Digest. Semver-Policy upstream ab v1.0.0 — passt zur Digest-Pinning-Konvention. - Kompatibilitaet gegen b12359 llama-server --help geprueft (Image lokal gepullt): alle 26 Exec=-Flags des Quadlets existieren; einzige entfernte Option --defrag-thold (#30565) wird nicht verwendet. ENTRYPOINT /app/llama-server unveraendert. Server-Haertung #32113/#32115/#32098 greift nicht (kein server-seitiges --tools/MCP, keine privaten Media-URLs, OpenWebUI ist Non-Browser-Client). libllama.so.1/36 entfernte Symbole betreffen nur native Bindings — wir sind HTTP-only. - Neue Optionen bewertet (GNAI-010 0.20): FP8-KV f8_e4m3 (auf gfx1030 emuliert, Benchmark vor Adoption, Default q8_0), --moe-cache-cpu-share (nur MoE), --sleep-idle-seconds (kontraproduktiv), --api-key (optional), die Opt-out-Flags --allow-unauthenticated-tools/--media-allow-private-urls bleiben aus. - Doku: GNAI-010 v0.20 + Abschnitt 3.1, GNAI-008 v0.23 + Probe-Abschnitt, Host-README Pin-Block (alter Pin als historisch markiert). - Quadlet Exec bleibt unveraendert. Rollout auf 10.0.0.90 steht noch aus. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
246 lines
9.3 KiB
Markdown
246 lines
9.3 KiB
Markdown
# LLM server host Ansible control project
|
|
|
|
This directory contains the reproducible control project for the shared host
|
|
baseline of `ghostnet-ki`. The LLM-Server host root is split into one main
|
|
directory per task area: `host/` (this project), `llm/` and `encoding/`; see
|
|
[../../README.md](../../README.md).
|
|
The read-only `playbooks/preflight.yml` verifies the documented host identity
|
|
and hardware. The shared `playbooks/host.yml` applies the host baseline,
|
|
Podman runtime, workload identities and narrowly scoped AMD device access
|
|
through the dedicated deployment account.
|
|
|
|
## Control-node setup
|
|
|
|
Use Python 3.12 through 3.14 and create a virtual environment at the repository
|
|
root:
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
.venv/bin/python -m pip install --upgrade pip
|
|
.venv/bin/python -m pip install -r shared/ansible/requirements-control.txt
|
|
```
|
|
|
|
Load the private key for the read-only `ghost-agent` account into an SSH agent.
|
|
Do not store private keys or passphrases in this repository:
|
|
|
|
```bash
|
|
ssh-add /path/to/ghost-agent-private-key
|
|
```
|
|
|
|
The workstation must already trust the verified SSH host key for `10.0.0.90`.
|
|
An unexpected host-key change must be checked through a trusted console before
|
|
continuing.
|
|
|
|
## Read-only preflight
|
|
|
|
Run the commands from the host project directory:
|
|
|
|
```bash
|
|
cd llm-server/host/ansible
|
|
../../../.venv/bin/ansible-inventory \
|
|
--inventory inventories/testing/hosts.yml \
|
|
--graph
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/testing/hosts.yml \
|
|
playbooks/preflight.yml \
|
|
--syntax-check
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/testing/hosts.yml \
|
|
playbooks/preflight.yml
|
|
```
|
|
|
|
The final recap must report `changed=0` and `failed=0`. The playbook verifies:
|
|
|
|
- hostname `ghostnet-ki`;
|
|
- Debian major version 13;
|
|
- default IPv4 address `10.0.0.90`;
|
|
- remote Python availability; and
|
|
- two AMD Navi 21 PCI devices with ID `1002:73bf`.
|
|
|
|
The inventory deliberately contains no private-key path. Each operator supplies
|
|
their authorized key through `ssh-agent` or a local Ansible override.
|
|
|
|
## One-time deployment-account bootstrap
|
|
|
|
Status 2026-09-19: The initial bootstrap has been applied to `ghostnet-ki`, and
|
|
the separate verification playbook confirmed SSH access and non-interactive
|
|
effective UID 0 with `changed=0` and `failed=0`. Run the bootstrap again only
|
|
for an intentional repair, rotation or idempotence check.
|
|
|
|
The bootstrap creates the dedicated `ansible-deploy` account. Its password is
|
|
locked, its committed public key is restricted to the LAN and WireGuard source
|
|
networks, and `/etc/sudoers.d/90-ansible-deploy` is validated with `visudo`
|
|
before installation. The account receives `NOPASSWD` root escalation because
|
|
the planned host, package, systemd and container management cannot be expressed
|
|
as a small stable command allowlist.
|
|
|
|
The matching private key stays outside this repository. On the workstation on
|
|
which the initial key was created, it is stored as:
|
|
|
|
```text
|
|
~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
```
|
|
|
|
The automation key has no interactive passphrase so a future runner can use it
|
|
non-interactively. It is protected by filesystem mode `0600`, SSH source-network
|
|
restrictions and the dedicated account. Compromise of this private key would
|
|
nevertheless grant root-equivalent deployment access; rotate it when moving the
|
|
deployment identity to the runner or after any suspected disclosure.
|
|
|
|
Load the existing personal administrator key into an SSH agent, then run the
|
|
bootstrap interactively from `llm-server/host/ansible/`. Ansible will ask for
|
|
the personal administrator's existing sudo password:
|
|
|
|
```bash
|
|
ssh-add /path/to/personal-ghost-private-key
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/bootstrap/hosts.yml \
|
|
playbooks/bootstrap-deploy-account.yml \
|
|
--ask-become-pass
|
|
```
|
|
|
|
Do not send the password through chat, store it in Git, or place it in a shell
|
|
history entry. The bootstrap is idempotent; a second run should report no
|
|
changes.
|
|
|
|
After a successful bootstrap, verify the new account independently:
|
|
|
|
```bash
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/verify-deploy-access.yml \
|
|
--extra-vars ansible_user=ansible-deploy \
|
|
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
```
|
|
|
|
The verification must report `changed=0` and `failed=0`. It confirms both the
|
|
unprivileged SSH identity and non-interactive escalation to effective UID 0.
|
|
|
|
## Shared host baseline
|
|
|
|
The host baseline is common to all workloads. It verifies the target identity
|
|
and at least 100 GiB free on the root filesystem, then manages these root-owned
|
|
parent and shared-data roots:
|
|
|
|
```text
|
|
/etc/ghostnet-ai
|
|
/var/lib/ghostnet-ai
|
|
/var/lib/ghostnet-ai/shared
|
|
```
|
|
|
|
It does not install software or start services. Use the dedicated host
|
|
inventory and supply the private deployment key outside the repository:
|
|
|
|
```bash
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/host.yml \
|
|
--syntax-check
|
|
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/host.yml \
|
|
--check --diff \
|
|
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/host.yml \
|
|
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
```
|
|
|
|
Run the final command a second time. The second recap must report `changed=0`
|
|
and `failed=0`; otherwise the role has not demonstrated convergence.
|
|
|
|
## Podman runtime
|
|
|
|
The shared host play also installs the Podman runtime from Debian 13. The role
|
|
requires systemd and cgroup v2, prepares `/etc/containers/systemd`, verifies the
|
|
overlay storage driver and exercises the installed Quadlet generator without
|
|
creating a workload.
|
|
|
|
The role explicitly keeps the Podman API socket and automatic image-update
|
|
timer disabled. It pulls no image. Apply and check it through the same
|
|
`playbooks/host.yml` commands shown above. The first run may install packages;
|
|
an immediate second run must report `changed=0`.
|
|
|
|
## Workload accounts
|
|
|
|
The shared host play creates three locked service identities with deterministic
|
|
UID/GID and subordinate-ID ranges:
|
|
|
|
| Account | UID/GID | Purpose | Rootless Podman graph root |
|
|
| --- | --- | --- | --- |
|
|
| `gnai-production` | 980 | stable AI Production | `/var/lib/ghostnet-ai/production/.local/share/containers/storage` |
|
|
| `gnai-testing` | 981 | temporary AI Testing | `/var/lib/ghostnet-ai/testing/.local/share/containers/storage` |
|
|
| `gn-encoding` | 982 | scheduled CPU video encoding | `/var/lib/ghostnet-encoding/.local/share/containers/storage` |
|
|
|
|
Each account has a persistent systemd user manager and a separate,
|
|
administrator-managed directory below `/etc/containers/systemd/users/<UID>`.
|
|
Configuration is root-owned and group-readable; workload data belongs to its
|
|
service account. The two AI identities have `render` as their explicit
|
|
supplementary-group baseline; the encoding identity has no GPU group. The role
|
|
creates no workload containers.
|
|
|
|
## AMD GPU access
|
|
|
|
The `amd_gpu_access` role verifies exactly two PCI devices `1002:73bf` using
|
|
the `amdgpu` kernel driver. It maps their sysfs identities to the corresponding
|
|
DRM render nodes, checks `/dev/kfd`, derives the device groups and confirms
|
|
read/write access as both AI workload identities.
|
|
|
|
On the current host, the verified devices are `/dev/kfd`,
|
|
`/dev/dri/renderD128` and `/dev/dri/renderD129`; all use group `render`.
|
|
`gnai-production` and `gnai-testing` can access them, while `gn-encoding`
|
|
remains outside that group. The final convergence run reported
|
|
`ok=55 changed=0 unreachable=0 failed=0`.
|
|
|
|
This proves host-level device permissions. The separate container probe below
|
|
verifies the next layer without creating a permanent service.
|
|
|
|
## Rootless ROCm container probe
|
|
|
|
The dedicated verification playbook uses `gnai-testing` and an immutable amd64
|
|
manifest of the official `llama.cpp` ROCm image:
|
|
|
|
```text
|
|
version: v1.0.0-b12359-b1b770d # OCI label; release tag v1.0.0 (2026-11-13), build b12359
|
|
revision: b1b770d90d94cad9e9ea5b993bdbc383a522fb28
|
|
digest: sha256:4bbbd2e33deb54330fd23edde3a2cdf12b05ce4415139c7e8639a9aebbacdcf2
|
|
```
|
|
|
|
Updated 2026-11-14 from the previous pin `b11058` / `f072b103…` /
|
|
`sha256:b429b8be…` to the v1.0.0 stable tag (`server-rocm-v1.0.0-b12359-b1b770d`,
|
|
amd64 manifest `sha256:abb28972…`).
|
|
|
|
It passes `/dev/kfd` and both discovered render nodes with
|
|
`--group-add=keep-groups`. `rocminfo` must report exactly two `gfx1030` agents,
|
|
and `llama-server --list-devices` must expose exactly two ROCm devices. No
|
|
architecture override is set.
|
|
|
|
Run the probe from `llm-server/host/ansible/`:
|
|
|
|
```bash
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/verify-rocm-container.yml \
|
|
--syntax-check
|
|
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/verify-rocm-container.yml \
|
|
--check \
|
|
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
|
|
../../../.venv/bin/ansible-playbook \
|
|
--inventory inventories/host/hosts.yml \
|
|
playbooks/verify-rocm-container.yml \
|
|
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
|
|
```
|
|
|
|
The verified live run on 2026-09-20 reported
|
|
`ok=33 changed=0 unreachable=0 failed=0`. The ephemeral probe containers are
|
|
removed; the pinned image remains in the separate Testing image store for the
|
|
next deployment step. This proves ROCm discovery and the `llama.cpp` backend,
|
|
but not model loading, multi-GPU splitting or inference.
|