ghostnet.ai/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

LLM-Server: ausführbares Projekt in die Hauptverzeichnisse host/, llm/ und encoding/ aufgeteilt.

Browse Source 
Grundkonfiguration des Servers liegt nun unter llm-server/host/ansible/
(vorher llm-server/ansible/), der LLM-Stack-Platzhalter unter llm-server/llm/
und der Encoding-Workload unter llm-server/encoding/. Dokumentation und
Pfadverweise (ansible.cfg roles_path, .venv-Tiefe, Markdown-Links) angepasst;
Linkvalidator meldet broken_local_links=0.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
spektr
2026-10-19 14:42:14 +02:00
parent 82af4e9235
commit 92c846f527
31 changed files with 114 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
llm-server/host/ansible/README.md
+241
View File
@@ -0,0 +1,241 @@
# LLM server host Ansible control project
This directory contains the reproducible control project for the shared host
baseline of `ghostnet-ki`. The LLM-Server host root is split into one main
directory per task area: `host/` (this project), `llm/` and `encoding/`; see
[../../README.md](../../README.md).
The read-only `playbooks/preflight.yml` verifies the documented host identity
and hardware. The shared `playbooks/host.yml` applies the host baseline,
Podman runtime, workload identities and narrowly scoped AMD device access
through the dedicated deployment account.
## Control-node setup
Use Python 3.12 through 3.14 and create a virtual environment at the repository
root:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -r shared/ansible/requirements-control.txt
```
Load the private key for the read-only `ghost-agent` account into an SSH agent.
Do not store private keys or passphrases in this repository:
```bash
ssh-add /path/to/ghost-agent-private-key
```
The workstation must already trust the verified SSH host key for `10.0.0.90`.
An unexpected host-key change must be checked through a trusted console before
continuing.
## Read-only preflight
Run the commands from the host project directory:
```bash
cd llm-server/host/ansible
../../../.venv/bin/ansible-inventory \
--inventory inventories/testing/hosts.yml \
--graph
../../../.venv/bin/ansible-playbook \
--inventory inventories/testing/hosts.yml \
playbooks/preflight.yml \
--syntax-check
../../../.venv/bin/ansible-playbook \
--inventory inventories/testing/hosts.yml \
playbooks/preflight.yml
```
The final recap must report `changed=0` and `failed=0`. The playbook verifies:
- hostname `ghostnet-ki`;
- Debian major version 13;
- default IPv4 address `10.0.0.90`;
- remote Python availability; and
- two AMD Navi 21 PCI devices with ID `1002:73bf`.
The inventory deliberately contains no private-key path. Each operator supplies
their authorized key through `ssh-agent` or a local Ansible override.
## One-time deployment-account bootstrap
Status 2026-09-19: The initial bootstrap has been applied to `ghostnet-ki`, and
the separate verification playbook confirmed SSH access and non-interactive
effective UID 0 with `changed=0` and `failed=0`. Run the bootstrap again only
for an intentional repair, rotation or idempotence check.
The bootstrap creates the dedicated `ansible-deploy` account. Its password is
locked, its committed public key is restricted to the LAN and WireGuard source
networks, and `/etc/sudoers.d/90-ansible-deploy` is validated with `visudo`
before installation. The account receives `NOPASSWD` root escalation because
the planned host, package, systemd and container management cannot be expressed
as a small stable command allowlist.
The matching private key stays outside this repository. On the workstation on
which the initial key was created, it is stored as:
```text
~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
```
The automation key has no interactive passphrase so a future runner can use it
non-interactively. It is protected by filesystem mode `0600`, SSH source-network
restrictions and the dedicated account. Compromise of this private key would
nevertheless grant root-equivalent deployment access; rotate it when moving the
deployment identity to the runner or after any suspected disclosure.
Load the existing personal administrator key into an SSH agent, then run the
bootstrap interactively from `llm-server/host/ansible/`. Ansible will ask for
the personal administrator's existing sudo password:
```bash
ssh-add /path/to/personal-ghost-private-key
../../../.venv/bin/ansible-playbook \
--inventory inventories/bootstrap/hosts.yml \
playbooks/bootstrap-deploy-account.yml \
--ask-become-pass
```
Do not send the password through chat, store it in Git, or place it in a shell
history entry. The bootstrap is idempotent; a second run should report no
changes.
After a successful bootstrap, verify the new account independently:
```bash
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/verify-deploy-access.yml \
--extra-vars ansible_user=ansible-deploy \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
```
The verification must report `changed=0` and `failed=0`. It confirms both the
unprivileged SSH identity and non-interactive escalation to effective UID 0.
## Shared host baseline
The host baseline is common to all workloads. It verifies the target identity
and at least 100 GiB free on the root filesystem, then manages these root-owned
parent and shared-data roots:
```text
/etc/ghostnet-ai
/var/lib/ghostnet-ai
/var/lib/ghostnet-ai/shared
```
It does not install software or start services. Use the dedicated host
inventory and supply the private deployment key outside the repository:
```bash
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/host.yml \
--syntax-check
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/host.yml \
--check --diff \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/host.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
```
Run the final command a second time. The second recap must report `changed=0`
and `failed=0`; otherwise the role has not demonstrated convergence.
## Podman runtime
The shared host play also installs the Podman runtime from Debian 13. The role
requires systemd and cgroup v2, prepares `/etc/containers/systemd`, verifies the
overlay storage driver and exercises the installed Quadlet generator without
creating a workload.
The role explicitly keeps the Podman API socket and automatic image-update
timer disabled. It pulls no image. Apply and check it through the same
`playbooks/host.yml` commands shown above. The first run may install packages;
an immediate second run must report `changed=0`.
## Workload accounts
The shared host play creates three locked service identities with deterministic
UID/GID and subordinate-ID ranges:
| Account | UID/GID | Purpose | Rootless Podman graph root |
| --- | --- | --- | --- |
| `gnai-production` | 980 | stable AI Production | `/var/lib/ghostnet-ai/production/.local/share/containers/storage` |
| `gnai-testing` | 981 | temporary AI Testing | `/var/lib/ghostnet-ai/testing/.local/share/containers/storage` |
| `gn-encoding` | 982 | scheduled CPU video encoding | `/var/lib/ghostnet-encoding/.local/share/containers/storage` |
Each account has a persistent systemd user manager and a separate,
administrator-managed directory below `/etc/containers/systemd/users/<UID>`.
Configuration is root-owned and group-readable; workload data belongs to its
service account. The two AI identities have `render` as their explicit
supplementary-group baseline; the encoding identity has no GPU group. The role
creates no workload containers.
## AMD GPU access
The `amd_gpu_access` role verifies exactly two PCI devices `1002:73bf` using
the `amdgpu` kernel driver. It maps their sysfs identities to the corresponding
DRM render nodes, checks `/dev/kfd`, derives the device groups and confirms
read/write access as both AI workload identities.
On the current host, the verified devices are `/dev/kfd`,
`/dev/dri/renderD128` and `/dev/dri/renderD129`; all use group `render`.
`gnai-production` and `gnai-testing` can access them, while `gn-encoding`
remains outside that group. The final convergence run reported
`ok=55 changed=0 unreachable=0 failed=0`.
This proves host-level device permissions. The separate container probe below
verifies the next layer without creating a permanent service.
## Rootless ROCm container probe
The dedicated verification playbook uses `gnai-testing` and an immutable amd64
manifest of the official `llama.cpp` ROCm image:
```text
version: b11058
revision: f072b103714dfa1eee531f80b24512faf38e3dd2
digest: sha256:b429b8be10d3d0353a09c98f4ab840fe516f779858a28f9fd4e5bdf10bc63cb4
```
It passes `/dev/kfd` and both discovered render nodes with
`--group-add=keep-groups`. `rocminfo` must report exactly two `gfx1030` agents,
and `llama-server --list-devices` must expose exactly two ROCm devices. No
architecture override is set.
Run the probe from `llm-server/host/ansible/`:
```bash
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/verify-rocm-container.yml \
--syntax-check
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/verify-rocm-container.yml \
--check \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
../../../.venv/bin/ansible-playbook \
--inventory inventories/host/hosts.yml \
playbooks/verify-rocm-container.yml \
--private-key ~/.ssh/id_ed25519_ghostnet_ai_ansible_deploy
```
The verified live run on 2026-09-20 reported
`ok=33 changed=0 unreachable=0 failed=0`. The ephemeral probe containers are
removed; the pinned image remains in the separate Testing image store for the
next deployment step. This proves ROCm discovery and the `llama.cpp` backend,
but not model loading, multi-GPU splitting or inference.
llm-server/host/ansible/ansible.cfg
+12
View File
@@ -0,0 +1,12 @@
[defaults]
home = .ansible
host_key_checking = True
interpreter_python = auto_silent
local_tmp = .ansible/tmp
retry_files_enabled = False
roles_path = roles:../../../shared/ansible/roles
timeout = 15
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
llm-server/host/ansible/files/ssh/ansible-deploy.pub
+1
View File
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJE3UGGHyB+CMA2sqPpI/6efn9u3eLelL2MkHqdt2naB ansible-deploy@ghostnet-control-2026-09
llm-server/host/ansible/inventories/bootstrap/hosts.yml
+8
View File
@@ -0,0 +1,8 @@
---
all:
children:
llm_servers:
hosts:
ghostnet-ki:
ansible_host: 10.0.0.90
ansible_user: ghost
llm-server/host/ansible/inventories/host/hosts.yml
+8
View File
@@ -0,0 +1,8 @@
---
all:
children:
llm_servers:
hosts:
ghostnet-ki:
ansible_host: 10.0.0.90
ansible_user: ansible-deploy
llm-server/host/ansible/inventories/testing/hosts.yml
+8
View File
@@ -0,0 +1,8 @@
---
all:
children:
llm_servers:
hosts:
ghostnet-ki:
ansible_host: 10.0.0.90
ansible_user: ghost-agent
llm-server/host/ansible/playbooks/bootstrap-deploy-account.yml
+98
View File
@@ -0,0 +1,98 @@
---
- name: Bootstrap the dedicated Ansible deployment account
hosts: llm_servers
become: true
gather_facts: true
vars:
ansible_deploy_user: ansible-deploy
ansible_deploy_public_key_file: >-
{{ playbook_dir }}/../files/ssh/ansible-deploy.pub
ansible_deploy_ssh_sources:
- 10.0.0.0/24
- 10.8.0.0/24
pre_tasks:
- name: Refuse to bootstrap through the target deployment account
ansible.builtin.assert:
that:
- ansible_user != ansible_deploy_user
fail_msg: >-
Run this one-time bootstrap through an existing personal administrator
account, not through {{ ansible_deploy_user }}.
- name: Verify the local deployment public key
ansible.builtin.assert:
that:
- lookup("ansible.builtin.file", ansible_deploy_public_key_file) is match("^ssh-ed25519 ")
fail_msg: >-
The expected Ed25519 public key is missing or malformed at
{{ ansible_deploy_public_key_file }}.
tasks:
- name: Create the dedicated deployment account with a locked password
ansible.builtin.user:
name: "{{ ansible_deploy_user }}"
comment: Ghostnet AI Ansible deployment account
create_home: true
home: "/home/{{ ansible_deploy_user }}"
password_lock: true
shell: /bin/bash
state: present
- name: Protect the deployment account home directory
ansible.builtin.file:
path: "/home/{{ ansible_deploy_user }}"
state: directory
owner: "{{ ansible_deploy_user }}"
group: "{{ ansible_deploy_user }}"
mode: "0750"
- name: Create the deployment SSH directory
ansible.builtin.file:
path: "/home/{{ ansible_deploy_user }}/.ssh"
state: directory
owner: "{{ ansible_deploy_user }}"
group: "{{ ansible_deploy_user }}"
mode: "0700"
- name: Install the restricted deployment public key
ansible.builtin.copy:
content: |
restrict,from="{{ ansible_deploy_ssh_sources | join(',') }}" {{ lookup('ansible.builtin.file', ansible_deploy_public_key_file) | trim }}
dest: "/home/{{ ansible_deploy_user }}/.ssh/authorized_keys"
owner: "{{ ansible_deploy_user }}"
group: "{{ ansible_deploy_user }}"
mode: "0600"
- name: Grant explicit non-interactive root escalation for Ansible
ansible.builtin.copy:
content: |
# Managed by Ghostnet AI Ansible bootstrap.
{{ ansible_deploy_user }} ALL=(root) NOPASSWD: ALL
dest: "/etc/sudoers.d/90-{{ ansible_deploy_user }}"
owner: root
group: root
mode: "0440"
validate: /usr/sbin/visudo -cf %s
- name: Read the effective deployment sudo policy
ansible.builtin.command:
argv:
- /usr/bin/sudo
- -l
- -U
- "{{ ansible_deploy_user }}"
changed_when: false
register: ansible_deploy_sudo_policy
- name: Verify non-interactive root escalation is effective
ansible.builtin.assert:
that:
- 'ansible_deploy_sudo_policy.stdout is search("NOPASSWD: ALL")'
fail_msg: >-
The deployment account exists, but its validated NOPASSWD sudo policy
is not effective.
success_msg: >-
The deployment account and its validated sudo policy are ready for
the separate access verification playbook.
llm-server/host/ansible/playbooks/host.yml
+15
View File
@@ -0,0 +1,15 @@
---
- name: Apply the shared Ghostnet AI host baseline
hosts: llm_servers
become: true
gather_facts: true
gather_subset:
- min
- network
- hardware
roles:
- role: host_baseline
- role: podman_runtime
- role: workload_accounts
- role: amd_gpu_access
llm-server/host/ansible/playbooks/preflight.yml
+60
View File
@@ -0,0 +1,60 @@
---
- name: Verify read-only access to the Ghostnet AI server
hosts: llm_servers
become: false
gather_facts: true
gather_subset:
- min
- hardware
- network
tasks:
- name: Verify the expected host identity and operating system
ansible.builtin.assert:
that:
- ansible_facts["hostname"] == "ghostnet-ki"
- ansible_facts["distribution"] == "Debian"
- ansible_facts["distribution_major_version"] == "13"
- ansible_facts["default_ipv4"]["address"] == "10.0.0.90"
- ansible_facts["python"]["version"]["major"] >= 3
fail_msg: >-
The connected host does not match the documented Ghostnet AI server
baseline. Stop before using any deployment playbook.
success_msg: The expected Debian 13 host ghostnet-ki is connected.
- name: Read the expected AMD GPU PCI devices
ansible.builtin.command:
argv:
- lspci
- -D
- -nn
- -d
- "1002:73bf"
check_mode: false
changed_when: false
register: preflight_gpu_devices
- name: Verify both documented AMD GPUs are present
ansible.builtin.assert:
that:
- preflight_gpu_devices.stdout_lines | length == 2
fail_msg: >-
Expected two AMD Navi 21 PCI devices with ID 1002:73bf, but found
{{ preflight_gpu_devices.stdout_lines | length }}.
success_msg: Both documented AMD Navi 21 PCI devices are present.
- name: Report the verified read-only baseline
ansible.builtin.debug:
msg:
hostname: "{{ ansible_facts['hostname'] }}"
operating_system: >-
{{ ansible_facts['distribution'] }}
{{ ansible_facts['distribution_version'] }}
default_ipv4: "{{ ansible_facts['default_ipv4']['address'] }}"
processor_vcpus: "{{ ansible_facts['processor_vcpus'] }}"
memory_mib: "{{ ansible_facts['memtotal_mb'] }}"
python_version: >-
{{ ansible_facts['python']['version']['major'] }}.
{{- ansible_facts['python']['version']['minor'] }}.
{{- ansible_facts['python']['version']['micro'] }}
amd_gpu_devices: "{{ preflight_gpu_devices.stdout_lines }}"
llm-server/host/ansible/playbooks/verify-deploy-access.yml
+41
View File
@@ -0,0 +1,41 @@
---
- name: Verify the dedicated Ansible deployment account
hosts: llm_servers
become: false
gather_facts: false
tasks:
- name: Verify Ansible module execution without privilege escalation
ansible.builtin.ping:
- name: Read the SSH account identity
ansible.builtin.command:
argv:
- /usr/bin/id
- -un
changed_when: false
register: deploy_account_identity
- name: Verify the expected SSH account
ansible.builtin.assert:
that:
- deploy_account_identity.stdout == "ansible-deploy"
fail_msg: The SSH connection did not use the ansible-deploy account.
- name: Read the effective identity through sudo
become: true
ansible.builtin.command:
argv:
- /usr/bin/id
- -u
changed_when: false
register: deploy_root_identity
- name: Verify non-interactive root escalation
ansible.builtin.assert:
that:
- deploy_root_identity.stdout == "0"
fail_msg: Non-interactive sudo did not produce effective UID 0.
success_msg: >-
The dedicated deployment account can connect and escalate privileges
non-interactively.
llm-server/host/ansible/playbooks/verify-rocm-container.yml
+11
View File
@@ -0,0 +1,11 @@
---
- name: Verify rootless ROCm access for the AI Testing account
hosts: llm_servers
become: true
gather_facts: true
gather_subset:
- min
roles:
- role: amd_gpu_access
- role: rocm_container_probe
llm-server/host/ansible/roles/amd_gpu_access/README.md
+16
View File
@@ -0,0 +1,16 @@
# AMD GPU access role
This shared host role verifies the two configured AMD GPUs and their `amdgpu`
kernel driver, discovers only the DRM render nodes belonging to their PCI IDs,
and checks `/dev/kfd`. It grants the Production and Testing service accounts
the minimal supplementary groups required by those device nodes.
When group membership changes, the role restarts both persistent systemd user
managers so later rootless Quadlets can use `GroupAdd=keep-groups`. It verifies
that each service account can read and write `/dev/kfd` and both selected
render devices.
The role installs no ROCm userspace packages, pulls no image and starts no
container. ROCm userspace will come from the pinned `llama.cpp` container. A
later workload check must still prove that the container detects both GPUs as
`gfx1030` and performs inference on them.
llm-server/host/ansible/roles/amd_gpu_access/defaults/main.yml
+10
View File
@@ -0,0 +1,10 @@
---
amd_gpu_vendor_id: "1002"
amd_gpu_device_id: "73bf"
amd_gpu_expected_count: 2
amd_gpu_workload_accounts:
- name: gnai-production
uid: 980
- name: gnai-testing
uid: 981
llm-server/host/ansible/roles/amd_gpu_access/handlers/main.yml
+10
View File
@@ -0,0 +1,10 @@
---
- name: Restart AI workload user managers
ansible.builtin.systemd_service:
name: "user@{{ amd_gpu_account.uid }}.service"
state: restarted
loop: "{{ amd_gpu_workload_accounts }}"
loop_control:
loop_var: amd_gpu_account
label: "{{ amd_gpu_account.name }}"
when: not ansible_check_mode
llm-server/host/ansible/roles/amd_gpu_access/tasks/main.yml
+212
View File
@@ -0,0 +1,212 @@
---
- name: Verify AMD GPU access declarations
ansible.builtin.assert:
that:
- amd_gpu_vendor_id is match("^[0-9a-fA-F]{4}$")
- amd_gpu_device_id is match("^[0-9a-fA-F]{4}$")
- amd_gpu_expected_count | int > 0
- >-
amd_gpu_workload_accounts | map(attribute="name") | unique | length ==
amd_gpu_workload_accounts | length
- >-
amd_gpu_workload_accounts | map(attribute="uid") | unique | length ==
amd_gpu_workload_accounts | length
fail_msg: AMD GPU identifiers and workload accounts must be valid and unique.
- name: Read matching PCI devices and kernel drivers
ansible.builtin.command:
argv:
- /usr/bin/lspci
- -Dnnk
- -d
- "{{ amd_gpu_vendor_id }}:{{ amd_gpu_device_id }}"
changed_when: false
check_mode: false
register: amd_gpu_lspci
- name: Verify expected GPUs use the amdgpu kernel driver
ansible.builtin.assert:
that:
- >-
amd_gpu_lspci.stdout_lines
| select("search", "(?i)" + amd_gpu_vendor_id + ":" + amd_gpu_device_id)
| list | length == amd_gpu_expected_count | int
- >-
amd_gpu_lspci.stdout
| regex_findall("Kernel driver in use: amdgpu")
| length == amd_gpu_expected_count | int
fail_msg: >-
Expected {{ amd_gpu_expected_count }} PCI devices
{{ amd_gpu_vendor_id }}:{{ amd_gpu_device_id }} using amdgpu.
- name: Find DRM render nodes in sysfs
ansible.builtin.find:
paths: /sys/class/drm
patterns: "renderD*"
file_type: link
recurse: false
register: amd_gpu_sysfs_render_nodes
- name: Read render node PCI vendor IDs
ansible.builtin.slurp:
src: "{{ amd_gpu_render_node.path }}/device/vendor"
loop: "{{ amd_gpu_sysfs_render_nodes.files }}"
loop_control:
loop_var: amd_gpu_render_node
label: "{{ amd_gpu_render_node.path | basename }}"
register: amd_gpu_render_vendor_ids
- name: Read render node PCI device IDs
ansible.builtin.slurp:
src: "{{ amd_gpu_render_node.path }}/device/device"
loop: "{{ amd_gpu_sysfs_render_nodes.files }}"
loop_control:
loop_var: amd_gpu_render_node
label: "{{ amd_gpu_render_node.path | basename }}"
register: amd_gpu_render_device_ids
- name: Initialize matching AMD render device list
ansible.builtin.set_fact:
amd_gpu_render_devices: []
- name: Select render devices belonging to the configured AMD GPUs
ansible.builtin.set_fact:
amd_gpu_render_devices: >-
{{ amd_gpu_render_devices
+ ['/dev/dri/'
+ (amd_gpu_sysfs_render_nodes.files[amd_gpu_render_index].path
| basename)] }}
loop: "{{ range(0, amd_gpu_sysfs_render_nodes.files | length) | list }}"
loop_control:
loop_var: amd_gpu_render_index
label: >-
{{ amd_gpu_sysfs_render_nodes.files[amd_gpu_render_index].path | basename }}
when:
- >-
amd_gpu_render_vendor_ids.results[amd_gpu_render_index].content
| b64decode | trim | lower == "0x" + amd_gpu_vendor_id | lower
- >-
amd_gpu_render_device_ids.results[amd_gpu_render_index].content
| b64decode | trim | lower == "0x" + amd_gpu_device_id | lower
- name: Verify one render device was found for each configured GPU
ansible.builtin.assert:
that:
- amd_gpu_render_devices | length == amd_gpu_expected_count
fail_msg: >-
Expected {{ amd_gpu_expected_count }} matching AMD render devices but found
{{ amd_gpu_render_devices | length }}: {{ amd_gpu_render_devices }}.
- name: Inspect the ROCm kernel device
ansible.builtin.stat:
path: /dev/kfd
register: amd_gpu_kfd
- name: Verify the ROCm kernel device exists
ansible.builtin.assert:
that:
- amd_gpu_kfd.stat.exists
- amd_gpu_kfd.stat.ischr | default(false)
- amd_gpu_kfd.stat.gr_name != "root"
fail_msg: /dev/kfd must exist as a character device with a dedicated group.
- name: Inspect matching AMD render devices
ansible.builtin.stat:
path: "{{ amd_gpu_render_device }}"
loop: "{{ amd_gpu_render_devices }}"
loop_control:
loop_var: amd_gpu_render_device
register: amd_gpu_render_device_stats
- name: Verify matching AMD render devices are usable character devices
ansible.builtin.assert:
that:
- amd_gpu_render_stat.stat.exists
- amd_gpu_render_stat.stat.ischr | default(false)
- amd_gpu_render_stat.stat.gr_name != "root"
loop: "{{ amd_gpu_render_device_stats.results }}"
loop_control:
loop_var: amd_gpu_render_stat
label: "{{ amd_gpu_render_stat.amd_gpu_render_device }}"
- name: Derive the minimal host groups required by the GPU devices
ansible.builtin.set_fact:
amd_gpu_required_group_names: >-
{{ ([amd_gpu_kfd.stat.gr_name]
+ (amd_gpu_render_device_stats.results
| map(attribute="stat.gr_name") | list))
| unique | sort | list }}
- name: Grant AI workload accounts access to the GPU device groups
ansible.builtin.user:
name: "{{ amd_gpu_account.name }}"
groups: "{{ amd_gpu_required_group_names | join(',') }}"
append: true
loop: "{{ amd_gpu_workload_accounts }}"
loop_control:
loop_var: amd_gpu_account
label: "{{ amd_gpu_account.name }}"
notify: Restart AI workload user managers
- name: Apply changed group memberships to persistent user managers
ansible.builtin.meta: flush_handlers
- name: Read effective AI workload account groups
ansible.builtin.command:
argv:
- /usr/bin/id
- -nG
- "{{ amd_gpu_account.name }}"
changed_when: false
loop: "{{ amd_gpu_workload_accounts }}"
loop_control:
loop_var: amd_gpu_account
label: "{{ amd_gpu_account.name }}"
register: amd_gpu_account_groups
when: not ansible_check_mode
- name: Verify AI workload account group memberships
ansible.builtin.assert:
that:
- >-
amd_gpu_required_group_names
| difference(amd_gpu_account_group.stdout.split()) | length == 0
fail_msg: >-
{{ amd_gpu_account_group.amd_gpu_account.name }} is missing a required GPU
device group.
loop: "{{ amd_gpu_account_groups.results }}"
loop_control:
loop_var: amd_gpu_account_group
label: "{{ amd_gpu_account_group.amd_gpu_account.name }}"
when: not ansible_check_mode
- name: Verify AI workload accounts can read and write required GPU devices
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ amd_gpu_access_pair.0.name }}"
- --
- /usr/bin/python3
- -c
- >-
import os, sys; raise SystemExit(
0 if os.access(sys.argv[1], os.R_OK | os.W_OK) else 1)
- "{{ amd_gpu_access_pair.1 }}"
changed_when: false
loop: >-
{{ amd_gpu_workload_accounts
| product(['/dev/kfd'] + amd_gpu_render_devices) | list }}
loop_control:
loop_var: amd_gpu_access_pair
label: "{{ amd_gpu_access_pair.0.name }} -> {{ amd_gpu_access_pair.1 }}"
when: not ansible_check_mode
- name: Report verified AMD GPU access
ansible.builtin.debug:
msg:
pci_id: "{{ amd_gpu_vendor_id }}:{{ amd_gpu_device_id }}"
gpu_count: "{{ amd_gpu_expected_count }}"
render_devices: "{{ amd_gpu_render_devices }}"
device_groups: "{{ amd_gpu_required_group_names }}"
accounts: "{{ amd_gpu_workload_accounts | map(attribute='name') | list }}"
llm-server/host/ansible/roles/host_baseline/README.md
+19
View File
@@ -0,0 +1,19 @@
# Shared host baseline
The `host_baseline` role manages state shared by the Production and Testing
environments on `ghostnet-ki`.
Its first version deliberately has a narrow scope:
- refuse changes when hostname, Debian major version or primary IPv4 address
do not match the declared target;
- require at least 100 GiB free on the root filesystem; and
- create root-owned parent directories and the future shared-data root.
Environment-specific directory ownership is managed by `workload_accounts` so
the shared baseline and workload identities do not compete over permissions.
It does not install packages, drivers, Podman or applications and does not
start or stop services. Ownership below the directory roots will be assigned
to dedicated service accounts when the corresponding application roles are
implemented.
llm-server/host/ansible/roles/host_baseline/defaults/main.yml
+16
View File
@@ -0,0 +1,16 @@
---
host_baseline_expected_hostname: ghostnet-ki
host_baseline_expected_distribution: Debian
host_baseline_expected_distribution_major_version: "13"
host_baseline_expected_ipv4_address: 10.0.0.90
# Preserve enough headroom for host operation and controlled model handling.
host_baseline_minimum_root_free_gib: 100
host_baseline_directories:
- path: /etc/ghostnet-ai
mode: "0755"
- path: /var/lib/ghostnet-ai
mode: "0755"
- path: /var/lib/ghostnet-ai/shared
mode: "0755"
llm-server/host/ansible/roles/host_baseline/tasks/main.yml
+59
View File
@@ -0,0 +1,59 @@
---
- name: Verify the target host before changing shared state
ansible.builtin.assert:
that:
- ansible_facts["hostname"] == host_baseline_expected_hostname
- ansible_facts["distribution"] == host_baseline_expected_distribution
- >-
ansible_facts["distribution_major_version"] ==
host_baseline_expected_distribution_major_version
- >-
ansible_facts["default_ipv4"]["address"] ==
host_baseline_expected_ipv4_address
fail_msg: >-
The connected host does not match the declared Ghostnet AI host. No
baseline changes may be applied.
success_msg: The expected Debian host ghostnet-ki is connected.
- name: Select the root filesystem facts
ansible.builtin.set_fact:
host_baseline_root_mount: >-
{{ (ansible_facts["mounts"]
| selectattr("mount", "equalto", "/")
| list
| first)
| default({}) }}
- name: Verify the root filesystem was discovered
ansible.builtin.assert:
that:
- host_baseline_root_mount | length > 0
- host_baseline_root_mount["size_available"] is defined
fail_msg: Root filesystem capacity could not be determined. No changes were made.
- name: Verify minimum free capacity on the shared root filesystem
ansible.builtin.assert:
that:
- >-
host_baseline_root_mount["size_available"] | int >=
host_baseline_minimum_root_free_gib | int * 1024 ** 3
fail_msg: >-
The root filesystem has only
{{ (host_baseline_root_mount["size_available"] | int / 1024 ** 3) | round(1) }}
GiB free. At least {{ host_baseline_minimum_root_free_gib }} GiB are
required before applying the shared host baseline.
success_msg: >-
The root filesystem has
{{ (host_baseline_root_mount["size_available"] | int / 1024 ** 3) | round(1) }}
GiB free.
- name: Create the shared configuration and data directory roots
ansible.builtin.file:
path: "{{ item.path }}"
state: directory
owner: root
group: root
mode: "{{ item.mode }}"
loop: "{{ host_baseline_directories }}"
loop_control:
label: "{{ item.path }}"
llm-server/host/ansible/roles/rocm_container_probe/README.md
+17
View File
@@ -0,0 +1,17 @@
# Rootless ROCm container probe
This verification role pulls one immutable amd64 manifest of the official
`llama.cpp` `server-rocm` image into the `gnai-testing` rootless Podman store.
It verifies the OCI image revision and build version, then runs
`rocminfo` and `llama-server --list-devices` in ephemeral containers.
The probe uses the same security-relevant options planned for the permanent
Quadlet: `/dev/kfd`, only the discovered AMD render nodes and
`GroupAdd=keep-groups`. It sets no graphics-architecture override, publishes no
port, mounts no model and removes the probe container after execution. The
pinned image remains in the Testing store for the subsequent `llama.cpp`
deployment.
Success proves that ROCm sees two `gfx1030` agents and that `llama.cpp` exposes
both as usable ROCm devices to the rootless Testing container. It does not yet
prove model loading, multi-GPU splitting or inference.
llm-server/host/ansible/roles/rocm_container_probe/defaults/main.yml
+12
View File
@@ -0,0 +1,12 @@
---
rocm_probe_account:
name: gnai-testing
uid: 981
home: /var/lib/ghostnet-ai/testing
rocm_probe_image: >-
ghcr.io/ggml-org/llama.cpp@sha256:b429b8be10d3d0353a09c98f4ab840fe516f779858a28f9fd4e5bdf10bc63cb4
rocm_probe_expected_image_revision: f072b103714dfa1eee531f80b24512faf38e3dd2
rocm_probe_expected_image_version: b11058
rocm_probe_expected_architecture: gfx1030
rocm_probe_expected_device_count: 2
llm-server/host/ansible/roles/rocm_container_probe/tasks/main.yml
+204
View File
@@ -0,0 +1,204 @@
---
- name: Verify ROCm probe declarations and discovered devices
ansible.builtin.assert:
that:
- rocm_probe_account.name | length > 0
- rocm_probe_account.uid | int > 0
- rocm_probe_account.home is match("^/")
- rocm_probe_image is match("^ghcr\.io/.+@sha256:[0-9a-f]{64}$")
- rocm_probe_expected_image_revision is match("^[0-9a-f]{40}$")
- rocm_probe_expected_image_version | length > 0
- rocm_probe_expected_architecture is match("^gfx[0-9]+$")
- rocm_probe_expected_device_count | int > 0
- amd_gpu_render_devices | length == rocm_probe_expected_device_count | int
fail_msg: ROCm probe inputs or AMD render-device discovery are invalid.
- name: Read rootless Podman OCI runtime
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ rocm_probe_account.name }}"
- --
- /usr/bin/env
- "HOME={{ rocm_probe_account.home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ rocm_probe_account.uid }}"
- /usr/bin/podman
- info
- --format=json
chdir: "{{ rocm_probe_account.home }}"
changed_when: false
check_mode: false
register: rocm_probe_podman_info_command
- name: Parse rootless Podman OCI runtime
ansible.builtin.set_fact:
rocm_probe_podman_info: "{{ rocm_probe_podman_info_command.stdout | from_json }}"
- name: Require crun for rootless keep-groups support
ansible.builtin.assert:
that:
- rocm_probe_podman_info.host.ociRuntime.name == "crun"
- rocm_probe_podman_info.host.security.rootless
fail_msg: The ROCm probe requires rootless Podman with the crun OCI runtime.
- name: Check whether the pinned ROCm image is already present
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ rocm_probe_account.name }}"
- --
- /usr/bin/env
- "HOME={{ rocm_probe_account.home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ rocm_probe_account.uid }}"
- /usr/bin/podman
- image
- exists
- "{{ rocm_probe_image }}"
chdir: "{{ rocm_probe_account.home }}"
changed_when: false
failed_when: rocm_probe_image_exists.rc not in [0, 1]
check_mode: false
register: rocm_probe_image_exists
- name: Pull the pinned ROCm image into the Testing store
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ rocm_probe_account.name }}"
- --
- /usr/bin/env
- "HOME={{ rocm_probe_account.home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ rocm_probe_account.uid }}"
- /usr/bin/podman
- pull
- "{{ rocm_probe_image }}"
chdir: "{{ rocm_probe_account.home }}"
changed_when: true
when:
- not ansible_check_mode
- rocm_probe_image_exists.rc == 1
- name: Inspect the pinned ROCm image
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ rocm_probe_account.name }}"
- --
- /usr/bin/env
- "HOME={{ rocm_probe_account.home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ rocm_probe_account.uid }}"
- /usr/bin/podman
- image
- inspect
- --format=json
- "{{ rocm_probe_image }}"
chdir: "{{ rocm_probe_account.home }}"
changed_when: false
register: rocm_probe_image_inspect_command
when: not ansible_check_mode
- name: Parse the pinned ROCm image metadata
ansible.builtin.set_fact:
rocm_probe_image_metadata: >-
{{ (rocm_probe_image_inspect_command.stdout | from_json) | first }}
when: not ansible_check_mode
- name: Verify the pinned ROCm image metadata
ansible.builtin.assert:
that:
- >-
rocm_probe_image_metadata.Labels['org.opencontainers.image.revision'] ==
rocm_probe_expected_image_revision
- >-
rocm_probe_image_metadata.Labels['org.opencontainers.image.version'] ==
rocm_probe_expected_image_version
fail_msg: The pulled ROCm image metadata does not match the pinned build.
when: not ansible_check_mode
- name: Detect both AMD GPU architectures through ROCm in a rootless container
ansible.builtin.command:
argv: >-
{{
[
'/usr/sbin/runuser',
'--user', rocm_probe_account.name,
'--',
'/usr/bin/env',
'HOME=' + rocm_probe_account.home,
'XDG_RUNTIME_DIR=/run/user/' + (rocm_probe_account.uid | string),
'/usr/bin/podman',
'run', '--rm', '--pull=never',
'--group-add=keep-groups'
]
+ ((['/dev/kfd'] + amd_gpu_render_devices)
| map('regex_replace', '^', '--device=') | list)
+ [
'--entrypoint=/usr/bin/rocminfo',
rocm_probe_image
]
}}
chdir: "{{ rocm_probe_account.home }}"
changed_when: false
register: rocm_probe_rocminfo_output
when: not ansible_check_mode
- name: Verify ROCm reported both expected GPU architectures
ansible.builtin.assert:
that:
- >-
(rocm_probe_rocminfo_output.stdout + rocm_probe_rocminfo_output.stderr)
| regex_findall("(?m)^\\s*Name:\\s+" ~ rocm_probe_expected_architecture ~ "\\s*$")
| length == rocm_probe_expected_device_count | int
fail_msg: >-
rocminfo did not report the expected architecture for both AMD GPUs.
when: not ansible_check_mode
- name: Detect both AMD GPUs through llama.cpp in a rootless container
ansible.builtin.command:
argv: >-
{{
[
'/usr/sbin/runuser',
'--user', rocm_probe_account.name,
'--',
'/usr/bin/env',
'HOME=' + rocm_probe_account.home,
'XDG_RUNTIME_DIR=/run/user/' + (rocm_probe_account.uid | string),
'/usr/bin/podman',
'run', '--rm', '--pull=never',
'--group-add=keep-groups'
]
+ ((['/dev/kfd'] + amd_gpu_render_devices)
| map('regex_replace', '^', '--device=') | list)
+ [rocm_probe_image, '--list-devices']
}}
chdir: "{{ rocm_probe_account.home }}"
changed_when: false
register: rocm_probe_device_output
when: not ansible_check_mode
- name: Verify llama.cpp reported both expected ROCm devices
ansible.builtin.assert:
that:
- >-
(rocm_probe_device_output.stdout + rocm_probe_device_output.stderr)
| regex_findall("(?m)^\s*ROCm[0-9]+:")
| unique | length == rocm_probe_expected_device_count | int
fail_msg: >-
The rootless llama.cpp ROCm image did not report both expected GPUs.
when: not ansible_check_mode
- name: Report verified rootless ROCm devices
ansible.builtin.debug:
msg:
account: "{{ rocm_probe_account.name }}"
image: "{{ rocm_probe_image }}"
image_version: "{{ rocm_probe_expected_image_version }}"
image_revision: "{{ rocm_probe_expected_image_revision }}"
architecture: "{{ rocm_probe_expected_architecture }}"
devices: "{{ amd_gpu_render_devices }}"
when: not ansible_check_mode
llm-server/host/ansible/roles/workload_accounts/README.md
+17
View File
@@ -0,0 +1,17 @@
# Workload accounts
The `workload_accounts` role creates separate locked service identities for AI
Production, AI Testing and CPU-based video encoding. Each identity receives a
stable UID/GID, a non-overlapping subordinate ID range, a private data root, an
administrator-managed rootless Quadlet directory and a persistent systemd user
manager.
Configuration remains owned by root and is readable by the corresponding
service group. Workload data and the rootless Podman store belong to the
service account. The role verifies that all three Podman stores are rootless
and use distinct graph roots.
The two AI accounts receive `render` as an explicit supplementary-group
baseline; the encoding account receives no GPU group. Hardware discovery and
effective device-access checks belong to `amd_gpu_access`. This role does not
define containers, networks, schedules or resource limits.
llm-server/host/ansible/roles/workload_accounts/defaults/main.yml
+45
View File
@@ -0,0 +1,45 @@
---
workload_accounts:
- name: gnai-production
uid: 980
gid: 980
comment: Ghostnet AI Production workload
home: /var/lib/ghostnet-ai/production
config_directory: /etc/ghostnet-ai/production
subid_start: 362144
supplementary_groups:
- render
- name: gnai-testing
uid: 981
gid: 981
comment: Ghostnet AI Testing workload
home: /var/lib/ghostnet-ai/testing
config_directory: /etc/ghostnet-ai/testing
subid_start: 427680
supplementary_groups:
- render
- name: gn-encoding
uid: 982
gid: 982
comment: Ghostnet video encoding workload
home: /var/lib/ghostnet-encoding
config_directory: /etc/ghostnet-encoding
subid_start: 493216
supplementary_groups: []
workload_subid_count: 65536
workload_quadlet_user_directory: /etc/containers/systemd/users
workload_data_directories:
- path: /var/lib/ghostnet-encoding/input
owner: 982
group: 982
mode: "0750"
- path: /var/lib/ghostnet-encoding/output
owner: 982
group: 982
mode: "0750"
- path: /var/lib/ghostnet-encoding/work
owner: 982
group: 982
mode: "0750"
llm-server/host/ansible/roles/workload_accounts/tasks/main.yml
+207
View File
@@ -0,0 +1,207 @@
---
- name: Verify workload account declarations
ansible.builtin.assert:
that:
- workload_accounts | map(attribute="name") | unique | length == workload_accounts | length
- workload_accounts | map(attribute="uid") | unique | length == workload_accounts | length
- workload_accounts | map(attribute="gid") | unique | length == workload_accounts | length
- >-
workload_accounts | map(attribute="subid_start") | unique | length ==
workload_accounts | length
fail_msg: Workload account names and numeric IDs must be unique.
- name: Create dedicated workload groups
ansible.builtin.group:
name: "{{ workload_account.name }}"
gid: "{{ workload_account.gid }}"
system: true
state: present
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
- name: Create locked workload service accounts
ansible.builtin.user:
name: "{{ workload_account.name }}"
uid: "{{ workload_account.uid }}"
group: "{{ workload_account.name }}"
comment: "{{ workload_account.comment }}"
home: "{{ workload_account.home }}"
shell: /usr/sbin/nologin
groups: "{{ workload_account.supplementary_groups | join(',') }}"
append: false
create_home: true
system: true
password_lock: true
state: present
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
- name: Assign deterministic subordinate UID ranges
ansible.builtin.lineinfile:
path: /etc/subuid
regexp: "^{{ workload_account.name | regex_escape }}:"
line: >-
{{ workload_account.name }}:{{ workload_account.subid_start }}:{{ workload_subid_count }}
owner: root
group: root
mode: "0644"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
- name: Assign deterministic subordinate GID ranges
ansible.builtin.lineinfile:
path: /etc/subgid
regexp: "^{{ workload_account.name | regex_escape }}:"
line: >-
{{ workload_account.name }}:{{ workload_account.subid_start }}:{{ workload_subid_count }}
owner: root
group: root
mode: "0644"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
- name: Assign workload data roots to their service accounts
ansible.builtin.file:
path: "{{ workload_account.home }}"
state: directory
owner: "{{ workload_account.uid }}"
group: "{{ workload_account.gid }}"
mode: "0750"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.home }}"
- name: Keep workload configuration root-managed
ansible.builtin.file:
path: "{{ workload_account.config_directory }}"
state: directory
owner: root
group: "{{ workload_account.gid }}"
mode: "0750"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.config_directory }}"
- name: Create workload-specific data directories
ansible.builtin.file:
path: "{{ workload_directory.path }}"
state: directory
owner: "{{ workload_directory.owner }}"
group: "{{ workload_directory.group }}"
mode: "{{ workload_directory.mode }}"
loop: "{{ workload_data_directories }}"
loop_control:
loop_var: workload_directory
label: "{{ workload_directory.path }}"
- name: Create the administrator-managed rootless Quadlet root
ansible.builtin.file:
path: "{{ workload_quadlet_user_directory }}"
state: directory
owner: root
group: root
mode: "0755"
- name: Create separate administrator-managed Quadlet directories
ansible.builtin.file:
path: "{{ workload_quadlet_user_directory }}/{{ workload_account.uid }}"
state: directory
owner: root
group: root
mode: "0755"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }} (UID {{ workload_account.uid }})"
- name: Enable persistent user managers for rootless Quadlets
ansible.builtin.command:
argv:
- /usr/bin/loginctl
- enable-linger
- "{{ workload_account.name }}"
creates: "/var/lib/systemd/linger/{{ workload_account.name }}"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
- name: Wait for persistent user managers
ansible.builtin.command:
argv:
- /usr/bin/systemctl
- is-active
- "user@{{ workload_account.uid }}.service"
changed_when: false
register: workload_user_manager_status
retries: 5
delay: 1
until: workload_user_manager_status.stdout == "active"
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
when: not ansible_check_mode
- name: Initialize and inspect each rootless Podman store
ansible.builtin.command:
argv:
- /usr/sbin/runuser
- --user
- "{{ workload_account.name }}"
- --
- /usr/bin/env
- "HOME={{ workload_account.home }}"
- "XDG_RUNTIME_DIR=/run/user/{{ workload_account.uid }}"
- /usr/bin/podman
- info
- --format=json
chdir: "{{ workload_account.home }}"
changed_when: false
register: workload_podman_info_commands
loop: "{{ workload_accounts }}"
loop_control:
loop_var: workload_account
label: "{{ workload_account.name }}"
when: not ansible_check_mode
- name: Parse rootless Podman information
ansible.builtin.set_fact:
workload_podman_infos: >-
{{ workload_podman_info_commands.results
| map(attribute="stdout")
| map("from_json")
| list }}
when: not ansible_check_mode
- name: Verify isolated rootless Podman stores
ansible.builtin.assert:
that:
- >-
workload_podman_infos | map(attribute="host.security.rootless")
| select("equalto", true) | list | length == workload_accounts | length
- >-
workload_podman_infos | map(attribute="store.graphRoot")
| unique | list | length == workload_accounts | length
fail_msg: Workload accounts do not have separate rootless Podman stores.
success_msg: All workload accounts use separate rootless Podman stores.
when: not ansible_check_mode
- name: Report workload account isolation
ansible.builtin.debug:
msg:
accounts: "{{ workload_accounts | map(attribute='name') | list }}"
uids: "{{ workload_accounts | map(attribute='uid') | list }}"
graph_roots: >-
{{ workload_podman_infos | map(attribute="store.graphRoot") | list }}
when: not ansible_check_mode