218 lines
8.4 KiB
Markdown
218 lines
8.4 KiB
Markdown
## 🤖 GhostNet OpenClaw & Multi-Vendor Ollama Pipeline
|
|
A highly optimized, hardware-accelerated local AI infrastructure leveraging OpenClaw, Ollama, and a modular MCP (Model Context Protocol) gateway running inside containerized isolation. Tailored specifically for Podman (rootless/SELinux) architectures on rolling-release host systems (like CachyOS / Arch Linux).
|
|
|
|
------------------------------
|
|
|
|
## Quick Start
|
|
|
|
### 1. Environment Setup
|
|
Clone the repository and prepare your environment configuration:
|
|
`cp .env.example .env`
|
|
|
|
Open `.env` and set your `GPU_TYPE` (amd, nvidia, or intel) and your desired models.
|
|
Set `OPENCLAW_DEFAULT_MODEL` to the model OpenClaw should use by default.
|
|
|
|
### 2. Boot the Stack
|
|
Use the integrated management script, which automatically selects the correct vendor-specific compose file:
|
|
|
|
```bash
|
|
# Start the stack in the background
|
|
./ghostnet.sh up
|
|
```
|
|
|
|
```bash
|
|
# For debugging: Stop everything and start in the foreground (Attached Mode)
|
|
./ghostnet.sh attached
|
|
```
|
|
|
|
### 3. Open the Interactive Chat
|
|
`podman exec -it openclaw-agent openclaw chat`
|
|
|
|
## Using the Stack with Cline in VS Code
|
|
|
|
You do not need a second local model runtime for Cline. The existing Ollama container is already published on the host at `http://127.0.0.1:11434`.
|
|
|
|
### 1. Configure Cline to use Ollama
|
|
|
|
In Cline, select the Ollama provider and use:
|
|
|
|
```text
|
|
Base URL: http://127.0.0.1:11434
|
|
Model: qwen3-coding
|
|
```
|
|
|
|
Use `ollama list` inside the container or on the host to see the exact model names currently available.
|
|
|
|
Recommended custom profiles for this stack:
|
|
|
|
```text
|
|
qwen3-general -> Qwen3 14B, universal profile for OpenClaw chats
|
|
qwen3-coding -> Qwen3 14B, balanced default for agentic coding
|
|
qwen25-coding -> Qwen2.5-Coder 14B, code-first edit and repair profile
|
|
deepseek-coding -> DeepSeek-Coder-V2 16B, long-context repo analysis profile
|
|
```
|
|
|
|
### 2. Reuse the existing MCP router for tools
|
|
|
|
The GhostNet router exposes an MCP SSE endpoint at `http://127.0.0.1:3000/sse`. Add it to your Cline MCP configuration:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"ghostnet-router": {
|
|
"transport": "sse",
|
|
"url": "http://127.0.0.1:3000/sse"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
This lets Cline reuse the same routed tool surface that OpenClaw uses internally.
|
|
|
|
### 3. What Cline does not need
|
|
|
|
Cline does not need to connect to the OpenClaw gateway on port 8080 for normal local usage. The direct path is:
|
|
|
|
* Cline -> Ollama for model inference
|
|
* Cline -> GhostNet router for MCP tools
|
|
|
|
If you change the host binding, use `TOOL_ROUTER_BIND` and `TOOL_ROUTER_HOST_PORT` in `.env`.
|
|
|
|
## Benchmarking Custom Models
|
|
|
|
The repository now ships two complementary benchmark entry points:
|
|
|
|
* `./scripts/benchmark-stack.sh` and `./scripts/benchmark-model-suite.sh` exercise the full OpenClaw agent path.
|
|
* `./scripts/benchmark-codeneedle-suite.sh` runs the upstream CodeNeedle recall benchmark against the same local Ollama instance.
|
|
|
|
The benchmark flow is split intentionally into runtime artifacts and curated results:
|
|
|
|
* Runtime artifacts go to `.cache/benchmarks/existing-suite/runs/$RUN_ID/`.
|
|
* Final benchmark outputs go to `docs/benchmarks/existing-suite/$RUN_ID/`.
|
|
|
|
Generated result sets now include more than raw `summary.txt` files:
|
|
|
|
* `report.html` per model for a quick interactive dashboard.
|
|
* `RESULTS.md` per model for git-friendly lab notes.
|
|
* `SUITE_COMPARISON.html` for side-by-side multi-model comparison.
|
|
|
|
These reports are generated automatically at the end of the benchmark scripts through `scripts/lib/report-generator.sh`.
|
|
|
|
Current timeout defaults were raised for slower local models and more reliable result capture:
|
|
|
|
* `benchmark-stack.sh`: 600s
|
|
* `benchmark-model-suite.sh`: 600s
|
|
* `benchmark-codeneedle-suite.sh`: 600s
|
|
|
|
The current Ollama tuning used for benchmark stability is:
|
|
|
|
* `OLLAMA_KEEP_ALIVE=-1`
|
|
* `OLLAMA_KV_CACHE_TYPE=q8_0`
|
|
* `OLLAMA_MAX_LOADED_MODELS=1`
|
|
* `OLLAMA_NUM_PARALLEL=1`
|
|
* `SHM_SIZE=16gb`
|
|
|
|
OpenClaw path examples:
|
|
|
|
```bash
|
|
./scripts/benchmark-stack.sh --model ollama/qwen3-coding:latest --lines 140 --timeout-sec 600 --no-start
|
|
./scripts/benchmark-model-suite.sh --lines 140 --timeout-sec 600
|
|
```
|
|
|
|
CodeNeedle path example:
|
|
|
|
```bash
|
|
./scripts/benchmark-codeneedle-suite.sh \
|
|
--model qwen3-coding:latest \
|
|
--model qwen25-coding:latest \
|
|
--model deepseek-coding:latest
|
|
```
|
|
|
|
`benchmark-codeneedle-suite.sh` clones the upstream CodeNeedle repository into `.cache/benchmarks/codeneedle/upstream`, builds its Dockerfile with the configured container engine, and benchmarks the standard `http_server` and `jquery` corpora against `http://127.0.0.1:11434`.
|
|
|
|
Use `--runtime-root`, `--results-root`, or the legacy `--artifact-root` overrides if you want to keep artifacts somewhere else for a run.
|
|
|
|
Typical curated benchmark outputs look like this:
|
|
|
|
```text
|
|
docs/benchmarks/existing-suite/model-suite-TIMESTAMP/
|
|
├── qwen3-coding/
|
|
│ ├── summary.txt
|
|
│ ├── assistant-output.txt
|
|
│ ├── report.html
|
|
│ └── RESULTS.md
|
|
├── qwen25-coding/
|
|
│ └── ...
|
|
└── SUITE_COMPARISON.html
|
|
```
|
|
|
|
## Technical Architecture & Key Highlights
|
|
This setup relies on unique architectural design patterns engineered to overcome container engines boundaries and system strictness:
|
|
|
|
```
|
|
+-------------------------------------------------------------+
|
|
| Host Hardware (GPU) |
|
|
+-------------------------------------------------------------+
|
|
^
|
|
| Passthrough (CDI / DRI / rwm)
|
|
v
|
|
+-------------------------------------------------------------+
|
|
| ollama (Container) |
|
|
+-------------------------------------------------------------+
|
|
^
|
|
| (Internal Bridge Net: Port 11434)
|
|
v
|
|
+-------------------------------------------------------------+
|
|
| openclaw-ollama-bridge (Sidecar) |
|
|
| - Tunnels 127.0.0.1:11434 directly to ollama:11434 |
|
|
+-------------------------------------------------------------+
|
|
^
|
|
| (Shared Network Namespace)
|
|
v
|
|
+-------------------------------------------------------------+
|
|
| openclaw-agent (Container) |
|
|
| - Runs 'openclaw gateway start' on the main thread |
|
|
| - Injects 'openclaw chat' (TUI) via podman exec |
|
|
+-------------------------------------------------------------+
|
|
|
|
```
|
|
|
|
## MCP Microservice Gateway (The GhostNet Router)
|
|
* **The Challenge:** OpenClaw needs to scale its capabilities (Search, Filesystem, DB) without bloating the main agent container or creating dependency hell.
|
|
* **The Solution:** A centralized **Python-based MCP Router**. It acts as a single API Gateway that aggregates multiple "Skills" (Bridges).
|
|
* **Key Feature:** The Router maintains persistent, asynchronous SSE connections to sub-services (like `searchfetch`) using a robust `maintain_connection` logic, ensuring the agent always has access to live tools without manual re-initialization.
|
|
|
|
## The Loopback TUI Bypass (The Socat Sidecar)
|
|
* **The Challenge:** The OpenClaw TUI strictly forces connections to `127.0.0.1:11434`, ignoring environment variables.
|
|
* **The Solution:** A Socat Sidecar (`network_mode: "service:agent"`) that tunnels local loopback traffic directly to the isolated Ollama container.
|
|
|
|
## Monolithic Multi-Vendor Composability
|
|
* Standalone compose files (`compose.amd.yaml` etc.) prevent the "Schema-Drop" bug of `podman-compose`, ensuring that `devices:` and `group_add:` mappings for ROCm/CUDA are never silently discarded.
|
|
|
|
## Rootless Storage Mandates (keep-id & ,U)
|
|
* Synchronization of host/container UIDs via `userns_mode: "keep-id"` combined with `:Z,U` flags. This allows the Node.js agent to write to host-mounted workspaces while maintaining strict SELinux compliance.
|
|
|
|
## Infrastructure Verification
|
|
|
|
```bash
|
|
# Inspect Engine Acceleration
|
|
# Success: PROCESSOR reads 100% GPU.
|
|
`podman exec -it ollama ollama ps`
|
|
```
|
|
|
|
```bash
|
|
# Check MCP Router Integrity
|
|
# Verify that the Gateway is alive and the Search-Bridge is successfully integrated.
|
|
curl -N http://tool-router:3000/sse
|
|
```
|
|
|
|
```bash
|
|
# Shut down the environment cleanly without state deadlocks
|
|
podman-compose -f compose.<vendor>.yaml down
|
|
```
|
|
|
|
```bash
|
|
# Run system integrity and validation checks
|
|
podman exec -it openclaw-agent openclaw doctor
|
|
```
|