jazzymc/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(services): correct Ollama iGPU entry — remove flash-attn/q8_0, note large-prompt crash
ci/woodpecker/push/woodpecker Pipeline was successful
Details

Browse Source 
The flash-attention + q8_0 KV cache settings were removed after they (and the
gfx1103 ROCm path generally) crash on large prompts. Document that the 780M iGPU
is direct/interactive-chat only, not an agent backend (Hermes ~15.5k-token prompt
crashed every call; reverted to cloud model).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kaloyan Danchev
2026-06-29 22:14:41 +03:00
parent 3a28e42543
commit 2a34c42270
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/03-SERVICES-OTHER.md
+2 -2
View File
@@ -510,13 +510,13 @@ Non-critical services that enhance functionality but don't affect core network o
|---------|-------| |---------|-------|
| `HSA_OVERRIDE_GFX_VERSION` | 11.0.0 (report 780M as gfx1100) | | `HSA_OVERRIDE_GFX_VERSION` | 11.0.0 (report 780M as gfx1100) |
| `OLLAMA_IGPU_ENABLE` | 1 (Ollama 0.30+ drops iGPUs otherwise) | | `OLLAMA_IGPU_ENABLE` | 1 (Ollama 0.30+ drops iGPUs otherwise) |
| `OLLAMA_FLASH_ATTENTION` | 1 |
| `OLLAMA_KV_CACHE_TYPE` | q8_0 |
| devices | `/dev/kfd`, `/dev/dri` | | devices | `/dev/kfd`, `/dev/dri` |
| group_add | `18` (video) | | group_add | `18` (video) |
**Verified:** Ornith 1.0 9B GGUF runs 100% on GPU (33/33 layers), ~11 tok/s. iGPU is RAM-bandwidth bound (shared GTT, ~23 GiB addressable). 9B fits; 35B RAM-tight (no swap); 397B not feasible. **Verified:** Ornith 1.0 9B GGUF runs 100% on GPU (33/33 layers), ~11 tok/s. iGPU is RAM-bandwidth bound (shared GTT, ~23 GiB addressable). 9B fits; 35B RAM-tight (no swap); 397B not feasible.
**⚠️ Limitation:** large prompts (~10k+ tokens) crash the iGPU (`ROCm unspecified launch failure` — gfx1103 rocBLAS). Tried flash-attn off, gfx1100/gfx1102, `num_batch=128` — all crash, so `OLLAMA_FLASH_ATTENTION`/`OLLAMA_KV_CACHE_TYPE=q8_0` were removed. Good for direct/interactive chat only, **not** an agent backend with big system prompts (tried as Hermes' model with a ~15.5k-token prompt — crashed every call, reverted to cloud).
--- ---
## Stopped/Disabled Services ## Stopped/Disabled Services