Case study · llama.cpp · Vulkan · 19 to 20 October 2026
A 3-second pause, taken apart
My voice assistant paused about 3 seconds before every answer. The model was fast, so the time had to be going somewhere else. Scroll through the eight measurements that found it.
The model is not the problem
llama-bench read 16 prompt tokens in 0.11 s. The server took 1.4 to 2.5 s for the same work. The model is a hybrid, Qwen3.6-35B-A3B: part of it keeps a running state instead of a normal attention cache.
Ruled out on a test server
Batch size, context size and slot count, each changed on a separate server on a spare port so production stayed untouched. None moved the delay.
Two snapshots, 0.6 s each
To resume a conversation after its text changes, the server saves snapshots of the model's running state: 63 MB each, two per request. Those two saves were the whole delay.
Faster voice, broken agent
With snapshots off, voice answered in 0.5 s. But the agent re-formats its previous reply slightly, and a hybrid model cannot step its state back without a snapshot, so it re-read its 13,700-token prompt every turn. My first tests used voice-style conversations only; the full test of every service caught it.
No change
A newer engine build and four Vulkan settings gave the same timings. The cost was in the code.
Skipping GPU waits saved nothing
Each saved tensor goes through a synchronous GPU barrier and wait. Skipping the redundant ones took a snapshot from 0.65 s to 0.61 s. Discarded.
114 MB per second
The waits took 14 ms in total. The CPU copy took 7,738 ms for 880 MB. On this integrated GPU the memory is visible to the CPU but not cached by it, so every read crawls.
Read directly only from cached memory
Otherwise let the GPU copy into a cached staging buffer first, then read that.
// ggml_vk_buffer_read_2d(): direct host read only when it is fast if (src->memory_property_flags & vk::MemoryPropertyFlagBits::eHostVisible && src->device->uma && src->memory_property_flags & vk::MemoryPropertyFlagBits::eHostCached) {
Verification
Same answers, word for word
- The same scripted six-turn conversation at temperature 0 ran on the stock and fixed builds, in two modes: history resent verbatim, and history altered so the server must restore a snapshot.
- Replies were identical word for word in both modes; snapshot restores 4 of 4 in each build.
- The exact deployment, one library swapped, was rehearsed on a copy of the engine folder first, with a backup and a one-step undo.
Outcome. After deploying, I found that the Vulkan backend's maintainer had merged the same fix two weeks earlier (llama.cpp #30049), prompted by a benchmark on the same GPU. My engine version predated it. This is an independent diagnosis, not an upstream contribution. I now check a file's recent history before patching it.
Results
| Measured | Before | After | Main cause |
|---|---|---|---|
| Voice, time to first word | ≈ 3 s | ≈ 0.7 s | Engine fix, voice-only sampling setting |
| Agent, each later turn | 3.5 s | 1.1 to 1.4 s | Engine fix, smaller prompt |
| Agent, new conversation | 2.5 s | 1.2 s | Engine fix |
| Agent, task with code tool | 43.5 s | 23.0 s | Engine fix |
| Agent, task with web search | 11.7 s | 7.1 s | Engine fix |
| One state snapshot | 0.61 s | 0.03 s | Cached read path |
| Agent prompt per request | 13,718 tok | 11,560 tok | Removed two unused tools |
The first turn after the model loads is unchanged: the agent still reads its 11,600-token instructions once, about 21 to 24 s.
Method
How it was built and tested
I designed and directed this work; the code, builds and measurements were carried out by Claude Code running as my agent, with every change to a running system approved by me first. I set the goals (faster voice and agent, no lost function, no cloud inference), chose the trade-offs, and decided what to deploy and when to revert.
llama.cpp (C++, Vulkan) · Visual Studio 2022 Build Tools · Vulkan SDK · llama-bench · Python harnesses · Docker · WSL