Rustycage

Long conversations do not run out of model weights, they run out of KV cache. The attention state for prior tokens routinely exceeds the size of the weights themselves, and on a 8 GB consumer card that ceiling arrives early and hard — as an out-of-memory crash rather than a graceful slowdown.

vLLM solved this with PagedAttention and CPU swap. That solution is Python, and CUDA-only, and carries an infrastructure footprint that does not fit a lab of mixed older GPUs. llama.cpp, which does fit, has had PagedAttention sitting open as issue #1955 since June 2023 with no implementation merged and no active PR. Nobody had shipped block-level KV tiering for llama.cpp.

Rustycage is the attempt: keep llama.cpp's portability and hardware tolerance, add the tiering that makes long context survivable, and wrap the whole thing in an API surface that existing Ollama and OpenAI clients can talk to without modification.

A six-crate Cargo workspace over llama.cpp via llama-cpp-2:

Three API surfaces from one process: OpenAI v1 on 8080, Ollama on 11434, and NIM-style health and Prometheus metrics. A model requested by name that is not loaded gets loaded automatically, which is what makes it a drop-in rather than a migration.

The tiering. The current implementation is state-based and coarse: tier 1 is KV cache on GPU with offload_kqv=true, tier 2 is host RAM with the flag off, tier 3 is a full state file on disk via llama.cpp's state_save_file(). It operates at conversation granularity — an entire context saves and restores as a unit, in seconds, and it cannot swap mid-generation.

That is a real limitation and it is written down as one. The forked path adds a tier field to llama_kv_cell and eviction logic in slot allocation, which moves granularity from conversation to block without touching the attention kernels. The assessment behind that decision is in docs/kv-cache-fork-assessment.md, including the two heavier paths that were costed and deliberately not taken yet.

Roughly five months, in three distinct bursts rather than a steady line.

March 2026 — the shape. Initial commit on 24 March as Nollama, renamed to Rustycage the same day. The full README with API tables landed in that first sitting, before most of the code did. By 30 March the fork existed: "Add KV cache tiering: fork llama.cpp with GPU→CPU→Disk eviction," plus an Ubuntu installer with a curated model set. Eight commits in a week.

April to June — the long tail. Two commits a month. HuggingFace search fixes, quantization parsing, prompt formatting, embedding server context. The one structural piece was 7 June: unifying the OpenAI and Ollama schemas so both ports serve the same shapes, which arrived as the repo's only pull request.

Between those, on 2 May, two build scripts that turned out to matter more than they look: a native CUDA rebuild for Wintermute's older glibc and nvcc, and a separate build-and-publish path from mrwick. Those exist because of the deploy problems in Pitfalls.

August 2026 — the fleet. Ten commits in two days, 12–13 August, and the project stopped being a single-host program. Soundgarden — a resident attendant running across all phases with peer heartbeats. Model quarantine with post-download load testing. Incidents made inspectable, with LLM-driven failure triage that files GitHub issues automatically. Negotiated cohort upgrade for multi-host GPU rollouts. Then three consecutive commits hardening that rollout: stdin and version verification, cmake and bindgen checks, a pinned CUDA toolkit path.

One commit message from 12 August is the honest summary of that week: "Require curl verification against a running node before calling work done."

The near work is finishing what the fork started. The metadata-first path is in place but the eviction policy is naive — LRU with sequence-aware grouping is the next correctness step, and async promotion is what turns cold-block access from a stall into a prefetch. Neither is useful without tier hit-rate and promotion-latency metrics to show whether the tiering is helping or just moving bytes around.

After that comes a decision point that was deliberately deferred: whether the custom ggml_backend_tiered wrapper earns its roughly 3,000 lines over the current approach. That is a measurement, not a preference, and it should not be made before the Path 1 metrics exist.

Physical block redesign — true PagedAttention with sub-4% memory waste — is costed at 5,000+ lines and two to three months, and requires modifying CUDA flash attention kernels. It stays in Later until something forces it.

Finish Path 1 properly — LRU eviction, async promotion, and the tier metrics that make the Path 2 decision a measurement rather than an argument. Replace the --version smoke test with a real one that loads a model and serves a completion before a cohort cutover is allowed to proceed. Cut a first tagged release, since five months and 24 commits without one makes the rollout scripts do work that version tags should be doing.