minfer Architecture Roadmap

Status: analysis + prioritized backlog (no code changed). Baseline: minfer HEAD = 293eb19 (2026-09-15, working tree clean). Method: direct reading of src/ (44,105 LOC) and docs/ (~15.6k LOC). Every claim about minfer carries a file:line anchor; claims taken from a document rather than code are marked "per docs".

Scope. Architecture-level work on the system layers: IR, scheduler, allocator, KV/sequence state, batching, backend abstraction, and model/quant coverage. Kernel micro-optimization is out of scope — CUDA_OPTIMIZATION.md and METAL_OPTIMIZATIONS.md cover it, and both campaigns are formally converged. Model-family selection is a separate document: docs/MODEL-SUPPORT-ROADMAP.md.

0. Verdict

minfer's kernel work is complete and competitive for the architectures it supports: eight quantized weight types on CPU, Metal and CUDA; int8 MMQ (quantized matrix-multiply on tensor cores) prefill and dp4a MMVQ (quantized matrix-vector multiply) decode on CUDA; simdgroup GEMM (general matrix-multiply) and flash attention on Metal; speculative decoding with adaptive draft depth. On the measured CUDA target, 7B Q4_K_M reaches ~3581 tok/s prefill and ~51.2 tok/s decode; on Metal, 7B decode is at parity and prefill reaches ~0.82–0.85× of the achievable ceiling, with the residual measured as not source-addressable.

The remaining work is at the system layer, and three items dominate it:

1. No multi-sequence batching. The IR, the attention kernels, the KV (Key/Value) allocator and the server are all single-sequence. E1/E1b made the attention side sequence-aware and E2 landed sequence-addressable batches and a batching server worker; the CPU payoff measured negative while the GPU payoff was 1.9x, so the default follows the device (E6: batches on CUDA and, since #44 part (b) landed the one-range attn_span read and the matching write/move side on a Mac (2026-10-06), on Metal too; off on CPU).
2. KV cache is a fixed per-layer buffer, not a sequence-addressable cell store: no sequence ids, no eviction/context shift, no defragmentation, no state save/restore, no quantized KV. (Phase C has since closed every one of those: C1/C2 the cell store and physical removal/shift, C3 the compaction — a pure planner, the counters, Backend::copy_cells on CPU and CUDA, the K re-rope a move needs while positions are cells, and a model-level continuation gate — C4 the packed Q8_0 cache (CPU, CUDA and Metal — the last in #310) and C5 the session container. What is still open is driving the compaction from the server's serving model, which reserves every slot once and packed, the CUDA packed-decode residual's stall attribution (#212 — #144/#186/#202 landed the fused epilogue, the dp4a dot and the FA prefill), and Metal's packed half (G5, #44; #310, enabled on Metal — mechanism A native packed decode + mechanism B f32 stage). See §2.4.)
3. The server has no persistent context. Every request builds a fresh GraphCache (KV regions + device pool re-allocated, CUDA Graph capture re-warmed) and re-prefills the whole prompt. Fixed in B2/B3 for the prefix-matched case: a slot keeps its cache and reuses the rows its prompt already covers (measured ≈11× TTFT on a second turn).

Everything else — IR expressiveness (views, multi-output), memory placement policy (VRAM budget, layer offload), backend pluggability, additional model families, quantizer tooling — is real but secondary, and mostly enabled by fixing (1) and (2) first.

Ranked backlog (full list with numbering in §3)

1. Current state

1.1 Scale

1.2 Landed surface

Compute graph (build → assign → fuse → alloc → execute) with params-only reuse (graph/cache.rs:47, graph/params.rs:51); per-op backend assignment (graph/scheduler.rs:60); liveness allocator with persistent per-layer KV regions (graph/alloc.rs:165, :385); eight quantized weight types (Q4_0/Q4_1/Q5_0/Q5_1/Q8_0/Q4_K/Q5_K/Q6_K) on CPU + Metal + CUDA; two model families (qwen2, qwen3 dense); GGUF v3 with split parts and Metal zero-copy mmap; self-contained BPE tokenizer; chat templates via minijinja; greedy/penalty/top-k/top-p/temperature sampling; speculative decoding with adaptive depth; multi-turn conversation CLI; OpenAI-compatible server (chat completions, streaming, multi-slot); a runtime graph introspection stack (MINFER_TRACE per-node capture, live SSE visualizer, pipeline view, DOT/JSON export).

1.3 Documentation hygiene

The index-level documents (README.md, ARCHITECTURE.md, CPU_OPTIMIZATIONS.md, QWEN3-SUPPORT-PLAN.md, OPENAI-CHAT-API-PLAN.md, MODEL-SUPPORT-ROADMAP.md) are the entry points AGENTS.md routes readers through, so a stale statement there costs more than one buried in a campaign record.

A pass on 2026-09-15 fixed these stale statements:

• README.md and ARCHITECTURE.md — the removed BiasRope fusion in the two architecture diagrams, plus the fusion.rs row in the module table (the pass produces only SwiGLU; the decode fusions are built by the model code, not by FusionPass).
• ARCHITECTURE.md — the "n_out tail-row optimization is not yet ported" claim (it landed as the G3 work), and the RopeStyle "already covers Qwen2 and Llama" claim (only the non-interleaved form is wired up).
• CPU_OPTIMIZATIONS.md — the pre-compute-graph snapshot is now banner-marked as historical, with the still-open AVX2 gap pointed at §2.7.
• QWEN3-SUPPORT-PLAN.md — the self-contradictory status line.
• OPENAI-CHAT-API-PLAN.md — the "KV is f32" note (f16 is auto-selected for 7B-class models).
• MODEL-SUPPORT-ROADMAP.md — the misleading status line.

Keep the invariant: when a change lands, update the index rows in the same commit. No outstanding doc-truth work item remains.

2. Layer-by-layer analysis

The layer names follow the vocabulary of docs/GLOSSARY.md, which classifies the campaign's terms into seven layers; §2.8 adds one cross-cutting layer for ops, safety, testing and observability.

Legend for Gap: 🔴 structural (blocks a class of use cases) · 🟠 material (measurable capability or performance loss) · 🟡 hygiene.

2.1 L1 — Graph IR and build

Today. ComputeGraph is a flat, topologically ordered vector of CNode (graph/mod.rs:83-115). Each node has exactly one output, a static [usize; 4] shape, and an Op carrying its full payload. Op::View { offset, shape }, Reshape, Permute exist (ops.rs:105-116) but both backends execute them as identity copies (cpu_backend.rs:384-388, cuda_backend.rs:433-438, metal_backend.rs likewise) — the allocator has no aliasing for views, so a view costs a full tensor copy. The builder appends sources before consumers, so node-id order is the execution order (scheduler.rs:214).

Gap. 🔴 Strided, zero-copy views and multi-output nodes were missing — closed by D1 (2026-09-19): views are zero-copy with allocator-known aliasing, and GraphBuilder::split_parts exposes a node's single output as independent parts, so what remains under this header is model work (item 17), not IR work. Without them the IR cannot express slicing, concatenation, or a copy between overlapping regions as compositions of primitive ops — which is why minfer carries four hand-written decode-specific fused ops instead: FusedQKV, QkvBiasRopeStore, FusedFFN, FusedQkvNorm (ops.rs:125-154). Each one costs changes in five places: the builder constructor (builder.rs:193-281), the allocator's special cases (alloc.rs:232-273), the scheduler's kv_pair resolution (scheduler.rs:261-271), every backend's supports_op + execute_node, and both models' build_graph. BatchMatMul is deferred for the same reason (COMPUTE-GRAPH-DESIGN.md §5.4).

Consequence for the model work. MODEL-SUPPORT-ROADMAP.md ranks new model families by how much of the existing graph they reuse. That ranking is accurate for parameter-isomorphic architectures (Llama/Mistral) but not for anything needing new structure: MoE (Mixture-of-Experts) needs mul_mat_id (3-D expert indexing), MLA (Multi-head Latent Attention) needs a different KV layout and a view-based latent split, and sliding-window attention needs a mask parameter threaded into attention. Each of those currently pays the five-site cost.

Recommendation. Add strided views with allocator-known aliasing, and let a node carry a small output list. Fusions can then be produced by the fusion pass from compositions rather than hand-written as IR variants, and the four decode fusions become compiler output instead of bespoke ops.

2.2 L2 — Scheduler and execution

Today. assign_backends (scheduler.rs:68-83) walks nodes in build order and assigns the highest-priority backend whose supports_op returns true. split_graph partitions into contiguous same-backend runs derived from that positional assignment (scheduler.rs:73-120); split inputs/outputs are the crossing edges. Execution then walks splits, calling sync_backend(prev) and copy_across for each cross edge (scheduler.rs:176-189), and finally sync_backend.

Gap. 🟠 Two distinct issues.

1. No cost model. Splits are positional. A single unsupported op in the middle of an otherwise GPU graph produces three splits with two host round trips; the assignment never considers the cost of the resulting movement, or whether a neighbouring op could be moved so the op becomes supportable. In practice this is masked today because the supported models build a single GPU split on both GPU backends (CUDA-BACKEND-DESIGN.md:313-315), but it is exactly what breaks the first time an op is unsupported — e.g. interleaved RoPE on CUDA (cuda_backend.rs:1522), or FusedQkvNorm, which CUDA does not advertise (cuda_backend.rs:1905) while Metal does (metal_backend.rs:686).
2. Synchronous, host-mediated cross-backend movement. — partly closed by F5 (#58, 2026-09-24): the boundary is now two registered phases. Phase A (copy_across → BackendEntry::copy_cross) enqueues the transfer — for a CUDA source a device→host cudaMemcpyAsync into a pinned slab plus a cudaEventRecord, where the pre-F5 code did copy_to_cpu (a full stream sync + a blocking cudaMemcpy) — and phase B (await_cross) waits on the recorded event at the documented synchronization point, once per staged input. The CPU's hooks are a synchronous host round trip and a no-op (it has no device memory); Metal runs the same two phases — a MTLBlitCommandEncoder copy into a shared staging buffer plus encodeSignalEvent on a MTLSharedEvent, waited once by a bounded waitUntilSignaledValue — ported by #137 and verified on a Mac (2026-10-05: blocking_host_copies 21 → 0, copies == waits, bitwise identical). Counters (graph/copystats.rs), the per-backend table, the enumeration of the synchronization points and the measured before/after are in docs/BACKEND-REGISTRY-DESIGN.md §11 and the plan's F5 record. What remains: nothing filed for the boundary copy. The wait is already deferred to the consumer's first use (#138), so a boundary with several staged inputs holds them in flight at once; true cross-split overlap has no target on the reachable macOS topology, and #300 recorded that negative result — a single device split feeding a host consumer whose first node reads the dominant staged tensor (docs/BACKEND-REGISTRY-DESIGN.md §11.6). Multi-device execution is still the place that would need it.

Also: the cross-boundary staging map is keyed by node id alone (alloc.rs:34), so a node consumed by two different foreign backends can only have one staging buffer; the scheduler compensates by filtering on backend (scheduler.rs:252-255), which is correct only while at most two backends are enabled. Fixed in A5: the map is keyed by (node, destination backend), one node can feed two foreign consumers, and the consumer-side filter is gone.

Recommendation. Add an assignment pass that (a) propagates support backwards from unsupported ops and (b) scores a candidate assignment by crossing count; and replace the host round trip with an async copy plus a recorded event when the source and destination are different devices (F5 landed the CUDA half of this; the Metal half and the actual consumer-side waiting that would let a copy overlap independent work are the remaining follow-ups). Neither is urgent today; both are prerequisites for multi-device execution and for any heterogeneous split.

2.3 L3 — Allocator and memory

Today. GraphAllocator::alloc_graph (alloc.rs:513-521) rebuilds the whole node→buffer mapping on every graph rebuild: it frees every previously live buffer back to the backend pool (:165-177), recomputes last_use over build order, and re-allocates. Buffer pools are per backend, not unified (alloc.rs:315-335), and allocation is by exact element count — both the Metal (metal_backend.rs:322-332) and CUDA (cuda_backend.rs:1333-1353) pools scan a free list for an exact byte-length match and otherwise allocate fresh. free_buffer never returns memory to the device (cuda_backend.rs:1355-1362).

Gap. 🟠 Three consequences.

1. Per-rebuild teardown. Non-persistent buffers are freed and re-derived on every topology change. For a decode loop with a stable shape this happens once; for anything with a varying n_tokens (speculative decoding with adaptive depth, server requests of differing prompt lengths, future continuous batching) it happens every time, together with a full device re-allocation for any size not seen before.
2. No size classes / no rounding. Because matching is exact, a workload touching k distinct activation shapes ends up with k sets of live buffers. The pool is a per-GraphCache high-water mark that never shrinks (documented as accepted debt, CUDA-BACKEND-DESIGN.md:447) — but that was reasoned about for a fixed-shape CLI run, not for a server whose prompt lengths vary per request. E4 S2 (2026-09-23) rounds every pooled activation to its class, so shapes inside one class share a buffer across rebuilds; E4 S3 then split reservation from assignment (a slots table per backend and class: a released buffer is re-assigned rather than returned to the pool, so a rebuild re-maps and CUDA's pool_gen — the capture generation — stops moving) and gave GraphCache one graph per GraphParams, so a switch is a re-map, not a build (§14 row 3, closed by E4 S3).
3. No VRAM budget or feasibility check. cudaMemGetInfo is queried once at init and only printed (src/cuda/methods/init.rs:126); the sole consumer of free-memory information today is a valve guarding the optional f16 weight cache (src/cuda/methods.rs:57-62) — the activation/KV allocator has no accounting at all. Out-of-memory surfaces as a null pointer that fails at execute time (cuda_backend.rs:1345-1348). There is no "this graph will not fit, offload the last n layers" fallback because there is no layer-offload concept at all (§2.6).

Recommendation. Split allocation into a reserve phase (size the graph for the worst-case shape the session will use: n_ctx, n_batch) and an assign phase that re-maps nodes into reserved regions without touching the device. Round pool sizes to a size-class ladder (e.g. 16/64/256 KiB steps) so distinct shapes share. Add real memory accounting (bytes_reserved, bytes_live, bytes_device_free) and make the prefill path consult it before allocating, with CPU execution as an explicit fallback decided at build time (consistent with the "no silent fallback" rule — the fallback would be visible in the backend assignment).

2.4 L4 — KV cache and sequence state 🔴

Today. Each layer owns two persistent contiguous regions, K and V, created on first use by ensure_kv(layer, backend, size) (alloc.rs:385-393), sized n_kv_embd × n_ctx f32 (or f16 when the type flag is set). They live in the allocator inside GraphCache and survive graph rebuilds (graph/cache.rs:69). Positions are data, injected per step through the positions input node, so the topology never depends on n_past (ops.rs:94-99, COMPUTE-GRAPH-DESIGN.md §1.4).

That design is a genuine strength — it is the precondition for graph reuse — but it stops at the single-sequence append-only case. What is missing:

Gap. 🔴 This is the single largest structural gap, because it blocks four separate user-visible capabilities at once: multi-slot serving throughput, cross-request prompt caching, long-conversation context handling without re-prefill, and any state-space/hybrid model family.

Two concrete defects live here as well:

• ensure_kv ignores the requested size after the first call (alloc.rs:385-393: the early if let Some(&pair) = self.kv.get(&layer) returns without comparing size). CParams.n_ctx is part of the reuse identity, so a session that changes n_ctx on the same GraphCache silently keeps the old region. The CPU backend then errors on out-of-range positions (cpu_backend.rs:170-172); the GPU backends do not check at all (below). Today no caller changes n_ctx on a live cache, so it is latent rather than live — but it is a landmine directly under the "grow the context" feature that a server wants.
• GPU KvcacheStore had no bounds validation — closed. A3 moved the check to the backend-agnostic GraphAllocator::fill_input_i32 (check_positions_bound), the single point where positions/cells become graph data, so CPU, CUDA and Metal all refuse an out-of-range row before execution; the Metal gap-table G1 ticket #38 then added the arm-level guard for fill paths that bypass fill_input_i32 (MetalBackend::check_kv_store_rows, covering KvcacheStore, FusedQKV and FusedQkvNorm). A caller violating the documented contract now gets an Err naming the cell and the arena instead of an out-of-bounds device write.

What C1/C2 landed (2026-09-16). src/graph/kvcache.rs now owns a per-layer arena with an owner per cell, and GraphAllocator::kv_rm(start, len, &rope) removes a row range and re-bases the rows after it — a physical operation, so cell == pos survives and no backend needed a new kernel. The conversation's overflow path uses it instead of dropping turns and re-prefilling them. Its one approximation is inherent and recorded in ARCHITECTURE-EXECUTION-PLAN.md §5 (C2 record): the retained rows hold the values they were written with, so rows that attended to the dropped turns keep that influence — no shift that avoids re-prefilling can avoid it (llama.cpp's context shift behaves the same way). What is exact is the mechanism, and that is what the tests pin bitwise: a tail removal leaves the retained head byte-identical (so continuing from it matches a fresh prefill exactly), a middle removal copies V verbatim and re-ropes only K, and the whole operation keeps the identity cell mapping C1's scheduler gate requires.

Recommendation. Redesign the KV layer as a sequence-addressable cell store before adding batched attention, because batching without per-sequence KV addressing cannot be correct. Concretely: a KvCache owning a per-layer arena of n_ctx cells, each carrying the set of sequence ids that own it; the store node resolves (layer, seq_id) → cell index on the host and passes an index array to the kernel; the attention kernel receives an explicit per-query allowed-cell mask instead of deriving the bound from positions. E1 landed that last part on CPU: attn_span carries each query's [lo, hi) cell range (resolved from ownership + the query's position), and a representation as a range is complete because a sequence's cells are contiguous — a per-cell mask is what a hole-creating layout would need (C3/D1). Build one such cache with an optional window parameter and an optional recurrent state, rather than a family of per-variant caches. That single abstraction simultaneously delivers prefix reuse (cells already owned by the matching prefix), eviction (drop cell ownership rather than re-prefill), and defragmentation (a cell-copy op that a strided-view IR makes expressible).

2.5 L5 — Batching and serving 🔴

Today. Single sequence by default on CPU — batching exists and its default follows the device (E6: on for CUDA, off for CPU/Metal; MINFER_BATCH=0/1 forces it), slower on CPU (0.49x) and 1.9x faster on the GPU, where the E2 acceptance is met. GraphParams carries no sequence count at all: E2 deleted n_seqs, which A7 had kept as reserved for item 3, once item 3 landed and showed the count is data rather than topology (CParams.explicit_span carries the only topology decision it can force — see ARCHITECTURE-EXECUTION-PLAN.md §8). The other dead identity field, CParams.n_batch, was deleted in Phase A7; chunked prefill (item 10) will reintroduce it with its real semantics. Attention derives its causal bound from the per-token positions input (cpu_backend.rs:411-416, cuda_backend.rs:1100-1102), so two sequences in one batch would attend to each other.

The prefill is one graph covering the entire prompt (main.rs:862-890): n_ctx = max(--n-ctx, prompt_len), one forward with nt = prompt_len. The server rejects any prompt longer than the slot context (chat.rs:118-124); new_slots (slot.rs:33-36) divides n_ctx_total equally per slot. worker_loop drains the queue serially, one slot at a time (chat.rs:452-518), and every request starts from a fresh GraphCache (chat.rs:77). Default --n-slots is 1 (main.rs:211), so the default server is strictly serial with a cold KV per request.

Gap. 🔴 This is the largest capability difference between what minfer does today and what a serving workload needs, and it costs throughput on exactly the workload the server exists for. With --n-slots N minfer gets N independent serial sessions, not N-way batching: aggregate decode throughput stays at batch-1 tokens/s × (fraction of time the GPU is busy).

Recommendation. Work in dependency order: (1) KV cells → (2) IR seq_id and mask inputs → (3) attention kernels with explicit masks → (4) batch composition in the scheduler/worker → (5) n_batch chunking. Do not start with (4); it cannot be made correct first.

Two smaller but immediate items sit in this layer:

• Persistent server context. chat.rs:491 discards the GraphCache per request. The comment justifies it as avoiding cross-request KV contamination (a real bug fixed in doc 97) — but the correct fix is sequence-aware KV invalidation (§2.4), not discarding the whole cache. As written, every request pays KV + pool re-allocation (≈235 MB for 7B at n_ctx=4096, f16) and invalidates the CUDA Graph capture warm-up (pool_gen bump, cuda_backend.rs:1342). Fixed in B2: the slot keeps its cache and a record of the tokens its rows hold; a request reuses the KV only when its prompt starts with exactly that sequence. Measured end-to-end (0.5B Q4_0, 219-token second turn): prefill 219 → 16 tokens, time-to-first-token ≈2.8 s → 0.25 s (≈11×), with the cold-slot turn unchanged.
• No panic isolation. Fixed in A4. The guard that existed (guarded_forward, chat.rs:585-597) only covered forward_graph_cached; the speculative path calls both models' forwards directly (spec.rs) and the tokenizer, sampler and stop-string paths were bare. A panic in any of them unwound worker_loop, dropping the bounded job channel (capacity 64, server/mod.rs:66) — later requests got 503 — and the queued jobs' StreamEvent senders, ending their SSE (Server-Sent Events) stream after an empty [DONE] rather than an error (server/mod.rs:198-216). The whole per-job body now runs under run_job_isolated, which turns a panic into a logged 500 for that request and keeps the worker draining the queue.

2.6 L6 — Backend abstraction and device reach

Today. Backend is a Copy/Hash/Eq/Ord handle over a fixed id space (graph/registry.rs, F4) and a trait (graph/backend.rs:21-98). The ids are a KV-session file-format contract, and the name-keyed registry carries each backend's priority, capability matrix and pool hooks; consumers read it instead of matching. Before F4, adding a backend meant teaching nine #[cfg]-laden match sites about it: GraphAllocator::supports (alloc.rs:384-386), alloc_in_pool/alloc_fresh_in/free_in_pool (:819, :955, :959), sync_backend (alloc.rs:2400), copy_across (alloc.rs:2449), and the scheduler's execute match (scheduler.rs:274-292); the functions are registry-driven now.

GPU participation is decided by an all-or-nothing model-level gate: every weight must be registered on the GPU or the model runs on CPU (ARCHITECTURE.md §5.2; CUDA-BACKEND-DESIGN.md:251-255). The engine enumerates devices and honours --gpu N but uses exactly one (src/cuda/methods/init.rs:63-81); there is no tensor_split, no peer copies, and no remote-device backend.

Gap. 🟠 Two axes.

Pluggability — F4 (2026-09-24) landed the registry half: the backend set is data (a name-keyed table of entries registered at startup) rather than nine match sites, and a name that is unknown, not compiled into this build, or not usable on this machine is a loud startup refusal (BACKEND-REGISTRY-DESIGN.md). What is still closed at compile time is the set itself: there is no dlopen/plugin path, so a Vulkan backend (the only route to Windows/Linux AMD and Intel GPUs) or a remote/RPC backend is added by registering an entry — but it must still be compiled in. That is what remains of this axis.

Placement policy — a model that does not fit in VRAM could not run at all. The docs already record this as a real limitation on 8 GB devices (DEVICE-ADAPTATION-PLAN.md §9.2: "7B Q8_0 will not fit (7.2 GB weights alone)"). E5 (2026-09-23) added the missing granularity and the fit: CNode.layer (stamped by the model builders), an OffloadPlan (graph/offload.rs) that the loader's registration filter, the builder's device-only fused gates and the assignment pass all read, --gpu-layers N / MINFER_GPU_LAYERS=N, the startup report, and a verified mixed CPU+CUDA run on the 0.5B — then S2's auto: per-block weight bytes from the GGUF index, the pure fit_blocks prefix search against a weight budget (MINFER_GPU_MEM, else three quarters of the device's free bytes) with a quarter held back for the KV arenas and the activation pool. The registry/BackendId half is done (F4), and the free-bytes query for Metal landed as #53's DeviceMemory answer (recommendedMaxWorkingSetSize), so auto and headroom_bytes() work on macOS too.

Recommendation. (a) Introduce a BackendRegistry with register(Box<dyn Backend>), supports(op, dtype) -> Option<BackendId> and per-backend allocation/sync as trait methods; replace the enum with an opaque BackendId. Done in F4 (2026-09-24) — the shape that landed is a name-keyed Registry of BackendEntry values (priority + caps + the pool hooks, registered at startup) and a Backend handle over a fixed id space, with the assignment order made an explicit pinned number (docs/BACKEND-REGISTRY-DESIGN.md). It keeps the capability matrices in the trait's own modules (the trait methods forward to them), so the registry and the trait cannot disagree. (b) Add a layer-offload budget policy that consumes the memory accounting from §2.3.

2.7 L7 — Models, quantization, tokenizer, sampling

Architectures. Two (qwen2, qwen3 dense), dispatched on general.architecture (models/mod.rs:105-121). Every other family is absent: MoE, MLA, sliding-window/hybrid, state-space (Mamba/RWKV), multimodal, and every non-Qwen dense family (Llama, Mistral, Phi, GLM, InternLM). The per-family decision, its prerequisites and its cost live in MODEL-SUPPORT-ROADMAP.md; the caveat that matters here is §2.1 — its Tier 1 estimate holds only while a family needs no new IR structure. 🟠

Quantization coverage. Eight quant types; no Q2_K/Q3_K/Q8_K, no I-quants, no MXFP4/NVFP4 (SUPPORT-MATRIX.md §Not Yet Supported; CUDA_OPTIMIZATION.md §1.4 marks IQ/Q2/Q3 "not planned"). BF16 landed as a weight type: the writer and the CPU path in #142, the device halves (CUDA and Metal) in #208, 2026-10-06. The notable one still for reach is Q2_K/Q3_K (running large models on small machines). 🟠 but legitimately deprioritized.

Quantizer tooling — closed by F6 (#49, 2026-09-24), and its f16-weight gap closed by #141 (2026-09-25). A GGUF v3 writer (src/gguf_write.rs), byte-verified weight encoders (src/quantize.rs), a HuggingFace Qwen2 converter that satisfies the strict loader (src/convert.rs), and the convert / quantize / split subcommands (src/tooling.rs, docs/GGUF-TOOLING.md). A converted f16 model runs on the CPU and on CUDA — the CPU graph path gained f16 weight dispatch in F6 and #141 vectorized that dot (AVX2 F16C / NEON FCVTL) and moved the multi-token row loop onto the shared CPU pool (3.2 → 207 tok/s prefill on the 0.5B), while CUDA registers the raw 2 B/element weights and converts in-register. Both supported architectures now: #141's f16 registration and graph type gate landed in qwen2's loader/graph only, so #167 completed the qwen3 pair and gave the two loaders one shared registration rule (models::weight_reg, which also carries the q4_K W_dsc plane gate). Metal refuses f16 weights until #164, and a rewrite/split round-trip is bit-exact, and the HF conversion's f16 output is byte-identical to llama.cpp's converter on the same checkpoint, per tensor. What remains is scope F6 did not claim: the K-quant/I-quant encoders are still refused by name (#140) and bf16 output is refused (#142). 🟢 for conversion/quantization of the supported set.

CPU SIMD. AVX2 (Advanced Vector Extensions 2) dot kernels exist for Q4_0/Q8_0 (src/quants/dot_q4_0.rs:1-9, src/quants/dot_q8_0.rs:28), and since #56 (2026-10-08) the K-quants too: Q4_K/Q5_K/Q6_K × Q8_K have AVX2+FMA kernels (src/quants/avx2.rs) and AVX-512/VNNI variants (src/quants/avx512.rs), dispatched AVX-512 → AVX2 → scalar at runtime (MINFER_NO_AVX512=1 drops to AVX2, MINFER_NO_AVX2=1 to scalar — the x86 counterparts of MINFER_NO_NEON) and gated bitwise against the scalar reference by quants::avx2_correctness; Q4_1/Q5_0/Q5_1 remain scalar on x86 and NEON-only on aarch64. The support matrix is updated (SUPPORT-MATRIX.md, AVX2 column) and CPU_OPTIMIZATIONS.md §P1 is closed. What remains is weight repacking — the standard fix (repack the quant blocks at load time into SIMD-friendly interleaved layouts) that would expose the kernel gain end-to-end, since the current matmul re-reads each weight row per token; there is also still no aarch64 i8mm path and no AMX. 🟡 for x86 (the dots landed; the end-to-end win awaits repacking).

Tokenizer. Byte-level BPE only, loaded from GGUF metadata (tokenizer.rs). F7 (#50, 2026-09-24) made tokenizer.ggml.pre authoritative with two hand-written rule sets (qwen2, qwen35), replaced the silent unwrap_or(0) with a checked byte fallback, and made every other pre-tokenizer value (including a missing one), a non-gpt2 model, an empty merge table and an incomplete byte vocabulary a loud load refusal. What remains: no SentencePiece/unigram or WordPiece path, and no ignore_merges/multi-regex pre-tokenizers (llama3, default, deepseek-*, …) — those refuse by name, so the coverage gap is explicit rather than a wrong split. Also no NFC normalization (llama.cpp does not apply one for BPE either) and special-token matching is still a per-model hand-extension (the DeepSeek-R1 fix noted in AGENTS.md), which does not scale. 🟡 for the Qwen/Llama-3-shaped families this engine supports; 🟠 for anything SentencePiece-based. Follow-up: #132.

Chat templates. Landed in F7 (#50, 2026-09-24). minijinja 2.21.0's set_unknown_method_callback is the extension point, so Qwen3's chat_template renders (think-block extraction, tool-call formatting, enable_thinking) with no dependency change; the reference renderings are committed and byte-for-byte gated, and any template the engine cannot render is a loud refusal naming the construct and line — validated at load, so the CLI exits and the server refuses to start. What remains: a --chat-template <FILE> override (a user-supplied template for a GGUF that has none or a wrong one), strftime_now, and the Python str methods outside the implemented set (each refused loudly). Design + accepted/refused sets: docs/CHAT-TEMPLATE-AND-TOKENIZER-DESIGN.md. 🟢 for the supported models.

Sampling. One SamplerConfig pipeline in sampler.rs (#48, landed 2026-09-24): logit bias → penalties → DRY → greedy shortcut → top-k → typical → top-p → min-p → XTC → temperature or mirostat v1/v2, with every new knob defaulting to a no-op so the pre-F3 chain is bit-identical. Landed from the old gap list: min-p, typical, XTC, DRY, mirostat v1/v2, logit bias (and frequency/presence penalties from the OpenAI plan). Grammar / JSON-schema constrained decoding landed in F2 (2026-09-24): src/grammar.rs compiles a GBNF grammar (or a JSON Schema, via a generated GBNF) into a pushdown automaton and masks the logits inside that one pipeline, between DRY and the greedy shortcut; CLI --grammar/--json-schema, server response_format (json_object / json_schema) plus a grammar field. Design and the exact accepted subset: docs/GRAMMAR-DESIGN.md. What it deliberately does not do (and refuses loudly rather than guessing): pattern/minLength/maxLength, multipleOf, real-valued numeric bounds, object property permutations, and a device-side mask — the mask is host work before any backend runs. Still missing: top-n-sigma, adaptive-p, infill. 🟡

No LoRA (Low-Rank Adaptation) / adapter support. There is no adapter path at all. minfer's weights_version field in GraphParams (params.rs:88-98) exists precisely to break reuse on a weight swap, but nothing produces such a swap. 🟡

2.8 L8 — Ops coverage, safety, testing, observability

Op coverage. The Op vocabulary is deliberately small; six variants are parity-only stubs that no architecture emits (Scale, Softmax, View, Reshape, Permute, AttnMode::Mha — COMPUTE-GRAPH-DESIGN.md §1.3), and View/Reshape/Permute execute as copies. CUDA additionally refuses transpose_b matmul (cuda_backend.rs:1454-1459) and FusedQkvNorm (cuda_backend.rs:1905), and Metal refuses QkvBiasRopeStore (metal_backend.rs:700) — so the two GPU backends do not implement the same op set, and a model's decode path differs by platform. Neither is documented in SUPPORT-MATRIX.md. Fixed in A8: SUPPORT-MATRIX.md now carries an "Operator Coverage by Backend" table with the four asymmetric rows and their consequences.

Guard asymmetry. Both halves are closed on a Mac (2026-10-06). The decode-fusion debug_assert!s (#39, gap-table G2): FusedFFN / FusedQKV / FusedQkvNorm each return Err naming the node and the observed nt before the weight lookup, so a release build no longer passes an invalid shape through. The weightless-RMSNorm substitution (#40, gap-table G3): Op::RmsNorm / Op::QkNorm now call MetalBackend::norm_weight, which returns Err naming the node and the missing tensor for both None meanings (weight_name absent, or unregistered on the device) instead of running the weightless kernel where CUDA errors (cuda_backend.rs norm_weight). Neither violates the project's rule that "kernel-invariant violations return Err, never a silent fallback" (AGENTS.md §GPU Safety) any more.

Testing. Five integration files, four of them #![cfg(target_os = "macos")] Metal kernel isolation tests; CPU/CUDA correctness rests on inline unit tests plus real-model tests that skip when a GGUF is not cached (tests/conversation_cli.rs:1-12). CI is cargo build --release on macOS only — no test run, no CUDA build, no Linux build. Fixed in A2: CI now runs cargo test --release on Linux/CPU, compiles the CUDA backend in NVIDIA's devel image, and keeps the macOS build. 🟠 The remaining highest-value addition is a systematic op × dtype × backend correctness matrix (every op checked on every backend against a CPU reference); it would have caught several items in §4 automatically — that is ticket A1.

Observability. F8 landed 2026-09-24 (#51): GET /metrics (Prometheus text) exports live KV/arena occupancy, queue depth and the running/in-flight counts; per-op timing is available under MINFER_OP_TIMING (off by default, measured at the scheduler's per-node dispatch); and SIGINT/SIGTERM drains gracefully, bounded by MINFER_DRAIN_MS. Still missing: structured logging and levels (the server prints free-form startup lines), and worker-restart supervision (a panicking job is isolated per request by run_job_isolated, but a dead worker thread is not restarted). The trace/viz stack remains a development instrument, not an operations one. 🟡 for a research engine, 🟠 the moment it is deployed.

3. Recommendation backlog

Effort: S ≤ 2 d · M ≤ 1 w · L ≤ 2 w · XL > 2 w. Backlog items tracked on GitHub link their issue (the open list); an item whose work predates the tracker has no issue, and the plan is its record.

P0 — structural, unblocks a class of use cases

P1 — material capability or performance

P2 — coverage

P3 — hygiene and operations

4. Concrete defects found (verified in code)

Ordered by severity. Items 1–6 are behavioural; 7–12 are hygiene; 13–14 are behavioural defects found while executing the plan, already fixed.

1. GPU KV store bounds check — closed. A3 moved the check to the backend-agnostic GraphAllocator::fill_input_i32 (check_positions_bound), so CPU, CUDA and Metal all refuse an out-of-range row before execution, and #38 added the arm-level Metal guard (MetalBackend::check_kv_store_rows) for the fill paths that bypass it — §2.4 states the same closure in the gap list.
2. ensure_kv ignores a changed size (alloc.rs:995-1002). The KV region is frozen at first allocation while CParams.n_ctx remains part of the reuse identity, so a size change is neither honoured nor detected.
3. Metal weakens kernel guards. Both halves are closed: the decode-fusion debug_assert!s (#39) return Err naming the node and the observed nt, and the silent weightless RMSNorm when a weight is missing (#40, MetalBackend::norm_weight) returns Err naming the node and the missing tensor instead. Neither contradicts docs/GPU_SAFETY.md and AGENTS.md's no-silent-fallback rule any more.
4. Server worker had no panic isolation outside the forward call (src/server/chat.rs::guarded_forward_batch): a panic anywhere else unwound the worker, permanently degrading the server (503 for new jobs, empty 200/SSE for queued ones) with no log. Fixed in A4 — the whole per-job body is now guarded.
5. Backend op-set asymmetry drives silent path changes. FusedQkvNorm is Metal-only (MetalBackend::supports_op) but absent from CUDA's supports_op, so Qwen3 decode takes the fused path on Metal and the unfused path on CUDA. QkvBiasRopeStore is the mirror case (MetalBackend::supports_op returns false for it). Neither is documented in SUPPORT-MATRIX.md. Documented in A8: SUPPORT-MATRIX.md now has an "Operator Coverage by Backend" table; the asymmetry is visible rather than silent.
6. CUDA RoPE is non-interleaved only (cuda_backend.rs:1522), so any model needing the interleaved style splits every layer between CUDA and CPU, producing two host round trips per layer (§2.2); docs/ARCHITECTURE.md:455-459 now states that limitation rather than advertising both styles.
7. Stale unreachable!("CUDA pool not implemented") in the non-CUDA arms — the string and any unreachable! are gone from alloc.rs. Fixed in #57 (cdf41b2).
8. Dead fields in the reuse identity: CParams.n_batch and GraphParams.n_seqs are compared by params_match (cache.rs:95-101) but no builder reads them; every construction site hard-codes 1 / n_tokens. Fixed in A7, closed in E2: n_batch is deleted; n_seqs was kept for item 3 and then deleted by it (the sequence count is data — the topology decision lives in CParams.explicit_span), with sequence_count_is_data_not_topology pinning that a sequence-count change no longer rebuilds an otherwise identical graph.
9. Single-entry cross-backend staging (GraphAllocator::cross_staging), mitigated by the consumer-side filter at scheduler::cross_input_ready. Fixed in A5 — keyed by (graph uid, node, dst_backend); two foreign consumers can now be served.
10. read_host returns None on CUDA (cuda_backend.rs:2268), so the trait's host-read contract is backend-dependent; the allocator compensates with copy_to_host (alloc.rs:2137-2143).
11. CUDA pool never releases device memory (cuda_backend.rs:2148-2156), documented as accepted debt but reasoned about for fixed-shape CLI runs; a varying-n_tokens workload accumulates one buffer set per distinct shape.
12. Tests: no Linux/CUDA/CPU in CI, four of five integration files macOS-only (.github/workflows/ci.yml, tests/*.rs). Fixed in A2 for the CI half (Linux/CPU tests, CUDA compile, plus --no-run test compilation); the four macOS-only kernel-isolation files remain macOS-only by nature.
13. Op::Softmax returned unnormalised values on the CPU backend (cpu_backend.rs): it called vec_soft_max_f32, which writes exp(x - max) and returns the sum, and discarded the return. No architecture emits a standalone Softmax node, so nothing exercised it. Found by the A1 op matrix and fixed: the arm now scales by 1/sum, with an op-matrix case pinning it.
14. The CPU attention was not nt-invariant. Each token's scores were padded to the batch-wide nkv, and the softmax, the normalisation and the weighted sum all ran over nkv instead of the token's own causal window vl = pos+1. The padded entries are -inf → 0, so the arithmetic reads as equivalent — but the reduction length, and with it the rounding, depended on how many tokens shared the batch. Measured on 0.5B: a one-token decode step differed from a single-shot prefill by max|Δlogits| = 0.41, and the same token's K rows differed from layer 3 on (nt=6 vs nt=13). Found while testing prefix reuse for B2 and fixed: restricting the window to vl makes incremental prefill, the decode loop and a single-shot prefill bitwise identical, and removes the padding pass. It also matters beyond B2 — the speculative-decoding identity gates assume exactly this property.

5. Suggested execution order

The dependency structure is more informative than the priority table alone:

                       ┌──────────────────────────────────────────┐
                       │ 1. KV cell store (seq-addressable)       │
                       └───────────────┬──────────────────────────┘
                                       │
        ┌──────────────────────────────┼───────────────────────────────┐
        ▼                              ▼                               ▼
 2. IR seq_id + mask         4. persistent server context    21. quantized KV
        │                              │
        ▼                              │
 3. continuous batching  ◄──────────────┘   (4 alone already removes
        │                                    per-request realloc/re-warm)
        │
        ├──► 10. chunked prefill
        └──► 9. layer offload ◄── 8. allocator reserve + accounting

  7. IR views/multi-output ──► 17. MoE, 11. CPU SIMD (independent)
  12. backend registry      ──► 14. async copy ──► multi-device execution

Items 5, 6, 13, 26–28 are independent, small, and can land at any time — they are the ones worth doing first simply because they are cheap and they remove hazards that the larger work would otherwise have to work around.

Appendix — evidence index