CPU Inference Path Analysis & Optimization Results

This document analyzes minfer's CPU inference performance, compares with llama.cpp, and documents optimization attempts and their outcomes.

Status and scope (updated 2026-10-08). This file is a layered record. The sections from the top down to §Conclusion are a 2026-07-01 snapshot taken before the compute-graph rewrite: their src/models/qwen2/forward.rs:NNN line references point at a file that was deleted in Phase 6, and their "minfer is single-threaded" premise no longer holds. The current numbers are in §"NEON + Threading Overhaul" (2026-08) below and in docs/PERF-QWEN3-4B-VS-LLAMACPP.md §3.

Four of the four "Remaining Optimization Opportunities" from that snapshot have since shipped: P4 multi-threading (persistent CPU thread pool, src/kernel.rs), P2 f16 KV cache (MINFER_CACHE_TYPE, auto-selected for 7B-class models), flash attention (on both GPU backends; the CPU path keeps the multi-pass form), and P1 — the AVX2/AVX-512 dot products for the K-quant types (#56, landed 2026-10-08; see §P1 below). The remaining piece of that lane is weight repacking, tracked in docs/ARCHITECTURE-ROADMAP.md §2.7 and docs/SUPPORT-MATRIX.md.

Current State

Baseline performance: 27.1 tok/s decode on Qwen2-0.5B (AVX2, i7-1260P). llama.cpp comparison: ~60-80 tok/s on the same model. Performance gap: ~2.5-3×.

Key finding: The codebase is already well-optimized. All major SIMD optimizations identified in initial analysis were already present in the codebase. Optimization attempts caused performance regressions due to function call overhead and interference with compiler auto-vectorization.

Existing Optimizations (Already Present)

The forward pass already uses SIMD-optimized operations throughout:

1. RMSNorm with Fused Weight Multiply

Location: src/models/qwen2/forward.rs:242-250

#![allow(unused)]
fn main() {
fn rms_norm(x: &[f32], eps: f32, out: &mut [f32], n: usize, d: usize, w: Option<&[f32]>) {
    for t in 0..n {
        let row = &x[t * d..(t + 1) * d];
        let dst = &mut out[t * d..(t + 1) * d];
        match w {
            Some(w) => crate::vec_ops::rms_norm_fused_f32(d, dst, row, w, eps),
            None => crate::vec_ops::rms_norm_f32(d, dst, row, eps),
        }
    }
}
}

Uses rms_norm_fused_f32 from vec_ops.rs:511-533 with AVX2 sum-of-squares computation using _mm256_fmadd_ps. Fuses RMSNorm + weight multiply in a single pass.

2. Vectorized Residual Add

Location: src/models/qwen2/forward.rs:110-115

#![allow(unused)]
fn main() {
unsafe {
    crate::vec_ops::vec_add_f32(hidden.len(),
        std::slice::from_raw_parts_mut(hidden.as_mut_ptr(), hidden.len()),
        std::slice::from_raw_parts(hidden.as_ptr(), hidden.len()),
        &bn);
}
}

Uses vec_add_f32 with AVX2 _mm256_add_ps (8-wide). Called twice per layer (48 times total for 24-layer model).

3. Vectorized Gate × Up Multiply

Location: src/models/qwen2/forward.rs:127-130

#![allow(unused)]
fn main() {
crate::vec_ops::vec_mul_f32(len,
    std::slice::from_raw_parts_mut(bg.as_mut_ptr(), len),
    std::slice::from_raw_parts(bg.as_ptr(), len),
    &bf);
}

Uses vec_mul_f32 with AVX2 _mm256_mul_ps. Called once per layer (24 times).

4. RoPE Sin/Cos Cache

Location: src/models/qwen2/forward.rs:257-275

#![allow(unused)]
fn main() {
let mut sin_cache = vec![0.0f32; half];
let mut cos_cache = vec![0.0f32; half];
for t in 0..pos.len() {
    let p = pos[t] as f32;
    for i in 0..half {
        let th = p * freqs[i];
        let (sn, cs) = th.sin_cos();
        sin_cache[i] = sn;
        cos_cache[i] = cs;
    }
    // Use cached values for all heads
    for h in 0..nh { ... }
}
}

Precomputes sin/cos once per position (64 values), shared across all 32 heads. Reduces sin_cos() calls from 2048 to 64 per token per layer (32× reduction).

5. SiLU Activation

Location: src/models/qwen2/forward.rs:120

#![allow(unused)]
fn main() {
crate::vec_ops::vec_silu_f32(len, &mut bg);
}

Uses vec_silu_f32 with AVX2 implementation.

Optimization Attempts & Results

Attempt 1: Activation Quantization Reuse

Goal: Quantize activation once, reuse Q8_0 buffer for Q/K/V matmuls.

Implementation: Added quant_matmul_f32_batch_prequantized and quantize_activation_f32 to kernel.rs. Pre-allocated Q8_0 buffer outside layer loop.

Result: 28.0 → 26.0 tok/s (-7% regression)

Why it failed:

• Added function call overhead for small batch sizes (nt=1 during decode)
• Extra memory traffic from separate quantization step
• Compiler couldn't optimize the separated code as well
• The original code's per-matmul quantization allows better inlining and register allocation

Attempt 2: Online Softmax

Goal: Merge 4-pass attention into single-pass online softmax.

Implementation: Replaced multi-pass softmax with running max/sum algorithm (ref: arxiv 2112.05682).

Result: 27.1 → 27.4 tok/s (+1% marginal gain, then regressed to 25.1 tok/s after further changes)

Why it failed:

• For small sequence lengths (typical in decode), the overhead of maintaining running max/sum outweighs the benefit
• Compiler auto-vectorization of the simple multi-pass version is very effective
• The attention loop is already memory-bound, not compute-bound

Attempt 3: RoPE Cache Stack Allocation

Goal: Avoid heap allocation in RoPE by using stack arrays.

Implementation: Changed vec![0.0f32; half] to [0.0f32; 64].

Result: 27.1 → 26.7 tok/s (-1.5% regression)

Why it failed:

• Stack allocation of 64 f32s (256 bytes) is not significantly faster than heap allocation for this size
• May have interfered with compiler's register allocation strategy
• The heap allocation happens once per layer (24 times), not per token

Attempt 4: Attention Head Parallelization (Multi-Threading)

Goal: Parallelize attention heads across CPU cores using std::thread.

Implementation: Added gqa_attn_parallel function that splits attention heads across multiple threads. Each thread processes a subset of heads independently, with its own scores buffer to avoid contention. Used std::mem::transmute to bypass Rust's Send trait check on closures capturing raw pointers (safe because each thread accesses non-overlapping output regions).

Two variants tested:

1. Always parallel: Parallelize for both decode (nt=1) and prefill (nt>1)
2. Prefill-only: Only parallelize when nt>1, keep original single-threaded path for decode

Results:

Why it failed:

• Thread creation overhead: Each gqa_attn call spawns 16 threads (one per head). With 24 layers, that's 384 thread create/join cycles per forward pass. Thread creation on Linux costs ~10-20μs each.
• Small workload per thread: For decode (nt=1), each head processes only hd=64 elements per KV position. The computation per thread is too small to amortize thread startup cost.
• Memory allocation per thread: Each thread allocates its own vec![0.0f32; nkv] scores buffer, adding heap allocation overhead.
• Code bloat: Adding the parallel path increases binary size, reducing instruction cache hit rate for the hot decode path.
• llama.cpp uses a thread pool: Unlike our per-call thread creation, llama.cpp uses a persistent thread pool (ggml_threadpool) where threads are created once and reused across all operations. This eliminates the creation overhead.

What would be needed to make this work:

1. Implement a persistent thread pool (create threads once at startup)
2. Use work-stealing or task queue pattern instead of per-call spawn
3. Consider using rayon crate which provides an efficient work-stealing thread pool with minimal overhead
4. Only beneficial for larger models (7B+) or longer sequences where the attention computation per head is substantial

Attempt 5: Crossbeam Work-Stealing & usize Pointer Trick

Goal: Research crossbeam's work-stealing deque and find a way to bypass Rust's Send/Sync checks on raw pointers in multi-threaded closures.

Key Discovery: usize Pointer Trick

Rust's type system prevents passing raw pointers (*const f32, *mut f32) across thread boundaries because they don't implement Send. Even wrapping them in structs with unsafe impl Send fails because the compiler recursively checks struct fields.

Solution: Convert raw pointers to usize before the closure, then reconstruct inside:

#![allow(unused)]
fn main() {
// Before closure (main thread)
let ptr = slice.as_ptr() as usize;  // usize is Send

std::thread::scope(|s| {
    s.spawn(move || {
        // Inside closure (worker thread)
        let slice = unsafe { std::slice::from_raw_parts(ptr as *const f32, len) };
        // Use slice...
    });
});
}

This works because:

• usize is just an integer type that implements Send
• The compiler doesn't see raw pointers in the closure capture
• The pointer values remain valid because std::thread::scope ensures all threads join before the scope exits (pointees stay alive)

Crossbeam Research Findings:

Tested both std::thread::scope and crossbeam::scope:

• Both require Send on closures passed to spawn()
• Both allow the outer closure to be non-Send
• crossbeam::deque provides work-stealing deques but doesn't solve the fundamental pointer issue
• The usize trick works with both std and crossbeam

Implementation:

Applied the usize pointer trick to gqa_attn with conditional parallelization:

• Only parallelize when nt > 8 (large prefill)
• Keep original inline single-threaded code for decode (nt=1)
• Each thread gets its own score buffer to avoid contention

Results:

Why it still failed:

• Thread creation overhead dominates: Even with conditional parallelization, creating 4-16 threads per attention call (24 layers) adds 240-960μs overhead per forward pass
• Small workload per head: For 0.5B model with hd=64, each head processes only 64 elements per KV position. Even with 35-token prefill, the work per thread is ~2240 operations, completing in ~10-20μs—not enough to amortize 10-50μs thread creation cost
• Compiler optimization interference: The presence of parallel code paths may prevent the compiler from fully optimizing the single-threaded path (instruction cache pressure, register allocation changes)
• Memory allocation overhead: Each thread allocates vec![0.0f32; nkv] score buffer

Crossbeam Work-Stealing Deque Analysis:

crossbeam::deque::Worker<T> + Stealer<T> pattern:

• Main thread creates Worker, pushes task indices
• Worker threads get Stealer handles, steal tasks when idle
• Good for load balancing when task sizes vary

However, for attention head parallelization:

• All heads have identical work (same hd, same nkv)
• No load imbalance to address
• Work-stealing overhead > benefit for uniform tasks
• Still requires persistent thread pool to avoid creation overhead

Conclusion:

The usize pointer trick successfully bypasses Rust's Send/Sync checks, enabling raw pointer sharing across threads. However, for small models (0.5B) and typical sequence lengths, the thread creation overhead dominates. Multi-threading would only benefit:

• Larger models (7B+) where attention computation per head is substantial
• Very long sequences (1000+ tokens) where work per head amortizes thread costs
• With a persistent thread pool (like llama.cpp's ggml_threadpool) to eliminate creation overhead

For this 0.5B model, the compiler's auto-vectorization and single-threaded execution remain more efficient than multi-threading.

Lessons Learned

1. Compiler Auto-Vectorization is Highly Effective

For small models (0.5B) and short sequences (decode phase with nt=1), the Rust compiler's auto-vectorization is extremely effective. Simple loops with clear patterns are automatically vectorized to AVX2.

Manual SIMD function calls add overhead:

• Function call overhead (even with #[inline])
• Memory traffic from explicit loads/stores
• Interference with compiler's optimization passes

2. Function Call Overhead Matters

For operations on small arrays (e.g., 128 elements in attention), the function call overhead can exceed the computation time. The existing code already uses SIMD functions where the array size justifies the call overhead.

3. Memory Bandwidth is the Bottleneck

During decode (nt=1), the inference is memory-bound, not compute-bound. The dominant cost is loading weights from memory, not computation. Optimizations that add memory traffic (separate quantization, intermediate buffers) hurt performance.

4. Don't Fix What Isn't Broken

The initial analysis incorrectly identified "scalar hotspots" as bottlenecks. In reality, the code already had SIMD optimizations. The perceived gap between minfer and llama.cpp is due to:

• llama.cpp's more aggressive optimizations (flash attention, tiled GEMM)
• Better memory layout (transposed K cache)
• Multi-threading (minfer is single-threaded)
• Model size differences (llama.cpp benchmarks often use larger models where amortization is better)

Remaining Optimization Opportunities

Superseded in part (2026-10-08). P2, P3 and P4 below have shipped since this section was written, and P1 — the K-quant AVX2/AVX-512 dots — landed 2026-10-08 (#56); only its weight-repacking half remains. See the status banner at the top of this document.

P1: K-quant AVX2 / AVX-512 Dot Products — DONE 2026-10-08 (#56)

Impact: Huge for Q4_K/Q5_K/Q6_K models (all matmul operations). Difficulty: High (requires SIMD bit manipulation).

Current state (closed): Q4_K/Q5_K/Q6_K × Q8_K have AVX2+FMA kernels in src/quants/avx2.rs and AVX-512/VNNI variants in src/quants/avx512.rs, dispatched AVX-512 → AVX2 → scalar at runtime (MINFER_NO_AVX512=1 drops to AVX2, MINFER_NO_AVX2=1 to scalar — the x86 counterparts of MINFER_NO_NEON). quants::avx2_correctness gates each kernel bitwise against its *_scalar reference, and its #[ignore]d kquant_simd_dot_speedup harness records the dot ratios on a 4096-element row: AVX2 3.39× / 2.80× / 1.79×, AVX-512 3.97× / 3.23× / 2.51× scalar for Q4_K / Q5_K / Q6_K (Intel i7-11700B, 2026-10-08). minfer bench on the cached Qwen2.5-0.5B Q4_K_M (16 threads, median of nine -r 3 samples, --n-ctx 656) moves prefill pp512 28.59 → 32.13 tok/s and decode tg128 11.20 → 12.54 tok/s; the modest end-to-end delta is the weight-row re-read per token, which the remaining weight-repacking increment addresses.

llama.cpp approach: ggml/src/ggml-cpu/arch/x86/quants.c uses the denibble()/maddubs AVX2 shape and the VNNI dpbusd AVX-512 shape; minfer's kernels follow the same structure and keep the scalar float order so the result is bitwise, not merely close.

P2: f16 KV Cache

Impact: 2× memory bandwidth reduction during attention. Difficulty: Medium.

Current state: the KV store is the graph allocator's persistent regions, whose width is the per-engine KvFormat (graph/kvformat.rs; f16 on the GPU backends, packed q8_0 on CPU + CUDA + Metal (#310), f32 default — MINFER_CACHE_TYPE). The pre-graph src/cache.rs Vec<f32> this line used to name was deleted in #252.

llama.cpp approach: Default is GGML_TYPE_F16, reducing memory by half.

Recommendation: Add Vec<half::f16> option. The half crate is already a dependency. Attention dot product would need f16→f32 widening.

P3: Flash Attention

Impact: 10-15% for long sequences, grows with context length. Difficulty: High.

Current state: Multi-pass softmax with strided KV access.

llama.cpp approach: Tiled flash attention with online softmax (ggml/src/ggml-cpu/ops.cpp:8347-8871).

Recommendation: Low priority for decode (short sequences). Only beneficial for prefill or long context. The current implementation is already efficient for typical use cases.

P4: Multi-Threading

Impact: Linear speedup with core count (expected +50-70% on 8-core CPUs). Difficulty: Medium.

Current state: no parallel libraries (rayon, crossbeam) in Cargo.toml, so a forward is one thread. The &mut KVCache this line used to blame was a dead parameter, not a constraint: it was never read and was deleted in #252, and the KV store the graph really uses lives inside the allocator (so top-level parallelization is bounded by the allocator's ownership, not by a caller-held cache handle).

llama.cpp approach: OpenMP parallelism across layers and attention heads.

Parallelization Opportunities

1. Attention Heads Parallel (Highest Priority)

Location: forward.rs:287 in gqa_attn function

#![allow(unused)]
fn main() {
for h in 0..nh {  // ← Can be parallelized!
    let hk = h / gqa;
    for t in 0..nt {
        // Each head writes to different output slice: out[os..os + hd]
        // Reads same Q, K, V caches (read-only)
    }
}
}

Why it can be parallel:

• Each head writes to independent memory region (out[h*hd .. (h+1)*hd])
• All heads read same Q/K/V caches (read-only access)
• Qwen2-0.5B has 16 heads, can use 16 threads

Expected gain: 4-6× speedup on attention portion (30-40% overall)

2. Q/K/V Matmul Batch Parallel

Location: forward.rs:95-99

#![allow(unused)]
fn main() {
crate::kernel::quant_matmul_f32_batch(&mut [
    (l.wq.as_ref().unwrap(), &mut bq, nqt),  // Independent output
    (l.wk.as_ref().unwrap(), &mut bk, nkt),  // Independent output
    (l.wv.as_ref().unwrap(), &mut bv, nkt),  // Independent output
], &bn, ne, nt);
}

Why it can be parallel:

• Three matmuls write to different buffers (bq, bk, bv)
• All read same input bn (read-only)
• Currently sequential, can run 3 threads in parallel

Expected gain: 2-3× speedup on matmul portion (10-15% overall)

3. FFN Gate/Up Parallel

Location: forward.rs:118-121

#![allow(unused)]
fn main() {
crate::kernel::quant_matmul_f32_batch(&mut [
    (l.ffn_gate.as_ref().unwrap(), &mut bg, nf),  // Independent
    (l.ffn_up.as_ref().unwrap(),   &mut bf, nf),  // Independent
], &ffn_in, ne, nt);
}

Why it can be parallel: Same as Q/K/V, two independent matmuls

Expected gain: 2× speedup on FFN matmuls (5-10% overall)

What Cannot Be Parallelized

1. Layer Loop (Sequential Dependency)

#![allow(unused)]
fn main() {
for il in 0..model.n_layer() {  // ← Must be sequential
    // layer[il] needs hidden output from layer[il-1]
    // Cannot parallelize different layers
}
}

2. KV Cache Writes

#![allow(unused)]
fn main() {
kv_cache.layers[il].store_multi(positions, &bk, &bv);
}

If multiple threads write to KV cache simultaneously, need locks or atomic operations, which would negate parallelization benefits.

Implementation Recommendations

Option 1: Rayon (Simplest)

# Cargo.toml
rayon = "1.10"

#![allow(unused)]
fn main() {
// forward.rs - modify gqa_attn
use rayon::prelude::*;

fn gqa_attn(...) {
    (0..nh).into_par_iter().for_each(|h| {
        // Each head's computation
        for t in 0..nt { ... }
    });
}
}

Pros: Minimal changes, automatic thread pool management Cons: Need to ensure output buffers don't conflict

Option 2: Crossbeam (More Control)

crossbeam = "0.8"

#![allow(unused)]
fn main() {
// Manual thread control
crossbeam::scope(|s| {
    for h in 0..nh {
        s.spawn(|_| {
            // Each head's computation
        });
    }
});
}

Pros: Fine-grained control Cons: Manual thread lifecycle management

Option 3: Pre-allocated Thread Pool (Highest Performance)

#![allow(unused)]
fn main() {
use std::sync::Arc;
use std::thread;

struct ThreadPool {
    workers: Vec<thread::JoinHandle<()>>,
    // ...
}

// Initialize once in main.rs, reuse across entire inference
}

Pros: Avoids thread creation overhead Cons: Complex implementation

Expected Performance Gains

Based on current 27.1 tok/s baseline:

This would close the gap with llama.cpp from 2.5-3× to 1.5-2×.

Recommendation: Start with Rayon for attention heads parallelization. This is the easiest change with the highest impact. Then add Q/K/V and FFN parallelization if needed.

Bottleneck Analysis (Updated)

Current Bottlenecks (in order of impact)

1. Memory bandwidth (70% of time): Loading Q4_0 weights from memory.

   • Mitigated by: Q4_0 quantization (4× compression vs f32)
   • Further optimization: Q4_K AVX2 (P1), better prefetching

2. Attention computation (15% of time): KV cache access + softmax.

   • Current implementation is already efficient for short sequences
   • Further optimization: Flash attention (P3), f16 KV cache (P2)

3. Element-wise operations (10% of time): RMSNorm, SiLU, residual add.

   • Already optimized with SIMD
   • Further optimization: marginal gains only

4. Quantization (5% of time): f32→Q8_0 conversion.

   • Already optimized in quants.rs
   • Further optimization: activation reuse (but this caused regressions)

Comparison with llama.cpp

Why llama.cpp is 2.5-3× faster

1. Multi-threading: llama.cpp uses OpenMP for parallel execution across layers and attention heads. minfer is single-threaded. This alone accounts for ~2× difference on modern CPUs with 8+ cores.

2. Flash attention: Tiled flash attention with online softmax reduces memory traffic and improves cache utilization. minfer uses multi-pass softmax.

3. Transposed K cache: llama.cpp stores K cache transposed (K_f32[dk][kv]) so the KV dimension is contiguous for SIMD. minfer uses position-major layout with strided access.

4. Better matmul kernels: llama.cpp uses tinyBLAS with register blocking and microkernels for matmul. minfer uses simple nested loops.

5. More aggressive quantization: llama.cpp supports Q4_K, Q6_K, Q8_0 for weights and KV cache. minfer primarily uses Q4_0.

What minfer does well

1. Clean Rust code: No unsafe code in most places, strong type safety.
2. Simplicity: Easy to understand and modify.
3. Good baseline optimizations: RMSNorm fusion, RoPE cache, vectorized element-wise ops are already present.
4. Fast for small models: 27.1 tok/s on 0.5B model is reasonable for single-threaded inference.

Recommendations

Short-term (1-2 weeks)

1. Implement Q4_K AVX2 dot product (P1)

   • Highest impact for Q4_K models
   • Reference: llama.cpp quants.c:1900-2076
   • Expected gain: 2-3× for Q4_K matmul operations

2. Add multi-threading (P4)

   • Use rayon or crossbeam for parallel layer execution
   • Expected gain: 2-4× on 8-core CPUs

Medium-term (1 month)

3. Add f16 KV cache (P2)

   • Reduces memory bandwidth by 2×
   • Expected gain: 10-20% for attention-bound workloads

4. Implement flash attention (P3)

   • Only beneficial for long sequences
   • Expected gain: 10-15% for prefill, minimal for decode

Long-term (2+ months)

5. Tiled GEMM with register blocking

   • Replace simple nested loops with microkernel approach
   • Reference: llama.cpp tinyBLAS (sgemm.cpp)
   • Expected gain: 20-30% for matmul operations

6. AVX-512 support

   • 16-wide f32 operations (vs 8-wide AVX2)
   • Requires runtime detection and fallback
   • Expected gain: 1.5-2× for compute-bound operations

Conclusion

Superseded (2026-09-15). This conclusion predates the 2026-08 NEON/threading work and the compute-graph rewrite: minfer is no longer single-threaded, and the CPU path described as "production-ready for single-threaded use cases" has since gained a persistent thread pool and Q8_K activation quantization. Read §"NEON + Threading Overhaul" for the current state.

The minfer codebase is already well-optimized for single-threaded CPU inference on small models. The initial analysis incorrectly identified "scalar hotspots" as bottlenecks, but these were already optimized with SIMD operations.

All optimization attempts (activation quantization reuse, online softmax, RoPE cache stack allocation) caused performance regressions due to:

• Function call overhead
• Interference with compiler auto-vectorization
• Extra memory traffic

The performance gap with llama.cpp (~2.5-3×) is primarily due to:

1. Multi-threading (llama.cpp uses OpenMP)
2. Flash attention with tiled execution
3. Better memory layout (transposed K cache)
4. More aggressive quantization support

Future optimizations should focus on:

• Q4_K AVX2 dot product (highest impact for Q4_K models)
• Multi-threading (easiest way to close the gap)
• f16 KV cache (reduces memory bandwidth)

The code is production-ready for small models and single-threaded use cases. Further optimizations require significant engineering effort and are only justified for larger models or multi-core servers.

NEON + Threading Overhaul (2026-08, Qwen3-4B Q4_K_M on M4 Pro)

The CPU path went from 1.1 tok/s decode (single-threaded scalar) to ~52–58 tok/s at -t 8 (~80–90 % of llama.cpp's 63–69), and prefill from ~1.1 to ~70–75 tok/s. Full analysis: docs/PERF-QWEN3-4B-VS-LLAMACPP.md §3.

What was added

1. aarch64 NEON/SDOT dot kernels (src/quants.rs) — all 8 quantized dot products (Q4_0/Q4_1/Q8_0/Q5_0/Q5_1/Q5_K/Q4_K/Q6_K) use the ARMv8.2 sdot instruction (16 MACs/instr) via stable inline asm (vdotq_s32 is unstable in std::arch). Bit-exact with the scalar kernels (exact int32 accumulation, per-block float ops in identical order); MINFER_NO_NEON=1 reverts. The x86 counterpart since #56 is MINFER_NO_AVX2=1 (the whole quants AVX2 layer) with MINFER_NO_AVX512=1 dropping just its AVX-512/VNNI layer to AVX2.
2. Q8_K activations for K-quant matmuls (src/block.rs Q8KB, src/quants.rs quantize + dots, src/kernel.rs dispatch) — llama.cpp's activation format: 256-element blocks with precomputed int16 per-subblock sums, so dots never re-reduce the activation and use one scale per 256 elements. Kernels restructured to llama's shape (scales applied to the SDOT vectors via vmlaq_n_s32 before the single horizontal reduce).
3. Persistent CPU thread pool + row-parallel matmuls (src/kernel.rs) — per-call thread::scope spawns cost ~170 µs (measured), so a persistent pool (atomic generation handoff, spin-then-yield workers, main thread participates as the last worker) dispatches each matmul in ~1–3 µs. Output is bit-identical to single-threaded (each row computed by one worker). Generic par_for extends it to other ops (attention over heads).
4. NEON vector ops (src/vec_ops.rs) — vec_dot/muladd/scale/add and an in-place softmax (polynomial exp); the attention/norm path was fully scalar on ARM before.
5. CLI -t/--threads <N> (default = macOS P-core count via hw.perflevel0.logicalcpu; measured best 8 for the 4B; E-cores hurt, matching llama).

Pitfalls hit

• vld1q_s16 loads 8 lanes only — loading 16 bsums needs two loads; vpaddq_s16(a, b) yields 8 lanes (a's 4 pairs then b's 4 pairs), and vget_high_s16 on an 8-lane array read undefined stack bytes. Both caused silent garbage; caught by NEON-vs-scalar unit tests on random data.
• int16 vmlal_s8 accumulation overflows for Q8_0 (127×127×4); SDOT's int32 accumulation avoids the whole class.
• Workers spinning between matmuls show up as ~50 % idle in profilers — that is the main thread's serial work (quantize, dispatch, attention), not a kernel problem; the attention-parallelization and NEON quantize closed most of it.

Decisions governing this document

This page is the current contract; the decisions behind it are frozen in the ADR corpus:

• ADR-0013 — The CPU quantizes activations to Q8_0; a device reads f32
• ADR-0007 — No ML frameworks: every operator is hand-written