minfer Debug Dump Mechanism

Feature: --features debug_dump Controlled by: MINFER_DUMP_DIR environment variable

Overview

The debug_dump feature writes raw f32 binary files of hidden states and logits at specific points during inference. Combined with the scripts/compare_layers.py tool (which compares against llama.cpp reference dumps), this enables precise layer-by-layer numerical debugging.

Performance: When not enabled (cargo build --release), all dump code is eliminated at compile time — zero instructions, zero branches. When enabled, overhead is one OnceLock env var read + file write per dump point.

Dump Points

│  forward pass
│
├─ ⓪ minfer_dump_prompt.txt ─────────────────────────────── text dump, NOT a hidden state
│     file: minfer_dump_prompt.txt
│     dump: rendered chat template prompt (for validation against llama prompt)
│
├─ ① embed_out ─────────────────────────────────────────── token_embd (Q5_0 dequant) → hidden
│     file: minfer_dump_embed_out.f32
│     shape: [nt * ne]   (nt = num tokens, ne = hidden dim)
│     dump: embedding lookup output, before any transformer layers
│
└─  for each layer N (0..23):
      │
      ├─ ⑥ layer0_bn ────────────────────────────── RMSNorm(hidden, attn_norm) → bn
      │     file: minfer_dump_layer0_bn.f32  (only layer 0)
      │     shape: [nt * ne]
      │     dump: RMSNorm output, verification target for verify_rmsnorm.py
      │
      ├─ WQ matmul: bn × WQ → bq  [Q5_0 dequant]
      │     dump ⑧: minfer_dump_layer0_bq.f32 (first 32 values)
      │     verification target for verify_matmul.py
      │
      ├─ add_bias(bq) / add_bias(bk) / add_bias(bv)
      ├─ RoPE(bq, bk)
      │     dump ⑫: minfer_dump_layer0_bq_rope.f32 (bq after RoPE)
      │
      ├─ store KV cache
      ├─ GQA attention (f32)
      │     dump ⑬: minfer_dump_layer0_ba.f32 (attention output)
      │
      ├─ WO matmul: ba × WO → bn                            [Q5_0 dequant]
      ├─ residual: hidden += bn
      │
      ├─ ② layer{N}_attn_out ──────────────────────────── attention output
      │     file: minfer_dump_layer{N}_attn_out.f32
      │     shape: [nt * ne]
      │     dump: hidden state after attention branch, before FFN
      │
      ├─ RMSNorm(hidden, ffn_norm) → ffn_in
      ├─ gate/up matmul: ffn_in × Wg/Wu → bg, bf           [Q5_0 dequant × 2]
      │     dump ⑨: minfer_dump_layer0_bg.f32 (first 32 values)
      │
      ├─ SwiGLU: silu(bg) * bf → bg
      │     dump ⑭: minfer_dump_layer0_swiglu.f32 (bg after SwiGLU)
      │
      ├─ down matmul: bg × Wd → bn                          [Q5_0 / Q4_K / Q6_K dequant]
      │     dump ⑩: minfer_dump_layer0_fd.f32 (first 32 values)
      │
      ├─ residual: hidden += bn
      │
      └─ ③ layer{N}_out ───────────────────────────────── FFN output
            file: minfer_dump_layer{N}_out.f32
            shape: [nt * ne]
            dump: hidden state after full layer (attention + FFN + residuals)

   after all layers:

   ├─ RMSNorm(hidden, output_norm) → bn
   ├─ LM head: bn × output.weight → logits                 [Q8_0 matmul]
   ├─ add output_bias
   │
   ├─ ⑤ last_norm ────────────────────────────────────── post-RMSNorm hidden
   │     file: minfer_dump_last_norm.f32
   │
   └─ ④ logits ────────────────────────────────────────── final logits
         file: minfer_dump_logits.f32
         shape: [nt * n_vocab]
         dump: raw logits before sampling


   ─── Utility dumps (not layer-specific) ───

   ⑦ minfer_dump_q8_quant_verify.txt ──────────────────── Q8_0 quantize
         fired once on first quantize_row_q8_0_buf call
         dump: amax, d, x[0], x[1], x[16], q[0], q[1], q[16]
         verification target for verify_q8_quant.py

minfer Side

Feature flag

# Cargo.toml
[features]
debug_dump = []

Core module: src/dump.rs

#![allow(unused)]
fn main() {
// For float arrays (hidden states, logits)
pub fn maybe_dump(name: &str, data: &[f32])

// For text (prompts)
pub fn maybe_dump_text(name: &str, text: &str)
}

• Both controlled by MINFER_DUMP_DIR env var
• If not set → no-op (returns immediately)
• If set → maybe_dump writes raw f32 bytes to {MINFER_DUMP_DIR}/{name}.f32
• If set → maybe_dump_text writes UTF-8 string to {MINFER_DUMP_DIR}/{name}.txt
• Uses OnceLock to cache the env var check (read once, reused)

Build & run

# Normal build — zero overhead, dump code eliminated at compile time
cargo build --release

# Debug build — dump enabled
cargo build --release --features debug_dump

# Run with CPU path + dump
MINFER_DISABLE_MPS=1 MINFER_DUMP_DIR=/tmp \
  cargo run --release --features debug_dump -- <model> "Hello"

Python Side

scripts/dump_llama_ref.py

Generates llama.cpp reference hidden states for comparison.

# Bare text (no chat template wrapping)
uv run python -m scripts.dump_llama_ref \
  --model <path-to-gguf> \
  --prompt "Hello" \
  --output ./llama_ref

# With chat template — reads tokenizer.chat_template from GGUF,
# renders with Jinja2, producing the same prompt minfer would use
uv run python -m scripts.dump_llama_ref \
  --model <path-to-gguf> \
  --prompt "Hello" --chat \
  --output ./llama_ref

Output (per layer):

How it works: Uses the "truncated model" technique — creates a fake GGUF with block_count=N (metadata only, zero-copy weights), runs llama.eval(), and extracts the embedding output via llama_get_embeddings(). Since a decoder-only transformer is feed-forward, the first N layers produce identical outputs to a full model.

When --chat is specified, the script reads tokenizer.chat_template from GGUF metadata, renders it with Jinja2 using messages=[{"role":"user", "content": prompt}] and add_generation_prompt=True, producing a prompt identical to minfer's chat template rendering.

scripts/compare_layers.py

Compares minfer dumps against llama.cpp reference.

uv run python -m scripts.compare_layers \
  --llama-dir ./llama_ref \
  --minfer-dir /tmp \
  --hidden-dim 896

Layer mapping: llama layer N (N=1..24) = minfer layer N-1 (0..23). Both dump the hidden state AFTER the corresponding layer's computation.

Prompt validation: Before layer comparison, the script reads prompt.txt (from llama reference) and minfer_dump_prompt.txt (from minfer dump). If both exist and differ, the script immediately aborts with PROMPT MISMATCH — comparison aborted and sys.exit(1).

Output: Per-layer:

Plus logits comparison with top token and cosine.

Full Workflow

# Step 1: Generate llama.cpp reference (one-time, ~20 min for 24-layer model)
#         --chat ensures the SAME prompt as minfer (rendered from tokenizer.chat_template)
uv run python -m scripts.dump_llama_ref \
  --model ~/.cache/minfer/models/hf/Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --prompt "Hello" --chat --output ./llama_ref

# Step 2: Run minfer with debug dump (CPU path)
#         MINFER_DUMP_DIR enables all dump points including prompt
MINFER_DISABLE_MPS=1 MINFER_DUMP_DIR=/tmp \
  cargo run --release --features debug_dump \
  -- <model> "Hello"

# Step 3: Compare
#         Script validates prompt files match before comparing layers
uv run python -m scripts.compare_layers \
  --llama-dir ./llama_ref --minfer-dir /tmp --hidden-dim 896

Diagnostic Logic

Given compare_layers.py output showing first divergence at some layer, use the sequence below to isolate the bug:

Layer numbering

Path Verification Status (2026-07-28)

All CPU inference paths verified correct through automated cross-validation against gguf.quants.dequantize() (validated against llama.cpp C reference in gguf-py/tests/test_quants.py).

Verification Results

Despite all 14 paths being verified correct, the Q5_K_M model still produces garbled output. Model file confirmed working with llama-cli. Root cause is a subtle integration issue not captured by individual verification.

Diagnostic Status (2026-07-29)

Verification Method

gguf.quants.dequantize() ── GGUF file ──→ Python reference values
        │                                       │
        │ (validated against C)                  │ compare
        │                                       │
minfer's own computation  ── forward pass ──→ minfer dump (.f32 / .txt)

The Python side uses gguf.quants.dequantize() from gguf-py, which is the same implementation validated in llama.cpp's test_quants.py (quantize + dequant must be bit-exact against the C reference). This eliminates the "Python formula might be wrong" concern — the verification chain traces back to llama.cpp's C implementation.

Row Stride Verification (2026-07-28)

The hypothesis that matmul ws (computed as (id/blck_size)*type_size) diverges from the GGUF physical row stride was tested and disproven:

The formulas are inherently consistent: both derive from (ne[0]/blck_size)*type_size.

Matmul Output Verification (2026-07-28)

All three matmul types verified correct using full forward-computation in Python, including bias and SwiGLU:

Despite all 10 paths being verified correct, the Q5_K_M model still produces garbled output. Remaining unverified: RoPE, GQA attention, SwiGLU, KV cache, residual connections. Root cause remains unidentified.

Verification Scripts

The verification scripts in scripts/verify_*.py provide standalone Python reference implementations for independent validation of minfer's computation. All dequantization uses gguf.quants.dequantize() — the same Python implementation validated against llama.cpp's C reference in gguf-py/tests/test_quants.py (quantize + dequant must be bit-exact).

These read the same GGUF file as minfer, perform the same computation using the validated gguf.quants.dequantize(), and print key values for comparison with minfer dumps (under --features debug_dump).

Two additional verification paths are covered by Rust unit tests:

Output Files (minfer)

Total: 24 × 2 × 105 KB + 17.4 MB ≈ 22 MB for a 24-layer model.