minfer

A minimal local LLM inference engine built from scratch in Rust, modeled on llama.cpp's ggml_cgraph + backend scheduler, with zero ML framework dependencies.

• Declarative compute graph — inference builds a ComputeGraph (pure IR), then assigns backends, fuses ops, allocates and executes via a scheduler. Params-only graph reuse (decode steps skip reconstruction).
• Backends — CPU + Apple MPS (Metal); optional CUDA (feature-gated).
• Models — Qwen2 / Qwen2.5 / Qwen3 (dense), GGUF v3 format.
• Tooling — OpenAI-compatible server, CLI multi-turn conversation, an interactive web visualizer, and per-layer debug dumps.

Explore

• Architecture — module map, pipeline, backend layering, adding an arch.
• Compute Graph — the declarative graph design + rewrite record.
• Architecture Roadmap — system-layer gaps and the prioritized backlog.
• Backends: Metal · CPU · GPU Safety.
• Qwen3 Support and the Model Support Roadmap.
• Tooling: Debug Dump, web visualizer (viz/).

Build & run

cargo build --release
./target/release/minfer <model.gguf> "hello"