▚ LOCAL LLM BENCHMARK SUITE // LFU CACHE & ACID AUDIT

// BENCHMARK_RESULTS .json

37 models graded on a strict 5-pillar / 100-pt rubric · O(1) LFU + ACID transactions · M3 Max · LM Studio · TOP: DeepSeek V4 Flash (CLOUD baseline)

Models Tested

Top Score

Average

64.9

Prod-Ready

1/37

▮ 5-PILLAR RADAR — TOP 3

▮ SCORE vs THROUGHPUT (tok/sec)

Local models only — cloud baseline (DeepSeek) excluded from the speed axis. Bars flagged ⚠ have suspected GPU-offload / inference issues (not representative of the model).

▮ KEY FINDINGS — patterns across 36 local models

12/36

RUN THEIR OWN TESTS

Half of local models crash before completing — runnability is the real filter.

22/36

CRITICAL BUGS

Cache corruption, evict-crashes, or fatal KeyErrors — not safe for systems work.

3/36

DECLARE __slots__

Rubric explicitly required it for memory efficiency; nearly all models miss it.

12/36

USE time.monotonic()

The rest use the system clock — NTP jumps corrupt TTL eviction.

Quant depth dominates quality. Same model, different quant: Qwen 3.6 35B-A3B scores 82 at 6-bit but 57 at 4-bit — a ~25-point drop. Aggressive quants cost real logic on systems code.
Speed ≠ quality. The 4-bit Qwen is the fastest (83 t/s) yet scores 57; the 6-bit is slower (69 t/s) but scores 82. Pick quants for correctness first, throughput second.
Concurrency is the killer pillar (avg 13.2/20). Local models most often break on async correctness — lock type mismatches, races, and non-reentrant-lock deadlocks.
The weakest pillar overall is Test Integrity (avg 10.5/20). Test suites that ship with crashing code validate nothing.
Only the cloud baseline (DeepSeek, 91) cleared Production-Ready. Best local scores cap at 82 — strong scaffolding, but every submission needs a human pass on __slots__, monotonic clocks, and lock granularity.

▮ CLOUD-OFFLOAD STRATEGY — what to actually run when rationing cloud credits

Goal: a trustworthy local backup for when your cloud-LLM plan runs low. A model only earns offload duty if its code runs — a high score that crashes wastes your time and still costs a cloud call afterward. Only 12/36 local models produced runnable code; 24/36 crashed.

✓ RUNS — offload candidates (verify output, then ship)

Muse Glimmer 28BGGUF17 t/s89TRUSTED Muse Glimmer 28BGGUF17 t/s88TRUSTED Gemma 4 26B-A4BMLX53 t/s86TRUSTED Qwen 3.6 35B-A3B6-bit MLX69 t/s82TRUSTED Gemma 4 26B-A4B8-bit58 t/s82TRUSTED Gemma 4 26B-A4B8-bit MLX49 t/s80TRUSTED Gemma 4 31BGGUF10 t/s78TRUSTED Qwen 3.6 27B8-bit12 t/s78TRUSTED Qwen 3.6 35B-A3B6-bit MLX69 t/s75VERIFY Gemma 4 31B QATQAT GGUF15 t/s70VERIFY Qwen3 Coder 30B6-bit MLX73 t/s58VERIFY Muse Glimmer 28BGGUF17 t/s58VERIFY

✗ CRASHES — do NOT offload (wasted cycle + cloud call anyway)

Muse Glimmer 28BGGUF20 t/s88AVOID Muse Glimmer 28BGGUF17 t/s85AVOID Muse Glimmer 28BGGUF19 t/s81AVOID Muse Glimmer 28BGGUF19 t/s76AVOID Gemma 4 26B-A4B8-bit MLX59 t/s72AVOID KAT-Coder v2.5 Dev XLMLX65 t/s72AVOID KAT-Coder v2.5 Dev XLMLX65 t/s65AVOID Qwen3 Coder 30B6-bit MLX73 t/s62AVOID KAT-Coder v2.5 Dev XLMLX65 t/s60AVOID Qwen 3.6 35B-A3B4-bit MLX83 t/s57AVOID Gemma 4 26B-A4B8-bit MLX59 t/s55AVOID Qwen3 Coder 30B6-bit MLX73 t/s54AVOID KAT-Coder v2.5 Dev XLMLX65 t/s52AVOID Qwen3 Coder 30B6-bit73 t/s50AVOID Qwen 3.6 35B-A3B6-bit MLX69 t/s50AVOID Qwen 3.6 35B-A3B (uncensGGUF63 t/s49AVOID Qwen 3.6 35B-A3B6-bit MLX70 t/s49AVOID KAT-Coder v2.5 Dev XLMLX65 t/s49AVOID Gemma 4 26B-A4B8-bit MLX59 t/s48AVOID Qwen 3.6 35B-A3B (uncensGGUF63 t/s46AVOID Qwen3 Coder 30B6-bit MLX73 t/s45AVOID Qwen3 Coder 30B6-bit MLX73 t/s44AVOID Gemma 4 12B Coder (fablemxfp8 MLX25 t/s43AVOID KAT-Coder v2.5 Dev XLMLX65 t/s36AVOID

Rule of thumb: route a task to local only if the top runner above handles it; if the local output doesn't run on first try, stop and use the cloud model — debugging bad local code usually costs more than the API call you tried to save.
Best pure-offload picks (runs clean + fast + ≥good): 6-bit MLX, 6-bit MLX, 6-bit MLX.
Don't trust the score alone. The 4-bit Qwen is the fastest local model (83 t/s) but crashes — speed is meaningless if the code doesn't run.

▮ LEADERBOARD

#	Model	Quant	Format	Speed	Score	Verdict	Best For
#1	DeepSeek V4 Flash (CLOUD baseline) lfu ⚠ speed suspect	n/a (cloud)	CLOUD	N/A t/s	91	PROD	Reference-quality baseline (91/100) — the bar the local mode…	DECODE ▸
#2	Muse Glimmer 28B automation ⚠ speed suspect	GGUF	GGUF	17.2 t/s	89	FLAWS	Reliable production-grade async batch processing with atomic…	DECODE ▸
#3	Muse Glimmer 28B data ⚠ speed suspect	GGUF	GGUF	17.2 t/s	88	FLAWS	Clean parameterized queries throughout (zero interpolation, …	DECODE ▸
#4	Muse Glimmer 28B mcp ⚠ speed suspect	GGUF	GGUF	20.2 t/s	88	FLAWS	Clean, idiomatic single-file MCP server with REAL SDK wiring…	DECODE ▸
#5	Gemma 4 26B-A4B data	MLX	MLX	53.5 t/s	86	FLAWS	Production-shaped async data-access layer with correct pooli…	DECODE ▸
#6	Muse Glimmer 28B rust ⚠ speed suspect	GGUF	GGUF	17.0 t/s	85	FLAWS	Production-grade async Rust service design with real tokio c…	DECODE ▸
#7	Qwen 3.6 35B-A3B lfu	6-bit MLX	MLX	68.9 t/s	82	FLAWS	Solid daily-driver scaffolding for ACID/async patterns — pro…	DECODE ▸
#8	Gemma 4 26B-A4B lfu	8-bit	MLX	58.3 t/s	82	FLAWS	Tied top local scorer (82). The only local model to use delt…	DECODE ▸
#9	Muse Glimmer 28B webhook ⚠ speed suspect	GGUF	GGUF	19.0 t/s	81	FLAWS	Carefully designed async webhook bridge with correct constan…	DECODE ▸
#10	Gemma 4 26B-A4B tts	8-bit MLX	MLX	49.5 t/s	80	FLAWS	DECISIVE: passes the TTS pipeline that Qwen 6-bit couldn…	DECODE ▸
#11	Gemma 4 31B lfu ⚠ speed suspect	GGUF	GGUF	10.1 t/s	78	FLAWS	Clean, correct, runnable code with solid O(1) structure and …	DECODE ▸
#12	Qwen 3.6 27B lfu ⚠ speed suspect	8-bit	MLX	12.3 t/s	78	FLAWS	Clean, correct, runnable — same tier as Gemma 4 31B (78). Go…	DECODE ▸
#13	Muse Glimmer 28B lfu ⚠ speed suspect	GGUF	GGUF	19.4 t/s	76	FLAWS	The strongest LFU result from a non-coder model in this benc…	DECODE ▸
#14	Qwen 3.6 35B-A3B webhook	6-bit MLX	MLX	69.2 t/s	75	FLAWS	Best non-LFU result for this model (75 vs TTS 49, Rust 50). …	DECODE ▸
#15	Gemma 4 26B-A4B rust	8-bit MLX	MLX	58.6 t/s	72	FLAWS	DECISIVE: Rust COMPILES CLEAN (0 errors with deps declared) …	DECODE ▸
#16	KAT-Coder v2.5 Dev XL webhook	MLX	MLX	65.3 t/s	72	CRIT	Reference design for the 5 required webhook mechanisms; need…	DECODE ▸
#17	Gemma 4 31B QAT lfu ⚠ speed suspect	QAT GGUF	GGUF	15.0 t/s	70	CRIT	Runnable and structurally sound, but the LFU eviction has a …	DECODE ▸
#18	KAT-Coder v2.5 Dev XL lfu	MLX	MLX	65.3 t/s	65	CRIT	Promising code-design instincts (cleanest abstractions and b…	DECODE ▸
#19	Qwen3 Coder 30B webhook	6-bit MLX	MLX	72.7 t/s	62	CRIT	A starting skeleton for a stdlib webhook bridge where someon…	DECODE ▸
#20	KAT-Coder v2.5 Dev XL automation	MLX	MLX	65.3 t/s	60	CRIT	Reference scaffold for an asyncio checkpointed batch process…	DECODE ▸
#21	Qwen3 Coder 30B automation	6-bit MLX	MLX	72.7 t/s	58	CRIT	Throwaway single-shot batch jobs where you only care that it…	DECODE ▸
#22	Muse Glimmer 28B tts ⚠ speed suspect	GGUF	GGUF	17.1 t/s	58	CRIT	Retry-heavy async job pipelines where backpressure-await is …	DECODE ▸
#23	Qwen 3.6 35B-A3B lfu	4-bit MLX	MLX	83.3 t/s	57	CRIT	Not recommended for systems code as-is. The 4-bit quant degr…	DECODE ▸
#24	Gemma 4 26B-A4B webhook	8-bit MLX	MLX	59.2 t/s	55	CRIT	Violated the stdlib-only constraint: used aiohttp (2 imports…	DECODE ▸
#25	Qwen3 Coder 30B rust	6-bit MLX	MLX	72.7 t/s	54	CRIT	Sketching idiomatic Rust type/trait/error-enum shapes when t…	DECODE ▸
#26	KAT-Coder v2.5 Dev XL tts	MLX	MLX	65.3 t/s	52	CRIT	Scaffolding async pipeline structure (queue+semaphore+retry+…	DECODE ▸
#27	Qwen3 Coder 30B lfu	6-bit	MLX	72.7 t/s	50	CRIT	Not usable as-is — transactions crash immediately due to an …	DECODE ▸
#28	Qwen 3.6 35B-A3B rust	6-bit MLX	MLX	69.0 t/s	50	CRIT	NOT usable for Rust as-is — 7 compile errors. Same model tha…	DECODE ▸
#29	Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive) lfu	GGUF	GGUF	62.5 t/s	49	CRIT	Not recommended for production code. Reasonable API shape an…	DECODE ▸
#30	Qwen 3.6 35B-A3B tts	6-bit MLX	MLX	69.9 t/s	49	CRIT	NOT a safe offload for async-pipeline / queue work despite s…	DECODE ▸
#31	KAT-Coder v2.5 Dev XL lfu	MLX	MLX	65.3 t/s	49	CRIT	Strong architectural instincts for async data-structure desi…	DECODE ▸
#32	Gemma 4 26B-A4B automation	8-bit MLX	MLX	59.2 t/s	48	CRIT	Doesn't run: SyntaxError (global processor declared aft…	DECODE ▸
#33	Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive) automation	GGUF	GGUF	62.5 t/s	46	CRIT	Generating plausible-looking async scaffolding that passes c…	DECODE ▸
#34	Qwen3 Coder 30B lfu	6-bit MLX	MLX	72.7 t/s	45	CRIT	Drafting async scaffolding and transaction API shapes when y…	DECODE ▸
#35	Qwen3 Coder 30B tts	6-bit MLX	MLX	72.7 t/s	44	CRIT	Scaffolding async job pipelines when you intend to rewrite t…	DECODE ▸
#36	Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge) lfu	mxfp8 MLX	MLX	25.3 t/s	43	CRIT	Not usable as-is — the cache cannot store its first key and …	DECODE ▸
#37	KAT-Coder v2.5 Dev XL rust	MLX	MLX	65.3 t/s	36	CRIT	Generating idiomatic tokio building blocks (task functions, …	DECODE ▸

▮ FORMAT & QUANT SHOWDOWN — same family, different quants/formats

Families with 2+ variants. Click a row for the full audit. Compare how quant depth and MLX-vs-GGUF change the score.

Muse Glimmer 7 variants · score spread 31 · best 89 (GGUF)

GGUFGGUF17 t/s89 GGUFGGUF17 t/s88 GGUFGGUF20 t/s88 GGUFGGUF17 t/s85 GGUFGGUF19 t/s81 GGUFGGUF19 t/s76 GGUFGGUF17 t/s58

Gemma 4 9 variants · score spread 43 · best 86 (MLX)

MLXMLX53 t/s86 8-bitMLX58 t/s82 8-bit MLXMLX49 t/s80 GGUFGGUF10 t/s78 8-bit MLXMLX59 t/s72 QAT GGUFGGUF15 t/s70 8-bit MLXMLX59 t/s55 8-bit MLXMLX59 t/s48 mxfp8 MLXMLX25 t/s43

Qwen 3.6 8 variants · score spread 36 · best 82 (6-bit MLX)

6-bit MLXMLX69 t/s82 8-bitMLX12 t/s78 6-bit MLXMLX69 t/s75 4-bit MLXMLX83 t/s57 6-bit MLXMLX69 t/s50 GGUFGGUF63 t/s49 6-bit MLXMLX70 t/s49 GGUFGGUF63 t/s46

KAT- Dev XL 6 variants · score spread 36 · best 72 (MLX)

MLXMLX65 t/s72 MLXMLX65 t/s65 MLXMLX65 t/s60 MLXMLX65 t/s52 MLXMLX65 t/s49 MLXMLX65 t/s36

Qwen3 6 variants · score spread 18 · best 62 (6-bit MLX)

6-bit MLXMLX73 t/s62 6-bit MLXMLX73 t/s58 6-bit MLXMLX73 t/s54 6-bitMLX73 t/s50 6-bit MLXMLX73 t/s45 6-bit MLXMLX73 t/s44