▚ LOCAL LLM BENCHMARK SUITE // LFU CACHE & ACID AUDIT

// BENCHMARK_RESULTS .json

13 models graded on a strict 5-pillar / 100-pt rubric · O(1) LFU + ACID transactions · M3 Max · LM Studio · TOP: DeepSeek V4 Flash (CLOUD baseline)

Models Tested

Top Score

Average

64.9

Prod-Ready

1/13

▮ 5-PILLAR RADAR — TOP 3

▮ SCORE vs THROUGHPUT (tok/sec)

Local models only — cloud baseline (DeepSeek) excluded from the speed axis. Bars flagged ⚠ have suspected GPU-offload / inference issues (not representative of the model).

▮ KEY FINDINGS — patterns across 12 local models

5/12

RUN THEIR OWN TESTS

Half of local models crash before completing — runnability is the real filter.

8/12

CRITICAL BUGS

Cache corruption, evict-crashes, or fatal KeyErrors — not safe for systems work.

1/12

DECLARE __slots__

Rubric explicitly required it for memory efficiency; nearly all models miss it.

3/12

USE time.monotonic()

The rest use the system clock — NTP jumps corrupt TTL eviction.

Quant depth dominates quality. Same model, different quant: Qwen 3.6 35B-A3B scores 82 at 6-bit but 57 at 4-bit — a ~25-point drop. Aggressive quants cost real logic on systems code.
Speed ≠ quality. The 4-bit Qwen is the fastest (83 t/s) yet scores 57; the 6-bit is slower (69 t/s) but scores 82. Pick quants for correctness first, throughput second.
Concurrency is the killer pillar (avg 13.7/20). Local models most often break on async correctness — lock type mismatches, races, and non-reentrant-lock deadlocks.
The weakest pillar overall is Test Integrity (avg 10.0/20). Test suites that ship with crashing code validate nothing.
Only the cloud baseline (DeepSeek, 91) cleared Production-Ready. Best local scores cap at 82 — strong scaffolding, but every submission needs a human pass on __slots__, monotonic clocks, and lock granularity.

▮ CLOUD-OFFLOAD STRATEGY — what to actually run when rationing cloud credits

Goal: a trustworthy local backup for when your cloud-LLM plan runs low. A model only earns offload duty if its code runs — a high score that crashes wastes your time and still costs a cloud call afterward. Only 5/12 local models produced runnable code; 7/12 crashed.

✓ RUNS — offload candidates (verify output, then ship)

Qwen 3.6 35B-A3B6-bit MLX69 t/s82TRUSTED Gemma 4 26B-A4B8-bit58 t/s82TRUSTED Gemma 4 31BGGUF10 t/s78TRUSTED Qwen 3.6 27B8-bit12 t/s78TRUSTED Gemma 4 31B QATQAT GGUF15 t/s70VERIFY

✗ CRASHES — do NOT offload (wasted cycle + cloud call anyway)

KAT-Coder v2.5 Dev XLMLX65 t/s65AVOID Qwen 3.6 35B-A3B4-bit MLX83 t/s57AVOID Qwen3 Coder 30B6-bit73 t/s50AVOID Qwen 3.6 35B-A3B6-bit MLX69 t/s50AVOID Qwen 3.6 35B-A3B (uncensGGUF63 t/s49AVOID Qwen 3.6 35B-A3B6-bit MLX70 t/s49AVOID Gemma 4 12B Coder (fablemxfp8 MLX25 t/s43AVOID

Rule of thumb: route a task to local only if the top runner above handles it; if the local output doesn't run on first try, stop and use the cloud model — debugging bad local code usually costs more than the API call you tried to save.
Best pure-offload picks (runs clean + fast + ≥good): 6-bit MLX, 8-bit.
Don't trust the score alone. The 4-bit Qwen is the fastest local model (83 t/s) but crashes — speed is meaningless if the code doesn't run.

▮ LEADERBOARD

#	Model	Quant	Format	Speed	Score	Verdict	Best For
#1	DeepSeek V4 Flash (CLOUD baseline) lfu ⚠ speed suspect	n/a (cloud)	CLOUD	N/A t/s	91	PROD	Reference-quality baseline (91/100) — the bar the local mode…	DECODE ▸
#2	Qwen 3.6 35B-A3B lfu	6-bit MLX	MLX	68.9 t/s	82	FLAWS	Solid daily-driver scaffolding for ACID/async patterns — pro…	DECODE ▸
#3	Gemma 4 26B-A4B lfu	8-bit	MLX	58.3 t/s	82	FLAWS	Tied top local scorer (82). The only local model to use delt…	DECODE ▸
#4	Gemma 4 31B lfu ⚠ speed suspect	GGUF	GGUF	10.1 t/s	78	FLAWS	Clean, correct, runnable code with solid O(1) structure and …	DECODE ▸
#5	Qwen 3.6 27B lfu ⚠ speed suspect	8-bit	MLX	12.3 t/s	78	FLAWS	Clean, correct, runnable — same tier as Gemma 4 31B (78). Go…	DECODE ▸
#6	Gemma 4 31B QAT lfu ⚠ speed suspect	QAT GGUF	GGUF	15.0 t/s	70	CRIT	Runnable and structurally sound, but the LFU eviction has a …	DECODE ▸
#7	KAT-Coder v2.5 Dev XL lfu	MLX	MLX	65.3 t/s	65	CRIT	Promising code-design instincts (cleanest abstractions and b…	DECODE ▸
#8	Qwen 3.6 35B-A3B lfu	4-bit MLX	MLX	83.3 t/s	57	CRIT	Not recommended for systems code as-is. The 4-bit quant degr…	DECODE ▸
#9	Qwen3 Coder 30B lfu	6-bit	MLX	72.7 t/s	50	CRIT	Not usable as-is — transactions crash immediately due to an …	DECODE ▸
#10	Qwen 3.6 35B-A3B rust	6-bit MLX	MLX	69.0 t/s	50	CRIT	NOT usable for Rust as-is — 7 compile errors. Same model tha…	DECODE ▸
#11	Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive) lfu	GGUF	GGUF	62.5 t/s	49	CRIT	Not recommended for production code. Reasonable API shape an…	DECODE ▸
#12	Qwen 3.6 35B-A3B tts	6-bit MLX	MLX	69.9 t/s	49	CRIT	NOT a safe offload for async-pipeline / queue work despite s…	DECODE ▸
#13	Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge) lfu	mxfp8 MLX	MLX	25.3 t/s	43	CRIT	Not usable as-is — the cache cannot store its first key and …	DECODE ▸

▮ FORMAT & QUANT SHOWDOWN — same family, different quants/formats

Families with 2+ variants. Click a row for the full audit. Compare how quant depth and MLX-vs-GGUF change the score.

Qwen 3.6 6 variants · score spread 33 · best 82 (6-bit MLX)

6-bit MLXMLX69 t/s82 8-bitMLX12 t/s78 4-bit MLXMLX83 t/s57 6-bit MLXMLX69 t/s50 GGUFGGUF63 t/s49 6-bit MLXMLX70 t/s49

Gemma 4 4 variants · score spread 39 · best 82 (8-bit)

8-bitMLX58 t/s82 GGUFGGUF10 t/s78 QAT GGUFGGUF15 t/s70 mxfp8 MLXMLX25 t/s43