⚠ SPEED CAVEAT: Slow deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning. Fast for quality, not throughput.
tok/sec17.09
tokens9639
TTFT2.83s
▮ PILLAR BREAKDOWN
Complexity (O(1))
16/20
Concurrency / Races
6/20
Error Handling
14/20
Resource & State Safety
10/20
Test Integrity
12/20
✓ WENT RIGHT
Complexity (O(1)) (16/20)
✗ WENT WRONG
Test Integrity (12/20)
Resource & State Safety (10/20)
Concurrency / Races (6/20)
▮ CRITICAL BUGS
submit() raises RuntimeError('Backpressure: queue full') when the deque hits 100 items instead of AWAITING a free slot — the spec explicitly requires blocking backpressure, not rejection. Same central failure as Qwen3-Coder (which scored 44 for the same reason). The backpressure test compounds this by asserting the raise is correct behavior, encoding the bug rather than catching it.
No shutdown/stop mechanism: the 4 worker tasks are infinite 'while True' loops with no way to cancel or join them. drain() busy-polls and returns, but the workers keep running forever — every TTSPipeline instance leaks 4 tasks.
Soft-cancel does not interrupt a job already inside mock_synthesize: the cancelled flag is only checked before start and between retry attempts. A first-try-success job cancelled mid-flight still completes and emits 'completed', not 'cancelled'.
▮ RECOMMENDED USE
Retry-heavy async job pipelines where backpressure-await is not required and a hard queue-reject is acceptable — the one weak prompt in muse-glimmer's battery. Retry/backoff and callback handling are sound, but the central bounded-concurrency mechanic is wrong.
▮ REFACTORED PATCH
# Fix 1 — backpressure by awaiting a slot, not raising.
class TTSPipeline:
def __init__(self):
self._slots = asyncio.Semaphore(4) # bounded concurrency at the gate
self._max_queued = 100
self._queue = collections.deque()
self._queue_lock = asyncio.Lock()
self._queue_not_full = asyncio.Condition(self._queue_lock)
self._queue_not_empty = asyncio.Condition(self._queue_lock)
async def submit(self, text, voice):
job_id = uuid.uuid4().hex
async with self._queue_lock:
while len(self._queue) >= self._max_queued:
await self._queue_not_full.wait() # BACKPRESSURE: await, never raise
self._queue.append({'id': job_id, 'text': text, 'voice': voice})
self._queue_not_empty.notify()
return job_id
# workers call self._queue_not_full.notify() after popleft()
# Fix 2 — add a real shutdown so workers don't leak
async def stop(self):
for w in self._workers: w.cancel()
await asyncio.gather(*self._workers, return_exceptions=True)
self._workers.clear()
# Fix 3 — correct the test to assert backpressure, not raise
async def flood():
for i in range(120): await slow.submit(f't{i}', 'v') # never raises
await asyncio.wait_for(flood(), timeout=30) # completes once workers drain