CrispASR

CrispASR — Performance benchmarks

Test audio: jfk.wav (11.0s), Q4_K quantization, greedy decode (-bs 1).

Metal im2col: the v0.17 sync silently dropped the batch-1 occupancy win — restored as kernel_im2col_flat, melotts hifigan back to ~1.85x (Apple M1, 2026-08-06)

The v0.17 ggml sync removed CRISPASR_METAL_IM2COL_OCC (upstream reworked the dispatch; see tools/upstream-prs/SYNC-v0.17-CONFLICTS.md “re-derive if regressions appear”) — which un-fixed the melotts/piper “P0 hifigan” row below: every batch-1 Metal conv went back to KH*KW-thread threadgroups. The re-derivation now lives in the shared fork as kernel_im2col_flat (89a2039d, branch sync/upstream-v0.17; authored against CrispEmbed’s PP-OCR profile, where the same kernel is 2.3x on the recognizer): one thread per dst element from a (ceil(OW*CHW/256), OH, N) grid, predicate N*KH*KW < 128 || IC == 1 (also covers the conv_2d_dw lowering the OCC variant missed), CRISPASR_METAL_IM2COL_FLAT=0 restores the standard kernel. A first cut with int64-divmod flat indexing was SLOWER than the broken dispatch — Apple GPUs emulate int64 division.

This pin bump (a0f7289d → 89a2039d) re-activates the win. Measured, M1, interleaved same-binary pairs:

The historical OCC numbers in the melotts row below (2–3x hifigan, 1.65x moonshine) were measured on the pre-v0.17 variant on a quiet box; today’s moonshine is closer to neutral because the current decode path spends its time elsewhere. CrispEmbed-side evidence (2.3x PP-OCR rec, 1.6x layout_detect, 18-fixture byte-identity) is in their PERFORMANCE.md. tools/upstream-prs/23 should be re-drafted from this kernel for upstream.

Backend × Optimization matrix

At-a-glance view of which performance knobs each backend supports today, and where the gaps are. Last refresh: 2026-05-04 (after PLAN §79 — 14-commit session that shipped #69a / #69b / #69e / #72 / #73).

Legend: ✓ = supported, opt-in via env var · F16 = stuck at F16 (quant cache types unavailable; attention path needs migration) · — = not applicable (no KV cache or no transformer blocks) · · = applicable but not yet wired (port deferred).

LLM-decoder ASR (high VRAM, autoregressive)

backend KV_QUANT KV_QUANT_K/_V KV_ON_CPU N_GPU_LAYERS weight residency
voxtral4b (4B) ✓ ✓ ✓ ✓ gpu
voxtral (3B) ✓ ✓ ✓ ✓ gpu
granite-speech (1B / 4.0 / 4.1 / 4.1-plus / 4.1-nar) ✓ ✓ ✓ ✓ gpu
gemma4-e2b (5B effective) ✓ ✓ ✓ ✓ gpu (FLIPPED §72)
mimo-asr (1.4B) ✓ ✓ ✓ ✓ gpu (FLIPPED §72)
qwen3-asr (0.6B) ✓ ✓ ✓ ✓ gpu
glm-asr (1B) ✓ ✓ ✓ ✓ gpu
omniasr-llm (300M) ✓ ✓ ✓ ✓ gpu
vibevoice (4B ASR mode) F16 F16 F16 ✓ gpu

Encoder-decoder ASR (medium VRAM, autoregressive)

backend KV_QUANT KV_QUANT_K/_V KV_ON_CPU N_GPU_LAYERS notes
canary (1B) ✓ ✓ ✓ · flash_attn_ext default, -17 % on JFK with q8_0/q4_0
cohere (2B) ✓ ✓ ✓ · cast-on-read default (13 % faster on 30 s chunks); CRISPASR_COHERE_FLASH=1 for unchunked long-form (-26 % win at 300 s) — see §5
kyutai-stt (1B) ✓ ✓ ✓ · flash_attn_ext native, quant-safe
firered-asr (900M) — — — — inline AED, no exposed transformer KV
moonshine-tiny / streaming — — — — tiny decoder, no exposed KV

Encoder-only ASR (low VRAM, single forward)

backend KV_QUANT KV_QUANT_K/_V KV_ON_CPU N_GPU_LAYERS notes
whisper (legacy) ✓ ✓ ✓ — upstream loader, separate path
parakeet (TDT) — — — — RNN-T transducer, no KV cache
fastconformer-ctc — — — — CTC head
wav2vec2 / hubert / data2vec — — — — CTC heads
omniasr (CTC variant) — — — — CTC head

TTS

backend KV_QUANT KV_QUANT_K/_V KV_ON_CPU N_GPU_LAYERS notes
orpheus (3B + DE / lex-au variants) ✓ ✓ ✓ ✓ shared Llama-3 path
chatterbox (T3 + CFG cache) ✓ ✓ ✓ · uses kv_self_attn natively
qwen3-tts (0.6B + 1.7B variants) ✓ talker ✓ talker ✓ talker · code-predictor cache stays F16 (separate path)
vibevoice (4B TTS mode) F16 F16 F16 ✓ KV migration still pending; layer offload routes tts_lm.layers.<N>.*
kokoro — — — — non-AR vocoder, no transformer KV

Batched classifier-free guidance (CFG) — the dispatch/bandwidth lever (§214)

Most TTS backends apply CFG by running the conditioned and unconditioned passes as two sequential forwards and blending the outputs. That’s correct and quant-safe, but on a single-token AR step (or per-diffusion-step) it reads the full weight set twice and doubles the dispatch count. Batching cond+uncond into one B=2 forward reads each weight once — the lever for any decode/step that is weight-bandwidth + dispatch bound. gianni-cor/chatterbox.cpp measured −42 % on the chatterbox T3 AR decode (its largest stage) from exactly this.

Who batches CFG, and how (full-tree audit, §214):

backend CFG mechanism batched on GPU? quant on GPU
s3gen CFM (chatterbox) B=2 (build_graph_unet1d_b2) ✓ dequant q*→F16 GPU-resident (dequant_cfm_f16)
chatterbox T3 (§214) B=2 (build_graph_t3_kv_b2) ✓ dequant q*→F16 GPU-resident (ensure_t3_b2_f16_weights)
dia B=2 encoder only ✓ n/a (F16/F32)
zonos two separate KV caches, sequential · n/a
tada two sequential B=1 passes · n/a
cosyvoice3 two sequential B=1 (explicitly chose not to batch) · n/a
f5 two sequential B=1 passes · n/a
voxcpm2 two sequential B=1 passes · n/a
kugelaudio CFG not implemented (TODO) — —

The Metal quant gotcha (and the fix). A B=2 matmul against quantized weights on Metal becomes a mat-vec with ne[1]=1, ne[2]=2 that does not dispatch the PREC_F32 mul_mv_q*_K exact-dot kernel the single-token (ne==1) path hits → it requantizes activations and drifts enough to wreck a greedy sampler (chatterbox T3: token divergence at step 2 → repetition collapse; same class as native batched-quant CFM NaN). F16 weights are fine batched. Fix (only solution in the tree): dequantize the batched-against weights q*→F16 GPU-resident once at first use (host to_float → ggml_fp32_to_fp16_row → upload to the GPU backend), then the B=2 matmuls take the correct mul_mm_f16 path. Both GPU-batched paths — s3gen CFM and chatterbox T3 — use this. CPU batches quantized weights natively (exact dot), no dequant needed.

Chatterbox T3 B=2 status (§214, CRISPASR_CHATTERBOX_T3_CFG_B2=1, default OFF): greedy-token bit-identical to the legacy sequential path on CPU (all quants) and GPU+F16; GPU+quant fixed via the F16 dequant above (ASR-roundtrips verbatim). Also the first working T3-on-GPU path on Metal — it rebuilds the step graph each step, so it sidesteps the §186 Lk-bucket buffer is nil crash that breaks legacy T3-GPU. CPU floor speedup ~34 % ms/tok (75→50) measured on a contended M1 (load 8–12, absolute numbers unreliable).

Per-step build+alloc is negligible — bucketing the B2 graph is a confirmed DUD (CHATTERBOX_BENCH_B2=1, 2026-06-21): CPU 0.41 ms/step = 0.8 % of step time (compute 53 ms/step); GPU+F16 1.55 ms = 0.7 % (compute 232 ms/step). Per §208, a cached graph only helps overhead-bound work — it doesn’t here, and a cached B2 graph would risk the §186 GPU bucket crash. Note GPU compute (232 ms/step) ≫ CPU (53 ms/step): B2 on GPU is not a speed win over the CPU default — its value is enabling T3-on-GPU (sidestepping the bucket crash) + the GPU+quant F16-dequant path. T3 stays CPU-default. See PLAN §214.

Where the gaps are

  1. Layer offload (N_GPU_LAYERS) on encoder-decoder ASR (canary, cohere, kyutai-stt). Their cross-attention layout doesn’t have the blk.<N>.* block-tagged tensors that the layer-split predicate recognises. Encoder-decoder offload is its own design problem — probably want to offload only the LLM/decoder side, but the tensor names (<arch>.dec.<N>.* etc.) need bespoke per-backend predicates.
  2. vibevoice quant K/V (both modes). The attention path uses the ggml_cpy(K_perm, view_into_kv_k) pattern that’s incompatible with quant K/V (see LEARNINGS.md “ggml_cont(ggml_permute(quant_tensor)) doesn’t move data”). Migration recipe is the canary/cohere ggml_flash_attn_ext port — ~50-80 LOC + F16 mask graph input. Layer offload (N_GPU_LAYERS) is independently shipped and works on F16 K/V; the migration only unlocks quant K/V on top.
  3. qwen3-tts code-predictor cache. Talker KV is fully covered via core_attn::kv_self_attn; the secondary code-predictor path doesn’t go through that helper, so its cache stays F16. Lower- priority since the talker dominates per-frame cost.
  4. Linux/CUDA validation of #72 GPU residency. mimo-asr 22 % / gemma4-e2b 2.2x speedups were measured on Apple Silicon Metal. dGPU should be even more favourable; deferred until a CUDA host is available. If a platform regresses, gate via env (CRISPASR_FORCE_CPU_WEIGHTS=1).

  5. Cohere flash_attn_ext: crossover confirmed (PLAN #73 closeout, 2026-06-04). Long-form rerun on FLEURS EN, VPS x86 CPU, 2 threads, cohere-transcribe-q4_k.gguf:

    audio flash-attn (s) cast-on-read (s) delta
    60 s 202.72 179.13 flash +13% slower
    300 s 820.37 1114.96 flash -26% faster

    Crossover is between 60 s and 300 s. Flash wins decisively on unchunked long-form (5+ min) due to O(n) vs O(n²) attention scaling. But with default 30 s auto-chunking, each decode pass is short-form. Recommendation: cast-on-read is now the cohere default (13% faster on the chunked path that all normal users hit). For unchunked long-form, set CRISPASR_COHERE_FLASH=1.

Stacking the four knobs

Each addresses an independent bottleneck:

knob addresses when to use
CRISPASR_KV_QUANT_K=q8_0 / _V=q4_0 KV size in VRAM always reasonable for LLM-decode ASR; quartered V cache on long context
CRISPASR_KV_ON_CPU=1 KV doesn’t fit in VRAM at all very long context with a tight VRAM budget
CRISPASR_N_GPU_LAYERS=N model itself doesn’t fit in VRAM model size > VRAM; spill the last (total-N) layers
CRISPASR_FORCE_CPU_WEIGHTS=1 (proposed) platform regressed on §72 GPU residency not yet wired — none seen on Apple Silicon 
# Maximum-memory-savings combo for a VRAM-tight host
CRISPASR_N_GPU_LAYERS=10 \
  CRISPASR_KV_ON_CPU=1 \
  CRISPASR_KV_QUANT_K=q8_0 \
  CRISPASR_KV_QUANT_V=q4_0 \
  ./build/bin/crispasr --backend voxtral4b -m auto -f long.wav

See docs/cli.md “Memory footprint” for the full env- var reference and the llama.cpp parity comparison table; HISTORY §79 for the implementation write-up.

Runtime Optimization Audit — Re-verification (2026-07-11)

Full re-sweep of the ASR + TTS + codec + pipeline paths against the matrix and the §232 campaign. Verified against current code, not carried from this doc. The matrix and per-model TTS section are stale on several flash/CFG claims — corrections below.

Stale claims corrected

Verified state

True remaining gaps (2026-07-11)

P Area Gap Impact
P0 firered_asr Decoder self-attention has no KV cache — growing vector, O(T²) recompute (firered_asr.cpp:2697) Highest-impact ASR gap
P0 relpos → P0 hifigan → RESOLVED ~2× (im2col batch-1 occupancy) melotts / piper relpos is a MEASURED DUD (2026-07-11): already GEMM’d, ~1.5% of synthesis. hifigan_decode is 73–92%. The prior “99.7% GPU-compute-bound, no cheap lever, do not fund without a new kernel insight” verdict was based on the IDEAL roofline (im2col ≈ 1%) — wrong. Per-node profiling (new CRISPASR_METAL_PROFILE=2) on a quiet M1 (2026-07-11) showed IM2COL = 58%, CONT 22%, MUL_MAT only 12% (the earlier 4.02e6 gpu_us was ~2× load-inflated; true quiet ≈ 1.75e6 = 13× off roofline). Root cause: ggml-metal-ops.cpp sized im2col thread-dim0 from batch N — at inference N=1, threadgroups ran only KH·KW (3–11) threads = ~10–34% of one simdgroup → im2col ~40× below BW. Fix (ggml fork, CRISPASR_METAL_IM2COL_OCC, auto-on N==1): block OW across thread-dim0 → threadgroups fill. Bit-exact (moonshine ASR occ-on==occ-off byte-identical; melotts ASR roundtrip valid), ~2–3× on hifigan (gpu 1.88→0.83 s quiet) + 1.65× moonshine, no regression across paraformer/melotts. A general batch-1 Metal conv win (whisper/all vocoders); CUDA unaffected (parallelizes threads over IC·KH·KW, not N). The existing ggml_conv_2d_direct is a dud (2.25× slower, naive scalar). Teardown assert FIXED (119ec75a). Was “compute floor”; the floor was a Metal occupancy bug — fixed
P0 voxcpm2_tts CPU-only (Metal SIGSEGV), manual per-step host KV re-upload (voxcpm2_tts.cpp:106-111) GPU-locked-out
P0 openvoice2 STFT scalar O(bins·win) DFT → shared radix-2 FFT: 1182 ms → 10.7 ms (110×, ~26% of convert) 2026-07-11. WaveNet already GEMM’d (§176d); ref-enc is 4% one-time. Remaining: hifigan_decode is 67% of convert — next target STFT fixed; vocoder dominant
P1 voxtral/voxtral4b enc, mimo LLM decoder Attention not on flash_attn_ext (O(T²) manual softmax) Enc mem+dispatch; mimo dispatch-bound
P1 dia / speecht5 / parler Manual soft_max + host KV re-uploaded per step Long outputs
P1 firered/glm/funasr/qwen3/omniasr/mimo Beam search is replay (no KV snapshot pool; canary/moonshine/kyutai have one) beam≥2 quadratic
P1 f5/dots/kugelaudio/pocket CFG serial / no persistent graph — un-migrated §232 targets ~halves DiT time
P1 granite/moss Metal decode Per-op dispatch ~100ms/step; ggml-metal has no ICB replay Dominant Metal decode cost
P2 Scalar CPU hotpaths RNN-T LSTM pred+joint; granite cpu_linear+depthwise; paraformer CIF; rvq encode; istft IRFFT; titanet mel front-end; diarize apply_xcorr Per-token/frame scalar loops
P2 CLOSED parakeet Batched TDT decode validated on Kaggle CPU (#81, 2026-07-18): 0.73–0.80× = SLOWER, byte-identical → keep default OFF, do NOT flip Predicted win, measured a loss
OPEN parakeet CPU vs onnx-asr crispasr parakeet-tdt CPU 4.0×/3.0× vs onnx-asr int8 8.6×/5.8× ⇒ ~2.1× slower on CPU (the real #81 residual). Needs a CPU BLAS/kernel lever — batched decode is ruled out GPU is fine (P100 36.9×/50.9×)
P2 align_wav2vec2_ctc Reloads the 300MB–1GB model every call (crispasr_aligner.cpp:315) — missing the §176e ctx-cache Concrete single-file win
P2 paraformer / voxcpm2 CPU-only, no GPU backend (paraformer leaks 256MB buffer) GPU offload available
P3 Threading Hardcoded default 4 threads in ~90 sites; only whisper-core caps to min(4, hw) Idle cores on big hosts
P3 Misc pyannote per-slice not once-over-audio (#107); RNNoise recreates state+resamplers/call; glm mel padded to 3000 always Localized

Highest-ceiling paths forward

dots.tts baseline and optimization queue (#319, 2026-07-28)

The first local low-memory benchmark used an Apple M1, Metal, a mixed Q4_K core (about 2.2 GiB; DiT tensors intentionally left in F16) and a Q8_0 vocoder (about 330 MiB). With an eight-patch cap, synthesis produced valid, recognizable audio and measured:

ODE steps Total Flow matching Vocoder decode
16 24.2 s 13.7 s 10.4 s
8 16.2 s 5.8 s 10.5 s

Model loading is excluded. The patch cap makes this a bounded smoke benchmark, not a real-time claim for full utterances. Eight steps is about 1.49x faster in this run, but the vocoder becomes the floor once DiT steps are reduced. The current Metal path already uses the persistent fused DiT graph by default; remaining high-value work is:

  1. Make vocoder decode reuse a fixed-shape graph and benchmark the batch-1 convolution occupancy fix against the existing HiFi-GAN measurements.
  2. Batch CFG conditional/unconditional work with a device-resident dequantized weight copy, preserving the quantization safety rule established by chatterbox.
  3. Add length-bucketed graph reuse for autoregressive patch counts, then measure the existing approximate CFG-interval mode separately from exact decoding.

Every dots.tts speed change must retain the audio roundtrip gate (TTS WAV → ASR) and compare decoded output, not only process exit status. Do not quantize the DiT tensors to F16 or increase the memory footprint as an optimization.

  1. Lk-bucketed decode-step graph caching generalized to the 30+ decoders that rebuild per step — templates: qwen3-tts (5 buckets), granite §210 gallocr, mimo step_t1_gf. Cache the decode-step graph, not the encoder.
  2. ggml-metal ICB replay — the Apple-side equivalent of CUDA-graph capture; decode is per-op-dispatch bound.
  3. BLAS/ggml the scalar hotpaths (melotts/piper relpos — measured DUD, already GEMM’d + ~1.5%; openvoice2 WaveNet — GEMM’d §176d; openvoice2 STFT DFT→FFT DONE 2026-07-11; rvq, titanet mel, RNN-T LSTM/joint still open) and extend §232’s CFG-batch + device-KV playbook to f5/dots/kugelaudio/pocket/dia/ speecht5/parler. The real cross-cutting TTS cost is HiFi-GAN decode (openvoice2 67%, melotts 73–92%) — GPU-resident but cold-JIT-confounded on Metal.

Do not re-enable encoder-graph caching (#235 UAF + measured dud), and do not CPU-batch decode that feeds a GPU pipeline (item 24).

Kaggle GPU — full backend sweep — 2026-06-20

Platform: Kaggle GPU worker (CUDA), tools/kaggle-benchmark-all-backends.py (kernel chr1s4/crispasr-full-backend-sweep). Commit: latest main. First full-coverage sweep — every ASR + TTS backend plus the two text-MT backends, 59 entries — with per-backend results streamed live to an HF dataset (cstr/crispasr-kaggle-progress/full-backend-sweep/latest/, resumable). Audio: jfk.wav (11 s). TTS phrase: “The quick brown fox…”.

Headline: ASR 33/35 pass · TTS 14/22 pass · MT 2/2 pass. (Per-backend JSON + summary.json in the dataset; model-size column is unreliable this run — the benchmark mis-detected several as ~55 MB — so it is omitted below.)

ASR — 33/35 pass (RTx, WER on JFK)

Backend RTx WER Backend RTx WER
SenseVoice Small 17.9x 0.0% Voxtral Mini 3B 3.3x 0.0%
Canary 1B 8.3x 0.0% Granite Speech 4.1 2B 3.1x 0.0%
Data2Vec Base 7.2x 4.5% Granite Speech 1B 3.0x 0.0%
FastConformer CTC 7.2x 0.0% Voxtral 4B Realtime 2.2x 0.0%
Moonshine Tiny 7.5x 9.1% Kyutai STT 1B 2.3x 0.0%
Cohere Transcribe 7.0x 0.0% VibeVoice ASR 1.9x 4.5%
HuBERT Large 6.8x 0.0% Nemotron Streaming 1.6x 9.1%
Wav2Vec2 XLSR-EN 6.9x 0.0% OmniASR LLM 300M 1.4x 4.5%
OmniASR CTC 1B 6.2x 22.7% Fun-ASR MLT Nano 1.1x 0.0%
FunASR Nano 6.1x 0.0% Gemma-4-E2B 0.9x 9.1%
Parakeet TDT 0.6B 5.9x 0.0% Granite 4.1 NAR 0.8x 0.0%
Whisper (base) 5.5x 0.0% Granite 4.1 2B+ 0.7x 0.0%
Qwen3 ASR 0.6B 5.3x 0.0% MOSS Audio 0.6x 0.0%
GLM ASR Nano 5.1x 0.0% FireRed ASR2 AED 0.6x 0.0%
Paraformer-zh 4.5x 0.0% MiMo-ASR (CPU #115) 0.5x 0.0%
Mega-ASR 1.7B 3.9x 0.0% mini-omni2 0.8x 0.0%
Granite Speech 4.1 2B 3.1x 0.0% moonshine-streaming 2.8x 0.0%

ASR failures (2): lfm2-audio — CRASH mid-run (~9.8 s) — FIXED §206 (embed device-ptr deref + a ggml_backend_sched weight-less-first-op cross-backend copy bug in the backbone; now computes directly on ctx->backend via gallocr, GPU transcribes verbatim); vibevoice-1.5b — ran (~17 s) but produced EMPTY transcript. Both are newly-covered backends.

TTS — 14/22 first pass → 16/22 after the voicefix re-test

First pass marked 8 TTS “fails”, but an audit found most were missing-args, not bugs: f5-tts/chatterbox/cosyvoice3/vibevoice-tts are voice-cloning models that need a reference voice, and vibevoice-1.5b had been mis-listed as ASR. The benchmark now passes the right voice per backend (--voice <ref.wav> --i-have-rights, fastpitch --voice 0) and vibevoice-1.5b moved to TTS. A fixed-subset re-test (run tag voicefix-retest) gives:

✓ pass (first pass) ✓ recovered by voicefix ✗ genuine fail (with correct args)
piper, kokoro, pocket-tts, bark, csm, parler-tts, dia, qwen3-tts-customvoice, indextts, zonos, melotts, outetts, tada, voxcpm2-tts vibevoice-1.5b (was run as ASR), vibevoice-tts (was no-voice) speecht5, fastpitch, orpheus, chatterbox, cosyvoice3, kugelaudio · f5-tts = runs-but-timeout (pending ≥240 s confirm)

MT — 2/2 pass

m2m100 (3.7 s), madlad (12.5 s) — en→de translation produced output.

Notes

Kaggle T4 GPU — 2026-06-03

Platform: Tesla T4 (16 GB VRAM), 4 CPU threads, CUDA. Commit: latest main (post b102060a). Run via tools/kaggle-benchmark-all-backends.py. 30 backends tested, 30 pass. First run to cover all backends added in the June 2-3 script completeness audit: granite-4.1-plus, granite-4.1-nar, fun-asr-mlt-nano, voxtral4b.

Speed ranking (11.0 s JFK, Q4_K unless noted, greedy)

Rank Backend RTx WER Architecture
1 SenseVoice Small 17.3x 0.0% Encoder (multitask)
2 FastConformer CTC Large 7.7x 0.0% Encoder-CTC
3 Data2Vec Base 6.9x 4.5% Encoder-CTC
4 Canary 1B 6.8x 0.0% Encoder-AED
5 Moonshine Tiny 6.7x 9.1% Encoder-Decoder
6 Wav2Vec2 XLSR-EN 6.4x 0.0% Encoder-CTC
7 HuBERT Large 6.2x 0.0% Encoder-CTC
8 OmniASR CTC 1B v2 6.2x 4.5% Encoder-CTC
9 Cohere Transcribe 6.1x 0.0% Encoder-AED
10 Fun-ASR Nano 2512 5.5x 0.0% Encoder-LLM (enc GPU, LLM CPU)
11 Parakeet TDT 0.6B 5.3x 0.0% Encoder-TDT
12 Paraformer-zh NAR 4.3x 0.0% Encoder (NAR)
13 Qwen3 ASR 0.6B 4.0x 0.0% Encoder-LLM
14 GLM ASR Nano 3.9x 0.0% Encoder-LLM
15 Mega-ASR 1.7B 2.6x 0.0% Encoder-LLM (qwen3)
16 Moonshine Streaming Tiny 2.5x 0.0% Encoder-Decoder
17 Granite Speech 1B 2.5x 0.0% Encoder-LLM
18 Granite Speech 4.1 2B 2.4x 0.0% Encoder-LLM
19 Voxtral Mini 3B 2.3x 0.0% Encoder-LLM
20 OmniASR LLM 300M 1.4x 4.5% Encoder-LLM
21 Kyutai STT 1B 1.3x 0.0% Encoder-AED
22 VibeVoice ASR 1.2x 4.5% Encoder-LLM
23 Voxtral 4B Realtime 0.9x 0.0% Encoder-LLM (streaming)
24 Granite Speech 4.1 2B+ 0.8x 0.0% Encoder-LLM
25 Gemma-4-E2B 2.3B 0.8x 9.1% Encoder-LLM
26 Granite Speech 4.1 NAR 0.6x 0.0% Encoder-CTC (non-AR)
27 FireRed ASR2 AED 0.5x 0.0% Encoder-AED
28 Whisper (base) 0.4x 0.0% Encoder-Decoder
29 MiMo-ASR 0.2x 0.0% Encoder-LLM (CPU-forced, #115)
30 Fun-ASR MLT Nano 2512 0.1x 0.0% Encoder-LLM (F16 on CPU)

Notes

First comprehensive TTS benchmark across 11 backends. Phrase: “The quick brown fox jumps over the lazy dog.” Each model auto-downloaded, synthesised, output WAV checked for >1 KB, then cleaned up.

Rank Backend Status Wall (s) WAV size Notes
1 Piper LessAC Medium PASS 3.7 103 KB Fastest TTS, 22 kHz VITS
2 SpeechT5 TTS PASS 4.6 56 KB 16 kHz, deterministic
3 Bark Small PASS 16.9 245 KB 3-stage GPT-2, 24 kHz
4 Kokoro 82M PASS 17.5 152 KB Needs espeak-ng (installed)
5 Pocket TTS 100M PASS 72.1 181 KB Continuous-latent AR, 24 kHz
6 CSM 1B PASS 213.4 165 KB Llama-3.2 + Mimi, 24 kHz
7 Orpheus 3B-FT PASS 269.7 205 KB Llama-3.2 + SNAC, 24 kHz
— FastPitch 60M TIMEOUT >30 0 Timeout too short (30s)
— F5-TTS v1 Base FAIL 4.1 0 Needs --voice <ref.wav>
— Parler TTS Mini v1.1 TIMEOUT >180 0 Too slow on CPU path
— Dia 1.6B TIMEOUT >240 0 Too slow on CPU path

7/11 pass. FastPitch needs a longer timeout (model download is slow, not inference). F5-TTS requires a reference audio for voice cloning. Parler-TTS and Dia are AR LLM-based and need GPU acceleration or longer timeouts.

Kaggle P100 GPU — 2026-05-31

Platform: Tesla P100-PCIE (16 GB VRAM), 4 CPU threads, CUDA sm_60 (auto-detected — the kernel pins CMAKE_CUDA_ARCHITECTURES from nvidia-smi compute_cap, so the build is correct whether the box is a T4/P100/A100/L4). Commit: 7bc3ef5b. Run via tools/kaggle-benchmark-all-backends.py (now on the shared tools/kaggle/kaggle_harness.py). 27 backends tested, 26 pass.

First run to cover all six newly-registered ASR backends (marked †): sensevoice, paraformer, mega-asr, granite-4.1, funasr, mimo-asr.

Speed ranking (11.0 s JFK, Q4_K unless noted, greedy)

Rank Backend RTx Time WER Architecture
1 SenseVoice Small † 19.8x 0.6s 0.0% Encoder (multitask)
2 FastConformer CTC 8.9x 1.2s 0.0% Encoder-CTC
3 Moonshine Tiny 8.6x 1.3s 9.1% Encoder-Decoder
4 Canary 1B 8.0x 1.4s 9.1% Encoder-AED
5 Data2Vec Base 7.5x 1.5s 4.5% Encoder-CTC
6 OmniASR CTC 1B 7.0x 1.6s 9.1% Encoder-CTC
7 HuBERT Large 6.9x 1.6s 0.0% Encoder-CTC
8 Wav2Vec2 XLSR-EN 6.9x 1.6s 0.0% Encoder-CTC
9 Cohere Transcribe 6.6x 1.7s 0.0% Encoder-AED
10 Parakeet TDT 0.6B 6.0x 1.8s 0.0% Encoder-TDT
11 Whisper base 5.6x 2.0s 0.0% Encoder-Decoder
12 Paraformer-zh NAR † 5.0x 2.2s 0.0% Encoder (NAR)
13 GLM ASR Nano 4.7x 2.4s 0.0% Encoder-LLM
14 Qwen3 ASR 0.6B 4.4x 2.5s 0.0% Encoder-LLM
15 Moonshine Streaming Tiny 2.9x 3.8s 0.0% Encoder-Decoder
16 Mega-ASR 1.7B † 2.9x 3.9s 0.0% Encoder-LLM (qwen3)
17 Granite Speech 1B 2.8x 3.9s 0.0% Encoder-LLM
18 Granite Speech 4.1 2B † 2.7x 4.1s 0.0% Encoder-LLM
19 Voxtral Mini 3B 2.5x 4.4s 0.0% Encoder-LLM
20 OmniASR LLM 300M 1.6x 7.0s 4.5% Encoder-LLM
21 Kyutai STT 1B 1.5x 7.5s 0.0% Encoder-AED
22 VibeVoice ASR 1.3x 8.6s 4.5% Encoder-LLM
23 Voxtral 4B Realtime 0.9x 11.9s 0.0% Encoder-LLM
24 Gemma-4-E2B 2.3B 0.8x 13.6s 0.0% Encoder-LLM
25 FireRed ASR2 AED 0.6x 19.2s 0.0% Encoder-AED
26 MiMo-ASR † 0.3x 38.0s 0.0% Encoder-LLM (CPU-forced, #115)
27 FunASR Nano † ~1.0x ~10.6s 0.0% Encoder-LLM (enc GPU, LLM CPU)

Notes

Kaggle T4 GPU — 2026-04-26

Platform: 2x Tesla T4 (15 GB VRAM each), 4 CPU threads, CUDA. Commit: b9fd8eb. All 19 backends pass.

By architecture

Encoder-CTC (non-autoregressive, single forward pass)

Backend Params Model MB WER RTx Time Notes
FastConformer CTC Large 120M 83 0.0% 9.6x 1.1s 18 FC layers
OmniASR CTC 1B v2 975M 551 4.5% 7.4x 1.5s w2v-BERT enc, 276ms GPU
Data2Vec Base 95M 78 0.0% 5.3x 2.1s 12 layers, pos_conv 735ms
Wav2Vec2 XLSR-EN 300M 212 0.0% 3.6x 3.1s 24 layers, pos_conv 1.6s
HuBERT Large 300M 212 0.0% 3.6x 3.1s Same runtime as wav2vec2

Encoder-TDT (non-autoregressive, transducer)

Backend Params Model MB WER RTx Time Notes
Parakeet TDT 0.6B 600M 466 0.0% 5.6x 2.0s 24 FC layers + joint net

Encoder-Decoder / AED (autoregressive, attention-based)

Backend Params Model MB WER RTx Time Notes
Whisper (base) 74M 141 0.0% 9.3x 1.2s Full GPU (upstream)
Moonshine Tiny 27M 20 9.1% 6.7x 1.6s CPU-only, tiny
Canary 1B 1B 672 0.0% 6.2x 1.8s GPU enc+dec, 32+8 layers
Cohere Transcribe 2B 1440 0.0% 5.2x 2.1s GPU enc, AED dec
Kyutai STT 1B 1B 636 4.5% 1.4x 7.7s 24-layer Mimi decoder
FireRed ASR2 AED 900M 918 0.0% 0.6x 19.0s CPU Q4_K SIMD dec (60ms/step greedy; beam batched through Q4_K mul_mat, 16.8s→3.1s at beam=3, §224); enc.* split-loaded to GPU by DEFAULT with use_gpu (transcript-identical, enc 2.2-2.3x on Metal/Vulkan/CUDA-P100, §224; CRISPASR_FIRERED_ENC_CPU=1 opts out)

Encoder-LLM (autoregressive, language model decoder)

Backend Params Model MB WER RTx Time Notes
Qwen3 ASR 0.6B 780M 515 0.0% 4.7x 2.3s 0.6B LLM
GLM ASR Nano 1.3B 1262 0.0% 4.6x 2.4s ~1B LLM
Voxtral Mini 3B 3B 2530 0.0% 2.4x 4.7s Mistral 3B LLM
OmniASR LLM 300M 1.6B 1018 4.5% 1.7x 6.4s LLaMA 1.3B dec
Granite Speech 1B 2.9B 2805 0.0% 1.7x 6.4s Granite LLM
VibeVoice ASR 4.5B 4589 4.5% 1.2x 8.8s ~4B LLM, JSON output
Voxtral 4B Realtime 4B 2407 0.0% 0.9x 12.8s Causal streaming arch (PLAN #7 streaming API; 1.6s first-text-token)

Speed ranking

Rank Backend RTx Time Architecture
1 FastConformer CTC 9.6x 1.1s Encoder-CTC
2 Whisper base 9.3x 1.2s Encoder-Decoder
3 OmniASR CTC 1B 7.4x 1.5s Encoder-CTC
4 Moonshine Tiny 6.7x 1.6s Encoder-Decoder
5 Canary 1B 6.2x 1.8s Encoder-AED
6 Parakeet TDT 0.6B 5.6x 2.0s Encoder-TDT
7 Data2Vec Base 5.3x 2.1s Encoder-CTC
8 Cohere Transcribe 5.2x 2.1s Encoder-AED
9 Qwen3 ASR 0.6B 4.7x 2.3s Encoder-LLM
10 GLM ASR Nano 4.6x 2.4s Encoder-LLM
11 Wav2Vec2 XLSR-EN 3.6x 3.1s Encoder-CTC
12 HuBERT Large 3.6x 3.1s Encoder-CTC
13 Voxtral Mini 3B 2.4x 4.7s Encoder-LLM
14 OmniASR LLM 300M 1.7x 6.4s Encoder-LLM
15 Granite Speech 1B 1.7x 6.4s Encoder-LLM
16 Kyutai STT 1B 1.4x 7.7s Encoder-AED
17 VibeVoice ASR 1.2x 8.8s Encoder-LLM
18 Voxtral 4B Realtime 0.9x 12.8s Encoder-LLM
19 FireRed ASR2 AED 0.6x 19.0s Encoder-AED

CPU-only VPS — 2026-04-24

Platform: x86_64, 4 threads, 7.6 GB RAM, AVX2, no GPU.

Backend RTx (CPU) Time (CPU) RTx (T4) Speedup
FastConformer CTC 9.4x 1.2s 9.6x 1.1x
Moonshine Tiny 16.8x 0.7s 6.7x 0.4x*
Parakeet TDT 0.6B 2.9x 3.8s 5.6x 1.9x
Canary 1B 2.7x 4.0s 6.2x 2.2x
Data2Vec Base 2.1x 5.2s 5.3x 2.5x
Qwen3 ASR 0.6B 1.7x 6.5s 4.7x 2.8x
Wav2Vec2 XLSR-EN 1.1x 9.9s 3.6x 3.2x
Cohere Transcribe 1.4x 7.7s 5.2x 3.7x
FireRed ASR2 AED 0.1x 123s 0.6x 6.5x

*Moonshine runs CPU-only on both (tiny model, no GPU benefit).

GPU acceleration is strongest for encoder-heavy models (2-6x). Decoder-bound models benefit less (FireRed decoder still runs on CPU even with GPU).

Per-phase breakdowns

wav2vec2 family (Kaggle T4)

Model CNN Pos conv Encoder Total
wav2vec2-large (24L) 215ms 1588ms 127ms 1941ms
hubert-large (24L) 227ms 1595ms 128ms 1960ms
data2vec-base (12L) 221ms 735ms 57ms 1023ms

Bottleneck: pos_conv (grouped conv1d on CPU) = 50-80% of total time. Encoder graph on GPU is only 57-128ms.

FireRed AED decoder (Kaggle T4)

Phase Time Notes
Fbank extraction ~50ms CPU
Conv2d subsampling ~100ms CPU
Hybrid encoder (16L) ~17s GPU matmuls + CPU attention, slow due to CPU weight copies
K/V precompute 433ms GPU (scheduler auto-copies)
Decoder (28 steps) 1695ms CPU Q4_K SIMD, 60.5ms/step
Total 19.0s Encoder dominates

OmniASR (Kaggle T4)

Model Encoder Prefill Decode Total RTx
CTC 1B v2 244ms — — 277ms 39.8x (encoder only)
LLM 300M v2 97ms 803ms 4028ms (103 steps) 5021ms 2.2x

Key observations

  1. CTC models dominate on speed. No decoder loop = one forward pass.
  2. Small LLM decoders (0.6-1B) are competitive — Qwen3 and GLM hit 4.5x+ realtime with 0% WER, close to encoder-only models.
  3. Large LLMs (3-4.5B) are 1-2x realtime on T4. Usable but not fast.
  4. Most WER=0% on jfk.wav. The 4.5% models have minor formatting differences, not actual transcription errors. Moonshine Tiny (9.1%) has a real word error.
  5. wav2vec2 pos_conv was the bottleneck — now 4.9x faster with ggml grouped conv (im2col + mul_mat SIMD). Was 1.6s (80% of runtime), now 324ms (~3.5%).
  6. FireRed encoder is slow because CPU weights auto-copy to GPU per-layer. Pre-loading encoder weights to GPU would save ~15s.

Optimization history

wav2vec2 grouped conv — 2026-04-27

Path pos_conv Notes
Manual C++ (OMP) 1588ms 4-thread OMP, plain float loops
ggml im2col + mul_mat 324ms 4.9x faster, SIMD kernels

The grouped positional conv (C=1024, K=128, G=16) is decomposed into G=16 independent ggml_pad_ext + ggml_im2col + ggml_mul_mat calls. The mul_mat output [cpg, T] is transposed to channel-first before reassembly. Applies to wav2vec2, data2vec, and hubert.

FireRed decoder — 2026-04-26

Path ms/step 28 tokens Why
Manual C++ F32 (original) 4400 123s No SIMD, no parallelism
+ OpenMP matmuls 2320 58s 2.1x from OMP
+ ggml Q4_K CPU native 70 2.0s 9.3x from fused SIMD kernel
ggml_vecmat on CUDA 2600 timeout CUDA launch overhead kills it
F32 dequant + cpu_matmul 590 16.5s No SIMD, OMP disabled on Kaggle
ggml_vecmat CPU (final) 60 1.7s Weights on CPU, native Q4_K

wav2vec2 CNN — 2026-04-24

Change CNN Total Speedup
Baseline (manual C++) 95.2s 108.4s 1.0x
ggml F32 im2col 2.4s 15.5s 7.0x
+ OpenMP pos_conv 2.3s 9.9s 10.9x

voxtral4b streaming — 2026-05 (PLAN #7 phases 1+1.5+2+3+4)

Native incremental encoder + streaming-prompt decode + speculative prefill + combined-chunk flush + live captions + decoder thread. M1 Q4_K JFK 11 s baseline, all variants bit-exact-batch:

Stage / phase Metric Before After Δ
Phase 1 (initial) first-text-token n/a 2674ms —
+ 240ms chunks (phase 2) feed total 23s 9.1s 2.5× faster
+ default-unification fix encoder drain 2064ms 1016ms -1.0s
+ fused QKV (Q4_K) per-decode-step 56ms 50.4ms -10%
+ combined-chunk flush (phase 3) encoder drain 990ms 307ms -683ms
+ speculative prefill (phase 3) first-text-token 921ms 650ms -271ms

Final: first-text-token 2674ms → 650ms (4.1× faster); sequential live decode (phase 3); decoder thread for non-blocking feed (phase 4, gated on CRISPASR_VOXTRAL4B_STREAM_DECODER_THREAD=1).

The remaining ~410ms gap to the model’s ≤240ms target is the architectural floor: 8 streaming-pad warmup steps × 50.4ms + LLM prefill = 655ms minimum on M1 Q4_K. Cross that floor only via a faster Q4_K Metal kernel or a model with a different prompt convention (no streaming-pad warmup).

Cross-backend portability of the fused-QKV Q4_K pattern:

Reproduce

# Per-backend timing
CRISPASR_VERBOSE=1 crispasr --backend firered-asr -m auto -f jfk.wav -v -bs 1

# wav2vec2 phase breakdown
WAV2VEC2_VERBOSE=1 crispasr --backend wav2vec2 -m auto -f jfk.wav -v

# Full Kaggle benchmark (all 19 backends)
# See tools/kaggle-benchmark-all-backends.py or gist:
# https://gist.github.com/CrispStrobe/c15f7a64878d93907a8a4a51b193b806

issue #81 / FastConformer family — CURRENT metrics (2026-07-12)

State on main: conv-pw Q8_0 fix (the quantizer skipped the 3D-stored conv pointwise weights, leaving them F16 in every quantized GGUF — ~35% of CPU encoder time) + fused QKV + strided flash-attn inputs; all 80 affected HF GGUFs re-quantized (transcript-verified); CUDA default = manual attention (flash_attn_ext rejects the per-head rel-pos mask there and bounced all 24 layers to CPU); Metal/CPU keep flash. Rounds 1-3, the May 2026 cross-comparisons, and all pre-fix numbers are superseded (git history).

M1 CPU-only (4 threads — optimal on M1; quiet box; JFK 11 s; medians; transcripts verified identical):

model x-RT
parakeet-ctc-0.6b Q8_0, re-quantized GGUF 18.4×
parakeet-ctc-0.6b Q8_0, old GGUF + runtime repack 15.0×
parakeet-ctc-0.6b Q4_K 10.3×
parakeet-tdt-0.6b-v3 Q4_K 10.7×
stt-en-fastconformer-ctc-large Q4_K 33×
onnx-asr parakeet-ctc int8 CPU-EP (same box, 2026-05 reference) 15.2×

M1 Metal (shared box, load 4-8 → treat ±10%): parakeet-ctc-0.6b Q8_0 16.2×; parakeet-tdt-0.6b-v3 Q4_K 13.8× (decode-bound — no encoder gain, needs a per-stage look).

P100 CUDA (transcripts verified):

measurement x-RT
CrispASR parakeet-ctc Q4_K, manual attn, 134 s varied, load-excl 158.6×
CrispASR parakeet-tdt Q4_K, backend joint projection, 134 s varied, load-excl 116.1×
same TDT, scalar CPU joint projection 52.4×
CrispASR parakeet-ctc Q8_0, manual attn, warm in-process, jfk×5 55 s 153× (11 s: 116×)
same, flash-with-CPU-fallback (old default) 48× (11 s: 61×)
CrispASR parakeet-ctc Q8_0, manual attn, 134 s varied, load-excl (honest) 137× (tdt 49.5×)
onnx-asr parakeet-ctc CUDA fp32, 134 s varied speech, in-process 214× (tdt 121×)
onnx-asr parakeet-ctc CPU int8, 134 s varied 5.8× (tdt 5.7×)

Q4 round-2 rejected arms (P100, 2026-09-07; Kaggle chr1str/crispasr-issue-81-q4-round-2, encoder matrix v1 at b82b7baf, TDT matrix v3 at c0527382). Device-side selection preserved exact transcripts but improved the experimental baseline by only 0.17% on the 134 s clip. Speculative joint batches lost 2.5–8.1%. The selection graph also made its argmax outputs part of the graph when the runtime switch was off, so that arm was removed rather than retained as a dormant option.

arm 134 s median x-RT vs same-run baseline
selection-graph baseline 1.2062 s 111.15× —
device argmax 1.2042 s 111.34× +0.17%
batch 4 1.2369 s 108.39× −2.48%
batch 8 1.2700 s 105.57× −5.02%
batch 4 + device argmax 1.2284 s 109.14× −1.81%
batch 8 + device argmax 1.3044 s 102.79× −7.52%

The preceding CTC matrix rejected the other proposed Q4 levers on the same 134 s varied clip. Direct standard/depthwise convolution reduced throughput, and enabling the fork’s per-head flash-attention path on sm_60 was more than 2× slower. Keeping selected FFN tensors at Q8 or F16 either failed transcript parity or lost speed.

CTC arm x-RT result
baseline 163.09× exact/stable
direct initial conv 157.78× −3.3%
direct depthwise conv 120.94× −25.8%
both direct convs 117.17× −28.2%
per-head flash attention 69.87× −57.2%
flash + both direct convs 60.11× −63.1%
selected FFN Q8 163.06× transcript changed
selected FFN F16 156.95× transcript changed and −3.8%

ggml v0.23 fork consolidation (P100 sm_60, 2026-09-07; Kaggle chr1s4/crispasr-ggml-v0-23-q4-a-b v3). The old fork pin and merged v0.23 runtime used separate source-compatible CrispASR checkouts, the same Q4 Parakeet TDT model and audio, and an account-matched warm ccache dataset. Transcripts were stable and byte-identical. v0.23 was neutral-to-faster:

clip old fork ggml v0.23 change
JFK, 11.00 s 85.44× 87.44× +2.34%
varied speech, 134.07 s 115.38× 115.99× +0.53%

The benchmark exercised ggml runtime commit 069a517d; the merged default commit 2dd13edd has the same runtime source tree (later commits only add patch guards, CI, and merge ancestry). Kaggle validation passed and the chr1s4/crispasr-ccache seed was refreshed to version 13.

CUDA rows resolved (2026-07-12, kernel issue81-onnx-bench v16, real 134 s varied LibriSpeech, load-excluded, 301-word proof-of-work, same-run onnx head-to-head): the honest-methodology re-run with the manual-attn default lands 137× ctc / 49.5× tdt — a 5.4× jump from the pre-manual-attn 25.3× ctc / 34.7× tdt (superseded). That cuts the warm gap to onnx-asr CUDA to ~1.56× (ctc) / ~2.4× (tdt), confirming the handover’s “~1.4×” claim on independent honest methodology. (The 153× in-process row is repeat-audio jfk×5, ~6% cache-friendly and warm — kept as the best-case in-process figure; the 137× row is the fair varied-audio comparison against onnx.)

Q4 follow-up (2026-09-07, kernel chr1str/crispasr-issue-81-q4-p100-profile v4, commit 39c2a7b2, P100 sm_60): profiling found that TDT spent about 1.40 s of its 2.56 s wall time in the encoder-to-joint projection, which still ran as scalar CPU code before the GPU decoder loop. One backend matmul reduced that projection to 4.0 ms and the total to 1.155 s: 52.4× → 116.1×, 2.21× faster overall and about 4% below the historical 121× onnx-asr TDT CUDA result. Every repeat produced the same 301-word transcript across both arms. CTC remained encoder-bound at 158.6×. Stable graph buckets of 25/50/100 mel frames reached 156.5×/158.2×/154.9× on the varied set, all below the unbucketed control, so CRISPASR_FC_BUCKET remains opt-in.

VPS 4-core x86 re-bench (2026-07-12, DONE): parakeet-ctc-0.6b q8_0, jfk 11 s, same-box A/B (load ~2.5) — new defaults 5.43 s (2.0× RT) vs legacy (CRISPASR_FC_PW_Q8=0 CRISPASR_FC_FUSED_QKV=0 CRISPASR_FC_ATTN_CONT=1) 6.30 s (1.7×); q4_k new 6.24 s. FC-perf gain on x86 is ~14% (vs M1’s 35%) — OpenBLAS already handles the F16 conv-pw GEMM well, so the Q8 repack recovers less than on ggml’s ARM CPU F16 path. Still beats onnx-CPU int8 (5.8× above). The re-quantized GGUF would not change this load-excluded inference RTF — it only saves one-time load (skips the runtime repack).

Gates: CRISPASR_FC_PW_Q8 (auto-on for quantized models) · CRISPASR_FC_FUSED_QKV (on) · CRISPASR_FC_ATTN_CONT=1 legacy conts · CRISPASR_FC_GPU_MANUAL_ATTN (auto = CUDA only) · CRISPASR_FC_BUCKET (opt-in bucketed persistent graph, output-equivalent) · CRISPASR_FC_PROFILE=1 per-node profiler · CRISPASR_FC_MAX_LAYERS=N bisection. Kernels: tools/kaggle/fc-unified-graph-ab (CUDA A/B), tools/kaggle/fc-pw-requant (fleet requant), tools/kaggle/issue81-onnx-bench (onnx head-to-head).

Long-audio coverage — 2026-07-04 final state (issue #89 closed out)

Platform: Apple M1 16 GB, Metal. Commits ca1e871b (VAD slice cap 12 s + per-slice single-pass) + 3a8141e3 (gap-fill second pass). Scoring: char-bigram recall/precision vs a whisper-large-v3-turbo reference with hiragana-reading normalization (pykakasi) — the coverage metric that erases kanji/kana spelling variants; latin stripped (the JA model renders English in katakana, which a latin reference can’t credit).

clip old default (streamed) shipped default precision notes
yt_60s (reporter’s clip) 64.2 % 97.2 % 90.5 % auto-VAD + 12 s cap + gap-fill
yt_120s 61.3 % 96.9 % 91.2 %  
first300 (reporter’s 300 s repro) — (June main-default.srt: 0.7 %) 95.9 % 93.4 %  
session ABI (yt_60s, bindings/server path) ~58 % 94.8 % 89.2 % energy slices instead of silero

Ceiling calibration: an independent SenseVoice-small run scores 97.2/96.8 % recall against the same reference at 78 % precision — the shipped pipeline sits AT the inter-model agreement ceiling; the residual is hearing variants, not missing content. Blueprint comparison on the same audio: NeMo 2.7.3 plain transcribe 11 %, local-attn [128,128] 46 % (both char-identical to our port — bit-faithfulness check), buffered BatchedFrameASRTDT 15-51 %. Gap-fill cost: one extra short encode per recovered gap (~1.3-2× wall on gap-heavy clips); CRISPASR_GAP_FILL=0 restores single-pass-per-slice. Reproduce: tools/asr_coverage_score.py <whisper-ref> <hyp> --strip-latin --reading (scorer) and tools/nemo_parakeet_blueprint.py (NeMo modes).

Long-form single-pass — qwen3-asr / glm-asr (#218, 2026-07-10)

Platforms: Apple M1 16 GB Metal + Kaggle T4 CUDA validation (tools/kaggle/qwen3-family-rebake/). Canonical clip: the reporter’s 145 s t32-145s.wav (noisy dialogue, quiet lead). Full detail in PLAN “#218 …” sections; the performance-relevant facts:

Encoder quantization floor (quality, not speed)

Sub-8-bit audio.* towers flip greedy LLM decode into loops/empty output on long audio — a behavioral step function, not gradual WER loss. crispasr-quantize now floors qwen3-asr audio.* at Q8_0 (opt-out CRISPASR_QWEN3ASR_QUANT_AUDIO=1):

model q4_k size encoder cos_mean vs F16 un-chunked 145 s
qwen3-asr-0.6b, old tower Q4 540 MB 0.9716 (cos_min 0.75 @1885 fr) loops / “language none”
qwen3-asr-0.6b, Q8 tower 631 MB 0.9997 (cos_min 0.992) clean+complete, raw (loop-fix off)
qwen3-asr-1.7b, old tower Q4 (#240) 1334 MB 0.9913 (cos_min 0.963 @jfk 143 fr!) empty on Metal
qwen3-asr-1.7b, Q8 tower (#240) 1490 MB 0.9998 (cos_min 0.9989) fixed
1.7b-ja-anime, Q8 tower 1421 MB (was 1334) — 1024 chars, max run 1
mega-asr, Q8 tower not shipped — still loops at bf16 too — LoRA-inherent, chunked-only

imatrix caveat: quantizing the tied LM head re-introduces long-form loops even with a Q8 tower, and a long-form-recalibrated imatrix still drifts (tested + rejected) — plain -q4_k/-q8_0 are the long-form recommendation; imatrix variants are for short clips.

Memory: full vs windowed encoder attention (qwen3-asr)

Default full self-attention is O(T²) in the audio tower: 145 s (1885 frames) fits easily on 16 GB; beyond ~10 min un-chunked it OOMs. KV is sized max(4096, prompt+max_new+16) + grow-on-demand (the old fixed 4096 capped un-chunked audio at ~5 min regardless of encoder).

CRISP_AUDIO_WINDOWED_ATTN=1 (opt-in, 46e08bc4) runs the FA2/cu_seqlens block-diagonal semantics natively: full 104-frame windows as ONE batched unmasked attention + ragged tail, no dense mask → O(N·W) memory, removing the encoder length cap. Encoder cos_mean 0.99953 vs a windowed bf16 reference. Default flip evaluated and REJECTED (2026-07-10), raw-decode A/B on M1 Metal (phrase-cycle metric, q4_k-v2):

clip mode wall peak RSS max uni-run / cycle complete
t32-145s (noisy) full ~400 s 1.65 GB 1 / 0 yes
t32-145s (noisy) windowed 658 s 1.99 GB 238 / 119 no
jfk×12 132 s (clean) full 271 s 1.99 GB 1 / 0 198 words
jfk×12 132 s (clean) windowed 397 s 1.97 GB 1 / 0 176 words

The windowed path attends the quiet/noisy lead and greedy decode collapses there — on Metal as well as CUDA (the earlier “Metal recovers” observation was not robust). It is also ~1.5× slower at these lengths (many 104² matmuls vs one big one), and the O(N·W) memory advantage only matters past full attention’s ~10-min OOM point. Default stays full attention + 30 s chunks; windowed remains the escape hatch for >10-min single-pass audio, with fix_loops ON. (jfk×12 note: both modes EOS early on genuinely 12×-repeated audio — model behaviour, not a loop bug.)

glm-asr multi-window single-pass

Blueprint path: 30 s sample windows, encode-then-trim, concatenated into ONE prompt/decode — cap 21 windows = 655 s (processor max_audio_len). The 145 s clip = 5 windows, 1812 audio tokens, 1827-token prompt, q4_k output 113 tokens vs bf16 blueprint’s 115 (near-verbatim). Note the blueprint (and therefore our single-pass) SKIPS quiet leading audio; the default 30 s-chunked path covers more content on such clips.

Long-audio coverage benchmark — 2026-05-21 (issue #89, historical)

Platform: x86_64 VPS, 4 threads, CPU-only, no GPU. Commit 5e16414 (30 s auto-chunk fallback + PR #116 VAD gate fix).

Test audio: first 60 s of the issue #89 reporter’s exact YouTube clip (o_9dWkRPYC0, Japanese podcast, 16 kHz mono PCM). Human estimate: 100-150 words in the first 60 s of continuous speech.

Issue #89 fix verification — parakeet-tdt-0.6b-ja

Final state — streamed encoding is always on (default CRISPASR_PARAKEET_STREAM_THRESHOLD=0, see commit “always route parakeet through streamed encode”).

Global z-norm + overlapping 8 s encoder chunks + single TDT decode pass.

The earlier “single-pass ≤60 s” default produced 99.5 % coverage on the cached MP3-derived copy of the reporter’s clip but only ~33 % on a fresh yt-dlp extract of the same YouTube video (lenhone, issue #89 comment 4529025103). Both extracts are perceptually identical (duration 60.000 s, 0.998 waveform correlation, ~0.3 % RMS diff from codec quantization) but the FastConformer encoder’s full-clip bidirectional attention amplified that quantization noise enough (encoder output std differed by 14 %: 0.2069 vs 0.2415) to drive the TDT decoder into emit-blank-forever past frame 250 (≈20 s). The streamed path keeps attention local to 8 s windows, so codec perturbations don’t amplify and the decoder runs to the end.

path audio chars first_ts last_ts coverage% gaps
streamed (default) reporter’s MP3-derived yt_60s.wav 309 0.00 55.84 ~99 0
streamed (default) fresh yt-dlp Opus→PCM o_9dWkRPYC0_60s.wav 314 0.00 55.84 ~99 0
old: single-pass ≤60 s reporter’s MP3-derived yt_60s.wav 309 0.16 59.84 99.5 0
old: single-pass ≤60 s fresh yt-dlp Opus→PCM o_9dWkRPYC0_60s.wav 91 0.16 20.08 ~33 0
--vad (silero) reporter’s MP3-derived 281 0.36 59.87 93.1 1
--vad --vad-model firered reporter’s MP3-derived 238 0.28 58.01 85.1 1
old: 30 s independent chunks reporter’s MP3-derived 195 0.16 58.02 59.7 2

Key findings:

Robustness validation — 2026-05-23

Full sweep on the reporter’s 60 s clip (commit 0c24178e, CPU-only). Streamed pipeline output is byte-identical to single-pass across every chunk/overlap combination tested — the global z-norm makes chunk boundaries transparent to the TDT decoder.

Caveat (added 2026-05-24). The “identical to single-pass” column below is only true on the cached MP3-derived audio (/mnt/storage/ samples/o_9dWkRPYC0.mp3 → yt_60s.wav). On a fresh yt-dlp extract of the same YouTube video, single-pass collapses to ~20 s of output while streamed still covers the whole clip — see the “single-pass not robust” finding above. The streamed-vs-streamed numbers across chunk and overlap sizes (300+ chars, 99 %+ coverage) hold across both audio derivations.

Chunk-size sweep (streamed, overlap=2 s):

chunk chars coverage% gaps identical to single-pass?
4 s 294 99.5 0 yes
6 s 294 99.5 0 yes
8 s (default) 294 99.5 0 yes
12 s 294 99.5 0 yes
16 s 294 99.5 0 yes
20 s 294 99.5 0 yes
30 s 294 99.5 0 yes

Overlap sweep (streamed, chunk=8 s):

overlap chars coverage% gaps identical?
0 s 294 99.5 0 yes
1 s 294 99.5 0 yes
2 s (default) 294 99.5 0 yes
3 s 294 99.5 0 yes
4 s 294 99.5 0 yes

300 s Japanese audio (streamed, default 8 s chunks):

VAD comparison (60 s):

mode chars coverage% gaps
auto (streamed) 294 99.5 0
--vad silero 281 93.1 1
--vad --vad-model firered 238 85.1 1

VAD produces fewer characters because it segments on speech boundaries and transcribes each segment independently. The auto/streamed path transcribes the full audio continuously and achieves higher coverage. Use VAD when you need per-utterance SRT entries; use auto when you want maximum transcription completeness.

Multi-backend Japanese comparison (60 s)

All backends on the same 60 s Japanese clip. “chars” counts non-space characters (Japanese has no word spaces; “words” column counts space-delimited tokens, which undercounts for CJK).

backend settings chars coverage% gaps wall_s rtf
parakeet-tdt-0.6b-ja auto (streamed) 294 99.5 0 ~60 ~1.0×
cohere-transcribe --vad 296 96.8 0 169.0 0.4×
parakeet-tdt-0.6b-ja --vad silero 281 93.1 1 50.7 1.2×
cohere-transcribe auto 242 87.4 1 199.2 0.3×
parakeet-tdt-0.6b-ja --vad firered 238 85.1 1 58.3 1.0×

Quality ranking for 60 s Japanese:

  1. Parakeet auto (streamed) — 99.5 % coverage, zero gaps, ~1× RT. The NeMo-style pipeline makes this the clear winner.
  2. Cohere + VAD — 96.8 %, zero gaps, but 3× slower.
  3. Parakeet + VAD silero — 93.1 %, 1 gap. Useful for per-utterance subtitle segmentation.

Cross-backend CAP_INTERNAL_CHUNKING — 2026-05-23

The 30 s auto-chunk fallback affected all CAP_UNBOUNDED_INPUT backends that use PerFeatureZ mel normalization, not just parakeet. Adding CAP_INTERNAL_CHUNKING to canary and fastconformer-ctc (commit 1dd247a7) lets them skip the auto-chunk and process full audio in a single encoder pass.

backend audio coverage (old 30 s chunks) coverage (new, single-pass)
parakeet-tdt 0.6b JA 60 s JA 59.7 % 99.5 %
parakeet-ctc 1.1b 60 s EN 74.6 % 98.5 %
canary-1b-v2 Q4_K 60 s EN broken (empty) 96.8 %

Not affected (different normalization or architecture): wav2vec2 / hubert / data2vec (GlobalClipMax, no PerFeatureZ drift); firered-asr (PerFeatureZ but inline AED — needs separate investigation); granite-nar (different architecture).

Benchmark framework

Results collected with tests/benchmark_asr.py:

# Quick single-backend triage:
python tests/benchmark_asr.py --audio myfile.wav --backend parakeet

# Full matrix across backends and settings:
python tests/benchmark_asr.py --corpus /mnt/storage/test-audio/corpus.json --all-settings

# Build the test audio corpus (en/de/ja/zh × 4 durations from FLEURS):
python tests/benchmark_corpus.py

Results are stored in /mnt/storage/benchmark-results/runs.jsonl (JSONL, one line per run). The framework computes: word count, char count, first/last timestamp, time coverage %, gap count/size, wall time, and realtime factor. See tests/benchmark_metrics.py for the metric definitions and tests/test_benchmark_metrics.py for 14 pytest unit tests that validate the computation (including the issue #89 failure signature: <5 % coverage on 300 s audio).

Multi-backend long-form Japanese — 120 s sweep (2026-05-24, issue #89 follow-up)

Test audio: first 120 s of the issue #89 reporter’s exact YouTube clip (o_9dWkRPYC0, fresh yt-dlp Opus→WAV extract, the file lenhone actually reports against — md5 d1f2ef…, not the cached MP3-derived copy). Apple M1 Metal, default flags unless noted.

Speech runs out around 01:37; the remaining ~22 s is short pause + follow-on talking. All “covers full speech” rows below land around 01:37 → 02:00.

backend mode segments first → last ts coverage notes
parakeet-tdt-0.6b-ja (default, streamed TDT) full 12 0:00 → 1:37.84 full speech post-fix 33f9a162
parakeet-tdt-0.6b-ja + --vad full 14 0:00 → 1:58.39 full speech cleaner per-utterance
parakeet-tdt-0.6b-ja + --parakeet-decoder ctc (hybrid CTC head) full 12 0:00 → 1:37.84 full speech byte-identical to streamed-TDT — confirms encoder is fine
parakeet-tdt-0.6b-ja + STREAM_THRESHOLD=999 (forced single-pass) full many 0:00 → ~14 s then kana-by-kana fragmentation broken the issue #89 bug; reproduces the lenhone complaint
sensevoice-small (CTC, multilingual) --vad 13 0:00 → 2:00 full speech accurate, minor JA glitches (スピーク**ジャ**プネス**ナチャパ**); 19.8× RT
voxtral-mini-3b (LLM AR, multilingual) default chunking partial 0:00 → 0:27 then 1:47 → 2:00 drops 0:27 → 1:47 (~80 s) LLM decoder loses a middle chunk
cohere-transcribe (Conformer, multilingual) default chunking 4 0:00 → 1:53 sparse with multi-tens-of-seconds gaps only ~0:00, 0:50, 1:18, 1:48 anchors
canary-1b-v2 (NeMo multilingual seq2seq) default broken n/a hallucinates "I am not aware of anything" in English needs proper language-prompt wiring; out of scope here

Best on this 120 s clip: parakeet-tdt-0.6b-ja (post-fix, streamed TDT) and sensevoice-small are tied — both produce full speech coverage with sentence-level segmentation. Parakeet via the CTC head produces byte-identical output to streamed-TDT, which confirms the encoder isn’t the problem: only TDT-decoded-over-the-full-utterance is.

The new finding (voxtral, cohere): these aren’t parakeet-specific failures. voxtral and cohere both drop the middle of a 120 s clip on this audio, with different symptoms:

Upstream behaviour for the same long-form failure:

Implication. The “default fine on a single transcribe() call over the whole file” affordance is fragile across this whole class of models. Going forward we should probably treat every CAP_UNBOUNDED _INPUT backend the way we now treat parakeet: ship a chunked / streamed default that the user doesn’t have to opt into. See PLAN #114 for the open architectural question and the per-backend ladder.

Cross-length × cross-backend matrix — 60 / 120 / 300 / 600 s (2026-05-25)

Same audio (lenhone’s fresh yt-dlp extract), extended to longer durations. Linux x86 CPU on the issue #89 VPS (168.119.190.252), sequential to avoid memory contention (we paused / split the queue when the kernel went into thrash territory). tools/longform_vps.sh is the harness; tools/analyze_longform.py parses the per-cell JSON output.

Matrix v1 vs v2 (2026-05-25 afternoon recheck). The numbers below were collected in two passes. The first pass (matrix v1) ran on the VPS binary bd8b98cf (May 24), which predates the per-backend opt-out fixes for cohere (dc2295b2), gemma4-e2b / glm-asr (46f6848d), kyutai-stt (eaee2319), and voxtral (6fef8790) that landed during the matrix run. Those fixes remove an external overlap-save context wrap that the LLM-decoder backends couldn’t trim back from correctly. Their pre-fix coverage of ~9-65 % at 120 s+ was driven by the wrap, not the model architecture. Matrix v2 (post-opt-out, rebuilt VPS binary 13059e0c) shows the true post-fix behaviour. Both passes are kept so the reader can see the cost of the missing opt-out.

Coverage % (covered span / clip duration, computed from segment timestamps). Higher = better. Bold = best at that length.

Matrix v2 (post-opt-out, what main looks like today):

backend / mode 60 s 120 s 300 s 600 s
parakeet streamed-TDT (default) 93.1 81.5 96.6 99.3
parakeet CTC head (byte-identical) 93.1 81.5 96.6 99.3
voxtral-mini-3b (default chunking) 100.0 100.0 100.0 wall-time timeout (rc=124 at 900 s in 15 min limit; LLM-AR CPU-bound, not a coverage failure)
voxtral-mini-3b streamed (option A — this PR’s pipeline) — single LLM context 100 %, 11 segs / 470 chars 100 %, 527 chars 100 %, 1276 chars / 863 tokens (post-a5165c84 max_new scaling fix; was 781 chars / 512-tok cap pre-fix) hung on contended M1 (80 MB free out of 16 GB → Metal allocator stall — see HISTORY 2026-05-25 (late) “Distinguishing slow vs hung run”)
cohere-transcribe (default chunking) 96.3 97.9 98.1 97.9 (22 segs, full 0:00 → 10:00, 577 s wall)
parakeet single-pass (STREAM_THRESHOLD=999, opt-in regression bait) 33.2 81.7 1.5 99.9
parakeet + --vad (silero) 86.7 82.0 76.3 84.0
canary-1b-v2 (still hallucinates English at every length — separate prompt-wiring bug, PLAN #114 P3)      

Largest gap (seconds) between consecutive segments, matrix v2:

backend / mode 60 s 120 s 300 s 600 s
parakeet streamed-TDT 0.0 0.0 0.0 0.0
voxtral default (post-opt-out) 0.0 0.0 0.0 timeout
cohere default (post-opt-out) 1.2 1.2 1.2 2.2

The pre-fix gap pathologies — voxtral 21.9 / 78.2 / 240.9 / 545.5 s and cohere up to 50 s — are entirely gone with the opt-out fixes. What remains is the cohere baseline ~1.2 s gap between chunks (natural energy-chunker boundaries, well under a sentence pause).

Wall time (s), matrix v2:

backend / mode 60 s 120 s 300 s 600 s
voxtral default (post-opt-out) 237 393 834 timeout (>900 s)
cohere default (post-opt-out) 70 125 290 577

cohere is consistently ~2× realtime at 300-600 s on VPS x86 CPU. voxtral-mini-3B is ~1.4-2× slower than cohere at the same length (LLM AR decode at 3 B params) and hits the wall around the 10 min mark. Apple Silicon Metal would close most of that gap — the LLM-AR rows on Mac are typically 5-10× faster than x86 CPU on this size class.

Matrix v1 (pre-opt-out, kept as historical reference for what we fixed):

backend / mode 60 s 120 s 300 s 600 s
parakeet streamed-TDT (default) 93.1 81.5 96.6 99.3
parakeet CTC head 93.1 81.5 96.6 99.3
parakeet single-pass (STREAM_THRESHOLD=999) 33.2 81.7 1.5 99.9
parakeet + --vad (silero) 86.7 82.0 76.3 84.0
voxtral-mini-3b (default chunking) 63.5 34.8 19.7 9.1
cohere-transcribe (default chunking) 95.0 91.5 58.8 61.8
cohere-transcribe + --vad 96.8 90.8 92.5 91.4
canary-1b-v2 (default) 99.7* 99.9* 99.3* OOM (rc=137)

*Canary’s coverage% is misleading — the transcribed text is "I am not aware of anything, I am not aware of…" (English) at every duration. It’s a separate language-prompt-wiring bug, not a long-audio bug. See PLAN #114 P3.

Largest gap (seconds) between consecutive emitted segments — catches the “drops a middle chunk” failure that the coverage% can under-report when the missing region is bracketed by emitted text on both sides.

backend / mode 60 s 120 s 300 s 600 s
parakeet streamed-TDT 0.0 0.0 0.0 0.0
parakeet CTC head 0.0 0.0 0.0 0.0
parakeet single-pass 0.0 0.0 0.0 0.0
parakeet + --vad 5.8 12.0 30.1 30.1
voxtral-mini-3b 21.9 78.2 240.9 545.5
cohere-transcribe 0.0 3.4 50.0 50.0
cohere-transcribe + --vad 1.4 4.2 4.2 19.0
canary-1b-v2 0.0 0.0 0.0 n/a

Wall time (s) / realtime factor. Apple Silicon would be 5-10× faster on the parakeet rows; numbers below are the Linux x86 VPS.

backend / mode 60 s 120 s 300 s 600 s
parakeet streamed-TDT 55 99 236 463
parakeet CTC head 54 102 236 462
parakeet single-pass 45 86 235 627
parakeet + --vad 55 97 225 457
voxtral-mini-3b 166 165 189 193
cohere-transcribe 79 144 349 673
cohere-transcribe + --vad 65 117 279 557
cohere-asr-ja (Q4_K, JA audio) 31 140 — —
cohere-asr-ja (Q4_K, EN audio) 39 104 — —
canary-1b-v2 68 122 381 OOM

(voxtral wall time is roughly constant because it silently skips most of the input — see the gap column.)

Per-backend take-aways from the matrix

parakeet (the post-fix default). streamed-TDT and CTC-head are byte-identical at every length (CTC head is a frame-synchronous fallback that bypasses the TDT blank-runaway entirely; streamed-TDT keeps the TDT decoder but bounds the encoder’s bidirectional attention to 8 s windows so it can’t accumulate the codec-noise amplification). The 120 s coverage dip to 81.5 % is the audio, not the model: the clip’s speech runs out at ~01:37 and the next ~22 s is silence + a sentence-start, so coverage measured against the full 120 s under-counts. Both paths produce the same actual content.

parakeet single-pass. Catastrophically non-monotonic: 33 % at 60 s, 82 % at 120 s, 1.5 % at 300 s, 99.9 % at 600 s. This is the “per-feature z-norm depends on the full audio’s mel statistics” problem manifesting as random walks across the stable/unstable boundary. The single-pass path is genuinely unsafe; the “works at 600 s” cell is luck, not a property.

parakeet + --vad. Coverage drops to 76-87 % across lengths because VAD trims silence (by design). Larger gaps at longer durations because the underlying clip has more silence stretches. Good for “I want per-utterance SRT entries” use cases, less so for “I want continuous transcription with maximum coverage.”

voxtral-mini-3b. In matrix v1: worst long-form behaviour we measured. Coverage halved with each length doubling: 64 → 35 → 20 → 9 %. In matrix v2 (post-opt-out, commit 6fef8790 removing the external overlap-save wrap), coverage jumps to 100 % at 60 / 120 / 300 s: the LLM AR decoder was processing all chunks fine; the matrix-v1 word-timestamp trim was discarding most of the emitted text because voxtral’s emitted word timestamps don’t honour the original slice frame, so the trim treated almost everything as “outside the slice range.” Two additional fixes shipped together with this matrix:

cohere-transcribe. In matrix v1: degraded from 95 % at 60 s to **59 % at 300 s and 62 % at 600 s, with 50 s gaps.** In matrix v2 (post-opt-out, commit dc2295b2 removing the external overlap-save wrap for cohere), default chunking jumps to 96-98 % at 60 / 120 / 300 s with gaps ≤ 1.2 s. The pre-fix gap-growth was driven by the overlap-save wrap, not the model itself. --vad is no longer a mandatory rescue — it’s available for users who want per-utterance SRT segmentation, but coverage parity is now native.

canary-1b-v2. Separate bug. Coverage looks fine because the decoder emits text for the full duration, but the text is English "I am not aware of anything" in a loop regardless of input language. Language-prompt wiring problem, not a long-audio problem. 600 s OOM-killed (rc=137) on the 7.6 GB VPS — likely the AED decoder’s hidden-state stack growing past the available memory.

What’s the right default per backend, post-matrix v2

backend recommended default why
parakeet (any variant) streamed-TDT (default since 33f9a162) best coverage at all lengths, byte-identical to CTC-head when available
voxtral-mini-3b streamed (this PR — Mistral apply_transcription_request shape) 100 % coverage at 60-300 s, single LLM context, denser segmentation; default-chunked + opt-out (6fef8790) also lands at 100 %
cohere-transcribe default chunking + opt-out (dc2295b2) 96-98 % at 60-300 s; --vad available but no longer required for coverage
canary-1b-v2 fix lang-prompt bug first; then streamed-encode port currently broken at all durations on JA; long-audio fix on hold
qwen3-asr / granite-speech / mimo-asr post-opt-out default chunking (audit pending) LLM-AR class — opt-out gate is glm-asr / gemma4-e2b / kyutai-stt (46f6848d, eaee2319); voxtral-style streamed is a follow-up improvement, not a coverage fix
fastconformer-ctc / wav2vec2 / firered-asr current single-pass (CTC is robust) no observed failure; defer streamed port until reported
sensevoice-small --vad already the recommendation; matrix v1 confirms 99 %+ at 120 s
whisper unchanged internal 30 s seek handles long audio by design

Reproducer

# Driver — runs all 32 cells sequentially with memory backpressure
bash tools/longform_vps.sh   # outputs to /mnt/akademie_storage/longform_results/

# Parser — JSON outputs → coverage table
python tools/analyze_longform.py /path/to/longform_results/

Both scripts in this commit. Audio: /mnt/akademie_storage/yt_{60,120,300,600}s.wav on the VPS (PCM s16le, 16 kHz mono, fresh yt-dlp extract of youtube.com/watch?v=o_9dWkRPYC0).

Beam search — quality vs speed (2026-05-23, PLAN #90)

Knob: --beam-size N (CLI) / CRISPASR_BEAM_SIZE=N (env) / crispasr_session_set_beam_size(session, N) (C API). Default N=1 (greedy). N > 1 activates beam search on supported backends. LLM-decoder backends (qwen3-asr, granite-speech, voxtral, gemma4-e2b) use core_beam_decode::run_with_probs (replay-from-prefix). Encoder-decoder backends (canary, cohere) use core_beam_decode::run_with_probs_branched (KV snapshots). Transducer backends (parakeet) use a dedicated TDT/RNNT label-looping beam search. CTC-only and NAR backends ignore the flag.

Benchmark script: tools/benchmark_vitw_beam.py — runs against zhifeixie/Voices-in-the-Wild-Bench (5 000 samples, 8 acoustic conditions, streamed — no full download needed).

Speed cost on JFK (11 s, M1 Metal, post-warmup)

backend beam=1 beam=2 beam=4
qwen3-asr 0.6B Q4_K 3.67 s (1×) 8.20 s (2.2×) 14.75 s (4.0×)
granite-speech 4.1 2B Q4_K 18.39 s (1×) 27.59 s (1.5×) 33.17 s (1.8×)
voxtral mini 3B Q4_K ~70 s (1×) ~56 s (0.8×)† ~77 s (1.1×)

†voxtral beam=2 < beam=1 is measurement noise — voxtral spends most of its time in the audio encoder; decoder token count for JFK is small enough that OS jitter dominates.

WER by condition (qwen3-asr, Voices-in-the-Wild-Bench, 8 EN samples each)

condition beam=1 beam=2 beam=4 beam=2 cost beam=4 cost
real_noise 0.125 0.144 0.136 1.7× 3.5×
syn_noise 0.167 0.167 0.167 2.6× 2.7×
real_dropout 0.045 0.045 0.041 1.9× 4.6×
real_obstructed 0.015 0.015 0.015 1.9× 3.3×
real_mixed 0.035 0.039 0.039 2.1× 4.8×
syn_mixed 0.089 0.089 0.080 1.3× 2.2×

Key findings

When to use

scenario recommendation
Clean / studio speech greedy (beam=1) — no quality gain, 2-5× cost
Noisy real speech, latency-insensitive beam=2 — marginal gain possible, 2× cost
Rare words / phonetic confusion, offline beam=4 — worth trying
Streaming / latency-critical greedy only — beam adds a full extra decode pass per token step

Reproduce:

python tools/benchmark_vitw_beam.py \
    --backends qwen3 \
    --splits real_noise,real_dropout,real_obstructed,real_mixed,syn_mixed \
    --n 8 --beams 1,2,4 \
    --json tools/vitw_beam_results.json

MAES beam search for transducers (2026-06-03, §134)

MAES (Modified Adaptive Expansion Search) is a transducer-specific beam search that’s more efficient than the label-looping beam above. It processes one encoder frame at a time with up to N adaptive non-blank expansions per frame, using gamma-threshold pruning to kill low-probability branches.

Knob: CRISPASR_PARAKEET_MAES=1 + --beam-size N (CLI), or --parakeet-decoder maes + --beam-size N, or parakeet_set_maes(ctx, true, num_steps, gamma, beta) (C API). Config: CRISPASR_MAES_NUM_STEPS (default 2), CRISPASR_MAES_GAMMA (2.3), CRISPASR_MAES_BETA (2).

Supports both TDT (Token-and-Duration Transducer) and pure RNNT models.

MAES vs greedy on FLEURS English (CPU, Hetzner CCX13, 4 threads)

Model Audio Greedy MAES beam=4 Speed cost
tdt-0.6b-v2 (1K vocab) 10s “…by 25%.” “…by 25 years.” —
tdt-0.6b-v3 (8K vocab) 10s “…by 25-30 years.” “…by 25 to 30 years.” —
tdt-0.6b-v2 60s 5 sentences 6 sentences (recovered full missing sentence) +35%
tdt-1.1b (8K vocab) 10s “…by twenty five to thirty years” identical —
tdt_ctc-110m (1K vocab) 10s garbled same garble —
rnnt-0.6b (8K vocab) 10s “…by twenty five to thirty years” identical —
rnnt-1.1b (8K vocab) 60s truncated at “lettering” identical +25%

When to use MAES vs standard beam

scenario recommendation
Parakeet TDT with small vocab (v2, 1K BPE) MAES beam=4 — measurable quality gain
Parakeet TDT/RNNT with large vocab (8K BPE) greedy — already strong baseline, MAES matches but costs 25-35%
Parakeet with hotwords (CTC-WS) label-looping beam — hotword trie not yet wired into MAES
Tiny model (110M) neither — model capacity is the bottleneck

CTC prefix beam search (2026-06-03, §134)

Shared core_ctc::prefix_beam_search() with optional gamma-threshold pruning. Available for any CTC backend via --beam-size N. Currently wired into: parakeet-CTC, sensevoice, wav2vec2 (16 languages).

CTC beam search has not yet been benchmarked for WER improvement — the primary benefit is expected to be on character-level CTC models (wav2vec2) where the small vocab makes greedy more error-prone than BPE models.

Transducer + encoder-decoder beam search (2026-06-02, issue #136 + §139)

Parakeet TDT/RNNT label-looping beam (b3cdcebd), canary + cohere AED branched-KV beam (§90 runtime, adapter wiring §139), gemma4-e2b replay-from-prefix beam (§139). All on VPS CPU-only (no GPU).

JFK 11 s — wall time (user time in parentheses)

backend model beam=1 beam=2 beam=4
parakeet parakeet-tdt-0.6b-v3 F16 27 s (30 s) 26 s (31 s) 15 s (32 s)
canary canary-1b-v2 Q4_K 27 s (37 s) 32 s (50 s) 42 s (68 s)
cohere cohere-transcribe F16 125 s (91 s) 104 s (107 s) 120 s (118 s)

Notes:

FLEURS 10 s — canary beam=4 vs greedy

beam output
1 “…Styles in the West could lag behind by twenty five percent. 25 to 30 years.”
4 “…styles in the west could lag behind by twenty five percent. 25 to 30 years.”

Minor capitalization difference (proper-noun casing on “Styles”/”West”).

FLEURS 60 s — parakeet beam=4 vs greedy

beam output diff
1 “…, and which was made famous…”
4 “… and which was made famous…” (comma dropped)

Both valid; stylistic punctuation variation.

Overhead summary (user time)

backend beam=2 beam=4 beam=8
parakeet (TDT LSTM) ~3 % ~7 % ~20 %
moonshine-streaming ~24 % ~56 % —
canary ~35 % ~84 % —
cohere ~18 % ~30 % OOM (F16)

Parakeet beam search is nearly free because the decoder is a tiny LSTM (~10 KB state per beam). Canary and cohere have 8-layer transformer decoders with full KV snapshot/restore, so the cost scales with decoder depth × sequence length × beam width.

Translation beam search (m2m100 + madlad/t5, 2026-06-02)

m2m100-418m Q8_0 — en→de (CPU-only VPS)

sentence beam=1 user beam=4 user output
“Hello world, how are you today?” 6 s 21 s (3.4×) “Hallo Welt, wie bist du heute?”
“The president said he would not attend…” 7 s 45 s (6.4×) “Der Präsident sagte, er würde wegen der Wetterbedingungen nicht an der Sitzung teilnehmen.”

Translation beam is expensive: the decoder-only replay cost is O(beam × T²) where T is the output length, and for translation the decoder does more work per token than for ASR. Identical output on these clean inputs; benefit is on ambiguous source text.

Multi-backend long-form comparison — 2026-05-26 (PLAN #114 P3 closeout)

Live runs on M1 Metal with the post-PLAN-#114-P3 binaries. Inputs from /Volumes/backups/code/audio_samples/ (mirrored from VPS — see that dir’s CLAUDE.md). All backends invoked with -l <lang>, -np, -nt, default settings (CANARY_STREAM_THRESHOLD_S=0 after 10c2fba5).

EN 60 s (FLEURS English, narration)

audio_samples/en/fleurs_60s.wav  (60 s, 16 kHz mono)
Backend Chars Notes
parakeet-tdt-0.6b-v3 ~217 → 520 (post e1904a1e) The 217 was with the c=8 chunk default that ships well for the JA-only model. Empirical sweep on EN+DE FLEURS 60s+300s showed c=8 collapses on the multilingual v3 model: EN 60s drops to 23% of the c=40 max. Fix e1904a1e adds a per-model chunk default keyed off vocab_size (< 4000 ⇒ JA model ⇒ c=8 preserved; ≥ 4000 ⇒ v3 / multilingual ⇒ c=30). v3+EN60 improved 186 → 520 chars (2.8×), v3+EN300 492 → 1550 chars (3.15×), v3+DE60 502 → 679, v3+DE300 2496 → 3064, ja+JA60 1674 unchanged. Root cause is encoder context, not decoder cold-start — see the corrected LEARNINGS section on the Independent-chunk failure mode.
canary-1b-v2 ~735 Full content but visible artifacts: "twenty five. to thirty" (model splits a number), "Save for You" (AED re-emits with different capitalization → after case-insensitive LCS the dup is dropped but the leftover "Save for You" reads as a sentence start), "Yeah, yeah, ×14" (degenerate-loop guard fired at the configured 14-token window).
voxtral-mini-3b-2507 ~826 Clean. Includes extra content like "in which was made famous to foreigners after a glowing account of its splendorous recorded by Lord Byron" that canary missed entirely. No boundary artifacts.
cohere-transcribe ~864 Clean. Similar coverage to voxtral, "world's" instead of "world" (model preference), "Northern Marianas" instead of "Northern Mariana's".

DE 60 s (FLEURS German, narration)

audio_samples/de/fleurs_60s.wav  (60 s, 16 kHz mono)
Backend Notes
canary-1b-v2 Full content but boundary dups: "Geld-Technologie-Technologie-Technologie-Technologie" (early-chunk loop, partly caught by the guard at 4 reps before the window opened), "T-Rex war war", "Rückseite der der Unabhängigkeitserklärung", "Männer und Frauen. Frauen", "Spitze. der Spitze". The LCS-merge + word-snap + case-insensitive LCS pipeline caught some but not all — these are exact-token re-emissions across chunks that an LCS strict-prefix match still leaks.
voxtral-mini-3b-2507 Clean, single-pass-quality. No boundary artifacts visible. Catches "Juden und Nicht-Juden gleichermaßen" (post-segment continuation) that canary missed.
cohere-transcribe Clean. "Tri-Rex" is a minor model error (not a boundary artifact), otherwise identical-shape transcript to voxtral.

Architectural takeaway

The data confirms the design-notes table in PLAN #114 (“Streaming-pattern design”): the voxtral-pattern backends (voxtral, cohere — 30 s disjoint chunks → audio embeds concat → one LLM AR decode) produce cleaner long-form output than the NeMo-pattern backends (canary — 8 s overlap chunks → per-chunk decode → LCS-merge + word-snap + case-insensitive LCS dedup). The voxtral pattern’s lack of overlap means no duplication enters the input, so no dedup is needed; the NeMo pattern’s overlap (necessary for bidirectional encoder context) requires dedup, and any imperfect dedup pass leaves visible artifacts.

This is not a universal win for the voxtral pattern though — it requires a long-context AR LLM (voxtral’s 3 B, cohere’s 1.3 B). Canary’s AED was trained on 8–30 s clips and cannot absorb a full 5 min in a single decode (<eos> lands at the first internal utterance boundary). Parakeet’s TDT could in principle use the voxtral pattern but doesn’t currently — something to revisit if parakeet’s truncation behaviour on the EN 60 s clip turns out to be a streamed-path bug rather than a one-off.

Six-commit canary thread that produced the “full content” column above

SHA What
dfe1af3b lang-whitelist (en/de/fr/es only) — refused unsupported langs before they could hallucinate
7177c931 canary_transcribe_streamed first cut (concat-then-decode → truncated at AED <eos>)
63fdbe46 NeMo FrameBatchMultiTaskAED analogon — per-chunk AED decode with prompt re-injection
62766dae LCS boundary dedup
10c2fba5 splice-punct cleanup + CANARY_STREAM_THRESHOLD_S=0 default
361df3e2 window-based degenerate-loop guard
935ffbee word-snap heuristic (extend LCS drop to next word-start)
5e402ee9 case-insensitive LCS (ASCII lowercase canonical id)

Before this thread canary truncated to ~460 chars on the 1.3 m De-Abwasch article and ~360 chars on EN FLEURS 60 s; after, full coverage (~1196 chars and ~735 chars respectively).

Parakeet long-form option matrix — 2026-05-26 (PLAN #114 follow-up)

Empirical sweep across all the dispatch knobs the parakeet backend exposes, on the same three 60 s fixtures used elsewhere in this section. Default mode includes the e1904a1e per-model chunk default (v3 → c=30 internal, ja → c=8 internal).

Mode v3 + EN 60s v3 + DE 60s v3 + JA 60s ja + JA 60s
default (backend streamed, c=auto) 520 679 605 1674
CRISPASR_PARAKEET_STREAM_CHUNK=8 forced 187 503 375 1674
CRISPASR_PARAKEET_STREAM_CHUNK=30 forced 520 679 605 508
CRISPASR_PARAKEET_STREAM_THRESHOLD=999 (single-pass) 626 621 599 271
--vad --vad-model silero 368 709 637 1627
--chunk-seconds 30 --chunk-overlap 0 (no LCS) 713 689 608 1413
--chunk-seconds 30 --chunk-overlap 3 (LCS) 755 665 660 1942

Headline finding: dispatcher-side --chunk-seconds 30 --chunk-overlap 3 wins on 3 of 4 cases — shipped as the new default in 98381810

The internal-streamed-path default that previously shipped was not the quality-optimal long-form mode. The CLI’s dispatcher-side chunking + overlap-save context wrap + LCS-merge dedup recovers more content than the backend’s single-pass-over-concat-encoder design.

Shipped as the new default 2026-05-26 (98381810) by dropping CAP_INTERNAL_CHUNKING from the parakeet backend’s capabilities declaration. The dispatcher’s should_auto_chunk_long fallback then fires for audio > 30 s — chunking at 30 s, overlap-save 3 s, LCS-merge dedup — exactly the matrix’s winning mode. Short audio (< 30 s) is unaffected: the dispatcher only auto-chunks past the threshold, so the 11 s JFK case still routes through a single backend call.

After-the-fix matrix (the previous matrix was with CAP_INTERNAL_CHUNKING declared, blocking the auto-chunk path):

case old default new default Δ
JFK 11s 109 109 unchanged
v3 + EN 60s 520 755 +45 %
v3 + DE 60s 679 665 -2 %
v3 + JA 60s 605 660 +9 %
ja + JA 60s 1674 1942 +16 %
v3 + EN 300s 1550 3865 +150 %
v3 + DE 300s 3064 3288 +7 %

The longer the audio, the bigger the win — EN 300 s scales from +45 % at 60 s to +150 % at 300 s. The internal-streamed-path’s quality degradation compounds with audio length; the dispatcher chunks scale linearly.

Wall time on M1 Metal: 300 s EN now takes ~86 s (was ~30 s) — 3.5× realtime. Acceptable for the quality gains; users can still pass CRISPASR_PARAKEET_STREAM_THRESHOLD=99999 to force the older single-pass path if the wall-time matters more than coverage.

Headline finding: dispatcher-side --chunk-seconds 30 --chunk-overlap 3 wins on 3 of 4 cases (original 4-trial sweep)

Why this works: the dispatcher splits the 60 s input into ~30 s chunks with ±3 s acoustic overlap, calls the backend once per chunk (each call sees a 33 s window), and LCS-merges the boundary tokens. Inside each backend call, parakeet’s internal streamed path now runs as a single 30 s encoder window (no further sub-chunking), which is exactly the encoder context size the v3 model was trained for. The backend’s own “streamed over 60 s with c=30” instead splits the 60 s mel into two ~30 s chunks internally — but the per-chunk encoder passes don’t see the bidirectional context across the cut as cleanly as the dispatcher’s per-call boundaries do (the dispatcher feeds each chunk independently with its own mel-norm; the backend’s streamed path applies global mel-norm first then splits).

When each mode wins

Audio profile Recommended mode Why
Continuous EN/DE long-form, supported v3 lang --chunk-seconds 30 --chunk-overlap 3 Highest coverage; modest wallclock overhead (12 s for 60 s audio)
JA model on JA long-form --chunk-seconds 30 --chunk-overlap 3 OR default Both recover most content; LCS edges default by 16 % on the tested clip
Short audio (< 30 s) default Single backend call, no dispatcher overhead
Speech-with-long-silences --vad --vad-model silero VAD trims silences and feeds the backend with bounded slices; can outperform chunking when speech density is uneven
Reference parity / debugging CRISPASR_PARAKEET_STREAM_THRESHOLD=999 Forces parakeet_transcribe_ex, the bit-exact single-pass path

Caveats

Reproduce

B=build/bin/crispasr
V3=/Volumes/backups/ai/crispasr/parakeet-tdt-0.6b-v3-q4_k.gguf
JA=/Volumes/backups/ai/crispasr/parakeet-tdt-0.6b-ja-q4_k.gguf
EN60=/Volumes/backups/code/audio_samples/en/fleurs_60s.wav
DE60=/Volumes/backups/code/audio_samples/de/fleurs_60s.wav
JA60=/Volumes/backups/ai/long-clips/yt_60s.wav

# default
$B --backend parakeet -m $V3 -f $EN60 -np -nt
# --chunk-seconds 30 --chunk-overlap 3
$B --backend parakeet -m $V3 -f $EN60 -np -nt --chunk-seconds 30 --chunk-overlap 3
# (etc)

Coverage parity check vs cohere / canary on 300 s — 2026-05-26

User direction: “are these after-numbers complete? compare to what other models deliver”. Right — char-count delta vs the previous parakeet default proves we improved, but says nothing about complete. Real test: how does parakeet’s new default compare against the best long-form-capable backends on the same audio.

Ran parakeet (post-98381810 default) vs cohere vs canary on the 300 s FLEURS clips. Voxtral skipped — its mem-thrash failure mode on M1 with the 300 s clip is documented in feedback_torch_omp_deadlock; sat at 5 s CPU / 30 min wall and was killed.

EN FLEURS 300 s:

Backend chars wall (s) × RT vs cohere
parakeet (default) 3865 66 5.0× -3 %
cohere 3994 94 3.2× (ref)
canary 2971 74 4.1× -26 %

DE FLEURS 300 s:

Backend chars wall (s) × RT vs cohere
parakeet (default) 3288 69 4.3× -0.3 %
cohere 3299 87 3.4× (ref)
canary 3532 273 1.1× +7 %, 3.1× slower

Headline. Parakeet’s post-fix default is now within 3 % of cohere on EN 300 s and within 0.3 % on DE 300 s at higher throughput (66 / 69 s wall vs cohere’s 94 / 87 s). The numbers are complete in the same sense as cohere — coverage parity with the best long-form backend, at faster wallclock. The previous default (CAP_INTERNAL_CHUNKING set) was 60 % below cohere on EN 300 s; the fix closes that gap.

Canary on DE wins on coverage (3532 chars) but pays 4× the wall time (273 s) — a different trade-off. For German-only workflows where wall time is bounded, canary’s per-chunk AED decode produces slightly more content; for general use, parakeet’s faster path with coverage-parity is the recommended default.

Runtime optimization audit — 2026-06-20

Full code-read survey of every runtime in the project: what optimization tricks each already implements, and where room exists for more. Covers 65+ backends across ASR, TTS, Audio-LLM, VAD, LID, speaker, translation, enhancement, alignment, punctuation, and diarization — plus the shared core infrastructure.

Legend

1. Cross-cutting optimization matrix

1a. AR decode infrastructure (backends with autoregressive token loops)

Backend KV cache Flash attn Fused QKV Graph cache CPU embd cache Layer offload Beam search
whisper Has Has — Gap — — Has (batch)
canary Has (quant) Has — Gap — Gap Has (branched)
cohere Has (quant) Has — Gap — Gap Has
kyutai-stt Has (quant) Has Has Gap — Gap Has
firered-asr Gap (vec) Gap — Gap — Gap Has (lazy)
moonshine Has Has — Gap — — Has (branched)
moonshine-stream Has Has — Gap — — Has
funasr Has (quant) Has (enc) Has Partial — Has (split) Has (replay)
voxtral Has (quant) Partial Has Gap — Has Has
voxtral4b Has (quant) Partial (enc gap) Has Gap — Has Has
glm-asr Has (quant) Has — Gap — Has (LLM) Has (replay)
granite-speech Has (quant) Has (LLM) — Gap — Has Has
qwen3-asr Has (quant) — Has Gap — Has —
omniasr Has (quant) Has — Gap — Has Has (replay)
mimo-asr Has (quant) Has (audio) Has Has (T=1) — Has Has
moss-audio Has Gap (enc) Gap Gap — Gap Has
gemma4-e2b Has (dual) Gap (enc) — Gap — Has Has
m2m100 Has (cross) Has — Gap — Gap Has (replay)
t5-translate Has (cross) Gap (rpe) — Gap — Gap —
orpheus Has (quant) Gap — Gap Gap Has —
outetts Has (quant) Gap — Gap Gap Gap —
kokoro — (NAR) Gap — Gap — — —
bark Has Has Has Gap — — —
melotts — Gap — Gap — — —
parler Has (host; device opt-in §176c) Gap — Has (Lk-bucket, opt-in §176b) — — —
speecht5 Has (host) Gap — Gap — — —
dia Has (host) Gap — Gap — — —
csm Has (device) Has (Mimi) Has — — — —
indextts Has (device) Has — — — — Has (device)
f5-tts — (diff) Has (DiT) — Gap — — —
fastpitch — (NAR) Has Has — — — —
zonos Has (quant) Gap Has Gap Gap Gap —
vibevoice Has (F16) Has — Partial Gap Has —
qwen3-tts Has (quant) Has Has Has (5-bucket) Has Partial —
cosyvoice3 Has Gap — Gap Gap Gap —
tada Has (quant) Gap — Gap Gap Gap —
voxcpm2 Has (host) Gap — Gap — Gap (CPU only) —
chatterbox Has (quant) Gap (stub) — Gap Gap Gap —
pocket-tts Has (host) Gap — Gap — — —
lfm2-audio Has (F16) Has Has Gap — Has Gap (stub)
mini-omni2 Has (quant) Has — Gap — — —
kugelaudio Has Has — Gap — — —
pcs — (enc) Has — — — — —

1b. Encoder-only / non-AR backends

Backend Flash attn BN fold Fused QKV Graph cache GPU path
paraformer Has — Has (cross) Gap Gap
parakeet Has (opt-in) Has — Gap Has
sensevoice Has — Has Gap Has
wav2vec2 Has — — Gap Has
fireredpunc Has — — Gap Has

1c. Support runtimes (VAD, LID, speaker, etc.)

Backend Context cache GPU inference Flash attn BN fold
silero-vad Has — — —
encdec-vad Gap Gap (CPU only) Has —
marblenet-vad Gap Gap (CPU only) — —
firered-vad Gap Gap (loads GPU, runs CPU) — —
pyannote-seg Gap — (correct: tiny model) — —
ecapa-lid Gap Has (partial) — Gap
silero-lid — Gap (loads GPU, runs CPU) Gap —
firered-lid — Has (via firered-asr) Has —
cld3 — (correct) — (correct) — —
fasttext-lid — — — —
titanet Gap Gap (loads GPU, runs CPU) — Has
audioseal Has (sched) Has — —
rnnoise Gap — — —

2. Per-runtime detail

ASR — Encoder-only / NAR

Paraformer (paraformer.cpp):

Parakeet (parakeet.cpp):

SenseVoice (sensevoice.cpp):

Wav2Vec2 (wav2vec2-ggml.cpp):

ASR — Encoder-decoder

Canary (canary.cpp):

Kyutai STT (kyutai_stt.cpp):

FireRed ASR (firered_asr.cpp):

Moonshine (moonshine.cpp):

Moonshine Streaming (moonshine_streaming.cpp):

ASR — LLM-decoder

FunASR (funasr.cpp):

Voxtral 3B (voxtral.cpp):

Voxtral 4B (voxtral4b.cpp):

GLM ASR (glm_asr.cpp):

Granite Speech (granite_speech.cpp):

Qwen3 ASR (qwen3_asr.cpp):

OmniASR (omniasr.cpp):

MIMO ASR (mimo_asr.cpp):

MOSS Audio (moss_audio.cpp):

Nemotron (nemotron.cpp):

Gemma4 E2B (gemma4_e2b.cpp):

TTS

Kokoro (kokoro.cpp):

Bark (bark_tts.cpp):

MeloTTS (melotts.cpp):

OuteTTS (outetts.cpp):

Orpheus (orpheus.cpp):

Parler TTS (parler_tts.cpp):

SpeechT5 (speecht5_tts.cpp):

F5-TTS (f5_tts.cpp):

Dia (dia_tts.cpp):

CSM (csm_tts.cpp):

IndexTTS (indextts.cpp):

FastPitch (fastpitch_tts.cpp):

OpenVoice2 (openvoice2.cpp):

Zonos (zonos_tts.cpp):

VibeVoice (vibevoice.cpp):

Qwen3 TTS (qwen3_tts.cpp):

CosyVoice3 (cosyvoice3_tts.cpp):

TADA (tada_tts.cpp):

VoxCPM2 (voxcpm2_tts.cpp):

Chatterbox (chatterbox.cpp):

Pocket TTS (pocket_tts.cpp):

Audio-LLM / S2S

LFM2 Audio (lfm2_audio.cpp):

Mini-Omni2 (mini_omni2.cpp):

KugelAudio (kugelaudio.cpp):

PCS (pcs.cpp):

VAD

Silero VAD (crispasr_vad.cpp):

WhisperEncDec VAD (crispasr_vad_encdec.cpp):

MarbleNet VAD (marblenet_vad.cpp):

FireRed VAD (firered_vad.cpp):

Pyannote (pyannote_seg.cpp):

LID

ECAPA-TDNN LID (ecapa_lid.cpp):

Silero LID (silero_lid.cpp):

FireRed LID (firered_lid.cpp):

Speaker

TitaNet (titanet.cpp):

Translation

M2M-100 (m2m100.cpp):

T5 Translation (t5_translate.cpp):

Other

AudioSeal (audioseal.cpp):

RNNoise Enhancement (crispasr_enhance.cpp):

FireRedPunc (fireredpunc.cpp):

CTC Forced Alignment (align.cpp):

Diarization (crispasr_diarize.cpp):

3. Shared core infrastructure

attention.h: flash attn, fused QKV, GQA (repeat_4d + native), KV quant (asymmetric K/V), KV on-CPU, on-device snapshot pool, dynamic index KV write, forced F32 dequant read. Gap: GQA repeat_4d materializes copy for non-granite models; Metal HOST snapshot still memcpys.

ffn.h: fused gate+up (saves ~30 ms/call per code comment), swiglu/ geglu/silu/gelu variants, no-op bias skip. Gap: no fused SiLU*mul kernel.

conv.h: convt1d_decomp (matmul+col2im), batch weight permutation, depthwise stride-2 k3. Gap: extra ggml_cont calls, single-threaded permute, F16 cast per call in depthwise.

mel.cpp: unified 9-family extractor, optional double precision, frame stacking zero-copy. Gap: single-threaded STFT loop (dominant cost for long audio), scalar mel projection (no BLAS — 31M multiply-adds for NeMo cluster), per-call power/mel buffer alloc.

kaldi_fbank.cpp: thread-local fb/window cache, iterative FFT, in- place pre-emphasis. Gap: per-call fft scratch alloc, scalar mel projection inner loop (no SIMD).

fft.h: thread-local scratch. Gap: recursive FFT with O(N log N) heap allocs — should use iterative version from kaldi_fbank.

greedy_decode.h: reserve, early exit, NoHook eliminated, inverse-CDF sampling, streaming callback, frequency penalty. Gap: sample_temp allocs vocab-sized double vector every token.

beam_decode.h: branched variant O(B×T) with VRAM snapshot pool, partial_sort, multi-EOS, RAII snapshot management. Gap: full-vocab idx alloc per beam per step, Beam deep-copies tokens/probs vectors.

ctc.h: gamma pruning, partial_sort. Gap: linear-scan prefix find_or_insert O(B×V), scalar posterior_pool.

cpu_ops.h: OMP layernorm, reused compute_meta. Gap: two-pass layernorm, matmul rebuilds graph every call.

gguf_loader.cpp: zero-copy mmap (CPU+Metal), WILLNEED→RANDOM madvise, mlock, preload, layer-split loader. Gap: two-pass tensor bind, sequential split partitions.

4. Top optimization opportunities by impact

Ordered by estimated breadth × depth of impact across the project:

Tier 1 — High impact, broadly applicable

  1. Wire flash_attn_ext in all AR decoders — Orpheus, OuteTTS, Chatterbox, Parler, SpeechT5, Dia, Zonos, TADA, Pocket-TTS, MOSS encoder, Voxtral/4B encoder, CSM backbone. The core_attn::kv_self_attn path already supports it; most backends just never pass the flag. Estimated: 2-5× attention bandwidth reduction at long sequences.

  2. Lk-bucketed graph caching for AR decode — applies to every backend that rebuilds its decode graph per step (30+ backends). Qwen3-TTS demonstrates the pattern with 5 buckets. The MIMO step_t1_gf is a simpler single-bucket variant. Graph rebuild overhead is 6-14 ms/step (measured in funasr); eliminating it across 100-1000 steps per synthesis/transcription is the largest single latency win.

  3. Migrate host-side KV re-upload to device-resident KV — SpeechT5, Dia, Parler, Pocket-TTS, VoxCPM2 all use std::vector<float> KV caches that grow and re-upload every step. IndexTTS and CSM demonstrate the correct pattern: 4D on-device tensor with ggml_view_4d + ggml_cpy writes. Eliminates O(step × layers × hidden) host↔device bandwidth.

  4. BLAS/ggml for scalar CPU matmul hotpaths — TitaNet ASP TDNN (9216×T), Silero LID attention (8 stages × O(T²)), FireRed VAD DFSMN (8 blocks), MeloTTS/Piper relpos attention (6 layers × O(T²)), OpenVoice2 WaveNet (16 layers), Granite cpu_linear, Parakeet LSTM predictor. These are the dominant compute in their respective runtimes and run as unvectorized nested loops.

Tier 2 — Medium impact, moderate effort

  1. Context caching for support runtimes — WhisperEncDec VAD, MarbleNet VAD, FireRed VAD, Pyannote, ECAPA-TDNN LID, RNNoise enhancement, CTC aligner, FireRedPunc graph context. The Silero VAD static-cache pattern is the template. Eliminates repeated init/free overhead in pipelines that call these per-segment.

  2. Parallel STFT in mel.cpp — the mel spectrogram is the entry point for ~40 backends. The STFT loop is single-threaded. Adding #pragma omp parallel for with per-thread scratch buffers would scale linearly with cores. Similarly, the mel projection matmul should use BLAS (cblas_sgemm).

  3. CPU embedding cache — Qwen3-TTS demonstrates caching quantized embedding bytes on CPU to avoid Metal command-buffer round-trips per AR step. Applies to Orpheus, OuteTTS, Chatterbox, Zonos, CosyVoice3, TADA, VibeVoice, Pocket-TTS.

  4. F5-TTS: collapse 22 per-block mini-graphs into one — currently 1408 alloc+compute+get round-trips per synthesis. Batch CFG as B=2 in a single graph would halve DiT inference time. Port Vocos + text encoder from scalar C++ to ggml.

  5. Cross-KV in F16 (not F32) — M2M-100, T5, SpeechT5, Dia, Parler all store cross-attention K/V as F32 on host. Cross-KV is read-only after projection; F16 halves memory with no accuracy impact.

  6. Replace recursive FFT (core/fft.h) with iterative — recursive variant allocates O(N log N) heap per call. kaldi_fbank.cpp already has the correct iterative in-place version. Also: CosyVoice3 HiFT DFT is O(n²) — must use FFT.

Tier 3 — Targeted wins

  1. FireRed ASR: add KV cache for decoder self-attention — currently grows a vector and does O(T²) scalar attention. Highest-impact single- backend optimization remaining.

  2. Kyutai STT RVQ encode: vectorized search or PQ — brute-force O(T×32×2048×256) codebook search is the dominant cost.

  3. Nemotron: ring buffer for streaming KV cache — vector erase from front is O(N); a ring buffer is O(1).

  4. VoxCPM2: fix Metal buffer type mismatch — currently CPU-only due to SIGSEGV from buffer placement. Unlocks GPU for the entire pipeline.

  5. embed_tokens micro-graph elimination — MOSTLY DONE. FunASR, GLM-ASR, MOSS-Audio, Qwen3-ASR, Gemma4-E2B shipped with direct CPU dequant (CRISPASR_XXX_EMBED_FAST). Orpheus/OuteTTS already had it. ~1.6× embed step. Only granite-speech remains.

  6. read_tensor_f32 weight pre-cache — DONE (piper 14%, melotts 16%). CRISPASR_PIPER_WEIGHT_CACHE / CRISPASR_MELOTTS_WEIGHT_CACHE.

VPS bench data (2026-06-20, 4-core CPU, Q4_K/F16, JFK 11s)

Backend Stage Time (ms)
paraformer-zh Q4_K (123 MB) fbank+lfr 142
  encoder 5007
  cif_predict 461
  decoder 522
nemotron Q4_K (458 MB) mel 143
  encoder 31411
  rnnt_decode 7838
canary-ctc Q4_K (434 MB) encoder+ctc 43235
piper-en F16 (30 MB) text_encoder 939
  flow_inverse 5187
  hifigan_decode 5487
  total 11790
melotts-en F16 (98 MB) text_encoder 407
  flow_inverse 2579
  hifigan_decode 17947
  total 26272
This site is open source. Improve this page.