CrispASR

Architecture

CrispASR is structured around three layers on top of whisper.cpp. The split between src/ (library) and examples/cli/ (presentation) is deliberate: every algorithm — VAD, diarization, LID, CTC alignment, HF download/cache, model registry — lives in src/ behind a stable C-ABI (src/crispasr_c_api.cpp), and every consumer (CLI, Dart, Python, Rust, Go, Java, Ruby) reaches it through the same symbols. The CLI keeps only presentation + UX policy.

┌───────────────────────────────────────────────────────────────────┐
│ examples/cli/cli.cpp (the crispasr binary)                        │
│   Parses CLI args, dispatches to backend when --backend           │
│   is set or GGUF arch is non-whisper; otherwise runs whisper_full │
│   unchanged                                                        │
├───────────────────────────────────────────────────────────────────┤
│ examples/cli/crispasr_*_cli.{h,cpp}                               │
│   Thin CLI shims for policy only — auto-download, TTY prompts,    │
│   sherpa-ONNX subprocess fallbacks. Delegate the algorithmic      │
│   work to the shared library below.                                │
├───────────────────────────────────────────────────────────────────┤
│ src/crispasr_c_api.cpp — C-ABI (shared with Dart / Python / Rust) │
│   crispasr_vad.{h,cpp}           Silero VAD + whisper-style       │
│                                  stitching, timestamp remap       │
│   crispasr_diarize.{h,cpp}       energy / xcorr / vad-turns /     │
│                                  native pyannote diarization      │
│   crispasr_speaker_embedder.{h,cpp}                                │
│                                  pluggable speaker-embedding      │
│                                  interface; TitaNet + IndexTTS    │
│                                  adapters                          │
│   crispasr_speaker_cluster.{h,cpp}                                 │
│                                  agglomerative cosine clustering  │
│                                  for globally stable speaker IDs  │
│   crispasr_lid.{h,cpp}           whisper-tiny + silero-native LID │
│   crispasr_aligner.{h,cpp}       canary-CTC + qwen3-forced-aligner│
│   crispasr_cache.{h,cpp}         HF download + ~/.cache/crispasr  │
│   crispasr_model_registry.{h,cpp} backend → canonical GGUF URL    │
├───────────────────────────────────────────────────────────────────┤
│ src/{crispasr,parakeet,canary,canary_ctc,cohere,qwen3_asr,        │
│      voxtral,voxtral4b,granite_speech,silero_lid,pyannote_seg,    │
│      lid_fasttext,lid_cld3,text_lid_dispatch}.cpp + ~65 more      │
│   Per-model runtimes (public C APIs). `crispasr.cpp` is the       │
│   whisper runtime (public header `include/crispasr.h`).           │
├───────────────────────────────────────────────────────────────────┤
│ src/core/      — shared model primitives (crispasr-core)          │
│   ~118 headers; the four load-bearing ones:                       │
│   mel.{h,cpp}          log-mel spectrogram (NeMo + HF clusters)   │
│   ffn.h                SwiGLU + SiLU FFN helpers                  │
│   attention.h          Llama-style self-attention + flash-attn    │
│   gguf_loader.{h,cpp}  Unified GGUF open / weight mmap / lookup   │
├───────────────────────────────────────────────────────────────────┤
│ ggml                                                               │
└───────────────────────────────────────────────────────────────────┘

src/ — shared library surface

Every algorithm listed below is exposed as extern "C" functions with a crispasr_ prefix. The CLI, Python, Rust, and Dart bindings all consume the same symbols.

File Role
crispasr_c_api.cpp The C-ABI. Exports session open/close/transcribe, VAD, diarize, LID, alignment, cache, registry — everything a wrapper needs.
crispasr_vad.{h,cpp} Silero VAD slicing + whisper-style stitching with timestamp remapping. Used by crispasr_session_transcribe_vad.
crispasr_diarize.{h,cpp} Five diarizers (CrispasrDiarizeMethod): energy (stereo), xcorr (stereo, TDOA), vad-turns (mono, timing), pyannote (mono, GGUF; #107 added cross-slice cache + segment splitting + overlap-aware scoring), foxnose (WeSpeaker embeddings + spectral clustering, #324 — see below). Both pyannote and sherpa/ecapa now run once globally on the full audio (#110), producing consistent speaker IDs across VAD slices.
crispasr_speaker_embedder.{h,cpp} Pluggable speaker-embedding interface (CrispasrSpeakerEmbedder base class + factory). Concrete adapters: TitaNet-Large (192-d, 16 kHz) and IndexTTS-BigVGAN ECAPA-TDNN (512-d, internally resamples 16→24 kHz). Add a third by subclassing and extending the factory dispatch.
crispasr_speaker_cluster.{h,cpp} Agglomerative single-linkage cosine clustering on speaker embeddings, with both a similarity-threshold stop and a hard max_speakers cap. Drives --diarize-embedder’s remap of pyannote-local track IDs into globally stable speaker IDs.
crispasr_lid.{h,cpp} whisper-tiny + silero-native audio-LID with process-wide whisper-context cache.
lid_fasttext.{h,cpp} Text-LID runtime for fastText supervised models — GlotLID-V3 (flat softmax, 2102 ISO 639-3 + script labels) and Facebook LID-176 (hierarchical softmax, 176 ISO 639-1 codes). Pure manual F32/F16 + on-the-fly dequant; no ggml graph.
lid_cld3.{h,cpp} Text-LID runtime for Google CLD3 — six feature extractors (4× cbog, RelevantScript, ScriptFeature) → 80-d concat → FC + ReLU → 208-d hidden → FC → softmax over 109 ISO 639-1 labels. Pure manual F32 forward.
text_lid_dispatch.{h,cpp} Backend-agnostic façade over lid_fasttext and lid_cld3. Peeks general.architecture at load time and dispatches to the matching backend; one C ABI for any text-LID GGUF. Powers crispasr-lid and --lid-on-transcript.
crispasr_aligner.{h,cpp} canary-CTC + Qwen3-ForcedAligner + wav2vec2 forced alignment behind one entry point; filename-based dispatch. Also the engine behind --align-only (standalone alignment without ASR, issue #217).
crispasr_cache.{h,cpp} WinHTTP / curl / wget download into ~/.cache/crispasr/; zombie-file detection.
crispasr_model_registry.{h,cpp} Backend → canonical GGUF URL table; exact default-bundle enumeration (primary, companion, extras, licence policy); fuzzy filename lookup for “did you mean …?” hints.

examples/cli/ — presentation + policy

File Role
cli.cpp crispasr entry point, extended with --backend dispatch branch.
crispasr_backend.{h,cpp} CrispasrBackend abstract class, capability bitmask, factory, GGUF auto-detect (crispasr_detect_backend_from_gguf).
crispasr_backend_*.cpp (75 files) Per-backend thin wrapper over each model’s C API — one file per backend, e.g. crispasr_backend_parakeet.cpp, crispasr_backend_crispasr.cpp (the whisper adapter). ASR backends emit crispasr_segments; TTS backends (vibevoice, qwen3_tts, orpheus, kokoro, chatterbox, moss_tts, miotts, …) implement synthesize(text) instead and write 24 kHz mono WAV via --tts-output; the translation backends (m2m100 for facebook m2m100 + WMT21, t5 for MADLAD-400 / future T5 translation) implement translate_text(text, src, tgt) and write UTF-8 to stdout; non-transcribe task backends (htdemucs, mel_band_roformer, crepe, btc, tabcnn, beat_this, piano_transcription) go through their own early CLI dispatchers (see below). ls examples/cli/crispasr_backend_*.cpp is the live list.
whisper_params.h Shared params struct (extracted from cli.cpp, extended).
crispasr_output.{h,cpp} TXT / SRT / VTT / CSV / JSON / LRC writers on crispasr_segment.
crispasr_vad_cli.{h,cpp} Delegates to src/crispasr_vad; adds auto-download for the Silero GGUF.
crispasr_lid_cli.{h,cpp} Delegates to src/crispasr_lid; adds auto-download + sherpa-ONNX subprocess fallback.
crispasr_diarize_cli.{h,cpp} Delegates to src/crispasr_diarize; adds sherpa subprocess fallback + pyannote GGUF auto-download. CrispasrSherpaCache (#110) pre-computes the global sherpa timeline; assign_speakers_from_global_sherpa() assigns + splits segments at speaker turns. Diarization runs after external CTC alignment (#267) so word timestamps are available for speaker-turn splitting; without words it falls back to segment-level dominant-speaker assignment.
crispasr_model_mgr_cli.{h,cpp} Delegates to src/crispasr_model_registry; adds “Download now? [Y/n]” prompt on TTY.
crispasr_aligner_cli.{h,cpp} Adapter converting CrispasrAlignedWord → the CLI’s crispasr_word shape.
crispasr_server.cpp HTTP server for the persistent-model mode + OpenAI-compatible endpoints.
crispasr_llm_pipeline.h Templated audio-LLM pipeline (mel → encoder → prompt → KV decode).
crispasr_run.cpp Top-level pipeline dispatch: resolve → detect → load → slice → transcribe → align → diarize → merge → cluster → Speaker DB → write (#267: align before diarize).

src/core/ — the shared model primitives

Duplicated scaffolding is bundled in a single static library, crispasr-core, linked into every model target (crispasr-lib, the whisper runtime, links it too — but only for the non-numeric helpers; it keeps its own mel / attention / GGUF code, see below).

Consumer counts below are direct #include "core/<x>.h" in src/*.cpp as of v0.8.30; representative names are listed, not the full set. grep -rl '#include "core/mel.h"' src/*.cpp is the live answer.

Header Replaces Consumers
core/mel.{h,cpp} 7× copy-pasted STFT + mel filterbank + log + norm 30 — parakeet, canary, canary_ctc, cohere, voxtral, voxtral4b, qwen3asr, granite_speech, granite_nle, gigaam, nemotron, ark_asr, higgs_stt, moss*, glm_asr, lfm2_audio, …
core/ffn.h 4× inline SwiGLU blocks 36 — qwen3asr, voxtral, voxtral4b, granite_speech, granite_nle, chatterbox, orpheus, vibevoice, moss*, omniasr, mimo_asr, csm_tts, …
core/attention.h Llama-style self-attention with NEOX RoPE + GQA + flash-attn 42 — voxtral, voxtral4b, granite_speech (via core_granite_llm), canary, cohere, qwen3_asr, kokoro, moonshine, kyutai_stt, zonos_tts, …
core/gguf_loader.{h,cpp} 8× identical two-pass GGUF load + mmap + tensor-map build 99 — effectively every non-whisper model
core/fft.h Radix-2 Cooley-Tukey FFT (4× duplicated) 17 — granite_speech, granite_nle, gigaam, sidon, htdemucs, chatterbox_{ve,s3tok,campplus}, indextts, piano_transcription, beat_this, … (≈28 models still carry a local FFT)
core/cpu_ops.h CPU LayerNorm + matmul fallbacks (when no GPU sched is available) 17 — granite_speech, granite_nle, canary, cohere, parakeet, sidon, sensevoice, piper_tts, moonshine_streaming, …
core/ctc.h posterior_weighted_pool + greedy_decode_with_blank granite_nle, parakeet, sensevoice, wav2vec2-ggml
core/fastconformer.h NeMo-style FastConformer block (conv subsampling + MHA RPE) parakeet, canary, canary_ctc, canary_qwen, gigaam, nemotron, indextts, firered_asr, lfm2_audio
core/conformer_ibm.h IBM Macaron Conformer block (FFN + Shaw RPE attn + conv module + FFN + Shaw lookup) — sibling of fastconformer.h, intentionally not merged granite_speech, granite_nle
core/granite_llm.h Granite-1B 40-block backbone (RMSNorm + GQA(16/4) flash-attn + RoPE + SwiGLU + µP residual scale); is_causal flag picks KV-cached prefill+decode (core_attn::kv_self_attn) vs non-causal flash (whole-sequence editing) granite_speech, granite_nle
core/qformer.h Windowed simplified Q-Former: pass A (LayerNorm + concat + linear + GELU) and per-window cross-attn + MLP cgraph builder granite_nle (NAR-only — granite_speech uses a different full BLIP-2 Q-Former)
core/bpe.h GPT-2 byte-level BPE encode + decode 31 — granite_speech, granite_nle, voxtral, qwen3asr, glm_asr, ark_asr, chatterbox, orpheus, moss*, outetts, …
core/greedy_decode.h Autoregressive greedy decode loop with EOS handling gemma4_e2b (+ crispasr_c_api.cpp); the other audio-LLM backends still carry their own decode loops
core/sanm.h FunASR SANM encoder block (MHA + FSMN depthwise conv) funasr, sensevoice, paraformer
core/asr_context_bias.h Aho-Corasick CTC-WS phrase-boost trie for --hotwords (#98). Per-beam state in TDT/RNNT beam search parakeet (CTC + TDT greedy + TDT beam); extensible to any CTC/TDT backend

core_mel::Params spans both algorithm clusters: the NeMo family (ln + per-mel z-score + (T, n_mels) layout) and the HF/Whisper family (log10 + global clip normalization + (n_mels, T) layout), with knobs for LogGuard (add-epsilon vs max-clip), MatmulPrecision (Float vs Double), FbLayout (MelsFreqs vs FreqsMels), drop_last_frame / drop_first_frame_if_odd, and pad_to_T.

core_gguf::WeightLoad owns the ggml_context, the ggml_backend_buffer_t, and the std::map<std::string, ggml_tensor*> in one struct that models std::move() into their own state. The mmap path has a pread / fseek fallback for filesystems that don’t support mmap.

Whisper is the reference implementation

src/crispasr.cpp is intentionally not migrated to src/core/ (yet) — it’s (for the time being) the battle-tested reference and the crispasr -m ggml-base.en.bin … code path is byte-identical to upstream whisper.cpp. This guarantee is a test gate: every CrispASR commit that touches the CLI is checked against it. It does pull three non-numeric core headers (core/gpu_backend_pref.h, core/whisper_special_tokens.h, core/ggml_cpu_backend.h) but none of core/mel.h, core/ffn.h, core/attention.h or core/gguf_loader.h.

Regression discipline

Every src/core/ migration commit includes a md5sum-level regression test against samples/jfk.wav:

Backend internals

Note: the snapshot below was last hand-edited in early 2026 and is not regenerated from the registry — treat it as a sketch, not ground truth. The authoritative source for what’s compiled in, per-backend GPU support, and current capability bits is src/crispasr_model_registry.cpp and the capabilities() returned by each adapter in examples/cli/crispasr_backend_*.cpp.

Backend Arch pattern ggml graph Flash attn KV cache GPU Shared core modules
whisper Enc-dec transformer ✔ ✔ ✔ CUDA / Metal / Vulkan (upstream)
parakeet FastConformer + TDT ✔ ✔ partial CPU mel, fastconformer, gguf_loader, ctc, cpu_ops, asr_context_bias
canary FastConformer + Transformer dec ✔ ✔ ✔ CUDA / Metal mel, fastconformer, gguf_loader, attention, cpu_ops
canary-qwen FastConformer + Qwen3-1.7B SALM ✔ ✔ ✔ CUDA mel, fastconformer, gguf_loader, kv_self_attn, swiglu, bpe
cohere Conformer + Transformer dec ✔ ✔ ✔ CUDA / Metal mel, gguf_loader, attention, cpu_ops
granite Conformer + Q-Former + LLM ✔ ✔ ✔ CPU mel, gguf_loader, kv_self_attn, swiglu, bpe, conformer_ibm, granite_llm, cpu_ops, fft
voxtral Whisper enc + Mistral LLM ✔ ✔ ✔ CUDA / Metal mel, gguf_loader, kv_self_attn, encoder_self_attn, swiglu, bpe
voxtral4b RoPE enc + 3.4 B LLM ✔ ✔ ✔ CUDA / Metal mel, gguf_loader, kv_self_attn, encoder_self_attn, swiglu, bpe
qwen3 Whisper enc + Qwen3 LLM ✔ ✔ ✔ CUDA / Metal mel, gguf_loader, kv_self_attn, swiglu, bpe
fc-ctc FastConformer + CTC ✔ ✔ — CPU mel, fastconformer, gguf_loader
wav2vec2 CNN + Transformer + CTC ✔ — — CUDA / Metal gguf_loader, ctc
glm-asr Whisper enc + Llama LLM ✔ ✔ ✔ CPU mel, gguf_loader, kv_self_attn, swiglu, bpe, beam_decode
kyutai-stt Mimi codec + causal LM (1B + 2.6B) ✔ ✔ ✔ CPU gguf_loader, attention, rvq, beam_decode
firered-asr Conformer + CTC + beam dec ✔ ✔ ✔ CPU gguf_loader, fastconformer, cpu_attention, repeat_break (its own mel, not core/mel.h)
moonshine Conv + 6L enc-dec ✔ ✔ ✔ CPU gguf_loader, attention, beam_decode, repeat_break
moonshine-streaming Sliding-window enc + dec ✔ ✔ ✔ CPU gguf_loader, cpu_ops, beam_decode
omniasr wav2vec2 enc + CTC / LLM ✔ ✔ CTC: — / LLM: ✔ CPU gguf_loader, kv_self_attn, swiglu, cpu_ops, beam_decode
gemma4-e2b Conformer enc + Gemma4 LLM ✔ ✔ ✔ CUDA / Metal mel, gguf_loader, kv_self_attn, swiglu, bpe, greedy_decode, beam_decode
mimo-asr wav2vec2 enc + Qwen2 LM ✔ ✔ ✔ CUDA / Metal gguf_loader, kv_self_attn, swiglu, bpe
ark-asr ⚠️exp/WIP Whisper enc (partial RoPE) + Qwen2.5-3B LM ✔ ✔ ✔ GPU default (Metal-validated; CRISPASR_ARKASR_CPU=1 forces CPU) mel, ffn, gguf_loader, kv_self_attn, swiglu, bpe
vibevoice σ-VAE + Qwen2 (ASR) / TTS LM (synth) ✔ ✔ ✔ CUDA / Metal gguf_loader, attention, ffn, bpe, conv
kokoro StyleTTS2 BERT + ProsodyPredictor + iSTFTNet ✔ — — CPU gguf_loader, attention, conv, lstm, dac_decoder, align
qwen3-tts Qwen3 talker + 12 Hz codec + code-predictor ✔ ✔ ✔ CUDA / Metal gguf_loader, kv_self_attn, swiglu
orpheus Llama-3.2 talker + SNAC RVQ codec ✔ ✔ ✔ CUDA / Metal gguf_loader, kv_self_attn, swiglu
chatterbox T3 (Llama / GPT-2) + S3Gen (Conformer + UNet1D CFM + HiFTGen) ✔ ✔ ✔ CUDA / Metal gguf_loader, kv_self_attn, swiglu, fft
zonos-tts 26L GQA transformer + 9-codebook DAC @ 44.1 kHz; CFG; voice cloning from WAV ✔ ✔ ✔ CUDA / Metal gguf_loader, kv_self_attn, dac_decoder (its GatedMLP is local, not core_ffn)
dots-tts Qwen2.5-1.5B LLM + 24L VAESemanticEncoder + 18L DiT flow-matching head (CFG Euler) + BigVGAN @ 48 kHz; continuous-latent AR; CAM++ voice cloning; incremental streaming PatchEncoder ✔ mixed (DiT must stay F16; LLM+penc Q8) ✔ Metal gguf_loader, kv_self_attn, swiglu, lstm, snake_beta (core/activation.h), adaln, conv, cpu_ops, audio_resample
fireredtts3 Qwen3-1.7B LLM (QK-norm, NEOX RoPE 1e6) + 8L CLS PatchEncoder + 11L AdaLN DiT flow head (+Conv1d branch, 10-step cosine Euler, CFG 2.0, sigmoid stop head) over continuous 64-d 25 Hz RedAE latents; RedAE = 18L sliding-window-64 Qwen3 encoder + CLS 2x downsample + 18L Qwen3 decoder + Vocos ISTFT head (n_fft 1920/hop 480, shipped window) @ 24 kHz; CAM++ 512-d ICL voice cloning (--voice ref.wav --ref-text), default English prompt baked into the core GGUF ✔ mixed (DiT+penc+RedAE stay F16; LLM Q4_K/Q8) ✔ — gguf_loader, bpe (qwen_pretokenize), istft, torch_rng, kaldi_fbank via chatterbox_campplus, audio_resample
m2m100 facebook/m2m100 12L+12L transformer (text-to-text translation; WMT21 4.7B variant via --backend m2m100-wmt21) ✔ — ✔ (cross-attn) CUDA / Metal gguf_loader, sentencepiece, beam_decode
madlad / t5 T5 encoder-decoder (MADLAD-400 12L+12L, gated-GELU, RMSNorm, bucketed rel-pos bias). Tokens match Python SP bit-by-bit; translation outputs match the HF reference. ✔ — ✔ (cross-attn) CUDA / Metal gguf_loader, beam_decode, repeat_break

Architecture families

Non-transcribe task surfaces

Not every backend produces text or synthetic speech. A task whose result is not crispasr_segments does not get layered onto transcribe() — it gets its own early CLI dispatcher, hit before the ASR backend is constructed, and its own capability bit (used for detection and help text, never for routing). docs/source-separation-surface.md settled this design; everything below follows it.

These are steps in the same music-transcription chain (separate → F0 → chords/beats → notes/tab), which is why they share the “early dispatcher, own result type” shape. Note that CAP_SEPARATE and CAP_STREAMING briefly collided on bit 22; streaming now owns bit 23.

Optimization opportunities

Per-backend architecture details

Detailed architecture notes for backends whose design warrants more than a one-line summary. The README backend table links here for each entry.

granite / granite-4.1 / granite-4.1-plus / granite-4.1-nar

granite (granite-speech-{3.2-8b, 3.3-2b, 3.3-8b}, granite-4.0-1b-speech): Conformer encoder + BLIP-2 Q-Former + Granite LLM (μP scaling).

granite-4.1 (granite-speech-4.1-2b): Same architecture as 4.0 (16-layer Conformer + Q-Former + Granite LLM); “2B” = full system. Encoder runs as a single ggml graph by default with per-layer Shaw RPE in attention (PLAN #16) — bit-near-identical to the per-op CPU loop, ~2.1× faster end-to-end on M1+Q4_K. CRISPASR_GRANITE_DISABLE_ENCODER_GRAPH=1 falls back to the CPU loop.

granite-4.1-plus (granite-speech-4.1-2b-plus): 4.1 + 2-layer encoder hidden-state concatenation (1024+1024=2048 projector input); emits punctuated / capitalised transcripts by default. cat_hidden_layers post-norm tensors are captured inline in the graph and ggml_concat-ed with the final encoder output, so PLUS rides the GPU path too (~2.5× end-to-end on M1+Q4_K).

granite-4.1-nar (granite-speech-4.1-2b-nar): 4.1 with non-autoregressive decoder — single LLM forward over [audio, text+slots]

kokoro

StyleTTS2 / iSTFTNet (BERT + ProsodyPredictor + iSTFTNet decoder, 82M params); per-voice GGUF; in-process libespeak-ng phonemizer with LRU cache; auto-routing for -l de swaps in the German-trained backbone + cascading voice fallback.

Models: hexgrad/Kokoro-82M

orpheus

Llama-3.2-3B-Instruct talker (28L, 3072 d) + SNAC RVQ codec (3 codebooks × 4096 @ 24 kHz); 8 baked English speakers (tara/leah/leo/…). Pick the speaker with --voice <name> and pass --temperature 0.6 (engine_class.py default — greedy loops).

Drop-in DE checkpoint variants:

chatterbox / chatterbox-turbo / chatterbox-nano / chatterbox-finnish-nano / kartoffelbox-turbo / lahgtna-chatterbox

Two-GGUF runtime: T3 AR text→speech-tokens + S3Gen flow-matching speech-tokens→24 kHz waveform.

T3 (Text-to-Tokens): Llama-30L for base/lahgtna, GPT-2-24L for turbo/kartoffelbox-turbo, and GPT-2-small for Nano and its Finnish fine-tune.

S3Gen (Tokens-to-Speech): UpsampleConformerEncoder + UNet1D CFM + HiFTGenerator vocoder. Turbo uses 2-step meanflow CFM (vs 10-step cosine for base). Default voice baked into T3 (conds.*); voice cloning goes through models/bake-chatterbox-voice-from-wav.py, which runs upstream prepare_conditionals(wav) (VoiceEncoder LSTM → 256-d speaker emb, CAMPPlus TDNN → 192-d x-vector, S3Tokenizer → prompt tokens, 24 kHz mel extractor → prompt mel) and writes a small voice GGUF (~150-200 KB) using the same tensor names the runtime already accepts for the built-in voice. --voice <voice.gguf> then loads it via chatterbox_load_voice_gguf into a separate voice_ctx_w / voice_buf_w and rebinds ctx->conds.* pointers, leaving the original baked-in default-voice tensors allocated but unreferenced. In-process WAV → cond extraction is fully ported across four modules — VE (src/chatterbox_ve.cpp), S3Tokenizer V2 (src/chatterbox_s3tok.cpp), CAMPPlus + Kaldi fbank (src/chatterbox_campplus.cpp + src/core/kaldi_fbank.{h,cpp}), and 24 kHz Matcha mel (in chatterbox_campplus.cpp) — all verified bit- or fp32-rounding-tight against PyTorch via crispasr-diff chatterbox. A polyphase Kaiser-windowed sinc resampler (src/core/audio_resample.{h,cpp}) handles the 16 ↔ 24 kHz conversion. Both 16 and 24 kHz .wav references trigger atomic cloning: all five conditionals are derived from one source, with the missing sample rate generated by the shared resampler. This avoids the former 16 kHz T3-only path, which silently rendered with S3Gen’s default voice and therefore did not actually clone the reference speaker. Two known issues affect end-to-end output: F16 T3 + GPU produces broken audio (pre-existing bug; use Q4_K + --no-gpu until fixed), and T3 sampling can drift on long prompts. The python baker workflow remains the recommended path for production-quality cloning. S3Gen GGUF is auto-discovered next to T3 or passed via --codec-model. See docs/tts.md for the workflow.

For multilingual base Chatterbox, -l <code> sets chatterbox_set_language() in the CLI adapter. The runtime validates the [code] token against the embedded tokenizer and prepends it to the text token sequence before the [STOP]-wrapped T3 prompt is built. The versioned chatterbox-v3-* artifacts use the paired grapheme_mtl_merged_expanded_v1.json tokenizer (2454 T3 text vocab entries, 2454 tokenizer tokens, 265 merges); older mismatched GGUFs are rejected at load time. They pin ResembleAI revision 5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18 and the production V3 pair (t3_mtl23ls_v3 + original S3Gen). A Q4_K smoke check confirmed -l fr inserts [fr] (id 634) and changes the generated speech tokens and waveform.

Variants:

Conformer rel-pos parity gap closed in §80 — encoder_out now bit-exact to Python reference.

zonos-tts

Zyphra Zonos-v0.1-transformer: 26-layer GQA transformer (d=2048, n_heads=16, n_kv=4, head_dim=128) conditioned on speaker embedding + phoneme tokens → 9-codebook DAC codes @ 86 Hz → 44.1 kHz PCM via the DAC decoder GGUF.

Conditioning prefix: speaker embedding (512-d float32 from a reference WAV, encoded externally or via --voice <ref.wav>) is projected through an MLP and prepended as prefix tokens before the phonemised text. CFG (classifier-free guidance) runs a conditioned path and an unconditioned path in parallel, blending with cfg_scale=2.0: uncond + 2*(cond − uncond).

Backbone: standard pre-norm transformer with RMSNorm, GatedMLP (fc2(y * silu(gate)), first chunk = y, second = gate), and consecutive-pair RoPE (GGML_ROPE_TYPE_NORMAL) matching x_transformers’ apply_rotary_emb (reshape to (…, d/2, 2), rotate paired elements). GQA with 4 KV heads shared across 16 query heads.

AR decode: 9-codebook delay pattern (codebook k shifted by k+1 positions). Each step samples one token per codebook via min-p=0.1 + temperature=1.0 + repetition penalty (factor=3.0, window=2). EOS is only predicted on codebook 0; other codebooks have EOS masked to −∞.

Quantisation: selective quantization is required. Uniform Q4_K inflates the EOS logit at AR step 0 by ~0.9 units (−1.125 → +0.21), making P(EOS) > 60 % and causing synthesis to fail on every seed. The crispasr-quantize tool keeps heads.*, embeddings.*, and prefix_conditioner.* at F16 while quantizing the 210 backbone projections — reducing the model to 931 MB (vs 872 MB for full-Q4_K). A 3-retry guard in the runtime handles residual step-0 failures (~25 % of seeds, 100 % resolved within 2 retries). Default via -m auto is Q8_0 (1.6 GB); Q4_K (931 MB) is safe with the above caveats; F16 (3.0 GB) is the reference.

DAC codec: the companion dac-44khz-f16.gguf (104 MB) is a purely convolutional architecture — all weight tensors have kernel-size ≤ 16 as ne[0], making block quantization impossible. F16 is the only usable quant.

Voice cloning: pass --voice <ref.wav> at the CLI or set CRISPASR_ZONOS_SPEAKER_EMB_PATH=/path/to/jfk_speaker_emb.bin (raw float32, 512 floats). The runtime calls zonos_tts_set_voice(ctx, path) which decodes the WAV via src/core/audio_resample and runs the VoiceEncoder MLP to produce the speaker embedding.

GGUFs: cstr/zonos-v0.1-transformer-GGUF (AR transformer) + cstr/dac-44khz-GGUF (DAC decoder, auto-discovered as a sibling or via --codec-model).

omniasr (CTC + LLM + Unlimited)

wav2vec2-style CNN frontend (7 layers, stride 5+2×6=320) + 24–48L transformer encoder + either CTC head or 12L LLaMA decoder (SwiGLU, RoPE, d=4096, 8 heads).

CTC variant (omniasr, omniasr-300m): greedy argmax with CTC blank collapse (blank = token 0 for both v1/fairseq2 and v2/HF formats). Only v2 (HF transformers) GGUFs work — the v1 fairseq2 .pt format produces empty output because fairseq2’s model loader applies weight transforms we cannot replicate. The 300M model’s positional encoding degrades beyond ~7 s; the runtime auto-chunks long audio into 5 s segments. The 1B model handles all lengths.

LLM variant (omniasr-llm-300m-v2): Encoder projection (1024→4096)

Unlimited variant (omniasr-llm-unlimited-300m-v2): Same architecture but trained with a streaming segment-token protocol. Audio is split into 15-second segments; each segment is decoded independently with a segment marker token that signals whether more audio follows. Three special tokens above vocab_size in tok_emb: streaming_lang (lid marker), last_segment, regular_segment. Auto-detected at load time from tok_emb shape (vocab_size + 3). Supports arbitrarily long audio input.

vibevoice

σ-VAE ConvNeXt encoders + Qwen2.5-7B decoder. Dual-mode: ASR (with timestamps, diarization, hotwords) and TTS (DPM-Solver++ flow matching).

The vibevoice-streaming checkpoint uses the same encoder family with a Qwen2.5-1.5B decoder and a different protocol. The prompt is prefixed once, then each 2.93-second audio chunk is encoded with 0.53 seconds of right lookahead. Speech features, generated text tokens, and a chunk-end token all advance one persistent decoder KV cache. vibevoice_stream_feed() retains the lookahead between calls and only pads the final partial chunk, so CLI and C ABI streaming do not re-run a growing audio window.

The ASR answer is JSON, and the adapter parses it. The system prompt says “transcribes audio input into text output in JSON format” and the user turn asks for the keys Start time, End time, Speaker ID, Content, so the model replies with an array of utterances:

[{"Start":0.0,"End":10.99,"Speaker":0,"Content":"…"},
 {"Start":11.32,"End":15.65,"Speaker":1,"Content":"…"}]

core_vibevoice::parse (src/core/vibevoice_transcript.h) turns that into one crispasr_segment per utterance — timings from Start/End offset into the chunk, speaker as the structured "(Speaker N) " label — which is what makes the diarization reachable in file output, --stream, --stream-json and the bindings. Until v0.8.24 the whole blob was one segment’s text (#300), so the labels were literal JSON and seg.speaker was never set.

It is a deliberately tolerant scanner rather than a strict JSON reader: a decode that hits the token cap ends mid-array, and a strict parse would discard every complete utterance before the cut. Unparseable output falls back to the raw string, and CRISPASR_VIBEVOICE_RAW_TRANSCRIPT=1 restores the pre-v0.8.24 single-segment behaviour for callers that parse the blob themselves. Because the model punctuates and sentence-cases its own Content, the adapter declares CAP_PUNCTUATION_NATIVE so the CLI’s FireRedPunc pass does not run over it.

mimo-asr

6L input_local_transformer (1024d) + 36L Qwen2 LM (4096d, 32Q/8KV); 8-channel RVQ codes from separate MiMo-Audio-Tokenizer GGUF. Mandarin (Wu/Cantonese/Hokkien/Sichuanese dialects) + English + code-switching.

Tokenizer is a separate file. --auto-download fetches both the LM (cstr/mimo-asr-GGUF) and the tokenizer (cstr/mimo-tokenizer-GGUF) into ~/.cache/crispasr/; the runtime auto-discovers mimo-tokenizer-q4_k.gguf next to the LM. Override with --codec-model PATH/mimo-tokenizer-q4_k.gguf if you keep the tokenizer elsewhere.

ark-asr

⚠️ Experimental / WIP. Validated verbatim on English + German (Q8_0, on both GPU and CPU), but rough edges remain (see below). Single self-contained GGUF.

AutoArk-AI/ARK-ASR-3B — 19-language ASR = Whisper-large-v3 encoder with partial interleaved RoPE (rot_dim 32 of head_dim 64, θ=10000, applied to Q+K only; k_proj has no bias; the encoder’s own final LayerNorm is dropped) + MLP adapter (LayerNorm → merge 4 consecutive frames → Linear 5120→4096 → GELU → Linear 4096→2048) + stock Qwen2.5-3B decoder (2048d, 36L, GQA 16Q/2KV, SwiGLU, RMSNorm, θ=1e6, tied embeddings). The <|audio|> (151663) placeholder embeddings are overwritten by the first N = ((mel_frames+1)//2)//4 adapter frames; mel is the stock WhisperFeatureExtractor recipe (128-bin, n_fft 400, hop 160). Convert with models/convert-arkasr-to-gguf.py (--outtype f16|q8_0); build Q4_K from the F16 with crispasr-quantize (the ark-asr rule keeps encoder/adapter/embeddings F16).

Validated against the original PyTorch model (trust_remote_code, bf16) via crispasr-diff arkasr on jfk: log-mel cos 0.999993, first decoder logits (Q8_0) cos 0.999646, audio-embeds mean cos 0.999445. Transcript verbatim (en + de). GGUFs published at cstr/ark-asr-3b-GGUF (f16 / q8_0 / q4_k).

Single-pass whole-audio (matches the reference): the RoPE encoder has no positional cap, so the whole clip is one encoder pass + one decode. Long audio falls back to internal 30 s chunking only above CRISPASR_ARKASR_MAX_SINGLE_PASS_S (default 300 s; 0 = never) to bound O(T²) encoder compute / decode length. The transcript-opening . token the model emits (the reference shows it too) is stripped in the output cleanup.

Known limitations (WIP):

bosonai/higgs-audio-v3-stt: Whisper-large-v3 audio encoder → depthwise-temporal-conv projector → Qwen3-1.7B-Base decoder. A <|AUDIO|> placeholder in a ChatML prompt (Transcribe the speech. Output only the spoken words in lowercase with no punctuation.) is replaced by the projected audio embeddings; the LLM greedily decodes the transcript, then a <think>…</think> strip + n-gram-loop collapse (ngram_loop_fix.py) clean it up.

Chunked encoder (the load-bearing detail). The model does not encode the clip as one padded 30 s Whisper window. It splits the waveform into chunk_size_seconds (4 s = 64000-sample) chunks — at inference vad_cut degenerates to a fixed ceil(total / 64000) split, last chunk = remainder — and encodes each chunk independently (chunk-local embed_positions[:T_enc], within-chunk attention) through the Whisper tower + AvgPool + projector, then concatenates the per-chunk audio embeds. Encoding one global window corrupts the conditioning (every valid frame attends to ~1900 silence-pad frames) and derails the decoder mid-sentence. Each chunk is encoded at its true length — the per-chunk GlobalClipMax norm is dominated by speech, so this matches the reference’s zero-padded+masked chunk for the valid frames, and the valid token count (((L-1)/2+1 - 2)/2+1 - 1)/2+1 (conv2 s2 → avgpool s2 → temporal-conv s2) falls out exactly. The backend declares CAP_INTERNAL_CHUNKING so the whole clip decodes in a single AR pass (no CLI window-splitting); the tied token_embd/output lm_head stays F16 in every quant. Validated verbatim against transcribe.py (bf16) on jfk and a 45 s / 12-chunk clip.

moss-audio

32-layer Whisper-style audio encoder (1280d, 20 heads, 128-mel, sliding-window attention w=100) with DeepStack 3-tap cross-layer injection + 36-layer Qwen3 LM (2560d, 32Q/8KV, SwiGLU, RoPE θ=1M). Apache-2.0. First audio-understanding backend — supports transcription, audio QA, scene description, and time-aware ASR via prompt.

DeepStack architecture: the encoder captures intermediate outputs at layers 8, 16, and 24. Each tap is projected through an independent GatedMLP (1280→8192→2560) into the LM embedding space. These projections are injected as residual adds at LM blocks 0, 1, and 2, preserving multi-resolution audio features (low-level prosody/transients alongside high-level semantics) through the LM’s early layers.

moss-transcribe

Dedicated ASR sibling of moss-audio (same author). Uses the stock Qwen3-Omni-MoE audio encoder: 128-mel → 3×Conv2d(stride 2, 480ch) → conv_out(7680→1280, no bias) → sinusoidal positions → 32 pre-LN Whisper-style layers (1280d, 20 heads, FFN 5120) with block-diagonal windowed attention (windows of n_window_infer/(2·n_window)=8 conv chunks) → ln_post → proj1(1280→1280)+GELU → proj2(1280→2048). A GatedMLP adapter (2048→8192→2048, SiLU gate) maps encoder frames into the LM embedding space, where they are masked_scatter-spliced into the prompt at audio-placeholder positions. The decoder is a Qwen3-1.7B LM (28L, 2048d, 16Q/8KV, head_dim 128, QK-norm, SwiGLU 6144, RoPE θ=1M, tied embeddings). No DeepStack. Apache-2.0, ~2.4B params. The prompt follows chat_template_default.py (ChatML): <|im_start|>user\n<|audio_start|>·audio·<|audio_end|><|im_end|>\n<|im_start|>assistant\n → transcript → <|im_end|>. The user/assistant framing is required — without it the model emits garbage instead of transcribing. ASR-only; output is lowercase, lightly punctuated.

Audio front-end: 128-bin log-mel → 3×Conv2d (stride 2 each, 8× downsample total) → stem_proj Linear(7680, 1280) → sinusoidal position embeddings → 32 encoder blocks. Slaney mel filterbank normalization. Encoder output padded to 3000 frames (Whisper 30s convention).

Prompt format: Qwen3 chat template with time-marker tokens inserted at fixed intervals between audio frame embeddings. Supports custom prompts via --prompt / set_ask() for audio understanding tasks beyond ASR.

moss-diarize

Joint ASR + speaker diarization + timestamps in a single 0.9B model (OpenMOSS-Team/MOSS-Transcribe-Diarize-0.9B, Apache-2.0). A stock Whisper encoder (80-mel → Conv1d stem → 24L pre-LN transformer, 1024d, 16 heads, global attention → ln_post) feeds into a 4× temporal merge (reshape T/4 × 4096) followed by a VQAdaptor (Linear(4096→1024) + SiLU

Output format: [start_time][Sxx]text[end_time] — each segment has a start/end timestamp in seconds and a speaker label ([S01], [S02], etc.). The CLI adapter parses these into crispasr_segment entries with native timestamps and speaker IDs.

Prompt format: ChatML with a system instruction requesting timestamped, speaker-labelled transcription. Hotwords are injected into the system prompt via 热词提示：word1, word2. The user turn wraps the audio pad sequence between <|audio_start|> and <|audio_end|>.

tiron ⚠️ experimental (#295)

Multi-speaker meeting ASR (Trelis/tiron, Apache-2.0). A drop-in WhisperForConditionalGeneration — Whisper large-v3 (128-mel, 32 enc + 32 dec, 1280d) with an extended vocab (51904): <|speaker1|>..<|speaker8|> (ids 51866–51873, contiguous above the 1501-token timestamp block) + <|nospeech|>. It runs on the whisper backend (alias tiron); the loader auto-detects the speaker tokens (whisper_has_speaker_tokens) and switches the decode.

Decode. Not plain greedy — a port of the harness’s constrained-decoding grammar (whisper_tiron_apply_grammar, from tiron/constraints.py; plain greedy loses ~5 cpWER): step 0 forces <|speaker1|>/<|nospeech|>; a speaker tag forces an opening timestamp; text runs until a closing timestamp; a closing timestamp allows EOS, another opening ts (same speaker continues), or the next speaker slot (speaker_blocks); no_repeat_ngram_size=15. Per-speaker timelines are non-monotonic (a later speaker opens earlier in the window), so the stock whisper “timestamps must increase / don’t go back in time” seek rules are disabled for a speaker vocab. Driven exactly as engine.py: a 0.75 s onset pad, fixed non-overlapping 30 s windows (the whisper adapter declares CAP_INTERNAL_CHUNKING so the CLI passes the whole clip), and an RMS silent- window gate. Validated byte-exact (f16 and q8_0 token stream) vs the Python reference (tools/reference_backends/tiron.py).

Speaker indices are window-LOCAL. crispasr_tiron_link_speakers (src/tiron_link.{h,cpp}) promotes them to meeting-level SPEAKER_NN by clustering per-(window, local-speaker) group voiceprints (TitaNet/ECAPA + agglomerative cosine), with a within-window must-link “spine”. Hoisted into the library (crispasr_tiron_link_transcript) so the CLI and server both apply it; opt-in via --diarize / --diarize-embedder. GGUFs at cstr/tiron-GGML (f16 + q4_k, quantized with crispasr-legacy-quantize — the whisper-bin quantizer).

qwen3-tts

Qwen3 talker LM + 12 Hz RVQ speech tokenizer. Three variants:

piano-transcription

ByteDance/Kong piano_transcription_inference (Apache-2.0) — CRNN-based piano transcription producing MIDI note events (88 keys, 100fps).

miotts

Aratako/MioTTS-0.6B (Apache-2.0) — Qwen3 (28L, 1024d, GQA 16/8) generating speech tokens decoded by MioCodec-25Hz-24kHz (FSQ + transformer + iSTFT → 24kHz). Single GGUF, tokenizer.json loaded at runtime.

moss-tts

OpenMOSS-Team/MOSS-TTS-v1.5 (MossTTSDelay, Apache-2.0) — a Qwen3-8B backbone (36L, 4096d, 32Q/8KV, head_dim 128, QK-norm, SwiGLU, NEOX RoPE θ=1e6) that autoregressively emits 32 RVQ audio codebooks under a delay pattern, decoded by a 1.6B pure-transformer codec to 24 kHz mono.

omnivoice

k2-fsa/OmniVoice (Apache-2.0) — masked iterative multi-codebook TTS. Qwen3-0.6B LLM backbone with:

Supports finetunes: ModelsLab/omnivoice-singing (same architecture).

csm

Sesame CSM-1B (sesame/csm-1b, Apache-2.0), one GGUF, 24 kHz. Three stages run per the original two-transformer + codec design:

GGUF built from the HF-transformers checkpoint (rotate_half layout → NEOX rope). Diff via crispasr-diff csm (backbone per-layer + depth + RVQ vs the manual PyTorch reference).

dia

Nari Labs Dia 1.6B (nari-labs/Dia-1.6B, Apache-2.0, ~1.6B params), single GGUF, 44.1 kHz mono. Dialogue TTS with inline [S1]/[S2] speaker tags. Architecture:

Classifier-Free Guidance (CFG) runs batch=2 (conditional + unconditional); logits = uncond + cfg_scale * (cond - uncond), default cfg_scale=3.0. Sampling: temperature (default 1.2), top-p (0.95), top-k (45), seedable.

speecht5

Microsoft SpeechT5 (microsoft/speecht5_tts, MIT, ~80M params), single GGUF (~300 MB F16), 16 kHz mono. Architecture:

Speaker conditioning requires a 512-d x-vector (e.g. from Matthijs/cmu-arctic-xvectors), passed as raw float32 .bin via --voice. The prenet uses “consistent dropout” at inference in the original; the C++ port omits it (deterministic).

fastpitch

NVIDIA FastPitch (nvidia/tts_en_fastpitch, CC-BY-4.0, ~60M params), single GGUF (~230 MB including HiFi-GAN vocoder), 22 kHz mono. Non-autoregressive — the entire mel spectrogram is generated in a single parallel forward pass (no AR loop, no sampling, no KV cache).

Deterministic output — same input always produces the same audio. Tokenizer: ARPABET vocabulary (115 tokens: space + 24 consonants + 45 stressed vowels + 26 lowercase chars + apostrophe + 15 punct + pad/blank/oov). Currently character-level; G2P not yet implemented.

bananamind-tts

BananaMind-TTS-V2.1 (Banaxi-Tech/BananaMind-TTS-V2.1-Preview, Apache-2.0, ~13M params), single GGUF (~50 MB F32) per locale, 22 kHz mono. Autoregressive Tacotron-lite with HiFi-GAN vocoder. Supports English (en-us, LJ Speech) and German (de-de, ThorstenVoice). Character-based tokenizer (39 symbols en-us, 43 de-de with umlauts).

Env vars: CRISPASR_BANANAMIND_DEBUG=1 (per-step decoder diagnostics), CRISPASR_BANANAMIND_TTS_BENCH=1 (per-stage timing).

Tacotron2 generalization note. BananaMind is a “Tacotron-lite” variant of the standard Tacotron2 architecture (NVIDIA, 1712.05884). The main difference is the decoder RNN: BananaMind uses 2× GRU cells with reduction_factor=4, while standard Tacotron2 uses 2-layer LSTM with reduction_factor=1 and always-on prenet dropout. The encoder, attention mechanism, postnet, and HiFi-GAN vocoder are architecturally identical. ~145 Tacotron2 models exist on HuggingFace (SpeechBrain, torchaudio, ESPnet frameworks), but most are small single-speaker models with low download counts and the architecture is largely superseded by VITS, VALLE, and flow-matching TTS. Porting a specific standard Tacotron2 model would require: (1) a per-framework converter (weight naming differs), (2) an LSTM decoder branch in run_decoder() alongside the existing GRU path (the gate math differs: 4 gates i/f/g/o vs 3 gates r/z/n), and (3) always-on prenet dropout. The BananaMind runtime is designed to serve as the template for this — all hyperparameters are already read from GGUF KV metadata, so a standard Tacotron2 GGUF just needs a decoder_rnn_type key to select the LSTM path.

pocket-tts

Kyutai Pocket TTS (100M, CC-BY-4.0 plus gated-use conditions). Continuous-latent AR TTS — architecturally unique: no codebook, no RVQ, no softmax.

Pipeline: SentencePiece (4000 vocab) → 4001×1024 embedding LUT → 6-layer causal transformer (24 layers for the French preview; 1024D, 16H, RoPE, pre-norm LN, GELU FFN) → consistency head (SimpleMLPAdaLN: 6 ResBlocks + FinalLayer, 512D, AdaLN conditioning from timestep embeddings + backbone output) → one-step Lagrangian Self Distillation (LSD) decode → continuous 32-dim float vectors at 12.5 Hz → Mimi VAE decoder (depthwise ConvTranspose1d upsample ×16 + 2L causal transformer with RoPE and LayerScale + SEANet CNN decoder with causal convolutions, ratios [6,5,4]) → 24 kHz mono PCM. Voice cloning: ref audio → Mimi VAE encoder → linear project → prepend to transformer KV cache. Model: kyutai/pocket-tts-without-voice-cloning (no encoder weights) or kyutai/pocket-tts (full, with encoder for voice cloning).

The registry ships English plus the upstream German, Spanish, Italian, Portuguese, and French checkpoints. --backend pocket-tts -m auto -l <code> selects the matching GGUF before download; the runtime itself remains a single metadata-driven implementation, including the checkpoint’s layer count.

parler-tts

Prompt-conditioned TTS from parler-tts/parler-tts-mini-v1.1 (Apache 2.0, ~900M params), single GGUF (~1.8 GB F16), 44.1 kHz mono.

Component Architecture Details
T5 encoder flan-t5-large encoder d=1024, 16 heads, 24 layers, gated-GELU FFN, RMSNorm, relative position bias
MusicGen decoder Causal transformer d=1024, 16 heads, 24 layers, 9 codebooks, sinusoidal PE, LayerNorm, delay pattern
DAC codec Descript Audio Codec 44 kHz 9 codebooks × 1024, Snake activations, 4 upsample blocks (8×8×4×2 = 512×)

Voice characteristics are controlled via --instruct (natural language description). Temperature=1.0 required (greedy produces degenerate output; the model is trained with stochastic sampling). --seed wired for reproducibility (note: C++ std::mt19937 differs from PyTorch RNG).

Tokenizer: LLaMA-2 sentencepiece BPE (90714 tokens). The GGUF stores the original sentencepiece scores and a parler.tokenizer.is_bpe=true flag so the runtime auto-selects core_spm::tokenize_bpe (iterative best-merge) instead of the Viterbi unigram path. Quantized GGUFs preserve DAC codec weights at F16 (audio codecs are precision-sensitive).

m2m100 / wmt21

12L encoder + 12L decoder transformer (d=1024, 16 heads, FFN=4096, ReLU, pre-norm) + SentencePiece BPE (128K vocab, 100 language codes) + sinusoidal positional embeddings + cross-attention KV cache + greedy decode. f16 == HF (AutoModelForSeq2SeqLM) exactly (token counts + output); q8_0 (~502 MB) matches except rare quant-floor decode flips; m2m100-f16 is registered for exact parity.

Tokenizer (2026-07 fix). M2M-100’s SP model is BPE, not Unigram — its tokenizer.ggml.scores are -merge_rank, and tokenization uses merge order via core_spm::tokenize_bpe (merge the highest-score adjacent pair whose concatenation is a vocab piece). The original backend used greedy longest-match (always, since the initial commit — never a regression), which mis-split multi-subword words and degraded translations. Two GGUF-side requirements make the BPE faithful: (1) score pieces by string (sp.piece_to_id), because vocab.json is not a constant-offset reorder of the SP model; (2) include SP pieces that only ever appear as intermediate merges (m2m100_418M’s vocab.json omits ~390, e.g. “esterd” building “esterday”) — the converter appends them at ids ≥ vocab_size with SP scores, and the engine maps any final token id ≥ vocab_size to <unk> (matching HF convert_token_to_id). wmt21’s vocab.json is already complete (0 intermediates). See LEARNINGS “SentencePiece tokenizer taxonomy”.

WMT21 (wmt21-dense-24-wide-en-x + wmt21-dense-24-wide-x-en): Same architecture scaled to 4.7B parameters (24L encoder, wider FFN). Won the WMT21 News competition. Routes through the same m2m100 runtime. Two separate checkpoints: en-x for English-source translation, x-en for English-target. Pick whichever matches your direction (-sl/-tl) — the auto-download path picks en-x by default; load x-en explicitly with -m <path> for X→English.

foxnose-diarize

Speaker diarization via --diarize-method foxnose (#324), an alternative to the pyannote path. Ported from the recipe in FoxNoseTech/diarize; the algorithms are standard published methods and the implementation is independent, so the tree stays MIT (see below).

speech regions (the caller's segments — no separate VAD)
  -> sliding windows: skip < 0.4 s, embed whole if <= 1.8 s, else 1.2 s / 0.6 s hop
  -> WeSpeaker ResNet34-LM  ->  256-d embedding per window
  -> speaker count: cosine-p10 veto -> PCA(8) -> full-covariance GMM BIC sweep
                    -> silhouette refinement (score = sil + 0.04*log k)
  -> spectral clustering on (cos+1)/2 affinity -> spherical centroid refinement
  -> temporal smoothing: Viterbi (0.18 switch penalty) -> restore sustained runs
                         -> collapse A-B-A islands <= 1.2 s
  -> merge adjacent same-speaker turns closer than 0.7 s

Components: src/wespeaker.{h,cpp} (embedder), src/core/spectral_diarize.{h,cpp} (counting + clustering), src/core/diarize_smooth.{h,cpp} (temporal), src/core/foxnose_pipeline.{h,cpp} (orchestration), src/core/der.h (the metric).

The pipeline takes its embedder as a function pointer rather than linking one in, which keeps core/ model-free and — more usefully — makes the whole orchestration testable with a synthetic embedder whose speakers are known by construction. A model-driven test cannot separate “the pipeline is wrong” from “the embedder is weak”.

Licensing. The WeSpeaker weights are CC-BY-4.0 (not Apache-2.0 as some downstream projects state), so redistributing the GGUF requires attribution — see THIRD_PARTY_NOTICES.txt. The upstream code is Apache-2.0 and none of it is incorporated: clustering.py is a sequence of scikit-learn calls rather than implementations, so there was nothing to translate, and what was taken is the recipe and its tuned constants — parameters and facts, not copyrightable expression.

Parity. Bit-exact agreement with scikit-learn is unachievable: its k-means++ seeding, GMM initialisation and ARPACK eigensolver all ride its own RNG stream. The gates are therefore known-answer unit tests (375 assertions over 57 hermetic cases) plus DER, not label equality. Output IS deterministic across runs — everything is explicitly seeded.

Speaker counting uses the upstream GMM/BIC + silhouette sweep by default. CRISPASR_DIARIZE_COUNT=eigengap selects an eigengap-of-the-Laplacian estimator instead (with row-wise affinity thresholding, without which the dense cosine affinity puts the largest gap at k=1 and it reports one speaker for everything). Eigengap is better on well-separated synthetic data and cheaper, but it UNDER-counts on real speech and scores materially worse: pooled DER over 8 VoxConverse dev files against human labels is 5.3% for bic and 11.4% for eigengap, against upstream Python’s 3.1%.

Benchmarked accuracy. Over 8 VoxConverse dev files against human labels (0.25 s collar, optimal 1:1 mapping), with Silero VAD supplying speech regions to both sides: this port 3.18 % DER, upstream Python 3.07 % — 0.11 points apart, with our speaker confusion actually lower (26.5 s vs 29.0 s). Feeding whole files with no VAD instead costs 2 points of pure false alarm (5.27 %), which is a property of the benchmark driver, not of the diarizer: the real CLI path takes the caller’s ASR/VAD segments. With the speaker count pinned equal on both sides the two agree with ZERO speaker confusion.

Speaker identity is consistent across slices: on the unified crispasr_run path FoxNose runs in ONE global pass after transcription (crispasr_apply_foxnose_global), taking the final segment list as its speech regions, and segments spanning several speakers are then split at word-aligned turn boundaries. Diarizing per slice cannot work — each slice clusters independently and restarts numbering at 0 — which is the same problem the pyannote path solves with a pre-computed posterior cache (#107).

Env gates: CRISPASR_DIARIZE_BIC_WINDOW=1 (restrict silhouette to a window around the BIC anchor instead of the full speaker range, which is the default), CRISPASR_WESPEAKER_BENCH=1, CRISPASR_WESPEAKER_DEBUG=1.

gigaam

ai-sage/GigaAM-v3 — Russian ASR. A 16-layer rotary Conformer encoder (220 M params, d_model=768, 16 heads) with either a CTC or an RNN-T head. Four shipped revisions of one architecture:

revision head vocabulary output
ctc CTC 33 Cyrillic chars lowercase, unpunctuated
rnnt RNN-T 33 Cyrillic chars lowercase, unpunctuated
e2e_ctc CTC SentencePiece 256 punctuation + casing + ITN
e2e_rnnt RNN-T SentencePiece 1024 punctuation + casing + ITN (best WER)

One GGUF carries the head type and tokenizer kind, so a single runtime (src/gigaam.cpp) serves all four.

Audio → log-mel (64 bins, n_fft=win=320, hop=160, center=False, htk, power=2,
                 ln(clamp(x, 1e-9)) — NO z-norm, NO pre-emphasis)
      → conv1d striding subsample: 2 x [Conv1d(k=5, s=2, p=2) + ReLU]  (4x)
      → 16 x Conformer block:
            FFN1(x0.5) -> rotary MHA -> conv(dw k=5 + LayerNorm) -> FFN2(x0.5) -> LN
      → CTC head (Conv1d k=1 + log_softmax + greedy collapse)
        or RNN-T head (Embedding + 1-layer LSTM predictor, joint enc/pred
        -> ReLU -> Linear, greedy with max 10 symbols per frame)

Three details are easy to get wrong and are worth stating explicitly — all three come from reading modeling_gigaam.py, not from Conformer convention:

rtt_half is the rotate-half (NEOX) pairing over head_dim=48, which maps onto ggml_rope_ext(..., GGML_ROPE_TYPE_NEOX) exactly. The FFN / conv / macaron halves are shape-identical to every other Conformer in the tree, so the weight container is core_conformer::BlockWeights; only the attention differs, which is why gigaam.cpp has its own block builder rather than calling the rel-pos core_conformer::build_block.

Batch is always 1 in this runtime, and the blueprint only builds an attention mask when batch > 1 — so full unmasked attention is correct and the conv module’s pad_mask is a no-op.

Preprocessing note: the Hann window and the mel filterbank are copied out of the checkpoint by models/convert-gigaam-to-gguf.py rather than rebuilt, which removes a whole class of window-convention and mel-normalization scale bugs. The mel is un-normalized log-mel in roughly [-20.7, +5], so encoder.pre.* stays at F32 in every quant (the same reasoning as nemotron’s pre-encode, #81), and the quantizer also keeps joint.*, decoder.* and head.ctc.* at source precision.

The adapter declares sole_language() == "ru", so -l auto resolves to Russian without downloading and running a whisper-tiny LID pass (#227), and CAP_PUNCTUATION_NATIVE for every revision — not only because the e2e_* ones already punctuate, but because the auto-enabled restorer (FireRedPunc) is a Chinese/English model that injects full-width CJK punctuation into Russian. An explicit --punc-model still applies.

Env gates: CRISPASR_GIGAAM_BENCH=1 (per-stage timings), CRISPASR_GIGAAM_DEBUG=1, CRISPASR_GIGAAM_FLASH=1 (flash attention in the encoder — opt-in until it has its own A/B), CRISPASR_GIGAAM_FORCE_SCALAR=1 (scalar LSTM/joint instead of cblas), CRISPASR_GIGAAM_QUANT_ALL=1 (quantize the heads too).

paraformer

FunASR Paraformer-zh — non-autoregressive (single-pass decode). 220M params, character-level tokenizer (8404 vocab), primarily Mandarin Chinese + English.

Audio → Kaldi fbank (80 mel) → LFR(7,6) → CMVN → 50 SANM encoder blocks
      → CIF predictor (Conv1d + sigmoid → fire-when-alpha≥1.0)
      → 16 NAR decoder blocks (FFN → FSMN → cross-attn)
      → decoders3 post block → after_norm → output_layer → argmax

Key architectural points:

F16 (421 MB), Q4_K (123 MB), Q8_0 (227 MB) at cstr/paraformer-zh-GGUF. All three produce byte-identical transcripts on Chinese + English test clips.

funasr / fun-asr-mlt-nano

FunAudioLLM Fun-ASR-Nano-2512 — 70 SANM encoder blocks + 2-block Transformer adaptor + Qwen3-0.6B LLM AR decoder. Uses the same SANM encoder primitive as paraformer + sensevoice (core_sanm::build_block). ChatML prompt template; audio embedded at <|startofspeech|> slot.

sensevoice

FunAudioLLM SenseVoice-Small — encoder-only multi-task ASR. Same 70-block SANM encoder as funasr, but paired with a CTC head (25K SentencePiece vocab) instead of an LLM. One forward pass emits transcript + language ID + audio-event tags. Non-autoregressive, 15× faster than Whisper-Large.

Upstream also emits an emotion tag in the same annotation prefix. CrispASR parses it only to strip it from the transcript and then discards it — see eu-ai-act.md.

madlad

T5 encoder-decoder (12L+12L, d=2048, gated-GELU FFN, RMSNorm, bucketed relative-position bias) + SentencePiece (256K vocab). Target language specified as <2xx> input prefix. Tokens match Python SentencePiece bit-by-bit; output matches HF reference.

melotts

myshell-ai/MeloTTS VITS2 architecture (~52M params, MIT). Text encoder (6-layer relative-position transformer with speaker conditioning at layer 2) → dual duration predictor (SDP spline flows + deterministic DP, blended via sdp_ratio) → TransformerCouplingBlock flow (4 blocks × 3 transformer layers) → HiFi-GAN vocoder (5 upsample stages) @ 44.1 kHz. Built-in English G2P via embedded CMU dictionary (129k entries) + rule-based LTS fallback for OOV words. 4 English speakers (US, BR, India, AU).

Optional BERT conditioning: loads a companion bert-base-uncased GGUF (238 MB) via melotts_load_bert(). Runs 10-layer BERT forward pass → hidden_states[-3] → word2ph expansion → ja_bert_proj (768→192) → added to text encoder embeddings. Improves contextual phoneme disambiguation (4/6 → 4/6 ASR roundtrip but fixes previously-broken sentences like “I enjoy reading”).

piper

rhasspy/piper VITS architecture (~60M params, Apache-2.0). Text encoder (TextConv + Transformer) → flow-based decoder → HiFi-GAN vocoder @ 22 kHz. Uses espeak-ng for phonemization. Single speaker per model. Lightweight, fast inference.

indextts

IndexTTS-1.5: GPT-2 AR (24L, 1280-d) mel-code generator + BigVGAN vocoder. Designed for Chinese + English. Zero-shot voice cloning from any reference WAV. Two GGUFs: GPT AR model + BigVGAN vocoder.

outetts

OuteTTS: OLMo-0.5B LM backbone AR codec-token generator + WavTokenizer codec (CC BY 4.0). Generates speech tokens autoregressively, decoded to PCM by the WavTokenizer.

voxcpm2-tts

VoxCPM2: Qwen2-2B backbone + flow matching + BigVGAN @ 48 kHz. Two-stage: AR text-to-semantic-tokens via the Qwen2 LM, then flow matching continuous diffusion + BigVGAN vocoder. Zero-shot voice cloning from reference WAV. Output is 48 kHz, decimated to 24 kHz for the standard CrispASR TTS pipeline.

voxcpm2-vae

The standalone VoxCPM2 AudioVAE backend exposes the model’s causal VAE as a speech-to-speech upscaler. Its encoder consumes 16 kHz mono PCM with rates [2, 5, 8, 8] (640x downsampling); its sample-rate-conditioned decoder uses rates [8, 6, 5, 2, 2, 2] (1920x upsampling), producing 48 kHz mono PCM. The output is cropped to exactly three times the 16 kHz input sample count. One call is capped at 960,000 input samples (60 seconds) before graph or activation allocation: the convolutional working set grows linearly but can still reach several GiB. Split longer audio, or override the cap with CRISPASR_VOXCPM2_VAE_MAX_SAMPLES when sufficient RAM/VRAM is available.

voxcpm2-vae has its own opaque context, backend handle, weight buffers, allocators, and reconstructed-weight caches. It can therefore coexist with a full voxcpm2-tts context without sharing live model state. Internally, both contexts call the same VAE graph implementation so fixes to the codec math do not need to be duplicated.

The selective loader accepts either a full VoxCPM2 GGUF or an AudioVAE-only GGUF, but allocates only vae.* tensors. A VAE-only conversion is smaller and auto-detects through general.architecture = "voxcpm2-vae":

python models/convert-voxcpm2-to-gguf.py \
  --input openbmb/VoxCPM2 \
  --output voxcpm2-vae-f32.gguf \
  --vae-only
crispasr -m voxcpm2-vae-f32.gguf -f input.wav \
  --s2s --s2s-output upscaled.wav

When using a full VoxCPM2 GGUF, select the component explicitly with --backend voxcpm2-vae; automatic detection of a full model intentionally continues to choose voxcpm2-tts.

cosyvoice3-tts

CosyVoice3 0.5B (FunAudioLLM, Apache-2.0): three-stage pipeline — Qwen2-0.5B AR speech-token LM → DiT-based conditional flow matching (10-step Euler ODE) → HiFT vocoder (NSF + iSTFT) @ 24 kHz. Supports 9 languages + 18 Chinese dialects. Zero-shot voice cloning via baked voice packs. Three separate GGUFs: LLM, flow, HiFT.

kugelaudio

KugelAudio-0-Open (MIT, 23 languages): hybrid AR + diffusion TTS based on VibeVoice. Qwen2.5-7B language model (28L, 3584d, GQA 28/4) generates constrained tokens; on speech_diffusion token, a 4-layer DiT prediction head (AdaLN + SwiGLU, v-prediction) runs 20-step SDE-DPMSolver++ with cosine beta schedule to produce 64-dim acoustic latents. An acoustic VAE decoder (6-stage ConvNeXt with depthwise conv mixer, transposed-conv upsample ratios [8,5,5,4,2,2] = 3200×) converts latents to 24 kHz mono PCM. Pre-encoded voice embeddings (acoustic connector: FC1→RMSNorm→FC2) inject speaker identity into the LM input sequence.

f5-tts

F5-TTS: DiT (Diffusion Transformer) for flow-matching text-to-speech. Converts text + reference audio to mel spectrograms via ODE-based diffusion (typically 32 Euler steps), then vocodes with a shared vocoder. Zero-shot voice cloning.

fireredtts3

FireRedTTS3 (FireRedTeam/FireRedTTS3, Apache-2.0): zero-shot in-context voice cloning over CONTINUOUS speech representations rather than discrete codec tokens. A Qwen3-1.7B backbone (28L, d=2048) runs over 64-d 25 Hz latents produced by RedAE, with an 8-layer PatchEncoder on the prompt side and an 11-layer DiT flow head that denoises the next latent window. RedAE is a Qwen3 autoencoder (18L encoder + 4L CLS downsample, 18L decoder, d=896, sliding window 64) whose Vocos-style ISTFT head reconstructs 24 kHz mono. Speaker identity comes from a CAM++ 512-d x-vector projected separately into the LLM (spk_proj_llm) and the DiT (spk_proj_dit).

Cloning is --voice ref.wav --ref-text "<transcript>"; the transcript is auto-transcribed when --ref-text is absent. Without --voice a default English prompt baked into the core GGUF is used.

The DiT flow head must stay F16 — crispasr-quantize quantizes only the LLM backbone, the same constraint dots-tts has.

Parity against the reference (kernel chr1s4/crispasr-fireredtts3-validate): penc_prompt, prefill_embeds, llm_prefill_out, stop_scores, latents_gen and dec_hidden all at cos 1.000000, gen_audio 0.999999, spk_llm 1.000000, spk_dit 0.999991, spk_emb 0.999452.

Two notes worth keeping, because both cost time:

lfm2-audio

LiquidAI LFM2.5-Audio (LFM Open v1.0, 1.5B): end-to-end multimodal ASR + TTS + speech-to-speech in a single model.

ASR path: 16 kHz mono PCM → 128-mel NeMo-style spectrogram (slaney filterbank, ln + per-feature-z normalization) → 17L FastConformer encoder (512-dim, 8 heads, rel-pos attention, dw-striding 8× subsampling) → audio adapter MLP (LayerNorm → Linear(512→2048) → GELU → Linear(2048→2048)) → 16L LFM2 hybrid backbone (2048-dim, 10 ShortConv + 6 GQA attention layers, SwiGLU FFN, RoPE θ=1M, QK layernorm) → greedy text decode via tied embed_tokens weight.

The LFM2 backbone interleaves two layer types:

Layer types follow the pattern ccaccaccacacacac (c=conv, a=attn).

TTS path: text tokenized via GPT-2 BPE → interleaved generation (6 text tokens + 9 audio frames alternating) → depthformer (6L transformer, 1024-dim, fused QKV, 8-codebook Mimi code generation) → ISTFT detokenizer (separate 8L LFM2 512-dim + Linear(512→1282) + ISTFT → 24 kHz PCM). The detokenizer loads from a companion *-detokenizer.gguf.

Speech-to-speech: combines the ASR conformer encoder (audio input) with interleaved generation (text + audio output). System prompt: “Respond with interleaved text and audio.”

GGUF: single file for the main model (encoder + backbone + depthformer + Mimi codec + audio embedding) + companion detokenizer. Quantization: Q5_K recommended for EN, Q4_K for JP. crispasr-quantize keeps encoder, adapter, embeddings, and Mimi codec at F16; only backbone + depthformer layers quantized.

Variants: English (LiquidAI/LFM2.5-Audio-1.5B) and Japanese (LiquidAI/LFM2.5-Audio-1.5B-JP).

nemotron

NVIDIA Nemotron-3.5-ASR-Streaming (NVOML, 600M): cache-aware streaming FastConformer + RNN-T. First truly streaming-native ASR backend — processes audio in fixed-size chunks with per-layer state caching.

Architecture: 16 kHz mono PCM → 80-mel NeMo spectrogram (no normalization) → causal 4× conv subsampling (pre-encode, asymmetric padding) → 24L Cache-Aware FastConformer (1024-dim, 8 heads, chunked_limited attention with rel-pos bias, depthwise conv kernel=9) → language prompt kernel (39 langs, one-hot conditioning) → 1L LSTM predictor → joint network → RNN-T decode with greedy (default), beam search (--beam-size N), or MAES (CRISPASR_NEMOTRON_MAES=1 --beam-size N).

Streaming: gated by CRISPASR_NEMOTRON_STREAMING=1. Per-layer caches:

Four attention context presets (CRISPASR_NEMOTRON_CONTEXT_PRESET=0..3) trade latency for accuracy: 160ms/7.67% WER to 1120ms/6.93% WER.

GGUF: cstr/nemotron-3.5-asr-streaming-0.6b-GGUF. F16 + Q4_K produce identical text. Pre-encode weights kept at F32 in the GGUF (F16 causes 1.56 max accumulation error across the 4352-dim projection). Auto-download: -m auto --backend nemotron.

bark

Suno Bark (MIT, ~400M): three-stage GPT-2 pipeline — text → semantic tokens (12L, 1024-d) → coarse acoustic tokens (12L, 1024-d) → fine acoustic tokens (12L, 1024-d) → EnCodec decoder @ 24 kHz. All sub-models packed into one GGUF with selective Q4_K quantization. Speaker conditioning via .npz voice prompts.

tada

HumeAI TADA-3B-ML (HumeAI/tada-3b-ml). Two GGUFs: backbone talker + codec.

Models: HumeAI/tada-3b-ml (backbone Q4_K ~2.2 GB) + companion codec GGUF (~1 GB). Pass --codec-model <codec.gguf>.

tada-encoder (voice reference creation)

The encoder pipeline converts audio + transcript → aligned acoustic features for voice cloning. Ported to C++ in src/tada_encoder.{h,cpp}:

GGUFs at cstr/tada-encoder-GGUF: tada-encoder-f16.gguf (178 MB, shared encoder) + tada-aligner-en.gguf (1.1 GB, wav2vec2-large + CTC head, loaded by wav2vec2_load()).

mini-omni2

gpt-omni/mini-omni2 (gpt-omni/mini-omni2). Multimodal speech model supporting ASR, TTS, and speech-to-speech.

Models: single GGUF (F16 ~1.6 GB) converted from lit_model.pth + small.pt. For TTS/S2S, also needs SNAC codec GGUF (--codec-model snac-24khz.gguf).

sidon

SaruLab Sidon v0.1 is a multilingual speech-restoration model exposed through the existing speech-to-speech API. It removes noise and reverberation and restores bandwidth while preserving speaker identity.

The CLI auto-detects general.architecture = "sidon" and exposes CAP_S2S:

crispasr -m sidon-v0.1-f16.gguf -f input.wav --s2s --s2s-output restored.wav

The C/Python session API accepts non-16 kHz input after setting the PCM sample rate; the unified S2S dispatch performs polyphase resampling before inference.

To reproduce the F16 GGUF from the upstream base model and released Sidon raw weights:

huggingface-cli download facebook/w2v-bert-2.0 --local-dir models/w2v-bert-2.0
huggingface-cli download sarulab-speech/sidon_raw_weight --local-dir models/sidon_raw_weight
python models/convert-sidon-to-gguf.py \
  --base models/w2v-bert-2.0 \
  --sidon models/sidon_raw_weight \
  --output models/sidon-v0.1-f16.gguf \
  --dtype f16

reazonspeech

ReazonSpeech NeMo v2 (reazon-research, Apache-2.0): 619M-param Japanese FastConformer-RNNT ASR model trained on the ReazonSpeech v2.0 corpus.

Audio → 80 log-mel (n_fft=512, Hann, per-feature z-norm)
      → dw_striding 8× subsampling (5× Conv2d + Linear)
      → 24 FastConformer blocks (rel_pos_local_attn, window=[128,128], 1 global token)
      → RNNT joint network (enc→640, pred→640, ReLU, →3001)
      → 2-layer LSTM predictor (embed(3001,640) + LSTM(640,640)×2)
      → RNNT greedy decode (n_tdt_durations=0)

Key architectural points:

Models at cstr/reazonspeech-nemo-v2-GGUF: F16 (1240 MB), Q8_0 (704 MB), Q4_K (455 MB).

parakeet-ctc-1.1b-ja

Community Japanese fine-tune of nvidia/parakeet-ctc-1.1b (grider-transwithai, Apache-2.0). 42-layer FastConformer-CTC, 1.1B params.

Audio → 80 log-mel (n_fft=512, Hann, per-feature z-norm)
      → dw_striding 8× subsampling (5× Conv2d + Linear)
      → 42 FastConformer blocks (rel_pos, full attention, xscaling)
      → CTC head (Conv1d(1024 → 4001, k=1))
      → CTC greedy decode

Key points:

Models at cstr/parakeet-ctc-1.1b-ja-GGUF: F16 (2.0 GB), Q8_0 (1.2 GB), Q4_K (631 MB).

tabcnn

Guitar tablature emission scorer (--tab). TabCNN (Wiggins & Kim, ISMIR 2019), 833,982 params — the smallest backend in the tree.

⚠️ fmin is C1, not the guitar’s low E. Assuming E2 is the obvious guess and it is wrong. Every wrong value still runs, producing plausible tensors that pass shape and cosine checks while the model emits garbage: measured on EGSet12 track 01, fmin C1 → tablature F1 0.771, E1 → 0.040, E2 at 44.1 kHz → 0.001. All front-end constants are stored as GGUF metadata and read back at load, so the runtime cannot drift from the reference dumper.

⚠️ Not a streaming surface. ref = max of the whole clip is a per-clip normalisation, so features cannot be computed chunked without changing them — two-pass by construction. Chunking here would reproduce the BTC chunked-CQT bug.

Validation. crispasr-diff tabcnn runs the full pipeline from the waveform, not replayed features: model.frontend is empty, the CQT lives outside the network, and a feature-replaying diff would never test it. All stages pass (cqt_db 0.9989 … logits 0.9997). End to end against EGSet12 JAMS ground truth: tablature F1 0.7732 vs the torch reference’s 0.7708 (Δ +0.0024, argmax agreement 98.57 %); the residual is the front end — direct Brown-kernel CQT against librosa’s recursive downsampling.

Quantization. Only two tensors are quantizable (dense0.weight 761 k and head.weight 16 k); the conv stack is 3×3 so ne0=3. K-quants are impossible — no tensor is 256-aligned, so --q4_k falls back to Q4_0. crispasr-quantize preserves head.weight: quantizing it costs 5.8 F1 points at Q4_0, while dense0 costs nothing.

Weights are CC BY 4.0 (https://zenodo.org/records/11406378), attribution required. See docs/cli.md and music-transcription/GUITAR_TAB_SPEC.md.

crepe

Monophonic pitch / F0 estimation — CREPE (Kim et al. 2018, MIT), from the torchcrepe weights. Neither ASR nor TTS: see Non-transcribe task surfaces. Driven by --pitch, capability bit CAP_PITCH (bit 24), C ABI crispasr_session_pitch*, runtime src/crepe.{h,cpp}.

16 kHz mono PCM
  → 1024-sample frames, hop 10 ms, zero-padded 512 at both edges
    (torchcrepe pad=True), each frame mean-centred and divided by its
    UNBIASED (n-1) std
  → 6 × [pad → conv1d → RELU → batchnorm → maxpool(2)]
  → flatten channel-fastest → Linear → 360 bins → sigmoid
  → decode: cents = 20·bin + 1997.3794084376191, Hz = 10·2^(cents/1200)
layer K stride pad (l, r) out ch (full / tiny) T
conv1 512 4 254, 254 1024 / 128 1024 → 256 → 128
conv2..4 64 1 31, 32 128 / 16 halves each layer
conv5 64 1 31, 32 256 / 32 16 → 8
conv6 64 1 31, 32 512 / 64 8 → 4

Three geometry decisions that are easy to get wrong:

Performance. CREPE is genuinely expensive per frame: at the reference 10 ms hop, full is 1409 MMAC/frame → 282 GFLOP per second of audio; tiny is 36.7 MMAC/frame → 7.3 GFLOP/s (38× cheaper). Measured on M1 over 10 s of audio: full RTF 2.0 (Metal) / ~40 (CPU), tiny RTF 0.28 (Metal) / ~2.4 (CPU). tiny is the shipping default and the GPU path is not optional. The graph batches 64 frames per dispatch and keeps the channel-fastest (OC, OL, N) layout throughout rather than letting ggml_conv_1d permute back every layer. Gates: CRISPASR_CREPE_NO_GPU, CRISPASR_CREPE_NO_BAKE_F32, CRISPASR_CREPE_DEBUG.

Batching CREPE surfaced a latent ggml_conv_1d bug: for N > 1 the final ggml_reshape_3d declares a shape that contradicts the data layout. Fixed in the fork (CrispStrobe/ggml@662b05fb); no other caller in either repo hits the N > 1 && OC > 1 case. See docs/music-transcription/PLAN.md.

Parity. tools/reference_backends/crepe.py → crispasr-diff crepe compares the 360-bin activation against torchcrepe per frame; tools/crepe_numpy_parity.py is the executable numpy spec for the graph, and tests/test_crepe_parity.cpp dumps activations for it to score. Measured on samples/jfk.wav (1101 frames), per-frame cos_min / same-argmax rate:

  f16 q8_0 q4_k
tiny 0.999999 · 100 % 0.999807 · 98.5 % 0.961643 · 85.2 %
full 1.000000 · 100 % 0.999937 · 99.5 % 0.992563 · 91.4 %

So q8_0 is the lowest safe quant; q4_k shifts the argmax on low-confidence frames, which for pitch means octave errors. Models at cstr/crepe-GGUF (f16 / q8_0 / q4_k × tiny / full; tiny-f16 is 1.0 MB, full-f16 44.5 MB).

Live test: tests/test_crepe_live.cpp, gated on CRISPASR_MODEL_CREPE.

This site is open source. Improve this page.