CrispASR

Streaming & live transcription

CrispASR supports three streaming modes — pipe input, microphone capture, and continuous live mode — and per-token confidence output. All work with every supported backend.

--backend vibevoice-streaming -m auto uses the model’s native streaming protocol: fixed 2.93-second chunks, 0.53-second lookahead, and one persistent decoder cache for the whole session. --stream-step, --stream-length, and --stream-keep therefore do not change its receptive field. A final flush pads one last partial chunk and does not decode retained lookahead twice.

Streaming TTS output (the reverse direction) is documented in its own section at the bottom — Streaming synthesized audio.

Over HTTP, the server exposes the same streaming decoder as a WebSocket endpoint — start it with --ws-port and send binary float32 PCM frames, or connect to ws-port + 1 for the JSON-based vLLM Realtime API endpoint. See server.md.

Pipe mode (--stream)

# Pipe audio from ffmpeg, sox, or any tool that outputs raw PCM:
ffmpeg -i audio.wav -f s16le -ar 16000 -ac 1 - | \
    crispasr --stream -m model.gguf

Sliding-window chunking, default 10 s rolling window with a 3 s step. Tune via --stream-step and --stream-length; --stream-keep is still parsed but is a no-op (see the note on issue #84).

Quality-control flags supported in streaming mode:

Notes:

Structured streaming output (--stream-json)

For wrappers (browser bridges, live-translation pipelines, captioning UIs) that need to distinguish a still-evolving partial from a finalized utterance, pass --stream-json. CrispASR then emits one JSON object per line on stdout — never plain text — and FireRed VAD diagnostics stay off stderr unless you opt in with --firered-vad-debug.

ffmpeg -i input.wav -f s16le -ar 16000 -ac 1 - 2>/dev/null \
  | crispasr --stream --stream-json -m model.gguf \
      --vad --vad-model firered-vad.gguf \
      --stream-final-on-silence-ms 800

Event types:

type When Fields
partial A streaming step produced new text for the open utterance. At most one partial per utterance_id per step — multiple VAD slices belonging to the same utterance within a step are concatenated. utterance_id, text, t0, t1
final Trailing silence ≥ --stream-final-on-silence-ms (default 800) after the last detected speech closed the open utterance. In the default --stream-final-mode redecode text is produced by re-running the backend on the buffered utterance PCM (covers [t0..t1]); in prefix mode text is a prefix accumulator stitched with the last partial. utterance_id, text, t0, t1, (optional) speaker
silence A streaming step produced no speech slices. Emitted regardless of whether an utterance is still open, so wrappers always see a timeline heartbeat. t

The optional speaker field on final events appears only with a backend that populates the structured speaker label (moss-diarize; vibevoice from v0.8.24; granite in speaker-aware --diarize mode) when the finalized utterance is single-speaker; its ordinals are utterance-local. See Speaker diarization while streaming.

Stream-contract guarantees:

Sample stream:

{"type":"partial","utterance_id":1,"text":"is that they can be tuned and adjusted","t0":10.20,"t1":13.20}
{"type":"partial","utterance_id":1,"text":"is that they can be tuned and adjusted for a specific","t0":10.20,"t1":16.20}
{"type":"final","utterance_id":1,"text":"is that they can be tuned and adjusted for a specific hardware target.","t0":10.20,"t1":17.80}
{"type":"silence","t":18.60}

Live-translation wrappers can show partial events in a draft pane and only ship final events to the translation API. Set --stream-final-on-silence-ms 0 to disable auto-finalization (useful when the wrapper finalizes on its own signal — e.g., a UI button — instead of trailing silence).

t0 / t1 are wall-clock seconds since stream start, derived from the cumulative sample count, so they map to the same timeline as the input PCM. t0 marks where the utterance started (first VAD speech frame, or first non-empty model decode in no-VAD mode); t1 marks the last detected speech frame for final events, or the current decoder-step time for partial.

Finalization timing

Finalization fires when there has been --stream-final-on-silence-ms worth of trailing silence after the last detected speech, not when the entire rolling window has decoded to empty. With VAD enabled the silence detector uses each VAD slice’s end time directly; without VAD the fallback is “the model decoded nothing for that long.”

The practical effect: a speaker who pauses mid-paragraph for ~800 ms gets a final per natural pause, instead of one giant final at the end of the recording. Set --stream-final-on-silence-ms higher (e.g. 2000) if you want fewer finalizations / longer-form chunks.

How final.text is built — --stream-final-mode

Two modes; redecode is the default.

# Best quality — re-runs the backend on the buffered utterance PCM at
# finalize time. final.text is guaranteed to cover [t0..t1] regardless
# of how the rolling window evicted audio.
crispasr --stream --stream-json --stream-final-mode redecode ...

# Cheaper — no extra encoder pass. final.text is built from a
# longest-common-prefix accumulator across consecutive partials, with
# the last partial appended. Subject to text duplication when the
# rolling window evicts mid-utterance audio.
crispasr --stream --stream-json --stream-final-mode prefix ...

In redecode mode CrispASR buffers the speech-region PCM in memory (capped at --stream-utterance-max-sec, default 60 s — about 4 MB at 16 kHz mono float). When the cap is hit the current utterance auto-finalizes and the next speech opens a new utterance with a fresh utterance_id. For most live-captioning / translation use cases the redecode path is what you want — its output covers the whole utterance the way t0/t1 advertise.

prefix mode preserves round-1 cost (no extra transcribe() call) at the price of imperfect text reconstruction on long utterances. Useful when the encoder is large and the per-chunk budget is tight.

Short-utterance fallback. Backends that use convolutional encoders (moonshine, parakeet, voxtral, …) abort with OW > 0 from ggml_im2col when handed audio shorter than the encoder’s first conv kernel — about 2 s at 16 kHz. When redecode would hit that limit (the VAD-trimmed [t0..t1] is under 2 s) CrispASR skips the extra backend pass and falls back to the prefix-mode stitcher for that one finalize. final.text is then the LCP-accumulated prefix plus the last partial — the same content the wrapper has already seen in partial events, never an empty string blanking a previously-emitted partial. The fallback is internal; no flag, no event change.

Streaming punctuation (--stream-punc)

When --stream-json --vad is combined with --punc-model, FireRedPunc can sit on either the partial path, the final path, both, or neither. PR #112 introduced the explicit knob; before that, partials and finals both ran through FireRedPunc (equivalent to today’s --stream-punc partial).

Mode Partials Finals Notes
off ❌ ❌ FireRedPunc is bypassed entirely on the streaming path. Both partials and finals come out unpunctuated — off is the most permissive setting, not just “off for partials”.
final (default) ❌ ✅ Recommended realtime mode. Live partials stay cheap; finals get punctuation once per utterance via either --stream-final-mode redecode (segments are punc’d before stitching) or the stitched-fallback path (the final string is punc’d in place).
partial ✅ ✅ Pre-#112 behaviour. Keep if every partial event needs punctuation downstream — the cost is one FireRedPunc forward per --stream-step.

Default change. Before PR #112 the de-facto default was equivalent to partial (no flag existed; every partial got punctuation). After #112 the default is final. Wrappers that relied on punctuated partials should pass --stream-punc partial explicitly to restore the old behaviour; everyone else gets the better latency profile for free.

Smoke results on 30 s of Cohere JA streaming (--stream-step 500 --stream-final-mode redecode):

mode wall_sec partials finals
off 35.5 36 11
final 44.3 36 11
partial 45.9 36 11

Event counts are identical across the three modes — the policy controls processing, not emission. The ~10 s gap between off and final/partial is FireRedPunc on the finals; the (smaller) gap between final and partial is FireRedPunc on the 36 partials. On longer audio or shorter --stream-step (more partials per second) the partial-vs-final gap widens proportionally.

--stream-punc is a no-op without --punc-model, and it gates only the FireRedPunc step. The truecasers (--truecase-model auto|crf|lstm|<path>) and PCS run on every mode. Note that both variants are selected by the value passed to a single flag — there are no separate --truecase-crf-model, --truecase-lstm-model or --pcs-model flags, and PCS is --punc-model pcs, which loads PCS instead of FireRedPunc (so on a PCS server --stream-punc has nothing to gate).

Microphone (--mic)

# Live microphone transcription (auto-detects arecord/sox/ffmpeg):
crispasr --mic -m model.gguf

CrispASR auto-detects whichever audio capture tool is on $PATH.

Continuous live mode (--live)

# Continuous live mode (prints each chunk as a new line, never stops):
crispasr --live -m model.gguf

# With progress monitor symbols (▶ processing, ✓ got text, · silence):
crispasr --live --monitor -m model.gguf

--live runs indefinitely, emitting one transcript line per processed chunk. --monitor adds visual feedback so you can tell processing state at a glance.

Per-token confidence

crispasr -m model.gguf -f audio.wav --alt

--alt prints alternative candidate tokens with probabilities — useful for filtering low-confidence file transcriptions or for downstream rescoring. Streaming modes do not currently emit this alternatives block.

Tuning the sliding window

Flag Default Effect
--stream-step N 3000 ms Step between consecutive windows. Smaller = more frequent partial transcripts.
--stream-length N 10000 ms Rolling context window cap. The decode buffer accumulates audio up to this many ms, then drops the oldest samples from the front. Larger = better accuracy on long-form content but higher per-step cost.
--stream-keep N 200 ms Legacy — kept for compatibility, currently a no-op. The rolling buffer above subsumes it (see issue #84).
--stream-partial-decode-ms N 0 ms JSON+VAD only. Minimum interval between live partial ASR decodes. 0 preserves the previous behavior and decodes every --stream-step; larger values keep VAD/final timing at --stream-step while reducing partial ASR cadence.
--stream-partial-tail-sec N 0 (off) JSON+VAD only (#404). Cap each live partial decode to the last ~N seconds of the open utterance. Text decoded ahead of the moving anchor is kept as a committed prefix, so partial.text still covers the whole utterance, and final.text is untouched (redecode mode re-decodes the full utterance regardless). Cuts land on the quietest 100 ms, the same boundary policy as the long-audio chunker. Effective floor ~4 s.

--stream-vad-merge-gap-ms defaults to 250 ms and applies only to --stream-json --vad. It merges adjacent VAD slices only across gaps smaller than that value. When --stream-final-on-silence-ms is enabled, the effective merge gap is clamped below the finalization threshold. Set it to 0 to disable this close-gap merge.

--stream-partial-decode-ms is useful when low-latency VAD/final timing is desired but partial ASR decode is too expensive to run every step. For example, --stream-step 500 --stream-partial-decode-ms 750 keeps VAD and silence finalization checks at 500 ms while allowing live partial ASR text at most every 750 ms. Steps that skip partial decode still keep VAD slice timing for the JSON utterance state machine. When trailing silence has crossed the finalization threshold, one step may bypass the partial-decode throttle before finalization so short-utterance fallback finals can use a fresh normal partial.

Related, and on by default: CRISPASR_STREAM_SLICE_MEMO memoizes each VAD-closed slice’s partial decode by its absolute sample range — a closed slice keeps the same audio while it stays in the rolling window, so re-decoding it every step repeated byte-identical work. Exact by decode determinism (finals and partials byte-equal in the A/Bs; wall −12 % CPU / −6 % GPU on an uncontended box); set =0 to restore the old re-decode path.

--stream-partial-tail-sec attacks the other axis of partial cost: not how often a partial decodes, but how much audio each one covers. Without it, the partial decode of an open utterance re-encodes the whole utterance-so-far (up to --stream-length) every time, so preview cost grows with utterance length even though only the tail changes. Encoder-state reuse cannot fix this exactly — a bidirectional encoder (e.g. cohere’s Conformer, unmasked relative-position attention over the whole window in every layer) makes every earlier frame’s encoding depend on later audio — so the incrementality lives at the text level instead: the region behind the cap is decoded once at a quiet cut, its text committed, and each subsequent partial decodes only [cut, now]. On CPU, where every encoder pass pays a large weights-bandwidth constant, pair it with --stream-partial-decode-ms; on GPU the per-decode saving dominates. Finals are exact either way: --stream-final-mode redecode (the default) re-decodes the buffered utterance PCM from scratch. Expect small cosmetic seams in the stitched partials (a capital letter or period where two independently-decoded regions join — e.g. “…that the Proposed…”); the final replaces them with the seamless full-utterance text.

The default value 0 means “follow --stream-step“ — the throttle is always conceptually present in the JSON+VAD path, but at 0 it locks to the step cadence so every step decodes (matching the pre-#113 behaviour). It is NOT “throttling disabled”; rather, the interval is set to one step’s worth of audio. Set --stream-partial-decode-ms to a value larger than --stream-step to actually space out partial decodes. Setting it smaller than the step has no effect — the gate only fires on stream-step boundaries, so the effective minimum is one step regardless of what you pass.

The first step of a stream is always allowed (so the first partial fires immediately), and --stream-partial-decode-ms is a no-op outside the --stream-json --vad combination — non-JSON streaming always decodes every step.

Note (issue #84). Before May 2026, --stream-length was a ceiling on keep + step rather than a true rolling cap, so --stream-length 18000 --stream-keep 200 --stream-step 3000 actually decoded ~3.4 s of audio per step instead of 18 s. The streaming loop was rewritten to accumulate up to length_samples and drop the oldest frame on overflow, which matches the documented behaviour. --stream-keep is now informational only.

Per-token streaming backends

All autoregressive ASR backends implement transcribe_streaming and emit tokens to the --stream callback as they are generated, without waiting for the full decode to finish:

Backend Token decode type Notes
granite (granite-speech) LLM greedy (Granite LLM) Standard run_with_probs_cb
voxtral4b LLM greedy (Mistral LLM) Per-step encoder-frame injection via pre_hook
glm-asr LLM greedy (GLM BPE) Adapter-side greedy loop using exported step APIs
moss-audio LLM greedy (GPT-2 BPE) Via moss_audio_process_cb
moss-transcribe LLM greedy (GPT-2 BPE) Via moss_transcribe_transcribe_cb
gemma4-e2b LLM greedy (SentencePiece) Via gemma4_e2b_transcribe_cb; control tokens filtered
moonshine-streaming LLM greedy (SentencePiece) Via moonshine_streaming_transcribe_cb
kyutai-stt LLM greedy (SentencePiece) Via kyutai_stt_transcribe_cb; padding tokens filtered in C lib
mimo-asr LLM greedy (GPT-2 BPE) Via mimo_asr_transcribe_cb
nemotron RNN-T (per non-blank frame) Via nemotron_transcribe_cb; fires per emitted frame
qwen3 (Qwen3-ASR; alias mega-asr) LLM greedy (Qwen3) Native
voxtral LLM greedy (Mistral LLM) Native

For these backends, --stream output grows one token at a time. For batch backends (whisper, parakeet, canary, funasr, etc.), each full chunk produces one update.

For native streaming-architecture backends (voxtral4b, moonshine-streaming, kyutai-stt, nemotron), the encoder also runs incrementally — the sliding window cost is lower than for batch backends.

Nemotron streaming (cache-aware FastConformer)

nemotron supports true cache-aware streaming via the NeMo cache_last_channel + cache_last_time architecture. Enable with:

CRISPASR_NEMOTRON_STREAMING=1 crispasr --backend nemotron -m model.gguf -f audio.wav

Four context presets trade latency for accuracy:

Preset Right-context Chunk size Approx latency Published WER
0 3 frames 4 frames ~160 ms 7.67 %
1 0 frames 1 frame ~80 ms 8.43 %
2 6 frames 7 frames ~560 ms 7.07 %
3 13 frames 14 frames ~1120 ms 6.93 %

Set via CRISPASR_NEMOTRON_CONTEXT_PRESET=N (default: 0).

Speaker diarization while streaming

Short answer: streaming carries whatever speaker information a backend produces per window/utterance, but not the cross-recording clustering pipeline or named-voiceprint identification — those two are recorded-file (offline) features by design. See diarization-speakers.md for the full diarization model.

Both backends issue #300 asked about produce speaker information while streaming, but by two different mechanisms — worth understanding because they behave differently downstream:

Backend How speaker info is produced In streaming you get
moss-diarize (MOSS-Transcribe-Diarize-0.9B, cstr/MOSS-Transcribe-Diarize-GGUF) a structured per-segment speaker label (seg.speaker), parsed from the model’s [Sxx] tags inline (Speaker N) in plain --stream; a "speaker" field on --stream-json final events
vibevoice (VibeVoice-ASR, cstr/vibevoice-asr-GGUF) a structured per-segment speaker label, parsed from the JSON array the model answers with (its prompt asks for “Start time, End time, Speaker ID, Content”) — from v0.8.24; before that the blob was passed through as raw text inline (Speaker N) in plain --stream; a "speaker" field on --stream-json final events. Set CRISPASR_VIBEVOICE_RAW_TRANSCRIPT=1 for the old raw-blob behaviour

Issue #300’s change surfaces the structured seg.speaker field in streaming — so it applies to moss-diarize, to granite in speaker-aware --diarize mode, and to vibevoice.

vibevoice needs v0.8.24. VibeVoice-ASR answers with a JSON array of utterances (Start / End / Speaker / Content), but until v0.8.24 the adapter handed that blob back as ONE segment’s text — so the labels reached you as literal JSON, seg.speaker was never populated, and the "speaker" field below could not fire for this backend at all. v0.8.24 reads the answer: one segment per utterance, native per-utterance timings, and the speaker in the structured field like any other native diarizer. CRISPASR_VIBEVOICE_RAW_TRANSCRIPT=1 restores the raw blob for callers that were parsing it themselves.

What works in streaming

moss-diarize and vibevoice populate the structured field, so under --stream / --mic / --live each decoded window carries its own speaker labels (substitute --backend vibevoice in either recipe below):

# Plain streaming — labels are prefixed inline, exactly like file-mode text output:
ffmpeg -i meeting.wav -f s16le -ar 16000 -ac 1 - 2>/dev/null \
  | crispasr --stream -m auto --backend moss-diarize
# (Speaker 1) welcome everyone
# (Speaker 2) thanks, glad to be here

# Structured streaming — a `final` event gains a "speaker" field when the
# finalized utterance is single-speaker (text stays clean, no inline labels):
ffmpeg -i meeting.wav -f s16le -ar 16000 -ac 1 - 2>/dev/null \
  | crispasr --stream --stream-json -m auto --backend moss-diarize \
      --vad --vad-model auto --stream-final-on-silence-ms 800
# {"type":"partial","utterance_id":1,"text":"welcome everyone","t0":0.30,"t1":1.80}
# {"type":"final","utterance_id":1,"text":"welcome everyone.","speaker":"(Speaker 1)","t0":0.30,"t1":2.10}

The speaker field is only present on final events, and only when every segment of that utterance shares one label (the common VAD-bounded case). A final whose redecode spanned a mid-utterance speaker turn, and every partial, omit the field — parse a missing speaker as “unlabeled”, not as a change of speaker. In plain (non-JSON) --stream, labels are prefixed inline into the text instead, matching the file-mode text/srt/vtt convention.

⚠ Speaker IDs are window/utterance-local, not globally stable. No cross-window clustering runs in streaming mode, so Speaker 1 in one step is not guaranteed to be the same physical voice as Speaker 1 in a later step — the same caveat the diarized file-mode JSON documents for per-chunk labels. If you need recording-stable labels, transcribe the recorded file offline (below), where global clustering runs across the whole audio.

What does NOT run in streaming (offline only)

For live captioning where per-utterance labels are enough, stream a native diarizer as above. For a transcript with recording-stable speaker labels (the same person keeps the same number end to end), run the diarizer — or any backend with --diarize-speakers — over the recorded file once capture ends:

crispasr -m auto --backend moss-diarize -f meeting.wav -ojf     # native labels + native timestamps
crispasr -m auto --backend cohere      -f meeting.wav --diarize-speakers -ojf   # any backend + clustering

Streaming synthesized audio (out)

TTS output can be streamed progressively — audio starts flowing before the whole clip is synthesized — from the CLI, the HTTP server, and the C ABI. All three split the input into sentence chunks and emit each chunk as soon as it is ready, so time-to-first-audio is one sentence, not the whole utterance.

CLI (--tts-stream)

Emits raw signed-16-bit little-endian mono PCM to stdout at the backend’s sample rate; all logs stay on stderr, so stdout is a clean stream to pipe into a player:

crispasr --backend irodori-tts -m model.gguf --codec-model dacvae-ja-32dim-f16.gguf \
    --tts "こんにちは。今日はいい天気ですね。" --tts-stream \
  | ffplay -f s16le -ar 48000 -nodisp -

The spoken AI-disclosure (voice-cloned output) is emitted first, each chunk is watermarked before emit, and a 200 ms gap separates chunks. Accepted on every TTS backend, but the granularity is always one sentence — unlike the server’s stream: true, this path calls synthesize() per chunk and does not use a backend’s CAP_STREAMING sub-sentence emit. The backends the chunker treats as single-shot (vibevoice*, qwen3-tts*, tada*, dots-tts*, omnivoice* — see server.md) therefore produce exactly one chunk, i.e. no early audio at all.

Note that the watermark is forced on here regardless of --no-watermark / CRISPASR_NO_WATERMARK: a raw PCM stream has no container, so no C2PA manifest can ride along and the watermark is the only machine-readable mark available.

Server (stream: true)

POST /v1/audio/speech with "stream": true and a PCM response_format (pcm, wav, or f32 — mp3/aac/opus return 400) streams the audio back with chunked transfer encoding. On a backend with CAP_STREAMING the chunks are the backend’s own codec chunks, so time-to-first-audio is roughly one chunk; on every other backend it is one chunk per sentence. The body is always raw int16 LE mono PCM at the backend’s native rate, served as Content-Type: audio/pcm — there is no RIFF header even when you asked for wav, so the client must know the rate out-of-band:

curl -N http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"irodori-tts","input":"…","stream":true,"response_format":"pcm"}' > out.pcm

C ABI (crispasr_session_synthesize_streaming)

For embedders/bindings — fires a callback per sentence chunk with that chunk’s watermarked PCM (backend-native sample rate, owned by the call). As with the other bindings, this path watermarks unconditionally; the --no-watermark / CRISPASR_NO_WATERMARK opt-out does not apply here (see bindings.md):

void on_chunk(const float* pcm, int n, int is_final, void* user) { /* play/queue */ }
crispasr_session_synthesize_streaming(session, "…", on_chunk, user);

This path has its own splitter, not the server/CLI one: it breaks on ASCII ., !, ? and newline, plus CJK 。, ！, ？ — no Devanagari danda, no 600-char run-on cap, and no per-backend single-shot exemption. It also inserts no silence between chunks (the caller concatenates), and applies --tts-pad-silence-ms only to the first chunk.

Note: for diffusion backends (e.g. irodori) the per-sentence granularity above is the real latency win — a diffusion utterance is generated in full before it is decoded, so there is no sub-sentence audio to emit early.

Source separation (htdemucs)

HTDemucs processes audio in overlapping chunks with cross-fading, but this is internal chunking for memory management, not real-time streaming. The streaming capability flag indicates support for chunked processing of long audio files.

This site is open source. Improve this page.