CrispASR

Music transcription in CrispASR

Porting the CometBeat / mus-textbook “transcription → SOTA” model roster (docs/TRANSCRIPTION_SOTA_HANDOFF.md in that repo) from ONNX to CrispASR ggml/GGUF backends.

NOW — active work

Performance — measured, M1, quiet box (load 4.0), 10 s audio, median of 3

model Metal CPU
full (44.5 MB f16) 20.0 s — RTF 2.0 ~400 s — RTF 40
tiny (1.0 MB f16) 2.8 s — RTF 0.28 ~24 s — RTF 2.4

CREPE is genuinely expensive: at the reference 10 ms hop it is 1409 MMAC per frame → 282 GFLOP per second of audio for full, and 36.7 MMAC/frame → 7.3 GFLOP/s for tiny (38× cheaper). So tiny is the shipping default — it is also what the handoff asks for (“smallest that hits accuracy”). full stays available and is the right choice offline. Neither is close to real-time on CPU; the GPU path is not optional here.

Three graph decisions got it from the first working version (RTF 31) to here:

  1. Batching (the big one). One frame per dispatch wastes the GPU on a model this small per-frame. kBatch = 64 makes each layer one large GEMM.
  2. Channel-fastest layout throughout. ggml_conv_1d ends by permuting back to (OL, OC, N), materializing the whole activation every layer. We keep the mul_mat’s native (OC, OL, N), do bias/relu/BN there — where a plain (OC) vector broadcasts along ne[0], ggml’s fast path, instead of a stride-0 (1, OC, 1) broadcast — and pool with ggml_pool_2d(k0=1, k1=2). The one transpose im2col forces is deferred until after the pool, so it moves half the bytes, and the last layer skips it entirely because (OC, OL, N) already is the channel-fastest flatten the classifier wants.
  3. F32-baked conv kernels (ggml_conv_1d casts an F16 kernel to F32 inside the graph — in a persistent graph that re-casts 44 MB per 10 ms frame). Gated CRISPASR_CREPE_NO_BAKE_F32=1. Honest note: this one measured neutral here, unlike qwen3-tts CODEC_FASTCONV. Kept gated-on because it is provably redundant work, but it was not the win.

Gates: CRISPASR_CREPE_NO_GPU=1, CRISPASR_CREPE_NO_BAKE_F32=1, CRISPASR_CREPE_DEBUG=1.

ggml_conv_1d returns a tensor whose declared shape contradicts its data for N > 1

Status: fixed in the fork (ggml/src/ggml.c), upstream PR drafted at tools/upstream-prs/24-conv-1d-batch-reshape.md + a standalone repro. NOT yet merged to main — one audit item is open, see below.

The im2col is the FIRST ggml_mul_mat argument, so the result’s ne is [N*OL, OC] (OC slowest). The final ggml_reshape_3d declares [OL, OC, N] (N slowest). Those expressions coincide exactly when N == 1 and differ otherwise — which is why every shipping caller is correct and this was invisible.

Repro (tools/upstream-prs/24-conv-1d-batch-reshape.repro.cpp, standalone, vs a hand-rolled direct convolution), before the fix:

N=1  cos=1.00000000  OK        N=2  cos=0.41129104  MISMATCH        N=3  cos=0.05935857  MISMATCH

After: all three cos=1.0. Fix reshapes to the true [OL, N, OC] then permutes; the N == 1 branch is the unmodified original statement, so batch-1 callers are bit-identical by construction, not merely by test.

Corroborating facts:

✅ AUDIT COMPLETE — landed in the fork (CrispStrobe/ggml@662b05fb)

The open question was whether any existing caller passes N > 1. Answered, and my original safety argument was wrong:

The two batched callers are aa_snake_beta_native in src/indextts_voc.cpp (:508 and :551), which deliberately maps channels onto the batch axis so one depthwise FIR runs across all C channels at once. So “the N == 1 branch is unmodified, therefore every caller is bit-identical” was false — those two take the new branch.

They are safe for a different reason: their filter is [K,1,1], i.e. OC == 1, and with OC == 1 both branches produce the identical flat layout n*OL+ol and the identical declared ne. Confirmed from the source (the shape is documented at indextts_voc.cpp:459-460 and enforced by a downstream ggml_reshape_2d nelements assert) and verified empirically on that exact shape class at N = 1..4. Neither site compensates for the old transpose, so nothing depended on the broken layout.

The branches diverge only when N > 1 AND OC > 1 — which no caller in either repo does. CREPE would have been the first, which is why it surfaced here.

Gates run: standalone repro (both shape classes, all N) cos = 1.0; CrispASR unit suite 1032/1032; CREPE parity unchanged at cos = 1.0.

Companion, now also landed (CrispStrobe/ggml@655c14e4): ggml_conv_1d_dw batch support. The first description of this (mine, repeating an agent’s) was wrong: it does NOT silently drop the batch dim. It reshapes to [T,1,C,N] and hits GGML_ASSERT(b->ne[3] == 1) at ggml.c:4476, i.e. it aborts — a safe failure, an unsupported case rather than a correctness bug. Verified by probing it rather than reading it. Fixed by folding the batch into the channel axis and tiling the kernel with ggml_repeat; verified N = 1..4 at cos = 1.0, max_abs = 0.0 exactly. No existing caller changes (N == 1 path untouched, and nothing can depend on an abort).

CREPE weights are published

https://huggingface.co/cstr/crepe-GGUF — all six files (f16/q8_0/q4_k × tiny/full), license: mit verified present on the card via model_info(expand=["cardData"]), public, ungated. So -m auto / --auto-download now resolves. Published deliberately before the accuracy eval, so that eval can be run on real music from the published artifacts.

✅ ACCURACY EVAL ON REAL MUSIC — the octave concern did NOT reproduce

Run on 10 monophonic instrumental recordings (violin arco + pizz, piano, glock, carillon, cello, flute, three folk melodies, brass), tools/crepe_music_eval.py. No hand-labelled F0, so two load-independent proxies: tiny-vs-full octave disagreement (|log2(a/b)| ~ 1) and in-tessitura rate on voiced frames (voiced_prob >= 0.5).

  tiny full
in-tessitura (weighted) 89.6% 89.0%
voiced frames 8166 9165
octave disagreement tiny-vs-full 2.3%

Conclusion: crepe-tiny is NOT meaningfully worse than crepe-full on real monophonic music — 0.6 pt apart on in-tessitura, 2.3% octave disagreement, and on 10_amazing_brass tiny is actually better (92.8% vs 83.9%). Given tiny is 38x cheaper (RTF 0.28 vs 2.0), tiny stays the default. The earlier samples/jfk.wav octave worry was archival speech — out of domain for a model being shipped for music — and it did not generalize.

Per-clip, the failures are domain limits shared by both capacities, not capacity defects:

⚠️ Caveats on this eval, stated so it is not over-trusted. (a) The tessitura bounds are hand-guessed per instrument, so the absolute in-range numbers are soft — 01_violin_scale reads 86%/80%, which is more likely my bounds than real error. The tiny-vs-full comparison is the robust part, since both are scored identically. (b) A first version of this script also reported “fraction within +/-50 cents of the nearest semitone” at exactly 100.0% for every clip and both models — that metric is vacuous by construction (deviation from the nearest semitone is bounded to +/-50c) and was removed. It is not evidence of anything. (c) Real per-frame ground truth (a labelled MIR dataset) is still the honest way to get an absolute note-F number against the handoff’s “note-F >= 0.9” gate.

Two measurement traps hit while benchmarking (both in the dev doc already)

CREPE blueprint — the geometry the C++ must hit

Traced from torchcrepe/model.py + core.py + convert.py (the source, see the warning below). Input is a 1024-sample 16 kHz frame, per-frame normalized (-= mean, /= max(std, 1e-10)); hop is 10 ms; pad=True zero-pads WINDOW_SIZE//2 each edge.

Per layer: F.pad -> conv -> F.relu -> batch_norm -> max_pool2d(2).

layer K stride pad (l, r) out ch (full / tiny) T out
conv1 512 4 254, 254 1024 / 128 1024 → 256 → 128
conv2 64 1 31, 32 128 / 16 128 → 64
conv3 64 1 31, 32 128 / 16 64 → 32
conv4 64 1 31, 32 128 / 16 32 → 16
conv5 64 1 31, 32 256 / 32 16 → 8
conv6 64 1 31, 32 512 / 64 8 → 4

Then permute to (T, C) — C is the fast axis — flatten to in_features (4 × 512 = 2048 full, 4 × 64 = 256 tiny), classifier Linear → 360, sigmoid. Decode: cents = 20 * bin + 1997.3794084376191, Hz = 10 * 2**(cents/1200).

Three traps, all now pinned by tools/crepe_numpy_parity.py:

  1. ReLU is BEFORE BatchNorm. So the conv+BN fold is invalid. BN ships as a standalone per-channel affine (_BN.scale, _BN.offset, computed in f64).
  2. conv2..6 padding is asymmetric (31, 32) and Metal rejects an asymmetric GGML_OP_PAD — use symmetric p=32 and drop output column 0.
  3. torchcrepe.convert.bins_to_cents applies dithering (triangular noise), so the reference is non-deterministic. Disable it when dumping parity fixtures, and do not implement it in C++. Also note torchcrepe’s default decoder is Viterbi, not the handoff’s weighted-average-around-argmax — implement local_average (original CREPE) and treat Viterbi as optional.

⚠️ Lesson (HARD RULE #1, the expensive way). The first converter folded BN into the conv, because a fetched summary of model.py listed the ops as “Batch Norm … ReLU activation” in that order. The real source has the relu first. The failure looked like plausible numerics, not a structural bug: layer 1 at cos=0.83 with ~2× the reference magnitude — because least-squares fitting an affine through a rectified signal recovers about half the true scale. What caught it in one run was printing |mine| and |ref| per stage and noticing |mine| was identical across four different input frames. A fetched summary of source is not reading the source.


Verdict: yes for the neural models, no for two of the seven

The handoff lists 7 workers. They are not the same kind of thing — four are neural models (a CrispASR port makes sense), two are pure score-level algorithms (they belong in Dart), and one is already shipped here.

Worker CrispASR? Status / why
W-SEP already done HTDemucs (src/htdemucs.cpp, §248 full parity, cstr/htdemucs-GGUF) + Mel-Band RoFormer (src/mel_band_roformer.cpp, waveform bit-exact 2.4e-7). Both shipped with --separate, auto-download, C ABI, Python Session.separate(). Don’t export Open-Unmix to ONNX — call CrispASR.
W-CREPE ✅ port — start here 6-layer 1D CNN on raw 16 kHz audio → 360-bin activation. No STFT, no attention, MIT. The single easiest port in the repo’s history.
W-PIANO (slice 1) ✅ port Kong/ByteDance high-res piano CNN + biGRU on log-mel. core/mel.h covers the front-end; needs a GRU in core/ (only LSTM exists today).
Basic Pitch ✅ port Already ONNX in the app; Apache-2.0, ~4 MB CNN over a harmonic-CQT stack. Needs a CQT front-end (absent).
W-HARMONY ⚠️ port, licence-gated Small CRNN/CQT chord model. Architecture is easy; the work is finding a checkpoint whose licence is actually permissive. Timebox the checkpoint hunt before the port.
W-DRUMS ⚠️ mostly DSP Onset + band-energy classification is DSP, and DSP belongs where the app is. Only worth a backend if a permissive drum-transcription CNN is chosen.
W-MT3 (slice 2) ⚠️ frontier, timebox T5 encoder-decoder over spectrogram frames → MIDI-like tokens, Apache-2.0. The architecture is well-trodden in ggml (easier than ONNX, honestly). The risk is the checkpoint format — T5X/JAX gin, not HF safetensors — so the converter is the whole job. Feasibility memo before committing.
W-METRE not CrispASR Downbeat DP + metrical quantisation. No model, no tensors. Pure algorithm over a RhythmGrid. Keep in Dart.
W-NOTATION not CrispASR Voice separation, staff split, enharmonic spelling — operates on crisp_notation score types, not audio. Keep in Dart.

So: 5 of 7 are worth porting, 1 is already done, 2 should stay in Dart.

Why port at all, given ONNX works

  1. W-SEP is the handoff’s “biggest lever” and it already exists here, at higher quality than the Open-Unmix fallback the handoff proposes, with per-stage cosine parity already validated. That alone justifies the seam.
  2. One runtime for the whole chain. Separation → F0 → notes currently means ONNX Runtime plus whatever runs the stems. CrispASR already owns the audio IO, resampling, chunking, and model auto-download.
  3. Quantization. crispasr-quantize gives q8_0/q4_k for free; these models ship as f32 ONNX. CREPE-full at q8_0 is a phone-sized model.
  4. Metal / CUDA / WASM come from ggml, not from a per-model ONNX EP story.

The counter-argument is honest and worth stating: for Basic Pitch and CREPE specifically, ONNX already works in the app today, and porting buys speed and packaging, not capability. The capability wins are W-SEP (done), piano, and MT3.


Architecture: a new task surface, not a transcribe() overload

docs/source-separation-surface.md already settled this argument for stems: a task that returns something other than crispasr_segments must not be layered onto transcribe(); it gets its own early dispatcher before the ASR backend is constructed. Music transcription (audio in → note events out) is the same shape, so it copies that design:

Contract compatibility. The handoff freezes contracts.dart (PitchFrame / NoteEvent / RhythmGrid). core/note_events.h is designed to be a 1:1 memory-layout match so the Dart FFI binding is a reinterpret, not a marshal. That is the whole point of the seam — an engine swaps behind it.


Phase 0 — infrastructure (blocks everything else)

The survey turned up three real gaps. None is hard; all are prerequisites.

  1. core/stft.h — forward STFT. core/istft.h exists but covers only the inverse. HTDemucs rolls its own (src/htdemucs.cpp:548 compute_stft) and mel-band-roformer has a second copy. A music backend would be the third copy. Extract now, before adding to the pile. ⚠️ This refactors two shipped backends → per the A/B rule, it needs byte-identical stem output on both before it lands, gated if not.
  2. core/cqt.h — constant-Q / harmonic-CQT. Absent entirely. Basic Pitch and every chord model want log-frequency bins. Built on (1).
  3. core/gru.h. core/lstm.h has uni/bidirectional LSTM; the piano model needs biGRU. Mirror the LSTM file’s structure.

Ordering: (1) → (3) can proceed in parallel with CREPE, which needs neither.

Why first: it needs zero new infrastructure. Raw 16 kHz waveform in (1024-sample frames), 6 conv+batchnorm+maxpool blocks, one 360-unit dense layer out. No STFT, no attention, no autoregression, no tokenizer. It exercises the entire new music surface end-to-end — CLI flag, capability bit, note-event result type, converter, registry, C ABI, bindings — against the simplest possible model, which is exactly how you want to debug a new surface.

Phase 2+ — piano, Basic Pitch, harmony, drums, MT3

Sequenced after phase 1 proves the surface. Each follows the same regime: blueprint read line-by-line → converter → per-stage diff → decoded-output gate → registry + 12-point checklist. MT3 gets a feasibility memo (checkpoint conversion viability) before any C++ is written.


Licence scoping of the remaining roster (2026-07-20)

Every candidate below was checked against the CometBeat HARD RULE — patent-free and MIT/Apache-2.0-compatible — by reading the actual LICENSE file or HF card, not from memory. Code licence and WEIGHTS licence are tracked separately, because for chords they diverge and that divergence is the whole problem.

Component Code Weights Verdict
CREPE (marl, torchcrepe) MIT MIT ✅ shipped
onnxcrepe (yqzhishen) MIT converted from torchcrepe + TF CREPE ✅ useful as an ONNX cross-check
mangio-crepe (Mangio-RVC-Fork) MIT — same CREPE weights nothing to port — see below
RMVPE (Dream-High) Apache-2.0 MIT (lj1995/VoiceConversionWebUI) ✅ clean — best quality tier
FCPE (CNChTu/TorchFCPE) MIT MIT repo ✅ clean — cheapest tier
w-okada/voice-changer MIT (6 holders incl. RVC, yxlllc) mixed; Beatrice v2 is a custom licence ⚠️ integration reference only — do NOT vendor
anyf0 (SoulMelody) MIT wraps crepe/fcpe/rmvpe ✅ good reference implementation
Basic Pitch (Spotify) Apache-2.0 Apache-2.0 ✅ clean — blocked on CQT, not licence
piano_transcription (Kong) MIT MIT ✅ clean — in flight
BTC-ISMIR19 (chords) MIT, ships btc_model{,_large_voca}.pt trained on Isophonics = CC BY-NC-SA ⚠️ THE GATE
MT3 Apache-2.0 T5X/JAX gin checkpoint ⚠️ converter is the whole job
madmom / Essentia / aubio / Vamp GPL/AGPL + Böck patents ❌ excluded by the hard rule

mangio-crepe needs no port

It is not a different model. Mangio-RVC-Fork’s contribution is a configurable crepe_hop_length on the same MIT CREPE weights; its own README recommends upstream RVC’s CREPE for artifact handling. Our src/crepe.cpp already exposes hop as a parameter, so this is covered. Worth stating plainly so nobody spends a week on it.

The chord problem is DATA provenance, not code

BTC-ISMIR19 is the obvious port — MIT code, pretrained checkpoints committed to the repo, architecture we can already build (bi-directional self-attention over CQT; every op exists in the CrispASR ggml stack). The catch is upstream of the code: its checkpoints were trained on Isophonics annotations, which are CC BY-NC-SA (non-commercial, share-alike), as are Robbie Williams and UsPop2002.

Whether NC-licensed annotations encumber the resulting weights is legally unsettled, and the repo ships them under MIT. But “unsettled” is not the bar this project set. Three options, in order of preference:

  1. Retrain on ChoCo’s permissive subset. ChoCo aggregates 18 chord corpora under CC BY 4.0, with only three NC exceptions to exclude (Chordify Annotator Subjectivity, Mozart Piano Sonata, JAAH). That leaves Billboard, Real Book, RWC-Pop, Weimar Jazz, Wikifonia, iReal Pro, Band-in-a-Box, When in Rome, Rock Corpus, Nottingham, Schubert-Winterreise — ample for a small CRNN, with commercially-clean provenance we can state in the model card.
  2. Synthetic audio. There is recent work on training chord recognisers on artificially generated audio (arXiv 2508.05878). Rendering progressions from permissive symbolic sources gives fully clean provenance and pairs well with option 1 as augmentation.
  3. Ship the chroma-template path (already in crisp_notation chroma_analysis.dart / analyze()) as the default and treat the neural chord model as a later premium tier.

DECISION (2026-07-20): port BTC now, gate the weights non-commercially

Superseding the “train first” recommendation above. We ship the BTC port with its upstream checkpoints, treated as non-commercial weights behind a download-time attestation — the same posture the repo already takes for Voxtral-4B-TTS (CC-BY-NC-4.0) and the German moonshine models (CC-BY-NC-SA-4.0).

Why this is sound:

Training a clean model on the ChoCo CC-BY subset (option 1 above) remains the target for a commercially usable chord tier. It is now a follow-up, not a prerequisite, and the BTC port is what proves the architecture + surface first.

Required mechanism — MIRROR CrispEmbed, do not invent one

--i-have-rights is the WRONG flag. It attests speaker consent for voice cloning — a third-party-rights question. Licence compliance is unrelated, and one flag must not silently grant two different permissions. A separate mechanism is required.

CrispEmbed already has the right one (examples/cli/model_mgr.{h,cpp}), and it is stricter than anything CrispASR does today. Mirror it rather than inventing a parallel design — the two repos should behave identically:

CrispEmbed (existing) CrispASR (to add)
license_requires_acceptance(spdx)cc-by-nc*, gemma, llama*, lfm1.0, other same predicate, same tag list
resolve_model(arg, auto_download, accepted_license) extend crispasr_resolve_model() with the same parameter
--accept-license <spdx> --accept-license <spdx>
CRISPEMBED_ACCEPT_LICENSE env fallback CRISPASR_ACCEPT_LICENSE
accepts the exact SPDX tag, or all / * same
TTY: prints licence + model-card URL, prompts [y/N] same
non-TTY without acceptance: refuses same
auto_download alone is NOT sufficient same — this is the key property

Two things CrispEmbed’s design gets right that a blanket NC flag would not:

  1. Acceptance is per-licence, not blanket. The user attests to a specific SPDX tag; all exists but is opt-in.
  2. SPDX tags, not substring matching. CrispASR currently tests license.find("NC"), and that same NC-detection logic is duplicated in three places (crispasr_model_registry.cpp + two spots in crispasr_model_mgr_cli.cpp). Moving to SPDX tags de-duplicates it.

What CrispASR must change:

CometBeat mirrors the same gate in its model store before fetching, and states the restriction in the UI at the point of download — not buried in an About box.

Follow-up once the ChoCo-trained model exists: it is Apache-2.0, needs no gate, and becomes the default; BTC stays as the opt-in higher-accuracy NC tier.

CQT is the shared unlock

Basic Pitch and the chord CRNN both need a constant-Q transform, which core/ does not have (only core/mel.h and core/fft.h). Building core/cqt.h once unblocks BOTH, and is the highest-leverage remaining infrastructure item — ahead of either model port.

F0 tier — CREPE is shipped, RMVPE is the quality upgrade

The handoff already flags RMVPE as “the quality tier after CREPE”, and the licence check confirms it is clean (Apache-2.0 code, MIT weights). It is also what w-okada’s guide recommends for all-purpose use, and it is robust to accompaniment — which matters because our W-SEP stems are not perfectly clean. FCPE is the cheap tier if CREPE-tiny proves too slow on low-end hardware. Priority: RMVPE > FCPE, and neither is urgent while CREPE-tiny hits RTF 0.28.


Additional CrispASR tasks from the cross-runtime scoping (2026-07-20)

Fell out of scoping which models CometBeat’s pure-Dart ONNX runtime can carry. That runtime can afford ~10–15 min for an offline whole-song analysis job, which is a very different budget from interactive use — and it changes what is worth having on the CrispASR side too.

A cheap-separator tier

CrispASR has the two best separators (HTDemucs, Mel-Band RoFormer) and neither of the cheap ones. That is a real gap for low-end hardware and for the pure-Dart path:

Candidate Licence Architecture Why
Spleeter 4-stem (Deezer) MIT 12-layer U-Net on magnitude spectrograms — all convs Cheapest separator that exists; 100x realtime on GPU. All-conv maps straight onto the existing im2col/GEMM path — no new op families.
Open-Unmix (umx/umxhq) MIT 3-layer BiLSTM on magnitude spectrograms Named in the handoff. Cheaper than HTDemucs but SEQUENTIAL, so it parallelises badly; expect it to lose to Spleeter despite fewer FLOPs.

Task: port Spleeter first (core/lstm.h is not even needed — it is pure conv), measure against HTDemucs on the same clip, and register it as the low-resource separation default. Open-Unmix only if Spleeter’s TF-checkpoint conversion turns out awkward.

Reference point for why this matters: HTDemucs costs ~103 s per 7.8 s segment in a pure-Dart runtime — ~45 min for a 3.5-minute song, over any usable budget — while an all-conv U-Net is roughly an order of magnitude cheaper.

Pitch tiers below and above CREPE

Not worth porting

Division of labour with the pure-Dart runtime

CrispASR is the native/performance path; the Dart runtime is the web/WASM path and the fallback. They should NOT both chase the same models. CrispASR keeps the heavy, highest-quality engines (HTDemucs, Mel-Band RoFormer, RMVPE, and MT3 if it ever lands); the Dart runtime takes the small permissive ones it can actually finish in-budget (Basic Pitch, CREPE-tiny, BTC chords, Spleeter). core/cqt.h and a Dart CQT are the one piece both need — worth keeping the two implementations diff-checkable against each other.

Open questions

Suggested order (highest leverage first)

  1. Licence-acceptance gate, ported from CrispEmbedDONE — land BEFORE any NC weights are registered, so there is never a window in which they are downloadable ungated. Also de-duplicates CrispASR’s three copies of substring-based NC detection.
  2. core/cqt.h — unblocks Basic Pitch AND the chord model. Infrastructure, no licence risk, reusable.
  3. BTC chords — architecture from the paper, weights gated NC.
  4. Finish piano_transcription — currently cos 0.971, below the 0.999 gate. It is the closest thing to a finished port that is not yet finished.
  5. Basic Pitch — Apache-2.0 end to end, and the app already depends on it via ONNX, so this is a like-for-like replacement with a known-good oracle.
  6. RMVPE — clean licence, real quality win on sung f0 over accompaniment.
  7. ChoCo-trained chord model — the commercially-clean tier, Apache-2.0.
  8. MT3 — feasibility memo on the T5X checkpoint conversion FIRST.

Everything above is CPU/Metal-verifiable locally. The Kaggle/CUDA run should wait until this roster is complete, so one clean CUDA session covers every backend at once rather than being repeated per port.


CometBeat Q&A — answers as of 2026-07-20 (§251)

Four asks came back from the CometBeat agent. Answers, with the facts checked against origin/main rather than assumed:

1. separate() Dart binding — DONE, stop shelling out to the CLI. Landed on main (05ee77b17). CrispasrSession.separate(Float32List pcmStereo) -> List<Stem> where Stem = ({String name, Float32List pcm}), plus a separateSampleRate probe. Verified end-to-end through the real FFI against cstr/htdemucs-GGUF q4_k: 4 stems (drums/bass/other/vocals), interleaved lengths exactly matching the input, ~4 s for 2 s of audio. ⚠️ Two contract points: input is interleaved stereo (L,R,L,R…) at 44100 Hz, and the native side counts samples per channel — an odd-length buffer now throws ArgumentError rather than being misread. To chain into pitch(), downmix to mono and resample 44100→16000 first; the dartdoc carries the recipe. Not on pub.dev yet — it needs an 0.8.17 release.

2. Piano — bindable TODAY, but through transcribe(), not a note-event API. piano-transcription is session-openable now (crispasr_session_open(path, backend: "piano-transcription"); it is in the C ABI backend list). There is no dedicated note-event C ABI. The CLI adapter converts each detected note into one crispasr_segment: t0/t1 are onset/offset, and text is the note name plus velocity (e.g. "C4 v=80"), with CAP_TIMESTAMPS_NATIVE. So loadCrispasrPiano can be un-stubbed immediately by opening that backend and parsing segments — but string-parsing "C4 v=80" back into a NoteEvent is lossy and ugly. A proper crispasr_session_piano_notes* C ABI returning {midi, onMs, offMs, velocity} is the right fix and is now a §251 item. Say if you want it prioritised — it is small, and it is the difference between a hack and a real seam.

3. The dylib — correcting an assumption: the pub package does NOT ship or download one. crispasr is a pure Dart FFI package; it opens whatever native library the host app provides. CrispasrSession.open probes, in order: libcrispasr.dylib, crispasr.framework/crispasr, libwhisper.dylib, whisper.framework/whisper on macOS/iOS (libcrispasr.so on Linux/Android), or you pass libPath: explicitly. Canonical filename: libcrispasr.dylib. Your cached 0.8.10 has no crispasr_session_pitch, which is exactly why your fallback fires — correct behaviour. You need a dylib built from 0.8.16+ for pitch + the crepe registry entry, and 0.8.17+ for separate(). Build it from the repo (cmake --build build --target crispasr) and bundle build/src/libcrispasr.dylib. Your tiny-q8_0 (~0.50 MB) choice is right — q4_k moves the argmax pitch bin on ~1 frame in 7 at tiny.

4. BTC licensing — your read is correct. Keep your MIT ONNX BTC plus the chroma-template fallback for the commercially-clean tier; do not ship our weights. Our port gates them behind an attestation precisely so the two can coexist. Cross-checking your harmony_cqt.dart against tools/cqt_librosa_parity.py is worthwhile even though yours is already librosa-validated — ours agrees with librosa at median 0.9999 per-frame shape correlation and 97.6% exact peak-bin, with disagreement confined to tone-transition frames (librosa’s per-octave recursive downsampling gives each octave a different group delay; our direct kernels are centred uniformly). If your numbers differ materially at the transitions, that difference is expected and not a bug in either.

New §251 item from this exchange


§251b — Model scoping, 18 candidates (2026-07-20)

Scoped 18 HuggingFace music models. Three are worth porting; 13 are licence-dead or off-contract; 2 are informative only. Licences were verified from LICENSE files and upstream repos, not just card YAML.

⚠️ PREMISE CORRECTION — upstream BTC is MIT, weights included

jayg996/BTC-ISMIR19 carries a plain MIT LICENSE (Copyright 2019 Jonggwon Park) and ships both checkpoints inside that MIT repo: test/btc_model.pt (12,154,754 B) and test/btc_model_large_voca.pt (12,229,576 B). Verified via the GitHub licence + contents API.

This contradicts the §BTC decision to gate the weights non-commercially behind crispasr_accept_noncommercial() / --accept-noncommercial. That apparatus may rest on a wrong premise — most likely a third-party HF mirror was mistaken for the source. Owner of the BTC work should confirm which artifact the NC belief came from before more is built on the gate. Not unilaterally removed here: the attestation machinery is still useful for genuinely NC models, and the call belongs to whoever made it.

✅ Worth porting

Model Licence Size Fills Front-end we already have
musetric/beat-this-onnx MIT (weights explicitly, upstream CPJKU/beat_this MIT) 83 MB, 20.25 M RhythmGrid — the one seam with no model 128 Slaney mel @22050, n_fft 1024 / hop 441 → core/mel.h; filterbank ships as a raw [513,128] f32 blob, load verbatim
musetric/chordmini-onnx MIT (upstream ptnghia-j/ChordMini MIT) 9.6 MB, ~2.4 M chord timeline librosa CQT, 144 bins, 24/oct, fmin C1, hop 2048 — exactly what core/cqt.h was validated against
livechord-music/livechord-beat-refiner Apache-2.0, code AND weights, ungated 12.8 MB, 3.19 M beat/downbeat refinement 84 CQT (C1, 12 bpo) + chroma + onset + RMS @22050/hop 2048 → core/cqt.h

beat-this is patent-clean and madmom-free — verified: its dependency list is numpy/torch/torchaudio/einops/rotary-embedding-torch/soxr. Its paper is literally “Accurate Beat Tracking Without DBN Postprocessing”, so no Böck-patented post-processing enters the chain even by distillation. That matters because CometBeat must avoid madmom’s DBN entirely.

chordmini is a distilled STUDENT of the BTC teacher — ~2.4 M params against BTC’s much larger body, MIT, 170-chord vocab in config.json. Worth an explicit decision by the BTC owner: it could validate the BTC port or supersede it at a fraction of the size.

beat-refiner needs a prior grid — it is a refiner, not a tracker. That is the opening rather than the limitation: CometBeat’s Ellis DP is patent-free but phase-drifty, and drift is what this was trained to fix (with random grid corruption, so it does not merely copy its input). Ellis DP → refiner is a fully patent-free chain. It has only ever been evaluated on beat_this input, never Ellis, so A/B before committing. Its chord-boundary head is explicitly not production-ready (cb F1 0.243) — ignore that head.

❌ Licence-dead or off-contract

ℹ️ Informative only

Porting gotchas that would silently corrupt a port

Order

⚠️ Training-data provenance is undocumented upstream for the musetric exports, and beat_this’s own README notes some training files are copyrighted or under limited CC licences. The weights licences are verified clean; provenance is a separate, unresolved upstream question.


§251b-1 — beat-this port: traced blueprint (converter DONE, runtime TODO)

models/convert-beat-this-to-gguf.py is landed and validated (147 tensors, 40.8 MB f16; fold exact at f32, rel 1.9e-4 at f16). What follows is the forward pass traced from beat_this/model/{beat_tracker,roformer}.py so the runtime can be written without re-deriving it.

Two subtleties that would silently break the port

1. RMSNorm here is written in a form that does not look like RMSNorm.

self.scale = size ** 0.5
self.gamma = nn.Parameter(torch.ones(size))
forward: F.normalize(x, dim=-1) * self.scale * self.gamma

F.normalize divides by the L2 norm, and RMS = L2 / sqrt(size), so normalize(x) * sqrt(size) == x / RMS(x). It is algebraically standard RMSNorm times gamma, and maps to ggml_rms_norm + ggml_mul. Do not reimplement the L2 form literally and do not assume a bias — there is none.

2. Attention has PER-HEAD sigmoid gating, applied before the out-projection.

gates = self.to_gates(x)                    # Linear(dim -> heads), WITH bias
out = out * gates.sigmoid()                 # broadcast over (head_dim, seq)
out = to_out(out)                           # Linear(dim_inner -> dim), NO bias

to_gates emits ONE scalar per head (not per channel), sigmoided and broadcast across that head’s head_dim and sequence. Miss it and the output is plausible but wrong. Note the bias asymmetry: to_qkv and to_out are bias-free, to_gates has a bias.

Block order

Attention(x):  RMSNorm -> to_qkv(no bias) -> RoPE(q), RoPE(k) -> attend
               -> * sigmoid(to_gates(x)) per head -> to_out(no bias)
FeedForward:   RMSNorm(net.0.gamma) -> Linear(net.1) -> GELU -> Linear(net.4)
               (indices 1 and 4 are the checkpoint's own numbering)

Frontend PartialFTTransformer runs attnF/ffF over the FREQUENCY axis then attnT/ffT over TIME, per block, with a shared RotaryEmbedding(32).

Tensor names (from the checkpoint, verified)

frontend.stem.bn1d.{scale,offset}                 <- pre-conv per-FREQ affine, NOT folded
frontend.stem.conv2d.{weight,bias}                <- BN folded in
frontend.blocks.{0,1,2}.partial.{attnF,attnT}.norm.gamma
frontend.blocks.{0,1,2}.partial.{attnF,attnT}.to_qkv.weight
frontend.blocks.{0,1,2}.partial.{attnF,attnT}.to_gates.{weight,bias}
frontend.blocks.{0,1,2}.partial.{attnF,attnT}.to_out.0.weight
frontend.blocks.{0,1,2}.partial.{ffF,ffT}.net.{0.gamma,1.weight,1.bias,4.weight,4.bias}
frontend.blocks.{0,1,2}.conv2d.{weight,bias}      <- BN (named `norm`!) folded in
frontend.linear.{weight,bias}                     <- 1024 -> 512
transformer_blocks.layers.{0..5}.{0=attn,1=ff}.*
transformer_blocks.norm.gamma                     <- norm_output
task_heads.beat_downbeat_lin.{weight,bias}        <- 512 -> 2
aux.mel_filterbank                                <- [513,128] baked verbatim

⚠️ The BN module name differs by location: bn2d in the stem, norm in the frontend blocks. Found only because the converter’s guard aborted rather than silently mis-folding.

Remaining work