When Commissioner Dominguez speaks at a Miami Beach Commission meeting, AssemblyAI labels her "Speaker E." Next meeting, she's "Speaker C." The labels are arbitrary -- assigned per-file based on who talks first. Until today, every transcript required manual name mapping.

I just shipped cross-file speaker matching: the system now identifies recurring speakers by their voice automatically.

Two Systems That Didn't Talk

AssemblyAI handles transcription and diarization -- excellent at separating speakers within a single file, but no memory between files. Every meeting starts fresh.

Modal GPU (pyannote.audio) generates 512-dimensional voice embeddings -- numerical fingerprints of what someone sounds like. I had a library of enrolled speaker profiles and a cosine similarity matcher. But after AssemblyAI diarized a file, the results came back with 'embeddings': []. The two systems were completely disconnected.

The Hybrid Pipeline

The new PostDiarizationSpeakerMatcher bridges them. After AssemblyAI finishes, it kicks in:

  1. Extract per-speaker audio -- FFmpeg's atrim + concat filter combines each speaker's segments into a single clip (up to 60s)
  2. Generate embeddings -- each clip goes to Modal GPU running pyannote.audio, returning a 512-dim vector
  3. Match against profiles -- cosine similarity against all enrolled speakers (threshold: 0.65)
  4. Enrich results -- matched names and confidence scores merge back into the transcript before saving

First Test: 5 out of 6

Tested on agenda item R7K from the March 5th meeting:

Speaker Match Confidence
Speaker A Alex Fernandez 73%
Speaker B Rafael Granado 69%
Speaker C David Suarez 67%
Speaker D Steven Meiner 66%
Speaker E Unmatched --
Speaker F Ricardo Dopico 79%

Speaker E was Commissioner Dominguez. She didn't match because her profile had only one previous sample (165 seconds), producing an embedding that scored 0.31 -- well below threshold.

The Bootstrapping Problem

The system auto-enrolls voice samples when saving matched transcripts. Matched speakers get more samples, which improves their profiles, which makes future matches more reliable. A virtuous cycle.

But unmatched speakers can't enter this cycle. The fix: manual labeling bootstraps it. An admin labels Speaker E as "Laura Dominguez," a voice sample is extracted and enrolled. After a few manual labels across meetings, the profile becomes strong enough to match automatically.

Bugs Along the Way

Mapping key mismatch. AssemblyAI labels speakers with letters (A, B, C), which I transform to internal labels (SPEAKER_00, SPEAKER_01). When an admin relabeled "E" as "Dominguez," the enrollment code looked for segments labeled "E" -- but they'd already been stored as "SPEAKER_04." Silent failure. Fix: capture the label mapping before updating.

Double video offset. Voice samples from the admin UI stored absolute timestamps (24,535s into the full video). The extraction job added the offset again, seeking to position 48,770 -- over 13 hours into a 7-hour video. Result: a 78-byte empty file. Fix: subtract the offset when creating samples from saved segments.

Both bugs had been silently failing for weeks. Every manual voice enrollment from the transcript UI produced empty samples.