Last night I discovered that my AI transcription system had been confidently transcribing the wrong city commission meeting -- and I didn't notice for hours.

The Setup

Democracy.cc provides AI-generated transcripts for Miami Beach Commission meetings. The pipeline downloads meeting video, extracts audio clips with FFmpeg, sends them to a Modal GPU worker for speaker diarization + Whisper transcription, and saves the results.

For the February 25th meeting, I renamed the video file from full_video.mp4 to meeting_5861.mp4 to bust a CDN cache issue. All agenda items were updated to point to the new filename.

The Symptom

I kicked off diarization for item C7U -- a heated 63-minute debate about bicycle and pedestrian paths near a golf course. 45 minutes later, the A100 GPU returned 106 transcript segments. I searched for "bicycle", "golf", "pedestrian" -- zero matches. Every segment was about zoning incentives and Washington Avenue revitalization. A completely different meeting.

The Investigation

I extracted a probe clip from the video at the right position -- correct content was right there. The video was fine. The timestamps were fine. Something between "look up the video URL" and "run FFmpeg" was wrong.

The Bug

private function getFullVideoUrl(AgendaItem $item): string
{
    $videoUrl = $item->video_url;

    if (str_contains($videoUrl, 'full_video.mp4')) {
        return preg_replace('/[?#].*$/', '', $videoUrl);
    }

    $meetingId = $item->meeting_id;
    return "https://...cdn.../meetings/{$meetingId}/full_video.mp4";
}

Hidden assumption: every video file is named full_video.mp4. When C7U's URL was meeting_5861.mp4?t=26708, the str_contains check failed. The fallback returned full_video.mp4 -- the old, wrong file from February 5th still sitting in cloud storage.

FFmpeg dutifully extracted audio from that file. The GPU dutifully transcribed it. The webhook dutifully saved 106 segments. Everything worked perfectly, just on the wrong meeting.

Why It Took So Long

  • No content validation. The pipeline doesn't check if transcribed text matches the agenda topic. Audio in, text out, success.
  • Plausible output. Same commissioners, similar procedural language, overlapping topics. You'd have to read the transcript to notice.
  • Timestamps looked correct. Right positions, wrong video file.

The Fix

private function getFullVideoUrl(AgendaItem $item): string
{
    if (!$item->video_url) {
        throw new \RuntimeException("No video URL for item {$item->item_number}");
    }
    return preg_replace('/[?#].*$/', '', $item->video_url);
}

Use the actual URL. Fail loudly if there isn't one. A wrong transcript is worse than no transcript.

The Lesson

When a pipeline processes the wrong input and produces plausible output, no amount of monitoring will catch it. The GPU didn't error. The webhook didn't fail. Every health check passed. The only signal was me reading the transcript and saying "this isn't about bicycle paths."