A transcript without speaker labels is like a play without character names. Speaker diarization -- identifying who spoke when -- requires GPU-accelerated AI models, and I needed a way to run them without expensive always-on infrastructure.

Three Platforms in Three Days

DigitalOcean GPU Droplets. The plan was clean: spin up a GPU droplet on demand, run diarization, destroy it. Reality was less clean -- manual SSH setup, dependency installation on every boot, 3-5 minute provisioning, inconsistent GPU availability. It worked, but it was fragile.

RunPod Serverless. Promised the dream: upload a Docker image, define an endpoint, workers scale to zero. I built a 23GB custom image with pyannote and Whisper, pushed it to three container registries, configured the endpoint. Workers refused to provision. Zero workers despite hours of waiting and endpoint recreation.

Modal. Instead of Docker images, you write Python and Modal builds the container. My entire diarization service -- pyannote speaker diarization, Whisper transcription, embedding generation -- is a single 300-line Python file. Deployment: modal deploy modal/diarization_app.py.

It worked on the first try. Within 30 minutes, I had my first successful job. A 46-minute agenda item processed in 16 minutes, producing 694 transcript segments with 15 speakers identified.

The Architecture

  1. Admin triggers diarization from the clips page
  2. Laravel queues batch jobs for all agenda items needing processing
  3. Each item is submitted to Modal with the audio URL and a webhook callback
  4. Modal spins up a GPU container, runs pyannote + Whisper large-v3
  5. Modal calls back with results (segments, speaker labels, embeddings)
  6. Laravel saves everything and matches anonymous speaker labels to known officials using voice embeddings

GPU Optimization

My first deployment used an A100 ($3.70/hr). Monitoring revealed GPU utilization was only 3.38% -- pyannote's spectral clustering runs entirely on CPU. The GPU bursts briefly for voice activity detection, then sits idle.

I downgraded to an L4 ($0.30/hr) -- 12x cheaper -- and upgraded Whisper from base to large-v3. Better transcription, comparable speed, 90% cost reduction. The CPU bottleneck means GPU power barely matters.

Speaker Matching

Raw diarization gives labels like "SPEAKER_00." Not helpful. I built a matching pipeline: extract voice samples from meetings where speakers are known, convert each to a 512-dimensional embedding, then compare against new speakers using cosine similarity. Score above 0.65 = match.

The system automatically identified commissioners, the City Clerk, and staff members -- all from voice patterns.