What is Speaker Diarization?
Also called diarization, who spoke when, speaker segmentation.
Speaker diarization is the process of partitioning an audio recording by speaker, answering who spoke when without necessarily knowing who anyone is. It segments the audio, embeds each segment as a voice representation, and clusters those embeddings into anonymous speaker labels. Attaching real names to those labels is a separate step called speaker identification.
A typical pipeline detects speech regions, cuts them into short segments, computes a fixed-length speaker embedding for each, and clusters the embeddings using agglomerative or spectral methods. The number of speakers may be supplied in advance or estimated from the clustering itself. End-to-end neural diarization models that emit per-speaker activity directly are an active alternative, especially for handling overlap.
Overlapping speech is the hardest case. When two people talk simultaneously, segment-level embeddings blend both voices and clustering degrades. Target-speaker extraction and permutation-invariant training address this, but accuracy on crosstalk-heavy recordings remains well below accuracy on orderly turn-taking. Short segments, similar voices, and channel changes mid-recording all add error.
Diarization becomes speaker identification only when labels are bound to known people. That binding can come from enrollment, where a person provides a reference sample matched against later embeddings, or from context, such as a conference platform tagging each participant's own audio stream. Where separate per-participant streams exist, diarization is often unnecessary because the channel already carries the attribution.
Voice embeddings are biometric data in several jurisdictions, including under frameworks that treat voiceprints as sensitive personal information. Systems that enroll and store reference voices generally need explicit consent, defined retention, and a deletion path, and the legal treatment differs enough between regions that policies are usually set per market.
Key points
- Answers who spoke when using anonymous speaker labels
- Segments audio, embeds voices, then clusters the embeddings
- Overlapping speech is the main source of error
- Identification requires enrollment or platform-supplied attribution
- Stored voiceprints are regulated biometric data in some regions
In practice
A four-person sales call is recorded as a single mixed audio file. Diarization splits it into segments and clusters them into four labels, producing a transcript where each line is tagged Speaker 1 through Speaker 4. The meeting platform then maps two of those labels to named participants using their individually captured streams, and a reviewer confirms the remaining two by reading the introductions at the start.