MEMORY NOTCH BLOG

What is speaker diarization, and where it fails

Speaker diarization splits a recording by voice and numbers each speaker. How it works, why it merges or invents voices, and what DER measures.

Published October 23, 2026 · The Memory Notch team

Speaker diarization is the step that works out who spoke when in a recording. It splits the audio by voice and gives each voice a label, such as Speaker 1 and Speaker 2, without knowing anyone's name. On September 23, 2026, NVIDIA released Nemotron 3 Diarization, an open model that labels up to 8 speakers live, about a second behind the audio.

What it answers: "who spoke when", not "who is this person".
What you get: stretches of time, each tagged with a numbered speaker.
Where it slips: two similar voices merged into one, a cough counted as a new person, people talking over each other, and more voices than the model has room for.
How it's scored: diarization error rate (DER), the share of speaking time labelled wrong.
Where it runs: on servers, and now on a laptop. Memory Notch, our Mac app, runs Nemotron 3 on the Mac.

Checked September 29, 2026.

What is speaker diarization?

Speaker diarization splits a recording into turns and groups the turns that sound like the same voice. The result is a timeline: Speaker 1 from 0:00 to 0:14, Speaker 2 from 0:14 to 0:31, Speaker 1 again after that.

NVIDIA's announcement puts it in one line: diarization "classifies who spoke when, helping applications connect what was said to the right participant." The speech-to-text model writes the words. The diarizer decides whose words they are. A transcript needs both before it reads like a conversation instead of one long paragraph.

How does diarization work?

A diarizer listens in slices of a few milliseconds, estimates which voices are active in each slice, and joins the slices into turns. There are two common designs.

The pipeline design runs several models in a row. One finds speech, one detects where the voice changes, one turns each stretch of voice into a numeric fingerprint (an embedding), and a clustering step groups fingerprints that sit close together. pyannote, a widely used open-source toolkit, is built from these parts. It can handle any number of speakers, because clustering can always add one more group.

The end-to-end design uses one model that outputs, for every short frame, a probability for each speaker slot. Nemotron 3 works this way. Its model card says it outputs a probability for each of 8 speaker slots every 10 milliseconds. Because each slot is scored separately, two slots can be active in the same frame, so overlapping speech is part of the output rather than an afterthought.

Then the words have to meet the speakers. The speech model gives each word a start and end time, and the diarizer's timeline says who was talking then. NVIDIA's blog describes a simple "midpoint rule" for this, which marks words spoken while two people are active as ambiguous rather than guessing.

Live diarization adds a time limit. The model has to commit to a label while the person is still talking. Nemotron 3 offers presets from 0.32 seconds to 30.4 seconds of delay. The longer presets see more audio before deciding, so they make fewer mistakes, but they can't label a live transcript.

In Memory Notch, a speaker counts as talking when the model's probability for them stays above one half for at least 0.2 seconds. That threshold is a product decision, not part of the model, and it's one example of how much of diarization happens around the model rather than inside it. Our guide to how AI transcription works covers the rest of that path, from the microphone to the note.

Diarization vs speaker identification: does it know names?

No. Diarization tells voices apart. Speaker identification matches a voice to a known person, and it only works after each person has recorded a voice sample (called enrollment).

pyannoteAI's explainer draws the same line. Diarization "requires no prior knowledge of the speakers." Identification asks "which person from this known pool is speaking?"

Task	Question it answers	Needs voice samples first?	Output
Diarization	Who spoke when?	No	Speaker 1, Speaker 2…
Speaker verification	Is this the person they claim to be?	Yes, one person	Yes or no
Speaker identification	Which of these known people is speaking?	Yes, everyone in the pool	A name per turn

So when a meeting app shows real names next to lines, it usually gets them from somewhere else: calendar invites, the call platform's participant list, voice profiles it keeps, or you typing them in.

Memory Notch numbers speakers and stops there. It keeps no voice profiles and doesn't match voices to people. Numbers are assigned in the order voices first appear, so the first person to talk is Speaker 1. You can put names back in with find-and-replace.

Why does diarization get speakers wrong?

Most errors come from four situations: two voices that sound alike, stray sounds taken for a new person, people talking at once, and more people than the model can track.

Merged voices

Two people with similar pitch and pace can end up under one label. A pipeline diarizer merges them when their fingerprints fall inside its clustering threshold. An end-to-end model does it by putting both into one slot. Either way, the transcript shows one speaker having a conversation with themselves.

Phantom speakers

The opposite error is a speaker who doesn't exist. A cough, a laugh, a door, or a second of echo can land in its own slot and become Speaker 4 for two seconds.

We see this in real recordings. On one 232-second call, the model gave two phantom speakers 0.3 seconds and 1.9 seconds of airtime, while the real voice held the rest. Real voices hold tens or hundreds of seconds, so airtime is what separates them. When Memory Notch saves a note, it drops any speaker with under 5 seconds of airtime and under 5% of that channel's speech, and moves their words to the nearest real voice.

The pruning applies to the saved note only. The live view keeps every label steady. A voice that looks like a phantom at second 10 may be a real person by second 60, and pruning mid-call would make labels flicker as the verdict changed.

Overlapping speech

When two people talk at once, the audio is a mix, and any model has less to go on. An end-to-end model can mark both as active, but the words still come out of one transcript, so someone's interruption can end up attributed to the wrong person. Overlapping voices, and several people sharing one microphone, remain hard to separate. That's a limit in Memory Notch too.

Speaker limits

An end-to-end model has a fixed number of slots. NVIDIA's Streaming Sortformer v2.1 has four, and its model card says "performance degrades on recordings with 5 and more speakers." A fifth voice has to share a slot with someone else. Nemotron 3 raises the ceiling to eight.

A higher ceiling doesn't make big groups easy. On DIHARD III, a public test set of hard, varied recordings, NVIDIA's own card lists Nemotron 3's error rate at 9.47% for one to four speakers and 28.65% for five to nine, at the 1.04-second preset. More voices means more chances to confuse them.

How accurate is speaker diarization?

Accuracy is measured as diarization error rate (DER): the share of speaking time that got the wrong label. A DER of 10% means that, across an hour of speech, about 6 minutes were attributed wrongly.

DER adds up three kinds of mistake, according to the pyannote.metrics documentation:

Missed speech: someone talked and no speaker was marked.
False alarm: a speaker was marked where nobody talked.
Confusion: the right time was labelled with the wrong speaker.

Many benchmarks also ignore a quarter-second on each side of every speaker change (the "collar"), because human reference labels can't be exact to the millisecond. Two DER figures are comparable only if they used the same test set and the same collar, and whether overlapping speech was counted.

Numbers vary a lot by audio. Broadcast debates, meeting-room recordings and phone calls give very different scores to the same model. In our own test of 24 recordings, run through Memory Notch's live pipeline, Nemotron 3 labelled 4.45% of speaking time wrong, against 14.07% for Sortformer. For how the two main open-source families compare, see pyannote vs NeMo diarization.

DER also misses something you notice as a reader. A model that gets 95% of the time right can still swap two people's labels halfway through, and one swap like that makes a transcript harder to follow than many small boundary errors.

Can speaker diarization run on your own computer?

Yes. Modern diarization models are small enough for a laptop. Nemotron 3 has 100 million parameters, and FluidAudio, an open-source Swift library for running speech models on Apple hardware, added it on the same day NVIDIA released it. NVIDIA's own card lists only its data-centre and desktop graphics cards, so running it on a Mac means a converted version like FluidAudio's.

Running it locally changes what you have to trust. The audio never has to go to a diarization service, and the labels arrive at the same pace online or off.

Memory Notch runs Nemotron 3 on the Mac through FluidAudio, next to its live speech model. It records two sources, your microphone and the call's audio, and diarizes each one separately, with up to 8 speakers per channel. That means the people on the call and the people in your room each get their own slots, instead of competing for one set.

A good model is only part of the result. The threshold, the airtime pruning, and keeping live labels steady all sit in the app around it.

What diarization can't do yet

It doesn't know names. Any tool that shows names gets them from somewhere other than the voice.
Overlapping speech stays hard. A model can mark two people as active, but the words in the overlap can still land on the wrong one.
Several people on one microphone are harder to separate than people on separate channels.
Past four or eight voices, depending on the model, accuracy drops, and even under the limit, larger groups score worse.
Published DER figures come from test sets that may not sound like your calls. Our own test used public meeting and broadcast recordings, not Memory Notch recordings.

In Memory Notch, recording, transcription and speaker labels run on your Mac. The app makes three network requests, and none of them carries a recording; the privacy policy lists them. Labels are numbered, not named, and there are no automatic summaries. Running the model locally doesn't replace the consent of the people you record. You can try it free for recordings of up to 5 minutes, with no account.

Frequently asked questions

What does diarization mean in speech recognition?

In speech recognition, diarization is the step that splits a transcript by speaker. The speech model turns audio into words, and the diarizer decides which voice said each stretch, labelling them Speaker 1, Speaker 2 and so on. Without it, a meeting transcript is one unbroken block of text with no sign of who said what.

Is speaker diarization the same as speaker recognition?

No. Diarization only tells voices apart within one recording and numbers them. Speaker recognition matches a voice to a known person, which needs a stored voice sample for each person first. Diarization needs no samples and never learns who anyone is, which is why its labels are numbers.

Can Whisper do speaker diarization?

Not on its own. OpenAI's Whisper is a speech-to-text model: it writes the words but doesn't label speakers. Tools that show speaker labels next to Whisper transcripts run a separate diarization model alongside it and combine the two by matching timestamps.

How many speakers can diarization handle?

The limit depends on the design. Pipeline tools that cluster voices have no fixed limit. End-to-end models have a fixed number of slots: four for Streaming Sortformer v2.1, and eight for Nemotron 3 Diarization, released September 23, 2026. Accuracy drops as the number of speakers rises, even below the limit.

What is a good diarization error rate?

There is no single good number, because DER depends heavily on the audio. On one hard test set, NVIDIA's own card shows Nemotron 3 under 10% with up to four speakers and near 29% with five to nine. Compare figures only when they come from the same test set, scored the same way.

Sources

Download Memory Notch · More from the blog