MEMORY NOTCH BLOG

pyannote vs NeMo diarization: past four speakers

pyannote groups any number of voices; NeMo’s Sortformer stopped at four. NVIDIA’s Nemotron 3 Diarization, out September 23, 2026, tracks eight live.

Published October 23, 2026 · The Memory Notch team

pyannote finds speech, turns each voice into a fingerprint and groups the fingerprints, so it has no fixed limit on speakers. NVIDIA NeMo’s Sortformer is one model that labels Speaker 1, 2 and so on in a single step, and it stops at four. On September 29, 2026, pyannoteAI published a comparison built around that four-speaker ceiling. It leaves out NVIDIA’s newer Nemotron 3 Diarization (September 23), which tracks eight.

Speaker diarization is the step that decides who spoke when. If the term is new, start with what speaker diarization is.

	pyannote Community-1	Streaming Sortformer v2.1	Nemotron 3 Diarization
Made by	pyannote (open source; pyannoteAI sells hosted models)	NVIDIA, in NeMo	NVIDIA, in NeMo
Design	Pipeline: find speech, fingerprint voices, cluster them	One end-to-end model	One end-to-end model, built on Sortformer
Speaker limit	None by design; you can set an exact count or a range	4	8
Live use	Works on a finished recording	Streaming, 1.04 s delay	Streaming at 1.04, 0.64 or 0.32 s, plus a 30.4 s offline mode
Licence	CC-BY-4.0	NVIDIA Open Model License	OpenMDW 1.1
Hugging Face downloads, last 30 days	5,443,619	110,001	30,931 (public for six days)

Checked September 29, 2026.

What changed this week

On September 23, 2026, NVIDIA released Nemotron 3 Diarization, a model that labels up to eight speakers in live audio. It runs in the same toolkit as the four-speaker model, so it belongs on that side of this comparison. The same day, FluidAudio, an open-source Swift library for speech models on Apple chips, added it.

Six days later, pyannoteAI published pyannote vs. NVIDIA NeMo, with a section titled “The four-speaker ceiling”. The post covers two versions of the four-speaker model and an older pipeline. Its top 7 diarization list, from the same day, repeats “a hard maximum of four speakers”. Neither post mentions the new model.

What pyannoteAI says about the models it names is accurate. It is a week out of date on the toolkit as a whole.

pyannote vs NeMo: what’s the difference?

pyannote is a pipeline of separate models, and NVIDIA’s newer diarizers are single end-to-end models. That one design choice explains most of the other differences.

The Community-1 pipeline runs three steps. It splits the audio into speech segments, computes a voice fingerprint for each, and clusters the fingerprints into speakers. Clustering groups whatever it finds, so the speaker count is an estimate, not a fixed setting. If you know the count, you can pass it in, or give a minimum and maximum.

NVIDIA’s toolkit has an older clustering pipeline too. Its newer route reads audio and outputs a label for each speaker slot directly, and the number of slots is fixed in training.

A single model can mark two slots as active at once, which is how it handles overlapping speech. The price is the ceiling. A pipeline has no ceiling, but it has more parts to tune.

Is the four-speaker ceiling still real?

For the older model, yes. For NVIDIA’s toolkit as a whole, the ceiling moved to eight on September 23, 2026.

The Sortformer v2.1 model card says it “can detect a maximum of 4 speakers; performance degrades on recordings with 5 and more speakers.” pyannoteAI quotes this correctly. As its post says, the limit is built into the model, not a setting you can raise.

Eight is still a limit, and pyannote’s point stands above it. Clustering has no cap, and a 14-speaker recording is beyond any current NVIDIA model. pyannoteAI also notes that counting speakers is hard “for every system, pyannote included.”

Four slots matter more than they sound on a video call. Everyone on the far end reaches your computer through one audio stream, so a six-person call already overflows them.

We tested this inside Memory Notch’s live pipeline, on 16 broadcast recordings with 5 to 8 speakers. The eight-speaker model’s error rate was 3.81%, against 18.20% for the four-slot one. pyannote was not part of that test.

Which is more accurate?

Each vendor publishes numbers that favour its own model, and we found no published table with all three on the same test and scoring. Read the tables as claims, not a ranking.

The measure is diarization error rate: the share of speaking time that is missed, falsely detected or given to the wrong speaker. Lower is better.

Claim (vendor-reported)	Source	Error rate
Community-1, AMI meetings (headset mics)	pyannote’s model card	17.0%
Community-1, DIHARD III	pyannote’s model card	20.2%
Community-1, VoxConverse	pyannote’s model card	11.2%
pyannoteAI’s hosted model, 196.6 hours in five languages	A late-2025 study, as cited by pyannoteAI	11.2%, best overall
pyannoteAI Precision-2, ten DIHARD domains	pyannoteAI’s own benchmark	Lowest in all ten

Claim (vendor-reported)	Sortformer v2.1	Nemotron 3
DIHARD III, full set, 1.04 s delay	19.60%	13.18%
DIHARD III, recordings with 5 to 9 speakers, 1.04 s	41.39%	28.65%
AMI test, mixed headset mics, 1.04 s	16.36%	9.48%

Checked September 29, 2026, on each model card.

NVIDIA’s launch post adds that its new model ranked first of 12 systems on one public diarization benchmark. The only outside comparison we found that includes both families comes from Baseten, which hosts the model. Baseten says it lost to Community-1 on only one test set of recorded meetings.

The 11.2% figure pyannoteAI leads with belongs to its paid hosted model, not the open-source one. The two model cards test on some of the same datasets, but with different microphone setups, so don’t subtract one card’s number from the other’s.

Which runs live, and which runs on a Mac?

Nemotron 3 and Sortformer stream. Open-source Community-1 works on a finished recording, and pyannoteAI’s live option is a hosted service.

pyannoteAI’s streaming service, Live-1, reached general availability on July 7, 2026. The streaming page lists a WebSocket API that takes audio in 100 ms pieces and answers in under 300 ms. It also mentions on-device use for wearables.

On a Mac, the open-source paths differ:

Question	pyannote	NeMo models
Official hardware	CPU by default, GPU through PyTorch	NVIDIA GPUs (Ampere and later, per the Nemotron 3 card)
Setup, per pyannoteAI	One pipeline call	Manifest files and Hydra configs
Apple-chip port in FluidAudio	Community-1 for files; pyannote 3.1 for streaming	Sortformer (4 speakers); Nemotron 3 (8 speakers) since v0.17.0

pyannote is easier to start with on any laptop, and that is a real advantage. NVIDIA’s own tooling expects its graphics cards, so on a Mac these models run only through a port like FluidAudio’s. A FluidAudio update on September 25 fixed a slow first load of the new model on some Apple chips.

Sortformer vs pyannote

Most of pyannoteAI’s post is Sortformer vs pyannote, and on those terms its conclusion is fair: pick Sortformer only when calls have four or fewer speakers.

Sortformer streams, handles overlapping speech in one model, and fits the stack many speech teams already run. Its limits are the four slots and, per pyannoteAI, training data that is mostly English.

pyannote’s strength is adoption. Its 3.1 pipeline had about 7.3 million downloads on Hugging Face in the last month. Community-1 had about 5.4 million more.

The pyannote-audio code has 10,604 GitHub stars. NVIDIA’s toolkit has 18,525, but it covers far more than diarization. Community-1 can also output one speaker at a time, which makes it easier to line words up with speakers.

Which should you pick?

Pick by speaker count, whether you need labels during the call, and where the audio may go.

Your job	Pick	Why
Label finished recordings, any number of speakers	pyannote Community-1	No speaker cap, runs on a CPU, millions of monthly downloads
More than eight speakers	pyannote	NeMo models stop at eight
Live labels from a hosted API	pyannoteAI Live-1	Streaming service with under 300 ms delay
Live labels on your own computer, up to eight speakers	Nemotron 3 Diarization	Streams, open weights, beats Sortformer in NVIDIA’s tests and ours
Live labels, four speakers or fewer	Nemotron 3 over Sortformer	Same toolkit, lower error in every comparison we found

For live meetings recorded on a laptop, pick Nemotron 3 Diarization. It streams by design, runs locally with open weights, and cut the older model’s error rate in NVIDIA’s tests and in ours. The model is only part of the job, as how AI transcription works explains. The audio handling around it decides much of what you see.

What these numbers can’t tell you

The new model is six days old. Its results so far come from its maker, a company that hosts it, and our own test. Outside benchmarks will take time.

Our test compared the two NVIDIA models only; we did not run pyannote. The audio was public meeting and broadcast recordings, not real calls captured on a laptop.

No model here separates overlapping voices well, and all of them label speakers with numbers, not names.

What Memory Notch runs

Memory Notch runs Nemotron 3 Diarization on your Mac through FluidAudio, which is bundled in the app. One copy listens to your microphone and one to the call’s audio, and each tracks up to eight speakers. The words come from a second model, the multilingual streaming version of Nemotron.

Speakers are numbered, not named, and the app writes no automatic summaries. Recording and transcription run on your computer. The app makes three network requests, and the privacy page shows that none carries a recording.

The offline transcription guide covers what works with no connection. Local processing doesn’t replace the consent of the people you record. You can try it free for recordings of up to 5 minutes.

Frequently asked questions

Is pyannote better than NeMo for diarization?

For files with many speakers, pyannote; for live labels, NeMo. pyannote Community-1 has no speaker cap and is easier to set up for files. NeMo’s Nemotron 3 Diarization streams live labels for up to eight speakers, and NVIDIA reports lower error than its older Sortformer model. We found no published table with both on one test.

How many speakers can NeMo diarization handle?

The Sortformer models handle up to four speakers, and their model card says accuracy drops with five or more. Nemotron 3 Diarization, released September 23, 2026, handles up to eight. pyannote’s clustering has no fixed cap.

Can pyannote do real-time diarization?

The open-source pyannote Community-1 pipeline works on finished recordings. pyannoteAI sells a streaming model, Live-1, as a hosted API. It answers in under 300 milliseconds and has been generally available since July. FluidAudio also ports an older pyannote 3.1 pipeline for streaming on Apple chips.

Does NeMo diarization run on a Mac?

Not through NVIDIA’s own tooling, which targets its graphics cards. FluidAudio, an open-source Swift library, ports Sortformer and Nemotron 3 Diarization to Apple chips. Memory Notch uses that port to run Nemotron 3 on the Mac.

Is pyannote free?

The Community-1 pipeline is free to download under a Creative Commons licence and runs on your own computer. pyannoteAI’s Precision models and its Live-1 streaming model are commercial services. Precision-3 is offered through its API and on-premises.

Sources

Download Memory Notch · More from the blog