MEMORY NOTCH BLOG

How does AI transcription work? Inside a Mac app

A speech model turns audio into words; the app does the rest: feeding it audio a tenth of a second at a time, telling who spoke, knowing when you stop.

Published October 23, 2026 · The Memory Notch team

How does AI transcription work? In two layers. A speech model turns short slices of audio into words. The app around it decides what audio the model hears, when a sentence ends and who said it. On September 23, 2026, NVIDIA released Nemotron 3 Diarization, which labels up to eight speakers. Our Mac app, Memory Notch, runs it for speaker labels as of version 0.5.7 (September 28).

The model: a neural network that maps sound to text. It is the engine.
The app: capture, audio format, timing, speaker labels, memory and error handling. It is the chassis.
Live vs file: a live transcript has to commit to words with about a second of audio in hand. A file can be read in bigger pieces.
Who spoke: a second model, called a diarizer, marks which voice is active every 10 milliseconds.
On a Mac: both models can run on Apple Silicon with no server. In our tests, the pair used about 300 megabytes of memory.

What changed this week

NVIDIA published Nemotron 3 Diarization on Hugging Face on September 23, 2026. It is a 100-million-parameter model. It tracks up to eight speakers, where its predecessor tracked four. In its launch post, NVIDIA reports an average 40 % relative cut in diarization error against that predecessor at a 1.04-second delay. That is NVIDIA's own measurement.

The model card lists only NVIDIA graphics cards. The same day, FluidAudio, an open-source Swift library for running speech models on Apple hardware, added it in v0.17.0. On September 25, v0.17.4 fixed loading on the Neural Engine of Macs with an M3 chip. Before that, every load there paid about 3 seconds for a failed compile.

Our app's version 0.5.7 shipped on September 28 with that FluidAudio release bundled. Its speaker labels come from Nemotron 3 Diarization, running on your Mac.

Checked September 29, 2026.

How does AI transcription work, step by step?

A transcription app records sound, converts it to the format the model expects, feeds it to the model in small pieces, and writes down the words the model emits. For a meeting, it also runs a second model to label the speakers and attaches each word to a voice.

Here is the path one sentence takes through our app:

Capture. The Mac records your microphone and the call's audio, at 48,000 samples a second.
Convert. The app turns that into 16,000 samples a second, mono. That is the rate the models take, and a third of the data.
Batch. Every 100 milliseconds, the app hands 1,600 of those samples to each model.
Encode. The speech model turns the audio into a sequence of numbers that describe the sounds. For live words we run the multilingual version of NVIDIA's Nemotron 3.5 streaming speech recognition model, with 600 million parameters, that covers 40 language-locales. From here on, "the speech model".
Decode. A second half of the network, called a transducer, turns those numbers into word pieces, with punctuation. Our speech model does this for every 1.12 seconds of audio, and it detects the language for each piece.
Label. A second model, the speaker model, reads the same audio and marks which voice is active every 10 milliseconds.
Join and save. The app matches words to voices and writes a plain-text note to ~/memory on your Mac.

Steps 4 and 5 are what people usually mean by "AI transcription". Every other step belongs to the app.

The model is the engine; the app is the chassis

A speech model decides how good the words can be; the app decides how many of them reach you. Think of the model as a big eight-cylinder engine and the app as everything that puts its power on the road: the chassis, the gearbox and the tyres. A Formula 1 engine bolted into a car that can't hold a corner loses to a smaller engine in a car that can.

Calling a speech model takes a few lines of code. Pass it a file and you get text back. The hard part is everything that file never had to deal with:

Chassis job	The problem	What our app does
Two channels	Your voice and the call's voices arrive mixed or separately, at different levels	Transcribes and labels the microphone and the system audio separately
Timing	A model wants audio in pieces; a meeting arrives one sample at a time	Batches every 100 ms; the speech model works in 1.12 s steps
Warmup	The first load after an update compiles the models for your chip	Loads the models at launch: 18.8 s cold, measured, against 0.15 s warm
An early stop	You press stop before the models are ready	Waits up to 60 s for models on disk, then transcribes the buffered audio
Phantom speakers	A cough or an echo gets its own speaker label	Drops voices with under 5 s of airtime from the saved note
Speakers vs headphones	The microphone hears the call playing from the speakers	Handles the echo
End of a sentence	Words arrive a step late, so a pause looks like the end	In dictation, pastes the text after 2 s of silence, timed on the audio
Memory	Two models, two channels, on a laptop	About 300 megabytes for both models on both channels, in our tests

The rest of this post takes those rows one at a time.

What happens between the microphone and the model?

Between the microphone and the model, the app captures two sources and converts them to 16 kHz mono. It feeds each one to its own copy of the pipeline in 100-millisecond batches. The model only ever sees those batches.

Two channels matter because a video call is two recordings at once. The microphone hears you and anyone in the room. The system audio carries the other people on the call, compressed by the call app's codec. The app transcribes each channel separately and numbers the speakers within each one. Your side and their side never compete for the same labels.

The two channels share one copy of the speech model in memory, which is loaded and compiled once. Each channel gets its own speaker model with its own working memory, because one shared buffer would mix the two sides mid-call.

Nothing is dropped while the models load. Audio that arrives early is buffered, so the first sample either model sees is the first sample of the recording. At 16 kHz, that buffer grows by 64 kilobytes a second. A minute of loading holds under 4 megabytes. When the models are ready, they catch up on the backlog in about a second.

Why is live transcription harder than transcribing a file?

Live transcription is harder because the model has to commit to words with about a second of audio in hand. With a file, it can read ahead, go back, and process large pieces with context on both sides.

A file model like Whisper or Parakeet reads long windows of audio and can use the rest of the sentence when it picks a word. A streaming model can't wait. Our speech model is built for this. NVIDIA's card describes a cache-aware encoder: it keeps what it has already worked out about past audio, so each new step only processes new sound. It supports steps from 80 ms to 1.12 s. We use the longest, because it is the shortest step at which FluidAudio recommends relying on punctuation.

The trade-off is that a word, once emitted, stays. The speech model does not go back and fix an earlier word when a later one makes the meaning clear. A person reading along sees a line settle once, not flicker.

The file case still has a place. After a recording, the app can make an Enhanced transcript: it runs the same speech model over the saved file in 2.24-second steps, since there is no call to keep up with. Speaker labels are redone too, in 30-second slices.

For how a streaming model compares with a file model from the same maker, see Nemotron vs Parakeet.

How does it know who is speaking?

A separate model, called a diarizer, listens to the audio and outputs, every 10 milliseconds, the chance that each of up to eight speakers is talking. The app turns those chances into turns and gives each word the speaker who was talking when it was said. The speaker model doesn't know names, so speakers are numbered.

Nemotron 3 Diarization outputs eight values per 10 ms frame, one per speaker slot, each between 0 and 1. The app counts a speaker as talking when the value stays above 0.5 for at least 0.2 seconds. Two slots can be active in the same frame, which is how the model marks two people talking at once. Our app runs the speaker model at a 1.04-second delay, so a label arrives about a second after the audio it describes.

Eight slots per channel matter on a call. Everyone on the far end arrives through one system-audio stream, so a six-person call puts five voices on one channel. A model with four slots has to fold two of those people into one label.

We benchmarked it on 24 recordings with three to eight speakers. Its diarization error rate was 4.45 %. NVIDIA's earlier four-speaker model, Streaming Sortformer v2.1, scored 14.07 %. That rate is roughly the share of speaking time missed, invented or given to the wrong voice.

If the word diarization is new, what is speaker diarization explains it from the start. The other widely used approach, which finds voices by clustering rather than tracking slots, is covered in pyannote vs NeMo diarization.

Can transcription run on your Mac, and what does it cost the Mac?

Yes. On an Apple Silicon Mac, both models run on the machine with no server involved, and the cost is memory, a one-time compile, and disk space. Memory Notch is about 1.6 gigabytes to download, and most of that is the models.

Memory. Our benchmark runs the app's own per-channel loop on a Mac with an M4 Max chip. There, the speech model and the speaker model on two channels used 301 megabytes. On a less busy run, they used 278 to 287 megabytes.

Speed. In the same test, both models together ran 18.4 times faster than real time. The machine was busy during that run, so treat the figure as a floor, not a promise.

Warmup. The first time a model loads after an install or an update, macOS compiles it for your Mac's chip. We measured that at 18.8 seconds. After that, a load takes 0.15 seconds. The app loads its models at launch, and holds the Record button until that load finishes, fails or gives up. A Mac whose models can't load still records audio, without live words.

Stopping early. If you stop a recording before the models are ready, the app waits up to 60 seconds for models already on disk, then transcribes everything it buffered. It never waits for a download.

Recording, transcription and speaker labels run on your Mac. The app makes three network requests, and none carries a recording: a signed list of revoked licences, a once-a-day update check you can turn off, and an update download when you choose one. The privacy page lists them. Running the models locally does not replace the consent of the people you record.

What goes wrong, and what a good app does about it

Phantom speakers, echo, cut-off sentences and crosstalk all happen around the model, so a good app fixes them before or after the model runs.

Phantom speakers. A speaker model sometimes gives a cough, a laugh or a burst of echo its own label. On one 232-second call we measured two phantoms holding 0.3 and 1.9 seconds, while the real voice held the rest. Our app drops any speaker with under 5 seconds of airtime and under 5 % of the channel's speech. It folds their words into the nearest real voice. On our benchmark, that raised the recordings with the right speaker count from 15 to 20, out of 24.

The prune only applies to the saved note, never the live view. A voice that looks spurious at second 10 can be a real person by second 60, and pruning live would make labels flicker.

Echo on speakers. With the call playing from your Mac's speakers, the microphone hears the other people too. Our app handles that echo when you're on speakers instead of headphones.

The end of a sentence. Words arrive about one model step late, so on their own they can't tell a pause from the end of a thought. Our app's dictation, a free feature in 0.5.7, shows the problem well. You double-tap ⌘ anywhere, speak, and after 2 seconds of silence the text is pasted into the app you're in. The silence clock runs on the audio itself, not just on the words, so a slow speaker isn't cut off mid-word. The clock gives up after 5 seconds with no words, and caps a phrase at 120 seconds. How software tells speech from silence in the first place is the subject of our comparison of two voice-activity detectors, Silero and WebRTC.

Overlapping voices. The speaker model can mark two people at once, but it still has to give each word to one of them. Crosstalk, and several people on one microphone, remain hard to separate.

What Memory Notch doesn't do

Memory Notch writes no automatic summaries. Speakers are numbered, not named. Library search covers titles and metadata, not every word of every transcript. Recording the Mac's own audio needs screen-recording permission in macOS.

Transcripts contain errors, as with every model, so check the audio when a word matters. Every benchmark number in this post comes from test sets of public meetings and broadcasts on one M4 Max machine, not from your Mac or your calls.

Memory Notch needs an Apple Silicon Mac on macOS 26 or later (specs). You can try it free for recordings of up to 5 minutes, with no account. For how local note takers compare, see local AI note takers for Mac.

Frequently asked questions

How does AI transcription work?

A speech model converts short slices of audio into words, using an encoder that describes the sounds and a decoder that picks the word pieces. The app around it records the audio, converts it to the model's format, feeds it in small batches, labels the speakers with a second model, and saves the result.

How does live transcription work?

A live transcriber feeds the model audio every fraction of a second and shows words as they come. Our app batches audio every 100 milliseconds, and its speech model emits words for every 1.12 seconds of sound. Once a word appears, it stays, so the transcript settles instead of rewriting itself.

Can AI transcription run offline on a Mac?

Yes. Memory Notch runs its speech model and speaker model on Apple Silicon Macs with macOS 26 or later. Recording and transcription work with no connection. The app makes three network requests for licences and updates, and none carries a recording.

How does AI transcription know who is speaking?

A second model, called a diarizer, estimates every 10 milliseconds which of up to eight speakers is talking. The app groups those estimates into turns and gives each word the speaker active when it was said. It numbers the speakers, because it doesn't know their names.

Why does the first recording after an update take longer to start?

The first load after an install or update compiles the models for your Mac's chip. We measured 18.8 seconds, against 0.15 seconds once cached. Our app does this at launch and holds the Record button until the load settles, so the wait doesn't land in the middle of your first recording.

Sources

Download Memory Notch · More from the blog