When two people talk in a video, translated captions need more than the right words. I also want to know who said them, while the conversation is still going. I built a local prototype on my Mac to see how close I could get. It shows the English, Japanese, and speaker labels side by side. The translation can be awkward, and the labels still make mistakes, but the 33-second clip below shows what is working.
Three jobs, one audio stream
Apple’s Speech framework turns incoming audio into text with time ranges. Nemotron 3 Diarization estimates who spoke when, returning speaker labels and their time ranges. It does not extract a separate audio track for each person. Finally, Apple’s Translation API translates completed lines. The required language and diarization models may need to be downloaded first.
What needed a second pass
The first fix was to prepare Apple’s speech model before capture and keep audio processing away from UI updates. Otherwise, model startup and screen work can hold up the incoming sound.
Speaker labels brought a different timing problem. The words and the speaker estimate do not necessarily arrive together. Matching them by arrival time could put the right words under the wrong speaker. I now match their audio time ranges and keep the timing of the words. If a speaker result comes in late, an earlier caption can be relabeled or split at the turn.
Translation waits for a completed line instead of chasing every partial word. The Japanese caption trails the English one, and the demo still has errors, especially when the speech moves quickly. I would not use this version for reliable interpretation yet.
Would a newer Mac change the result?
I recorded this on an M1 Mac, and I’m curious what will change when I run the same clip on my M5 MacBook Pro. Apple reports better dictation with AFM 3 Core Advanced on M3-or-later Macs with at least 12GB of memory running macOS 27. This demo gets its English text from SpeechTranscriber, so I want to see what improves in practice and where the mistakes begin. If the English is off, I’d compare Apple’s transcription with Whisper; if the English is right but the Japanese reads strangely, I’d try Apple’s higher-quality translation option. I’d time both, too. A better caption is less useful if it arrives after the conversation has moved on.