How OpenAI delivers low-latency voice AI at scale

openai.com

383 points by Sean-Der 12 hours ago


Sean-Der - 10 hours ago

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works.

[0] https://github.com/pion/webrtc

[1] https://webrtcforthecurious.com

legohead - 10 hours ago

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away.

I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and think out the whole sentence in my head before I say anything -- not very natural.

vjay15 - 7 minutes ago

This is such a good write up, WebRTC is one of the coolest things ever! It's kinda genius to use the VIP approach, SFU is also pretty scalable but now they dont even have to do that

Lucasoato - 10 hours ago

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts.

Still, it’s worth to keep in mind that these are not frontier models, differently from when they were released.

(Please Sam, if you read this, release the new realtime audio models)

thimabi - 11 hours ago

> Voice AI only feels natural if conversation moves at the speed of speech […] At OpenAI’s scale, that translates into three concrete requirements: Global reach for more than 900 million weekly active users

Surely the number refers to the total users of ChatGPT overall, and the fraction of those who use voice features is considerably smaller, is it not?

That’s the kind of thing that influences business decisions like knowing how much hardware and software optimization to throw at a problem.

amirathi - 2 hours ago

I find OpenAI's speech-to-text model the best of the lot. It can handle my & my 5-year old daughter's Indian accent pretty well.

I wonder if they run the STT model's output through the current model (that we're chatting with) as a final pass - since the text seem to be well aligned to the current conversation context.

For long prompts, I often speak to OAI web/app and copy-paste the text to Claude / Gemini :)

Aeroi - 11 hours ago

if anyone is looking to get into this. pipecat is a great open-source repo and community. https://github.com/pipecat-ai/pipecat

tracyhenry - 2 hours ago

After all these, I still feel their voice AI interrupts quite a lot, especially when I pause just for 0.5 sec. Interestingly, when I tell it to interrupt less, it seems to be better.

didibus - 11 hours ago

I wouldn't mind waiting longer for answers that would go through a better model with more thinking. As long as it has good support for interrupting and also it doesn't start answering as soon as I pause for 1 second and it's smart about knowing I'm done speaking.

deferredgrant - 2 hours ago

People are very sensitive to timing in conversation. Even if the words are good, a slightly wrong pause or interruption can make the system feel much less intelligent.

qrush - 10 hours ago

Am I reading this right that OpenAI is not using Livekit for WebRTC/audio anymore?

whateveracct - an hour ago

why is the "How" included here? it is often removed

hiroakiaizawa - an hour ago

Interesting. What are the main latency bottlenecks in practice?

logickkk1 - 10 hours ago

IMO this probably isn't just about latency. keeping people in voice gives them training data text never will. is that why they were fine going transceiver over sfu and mostly ignoring multi-party?

hnav - 8 hours ago

RFC 9297 support can't come quick enough in browsers. Would obviate having to deal with WebRTC in a client-server scenario.

charisma123 - 11 hours ago

If a transceiver crashes during a stream, how is the active session recovered? Does the system automatically re-establish the context in a new WebRTC session?

furyofantares - 11 hours ago

> Global reach for more than 900 million weekly active users

lol, definitely didn't need to know there's 900M weekly users for this post. I mean yeah, there's a lot of users and they serve globally, that's relevant. But this is just pulling out your biggest stat because you can. How many voice users you have would actually be relevant and interesting but, to baselessly speculate on motivation here, might be a number that doesn't add as much fuel to an upcoming IPO as reminded people that you're almost at a billion users does.

- 10 hours ago
[deleted]
anzerarkin - 11 hours ago

I hate the voice ai though, it's so much dumber

CrzyLngPwd - 10 hours ago

It's bad enough having to speed-read the waffle of its written answers; even when told to be concise, the thought of having to listen to it waffle on in its smarmy, sycohpantic fashion makes me want to reach for the sick bag.

doctorpangloss - 11 hours ago

what i learned from making a webrtc+kubernetes game streaming product:

- openai is wrong. almost of the issues they described are issues with libwebrtc, not with webrtc, kubernetes, network architecture, etc. the clue was when they said "the conventional one-port-per-session WebRTC model."

- there are no alternatives worth trying. everything else open source in the ecosystem, like pion, coturn, stunner, are too immature.

- libwebrtc is the only game in town.

- they haven't discovered libwebrtc feature flags or how it works with candidates, which directly fix a bunch of latency issues they are discovering. a correct feature flag can instantly reduce latency for free, compared to pay for twilio network traversal style solutions

- 99% of low latency voice END USERS will be in a network situation that can eliminate relays, transceivers, etc. it is totally first class on kubernetes. but you have to know something :)

this is the first time i'm experiencing gell mann amnesia with openai! look those guys are brilliant, but there is hardly anyone in the world who is doing this stuff correctly.

tom1IIIl1iIL - 10 hours ago

I think it's better to join some kind of club if you want to make friends?

AIorNot - 11 hours ago

so is the answer

WebRTC + Kubernetes

devopsengine - 5 hours ago

Inspired

rvz - 10 hours ago

OpenAI uses Go for the networking implementation for the relays and the services, which makes a ton of sense, instead of something immature as TypeScript / Node or whatever.

Yet another reason to not consider anything else like that for low-latency networking. Golang (or even Rust and C++) is unmatched for this use-case.

mt_ - 7 hours ago

[dead]

DumpoLumbo - 10 hours ago

[dead]

DumpoLumbo - 10 hours ago

[flagged]

Dorrell - 11 hours ago

[flagged]

charcircuit - 6 hours ago

[dead]

testing_auth - 10 hours ago

[dead]

testing_auth - 10 hours ago

[dead]

flakiness - 11 hours ago

Should I or shouldn't I be glad to see zero mention on Codex.

jonahs197 - 10 hours ago

Who cares? Their company is dying.

cdrnsf - 11 hours ago

It's missing the part where they explain how they obtained the training data for their voice AI.