OpenAI rebuilt ChatGPT's voice stack so GPT-Live can listen while speaking

Why it matters

GPT-Live separates real-time conversation from slower reasoning and tool use, establishing a new infrastructure baseline for developers building voice agents.

OpenAI engineers Justin Uberti and Zahan Malkani on August 3rd detailed a six-month rebuild of ChatGPT's voice infrastructure that lets GPT-Live listen and speak simultaneously while other models handle search, reasoning and tool calls in the background.

The engineers described the architecture in an OpenAI engineering post and a thread on X, nearly four weeks after OpenAI began rolling GPT-Live out globally on July 8th. The technical account shows how much infrastructure sits behind a product behavior that users experience as conversational timing.

Uberti, one of WebRTC's original architects, and Malkani are members of OpenAI's technical staff. Uberti's background is central to the system: GPT-Live relies on WebRTC, the communications standard built for low-latency audio and video, as its transport foundation. OpenAI previously credited Uberti and Pion creator Sean DuBois with helping guide its work on real-time AI and WebRTC infrastructure.

A fast path for speech

Earlier ChatGPT voice systems treated conversation as a sequence of turns. A detector tried to decide when the user had stopped talking before the model began responding. That design created an uncomfortable tradeoff: an early decision could cut off the user, while a late one introduced an audible pause.

GPT-Live removes the separate turn detector from the audio path. Its full-duplex model continuously processes incoming speech while generating outgoing audio, allowing it to decide several times per second whether to listen, speak, pause, interrupt or invoke another system.

OpenAI separated that continuous audio loop from the rest of ChatGPT's application logic. Speech travels between the client and GPT-Live through a dedicated fast path. Search, tool calls, persistence and deeper reasoning sit behind an asynchronous boundary, so a slow service should delay its own result without stopping the model from listening or talking.

That division allows GPT-Live to keep a conversation moving while GPT-5.5 works on a more demanding request. OpenAI can also replace the delegated reasoning model as newer systems become available without rebuilding the voice model around each release.

The architecture amounts to a two-model system presented as one assistant. GPT-Live manages timing and speech, while a frontier model handles tasks that need additional computation. OpenAI's product announcement says GPT-Live-1 Instant and GPT-Live-1 mini use GPT-5.5 Instant in the background, while Medium and High settings delegate to GPT-5.5 Thinking with different reasoning levels.

OpenAI replaced Python code and compressed the handshake

OpenAI rewrote the media frontend and inference logic in Go, replacing an earlier Python asyncio implementation. OpenAI says the new system's 95th-percentile frame-delivery performance matched the previous system's median, indicating that the rebuild targeted consistency across sessions rather than a faster best-case demonstration.

Long conversations created another problem. The context grows while model instances may need to be replaced as capacity changes. OpenAI built a handoff process that warms a replacement instance, loads it with the current context and runs both instances in parallel before switching traffic. The same mechanism lets OpenAI compact an oversized context without pausing the conversation.

Session startup also required work below the model layer. Standard WebRTC setup involves several protocol handshakes. OpenAI developed the WebRTC Abridged Roundtrip Protocol, or WARP, which it says reduced media and data startup from six network round trips to one. A related system called Instant Connect negotiates session parameters ahead of time, allowing the client to begin a session with a single UDP packet when those parameters remain valid.

OpenAI is advancing WARP proposals through an Internet Engineering Task Force working group. Support has already been added to libwebrtc and Pion, according to the engineering post, giving other real-time communications developers a route to adopt parts of the work.

Production traffic exposed the real bottlenecks

Before the July 8th launch, OpenAI silently routed a gradually increasing share of production voice sessions through the new system while its existing Advanced Voice Mode continued serving users. The shadow deployment exposed the replacement stack to real networks, session lengths and geographic traffic without changing what users heard.

The testing found that GPU throughput alone was a poor measure of capacity. Voice calls keep connections open and send frames continuously, placing sustained pressure on CPU stream handlers, queues and network services. OpenAI said one supporting component saturated earlier than its load tests predicted, causing inference requests and latency to accumulate.

The finding matters at OpenAI's stated scale. OpenAI says more than 150 million people use ChatGPT Voice or Dictation each week. At that volume, natural conversation depends on regional routing, connection setup, state management and recovery behavior alongside model inference.

GPT-Live is currently powering ChatGPT Voice, with GPT-Live-1 designated for Go, Plus and Pro users and GPT-Live-1 mini for Free users, according to OpenAI's July 8th launch announcement. OpenAI also says the architecture will support an upcoming GPT-Live API.

That API will give developers access to a voice system built around continuous interaction and background delegation. Customer support, tutoring and desktop-agent products will face a higher technical bar as users begin to expect assistants to remain conversational while searches and tool calls are still running.