Cloudflare's Meerkat takes aim at the timeout problem inside global consensus

The internal research project uses QuePaxa, a 2023 EPFL algorithm, to keep control-plane writes moving across Cloudflare's global network.

By ยท Published

Why it matters

Meerkat shows Cloudflare investing below the product surface: stronger global control-plane coordination could make its AI, Workers and security services harder to knock out when links or leaders fail.

Cloudflare's Meerkat takes aim at the timeout problem inside global consensus

Cloudflare research engineers James Larisch (@jameslarisch), Bob Halley and Joao Pedro Leite introduced Meerkat on July 8, an internal distributed consensus service built to solve a problem Cloudflare created for itself by running one network across hundreds of locations.

In a Cloudflare Blog post, the three engineers described Meerkat as an experimental Cloudflare Research project that uses QuePaxa, a consensus algorithm published in 2023 by researchers at EPFL, to let Cloudflare services read and modify shared control-plane state across its global network without relying on a single write leader.

That makes Meerkat less of a product launch than an unusually direct look at Cloudflare's infrastructure pressure. Cloudflare says many of its internal services need to update small pieces of control-plane data from across 330+ global data centers. Examples in the post include placement information, such as where an AI model instance is stored, and leadership information, such as which machine is allowed to write to a replicated database.

Cloudflare says the hard requirement is linearizability: after a write succeeds, later reads should see it, even when requests hit different replicas around the world. The company also wants the system to stay available for reads and writes as long as a majority of machines are alive and can communicate. That is the familiar promise of consensus, framed against the less familiar operating conditions of Cloudflare's network: failed servers, down data centers, full queues, degraded links and cut cables.

The disclosure lands a week after RuntimeWire reported on Cloudflare's push to turn AI crawler behavior into publisher-facing controls, including Attribution Business Insights on July 2 and an x402 monetization gateway for AI agents on July 1. Those were outward-facing product moves. Meerkat is the other side of the same company story: Cloudflare is selling itself as the network where applications, agents and security policies run, and that pitch depends on internal coordination systems that survive the Internet's failures.

Why Cloudflare is moving beyond Raft

The direct target in the Meerkat post is Raft, one of the standard consensus algorithms used in distributed systems. Raft is leader-based: a leader replica drives writes, and if that leader crashes or becomes unreachable, writes stop until the remaining replicas elect a replacement. The Raft project describes consensus as multiple servers agreeing on values, with typical algorithms continuing to make progress when a majority of servers is available.

Cloudflare's objection is operational. In a wide-area network, the timeout that triggers a leader election is hard to set. Make it too short and normal latency can look like failure. Make it too long and the system waits while a dead or degraded leader blocks writes. Larisch, Halley and Leite wrote that Cloudflare has seen multiple incidents caused by unavailable leaders in consensus-driven systems.

QuePaxa attacks that specific failure mode. The QuePaxa paper, titled "Escaping the Tyranny of Timeouts in Consensus," argues that leader-based consensus algorithms perform well under normal conditions but lose resilience when timeouts, slow leaders or network attacks interfere with progress. Its authors report WAN-scale prototype experiments, including adverse conditions such as denial-of-service attacks, misconfigurations or slow leaders, and show high throughput with sub-second median latencies.

Cloudflare is careful with its own claims. Meerkat is still in development and internal-only for the immediate future. It is designed first for small pieces of control-plane state, such as leadership for replicated databases, rather than as a general-purpose database system. The engineers also acknowledge the fundamental cost of consensus: coordination requires round trips between proposers and a majority of replicas, with added delay under contention.

That tradeoff explains the initial scope. The first useful workloads are low-write, high-correctness coordination tasks where stale or conflicting state can cause an outage. A global user-facing database would expose the latency cost immediately. A control-plane service that decides which machine owns a lock, where a resource is placed or which replica is allowed to write can justify a slower path if it prevents inconsistent state.

The founder imprint is still visible

Meerkat is a research project, not a startup, and Cloudflare has not identified a standalone founder for it. The founder context still matters because Cloudflare's technical choices have long followed the original company bet made by Matthew Prince, Michelle Zatlyn and Lee Holloway: replace network appliances with cloud services that run on a shared global platform.

In their 2019 founders' letter, Prince and Zatlyn wrote that Cloudflare was formed as firewalls, load balancers and other network appliances were moving from hardware customers bought to cloud services they rented. They also wrote that Holloway architected Cloudflare's early platform and recruited and led the early technical team. That architecture goal, every server in every city able to run every service, is exactly the kind of decision that later creates a need for a Meerkat-like coordination layer.

Cloudflare's current scale raises the stakes. The company says on its about page that it serves 102 million HTTP requests per second on average and serves data from 335 cities in more than 125 countries. Its homepage says Cloudflare reaches 95% of the world's Internet-connected population within 50 ms and powers 42% of the Fortune 500. Those are company-provided numbers, but they show why a consensus problem that sounds academic becomes a production issue: tiny control-plane decisions can affect a very large surface area.

The Meerkat post also fits Cloudflare Research's stated charter. Cloudflare Research describes its work as bridging theory and Internet-scale application, with reliability at global scale listed as one of its focus areas. Meerkat is a literal example: Cloudflare is taking a 2023 academic consensus protocol and trying to turn it into a working internal service.

What Cloudflare has proved so far

Cloudflare says Meerkat is not in production. The post frames this as an introduction to an experimental service the Research team has been building for the past year.

Cloudflare also says it is layering applications on top of Meerkat's consensus log, including a transactional key-value store and a leasing system.

The open questions are product questions, even if Meerkat stays internal. Cloudflare has not disclosed production rollout timing, production latency numbers, the exact start date beyond "the past year," or whether Meerkat will eventually underpin customer-visible products. Cloudflare says, to its knowledge, Meerkat would be the first industrial deployment of QuePaxa at global scale. That claim is attributed to Cloudflare because the company did not provide independent evidence for the market-wide comparison.

For now, the practical read is simpler. Cloudflare is taking a known pain point in distributed infrastructure, leader failure under messy network conditions, and trying to remove the leader from the write path. If Meerkat works, Cloudflare gets a more resilient internal coordination layer for the services that decide where work runs and who is allowed to change state. If it fails, the company still has Raft and the existing consensus stack, plus a research program that surfaced the right problem: global platforms do not fail only when machines die. They fail when coordination assumes the network behaves better than it does.

Reader comments

Conversation for this story loads after sign-in.