Perplexity publishes its three-layer stack for search embeddings and ranking

Why it matters

Perplexity is bringing model serving in-house for embedding, ranking, classification, scoring and large language model workloads. That control could lower latency and cost across its consumer product and developer APIs, although the company has not published production benchmarks supporting those claims.

Perplexity, the AI search company co-founded by CEO Aravind Srinivas, published the architecture behind the embedding and ranking workloads that feed its search product. The three-layer internal stack is designed to prevent request preparation, batch scheduling and GPU execution from holding one another up.

Perplexity's architecture thread

The September 4th thread names those layers Ivy, Tulip and ROSE. Together, they handle the path from an incoming search query to the transformer models that embed, retrieve, rank and score information before Perplexity generates an answer.

The disclosure fits the founders who built Perplexity. Srinivas studied electrical engineering at IIT Madras, completed a computer science Ph.D. at UC Berkeley and worked as a research scientist at OpenAI before co-founding Perplexity in August 2022, according to an IIT Madras profile. Co-founder and CTO Denis Yarats previously worked on Bing at Microsoft and in machine learning at Quora before serving as an AI research scientist at Facebook AI Research. Founding president Andy Konwinski helped create Apache Mesos and Spark before co-founding Databricks.

That mix of search, machine learning and distributed-systems experience shows up in a serving stack designed around a familiar infrastructure problem: GPUs are expensive, and much of the engineering work happens before a model can usefully occupy one.

Perplexity has ample capital to do that work internally. TechCrunch reported in September 2025 that the company had raised another $200 million at a $20 billion valuation, bringing its reported funding to roughly $1.5 billion. Named investors across its financings include Accel, Institutional Venture Partners, Nvidia, Jeff Bezos, New Enterprise Associates, Databricks, Bessemer Venture Partners, SoftBank Vision Fund 2, Sequoia Capital, Andreessen Horowitz and T. Rowe Price.

Two embedding jobs with different clocks

Perplexity maps queries and documents into a shared vector space, then retrieves documents whose vectors are closest to the query. The method is widely used in semantic search, but Perplexity has to serve it under two different operating conditions.

Indexing and scoring create large batch workloads where throughput matters most. A live user query creates a smaller online workload where latency matters. Perplexity's published search architecture says retrieval begins through both lexical and semantic modalities, with results merged into a hybrid candidate set, so the pipeline is not wholly blocked on the query embedding. Semantic retrieval nevertheless puts embedding generation on a latency-sensitive path for each live search.

Perplexity separated the serving path into three components so each layer can be tuned without forcing changes through the entire system.

Ivy is the Rust HTTP gateway. It parses incoming requests, tokenizes text, applies templates and splits batches that are too large before sending work over gRPC. Keeping that CPU-heavy preparation outside the inference server lets Perplexity change request formatting and tokenization without modifying the GPU-facing components.

Tulip is a lightweight Rust gRPC server that gathers requests and schedules batches for the GPU. Perplexity says runtime for its smaller embedding models is driven mainly by the number of tokens rather than the number of individual queries. Roughly 512 tokens can therefore be enough to fill the GPU, according to the thread.

That detail explains why batching is central to the design. Tulip can combine several small requests until it has enough tokens to use the hardware efficiently, while avoiding the latency penalty of waiting too long for a batch to fill.

ROSE is written in Python, performs the model forward passes and manages CUDA graph execution. Perplexity says the engine reuses many of the same kernels for large language models and embedding models. Embedding workloads skip the key-value cache used during autoregressive generation and rely on ragged attention instead of paged attention. ROSE can also select among attention backends based on model shape and sequence length.

Keeping the CPU ahead of the GPU

Tulip uses CUDA graphs to record a sequence of GPU operations so the work can be launched with a single call. That reduces repeated CPU launch overhead, which can become significant when the underlying model is small and each inference job completes quickly.

A second mechanism called LazyTensors tracks results asynchronously. The CPU can begin preparing the next batch while the GPU completes the current one, reducing the idle gaps between jobs. The goal is a continuous pipeline in which request handling, batch assembly and model execution overlap instead of running as a strict sequence, according to Perplexity's description of CUDA graphs and LazyTensors.

Perplexity says the combination improves latency and throughput across online and batch embedding workloads while costing less than off-the-shelf serving systems. The company did not attach production benchmarks to that performance claim. The thread contains no figures for embedding latency, requests per second, GPU utilization, fleet size or per-query cost, so the size of the improvement cannot be independently assessed.

Perplexity has disclosed broader search metrics elsewhere. In a September 25th, 2025 technical account of its search architecture, the company said its production infrastructure processed 200 million queries a day and reported a median Search API latency of 358 milliseconds for requests initiated from AWS us-east-1. Those figures describe the broader search system, not the latency or savings attributable to Ivy, Tulip and ROSE.

ROSE has grown beyond embeddings

The September thread gives Ivy and Tulip their clearest public descriptions, but ROSE predates the disclosure. A May 6th technical post from Perplexity's AI Inference team says the engine was initially built to serve customized Llama models for language decoding and classification.

Perplexity's later technical material describes ROSE as a broader model engine serving ranking, classification, scoring, embeddings and large language model workloads. The same post says Perplexity uses ROSE to host custom models on Nvidia Hopper and Blackwell GPUs and that the engine sits behind its Sonar, Search and Embeddings APIs. The newly described embedding pipeline is one workload inside a broader inference system.

Perplexity also sells embeddings through a public API for semantic search and retrieval-augmented generation. Perplexity's Embeddings API documentation lists standard and contextualized models at 0.6 billion and 4 billion parameter scales. A separate retrieval-augmented generation guide describes batch embedding and a vector-index workflow covering chunking, embedding, indexing and similarity search. The architecture thread does not specify which parts of Ivy, Tulip and ROSE support external API traffic or identify the production models used inside Perplexity's consumer search product.

Infrastructure control becomes more valuable as the number of workloads grows. A shared engine can support an internal search product, developer APIs and new model architectures without requiring a separate serving stack for each one. It also lets researchers test a model without waiting for an outside inference project to support each new attention pattern or kernel.

Search infrastructure is becoming a product category

Perplexity is building this stack while specialist companies sell search infrastructure directly to AI developers. Exa said in May that it raised a $250 million Series C at a $2.2 billion valuation and planned to scale its infrastructure to hundreds of thousands of searches per second. Tavily raised a $25 million Series A in August 2025 for its web-search tools for AI agents.

Those companies sell search and web-access services. Perplexity's September disclosure concerns the internal serving path for embeddings and ranking, with some of that infrastructure also supporting public APIs. Their funding shows why the machinery below an answer engine now attracts investment in its own right. Developers building agents and retrieval systems need fresh web data, ranking and predictable response times without operating the entire search stack themselves.

Publishing the architecture also serves Perplexity's developer business. Embeddings API customers are being asked to trust retrieval infrastructure hidden behind an endpoint. Naming the gateway, scheduler and engine gives them a more concrete account of the system, even though Perplexity has not quantified its cost and performance claims.

Srinivas started Perplexity around the premise that search should produce direct, sourced answers. Four years later, the company describes an internal serving stack with an HTTP gateway, a batching and scheduling server, and a model engine containing custom GPU kernels.

Search quality still depends on the index, retrieval method, ranking models, sources and answer generation above this stack. The September 4th publication establishes how much engineering Perplexity has put beneath those systems. It does not establish that Perplexity ranks better than Google, Microsoft, OpenAI or other search competitors, but it explains how the company intends to run its models quickly enough, and cheaply enough, to compete with them.