Qdrant releases a 10-billion-vector benchmark and an engine to rerun it

Why it matters

Qdrant is trying to make internet-scale reproducibility part of how vector databases are evaluated. Adoption would give buyers better tests and Qdrant a standard aligned with its engineering strengths.

Andre Zayarni and Andrey Vasnetsov's Qdrant released a 10-billion-vector search benchmark and the open-source engine used to build it, giving database developers a test that begins where many existing benchmarks end. The September 1st release pairs Qdrant-FineWeb-10B with Supernova, a framework for embedding data, calculating exact results, loading databases and running concurrent search workloads.

The project follows the same problem-first path that produced Qdrant. According to Qdrant's company biography, Zayarni and Vasnetsov began working together in 2021 on a matching engine for unstructured data. Qdrant says existing tools, including Meta's FAISS library, lacked the production features and scale they needed, so Vasnetsov built a vector search engine from scratch and published it on GitHub. Developer interest turned that engine into Qdrant, with Zayarni as CEO and Vasnetsov as CTO.

FineWeb-10B extends that founding thesis into benchmarking. Qdrant argues that research built around tidy collections of a few million vectors says little about systems expected to index changing data, apply filters during retrieval and maintain predictable tail latency across billions of records. Publishing a much larger corpus lets Qdrant help define the tests by which vector databases are judged, while Supernova gives engineers the code needed to challenge the results.

That standard-setting push follows Qdrant's $50 million Series B on March 12th. AVP led the round, with Bosch Ventures, Unusual Ventures, Spark Capital and 42CAP participating. In that announcement, Zayarni described retrieval as core infrastructure for AI agents, search and multimodal systems. FineWeb-10B is a concrete extension of that pitch: Qdrant is spending investor capital to make production-scale retrieval easier to measure, reproduce and, eventually, purchase.

A benchmark built to move the denominator

The dataset card describes roughly 10.07 billion dense and sparse vectors derived from Hugging Face's FineWeb corpus, alongside source text and metadata. It says each source document has a 768-dimensional dense embedding and a sparse representation generated with Alibaba-NLP's gte-multilingual-base. Qdrant reports 24.47 TB of vector data and 28.66 TB of text and metadata.

The query count needs more precision than the announcement's round number suggests. Qdrant's release emphasizes exact top-1,000 results for 100,000 queries. The directly verified dataset card lists 119,953 queries across four sets: 100,000 dense queries, 10,000 sparse queries, 4,953 text-filtered dense queries and 5,000 structured-filtered dense queries. The release's 100,000 figure describes the main dense set, while the dataset card's larger total includes the additional sparse and filtered workloads.

Qdrant says calculating the ground truth required more than one quadrillion distance computations across the full corpus. Vultr supplied the compute infrastructure and ran the initial embedding-generation pipeline. Hugging Face provided the hosting platform and a storage grant for the finished dataset.

The scale is the point. Qdrant says common vector search benchmarks stop between 10 million and 100 million embeddings, frequently omit exact ground truth and underrepresent sparse, multivector and filtered retrieval. Those omissions favor clean algorithm comparisons over the messy work of operating search in production, where ingestion, filters, retrieval depth and high-percentile latency can determine whether an application works.

Supernova is the strategic product

The dataset will age as embedding models and retrieval techniques change. Supernova is Qdrant's attempt to make the process repeatable. Its modules handle embedding generation, GPU-based brute-force ground-truth calculation, database ingestion and load testing. Workloads are configured through YAML files and can be divided among distributed workers, with SkyPilot used to provision and schedule infrastructure across cloud and high-performance computing environments.

Supernova's load and stress-testing tools support Qdrant, Milvus and Elasticsearch, according to the release. Tests can track ingestion throughput, queries per second, recall and latency at the 50th, 95th and 99th percentiles. That support for competing engines matters because a Qdrant-only harness would have been a product demonstration with a very large download attached. An open workflow gives Milvus and Elasticsearch engineers a path to inspect configurations, run their own hardware and contest an unfavorable result.

Qdrant still benefits if Supernova becomes widely used. Benchmark authors choose the workloads, defaults and metrics that frame a market. FineWeb-10B emphasizes scale, hybrid retrieval, filters and predictable latency, all areas around which Qdrant has built its Rust-based engine and commercial positioning. The framework can be genuinely useful while also moving buyer attention toward Qdrant's preferred criteria.

Vendor authorship complicates neutrality

Qdrant produced the benchmark, selected its initial embedding model and wrote the evaluation framework. Independent teams still need to reproduce the pipeline and tune each supported database before FineWeb-10B can serve as credible evidence in vendor comparisons. Qdrant's benchmark FAQ acknowledges that its engineers can misconfigure other engines or run them inefficiently and invites outside contributions. That is the correct posture, though code availability does not eliminate differences in operator expertise.

The dataset card also documents a numerical reproducibility wrinkle. Qdrant calculated the original results with bfloat16 GPU arithmetic, while the supplied regeneration scripts produce float32 embeddings. Qdrant says the differences are small, but they can reorder tied results or change membership near the top-1,000 cutoff. Researchers comparing exact reproductions will need to account for that boundary behavior.

Licensing adds another constraint. The dataset card says Qdrant-FineWeb-10B is released under ODC-BY-1.0, while the underlying web pages remain subject to their own terms and FineWeb's Common Crawl conditions. The MS MARCO queries used for ground truth retain Microsoft's licensing terms, including restrictions to non-commercial research. Downstream use therefore remains governed by several upstream layers.

For Zayarni and Vasnetsov, the release turns Qdrant's original frustration with research-oriented search tools into a bid to shape production research itself. FineWeb-10B supplies the intimidating number. Supernova carries the longer-term bet: developers will trust benchmark claims when they can inspect the machinery, rerun the workloads and argue over the configuration in public.