NVIDIA and AI4Bharat announce a 2.8T-token corpus for 22 Indian languages

AI4Bharat head Mitesh Khapra co-authored an announcement for a 22-language corpus and reusable data pipeline, both listed as coming soon.

By · Published

Primary source: Hugging Face Newsroom

Why it matters

NVIDIA and AI4Bharat announced plans to release a large Indic-language training corpus and the recipe used to generate it. The token total alone does not establish language-level quality or downstream performance.

NVIDIA and AI4Bharat build a 2.8T-token corpus for Indian-language models — AI4Bharat head Mitesh Khapra helped shape the release, which pairs translated training data across 22 languages with a reusable generation pipeline.

Mitesh Khapra, who leads AI4Bharat at IIT Madras, is helping NVIDIA tackle a problem he has described plainly: Indian-language AI lacks data. On October 16th, NVIDIA and AI4Bharat announced Nemotron-pretraining-Indic-v1, a corpus of about 2.8 trillion tokens across 22 Indian languages, alongside an open pipeline intended to let other teams generate and assess similar data.…

Reader comments

Conversation for this story loads after sign-in.