A new open pipeline released 2.8 trillion tokens for Indic language pretraining, helping reduce the English-data dominance in large models. The release makes it easier for researchers to train models that…

#AI #NLP #IndicAI #OpenData
https://huggingface.co/blog/nvidia/scaling-indic-pretraining