インド言語の大規模事前学習：2.8兆トークンとオープンデータ生成パイプライン

NVIDIAとAI4Bharatが公開した2.8兆トークン規模のインド言語事前学習データセットと生成パイプラインの技術解説。

#LLM #Pretraining #NVIDIA #OpenSource #Multilingual

DeBLAS: Accelerate LLM Pretraining by Length-based Sequence Scheduling

Geng Yu, Jianing Zhu, Jiangchao Yao, Bo Han, Gang Niu, Masashi Sugiyama

Action editor: Andriy Mnih

https://openreview.net/forum?id=s0MTAeYvww

#pretraining #batches #training

New arXiv paper defines three distinct signals for synthetic pretraining tasks—diagnostic, teachable, and transferable—and finds only 14 of 27 tasks are teachable under controlled Python pretraining. Useful framing for…

#OpenSourceAI #LLM #CodeLLMs #Pretraining
https://arxiv.org/abs/2610.11548

what pretraining looks like #CATCHAT is using a stat-of-the-art GRU-L3-V3a #LLM written with #LLVM #CPP and #Vulkan #compute #shaders . no python, training on our first #dataset of dialogue 397M dialogue turns tokens and 78.9m parameters as for our phase0 #pretraining running on 1x L40S by #NVIDIA