Ai2

@ai2.bsky.social

Breakthrough AI to solve the world's biggest problems.

› Join us: http://allenai.org/careers
› Get our newsletter: https://share.hsforms.com/1uJkWs5aDRHWhiky3aHooIg3ioxm

@ai2.bsky.social

Over a 30-day test with our new scheduler, teams received 98% of the GPU hours they were owed, accounting for actual demand. Cluster occupancy stayed at 98%, with spare capacity available to interruptible work that didn’t draw down a team’s budget.

@ai2.bsky.social

Jobs declare the minimum runtime needed to make progress. Budgeted work is protected until then. Afterward, it can keep running or be interrupted; resumable jobs return to the queue.

@ai2.bsky.social

Managers now assign GPU-time budgets to research programs & projects. Our fair-share scheduler compares recent use with those budgets, moving work from underused allocations ahead in-queue.

@ai2.bsky.social

Under our old scheduler, every scheduled workload eventually used HIGH priority. Some researchers kept idle jobs running to reserve GPUs for experiments. The incentives rewarded holding on to capacity even when there were no active jobs.

@ai2.bsky.social

We’re sharing the engineering work behind Ai2’s GPU scheduler—our new approach to allocating compute across research teams.

On our largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds. 🧵

buff.ly/4UsJEYe

@bminixhofer.bsky.socialReposted by @ai2.bsky.social

Super excited to share that our method to convert language models to byte-level has been published in @nature.com!

This was originally the approach behind Bolmo. We now generalized it to other model families.

All checkpoints (including new Qwen and Llama models) and data are public!

@ai2.bsky.social

We hope Bolmo opens a path to larger models that adapt how they represent information across languages & domains. Because bytes also represent images & audio, the same ideas could eventually extend beyond text.

Learn more in our blog: buff.ly/eAfLyFG

@ai2.bsky.social

We're making new Stage 1 checkpoints available with the byte-level components already trained & the original model's weights unchanged. Researchers can build on these to test new architectures & train the full system without repeating that initial stage.

@ai2.bsky.social

We byteified Qwen 3 8B & Llama 3 8B to create Bwen 8B & Blama 8B.

Both come close to matching their source models' performance in our evaluations. Bwen 8B also outperforms Bolmo 7B across our aggregate evaluation suite.

buff.ly/9BSybFo

@ai2.bsky.social

Our recipe adapts existing models to bytes with a short additional training run. It keeps the model's core, adding components that group bytes into variable-length patches for it to process, then expand its outputs back to byte-level representations to predict the next byte.

@ai2.bsky.social

Most language models split text into subwords—words or fragments of words from a fixed vocabulary. This can obscure spelling details across writing systems & split meaningful units in code or math. Byte-level models work directly with the bytes computers use to represent text.

@ai2.bsky.social

Our paper on retrofitting language models to operate over bytes – the approach behind Bolmo – has been accepted to Nature! 🎉

We’re also releasing new checkpoints that extend our method from Olmo to Qwen & Llama. 🧵

buff.ly/l5ILoNT

Older posts