Pluralis Research (@Pluralis) / X

Pinned

Today we're releasing Agora: the first ever pretraining stack that allows non-collocated consumer GPUs to be competitive with centralized clusters Agora is 15x faster than Megatron-LM in this setting and is only 1.5x less efficient in terms of tokens per unit compute than

Training over the internet is bottlenecked on communication between GPUs. We’re launching Protocol NanoGPT, a speedrun challenge for multi-party LLM training over low-bandwidth links adapted from @kellerjordan0’s modded-nanogpt. Winners will be announced at the CODEC-FM workshop

We are fortunate to have 3 papers accepted to this years NeurIPS. The first is AsyncMesh - a technique for asynchronous updates across both data and pipeline parallelism simultaneously. Second is NuMuon, which shows the low-rank structure we observe in Subspace Networks is also

This exceeded all expectations and continued to train with 40+ nodes in a single stage, 370 across the swarm and GPUs in multiple continents. TPS reached over 250k tok/s. Compute wise, this local hardware swarm is equivalent to 40 colocated B200's and would run $150k per month,

Pluralis Research

@Pluralis

Sep 4

A 13B stress-test has just finished in Agora and instead of winding down we're going to ramp the contributor count until it blows up. If you have access to good local ML hardware, point your agent at agora-testing.pluralis.ai and join the pre-train for a few hours. Currently at 165

A 13B stress-test has just finished in Agora and instead of winding down we're going to ramp the contributor count until it blows up. If you have access to good local ML hardware, point your agent at agora-testing.pluralis.ai and join the pre-train for a few hours. Currently at 165

3