Google updates MaxText with native support for 14-million-token context training
In short: Google has updated its open-source JAX training framework, MaxText, to support training LLMs at extreme sequence lengths on Cloud TPU hardware. By introducing four distinct context-parallelism strategies, the update enables efficient training of models with up to 14 million tokens, overcoming memory and compute bottlenecks.
This summary was generated automatically by AI from Google Developers's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1Introduction of four context-parallelism strategies: All-gather, A2A (Ulysses), P2P (Ring Attention), and A2A+P2P (USP).
- 2Compatibility with TPU 5p and TPU 7 hardware.
- 3Support for 14-million-token sequence lengths.
- 4Integration of Tokamax Splash Attention kernels for efficient block-sparse attention.
| Parameter | Before | Now |
|---|---|---|
| Supported sequence length | Not specified | Up to 14 million tokens |
| Framework version | Not specified | v0.2.4 |
Why it matters
For organizations training frontier models, these tools offer a way to bypass out-of-memory errors and communication bottlenecks. This allows for significantly larger input windows—useful for processing entire codebases or long video files—without sacrificing hardware utilization efficiency.
What it means for AI agents and contact centers
If you maintain infrastructure for large-scale model training or fine-tuning, this update provides a roadmap for scaling context windows. When testing, prioritize comparing P2P (Ring Attention) against All-gather to see which strategy maintains higher MFU for your specific model architecture and head count.
🧪 Worth Testing
It is worth testing if you require custom-trained models capable of reasoning over massive datasets or long-duration inputs.
Sources
- Google DevelopersOfficialPrimary sourceOriginal article →„Announcing Long-Context Training on TPUs with MaxText Context Parallelism“9 Nov 2026, 02:00
- Published by source
- 9 Nov 2026, 02:00
- Found by our system
- 9 Nov 2026, 22:52
- Summary generated
- 9 Nov 2026, 22:53
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy