Amazon's 'Priming' Turns Pretrained Transformers Into Efficient Hybrid Models for Under 1% of Training Cost
In short: Amazon Web Services researchers published a method called 'priming' that converts pretrained Transformer models into hybrid architectures mixing attention and state space model (SSM) layers, reusing existing weights instead of retraining from scratch. The approach cuts pretraining costs by over 99% and lets a 32-billion-parameter hybrid model match its source Transformer's quality while decoding up to twice as fast. Amazon tested three SSM types and released an open-source model zoo, training code, and serving tools under Apache 2.0.
This summary was generated automatically by AI from Amazon Science's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1Priming converts attention layers to SSM layers using less than 0.5% of a model's original pretraining token budget
- 2Among tested SSM types, Gated KalmaNet (GKA) outperformed Gated DeltaNet (GDN) and Mamba-2
- 3A 32B GKA hybrid matches its source Transformer within 1% accuracy using half the KV cache
- 4SSM-based hybrids outperformed sliding-window attention hybrids even with 2-4x less memory
- 5GKA allows adjustable inference-time compute (e.g., 10 vs 30 iterations) trading ~1% accuracy for 10% faster decoding
- 6Open-sourced 8B and 32B model zoo, training code, optimized kernels, and vLLM plugin under Apache 2.0
| Parameter | Before | Now |
|---|---|---|
| Training cost vs scratch | 100% (full pretraining) | Less than 1% (under 0.5% of token budget) |
| Throughput | 1x (source Transformer) | Up to 2x at 128,000 tokens context |
| KV cache size | Full attention cache | Half the KV cache at inference (32B GKA hybrid) |
| Context length tested | Not specified | 128,000 tokens |
| License | Not specified | Apache 2.0 |
Why it matters
Hybrid architectures reduce the memory and compute cost of long-context inference, which directly affects latency and throughput for applications handling long conversations, reasoning chains, or large documents — a core bottleneck for scaling AI agents.
What it means for AI agents and contact centers
For teams running long voice-agent sessions, large call transcripts, or reasoning-based automation, these open, efficient hybrid models could offer lower latency and cost per token at similar accuracy to standard LLMs — worth benchmarking against current STT/LLM pipelines for throughput and context handling in Lithuanian or other smaller-language workloads.
🧪 Worth Testing
The open-sourced 8B/32B hybrid models under Apache 2.0 promise up to 2x throughput and smaller memory footprint at comparable quality, making them a practical candidate to test for latency-sensitive, long-context voice and automation workloads.
Priming (hybrid state space models from pretrained Transformers)· New
Sources
- Amazon ScienceOfficialPrimary sourceOriginal article →„Can we change an AI model’s architecture without retraining it from scratch?“12 Oct 2026, 20:36
- Published by source
- 12 Oct 2026, 20:36
- Found by our system
- 12 Oct 2026, 20:40
- Summary generated
- 12 Oct 2026, 20:42
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy