Zyphra Releases ZAYA1-8B, a Reasoning Model Trained on AMD

Zyphra has released ZAYA1-8B, a compact Mixture-of-Experts reasoning model that scores in the same range as much larger systems on mathematical benchmarks. The release is notable for two reasons: the model is available under an Apache 2.0 licence on Hugging Face, and it was trained end to end on AMD hardware rather than the NVIDIA stack that dominates frontier AI development.

The architecture totals 8.4 billion parameters but activates only about 760 million per token, a ratio Zyphra calls "intelligence density per parameter." On the published numbers, ZAYA1 is competitive on hard mathematical reasoning tasks with first-generation frontier reasoning models including DeepSeek-R1-0528, Gemini 2.5 Pro and Claude 4.5 Sonnet, according to Zyphra's announcement. MarkTechPost and VentureBeat have reported the benchmark figures independently.

An AMD-only training run

The training infrastructure is where the announcement matters most for the broader field. The full pretraining, midtraining and supervised fine-tuning pipeline ran on a cluster of 1,024 AMD Instinct MI300x nodes, connected with AMD's Pensando Pollara interconnect and built in collaboration with IBM. Until recently, AMD-only frontier training was rare enough to be remarkable on its own. ZAYA1 follows Zhipu AI's GLM-4.7, which earlier this year was trained without NVIDIA hardware at all, in suggesting that the alternatives now work at production scale rather than as proof-of-concept exercises.

What is new in the architecture

Architecturally, the model is built on what Zyphra calls MoE++, a variant of the standard Mixture-of-Experts design with three distinct modifications. The team has replaced standard attention with Compressed Convolutional Attention, which operates in a compressed latent space and yields an eight-fold reduction in KV-cache size. The router is MLP-based and stabilised with PID-controller-style bias balancing. A learned residual scaling factor keeps the residual norm under control through depth. The cumulative effect is a model that activates roughly the same number of parameters per token as a 1B dense network, but that performs in line with much larger systems.

For developers, the model is already live on Hugging Face and as a serverless endpoint on Zyphra Cloud. Zyphra also released a separate ZAYA1-8B-Diffusion-Preview on 15 May, an MoE diffusion variant converted from the autoregressive base model that promises up to a 7.7x speedup on certain workloads. For the open-source ecosystem more broadly, the release lands at a moment when DeepSeek, Mistral and Llama have already closed much of the gap to closed-weight models. ZAYA1 adds another option, and an AMD-trained one at that.