Introduction
GPU performance from 2025 through the end of the decade is growing exponentially, but the more fundamental driver of infrastructure change is what is happening to the models themselves. The shift from dense transformer architectures to Mixture of Experts designs, now dominant across frontier model development, is transforming not just compute requirements but the nature of the traffic that must flow between GPUs. Training clusters, inference clusters, and hybrid designs each serve different purposes, yet all face the same imperative. MoE architectures, exploding context windows, and agentic workloads demand larger pods, flatter fabrics, and entirely new approaches to memory that cannot be satisfied by incrementally upgrading what came before.
How AI Models Are Evolving: The Rise of Mixture of Experts
Until recently the dominant model architecture was the dense transformer. Every input token flows through every layer, activating all model parameters on every forward pass. This made the relationship between model capability and compute cost nearly linear. A larger model required proportionally more memory and compute for every inference.
Mixture of Experts breaks that relationship. In an MoE model, each transformer layer is replaced by a set of specialized subnetworks called experts, perhaps 8, 64, or hundreds of them, paired with a learned router that selects only a small subset to process each token. A model might have hundreds of billions of total parameters while activating only a fraction of them per token. The result is a model that achieves the capability of a much larger dense model at a fraction of the per token compute cost.
Given those benefits, MoE has become the architecture of choice for frontier models. Models are being built with more total experts and more active experts per token, increasing the demand for high bandwidth communications between a greater and greater number of accelerators.
Context windows are following a similar trajectory. Personalization and multimodal inputs are driving context lengths into the millions of tokens, resulting in larger KV cache data that must be moved across the network and accessed at low latency during inference. Agentic workflows, where a single user prompt triggers hundreds of model API calls through orchestrated chains of agents, add a further multiplier to both traffic volume and context management requirements.
Why MoE Transforms Network Requirements
Dense models rely primarily on tensor parallelism. Weights are split across GPUs and synchronized with all reduce operations at each layer, a communication pattern that is well understood and relatively predictable. MoE adds expert parallelism on top of this. Because each GPU holds only a subset of the experts, tokens must be dynamically routed across the network to whichever GPU holds the selected expert for that token, then results must be gathered back. This creates all to all communication patterns at every MoE layer, where every GPU is potentially exchanging data with every other GPU simultaneously.
As models scale to dozens or hundreds of experts with rising numbers of active experts per token, this all to all traffic grows dramatically. What delivered acceptable performance last year collapses under today’s workloads. The speed of model iteration leaves hyperscalers, neoclouds, and sovereign AI operators with almost no runway for gradual upgrades. Every additional expert layer multiplies bandwidth pressure inside pods and across disaggregated racks. Only large, single tier, high radix fabrics paired with pod level context memory can absorb these demands, maintaining predictable, high performance without inflating costs or power demands.
Diving Deeper into Disaggregated Inference
While training clusters run sustained collective operations across thousands of GPUs and need massive east west bandwidth with minimal jitter, inference clusters are increasingly diversifying to optimize for low latency token generation and high token throughput. Disaggregated or hybrid inference architectures split the work across different compute racks. Prefill runs on dense GPU racks while decode shifts to specialized accelerators, and all of them share a pod level memory pool.
For example, NVIDIA pairs the Vera Rubin NVL72 GPU rack with Groq 3 LPX systems that pack hundreds of LPUs. The GPU handles heavy parallel prefill and expert routing. The LPU accelerates deterministic low latency decode with 150 TB/s on chip SRAM bandwidth and delivers up to 35 times higher inference throughput per megawatt in early deployments.
Splitting inference processing across different types of accelerators in multiple racks drives the need for a new approach to shared memory. The NVIDIA CMX Context Memory Storage Appliance creates a pod level memory tier that pools KV cache and long context data across the entire pod. This shared appliance supports larger context windows, personalization, multimodal workloads, and the hundreds of API calls typical in agentic prompts. More memory removes previous bottlenecks and moves managed agents into production at scale.
Large Pod Sizes with Single tier Switching Deliver Decisive Gains
Clearly this type of disaggregation drives new networking demands, but it also opens up the opportunity to rethink network architectures.
Imagine being able to place thousands of diverse xPUs (GPUs, LPUs, CPUs) in dozens of racks within a large pod connected by a single tier of high radix switches. This large single pod design eliminates extra switch hops, minimizes latency, and saves power while delivering the highest performance. The industry needs that high radix today, and each new generation of accelerators and new innovation in models will drive demand for more.
Conclusion
The need for high radix is already apparent and it will only become more urgent. MoE’s all to all expert parallelism, expanding context windows, and agentic workflows have already changed the nature of network traffic, and every subsequent model generation will push further in the same direction. Disaggregated inference demands larger high bandwidth domains that cross multiple racks of heterogeneous compute and memory resources while maintaining single hop, low latency connectivity. High radix, single tier, large pod architectures are the only designs that can scale with this demand. Operators who commit to high radix network designs will find their infrastructure aligned with every future advance in MoE models and disaggregated compute architectures.