NVIDIA Claims 30x AI Throughput per Dollar With Memory Codesign
NVIDIA says full-stack codesign lifts agentic AI throughput per dollar up to 30x by shrinking weights, tiering KV cache and keeping RAG indexes off HBM.
Summary
On November 5, 2026, NVIDIA said larger models, longer contexts and agentic apps are squeezing scarce, costlier GPU memory. Frontier models have trillions of parameters; agentic trajectories can process up to 3 million tokens, while RAG adds context. HBM holds weights and KV cache, but bandwidth, power and latency create a memory wall that can strand added capacity. First to integrate HBM3e and HBM4, NVIDIA codesigns GPUs, CPUs, networking, storage and software across HBM, CPU memory, local SSDs and shared storage. From NVIDIA Blackwell Ultra to Vera Rubin, HBM per GPU rose about 3%, while SemiAnalysis AgentX throughput per dollar rose up to 30x.
NVFP4 shrinks weights and KV cache versus FP8, NVIDIA Hopper’s lowest precision. NVFP4 checkpoints include FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL. NVL72 distributes weights across up to 72 GPUs. NVIDIA Vera CPU accelerates tools and uses LPDDR linked to Rubin GPUs by NVLink-C2C; NVIDIA Dynamo moves idle context into shared, cheaper NVIDIA CMX context memory storage and reuses cache across GPU, CPU and storage. CMX partners include Cloudian, DDN, Dell Technologies, Everpure, previously Pure Storage, HPE, IBM, NetApp, VAST and WEKA. NVIDIA Nemotron Retriever makes smaller, more accurate embeddings, while NVIDIA cuVS serves RAG indexes outside HBM. On SemiAnalysis AgentX DeepSeek V4 Pro, NVIDIA GB300 NVL72 delivered up to 10x NVIDIA H200 throughput per dollar at 100 tokens per second per user, with roughly double the HBM per GPU; Vera Rubin NVL72 reached up to 30x GB300 at 125 tokens per second per user with about 3% more HBM.
Positives
- Vera Rubin NVL72 reached up to 30x GB300 throughput per dollar at 125 tokens per second per user with about 3% more HBM.
- GB300 NVL72 delivered up to 10x H200 throughput per dollar at 100 tokens per second per user with roughly twice the HBM.
- NVL72 distributes model weights across up to 72 GPUs, freeing each GPU’s HBM for KV cache and concurrent users.
- NVFP4 checkpoints from six model builders demonstrate growing adoption of memory-efficient 4-bit inference.
- Dynamo reuses KV cache across GPU memory, CPU memory and storage instead of recomputing context.
- Nemotron Retriever and cuVS keep compact, accurate RAG indexes outside scarce HBM.
Risks & concerns
- GPU memory supply is tightening and prices are rising as larger models, longer contexts and agentic applications increase demand.
- Agentic trajectories can process up to 3 million tokens, causing each request’s working memory to grow over repeated model and tool calls.
- Frontier models contain trillions of parameters, leaving model weights and KV cache competing for limited HBM.
- Every additional user shares the same compute and memory bandwidth, eventually limiting response speed despite added HBM capacity.
- Beyond the memory wall, extra HBM can remain unusable while still increasing system cost and power consumption.