Compute scales about 3x every two years; memory bandwidth scales under 2x. NVIDIA, AMD, Google, Intel, and OpenAI each answered that gap differently.
Hot Chips 2026 brought a familiar flood of peak-compute claims, but they did not describe the same thing. NVIDIA showed 2 zettaFLOPS of NVFP4 inference at a modeled 100-megawatt AI-factory scale; AMD rated one 72-GPU Helios rack at 2.9 exaFLOPS of OCP MXFP4; and Google rated a 9,600-chip TPU 8t superpod at 121 exaFLOPS of FP4. Those figures are not directly comparable. The useful common thread came from Micron's memory tutorial: accelerator compute has been scaling at roughly 3x every two years while attached-memory bandwidth has improved by less than 2x. Across the GPU, custom-ASIC, memory, and networking sessions, HC38 (Aug. 23 to 25 at Stanford's Memorial Auditorium) showed vendors trading different resources to keep more of their peak arithmetic busy.
Start with the two flagship GPU racks, because that is where scale mixing does the most damage. NVIDIA's Rubin figures separate by boundary. At the modeled 100-megawatt AI-factory level, NVIDIA cites about 2 zettaFLOPS of NVFP4 inference and 1.4 zettaFLOPS of training, with roughly 11 petabytes of HBM4 and 800 petabytes per second of aggregate bandwidth. Those are factory aggregates, and they appear on no NVIDIA-published page: they are conference figures for a reference configuration. One Vera Rubin NVL72 rack, by NVIDIA's own published numbers, is about 3.6 exaFLOPS of NVFP4 inference and 2.5 exaFLOPS of training, with roughly 20.7 terabytes of HBM4 and about 1.6 petabytes per second of peak HBM bandwidth. The zettaFLOPS headline is real, but it belongs to the building, and the FLOPS exist only at NVFP4, NVIDIA's four-bit format, so they do not line up with FP8 or FP16 numbers from anyone else. The Rubin talk spent much of its time on reliability engineering: hot-swappable NVLink switch trays, liquid cooling, RAS, and power smoothing. Our read is that at 100-megawatt scale, keeping the machine running is at least as hard as raising its peak, though NVIDIA did not put it that way. And Rubin was already in production before Hot Chips; HC38 added architecture detail.
AMD's Helios rack is the one place among the opening numbers where the figure is genuinely rack-scale. AMD specifies 72 MI455X accelerators at 2.9 exaFLOPS of OCP MXFP4, 31 terabytes of HBM4, and 1.67 petabytes per second of peak theoretical memory bandwidth. The MI455X is a chiplet part: compute dies on TSMC's N2, with fabric, cache, and I/O dies on N3P. On peak specifications, that is competitive with a Rubin rack, though "parity" is the wrong word, since it turns on precision, workload, power, and measured utilization. AMD's harder problem is not the silicon. It is software portability: whether ROCm runs the breadth of models and kernels that CUDA does without hand-tuning is a real and testable question, and one SCN has framed as a composability problem. AMD used the show to press its software case; the show did not settle it. Helios volume deployments are targeted for the second half of 2026.
"The memory wall" is shorthand for several constraints that bite at different moments, and LLM inference hits them in sequence. Prefill, the phase that ingests a prompt, can be compute-bound. Decode, the token-by-token generation phase, is often bandwidth-bound. A distributed mixture-of-experts model adds a communication bottleneck between chips. And across all of them, software decides how much of the peak a workload actually reaches. OpenAI drew this distinction in its own Jalapeño talk. So memory bandwidth does not set a ceiling on every FLOPS number, which would be a category error, because peak FLOPS are an arithmetic-throughput specification. Memory bandwidth, capacity, locality, and interconnect often limit how much of that peak a given workload, at a given operating point, can actually use. That thread runs through HC38.
If bandwidth is one ceiling, capacity is the other axis, and Intel's Crescent Island pushed on it. Intel described a 350-watt, air-cooled inference GPU carrying 160 gigabytes of LPDDR5X, configurable up to 480 gigabytes on ODM designs, with customer sampling targeted for the second half of 2026. Intel frames it as trading HBM bandwidth for LPDDR capacity and air-cooling economics, aimed at inference that needs room to hold long contexts and KV-caches more than it needs maximum bandwidth. It is a real architectural fork, and it arrives against Intel's recent accelerator record, including the cancellation of Falcon Shores as a commercial product, which is the context a buyer will weigh.
Memory ran through HC38 from Sunday's tutorial to Tuesday's session, and the options varied enough in maturity to warrant separating them. In the session, Samsung presented what it and the program call the world's first LPDDR5X-based processing-in-memory part for inference, with productization discussed for 2026, and XCENA, co-presenting with Samsung, showed a CXL computational-memory device, part of the broader fork SCN has mapped between CXL pooling and processing near memory. The tutorial carried the earlier-stage work: d-Matrix and Meta on a 3D-DRAM accelerator for generative inference, and Oxmiq Labs walking through where high-bandwidth flash pencils out and where it does not. HBF is worth stating precisely, because it is routinely described as a middle tier and is not one: it is NAND in an HBM-style package, block-accessed like storage rather than addressed like memory, and Oxmiq's own simulation put it at roughly 14 times the capacity for about 0.6 times the bandwidth. These sit at different points from productization to research, and the honest summary is that the industry has several plausible ways to widen the memory path and no single proven one at volume. The per-stack arithmetic is worth pinning down while here, because two different figures circulate and both trace to AMD. Its MI455X datasheet gives 432 gigabytes and 23.3 terabytes per second per GPU, which works out to about 1.94 across twelve stacks; its Helios page still carries the pre-production 19.6 alongside the shipping number, which is where the roughly 1.63 per-stack figure comes from. Micron specifies more than 2.8 terabytes per second per 36-gigabyte stack and Samsung up to 3.3. SCN has traced why that HBM supply concentrates in a few vendors.
The freshest material was the custom silicon. OpenAI gave the first detailed technical account of Jalapeño, its inference ASIC built with Broadcom, which the two announced in June; Hot Chips supplied the architecture and the benchmarks. Richard Ho, Ravi Narayanaswami, and Chris Leary described a nine-month path from initial design to tapeout, an architecture built around HBM4 and a spatial programming model, and a design tuned to inference metrics, time-to-last-token, and tokens per joule, rather than the time-to-first-token SCN has argued is not the metric that defines the era. The benchmark numbers are OpenAI's own, run on InferenceX, SemiAnalysis's public benchmark, and normalized by published package power: a 700-watt Jalapeño, which OpenAI says sustains at or below 550 watts on the tested workloads, against a 1.2-kilowatt GB200 and a 1.4-kilowatt GB300. On GPT-OSS 120B, OpenAI reports 85,448 versus 44,960 mixed tokens per second per kilowatt, roughly 1.9 times, against a GB200 at 1.2 kilowatts. Two caveats belong beside that figure. OpenAI's chart is labeled single-token prediction throughout, so it is not the same comparison as SemiAnalysis's own perf-per-watt chart, where Jalapeño runs single-token prediction against NVIDIA parts in their best multi-token configurations; the two get quoted interchangeably and should not be. And nobody has reproduced this: SemiAnalysis says it witnessed the InferenceX runs in the lab but did not run the suite itself. It is a vendor-reported result on engineering silicon, which OpenAI says it ran GPT-OSS 120B, DeepSeek R1 670B, and the trillion-parameter Kimi K2.5 on after first silicon, organized as a 128-chip local domain and a 2,048-chip global domain, with deployment targeted by year-end.
The rest of the track showed mixed readiness. Google detailed its eighth-generation TPU, the first it has launched as two purpose-built chips from the outset, the 8t for training and the 8i for serving, though inference-specific TPUs go back to TPUv4i. Its Cloud Next disclosure puts an 8t superpod at 9,600 chips, 2 petabytes of shared HBM, and 121 exaFLOPS, FP4 by the per-chip specs, with both chips generally available later this year. Meta's MTIA 400 broadened the accelerator beyond ranking and recommendation into general GenAI workloads, adding hardware MXFP4, a 72-chip scale-up domain, and about 12 petaFLOPS of FP4 per accelerator, on HBM3E rather than HBM4. Microsoft presented Maia 200 as first-party silicon with 216 gigabytes of HBM3E, naming no external design partner. Marvell Technology is widely reported as a co-design partner on the Maia line and is the named candidate for Maia 300, but no source places it on Maia 200. Cerebras took its wafer-scale engine rack-scale, and NVIDIA presented a language processing unit it did not originate: the NVIDIA Groq 3 LPU, a Groq design it now brands and ships as its own, reportedly via an acquihire of the company, and in full production before the show. It ships as a 256-LPU LPX rack aimed at the low-latency decode phase where GPUs are weakest, and NVIDIA framed it as heterogeneous compute filling a gap in the Rubin stack rather than an answer to the custom-inference wave. Counterpoint Research projects Broadcom holding 60 percent of AI server compute ASIC design-partner share in 2027, a design-services measure it explicitly decouples from shipment volume. Their captivity varies too. The hyperscaler ASICs, TPU, MTIA, and Maia, are built first for their owners' clouds, though Google also sells third-party TPU cloud access and reports TPU-system sales, while Cerebras and NVIDIA sell merchant hardware. The trend SCN has tracked is that custom inference silicon could pressure NVIDIA's inference margins well before it dents its merchant position.
The networking track was the same portability fight one layer down. Broadcom's Thor Ultra is an 800-gigabit scale-out Ethernet NIC compliant with the Ultra Ethernet Consortium, not a UALink scale-up product. That compliance buys Ultra Ethernet's multipath reliability layer, which SCN has covered at 75,000-GPU scale. NVIDIA's fabric is three distinct things: NVLink for scale-up inside a rack, Spectrum-X Ethernet for scale-out across racks, and BlueField-4 for scale-in infrastructure offload. Spectrum-X runs on Ethernet, and proprietary implementation features do not make Ethernet itself a closed standard. The reason it matters to the memory story is that moving data between chips- model weights and KV-cache, but also training gradients, expert routing, storage, and control traffic- is what turns a pile of accelerators into one machine, and bandwidth there is as scarce as it is on the memory bus. Whether open multivendor Ethernet or NVIDIA's integrated stack prevails is not something a conference settles; both had disclosures, and BlueField-4 partner availability, like several parts here, is a second-half-2026 date.
The CPU sessions are best read as one observer's synthesis, because the talks pointed in different ways. Intel's Diamond Rapids still emphasized core count, up to 256; Fujitsu's MONAKA emphasized efficiency; Arm's chiplet-based reference server SoC, called AGI, emphasized coherency and chiplet bandwidth; NVIDIA's Arm-based Vera is the host half of Vera Rubin; and IBM's future Z processor emphasized dual-ISA operation and on-die AI inference. What several of them share, and what we would draw out, is that the interesting axis is increasingly memory bandwidth and coherency alongside cores, the case SCN made for the CPU in the agentic data center. One number to correct: Arm's AGI SoC is not built for terabyte-per-second DRAM. Its twelve-channel DDR5 controller runs about 845 gigabytes per second per socket at DDR5-8800, and Arm's own launch figures imply roughly 816; the 2-terabyte-per-second number from the talk is aggregate UCIe die-to-die bandwidth. Arm sized the die-to-die links deliberately above memory bandwidth, which is the more interesting fact and the one the talk's title points to.
Underneath sat the physical substrate. Higher-voltage rack distribution, the 800-volt DC transition, is an engineering answer to very high per-rack power, not a blanket requirement of every large site. And chiplet integration is now load-bearing for several of these parts, the MI455X, MTIA 400, and Arm's AGI SoC among them, though not for everything shown, part of the shift SCN traced from architecture slides to production silicon.
HC38 put mature platforms, engineering silicon, near-term deployments, and research proposals on the same stage. Their common problem was not that FLOPS are fictitious, but that peak arithmetic increasingly outruns the systems that must feed it. Rubin and Helios attack it with large HBM4 pools and rack-scale fabrics; Intel trades HBM bandwidth for LPDDR capacity; Google separates training and inference TPUs; OpenAI co-designs memory locality, networking, and software for inference; and the memory sessions explored less mature alternatives. The next test is commercial: which of these improve measured workload performance, at acceptable power and cost, in systems customers can deploy.
The SCN Staff is a small AI editorial squad working under human direction. Each agent owns one job.
Scout does the research. It runs down primary sources and checks what's already been published, on SCN and everywhere else, before a story gets written. If a claim can't be traced back to a real document, Scout flags it.
Forge writes. It takes what Scout found and turns it into a draft, argument and sentences and all. Every SCN piece starts here, then gets sharpened.
Cipher handles search: the titles, descriptions, and keyphrase work that decides whether a good article ever gets found. Least glamorous job on the squad. Also one that matters more than it looks.
Pixel makes the visuals. Images, charts, the occasional diagram, all built to SCN's brand instead of pulled from a stock library. When something's easier to see than to read, it goes to Pixel.
Editorial judgment and the final call stay with the humans. So does the fact-checking.
