NVIDIA has priced training across 1,000 km; Google trains synchronously across data centers. Missing: what one job pays to span mixed silicon and fabrics.
Meta's next flagship AI training system is not a building. Prometheus, which the company still expects to bring online by the end of 2026 (a schedule reaffirmed at this week's IEEE Hot Interconnects keynote, per HPCwire), is a collection of data centers in Ohio that Omar Baldonado, Meta's senior director of data center and AI networking, described as a mix of liquid-cooled and air-cooled buildings holding unequal amounts of compute. Its successor, Hyperion, in Richland Parish, Louisiana, is designed to expand to 5 GW of compute capacity, a ceiling with no date attached. Meta has not published an energized-megawatt figure for either site; the Prometheus and Hyperion gigawatt numbers are targets and design ceilings.
The engineering record splits in two. Distance has a measured cost, and in the one configuration NVIDIA has published, 3,072 GPUs split across roughly 1,000 kilometers, it was small: about two points of model FLOPS utilization, under 4% of throughput. Heterogeneity, the mix of silicon, fabrics and cooling that accumulates when buildings energize years apart, has no comparable public number. Even Meta's own benchmark work, published this spring, stops at the homogeneous case and names the mixed one as future work.
A gigawatt of training capacity arrives as a set of buildings sited where interconnection capacity and turbine delivery slots clear. Baldonado spent his keynote on what it takes to make those pieces behave as one machine: scale-across networking, the layer above the scale-up domain in the rack and the scale-out fabric in the building.
The boundaries are physical, though the domain names are architectural conventions rather than formal standards. Scale-up is the domain that behaves as a single computer: copper reaches roughly five meters at current high-speed interconnect rates, and every accelerator inside it expects to synchronize with no perceptible delay. Scale-out is the fabric that joins those systems across a building; Baldonado put the practical edge of InfiniBand and RDMA over Converged Ethernet at "a few hundred meters," allowing that "some RoCE systems can handle distances up to 50 kilometers away." Scale-across is everything past that point; in his definition, it is also everything a single site's energy supply can no longer feed. SCN has covered the scale-up bottleneck from the optics side; this is the layer above it.
Not all of a training job has to cross that boundary. Tensor parallelism, "the tightest computation loop of AI jobs" in Baldonado's phrase, synchronizes at every matrix multiply and stays inside the scale-up domain. NVIDIA's NeMo team reached the same placement in a May 2025 technical post on long-haul training: tensor, context, and expert parallelism stay local because they "demand frequent, high-bandwidth synchronization," and only data and pipeline parallelism cross the wide-area link.
The numbers in that post are the distance answer. NVIDIA split 3,072 GPUs evenly between Chicago and Ashburn (roughly 1,000 kilometers, 21 milliseconds round trip) and trained Nemotron-4 340B at what it reports as over 96% scaling efficiency against a single-site baseline, with model FLOPS utilization slipping from 51% to 49%. One self-reported vendor benchmark does not make a law. But Google has stated a matching conclusion from production: the Gemini 2.5 technical report says the model "employed synchronous data-parallel training to parallelize over multiple 8960-chip pods of Google's TPUv5p accelerators, distributed across multiple datacenters." Meta is not first across this boundary, just the operator most willing to publish the plumbing. Researchers who tolerate asynchrony go further: in a simulated 100-billion-parameter configuration, an overlapped FP4 variant of Streaming DiLoCo reaches 95% compute utilization at 1.1 Gbit/s, against roughly 391 Gbit/s modeled for conventional data parallelism. Its empirical runs stop at billion scale, and the approach remains a research result.
Baldonado was explicit about which problem is his. "Heterogeneity is really the problem that scale-across deals with," he said, per HPCwire. When people think about scale-across, he added, "it's only about... distance, which is an important factor. But you have to really think about what it is that you're connecting."
The word covers at least four things: accelerator generation (buildings energized years apart hold different silicon vintages), accelerator vendor and instruction set, network fabric (different scheduling and buffering behavior), and facility design (liquid-cooled buildings next to air-cooled ones). They tax a training job differently. Cooling mostly constrains where hardware can go and how densely it can be packed. Mixed accelerator performance and mixed fabric behavior enter the collectives directly, where the slowest painside a 150 MW, 83,000-GPU Meta facilityrticipant sets the pace.
Meta's fleet exhibits all four. It deploys accelerators from NVIDIA and AMD alongside its own MTIA parts, and operates two fabric architectures: the Disaggregated Scheduled Fabric, a credit-based design that runs non-blocking up to 18,432 accelerators, and the Non-Scheduled Fabric, a shallow-buffer design with adaptive routing, disclosed at the OCP Summit in October 2025. But fleet diversity and job diversity are different claims. No public disclosure shows Meta running one synchronous training job across NVIDIA, AMD, and MTIA silicon at once. The scale-across problem lives between those two states.
The joining layer is documented. Backend Aggregation is a centralized Ethernet super-spine that Meta illustrates with a five-building example, with inter-region capacity Meta puts at 16 to 48 petabits per second and oversubscription around 4.5:1. Its line cards are built on Broadcom Jericho3 silicon; the same OCP Summit disclosures include Minipack3N, a 51.2 Tbps switch built on NVIDIA's Spectrum-4 ASIC. Meta also keeps runs from the fabric edge short, since shallow-buffer switches punish long links. SCN has previously reported on a 150 MW, 83,000-GPU Meta facility described as one slice of a 1 GW buildout, consistent with Prometheus's scale, though Meta has not publicly connected the two.
Meta has begun to benchmark the layer. In May, Meta and Harvard researchers posted ScaleAcross Explorer, an optimizer that searches jointly over parallelism placement, scheduling, and network design for cross-building training, drawing on production experience. It reports up to a 64.62% speedup over Meta's manually tuned production configuration, evidence that cross-site placement is unsettled even inside the company running it. The experiments, though, are homogeneous: 64 to 128 H100 GPUs with fiber-emulated distance, plus packet-level simulation at 100,000-GPU scale, H100s throughout. The paper's future-work text names what is missing: buildings "may house different generations and numbers of accelerators and intra-building topologies, and may have heterogeneous distances." So Meta itself draws the boundary of public knowledge. A scale-across benchmark now exists for the homogeneous case. No public apples-to-apples benchmark exists for one synchronous training job spanning mixed accelerator generations and mixed fabrics.
Why heterogeneity accumulates is a power story SCN has covered all year. Grid interconnection timelines now run longer than training roadmaps, and the queue for new gas turbines stretches to 2031. A gigawatt of contiguous interconnection is rarely available on a frontier lab's schedule, so clusters get sited in pieces that energize years apart. Sequencing on that timescale does not guarantee mixed hardware, but it systematically raises the odds, with silicon refresh cycles, procurement, leasing, and construction timing doing the rest. A building that comes up in 2026 is likely to reflect 2026 silicon and 2026 cooling design; a 2029 building is not. The hardest problem Meta's network team describes is the downstream invoice for a power-market condition that is reorganizing supply chains far beyond networking. No collective-communications library fixes a procurement history.
The fiber market is planning on the same logic. Zayo's April announcement of an 8,000-route-mile build, its largest ever, prioritizes corridors "where power availability is driving data center demand and those data centers need to connect to adjacent metros." The April release described an unnamed anchor customer; an August release names NVIDIA as a partner on the build. Neither states a budget or an in-service date. Microsoft says it delivered over 120,000 new fiber miles in a single year for the AI WAN that lets it "directly connect different generations of supercomputers." Route miles and fiber miles do not compare, but both point the same way: long-haul fiber is organizing around AI power geography.
The on-record reach numbers diverge because vendors and operators measure different things. Baldonado, per HPCwire, put RoCE's practical ceiling at 50 kilometers. Broadcom claims lossless RoCE transport beyond 100 kilometers for its Jericho4 silicon. Cisco's 8223 router, built on Silicon One P200, "features 800GE coherent optics support, enabling data center interconnect and metro applications reaching up to 1000km": an optics-reach figure, not a demonstrated lossless-RDMA domain, and not like-for-like with Broadcom's claim. Three quantities travel under one word: a vendor's maximum spec, the physical reach of coherent optics, and the distance an operator will accept for a production job burning capital by the hour. The spread is not evidence that anyone is wrong. Ciena, whose keynote follows Baldonado's, lists effects in its abstract that help explain it: scale-across "intensif[ies] familiar interconnect challenges such as jitter, burstiness, congestion, and long-tail latency," alongside optics link budgets, propagation delay, transport semantics and recovery behavior.
NVIDIA's entry, Spectrum-XGS, announced a year ago this week with distance-adjusted congestion control as its stated differentiator, claims to nearly double collective-communications performance, a figure published with no distance, topology, model, or GPU count attached. CoreWeave's CTO said at launch that the technology means "we can connect our data centers into a single, unified supercomputer." Can, not have: no second publicly named customer, and no published production deployment, appears in NVIDIA's materials as of August 20, 2026.
The supercomputing category's newest networking layer has one half in production and the other in preprints. Distance has published methods and a measured cost in at least one named configuration. Heterogeneity has an architecture disclosure, a homogeneous benchmark, and a future-work section. A meaningful result is specifiable now: one synchronous training job, same model and global batch size, run across sites with the accelerator generations and fabrics in each named; homogeneous single-site and multi-site baselines; throughput or MFU reported with collective tail latency, straggler behavior, and failure recovery. That is a prediction of a paper, not a report of one. Based on this week's keynote and who wrote ScaleAcross Explorer, Meta is the operator most likely to publish it.
The SCN Staff is a small AI editorial squad working under human direction. Each agent owns one job.
Scout does the research. It runs down primary sources and checks what's already been published, on SCN and everywhere else, before a story gets written. If a claim can't be traced back to a real document, Scout flags it.
Forge writes. It takes what Scout found and turns it into a draft, argument and sentences and all. Every SCN piece starts here, then gets sharpened.
Cipher handles search: the titles, descriptions, and keyphrase work that decides whether a good article ever gets found. Least glamorous job on the squad. Also one that matters more than it looks.
Pixel makes the visuals. Images, charts, the occasional diagram, all built to SCN's brand instead of pulled from a stock library. When something's easier to see than to read, it goes to Pixel.
Editorial judgment and the final call stay with the humans. So does the fact-checking.
