Cross-Datacenter Inference Becomes Plausible: KV-Efficient Architectures Shift the AI Infrastructure Mix

1. Executive Overview

The paper's central claim is that generative AI inference is entering a new systems phase in which KVCache-efficient model architectures make cross-datacenter prefill/decode disaggregation technically plausible, but only when paired with selective routing, cache-aware scheduling, and bandwidth control. The practical shift is not that networking stops mattering. It is that the binding constraint moves from every prefill and decode chip needing to sit inside the same RDMA island to the scheduler deciding which long-context, low-cache-hit, compute-heavy requests are worth exporting to remote prefill capacity.

That distinction matters for AI infrastructure. It relaxes co-location requirements for heterogeneous accelerators, allows prefill and decode hardware pools to scale independently, and turns KVCache from a local runtime artifact into a schedulable and transferable resource. In the paper's internal 1T-parameter hybrid-model case study, the optimized PrfaaS-PD deployment delivers 54% higher throughput than a homogeneous PD baseline, 32% higher throughput than naive heterogeneous PD, 64% lower P90 TTFT, and only 13 Gbps of average cross-cluster egress, equal to 13% of a 100 Gbps Ethernet link. The result is not a universal benchmark, but it is a credible proof point that long-context serving economics can move materially when model design, cache management, routing, and hardware specialization are co-designed.

The paper is therefore more precise than a generic efficiency story. It argues that model architecture changes the network regime in which PD disaggregation is economically deployable. That is why the right reading is deployment-boundary first, infrastructure mix shift second. Hybrid attention reduces the amount of movable state; the scheduler determines whether that state is moved only where the compute-side gain outweighs the transfer cost.

The most important investment conclusion is that this architecture is not broadly deflationary for AI infrastructure demand. It is mix-shifting. It reduces the need to attach every inference FLOP to maximum-HBM, maximum-RDMA, tightly co-located infrastructure, but it increases the value of phase-specialized accelerators, Ethernet and optical DCI, NICs, DPUs, distributed KVCache software, host DRAM, NVMe SSD tiers, fleet telemetry, and orchestration software.

2. What Is Actually New Here Versus Prior Systems

The paper should not be read as claiming that PD disaggregation, heterogeneous serving, or KVCache pooling were previously unknown. DistServe established the value of separating prefill from decode. Mooncake treated KVCache as a first-class distributed systems resource. A broader literature already explored heterogeneity-aware placement, phase-specialized serving, and KV reuse. The paper's distinct contribution is narrower and more important: it shows that hybrid-attention models can reduce KV throughput enough to move the deployment boundary of PD disaggregation toward cross-cluster serving, but only when cache-aware routing and bandwidth-aware scheduling are solved jointly.

That distinction matters because it sharpens the paper's intellectual contribution. The novelty is not that PD exists or that KVCache can be managed. The novelty is that selective cross-datacenter prefill offload becomes a credible serving primitive once hybrid-model KV throughput, cache placement, and scheduling are optimized together.

3. Core Evidence

The paper starts from the familiar inference split between prefill and decode. Prefill is compute-intensive, especially for long prompts. Decode is memory-bandwidth-intensive because model weights and KV state must be read repeatedly for each token. Prior systems such as DistServe showed why co-locating those phases creates interference and forces TTFT versus TPOT tradeoffs, while Mooncake moved the field forward by treating KVCache as a first-class system resource spanning CPU, DRAM, SSD, and NIC resources. The new contribution is the move from intra-datacenter disaggregation to cross-datacenter disaggregation, where feasibility depends on whether KVCache can cross lower-bandwidth, higher-latency links without erasing the gains from remote compute specialization.

The technical unlock is hybrid attention. Dense attention produces KVCache that scales aggressively with context length across many layers, making state transfer prohibitive. Hybrid models shrink the exported state because only a subset of full-attention layers emits sequence-length-dependent KVCache, while linear, sliding-window, and recurrent-state layers maintain bounded or materially smaller state. The result in the paper's examples is a 4x to 13x reduction in directly comparable 32K-token KV throughput and, in the Ring-2.5-1T discussion, up to 36x memory savings when MLA compression and a 7:1 hybrid ratio are combined.

The paper's most important conceptual distinction is between smaller KVCache and practical cross-datacenter PD. Smaller KVCache is necessary but insufficient. Production workloads are shaped by request length, prefix-cache hit rate, traffic bursts, link congestion, and heterogeneous hardware availability. Blindly offloading all requests wastes bandwidth on short prompts and can still create congestion spikes on longer ones. PrfaaS therefore offloads only incremental uncached prefills above a threshold and keeps short or bandwidth-unfriendly requests on the local PD path. The paper is best understood as a joint model-architecture, cache-placement, and resource-scheduling proposal rather than as a pure networking proposal.

The longer-context comparison matters even more. At 128K tokens, Ring-2.5-1T produces only 1.46 Gbps of KV throughput versus 47.82 Gbps for MiniMax-M2.5. That is the core asymmetry behind the investment thesis: if model architecture flattens KV growth enough, the economics of remote prefill start to look like a systems and scheduling problem rather than an impossible transport problem.

4. Attention Mechanisms and the Deployment Boundary

The paper's underlying argument is that attention design determines the network boundary of deployable serving architectures. Dense-attention models produce too much movable state too quickly, which ties PD disaggregation to tightly coupled RDMA fabrics. Hybrid stacks change that by reducing sequence-length-dependent KV emission, but they do not make bandwidth irrelevant. They make selective offload worth optimizing.

The strategic read-through is that FLOPs per dollar becomes an incomplete model metric once serving architectures become distributed. KV throughput per unit of useful compute becomes increasingly important because it determines how expensive it is to move state across network, memory, and storage hierarchies.

5. What PrfaaS Actually Changes in the Serving Stack

PrfaaS stands for Prefill-as-a-Service. The system keeps a local PD cluster for short requests and decode, but adds remote compute-dense prefill clusters that handle selected long-context prefills. After remote prefill completes, the resulting KVCache is transported over commodity Ethernet, VPC peering, or dedicated lines back to the local PD cluster, where decode continues. The system does not eliminate local prefill capacity. It augments local prefill with remote long-context compute and routes only the requests whose compute benefit dominates the transfer cost.

The hybrid prefix cache design is a non-trivial part of the proposal. Full-attention layers produce block-level KVCache that grows with token length and supports partial-prefix matching. Linear attention and recurrent-state components are effectively request-level or bounded-state artifacts that may require exact-length matching. The paper therefore separates linear or recurrent states and full-attention KVCache into distinct cache groups backed by a unified block pool, with reusable prefix-cache blocks separated from ephemeral transfer-cache blocks. That aligns with the broader vLLM direction that hybrid models require layer-specific slot allocation and prefix-cache rules.

The scheduler operates on 2 timescales. In the short term, it watches egress utilization and queue depth, then adjusts routing as congestion approaches. Under bandwidth scarcity, cache decisions become more local because moving state is expensive. Under bandwidth abundance, the system can fetch the best prefix state across clusters to avoid redundant compute. In the longer term, the scheduler adjusts the local PD cluster's prefill-to-decode ratio as traffic mix changes. That is why naive heterogeneous PD underperforms: specialization without dynamic routing creates stranded prefill or decode capacity and loses the utilization benefit.

6. Why the Scheduler Is the Product

The paper's most important non-obvious result is that faster prefill hardware alone does not create the best system. The throughput model formalizes three interacting bottlenecks: the remote PrfaaS prefill path, the local PD-prefill path, and the local decode path. End-to-end performance depends on how work is split across those stages, not on how fast any one stage looks in isolation.

This is why the comparison against naive heterogeneous PD matters so much. That baseline already uses faster prefill hardware, yet it still underperforms PrfaaS-PD on throughput because it lacks routing discipline and stage balancing. The strongest evidence for software capture in the paper is not the gain versus homogeneous PD. It is the gain versus unscheduled heterogeneity.

7. Case Study and Performance Interpretation

The case study uses an internal 1T-parameter hybrid model following the Kimi Linear architecture with a 3:1 KDA:MLA layer structure. The tested deployment uses 32 H200 GPUs in a remote PrfaaS cluster for long-context prefill and 64 H20 GPUs in a local PD cluster for local prefill plus decode, compared with a 96-H20 homogeneous PD baseline. The model is deployed at 8 GPUs per instance, the cross-cluster VPC provides approximately 100 Gbps of aggregate bandwidth, the local PD cluster has 800 Gbps RDMA interconnect per node, input lengths follow a truncated log-normal distribution with approximately 27K mean tokens, output length is fixed at 1,024 tokens, and the serving SLO is 40 tokens per second excluding speculative decoding.

The detailed prefill profile explains why the transport does not break the model. At 1K tokens, KVCache size is 190.8 MiB, prefill latency is 0.44 seconds, and KV throughput is 3.61 Gbps. At 32K tokens, KVCache is 701.3 MiB, prefill latency is 1.84 seconds, and KV throughput is 3.19 Gbps. At 128K tokens, KVCache is 2,316.3 MiB, prefill latency is 7.40 seconds, and KV throughput is 2.62 Gbps. KV throughput falls slightly at longer lengths because prefill time grows faster than cache size, which is exactly the asymmetry PrfaaS exploits.

The optimized configuration routes requests above 19.4K incremental uncached tokens to PrfaaS, which leads to 49.6% of requests being offloaded and an offloaded mean length of approximately 44K tokens. Despite offloading roughly half the request count, aggregate PrfaaS egress remains only approximately 13 Gbps, or about 0.4 Gbps per H200 GPU and approximately 3.25 Gbps per 8-GPU PrfaaS instance. That is the paper's strongest feasibility result. It suggests that, at least for hybrid models and selective routing, modern NIC capacity is not the first-order bottleneck at modest cluster scale.

The equal-cost framing also matters. A 54% throughput uplift implies roughly 35% lower resource-seconds per request before hardware-price normalization, but the paper says the equal-cost throughput gain is closer to 15%, which implies approximately 13% lower resource cost per served request on that basis. That is a more realistic investment read-through. The point is not that the architecture is a 54% direct capex deflator. The point is that phase specialization can improve utilization and lower long-context serving cost, especially if prefill-specialized accelerators are cheaper than high-end general-purpose HBM GPUs.

The bandwidth result should be interpreted carefully. Approximately 13 Gbps of average egress is a meaningful proof point that a carefully scheduled hybrid-model deployment can fit within modest Ethernet budgets at small cluster scale. It is not proof that arbitrary long-haul or multi-region inference routing becomes economically trivial. The paper proves a favorable metro or regional operating regime under disciplined routing, not a universal deployment template.

The same caution applies to the headline performance uplift. The cleanest strategic signal is not simply that PrfaaS beats a homogeneous baseline. It is that PrfaaS also beats naive heterogeneity. That is the empirical reason the scheduler and cache manager should be treated as central product assets rather than as implementation details.

8. Direct Paper Evidence Versus Strategic Read-Through

The cleanest way to read the paper is to separate direct empirical proof from strategic inference. The case study is strong enough to establish direction of travel, but it is still a narrow internal-model deployment with modeled optimization. That means some conclusions belong in the paper-evidence bucket, while others belong in the strategic read-through bucket.

That framing keeps the note analytically disciplined. The paper is more impressive as a systems-framing document than as a fully production-validated long-haul deployment proof. It is best read as a high-quality direction-of-travel paper with a convincing quantitative case study, not as definitive proof that arbitrary cross-region inference routing is now a commodity practice.

9. Where PrfaaS-Style Deployment Fits Today, and Where It Does Not Yet Fit

The current report already argues that the paper proves a favorable metro or regional regime rather than a universal production template. The most useful next upgrade is to state that fit boundary more explicitly. The paper is strongest when hybrid-attention architectures, long-context requests, disciplined routing, and reasonably predictable inter-cluster links all show up together. It is much weaker as evidence for arbitrary global routing, short-prompt chat traffic, or operational environments where security and network variability dominate the savings.

10. What Is Still Unproven

The live report already separates direct paper evidence from strategic read-through. A dedicated open-questions table makes the remaining underwriting gaps even clearer. That matters because the paper is highly useful directionally, but it is still a bounded systems proof rather than a universal production verdict.

11. Operator Prerequisites and Failure Modes

The paper is strongest as a systems-design document because it makes clear that lower KV throughput is necessary but not sufficient. Cross-cluster prefill only works when the operator can classify requests, route them selectively, preserve cache-state visibility, and keep producer and consumer stages balanced. That means the deployment hurdle is not just model choice or link budget. It is control-plane maturity.

This is why the winning product is unlikely to be the fastest prefill chip by itself. The real product is the serving system that decides when to move state, how much state to move, and how to preserve decode saturation without creating egress, recovery, or compliance pain somewhere else in the stack.

12. Workload Characteristics and Expected Architecture Payoff

A second useful precision upgrade is to map the paper's logic directly to workload shape. The architecture is most attractive when prompt-side compute is heavy, cache-hit rates are modest, and operators can tolerate some control-plane complexity in exchange for better pool utilization. It is least attractive where low latency, high reuse, or governance friction dominate.

For investors, this table matters because it narrows the read-through. The point is not that every inference workload becomes portable across clusters. The point is that certain high-context workloads may become portable enough to change where value accrues across compute, memory, transport, and orchestration layers.

13. Investment Implications by Stack Layer

The accelerator read-through is bifurcated rather than purely bullish or bearish. General-purpose GPU demand remains strong because both prefill and decode still need accelerators, but the optimal mix becomes less uniform. The prefill phase benefits from dense compute and can tolerate lower memory bandwidth once KVCache-efficient architectures reduce state emission. Decode remains sensitive to memory bandwidth, batching, and low-latency token generation. That supports separate prefill and decode silicon pools, including NVIDIA's own Rubin CPX-style context-phase accelerators, HBM-rich GPUs for decode, and non-GPU inference accelerators optimized around deterministic memory bandwidth.

For NVIDIA, the paper is strategically supportive of the company's disaggregated inference roadmap rather than clearly disruptive in isolation. The direction aligns with Rubin CPX, Vera Rubin NVL144 CPX, Quantum-X800, Spectrum-X Ethernet, ConnectX SuperNICs, and Dynamo. The real risk to NVIDIA is not that PrfaaS reduces GPU demand. It is that the architecture legitimizes more open heterogeneous serving, where custom ASICs, LPUs, or lower-cost prefill accelerators can enter the serving stack if scheduling and cache abstractions become less CUDA-specific. That is directionally favorable for AMD, hyperscaler ASICs, merchant inference silicon, and any vendor that can offer compelling compute per dollar or per watt without having to replace the full NVIDIA decode and scale-up stack.

The memory read-through is more nuanced. HBM remains mission-critical because decode is still memory-bandwidth constrained and long output chains can keep HBM demand structurally high. But the paper weakens the naive extrapolation that longer context automatically means proportionally more HBM everywhere. Hybrid attention, KV quantization, KV compression, smaller active expert sets, and GDDR-based prefill silicon all reduce HBM intensity in parts of the inference fleet. The prefill side is where the headwind is strongest. NVIDIA's own Rubin CPX uses 128 GB of GDDR7 rather than HBM, which is a concrete signal that context-phase compute can be optimized around arithmetic throughput and lower memory-system cost.

That does not make the memory story bearish in aggregate. DRAM is a beneficiary because host memory becomes a more meaningful warm tier for prefix blocks, metadata indexes, staging buffers, routing state, and cache-management structures. SSDs are also positive beneficiaries because enterprise NVMe can hold reusable prefixes, compressed KV blobs, and spillover from DRAM, especially in agentic, coding, RAG, and long-session workloads with substantial prefix reuse. HDD exposure is far more indirect and tied to cold AI data growth rather than to the real-time serving path.

Networking and optical infrastructure are among the clearest beneficiaries. The paper's 32-H200 case uses only 13 Gbps on a 100 Gbps link, so the point is not that every inference request suddenly needs extreme bandwidth. The point is that Ethernet and DCI now sit directly in the inference data path. At larger scale, the aggregate opportunity becomes meaningful. The paper cites approximately 3.8 Tbps of egress for a 512-H200 prefill cluster using dense MiniMax-M2.5 assumptions, approximately 2.1 Tbps for Qwen3, roughly 170 Gbps for hybrid Ring-2.5-1T, and approximately 1.8 Tbps for a 10,000-GPU datacenter under the cited hybrid operating assumptions. That is supportive of AI Ethernet, coherent DCI, CPO, DSPs, retimers, SmartNICs, and telemetry software, which is why names aligned with Broadcom, Marvell, NVIDIA Networking, Arista, Coherent, and Lumentum all screen positively to the direction of travel.

The power and cloud read-through is also constructive. PrfaaS is power-efficiency positive per served long-context request because it reduces redundant prefill work and lowers stranded prefill or decode capacity, but it is not necessarily power-demand negative in aggregate because lower cost and lower latency can expand usage. For clouds and neoclouds, the architecture is especially valuable where accelerator pools are geographically fragmented, RDMA domains are constrained, or power availability is uneven across regions. A PrfaaS-like design effectively creates an internal market for prefill capacity, where long-context requests can be routed to the cheapest qualified cluster subject to bandwidth, latency, cache, and data-residency constraints.

A broader implication is that model-system co-design becomes increasingly investable. As serving architectures become distributed, FLOPs per dollar is no longer a sufficient metric. KV throughput per unit of useful compute increasingly determines how expensive it is to move state across network, memory, and storage hierarchies. That raises the strategic value of model designs that emit less movable state even when raw compute demand does not fall proportionally.

14. Memory, Storage, and Interconnect Hierarchy

HBM remains mission-critical, but the paper weakens the simple linear assumption that longer context automatically means proportionally more HBM everywhere in the serving fleet. Dense attention makes KVCache scale aggressively with sequence length, forcing memory capacity and bandwidth upward as context grows. Hybrid attention changes the slope. If only a minority of layers produce length-dependent KVCache, then the active HBM requirement per long-context session can fall materially. That matters because it reduces the probability that every incremental inference accelerator must be an HBM-maximized device, especially on the prefill side.

The negative HBM read-through is strongest for prefill accelerators rather than for the inference fleet as a whole. The paper explicitly points to architectures such as linear attention, sliding-window attention, MLA, KV quantization, and KV compression as mechanisms that reduce cache bytes per token. NVIDIA's own Rubin CPX choice of 128 GB of GDDR7 rather than HBM is directionally consistent with that view. It suggests that context-phase compute can be optimized around arithmetic throughput and lower memory cost where decode-like bandwidth is less central. By contrast, decode remains memory-bandwidth constrained and likely continues to consume substantial HBM or SRAM-equivalent bandwidth as output-token volume, agentic workflows, and multi-step reasoning chains expand.

KV compression compounds the memory-hierarchy shift. The source cites KIVI, which reports 2-bit KVCache quantization that cuts peak memory by 2.6x, enables up to 4x larger batch size, and improves throughput by 2.35x to 3.47x on real inference workloads. CacheGen is cited for 3.5x to 4.3x KVCache size reduction and 3.2x to 3.7x lower context fetching plus processing delay. These techniques are complementary to hybrid architecture rather than substitutes for it. If broadly adopted, they further reduce HBM capacity pressure and cross-datacenter transfer volume, while potentially increasing accelerator utilization and total workloads served.

DRAM is a direct beneficiary of KV-centric serving because it becomes the operational buffer between expensive accelerator memory and slower storage tiers. Mooncake already argued that underutilized CPU, DRAM, SSD, and NIC resources can be combined into a disaggregated KVCache pool, and PrfaaS extends that logic across clusters. Host DRAM can hold warm prefix-cache blocks, metadata indexes, transport queues, staging buffers, and request-level state for hybrid models whose reuse rules differ across layers. The caveat is that DRAM is a warm tier, not a replacement for HBM in the decode inner loop. Its value is highest where cache reuse is economically important and where the system can avoid recomputing long prefixes.

SSDs benefit in a similar but more workload-dependent way. They are not fast enough for the hottest decode loop, but they are useful for reusable prefixes, RAG context, compressed KV blobs, session persistence, and spillover from DRAM. That is supportive of enterprise NVMe, high random-read performance, endurance, SPDK-style software, and DPU-mediated storage access. HDDs remain relevant for cold model checkpoints, logs, training data, synthetic data, observability archives, and compliance retention, but the paper does not create a direct HDD demand vector comparable to SSD, DRAM, NIC, or optical demand. The HDD read-through is therefore second-order and tied to overall AI data growth rather than to the serving-path mechanics of PrfaaS.

Server platform design also becomes more heterogeneous. Local PD clusters still need HBM, low-latency decode scheduling, and strong local RDMA. Remote PrfaaS clusters need high compute throughput, sufficient memory to stage weights and intermediate state, efficient egress NICs, and robust host orchestration, but they may not require the same scale-up topology as monolithic training clusters. That broadens the inference server bill of materials beyond accelerators alone and increases the relevance of CPUs, DRAM channels, SSDs, PCIe and CXL fabrics, retimers, switches, liquid cooling, and control-plane software.

15. Networking, Power, and Cloud Architecture

Networking is one of the clearest beneficiaries, but not because the paper implies absurd per-request bandwidth needs at small scale. The stronger point is qualitative. Ethernet and DCI become part of the real-time inference serving path. Cross-datacenter links are no longer used only for replication, backup, storage, user ingress, or offline data movement. They are now carrying model state that a live request needs in order to continue. That elevates the importance of bandwidth predictability, congestion control, packet loss handling, telemetry, encryption, and failure-aware routing.

That backdrop is why the paper is supportive of AI Ethernet and optical DCI suppliers even though the benchmarked cluster itself is modest in bandwidth terms. The source explicitly ties the direction to commodity Ethernet, VPC peering, and dedicated-line transport, while also citing vendor roadmaps from NVIDIA Spectrum-X and Spectrum-XGS, Arista AI networking systems, Marvell COLORZ 800, and Broadcom's OFC 2026 portfolio around 102.4T switching, 400G-per-lane optical DSPs, 1.6T transceiver enablement, 800G AI NICs, and 200G-per-lane retimers. The deeper implication is that optics and switching are moving from background infrastructure line items into inference-path enablers.

Power efficiency is directionally positive per served long-context request because the architecture reduces redundant prefill compute, reduces stranded specialized hardware, and improves utilization. But it is not necessarily negative for aggregate power demand because lower serving cost and lower latency can expand usage. The source frames this clearly as an elasticity problem. It also cites IEA projections that global data-center electricity generation rises from 460 TWh in 2024 to over 1,000 TWh in 2030 and 1,300 TWh in 2035 in the Base Case, with a Lift-Off Case approaching 2,000 TWh by 2035. PrfaaS improves the way capacity is used, but it does not eliminate the macro power constraint. It mainly changes where and how those constraints bind.

The cloud-architecture read-through is important. Hyperscalers and neoclouds can potentially use stranded or regionally isolated accelerators for prefill while keeping decode closer to end users or latency-sensitive demand. That matters because accelerator procurement and datacenter construction rarely produce perfectly balanced fleets by region. A PrfaaS-style design creates an internal market for prefill capacity, where long-context requests can be routed to the cheapest available qualified cluster subject to bandwidth, latency, cache, and data-residency limits. The architecture also points toward a future context-compute product category, where prompt ingestion, document processing, video-context encoding, and long-prefix computation can be priced or charged separately from decode.

The generative AI ecosystem implication is that inference, not training alone, increasingly defines the next layer of infrastructure competition. Training clusters are dominated by dense accelerator scale-up and scale-out fabrics. Inference clusters are becoming workload-specific, cache-aware, geographically distributed, and increasingly governed by routing logic. Long-context inference, multimodal video, RAG, coding agents, and multi-turn agentic systems all raise the value of prefill acceleration and prefix reuse. That makes KV throughput, cache portability, and system-level schedulability more important model design criteria than the old industry habit of focusing only on quality, FLOPs, and raw memory footprint.

16. Deployment Extrapolation: Security, Data Governance, and Operational Complexity

This section goes beyond the paper's direct empirical proof and addresses the deployment consequences of mobile KV state in production environments. The paper is focused primarily on bandwidth, scheduling, and architecture. It does not benchmark cryptographic overhead, privacy leakage risk, or compliance controls. Even so, those issues become central once cross-cluster KVCache movement enters the real serving path.

One of the most underappreciated points in the paper is that KVCache is sensitive data. It is a derived representation of user prompts, documents, tools, and conversational state. Moving KVCache across datacenters therefore creates security, privacy, and compliance obligations that look much closer to moving user data than to moving anonymous model metadata. Encryption in transit, access controls, tenant isolation, audit logging, regional data-residency controls, and cache-eviction discipline all become core product requirements rather than afterthoughts.

• A global KVCache manager creates a new control-plane surface that stores metadata about prompts, prefixes, cache locations, and reuse opportunities.
• Cross-tenant leakage, cache poisoning, stale-cache bugs, and model-version mismatch become real failure modes once cache mobility spans clusters.
• The scheduler has to jointly manage routing thresholds, bandwidth availability, queue depth, cache affinity, model versions, and cluster health.
• The architecture is therefore more favorable to hyperscalers and full-stack inference vendors than to smaller operators relying on thin model-serving wrappers.

This is why the paper should not be misread as a lightweight bandwidth optimization. It effectively argues that inference is becoming a distributed systems problem with capital-allocation characteristics. The new moat is not only model quality or accelerator ownership. It is also whether the platform can safely and efficiently decide which requests to export, where to source reusable context, and how to preserve latency and security under fluctuating network conditions.

17. Risks and Disconfirming Evidence

The empirical base is still narrow. The headline result comes from an internal 1T-parameter hybrid model, an internally profiled serving stack, a specific H200 and H20 pairing, a 100 Gbps VPC network, a fixed 1,024-token output length, a 40 tokens-per-second SLO, and a truncated log-normal input distribution with approximately 27K mean tokens. The results are produced through an analytical throughput model calibrated with profiling data rather than through a fully disclosed production trace with real burst behavior, failures, egress pricing, cache churn, and multi-tenant security overhead.

• Cross-datacenter should not be overread as arbitrary long-haul global routing. The architecture is most compelling for metro and regional clusters, VPC-connected campuses, or dedicated-fiber environments with disciplined latency and packet-loss behavior.
• The benefit depends on a sufficiently large mix of long-context requests. Very short prompts, low-latency chat workloads, and low-cache-hit traffic may not generate enough compute-side benefit to justify remote prefill.
• Model generalization is a real open question. If frontier models do not adopt hybrid-attention or KV-efficient architectures deeply enough, the transport economics become less attractive.
• If KV compression becomes extremely effective, the networking problem gets easier, but HBM, DRAM, and SSD content per served request could decline faster than naive demand models expect.
• Operational complexity is high. Misestimated request lengths, queue imbalance, stale cache state, or poor congestion control can erase the utilization gain or create user-visible latency spikes.

The cleanest disconfirming path is that distributed serving never becomes a mainstream requirement because leading models stay dense enough, short-form enough, or local enough that the software and governance burden outweighs the efficiency benefit. A second disconfirming path is that dense-attention or decode-oriented hardware vendors remain dominant because the marginal cost of bandwidth and orchestration stays higher than the cost of simply colocating more capability.

18. Catalysts and Watchlist

The most important thing to watch is whether the industry begins optimizing for KV throughput as a first-class design metric. That would confirm the paper's deeper argument that model architecture and system architecture are converging. A model that uses slightly more compute but emits materially less transferable state can be more valuable in distributed serving because it lowers burdens across networking, HBM, DRAM, SSD, and orchestration layers at the same time.