RDMA: The Reason Remote Storage Feels Like It's Plugged Into Your Motherboard

Your SSD does 4K reads in 80 microseconds. Your network stack just added 150 µs. That was the whole problem — until RDMA removed the network from the latency math.

The bottleneck nobody talks about

For most of the 2010s, the rule was simple: if you wanted fast storage, you kept it local. NVMe SSDs kept getting faster — 4K random reads in the tens of microseconds — while the network path stayed stubbornly expensive. Every byte pushed to a remote drive had to survive the kernel network stack: socket buffers, interrupts, context switches, multiple memory copies between user space and the kernel. The network didn't just add latency; it burned CPU — 2 to 4 cores per million IOPS in software, just to move data around.

Remote Direct Memory Access (RDMA) is what broke this trade-off. It lets one computer read and write another computer's memory directly, with no involvement from either machine's CPU or operating system on the data path.

What RDMA actually does (three tricks)

1. Zero-copy. Data moves from application buffer to application buffer. No kernel socket buffers, no user-space-to-kernel copies. The bytes land exactly where the application wants them, directly.

2. Kernel bypass. The NIC itself — an RDMA-capable NIC (RNIC) — runs the transport in hardware. No syscalls, no interrupts, no context switches on the data path. Your CPU never touches the packet.

3. One-sided operations. This is the "wait, what?" moment. RDMA READ and RDMA WRITE execute against remote memory without the remote CPU doing anything. No receive call, no wakeup, no cycle spent. The data just appears in the remote application's memory.

The programming model is small: register (pin) memory so the NIC can DMA into it, create queue pairs (a send queue plus a receive queue, managed in hardware), post work requests, poll for completions. Here's the shape of it in C using the standard libibverbs API — simplified, but this is genuinely all there is to it:

/* 1. Register (pin) the buffer so the NIC can DMA into it */
struct ibv_mr *mr = ibv_reg_mr(pd, buf, len,
    IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE);

/* 2. Post an RDMA WRITE: bytes go straight into the
      remote machine's memory. The remote CPU never wakes up. */
struct ibv_send_wr wr = {
    .opcode   = IBV_WR_RDMA_WRITE,
    .sg_list  = &(struct ibv_sge){ .addr   = (uintptr_t)buf,
                                   .length = len,
                                   .lkey   = mr->lkey },
    .num_sge  = 1,
    .wr.rdma  = { .remote_addr = remote_addr, .rkey = rkey },
};
ibv_post_send(qp, &wr, NULL);

/* 3. Poll for completion — no interrupts involved */
while (ibv_poll_cq(cq, 1, &wc) == 0) { /* busy-wait, that's the point */ }

That busy-wait at the end is not a bug — it's the philosophy. Interrupts are latency. Polling is how you buy microseconds.

Three flavors of the same idea

	InfiniBand	RoCE(RDMA over Converged Ethernet) v1	RoCE v2	iWARP
Runs on	Native IB fabric	Ethernet (L2)	Ethernet over UDP/IP (L3)	TCP/IP
Routable across subnets	No	No	Yes	Yes
Latency	Lowest	Very low	Very low	Low
Fabric requirements	IB switches	Lossless Ethernet	Lossless Ethernet (PFC + ECN)	Any IP network
Where you see it	HPC, AI training	Legacy data centers	Cloud data centers	Enterprise Ethernet

InfiniBand is the original: a purpose-built fabric with the lowest latency, still the king of large AI training clusters. RoCE v2 is what most cloud data centers actually run — RDMA semantics over UDP/IP, routable, on standard Ethernet. iWARP is the pragmatist's choice: RDMA layered on TCP, so it works on any IP network with no special fabric configuration, at a small latency cost.

Where it matters most: NVMe over Fabrics

The killer app for RDMA in storage is NVMe over Fabrics (NVMe-oF). NVMe-oF lets a host talk to a remote SSD using the same command set as a local drive — the drive looks local, but it lives across the network. The transport you pick decides the tax:

Metric	NVMe / RDMA (RoCE v2)	NVMe / TCP	NVMe / InfiniBand
Fabric-added latency (above the SSD)	~5–20 µs	~50–150 µs	~3–10 µs
4K random read IOPS (one 100 GbE port)	2.5–3.5 M	1.5–2.5 M	3–4 M
4K read latency at queue depth 1	~25–40 µs	~80–150 µs	~20–35 µs
Host CPU per 1M IOPS	~0.5 core	~2–4 cores (software)	~0.5 core

Here's why those numbers decide architectures: a mainstream enterprise NAND SSD does 4K random reads in roughly 70–120 µs. Add a 100 µs TCP tax and you've nearly doubled your latency — and doubled your tail. Add a 10 µs RDMA tax and it's noise. One published comparison put it starkly: local NVMe 4K reads at 15 µs, NVMe-oF over RDMA at 82 µs, NVMe-oF over TCP at 382 µs.

The CPU column is the sleeper story. At millions of IOPS, TCP's software stack eats whole cores just moving data. RDMA offloads the transport to the NIC, so roughly half a core does work that costs TCP two to four cores. When you're disaggregating storage across a fleet, that CPU delta is real money.

RDMA in the wild

This isn't lab technology. It's the plumbing under production systems you already use:

• SPDK (Storage Performance Development Kit): the open-source toolkit that moved NVMe drivers into user space with poll-mode, lock-free I/O — and ships a spec-compliant NVMe-oF target and host with RDMA transport. It's the reference implementation for squeezing line rate out of NVMe-oF.
• Ceph: ships an RDMA messenger so OSD replication traffic runs at near wire speed with minimal CPU — exactly the kind of east-west replication traffic that used to saturate storage nodes.
• The hyperscalers: Microsoft Azure and Google Cloud run RDMA for internal storage traffic. GCP's A3 instances ship NVIDIA ConnectX-7 NICs with RoCE v2 for exactly this reason.
• Enterprise arrays: Pure Storage's DirectFlash Fabric replaced iSCSI with NVMe-oF over RoCE and reported roughly 50% lower latency than iSCSI and about 20% better performance than Fibre Channel.
• AI infrastructure: GPUDirect Storage moves data from NVMe-oF/RDMA storage straight into GPU memory, bypassing the CPU entirely — the same zero-copy philosophy, extended to the accelerator. Checkpoint-heavy training loops live or die on this path.

Why AI made this urgent (and why this blog exists)

Here's the honest answer to "why write about RDMA now": AI turned storage latency from a database problem into a GPU-economics problem.

A modern training cluster is thousands of GPUs costing dollars per hour each. Every minute they sit idle waiting on I/O is money burned at datacenter scale. And AI workloads are uniquely brutal on storage:

Checkpoints arrive as bursts. Training periodically writes multi-terabyte model snapshots — and every rank writes at once. Meta's Llama 3 training storage (Tectonic: 240 PB across 7,500 SSD servers, 2 TB/s sustained, 7 TB/s peak) called "highly bursty checkpoint writes that saturate the storage fabric for short durations" a major challenge. When a checkpoint write stalls, training stalls with it — thousands of GPUs waiting on a storage fabric.

Data loading starves GPUs quietly. Training reads continuously: many small files, random access, CPU-bound decoding. If storage can't feed the GPUs, you don't get an error — you get 30% utilization on hardware you're paying 100% for. One published Alluxio case study took a fleet from 30% to 90% GPU utilization largely by fixing the data path.

Inference moved the problem. Disaggregated serving splits prefill and decode across workers, and the KV cache has to move between them — fast. Together AI's cache-aware prefill-decode disaggregation runs its KV-cache hierarchy over RDMA for exactly this reason. And cold starts are a storage problem too: pulling a 400 GB checkpoint over a slow link makes every scale-up take minutes.

This is where RDMA stops being a storage-geek topic and becomes infrastructure strategy. GPUDirect Storage already moves data between GPU memory and NVMe-oF storage over RDMA, cutting the CPU and its bounce buffers out of the path. NVIDIA's 2025 SCADA work goes further: letting GPUs initiate and control storage I/O directly, which beats even GPUDirect for the small-block reads inference needs. The direction is unmistakable — the storage path is being rebuilt around the same zero-copy, kernel-bypass ideas this post describes.

And that's why this blog exists. The industry conversation about AI infrastructure is 90% GPUs and collectives — NCCL, InfiniBand, topology. The storage half of the cluster gets a fraction of the attention, yet it's where checkpoint stalls, data-loader starvation, and cold-start latency actually live. If you're building or buying storage for AI workloads in 2026, the RDMA-vs-TCP decision is one of the highest-leverage choices you'll make — and it's the one nobody wrote the plain-English guide for. Until now.

The catch (because there's always one)

RDMA's performance is real; so is its operations tax.

RoCE wants a lossless fabric. To hit those latency numbers, RoCE v2 expects Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) tuned on every switch port in the path. Get it wrong and you get head-of-line blocking or congestion collapse — RDMA is famously unforgiving of congestion, degrading dramatically instead of gracefully. "Just turn on RoCE" is never just turning on RoCE; it's a conversation with your network team.

Complexity has a price. Special NICs, fabric tuning, RDMA-aware monitoring — all of it is operational surface area. NVMe/TCP exists precisely because sometimes you don't want any of that: it runs on any network, needs no special hardware, and degrades gracefully under congestion instead of falling over. If your workload's latency budget is measured in milliseconds, TCP is the right answer and RDMA is expensive theater.

Know which game you're playing. The honest rule of thumb: if your SSD's latency budget is measured in tens of microseconds, your fabric has to be too — and only RDMA gets you there today. If it's measured in milliseconds, save yourself the fabric surgery.

What's next

• Ultra Ethernet: the Ultra Ethernet Consortium (founded 2023 — AMD, Broadcom, Cisco, Intel, Meta, Microsoft and others) is building a purpose-built Ethernet transport designed for lossy networks, multipath operation, and massive scale — explicitly fixing RoCE's lossless-fabric requirement. Its storage layer builds on NVMe-oF and RDMA rather than replacing them.

• DPUs / SmartNICs: BlueField, Pensando, Intel IPU — offloading the storage data path (including NVMe/TCP) onto the NIC itself, so the host CPU does even less.

• CXL: Compute Express Link 3.0 brings rack-scale memory pooling and sharing — a different answer to "remote memory access" that may complement or compete with RDMA. The industry hasn't decided yet, which makes it interesting.

The bottom line

RDMA didn't make networks faster. It made the network disappear from the storage path — no CPU, no kernel, no copies, and a remote SSD that behaves like it's on your PCIe bus. The price is operational: lossless fabrics, tuned congestion control, and hardware that speaks the protocol. Pay it when microseconds matter. Skip it when they don't.

If you've run RoCE in production — or deliberately chosen TCP instead — I'd love to hear how that trade-off played out in the comments.