throughput: send-side UDP GSO on the poll path (lever 4a) #55

Merged

vxfemboy merged 10 commits from feat/throughput-gso-4a into main

2026-07-12 04:23:36 +00:00

vxfemboy commented

2026-07-12 04:09:53 +00:00

(Migrated from github.com)

What

Wires send-side UDP GSO (UDP_SEGMENT) into the default poll data-plane loop. run_poll's flush_tx now coalesces same-destination, same-length, distinct-FEC-object datagrams into one sendmsg, cutting the per-datagram kernel send-stack cost that real-hardware profiling pinned as the single-core bottleneck. Lever 4a of a measure-gated campaign (4a GSO → 4b recv GRO → 4c MTU).

Why

After the merged FEC (#51), AEAD (#52), and batched-I/O (#54) levers, kernel-stack sampling of yipd under load on a real target VPS showed __sys_sendmmsg as the single largest CPU slice (~40% of samples) — the dominant remaining single-core cost is the UDP send path, not crypto/FEC. Batched I/O cut syscall count; each datagram still traverses the full kernel UDP stack. GSO amortizes that stack traversal across a batch.

How

crates/yip-io/src/gso.rs (new): the one FEC-safety rule, shared by both drivers — can_coalesce / partition_fate_safe / max_gso_run_len. A GSO run is same-dst, same-length, pairwise-distinct fate (FEC object id). The UringDriver GSO path is refactored to delegate to it (behavior-preserving), so the correctness-critical rule lives in exactly one place.
crates/yip-io/src/poll.rs: send_gso_payload (one sendmsg + UDP_SEGMENT cmsg; the only new unsafe), send_gso/send_gso_indexed wrappers, and a rewritten flush_tx that partitions its egress batch into fate-safe runs, GSO-sends each run of ≥2, plain-sends singletons, and — after latching a per-run_poll "GSO unavailable" flag on EIO/EINVAL — falls back to send_mmsg. GsoScratch is threaded through drain_udp/drain_tun.

FEC loss-independence preserved: the distinct-fate rule guarantees a GSO skb never carries two symbols of one FEC object, so even a whole-skb drop costs each object at most one symbol — exactly recoverable. Wire-identical, latency-neutral, opportunistic (coalesces only what a burst already queued; a lone datagram sends plain). unsafe stays confined to yip-io; yipd remains #![forbid(unsafe_code)].

Results

Decision-gate spike (throwaway, on a real EPYC virtio box): UDP_SEGMENT vs plain sendmmsg = 2.6× datagrams per CPU-second (462k vs 181k) — well above the 1.3× gate.

Real-hardware A/B (two 1-core AMD EPYC / 967 MB virtio VPSes, same session, baseline = main e030a39 #54 no-GSO vs this branch):

Test	Baseline peak	4a (GSO) peak	at equal single-core CPU
Direct (core-shared)	195 Mbit/s	255	+31%
Isolated (yipd owns core, NAT chain)	157 Mbit/s	196	+25%

The end-to-end gain is smaller than the spike's 2.6× because send-side GSO only amortizes the transmit stack — recv/TUN/conntrack/IRQ don't benefit. Full tables in crates/yip-bench/RESULTS.md.

Correctness: cargo test --workspace 0 failures; yip-io 46/46; clippy --all-targets -D warnings clean. netns ping_across_yipd_tunnel, ..._under_loss (10% netem — FEC still recovers with GSO), and arq_recovers_bulk_loss all PASS under both the poll and uring drivers.

Review

Executed via subagent-driven-development: 6 tasks, each with a dedicated spec+quality review; two review-caught findings fixed (a release-compiled-out control-buffer guard → real Err; the GSO latch now honored within a burst, not just across calls). Final whole-branch verdict: READY WITH FOLLOW-UPS, no Critical/Important.

Non-blocking follow-ups

flush_tx now runs tx.clear() on the Err path too (harmless — every caller tears down the loop on Err).
A f as u8 cast in new test-only code.
uring.rs:1852 comment still names two constants that moved to gso.rs.
Pre-existing doc-debt (out of scope here): README/CLAUDE.md still say "adaptive RaptorQ FEC" after the Reed–Solomon swap in #50 — needs its own doc-sweep PR.

Design docs

Spec: docs/superpowers/specs/2026-07-11-throughput-gso-send-design.md
Plan: docs/superpowers/plans/2026-07-11-throughput-gso-send.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01RVP6NnbDMAg1iTsMMTfL86

## What Wires **send-side UDP GSO (`UDP_SEGMENT`)** into the default poll data-plane loop. `run_poll`'s `flush_tx` now coalesces same-destination, same-length, distinct-FEC-object datagrams into one `sendmsg`, cutting the per-datagram kernel send-stack cost that real-hardware profiling pinned as the single-core bottleneck. Lever **4a** of a measure-gated campaign (4a GSO → 4b recv GRO → 4c MTU). ## Why After the merged FEC (#51), AEAD (#52), and batched-I/O (#54) levers, kernel-stack sampling of `yipd` under load on a real target VPS showed **`__sys_sendmmsg` as the single largest CPU slice** (~40% of samples) — the dominant remaining single-core cost is the UDP *send* path, not crypto/FEC. Batched I/O cut syscall *count*; each datagram still traverses the full kernel UDP stack. GSO amortizes that stack traversal across a batch. ## How - **`crates/yip-io/src/gso.rs` (new):** the one FEC-safety rule, shared by both drivers — `can_coalesce` / `partition_fate_safe` / `max_gso_run_len`. A GSO run is same-`dst`, same-length, **pairwise-distinct `fate`** (FEC object id). The `UringDriver` GSO path is refactored to delegate to it (behavior-preserving), so the correctness-critical rule lives in exactly one place. - **`crates/yip-io/src/poll.rs`:** `send_gso_payload` (one `sendmsg` + `UDP_SEGMENT` cmsg; the only new `unsafe`), `send_gso`/`send_gso_indexed` wrappers, and a rewritten `flush_tx` that partitions its egress batch into fate-safe runs, GSO-sends each run of ≥2, plain-sends singletons, and — after latching a per-`run_poll` "GSO unavailable" flag on `EIO`/`EINVAL` — falls back to `send_mmsg`. `GsoScratch` is threaded through `drain_udp`/`drain_tun`. **FEC loss-independence preserved:** the distinct-`fate` rule guarantees a GSO skb never carries two symbols of one FEC object, so even a whole-skb drop costs each object at most one symbol — exactly recoverable. **Wire-identical, latency-neutral, opportunistic** (coalesces only what a burst already queued; a lone datagram sends plain). `unsafe` stays confined to `yip-io`; `yipd` remains `#![forbid(unsafe_code)]`. ## Results **Decision-gate spike** (throwaway, on a real EPYC virtio box): `UDP_SEGMENT` vs plain `sendmmsg` = **2.6× datagrams per CPU-second** (462k vs 181k) — well above the 1.3× gate. **Real-hardware A/B** (two 1-core AMD EPYC / 967 MB virtio VPSes, same session, baseline = main `e030a39` #54 no-GSO vs this branch): | Test | Baseline peak | 4a (GSO) peak | at equal single-core CPU | |---|---|---|---| | Direct (core-shared) | 195 Mbit/s | **255** | **+31%** | | Isolated (yipd owns core, NAT chain) | 157 Mbit/s | **196** | **+25%** | The end-to-end gain is smaller than the spike's 2.6× because send-side GSO only amortizes the transmit stack — recv/TUN/conntrack/IRQ don't benefit. Full tables in `crates/yip-bench/RESULTS.md`. **Correctness:** `cargo test --workspace` 0 failures; `yip-io` 46/46; clippy `--all-targets -D warnings` clean. netns `ping_across_yipd_tunnel`, **`..._under_loss` (10% netem — FEC still recovers with GSO)**, and `arq_recovers_bulk_loss` all PASS under **both** the poll and uring drivers. ## Review Executed via subagent-driven-development: 6 tasks, each with a dedicated spec+quality review; two review-caught findings fixed (a release-compiled-out control-buffer guard → real `Err`; the GSO latch now honored within a burst, not just across calls). Final whole-branch verdict: **READY WITH FOLLOW-UPS**, no Critical/Important. ## Non-blocking follow-ups - `flush_tx` now runs `tx.clear()` on the `Err` path too (harmless — every caller tears down the loop on `Err`). - A `f as u8` cast in new **test-only** code. - `uring.rs:1852` comment still names two constants that moved to `gso.rs`. - Pre-existing doc-debt (out of scope here): README/CLAUDE.md still say "adaptive RaptorQ FEC" after the Reed–Solomon swap in #50 — needs its own doc-sweep PR. ## Design docs - Spec: `docs/superpowers/specs/2026-07-11-throughput-gso-send-design.md` - Plan: `docs/superpowers/plans/2026-07-11-throughput-gso-send.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RVP6NnbDMAg1iTsMMTfL86