throughput 4c: MTU-aware packetization to cut packet count (fewer per-packet syscalls / TUN / SipHash) #59

New issue

Open

opened 2026-07-12 23:50:06 +00:00 by vxfemboy · 1 comment

vxfemboy commented

2026-07-12 23:50:06 +00:00

(Migrated from github.com)

Lever 4c of the single-core throughput campaign (4a send GSO ✓ #55, 4b TUN offload ✓ #56).

The post-4a/4b profiling showed the single-core ceiling is per-packet cost (kernel UDP send sendmsg ~23%, TUN write ~20%, SipHash ~9% — all charged per packet). Raising the tunnel MTU cuts the packet count for the same bytes, so it reduces every per-packet cost proportionally — the cheapest, most broadly-applicable remaining lever.

Scope

Make the tunnel MTU path-aware rather than a fixed conservative value; probe (PMTUD / PLPMTUD-style) and adapt, falling back cleanly to 1500-safe where the path can't carry more.
Interacts with FEC symbol sizing and the outer-UDP overhead budget — needs its own design (spec).

Note

Jumbo frames only help where the whole path supports them; 1500 stays the internet-safe default. The win is largest on controlled/LAN paths.
Related parked idea: inner-header compression (ROHC-style) — assessed low-priority for yip (loss-fragile + anti-DPI conflict); revisit only after 4c if a header-overhead bottleneck survives.

Lever **4c** of the single-core throughput campaign (4a send GSO ✓ #55, 4b TUN offload ✓ #56). The post-4a/4b profiling showed the single-core ceiling is **per-packet** cost (kernel UDP send `sendmsg` ~23%, TUN write ~20%, SipHash ~9% — all charged *per packet*). Raising the tunnel MTU cuts the **packet count** for the same bytes, so it reduces *every* per-packet cost proportionally — the cheapest, most broadly-applicable remaining lever. ### Scope - Make the tunnel MTU path-aware rather than a fixed conservative value; probe (PMTUD / PLPMTUD-style) and adapt, falling back cleanly to 1500-safe where the path can't carry more. - Interacts with FEC symbol sizing and the outer-UDP overhead budget — needs its own design (spec). ### Note - Jumbo frames only help where the *whole path* supports them; 1500 stays the internet-safe default. The win is largest on controlled/LAN paths. - Related parked idea: inner-header compression (ROHC-style) — assessed low-priority for yip (loss-fragile + anti-DPI conflict); revisit only after 4c if a header-overhead bottleneck survives.

vxfemboy commented

2026-07-22 17:54:35 +00:00

(Migrated from github.com)

Investigation complete (2026-07-22): levers measured and bounded

Ran a measurement-first spike (real Transport::encode path + netns) before committing to a mechanism. Full write-up: docs/research/2026-07-22-mtu-packetization-findings.md. Summary:

The send path, and where the cost is

inner → +16 AEAD → FEC symbols (size 1200, **last zero-padded to full symbol_size on the wire**) → +23 wire frame → +11 obf → +28/48 IP/UDP. Two cost tiers: per-inner-packet (AEAD, TUN-write — the two biggest) and per-symbol (sendmsg ~23%, SipHash ~9%, framing). yipd does not currently set the TUN MTU (kernel default 1500); the QUIC MtuDiscovery is only the mimicry transport, not the data plane.

Finding 1 — wire padding makes `symbol_size` double-edged

Every FEC symbol is zero-padded to full symbol_size on the wire (fec.rs "last zero-padded"; reassembler rejects data.len() != symbol_size; framer sends it untrimmed). So a blind symbol_size increase is a bandwidth regression for small packets (a 576 B packet at symbol_size 9000 → two 9000 B symbols, 3047% overhead).

Finding 2 — two MTU levers, very different reach

Lever 1 (symbol_size → ~1500): collapses a 1448 B packet from 2 source symbols to 1 (−33% datagrams). Reduces only per-symbol costs, not AEAD/TUN-write. Internet-safe but pays Finding 1's padding penalty on small packets. (Note: 1452 doesn't fit 1448+16 — need 1500.)
Lever 2 (jumbo inner MTU ~9000 + matching symbol_size): ~6× fewer of everything incl. AEAD + TUN-write. But only where the whole path carries jumbo, and only helps jumbo-sized traffic. This is the issue's "largest on controlled/LAN paths" win.

The issue's premise ("raising MTU reduces every per-packet cost proportionally") holds only for Lever 2.

Finding 3 — the FEC-rate lever is a MIRAGE (load-bearing, not waste)

The bench's "2.00 symbols/packet" turned out to be the permanent steady state of the Default class (DSCP 0 = ~all traffic): non-ARQ, min_ratio floors at 0.10, so it emits 1 source + 1 repair forever, even on a pristine link. Tempting to decay it to 0. Measured the experiment (floor→0):

Clean-link: 2.000 → 1.000 datagrams/packet (real 2× send-path reduction). ✅
10% netem, low-rate ping: baseline 10/10 → experiment 7/10, 9/10, 8/10 (~20% lost, unrecovered). ❌ Sparse traffic can't keep the adaptive controller armed; the ratio decays to ~0 between packets and losses go unprotected (Default is non-ARQ — nothing catches them).
The change breaks non_arq_class_keeps_floor, a deliberate tested invariant. The floor protects exactly the low-rate latency-sensitive traffic (SSH/DNS/VoIP/gaming) the class exists for.

Conclusion: the current FEC tuning is correct; do not pursue the FEC-rate change.

Net recommendation

The only defensible remaining work is static configurable MTU + last-symbol wire-trim (Finding 4: pad for the GF math, trim the partial last symbol on the wire — removes Lever 1's padding penalty), worthwhile for controlled/LAN/jumbo deployments, not worth the codec risk for internet-only paths. Suggest either closing this as investigated (revisit for a controlled-path deployment) or descoping to the static-MTU + wire-trim work filed for when it pays.

## Investigation complete (2026-07-22): levers measured and bounded Ran a measurement-first spike (real `Transport::encode` path + netns) before committing to a mechanism. Full write-up: `docs/research/2026-07-22-mtu-packetization-findings.md`. Summary: ### The send path, and where the cost is `inner → +16 AEAD → FEC symbols (size 1200, **last zero-padded to full symbol_size on the wire**) → +23 wire frame → +11 obf → +28/48 IP/UDP`. Two cost tiers: **per-inner-packet** (AEAD, TUN-write — the two biggest) and **per-symbol** (sendmsg ~23%, SipHash ~9%, framing). `yipd` does not currently set the TUN MTU (kernel default 1500); the QUIC MtuDiscovery is only the mimicry transport, not the data plane. ### Finding 1 — wire padding makes `symbol_size` double-edged Every FEC symbol is zero-padded to full `symbol_size` on the wire (`fec.rs` "last zero-padded"; reassembler rejects `data.len() != symbol_size`; framer sends it untrimmed). So a blind `symbol_size` increase is a **bandwidth regression** for small packets (a 576 B packet at symbol_size 9000 → two 9000 B symbols, 3047% overhead). ### Finding 2 — two MTU levers, very different reach - **Lever 1 (symbol_size → ~1500):** collapses a 1448 B packet from 2 source symbols to 1 (−33% datagrams). Reduces only per-symbol costs, not AEAD/TUN-write. Internet-safe but pays Finding 1's padding penalty on small packets. (Note: 1452 doesn't fit 1448+16 — need 1500.) - **Lever 2 (jumbo inner MTU ~9000 + matching symbol_size):** ~6× fewer of everything incl. AEAD + TUN-write. But only where the **whole path** carries jumbo, and only helps jumbo-sized traffic. This is the issue's "largest on controlled/LAN paths" win. The issue's premise ("raising MTU reduces every per-packet cost proportionally") holds only for **Lever 2**. ### Finding 3 — the FEC-rate lever is a MIRAGE (load-bearing, not waste) The bench's "2.00 symbols/packet" turned out to be the **permanent steady state** of the Default class (DSCP 0 = ~all traffic): non-ARQ, `min_ratio` floors at 0.10, so it emits 1 source + 1 repair forever, even on a pristine link. Tempting to decay it to 0. **Measured the experiment (floor→0):** - Clean-link: 2.000 → **1.000** datagrams/packet (real 2× send-path reduction). ✅ - 10% netem, low-rate ping: baseline **10/10** → experiment **7/10, 9/10, 8/10** (~20% lost, unrecovered). ❌ Sparse traffic can't keep the adaptive controller armed; the ratio decays to ~0 between packets and losses go unprotected (Default is non-ARQ — nothing catches them). - The change breaks `non_arq_class_keeps_floor`, a **deliberate tested invariant**. The floor protects exactly the low-rate latency-sensitive traffic (SSH/DNS/VoIP/gaming) the class exists for. **Conclusion:** the current FEC tuning is correct; do not pursue the FEC-rate change. ### Net recommendation The only defensible remaining work is **static configurable MTU + last-symbol wire-trim** (Finding 4: pad for the GF math, trim the partial last symbol on the wire — removes Lever 1's padding penalty), worthwhile for controlled/LAN/jumbo deployments, not worth the codec risk for internet-only paths. Suggest either **closing this** as investigated (revisit for a controlled-path deployment) or **descoping** to the static-MTU + wire-trim work filed for when it pays.