perf: yip-wire SipHash header-auth is ~9% of receiver CPU under load — investigate/optimize #58

New issue

Open

opened 2026-07-12 23:50:05 +00:00 by vxfemboy · 3 comments

vxfemboy commented

2026-07-12 23:50:05 +00:00

(Migrated from github.com)

The post-4a perf re-profile (bulk traffic, 1-core AMD EPYC target box) surfaced the yip-wire coverage-auth SipHash (Sip24) as ~9% of receiver CPU — a surprising userspace hotspot, second only to the kernel TUN-write path (~20%) and well above AEAD decrypt (~4.4%) and RS FEC decode (~1.6%).

See crates/yip-bench/RESULTS.md (the 4b re-profile section) for the profile.

Investigate

Is the SipHash recomputed redundantly (e.g. over more bytes than necessary, or more than once per packet)?
Can the hashed region be reduced (auth only the header, not the full covered payload, if the AEAD tag already covers the payload)?
Is Sip24 (24 rounds) needed, or would SipHash-1-3 / a keyed-BLAKE2s-short / an AES-NI-based one-block MAC be materially cheaper at equal security for a per-packet keyed header tag?
Measure the candidate on the target box before committing (spike-first, like the throughput levers).

Constraints

The header-auth is load-bearing for the anti-DPI wire design (keyed header protection, no fixed bytes). Any change must preserve the security property and the "no fixed signature" goal.

The post-4a `perf` re-profile (bulk traffic, 1-core AMD EPYC target box) surfaced the **`yip-wire` coverage-auth SipHash (Sip24) as ~9% of receiver CPU** — a surprising userspace hotspot, second only to the kernel TUN-write path (~20%) and well above AEAD decrypt (~4.4%) and RS FEC decode (~1.6%). See `crates/yip-bench/RESULTS.md` (the 4b re-profile section) for the profile. ### Investigate - Is the SipHash recomputed redundantly (e.g. over more bytes than necessary, or more than once per packet)? - Can the hashed region be reduced (auth only the header, not the full covered payload, if the AEAD tag already covers the payload)? - Is Sip24 (24 rounds) needed, or would SipHash-1-3 / a keyed-BLAKE2s-short / an AES-NI-based one-block MAC be materially cheaper at equal security for a per-packet keyed header tag? - Measure the candidate on the target box before committing (spike-first, like the throughput levers). ### Constraints - The header-auth is load-bearing for the anti-DPI wire design (keyed header protection, no fixed bytes). Any change must preserve the security property and the "no fixed signature" goal.

vxfemboy commented

2026-07-13 02:15:35 +00:00

(Migrated from github.com)

Investigation (2026-07-12): traced the pipeline — SipHash-2-4 auth_tag runs over the full ~1400-byte FEC symbol per packet (~9% receiver CPU). It is NOT redundant with the AEAD: order is encrypt → FEC-encode ciphertext → per-symbol SipHash; on receive deframe(verify SipHash) → FEC-decode → AEAD-decrypt. The SipHash authenticates each FEC symbol before the decoder, keeping forged/corrupted symbols out of reassembly (load-bearing for loss recovery) + cheap-reject + anti-DPI. So we can't shrink it to the header — a faster MAC over the same bytes is the only lever. Options: SipHash-1-3 (~2×, minor margin cut), keyed BLAKE3 (SIMD, constant-time, no AES-NI). AES-NI MAC rejected on principle (reintroduces the AES-NI dependency + software-AES timing side channel that yip chose ChaCha20 to avoid). Parked as low-yield on current hardware: ~4.5% best-case CPU saving that does NOT move end-to-end throughput on the RTT/window-capped target boxes (same lesson as 4b), for a wire-format change to security-critical code. Revisit if/when a box is genuinely MAC-bound.

Investigation (2026-07-12): traced the pipeline — SipHash-2-4 `auth_tag` runs over the full ~1400-byte FEC symbol per packet (~9% receiver CPU). It is **NOT redundant with the AEAD**: order is encrypt → FEC-encode ciphertext → per-symbol SipHash; on receive deframe(verify SipHash) → FEC-decode → AEAD-decrypt. The SipHash authenticates each FEC symbol *before* the decoder, keeping forged/corrupted symbols out of reassembly (load-bearing for loss recovery) + cheap-reject + anti-DPI. So we can't shrink it to the header — a faster MAC over the same bytes is the only lever. Options: SipHash-1-3 (~2×, minor margin cut), keyed BLAKE3 (SIMD, constant-time, no AES-NI). **AES-NI MAC rejected on principle** (reintroduces the AES-NI dependency + software-AES timing side channel that yip chose ChaCha20 to avoid). **Parked as low-yield on current hardware:** ~4.5% best-case CPU saving that does NOT move end-to-end throughput on the RTT/window-capped target boxes (same lesson as 4b), for a wire-format change to security-critical code. Revisit if/when a box is genuinely MAC-bound.

vxfemboy commented

2026-07-26 02:49:07 +00:00

(Migrated from github.com)

Spike done — measured both candidates on the EPYC target box (PR #114)

Isolated keyed-MAC microbench over the exact covered region (header‖symbol, 8-byte tag), on the EPYC 4b-profile box and a dev box — ratios agree:

covered	candidate	EPYC ns	vs SipHash-2-4
1415 B	siphash13	420	0.55× (−45%)
1415 B	siphash24 (today)	758	1.00×
1415 B	blake3_keyed	1888	2.49× (slower)
63 B	siphash13	31.8	0.63×
63 B	siphash24	50.5	1.00×
63 B	blake3_keyed	104	2.06× (slower)

Findings

SipHash-1-3 is the only real lever: ~45% cheaper at packet size → ≈4% of receiver CPU. Confirms the earlier ~4.5% estimate with real numbers.
BLAKE3 is rejected on data, not just principle: ~2.5× slower at a 1.4 KB symbol — per-call key-schedule + finalize overhead dominates its SIMD throughput at packet sizes. (Runtime AVX2 dispatch confirmed active on both boxes, so this is not a fallback artifact.)

Decision: stay parked. The ~4% receiver-CPU saving is real but does not move end-to-end throughput on the RTT/window-capped boxes (same lesson as 4b), and it would mean a reduced-round MAC on security-critical wire code. Not worth it now.

Revisit trigger: a genuinely MAC-bound (CPU-bound, not RTT-bound) receiver — e.g. short-RTT/LAN paths or enough parallel sessions to saturate a core. The bench (crates/yip-bench/benches/mac_candidates.rs) stays as the decision artifact + regression guard; re-run it on the candidate box before reopening.

## Spike done — measured both candidates on the EPYC target box (PR #114) Isolated keyed-MAC microbench over the exact covered region (header‖symbol, 8-byte tag), on the EPYC 4b-profile box and a dev box — ratios agree: | covered | candidate | EPYC ns | vs SipHash-2-4 | |--:|--|--:|--:| | 1415 B | **siphash13** | **420** | **0.55× (−45%)** | | 1415 B | siphash24 (today) | 758 | 1.00× | | 1415 B | blake3_keyed | 1888 | 2.49× (slower) | | 63 B | siphash13 | 31.8 | 0.63× | | 63 B | siphash24 | 50.5 | 1.00× | | 63 B | blake3_keyed | 104 | 2.06× (slower) | **Findings** 1. **SipHash-1-3 is the only real lever:** ~45% cheaper at packet size → ≈4% of receiver CPU. Confirms the earlier ~4.5% estimate with real numbers. 2. **BLAKE3 is rejected on data, not just principle:** ~2.5× *slower* at a 1.4 KB symbol — per-call key-schedule + finalize overhead dominates its SIMD throughput at packet sizes. (Runtime AVX2 dispatch confirmed active on both boxes, so this is not a fallback artifact.) **Decision: stay parked.** The ~4% receiver-CPU saving is real but does not move end-to-end throughput on the RTT/window-capped boxes (same lesson as 4b), and it would mean a reduced-round MAC on security-critical wire code. Not worth it now. **Revisit trigger:** a genuinely MAC-bound (CPU-bound, not RTT-bound) receiver — e.g. short-RTT/LAN paths or enough parallel sessions to saturate a core. The bench (`crates/yip-bench/benches/mac_candidates.rs`) stays as the decision artifact + regression guard; re-run it on the candidate box before reopening.

vxfemboy commented

2026-07-26 03:14:50 +00:00

(Migrated from github.com)

Correction / follow-up: the "doesn't move throughput" line was regime-specific

The decision above rests on "the ~4% saving does not move end-to-end throughput on the RTT/window-capped boxes." A CPU-bound-regime spike run afterward (PR #115) shows that premise is regime-specific to the 24 ms single-flow WAN path, not general:

Under UDP blast the single receiver core saturates (~0.97) and caps at ~1.2 Gbps with ~80% drop at every RTT — a hard single-core processing ceiling.
TCP is CPU-bound at low RTT (0 ms: 0.97 Gbps, core 0.86) and window-bound only at high RTT (24 ms: 0.42 Gbps, core idle 0.38).

So in short-RTT / regional / aggregate-parallel deployments the receiver is CPU-bound, and there the ~4% MAC saving ≈ +50 Mbps and codec-path wins scale directly. #58 stays parked (SipHash is third-tier behind TUN-write and AEAD), but the reason is "not the biggest lever," not "CPU never matters." See crates/yip-bench/cpu-bound-regime.md.

### Correction / follow-up: the "doesn't move throughput" line was regime-specific The decision above rests on "the ~4% saving does not move end-to-end throughput on the RTT/window-capped boxes." A CPU-bound-regime spike run afterward (PR #115) shows that premise is **regime-specific to the 24 ms single-flow WAN path**, not general: - Under UDP blast the single receiver core **saturates (~0.97) and caps at ~1.2 Gbps** with ~80% drop at every RTT — a hard single-core processing ceiling. - TCP is **CPU-bound at low RTT** (0 ms: 0.97 Gbps, core 0.86) and window-bound only at high RTT (24 ms: 0.42 Gbps, core idle 0.38). So in short-RTT / regional / aggregate-parallel deployments the receiver **is** CPU-bound, and there the ~4% MAC saving ≈ +50 Mbps and codec-path wins scale directly. #58 stays parked (SipHash is third-tier behind TUN-write and AEAD), but the reason is "not the biggest lever," **not** "CPU never matters." See `crates/yip-bench/cpu-bound-regime.md`.