fix: remaining netns-tunnel-test flakes (io_uring RX drop + flowshape spread gate) #161

Merged

vxfemboy merged 4 commits from fix/uring-recv-depth into main 2026-08-04 05:49:31 +00:00

Conversation 0 Commits 4 Files changed 5 +159 -67

vxfemboy commented 2026-08-04 03:50:24 +00:00

Owner

Copy link

Supersedes #160 (which was the diagnostic step; this PR includes that instrumentation plus the actual fix).

Root cause

The netns-tunnel-test job's arq_recovers_bulk_loss failed under the io_uring driver: 80–96% delivery vs the poll driver's 99.3% in the same run, with no driver fallback (the log had no falling back to PollDriver line, so it wasn't the teardown path).

The io_uring backend kept only UDP_RECV_DEPTH=16 single-shot recvs outstanding, so one poll_once harvests at most 16 datagrams (~3 ms of a 5000 pps flow) before re-arming. The poll backend's drain_udp drains the whole socket backlog per wakeup (recvmmsg(64) until a short read). Under the CI container's CPU contention the socket receive queue outpaced the 16-deep uring drain and overflowed a clamped net.core.rmem_max → kernel-level UDP drops the poll driver never saw.

Fix

• UDP_RECV_DEPTH 16 → 256 (matches RING_BUFS; ~590 KB of slots) — one poll_once now drains up to ~256 datagrams, closing the gap to poll.
• RING_ENTRIES 512 → 1024 (CQ → 2048) — headroom for the deeper recv queue + in-flight sends + TUN reads.

Evidence / how it was found

• Diagnostic (in this branch): reverted the earlier (refuted) receiver-timeout theory and added /proc/net/snmp UdpRcvbufErrors deltas + udp_rx.py stop=/idle_gap=. The refuting datapoint: a uring run printed stop=idle idle_gap=30.0 at 80.1% — the receiver waited the full 30 s and packets never came, so it was not receiver patience.
• The failing run stayed on io_uring the whole flow (no fallback) → drain-depth deficit, not a crash.
• An independent io_uring driver review reached the same drain-depth conclusion.

Verification

Local: 60 yip-io tests pass (all io_uring driver tests), arq passes on both drivers, RcvbufErrors=0. Not reproducible on bare metal (both drivers ≥99.2% even under load) — the CI container is the constrained regime, so this PR's CI is the real validation (uring arq should recover to ~99%, RcvbufErrors ~0).

Follow-ups (found in the same review, NOT in this PR)

• CRITICAL-ish correctness/DoS: a single bad UDP recv CQE tears down the whole driver and discards the rest of the reaped batch — reachable by any peer sending a >2048-byte datagram to the port (uring.rs:1093/1146), or on ECONNREFUSED/ENOMEM.
• Redundant submit + submit_and_wait(1) (two io_uring_enter) per idle iteration.
• RecvMsgMulti (multishot recvmsg, recovers src addr per completion) as the structural fix that removes the fixed depth entirely.
• No CQ-overflow observability; hot-path synchronous eprintln! cost.

Supersedes #160 (which was the diagnostic step; this PR includes that instrumentation **plus** the actual fix). ## Root cause The `netns-tunnel-test` job's `arq_recovers_bulk_loss` failed under the **io_uring** driver: 80–96% delivery vs the **poll** driver's 99.3% in the *same* run, with **no driver fallback** (the log had no `falling back to PollDriver` line, so it wasn't the teardown path). The io_uring backend kept only **`UDP_RECV_DEPTH=16`** single-shot recvs outstanding, so one `poll_once` harvests at most 16 datagrams (~3 ms of a 5000 pps flow) before re-arming. The poll backend's `drain_udp` drains the **whole** socket backlog per wakeup (`recvmmsg(64)` until a short read). Under the CI container's CPU contention the socket receive queue outpaced the 16-deep uring drain and overflowed a clamped `net.core.rmem_max` → kernel-level UDP drops the poll driver never saw. ## Fix - `UDP_RECV_DEPTH` **16 → 256** (matches `RING_BUFS`; ~590 KB of slots) — one `poll_once` now drains up to ~256 datagrams, closing the gap to poll. - `RING_ENTRIES` **512 → 1024** (CQ → 2048) — headroom for the deeper recv queue + in-flight sends + TUN reads. ## Evidence / how it was found - **Diagnostic** (in this branch): reverted the earlier (refuted) receiver-timeout theory and added `/proc/net/snmp` `UdpRcvbufErrors` deltas + `udp_rx.py` `stop=/idle_gap=`. The refuting datapoint: a uring run printed `stop=idle idle_gap=30.0` at 80.1% — the receiver waited the full 30 s and packets never came, so it was not receiver patience. - The failing run stayed on io_uring the whole flow (no fallback) → drain-depth deficit, not a crash. - An independent io_uring driver review reached the same drain-depth conclusion. ## Verification Local: 60 `yip-io` tests pass (all io_uring driver tests), arq passes on both drivers, `RcvbufErrors=0`. Not reproducible on bare metal (both drivers ≥99.2% even under load) — the CI container is the constrained regime, so **this PR's CI is the real validation** (uring arq should recover to ~99%, RcvbufErrors ~0). ## Follow-ups (found in the same review, NOT in this PR) - **CRITICAL-ish correctness/DoS:** a single bad UDP recv CQE tears down the whole driver *and discards the rest of the reaped batch* — reachable by any peer sending a **>2048-byte datagram** to the port (`uring.rs:1093/1146`), or on `ECONNREFUSED`/`ENOMEM`. - Redundant `submit` + `submit_and_wait(1)` (two `io_uring_enter`) per idle iteration. - `RecvMsgMulti` (multishot recvmsg, recovers src addr per completion) as the structural fix that removes the fixed depth entirely. - No CQ-overflow observability; hot-path synchronous `eprintln!` cost.

vxfemboy added 3 commits 2026-08-04 03:50:24 +00:00

fix(test): cover arq-uring CI flake by outlasting the retransmit tail ... 56315d66b1

The privileged sudo CI job's `arq_recovers_bulk_loss` intermittently failed
under the io_uring driver: one run delivered 96.4% (< 98% required), received
19274/20000 with ARQ retransmits=193 — the sender was still actively
retransmitting when the receiver stopped counting. That truncated-tail
signature (in-flight retransmits + an early-terminating receiver) points at
receiver patience, not recovery capacity: udp_rx.py's idle timeout was 10s,
and io_uring inside the runner container evidently stalls retransmit delivery
past 10s of silence.

Not reproducible on bare metal: delivery held >=99.2% on both drivers, idle
and under 3x CPU oversubscription, and even with the idle timeout forced down
to 2s — this hardware never reaches the container's stall regime, so the
mechanism is supported by the CI signature, not proven locally.

Fix (mechanism-targeted, not a retry): raise the receiver idle timeout 10 -> 30s
so it outlasts a >10s retransmit stall and counts the recovered tail instead of
quitting mid-recovery. A retry/best-of-N was rejected: if the container stall
is deterministic it would fail every re-draw and fix nothing, whereas the
timeout bump catches the tail whether the stall is intermittent or not.

Also instrument udp_rx.py: it now prints stop=<idle|complete> and idle_gap, so
the next occurrence shows directly whether the receiver quit early on a slow
tail or the packets never came — turning inference into a read.

Cost: delivery is never 100% (some loss is always unrecovered within the flow
window), so udp_rx.py always exits via the idle path; the full 30s now lands on
every arq run on both driver passes, adding ~20s x2 to the test's wall-clock.

Verified locally: both drivers pass at 99.2%, stop=idle idle_gap=30.0.
This covers and instruments the flake; the new diagnostic will confirm or
refute the mechanism on the next CI occurrence.

test(arq): revert the refuted timeout bump; instrument the real uring RX drop ... f6f3d2de3c

The 10->30 idle-timeout bump from the previous commit did not fix the flake:
the next uring CI run printed `stop=idle idle_gap=30.0` at 80.1% delivery — the
receiver waited the full 30s and the packets never came. So it is NOT receiver
patience (that theory is refuted); the io_uring RX path itself drops datagrams
under the container's load, while poll delivers 99.3% in the same run.

- Revert RX_IDLE default 30 -> 10 (keep it overridable; keep udp_rx.py's
  stop=/idle_gap= diagnostic, which is what refuted the timeout theory).
- Add kernel-truth RX-drop instrumentation: capture /proc/net/snmp UdpRcvbufErrors
  (+ InErrors, InDatagrams) in both netns before/after the blast and print the
  delta. A large RcvbufErrors on arqA under uring, ~0 under poll, localizes the
  loss to the underlay socket overflowing because the io_uring RX path drains too
  slowly (only UDP_RECV_DEPTH=16 single-shot recvs outstanding vs poll's batched
  recvmmsg). The next CI run will read this out.

Not reproducible on bare metal: RcvbufErrors=0 and >=99.2% delivery on both
drivers here, idle and under load — this box's io_uring drains fast enough. The
instrumentation is the way to see inside the CI container's behavior.

fix(io): deepen io_uring UDP recv queue so it drains like the poll driver ... 432585dff0

Root cause of the arq-uring CI drop (80-96% delivery vs poll's 99.3% in the
same run, no driver fallback): the io_uring backend kept only UDP_RECV_DEPTH=16
single-shot recvs outstanding, so one poll_once could harvest at most 16
datagrams (~3ms of a 5000pps flow) before re-arming. The poll backend's
drain_udp drains the WHOLE socket backlog per wakeup (recvmmsg(64) until short
read). Under the CI container's CPU contention the socket receive queue outpaced
the 16-deep uring drain and overflowed (a clamped net.core.rmem_max), dropping
datagrams the poll driver did not.

- Raise UDP_RECV_DEPTH 16 -> 256 (matches RING_BUFS; ~590 KB of slots) so one
  poll_once drains up to ~256 datagrams, closing the gap to the poll driver.
- Raise RING_ENTRIES 512 -> 1024 (CQ 1024 -> 2048) so the deeper recv queue plus
  in-flight sends and TUN reads fit without SQ-fill submit-retries or CQ
  overflow into the NODROP backlog.

Confirmed via: the failing run stayed on io_uring the whole flow (no "falling
back to PollDriver" line) — ruling out the driver-teardown path — and an
independent driver review reaching the same drain-depth conclusion. The
/proc/net/snmp UdpRcvbufErrors instrumentation added earlier in this branch will
show the socket overflow going to ~0 with this change.

Not reproducible on bare metal (both drivers hold >=99.2% even under load); the
CI container is the constrained regime. Verified locally: 60 yip-io tests pass
(all io_uring driver tests), arq passes on both drivers, RcvbufErrors=0.

Follow-ups identified in the same review, not in this change: single-bad-recv-CQE
tears down the whole driver and drops the rest of the batch (e.g. any peer
sending a >2048B datagram); redundant submit+wait syscall per idle iteration;
RecvMsgMulti as the structural fix removing the fixed depth entirely; CQ-overflow
observability; hot-path eprintln! cost.

vxfemboy added 1 commit 2026-08-04 04:25:51 +00:00

fix(test): gate flowshape on span SPREAD, not an absolute floor (CI flake) ... 8c94d49674

The flowshape fix from #159 (dense-cluster span > CLUSTER_MIN=12) started
flaking on CI: a session measured span=12 exactly and failed `span > 12`. Under
the CI runner's contention the intra-burst gaps balloon toward the ~3ms
separator floor, fragmenting the dense clusters and pulling junk-ON spans down
from 15-32 into the 12-16 band — colliding with any absolute floor above the
junk-OFF ceiling of 10. The 10-vs-12 gap is simply too narrow to gate on
absolutely under CI variance.

Fix: gate on the level-INDEPENDENT signal instead. Each side's Jc in [3,12] is
redrawn per session, so junk-ON spans vary widely (spread max-min = 14-19 on CI,
13 locally) while a junk-free/fixed-junk handshake is near-constant (spread <= 4).

- Gate (b) PRIMARY: spread (max-min) >= SPREAD_MIN (6). Proves BOTH junk-present
  and randomized in one shot; a junk-free run cannot manufacture the spread.
  Contention shifts the absolute spans but not the spread, so it does not
  false-fail the way the floor did.
- Gate (a) demoted to a loose sanity floor: every span > SPAN_FLOOR (4), only
  rejecting a broken capture (junk-on clears it with huge margin).

Verified locally: spans 15 20 27 28 22 16 17 22, spread=13 >= 6, both gates
pass. junk-off spread <= 4 < 6 keeps the gate meaningful.

vxfemboy changed title from fix(io): deepen io_uring UDP recv queue (fixes arq-uring CI drop) to fix: remaining netns-tunnel-test flakes (io_uring RX drop + flowshape spread gate) 2026-08-04 04:25:54 +00:00

vxfemboy merged commit cac09dea84 into main 2026-08-04 05:49:31 +00:00

vxfemboy referenced this pull request from a commit 2026-08-04 05:49:31 +00:00

Merge pull request 'fix: remaining netns-tunnel-test flakes (io_uring RX drop + flowshape spread gate)' (#161) from fix/uring-recv-depth into main

vxfemboy referenced this pull request 2026-08-04 05:52:40 +00:00

netns-tunnel-test: two intermittent failures (cert-revocation budget vs documented discovery time; ARQ floor inside noise band) #157

vxfemboy referenced this pull request 2026-08-04 05:52:52 +00:00

test(arq): instrument the io_uring RX drop (timeout theory refuted) #160

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

femboy/yip!161

No description provided.