femboy/yip
Watch
1
0
Fork
You've already forked yip
 0
Code Issues 28 Pull requests 11 Projects Releases Packages Wiki Activity Actions 1

test(arq): instrument the io_uring RX drop (timeout theory refuted) #160

Closed
vxfemboy wants to merge 0 commits from fix/arq-rx-idle-timeout into main
pull from: fix/arq-rx-idle-timeout
merge into: femboy:main
Conversation 1 Commits - Files changed -
vxfemboy commented 2026-08-04 00:58:20 +00:00
Owner
Copy link

What (repurposed)

The original timeout-bump theory was refuted by its own diagnostic: the next uring CI run printed stop=idle idle_gap=30.0 at 80.1% delivery — the receiver waited the full 30s and the packets never came. So it is not receiver patience; the io_uring RX path itself drops datagrams under the container's load, while poll delivers 99.3% in the same run.

This PR is now a diagnostic, not a fix:

  • Revert RX_IDLE 30→10 (kept overridable; kept udp_rx.py stop=/idle_gap= which refuted the theory).
  • Add kernel-truth RX-drop instrumentation: /proc/net/snmp UdpRcvbufErrors deltas around the blast. A large RcvbufErrors on arqA under uring, ~0 under poll, localizes the loss to the underlay socket overflowing because uring drains too slowly (UDP_RECV_DEPTH=16 single-shot recvs vs poll's batched recvmmsg).

The next CI run reads this out. The real fix (deeper recv depth / multishot recv / force rmem) follows once the counter confirms the mechanism. Not reproducible on bare metal (RcvbufErrors=0, ≥99.2% both drivers, even under load).

## What (repurposed) The original timeout-bump theory was **refuted by its own diagnostic**: the next uring CI run printed `stop=idle idle_gap=30.0` at **80.1%** delivery — the receiver waited the full 30s and the packets never came. So it is **not** receiver patience; the **io_uring RX path itself drops datagrams** under the container's load, while poll delivers 99.3% in the same run. This PR is now a **diagnostic**, not a fix: - Revert `RX_IDLE` 30→10 (kept overridable; kept `udp_rx.py` `stop=/idle_gap=` which refuted the theory). - Add kernel-truth RX-drop instrumentation: `/proc/net/snmp` `UdpRcvbufErrors` deltas around the blast. A large `RcvbufErrors` on arqA under uring, ~0 under poll, localizes the loss to the underlay socket overflowing because uring drains too slowly (`UDP_RECV_DEPTH=16` single-shot recvs vs poll's batched recvmmsg). The next CI run reads this out. The real fix (deeper recv depth / multishot recv / force rmem) follows once the counter confirms the mechanism. Not reproducible on bare metal (RcvbufErrors=0, ≥99.2% both drivers, even under load).
fix(test): cover arq-uring CI flake by outlasting the retransmit tail
Some checks failed
CI / shear (pull_request) Successful in 6m5s
Details
CI / deny (pull_request) Successful in 11m3s
Details
Coverage / coverage (pull_request) Successful in 21m13s
Details
CI / build-test (pull_request) Successful in 21m52s
Details
Integration (privileged) / bench-smoke (pull_request) Successful in 1m55s
Details
Integration (privileged) / netem-comparison (pull_request) Successful in 9m37s
Details
Integration (privileged) / device-tests (pull_request) Successful in 1m0s
Details
Integration (privileged) / dpi-undetectability (pull_request) Successful in 18m38s
Details
Integration (privileged) / netns-tunnel-test (pull_request) Failing after 20m46s
Details
56315d66b1 
The privileged sudo CI job's `arq_recovers_bulk_loss` intermittently failed
under the io_uring driver: one run delivered 96.4% (< 98% required), received
19274/20000 with ARQ retransmits=193 — the sender was still actively
retransmitting when the receiver stopped counting. That truncated-tail
signature (in-flight retransmits + an early-terminating receiver) points at
receiver patience, not recovery capacity: udp_rx.py's idle timeout was 10s,
and io_uring inside the runner container evidently stalls retransmit delivery
past 10s of silence.

Not reproducible on bare metal: delivery held >=99.2% on both drivers, idle
and under 3x CPU oversubscription, and even with the idle timeout forced down
to 2s — this hardware never reaches the container's stall regime, so the
mechanism is supported by the CI signature, not proven locally.

Fix (mechanism-targeted, not a retry): raise the receiver idle timeout 10 -> 30s
so it outlasts a >10s retransmit stall and counts the recovered tail instead of
quitting mid-recovery. A retry/best-of-N was rejected: if the container stall
is deterministic it would fail every re-draw and fix nothing, whereas the
timeout bump catches the tail whether the stall is intermittent or not.

Also instrument udp_rx.py: it now prints stop=<idle|complete> and idle_gap, so
the next occurrence shows directly whether the receiver quit early on a slow
tail or the packets never came — turning inference into a read.

Cost: delivery is never 100% (some loss is always unrecovered within the flow
window), so udp_rx.py always exits via the idle path; the full 30s now lands on
every arq run on both driver passes, adding ~20s x2 to the test's wall-clock.

Verified locally: both drivers pass at 99.2%, stop=idle idle_gap=30.0.
This covers and instruments the flake; the new diagnostic will confirm or
refute the mechanism on the next CI occurrence.
test(arq): revert the refuted timeout bump; instrument the real uring RX drop
Some checks failed
Integration (privileged) / bench-smoke (pull_request) Successful in 3m24s
Details
Integration (privileged) / device-tests (pull_request) Successful in 44s
Details
CI / shear (pull_request) Successful in 4m10s
Details
CI / deny (pull_request) Successful in 7m32s
Details
CI / build-test (pull_request) Successful in 20m9s
Details
Coverage / coverage (pull_request) Successful in 21m7s
Details
Integration (privileged) / netem-comparison (pull_request) Successful in 19m48s
Details
Integration (privileged) / netns-tunnel-test (pull_request) Failing after 24m59s
Details
Integration (privileged) / dpi-undetectability (pull_request) Successful in 14m56s
Details
f6f3d2de3c 
The 10->30 idle-timeout bump from the previous commit did not fix the flake:
the next uring CI run printed `stop=idle idle_gap=30.0` at 80.1% delivery — the
receiver waited the full 30s and the packets never came. So it is NOT receiver
patience (that theory is refuted); the io_uring RX path itself drops datagrams
under the container's load, while poll delivers 99.3% in the same run.

- Revert RX_IDLE default 30 -> 10 (keep it overridable; keep udp_rx.py's
  stop=/idle_gap= diagnostic, which is what refuted the timeout theory).
- Add kernel-truth RX-drop instrumentation: capture /proc/net/snmp UdpRcvbufErrors
  (+ InErrors, InDatagrams) in both netns before/after the blast and print the
  delta. A large RcvbufErrors on arqA under uring, ~0 under poll, localizes the
  loss to the underlay socket overflowing because the io_uring RX path drains too
  slowly (only UDP_RECV_DEPTH=16 single-shot recvs outstanding vs poll's batched
  recvmmsg). The next CI run will read this out.

Not reproducible on bare metal: RcvbufErrors=0 and >=99.2% delivery on both
drivers here, idle and under load — this box's io_uring drains fast enough. The
instrumentation is the way to see inside the CI container's behavior.
vxfemboy changed title from fix(test): cover arq-uring CI flake (outlast retransmit tail) to test(arq): instrument the io_uring RX drop (timeout theory refuted) 2026-08-04 03:38:17 +00:00
vxfemboy commented 2026-08-04 05:52:52 +00:00
Author
Owner
Copy link

Superseded by #161 (merged). This branch started as a receiver-timeout fix, which its own diagnostic (udp_rx.py stop=/idle_gap=) then refuted (idle_gap=30.0 at 80.1% delivery = the receiver waited the full timeout, packets never came). #161 carries this branch's instrumentation (UdpRcvbufErrors deltas + the udp_rx.py diagnostic) plus the actual fix (io_uring UDP_RECV_DEPTH 16→256). Closing in favor of #161.

Superseded by #161 (merged). This branch started as a receiver-timeout fix, which its own diagnostic (`udp_rx.py` `stop=/idle_gap=`) then refuted (`idle_gap=30.0` at 80.1% delivery = the receiver waited the full timeout, packets never came). #161 carries this branch's instrumentation (`UdpRcvbufErrors` deltas + the `udp_rx.py` diagnostic) **plus** the actual fix (io_uring `UDP_RECV_DEPTH` 16→256). Closing in favor of #161.
vxfemboy closed this pull request 2026-08-04 05:52:58 +00:00
Some checks failed
Integration (privileged) / bench-smoke (pull_request) Successful in 3m24s
Details
Integration (privileged) / device-tests (pull_request) Successful in 44s
Details
CI / shear (pull_request) Successful in 4m10s
Details
CI / deny (pull_request) Successful in 7m32s
Details
CI / build-test (pull_request) Successful in 20m9s
Details
Coverage / coverage (pull_request) Successful in 21m7s
Details
Integration (privileged) / netem-comparison (pull_request) Successful in 19m48s
Details
Integration (privileged) / netns-tunnel-test (pull_request) Failing after 24m59s
Details
Integration (privileged) / dpi-undetectability (pull_request) Successful in 14m56s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
femboy/yip!160
No description provided.