Intermittent: punch→relay escalation reuses the punch ephemeral (hardening.36 money test flakes on main) #116

New issue

Closed

opened 2026-07-26 03:14:38 +00:00 by vxfemboy · 0 comments

vxfemboy commented

2026-07-26 03:14:38 +00:00

(Migrated from github.com)

Summary

bin/yipd/tests/run-netns-pathswitch-rehandshake.sh (CI job hardening.36 path-switch convergence money test) fails intermittently on main. Investigation shows this is not a test flake — it's a real intermittent control-plane bug: on a punch→relay escalation, the initiator (A) sometimes reuses the punch attempt's ephemeral instead of drawing a fresh one, reproducing the pre-#36-inversion "ephemeral preservation" behavior the test now guards against (reopens the #34 downgrade window).

Evidence (CI witness output)

The rekey_epoch_witness counts A's Init ephemerals. Same gate, deterministic message, intermittent outcome:

run	HANDSHAKE_INIT_PKTS	DISTINCT_INIT_EPHEMERALS	result
main `a7628e3` (30183481851)	5	1	FAIL
main `98dba84` (30182458064)	5	1	FAIL
#113 re-run attempt-1 (30184970902)	5	1	FAIL
#114 run (30184978853), poll	10	10	PASS
#114 run (30184978853), uring	5	2	PASS

In the failing case A sends 5 Inits all carrying one ephemeral — so it is not a pcap capture race (5 were captured; a fresh one would be among them). Convergence still PASSES in the failing case (0% steady-state loss, relay-forwarded > 0) because A converges via the reused ephemeral — which is exactly the regression: a preserved ephemeral is a bare retransmit the freshness gate would refuse as stale on a genuine replay.

Why it matters

The test's headline assertion (post-#34 Task 4 / #36-inversion) is that punch→relay retarget must draw a FRESH ephemeral (state = Idle; begin_handshake(..)). Intermittent reuse means the #34 downgrade protection is not reliably in force on escalation.

Likely area (not yet root-caused)

bin/yipd/src/path.rs — Direct→Punch→Relay escalation state machine.
bin/yipd/src/peer_manager.rs:2160 — comment already flags a "stale relay flag from a raced escalation".
bin/yipd/src/handshake.rs:452 — "a NEW handshake attempt draws a fresh ephemeral"; the escalation path apparently sometimes retransmits the existing attempt instead of starting a new one.
bin/yipd/src/peer_manager.rs:1718 — state = PeerState::Idle reset that should precede the fresh begin_handshake.

The intermittency points to a race between the punch→relay escalation trigger and the handshake-retransmit path: when the retransmit timer fires the escalation as a bare resend, the ephemeral is preserved; when it goes through Idle→begin_handshake, a fresh one is drawn.

Impact on CI

This gate blocks any PR whose integration run happens to hit the reuse case (e.g. #113). It is pre-existing on main and unrelated to those PRs.

Root-cause the escalation trigger and ensure punch→relay retarget always goes Idle→begin_handshake (fresh ephemeral). Do not relax the test — it is correctly catching this.

## Summary `bin/yipd/tests/run-netns-pathswitch-rehandshake.sh` (CI job **hardening.36 path-switch convergence money test**) fails **intermittently on `main`**. Investigation shows this is **not a test flake** — it's a real intermittent control-plane bug: on a punch→relay escalation, the initiator (A) *sometimes* reuses the punch attempt's ephemeral instead of drawing a fresh one, reproducing the pre-#36-inversion "ephemeral preservation" behavior the test now guards against (reopens the #34 downgrade window). ## Evidence (CI witness output) The `rekey_epoch_witness` counts A's Init ephemerals. Same gate, deterministic message, intermittent outcome: | run | HANDSHAKE_INIT_PKTS | DISTINCT_INIT_EPHEMERALS | result | |--|--:|--:|--| | main `a7628e3` (30183481851) | 5 | **1** | FAIL | | main `98dba84` (30182458064) | 5 | **1** | FAIL | | #113 re-run attempt-1 (30184970902) | 5 | **1** | FAIL | | #114 run (30184978853), poll | 10 | 10 | PASS | | #114 run (30184978853), uring | 5 | 2 | PASS | In the failing case A sends **5 Inits all carrying one ephemeral** — so it is *not* a pcap capture race (5 were captured; a fresh one would be among them). Convergence still PASSES in the failing case (0% steady-state loss, relay-forwarded > 0) because A converges **via the reused ephemeral** — which is exactly the regression: a preserved ephemeral is a bare retransmit the freshness gate would refuse as stale on a genuine replay. ## Why it matters The test's headline assertion (post-#34 Task 4 / #36-inversion) is that punch→relay retarget must draw a FRESH ephemeral (`state = Idle; begin_handshake(..)`). Intermittent reuse means the #34 downgrade protection is not reliably in force on escalation. ## Likely area (not yet root-caused) - `bin/yipd/src/path.rs` — Direct→Punch→Relay escalation state machine. - `bin/yipd/src/peer_manager.rs:2160` — comment already flags a "stale `relay` flag from a **raced escalation**". - `bin/yipd/src/handshake.rs:452` — "a NEW handshake attempt draws a fresh ephemeral"; the escalation path apparently sometimes retransmits the existing attempt instead of starting a new one. - `bin/yipd/src/peer_manager.rs:1718` — `state = PeerState::Idle` reset that should precede the fresh `begin_handshake`. The intermittency points to a race between the punch→relay escalation trigger and the handshake-retransmit path: when the retransmit timer fires the escalation as a bare resend, the ephemeral is preserved; when it goes through Idle→begin_handshake, a fresh one is drawn. ## Impact on CI This gate blocks any PR whose integration run happens to hit the reuse case (e.g. #113). It is pre-existing on `main` and unrelated to those PRs. ## Next Root-cause the escalation trigger and ensure punch→relay retarget always goes Idle→begin_handshake (fresh ephemeral). Do **not** relax the test — it is correctly catching this.