[lab] deception-resilience gate hangs at connect-flood (600 conns) — disabled in CI #136

New issue

Closed

opened 2026-07-07 00:08:22 +00:00 by vxfemboy · 1 comment

vxfemboy commented

2026-07-07 00:08:22 +00:00

(Migrated from github.com)

The deception-resilience lab gate hangs at step 2 (the connect-flood --concurrency 600 --duration 8 assert) and never terminates within the gate timeout — reproduced locally (exit 124, only ok 1 - wait port-open completes). Disabled in CI (if: false) in the lab-timeout PR so it can't starve the rest of the suite.

Context / history:

Originally the scenario used in-netns cargo run --offline -p blackwall-trafficgen (the #88 cargo-in-scenario anti-pattern) — that was fixed to the pre-built target/debug/trafficgen binary in the same PR. The hang persists with the binary, so it's a separate issue from the cargo-run wedge.
The #9 (DeceptionTransport) work noted this scenario's cargo run failure "reproduces identically on unmodified main," suggesting it predates recent changes.

To investigate:

Is the hang in the trafficgen connect-flood subcommand (does it fail to terminate after --duration when connections are dropped/held?) or in the engine's drop-at-cap path holding connections open?
Rule out a regression from the DeceptionTransport #9 refactor or the stateless-tier work: run the scenario against a commit before those changes. The single-connection deception gate passes, so basic serve() works; this is load-specific.
Fix the terminating behavior (connect-flood must always exit at --duration, closing in-flight sockets), then re-enable the gate.

Blocks: nothing critical (CI is green without it), but it's a resilience/correctness gate (#67) we want back.

The `deception-resilience` lab gate hangs at step 2 (the `connect-flood --concurrency 600 --duration 8` assert) and never terminates within the gate timeout — reproduced locally (exit 124, only `ok 1 - wait port-open` completes). Disabled in CI (`if: false`) in the lab-timeout PR so it can't starve the rest of the suite. **Context / history:** - Originally the scenario used in-netns `cargo run --offline -p blackwall-trafficgen` (the #88 cargo-in-scenario anti-pattern) — that was fixed to the pre-built `target/debug/trafficgen` binary in the same PR. The hang persists with the binary, so it's a *separate* issue from the cargo-run wedge. - The #9 (DeceptionTransport) work noted this scenario's `cargo run` failure "reproduces identically on unmodified main," suggesting it predates recent changes. **To investigate:** 1. Is the hang in the trafficgen `connect-flood` subcommand (does it fail to terminate after `--duration` when connections are dropped/held?) or in the engine's drop-at-cap path holding connections open? 2. **Rule out a regression** from the DeceptionTransport #9 refactor or the stateless-tier work: run the scenario against a commit before those changes. The single-connection `deception` gate passes, so basic serve() works; this is load-specific. 3. Fix the terminating behavior (connect-flood must always exit at `--duration`, closing in-flight sockets), then re-enable the gate. Blocks: nothing critical (CI is green without it), but it's a resilience/correctness gate (#67) we want back.

vxfemboy commented

2026-07-09 02:29:58 +00:00

(Migrated from github.com)

deception-resilience gate re-enabled and green (#159): the connect-flood deadline fix here + the tproxy accept fix (#157) together let it serve under load and terminate cleanly. Closing.