[lab/CI] Killed `lab test` orphans self-daemonizing processes that wedge the CI step past its timeout #137

New issue

Closed

opened 2026-07-07 00:51:14 +00:00 by vxfemboy · 3 comments

vxfemboy commented

2026-07-07 00:51:14 +00:00

(Migrated from github.com)

Several lab gates wedge in CI: the step stays in_progress far past its timeout-minutes (observed 30+ min on a 6-min step) until the whole job is cancelled at the job cap. It was latent for many commits because the flow-live gate (earlier in the sequence) always wedged first and cancelled the job before later gates ran.

Confirmed wedging gates: flow-live (#88), deception-resilience (#136), flowspec, flowspec-auto. All run long-lived / self-daemonizing helpers (hsflowd, the deception engine under load, BIRD). Gates that reap cleanly (deception, flow, bgp, rtbh, deception-syncookie, deception-syncookie-v6, xdp) pass.

Root cause (hypothesis): when lab test is killed before its normal teardown runs — by the #88 in-process watchdog (std::process::exit(124), which skips the Drop-based Teardown) or by the outer sudo timeout — the scenario's daemons are orphaned. Some (e.g. self-daemonizing BIRD, hsflowd) survive and keep the CI step's stdout/stderr pipe open, so GitHub's runner never sees EOF and the step's own timeout-minutes kill doesn't finalize it. The step only ends when the job cap cancels everything.

Fix directions:

Make the watchdog reap the scenario's daemon process groups before std::process::exit (teardown-then-exit instead of bare exit), so no orphan survives a watchdog trip.
And/or run each gate's lab test in its own session and have the CI wrapper kill the whole session/process-group on timeout (e.g. setsid + group-kill), and close inherited fds (3/4 from the exec redirect) so a lingering daemon can't hold the runner pipe.
Verify by re-enabling each quarantined gate.

Currently quarantined in CI (via if: false, with the timeout PR) so the suite is green: flow-live, deception-resilience, flowspec, flowspec-auto. Re-enable as each is fixed.

Several lab gates wedge in CI: the step stays `in_progress` far past its `timeout-minutes` (observed 30+ min on a 6-min step) until the whole job is cancelled at the job cap. It was latent for many commits because the `flow-live` gate (earlier in the sequence) always wedged first and cancelled the job before later gates ran. **Confirmed wedging gates:** `flow-live` (#88), `deception-resilience` (#136), `flowspec`, `flowspec-auto`. All run long-lived / self-daemonizing helpers (hsflowd, the deception engine under load, BIRD). Gates that reap cleanly (`deception`, `flow`, `bgp`, `rtbh`, `deception-syncookie`, `deception-syncookie-v6`, `xdp`) pass. **Root cause (hypothesis):** when `lab test` is killed before its normal teardown runs — by the #88 in-process watchdog (`std::process::exit(124)`, which skips the `Drop`-based `Teardown`) or by the outer `sudo timeout` — the scenario's daemons are orphaned. Some (e.g. self-daemonizing BIRD, hsflowd) survive and keep the CI step's stdout/stderr pipe open, so GitHub's runner never sees EOF and the step's own `timeout-minutes` kill doesn't finalize it. The step only ends when the *job* cap cancels everything. **Fix directions:** 1. Make the watchdog reap the scenario's daemon process groups **before** `std::process::exit` (teardown-then-exit instead of bare exit), so no orphan survives a watchdog trip. 2. And/or run each gate's `lab test` in its own session and have the CI wrapper kill the whole session/process-group on timeout (e.g. `setsid` + group-kill), and close inherited fds (3/4 from the `exec` redirect) so a lingering daemon can't hold the runner pipe. 3. Verify by re-enabling each quarantined gate. **Currently quarantined in CI** (via `if: false`, with the timeout PR) so the suite is green: flow-live, deception-resilience, flowspec, flowspec-auto. Re-enable as each is fixed.

vxfemboy commented

2026-07-07 08:12:35 +00:00

(Migrated from github.com)

New data point: with the #136 connect-flood fix, re-enabling the deception-resilience gate makes it pass green in CI — but the very next gate, rtbh (BIRD), then wedged (33min on a 6min step), reproducibly only when deception-resilience runs before it. So deception-resilience's teardown appears to leave residue (a lingering engine/flood daemon or netns) that wedges the following BIRD gate — the same daemon-orphan class this issue tracks. The systemic teardown-on-kill fix should let ALL four quarantined gates (flow-live, deception-resilience, flowspec, flowspec-auto) re-enable together; re-enabling them one at a time hits these cross-gate wedges.

New data point: with the #136 connect-flood fix, re-enabling the `deception-resilience` gate makes it **pass green in CI** — but the very next gate, `rtbh` (BIRD), then wedged (33min on a 6min step), reproducibly only when deception-resilience runs before it. So deception-resilience's teardown appears to leave residue (a lingering engine/flood daemon or netns) that wedges the following BIRD gate — the same daemon-orphan class this issue tracks. The systemic teardown-on-kill fix should let ALL four quarantined gates (flow-live, deception-resilience, flowspec, flowspec-auto) re-enable together; re-enabling them one at a time hits these cross-gate wedges.

vxfemboy commented

2026-07-07 15:23:13 +00:00

(Migrated from github.com)

Reverted (#152). The residue-reap fix (#150) regressed previously-green daemon-gates: deception-syncookie-v6 wedged ~24min on main, and deception-syncookie wedged in a follow-up PR — both stable for many runs before #137. Cause: the unbounded pre-gate sudo lab down + the reap_orphan_lab_procs() pkill intermittently hang/race the daemon-gates' engines.

The root-cause finding stands (orphans survive teardown into anonymous, name-deleted namespaces — kill-before-delete + ip netns pids reaping is the right idea). But a safe re-land must: (1) timeout-wrap every sweep/reap so it can NEVER hang a gate; (2) make the orphan pkill signature strictly scoped so it cannot match a live gate's own engine or race it; (3) verify on a full CI run that no previously-green daemon-gate (deception, deception-syncookie, deception-syncookie-v6, trafficgen) regresses BEFORE re-enabling any quarantined gate. Needs a CI-reproduction loop to iterate safely — deferred.

**Reverted (#152).** The residue-reap fix (#150) regressed previously-green daemon-gates: `deception-syncookie-v6` wedged ~24min on main, and `deception-syncookie` wedged in a follow-up PR — both stable for many runs before #137. Cause: the unbounded pre-gate `sudo lab down` + the `reap_orphan_lab_procs()` `pkill` intermittently hang/race the daemon-gates' engines. The **root-cause finding stands** (orphans survive teardown into anonymous, name-deleted namespaces — kill-before-delete + `ip netns pids` reaping is the right idea). But a safe re-land must: (1) **timeout-wrap** every sweep/reap so it can NEVER hang a gate; (2) make the orphan `pkill` signature strictly scoped so it cannot match a live gate's own engine or race it; (3) verify on a full CI run that no previously-green daemon-gate (deception, deception-syncookie, deception-syncookie-v6, trafficgen) regresses BEFORE re-enabling any quarantined gate. Needs a CI-reproduction loop to iterate safely — deferred.

vxfemboy commented

2026-07-09 02:29:56 +00:00

(Migrated from github.com)

Resolved for 3 of 4 gates (#159). The systemic cross-gate wedge root cause was broken tproxy, not (only) a teardown-thoroughness bug: the deception-TCP tproxy rule set the tproxy verdict but no terminal accept, so the interactive tier didn't deliver connections — the engine hung on the broken path and left orphaned processes that wedged the next gate (rtbh/flowspec). Fixing the tproxy rule to mark set → tproxy → accept (#157, plus the #136 connect-flood deadline) makes the engine serve under load AND tear down cleanly. With that on main, deception-resilience, flowspec, and flowspec-auto were re-enabled and pass green (twice, #159).

My earlier teardown-reap attempt (#150, reverted #152) was treating the symptom; the tproxy fix removed the cause. Remaining: only flow-live, whose wedge is the separate in-step hsflowd-pipe issue tracked in #88 — closing this as resolved and leaving flow-live to #88.

**Resolved for 3 of 4 gates (#159).** The systemic cross-gate wedge root cause was **broken tproxy**, not (only) a teardown-thoroughness bug: the deception-TCP tproxy rule set the tproxy verdict but no terminal `accept`, so the interactive tier didn't deliver connections — the engine hung on the broken path and left orphaned processes that wedged the *next* gate (rtbh/flowspec). Fixing the tproxy rule to `mark set → tproxy → accept` (#157, plus the #136 connect-flood deadline) makes the engine serve under load AND tear down cleanly. With that on main, `deception-resilience`, `flowspec`, and `flowspec-auto` were re-enabled and pass green (twice, #159). My earlier teardown-reap attempt (#150, reverted #152) was treating the symptom; the tproxy fix removed the cause. Remaining: only **flow-live**, whose wedge is the *separate* in-step hsflowd-pipe issue tracked in #88 — closing this as resolved and leaving flow-live to #88.