fix(#137): reap lab teardown residue (orphans in anonymous namespaces) #150

Merged

vxfemboy merged 3 commits from sp-fix-137-teardown into main

2026-07-07 14:49:47 +00:00

vxfemboy commented

2026-07-07 13:35:28 +00:00

(Migrated from github.com)

Fixes the systemic lab-teardown residue behind #137. Root cause (proven with live evidence): ~18 orphaned blackwall-deception-interop processes were found running with an empty ip netns list — daemons survive teardown orphaned into anonymous (name-deleted) namespaces, invisible to any bw-* sweep, still holding ports/state. Caused by lab test being force-killed before its Drop teardown, and down_all deleting netns names before killing attached processes.

Fix: kill-before-delete ordering (ip netns pids <ns> → kill, then netns_del); a command-signature-scoped orphan pkill in lab down; idempotent host-inet blackwall nft/route clear; and a defensive pre-gate lab down in each CI step.

Verified in CI: all 14 active gates pass with the hardening + pre-gate sweep (safe). Locally, lab down reaps orphans and flowspec/rtbh/flow scenarios tear down clean (no leftover netns/orphans/host table).

Gate re-enables deferred (kept if: false): re-enabling the 4 quarantined gates revealed each has its OWN reliability issue, separate from the residue fix:

flow-live (#88): in-step hsflowd holds the CI step's stdout pipe past the timeout (still wedges even with teardown fixed).
deception-resilience: flaky/borderline under its 600-conn tproxy load — wedged in this PR's run though it passed in #145.
flowspec / flowspec-auto: never ran (behind the above); unverified.
Each needs dedicated per-gate work (tracked in #137/#88); this PR lands only the systemic residue fix, which is safe and green.

Gate: fmt + clippy --deny warnings clean, workspace builds, coverage 95.46%, YAML valid.

Fixes the systemic lab-teardown residue behind #137. **Root cause (proven with live evidence):** ~18 orphaned `blackwall-deception-interop` processes were found running with an **empty `ip netns list`** — daemons survive teardown orphaned into *anonymous* (name-deleted) namespaces, invisible to any `bw-*` sweep, still holding ports/state. Caused by `lab test` being force-killed before its `Drop` teardown, and `down_all` deleting netns *names* before killing attached processes. **Fix:** kill-before-delete ordering (`ip netns pids <ns>` → kill, *then* `netns_del`); a command-signature-scoped orphan `pkill` in `lab down`; idempotent host-`inet blackwall` nft/route clear; and a defensive pre-gate `lab down` in each CI step. **Verified in CI:** all 14 active gates pass with the hardening + pre-gate sweep (safe). Locally, `lab down` reaps orphans and flowspec/rtbh/flow scenarios tear down clean (no leftover netns/orphans/host table). **Gate re-enables deferred** (kept `if: false`): re-enabling the 4 quarantined gates revealed each has its OWN reliability issue, separate from the residue fix: - **flow-live** (#88): in-step hsflowd holds the CI step's stdout pipe past the timeout (still wedges even with teardown fixed). - **deception-resilience**: flaky/borderline under its 600-conn tproxy load — wedged in this PR's run though it passed in #145. - **flowspec / flowspec-auto**: never ran (behind the above); unverified. Each needs dedicated per-gate work (tracked in #137/#88); this PR lands only the systemic residue fix, which is safe and green. Gate: fmt + clippy `--deny warnings` clean, workspace builds, coverage 95.46%, YAML valid.