fix(lab): select hsflowd veth by global-scope address (flow-live root cause) #259

Merged

vxfemboy merged 2 commits from fix/hsflowd-veth-select into main

2026-08-03 22:54:32 +00:00

vxfemboy commented

2026-08-03 20:27:29 +00:00

Owner

Why flow-live never fired

flow-sflow-live drives real hsflowd (mod_pcap) sampling the victim veth
into real sFlow v5, then the production collector + detector must fire. It has
been failing (no live volumetric detection fired, empty hsflowd.log) and was
quarantined with continue-on-error (#88) as "environmental". It isn't.

spawn_hsflowd discovered the veth as the first non-lo/ifb link from
ip link show. A fresh netns on the CI runner auto-creates tunnel stubs
(sit0/tunl0/ip6tnl0) that sort ahead of the veth and carry no address.
mod_pcap on a stub samples zero packets, hsflowd exports no flow samples, the
detector never opens, and the file-present probe times out.

This is the same bug family already fixed twice this cycle:

trafficgen (io/mod.rs, #256/#258) - select by IPv4 address
deception (tests/interop.rs, #254) - select by scope global

The tell: in this same scenario step 1 (trafficgen send) passes (fixed
selector) while step 2 (hsflowd) fails (unfixed selector). Same netns, same
stubs, opposite outcomes.

Change

spawn_hsflowd now selects the veth via ip -o addr show scope global,
skipping address-less stubs. scope global (not -4) also covers a v6-only
netns, avoiding the syncookie-v6 regression -4 caused last time.
Writes a hsflowd-iface.log diagnostic (chosen iface + the addresses picked
from) under the node scratch dir. The failure log-dump globs *.log, so the
next run shows exactly which iface mod_pcap sampled - making a wrong-iface
pick distinguishable from a hsflowd-not-exporting problem.

Quarantine stays this PR

flow-live keeps continue-on-error: true. Lab gates are sequential fail-stop;
a hard failure there would skip the whole tail (deception-resilience,
syncookie x2, rtbh, bird-gen, flowspec x2, armed x3, xdp x3). Un-quarantine only
after a lab run is observed passing flow-live.

Local: cargo build/clippy -p blackwall-lab clean. The gate itself needs the
privileged lab runner (root + netns + hsflowd).

## Why flow-live never fired `flow-sflow-live` drives **real** hsflowd (`mod_pcap`) sampling the victim veth into real sFlow v5, then the production collector + detector must fire. It has been failing (`no live volumetric detection fired`, empty `hsflowd.log`) and was quarantined with `continue-on-error` (#88) as "environmental". It isn't. `spawn_hsflowd` discovered the veth as the **first non-`lo`/`ifb` link** from `ip link show`. A fresh netns on the CI runner auto-creates tunnel stubs (`sit0`/`tunl0`/`ip6tnl0`) that sort ahead of the veth and carry **no address**. `mod_pcap` on a stub samples zero packets, hsflowd exports no flow samples, the detector never opens, and the `file-present` probe times out. This is the **same bug family** already fixed twice this cycle: - trafficgen (`io/mod.rs`, #256/#258) - select by IPv4 address - deception (`tests/interop.rs`, #254) - select by `scope global` The tell: in this same scenario **step 1 (`trafficgen send`) passes** (fixed selector) while **step 2 (hsflowd) fails** (unfixed selector). Same netns, same stubs, opposite outcomes. ## Change - `spawn_hsflowd` now selects the veth via `ip -o addr show scope global`, skipping address-less stubs. `scope global` (not `-4`) also covers a v6-only netns, avoiding the syncookie-v6 regression `-4` caused last time. - Writes a `hsflowd-iface.log` diagnostic (chosen iface + the addresses picked from) under the node scratch dir. The failure log-dump globs `*.log`, so the next run shows exactly which iface `mod_pcap` sampled - making a wrong-iface pick distinguishable from a hsflowd-not-exporting problem. ## Quarantine stays this PR flow-live keeps `continue-on-error: true`. Lab gates are sequential fail-stop; a hard failure there would skip the whole tail (deception-resilience, syncookie x2, rtbh, bird-gen, flowspec x2, armed x3, xdp x3). Un-quarantine only after a lab run is observed passing flow-live. Local: `cargo build`/`clippy -p blackwall-lab` clean. The gate itself needs the privileged lab runner (root + netns + hsflowd).

vxfemboy added 1 commit

2026-08-03 20:27:29 +00:00

fix(lab): select hsflowd veth by global-scope address, not link order

CI / check (pull_request) Failing after 3m10s

Details

CI / lab (pull_request) Has been skipped

Details

fdcda34579

The flow-sflow-live gate's hsflowd samples the node veth via mod_pcap but
picked it as the first non-lo/ifb link from `ip link show`. A fresh netns on
the CI runner auto-creates tunnel stubs (sit0/tunl0/ip6tnl0) that sort ahead
of the veth and carry no address; mod_pcap on a stub samples zero packets, so
hsflowd exports no flow samples and the live detector never fires (empty
hsflowd.log, the file-present probe times out).

Select by `ip -o addr show scope global` instead, the same fix already applied
to the trafficgen (#256/#258) and deception (#254) selectors; `scope global`
(not `-4`) also covers a v6-only netns. Also write a hsflowd-iface.log
diagnostic (chosen iface + addresses) under the node scratch dir so a future
"no live detection" can be told apart from a wrong-iface pick.

flow-live stays continue-on-error (#88) until a lab run confirms the fix.

vxfemboy added 1 commit

2026-08-03 20:47:09 +00:00

style(lab): rustfmt wrap hsflowd ip-addr args array

CI / check (pull_request) Successful in 9m56s

Details

CI / lab (pull_request) Failing after 9m11s

Details

84aa57e222

~~vxfemboy referenced this pull request 2026-08-03 21:48:37 +00:00~~

perf(deception): sample per-packet reply-send drop log (log-flood hygiene) #260

vxfemboy referenced this pull request

2026-08-03 22:36:43 +00:00