feat(lab): dump node logs on first step failure (self-diagnosing gates) #251

Merged

vxfemboy merged 1 commit from feat/lab-dump-node-stderr-on-timeout into main

2026-08-02 17:52:06 +00:00

vxfemboy commented

2026-08-02 17:33:55 +00:00

Owner

Makes a failing lab gate show why a node never came up, instead of only an opaque probe timed out.

Motivation. deception-nft fails in CI with not ok 1 - wait port-open:61000 timed out — the engine (serves_deception_banner) never binds :61000 in the victim netns. But the engine's own panic/error goes to a per-node run.log that CI never prints, so the actual cause is invisible. (Passes on bare metal, so it's a container-netns gap — but we can't confirm which .expect() dies without the node's output.)

Change. On the first step failure, the runner dumps every *.log under the run's scratch dir (/run/blackwall-lab/<id>/) as TAP # comment lines — last 80 lines each, earlier lines elided with a count. Emitted on stdout inline under the failing step, so CI's existing cat lab-gate.log shows it. Best-effort and read-only: a missing dir or unreadable file yields nothing (never fails the run), JUnit output is unchanged, and passing runs never trigger it.

Tests. format_log_dump (TAP-comment prefixing + tail/elision) and collect_run_logs (nested .log discovery, sorted, non-.log ignored). cargo test -p blackwall-lab 62/62; clippy --deny warnings and fmt clean.

Follow-up (separate): with this landed, the next deception-nft CI failure will show the engine's panic, which pinpoints the container-netns fix.

Makes a failing lab gate show **why** a node never came up, instead of only an opaque `probe timed out`. **Motivation.** `deception-nft` fails in CI with `not ok 1 - wait port-open:61000 timed out` — the engine (`serves_deception_banner`) never binds :61000 in the victim netns. But the engine's own panic/error goes to a per-node `run.log` that CI never prints, so the actual cause is invisible. (Passes on bare metal, so it's a container-netns gap — but we can't confirm which `.expect()` dies without the node's output.) **Change.** On the **first** step failure, the runner dumps every `*.log` under the run's scratch dir (`/run/blackwall-lab/<id>/`) as TAP `#` comment lines — last 80 lines each, earlier lines elided with a count. Emitted on stdout inline under the failing step, so CI's existing `cat lab-gate.log` shows it. Best-effort and read-only: a missing dir or unreadable file yields nothing (never fails the run), JUnit output is unchanged, and passing runs never trigger it. **Tests.** `format_log_dump` (TAP-comment prefixing + tail/elision) and `collect_run_logs` (nested `.log` discovery, sorted, non-`.log` ignored). `cargo test -p blackwall-lab` 62/62; clippy `--deny warnings` and fmt clean. Follow-up (separate): with this landed, the next `deception-nft` CI failure will show the engine's panic, which pinpoints the container-netns fix.

vxfemboy added 1 commit

2026-08-02 17:33:55 +00:00

feat(lab): dump node logs on first step failure

CI / lab (pull_request) Has been cancelled

Details

CI / check (pull_request) Has been cancelled

Details

33a3dc4e3f

A `wait port-open` timeout previously surfaced only as an opaque
"probe timed out" — the failing node's own output (e.g. an engine
panic before it could bind :61000) went to a per-node run.log that CI
never showed. On the first step failure, dump every *.log under the
run's scratch dir as TAP # comments (last 80 lines each) so CI shows
why a node never came up. Best-effort, read-only; passing runs and the
JUnit output are unaffected.

vxfemboy force-pushed feat/lab-dump-node-stderr-on-timeout from 33a3dc4e3f

CI / lab (pull_request) Has been cancelled

Details

CI / check (pull_request) Has been cancelled

Details

to 7fe11a6bdf

CI / check (pull_request) Successful in 15m0s

Details

CI / lab (pull_request) Failing after 19m13s

Details

2026-08-02 17:35:56 +00:00

Compare