fix(ci): install iproute2/ping/tcpdump in netns jobs; surface sshd failure #156

Open

vxfemboy wants to merge 6 commits from fix/ci-missing-netns-tooling into main

pull from: fix/ci-missing-netns-tooling

merge into: femboy:main

femboy:main

femboy:nop/time-0.x-lockfile

femboy:nop/rustls-0.x-lockfile

femboy:renovate/actions-checkout-7.x

femboy:renovate/socket2-0.x

femboy:deps/remaining-trivial

femboy:fix/deps-rng-ecosystem

femboy:fix/deps-hashes-0.11

femboy:fix/deps-ml-kem-0.3

femboy:fix/deps-boring-5

femboy:fix/deps-p256-0.14

femboy:fix/deps-rcgen-0.14

femboy:spike/sharding-scale

femboy:fix/ci-forgejo-build-deps

femboy:refactor/peer-manager-module-split

femboy:feat/antidpi-tls-mimicry-3c2

femboy:fix/replay-window-before-auth

femboy:docs/fox-theme-agpl-hygiene

femboy:feat/throughput-batched-io

femboy:feat/antidpi-3c1

femboy:feat/antidpi-3b

femboy:docs/user-guide-and-config

femboy:feat/antidpi-3a

femboy:feat/discovery-2c

femboy:feat/rendezvous-2b

femboy:feat/uring-gso-fate-tags

femboy:chore/uring-docs-cleanup

femboy:feat/l2-tap

femboy:chore/cursor-review-followups

femboy:fix/uring-enobufs

femboy:fix/orchestration-spec-scope

femboy:io-uring-busy-poll

femboy:ci-guard-netns-tests

femboy:data-plane-followups

femboy:data-plane-feedback-arq

femboy:data-plane-throughput-pass

Conversation 0 Commits 6 Files changed 3 +72 -8

vxfemboy commented 2026-08-02 17:44:33 +00:00

Owner

Copy link

Run 277's three integration failures were all repo-side, not runner faults. Each stayed failed across every retry with identical log length; bench-smoke and device-tests passed on attempt 1 on the same privileged runner in the same run.

Root cause

catthehacker/ubuntu:act-22.04 ships none of ip, tc, ping, wg, iperf3, tcpdump. Verified against the image, not inferred — and its PATH does include /usr/sbin and /sbin, which rules out a PATH problem. (ssh, scp, sshd, jq, openssl are present.)

Three jobs drove ip without declaring iproute2:

• netns-tunnel-test — died at run-netns-tunnel.sh:86
• dpi-undetectability — died at run-ndpi-oracle.sh:129
• netem-comparison — died at run-yip-netem.sh:99

netem-comparison is the interesting one. It used to work because wireguard-tools lists iproute2 and iputils-ping as Recommends — not Depends. That resolution changed between 2026-08-01 and 2026-08-02: run 277 logs Setting up iproute2, run 290 with the identical apt line does not. A CI job that only works because of a transitive Recommends is a job waiting to break, and this one did, mid-fix.

Every job that needs a tool now declares it.

Also

start_sshd routed sshd's diagnostics to a file via -E "$logfile", which the cleanup trap then removed — so a failed start surfaced as nothing but harness failed. It now prints sshd's log and exit status before propagating, guarded on [ -s ] rather than cat's exit status so an empty log can't recreate the same blank wall.

Verified in run 290 (first two fixes)

• dpi-undetectability — passes, for the first time in its recorded history
• netns-tunnel-test — got past ip; ping_across_yipd_tunnel and ping_across_yipd_tunnel_under_loss now pass, both for the first time

Removing the mask surfaced one real defect, left unfixed here because it is a yip behavior question, not a CI one: arq_recovers_bulk_loss fails with delivery 81.5% < 98% under 5% bulk loss. ARQ does fire (72 retransmits), so the machinery works — FEC+ARQ just doesn't recover to the asserted floor. Needs a decision on whether the 98% floor or the recovery behavior is wrong.

Noted, not fixed: the runner kernel (6.18.35_1) has no wireguard module, so the WG comparison columns stay skipped. The workflow already tolerates that (modprobe wireguard || echo ...).

Run 277's three integration failures were all repo-side, not runner faults. Each stayed failed across every retry with identical log length; `bench-smoke` and `device-tests` passed on attempt 1 on the same `privileged` runner in the same run. ## Root cause `catthehacker/ubuntu:act-22.04` ships **none** of `ip`, `tc`, `ping`, `wg`, `iperf3`, `tcpdump`. Verified against the image, not inferred — and its `PATH` does include `/usr/sbin` and `/sbin`, which rules out a PATH problem. (`ssh`, `scp`, `sshd`, `jq`, `openssl` are present.) Three jobs drove `ip` without declaring iproute2: - `netns-tunnel-test` — died at `run-netns-tunnel.sh:86` - `dpi-undetectability` — died at `run-ndpi-oracle.sh:129` - `netem-comparison` — died at `run-yip-netem.sh:99` `netem-comparison` is the interesting one. It *used* to work because `wireguard-tools` lists `iproute2` and `iputils-ping` as **Recommends** — not Depends. That resolution changed between 2026-08-01 and 2026-08-02: run 277 logs `Setting up iproute2`, run 290 with the identical apt line does not. A CI job that only works because of a transitive Recommends is a job waiting to break, and this one did, mid-fix. Every job that needs a tool now declares it. ## Also `start_sshd` routed sshd's diagnostics to a file via `-E "$logfile"`, which the cleanup trap then removed — so a failed start surfaced as nothing but `harness failed`. It now prints sshd's log and exit status before propagating, guarded on `[ -s ]` rather than `cat`'s exit status so an empty log can't recreate the same blank wall. ## Verified in run 290 (first two fixes) - `dpi-undetectability` — **passes**, for the first time in its recorded history - `netns-tunnel-test` — got past `ip`; `ping_across_yipd_tunnel` and `ping_across_yipd_tunnel_under_loss` now pass, both for the first time Removing the mask surfaced one real defect, left unfixed here because it is a yip behavior question, not a CI one: `arq_recovers_bulk_loss` fails with `delivery 81.5% < 98%` under 5% bulk loss. ARQ does fire (72 retransmits), so the machinery works — FEC+ARQ just doesn't recover to the asserted floor. Needs a decision on whether the 98% floor or the recovery behavior is wrong. Noted, not fixed: the runner kernel (`6.18.35_1`) has no wireguard module, so the WG comparison columns stay skipped. The workflow already tolerates that (`modprobe wireguard || echo ...`).

vxfemboy added 1 commit 2026-08-02 17:44:33 +00:00

fix(ci): install iproute2/ping/tcpdump in netns jobs; surface sshd failure ... a207ba15e0

Three jobs in run 277 failed and stayed failed across every retry (2-3
attempts each, byte-identical logs) — deterministic environment defects,
not runner flakes. bench-smoke and device-tests passed first try on the
same `privileged` runner in the same run.

netns-tunnel-test and dpi-undetectability died at
`ip: command not found` (run-netns-tunnel.sh:86, run-ndpi-oracle.sh:129).
Neither job installed iproute2; both scripts drive `ip` heavily (42x and
33x) plus ping, and the oracle also shells out to tcpdump. Only
netem-comparison had `ip` at all, and only as a transitive dependency of
wireguard-tools — an accident, not a declaration. Declare them.

netem-comparison failed differently: sshd would not start, and
start_sshd's `-E "$logfile"` had put the reason in a file the cleanup
trap deleted, so the harness surfaced nothing but "harness failed".
Dump sshd's log and exit status before propagating.

Unrelated but noted while reading the logs: the runner's kernel
(6.18.35_1) has no wireguard module, so the WG comparison columns stay
skipped. The workflow already tolerates that.

vxfemboy added 1 commit 2026-08-02 18:18:52 +00:00

fix(ci): declare iproute2/ping/iperf3 in netem-comparison too ... ec3ecd6f31

Run 290 proved the earlier reasoning right about the cause and wrong
about the blast radius. netem-comparison had `ip` only because
wireguard-tools lists iproute2 and iputils-ping as *Recommends* — not
depends. That resolution changed between 2026-08-01 and 2026-08-02: run
277 logs "Setting up iproute2", run 290 (identical apt line, one day
later) does not, and the job died at run-yip-netem.sh:99 with
`ip: command not found` — one test earlier than before.

Verified against the image rather than inferred: catthehacker/ubuntu:
act-22.04 ships no ip, tc, ping, wg, iperf3 or tcpdump, and its PATH does
include /usr/sbin and /sbin, so this is a missing package and never a
PATH problem. ssh/scp/sshd/jq/openssl are present.

The five harness scripts this job drives (run-yip-netem, run-compare,
run-scp-compare, run-fec-compare, run-iperf-compare) need ip+tc, ping and
iperf3 between them. Declare all three rather than inherit them.

Also: guard the sshd diagnostic on `[ -s ]` rather than cat's exit
status. An existing-but-empty log would make cat succeed silently and
reproduce the blank wall this branch exists to remove.

vxfemboy added 1 commit 2026-08-02 18:49:44 +00:00

fix(ci): create /run/sshd before starting sshd in the scp harness ... 01b307f943

The diagnostic added earlier in this branch immediately paid for itself:

  [error] sshd failed to start in netns yipA (exit 255)
  [error] sshd log follows:
  Missing privilege separation directory: /run/sshd

sshd will not start without its privsep directory. The openssh-server
postinst normally creates it, but the runner image ships sshd without
having run it and /run is a fresh tmpfs per container, so it never
existed. Create it in the script rather than the workflow so a standalone
run gets the same treatment.

Two more tests reached and passed on the way to this one, both for the
first time in CI: yip_tunnel_under_netem_loss and
comparison_under_netem_loss (the latter correctly SKIPping its WG column,
since the runner kernel has no wireguard module).

vxfemboy added 1 commit 2026-08-02 19:18:13 +00:00

fix(ci): install tcpdump for netns-tunnel-test; correct 'preinstalled' claim ... 41c2b3a8dc

flowshape_not_obviously_constant died on `exec of "tcpdump" failed: No
such file or directory`. The tshark step's comment asserted tcpdump was
preinstalled on the runner image; it is not, and that wrong belief is
why it was never declared. Ten of this job's ~30 harness scripts capture
with tcpdump (flowshape, rekey, rekey-relay, replay-hijack, path-switch,
relay-tls, port-plausibility, and the QUIC/TLS/nDPI mimicry oracles).

Also: arq_recovers_bulk_loss passed this run at 99.4% delivery after
failing the prior run at 81.5% against its 98% floor. It is marginal
under randomized netem loss, not deterministically broken.

vxfemboy added 1 commit 2026-08-02 21:11:49 +00:00

Merge main into fix/ci-missing-netns-tooling ... 93278d59d4

f503ca6 on main fixed the same netem-comparison defect independently.
Kept main's step and comment verbatim -- its note that installing on the
runner-priv container does nothing (act_runner maps `privileged` to
docker://catthehacker/ubuntu:act-22.04, so steps run inside the image) is
the part that is easiest to get wrong -- and folded iperf3 into its
package list rather than keeping a second, competing install step.

iperf3 is not cosmetic here: without it run-fec-compare.sh and
run-iperf-compare.sh SKIP their columns and the job reports green while
measuring less than it claims.

vxfemboy added 1 commit 2026-08-02 21:46:25 +00:00

fix(ci): set YIP_SKIP_PRIVILEGED_TESTS in the coverage job ... 3cf588fd12

Coverage has failed on every PR since #149 with four netem_bench
failures -- yip_tunnel_under_netem_loss, comparison_under_netem_loss,
scp_throughput_comparison, udp_loss_recovery_comparison -- all dying on
`ip: command not found`.

The root gate was never missing. netem_bench.rs:31 already emits
`SKIP <test>: needs root` and integration.yml already greps for exactly
that string. The gate keys off is_root(), which honours
YIP_SKIP_PRIVILEGED_TESTS -- and ci.yml:40 has set it for build-test since
the gate was written. coverage.yml simply never opted in, so is_root()
saw uid 0 in the container and ran netns tests the job was never meant to
run (its own step is named 'netns-gated').

One env var, no test-script changes: the privileged jobs leave it unset,
so they still run these for real and their honesty guards still enforce
it.

Some checks failed

Hide checks

Integration (privileged) / bench-smoke (pull_request) Successful in 3m12s

Details

Integration (privileged) / device-tests (pull_request) Successful in 43s

Details

CI / shear (pull_request) Successful in 4m0s

Details

CI / deny (pull_request) Successful in 6m4s

Details

CI / build-test (pull_request) Successful in 11m24s

Details

Coverage / coverage (pull_request) Successful in 12m1s

Details

Integration (privileged) / netem-comparison (pull_request) Successful in 14m36s

Details

Integration (privileged) / dpi-undetectability (pull_request) Successful in 12m10s

Details

Integration (privileged) / netns-tunnel-test (pull_request) Failing after 30m51s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/ci-missing-netns-tooling:fix/ci-missing-netns-tooling

git switch fix/ci-missing-netns-tooling

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff fix/ci-missing-netns-tooling

git switch fix/ci-missing-netns-tooling

git rebase main

git switch main

git merge --ff-only fix/ci-missing-netns-tooling

git switch fix/ci-missing-netns-tooling

git rebase main

git switch main

git merge --no-ff fix/ci-missing-netns-tooling

git switch main

git merge --squash fix/ci-missing-netns-tooling

git switch main

git merge --ff-only fix/ci-missing-netns-tooling

git switch main

git merge fix/ci-missing-netns-tooling

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

femboy/yip!156

No description provided.