bench: multi-core sharding scaling spike — GO for Way A (#10) #143

Merged

vxfemboy merged 11 commits from spike/sharding-scale into main 2026-07-29 04:12:44 +00:00

Conversation 0 Commits 11 Files changed 6 +1478 -1

vxfemboy commented 2026-07-28 21:01:50 +00:00

Owner

Copy link

De-risking spike for multi-core sharding (Way A, #10), sub-project 2 part 1. Answers whether the yip receive path scales across cores before building the sharded yipd. No production code touched — a standalone yip-bench example + a benchmark workflow.

Verdict: GO for Way A

Level-1 compute scaling on a 6-physical-core / 12-SMT box (real receive chain: deframe → FEC-decode → AEAD decrypt, per-worker distinct L3-exceeding fixture, topology-aware pinning, start barrier):

N	Gbps	efficiency	avg MHz
1	1.80	1.00	4620
4	6.50	0.90	4428
6	8.99	0.83	4262
12	10.55	0.49	3989

Near-linear across physical cores (0.90 at N=4, 0.83 at N=6 — ~0.90 clock-normalized after backing out the 7.7% laptop DVFS droop). The N=8/12 sub-linearity is SMT sharing (only 6 physical cores; SMT adds ~20-25% for this compute-bound path), not a memory-bandwidth wall. The ~1.2 Gbps single-core ceiling multiplies with physical cores → on fixed-clock server silicon, expect ~physical_cores × 1.2 Gbps. Building the sharded yipd (part 2) is justified.

Level 2 confirms SO_REUSEPORT distributes fairly across N sockets (the inbound steering mechanism Way A relies on); its absolute throughput is blaster-/SMT-/loopback-bound and is not a receiver ceiling.

What's here

• crates/yip-bench/examples/sharding_scale.rs — the spike (level 1 + level 2), core-count-safe sweep, non-fatal pinning.
• crates/yip-bench/sharding-scale.md — findings + verdict + caveats.
• docs/2026-07-26-multicore-sharding-scale-spike-design.md — the spec.
• .forgejo/workflows/benchmarks.yml — runs hotpath + mac_candidates + sharding_scale on the existing runner.

How it was built

9 commits, subagent-driven with per-task review. Level 1 (the gate) got deep scrutiny — pinning verified against live sysfs, fixture confirmed >L3, no scaling-inflation (the N=1 baseline is cache-favored, so efficiency is conservative). A whole-branch opus review confirmed the GO is sound and caught the sweep-panic + framing fixes now applied.

Caveats (in the findings doc)

Laptop, not server silicon: absolute Gbps optimistic (loopback), DVFS is a mobile artifact, and 32-64-core memory/interconnect behavior is unverified here — re-run on target hardware (via the workflow) before finalizing shard-count defaults.

Next after this lands: part 2 — the sharded-yipd architecture spec, whose central unsolved problem is the outbound TUN path (routing each inner packet to the core owning its destination peer).

De-risking spike for multi-core sharding (Way A, #10), sub-project 2 part 1. Answers whether the yip receive path scales across cores **before** building the sharded `yipd`. No production code touched — a standalone `yip-bench` example + a benchmark workflow. ## Verdict: GO for Way A Level-1 compute scaling on a 6-physical-core / 12-SMT box (real receive chain: deframe → FEC-decode → AEAD decrypt, per-worker distinct L3-exceeding fixture, topology-aware pinning, start barrier): | N | Gbps | efficiency | avg MHz | |--:|--:|--:|--:| | 1 | 1.80 | 1.00 | 4620 | | 4 | 6.50 | 0.90 | 4428 | | 6 | 8.99 | **0.83** | 4262 | | 12 | 10.55 | 0.49 | 3989 | **Near-linear across physical cores** (0.90 at N=4, 0.83 at N=6 — ~0.90 clock-normalized after backing out the 7.7% laptop DVFS droop). The N=8/12 sub-linearity is SMT sharing (only 6 physical cores; SMT adds ~20-25% for this compute-bound path), not a memory-bandwidth wall. The ~1.2 Gbps single-core ceiling multiplies with physical cores → on fixed-clock server silicon, expect ~`physical_cores × 1.2 Gbps`. **Building the sharded `yipd` (part 2) is justified.** Level 2 confirms `SO_REUSEPORT` distributes fairly across N sockets (the inbound steering mechanism Way A relies on); its absolute throughput is blaster-/SMT-/loopback-bound and is not a receiver ceiling. ## What's here - `crates/yip-bench/examples/sharding_scale.rs` — the spike (level 1 + level 2), core-count-safe sweep, non-fatal pinning. - `crates/yip-bench/sharding-scale.md` — findings + verdict + caveats. - `docs/2026-07-26-multicore-sharding-scale-spike-design.md` — the spec. - `.forgejo/workflows/benchmarks.yml` — runs hotpath + mac_candidates + sharding_scale on the existing runner. ## How it was built 9 commits, subagent-driven with per-task review. Level 1 (the gate) got deep scrutiny — pinning verified against live sysfs, fixture confirmed >L3, no scaling-inflation (the N=1 baseline is cache-favored, so efficiency is conservative). A whole-branch opus review confirmed the GO is sound and caught the sweep-panic + framing fixes now applied. ## Caveats (in the findings doc) Laptop, not server silicon: absolute Gbps optimistic (loopback), DVFS is a mobile artifact, and 32-64-core memory/interconnect behavior is unverified here — re-run on target hardware (via the workflow) before finalizing shard-count defaults. **Next after this lands:** part 2 — the sharded-`yipd` architecture spec, whose central unsolved problem is the outbound TUN path (routing each inner packet to the core owning its destination peer).

vxfemboy added 10 commits 2026-07-28 21:01:50 +00:00

docs(spec): multi-core sharding scaling spike + benchmark runner (#10, sub-project 2 part 1) ... f7bfdd3e05

De-risk Way A before building it: a standalone sharding_scale bench (level 1
compute-scaling gate + level 2 SO_REUSEPORT delivery) on the 12-core dev box to
measure whether the receive path scales across cores past the ~1.2 Gbps/core
ceiling, plus a benchmarks.yml workflow on the existing Forgejo runner for
ongoing regression tracking. No production yipd change; full sharded architecture
is part 2, specced only if the numbers justify it.

bench(sharding): single-worker receive-chain scaling loop (level 1, N=1) 4354908600

bench(sharding): core-pinned N-worker level-1 sweep + efficiency table 637c05662a

bench(sharding): fix round 1 -- add N=6, per-core avg_MHz column, correct inert-5-tuple comment, warn on topology fallback ... 1c7230990a

Review of the level-1 sweep flagged 4 items:
- add N=6 (last all-physical-core point on this box) so the 4->8 jump
  no longer has to be extrapolated
- sample and print average per-core clock (MHz) over the same timing
  window as the throughput, via a sampler thread on the same start
  barrier, so DVFS clock-droop (mobile APU, package power limit) can be
  told apart from real sharding/SMT contention
- correct the build_fixture comment: the per-payload 5-tuple mutation
  does not exercise the classifier's flow table (sample_inner's fixed
  EF DSCP short-circuits classify() before the flow table is
  consulted); the mutation only guarantees byte-distinctness, which is
  what actually defeats cache residency
- warn (eprintln) when sysfs SMT-topology detection fails and the code
  falls back to naive (0..n) core ordering, so a future box doesn't
  silently regress into the same-physical-core pinning bug fixed in
  the previous commit

bench(sharding): SO_REUSEPORT delivery variant (level 2) ... 1d8d39ae1a

N core-pinned receiver sockets bound to one port with SO_REUSEPORT + a
multi-source-port loopback blaster on a disjoint core set. Worker::receive runs
the real receive half (deframe -> FEC-decode -> open) on each datagram; reports
recv_Gbps, per-socket imbalance ratio, and offered/received drop%.

Result (this box): SO_REUSEPORT distributes fairly (imbalance 1.0-1.95, roughly
even per-socket counts) — the delivery mechanism Way A relies on works. Absolute
recv throughput is blaster-/loopback-bound (senders and receivers share the box;
sender cores shrink as receiver count grows), so level-2 throughput scaling is
confounded and NOT a clean receiver ceiling — level 1 already gave the clean
compute-scaling answer. Level 2's job is confirming even kernel distribution,
which it does.

bench(sharding): fix round 2 -- correct level-2 open()-always-fails framing, remove fabricated brief citation, surface drop% in summary 2a15f8c126

bench(sharding): tighten disjoint-by-index wording in level-2 module doc ... ef137be934

Follow-up to the fix-round-2 commit: the opening module-doc paragraph
still said the blaster group is pinned to a "disjoint core set" without
qualification, which reads inconsistently right above the new SMT-overlap
correction (blaster and receiver DO share physical cores; only core index
is disjoint). Matches the wording already used in level2_sweep's and
MIN_BLASTER_CORES's doc comments.

docs(bench): sharding-scale spike findings — GO for Way A (near-linear across physical cores) 31327a02b8

ci(bench): benchmarks workflow on the existing runner (hotpath, mac_candidates, sharding_scale) 120bbfe52e

fix(bench): clamp sharding sweep to available cores + non-fatal pin failure; correct sharding-scale.md SMT/bandwidth/gate framing (final review) 305126e7c6

vxfemboy added 1 commit 2026-07-29 00:20:58 +00:00

Merge branch 'main' into spike/sharding-scale 7e64b6742c

vxfemboy merged commit 6f2b2d250f into main 2026-07-29 04:12:44 +00:00

vxfemboy referenced this pull request from a commit 2026-07-29 04:12:45 +00:00

Merge pull request 'bench: multi-core sharding scaling spike — GO for Way A (#10)' (#143) from spike/sharding-scale into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

femboy/yip!143

No description provided.