toy-relay-afxdp

Imported from romshark/toy-relay-afxdp (AGENTS.md). Install upstream with npx skills add romshark/toy-relay-afxdp. Copyright stays with the author.

Repository

This repository compares the same UDP relay implemented in Go, C and Rust on one core and 25 GbE hardware. Every implementation must accept the same packet format, pass the same conformance suite and report the same counters.

Preserve comparability and readability. A faster implementation is not useful if it changes protocol behavior or obscures the forwarding path.

Architecture

• wire/ defines the packet format and MAC. Mirror protocol changes in relay-c/ and relay-rs/ in the same change.
• wire/fastsigner/ is experimental and belongs only to cmd/go_afxdp_o/.
• internal/impls/ is the shared deployment registry for benchmarks and acceptance tests.
• Put rig constants in internal/rig/ and keep them consistent with scripts/topology.sh.
• Keep each relay's forwarding loop explicit. Do not hide it behind a shared interface or abstraction.
• Keep measurement and deployment code out of relay packages when possible.
• Keep changes to vendored underlay/ code small and document them in ideas.md.

Code

• Follow standard Go conventions.
• Use the standard testing package directly. Do not add testify outside vendored code.
• Prefer table-driven tests. Use case names as map keys when test order should vary.
• Follow the kernel-style C conventions enforced by relay-c/.clang-tidy.
• Do not add Rust unsafe outside relay-rs/src/xsk/ring.rs. Explain the need before changing the existing unsafe code.
• Prefer the standard library. Propose a new direct dependency before adding it, including what it replaces and its transitive cost.
• After changing go.mod, run go mod tidy and make test.

Generated Files

Change the source and regenerate these files; do not edit them directly:

• underlay/ebpf/sockfilter_bpfel.go and underlay/ebpf/sockfilter_bpfel.o from underlay/ebpf/sockfilter.c.
• relay-rs/bpf/xdp_pass.o from relay-rs/bpf/xdp_pass.c.
• cmd/*/default.pgo with make pgo on the rig.

Build and Test

Use the top-level Makefile:

• make: build all implementations into bin/.
• make test: run Go, C and Rust tests.
• make lint: lint all three languages.
• make acceptance: run conformance tests; requires root and the rig.
• make bench: run benchmarks; requires root and the rig.
• make pgo: regenerate PGO profiles; requires root and the rig.

Run make lint and make test before reporting a change complete. State which checks ran and any that did not.

Rig

• Run NIC operations as root and in the correct network namespace.
• Interrupt relays cleanly. A killed relay can leave the RSS indirection table pinned to queue 0. Restore it with ethtool -X <iface> default.
• Leave NIC, namespace, XDP and RSS state as found.
• Compare performance measurements only within one run. Report the conditions with every result.

Writing

Applies to chat replies, code comments, documentation and commit messages. Write like an engineer reporting findings to another engineer: plain, specific, with facts, non-verbose.

Styles:

• BLUF (bottom line up front), US military staff writing.
• Inverted pyramid, newswire reporting.
• Plain English: Gowers' Plain Words, Cutts' Oxford Guide to Plain English.
• Orwell's six rules from Politics and the English Language.
• Strunk & White: omit needless words, use the active voice.
• IMRaD Results sections: every claim carries a measurement or a citation.
• SRE postmortems: timeline, root cause, action items, no blame, no drama.
• Aviation and maritime logs: one fact per line, interpretation kept separate.
• Unix man pages and RFCs: terse, imperative, everything named exactly.

Rules:

• Prefer simple (near primitive) technical English.
• Lead with the conclusion, then the facts that support it.
• Name concrete things: files, lines, symbols, values, error codes. Write internal/ether/ether.go:142, not "the place where the parser reads it".
• End with what to do about it, or say plainly that nothing needs doing.
• Don't paraphrase code changes, the diff already shows them. Name the file and what it does now. Prose is for the non-obvious what the diff can't show: why, what was left out, what could break.
• Say what was verified and how: "make test passes", "make lint clean", "not run on the rig". Never imply a check that didn't happen. A number quoted without saying where it was measured is not a result.
• Report failures, dead ends and skipped work as plainly as successes.
• Answer at the length the question needs. "Is Cache.Get safe for concurrent use?" is answered by "No, it writes c.entries without holding c.mu." and nothing else. Don't pad a short answer out to look thorough, and don't compress a real explanation into three bullets.
• Stop when the information is delivered. No preamble, no restating the request, no closing summary of what was just said.

Avoid:

• Suspense and buildup: "here's where it gets interesting", "and this is the kicker", "the third one is the most instructive".
• Hype and intensifiers: "not just X, it's THE Y", "load-bearing", "crucial", "powerful", "seamless", "robust", "comprehensive", "deep dive".
• Counting the items instead of naming them. "Two things: X and Y" is "X and Y". A teaser count with no items after it ("three things jumped out at me") is worse.
• Figurative language where a plain word fits: "buys", "drives", "unlocks", "wins", "kills", "shines", "leaves the reader hunting". These are examples, not the whole set. The test is whether the sentence says what literally happens: nothing buys, drives or hunts (unless it literally does). Write "a value receiver prevents mutation", not "a value receiver buys us immutability". Write "the pointer saves no allocation here", not "the pointer buys nothing here". Write "the tests that send it requests over HTTP", not "the tests that drive it over HTTP". Write "the reader cannot tell what is meant", not "it leaves the reader hunting for what was meant".
• Restating a general principle the facts already show. Start with the example.
• Rhetorical questions as headings: "So what does this mean?".
• Filler transitions: "let's dive in", "at the end of the day", "it's worth noting that", "as we can see".
• Praise of the user or of the question: "great question", "you're absolutely right".
• Typographic drama: spaced-out words, all-caps emphasis, exclamation marks, emoji, bold scattered over half the sentences.
• Em-dashes and ", so ..." clauses. Use a colon, a full stop or "which ...".
• Non-ASCII characters where ASCII exists: curly quotes, ellipsis, arrows, non-breaking spaces. Write ', ", ..., -> and a plain space.
• Stating what did not change, stayed, or was already correct.
• Hedging where a check would settle it. Check, then state the answer.
• Apologies and post-mortems after a mistake. Correct it and continue.

Bad:

Here's where it gets interesting: the retry logic isn't just a nice-to-have, it's the load-bearing assumption of the entire sync pipeline. Three things jumped out at me, and the third one is the most instructive yet. [...] And third, and this is the kicker, the dedupe key includes a timestamp, which means retries are never actually deduplicated.

Good:

The sync pipeline's retry logic has three bugs. syncQueue.ts:142 swallows ETIMEDOUT instead of re-queuing the job. backoff.ts:31 caps the delay at 2s, under the 8s p99 LTE reconnect time in bench/network.json. dedupe.ts:77 puts a timestamp in the key, which means retries never deduplicate. All three reproduce in syncQueue_test.ts against the network stub. Fix: re-throw the timeout, raise the cap to 30s and strip the timestamp from the key.

Code comments

Write a comment only if it adds value to the reader by providing non-obvious context information that the reader cannot easily infer from the code itself.

Don't write:

• Comments that repeat the next line: // increment i above i++, // loop over users, // return the result, // error handling.
• Doc comments that only spell out the name: // UserID is the user ID.
• Banners and dividers: // --- helpers ---, // BEGIN, // END.
• History: // added in v2, // fixed #142, // was int64 before. All history is in Git. Comments must only explain current code at hand, unless history is crucial context, e.g. when the code makes no sense without it.
• Commented-out code. Delete it, git has it.
• Comments written for the reviewer of the diff instead of the reader of the code: // Note: now also handles nil, // Changed to a map for speed, // For simplicity we just skip this.
• Comments that teach Go or the standard library: // mu guards concurrent access above a plain mutex, // defer closes the file, // ok is false when the key is missing. Assume that the reader is a seasoned software engineer with good understanding of Go.
• File and line references: // see decoder.go:212. They break as soon as the code moves and nothing checks them. Name the symbol with a doc link instead: [Decoder.Next].
• Positional references: // here and not above, // unlike the check below, // as mentioned earlier. The reader cannot tell what is meant, and the words stop being true when the code moves. Name the symbol, or state the fact on its own.

Do write:

• Why the code works this way, when that isn't clear from reading it: why this order, this algorithm, this lock, what breaks without it.
• Where a value comes from: a constant, a timeout, a buffer size. Name the spec section, benchmark, RFC or issue and link it.
• Rules the types can't state: what the caller must guarantee, what the function assumes.
• Why the obvious approach wasn't used: name it and say what went wrong with it.
• TODO: ... with what to do and what unblocks it, not a bare // TODO.

Form:

• Respect writing.
• Plain sentences, present tense, within the line limit.
• Go doc comments on exported names start with the name. This is the one place where repeating the name is required.
• Put the comment above the code, not after it. Comments after the code are for short notes on struct fields, enum values and table-test rows.
• Reference other code with Go doc links in square brackets: [Parse], [Signer.Verify], [github.com/romshark/toy-relay-afxdp/wire.Header]. C and Rust have no doc links: name the symbol and the file it's in, and keep the name unique enough to grep.
• A doc comment on a symbol says what the symbol does: // Close ends the open requests and is safe to call repeatedly.
• A doc comment on a test explains what scenario is tested: // TestCloseIdempotent tests that repeated Close calls all return nil.

Bad:

// Timeout for the request.
const timeout = 8 * time.Second

// Loop over the items and check each one.
for _, it := range items {
	// Skip if nil.
	if it == nil {
		continue // skip
	}
}

Good:

// timeout covers the p99 LTE reconnect time measured in bench/network.json.
// Below 8s the sync queue retries before the radio is back up.
const timeout = 8 * time.Second

for _, it := range items {
	// [Decoder.Next] emits nil for unknown tags.
	if it == nil {
		continue
	}
}

Git Commits

• Respect writing.
• Title: type: Summary, imperative, capitalized after the prefix, 50 characters or less, no trailing period.
• Types: feat, fix, perf, refactor, test, chore, ci, docs. Suffix the type with ! for a breaking change.
• Wrap the description at 72 characters.
• Imperative mood in the title, never past tense: Add cache, not Added cache or Adds cache. The description uses present tense and describes the code as it's now: Move foo to package buzz, not foo is now in package buzz. No first person, no we.
• Don't paraphrase the code changes, the diff already shows them. The description is for what the diff can't show: why, scope, what was left out. Leave it empty when the title says everything.
• Don't list the files touched, don't restate the title in longer words and don't close with a summary of the commit.
• A commit that bundles several changes lists them as bullets, each with its own conventional prefix. List only the main changes. Tests, docs and call-site updates that come with a change are part of it, not separate bullets:

feat: Add funcFoo <- keep, this is the change
docs: Add docs for funcFoo <- drop
test: Add tests for funcFoo <- drop
refactor: Use funcFoo everywhere <- drop
test: Add benchmark to test funcFoo against funcBar <- drop

• A change that makes the three implementations disagree is not allowed to land alone. Say in the body which of Go, C and Rust moved and that the acceptance suite was run, or that it was not and why.
• No tool attribution or Co-Authored-By trailers.

Bad:

feat: added batching to the relay and updated some files

This commit adds batching to the AF_XDP relay so that it's faster. I changed
cmd/go_afxdp/main.go, internal/xdprelay/xdprelay.go and wire/wire.go to add
the batch and wire it up. Overall this makes the relay faster and cleaner.

Co-Authored-By: Some Tool <tool@example.com>

Good:

perf: Drive SHA-256 assembly without crypto/hmac

Around identical SHA-NI work, crypto/hmac and crypto/sha256 spend about
1,100 instructions per packet where OpenSSL spends 400, unmarshalling the
keyed states and copying digests. The key schedule is computed once at
startup and the assembly called directly.

A second binary rather than a flag, because the signer is a type.
go_afxdp keeps the standard library, which is what the comparison is for.

5.19 -> 6.84 M pps (go_afxdp_ozc, 66-byte frames, tuned, one core).

Good, a fix that has to name what it did not fix:

fix: Reject a leading + in the hex MAC key

u8::from_str_radix accepts a sign, so a WIRE_KEY containing + parsed here
and was rejected by key.Parse on the Go side. The three relays have to
agree on the key: when one accepts what another refuses, the run looks
like a relay silently forwarding nothing.

relay-c has the same hole in hex_decode, plus - and leading whitespace,
and is not fixed here: see review-c.md.

make test and make lint pass. Not run on the rig.