profiling-software-performance

Imported from vrazdalovschi/dotfiles (agents/skills/shared/profiling-software-performance/SKILL.md). Install upstream with npx skills add vrazdalovschi/dotfiles --skill profiling-software-performance. Copyright stays with the author.

Profiling Software Performance

One Rule: Ranked evidence before any optimization. No hotspot list → no change.

Table of Contents

One Rule · The Loop · Pre-Flight Checklist · Quick Triggers · Build Flags · Scenario Fingerprint · Samplers · Metrics · Instrumentation · OS Tuning · Hotspot Table · Cheat Sheet · Anti-Patterns · Hand-off · Reference Index

This skill produces the input that extreme-software-optimization consumes: a ranked hotspot table, a baseline fingerprint, and a hypothesis ledger. It does not change code to make things faster — that's the next skill. Measurement-only instrumentation is allowed when needed to attribute cost; keep it behind an env flag, document it as profiling-only, and do not mix it with optimization changes. Stop at the hand-off.

The Loop

1. DEFINE       → scenario, metric, budget, golden output — declare what "fast" means
2. ENVIRONMENT  → fingerprint host; ASK before tuning kernel/governor
3. BASELINE     → hyperfine / criterion / bench_baseline.sh → p50/p95/p99/p99.9, ops/sec, RSS
4. INSTRUMENT   → spans, histograms, sentinel frames, structured `perf.profile.*` logs
5. PROFILE      → CPU + alloc + I/O + off-CPU + contention samplers
6. INTERPRET    → ranked hotspots + scaling law + hypothesis ledger
7. HAND OFF     → extreme-software-optimization with Opportunity Matrix inputs

Each phase emits a concrete artifact under tests/artifacts/perf/ (or equivalent). No artifact → phase isn't done.

Pre-Flight Checklist

Run through this before touching any profiler — the most common "bad profile" root cause is skipping one of these.

• Scenario is written down — name, inputs, expected output, success metric (p95/throughput/RSS). "It's slow" is not a scenario.
• Build profile is profilable — debug info on, frame pointers on, strip=false (see Build Flags below).
• Fingerprint captured — scripts/env_fingerprint.sh > fingerprint.json (CPU, governor, kernel, toolchain, FS). Without this, the number isn't comparable to anything.
• Baseline recorded — ≥ 20 runs via scripts/bench_baseline.sh; variance within envelope (check with scripts/variance_envelope.py).
• Same-host discipline — no other heavy jobs, no cross-host comparisons, same power profile.
• One lever per run — don't change code and profiler settings together; you lose attribution.
• Kernel tuning asked-and-approved — scripts/profile_init.sh prints the plan and prompts before applying.

If any box is empty, stop and resolve it first. Profiling under a broken precondition wastes the whole session.

Quick Triggers

Build Flags That Unbreak Profilers

Release builds strip the symbols profilers need. Always add a profiling (or release-perf) profile — never profile the size-optimized release binary.

Rust (Cargo.toml):

[profile.release-perf]          # or "profiling"
inherits = "release"
opt-level = 3
lto = "thin"
codegen-units = 1
debug = "line-tables-only"      # cheap, preserves frames; use `true` for deepest unwinds
strip = false

export RUSTFLAGS="-C force-frame-pointers=yes"
cargo build --profile release-perf

Go: go build -gcflags='all=-N -l' kills inlining — only for deep attribution; otherwise leave optimized and rely on pprof DWARF.

C/C++: -O2 -g -fno-omit-frame-pointer -gdwarf-4.

Node: run with --cpu-prof --heap-prof; add NODE_OPTIONS="--inspect=0.0.0.0:9229" for DevTools.

Python: no flag changes — py-spy / memray / scalene attach to the running interpreter.

Scenario Fingerprint (required)

Every profile run must carry this header (see ARTIFACTS.md for the template). Without it the number is not comparable to anything:

CPU model + cores + SMT state + governor + no_turbo
RAM size + swap
Storage + filesystem + mount options
Kernel version
Toolchain version (rustc, go, node, python)
Build profile (opt-level, lto, codegen-units, debug, panic)
Workload isolation (taskset, cgroup, rch, bare)
Cold vs warm cache
Run ID + git SHA + timestamp

The bundled scripts/bench_baseline.sh captures this into fingerprint.json per run — copy that pattern into the project under measurement.

Sampler Picks (default first, others only if first fails)

Full install + flags per language: LANGUAGE-GUIDES.md.

Metrics You Report (not optional)

latency:    p50, p95, p99, p99.9, p99.99, max  (flag p99.9/p99.99 as conservative if samples < 1000)
throughput: ops/sec + bytes/sec
memory:     peak RSS (`/usr/bin/time -v`), heap high-water (allocator), PSS via smaps_rollup
cpu:        avg process CPU% + per-core
scaling:    p95 ratio vs 1-unit baseline across 1/10/50/100/500/1000

Variance envelope (same-host):

• ≤ 10% p95 drift → noise, ignore
• > 10% → investigate
• > 20% or 3 consecutive > 10% → escalate

Instrumentation Shortlist

Add these before profiling, keep them behind an env flag:

Structured logging contract (scripts/render_hotspot_table.py accepts either naming):

perf.profile.run_start             { scenario, rust_version, hardware }
perf.profile.sample_collected      { sample_count, duration_sec }
perf.profile.span_summary          { span | span_name, cumulative_us, count,
                                     p50_us | p50, p95_us | p95,
                                     category, evidence }
perf.profile.hypothesis_evaluated  { name, verdict: "supports"|"rejects", evidence }
perf.profile.run_complete          { duration_sec, artifacts_written }

Full guidance: INSTRUMENTATION.md.

OS Tuning (ASK FIRST)

These kernel knobs visibly affect numbers but require sudo and change global state. Always present the list and ask "apply these?" before running. Revert commands are in OS-TUNING.md.

Minimum set for Linux profiling accuracy:

sudo sysctl -w kernel.perf_event_paranoid=-1   # or 1 for user-space profiling
sudo sysctl -w kernel.kptr_restrict=0          # resolve kernel symbols in stacks
sudo sysctl -w kernel.nmi_watchdog=0           # frees a PMU counter
sudo cpupower frequency-set -g performance     # no P-state jitter
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
taskset -c 2,3 ./bin                           # pin to isolated cores (needs isolcpus= boot flag)
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches   # cold-cache runs only

macOS: SIP cripples dtrace; prefer samply, xctrace record, sample <pid>, spindump. Full matrix + restore commands in OS-TUNING.md.

Hotspot Table (the hand-off artifact)

| Rank | Location           | Metric        | Value   | Category | Evidence            |
|------|--------------------|---------------|---------|----------|---------------------|
| 1    | record_decode      | cumulative    | 841ms   | CPU      | flame_1.svg:0.23    |
| 2    | parser_ast_churn   | cumulative    | 627ms   | CPU/alloc| dhat-heap.json#L412 |
| 3    | flush_async_commits| wait          | 82ms    | I/O      | span_summary.json   |
| 4    | btree_seek         | count*avg     | 3063×x  | CPU      | perf.data           |
| 5    | page_lock_wait     | cumulative    | 0ns     | lock     | off-cpu.svg         |

Each row must cite an artifact path — no unreferenced claims.

Also ship a hypothesis ledger — name each candidate explanation, mark supports / rejects with evidence:

coalescer batching : rejects — write_message_batch dominates at 25.4s vs flush 0.87s
fsync per msg       : rejects — wbq_flush cumulative 0 vs flush_async_commits 873ms
file layout         : supports — per-msg archive burst dominates profile (25.4s)
lock thrash         : rejects — scaling is sublinear through batch-1000

Workflow Cheat Sheet

# Rust
cargo bench -p <crate> --bench <name> -- --save-baseline before
samply record --save-only -o cpu.json -- ./target/release-perf/bin
heaptrack ./target/release-perf/bin

# Go
go test -bench=. -cpuprofile=cpu.p -memprofile=mem.p -benchmem
go tool pprof -http=:8080 cpu.p
go tool trace trace.out

# Node
node --cpu-prof --cpu-prof-dir=./prof app.js
clinic doctor -- node app.js
autocannon -c 100 -d 30 http://localhost:3000

# Python
py-spy record -o flame.svg --pid $PID --duration 30
memray run --live-remote prog.py
scalene --cpu --memory prog.py

# Wall-clock A/B
hyperfine --warmup 3 --runs 20 --export-json r.json 'cmdA' 'cmdB'

Anti-Patterns

Hand-off

When the hotspot table + hypothesis ledger + baseline fingerprint are in place, stop.

Say to the user:

Baseline captured. Top 5 hotspots ranked with evidence in tests/artifacts/perf/<run-id>/. Ready for extreme-software-optimization to score targets (Impact × Confidence / Effort ≥ 2.0) and apply one lever at a time.

That's the contract.

References

Core methodology & artifacts

Operator library & prompts

Reading & interpretation

Systems & tools

CI, skill hand-off, and creative expansion

Scripts