CI / fmt + clippy + test (Linux x86_64) (push) Waiting to run
CI / schema instance validation + consumers (push) Waiting to run
CI / runtime contract (macos-latest) (push) Waiting to run
CI / runtime contract (windows-latest) (push) Waiting to run
Linux strict security lane (G-C) / strict isolation escape matrix (push) Waiting to run
ci/woodpecker/push/woodpecker Pipeline was canceled
Replace the three-run baseline profile with tools/perf/runner.py: one fixed release binary, per-child rusage via posix_spawn/wait4, 5 warmups and 20 samples by default, SHA-derived interleaved order, p50/p95/MAD, CPU user/sys and peak RSS. Every warmup output is normalized and compared with a committed wire digest before timing; a mismatch is CORRECTNESS_FAIL, never a fast sample. Short or unapproved-host runs are SMOKE, and baselines from another platform_id/profile are refused. Add workloads.json, expected digests for nine existing workloads, a dedicated fixed-node workflow (manual, self-hosted, requires --cpu-list), and docs/performance.md. bench.py stays as a compatibility shim. This is step 5 of the CI/security RFC (B08).
46 lines
2.1 KiB
Markdown
46 lines
2.1 KiB
Markdown
# Performance sampler (marqdo.perf.v2)
|
|
|
|
`runner.py` measures a fixed release binary and refuses to report timings for a
|
|
workload whose wire output did not match the committed expected digest.
|
|
|
|
## Rules
|
|
|
|
* Build one release binary with `cargo build --release --locked -p marqdo-cli`
|
|
and keep it fixed for the whole run. The report records its SHA256, the
|
|
`Cargo.lock` digest, `rustc -vV`, source SHA and profile.
|
|
* Correctness comes first: every warmup run is parsed and hashed. A mismatch
|
|
is `CORRECTNESS_FAIL`; its faster wall time is never reported.
|
|
* Release evidence needs at least 5 warmups and 20 samples per workload with
|
|
`--approved-node` on a dedicated, stable performance node. Shorter runs are
|
|
`SMOKE` and must not be cited as a release gate.
|
|
* Do not compare timings across different `platform_id`s, CPU models, kernels,
|
|
toolchains or load conditions. `--baseline` refuses a different
|
|
`platform_id`/profile.
|
|
* Avoid TCG/QEMU for final performance numbers; use native or KVM with fixed
|
|
CPU affinity (`--cpu-list`) and an administrator-approved governor.
|
|
* The suggested investigation thresholds are p50 +15% and peak RSS +20%; they
|
|
are not automatic rollback rules. Investigate before freezing a hard gate.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# Record/review expected wire digests after an intentional behaviour change.
|
|
python3 tools/perf/runner.py --record-expected --warmups 1 --samples 1 \
|
|
--allow-smoke --only hello-check
|
|
|
|
# Non-release smoke run on the development host.
|
|
python3 tools/perf/runner.py --warmups 1 --samples 3 --allow-smoke \
|
|
--platform-id local-unapproved --out /tmp/marqdo-perf-smoke.json
|
|
|
|
# Release measurement on the dedicated node.
|
|
python3 tools/perf/runner.py --warmups 5 --samples 20 --approved-node \
|
|
--platform-id linux-x86_64-fixed-node-01 --cpu-list 2,3 \
|
|
--out perf/2026-11-21.json
|
|
```
|
|
|
|
Expected digests live in `expected/<workload>.json` and are versioned with the
|
|
source. Running `--record-expected` rewrites them: inspect the diff and commit
|
|
it only when the output change is the intended one.
|
|
|
|
`bench.py` remains only as a compatibility shim.
|