ci: aggregate SHA-bound release evidence and clear status drift
ci/woodpecker/push/woodpecker Pipeline was successful
Linux strict security lane (G-C) / strict isolation escape matrix (push) Canceled after 0s
CI / fmt + clippy + test (Linux x86_64) (push) Canceled after 0s
CI / schema instance validation + consumers (push) Canceled after 0s
CI / runtime contract (macos-latest) (push) Canceled after 0s
CI / runtime contract (windows-latest) (push) Canceled after 0s
CI / release evidence manifest (push) Canceled after 0s
CI / release candidate artifact (macos-latest) (push) Canceled after 0s
CI / release candidate artifact (ubuntu-latest) (push) Canceled after 0s
ci/woodpecker/push/woodpecker Pipeline was successful
Linux strict security lane (G-C) / strict isolation escape matrix (push) Canceled after 0s
CI / fmt + clippy + test (Linux x86_64) (push) Canceled after 0s
CI / schema instance validation + consumers (push) Canceled after 0s
CI / runtime contract (macos-latest) (push) Canceled after 0s
CI / runtime contract (windows-latest) (push) Canceled after 0s
CI / release evidence manifest (push) Canceled after 0s
CI / release candidate artifact (macos-latest) (push) Canceled after 0s
CI / release candidate artifact (ubuntu-latest) (push) Canceled after 0s
Add tools/ci/release_evidence.py (marqdo.release-gates.v1) plus unit tests: missing/skipped/FAIL/UNVERIFIED and tested_sha mismatches stay DO_NOT_RELEASE, performance may be PASS/WARN, unknown gate names are rejected. The GitHub CI now has a release-evidence job that maps the Linux/schema and native matrix results onto the manifest and uploads it, and both CI systems run the evidence-tool tests. Clear the contradictions the RFC calls out: remove the stale CURRENT.md rows that claimed DNS/TLS/UDP and fs metadata were absent while the same table said they were implemented; correct docs/ci.md to state that the GitHub-hosted Windows/macOS matrix exists (Woodpecker has no such agent); keep the old convergence plan historical; require the evidence manifest in releasing.md; and add a docs-consistency guard for these claims. This is step 6 of the CI/security RFC (B05/B06/B07).
This commit is contained in:
1 parent
d2267e912a
commit
0d7e11b6f1
12 files changed
+483
-36
No files matched your search
@@ -0,0 +1,37 @@
|
||||
# CI evidence tools
|
||||
|
||||
## `strict-evidence.py`
|
||||
|
||||
Writes `marqdo.strict-evidence.v1`, called by `tests/security/run-lane.sh` when
|
||||
`MARQDO_STRICT_EVIDENCE` is set and by the self-hosted security workflow. It
|
||||
records only measured or explicitly supplied facts:
|
||||
|
||||
* `tested_sha` and whether it came from `git rev-parse` or an operator env;
|
||||
* source archive SHA256 (and whether it matched the expected digest);
|
||||
* kernel/uname/cgroup2/container, `uid`/root, bwrap/rustc/cargo/cc versions;
|
||||
* runner kind, image digest, run URL, start/finish and outcome.
|
||||
|
||||
A root-only `PASS` is rejected. `PASS` from a container or a persistent
|
||||
self-hosted runner with production credentials is not valid G-C evidence; see
|
||||
[`docs/security-vm.md`](../../docs/security-vm.md).
|
||||
|
||||
## `release_evidence.py`
|
||||
|
||||
Aggregates per-gate results into `marqdo.release-gates.v1`:
|
||||
|
||||
```bash
|
||||
python3 tools/ci/release_evidence.py \
|
||||
--target-sha "$SHA" \
|
||||
--gate "linux=PASS:$SHA:https://ci.example/run/1" \
|
||||
--gate "windows=PASS:$SHA:https://ci.example/run/2" \
|
||||
--gate "macos=PASS:$SHA:https://ci.example/run/3" \
|
||||
--gate "strict_vm=PASS:$SHA:https://ci.example/run/4" \
|
||||
--gate "performance=WARN:$SHA:https://ci.example/run/5" \
|
||||
--out "docs/evidence/ci/$SHA/manifest.json"
|
||||
```
|
||||
|
||||
A missing gate becomes `UNVERIFIED`. Any non-PASS required gate, a
|
||||
`tested_sha` different from `target_sha`, an unknown gate name or a malformed
|
||||
SHA forces `DO_NOT_RELEASE`. `performance` may be `PASS` or `WARN`.
|
||||
|
||||
Unit tests: `python3 -m unittest discover -s tools/ci -p 'test_*.py'`.
|
||||
Executable
+128
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate per-gate CI evidence into a SHA-bound release manifest.
|
||||
|
||||
Schema: marqdo.release-gates.v1. A missing, skipped, timed-out, failed or
|
||||
SHA-mismatched gate is never PASS. The tool only records supplied facts; it
|
||||
does not query CI and cannot turn an UNVERIFIED gate into a release.
|
||||
|
||||
Gate spec: `name=STATE:tested_sha:run_url` (url may contain colons).
|
||||
Required gates: linux, windows, macos, strict_vm; performance must be PASS or
|
||||
WARN. Any other gate name is rejected so a typo cannot silently disappear.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCHEMA = "marqdo.release-gates.v1"
|
||||
REQUIRED = ("linux", "windows", "macos", "strict_vm")
|
||||
OPTIONAL = ("performance",)
|
||||
STATES = {
|
||||
"linux": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"windows": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"macos": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"strict_vm": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"performance": {"PASS", "WARN", "FAIL", "UNVERIFIED"},
|
||||
}
|
||||
SHA_RE = re.compile(r"^[0-9a-f]{40}$")
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return (
|
||||
datetime.datetime.now(datetime.timezone.utc)
|
||||
.replace(microsecond=0)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def parse_gate(text: str) -> tuple[str, dict]:
|
||||
if "=" not in text:
|
||||
raise ValueError(f"gate must be name=STATE:sha:url, got {text!r}")
|
||||
name, rest = text.split("=", 1)
|
||||
parts = rest.split(":", 2)
|
||||
if len(parts) < 3:
|
||||
raise ValueError(f"gate must be name=STATE:sha:url, got {text!r}")
|
||||
state, tested_sha, run_url = parts
|
||||
if name not in STATES:
|
||||
raise ValueError(f"unknown gate {name!r}; expected {list(STATES)}")
|
||||
if state not in STATES[name]:
|
||||
raise ValueError(f"invalid state {state!r} for gate {name!r}")
|
||||
if tested_sha and not SHA_RE.fullmatch(tested_sha):
|
||||
raise ValueError(f"gate {name!r} tested_sha is not a full 40-hex SHA: {tested_sha!r}")
|
||||
return name, {"state": state, "tested_sha": tested_sha, "run_url": run_url}
|
||||
|
||||
|
||||
def build_manifest(target_sha: str, gates: list[str]) -> dict:
|
||||
if not SHA_RE.fullmatch(target_sha):
|
||||
raise ValueError(f"target SHA must be 40 lowercase hex characters: {target_sha!r}")
|
||||
parsed = {}
|
||||
for spec in gates:
|
||||
name, gate = parse_gate(spec)
|
||||
if name in parsed:
|
||||
raise ValueError(f"duplicate gate {name!r}")
|
||||
parsed[name] = gate
|
||||
|
||||
reasons = []
|
||||
for name in REQUIRED + OPTIONAL:
|
||||
gate = parsed.get(name)
|
||||
if gate is None:
|
||||
parsed[name] = {"state": "UNVERIFIED", "tested_sha": "", "run_url": ""}
|
||||
reasons.append(f"{name}: missing gate")
|
||||
continue
|
||||
if gate["tested_sha"] != target_sha:
|
||||
reasons.append(
|
||||
f"{name}: tested_sha {gate['tested_sha'] or '(missing)'} != target_sha {target_sha}"
|
||||
)
|
||||
elif gate["state"] == "FAIL":
|
||||
reasons.append(f"{name}: FAIL")
|
||||
elif gate["state"] == "UNVERIFIED":
|
||||
reasons.append(f"{name}: UNVERIFIED")
|
||||
elif name in REQUIRED and gate["state"] != "PASS":
|
||||
reasons.append(f"{name}: {gate['state']} is not PASS")
|
||||
elif name == "performance" and gate["state"] not in {"PASS", "WARN"}:
|
||||
reasons.append(f"performance: {gate['state']} is not PASS/WARN")
|
||||
|
||||
decision = "RELEASE_CANDIDATE" if not reasons else "DO_NOT_RELEASE"
|
||||
recorded = utc_now()
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"target_sha": target_sha,
|
||||
"recorded_utc": recorded,
|
||||
"gates": {
|
||||
name: {
|
||||
**gate,
|
||||
"recorded_utc": recorded,
|
||||
}
|
||||
for name, gate in parsed.items()
|
||||
},
|
||||
"release_decision": decision,
|
||||
"reasons": reasons,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--target-sha", required=True)
|
||||
parser.add_argument("--gate", action="append", default=[])
|
||||
parser.add_argument("--out", default="")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
manifest = build_manifest(args.target_sha, args.gate)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
text = json.dumps(manifest, ensure_ascii=False, indent=2) + "\n"
|
||||
if args.out:
|
||||
out = Path(args.out)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(text)
|
||||
sys.stdout.write(text)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,94 @@
|
||||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
MODULE = Path(__file__).with_name("release_evidence.py")
|
||||
spec = importlib.util.spec_from_file_location("release_evidence", MODULE)
|
||||
release_evidence = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(release_evidence)
|
||||
|
||||
SHA = "a" * 40
|
||||
OTHER = "b" * 40
|
||||
|
||||
|
||||
def gate(name: str, state: str, sha: str = SHA, url: str = "https://example.test/run") -> str:
|
||||
return f"{name}={state}:{sha}:{url}"
|
||||
|
||||
|
||||
class ReleaseEvidenceTests(unittest.TestCase):
|
||||
def test_all_required_pass_is_candidate(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("windows", "PASS"),
|
||||
gate("macos", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("performance", "WARN"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "RELEASE_CANDIDATE")
|
||||
self.assertEqual(manifest["reasons"], [])
|
||||
|
||||
def test_missing_strict_vm_is_not_release(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("windows", "PASS"),
|
||||
gate("macos", "PASS"),
|
||||
gate("performance", "PASS"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
self.assertTrue(any("strict_vm" in reason for reason in manifest["reasons"]))
|
||||
|
||||
def test_sha_mismatch_is_not_release(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("windows", "PASS", OTHER),
|
||||
gate("macos", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("performance", "PASS"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
self.assertTrue(any("windows" in reason for reason in manifest["reasons"]))
|
||||
|
||||
def test_failure_is_not_release(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "FAIL"),
|
||||
gate("windows", "PASS"),
|
||||
gate("macos", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("performance", "PASS"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
|
||||
def test_unverified_performance_is_not_release(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("windows", "PASS"),
|
||||
gate("macos", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("performance", "UNVERIFIED"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
|
||||
def test_unknown_gate_and_bad_sha_are_rejected(self) -> None:
|
||||
with self.assertRaises(ValueError):
|
||||
release_evidence.build_manifest(SHA, [gate("solaris", "PASS")])
|
||||
with self.assertRaises(ValueError):
|
||||
release_evidence.build_manifest("not-a-sha", [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user