release(3.0.0-beta.1): product version migration, beta gates, archive tooling and developer docs
ci/woodpecker/push/main Pipeline was successful
ci/woodpecker/manual/performance Pipeline was successful
Linux strict security lane (G-C) / strict isolation escape matrix (push) Canceled after 0s
CI / fmt + clippy + test (Linux x86_64) (push) Canceled after 0s
CI / schema instance validation + consumers (push) Canceled after 0s
CI / runtime contract (macos-latest) (push) Canceled after 0s
CI / runtime contract (windows-latest) (push) Canceled after 0s
CI / release evidence manifest (push) Canceled after 0s
CI / release candidate artifact (macos-latest) (push) Canceled after 0s
CI / release candidate artifact (ubuntu-latest) (push) Canceled after 0s
ci/woodpecker/push/main Pipeline was successful
ci/woodpecker/manual/performance Pipeline was successful
Linux strict security lane (G-C) / strict isolation escape matrix (push) Canceled after 0s
CI / fmt + clippy + test (Linux x86_64) (push) Canceled after 0s
CI / schema instance validation + consumers (push) Canceled after 0s
CI / runtime contract (macos-latest) (push) Canceled after 0s
CI / runtime contract (windows-latest) (push) Canceled after 0s
CI / release evidence manifest (push) Canceled after 0s
CI / release candidate artifact (macos-latest) (push) Canceled after 0s
CI / release candidate artifact (ubuntu-latest) (push) Canceled after 0s
REL-01: workspace/CLI/Analysis product version 3.0.0-beta.1 with updated Cargo.lock; project.marqdo is enforced against the running core with package.marqdo_range_invalid/package.marqdo_incompatible fail-closed diagnostics; Tool SemVer prerelease negotiation tests; init template and examples use the beta range. Verify init derives the project name from the target directory instead of falling back to 'example'. REL-02: beta support matrix is Linux x86_64 only; macOS/Windows are explicitly experimental/unclaimed; strict policy and G4 Beta-only WARN are documented for the release evidence. REL-03: deterministic release packaging (tools/release/package.py) ships the CLI and provider worker together with SOURCE_SHA, RELEASE_INFO.json, known-limitations, specs, developer docs, examples incl. built plugin/provider artifacts and consumers; tools/release/smoke.py cold-install tests the archive. REL-04: docs/developers entry pages; corrected and self-contained third-party integration handbook; stale Proposed/native-only/informational spec claims fixed; release/status/compatibility/changelog/releasing docs updated. REL-05: release_evidence.py gains an explicit beta profile (BETA_RELEASE_CANDIDATE) distinct from the stable profile; Woodpecker build step now packages and smoke-tests the archive on every pipeline. Adds a runnable Resource Provider v2 example and corrects check --emit to Analysis v2.
This commit is contained in:
1 parent
56189b176c
commit
2c7c4f31e7
55 files changed
+3136
-453
No files matched your search
@@ -1,13 +1,35 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate per-gate CI evidence into a SHA-bound release manifest.
|
||||
|
||||
Schema: marqdo.release-gates.v1. A missing, skipped, timed-out, failed or
|
||||
Schema: `marqdo.release-gates.v1`. A missing, skipped, timed-out, failed or
|
||||
SHA-mismatched gate is never PASS. The tool only records supplied facts; it
|
||||
does not query CI and cannot turn an UNVERIFIED gate into a release.
|
||||
|
||||
Two profiles exist and both are explicit in the output:
|
||||
|
||||
* `stable` — every required platform gate must PASS on the target SHA.
|
||||
* `beta` — the supported platform set is the one declared in
|
||||
`COMPATIBILITY.md` (Linux x86_64 for `3.0.0-beta.1`). `windows`/`macos` must
|
||||
be explicitly `UNSUPPORTED` (not silently omitted). `strict_vm` may be PASS
|
||||
with same-SHA disposable-VM evidence, or `UNSUPPORTED` when the tested policy
|
||||
is an explicit fail-closed refusal; `extension`, `clean_install` and
|
||||
`version_compat` must PASS. `performance` may be `WARN` only in beta and is
|
||||
recorded with its reason.
|
||||
|
||||
Gate spec: `name=STATE:tested_sha:run_url` (url may contain colons).
|
||||
Required gates: linux, windows, macos, strict_vm; performance must be PASS or
|
||||
WARN. Any other gate name is rejected so a typo cannot silently disappear.
|
||||
|
||||
Usage:
|
||||
|
||||
python3 tools/ci/release_evidence.py --target-sha "$SHA" --profile beta \
|
||||
--gate linux=PASS:$SHA:https://ci.cflmy.cn/... \
|
||||
--gate strict_vm=PASS:$SHA:docs/evidence/... \
|
||||
--gate extension=PASS:$SHA:docs/evidence/extension/$SHA/manifest.json \
|
||||
--gate clean_install=PASS:$SHA:docs/evidence/... \
|
||||
--gate version_compat=PASS:$SHA:cargo-test \
|
||||
--gate windows=UNSUPPORTED:$SHA:supported-platforms-exclude \
|
||||
--gate macos=UNSUPPORTED:$SHA:supported-platforms-exclude \
|
||||
--gate performance=WARN:$SHA:docs/evidence/... \
|
||||
--out/docs/evidence/ci/$SHA/manifest.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -18,13 +40,40 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCHEMA = "marqdo.release-gates.v1"
|
||||
REQUIRED = ("linux", "windows", "macos", "strict_vm")
|
||||
OPTIONAL = ("performance",)
|
||||
PROFILES = {
|
||||
"stable": {
|
||||
"required": ("linux", "windows", "macos", "strict_vm"),
|
||||
"optional": ("performance",),
|
||||
"windows_macos_unsupported_allowed": False,
|
||||
"strict_unsupported_allowed": False,
|
||||
"decision": "RELEASE_CANDIDATE",
|
||||
},
|
||||
"beta": {
|
||||
"required": ("linux", "strict_vm", "extension", "clean_install", "version_compat"),
|
||||
"optional": ("windows", "macos", "performance"),
|
||||
"windows_macos_unsupported_allowed": True,
|
||||
"strict_unsupported_allowed": True,
|
||||
"decision": "BETA_RELEASE_CANDIDATE",
|
||||
},
|
||||
}
|
||||
GATES = (
|
||||
"linux",
|
||||
"windows",
|
||||
"macos",
|
||||
"strict_vm",
|
||||
"extension",
|
||||
"clean_install",
|
||||
"version_compat",
|
||||
"performance",
|
||||
)
|
||||
STATES = {
|
||||
"linux": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"windows": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"macos": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"strict_vm": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"windows": {"PASS", "FAIL", "UNVERIFIED", "UNSUPPORTED"},
|
||||
"macos": {"PASS", "FAIL", "UNVERIFIED", "UNSUPPORTED"},
|
||||
"strict_vm": {"PASS", "FAIL", "UNVERIFIED", "UNSUPPORTED"},
|
||||
"extension": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"clean_install": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"version_compat": {"PASS", "FAIL", "UNVERIFIED"},
|
||||
"performance": {"PASS", "WARN", "FAIL", "UNVERIFIED"},
|
||||
}
|
||||
SHA_RE = re.compile(r"^[0-9a-f]{40}$")
|
||||
@@ -48,7 +97,7 @@ def parse_gate(text: str) -> tuple[str, dict]:
|
||||
raise ValueError(f"gate must be name=STATE:sha:url, got {text!r}")
|
||||
state, tested_sha, run_url = parts
|
||||
if name not in STATES:
|
||||
raise ValueError(f"unknown gate {name!r}; expected {list(STATES)}")
|
||||
raise ValueError(f"unknown gate {name!r}; expected {list(GATES)}")
|
||||
if state not in STATES[name]:
|
||||
raise ValueError(f"invalid state {state!r} for gate {name!r}")
|
||||
if tested_sha and not SHA_RE.fullmatch(tested_sha):
|
||||
@@ -56,9 +105,13 @@ def parse_gate(text: str) -> tuple[str, dict]:
|
||||
return name, {"state": state, "tested_sha": tested_sha, "run_url": run_url}
|
||||
|
||||
|
||||
def build_manifest(target_sha: str, gates: list[str]) -> dict:
|
||||
def build_manifest(target_sha: str, gates: list[str], profile: str = "stable") -> dict:
|
||||
if not SHA_RE.fullmatch(target_sha):
|
||||
raise ValueError(f"target SHA must be 40 lowercase hex characters: {target_sha!r}")
|
||||
try:
|
||||
rules = PROFILES[profile]
|
||||
except KeyError:
|
||||
raise ValueError(f"unknown profile {profile!r}; expected {sorted(PROFILES)}") from None
|
||||
parsed = {}
|
||||
for spec in gates:
|
||||
name, gate = parse_gate(spec)
|
||||
@@ -67,29 +120,41 @@ def build_manifest(target_sha: str, gates: list[str]) -> dict:
|
||||
parsed[name] = gate
|
||||
|
||||
reasons = []
|
||||
for name in REQUIRED + OPTIONAL:
|
||||
for name in rules["required"] + rules["optional"]:
|
||||
gate = parsed.get(name)
|
||||
if gate is None:
|
||||
parsed[name] = {"state": "UNVERIFIED", "tested_sha": "", "run_url": ""}
|
||||
reasons.append(f"{name}: missing gate")
|
||||
if name in rules["required"]:
|
||||
reasons.append(f"{name}: missing gate")
|
||||
continue
|
||||
if gate["tested_sha"] != target_sha:
|
||||
reasons.append(
|
||||
f"{name}: tested_sha {gate['tested_sha'] or '(missing)'} != target_sha {target_sha}"
|
||||
)
|
||||
elif gate["state"] == "FAIL":
|
||||
continue
|
||||
if gate["state"] == "FAIL":
|
||||
reasons.append(f"{name}: FAIL")
|
||||
elif gate["state"] == "UNVERIFIED":
|
||||
reasons.append(f"{name}: UNVERIFIED")
|
||||
elif name in REQUIRED and gate["state"] != "PASS":
|
||||
elif gate["state"] == "UNSUPPORTED":
|
||||
allowed = False
|
||||
if name in {"windows", "macos"}:
|
||||
allowed = rules["windows_macos_unsupported_allowed"]
|
||||
elif name == "strict_vm":
|
||||
allowed = rules["strict_unsupported_allowed"]
|
||||
if not allowed:
|
||||
reasons.append(f"{name}: UNSUPPORTED is not allowed in the {profile} profile")
|
||||
elif name == "performance":
|
||||
if gate["state"] not in {"PASS", "WARN"}:
|
||||
reasons.append(f"{name}: {gate['state']} is not PASS/WARN")
|
||||
elif gate["state"] != "PASS":
|
||||
reasons.append(f"{name}: {gate['state']} is not PASS")
|
||||
elif name == "performance" and gate["state"] not in {"PASS", "WARN"}:
|
||||
reasons.append(f"performance: {gate['state']} is not PASS/WARN")
|
||||
|
||||
decision = "RELEASE_CANDIDATE" if not reasons else "DO_NOT_RELEASE"
|
||||
decision = rules["decision"] if not reasons else "DO_NOT_RELEASE"
|
||||
recorded = utc_now()
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"profile": profile,
|
||||
"target_sha": target_sha,
|
||||
"recorded_utc": recorded,
|
||||
"gates": {
|
||||
@@ -107,11 +172,12 @@ def build_manifest(target_sha: str, gates: list[str]) -> dict:
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--target-sha", required=True)
|
||||
parser.add_argument("--profile", choices=sorted(PROFILES), default="stable")
|
||||
parser.add_argument("--gate", action="append", default=[])
|
||||
parser.add_argument("--out", default="")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
manifest = build_manifest(args.target_sha, args.gate)
|
||||
manifest = build_manifest(args.target_sha, args.gate, args.profile)
|
||||
except ValueError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
@@ -83,6 +83,96 @@ class ReleaseEvidenceTests(unittest.TestCase):
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
|
||||
def test_beta_profile_accepts_declared_linux_scope_with_warn_performance(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("extension", "PASS"),
|
||||
gate("clean_install", "PASS"),
|
||||
gate("version_compat", "PASS"),
|
||||
gate("windows", "UNSUPPORTED", url="linux-only-support-matrix"),
|
||||
gate("macos", "UNSUPPORTED", url="linux-only-support-matrix"),
|
||||
gate("performance", "WARN"),
|
||||
],
|
||||
profile="beta",
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "BETA_RELEASE_CANDIDATE")
|
||||
self.assertEqual(manifest["profile"], "beta")
|
||||
self.assertEqual(manifest["reasons"], [])
|
||||
|
||||
def test_beta_profile_allows_tested_fail_closed_strict_policy(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("strict_vm", "UNSUPPORTED", url="strict.fail_closed_test"),
|
||||
gate("extension", "PASS"),
|
||||
gate("clean_install", "PASS"),
|
||||
gate("version_compat", "PASS"),
|
||||
gate("windows", "UNSUPPORTED"),
|
||||
gate("macos", "UNSUPPORTED"),
|
||||
gate("performance", "WARN"),
|
||||
],
|
||||
profile="beta",
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "BETA_RELEASE_CANDIDATE")
|
||||
|
||||
def test_beta_profile_requires_explicit_platform_and_extension_gates(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("clean_install", "PASS"),
|
||||
gate("version_compat", "PASS"),
|
||||
gate("performance", "WARN"),
|
||||
],
|
||||
profile="beta",
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
joined = " ".join(manifest["reasons"])
|
||||
self.assertIn("extension: missing gate", joined)
|
||||
self.assertNotIn("windows: missing gate", joined)
|
||||
|
||||
def test_beta_profile_rejects_wrong_sha_and_unverified_gates(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS", OTHER),
|
||||
gate("strict_vm", "PASS"),
|
||||
gate("extension", "UNVERIFIED"),
|
||||
gate("clean_install", "PASS"),
|
||||
gate("version_compat", "PASS"),
|
||||
gate("windows", "UNSUPPORTED"),
|
||||
gate("macos", "UNSUPPORTED"),
|
||||
gate("performance", "PASS"),
|
||||
],
|
||||
profile="beta",
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
joined = " ".join(manifest["reasons"])
|
||||
self.assertIn("linux", joined)
|
||||
self.assertIn("extension", joined)
|
||||
|
||||
def test_stable_profile_rejects_beta_unsupported_states(self) -> None:
|
||||
manifest = release_evidence.build_manifest(
|
||||
SHA,
|
||||
[
|
||||
gate("linux", "PASS"),
|
||||
gate("windows", "UNSUPPORTED"),
|
||||
gate("macos", "UNSUPPORTED"),
|
||||
gate("strict_vm", "UNSUPPORTED"),
|
||||
gate("performance", "WARN"),
|
||||
],
|
||||
)
|
||||
self.assertEqual(manifest["release_decision"], "DO_NOT_RELEASE")
|
||||
|
||||
def test_unknown_profile_is_rejected(self) -> None:
|
||||
with self.assertRaises(ValueError):
|
||||
release_evidence.build_manifest(SHA, [], profile="lts")
|
||||
|
||||
def test_unknown_gate_and_bad_sha_are_rejected(self) -> None:
|
||||
with self.assertRaises(ValueError):
|
||||
release_evidence.build_manifest(SHA, [gate("solaris", "PASS")])
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the Marqdo release archive from a clean, pinned commit.
|
||||
|
||||
The archive is deterministic (sorted entries, commit timestamp, uid/gid 0) and
|
||||
contains the CLI, the provider worker helper (same directory, same target), the
|
||||
licenses/changelog/compatibility matrix, the public specs, the developer docs,
|
||||
the runnable examples and the public protocol consumers.
|
||||
|
||||
It refuses to package a dirty tree, a version that does not match the Cargo
|
||||
workspace, or a HEAD that differs from `--source-sha`, and it fails if the
|
||||
example extension artifacts cannot be built on the host. Nothing here publishes
|
||||
anything; tagging is a separate reviewed step.
|
||||
|
||||
Usage (from the repository root, after
|
||||
`cargo build --release --locked -p marqdo-cli -p marqdo-provider-worker`):
|
||||
|
||||
python3 tools/release/package.py \
|
||||
--source-sha "$(git rev-parse HEAD)" \
|
||||
--version 3.0.0-beta.1
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
BINARIES = ("marqdo", "marqdo-provider-worker")
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"package: error: {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run(command: list[str], **kwargs) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(command, check=True, text=True, **kwargs)
|
||||
|
||||
|
||||
def git(*args: str) -> str:
|
||||
return run(["git", *args], cwd=ROOT, capture_output=True).stdout.strip()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def host_target() -> str:
|
||||
out = run(["rustc", "-vV"], capture_output=True).stdout
|
||||
for line in out.splitlines():
|
||||
if line.startswith("host: "):
|
||||
return line[len("host: ") :]
|
||||
fail("rustc -vV has no host line")
|
||||
|
||||
|
||||
def workspace_version() -> str:
|
||||
metadata = json.loads(
|
||||
run(
|
||||
["cargo", "metadata", "--no-deps", "--format-version", "1", "--offline"],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
).stdout
|
||||
)
|
||||
for package in metadata["packages"]:
|
||||
if package["name"] == "marqdo-cli":
|
||||
return package["version"]
|
||||
fail("cargo metadata has no marqdo-cli package")
|
||||
|
||||
|
||||
def toolchain() -> dict:
|
||||
rustc = run(["rustc", "-vV"], capture_output=True).stdout
|
||||
release = ""
|
||||
commit = ""
|
||||
for line in rustc.splitlines():
|
||||
if line.startswith("release: "):
|
||||
release = line[len("release: ") :]
|
||||
elif line.startswith("commit-hash: "):
|
||||
commit = line[len("commit-hash: ") :]
|
||||
cargo = run(["cargo", "--version"], capture_output=True).stdout.strip()
|
||||
return {"rustc": release, "rustc_commit": commit, "cargo": cargo}
|
||||
|
||||
|
||||
def check_clean(source_sha: str) -> None:
|
||||
head = git("rev-parse", "HEAD")
|
||||
if head != source_sha:
|
||||
fail(f"HEAD {head} does not match --source-sha {source_sha}")
|
||||
status = git("status", "--porcelain")
|
||||
allowed = [
|
||||
line
|
||||
for line in status.splitlines()
|
||||
if not (line.startswith("?? dist/") or line.startswith("!! dist/"))
|
||||
]
|
||||
if allowed:
|
||||
fail("working tree is not clean:\n" + "\n".join(allowed))
|
||||
|
||||
|
||||
def build_examples() -> dict:
|
||||
outputs = {}
|
||||
for name in ("plugin-demo", "plugin-multi-export", "provider-demo"):
|
||||
script = ROOT / "examples" / name / "build.sh"
|
||||
run(["bash", str(script)], cwd=ROOT)
|
||||
outputs[name] = ROOT / "examples" / name / "build"
|
||||
for path in sorted(outputs[name].rglob("*")):
|
||||
if path.is_file():
|
||||
outputs.setdefault("files", {})[str(path.relative_to(ROOT))] = sha256_file(path)
|
||||
return outputs
|
||||
|
||||
|
||||
def copy_tree(source: Path, destination: Path) -> None:
|
||||
shutil.copytree(source, destination, dirs_exist_ok=True)
|
||||
|
||||
|
||||
def stage(stage_dir: Path, version: str, source_sha: str, target: str) -> dict:
|
||||
stage_dir.mkdir(parents=True)
|
||||
release_dir = ROOT / "target" / "release"
|
||||
for name in BINARIES:
|
||||
binary = release_dir / name
|
||||
if not binary.is_file():
|
||||
fail(f"missing release binary {binary}; build with --release --locked first")
|
||||
installed = stage_dir / name
|
||||
shutil.copy2(binary, installed)
|
||||
installed.chmod(0o755)
|
||||
|
||||
for name in ("README.md", "CHANGELOG.md", "COMPATIBILITY.md"):
|
||||
shutil.copy2(ROOT / name, stage_dir / name)
|
||||
for license_name in ("LICENSE-MIT", "LICENSE-APACHE"):
|
||||
shutil.copy2(ROOT / license_name, stage_dir / license_name)
|
||||
|
||||
copy_tree(ROOT / "spec", stage_dir / "spec")
|
||||
copy_tree(ROOT / "docs" / "developers", stage_dir / "docs" / "developers")
|
||||
|
||||
# Examples without local build output, then the freshly built extension
|
||||
# artifacts so a clean install can run every example without a compiler.
|
||||
copy_tree(ROOT / "examples", stage_dir / "examples")
|
||||
for build_dir in stage_dir.glob("examples/*/build"):
|
||||
shutil.rmtree(build_dir, ignore_errors=True)
|
||||
for example in ("plugin-demo", "plugin-multi-export", "provider-demo"):
|
||||
copy_tree(ROOT / "examples" / example / "build", stage_dir / "examples" / example / "build")
|
||||
|
||||
copy_tree(ROOT / "tests" / "consumers", stage_dir / "tests" / "consumers")
|
||||
|
||||
limitations = ROOT / "docs" / "developers" / "release-limits.md"
|
||||
shutil.copy2(limitations, stage_dir / "KNOWN_LIMITATIONS.md")
|
||||
|
||||
commit_time = int(git("show", "-s", "--format=%ct", "HEAD"))
|
||||
binaries = {
|
||||
name: {
|
||||
"sha256": sha256_file(stage_dir / name),
|
||||
"size": (stage_dir / name).stat().st_size,
|
||||
}
|
||||
for name in BINARIES
|
||||
}
|
||||
info = {
|
||||
"schema": "marqdo.release-info",
|
||||
"schema_version": 1,
|
||||
"product": "marqdo-cli",
|
||||
"product_version": version,
|
||||
"source_sha": source_sha,
|
||||
"target_triple": target,
|
||||
"build_profile": "release",
|
||||
"commit_time_utc": time_utc(commit_time),
|
||||
"toolchain": toolchain(),
|
||||
"binaries": binaries,
|
||||
"protocols": {
|
||||
"analysis": 2,
|
||||
"trace": 2,
|
||||
"tool": 1,
|
||||
"plugin_abi": "1.x",
|
||||
"foreign_wire": 1,
|
||||
"provider_manifest": 2,
|
||||
"lock": 4,
|
||||
"markdown_profile": "marqdo.md.v1",
|
||||
},
|
||||
"supported_platforms": ["x86_64-unknown-linux-gnu"],
|
||||
"known_limitations": "KNOWN_LIMITATIONS.md",
|
||||
"notes": [
|
||||
"Checksums prove content integrity only; they are not a publisher signature.",
|
||||
"The provider worker must stay in the same directory as marqdo.",
|
||||
"Run `python3 tools/release/smoke.py --archive <this tarball>` before publishing.",
|
||||
],
|
||||
}
|
||||
(stage_dir / "RELEASE_INFO.json").write_text(
|
||||
json.dumps(info, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
||||
)
|
||||
(stage_dir / "SOURCE_SHA").write_text(source_sha + "\n", encoding="utf-8")
|
||||
return {"info": info, "commit_time": commit_time}
|
||||
|
||||
|
||||
def time_utc(epoch: int) -> str:
|
||||
import datetime
|
||||
|
||||
return (
|
||||
datetime.datetime.fromtimestamp(epoch, datetime.timezone.utc)
|
||||
.replace(microsecond=0)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def make_archive(stage_dir: Path, archive: Path, commit_time: int) -> None:
|
||||
entries = sorted(path for path in stage_dir.rglob("*") if path.is_file())
|
||||
entries.sort(key=lambda path: str(path.relative_to(stage_dir.parent)))
|
||||
with archive.open("wb") as raw:
|
||||
with gzip.GzipFile(filename="", mode="wb", fileobj=raw, mtime=0) as compressed:
|
||||
with tarfile.open(fileobj=compressed, mode="w", format=tarfile.GNU_FORMAT) as tar:
|
||||
for path in entries:
|
||||
arcname = str(path.relative_to(stage_dir.parent))
|
||||
info = tar.gettarinfo(str(path), arcname=arcname)
|
||||
info.uid = 0
|
||||
info.gid = 0
|
||||
info.uname = ""
|
||||
info.gname = ""
|
||||
info.mtime = commit_time
|
||||
# Preserve the executable bit of scripts/artifacts; only
|
||||
# force the two release binaries themselves to 0755.
|
||||
if path.name in BINARIES:
|
||||
info.mode = 0o755
|
||||
else:
|
||||
info.mode = path.stat().st_mode & 0o777
|
||||
with path.open("rb") as handle:
|
||||
tar.addfile(info, handle)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source-sha", required=True)
|
||||
parser.add_argument("--version", required=True)
|
||||
parser.add_argument("--target", default=None)
|
||||
parser.add_argument("--out-dir", default=str(ROOT / "dist"))
|
||||
parser.add_argument("--skip-example-build", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if not args.source_sha or len(args.source_sha) != 40 or any(
|
||||
ch not in "0123456789abcdef" for ch in args.source_sha
|
||||
):
|
||||
fail("--source-sha must be a full lowercase 40-character commit SHA")
|
||||
check_clean(args.source_sha)
|
||||
version = workspace_version()
|
||||
if version != args.version:
|
||||
fail(f"workspace version is {version}, not {args.version}")
|
||||
target = args.target or host_target()
|
||||
if target != "x86_64-unknown-linux-gnu":
|
||||
fail(f"this beta release procedure supports x86_64-unknown-linux-gnu only, got {target}")
|
||||
if not args.skip_example_build:
|
||||
build_examples()
|
||||
|
||||
out_dir = Path(args.out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
stage_dir = out_dir / f"marqdo-{version}-{target}"
|
||||
shutil.rmtree(stage_dir, ignore_errors=True)
|
||||
staged = stage(stage_dir, version, args.source_sha, target)
|
||||
archive = out_dir / f"marqdo-{version}-{target}.tar.gz"
|
||||
archive.unlink(missing_ok=True)
|
||||
make_archive(stage_dir, archive, staged["commit_time"])
|
||||
digest = sha256_file(archive)
|
||||
checksum = out_dir / f"{archive.name}.sha256"
|
||||
checksum.write_text(f"{digest} {archive.name}\n", encoding="utf-8")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"archive": str(archive),
|
||||
"sha256": digest,
|
||||
"checksum_file": str(checksum),
|
||||
"size": archive.stat().st_size,
|
||||
"source_sha": args.source_sha,
|
||||
"version": version,
|
||||
"target": target,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,339 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cold-install smoke test for a built Marqdo release archive.
|
||||
|
||||
Extracts the archive into a fresh temporary directory and exercises the shipped
|
||||
binaries only (never the development source tree): version/help, check/run/call,
|
||||
lock/sync for a path dependency, Analysis v2 + Trace v2 consumers, Tool
|
||||
install/dispatch, native and worker plugin transports, foreign kernel, the
|
||||
Provider v2 worker (positive, unauthorized and missing-helper fail-closed), and
|
||||
strict mode when the host genuinely supports it.
|
||||
|
||||
Usage:
|
||||
python3 tools/release/smoke.py --archive dist/marqdo-3.0.0-beta.1-x86_64-unknown-linux-gnu.tar.gz
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"smoke: FAIL: {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
class Runner:
|
||||
def __init__(self, root: Path):
|
||||
self.root = root
|
||||
self.marqdo = root / "marqdo"
|
||||
self.env = dict(os.environ)
|
||||
self.env.pop("MARQDO_MODE", None)
|
||||
self.env["MARQDO_TOOL_HOME"] = str(root / "tool-home")
|
||||
|
||||
def run(
|
||||
self,
|
||||
args: list[str],
|
||||
*,
|
||||
expect: int = 0,
|
||||
contains: str | None = None,
|
||||
cwd: Path | None = None,
|
||||
env: dict | None = None,
|
||||
) -> subprocess.CompletedProcess:
|
||||
command = [str(self.marqdo), *args]
|
||||
merged = dict(self.env)
|
||||
if env:
|
||||
merged.update(env)
|
||||
result = subprocess.run(
|
||||
command,
|
||||
cwd=cwd or self.root,
|
||||
env=merged,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120,
|
||||
)
|
||||
shown = " ".join(command[1:])
|
||||
if result.returncode != expect:
|
||||
fail(
|
||||
f"`marqdo {shown}` exited {result.returncode}, expected {expect}\n"
|
||||
f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
|
||||
)
|
||||
if contains is not None:
|
||||
haystack = result.stdout + result.stderr
|
||||
if contains not in haystack:
|
||||
fail(f"`marqdo {shown}` output does not contain {contains!r}:\n{haystack}")
|
||||
return result
|
||||
|
||||
def json(self, args: list[str]) -> dict:
|
||||
result = self.run([*args, "--json"])
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except json.JSONDecodeError as error:
|
||||
fail(f"`marqdo {' '.join(args)} --json` is not JSON: {error}\n{result.stdout}")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--archive", required=True)
|
||||
parser.add_argument("--workdir", default=None, help="keep the extracted tree here")
|
||||
parser.add_argument("--skip-strict", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
archive = Path(args.archive).resolve()
|
||||
if not archive.is_file():
|
||||
fail(f"archive not found: {archive}")
|
||||
checksum_file = archive.with_name(archive.name + ".sha256")
|
||||
if checksum_file.is_file():
|
||||
expected = checksum_file.read_text(encoding="utf-8").split()[0]
|
||||
actual = sha256_file(archive)
|
||||
if expected != actual:
|
||||
fail(f"archive sha256 {actual} does not match {checksum_file}: {expected}")
|
||||
|
||||
if args.workdir:
|
||||
root = Path(args.workdir).resolve()
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
root.mkdir(parents=True)
|
||||
else:
|
||||
root = Path(tempfile.mkdtemp(prefix="marqdo-smoke-"))
|
||||
try:
|
||||
shutil.unpack_archive(str(archive), str(root))
|
||||
entries = [path for path in root.iterdir() if path.is_dir()]
|
||||
if len(entries) != 1:
|
||||
fail(f"archive must contain exactly one top-level directory, found {entries}")
|
||||
install = entries[0]
|
||||
info = json.loads((install / "RELEASE_INFO.json").read_text(encoding="utf-8"))
|
||||
version = info["product_version"]
|
||||
source_sha = info["source_sha"]
|
||||
if (install / "SOURCE_SHA").read_text(encoding="utf-8").strip() != source_sha:
|
||||
fail("SOURCE_SHA does not match RELEASE_INFO.json")
|
||||
for name in ("marqdo", "marqdo-provider-worker"):
|
||||
binary = install / name
|
||||
if not binary.is_file() or not os.access(binary, os.X_OK):
|
||||
fail(f"{name} is missing or not executable in the archive")
|
||||
recorded = info["binaries"][name]["sha256"]
|
||||
if sha256_file(binary) != recorded:
|
||||
fail(f"{name} sha256 does not match RELEASE_INFO.json")
|
||||
|
||||
runner = Runner(install)
|
||||
print(f"smoke: archive version {version}, source {source_sha}")
|
||||
|
||||
version_out = runner.run(["--version"]).stdout.strip()
|
||||
if version_out != f"marqdo {version}":
|
||||
fail(f"--version printed {version_out!r}, expected 'marqdo {version}'")
|
||||
runner.run(["--help"])
|
||||
runner.run(["check", "examples/multi-module/index.mq.md"])
|
||||
hello = runner.run(["run", "examples/hello.mq.md"]).stdout
|
||||
if "你好" not in hello:
|
||||
fail(f"hello example printed {hello!r}")
|
||||
factor = runner.run(
|
||||
["call", "examples/factorial.mq.md", "阶乘", "--arg", "n=5"]
|
||||
).stdout.strip()
|
||||
if factor != "120":
|
||||
fail(f"factorial call printed {factor!r}, expected 120")
|
||||
|
||||
# Static Analysis v2 + Trace v2 through the public consumers.
|
||||
analysis = install / "analysis.json"
|
||||
trace = install / "trace.json"
|
||||
runner.run(
|
||||
[
|
||||
"check",
|
||||
"examples/multi-module/index.mq.md",
|
||||
"--emit",
|
||||
str(analysis),
|
||||
]
|
||||
)
|
||||
runner.run(["check", "examples/multi-module/index.mq.md", "--refs"])
|
||||
document = json.loads(analysis.read_text(encoding="utf-8"))
|
||||
if document.get("schema") != "marqdo.analysis" or document.get("schema_version") != 2:
|
||||
fail("--emit did not write Analysis v2")
|
||||
if document.get("producer", {}).get("version") != version:
|
||||
fail("Analysis producer.version does not match the released product version")
|
||||
runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/multi-module/index.mq.md",
|
||||
"--trace-file",
|
||||
str(trace),
|
||||
]
|
||||
)
|
||||
run_trace = json.loads(trace.read_text(encoding="utf-8"))
|
||||
if run_trace.get("schema") != "marqdo.trace" or run_trace.get("schema_version") != 2:
|
||||
fail("--trace-file did not write Trace v2")
|
||||
for consumer in ("analysis_consumer.py", "trace_consumer.py"):
|
||||
command = [
|
||||
sys.executable,
|
||||
str(install / "tests" / "consumers" / consumer),
|
||||
str(analysis),
|
||||
]
|
||||
if consumer == "trace_consumer.py":
|
||||
command.append(str(trace))
|
||||
result = subprocess.run(command, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
fail(f"{consumer} failed:\n{result.stdout}\n{result.stderr}")
|
||||
validate = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(install / "tests" / "consumers" / "validate_protocols.py"),
|
||||
str(analysis),
|
||||
str(trace),
|
||||
str(install / "examples" / "tool-unknown-name" / "tool.json"),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if validate.returncode != 0:
|
||||
if "jsonschema" in validate.stderr and "ModuleNotFoundError" in validate.stderr:
|
||||
print("smoke: note: jsonschema is not installed; schema validation skipped")
|
||||
else:
|
||||
fail(f"validate_protocols.py failed:\n{validate.stdout}\n{validate.stderr}")
|
||||
|
||||
# Tool Protocol v1: install and dispatch an independent process.
|
||||
runner.run(["tool", "install", "examples/tool-unknown-name/tool.json"])
|
||||
listed = runner.run(["tool", "list"]).stdout
|
||||
if "benchmark" not in listed:
|
||||
fail(f"tool list does not contain benchmark:\n{listed}")
|
||||
benchmark = runner.run(["benchmark", "run", "."]).stdout
|
||||
if "benchmark tool:" not in benchmark:
|
||||
fail(f"tool dispatch output unexpected:\n{benchmark}")
|
||||
|
||||
# Callable plugin ABI v1: native (explicit trust) and worker transports.
|
||||
native = runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/plugin-demo/plugin-call.mq.md",
|
||||
"--plugin",
|
||||
"examples/plugin-demo/build",
|
||||
"--trust-native-plugin",
|
||||
"--allow-plugin",
|
||||
"demo.solver",
|
||||
]
|
||||
).stdout.strip()
|
||||
if native != "42":
|
||||
fail(f"native plugin example printed {native!r}, expected 42")
|
||||
runner.run(
|
||||
["plugin", "inspect", "examples/plugin-demo/build", "--json"]
|
||||
)
|
||||
worker_dir = install / "worker-plugin"
|
||||
worker_dir.mkdir()
|
||||
shutil.copy2(install / "examples/plugin-demo/build/solver.so", worker_dir / "solver.so")
|
||||
worker_manifest = (
|
||||
install / "examples/plugin-demo/build/plugin.yaml"
|
||||
).read_text(encoding="utf-8").replace("kind: native", "kind: worker")
|
||||
(worker_dir / "plugin.yaml").write_text(worker_manifest, encoding="utf-8")
|
||||
worker = runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/plugin-demo/plugin-call.mq.md",
|
||||
"--plugin",
|
||||
str(worker_dir),
|
||||
"--allow-plugin",
|
||||
"demo.solver",
|
||||
]
|
||||
).stdout.strip()
|
||||
if worker != "42":
|
||||
fail(f"worker plugin example printed {worker!r}, expected 42")
|
||||
|
||||
# Foreign kernel: no core change for a new language tag.
|
||||
kernel = runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/kernel-mock/demo.mq.md",
|
||||
"--kernel",
|
||||
"examples/kernel-mock/kernel.json",
|
||||
"--allow-foreign",
|
||||
]
|
||||
).stdout.strip()
|
||||
if kernel != "42":
|
||||
fail(f"foreign kernel example printed {kernel!r}, expected 42")
|
||||
|
||||
# Resource Provider v2 through the shipped worker helper.
|
||||
provider = "examples/provider-demo/build/provider.json"
|
||||
provider_run = runner.json(
|
||||
[
|
||||
"run",
|
||||
"examples/provider-demo/demo.mq.md",
|
||||
"--provider",
|
||||
provider,
|
||||
"--allow-provider",
|
||||
"demo.sensor",
|
||||
]
|
||||
)
|
||||
if not provider_run.get("success"):
|
||||
fail(f"provider run was not successful: {provider_run}")
|
||||
reads = [
|
||||
event
|
||||
for event in provider_run.get("trace", [])
|
||||
if event.get("op") == "resource.read"
|
||||
]
|
||||
if not reads or "count=5" not in str(reads[-1].get("result")):
|
||||
fail(f"provider read did not report five bytes: {reads}")
|
||||
runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/provider-demo/unauthorized.mq.md",
|
||||
"--provider",
|
||||
provider,
|
||||
],
|
||||
expect=1,
|
||||
contains="permission.denied",
|
||||
)
|
||||
worker_binary = install / "marqdo-provider-worker"
|
||||
shutil.move(str(worker_binary), str(worker_binary.with_suffix(".hidden")))
|
||||
try:
|
||||
runner.run(
|
||||
[
|
||||
"run",
|
||||
"examples/provider-demo/demo.mq.md",
|
||||
"--provider",
|
||||
provider,
|
||||
"--allow-provider",
|
||||
"demo.sensor",
|
||||
],
|
||||
expect=1,
|
||||
contains="ext.worker_unavailable",
|
||||
)
|
||||
finally:
|
||||
shutil.move(str(worker_binary.with_suffix(".hidden")), str(worker_binary))
|
||||
|
||||
# pure mode: language execution without project grants.
|
||||
runner.run(["run", "examples/hello.mq.md", "--mode", "pure"])
|
||||
|
||||
if not args.skip_strict:
|
||||
platform = runner.json(["inspect", "examples/hello.mq.md", "--platform"])
|
||||
strict = platform.get("platform", {}).get("strict", {})
|
||||
if strict.get("state") == "available":
|
||||
runner.run(["run", "examples/hello.mq.md", "--mode", "strict"])
|
||||
print(f"smoke: strict available ({strict.get('mechanism')}) and exercised")
|
||||
else:
|
||||
refused = runner.run(
|
||||
["run", "examples/hello.mq.md", "--mode", "strict"],
|
||||
expect=1,
|
||||
)
|
||||
if "strict" not in (refused.stdout + refused.stderr):
|
||||
fail("unsupported strict mode did not fail closed with a strict diagnostic")
|
||||
print(f"smoke: strict unsupported on this host; fail-closed refused: {strict}")
|
||||
|
||||
print("smoke: PASS")
|
||||
return 0
|
||||
finally:
|
||||
if not args.workdir:
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
Reference in new issue
Block a user