Linux strict security lane (G-C) / strict isolation escape matrix (push) Waiting to run
ci/woodpecker/push/main Pipeline was successful
ci/woodpecker/manual/performance Pipeline was successful
CI / fmt + clippy + test (Linux x86_64) (push) Waiting to run
CI / schema instance validation + consumers (push) Waiting to run
CI / runtime contract (macos-latest) (push) Waiting to run
CI / runtime contract (windows-latest) (push) Waiting to run
REL-01: workspace/CLI/Analysis product version 3.0.0-beta.1 with updated Cargo.lock; project.marqdo is enforced against the running core with package.marqdo_range_invalid/package.marqdo_incompatible fail-closed diagnostics; Tool SemVer prerelease negotiation tests; init template and examples use the beta range. Verify init derives the project name from the target directory instead of falling back to 'example'. REL-02: beta support matrix is Linux x86_64 only; macOS/Windows are explicitly experimental/unclaimed; strict policy and G4 Beta-only WARN are documented for the release evidence. REL-03: deterministic release packaging (tools/release/package.py) ships the CLI and provider worker together with SOURCE_SHA, RELEASE_INFO.json, known-limitations, specs, developer docs, examples incl. built plugin/provider artifacts and consumers; tools/release/smoke.py cold-install tests the archive. REL-04: docs/developers entry pages; corrected and self-contained third-party integration handbook; stale Proposed/native-only/informational spec claims fixed; release/status/compatibility/changelog/releasing docs updated. REL-05: release_evidence.py gains an explicit beta profile (BETA_RELEASE_CANDIDATE) distinct from the stable profile; Woodpecker build step now packages and smoke-tests the archive on every pipeline. Adds a runnable Resource Provider v2 example and corrects check --emit to Analysis v2.
388 lines
18 KiB
Markdown
388 lines
18 KiB
Markdown
# Marqdo Project Package Format v4
|
|
|
|
Status: **lock_version 4 package-instance graph plus extension identities
|
|
(RFC-EXT-0001 §6, ADR-0056).** Path and pinned Git sources, transitive closure
|
|
locking, fail-closed locked resolution, strict `sync`, content-profile-1
|
|
file-set hashing/materialization, and explicit `extensions:` activation
|
|
sources/identities are implemented. Registries and SemVer solving remain out of
|
|
scope; see §8. Lock v3 is rejected with an explicit `marqdo lock` migration
|
|
error; it is never reinterpreted silently.
|
|
|
|
Implementation: `crates/marqdo-package/src/lib.rs` (manifest/lock types and
|
|
validation), `crates/cli/src/commands.rs` (`init`, `add`, `remove`, `lock`,
|
|
`sync`).
|
|
|
|
## 1. Project root
|
|
|
|
The project root is the nearest ancestor directory of the entry document that
|
|
contains `marqdo.yaml` (`marqdo_package::discover_root`). If no manifest exists
|
|
anywhere above the entry, the entry file's parent directory is used as the
|
|
root (a document can be checked/run without a project).
|
|
|
|
`ModuleId`s are package-instance + package-relative paths (`spec/references.md`
|
|
§2.1), so the project root selects the entry package instance; physical paths do
|
|
not form module identity. `add`/`remove`/`lock`/`sync` discover the
|
|
root from the **current working directory**, not from a document path. There is
|
|
no `--project` flag in this build.
|
|
|
|
## 2. `marqdo.yaml`
|
|
|
|
```yaml
|
|
project:
|
|
name: example
|
|
version: 0.1.0
|
|
marqdo: ">=3.0.0-beta.1,<4.0.0" # optional; enforced against the running core
|
|
dependencies:
|
|
geometry:
|
|
path: ./packages/geometry # exactly one of path / git+rev
|
|
quantum:
|
|
git: https://github.com/example/quantum-mq
|
|
rev: 0123456789abcdef0123456789abcdef01234567
|
|
plugins:
|
|
solver:
|
|
version: "^1.0" # requirement only; not installed or authorized
|
|
capabilities: # requests only; not grants
|
|
fs:
|
|
read: ["./data/**"]
|
|
write: ["./output/**"]
|
|
net:
|
|
hosts: []
|
|
process: false
|
|
extensions: # explicit activation sources; never grants
|
|
plugins:
|
|
- from: ./adapter/plugin.yaml
|
|
kernels: []
|
|
providers:
|
|
- from: "@quantum/adapter/provider.json"
|
|
```
|
|
|
|
### 2.1 Fields
|
|
|
|
| Field | Type | Required | Semantics |
|
|
|---|---|---|---|
|
|
| `project.name` | string | yes | non-empty package name |
|
|
| `project.version` | string | yes | non-empty version string (not semver-validated in the manifest) |
|
|
| `project.marqdo` | string | no | Marqdo core SemVer requirement; enforced against the running core, incompatible ranges are hard errors |
|
|
| `dependencies.<name>.path` | path string | one of path/git | project-relative path to a dependency directory or `.mq.md` file |
|
|
| `dependencies.<name>.git` | URL string | one of path/git | git source (must be pinned with `rev`) |
|
|
| `dependencies.<name>.rev` | string | with `git` | exact revision; a bare `git` without `rev` is rejected |
|
|
| `plugins.<name>.version` | string | per entry | non-empty plugin version requirement; declarative only |
|
|
| `capabilities.fs.read` / `.write` | list of strings | no | requested project-relative globs |
|
|
| `capabilities.net.hosts` | list of strings | no | requested hosts |
|
|
| `capabilities.process` | bool | no | requested process execution |
|
|
| `extensions.plugins[]` / `.kernels[]` / `.providers[]` | `{from}` | no | explicit extension activation sources; `from` is `./relative/path` in the project or `@alias/relative/path` in a declared dependency. The alias must be in `dependencies`. Absolute paths, URLs, globs and `..` traversal are rejected. A declaration is not a grant. |
|
|
|
|
Unknown fields are ignored (the types do not `deny_unknown_fields`). Duplicate
|
|
YAML keys follow `serde_yaml` behavior (last one wins); there is no duplicate-key
|
|
diagnostic.
|
|
|
|
### 2.2 Validation (`Manifest::validate`)
|
|
|
|
`marqdo.yaml` is loaded with `Manifest::from_yaml`/`Manifest::load`, which
|
|
validates:
|
|
|
|
* `project.name` and `project.version` are non-empty after trimming;
|
|
* when `project.marqdo` is present it must parse as a `semver::VersionReq`
|
|
and match the running core version (`package.marqdo_range_invalid` /
|
|
`package.marqdo_incompatible`). SemVer prerelease rules apply: a stable-only
|
|
range such as `>=3.0.0,<4.0.0` does **not** accept `3.0.0-beta.1`; prerelease
|
|
users must opt in with a range whose comparator carries the prerelease
|
|
(for example `>=3.0.0-beta.1,<4.0.0`);
|
|
* for every dependency, exactly one source is present:
|
|
* both `path` and `git` → error "declares both path and git";
|
|
* neither → error "must declare a path or a pinned git source";
|
|
* `git` without `rev` → error requiring an exact revision;
|
|
* an absolute `path` → error "path must be project-relative";
|
|
* for every plugin requirement, `version` is non-empty;
|
|
* every `extensions.*[].from` is one of the two allowed forms and resolves to a
|
|
regular file inside its owning package; duplicates within a kind are
|
|
rejected; `@alias` must name a declared dependency. Resolution happens when
|
|
the project is loaded (`PackageResolver::resolve_extension_sources`), not in
|
|
plain `Manifest::validate`.
|
|
|
|
A pinned *git* dependency is resolvable, lockable and runnable; `lock`/`sync`
|
|
clone it into the content-addressed cache and verify the checked-out commit
|
|
and tree digest (§4.3).
|
|
|
|
### 2.3 `capabilities` are requests
|
|
|
|
The manifest is the project's **request** for permissions. It is never read
|
|
when constructing the host capability set; only CLI flags/grant APIs create
|
|
grants (`spec/security.md` §2). `net` has no implementing host op, and
|
|
`process`/foreign execution still needs `--allow-foreign`.
|
|
|
|
## 3. `marqdo.lock`
|
|
|
|
`lock_version: 4` records the complete package-instance closure reachable from
|
|
the root manifest plus the exact identity of every declared extension.
|
|
Packages are sorted by `instance_id`; extensions by `(kind, name,
|
|
logical_owner)`:
|
|
|
|
```yaml
|
|
lock_version: 4
|
|
content_profile_version: 1
|
|
root:
|
|
package: root:quantum@0.1.0
|
|
manifest_digest: 4f5c...
|
|
packages:
|
|
- instance_id: dep:math@1.0.0#9e2...
|
|
name: math
|
|
version: 1.0.0
|
|
source:
|
|
kind: path
|
|
path: deps/math
|
|
content_digest: 9e2...
|
|
manifest_digest: 7a1...
|
|
dependencies:
|
|
helper: dep:helper@2.0.0#1b3...
|
|
- instance_id: dep:helper@2.0.0#1b3...
|
|
name: helper
|
|
version: 2.0.0
|
|
source:
|
|
kind: git
|
|
url: https://example.invalid/helper.git
|
|
rev: 0123456789abcdef0123456789abcdef01234567
|
|
content_digest: 1b3...
|
|
manifest_digest: 3c9...
|
|
dependencies: {}
|
|
root_dependencies:
|
|
math: dep:math@1.0.0#9e2...
|
|
extensions:
|
|
- kind: plugin
|
|
logical_owner: root:quantum@0.1.0
|
|
name: demo.solver
|
|
version: 0.1.0
|
|
manifest_digest: 4f5c... # sha256 of the exact plugin.yaml bytes
|
|
artifact_digest: 9e2... # selected target artifact sha256
|
|
selected_target: x86_64-unknown-linux-gnu
|
|
source: ./adapter/plugin.yaml
|
|
```
|
|
|
|
An extension lock record binds:
|
|
|
|
* `kind` — `plugin` | `kernel` | `provider`;
|
|
* `logical_owner` — root instance or dependency package instance id;
|
|
* `name` / `version` — stable declared identity (kernel descriptors have no
|
|
version, so the lock records `wire:<n>`);
|
|
* `manifest_digest` — SHA-256 of the exact manifest/descriptor bytes;
|
|
* `artifact_digest` — declared selected-target artifact SHA-256 (for a kernel,
|
|
the pinned provider executable digest);
|
|
* `selected_target` — the host triple selected at lock time;
|
|
* `source` — the `from` string exactly as declared.
|
|
|
|
`marqdo lock` records these identities statically (no extension is executed).
|
|
`check`/`sync` compare the current declaration to the lock and fail with
|
|
`ext.extension_lock_mismatch` on a missing, extra or changed identity; `run`
|
|
additionally verifies the real artifact bytes before starting any worker.
|
|
`extensions` is omitted when no extension is declared.
|
|
|
|
### 3.1 Lock v4 migration
|
|
|
|
`from_yaml` of a v3 document returns a deterministic error: lock v3 is
|
|
superseded by v4 (extension identities), and the user must run `marqdo lock`
|
|
to migrate explicitly. No v3 field is silently reinterpreted. Lock v1/v2 keep
|
|
their existing relock error with the v4 requirement.
|
|
|
|
| Field | Semantics |
|
|
|---|---|
|
|
| `lock_version` | writers emit `4`; readers accept only `4`; a v3 document gets the explicit "run `marqdo lock`" migration error and v1/v2 get the relock error |
|
|
| `content_profile_version` | file-set/digest profile; only `1` is accepted |
|
|
| `root.package` | logical root instance `root:<name>@<version>` |
|
|
| `root.manifest_digest` | lowercase hex SHA-256 of the canonical `marqdo.yaml` |
|
|
| `packages[].instance_id` | `dep:<name>@<version>#<full content digest>` |
|
|
| `packages[].source` | `{kind: path, path}` (project-relative) or `{kind: git, url, rev}` (full commit hash) |
|
|
| `packages[].content_digest` | 64-character lowercase hex SHA-256 of the package file set |
|
|
| `packages[].manifest_digest` | optional lowercase hex digest of the dependency's own manifest |
|
|
| `packages[].dependencies` | this instance's alias → target instance edges |
|
|
| `root_dependencies` | root manifest alias → target instance edges |
|
|
|
|
The graph is the unit of reproducibility: a transitive dependency that can be
|
|
imported is also locked. Edges are keyed by `(parent instance, alias)`, so the
|
|
same alias in two packages may resolve to different instances.
|
|
|
|
### 3.2 Serialization and validation
|
|
|
|
`lock` writes atomically (temporary file, `sync_all`, rename); a failed
|
|
resolution leaves the previous lock untouched. `LockFile::to_yaml` sorts
|
|
packages by `instance_id`. The format contains no timestamps, absolute paths or
|
|
machine-specific values, so repeated `lock` runs over unchanged inputs are
|
|
byte-identical.
|
|
|
|
`LockFile::from_yaml` validates before returning:
|
|
|
|
* `lock_version == 4`, `content_profile_version == 1`;
|
|
* unique ids, each consistent with `dep:<name>@<version>#<content_digest>`;
|
|
* every digest is 64 lowercase hex characters;
|
|
* every source is normalized (path relative to the project root without `..`,
|
|
Git revision a full lowercase 40/64-character hash);
|
|
* every edge target exists and every package is reachable from
|
|
`root_dependencies`; empty aliases and self-edges are errors.
|
|
|
|
v1/v2 locks are rejected, not silently upgraded: a flat lock cannot represent
|
|
the closure and treating it as complete would re-open fail-open loading.
|
|
|
|
### 3.3 Tree hash
|
|
|
|
For a path or Git dependency, `lock` hashes the resolved package tree with
|
|
`content_profile_version: 1` (`crates/marqdo-package/src/fileset.rs`, ADR-0016):
|
|
|
|
1. walk the dependency directory with `walkdir` (`sort_by_file_name`,
|
|
`follow_links(false)`), propagating every walk error;
|
|
2. exclude any path component `.git` or `.marqdo`; include `target/`;
|
|
3. accept regular files only. Symlinks and special files are hard errors, not
|
|
skipped or followed; paths must be UTF-8, `/`-separated, relative and
|
|
duplicate-free;
|
|
4. sort files by root-relative path (UTF-8 bytes); enforce budgets of 10 000
|
|
files, 64 MiB per file, 512 MiB total and depth 64;
|
|
5. hash with domain separation and length prefixes:
|
|
`b"marqdo.package.file-set\0"`, `profile_version: u32le`,
|
|
`entry_count: u64le`, then for each file
|
|
`rel_len: u64le`, `rel_bytes`, `content_len: u64le`, `content_bytes`;
|
|
6. hex-encode the digest.
|
|
|
|
Materialization uses the same selection and reads each file once, hashing the
|
|
exact bytes it writes; the published environment is accepted only when the
|
|
final digest equals the lock. This is a deterministic content fingerprint, not
|
|
a Merkle tree and not a commitment to file modes or empty directories. Modes
|
|
are normalized on materialization (`0644` files, `0755` directories on Unix).
|
|
|
|
### 3.4 Lock generation rules
|
|
|
|
`marqdo lock` (run in a project):
|
|
|
|
1. discovers the project root from the cwd;
|
|
2. loads and validates `marqdo.yaml`;
|
|
3. traverses the full dependency closure in sorted manifest order:
|
|
* a `path` source must resolve inside the project root; its module root is
|
|
hashed and recorded as a project-relative path;
|
|
* a `git` source requires a full lowercase commit hash. `lock` clones into
|
|
`.marqdo/cache/git/<url-digest>-<rev>`, checks out the revision, verifies
|
|
`git rev-parse HEAD` equals it, strips `.git`, hashes the tree, and writes
|
|
a sibling `.meta.json` marker (`url`, `rev`, profile, digest). Install
|
|
hooks and submodules are not run;
|
|
* package cycles, duplicate ids with conflicting content, depth >128 and
|
|
more than 4096 instances are errors;
|
|
4. records every package instance with its alias→instance edges, plus the root
|
|
manifest digest and root edges;
|
|
5. validates the graph and atomically writes `marqdo.lock` (exit 0,
|
|
`locked N package(s)`).
|
|
|
|
What it does **not** do: resolve SemVer ranges, download a registry, verify
|
|
publisher signatures, install packages or run dependency scripts. A dependency's
|
|
own `marqdo.yaml` supplies its dependency edges.
|
|
|
|
## 4. `sync`
|
|
|
|
`marqdo sync [--locked]` (run in a project):
|
|
|
|
1. discovers the root; loads and validates the existing `marqdo.lock`
|
|
(missing or v1/v2 lock → exit 1 with a relock instruction);
|
|
2. verifies the lock belongs to the current root manifest
|
|
(`root.package` and `root.manifest_digest`); drift fails closed;
|
|
3. materializes **every** package in the locked closure into a project-private
|
|
`.marqdo/env/pkg-<sha256(instance-id)>/` tree using the profile-1 file set
|
|
and the lock's content digest;
|
|
4. writes `.marqdo/env/env.json` schema 2 with the content profile, canonical
|
|
lock digest, root edges and one record per package; publishes via staging +
|
|
rename, restoring the previous environment on failure;
|
|
5. never rewrites the lock. Plain `sync` may fetch a pinned Git revision named
|
|
by the lock; `--locked` forbids fetching. Both modes verify contents and
|
|
fail with "lockfile is out of date or dependency contents changed; run
|
|
`marqdo lock`" on drift.
|
|
|
|
`run`, `call`, `check` and `inspect` never fetch and never write the lock. With
|
|
declared dependencies they require a lock. They resolve each `(parent instance,
|
|
alias)` edge in the lock; missing entries, digest/source drift and root-manifest
|
|
drift are errors. A materialized env or Git cache directory is trusted only
|
|
after its file-set digest and cache marker verify.
|
|
|
|
## 5. `init`
|
|
|
|
`marqdo init [dir]`:
|
|
|
|
* refuses to overwrite an existing `marqdo.yaml` (exit 1);
|
|
* creates the directory if needed;
|
|
* derives the project name from the directory's final path component (falling
|
|
back to `example`);
|
|
* writes `marqdo.yaml` from `Manifest::template`:
|
|
|
|
```yaml
|
|
project:
|
|
name: <name>
|
|
version: 0.1.0
|
|
marqdo: '>=3.0.0-beta.1,<4.0.0'
|
|
```
|
|
|
|
(empty dependencies/plugins/capabilities are omitted);
|
|
* writes `index.mq.md` using the RFC-LANG-0002 syntax:
|
|
|
|
```markdown
|
|
## @exec main
|
|
|
|
`问候 = "你好,Marqdo"`
|
|
|
|
`打印(问候)`
|
|
|
|
`return 问候`
|
|
```
|
|
|
|
* writes `.gitignore` containing `.marqdo/`;
|
|
* prints the next-step hint `marqdo check index.mq.md && marqdo run index.mq.md`.
|
|
|
|
`init` does not create `marqdo.lock`, `.marqdo/`, a plugin directory or a git
|
|
repository.
|
|
|
|
## 6. `add` / `remove`
|
|
|
|
Both commands edit **only** `marqdo.yaml` (and, for `remove`, prune a matching
|
|
stale lock entry). They do not fetch, install, resolve or authorize anything.
|
|
|
|
* `marqdo add <name> --path <dir>` or
|
|
`marqdo add <name> --git <url> --rev <sha>`:
|
|
validate the dependency shape (same rules as §2.2), refuse an existing name
|
|
(exit 1), insert it, write the manifest, and tell the user to run
|
|
`marqdo lock`. A pinned Git dependency is lockable, resolvable and runnable.
|
|
* `marqdo remove <name>`: remove the dependency from the manifest (exit 1 if
|
|
absent), and drop a same-named entry from an existing lock file. It does not
|
|
re-hash or re-lock the rest of the project.
|
|
|
|
This differs from the original task brief that listed `add`/`remove` as
|
|
"not supported yet": the CLI does implement them as manifest editors. They
|
|
are intentionally shallow and should not be confused with a dependency
|
|
resolver.
|
|
|
|
## 7. JSON output
|
|
|
|
`check`, `inspect`, `run`, `call`, `plugin inspect`, `add`, `remove`, `lock`
|
|
and `sync` accept `--json` and print a single JSON object on stdout. Relevant
|
|
package shapes:
|
|
|
|
* `lock`: `{"command":"lock","success":true,"packages":[...]}`
|
|
* `sync`: `{"command":"sync","success":true,"locked":<bool>,"packages":[...]}`
|
|
* `add`: `{"command":"add","success":true,"dependency":"<name>","manifest":"<path>"}`
|
|
* `remove`: `{"command":"remove","success":true,"dependency":"<name>"}`
|
|
|
|
Errors in these commands are printed as plain `error: ...` on stderr even when
|
|
`--json` is passed (the JSON success envelope is not emitted; this is an
|
|
inconsistency to fix).
|
|
|
|
## 8. Not supported in this build (explicit)
|
|
|
|
| Capability | Status |
|
|
|---|---|
|
|
| Git dependency resolution (pinned rev, clone/fetch/checkout) | Implemented for exact revisions; no floating branches, submodules or dependency scripts. |
|
|
| Registry / package publishing / SemVer resolution | Not implemented. |
|
|
| Transitive dependency graph and conflict resolution | The lock records the full transitive closure with per-instance edges; two versions of one package can coexist. A general SemVer conflict solver is not implemented. |
|
|
| `tree`, `update`, `vendor`, `package` commands | Not implemented. |
|
|
| `--offline`, `--locked` on `lock` | Not implemented; missing cache fails instead of fetching during check/run. |
|
|
| Environment materialization (`.marqdo/env`, `cache`) | Implemented: `sync` materializes the full locked closure into `.marqdo/env` atomically; `lock`/`sync` populate a verified Git cache. Kernels/sessions are not materialized. |
|
|
| Dependency signatures / trusted publishers / supply-chain scanning | Not implemented. |
|
|
| Multiple/nested project boundary diagnostics | Not implemented; nearest `marqdo.yaml` wins. |
|
|
| Lock file migration between `lock_version`s | Only v4 is read. v3 gets the explicit "lock v3 is superseded by lock v4; run `marqdo lock`" error; v1/v2 get the relock instruction. `lock` regenerates the graph. |
|
|
| Plugin installation / plugin dependency management | No package-manager install. `plugins:` is a declarative version requirement; explicit `extensions.plugins` activation plus lock v4 manifest/artifact identity is implemented and re-verified at run time. |
|
|
|
|
The lock/package format is deliberately small and deterministic so these can
|
|
be added later without changing the identity model. Changing `lock_version`,
|
|
`source` syntax or the tree-hash algorithm requires an ADR and a migration
|
|
plan.
|