Files
marqdo-cli/spec/package-format.md
T
cflmy 2c7c4f31e7
Linux strict security lane (G-C) / strict isolation escape matrix (push) Waiting to run
ci/woodpecker/push/main Pipeline was successful
ci/woodpecker/manual/performance Pipeline was successful
CI / fmt + clippy + test (Linux x86_64) (push) Waiting to run
CI / schema instance validation + consumers (push) Waiting to run
CI / runtime contract (macos-latest) (push) Waiting to run
CI / runtime contract (windows-latest) (push) Waiting to run
release(3.0.0-beta.1): product version migration, beta gates, archive tooling and developer docs
REL-01: workspace/CLI/Analysis product version 3.0.0-beta.1 with updated
Cargo.lock; project.marqdo is enforced against the running core with
package.marqdo_range_invalid/package.marqdo_incompatible fail-closed
diagnostics; Tool SemVer prerelease negotiation tests; init template and
examples use the beta range. Verify init derives the project name from the
target directory instead of falling back to 'example'.

REL-02: beta support matrix is Linux x86_64 only; macOS/Windows are explicitly
experimental/unclaimed; strict policy and G4 Beta-only WARN are documented for
the release evidence.

REL-03: deterministic release packaging (tools/release/package.py) ships the
CLI and provider worker together with SOURCE_SHA, RELEASE_INFO.json,
known-limitations, specs, developer docs, examples incl. built plugin/provider
artifacts and consumers; tools/release/smoke.py cold-install tests the archive.

REL-04: docs/developers entry pages; corrected and self-contained third-party
integration handbook; stale Proposed/native-only/informational spec claims
fixed; release/status/compatibility/changelog/releasing docs updated.

REL-05: release_evidence.py gains an explicit beta profile
(BETA_RELEASE_CANDIDATE) distinct from the stable profile; Woodpecker build
step now packages and smoke-tests the archive on every pipeline.

Adds a runnable Resource Provider v2 example and corrects check --emit to
Analysis v2.
2026-11-22 02:31:32 +08:00

18 KiB

Marqdo Project Package Format v4

Status: lock_version 4 package-instance graph plus extension identities (RFC-EXT-0001 §6, ADR-0056). Path and pinned Git sources, transitive closure locking, fail-closed locked resolution, strict sync, content-profile-1 file-set hashing/materialization, and explicit extensions: activation sources/identities are implemented. Registries and SemVer solving remain out of scope; see §8. Lock v3 is rejected with an explicit marqdo lock migration error; it is never reinterpreted silently.

Implementation: crates/marqdo-package/src/lib.rs (manifest/lock types and validation), crates/cli/src/commands.rs (init, add, remove, lock, sync).

1. Project root

The project root is the nearest ancestor directory of the entry document that contains marqdo.yaml (marqdo_package::discover_root). If no manifest exists anywhere above the entry, the entry file's parent directory is used as the root (a document can be checked/run without a project).

ModuleIds are package-instance + package-relative paths (spec/references.md §2.1), so the project root selects the entry package instance; physical paths do not form module identity. add/remove/lock/sync discover the root from the current working directory, not from a document path. There is no --project flag in this build.

2. marqdo.yaml

project:
  name: example
  version: 0.1.0
  marqdo: ">=3.0.0-beta.1,<4.0.0"   # optional; enforced against the running core
dependencies:
  geometry:
    path: ./packages/geometry # exactly one of path / git+rev
  quantum:
    git: https://github.com/example/quantum-mq
    rev: 0123456789abcdef0123456789abcdef01234567
plugins:
  solver:
    version: "^1.0"            # requirement only; not installed or authorized
capabilities:                  # requests only; not grants
  fs:
    read: ["./data/**"]
    write: ["./output/**"]
  net:
    hosts: []
  process: false
extensions:                    # explicit activation sources; never grants
  plugins:
    - from: ./adapter/plugin.yaml
  kernels: []
  providers:
    - from: "@quantum/adapter/provider.json"

2.1 Fields

Field Type Required Semantics
project.name string yes non-empty package name
project.version string yes non-empty version string (not semver-validated in the manifest)
project.marqdo string no Marqdo core SemVer requirement; enforced against the running core, incompatible ranges are hard errors
dependencies.<name>.path path string one of path/git project-relative path to a dependency directory or .mq.md file
dependencies.<name>.git URL string one of path/git git source (must be pinned with rev)
dependencies.<name>.rev string with git exact revision; a bare git without rev is rejected
plugins.<name>.version string per entry non-empty plugin version requirement; declarative only
capabilities.fs.read / .write list of strings no requested project-relative globs
capabilities.net.hosts list of strings no requested hosts
capabilities.process bool no requested process execution
extensions.plugins[] / .kernels[] / .providers[] {from} no explicit extension activation sources; from is ./relative/path in the project or @alias/relative/path in a declared dependency. The alias must be in dependencies. Absolute paths, URLs, globs and .. traversal are rejected. A declaration is not a grant.

Unknown fields are ignored (the types do not deny_unknown_fields). Duplicate YAML keys follow serde_yaml behavior (last one wins); there is no duplicate-key diagnostic.

2.2 Validation (Manifest::validate)

marqdo.yaml is loaded with Manifest::from_yaml/Manifest::load, which validates:

  • project.name and project.version are non-empty after trimming;
  • when project.marqdo is present it must parse as a semver::VersionReq and match the running core version (package.marqdo_range_invalid / package.marqdo_incompatible). SemVer prerelease rules apply: a stable-only range such as >=3.0.0,<4.0.0 does not accept 3.0.0-beta.1; prerelease users must opt in with a range whose comparator carries the prerelease (for example >=3.0.0-beta.1,<4.0.0);
  • for every dependency, exactly one source is present:
    • both path and git → error "declares both path and git";
    • neither → error "must declare a path or a pinned git source";
    • git without rev → error requiring an exact revision;
    • an absolute path → error "path must be project-relative";
  • for every plugin requirement, version is non-empty;
  • every extensions.*[].from is one of the two allowed forms and resolves to a regular file inside its owning package; duplicates within a kind are rejected; @alias must name a declared dependency. Resolution happens when the project is loaded (PackageResolver::resolve_extension_sources), not in plain Manifest::validate.

A pinned git dependency is resolvable, lockable and runnable; lock/sync clone it into the content-addressed cache and verify the checked-out commit and tree digest (§4.3).

2.3 capabilities are requests

The manifest is the project's request for permissions. It is never read when constructing the host capability set; only CLI flags/grant APIs create grants (spec/security.md §2). net has no implementing host op, and process/foreign execution still needs --allow-foreign.

3. marqdo.lock

lock_version: 4 records the complete package-instance closure reachable from the root manifest plus the exact identity of every declared extension. Packages are sorted by instance_id; extensions by (kind, name, logical_owner):

lock_version: 4
content_profile_version: 1
root:
  package: root:quantum@0.1.0
  manifest_digest: 4f5c...
packages:
- instance_id: dep:math@1.0.0#9e2...
  name: math
  version: 1.0.0
  source:
    kind: path
    path: deps/math
  content_digest: 9e2...
  manifest_digest: 7a1...
  dependencies:
    helper: dep:helper@2.0.0#1b3...
- instance_id: dep:helper@2.0.0#1b3...
  name: helper
  version: 2.0.0
  source:
    kind: git
    url: https://example.invalid/helper.git
    rev: 0123456789abcdef0123456789abcdef01234567
  content_digest: 1b3...
  manifest_digest: 3c9...
  dependencies: {}
root_dependencies:
  math: dep:math@1.0.0#9e2...
extensions:
- kind: plugin
  logical_owner: root:quantum@0.1.0
  name: demo.solver
  version: 0.1.0
  manifest_digest: 4f5c...            # sha256 of the exact plugin.yaml bytes
  artifact_digest: 9e2...             # selected target artifact sha256
  selected_target: x86_64-unknown-linux-gnu
  source: ./adapter/plugin.yaml

An extension lock record binds:

  • kind — plugin | kernel | provider;
  • logical_owner — root instance or dependency package instance id;
  • name / version — stable declared identity (kernel descriptors have no version, so the lock records wire:<n>);
  • manifest_digest — SHA-256 of the exact manifest/descriptor bytes;
  • artifact_digest — declared selected-target artifact SHA-256 (for a kernel, the pinned provider executable digest);
  • selected_target — the host triple selected at lock time;
  • source — the from string exactly as declared.

marqdo lock records these identities statically (no extension is executed). check/sync compare the current declaration to the lock and fail with ext.extension_lock_mismatch on a missing, extra or changed identity; run additionally verifies the real artifact bytes before starting any worker. extensions is omitted when no extension is declared.

3.1 Lock v4 migration

from_yaml of a v3 document returns a deterministic error: lock v3 is superseded by v4 (extension identities), and the user must run marqdo lock to migrate explicitly. No v3 field is silently reinterpreted. Lock v1/v2 keep their existing relock error with the v4 requirement.

Field Semantics
lock_version writers emit 4; readers accept only 4; a v3 document gets the explicit "run marqdo lock" migration error and v1/v2 get the relock error
content_profile_version file-set/digest profile; only 1 is accepted
root.package logical root instance root:<name>@<version>
root.manifest_digest lowercase hex SHA-256 of the canonical marqdo.yaml
packages[].instance_id dep:<name>@<version>#<full content digest>
packages[].source {kind: path, path} (project-relative) or {kind: git, url, rev} (full commit hash)
packages[].content_digest 64-character lowercase hex SHA-256 of the package file set
packages[].manifest_digest optional lowercase hex digest of the dependency's own manifest
packages[].dependencies this instance's alias → target instance edges
root_dependencies root manifest alias → target instance edges

The graph is the unit of reproducibility: a transitive dependency that can be imported is also locked. Edges are keyed by (parent instance, alias), so the same alias in two packages may resolve to different instances.

3.2 Serialization and validation

lock writes atomically (temporary file, sync_all, rename); a failed resolution leaves the previous lock untouched. LockFile::to_yaml sorts packages by instance_id. The format contains no timestamps, absolute paths or machine-specific values, so repeated lock runs over unchanged inputs are byte-identical.

LockFile::from_yaml validates before returning:

  • lock_version == 4, content_profile_version == 1;
  • unique ids, each consistent with dep:<name>@<version>#<content_digest>;
  • every digest is 64 lowercase hex characters;
  • every source is normalized (path relative to the project root without .., Git revision a full lowercase 40/64-character hash);
  • every edge target exists and every package is reachable from root_dependencies; empty aliases and self-edges are errors.

v1/v2 locks are rejected, not silently upgraded: a flat lock cannot represent the closure and treating it as complete would re-open fail-open loading.

3.3 Tree hash

For a path or Git dependency, lock hashes the resolved package tree with content_profile_version: 1 (crates/marqdo-package/src/fileset.rs, ADR-0016):

  1. walk the dependency directory with walkdir (sort_by_file_name, follow_links(false)), propagating every walk error;
  2. exclude any path component .git or .marqdo; include target/;
  3. accept regular files only. Symlinks and special files are hard errors, not skipped or followed; paths must be UTF-8, /-separated, relative and duplicate-free;
  4. sort files by root-relative path (UTF-8 bytes); enforce budgets of 10 000 files, 64 MiB per file, 512 MiB total and depth 64;
  5. hash with domain separation and length prefixes: b"marqdo.package.file-set\0", profile_version: u32le, entry_count: u64le, then for each file rel_len: u64le, rel_bytes, content_len: u64le, content_bytes;
  6. hex-encode the digest.

Materialization uses the same selection and reads each file once, hashing the exact bytes it writes; the published environment is accepted only when the final digest equals the lock. This is a deterministic content fingerprint, not a Merkle tree and not a commitment to file modes or empty directories. Modes are normalized on materialization (0644 files, 0755 directories on Unix).

3.4 Lock generation rules

marqdo lock (run in a project):

  1. discovers the project root from the cwd;
  2. loads and validates marqdo.yaml;
  3. traverses the full dependency closure in sorted manifest order:
    • a path source must resolve inside the project root; its module root is hashed and recorded as a project-relative path;
    • a git source requires a full lowercase commit hash. lock clones into .marqdo/cache/git/<url-digest>-<rev>, checks out the revision, verifies git rev-parse HEAD equals it, strips .git, hashes the tree, and writes a sibling .meta.json marker (url, rev, profile, digest). Install hooks and submodules are not run;
    • package cycles, duplicate ids with conflicting content, depth >128 and more than 4096 instances are errors;
  4. records every package instance with its alias→instance edges, plus the root manifest digest and root edges;
  5. validates the graph and atomically writes marqdo.lock (exit 0, locked N package(s)).

What it does not do: resolve SemVer ranges, download a registry, verify publisher signatures, install packages or run dependency scripts. A dependency's own marqdo.yaml supplies its dependency edges.

4. sync

marqdo sync [--locked] (run in a project):

  1. discovers the root; loads and validates the existing marqdo.lock (missing or v1/v2 lock → exit 1 with a relock instruction);
  2. verifies the lock belongs to the current root manifest (root.package and root.manifest_digest); drift fails closed;
  3. materializes every package in the locked closure into a project-private .marqdo/env/pkg-<sha256(instance-id)>/ tree using the profile-1 file set and the lock's content digest;
  4. writes .marqdo/env/env.json schema 2 with the content profile, canonical lock digest, root edges and one record per package; publishes via staging + rename, restoring the previous environment on failure;
  5. never rewrites the lock. Plain sync may fetch a pinned Git revision named by the lock; --locked forbids fetching. Both modes verify contents and fail with "lockfile is out of date or dependency contents changed; run marqdo lock" on drift.

run, call, check and inspect never fetch and never write the lock. With declared dependencies they require a lock. They resolve each (parent instance, alias) edge in the lock; missing entries, digest/source drift and root-manifest drift are errors. A materialized env or Git cache directory is trusted only after its file-set digest and cache marker verify.

5. init

marqdo init [dir]:

  • refuses to overwrite an existing marqdo.yaml (exit 1);

  • creates the directory if needed;

  • derives the project name from the directory's final path component (falling back to example);

  • writes marqdo.yaml from Manifest::template:

    project:
      name: <name>
      version: 0.1.0
      marqdo: '>=3.0.0-beta.1,<4.0.0'
    

    (empty dependencies/plugins/capabilities are omitted);

  • writes index.mq.md using the RFC-LANG-0002 syntax:

    ## @exec main
    
    `问候 = "你好,Marqdo"`
    
    `打印(问候)`
    
    `return 问候`
    
  • writes .gitignore containing .marqdo/;

  • prints the next-step hint marqdo check index.mq.md && marqdo run index.mq.md.

init does not create marqdo.lock, .marqdo/, a plugin directory or a git repository.

6. add / remove

Both commands edit only marqdo.yaml (and, for remove, prune a matching stale lock entry). They do not fetch, install, resolve or authorize anything.

  • marqdo add <name> --path <dir> or marqdo add <name> --git <url> --rev <sha>: validate the dependency shape (same rules as §2.2), refuse an existing name (exit 1), insert it, write the manifest, and tell the user to run marqdo lock. A pinned Git dependency is lockable, resolvable and runnable.
  • marqdo remove <name>: remove the dependency from the manifest (exit 1 if absent), and drop a same-named entry from an existing lock file. It does not re-hash or re-lock the rest of the project.

This differs from the original task brief that listed add/remove as "not supported yet": the CLI does implement them as manifest editors. They are intentionally shallow and should not be confused with a dependency resolver.

7. JSON output

check, inspect, run, call, plugin inspect, add, remove, lock and sync accept --json and print a single JSON object on stdout. Relevant package shapes:

  • lock: {"command":"lock","success":true,"packages":[...]}
  • sync: {"command":"sync","success":true,"locked":<bool>,"packages":[...]}
  • add: {"command":"add","success":true,"dependency":"<name>","manifest":"<path>"}
  • remove: {"command":"remove","success":true,"dependency":"<name>"}

Errors in these commands are printed as plain error: ... on stderr even when --json is passed (the JSON success envelope is not emitted; this is an inconsistency to fix).

8. Not supported in this build (explicit)

Capability Status
Git dependency resolution (pinned rev, clone/fetch/checkout) Implemented for exact revisions; no floating branches, submodules or dependency scripts.
Registry / package publishing / SemVer resolution Not implemented.
Transitive dependency graph and conflict resolution The lock records the full transitive closure with per-instance edges; two versions of one package can coexist. A general SemVer conflict solver is not implemented.
tree, update, vendor, package commands Not implemented.
--offline, --locked on lock Not implemented; missing cache fails instead of fetching during check/run.
Environment materialization (.marqdo/env, cache) Implemented: sync materializes the full locked closure into .marqdo/env atomically; lock/sync populate a verified Git cache. Kernels/sessions are not materialized.
Dependency signatures / trusted publishers / supply-chain scanning Not implemented.
Multiple/nested project boundary diagnostics Not implemented; nearest marqdo.yaml wins.
Lock file migration between lock_versions Only v4 is read. v3 gets the explicit "lock v3 is superseded by lock v4; run marqdo lock" error; v1/v2 get the relock instruction. lock regenerates the graph.
Plugin installation / plugin dependency management No package-manager install. plugins: is a declarative version requirement; explicit extensions.plugins activation plus lock v4 manifest/artifact identity is implemented and re-verified at run time.

The lock/package format is deliberately small and deterministic so these can be added later without changing the identity model. Changing lock_version, source syntax or the tree-hash algorithm requires an ADR and a migration plan.