From 0116160399fda6df69a72da438e04b0072566b0f Mon Sep 17 00:00:00 2001 From: cflmy Date: Fri, 20 Nov 2026 10:56:26 +0800 Subject: [PATCH] docs: formal specs, ADRs, STATUS matrix, contributor guide and honest README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete the documentation set for the M0.5 toolchain, grounded in the code: - spec/references.md: import/reference/call edges, identities, loader rules, anchors, `ref` resolution/ambiguity, `check --refs` validation. - spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal string rule, special floats, limits and exact float round-trip. - spec/security.md: modes (strict refuses without OS isolation), capabilities, path rules, budgets, tracing, foreign process caveats, plugin trust. - spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync, unsupported registry/git/environment work. - spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the implemented CLI plugin-invocation flow with its limitations. - spec/math.md and spec/foreign.md: the implemented math subset and the foreign request/response/timeout/capability contract. - docs/adr/0001..0007: accepted decisions for the span foundation, inert executables outside function bodies, ref ambiguity (S-07), strict refusal, AST normalization, CLI contract and BigInt/typed wire (S-08). - docs/contributor-guide.md: repo map, workflow, fixture/golden updates, quality gate, ADR/commit policy. - docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18 Alpha checklist (conclusion: not yet Alpha-complete), including known divergences rather than overclaims. - README.md rewritten from "design-only starter" to the working implementation with quickstart, implemented phases, deliberate limits and doc index. - AGENTS.md: updated pointers/workflow only; architectural rules unchanged. - Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside functions, unsupported math.simplify/differentiate). --- AGENTS.md | 6 +- README.md | 214 +++++++-- docs/STATUS.md | 217 +++++++++ .../0001-crate-graph-and-span-foundation.md | 139 ++++++ ...002-executables-only-in-function-bodies.md | 140 ++++++ docs/adr/0003-ref-resolution-and-ambiguity.md | 157 +++++++ ...4-strict-mode-refuses-without-isolation.md | 133 ++++++ docs/adr/0005-ast-normalization.md | 174 ++++++++ docs/adr/0006-cli-contract.md | 225 ++++++++++ .../0007-arbitrary-precision-int-and-wire.md | 178 ++++++++ docs/contributor-guide.md | 286 ++++++++++++ spec/evaluation.md | 4 +- spec/foreign.md | 221 +++++++++ spec/math.md | 227 ++++++++++ spec/package-format.md | 258 +++++++++++ spec/plugin-abi.md | 419 ++++++++++++++++++ spec/references.md | 362 +++++++++++++++ spec/security.md | 276 ++++++++++++ spec/value-wire.md | 161 +++++++ 19 files changed, 3767 insertions(+), 30 deletions(-) create mode 100644 docs/STATUS.md create mode 100644 docs/adr/0001-crate-graph-and-span-foundation.md create mode 100644 docs/adr/0002-executables-only-in-function-bodies.md create mode 100644 docs/adr/0003-ref-resolution-and-ambiguity.md create mode 100644 docs/adr/0004-strict-mode-refuses-without-isolation.md create mode 100644 docs/adr/0005-ast-normalization.md create mode 100644 docs/adr/0006-cli-contract.md create mode 100644 docs/adr/0007-arbitrary-precision-int-and-wire.md create mode 100644 docs/contributor-guide.md create mode 100644 spec/foreign.md create mode 100644 spec/math.md create mode 100644 spec/package-format.md create mode 100644 spec/plugin-abi.md create mode 100644 spec/references.md create mode 100644 spec/security.md create mode 100644 spec/value-wire.md diff --git a/AGENTS.md b/AGENTS.md index d50ff76..decc482 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Marqdo CLI — Agent / Contributor Operating Rules -**Start here:** read `docs/DEVELOPMENT_BLUEPRINT.md` before proposing code. It is the project architecture baseline. Then read the relevant `spec/` files and conformance fixtures. Do not assume any current implementation or legacy Marqdo syntax is authoritative. +**Start here:** read `docs/DEVELOPMENT_BLUEPRINT.md` before proposing code. It is the project architecture baseline. Then read `docs/STATUS.md` (what is actually implemented), the relevant `spec/` files, `docs/adr/` for frozen decisions, `docs/contributor-guide.md` for the workflow, and `tests/README.md` plus the conformance fixtures. Do not assume any current implementation or legacy Marqdo syntax is authoritative. ## Non-negotiable architectural constraints @@ -20,8 +20,8 @@ 1. Identify the affected rule and cite the section of `docs/DEVELOPMENT_BLUEPRINT.md`. 2. Add a small `.mq.md` positive/negative conformance fixture and define the expected Markdown AST and execution IR. 3. Implement only the smallest module that satisfies this fixture. Keep Markdown parser, semantic mapper, runtime and host separated. -4. Run formatting, clippy, unit tests, gold tests and relevant security/ABI compatibility tests. -5. Write the tradeoff and state of completion in the PR. Never say a feature is finished because a design example looks executable. +4. Run formatting, clippy, unit tests, gold tests and relevant security/ABI compatibility tests (`cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`, `cargo test -p marqdo-cli`). +5. Write the tradeoff and state of completion in the PR. Never say a feature is finished because a design example looks executable. Update `spec/` and `docs/STATUS.md` when behavior or status changes; the docs describe what the code does and flag blueprint divergences explicitly. ## Unresolved design topics diff --git a/README.md b/README.md index d6a19a3..63478e8 100644 --- a/README.md +++ b/README.md @@ -1,39 +1,203 @@ -# Marqdo CLI — next-generation design starter +# Marqdo CLI -> **Design-only starter package. This archive does not contain an executable CLI or a completed Rust implementation.** +Marqdo is a **Markdown-native, independently interpreted programming language**: +standard CommonMark/GFM structures carry program control (headings scope, +`**strong**` executes, `*emph*` returns, ordered lists branch, bullet lists +loop, block quotes call), prose and executable nodes mix in one document, and +all side effects go through a capability-checked host boundary. The official +implementation is a Rust workspace; Python/Node are optional foreign kernels, +never the compiler target. -Marqdo aims to be a **Markdown-native, independently interpreted programming language**: Markdown AST structures carry program control semantics; readable prose can coexist with execution, while runtime effects are explicit and controlled. The new toolchain is independent of the original `cflmy/marqdo` repository. +This repository is the new, independent Marqdo toolchain (M0.5). It is **not** +a compatibility rewrite of `cflmy/marqdo`; the old syntax, package format and +plugin ABI v2 are not claimed to work here. -## Read first +> **Status: working implementation, not an Alpha release.** P0/P1 and the +> documented Math/Foreign subset are implemented and tested; modules/package/ +> security/plugin support is functional but partial; strict isolation, worker +> plugins, git packages and performance/release work are not done. See +> [`docs/STATUS.md`](docs/STATUS.md) and "Deliberate limits" below. -- [`docs/DEVELOPMENT_BLUEPRINT.md`](docs/DEVELOPMENT_BLUEPRINT.md) — the comprehensive Chinese design and development specification: motivation, formal mapping, functions/parameters/calls/returns, execution, references, Math/Foreign kernels, environment and package manager, sandbox, plugin ABI/SDK, milestones, tests and related projects. -- [`AGENTS.md`](AGENTS.md) — succinct onboarding instructions for coding agents and new contributors. +## Build -## Planned CLI surface +Requires Rust 1.82+. ```bash -marqdo init -marqdo run index.mq.md -marqdo call index.mq.md main -marqdo check index.mq.md -marqdo inspect index.mq.md --ast -marqdo add -marqdo lock && marqdo sync -marqdo plugin inspect +cargo build -p marqdo-cli +# binary: target/debug/marqdo (or $CARGO_TARGET_DIR/debug/marqdo) ``` -These commands are **planned APIs**, not yet available in this starter package. Begin by implementing P0 tests and a reference Markdown AST explorer; do not start with a monolithic CLI or port old parser heuristics. +## Quickstart -## Design status +```bash +marqdo init demo +cd demo +marqdo check index.mq.md +marqdo run index.mq.md # executes ## main +marqdo inspect index.mq.md --ast +marqdo inspect index.mq.md --signature +marqdo call index.mq.md main +``` -The language direction and architecture principles have been selected. Detailed proposals still require ADRs and conformance testing before a stable version. See blueprint section 17 for unresolved items and section 18 for the Alpha definition of done. +Example project from scratch (`index.mq.md`): + +````markdown +# 数学 + +## 阶乘 + +1. `n <= 1` + + *1* + +2. `n > 1` + + **结果 = n * 阶乘(n=n-1)** + + *结果* +```` + +```bash +marqdo check examples/factorial.mq.md +marqdo call examples/factorial.mq.md 数学.阶乘 --arg n=5 # 120 +``` + +The parser is frozen: `marqdo.md.v1` = CommonMark 0.31.2 + GFM tables, +strikethrough, task lists + `$...$`/`$$...$$` math (comrak `=1.0.0`). A `.mq.md` +file still renders normally in any compatible Markdown reader; rendering never +executes anything. + +## What works (implemented and tested) + +* **P0 contracts** — frozen Markdown profile, AST → program mapping, strict + expression grammar, evaluation semantics; 150 AST goldens and 48 negative + compile-error fixtures (`tests/`). +* **P1 interpreter** — native Rust tree-walk interpreter: values, scopes, + functions, static parameter inference, assignments, branches, loops, + returns, recursion, structured errors, fuel/stack/output budgets. +* **P2 references + CLI** — explicit YAML front-matter imports, `ModuleId`/ + `UnitId`/anchors, import cycle detection, multi-module calls, + `run/call/check/inspect/fmt`, reference validation (`check --refs`) and the + static link graph (`inspect --refs`). +* **P3 project** — `marqdo.yaml` validation, deterministic `marqdo.lock` for + path dependencies, `init`, `add`/`remove` (manifest editing), `lock`, + `sync --locked`. +* **P4 security core** — capability-checked host operations, project-relative + FS globs with traversal rejection, budgets, effect trace, `pure`/`project` + modes, and **`strict` mode refuses to run** because no OS isolation adapter + exists. +* **P5 Math/Foreign** — native `Formula` value from `$$...$$`; `math.parse/ + eval/substitute` over a documented exact-rational subset; one-shot Python/ + Node kernels via `foreign.run/call` with typed wire protocol, timeout and + output limits. +* **P6 plugin ABI** — frozen C ABI v1 header (`include/marqdo_plugin_abi_v1.h`), + host bindings, Rust SDK (`mq_plugin_entry!`), `plugin.yaml` validation, + target-triple + sha256 artifact verification, C demo, and explicitly trusted + in-process plugin calls (`--plugin --trust-native-plugin --allow-plugin`). + +Run everything: + +```bash +cargo test --workspace +cargo test -p marqdo-cli # conformance corpus + CLI contract tests +cargo fmt --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +## Deliberate limits (do not assume) + +* **Strict sandbox is not implemented.** `--mode strict` exits 1; there is no + OS isolation or isolated plugin worker. `project` mode is cooperative and + in-process. +* **Native plugins run in-process** and require an explicit unsafe trust flag. + A plugin can crash or read the host; a C ABI is not a sandbox. +* **Package management is path-only in practice.** Git dependencies can be + declared but cannot be resolved/locked; there is no registry, transitive + resolution, `--offline`, `tree`, `update` or environment materialization. + Manifest `capabilities` are requests, never grants. +* **Reference validation is opt-in**: `check --refs` reports broken links; + plain `check` ignores them because links are static knowledge references. + Link-form calls cannot cross modules; cross-module calls use imported alias + expressions such as `geometry.圆面积(r=2)`. +* **`ref("#anchor")` is module-local** and errors with `ref.ambiguous` when a + heading range contains more than one formula/code block instead of picking + the first. +* **`fmt` is minimal** (line endings/trailing newlines), not AST-preserving + reformatting. +* **Math is a limited calculator**, not a CAS/prover: no `simplify`/ + `differentiate`/matrices/series; equation solving is affine-only. +* **Foreign kernels are unsandboxed one-shot child processes** (no sessions, + no state bridge); they run with the host user's filesystem/network access. +* Finite floats are IEEE-754 binary64; the typed wire round-trip is exact + (`serde_json/float_roundtrip`), tagged integers never lose digits, and + display formatting is decimal rather than bit-identical. + +## Repository layout + +```text +crates/marqdo-span shared byte spans/source maps (leaf) +crates/marqdo-value Value model, diagnostics, typed wire encoding +crates/marqdo-expression strict expression/statement parser +crates/marqdo-markdown frozen comrak profile, canonical AST, front matter +crates/marqdo-ir module/unit/function/statement IR, builtin table +crates/marqdo-frontend heading scopes, mapping, inference, static calls +crates/marqdo-math exact-rational Formula subset +crates/marqdo-foreign Python/Node one-shot kernel supervisor +crates/marqdo-sandbox mode parsing + platform isolation detection +crates/marqdo-plugin-abi C ABI v1 mirrors, manifest, artifacts, loader +crates/marqdo-plugin-sdk Rust plugin SDK and entry macro +crates/marqdo-host CapabilityHost, budgets, FS, math/foreign/plugin adapters +crates/marqdo-runtime tree-walk interpreter +crates/marqdo-package marqdo.yaml / marqdo.lock / root discovery +crates/cli CLI composition root (binary `marqdo`) +include/ frozen plugin C ABI header +spec/ normative language/format/ABI specs +docs/ blueprint, ADRs, status, contributor guide +examples/ runnable documents + multi-module + plugin demo +tests/ conformance/negative corpus (see tests/README.md) +``` + +## Documentation index + +* [`AGENTS.md`](AGENTS.md) — operating rules for contributors and coding agents. +* [`docs/DEVELOPMENT_BLUEPRINT.md`](docs/DEVELOPMENT_BLUEPRINT.md) — architecture + baseline (Chinese; the authoritative design document). +* [`docs/contributor-guide.md`](docs/contributor-guide.md) — repo map, workflow, + golden updates, quality gate. +* [`docs/STATUS.md`](docs/STATUS.md) — P0–P7 implementation matrix and honest + open items; blueprint §18 Alpha checklist. +* [`docs/adr/`](docs/adr/) — accepted decisions (crate graph, inert executables, + `ref` ambiguity, strict refusal, AST normalization, CLI contract, BigInt/wire). +* Specs: [`markdown-profile.md`](spec/markdown-profile.md), + [`mapping.md`](spec/mapping.md), [`expression.ebnf`](spec/expression.ebnf), + [`evaluation.md`](spec/evaluation.md), [`references.md`](spec/references.md), + [`value-wire.md`](spec/value-wire.md), [`security.md`](spec/security.md), + [`package-format.md`](spec/package-format.md), + [`plugin-abi.md`](spec/plugin-abi.md), [`math.md`](spec/math.md), + [`foreign.md`](spec/foreign.md). +* [`tests/README.md`](tests/README.md) — conformance corpus and `UPDATE_GOLDENS` + workflow. + +## Examples + +```bash +marqdo run examples/hello.mq.md +marqdo call examples/factorial.mq.md 数学.阶乘 --arg n=5 +marqdo run examples/multi-module/index.mq.md +marqdo run examples/python-foreign.mq.md --allow-foreign # needs python3 or node +marqdo inspect examples/references.mq.md --refs +marqdo inspect examples/hello.mq.md --platform # strict: unavailable + +./examples/plugin-demo/build.sh +marqdo plugin inspect examples/plugin-demo +marqdo run examples/plugin-demo/plugin-call.mq.md \ + --plugin examples/plugin-demo --trust-native-plugin --allow-plugin demo.solver +``` ## References -- [CommonMark](https://spec.commonmark.org/0.31.2/) -- [GFM](https://github.github.io/gfm/) -- [Comrak](https://comrak.ee/) -- [Cargo](https://doc.rust-lang.org/cargo/) -- [uv](https://docs.astral.sh/uv/) -- [Wasmtime Component Model](https://component-model.bytecodealliance.org/) -- [Legacy Marqdo source](https://github.com/cflmy/marqdo) +* [CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/) · + [GFM](https://github.github.io/gfm/) · [Comrak](https://comrak.ee/) +* [Cargo](https://doc.rust-lang.org/cargo/) · [uv](https://docs.astral.sh/uv/) +* [Wasmtime Component Model](https://component-model.bytecodealliance.org/) +* Legacy Marqdo source (reference only): diff --git a/docs/STATUS.md b/docs/STATUS.md new file mode 100644 index 0000000..d17b4ff --- /dev/null +++ b/docs/STATUS.md @@ -0,0 +1,217 @@ +# Marqdo Implementation Status (M0.5) + +Snapshot: **2026-11-20**, workspace version `0.1.0`, plugin ABI `1.0`, +`wire_version` 1, Markdown profile `marqdo.md.v1`. + +This file maps the blueprint's phases (`docs/DEVELOPMENT_BLUEPRINT.md` §13) to +concrete evidence in the repository and lists honest open items. It is a status +report, not a promise. "Done" means implemented **for the documented subset** +and covered by tests; it does not mean the blueprint's full ambition is +complete. + +Reproduce the evidence: + +```bash +export PATH=/root/.cargo/bin:$PATH +export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo +export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target +cargo test --workspace +cargo test -p marqdo-cli +cargo fmt --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +## 1. Phase matrix + +| Phase | Status | Main evidence | +|---|---|---| +| **P0** Formal specs + parser goldens | **Done** | `spec/markdown-profile.md`, `spec/mapping.md`, `spec/expression.ebnf`, `spec/evaluation.md`; frozen profile test `crates/marqdo-markdown/src/profile.rs`; 150 AST fixtures + goldens in `tests/conformance/ast/`; 48 compile-error fixtures in `tests/negative/`; `inspect --ast`; `conformance_ast.rs`/`negative.rs` harnesses. | +| **P1** Interpreter, expressions, control flow | **Done** | `marqdo-expression`, `marqdo-ir`, `marqdo-frontend`, `marqdo-runtime`; factorial/countdown examples run without Python (`examples/factorial.mq.md`, `examples/hello.mq.md`); 36 execution goldens in `tests/conformance/semantics/`; runtime tests `crates/marqdo-runtime/tests/interpreter.rs`; prose invariance `crates/cli/tests/properties.rs`. | +| **P2** Modules, references, CLI experience | **Partial** | Imports/ModuleId/UnitId/anchor/cycle detection + `run/call/check/inspect/fmt` (`crates/cli`), multi-module example and `multi_module_imports_and_cycles` CLI test; `check --refs` reference validation (all loaded documents, on-demand static analysis of link targets) and `inspect --refs` static graph. Missing: cross-module link calls, rename/interface-lock tooling, canonical AST-preserving `fmt` (see §3). | +| **P3** Package environment | **Partial** | `marqdo.yaml`/`marqdo.lock` types+validation (`marqdo-package`), `init`, `add`/`remove` (manifest editing), deterministic `lock`, `sync --locked`; CLI test `lock_and_sync_locked_detect_dependency_tampering`; `examples/multi-module`. Missing: git resolution, transitive deps, registry, `tree/update/vendor`, `--offline`, environment materialization. | +| **P4** Security core and isolation | **Partial** | `CapabilityHost`, capabilities at execution points (console and math included), project-rooted FS with traversal rejection, budgets, effect trace, pure/project modes, **strict refuses without OS isolation** (`marqdo-sandbox`); CLI test `strict_mode_refuses_without_isolation`; host unit tests. Missing: OS isolation adapter, plugin worker, network enforcement. | +| **P5** Math and Foreign | **Partial (documented subset)** | `marqdo-math` exact-rational formula kernel for the grammar in `spec/math.md`; `math.parse/eval/substitute` host ops; `marqdo-foreign` Python/Node one-shot kernels with typed `foreign.call`; tests `crates/marqdo-math/tests/kernel.rs`, `crates/marqdo-foreign/tests/kernels.rs`; examples `math.mq.md`, `python-foreign.mq.md`. Missing: `simplify`/`differentiate`, matrices/series/symbolic algebra, foreign sessions, Python environment management. | +| **P6** Plugin ABI and developer experience | **Partial** | Frozen `include/marqdo_plugin_abi_v1.h`; C ABI mirrors/layout assertions, version negotiation, `plugin.yaml` validation, target triple + sha256 verification, wire envelopes, native loader (`marqdo-plugin-abi`); Rust SDK macro (`marqdo-plugin-sdk`); C demo (`examples/plugin-demo`); CLI `plugin inspect` and explicitly trusted in-process invocation (`--plugin`, `--trust-native-plugin`, `--allow-plugin`); tests `plugin_inspect_and_explicit_trusted_invocation`, `crates/marqdo-plugin-abi/tests/loader_unix.rs`. Missing: isolated worker transport, plugin capability bridge, installation, Rust plugin example crate. | +| **P7** Performance, release, CI | **Partial** | `.github/workflows/ci.yml` runs fmt/clippy/workspace tests plus CLI example smoke tests (including plugin inspect and invocation) on Linux x86_64. Missing: benchmarks, bytecode backend, cross-platform matrix, signed releases and supply-chain checks. Tree-walk interpreter only. | + +## 2. Feature evidence (selected commands) + +| Command | Verified behavior | +|---|---| +| `marqdo init demo` | creates `marqdo.yaml` (`>=0.1,<1`), `index.mq.md`, `.gitignore` (`.marqdo/`); refuses existing manifest (exit 1). | +| `marqdo check index.mq.md` | static only; 0 when no error diagnostics (warnings allowed); 1 on errors; JSON `command/success/errors/diagnostics`. | +| `marqdo run index.mq.md` | executes `main` through the Rust interpreter/host; program stdout on stdout; exit 1 on runtime/compile/strict errors. | +| `marqdo call file fn --arg n=5` | typed constant arguments; prints the non-`Unit` return value. | +| `marqdo inspect file --ast/--units/--signature/--ir/--refs/--platform --json` | static dashboards; `--platform` reports strict unavailable. | +| `marqdo fmt [--check]` | normalizes CRLF and trailing newlines only; exit 1 if `--check` detects a change. | +| `marqdo plugin inspect dir` | validates manifest + verifies sha256; never `dlopen`s (`loaded: false`). | +| `marqdo run doc --plugin dir --trust-native-plugin --allow-plugin demo.solver` | calls `solver.add(a=20,b=22)` and prints `42`; without trust/grant it exits 1. | +| `marqdo lock` / `sync --locked` | deterministic path-dependency lock; `--locked` fails on manifest/dependency drift. | + +Test corpus at this snapshot (`tests/README.md`): + +* 150 AST fixtures + goldens (`tests/conformance/ast/`); +* 36 semantic execution fixtures + goldens (`tests/conformance/semantics/`); +* 48 compile-time hard-error fixtures, 6 inert-executable warning fixtures, + 10 runtime-error fixtures (`tests/negative/`); +* `crates/cli/tests/` harnesses: `conformance_ast`, `conformance_semantics`, + `negative`, `properties`, `integration_cli`; +* crate tests across all 15 crates, including real `python3`/`node` foreign + tests (skipped if unavailable) and a compiled C plugin test (skipped without + `cc`). + +At this snapshot `cargo test -p marqdo-cli` passes all 21 harness tests +(2 CLI unit + AST/semantics/negative/properties/integration) and +`cargo test --workspace` passes. The fixture counts above match +`tests/README.md`; empty `tests/integration`, `tests/properties`, +`tests/security` and `tests/abi` directories are reserved for future +workstreams — the active corpus lives in `tests/conformance`, `tests/negative` +and `crates/cli/tests`. + +## 3. Open items by phase + +### P0 + +* Wire floats now round-trip exactly: `serde_json`'s `float_roundtrip` + feature is enabled and a Python `math.pi * 9` fixture pins the exact value + (`spec/value-wire.md` §3). +* Profile change safety relies on the frozen-options unit test; no CI gate + regenerates goldens (intentional). + +### P1 + +* The blueprint asks for a draft Turing-completeness argument. The language + supports mutable locals, branch-first-match, per-item `while` loops and + recursion with bounded budgets; a formal encoding write-up is still missing. +* The default cooperative stack budget is 128 frames so very deep recursion + yields a structured `resource.exhausted` instead of exhausting the debug-build + native stack; `--stack` raises it deliberately. A true guard-page/iterative + interpreter hardening is still open. + +### P2 + +* `fmt` is intentionally minimal (line endings/trailing newlines only), not the + blueprint's "canonical AST-equivalent formatting"; no semantic formatting, + AST-equivalence check or render-preserving rewrite exists. +* Reference validation is opt-in: `check --refs` (and `inspect --refs`) resolve + links; plain `check` does not fail on broken links because links are static + knowledge references rather than execution inputs. +* Link-form calls cannot cross modules; cross-module calls use imported alias + expressions. The blueprint's `> [f](./other.mq.md#f)(...)` example is not + supported. +* `ref("#...")` is module-local and does not accept a selector for units with + multiple resources (it errors with `ref.ambiguous`, ADR-0003). +* No rename tooling, interface lock/ABI diffing, or incremental cache. +* Duplicate/nested project boundaries are not diagnosed (nearest + `marqdo.yaml` wins). + +### P3 + +* Git dependencies can be declared (`git` + `rev`, validated) but cannot be + resolved or locked: `lock` errors "git dependencies are not supported by this + build". +* No transitive dependency graph, conflict resolution, registry, publishing, + signing, `tree`, `update`, `vendor`, `--offline`, or `lock --locked`. +* No environment materialization (`.marqdo/env`, caches, kernels, sessions); + `sync` verifies but never installs or rewrites. +* `add`/`remove` are shallow manifest editors; adding a git dependency succeeds + at the manifest level even though `lock` will later refuse it. +* JSON errors from `add`/`remove`/`lock`/`sync` are plain stderr even with + `--json`. + +### P4 + +* **Strict mode cannot run at all**: no OS isolation adapter (Linux + namespaces/seccomp/cgroups, Windows Job Object/appcontainer, macOS sandbox) + and `worker_processes: false`; `run --mode strict` exits 1 (ADR-0004). +* `console` capability is always on; `print` is not capability-checked. +* Network capability (`net_hosts`, `check_net`) exists but no host operation + performs network I/O and there is no `--allow-net` flag. +* Write-path symlink TOCTOU caveat (`spec/security.md` §2.1); symlink/race + hardening open. +* Budgets do not cover every allocation path; the interpreter's containers are + bounded only by fuel and available memory. +* Effect trace does not record argument values or paths (privacy-positive, but + less diagnostic detail); trace is opt-in and implicitly on for `--json`. +* `plugin_handle` wire values are structurally reconstructable, not + unforgeable capability tokens. + +### P5 + +* Math is a deliberately limited calculator subset: no `simplify`, + `differentiate`, matrices, sums/integrals, trig/log functions, units, + assumptions or theorem proving. `math.simplify`/`math.differentiate` are + host ops that always return `math.unsupported`. +* Equation solving is affine-only, single target, target must occur exactly + once; other shapes are `math.unsupported`. +* Division by zero inside a formula maps to `runtime.type` (via + `MathErrorKind::Type`), not `runtime.division_by_zero`; documented divergence. +* Foreign kernels are one-shot only: no sessions, state bridge, Python/Node + environment management, or network policy for child processes. +* Foreign child processes are terminated only when the direct child is killed; + grandchildren are not tracked. +* Foreign per-call timeout (10 s) and output cap (1 MiB) are not CLI-configurable. + +### P6 + +* No isolated plugin worker (length-prefixed IPC) or Wasm transport; only + explicitly trusted in-process native loading. A native plugin crash is a host + crash (`spec/plugin-abi.md` §1/§9). +* No plugin capability bridge from the CLI: plugins are loaded with no + capability handler, so every plugin `request_capability` is denied. +* Plugin calls are not statically validated; unknown namespaces are runtime + `symbol.unknown`. Positional plugin arguments are rejected. +* One namespace per plugin (first export prefix); multi-prefix manifests are not + reachable beyond the first prefix. +* No plugin installation/package management; `plugins:` in `marqdo.yaml` is + declarative only. No Rust plugin example crate (SDK has doc examples/tests). +* No hot unload/reload (ABI v1 design). + +### P7 + +* Performance work is untouched: no benchmark suite, no bytecode backend or + differential tests against the tree-walker, no parse/compile caching. +* No CI workflows (`.github/` is empty of workflow files); no + Windows/macOS/Linux matrix, sanitizers/fuzz gates, signing, release + automation or supply-chain scanning. +* No published version, no compatibility matrix beyond the frozen ABI/format + documents. + +## 4. Blueprint §18 "Alpha Definition of Done" mapping + +| §18 item | Status | Evidence / gap | +|---|---|---| +| Run/call non-trivial recursive, branch, loop, call, parameter-inference programs without Python | **Met** | factorial/recursion (`examples/factorial.mq.md`, runtime + semantic goldens), countdown loop, multi-module alias call; interpreter is native Rust. | +| `.mq.md` readable under the chosen CommonMark/GFM/Math profile; standard AST behavior covered by goldens | **Met** | frozen `marqdo.md.v1`, 150 AST goldens, profile option test, `spec/markdown-profile.md`. | +| No heuristic downgrade of execution semantics; accurate source locations for invalid execution nodes | **Mostly met** | hard errors with spans for function-body executable shapes and negative corpus. Gaps: inert executable outside functions is a warning by design (ADR-0002); cross-module runtime error spans resolve against the entry file only; some lazy nested nodes are inert without a warning. | +| `check`/`inspect` side-effect free; imports, calls and static references verifiable across files | **Mostly met** | static load/analysis only; imports + cycle detection; cross-module calls validated; `check --refs` validates references across loaded documents and statically analyses link targets; `inspect --refs` prints the entry graph. | +| Minimal manifest, lock file, path/Git dependencies and environment sync | **Partial** | manifest/lock/path works, `sync --locked` verifies; git resolution, transitive deps, environment materialization and `--offline` missing. | +| All host side effects capability-authorized; untrusted code/native plugins cannot masquerade as strict sandbox | **Mostly met** | capability checks at execution points; strict refuses; native plugins require explicit trust and are documented as unsandboxed. Gaps: network enforcement not implemented (no network builtin is exposed), no worker isolation. | +| Formula computational subset explicit/testable; Foreign I/O, timeout and permissions explicit | **Met for the documented subset** | `spec/math.md`, `spec/foreign.md`; math kernel tests; foreign tests (timeout, protocol, output cap, env); permission gating tests/smoke. Simplification/differentiation explicitly unsupported. | +| Plugin C ABI v1, Rust SDK, C example, isolated worker basic path, ABI compat error tests | **Partial** | ABI/SDK/C example/loader/ABI-error tests exist; **isolated worker path missing**. | +| CI has at least one full research pipeline: Markdown function → Math → Foreign → Plugin → controlled output | **Partial** | CI now runs fmt/clippy/tests and CLI smoke tests; individual math/foreign/plugin paths are covered by examples and tests, but no single end-to-end pipeline document/test exists yet. | +| Examples, README, AGENTS and contributor docs allow a first change without legacy context | **Met** | `README.md`, `AGENTS.md`, this guide, `docs/STATUS.md`, specs, ADRs, `tests/README.md`, examples. | + +**Conclusion: the Alpha definition of done is not fully met.** P0/P1 and the +documented P5 subset are solid; P2/P3/P4/P6 are functional but partial; P7 is +absent. The blocking items for an Alpha claim are the strict sandbox/worker +isolation story (P4/P6), git/environment package management (P3), and a CI +end-to-end pipeline (P7). + +## 5. What is deliberately limited (do not overclaim) + +* Only a single, frozen Markdown profile (`marqdo.md.v1`); changing parser + options/versions is a language change. +* Strict mode is a refusal, not an isolation implementation. `project` mode is + cooperative and in-process. +* Native plugins are trusted in-process code; there is no isolated plugin + worker, and a plugin crash takes down the host. +* Ints are arbitrary precision; finite floats are binary64 and wire round-trip + is exact (`float_roundtrip` enabled), but decimal-vs-binary formatting still + applies when displaying values. +* Math is a limited exact-rational calculator, not a CAS or a prover. +* Foreign kernels are optional, one-shot, unsandboxed child processes. +* `fmt` is minimal; `check` does not yet validate Markdown links; link-form + calls cannot cross modules. +* Package management is path-only in practice; `marqdo.yaml` capabilities are + requests, never grants. diff --git a/docs/adr/0001-crate-graph-and-span-foundation.md b/docs/adr/0001-crate-graph-and-span-foundation.md new file mode 100644 index 0000000..f06f0c0 --- /dev/null +++ b/docs/adr/0001-crate-graph-and-span-foundation.md @@ -0,0 +1,139 @@ +# ADR-0001: Crate graph and the `marqdo-span` foundation + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +`docs/DEVELOPMENT_BLUEPRINT.md` §11.1 fixes the dependency direction for the +workspace and states: "`markdown ->` zero other Marqdo crate; responsible for a +stable view of the standard AST." The implemented workspace instead has +fifteen crates, and `marqdo-markdown` depends on `marqdo-span`. + +Why the divergence was necessary: + +* Every layer needs identical byte spans: the Markdown AST + (`crates/marqdo-markdown/src/ast.rs`), expression AST + (`crates/marqdo-expression`), IR (`crates/marqdo-ir`), diagnostics and runtime + errors (`crates/marqdo-value`), effect traces (`crates/marqdo-host`), plugin + errors (`crates/marqdo-plugin-abi`) and the CLI renderer. +* A `Span` must be the same type across boundaries or every layer needs + conversion functions. Conversions are exactly where off-by-one bugs and + broken source locations appear, and they would make `Expr`/`Stmt` + serde output inconsistent between `inspect --ast` and `inspect --ir`. +* The blueprint's "zero Marqdo dependencies" rule exists to keep the parser + independent of the semantic layers. A tiny, effect-free span crate preserves + that property while satisfying the shared-data-contract requirement. + +## Decision + +1. Add `crates/marqdo-span` as a **leaf** crate with no Marqdo dependencies and + only `serde` as an external dependency. It defines: + * `Span { start: usize, end: usize }` — half-open byte range into the + original UTF-8 source, with `merge`, `shift`, `contains`, `len`; + * `LineCol { line, column }` — 1-based, byte columns; + * `LineIndex` — precomputed line starts for byte→line/column conversion. +2. Treat `Span` as part of the toolchain's shared data contract. Crates that + produce or consume source locations depend on `marqdo-span` directly; they + do **not** wrap it in local span types. +3. `marqdo-markdown` depends on `marqdo-span` (and `comrak`) but on **no + semantic crate**. It still owns only the canonical AST view, front matter + extraction and source maps; it performs no mapping, resolution or execution. +4. Keep the rest of the blueprint's direction: + `value -> span`, `expression -> span`, `ir -> span + expression`, + `markdown -> span + comrak`, `frontend -> markdown + expression + ir + value`, + `host -> value + math + foreign + sandbox`, `runtime -> ir + value + host + + expression`, `cli -> all` (composition root). There is no `runtime -> cli` + edge and no cycle; `cargo tree` is expected to stay acyclic. +5. Do not introduce the blueprint's separate `sema` or `environment` crates in + M0.5: semantic analysis lives in `marqdo-frontend` + (`analyze`, `calls`, `infer`, `signature`, `units`, `map`), and package/ + environment formatting lives in `marqdo-package` plus the CLI. Splitting + them is a refactor, not a semantic change. + +Actual dependency graph (workspace `Cargo.toml` plus crate manifests): + +```text +span serde +value span, serde, serde_json, num-bigint, num-traits +expression span, serde, num-bigint, unicode-ident +markdown span, comrak, serde, serde_yaml, unicode-normalization +ir span, expression, serde +frontend span, markdown, expression, ir, value, sha2, + unicode-normalization, unicode-ident +math value, num-bigint, num-rational, num-traits, serde +foreign span, value, serde_json, thiserror +sandbox serde +plugin-abi span, value, serde, serde_json, serde_yaml, thiserror, + libloading, sha2, hex +plugin-sdk plugin-abi, serde, serde_json +host span, value, math, foreign, sandbox, serde, serde_json, + thiserror, glob +runtime span, value, ir, expression, host, num-bigint, + num-rational, num-traits, thiserror +package serde, serde_yaml +cli span, value, expression, markdown, ir, frontend, math, + foreign, plugin-abi, host, runtime, package, sandbox, + clap, serde, serde_json, walkdir, hex, sha2 +``` + +## Alternatives + +1. **Put `Span` in `marqdo-value`.** Then `marqdo-markdown` would depend on the + runtime value model, which is a semantic layer and pulls in `num-bigint` and + `serde_json`. Worse layering than a dedicated leaf. +2. **Duplicate `Span` in each crate and convert at boundaries.** Avoids a crate + but guarantees conversion code, mismatched serde shapes and duplicated + tests; rejected. +3. **Define `Span` in `marqdo-markdown` and let others depend on it.** Makes + the parser a de-facto foundation crate and lets non-Markdown layers depend + on `comrak`'s dependencies; rejected. +4. **Use a third-party range type.** Adds a dependency and gives no benefit + over ~170 lines; rejected. +5. **Keep the blueprint's literal "markdown has zero Marqdo deps".** Would + force either duplication or a span type in the parser that cannot be shared + with diagnostics; rejected in favor of a documented, narrow exception. + +## Markdown AST shape + +No syntax change. Every `MdNode` still has `span: Span` in the original file +(`crates/marqdo-markdown/src/ast.rs`), and the JSON inspection view is +`{"kind":"heading","level":2,"span":{"start":0,"end":7},...}`. Spans are +half-open byte offsets, BOM-rebased, and byte columns for diagnostics. + +## Compatibility + +* This is a **new repository**; there is no legacy compatibility burden. +* Public Rust API: moving `Span` later is a workspace-wide API change but has + no language/ABI/package-format impact. It would require updating every + `crates/*/Cargo.toml`. +* Stable observable contracts (AST JSON `span`, diagnostic JSON `span`, IR + `span`) are unchanged by the crate boundary. Changing span semantics + (e.g. line/column units) requires a spec change and ADR. +* The divergence from blueprint §11.1 is recorded here; future crates should + use `marqdo-span` rather than introducing another location type. + +## Security + +No new capability or effect. `marqdo-span` is `serde`-only, performs no I/O and +cannot execute anything. Byte offsets are derived from already-parsed source; +`LineIndex::line_col` clamps out-of-range offsets instead of panicking. + +## Testing + +* `crates/marqdo-span/src/lib.rs` tests multi-byte UTF-8 line/column mapping, + `merge`/`shift`/`contains`, and clamping. +* `crates/marqdo-markdown/src/profile.rs` tests BOM rebasing and that every AST + node carries a byte span; frontend tests assert diagnostics point at the + expected source line/column. +* `cargo test --workspace` must pass; `cargo tree` must remain acyclic (no + automated CI gate yet — this is a P7 item). + +## Migration & rollout + +Immediate: the crate exists and all layers already use it. Future work may +split the frontend into `sema`/`environment` crates; that must preserve the +graph above (`runtime` must never depend on `cli`, `markdown` must never depend +on semantic crates). If a second span representation is ever needed (for +example for generated code), add it as a conversion in `marqdo-span`, not as a +per-crate duplicate. diff --git a/docs/adr/0002-executables-only-in-function-bodies.md b/docs/adr/0002-executables-only-in-function-bodies.md new file mode 100644 index 0000000..ac0cc1f --- /dev/null +++ b/docs/adr/0002-executables-only-in-function-bodies.md @@ -0,0 +1,140 @@ +# ADR-0002: Executable nodes execute only inside level-2 function bodies + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +The blueprint (§1.2 execution context, §3.2 control flow, unresolved issue +**S-03**) establishes that `Strong` (execute), `Emph` (return), ordered lists +(branch), bullet lists (loop) and block quotes (call) have program meaning +**inside a function body**, i.e. inside the range of a level-2 heading. It +leaves the module top level under-specified: if a document has `**x = 1**` +before any function, should that be a hard error, a silent no-op, or an +explicit top-level statement? + +Two failure modes had to be avoided: + +1. **Silent execution at module scope.** There is no function frame, no return + target and no entry point, so top-level effects would be surprising and + would make `check`/`inspect` harder to reason about. +2. **Hard error.** Ordinary Markdown documents routinely use `**bold**`, + `*italic*`, `1. ordered` and `> quotes` in prose. Treating every such node + outside a function as a fatal compile error would make the language hostile + to normal documentation and would break the "renders normally" goal. + +The chosen baseline was: executable nodes only execute inside function bodies; +elsewhere they are ordinary document content, but this must not happen +silently. + +## Decision + +1. A **function body** is the child-block range of a `heading(level=2)` up to + the next heading of level ≤ 2 (`crates/marqdo-frontend/src/units.rs`, + `FunctionScope.body_start..body_end`). Executable roles are assigned only + inside those ranges by `crates/marqdo-frontend/src/map.rs`. +2. Outside every function body, `strong`, `emph`, `list` and `block_quote` + nodes are **inert document content**. They are not mapped to IR, are not + executed, and prose is not inspected for keywords. +3. The frontend emits a **warning** `markdown.inert_executable` for every + would-be execution node found outside a function body. The document still + compiles and runs; only `check`'s diagnostic output reflects the warning. + The warning is produced once per top-level node and has a hint to move the + construct under a `## function` heading. + * `block_quote` → "a block quote invocation ... is inert document content"; + * `list` → "a list control structure ..."; + * `paragraph` → the first `strong`/`emph` found in the paragraph + ("an execution statement" / "a return statement"). + * warnings are emitted only while scanning the document's direct top-level + blocks: headings, tables, code/math blocks and other non-paragraph blocks + produce none. A top-level paragraph is walked recursively, so a + `strong`/`emph` nested under a link or strikethrough inside that paragraph + can still produce the inert warning. Nodes inside a function body produce + no warning (they are mapped or rejected by the strict executable-shape + rules instead). +4. Warnings do not change exit codes: `check` exits 0 when there are no error + diagnostics even if warnings exist; `run`/`call` execute normally. JSON + `check` output sets `success = (errors == 0)` and includes the warning + entries in `diagnostics`. + +Rationale for the nodes covered: these are exactly the block-level constructs +that `map.rs` would otherwise turn into statements if a function body existed. +Inline nodes inside a paragraph are covered by one paragraph-level warning +rather than one per node, which keeps normal prose readable. + +## Alternatives + +1. **Hard error for every executable-looking node outside a function.** Rejected + (see context); it forbids emphasis in ordinary Markdown and makes writing a + README a compile failure. +2. **Silent inert behaviour with no diagnostic.** Rejected: an author could + believe top-level `**x = 1**` had executed. The blueprint explicitly wants + invalid execution intent surfaced. +3. **Execute top-level nodes in an implicit module frame.** Rejected: no + defined entry semantics, no return target, would make imported modules + potential side-effect roots (violating "modules register definitions only"). +4. **Compile-time error only when the payload parses as a statement.** Rejected: + this is the forbidden heuristic ("parses as code → execute") from blueprint + §1.3; role must come from node type and position, not from guessing. +5. **Warning per AST node.** Rejected as noisy for nested content; one warning + per top-level construct is enough to point at the problem. + +## Markdown AST shape + +No new node kinds. Relevant structure: + +```text +document + heading(level=2) "f" <-- opens a function body + paragraph -> strong(text "x = 1") <-- executes (Stmt::Assign) + paragraph (before any heading) + strong(text "x = 1") <-- inert; emits markdown.inert_executable + list(ordered) <-- inert outside function; warning + block_quote <-- inert outside function; warning +``` + +A level-2 heading without a preceding level-1 heading is a module-root function +(`## main`) and its body executes normally. A `###` section inside the function +range stays inside the function body; headings never execute. + +## Compatibility + +* Warning code `markdown.inert_executable` is part of the CLI diagnostic + contract. Renaming it requires an ADR and a CLI version note. +* Making this a hard error later would be a breaking language change: documents + that currently compile (with warnings) would stop. If that is ever desired it + needs a migration plan (e.g. an opt-in strict lint). +* No package/ABI/IR shape change: inert nodes are simply absent from the IR. +* Prose invariance is preserved: editing narrative text outside functions + cannot change the IR. + +## Security + +No effect: inert nodes never reach the interpreter, host or kernels. The +warning reduces the chance of a false belief that a side effect happened, +which is a security-relevant honesty property. A document cannot cause an +execute-by-accident through top-level prose. + +## Testing + +* `crates/marqdo-frontend/src/units.rs::tests:: + executable_outside_function_warns_but_does_not_execute` asserts the warning + and that the function index is still built. +* `units.rs` tests for function ranges, nested sections, root-level `## main` + and duplicate/empty headings. +* `crates/marqdo-frontend/src/analyze.rs` tests hard errors for invalid + executable shapes inside function bodies (the companion rule: when a node + *is* executable, its shape is strictly checked). +* Conformance/negative fixtures under `crates/cli/tests` and `tests/` + (work in progress by the fixtures workstream) are expected to cover the + warning and the inert behavior; `spec/mapping.md` §4 documents the contract. + +## Migration & rollout + +M0.5 ships with warnings. Rollout guidance: + +* document authors move executable constructs under `## function` headings; +* tooling should show the warning in editors (the diagnostic already has a + precise span); +* if a future release makes inert execution a hard error, add a deprecation + window and a `check` flag so existing documents can be migrated. diff --git a/docs/adr/0003-ref-resolution-and-ambiguity.md b/docs/adr/0003-ref-resolution-and-ambiguity.md new file mode 100644 index 0000000..f59ae07 --- /dev/null +++ b/docs/adr/0003-ref-resolution-and-ambiguity.md @@ -0,0 +1,157 @@ +# ADR-0003: `ref` resolution order and `ref.ambiguous` (S-07) + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +`ref("#anchor")` is the only way to turn a heading range into an inert +resource value (`Formula` or `Code`) or a `FunctionRef`. The blueprint +(§5.2, unresolved issue **S-07**) is explicit that a heading range can contain +several formulas or code fences, and that the implementation must **never +silently pick the first one**: + +> 一个范围存在多段公式时,默认对象选择必须是明确规则(例如禁止无选择器,要求 +> `math.formulas(ref(...))` 列出候选;不选"第一条")。 + +The failing example is easy to write: + +```markdown +## f + +### 公式 + +$$a = 1$$ + +$$b = 2$$ + +**x = ref("#公式")** +``` + +If `ref` returned `a = 1`, the program would be non-deterministic with respect +to document structure (insert an earlier formula and the meaning changes) and +would hide a genuine authoring error. + +At the same time the common case — one formula per heading — must stay simple; +requiring a selector for every reference would make `examples/math.mq.md` +awkward. + +## Decision + +`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref` implements this +algorithm: + +1. The target must be a `Text` beginning with `#`; otherwise + `symbol.unknown` ("must be a `#heading` anchor"). `ref` is not a URL + resolver and does not cross modules. +2. The fragment is looked up in the **current module's anchor map** + (`Module::anchors`, GitHub-style slugs computed in + `crates/marqdo-frontend/src/units.rs`). `ref` does not fall back to + canonical unit names and does not search imported modules. +3. Collect the resources attributed to the resolved unit (innermost unit by + span, `crates/marqdo-frontend/src/resources.rs`). + * **Exactly 1** → return `Formula` or `Code` (with `unit` and `span` + provenance). + * **0** → if a function's unit equals the resolved unit, return + `FunctionRef { module, name }`; otherwise `symbol.unknown`. + * **≥ 2** → return **`ref.ambiguous`**: + "`#...` contains N resources; select one explicitly instead of relying on + the first". Never choose a resource. +4. Function references are callable through the normal call path; formulas and + code remain inert until `math.eval`/`foreign.*` is explicitly called (those + are separately capability-checked). + +Resource attribution is deterministic: a resource belongs to its innermost +enclosing heading unit, resources keep document order, and a resource is a +Markdown `math`/`math_block` or `code_block` node (inline code is not a +resource). + +The `ref.ambiguous` category is `ErrorCategory::RefAmbiguous` with the stable +code `ref.ambiguous` (`crates/marqdo-value/src/error.rs`). + +## Alternatives + +1. **First resource wins** (blueprint's rejected baseline). Rejected by S-07: + document edits would silently change program meaning, and there is no + diagnostic for the author. +2. **Always require a selector argument** + (`ref("#f", selector="second")`). Rejected for M0.5: no selector syntax or + addressing scheme is specified yet, and the single-resource case is by far + the common one. A future ADR can add an explicit selector while keeping the + ambiguity rule. +3. **Error on every heading with ≥ 2 resources at compile time**, even if never + referenced. Rejected: a document may legitimately contain several formulas + for prose and never `ref` them; only the reference is ambiguous. +4. **Return a list of all candidates.** Rejected for v1: it changes the return + type and forces every caller to handle a selector; a dedicated API + (`math.formulas(ref(...))`) would be a new spec, not a silent widening. +5. **Cross-module `ref`.** Rejected for v1: the wire/identity model and + `ModuleId` resolution live in the loader; `ref` is a pure runtime builtin + over the current module. Cross-module resources are an open item. + +## Markdown AST shape + +```text +heading(level=2) "f" + heading(level=3) "公式" + math_block "/$$ a = 1 $$/" -> Resource::Formula { unit: <#f/公式> } + math_block "/$$ b = 2 $$/" -> Resource::Formula { unit: <#f/公式> } + paragraph -> strong("x = ref(\"#公式\")") +``` + +Anchors are separate from unit paths: `#公式` is a GitHub slug for the heading +name, while the canonical `UnitId` is `#f/公式`. Colliding slugs get +`-1`/`-2` suffixes; canonical ids never do. + +## Compatibility + +* `ref.ambiguous` is a stable runtime code; callers may branch on it. +* The single-resource and function-reference cases are frozen. Adding a + selector must be additive (new argument or new builtin) and must not change + the ≥ 2 error into a silent choice. +* `ref` target syntax (`#fragment`) is frozen; changing it to accept file paths + would need a new ADR. +* No package/ABI/wire change: formula/code values already carry + `unit`/`span`/`content_hash`. + +Example behavior, verified by +`crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`: + +```text +$ marqdo run index.mq.md # heading with two $$...$$ blocks and ref("#公式") +runtime error[ref.ambiguous]: `#公式` contains 2 resources; select one explicitly instead of relying on the first +``` + +## Security + +* `ref` is a pure builtin: no host call, no capability required, no side + effect. It only returns inert values or a function reference. +* An inert `Formula`/`Code` does not execute. Evaluation/execution requires a + separate explicit call (`math.eval`, `foreign.run/call`), which is capability + checked at the host boundary. +* Refusing ambiguity prevents a class of "wrong formula executed" mistakes, + which matters for scientific documents where formulas are evidence. + +## Testing + +* `crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error` + drives a real document through frontend + runtime and asserts the + `ref.ambiguous` code. +* `crates/marqdo-frontend/src/resources.rs` tests attribution to innermost + functions, multiple resources kept in one unit, and stable SHA-256 code + hashes. +* `crates/marqdo-markdown`/`frontend` tests cover anchors and slug collisions. +* `examples/math.mq.md` is the positive single-formula case; it runs under + `marqdo run` only via a function (the example itself defines callable + functions, no `main`). +* Missing anchor/not-a-function cases are covered by the `symbol.unknown` + paths; conformance fixtures are being added by the fixtures workstream. + +## Migration & rollout + +No migration needed for documents with exactly one resource per referenced +heading. Documents that relied on "first formula" behavior (there are none in +the examples) must split headings or wait for an explicit selector API. If a +selector is added later, keep `ref("#...")` behavior unchanged and introduce +the selector as a new argument/builtin, with tests for both the ambiguous and +the selected case. diff --git a/docs/adr/0004-strict-mode-refuses-without-isolation.md b/docs/adr/0004-strict-mode-refuses-without-isolation.md new file mode 100644 index 0000000..b8dfa26 --- /dev/null +++ b/docs/adr/0004-strict-mode-refuses-without-isolation.md @@ -0,0 +1,133 @@ +# ADR-0004: Strict mode refuses to run without OS isolation + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +Blueprint §9 defines three execution modes and one non-negotiable rule: + +> `strict`: untrusted documents/code/plugins — independent restricted process, +> OS-level resource and syscall/file/network restrictions; if unavailable, +> **refuse to run**. + +A cooperative in-process capability check cannot contain hostile native code +or an unrestricted child process. If `strict` silently degraded to `project`, +users would believe they had isolation they did not have. That is a security +bug, not a missing feature. + +M0.5 implements capability checks, budgets and process supervision but **no OS +isolation adapter** (no Linux namespaces/seccomp/cgroups, no Windows Job +Object/appcontainer, no macOS sandbox), and the isolated plugin worker +transport is not implemented either. + +## Decision + +1. `crates/marqdo-sandbox` is honest about platform support: + * `PlatformCapabilities::detect()` always reports + `StrictIsolation::Unavailable { reason }` in this build, with a + platform-specific reason (Linux: "no OS isolation adapter + (namespaces/seccomp/cgroups) is implemented in this build"; macOS/Windows/ + other targets have analogous messages); + * `worker_processes: false`. +2. `marqdo_sandbox::enforce(mode, platform)`: + * `pure` and `project` → `Ok(())`; + * `strict` → `Err(SandboxError::StrictUnavailable { reason })` unless + `platform.strict.is_available()`. +3. `run`/`call` call `enforce` **before** constructing the host, loading + plugins or executing anything. On failure they print + "refusing to run in strict mode: ..." (or a JSON `{"success":false, + "error":...}` envelope) and exit 1. +4. There is **no fallback**: the CLI never downgrades a requested strict run to + project mode. `SandboxMode::parse("strict")` succeeds (the mode name is + valid); only execution is refused. +5. `inspect --platform` exposes the detection report so users can see why: + `{"os":"linux","arch":"x86_64","strict":{"state":"unavailable","reason":...}, + "worker_processes":false}`. +6. The rest of the system is labelled accordingly: `project` mode is + cooperative, in-process, and is not a sandbox; foreign kernels are + process-isolated but not confined; native plugins run in-process and require + explicit trust (ADR-0006/security spec). + +## Alternatives + +1. **Best-effort strict** (run with the same checks as project). Rejected: it + directly violates the blueprint's "cannot provide → refuse" rule and hides a + missing security boundary. +2. **Warn and continue.** Rejected for the same reason; a warning is not a + sandbox, and CI/agents may not read warnings. +3. **Implement a Linux namespaces/seccomp adapter now.** Out of scope for + M0.5: it needs platform-specific code, integration tests, resource limits, + CI on multiple kernels, and a way to supervise the worker protocol. This is + P4 work and belongs in its own ADR when implemented. +4. **Use Wasm/WASI for strict.** Attractive for plugins later, but it does not + sandbox arbitrary Python/Node processes or native `.so` files, and the + runtime is not implemented. Rejected for now. +5. **Claim strict via the process isolation of foreign kernels.** Rejected: + a child process inherits the user's privileges and filesystem/network + access; changing cwd and clearing env is hygiene, not confinement. + +## Markdown AST shape + +No language or AST change. A strict run is rejected before any document is +executed (after static loading/diagnostics); the CLI contract is: + +```text +$ marqdo run index.mq.md --mode strict +error: refusing to run in strict mode: no OS isolation adapter + (namespaces/seccomp/cgroups) is implemented in this build. Strict mode + requires verified OS-level isolation and never falls back to an + unrestricted process. +$ echo $? +1 +``` + +(Exact wording comes from +`crates/marqdo-sandbox/src/lib.rs::SandboxError::display`.) + +## Compatibility + +* Accepting `--mode strict` as a valid *name* is part of the CLI contract; the + refusal is a documented behavior, not a parse error. A future release that + implements isolation flips the detection report and starts executing strict + runs — that is the intended upgrade path. +* Exit code 1 (runtime/security refusal) is stable; usage errors are 2. +* `PlatformCapabilities` JSON shape (`state: available|unavailable`) is + versioned by documentation and should not change without an ADR. +* No package/ABI/wire impact. + +## Security + +* The guarantee "requested strict → refused rather than downgraded" is + enforced in code (`enforce`), not documentation. +* Users cannot accidentally believe untrusted docs are sandboxed by this + build; the platform report is the evidence. +* Project mode remains explicitly non-sandboxed. Native plugins (in-process) + and foreign child processes are recorded as trust decisions, not isolation. +* Secrets: strict refusal happens before plugin loading or foreign spawning, + so no untrusted code is touched by a refused command. + +## Testing + +* `crates/marqdo-sandbox/src/lib.rs::tests:: + strict_refuses_when_isolation_is_unavailable` asserts the refusal and that + `pure`/`project` still succeed. +* `mode_names_round_trip` covers parsing/printing. +* CLI behavior (`--mode strict` exit 1, message) is covered by manual/CLI + smoke checks; a CLI integration test should be added with the conformance + harness. +* `inspect --platform` JSON is exercised manually and in the CLI help/example + documentation; it should get a golden test in the fixture workstream. + +## Migration & rollout + +When an OS isolation adapter lands: + +1. implement a `StrictIsolation::Available { mechanism }` report and a real + supervisor/worker; +2. add integration tests for filesystem/network/process restrictions and + resource limits per platform, plus the platform capability matrix; +3. update `spec/security.md`, this ADR (or supersede it), `docs/STATUS.md` and + the CLI docs; +4. only then change `PlatformCapabilities::detect`; do not add a flag that + bypasses the refusal before that work is done. diff --git a/docs/adr/0005-ast-normalization.md b/docs/adr/0005-ast-normalization.md new file mode 100644 index 0000000..3b025c6 --- /dev/null +++ b/docs/adr/0005-ast-normalization.md @@ -0,0 +1,174 @@ +# ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough) + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +The language derives program meaning from the **canonical AST**, never from +source spelling (`spec/markdown-profile.md`, `spec/mapping.md` §1). The parser +(`comrak =1.0.0`, frozen options) normalizes several syntaxes before Marqdo +sees them, and a few normalizations are lossy: + +* `# H` and Setext `H\n===` both become a heading node; the source spelling is + lost unless recorded; +* GFM task list items (`- [x] text`) produce a task-specific item node; +* inline code spans are stored as a literal without the backtick delimiters, + but source spans include them; +* `~~strikethrough~~` is enabled by the profile and wraps arbitrary inlines. + +The language needs a single, documented answer per question, because node type +and structural position decide executable roles. + +## Decision + +The profile (`marqdo.md.v1`) and the canonical AST +(`crates/marqdo-markdown/src/ast.rs`, `src/profile.rs`) freeze these rules: + +1. **Setext headings are headings.** `# name` and a Setext level-1/2 underline + both produce `heading { level, setext }`; `setext` is inspection-only and + **never affects semantics**. `## name` and a Setext level-2 heading are both + functions. The profile keeps `parse.ignore_setext = false`. +2. **Task-list items are `item` nodes with a task flag.** + `item { task: Some(true|false) }` for GFM task items, `task: None` + otherwise. Node type stays `item`, so list structure (ordered vs bullet) + still determines branch/loop roles; the checkbox flag is inert data and does + not change execution. A task list item in an executable position still + requires its condition header like any other item. +3. **Code spans carry delimiter tick counts.** Inline `code { ticks }` records + the number of backticks, and the source span includes them; the literal is + the inner payload. This lets the frontend map expression payload spans back + to exact source bytes for diagnostics. "Ticks" is not user-visible program + meaning. +4. **Strikethrough is inert/lazy.** `strikethrough` is an inert wrapper: it is + never traversed for execution, its content is prose, and a `strong`/`emph` + inside it does not execute. It is not an error to write it; the language + simply does not use formatting as a control-flow mechanism here. (A code + span or math inside it is likewise inert.) +5. **Unmodelled nodes are preserved** as `other { tag }` with children and raw + literal. They are inert but never silently dropped from `inspect --ast`. +6. **Literal payloads are kept** on inert leaves: `text`, `code`, `code_block`, + `math`, `math_block`, `html_block`, `html_inline`, `front_matter`. +7. **Other frozen normalizations:** + * `**x**`/`__x__` and `*x*`/`_x_` are the same node kinds (`strong`/`emph`); + * ordered delimiters `1.`/`1)` and bullets `-`/`*`/`+` are the same + `list`/`item` structure; role depends on `ordered`, not marker; + * tight vs loose lists only affect rendering (`tight` is recorded, inert); + * `link`/`image` keep `url`/`title`; `table` keeps column alignments, + `table_row` keeps `header`; `heading` keeps `level`/`setext`; + * a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the + original file; + * front matter is recognized only at the document start (after BOM), with + `---` opening and `---`/`...` closing; a later `---` is a thematic break + or Setext underline. + +These rules are part of profile id `marqdo.md.v1`. Changing them (or bumping +the comrak version/options) requires a profile id bump, updated goldens and this +ADR being superseded. + +## Alternatives + +1. **Distinguish Setext vs ATX headings semantically.** Rejected: identical AST + structure outside the marker must have identical meaning (`## H` and Setext + H2 are the same function). Keeping `setext` only for inspection avoids + silently diverging programs when a document changes style. +2. **Introduce a separate `task_item` node kind.** Rejected: it would require + every list-role check to special-case two item kinds and could be used to + hide a control structure behind a checkbox. Normalizing to `item { task }` + keeps list structure authoritative. +3. **Strip the tick information and store only the literal.** Rejected: payload + spans returned by the expression parser are relative to the concatenated + payload; without delimiter lengths, error spans cannot be mapped back to + file bytes precisely. +4. **Traverse strikethrough content for execution.** Rejected: it is a lazy + region by profile; executing inside it would make formatting semantics + surprising ("struck-through code runs"). It would also create a second way + to hide executable nodes. +5. **Drop unknown parser nodes.** Rejected: `inspect --ast` must show what the + parser produced; dropping syntax hides profile drift. +6. **Preserve raw source and re-parse with a custom parser.** Rejected by the + project's core rule: one standard Markdown parser, no second Markdown AST. + +## Markdown AST shape + +Frozen JSON shapes (`inspect --ast --json`): + +```json +{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7}, + "children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]} +``` + +```json +{"kind":"list","ordered":false,"tight":true,"span":{...}, + "children":[{"kind":"item","task":true,"span":{...},"children":[...]}]} +``` + +```json +{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"} +``` + +```json +{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]} +``` + +Attribute order in the JSON is fixed by the serializer; goldens are regenerated +only through the explicit update workflow. + +### 5.1 Known interaction to keep in mind + +Inside a function body, a `**strong**` (or an ordered/bullet list) nested under +`strikethrough` is inert and never executes; because the mapper does not +traverse strikethrough, it also produces no warning. Outside function bodies, +the `markdown.inert_executable` scan walks a top-level paragraph recursively, +so such a nested node may still produce the inert warning even though it would +not execute if the paragraph were inside a function. This is consistent with +"formatting never hides execution", but an author who intends execution must +not wrap it in strikethrough. Warnings for lazy-region content inside functions +are a possible future refinement; no behavior change is planned without an ADR. + +## Compatibility + +* Profile id `marqdo.md.v1` is frozen with comrak `=1.0.0` and the exact + extension flags; the unit test + `crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen` + asserts the option table (tables, tasklist, strikethrough, math dollars, + front matter; all other extensions disabled). +* Adding an AST field is a JSON output change for `inspect --ast`; it needs a + goldens update and a profile note. Changing a semantic rule (e.g. executing + inside strikethrough) is a language change and requires an ADR. +* No package/ABI/wire impact. + +## Security + +* The lazy-region rules ensure inline formatting inside headings, tables, + links, code, math, HTML and front matter never executes — a context-aware + visitor is used (`crates/marqdo-frontend/src/map.rs` only descends from + paragraphs into `strong`/`emph`, and ignores every other inline node). +* Strikethrough cannot be used to smuggle executable content; it is inert. +* Normalizing task items to `item` prevents a checkbox from hiding a + branch/loop header; the condition-header check still applies. +* No new capabilities or host calls. + +## Testing + +* `crates/marqdo-markdown/src/profile.rs`: + * `profile_options_are_frozen` — exact extension/parse option table; + * `setext_headings_are_heading_nodes` — Setext produces `heading { setext: true }`; + * `tables_and_task_lists_are_supported` — task items carry `task: Some(_)`; + * span/BOM tests. +* Frontend mapping tests use code spans as conditions and payloads, exercising + `ticks`/payload span mapping. +* `spec/markdown-profile.md` §5/§6 is the normative JSON shape; AST goldens + under `crates/cli/tests` (fixtures workstream) regenerate only with the + documented update command. +* Conformance fixtures are expected to include `**a**` vs `__a__`, Setext vs + ATX, task/list variants and strikethrough inertness. + +## Migration & rollout + +No migration for documents: Setext and ATX were always the same function under +this profile. When comrak is upgraded or the extension set changes, bump the +profile id, regenerate AST goldens, re-run all specs/ADRs that reference node +shapes, and document the change here (superseding this ADR if a rule changes). +Code-span `ticks` and `task` are additive inspection fields; keep them optional +in consumers that parse the JSON. diff --git a/docs/adr/0006-cli-contract.md b/docs/adr/0006-cli-contract.md new file mode 100644 index 0000000..551a8c5 --- /dev/null +++ b/docs/adr/0006-cli-contract.md @@ -0,0 +1,225 @@ +# ADR-0006: CLI output, exit-code and JSON contract + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +CI, scripts and coding agents need a stable CLI contract, not just human +output. The blueprint (§12) requires: all commands available with `--json`, +non-zero error exit codes, program stdout kept separate from diagnostics, and +machine-readable error codes. The implemented CLI must be documented precisely +because scripts will depend on stream separation and exit status. + +Implementation: `crates/cli/src/main.rs` (clap definitions, process exit), +`crates/cli/src/commands.rs` (behavior), `crates/cli/src/output.rs` +(diagnostic/JSON rendering), `crates/cli/src/args.rs` (`--arg` constants). + +## Decision + +### 1. Commands and flags (implemented surface) + +| Command | Arguments / flags | What it does | +|---|---|---| +| `init [dir]` | — | create `marqdo.yaml`, `index.mq.md`, `.gitignore`; refuses an existing manifest | +| `run [file]` | `--mode`, `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin`, `--fuel`, `--stack`, `--max-output`, `--timeout`, `--plugin`, `--trust-native-plugin`, `--json`, `--trace` | execute `main` (or default `index.mq.md`) | +| `call ` | `--arg NAME=VALUE` (repeatable) plus all `run` flags | execute one named function | +| `check [file]` | `--json` | static check; never executes | +| `inspect [file]` | `--ast`, `--units`, `--signature`, `--ir`, `--refs`, `--platform`, `--json` | static inspection; never executes | +| `fmt [file]` | `--check` | normalize CRLF and trailing newlines | +| `plugin inspect ` | `--json` | validate manifest + verify artifact hash; never loads code | +| `add ` | `--path`, `--git`, `--rev`, `--json` | edit `marqdo.yaml` | +| `remove ` | `--json` | edit `marqdo.yaml` (and prune a stale lock entry) | +| `lock` | `--json` | write deterministic `marqdo.lock` | +| `sync` | `--locked`, `--json` | verify the lock is current | + +There is no `test`, `tree`, `update`, `vendor`, `add` for plugins, `kernel` +command or `--project`/`--offline` flag in this build. `run` defaults the +file to `index.mq.md`; `check`/`inspect`/`fmt` do too. `call` requires an +explicit file path plus a function name. + +`--mode` is only on `run`/`call` and accepts `pure|project|strict` +(default `project`). `--trust-native-plugin`/`--plugin` are only meaningful on +`run`/`call`. `inspect --platform` still requires a document argument (or +defaults to `index.mq.md`), because project loading happens first. + +### 2. Exit codes + +| Code | Meaning | +|---|---| +| `0` | success (for `check`: no **error** diagnostics; warnings do not fail) | +| `1` | a checked document has errors; a runtime/security failure; strict refusal; plugin load/inspect failure; lock/sync mismatch; `fmt --check` would reformat; `init` found an existing manifest; `add` duplicate; `remove` missing dependency; `sync` stale lock | +| `2` | usage error (clap), unreadable entry file, invalid `--mode` name, invalid capability combination (e.g. `--allow-read` in `pure`), invalid `--arg` value, unreadable manifest for project commands, unreadable file for `fmt` | + +Codes are stable contract. Diagnostic **codes** (for example +`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`, +`permission.denied`, `ref.ambiguous`) are stable; message wording is not. + +### 3. Stream separation + +* **Program output** (`打印`/`print`) goes to stdout in text mode. In JSON mode + it is embedded as the `stdout` string; it is never interleaved with JSON. +* **Diagnostics and effect traces** go to stderr in text mode. +* `run` in text mode writes only program output to stdout (no implicit return + value). `call` in text mode additionally prints the returned value (using + `Value::display`) when it is not `Unit`. +* `check` text mode: diagnostics to stderr; `checked N module(s): ok` to stdout + on success; a `check failed: ...` summary to stderr on failure. +* `inspect` prints pretty JSON-ish structures to stdout; load errors to stderr. +* JSON mode prints exactly one pretty JSON object to stdout per invocation. + `add`/`remove`/`lock`/`sync` still print errors as plain `error: ...` on + stderr even with `--json` (JSON success envelope only) — an inconsistency to + fix before the contract is declared final. + +### 4. JSON output shapes + +`check`: + +```json +{"command":"check","success":true,"errors":0, + "diagnostics":[{"file":"index.mq.md","module":"index.mq.md", + "line":3,"column":5,"severity":"error","code":"expr.parse", + "message":"...","hints":["..."]}]} +``` + +`inspect`: + +```json +{"command":"inspect","module":"index.mq.md", + "ast":{...},"units":[...],"signatures":[...],"ir":[...], + "refs":[...],"platform":{...}} +``` + +Only the requested sections are present. With no section flags, a `summary` +with `modules`, `functions`, `units`, `resources` is emitted instead. + +`run`/`call` (success): + +```json +{"command":"run","success":true, + "value":{"type":"text","value":"..."}, + "stdout":"program stdout text", + "trace":[{"op":"print","module":"index.mq.md", + "span":{"start":43,"end":57},"duration_micros":3,"result":"ok"}]} +``` + +Note: `call --json` also emits `"command":"run"` in this build (shared +implementation). Consumers should accept both names or key on the presence of +`diagnostics`/`value`; this is a known quirk. + +`run`/`call` (failure): `{"command":"run","success":false,"stdout":..., +"error":{"code":"...","message":"...","span":{"line":..,"column":..,"file":..}, +"frames":[...]},"trace":[...]}`. Runtime error spans are currently resolved +against the **entry file** only, even when the error came from an imported +module; frames carry function names, not module ids. This is a known precision +limitation. + +`plugin inspect`: + +```json +{"command":"plugin.inspect","success":true,"plugin":"demo.solver", + "version":"0.1.0","abi":"^1.0","abi_compatibility":"compatible", + "target":"x86_64-unknown-linux-gnu", + "artifact":{"file":"solver.so","path":"...","sha256":"...","verified":true}, + "exports":[{"name":"solver.add","signature":"($a: Int, $b: Int) -> Int"}], + "loaded":false} +``` + +`load_project` failures with `--json` print `{"success":false,"error":"..."}` +without a `command` field. + +There is no `json_schema_version` field yet. The blueprint asks for a +versioned JSON schema; until it exists, treat these keys as the stable subset +and ignore unknown keys. Adding keys is not considered breaking; renaming or +removing one is. + +### 5. `--arg` values + +`--arg NAME=VALUE` parses `VALUE` with the **expression grammar restricted to +constants** (`crates/cli/src/args.rs`): integers, floats, double-quoted text +(escapes supported), `true`/`false`/`None`, lists, records, and unary `-`/`+` +over constants. Calls and identifiers are rejected ("must be a literal +constant"). Text must be double-quoted; there is no bare-word fallback. + +### 6. Determinism and safety of read-only commands + +* `check`/`inspect` never execute project code (ADR context; see + `spec/security.md` §7). +* `plugin inspect` never loads plugin code. +* `fmt` only rewrites the file when not `--check`; exit 1 when `--check` + detects a difference. + +## Alternatives + +1. **Use exit code 1 for all failures.** Rejected: scripts need to distinguish + usage errors (2) from document/runtime failures (1); it also mirrors common + CLI conventions and clap's own usage exit code. +2. **Print JSON even for usage errors.** Rejected for now: clap owns usage + output before our code runs; wrapping it would mean intercepting clap + errors. Documented as a limitation. +3. **Interleave program output and JSON.** Rejected: it makes JSON + unparseable. Program stdout is embedded as a string instead. +4. **Return values from `run` on stdout in text mode.** Rejected: `run` is the + batch entry; implicit value printing would pollute program output. `call` + prints the return value exactly when non-`Unit`. +5. **Human-only messages with numeric exit codes only.** Rejected: diagnostics + and runtime errors have stable machine codes in both text and JSON. + +## Markdown AST shape + +The CLI itself does not change the AST. What it exposes: + +* `inspect --ast` = canonical `MdNode` tree (see ADR-0005); +* `inspect --units` = `UnitInfo[]` (`marqdo-ir`); +* `inspect --signature` = `{name, local_name, signature: explicit|inferred, + params:[{name,type,has_default}]}`; +* `inspect --ir` = `Function[]` with `Stmt`/`Expr` and spans; +* `inspect --refs` = link edge array (`spec/references.md` §4); +* `inspect --platform` = `PlatformCapabilities` JSON. + +## Compatibility + +* Exit codes, stream separation and the documented JSON keys are the public + CLI contract. Changing them requires this ADR to be superseded and a + changelog entry. +* The old "design-only starter" README listed planned commands (`add`, + `lock && sync`, `plugin inspect`); all listed commands except a versioned + JSON schema are now implemented as described here. +* Diagnostic code renames are breaking for CI; message changes are not. +* No package/ABI/wire changes. + +## Security + +* `--json`/`--trace` enable effect tracing; traces contain op/module/span/ + duration/result summary, never argument values (no secrets) — see + `spec/security.md` §4. +* JSON mode embeds program stdout, which is program-controlled data; consumers + must not treat it as trusted. +* Refusing to run with compile errors happens before host creation, so JSON + failure output cannot leak effects. +* Strict-mode refusal and capability denials produce stable codes for CI. + +## Testing + +* CLI behavior is currently covered by manual smoke checks and library-level + tests; the fixtures workstream is adding CLI integration tests under + `crates/cli/tests` (goldens/negative cases) and `tests/`. +* `crates/cli/src/args.rs` unit tests cover typed `--arg` parsing and rejection + of non-constant values. +* `crates/cli/src/output.rs` has no unit tests; diagnostic JSON is exercised + through check/run smoke tests. +* Add contract tests for: exit codes per failure class, stdout/stderr + separation, JSON key presence, `call --json` command naming, and runtime + error span from a non-entry module before calling the schema "final". + +## Migration & rollout + +1. Keep the current shapes stable while adding tests. +2. Add a `schema_version` to JSON envelopes and document the versioned schema; + then document that unknown keys may be added (forward compatible) while + existing keys are frozen. +3. Fix the `call --json` `"command":"run"` quirk and JSON errors in + `add/remove/lock/sync`; both are additive/corrective and should be + announced in the changelog. +4. If a command is renamed or an exit code changes, update this ADR, the + README quickstart, `docs/STATUS.md` and CI scripts in the same change. diff --git a/docs/adr/0007-arbitrary-precision-int-and-wire.md b/docs/adr/0007-arbitrary-precision-int-and-wire.md new file mode 100644 index 0000000..2e110dc --- /dev/null +++ b/docs/adr/0007-arbitrary-precision-int-and-wire.md @@ -0,0 +1,178 @@ +# ADR-0007: Arbitrary-precision `Int` and the typed wire encoding (S-08) + +Status: **Accepted** +Date: 2026-11-20 + +## Context + +The blueprint (§3.1 and unresolved issue **S-08**) requires a numeric model +that does not silently lose precision at language, JSON/ABI and math +boundaries. It recommends arbitrary-precision integers for `Int` and type tags +plus string forms for large integers on the wire. + +Three independent places could lose information: + +1. **Language arithmetic.** A fixed-width `i64` would make `Int` silently + overflow or wrap. Factorials, powers and scientific constants routinely + exceed 64 bits. The expression grammar already has arbitrary-length integer + literals (`spec/expression.ebnf`). +2. **JSON / wire.** JSON has one number type (IEEE-754 double in most parsers). + Encoding `12345678901234567890` as a JSON number loses digits in + JavaScript/Go consumers; encoding `1` as a number makes `Int` and `Float` + indistinguishable. +3. **Math kernel.** The exact-rational engine needs an integer type that is + the *same* type as `Value::Int`, or every formula/binding crossing the host + boundary needs a lossy conversion. + +The chosen dependency set (`Cargo.toml`) is: + +```toml +num-bigint = { version = "0.4", features = ["serde"] } +num-rational = { version = "0.4", features = ["serde"] } +num-traits = "0.2" +``` + +## Decision + +### 1. `Int` is arbitrary precision (`num-bigint::BigInt`) + +* `Value::Int(BigInt)` (`crates/marqdo-value/src/value.rs`) is the only integer + representation in the runtime and the expression AST + (`crates/marqdo-expression/src/ast.rs`). +* Expression integer literals are arbitrary-precision decimal; there is no + fixed-width parsing step. +* Arithmetic (`crates/marqdo-runtime/src/value_ops.rs`): + * `+ - *` on two `Int` are exact `BigInt` operations; + * `/` **always** produces `Float` (exact ratio converted to binary64), so + there is no integer division surprise; + * `//` and `%` require `Int` operands; floor division is mathematical floor + and `a % b == a - b * (a // b)`, with the sign following the divisor; + * mixed `Int`/`Float` arithmetic produces `Float`; converting an `Int` + larger than 2^53 may lose precision (documented, not silently exported as + an `Int`); + * `Int`/`Float` comparisons are **exact**: the float is treated as the exact + binary rational it is (`BigRational::from_float`), so + `9007199254740993 > 9007199254740992.0` is `true`; + * `**` on `Int` is exact exponentiation, with a negative exponent rejected + as a type error and guards `MAX_EXACT_EXPONENT = 100_000` and + `MAX_POWER_BITS = 1_000_000` (`resource.exhausted`). +* There is no truthiness and no implicit string↔number conversion; `整数`/`int` + performs explicit conversions and rejects non-integral floats. + +### 2. `num-bigint` 0.4 is pinned to match `num-rational` 0.4 + +`num-rational`'s generic `Ratio` is instantiated as +`Ratio` in `crates/marqdo-math/src/ast.rs` +(`pub type Rational = Ratio`). The two crates must resolve to the same +`num-bigint` version, otherwise `BigInt` values from the value model cannot be +passed to `Ratio` without conversion (or a duplicate type). The workspace pins +both at the `0.4` line and the lockfile resolves one `num-bigint`. Upgrading +either crate requires checking that this shared-type invariant still holds and +running the math + runtime test suites. + +### 3. The wire encoding never sends an integer as a JSON number + +Canonical encoding (`crates/marqdo-value/src/wire.rs`; full contract in +`spec/value-wire.md`): + +```json +{"type":"int","decimal":"12345678901234567890"} +``` + +* Every value is a JSON object with a `"type"` tag. +* `Int` uses a base-10 string. All digits, sign and magnitude survive any JSON + consumer including JavaScript. +* `Float` uses a JSON number only when finite; `NaN`, `Infinity`, + `-Infinity` use explicit `"special"` tags. +* JSON numbers appear only in non-value positions (`span.start/end`, + `plugin_handle.id`). +* Decoding is validated: unknown tags, missing fields, invalid decimals, wrong + types and limits are `WireError`s, never coercions. +* The same encoding is used for plugin ABI request/response values, foreign + kernel stdin `input`, foreign typed return values and `run --json` values. + +### 4. Math results preserve exactness where possible + +The math kernel keeps exact rationals and returns `Value::Int` exactly when the +exact result is an integer, otherwise `Value::Float`. Decimal literals are +exact rationals (`0.25` = `1/4`), and `\frac{1}{3} * 3` is exactly `1`. Constants +(`\pi`, `\e`) and non-perfect square roots introduce binary64 approximations and +the result is a `Float` (`spec/math.md` §3). + +## Alternatives + +1. **`i64`/`u64` integers with overflow errors.** Rejected: common programs + (factorials, powers) would need explicit bignum workarounds; the language + would inherit fixed-width semantics not implied by its arithmetic. +2. **`i128` integers.** Same problem, just a bigger ceiling; and the wire + precision problem remains for values beyond 2^53. +3. **Encode integers as JSON numbers and hope consumers use bignum JSON.** + Rejected: the JSON standard/ecosystem does not guarantee it; JavaScript's + `JSON.parse` loses digits; this is the exact S-08 failure. +4. **Encode every number as a string without type tags.** Rejected: loses the + `Int`/`Float` distinction and makes all consumers parse strings; type tags + are explicit and cheap. +5. **Use `f64` for all numbers.** Rejected: contradicts exact `Int` + semantics, the math kernel's exact rational requirement, and the + no-silent-precision-loss rule. +6. **Different integer crates in value vs math.** Rejected: forces lossy or + string conversions at every kernel boundary and duplicates tests. + +## Markdown AST shape + +No AST change. Numeric literals live inside executable payloads only (code +spans/plain text of `strong`/`emph`/conditions). A `math` node is inert LaTeX; +its numeric interpretation is the math kernel's job. The JSON inspection view +of IR/AST contains spans as JSON numbers, not user numeric values. + +## Compatibility + +* `Int` semantics (arbitrary precision, `/` → `Float`, exact comparisons, + `//`/`%` sign rules) are frozen language behavior; changing them requires an + ADR and conformance fixtures. +* `wire_version = 1` with typed tags and decimal-string integers is frozen. + Changing the integer representation is a wire-breaking change and needs a + new `wire_version` plus plugin ABI considerations. +* The dependency pin may move within `0.4` for security fixes, provided + `num-bigint` remains a single resolved version shared with `num-rational`. +* Package/lock formats are unaffected. + +## Security + +* Untrusted wire data cannot trigger huge allocations through the integer + parser without also counting against value depth/string limits: a decimal + string is bounded by `max_string_bytes` (16 MiB default) and `BigInt` parsing + is linear. The current decoder does not separately cap the *digit count* of + an `Int`; the envelope/string caps are the effective bound (open item). +* Arithmetic can grow memory; the runtime bounds it with fuel and the + `Int` power guards (`MAX_EXACT_EXPONENT`, `MAX_POWER_BITS`) and the math + kernel bounds exact numerator/denominator bits (`MAX_RESULT_BITS`). +* Exact `Int`/`Float` comparison prevents a class of "looks equal but is not" + authorization or verification bugs. +* No new capabilities; numeric parsing is pure. + +## Testing + +* `crates/marqdo-value/src/value.rs`: `big_integers_are_arbitrary_precision`, + `float_to_bigint_requires_integral_value`, display/format tests. +* `crates/marqdo-value/src/wire.rs`: `big_integers_keep_all_digits`, + `special_floats_are_tagged`, scalar round trips, untrusted-input limits. +* `crates/marqdo-runtime/src/value_ops.rs`: exact integer arithmetic, + `/`→Float, floor/mod sign rules, exact mixed comparisons, power guards, + division-by-zero. +* `crates/marqdo-math/tests/kernel.rs`: exact big integer arithmetic, exact + rational division, decimal literals as rationals, `pi`/`sqrt` approximating + to `Float`, `Int` vs `Float` result typing. +* `crates/marqdo-plugin-abi/src/wire.rs`: request round trips preserving tags + and refusing coercible-looking values beyond `i64`. +* `crates/marqdo-foreign/tests/kernels.rs`: typed foreign return values decode + exactly. + +## Migration & rollout + +No migration: the implementation has used `BigInt` and the typed wire format +since M0.5. For future dependency upgrades: run `cargo test --workspace`, +verify one resolved `num-bigint` (`cargo tree -i num-bigint`), and re-run wire +compatibility tests. For any change to the wire integer encoding, bump +`wire_version`, update `spec/value-wire.md`, the plugin ABI documentation and +the foreign protocol in one coordinated change. diff --git a/docs/contributor-guide.md b/docs/contributor-guide.md new file mode 100644 index 0000000..9b33bb0 --- /dev/null +++ b/docs/contributor-guide.md @@ -0,0 +1,286 @@ +# Marqdo Contributor Guide + +This guide is for people changing the Marqdo CLI workspace. It assumes you have +read `AGENTS.md` (the non-negotiable architectural rules) and +`docs/DEVELOPMENT_BLUEPRINT.md` §0/§1/§11/§13/§14. Formal contracts live in +`spec/`; decisions live in `docs/adr/`. + +## 1. Repository map + +```text +marqdo-cli/ +├── Cargo.toml workspace (15 crates, resolver 2) +├── AGENTS.md operator rules for contributors/agents +├── README.md user-facing quickstart and limits +├── docs/ +│ ├── DEVELOPMENT_BLUEPRINT.md architecture baseline +│ ├── STATUS.md implementation matrix (P0–P7) +│ ├── contributor-guide.md this file +│ └── adr/0001..0007-*.md accepted decisions +├── spec/ +│ ├── markdown-profile.md frozen parser profile `marqdo.md.v1` +│ ├── mapping.md AST → program roles +│ ├── expression.ebnf executable payload grammar +│ ├── evaluation.md values, numbers, control flow, builtins +│ ├── references.md imports/references/calls/ref() +│ ├── value-wire.md typed wire_version 1 values +│ ├── security.md modes, capabilities, budgets, trust +│ ├── package-format.md marqdo.yaml / marqdo.lock / init +│ ├── plugin-abi.md C ABI v1, SDK, host lifecycle +│ ├── math.md Formula subset and math.* semantics +│ └── foreign.md Python/Node kernel protocol +├── crates/ +│ ├── marqdo-span/ Span, LineCol, LineIndex (leaf) +│ ├── marqdo-value/ Value, diagnostics, errors, wire encoding +│ ├── marqdo-expression/ lexer/parser/AST for executable payloads +│ ├── marqdo-markdown/ comrak profile, canonical AST, front matter +│ ├── marqdo-ir/ Module/Unit/Function/Stmt IR, builtin table +│ ├── marqdo-frontend/ units, map, signature, infer, calls, resources +│ ├── marqdo-math/ Formula parser/evaluator (exact rationals) +│ ├── marqdo-foreign/ Python/Node one-shot kernel supervisor +│ ├── marqdo-sandbox/ modes + platform capability detection +│ ├── marqdo-plugin-abi/ C ABI mirrors, manifest, artifact, loader, wire +│ ├── marqdo-plugin-sdk/ Rust plugin SDK and `mq_plugin_entry!` +│ ├── marqdo-host/ CapabilityHost, budgets, FS, math/foreign/plugin adapters +│ ├── marqdo-runtime/ tree-walk interpreter, value ops, builtins +│ ├── marqdo-package/ marqdo.yaml / marqdo.lock types + discovery +│ └── cli/ composition root: flags, commands, loader, output +├── include/marqdo_plugin_abi_v1.h frozen C ABI v1 header +├── examples/ hello, factorial, math, references, python-foreign, +│ multi-module/, plugin-demo/ +├── tests/ +│ ├── conformance/ast/ *.mq.md + *.ast.json AST goldens +│ ├── conformance/semantics/ *.mq.md + *.expected.json execution goldens +│ ├── negative/ compile-time + runtime hard-error fixtures +│ ├── README.md corpus layout and golden workflow +│ ├── integration/ properties/ security/ abi/ reserved dirs (see STATUS) +└── crates/cli/tests/ Rust harnesses for the corpus and CLI +``` + +## 2. Dependency direction (must stay acyclic) + +```text +span -> serde +value -> span +expression -> span +markdown -> span, comrak (no semantic crate!) +ir -> span, expression +frontend -> markdown, expression, ir, value +host -> value, math, foreign, sandbox +runtime -> ir, value, host, expression +math -> value +foreign -> value +plugin-abi -> value +plugin-sdk -> plugin-abi +package -> serde only +sandbox -> serde only +cli -> everything (composition root) +``` + +Rules (from blueprint §11.1, adjusted by ADR-0001): + +* `markdown` owns the only Markdown parser; it must not depend on semantic + crates, the host or the runtime. The single exception to "zero Marqdo deps" + is `marqdo-span`, which is a dep-free leaf shared by all layers. +* `frontend`/`ir`/`runtime` never depend on `cli`. +* `runtime` never parses Markdown and never performs I/O directly: all effects + go through `marqdo_host::Host`. +* `host` never depends on the CLI; kernels/plugins are adapted here. +* The CLI is the composition root: flag parsing, project loading, rendering and + process exit. It must not become a second interpreter. +* Check the graph with `cargo tree` (or `cargo tree -i `); `runtime -> + cli`, `markdown -> frontend`, and dependency cycles are review blockers. + +## 3. Adding a feature or fixture (workflow) + +The project uses a fixture-first workflow (blueprint §13/§14, AGENTS § +"Development workflow"). The expected sequence is: + +1. **Find the rule.** Identify the affected spec section and blueprint section. + If the change alters observable language semantics, package/lock format, + ABI, authorization defaults or public CLI behavior, write/supersede an ADR + first (`docs/adr/`, template in blueprint §16.2). +2. **Write a failing fixture.** Add a small `.mq.md` positive and/or negative + case under the right directory: + * AST shape: `tests/conformance/ast/NNN-name.mq.md` + golden + `NNN-name.ast.json`; + * semantics: `tests/conformance/semantics/NNN-name.mq.md` + + `NNN-name.expected.json` (typed wire `args`/`value` + captured `stdout`); + * hard error: `tests/negative/NNN-name.mq.md` + `{"codes":[...]}`; + * inert-executable warning: `tests/negative/warnings/`; + * runtime error: `tests/negative/runtime/` with `{"function","args","codes"}` + (optional `grants`, `stack_limit`). + Run the relevant harness (see §6) and confirm it fails for the expected + reason. +3. **Update the spec.** `spec/*.md` is normative. If implementation and spec + disagree, fix whichever is wrong and note the tradeoff. Do not silently + invent behavior in code and leave the spec stale. +4. **Implement the smallest module.** Keep the layers separated: parser + (`marqdo-markdown`), semantic mapping (`marqdo-frontend`), IR + (`marqdo-ir`), interpreter (`marqdo-runtime`), effects (`marqdo-host`), + composition (`cli`). Never add a second Markdown parser, a regex-based + syntax decision, or a host bypass. +5. **Refresh goldens deliberately** (§4) and run the full gate (§6). +6. **Write the tradeoff/limitations** in `docs/STATUS.md` when a P-level claim + changes, and in the PR description (paths, commands run, skipped tests). + +If a fixture reveals an implementation bug, fix the code, not the expected +behavior. Negative fixtures must assert hard errors — never weaken them to +warnings or prose fallbacks. + +## 4. Golden update workflow + +AST goldens (`tests/conformance/ast/*.ast.json`) are the serialized canonical +AST from `marqdo_markdown::parse(source).root`. Regenerate **only** for an +intentional, ADR-backed profile change: + +```bash +UPDATE_GOLDENS=1 cargo test -p marqdo-cli --test conformance_ast +``` + +The update run rewrites every golden, prints the count and passes. Re-run +without the variable and review the diff: **a golden change is a language +change, not a bug fix**. A fixture without a golden and a golden without a +fixture are both errors. + +Semantic goldens (`*.expected.json`): `UPDATE_SEMANTICS_GOLDENS=1` exists for +authoring convenience, but never use it blindly — verify every value against the +fixture source first. Invalid, inconsistent or unverified expected values defeat +the corpus. + +Negative goldens are hand-written expected code lists (exact sorted multiset for +compile-time errors; warning suites; runtime suites). Do not auto-generate them. + +Adding/removing a fixture and its golden must happen in the same change. +Fixtures are data: tests must not commit, copy or rewrite files outside a +throw-away temp directory. `tests/README.md` is the in-repo reference for corpus +counts, harness responsibilities and known observations. + +## 5. Coding conventions + +* Rust edition 2021, `rust-version = 1.82`, workspace version `0.1.0`. +* `unsafe` is forbidden in pure crates (`marqdo-math`, `marqdo-expression` use + `#![forbid(unsafe_code)]`); the ABI/loader crates contain the only + `unsafe` code and every block has a SAFETY comment explaining the contract. +* No panics on untrusted input: Markdown, expressions, LaTeX, wire data, + manifests and foreign output all produce structured errors. `unwrap()`/ + `expect()` are acceptable only for genuinely impossible invariants or in + tests. +* Stable diagnostic/runtime **codes** are part of the contract; human messages + are not. Add new codes to `crates/marqdo-value/src/diagnostic.rs` and map + categories in `error.rs`. +* Values cross layers via `marqdo_value::Value` and the typed wire format + (`spec/value-wire.md`); never serialize `Int` as a JSON number. +* `check`/`inspect` must remain side-effect free: no execution, no foreign + process, no plugin `dlopen`, no file writes. +* Prose and executable nodes mix freely; prose never causes an action. +* Formatting: `rustfmt.toml` is authoritative; keep `cargo fmt --check` clean. + +## 6. Build, test and quality gate + +The workspace is offline-cached. Use the workspace cache as documented in +`tests/README.md` / AGENTS.md: + +```bash +export PATH=/root/.cargo/bin:$PATH +export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo +export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target +``` + +Commands: + +```bash +cargo build -p marqdo-cli # binary: target/debug/marqdo +cargo test --workspace # all crate unit + integration tests +cargo test -p marqdo-cli # conformance corpus + CLI integration tests +cargo fmt --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +Harness map (`crates/cli/tests/`): + +| Harness | Proves | +|---|---| +| `conformance_ast.rs` | frozen profile parses fixtures into checked-in AST (kinds, attributes, literals, spans, order) | +| `conformance_semantics.rs` | frontend + IR + runtime + host produce expected wire value/stdout | +| `negative.rs` | invalid executable shapes are hard errors; warning and runtime suites | +| `properties.rs` | prose-invariance (pure narrative edits do not change executable IR modulo spans) | +| `integration_cli.rs` | real CLI contract: init/check/run/call/inspect/fmt, multi-module, lock/sync, plugin trust and invocation | + +Crate-level tests worth knowing: + +* `marqdo-expression` — grammar/priority/nesting/limit tests; +* `marqdo-math/tests/kernel.rs` — exact arithmetic, bounds, equations, unsupported constructs; +* `marqdo-foreign/tests/kernels.rs` — real python3/node (skips if absent), timeout, output caps, env, wire protocol; +* `marqdo-plugin-abi/tests/loader_unix.rs` — compiles the C demo with `cc`, loads it, ABI-mismatch/negative cases (skips without `cc`); +* `marqdo-runtime/tests/interpreter.rs` — recursion, loops, returns, budgets, ambiguity; +* `marqdo-host` unit tests — capabilities, project FS, budgets. + +Tests that need Python/Node/`cc` print a skip notice and pass when unavailable; +record such skips in the PR. + +## 7. Running examples + +From the repository root (after `cargo build -p marqdo-cli`): + +```bash +B=target/debug/marqdo +$B run examples/hello.mq.md +$B call examples/factorial.mq.md 数学.阶乘 --arg n=5 # prints 120 +$B check examples/references.mq.md +$B inspect examples/references.mq.md --refs +$B inspect examples/hello.mq.md --ast --json +$B run examples/math.mq.md # has no main: use call +$B call examples/math.mq.md 数学.计算圆面积 +$B run examples/python-foreign.mq.md --allow-foreign # needs python3 +$B run examples/multi-module/index.mq.md +(cd examples/multi-module && ../../$B lock && ../../$B sync --locked) + +# Plugin demo (build the artifact + refresh its sha256 first) +./examples/plugin-demo/build.sh +$B plugin inspect examples/plugin-demo +$B run examples/plugin-demo/plugin-call.mq.md \ + --plugin examples/plugin-demo --trust-native-plugin --allow-plugin demo.solver +``` + +Notes: `run` defaults to `index.mq.md` and executes `main`; `call` requires a +file and function; root-only examples (for example `math.mq.md`) define no +`main`. `--mode strict` is expected to refuse to run (ADR-0004). + +## 8. Docs, ADRs and commit policy + +* `spec/` = normative contracts; `docs/` = architecture/decisions/status; + `README.md` = user quickstart; `AGENTS.md` = contributor rules. +* Changes to language semantics, AST mapping, wire format, plugin ABI, + package/lock format, authorization defaults or public CLI behavior require an + **Accepted ADR** (or superseding one) plus spec updates and tests. Use the + blueprint §16.2 template: Context / Decision / Alternatives / Markdown AST + shape / Compatibility / Security / Testing / Migration. +* ADRs are immutable once Accepted: add a new ADR with + `Status: Superseded by ADR-NNNN` rather than rewriting history (typo fixes + aside). +* Docs must describe what the code does. When implementation and blueprint + disagree, document the implementation and flag the divergence explicitly as + a limitation/open item; never invent behavior. +* Do not edit generated/vendored state, and never modify tests to hide a + regression. +* Commit in reviewable units with a message stating the affected spec/ADR and + the commands run. Do not commit built plugin artifacts (`*.so` is + gitignored), the cargo cache, or temp projects. +* There is no git-commit automation in this workspace; contributors commit + manually after review. + +## 9. Known gotchas + +* `.cache/` is the offline cargo cache/target used in this environment; it is + not part of the deliverable. +* Foreign availability (`python3`/`node`) and a C compiler (`cc`) determine + which tests run; a skipped test is not a passed feature. +* The strict sandbox, plugin worker isolation and package registry are not + implemented. Do not write docs or tests that pretend they are — see + `docs/STATUS.md` and `spec/security.md`. +* Runtime error spans in the CLI currently resolve against the entry file only; + imported-module error locations are approximate (ADR-0006). +* The semantic corpus may pin current quirks (for example the external-link + call code, or a float's last digit). A fixture diff for such a quirk is a + deliberate change, not a free update. diff --git a/spec/evaluation.md b/spec/evaluation.md index ae8e1d4..f48fe40 100644 --- a/spec/evaluation.md +++ b/spec/evaluation.md @@ -124,7 +124,7 @@ host boundary. Names are stable; adding one requires a spec update. | `整数` / `int` | `(x) -> Int` | from Int/Float(if integral)/Text(decimal) | | `小数` / `float` | `(x) -> Float` | from Int/Float/Text | | `表` / `table` | `(header: List, rows: List) -> Table` | validates row widths | -| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address | -| `math.parse/eval/simplify/differentiate` | see `spec/math.md` | P5 | +| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address; module-local, refuses ambiguous resource units (`ref.ambiguous`) | +| `math.parse/eval/substitute` | see `spec/math.md` | P5; `math.simplify`/`math.differentiate` are declared but return `math.unsupported` in this build | | `foreign.run/call` | see `spec/foreign.md` | P5, capability-gated | | `fs.read/write` | see `spec/security.md` | capability-gated | diff --git a/spec/foreign.md b/spec/foreign.md new file mode 100644 index 0000000..d9d009f --- /dev/null +++ b/spec/foreign.md @@ -0,0 +1,221 @@ +# Marqdo Foreign Kernels v1 + +Status: **one-shot kernels for Python and Node/JavaScript**, frozen protocol for +M0.5. There are no sessions or persistent kernels. + +Implementation: `crates/marqdo-foreign/src/lib.rs` (process supervision and +protocol), `crates/marqdo-host/src/kernels.rs::foreign` (capability check and +Marqdo mapping), runtime tests `crates/marqdo-foreign/tests/kernels.rs`, +example `examples/python-foreign.mq.md`. + +## 1. A code fence is a value, not a process + +A `code_block` node becomes an inert `Code { language, source, content_hash, +unit }` resource (`crates/marqdo-frontend/src/resources.rs`, +`marqdo_value::CodeValue`). Reading, checking, rendering or importing a +document never starts a foreign process. A process is created only when +`foreign.run` or `foreign.call` is executed after the capability check. + +Access is by `ref("#heading")` (exactly one code/formula resource in that +heading range; `ref.ambiguous` otherwise — see `spec/references.md` §5): + +````markdown +## Python 校验 + +```python +import json, math, sys +envelope = json.load(sys.stdin) +r = int(envelope["input"]["entries"]["r"]["decimal"]) +print(json.dumps({"type": "float", "value": math.pi * r ** 2})) +``` + +## main + +**结果 = foreign.call(ref("#python-校验"), {"r": 3})** + +> 打印(结果) +```` + +The anchor is the GitHub-style slug of the heading (`#python-校验`), not the +heading text. + +## 2. Supported languages + +`normalize_language` is ASCII case-insensitive and trims surrounding +whitespace: + +| Canonical | Accepted aliases | Interpreter lookup | +|---|---|---| +| `python` | `python`, `python3`, `py` | `python3`, then `python` on `PATH` | +| `javascript` | `javascript`, `js`, `node`, `nodejs` | `node` on `PATH` | + +Any other language yields `ForeignError::KernelUnavailable` (the `Code` value is +still valid data). `detect(language)` returns `KernelInfo { language, program, +version }` by scanning `PATH` and running the interpreter with `--version` +(5 s timeout, 64 KiB output cap, cleared environment except `PATH`); the +`version` is the first non-empty line of stdout or stderr. `detect` and +execution use the same `PATH`-only environment. + +## 3. `RunRequest` / `RunResult` + +```rust +pub struct RunRequest { + pub language: String, + pub source: String, + pub input: Option, // an already-typed wire value + pub timeout: std::time::Duration, + pub max_output_bytes: usize, +} + +pub struct RunResult { + pub status: RunStatus, // Success | Failed | TimedOut + pub stdout: String, // lossy UTF-8 for `run` + pub stderr: String, // preserved on failure + pub exit_code: Option, // None on signal/timeout + pub duration: std::time::Duration, + pub value: Option, // only `call` on success +} +``` + +`RunRequest::new` defaults to a 30 s timeout and 1 MiB output cap **for direct +library use**. The CLI host constructs requests with `StdHost` defaults: +`foreign_timeout = 10 s`, `foreign_output_limit = 1 MiB` (stdout+stderr +combined). The CLI has no per-kernel flag for these; `--timeout` only sets the +interpreter wall-clock deadline. + +`RunStatus::as_str()` yields `success`, `failed`, `timed_out`. + +## 4. stdin envelope + +The program is executed as a file; stdin receives exactly one JSON object +(`crates/marqdo-foreign/src/lib.rs::request_envelope`): + +```json +{ + "wire_version": 1, + "kind": "request", + "input": +} +``` + +`input` is **already encoded with the type-tagged value format** +(`spec/value-wire.md`), not a plain JSON mirror of the Marqdo value. For +example `{"r": 3}` becomes +`{"type":"map","entries":{"r":{"type":"int","decimal":"3"}}}`. The example +kernel reads `envelope["input"]["entries"]["r"]["decimal"]` accordingly. When +`input` is absent, the field is JSON `null` and the write side closes stdin +right after writing (EOF). + +Marqdo values that cannot cross the wire (none in v1, but see +`spec/value-wire.md` limits) are encoded before the process starts; the +envelope may be size-limited by the host's output/wire budgets. + +## 5. `run` versus `call` + +| | `foreign.run(ref, input?)` | `foreign.call(ref, input)` | +|---|---|---| +| One-shot isolated execution | yes | yes | +| stdin envelope | yes (input optional) | yes (input required) | +| stdout contract | captured as data; any content | must be **exactly one** `wire_version` 1 typed value; no logs/trailing noise | +| result | `Map` with `status`, `stdout`, `stderr`, `exit_code`, `duration_ms`, `value` (always `None` for `run`) | decoded `Value` | +| non-zero exit | `status: "failed"` in the returned map | `Ok(RunResult { status: Failed })` from the library, but the host turns it into `foreign.failure` runtime error | +| timeout | library returns `Ok` with `status: "timed_out"`; host returns the map | library returns `ForeignError::Timeout`; host maps to `resource.exhausted` | +| stdout pollution / invalid wire | irrelevant | `ForeignError::Protocol` → `foreign.failure` | + +`call` parses stdout with `serde_json` and then `marqdo_value::from_wire` with +default limits; an unknown `type` tag, bad decimal, JSON syntax error, invalid +UTF-8 or trailing bytes are all protocol errors. A successful call whose stdout +has no typed value is a protocol error at the host boundary ("foreign call +returned no typed value" / `foreign.failure`). + +The host result map from `foreign.run` (`crates/marqdo-host/src/kernels.rs:: +foreign_result_map`) is: + +```text +{status: "success"|"failed"|"timed_out", + stdout: Text, stderr: Text, exit_code: Int|None, + duration_ms: Int, value: None} +``` + +## 6. Process model and limits + +`crates/marqdo-foreign/src/lib.rs`: + +1. normalize the language (unknown → `KernelUnavailable` with install hint); +2. find the interpreter on `PATH`; +3. create a **private unique temp working directory** (mode 0700 on Unix) and + write the program to a private temp file (`.py`/`.js`, mode 0600); +4. spawn the interpreter **directly with `argv`** (no shell, no string + interpolation), cwd = the fresh temp dir, environment cleared except + `PATH`, stdin/stdout/stderr piped; +5. a dedicated thread writes the stdin envelope and closes the pipe; reader + threads capture stdout and stderr against a **shared** `max_output_bytes` + budget; the supervisor polls `try_wait` and kills the child on the wall-clock + deadline or output overflow; +6. after the child is reaped, readers get a 2 s grace to finish; cleanup (Drop) + removes the temp file and directory on success and error paths. + +Bounds: wall-clock `timeout` (10 s in the CLI), combined output +`max_output_bytes` (1 MiB in the CLI; stderr counts too), reader grace 2 s, +detection timeout 5 s / 64 KiB. A kernel that ignores stdin and exits does not +hang the host. `OutputTooLarge` drops the captured output and kills the child. +Grandchild processes are not killed when the direct child is killed. + +Environment: only `PATH` is inherited, so Marqdo secrets in other environment +variables do not leak by default. This is hygiene, not a security boundary. + +## 7. Capability gating + +`foreign.run`/`foreign.call` are effectful builtins +(`crates/marqdo-ir/src/builtins.rs`). The host adapter +(`crates/marqdo-host/src/kernels.rs::foreign`) first checks +`Capabilities::check_foreign`, which requires both `foreign` and `process`. + +* default (`Capabilities::pure()`/`project()`): denied → + `permission.denied` "foreign kernel execution was not granted"; +* `--allow-foreign` in `project` mode grants both flags; +* `pure` mode rejects `--allow-foreign` at CLI parse time (exit 2); +* `strict` mode refuses to run before any kernel is considered; +* a manifest `capabilities.process`/foreign request never grants anything; +* the check is repeated for every call (not cached at parse time). + +If the first argument is not a `Code` value, the host returns `runtime.type` +("foreign kernel expects a Code value from `ref("#代码块")`"). A code fence with +an empty language is `kernel.unavailable` ("no kernel can be selected"). +Unsupported languages are `kernel.unavailable` with an install hint. + +## 8. Error mapping + +| Library error | Host runtime code | +|---|---| +| `KernelUnavailable { language, hint }` | `kernel.unavailable` (message includes hint) | +| `Spawn { message }` | `host.error` | +| `Io { message }` | `host.error` | +| `Timeout { seconds }` | `resource.exhausted` | +| `Protocol { message }` | `foreign.failure` | +| `OutputTooLarge { limit }` | `resource.exhausted` | + +`foreign.call` with a non-zero exit becomes `foreign.failure` with the stderr +text preserved; a successful process whose stdout is not a typed value is also +`foreign.failure`. + +## 9. Security caveats + +* **Process isolation is not a sandbox.** The child runs as the host user and + can read/write any file the user can, open sockets, spawn processes and + persist beyond the one-shot execution. Fresh cwd, cleared environment and a + killed direct child are limited hygiene measures. Only `strict` mode + (currently refused) would promise OS-level confinement. +* There is no network policy for foreign kernels; `net` capability applies to + host operations, not to what a child process does. +* Output is captured, not inherited; the child cannot write to the CLI's + stdout/stderr directly, but it can still affect the filesystem. +* Kernel availability is environment-dependent (`python3`/`python`/`node` on + `PATH`). Missing kernels must be fixable by the user (the error includes an + install hint). +* `foreign.run` returns raw stdout as data; consumers must not treat it as a + typed value unless they parse it themselves. `foreign.call` is the typed + path. +* Sessions, shared state, persistent kernels and automatic environment + management (uv/venv) are not implemented. Python/Node are optional foreign + kernels, never the official execution backend. diff --git a/spec/math.md b/spec/math.md new file mode 100644 index 0000000..32d4950 --- /dev/null +++ b/spec/math.md @@ -0,0 +1,227 @@ +# Marqdo Math v1 + +Status: **frozen subset for M0.5.** The kernel computes numbers; it never proves +theorems. `math.simplify` and `math.differentiate` are **not implemented** in +this build. + +Implementation: `crates/marqdo-math` (`parse`, `evaluate`, `substitute`, +`MathError`), host adapter `crates/marqdo-host/src/kernels.rs::math`, builtin +metadata `crates/marqdo-ir/src/builtins.rs`, tests +`crates/marqdo-math/tests/kernel.rs`. + +## 1. From Markdown to a value + +A `math` / `math_block` node is an **inert document resource**, never an +operation. The frontend attributes every formula to its innermost enclosing +heading unit (`crates/marqdo-frontend/src/resources.rs`). `ref("#heading")` +returns a `Formula` value only when that unit contains exactly one resource; +with two or more formulas the runtime raises `ref.ambiguous` and refuses to +pick one (`spec/references.md` §5, ADR-0003). + +```markdown +## 圆面积公式 + +$$ +A = \pi r^2 +$$ + +## 计算圆面积 + +**面积 = math.eval(ref("#圆面积公式"), {"r": 4}, target="A")** + +*面积* +``` + +Nothing is evaluated by reading or rendering the document. `math.*` must be +called explicitly and the host checks the `math` capability at execution time +(default: granted in `pure`/`project`; there is no CLI flag to revoke it today). + +## 2. Grammar + +The parser accepts exactly this LaTeX subset (whitespace insignificant; byte +offsets are reported for parse errors): + +```text +formula = expr [ "=" expr ] ; single '=' only, never assignment +expr = term { ("+" | "-") term } ; +term = unary { ("*" | "/" | "·" | "×" | implicit) unary } ; +unary = ("+" | "-") unary | power ; +power = primary [ "^" unary ] ; right associative +primary = number + | identifier Unicode letter (plus digits/underscore) + | "\pi" | "pi" | "\e" | "e" constants + | "(" expr ")" | "\left(" expr "\right)" + | "{" expr "}" grouping + | "\frac{" expr "}{" expr "}" exact division + | "\sqrt{" expr "}" real square root + ; +number = digits [ "." digits ] | "." digits ; decimal, no exponent form +``` + +Rules and edge cases: + +* `\cdot` and `\times` are accepted as synonyms for `*`. +* Implicit multiplication between adjacent factors: `2x`, `\pi r^2`, `(a)(b)`, + `m a`. Concatenated letters form one identifier (`ma` is the variable `ma`; + write `m a` or `m * a` for a product). Two adjacent number literals are a + parse error ("ambiguous"). +* Unicode identifiers are supported (`面积`, `r`, `速度`). Plain multi-letter + identifiers are variables, but names of common elementary functions (`sin`, + `cos`, `log`, `exp`, ...) are reserved and rejected as unsupported so + `sin(x)` is not silently read as a product. +* Underscores are ordinary identifier characters: `x_1` is the variable `x_1`, + not a subscript. +* A single top-level `=` creates an `Equation { left, right }`; it is **not** + assignment and never writes a Marqdo binding. A second `=` is a parse error. +* `\frac{a}{b}` is exact division (`Div`); `1.5 + 2.5` is an exact rational + computation, not binary floating point. +* `\sqrt{x}` is real-valued: a negative radicand is a `Type` error. +* Decimal literals have no exponent form (`1e3` is rejected); `2e` is + `2 * e` (number times the constant), while `2e3` is rejected rather than + reinterpreted. + +### 2.1 Unsupported constructs + +Everything outside the grammar is a structured `Unsupported` or `Parse` error: +matrices, sums/integrals, function applications (`\sin`, `\cos`, ...), +multi-letter commands, `\left[`/`\right]`, mismatched/missing braces, +scientific notation, commas, arrays, subscripts, superscript commands other +than the `^` operator, `\frac`/`\sqrt` without braced arguments, labels, +alignment characters, line breaks. There is no `simplify`, `differentiate`, +`integrate`, `matrix` or theorem/proof operation in this build. + +## 3. Numeric semantics + +* Exact rational arithmetic uses `num-rational::Ratio` over the + `num-bigint::BigInt` type that also backs `Value::Int`; integers are + arbitrary precision. +* Decimal literals are exact rationals (`0.25` = `1/4`). `Int` bindings become + exact integers. `Float` bindings are interpreted as the **exact binary + rational value** of the `f64`; non-finite `Float` bindings are a `Type` + error. +* `+ - * / ^` are exact when both operands are exact. Division keeps a + rational, so `\frac{1}{3} * 3` is exactly `1`. +* The final value is a Marqdo `Int` exactly when the exact result is an + integer, and a `Float` otherwise (`crates/marqdo-math/src/eval.rs`, the + `Numeric` enum). Examples: `1.5 + 2.5` → `Int(4)`; `1 / 2` → `Float(0.5)`; + `100 / 4` → `Int(25)`. +* `\pi`, `\e` and a `\sqrt` of a non-perfect square introduce an `f64` + approximation. Anything derived from them is a `Float`. Precision follows + IEEE-754 binary64 (about 15–17 significant decimal digits); approximate + results are never presented as exact. +* `\sqrt` of a perfect square of an exact rational stays exact (`\sqrt{4}` → + `Int(2)`); `\sqrt{1/4}` is `Float(0.5)` in this build (it becomes + approximate via the root path, matching the documented test). +* The exponent of `^` must evaluate to an integer. Magnitude is bounded by + `MAX_EXPONENT`. Negative integer exponents are exact reciprocals (the result + is therefore a `Float` unless it is integral). `0 ^ 0` is a `Type` error. +* Exact intermediate/result numerators and denominators are bounded by + `MAX_RESULT_BITS`; exceeding the bound returns a structured `Unsupported` + error rather than hanging or exhausting memory. +* Division by zero is a `Type` error inside the kernel. The host maps all + `MathErrorKind::Type` to `runtime.type`; it does **not** produce + the interpreter's `runtime.division_by_zero` code for a formula. This is a + known divergence from the module doc comment in + `crates/marqdo-math/src/lib.rs` and should be fixed or documented in an ADR. +* Constants are reserved: a binding named `pi` or `e` never overrides the + constant, and they cannot be solve targets (use `E`, `x`, `r`, ...). + +### 3.1 Bounds + +| Constant | Value | Location | +|---|---|---| +| `MAX_SOURCE_BYTES` | 1,000,000 | `crates/marqdo-math/src/parse.rs` | +| `MAX_PARSE_DEPTH` | 256 | parse.rs | +| `MAX_AST_NODES` | 512 | parse.rs | +| `MAX_EXPONENT` | 10,000 | `crates/marqdo-math/src/eval.rs` | +| `MAX_RESULT_BITS` | 1,000,000 | eval.rs | + +The AST-node bound also bounds later recursion for long operator chains, so +hostile input cannot overflow the stack during evaluation, rendering or +substitution. No code path panics on untrusted LaTeX; all failures are +`MathError`. + +## 4. Equations and solving a target + +`FormulaKind` is `Expression(expr)` or `Equation { left, right }`. + +`evaluate(formula, bindings, target)`: + +* for a plain expression, `target` must be `None`; supplying one is + `Unsupported` ("a target variable is only meaningful for an equation"); +* for an equation, `target` is required (`Unsupported` otherwise); an empty + target is `Unsupported`; +* the target must occur **exactly once** and the equation must be **affine in + the target**. Other variables are substituted from `bindings` first; a + missing binding is `UndefinedVariable`. + +Affine solving supported by `crates/marqdo-math/src/eval.rs`: + +* `+`, `-`, multiplication by a target-free side, division where the target is + in the numerator, and inversion of those operations in either direction: + `2x = 4`, `4 = 2x`, `x / 2 = 3`, `3 = x / 2`, `x + 2 = 5`, `2x + 1 = 7`, + `(x + 1) * 2 = 8`, `x = 2y + 1` (with `y` bound); +* `F = m a` solved for `F`, `m` or `a`. + +Rejected as `Unsupported`: target occurs zero times or more than once, target +in a denominator, under a root, in an exponent, or in a non-affine product +(e.g. `x * x`). Equations never mutate bindings and never create Marqdo +variables. + +## 5. Substitution + +`substitute(formula, bindings)` replaces bound variables with exact numeric +literals and keeps unbound variables symbolic. The returned `Formula.source` is +a deterministic canonical rendering that `parse` accepts again; it is not +byte-identical to the input formatting. Host-facing `math.substitute` wraps the +result in a `Formula` value with `unit: None`, `display: false`, span of the +call site. + +## 6. Host-facing surface (`math.*`) + +All operations are builtins in `crates/marqdo-ir/src/builtins.rs`, dispatched +in `crates/marqdo-runtime/src/builtins_impl.rs`, and executed in the host +adapter `crates/marqdo-host/src/kernels.rs`. `math` capability is checked +before touching the kernel. + +| Builtin | Parameters | Result | +|---|---|---| +| `math.parse` | `formula: Formula or Text` | `Formula` (source = input; `unit=None`, `display=false`); parse errors are `math.unsupported` | +| `math.eval` | `formula: Formula or Text`, `bindings: Map`, optional `target: Text` | `Int` or `Float`; equation requires target | +| `math.substitute` | `formula: Formula or Text`, `bindings: Map` | `Formula` with canonical source | +| `math.simplify` | `formula` | **not implemented**: runtime error `math.unsupported` ("not part of the supported math subset in this build") | +| `math.differentiate` | `formula`, `变量` | **not implemented**: `math.unsupported` | + +Input typing: + +* formula arguments accept a `Formula` value or `Text`; anything else is + `runtime.type`; +* `bindings` must be a `Map` whose values are `Int` or `Float`; any other + value type is `runtime.type` with a message naming the binding; +* `target` must be `Text` or `None`; other types are `runtime.type`; +* builtin arity/parameter-name mistakes are catchable statically + (`call.arity`, see `spec/mapping.md` §6); +* the math kernel is a pure computation: no file, network, process or plugin + effects, and nothing it returns is a capability. + +### 6.1 Error mapping + +Host mapping (`crates/marqdo-host/src/kernels.rs::math_error`): + +| `MathErrorKind` | Runtime code | +|---|---| +| `Parse` | `math.unsupported` (message includes the byte offset when known) | +| `Unsupported` | `math.unsupported` | +| `UndefinedVariable` | `symbol.unknown` | +| `Type` | `runtime.type` | + +## 7. What this is not + +* Not a computer-algebra system: no simplification, differentiation, + integration, matrices, series, assumptions, units or symbolic rewriting. +* Not a theorem prover: `F = m a` is an equation value, not a proof, and a + numeric solve is not a certificate. +* Not a LaTeX renderer: rendering is the Markdown reader's job; the kernel only + receives the raw `latex` text of a `math` node. +* Not error-free in the floating-point sense: approximation precision is + documented, not guaranteed beyond binary64. diff --git a/spec/package-format.md b/spec/package-format.md new file mode 100644 index 0000000..f40eccf --- /dev/null +++ b/spec/package-format.md @@ -0,0 +1,258 @@ +# Marqdo Project Package Format v1 + +Status: **v1 format as implemented in M0.5.** Git resolution, registries, +transitive dependency solving and environment materialization are **not** +implemented; see §8. + +Implementation: `crates/marqdo-package/src/lib.rs` (manifest/lock types and +validation), `crates/cli/src/commands.rs` (`init`, `add`, `remove`, `lock`, +`sync`). + +## 1. Project root + +The project root is the nearest ancestor directory of the entry document that +contains `marqdo.yaml` (`marqdo_package::discover_root`). If no manifest exists +anywhere above the entry, the entry file's parent directory is used as the +root (a document can be checked/run without a project). + +`ModuleId`s are project-root-relative paths (`spec/references.md` §2.1), so the +root choice affects import identity. `add`/`remove`/`lock`/`sync` discover the +root from the **current working directory**, not from a document path. There is +no `--project` flag in this build. + +## 2. `marqdo.yaml` + +```yaml +project: + name: example + version: 0.1.0 + marqdo: ">=0.1,<1" # optional; informational in this build +dependencies: + geometry: + path: ./packages/geometry # exactly one of path / git+rev + quantum: + git: https://github.com/example/quantum-mq + rev: 0123456789abcdef0123456789abcdef01234567 +plugins: + solver: + version: "^1.0" # requirement only; not installed or authorized +capabilities: # requests only; not grants + fs: + read: ["./data/**"] + write: ["./output/**"] + net: + hosts: [] + process: false +``` + +### 2.1 Fields + +| Field | Type | Required | Semantics | +|---|---|---|---| +| `project.name` | string | yes | non-empty package name | +| `project.version` | string | yes | non-empty version string (not semver-validated in the manifest) | +| `project.marqdo` | string | no | required Marqdo runtime range; stored, not enforced by this build | +| `dependencies..path` | path string | one of path/git | project-relative path to a dependency directory or `.mq.md` file | +| `dependencies..git` | URL string | one of path/git | git source (must be pinned with `rev`) | +| `dependencies..rev` | string | with `git` | exact revision; a bare `git` without `rev` is rejected | +| `plugins..version` | string | per entry | non-empty plugin version requirement; declarative only | +| `capabilities.fs.read` / `.write` | list of strings | no | requested project-relative globs | +| `capabilities.net.hosts` | list of strings | no | requested hosts | +| `capabilities.process` | bool | no | requested process execution | + +Unknown fields are ignored (the types do not `deny_unknown_fields`). Duplicate +YAML keys follow `serde_yaml` behavior (last one wins); there is no duplicate-key +diagnostic. + +### 2.2 Validation (`Manifest::validate`) + +`marqdo.yaml` is loaded with `Manifest::from_yaml`/`Manifest::load`, which +validates: + +* `project.name` and `project.version` are non-empty after trimming; +* for every dependency, exactly one source is present: + * both `path` and `git` → error "declares both path and git"; + * neither → error "must declare a path or a pinned git source"; + * `git` without `rev` → error requiring an exact revision; + * an absolute `path` → error "path must be project-relative"; +* for every plugin requirement, `version` is non-empty. + +Note: a *git* dependency passes manifest validation when pinned, but the +dependency is not resolvable or lockable in this build (§5). + +### 2.3 `capabilities` are requests + +The manifest is the project's **request** for permissions. It is never read +when constructing the host capability set; only CLI flags/grant APIs create +grants (`spec/security.md` §2). `net` has no implementing host op, and +`process`/foreign execution still needs `--allow-foreign`. + +## 3. `marqdo.lock` + +`lock_version: 1`, packages sorted by name: + +```yaml +lock_version: 1 +packages: +- name: geo + source: path:./packages/geo + sha256: 101a6f6db4cc48d8c138b4716028da9bc8934b5f02b2235e020fbcabba15b757 +``` + +| Field | Semantics | +|---|---| +| `lock_version` | must be `1`; any other value is rejected ("unsupported lock_version N (expected 1)") | +| `packages[].name` | dependency name from the manifest | +| `packages[].source` | `path:`; a git source would be `git:...@rev`, but git is not implemented in this build | +| `packages[].sha256` | optional; lowercase hex SHA-256 of the dependency tree | + +### 3.1 Deterministic serialization + +`LockFile::to_yaml` sorts packages by name before serializing. The format +contains no timestamps, no absolute paths and no machine-specific values, so +repeated `marqdo lock` runs over unchanged dependencies are byte-identical. +`LockFile::from_yaml` also sorts packages on load, so equality comparison is +order-insensitive. + +### 3.2 Tree hash + +For a path dependency, `lock` hashes the dependency tree +(`crates/cli/src/commands.rs::hash_tree`): + +1. walk the dependency directory with `walkdir` (`sort_by_file_name`), keeping + regular files only (symlinks are not followed and are skipped); +2. sort files by their root-relative path (normalized to `/`); +3. feed into SHA-256, for each file in order: + `relative_path bytes`, a `0x00` byte, `file bytes`, a `0x00` byte; +4. hex-encode the digest. + +This is a deterministic content fingerprint, not a Merkle tree and not a +cryptographic commitment to file modes, symlinks or empty directories. It +detects content changes and additions/removals of regular files. + +### 3.3 Lock generation rules + +`marqdo lock` (run in a project): + +1. discovers the project root from the cwd; +2. loads and validates `marqdo.yaml`; +3. for each manifest dependency, in map (sorted) order: + * a git source → error "git dependencies are not supported by this build + (path dependencies only)"; no lock is written; + * no path source → error; + * the dependency path must exist; the tree is hashed; +4. writes `marqdo.lock` (exit 0, `locked N package(s)`). + +What it does **not** do: resolve semver ranges, traverse a dependency's own +`marqdo.yaml`/transitive dependencies, clone git repositories, download +registries, verify package signatures, or install anything. + +## 4. `sync` + +`marqdo sync [--locked]` (run in a project): + +1. discovers the root; loads the manifest and the existing `marqdo.lock` + (missing lock → error "run `marqdo lock` first", exit 1); +2. recomputes the expected lock file exactly as `lock` would (including tree + hashes); +3. with `--locked`, compares the expected lock with the on-disk lock; a + difference (manifest change or dependency content change) prints + "lockfile is out of date or dependency contents changed; run `marqdo lock`" + and exits 1; +4. without `--locked`, prints `synchronized N package(s)` and exits 0. + +Current behavior: `sync` **never rewrites** the lock file and never materializes +a `.marqdo/env` directory. It is a verification command only. There is no +`--offline` flag and no `lock --locked` flag (the blueprint example +`marqdo lock --locked` is not implemented; use `sync --locked`). + +## 5. `init` + +`marqdo init [dir]`: + +* refuses to overwrite an existing `marqdo.yaml` (exit 1); +* creates the directory if needed; +* derives the project name from the directory's final path component (falling + back to `example`); +* writes `marqdo.yaml` from `Manifest::template`: + + ```yaml + project: + name: + version: 0.1.0 + marqdo: '>=0.1,<1' + ``` + + (empty dependencies/plugins/capabilities are omitted); +* writes `index.mq.md`: + + ```markdown + ## main + + **问候 = "你好,Marqdo"** + + > 打印(问候) + + *问候* + ``` + +* writes `.gitignore` containing `.marqdo/`; +* prints the next-step hint `marqdo check index.mq.md && marqdo run index.mq.md`. + +`init` does not create `marqdo.lock`, `.marqdo/`, a plugin directory or a git +repository. + +## 6. `add` / `remove` + +Both commands edit **only** `marqdo.yaml` (and, for `remove`, prune a matching +stale lock entry). They do not fetch, install, resolve or authorize anything. + +* `marqdo add --path ` or + `marqdo add --git --rev `: + validate the dependency shape (same rules as §2.2), refuse an existing name + (exit 1), insert it, write the manifest, and tell the user to run + `marqdo lock`. Adding a git dependency is allowed at the manifest level even + though `lock` will refuse it later. +* `marqdo remove `: remove the dependency from the manifest (exit 1 if + absent), and drop a same-named entry from an existing lock file. It does not + re-hash or re-lock the rest of the project. + +This differs from the original task brief that listed `add`/`remove` as +"not supported yet": the CLI does implement them as manifest editors. They +are intentionally shallow and should not be confused with a dependency +resolver. + +## 7. JSON output + +`check`, `inspect`, `run`, `call`, `plugin inspect`, `add`, `remove`, `lock` +and `sync` accept `--json` and print a single JSON object on stdout. Relevant +package shapes: + +* `lock`: `{"command":"lock","success":true,"packages":[...]}` +* `sync`: `{"command":"sync","success":true,"locked":,"packages":[...]}` +* `add`: `{"command":"add","success":true,"dependency":"","manifest":""}` +* `remove`: `{"command":"remove","success":true,"dependency":""}` + +Errors in these commands are printed as plain `error: ...` on stderr even when +`--json` is passed (the JSON success envelope is not emitted; this is an +inconsistency to fix). + +## 8. Not supported in this build (explicit) + +| Capability | Status | +|---|---| +| Git dependency resolution (clone/fetch/checkout) | Not implemented. Manifest accepts `git`+`rev`; `lock` refuses it. | +| Registry / package publishing / SemVer resolution | Not implemented. | +| Transitive dependency graph and conflict resolution | Not implemented; only direct manifest dependencies are hashed. | +| `tree`, `update`, `vendor`, `package` commands | Not implemented. | +| `--offline`, `--locked` on `lock` | Not implemented. | +| Environment materialization (`.marqdo/env`, `cache`, `kernels`, `sessions`) | Not implemented; `init` only writes `.marqdo/` to `.gitignore`. | +| Dependency signatures / trusted publishers / supply-chain scanning | Not implemented. | +| Multiple/nested project boundary diagnostics | Not implemented; nearest `marqdo.yaml` wins. | +| Lock file migration between `lock_version`s | Only version 1 is accepted. | +| Plugin installation / plugin dependency management | Not implemented; `plugins:` in the manifest is declarative only. | + +The lock/package format is deliberately small and deterministic so these can +be added later without changing the identity model. Changing `lock_version`, +`source` syntax or the tree-hash algorithm requires an ADR and a migration +plan. diff --git a/spec/plugin-abi.md b/spec/plugin-abi.md new file mode 100644 index 0000000..ad03dff --- /dev/null +++ b/spec/plugin-abi.md @@ -0,0 +1,419 @@ +# Marqdo Plugin ABI v1 + +Status: **frozen ABI v1.0** (`MQ_ABI_MAJOR=1`, `MQ_ABI_MINOR=0`). The canonical +C header is `include/marqdo_plugin_abi_v1.h`; the Rust mirrors are in +`crates/marqdo-plugin-abi/src/abi.rs`; the plugin-side SDK is +`crates/marqdo-plugin-sdk`. + +> **A C ABI is not a sandbox.** A native plugin is loaded into the host +> process. It can read host memory, call arbitrary syscalls, bypass the +> capability model and crash the host. Loading requires an explicit trust +> decision. The isolated worker transport described by the blueprint is **not +> implemented** in this build. + +## 1. Independent versions + +Three versions exist and are never conflated: + +| Version | Current | Where | +|---|---|---| +| Marqdo language/semantics | M0.5, `spec/*` | `docs/DEVELOPMENT_BLUEPRINT.md`, specs | +| Plugin ABI major.minor | `1.0` | `MQ_ABI_MAJOR`/`MQ_ABI_MINOR`, `marqdo_plugin_abi::ABI_MAJOR/ABI_MINOR` | +| Plugin package SemVer | e.g. `0.1.0` | `plugin.yaml` `plugin.version` | + +The JSON wire protocol has its own `wire_version = 1` +(`spec/value-wire.md`); it is not the ABI major version. + +## 2. Stable-layout types + +The header defines only fixed-width integers, `size_t`-based length+pointer +buffers, opaque host-owned handles and `extern "C"` function pointers. Rust +`String`/`Vec`/trait objects/`Result`/panics, C++ exceptions, Go objects and +raw host pointers never cross the boundary. + +On a 64-bit target (`include/marqdo_plugin_abi_v1.h` static assertions and +`crates/marqdo-plugin-abi/src/abi.rs` compile-time assertions): + +| Type | Size | Align | +|---|---|---| +| `MqBytes` | 16 | 8 | +| `MqOwnedBytes` | 16 | 8 | +| `MqPluginApi` | 32 | 8 | +| `MqHostApi` | 56 | 8 | +| `MqPluginInvoke` (fn ptr) | 8 | — | + +On a 32-bit target the Rust side asserts `MqBytes`/`MqOwnedBytes` = 8, +`MqPluginApi` = 20 and `MqHostApi` = 28. The layout uses `size_t`, so an +artifact is only loadable by a host built for the **same target triple, pointer +width and byte order**. `(NULL, 0)` is the only empty buffer; a non-empty +buffer with a NULL pointer is a protocol error. + +### 2.1 Ownership and lifetimes + +| Buffer | Producer | Lifetime / free rule | +|---|---|---| +| `args_wire`, `export_name`, `signature_wire`, capability `request_wire`, `plugin_metadata_wire` (as seen by host) | caller passing it | borrowed for the duration of the call; consumer must copy and must not retain or free | +| `result_wire`, `error_wire`, capability `reply_wire` | the producing side | must be allocated with `host->alloc(host->host_data, size)` and is released with `host->free(host->host_data, data)` by the consumer after copying; never `malloc`/`new`/GC memory | +| `plugin_metadata_wire` (as returned by plugin) | plugin static storage | valid only during `mq_plugin_entry_v1`; host copies it immediately | + +`MqCallContext` is opaque, host-owned and valid only for the invocation it was +minted for. Plugins must not dereference, cache or copy it. + +### 2.2 Size limits (enforced by host / SDK) + +| Constant | Bytes | +|---|---| +| `MQ_MAX_EXPORT_NAME_BYTES` | 256 | +| `MQ_MAX_SIGNATURE_WIRE_BYTES` | 4096 | +| `MQ_MAX_METADATA_WIRE_BYTES` | 65536 | +| `MQ_MAX_REQUEST_WIRE_BYTES` | 16 MiB (16,777,216) | +| `MQ_MAX_RESULT_WIRE_BYTES` | 64 MiB (67,108,864) | +| `MQ_MAX_ERROR_WIRE_BYTES` | 1 MiB (1,048,576) | +| `MQ_MAX_CAPABILITY_REPLY_BYTES` | 16 MiB | + +Wire JSON decoding additionally applies the default `WireLimits` from +`spec/value-wire.md` (depth 64, 1,000,000 items, 16 MiB strings). Oversized +payloads are rejected, never truncated and reparsed. + +## 3. Exported entry point and tables + +Required symbol: `mq_plugin_entry_v1` (`MQ_PLUGIN_ENTRY_SYMBOL`): + +```c +int32_t mq_plugin_entry_v1(const MqHostApi *host, MqPluginApi *plugin); +``` + +Called exactly once, by the loading thread: + +* the host supplies a valid `MqHostApi` table and a zeroed `MqPluginApi` whose + `struct_size = sizeof(MqPluginApi)`; +* the plugin must validate the host table, fill `plugin.abi_major`, + `plugin.abi_minor`, `plugin.plugin_metadata_wire` and register every export + via `host->register_function`; +* it returns `MQ_OK` only after all exports are registered; on non-zero the + host ignores the plugin table and fails the load. + +`MQ_OK` and error codes (stable integers): + +| Code | Name | Meaning | +|---|---|---| +| 0 | `MQ_OK` | success | +| 1 | `MQ_ERR_INVALID_ARGUMENT` | NULL pointer, bad pointer/length pair | +| 2 | `MQ_ERR_ABI_MISMATCH` | ABI major mismatch or incompatible `struct_size` | +| 3 | `MQ_ERR_REGISTRATION` | an export could not be registered | +| 4 | `MQ_ERR_UNSUPPORTED` | operation/type/capability unsupported | +| 5 | `MQ_ERR_OUT_OF_MEMORY` | host allocation failed | +| 6 | `MQ_ERR_BAD_WIRE` | payload is not valid `wire_version` 1 JSON | +| 7 | `MQ_ERR_INTERNAL` | unexpected failure, including caught panic | +| 8 | `MQ_ERR_CAPABILITY` | capability request denied/failed | + +### 3.1 Version negotiation + +Host side (`negotiate` in `crates/marqdo-plugin-abi/src/abi.rs`): + +* `plugin.abi_major != 1` → load fails with `abi_major_mismatch`; +* `plugin.abi_minor` greater than the host minor is accepted but **clamped** to + the host minor (currently 0); only features defined at that minor may be + used; +* the plugin must reject a host table with `host == NULL`, + `host->struct_size < sizeof(MqHostApi)`, `host->abi_major != 1` or NULL + alloc/free/register_function/request_capability pointers. + +`struct_size` is the extension mechanism: fields may only be appended in a +later ABI minor, never reordered or resized; a consumer only reads a field when +`struct_size` covers it. + +### 3.2 Threading and lifetime + +v1 is single-threaded and synchronous: entry is called once; invocations and +host callbacks for one plugin instance run on that thread with at most one call +in flight. A plugin must not start threads that call back into `MqHostApi`. +v1 has no shutdown callback and no hot unload/reload; the `LoadedPlugin` +wrapper unloads only when dropped, and the docs require one instance per +artifact path (ABI v1 plugins may keep the host table in a process-global). +No panic/exception/longjmp may cross the boundary; the SDK catches Rust panics +and returns `MQ_ERR_INTERNAL` with a structured error envelope. + +## 4. `plugin.yaml` + +```yaml +plugin: + name: demo.solver + version: 0.1.0 + abi: "^1.0" + kind: native + exports: + - name: solver.add + signature: "($a: Int, $b: Int) -> Int" + requires: + - math.compute + artifacts: + - target: x86_64-unknown-linux-gnu + file: solver.so + sha256: "<64 lowercase hex>" +``` + +Validation (`crates/marqdo-plugin-abi/src/manifest.rs::PluginManifest::validate`): + +* `name`: dotted segments, each starting with a lowercase ASCII letter and + containing only lowercase letters, digits, `-`/`_`; ≤ 256 bytes; package + names starting with `marqdo.` or `builtin.` are reserved; +* `version`: semver-ish `major.minor.patch` with optional `-prerelease` and + `+build`; numeric components have no leading zeros; not a SemVer range + resolver; +* `abi`: caret-style requirement `^1`, `^1.0`, `1` or `1.0`. Caret semantics: + `^M.m` means `>=M.m, `, no control characters; +* `requires`: at most 64, unique names from the whitelist `log`, + `math.compute`, `fs.read`, `fs.write`, `net.request`, `clock`, `random`, + `artifact.record`; this is a declaration, not a grant; +* `artifacts`: 1–32, one per target triple; target parses as a 3–4 component + triple (lowercase, alphanumeric/underscore components); `file` is a + non-empty relative path ≤ 256 bytes with no absolute/`..` escape; `sha256` is + exactly 64 hex characters. + +Manifest parsing/validation never executes plugin code and performs no +filesystem access (artifact verification is a separate step). + +## 5. Artifact selection and verification + +* `HostTarget::current()` derives a conventional triple from + `std::env::consts` (`x86_64-unknown-linux-gnu`, `aarch64-apple-darwin`, + `x86_64-pc-windows-msvc`, ...). +* `resolve_artifact` requires **exact triple equality**; pointer width and byte + order follow from the triple. No fallback to another architecture. +* `verify_artifact` resolves the path below the manifest directory and compares + the file's SHA-256 with the manifest (case-insensitive hex). A missing file + is `missing_artifact`; a digest mismatch is `hash_mismatch`. +* Verification proves the artifact is the file described by the manifest; it + proves nothing about trust or authorization. + +## 6. Loader lifecycle (host side) + +`crates/marqdo-plugin-abi/src/loader.rs::LoadedPlugin::open_with`: + +1. `manifest.validate()`; +2. require `kind == native` (otherwise `plugin_kind_not_loadable_in_process`); +3. resolve artifact for the host target and verify its sha256; +4. `libloading::Library::new` (a trust decision, not a sandbox); +5. build the host API table and `HostState`, initialize `MqPluginApi`; +6. call `mq_plugin_entry_v1` once; non-zero → structured load error; +7. require `plugin_api.struct_size >= sizeof(MqPluginApi)`; +8. negotiate ABI major/minor; +9. copy `plugin_metadata_wire` and require metadata `name`/`version` to match + the manifest exactly; +10. take registered exports and cross-check against the manifest: duplicate + registrations, missing declared exports, signature mismatches and + undeclared exports all fail the load; +11. keep the library mapped for the lifetime of `LoadedPlugin`. + +Invocation (`LoadedPlugin::invoke`): validate the request envelope, check the +request size, find the registered export, mint a `MqCallContext`, call the +plugin's `MqPluginInvoke`, copy and free both output buffers, then enforce the +envelope contract: + +* success (`MQ_OK`): `error_wire` must be empty; `result_wire` must be a + non-empty `wire_version` 1 success envelope; the typed value is optional + (absent `value` means `Unit`); +* failure: `error_wire` must hold a `wire_version` 1 error envelope; a missing + or invalid error envelope still fails the call (structured + `plugin_call_failed`). + +Capability requests from the plugin go through +`MqHostApi::request_capability` into the host's `CapabilityHandler`, on the +same thread, one at a time. The handler receives the capability name and +request bytes and returns reply bytes or an error. When no handler is +installed (the CLI's case) **every request is denied** (`MQ_ERR_CAPABILITY`). + +## 7. Wire envelopes + +All payloads are UTF-8 JSON with `wire_version = 1`. Values inside `args` and +`value` use the type-tagged encoding of `spec/value-wire.md`. + +Request: + +```json +{ + "wire_version": 1, + "kind": "request", + "function": "solver.add", + "args": { + "a": {"type": "int", "decimal": "20"}, + "b": {"type": "int", "decimal": "22"} + }, + "context": {"trace_id": "optional", "capability_handle": "opaque-and-scoped"} +} +``` + +Response (success): + +```json +{"wire_version":1,"kind":"response","value":{"type":"int","decimal":"42"}} +``` + +A success response with neither `value` nor `error` means `Unit`. A response +with both is rejected (`ambiguous_response`). Error: + +```json +{"wire_version":1,"kind":"response","error":{ + "kind":"invoke","code":"missing_argument", + "message":"required argument `a` is missing","recoverable":true +}} +``` + +Envelope limits: at most `MAX_ARGS = 1024` request arguments; request ≤ 16 MiB, +response ≤ 64 MiB, error ≤ 1 MiB. `context` carries no secrets and no +reconstructable OS handles. `signature_wire` is +`{"signature":""}`; `plugin_metadata_wire` is +`{"name":"","version":""}` (extra fields ignored). + +## 8. Error model + +Host-side errors are `PluginError { kind, code, message, detail?, recoverable }` +with stable `kind` values: `manifest`, `abi`, `artifact`, `load`, `wire`, +`capability`, `invoke`, `unsupported`. Examples: `abi_major_mismatch`, +`no_matching_artifact`, `hash_mismatch`, `missing_entry_symbol`, +`metadata_name_mismatch`, `export_signature_mismatch`, +`unsupported_wire_version`. `message`/`detail` must never contain secrets. +They are serializable and are used as the wire error body. + +When a plugin is used from Marqdo code, host errors become structured +`RuntimeError`s (`crates/marqdo-host/src/host.rs::map_plugin_error`): + +| Plugin error kind | Runtime code | +|---|---| +| `abi` | `plugin.abi_mismatch` | +| `capability` | `permission.denied` | +| `wire` | `runtime.type` | +| `unsupported` | `kernel.unavailable` | +| `manifest`/`artifact`/`load`/`invoke` | `plugin.failure` | + +## 9. Using a plugin from `.mq.md` + +This build supports explicitly trusted **in-process native** plugins only. + +```bash +# 1. Build the C example and refresh its sha256 in plugin.yaml +./examples/plugin-demo/build.sh + +# 2. Inspect without loading +marqdo plugin inspect examples/plugin-demo + +# 3. Load with an explicit trust decision and a capability grant, then run +marqdo run examples/plugin-demo/plugin-call.mq.md \ + --plugin examples/plugin-demo \ + --trust-native-plugin \ + --allow-plugin demo.solver +``` + +Flags (`run` and `call`): `--plugin ` (repeatable), +`--trust-native-plugin`, `--allow-plugin `. `--allow-plugin` +matches the manifest `plugin.name` (for example `demo.solver`), not the call +namespace (`solver`). + +Document syntax (`examples/plugin-demo/plugin-call.mq.md`): + +```markdown +## main + +**结果 = solver.add(a=20, b=22)** + +> 打印(结果) +``` + +Semantics: + +* the call namespace is derived from the **first** manifest export prefix + (`solver.add` → namespace `solver`); a manifest whose exports use several + prefixes registers only the first namespace, so exports under other prefixes + are not reachable in this build; +* calls accept **named arguments only**; positional arguments are a runtime + type error ("plugin calls require named arguments"); +* each call becomes `HostOp::PluginInvoke` with a typed wire request; the + plugin's typed response becomes a Marqdo value (`Unit` when absent); +* the capability grant is re-checked on every call, not only at load; +* plugin namespace calls are **not statically validated**: `check` passes a + document that calls a namespace no plugin provides; at runtime the call fails + with `symbol.unknown` ("no plugin namespace ... is loaded") unless + `--plugin` loaded it; +* plugin host-capability requests are denied because the CLI installs no + capability handler (`LoadOptions::default()`); +* `--mode strict` refuses to run before any plugin is loaded; `--mode pure` + cannot grant `--allow-plugin`, so loading fails with `permission.denied`. + +Trust boundary: `--trust-native-plugin` is an explicit unsafe decision; the +process shares its address space with the plugin. In-process load is the only +transport implemented — there is no worker process, Wasm component, hot +unload, or plugin capability bridge. A native plugin crash is a host crash. + +## 10. SDK usage (`crates/marqdo-plugin-sdk`) + +The SDK hides the raw C layout and exports the entry symbol with +`mq_plugin_entry!`. A complete Rust plugin crate: + +```rust +use marqdo_plugin_sdk::{mq_plugin_entry, CallContext, PluginError, Value}; + +fn add(_ctx: &mut CallContext, args: &[u8]) -> Result, PluginError> { + let request = marqdo_plugin_sdk::wire::decode_request(args)?; + let values = request.typed_args()?; + let a = marqdo_plugin_sdk::wire::int_arg(&values, "a")?; + let b = marqdo_plugin_sdk::wire::int_arg(&values, "b")?; + marqdo_plugin_sdk::wire::success(Some(Value::Int((a + b).into()))) +} + +mq_plugin_entry! { + name: "demo.solver", + version: "0.1.0", + exports: { + "solver.add" => "($a: Int, $b: Int) -> Int" => add, + } +} +``` + +The macro validates the host table, reports the plugin ABI version and metadata, +registers every export and returns `MQ_OK` only if all registrations succeed. +Handlers receive a `&mut CallContext`; use +`CallContext::capability_request(name, request_bytes)` for an already-authorized +host capability (the CLI grants none today), and +`CallContext::log(level, message)` as a convenience. Panics are caught at the +ABI boundary and converted to a structured error. The SDK depends only on +`marqdo-plugin-abi`, never on host-only crates or the interpreter. + +## 11. C demo (`examples/plugin-demo`) + +`examples/plugin-demo/solver.c` is a dependency-free C implementation of one +export (`solver.add`, `Int + Int`) using a minimal JSON scanner; it validates +the host table, reports metadata and registers the export. `build.sh` compiles +it (`cc -shared -fPIC -I include`) and rewrites the `sha256` in `plugin.yaml` +(the checked-in manifest uses a zero placeholder until you build). The +`examples/plugin-demo/README.md` walks through the security model, build, +inspect and call flow. + +The integration test `crates/marqdo-plugin-abi/tests/loader_unix.rs` compiles +this exact `solver.c` with `cc`, loads it with `libloading`, round-trips a wire +request, and rebuilds it with `-DMQ_SOLVER_REPORT_ABI_MAJOR=2` to prove that a +wrong ABI major fails the load. It also covers a missing entry symbol, missing +artifact, hash mismatch, malformed manifest and target mismatch. The tests skip +when no C compiler is available. + +## 12. Not implemented / explicit limitations + +| Item | Status | +|---|---| +| Isolated plugin worker transport (length-prefixed IPC) | Not implemented; `PlatformCapabilities::worker_processes = false`. | +| Wasm Component + WIT | Not implemented. | +| Hot unload/reload | Not supported by ABI v1 (no shutdown callback). | +| Plugin capability bridge from the CLI | Not implemented; CLI supplies no handler, so all plugin host-capability requests are denied. | +| Plugin installation by package manager | Not implemented; `marqdo.yaml` `plugins:` is declarative only. | +| Static validation of plugin namespaces/exports/signatures against loaded plugins | Not implemented; errors are runtime-only. | +| Multiple namespaces per plugin | Not implemented; single namespace from the first export prefix. | +| Positional plugin arguments | Not supported; named only. | +| Thread-safe / concurrent invocation | Not supported (single-threaded v1 contract). | +| Old `marqdo_abi.h` v2 compatibility | Not implemented and not claimed; a separate adapter would be required. | diff --git a/spec/references.md b/spec/references.md new file mode 100644 index 0000000..71701bf --- /dev/null +++ b/spec/references.md @@ -0,0 +1,362 @@ +# Marqdo References, Imports and Calls v1 + +Status: **frozen for M0.5 except where marked "current limitation".** Language +changes require an ADR plus updated tests. + +Normative implementation: + +| Concern | Code | +|---|---| +| Front matter parsing | `crates/marqdo-markdown/src/frontmatter.rs` | +| Module loading, project root, cycles | `crates/cli/src/loader.rs` | +| Module/Unit/Function identities | `crates/marqdo-ir/src/lib.rs` | +| Heading scopes, anchors | `crates/marqdo-frontend/src/units.rs` | +| Resource collection and attribution | `crates/marqdo-frontend/src/resources.rs` | +| Link-call mapping | `crates/marqdo-frontend/src/map.rs` | +| `ref(...)` runtime resolution | `crates/marqdo-runtime/src/builtins_impl.rs` | +| Static link graph (`inspect --refs`) | `crates/cli/src/commands.rs` | +| Cross-module call validation | `crates/marqdo-frontend/src/calls.rs` | + +This file complements [`mapping.md`](mapping.md) (AST roles) and +[`package-format.md`](package-format.md) (manifest/lock file). + +## 1. Three edge kinds, never conflated + +Blueprint §4.1 distinguishes three relations. The implementation keeps them +separate and **only the call edge can execute code**. + +| Edge | Written as | Resolved when | Executes? | +|---|---|---|---| +| **Import** | YAML front matter `import:` mapping | module load / static analysis | No. Imported modules only register definitions; their `main` never runs. | +| **Reference** | Markdown `[label](url)` | `inspect --refs` (static graph) | No. Rendered/read as documentation only. | +| **Call** | `> f(...)` block quote, link-form block quote `> [f](#f)(...)`, or a `CallExpr` in an executable payload | static validation, then execution | Yes, at runtime, through the interpreter. | + +Consequences: + +* A Markdown link never calls, imports or loads anything. `[计算作用力](#计算作用力)` + is a reference; `> [计算作用力](#计算作用力)()` is a call. +* `ref("#anchor")` is a **runtime builtin**, not an import; it only resolves an + inert resource or function reference in the running module. +* The reference graph is static metadata and creates no proof obligation. + +## 2. Canonical identities + +These types live in `crates/marqdo-ir/src/lib.rs`. + +### 2.1 `ModuleId` + +```rust +pub struct ModuleId(pub String); +``` + +* A canonical **project-root-relative path with `/` separators**, for example + `index.mq.md`, `packages/geometry/index.mq.md`. +* Computed in `crates/cli/src/loader.rs::Loader::module_id_for` by taking the + canonicalized file path, stripping the project root, and joining the path + components with `/`. +* There is no "module name" identity separate from the path: the module id is + the file location inside the project. Imported `@dependency` modules keep + their project-root-relative path (for example + `packages/geometry/index.mq.md`). +* A file that canonicalizes outside the project root is rejected with + "outside the project root" (entry: fatal load error; imported module: + `module.unresolved` diagnostic). +* Module ids use `BTreeMap` ordering, so program iteration and JSON output are + deterministic. + +### 2.2 `UnitId` + +```rust +pub struct UnitId { pub module: ModuleId, pub path: Vec } +``` + +* One addressable heading range. `path` is the canonical heading path from the + outermost heading to this unit (root unit has an empty path). +* Canonical heading names are NFC-normalized, trimmed, with internal + whitespace collapsed to one space (`marqdo_markdown::normalize_name`). +* `Display` renders `#`; the root unit renders as + `#`. +* `UnitKind` is `root`, `namespace` (`#`), `function` (`##`) or `section` + (`###`+). Sections are addressable but not callable. + +### 2.3 `FunctionId` + +```rust +pub struct FunctionId { pub module: ModuleId, pub name: String } +``` + +* `name` is the **namespace-qualified canonical function name**: `ns.fn` when + the level-2 heading is inside a level-1 namespace, otherwise `fn` + (`crates/marqdo-frontend/src/units.rs`). `Display` is `::`. +* A function heading under a namespace yields both `FunctionId.name` (`ns.fn`) + and `Function.local_name` (`fn`) plus `namespace: Some("ns")`. +* Lookup (`Module::function`) first matches `id.name`, then a unique + `local_name`; if several namespaces define the same local name the lookup + returns `None` rather than guessing. +* No overloads: duplicate canonical names in one scope are hard errors + (`markdown.invalid_construct`, "duplicate heading name ... in the same + scope") and are never auto-suffixed. + +### 2.4 Anchors are a separate namespace, not identity + +`UnitIndex.anchors: BTreeMap` maps GitHub-style heading slugs +to units (`crates/marqdo-frontend/src/units.rs::github_slug`): + +* digits/letters/`_` are kept and lowercased; spaces, `-` and tabs become `-`; + all other punctuation is dropped; +* collisions receive `-1`, `-2`, ... suffixes in document order (separate + counters per base slug); +* anchors exist for navigation and `ref("#...")`; they are **not** identity. + Internal ids stay canonical, so a heading rename is a link-breaking change + but never changes the canonical unit path. + +## 3. Front matter imports + +Imports exist only as an explicit YAML mapping in the document's front matter +(recognized by the Markdown profile only at the start of the file): + +```markdown +--- +import: + geometry: "@geometry" + util: "./lib/util.mq.md" +--- + +# main +``` + +Parsing rules (`crates/marqdo-markdown/src/frontmatter.rs::imports`): + +* the `import` key must be a mapping of **string alias → string target**; +* aliases must be non-empty and unique within the document; +* any other front matter key is metadata for tools and is ignored by the + interpreter; invalid YAML is a `frontmatter.invalid` diagnostic, never + silently dropped; +* imports are never inferred: a link, a file name or a directory scan does not + create an import; +* the diagnostic span recorded for all imports of a file is the whole front + matter span (`ImportDecl.span`). + +### 3.1 Target forms + +`Loader::resolve_import` recognizes exactly two forms: + +| Target | Resolution | +|---|---| +| `@dependency` | Look up the alias after `@` in `marqdo.yaml` `dependencies`. The dependency must declare a **path** source. Resolve `project_root / path`; if it is a directory, append `index.mq.md`; canonicalize. A git-only dependency cannot be resolved in this build (`dependency.path` is `None` → unresolved). | +| any other string (intended: `./file.mq.md`) | Join the string to the **importing file's directory**, then canonicalize. There is no extension inference, no search path and no fallback to a package directory. | + +The resulting file is loaded as a module. If it canonicalizes outside the +project root, the loader reports "outside the project root" and the import +becomes a `module.unresolved` diagnostic. + +### 3.2 Loader rules + +`crates/cli/src/loader.rs` implements a static, non-executing load: + +1. **Project root discovery** (`marqdo_package::discover_root`): walk up from + the entry file (starting at its parent directory) to the nearest directory + containing `marqdo.yaml`; if none is found, the entry file's parent + directory is the root. There is no `--project` flag in this build and nested + project boundaries are not diagnosed — the nearest manifest wins (open + item). +2. **Manifest** is optional at run time; if a `marqdo.yaml` exists in the root + it is parsed and validated. An invalid manifest is a fatal load error + (exit code 2), not a diagnostic list. +3. **Entry module** is loaded first; `Program.main_module` is set to its + module id. +4. **Imports** are processed depth-first in front matter order. Each import + target is resolved as in §3.1, then loaded recursively. +5. **Cycle detection** uses the path stack: if the canonical target path is + already being loaded, the importing file gets a `module.cycle` diagnostic + ("import ... creates an import cycle") and the import edge is skipped. + Modules are otherwise loaded once; a second import of an already-loaded + module reuses it. +6. **Definitions only**: loading parses, indexes and analyzes source. No + function is called, no `main` is run, no foreign/plugin code is touched. +7. After loading, `calls::validate_program` validates cross-module and + qualified-builtin calls and attaches diagnostics to the emitting file. +8. Diagnostics are collected in deterministic (load) order; the CLI renders + them on stderr (or as JSON). + +Failure mapping: a failure to load the entry file or read the manifest is a +**fatal load error** (`exit 2`, `error: ...` on stderr). A failure to resolve +an imported module becomes a `module.unresolved` **diagnostic** on the +importing file and `check`/`run` then exit 1 (run refuses to execute a project +with compile errors). + +## 4. Markdown reference edges (`inspect --refs`) + +`marqdo inspect --refs` walks every `link` node in the entry document +and prints/writes an edge array (`crates/cli/src/commands.rs::refs_json`, +`resolve_link`). Each edge has: + +```json +{ + "source": "references.mq.md", + "url": "./math.mq.md#计算圆面积", + "span": {"start": 54, "end": 95}, + "resolved": true, + "target_module": "math.mq.md", + "target_unit": "math.mq.md#数学/计算圆面积" +} +``` + +Resolution rules: + +1. `#fragment` → look up the entry module's anchors, then fall back to a unit + whose canonical **name** equals the fragment. +2. `http://` / `https://` URL → counted as resolved with no local target. + External links are never downloaded, imported or executed. +3. anything else → split optional `#fragment`, canonicalize the path relative + to the **entry file's directory**, require the target to be inside the + project root, derive its `ModuleId`, and look up anchors/names in that + module. If the target module is not imported, its source is **read and + analyzed statically on demand** (`analyze_source`) so broken anchors can be + detected without execution. +4. `resolved: false` means the file or anchor could not be resolved. + +Current limitations (documented, not hidden): + +* `check --refs` validates every loaded document's link nodes and reports + broken same-document anchors, missing files and missing target anchors as + `ref.unresolved` errors, changing the command's exit status. Plain `check` + (without `--refs`) does not treat reference problems as errors, because links + are static knowledge references rather than execution inputs. +* `inspect --refs` prints the edge graph for the entry document only; other + loaded modules are still covered by `check --refs`. +* The graph is not emitted for call edges or import edges; imports are visible + through module counts/`inspect` summaries, calls through `inspect --ir`. + +## 5. `ref("#anchor")` runtime semantics + +`ref` is a **pure builtin** (`crates/marqdo-ir/src/builtins.rs`, +`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref`). It never evaluates +the referenced resource and requires no capability. + +Signature: `ref(target: Text) -> Formula | Code | FunctionRef`. + +Algorithm: + +1. `target` must start with `#`; otherwise `symbol.unknown` with the message + "ref target ... must be a `#heading` anchor". It is not a URL resolver. +2. The fragment is looked up **only in the running module's anchor map** + (`Module::anchors`). In this build `ref` does not fall back to canonical + unit names and does not search imported modules; a fragment that is not a + GitHub slug yields `symbol.unknown` ("no heading matches ..."). +3. The matching unit's resources (`Module::resources_of`, computed from the + unit's heading range) are examined: + * **exactly one resource** → `Formula { latex, display, unit }` or + `Code { language, source, content_hash, unit }`. `unit` is the + `UnitId::Display` string. + * **zero resources** → if a function's unit equals the resolved unit, return + a `FunctionRef`; otherwise `symbol.unknown`. + * **more than one resource → `ref.ambiguous`** with the message + "`#...` contains N resources; select one explicitly instead of relying on + the first". The runtime **never** picks the first resource. This is + blueprint issue S-07 and is frozen by ADR-0003. +4. Resource attribution (`resources.rs`): a formula or code fence belongs to + the **innermost enclosing heading unit** by source span; resources are kept + in document order. Markdown `math`/`math_block` and `code_block` nodes + become resources; inline `code` spans do not. + +Example (from `examples/math.mq.md`): + +```markdown +## 圆面积公式 + +$$ +A = \pi r^2 +$$ + +## 计算圆面积 + +**面积 = math.eval(ref("#圆面积公式"), {"r": 4}, target="A")** + +*面积* +``` + +`ref("#圆面积公式")` resolves the single formula under that heading; the +formula is still inert until `math.eval` consumes it. + +### 5.1 Error model of `ref` + +| Situation | Runtime code | +|---|---| +| target not starting with `#` | `symbol.unknown` | +| no anchor/unit matches | `symbol.unknown` | +| unit has no formula/code/function | `symbol.unknown` | +| unit has ≥ 2 resources | `ref.ambiguous` | +| module not present in the program | `symbol.unknown` | + +`ref.unresolved` (`crates/marqdo-value/src/diagnostic.rs`) is a **compile-time** +code used by link-form call mapping, not by the `ref` builtin. + +## 6. Call edges + +### 6.1 Expression calls + +Inside a function body, `CallExpr` nodes call: + +* a local function (same module, namespace-aware resolution: current namespace + first, then canonical name, then a unique local name); +* an imported function through its alias: `geometry.圆面积(r=2)`; +* a builtin (`打印`, `math.eval`, ...) by canonical name or alias; +* a `FunctionRef` value obtained earlier (for example from `ref`). + +Static validation (`crates/marqdo-frontend/src/calls.rs`) checks arity, +argument names and the named-only rule for inferred signatures. Type errors are +runtime concerns in v1. + +### 6.2 Block-quote calls + +`> f(...)` is a call statement; the payload must parse as a complete +`CallExpr` (see `mapping.md` §4.5). Its result is discarded. + +Link form `> [label](#anchor)(args)` resolves the link target **within the +current module** using the anchor map, then the canonical unit-name fallback, +and requires the target to be a function unit. Notes: + +* a zero-argument link call must still write `()`; proxy `> [f](#f)` is a hard + error; +* `http(s)` link targets used as calls are a hard `ref.unresolved` error; + external links are never downloaded or called; +* a relative-path link target such as `> [f](./other.mq.md#f)(x=1)` is + **not supported** in this build: mapping reports `module.unresolved` with + the hint to import the module in front matter and call through its alias. + Cross-module calls therefore go through an alias expression + (`geometry.f(...)`), not through a Markdown link. Blueprint §2.4's + `> [计算面积](./math.mq.md#计算面积)(r=3)` example is a **known divergence**. + +### 6.3 Imported alias calls + +At runtime an alias call resolves via `Program::resolve_import` to the target +module id; an unresolved alias is `symbol.unknown`. The static validator +checks the target module exists and the function/arity match when the whole +program is loaded, and emits `symbol.unknown` for an unknown function. + +## 7. Diagnostics for references/imports + +| Code | Severity | Where produced | +|---|---|---| +| `module.unresolved` | error | loader (bad import target), call mapping (relative-path link call) | +| `module.cycle` | error | loader | +| `ref.unresolved` | error | link-form call target mapping, external links used as calls, `check --refs` link validation | +| `ref.ambiguous` | runtime error | `ref` builtin (≥ 2 resources) | +| `symbol.unknown` | error / runtime error | static call validation; `ref` anchor miss at runtime | +| `frontmatter.invalid` | error | analyser (bad YAML/import shape) | +| `markdown.invalid_construct` | error | empty/duplicate heading names, invalid link-call shape | + +## 8. Current limitations (open ADR candidates) + +1. `ref("#...")` is module-local and has no cross-module form; there is no + documented syntax for referencing a resource in an imported module. +2. Ordinary Markdown link resolution is inspection-only; `check` does not + validate links and there is no `--refs` flag on `check`. +3. Link-form calls cannot cross modules (see §6.2). +4. Headings inside container blocks or fenced code never create units, and a + heading whose name is empty is a hard error rather than an ignored heading. +5. Anchor lookup in `ref` does not fall back to canonical unit names, while + link-form calls and `inspect --refs` do. This asymmetry is intentional for + the frozen build but should be unified by a future ADR (it is easy to get + surprising results with headings containing punctuation). diff --git a/spec/security.md b/spec/security.md new file mode 100644 index 0000000..6991a08 --- /dev/null +++ b/spec/security.md @@ -0,0 +1,276 @@ +# Marqdo Security Model (implemented status) + +Status: **describes the M0.5 implementation, including honest gaps.** This is +not a security audit; it documents what the code actually enforces. + +Normative implementation: + +| Concern | Code | +|---|---| +| Sandbox modes, strict refusal | `crates/marqdo-sandbox/src/lib.rs` | +| Capability sets and checks | `crates/marqdo-host/src/capability.rs` | +| Host dispatch and per-op checks | `crates/marqdo-host/src/host.rs` | +| Project-rooted filesystem | `crates/marqdo-host/src/fs.rs` | +| Budgets | `crates/marqdo-host/src/budget.rs` | +| Effect tracing | `crates/marqdo-host/src/trace.rs` | +| Math/foreign adapters | `crates/marqdo-host/src/kernels.rs` | +| Foreign child processes | `crates/marqdo-foreign/src/lib.rs` | +| Plugin loading / in-process plugin calls | `crates/marqdo-host/src/host.rs` (`load_plugin`, `op_plugin_invoke`), `crates/marqdo-plugin-abi/src/loader.rs` | +| CLI flags and capability construction | `crates/cli/src/main.rs`, `crates/cli/src/commands.rs` | + +## 1. Modes + +`SandboxMode` has three values (`pure`, `project`, `strict`), parsed by +`SandboxMode::parse` and selectable on `run`/`call` with +`--mode ` (default: `project`). + +| Mode | Intended guarantee | Actual status in this build | +|---|---|---| +| `pure` | No external I/O; memory computation and budgets only. | **Enforced.** `Capabilities::pure()` = `console`, `math` only; every FS/foreign/plugin grant flag makes the CLI exit 2 with "pure mode prohibits external I/O grants". | +| `project` | Operator-approved project resources under `CapabilityHost` + bounded resources. | **Enforced for the implemented operations.** Defaults equal `pure`; `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin` add grants. Sandboxing is cooperative (in-process checks), not OS isolation. | +| `strict` | Untrusted documents/code/plugins under real OS isolation. | **Refused.** `PlatformCapabilities::detect()` always reports `strict: Unavailable` in this build because no OS isolation adapter (namespaces/seccomp/cgroups, Job Object, macOS sandbox) is implemented, and `worker_processes: false`. `marqdo_sandbox::enforce` returns `StrictUnavailable`; `run`/`call` print the reason and exit 1. There is **no fallback** to an unrestricted process. | + +`marqdo inspect --platform` prints the detection report, for example: + +```json +{ + "os": "linux", + "arch": "x86_64", + "strict": {"state": "unavailable", + "reason": "no OS isolation adapter (namespaces/seccomp/cgroups) is implemented in this build"}, + "worker_processes": false +} +``` + +Design rule (blueprint §9): a cooperative in-process check is not a sandbox. +Strict mode will remain refused until a platform isolation adapter exists; see +ADR-0004. + +## 2. Capability grants + +`Capabilities` (`crates/marqdo-host/src/capability.rs`) is the effective grant +set for one run: + +| Field | Default (`pure`/`project`) | Used by | +|---|---|---| +| `console: bool` | `true` | currently **not checked** by `op_print`; print is always allowed and only bounded by the output budget (open item) | +| `math: bool` | `true` | `math.*` host ops | +| `fs_read: Vec` | empty | `fs.read` | +| `fs_write: Vec` | empty | `fs.write` | +| `net_hosts: Vec` | empty | `Capabilities::check_net` exists, but no host op or builtin performs network access in this build | +| `process: bool` | `false` | required together with `foreign` for `foreign.run`/`foreign.call` | +| `foreign: bool` | `false` | foreign kernels | +| `plugins: Vec` | empty | `plugin.invoke` and `StdHost::load_plugin` | + +Requests versus grants: + +* `marqdo.yaml` `capabilities:` is a **request**. It is parsed and validated + (`crates/marqdo-package/src/lib.rs`) but is not consulted when building the + host capability set: only CLI flags/grant APIs produce grants. Declaring a + capability never grants it. +* `marqdo.yaml` `plugins:` likewise declares requirements only; it does not + install, load or authorize a plugin. +* CLI grant flags (`crates/cli/src/main.rs::SandboxArgs`): + * `--allow-read ` and `--allow-write ` (repeatable, project mode + only); + * `--allow-foreign` (project mode only; sets `foreign = true` and + `process = true`); + * `--allow-plugin ` (repeatable, project mode only; matches the + manifest `plugin.name`, not the call namespace); + * `--plugin ` (repeatable, loads a plugin after the trust check); + * `--trust-native-plugin` (explicit in-process native trust decision). +* There is no network grant flag and no plugin-worker flag in this build. + +### 2.1 Filesystem grants are project-relative globs + +`fs.read`/`fs.write` paths go through `ProjectFs::resolve` **before** the +capability check: + +* empty paths, absolute paths, `..` components and root/prefix components are + rejected with `permission.denied`; +* `.` components are dropped; the normalized relative string uses `/`; +* the capability patterns are matched with the `glob` crate against that + normalized relative path; a leading `./` is stripped on both sides, so + `--allow-read ./data/**` matches `data/a.csv` and does not match + `secrets/key`; `--allow-write output/**` does not grant reads; +* an invalid glob pattern is treated as non-matching (`Pattern::new(...). + unwrap_or(false)`), so it silently grants nothing (there is no diagnostic). + +Additional FS limits (`crates/marqdo-host/src/fs.rs`): + +* reads are capped at `MAX_READ_BYTES = 8 MiB`, must be regular files, must be + UTF-8, and the canonicalized path must stay inside the project root; +* writes create parent directories; before writing, the **closest existing + ancestor** is canonicalized and must stay inside the project root. + +Known caveat: for writes, only the closest existing ancestor is checked and the +final write uses the non-canonical path. A symlink created between the check +and the write (TOCTOU) could escape the root. Reads canonicalize the final path +before reading. Full symlink/race hardening is an open P4 item. + +### 2.2 Checks happen at execution points, not only at parse time + +`Host::invoke` is the only path to effects. `StdHost::dispatch` routes every +`HostOp` and the checks run immediately before the effect: + +* `fs.read` → `capabilities.check_read(path)` then `ProjectFs::read`; +* `fs.write` → `check_write` then `ProjectFs::write`; +* math ops → `capabilities.math` (in `crates/marqdo-host/src/kernels.rs`); +* foreign ops → `capabilities.check_foreign` (requires `foreign && process`); +* plugin invoke → the namespace must be loaded, then + `capabilities.check_plugin(package)` on **every call**; loading also checks + the same grant once; +* print → output budget only. + +Static analysis may warn, but it never replaces these runtime checks. There is +no host bypass for tests: tests use the same `StdHost`/`Capabilities` types. + +## 3. Resource budgets + +`Budget` (`crates/marqdo-host/src/budget.rs`) is per run: + +| Limit | Default | Enforcement | +|---|---|---| +| evaluation fuel | `DEFAULT_FUEL = 5_000_000` | 1 unit per statement, 1 per expression node, 1 per branch condition, 1 per loop iteration; exhaustion → `resource.exhausted` | +| call stack depth | `DEFAULT_STACK_LIMIT = 256` | checked before every function call → `resource.exhausted` | +| program output bytes | `DEFAULT_OUTPUT_BYTES = 1 MiB` | `print` output and `fs.read` contents are charged; overflow → `resource.exhausted` | +| wall clock | none by default | `--timeout ` sets a deadline; checked between statements/iterations and before loop conditions → `resource.exhausted`. It does **not** preempt a running host call. | + +CLI overrides (`run`/`call`): `--fuel`, `--stack`, `--max-output`, +`--timeout`. Additional fixed limits: + +* foreign kernel: `StdHost.foreign_timeout = 10s` and + `foreign_output_limit = 1 MiB` (stdout+stderr combined) per call. + These are not configurable from the CLI in this build; +* plugin wire: request ≤ 16 MiB, result ≤ 64 MiB, error ≤ 1 MiB, metadata + ≤ 64 KiB (see `include/marqdo_plugin_abi_v1.h`); +* math kernel: 1 MB source, 256 parse depth, 512 AST nodes, exponent ≤ 10,000, + exact-result ≤ 1,000,000 bits (`crates/marqdo-math`). + +These are cooperative budgets in the interpreter and the supervisor. Only +foreign child processes are terminated by the host (kill on timeout/output +overflow). In-process math and native plugins are not OS-limited. + +## 4. Effect tracing + +`EffectTrace` records one entry per `Host::invoke` when enabled +(`--trace`, or implicitly `--json` for `run`/`call`). Entry fields: + +```json +{"op":"print","module":"hello.mq.md","span":{"start":43,"end":57}, + "duration_micros":3,"result":"ok"} +``` + +* `op` is the stable `HostOp::name()` (`print`, `math.eval`, `foreign.call`, + `fs.write`, `plugin.invoke`, ...); +* `module` is the initiating module id; `span` is the call site when known; +* `result` is `"ok"` or the failure code (for example + `permission.denied`); +* argument values and file contents are **not** recorded, so ordinary secrets + do not leak into traces by design. Paths are also not recorded today. +* In text mode trace entries are printed to stderr (`trace: ...`); in JSON + mode they are a `trace` array next to the program stdout string. Program + output is always separate from diagnostics/trace (stdout vs stderr). + +## 5. Foreign kernels: process isolation, not a sandbox + +`crates/marqdo-foreign/src/lib.rs` runs Python/Node one-shot: + +* the program is written to a private temp file (mode 0600 on Unix), the child + runs with a fresh private temp directory (mode 0700) as cwd, and both are + removed on drop; +* direct `argv` exec (no shell, no interpolation); stdin carries the versioned + request envelope; stdout/stderr are always piped and captured; +* the environment is cleared except `PATH`; +* the wall-clock deadline is enforced by polling `try_wait` and killing the + direct child; output flooding kills the child once the combined byte budget + is exceeded; a 2-second reader grace prevents a grandchild holding the pipes + from hanging the host; +* a non-zero exit is data (`RunStatus::Failed`, stderr preserved); a timeout is + `TimedOut` for `run` and an error for the typed `call`. + +What this is **not**: an OS sandbox. The child runs with the host user's +privileges, can open absolute paths, use the network and spawn further +processes. Clearing the environment and changing cwd are hygiene, not +confinement. Grandchildren are not killed when the direct child is killed. +Strict mode refuses to run at all, so foreign execution is only reachable in +`project` mode with `--allow-foreign`. + +## 6. Plugins: in-process native trust boundary + +* The only plugin transport implemented in this build is **in-process native + loading** (`libloading`) after `manifest.validate()`, exact target-triple + artifact resolution, sha256 verification, ABI negotiation and export + metadata cross-checking (`crates/marqdo-plugin-abi/src/loader.rs`). +* Loading requires **three** independent things: + 1. `--plugin ` names the plugin; + 2. `--trust-native-plugin` is an explicit unsafe trust decision; + 3. the manifest package name is granted with `--allow-plugin ` (checked + again on every invocation). + Without trust, `StdHost::load_plugin` returns `permission.denied` with the + hint that no isolated worker transport exists. There is no isolated worker, + Wasm component or out-of-process plugin path in this build. +* A native plugin runs in the host address space. It can read host memory, + call arbitrary syscalls, bypass the capability model entirely and crash or + corrupt the host process. **A C ABI is not a sandbox.** `--trust-native-plugin` + must be treated as equivalent to running the plugin's code as the host user. +* Plugin call sites are written as `.(...)` in a function + body (for example `solver.add(a=20, b=22)`). The namespace is derived from + the **first** manifest export prefix. Calls are made through + `HostOp::PluginInvoke` with a typed `wire_version` 1 request; the plugin's + typed response becomes a Marqdo value (`Unit` when absent). Named arguments + are required in this build; positional plugin arguments are `runtime.type`. +* Plugin errors map to structured runtime errors: ABI → `plugin.abi_mismatch`, + capability → `permission.denied`, wire/value → `runtime.type`, unsupported → + `kernel.unavailable`, everything else → `plugin.failure`. An unloaded + namespace is `symbol.unknown`. +* The plugin's host-capability requests are denied by default: the CLI loads + plugins with `LoadOptions::default()`, i.e. **no capability handler**, so + every `request_capability` call fails. The `requires:` list in `plugin.yaml` + is only a declaration. +* `marqdo plugin inspect` never loads code: it validates the manifest and + verifies the artifact hash only. +* Installation, inspection, loading and authorization are four separate + decisions. Copying a plugin into the project or declaring it in + `marqdo.yaml` grants nothing. + +Known limitations: single namespace per plugin (derived from the first export), +no hot unload/reload, no plugin capability bridge from this CLI, native crash +is a host crash, `plugin.invoke` cannot be used in `pure`/`strict` mode. + +## 7. `check` and `inspect` are side-effect free + +Neither command executes project code. Specifically: + +* `check` parses, indexes, maps to IR, runs static call validation and reports + diagnostics. It never calls a function, never spawns a foreign kernel, never + loads a native plugin and never writes files. +* `inspect` with `--ast`, `--units`, `--signature`, `--ir`, `--refs` or + `--platform` does the same static work and prints metadata. `--refs` may + **read** other project source files on demand to resolve link targets, but + only parses/analyzes them; nothing is executed or written. +* `plugin inspect` reads `plugin.yaml`, validates it, resolves the artifact for + the host target and computes/compares its sha256. It does not `dlopen`, + instantiate or call the plugin (`"loaded": false` in JSON output). +* `run`/`call` first perform the same static load; if any diagnostic has error + severity, execution is refused before the host or any kernel is touched. +* `fmt` only normalizes line endings/trailing newlines and writes the document + when not in `--check` mode; it does not execute anything. + +## 8. Open security items + +1. Strict mode: no OS isolation adapter (P4); worker processes report + `false`. +2. `console` capability is not enforced; print is always allowed. +3. Network capability exists in the model but no host operation uses it; there + is no `--allow-net` flag. +4. Write-path symlink TOCTOU (see §2.1). +5. Plugin worker isolation and a plugin capability bridge are unimplemented; + only trusted in-process native plugins are possible. +6. `plugin_handle` wire values are structurally reconstructable (see + `value-wire.md` §4.1); no unforgeable capability token. +7. Budgets do not cover all allocation paths (for example list/table growth + inside the interpreter is bounded only by fuel and memory). +8. `--json` run output includes the full program stdout string; that is the + program's own output, not a host secret, but it is embedded in the JSON + document consumers must parse. diff --git a/spec/value-wire.md b/spec/value-wire.md new file mode 100644 index 0000000..990d165 --- /dev/null +++ b/spec/value-wire.md @@ -0,0 +1,161 @@ +# Marqdo Value Wire Format v1 + +Status: **frozen for M0.5**. Changes require an ADR, a `wire_version` bump and +cross-ABI compatibility tests. + +Canonical implementation: `crates/marqdo-value/src/wire.rs` +(`to_wire`, `from_wire`, `from_wire_with_limits`, `WireLimits`, `WireError`). +The plugin envelope that *carries* these values is defined in +`crates/marqdo-plugin-abi/src/wire.rs` and documented in +[`plugin-abi.md`](plugin-abi.md); the foreign-kernel stdin envelope is +documented in [`foreign.md`](foreign.md) and implemented in +`crates/marqdo-foreign/src/lib.rs`. + +## 1. Why a typed wire format + +JSON has one numeric type. Marqdo has arbitrary-precision `Int`, IEEE-754 +`Float` (including `NaN` and both infinities) and 13 distinct runtime value +types (`crates/marqdo-value/src/value.rs`). Encoding an `Int` as a JSON number +would silently lose digits above 2^53 and would make `Int(1)` and +`Float(1.0)` indistinguishable after a round trip. The wire format is +therefore **fully type-tagged** and is not plain JSON. + +The rules that follow from this: + +1. Every value is a JSON **object** with a mandatory `"type"` string. +2. **Integers never travel as JSON numbers.** `Int` is + `{"type":"int","decimal":""}`, which preserves every digit, + sign and leading-zero-free canonical form. +3. `Float` uses a JSON number only for finite values; `NaN`, `Infinity` and + `-Infinity` use an explicit `"special"` tag. +4. Unknown tags, missing fields, wrong JSON types and invalid payloads are + structured decode errors, never silent coercions. + +JSON numbers still appear in two non-value positions: `plugin_handle.id` (a +host generation counter) and `span.start` / `span.end` (byte offsets). Neither +is a user arithmetic value. + +## 2. Version and entry points + +* `WIRE_VERSION: u32 = 1` (`marqdo_value::WIRE_VERSION`). +* `to_wire(&Value) -> serde_json::Value` encodes. +* `from_wire(&Json) -> Result` decodes with + `WireLimits::default()`. +* `from_wire_with_limits(&Json, WireLimits) -> Result` + decodes with explicit limits. +* Decoding is recursive and never panics on untrusted input; every failure is + a `WireError`: + * `UnknownType(String)` — missing/non-string/unknown `"type"`; + * `MissingField(&'static str)` — required field absent; + * `Invalid(String)` — wrong JSON type, invalid decimal, bad table shape, + unknown float `special`, non-object value; + * `LimitExceeded(&'static str)` — nesting depth, list/map length, string or + map-key length. + +`WireError` implements `Display` but has no stable numeric code of its own. +Boundary crates wrap it in their own stable codes: the plugin layer uses +`invalid_argument_wire` / `invalid_result_wire` +(`crates/marqdo-plugin-abi/src/wire.rs`), and the foreign layer maps it to +`Protocol` (`crates/marqdo-foreign/src/lib.rs`). + +## 3. Value encodings + +The table is normative and mirrors `encode`/`decode` in +`crates/marqdo-value/src/wire.rs`. + +| Marqdo type | Wire JSON | Notes | +|---|---|---| +| `Unit` | `{"type":"unit"}` | No result; distinct from `None`. | +| `None` | `{"type":"none"}` | Explicit empty value. | +| `Bool` | `{"type":"bool","value":true}` | `value` must be a JSON boolean. | +| `Int` | `{"type":"int","decimal":"-123..."}` | Decimal string; decoded with `num_bigint::BigInt::from_str`. No `+`, no separators, no hex. `-0` decodes to `0` because `BigInt` has no signed zero. | +| `Float` (finite) | `{"type":"float","value":1.5}` | JSON number decoded with `as_f64()`. `-0.0` round-trips as a JSON negative-zero number and decodes to `Float(-0.0)`. | +| `Float` (`NaN`) | `{"type":"float","special":"nan"}` | Never a JSON number. | +| `Float` (`+∞`) | `{"type":"float","special":"infinity"}` | | +| `Float` (`−∞`) | `{"type":"float","special":"-infinity"}` | | +| `Text` | `{"type":"text","value":"..."}` | UTF-8 JSON string. | +| `List` | `{"type":"list","items":[, ...]}` | Order preserved. | +| `Map` | `{"type":"map","entries":{"key":, ...}}` | Decoding rebuilds a `BTreeMap`, so key order is canonical (sorted) after a round trip. Duplicate keys cannot occur in a JSON object. | +| `Table` | `{"type":"table","header":["a","b"],"rows":[[,], ...]}` | Header cells must be JSON strings; every row must have exactly `header.len()` cells or decoding fails. | +| `Formula` | `{"type":"formula","latex":"...","display":false}` plus optional `"unit"` (string) and `"span":{"start":N,"end":M}` | `display` defaults to `false` on decode when absent. The formula is inert data; this encoding never evaluates it. | +| `Code` | `{"type":"code","language":"python","source":"...","content_hash":""}` plus optional `"unit"` and `"span"` | `language` and `content_hash` default to `""` on decode when absent; `source` is required. | +| `FunctionRef` | `{"type":"function_ref","module":"index.mq.md","name":"main"}` | Both fields required strings. | +| `PluginHandle` | `{"type":"plugin_handle","plugin":"demo.solver","export":"solver.add","id":1}` | `id` must be a non-negative JSON integer (`as_u64`). | + +Optional fields (`unit`, `span`, `display`, `language`, `content_hash`) are +emitted only when present on the value. Encoders always emit `display` for +`Formula` and `language`/`content_hash`/`source` for `Code`. + +### 3.1 Round-trip guarantees + +For the values covered by `wire::tests::scalar_roundtrip`, +`from_wire(to_wire(v)) == v`. Exact guarantees: + +* all `Int` digits survive (`big_integers_keep_all_digits`); +* `NaN`/`±Infinity` survive with their sign/tag (`special_floats_are_tagged`); +* `Unit` and `None` never collapse into each other; +* `Map` iteration order after decoding is sorted, which is the canonical order + used by `Value::display`/`repr`. + +Finite `Float` values are encoded as JSON numbers. The workspace enables +`serde_json`'s `float_roundtrip` feature, so a finite binary64 value survives an +encode/decode cycle exactly (verified by the Python `math.pi * 9` semantic +fixture: `28.274333882308138` stays `28.274333882308138`). `NaN`/infinities are +tagged and exact. Use `Text` when a decimal string must be exchanged verbatim +rather than as a binary64 float. + +## 4. Limits + +`WireLimits::default()` (`crates/marqdo-value/src/wire.rs`): + +| Limit | Default | Applies to | +|---|---|---| +| `max_depth` | 64 | Nesting of *values*: the top-level object is depth 0, every child decoded through `list.items`, `map.entries`, `table.rows` or `table.header` cells is depth+1. A value at depth > 64 is rejected with `LimitExceeded("nesting depth")`. | +| `max_items` | 1,000,000 | `list.items` length and `map.entries` key count. | +| `max_string_bytes` | 16 MiB | `text.value` byte length and map key byte length. | + +### 4.1 Known gaps in the current decoder + +The following are **not** bounded by `WireLimits` in +`crates/marqdo-value/src/wire.rs` today and must be treated as open items +(they are also bounded indirectly at the ABI layer by envelope size limits): + +* `table.header` length, `table.rows` length and the number of columns are not + checked against `max_items` (only row/header width equality is checked); +* table header cell strings are not checked against `max_string_bytes`; +* `formula.latex`, `code.source`, `function_ref.module`/`name`, + `plugin_handle.plugin`/`export` are not checked against + `max_string_bytes`; +* a `plugin_handle` decoded from wire data is a structurally reconstructed + `Value::PluginHandle`; this build does not carry an unforgeable capability + token. Host code must therefore treat a plugin handle as opaque *intent* and + re-authorize any operation, not as proof of prior authorization. The header + comment "cannot be forged from wire data" is aspirational for this point and + should be fixed or reworded in a future ADR. +* JSON itself may contain duplicate object keys; `serde_json` keeps the last + one, so `map.entries` with duplicate keys is not rejected as an error. + +## 5. Where the format is used + +| Boundary | Envelope | Value positions | +|---|---|---| +| Plugin ABI v1 request/response | `{"wire_version":1,"kind":"request"/"response",...}` (`crates/marqdo-plugin-abi/src/wire.rs`) | `request.args` is an object of name → typed value; `response.value` is one typed value. | +| Plugin host capability request/reply | opaque bytes, shape per capability | defined by the capability, not by this file. | +| Foreign kernel stdin | `{"wire_version":1,"kind":"request","input":}` (`crates/marqdo-foreign/src/lib.rs`) | `input` is a `to_wire(value)` result, or JSON `null` when absent. | +| Foreign typed return (`foreign.call`) | stdout must be exactly one typed value | decoded with `from_wire` and default limits. | + +The `--json` output of `run`/`call` embeds `"value": to_wire(outcome.value)` +(`crates/cli/src/commands.rs`), so CLI JSON consumers see the same typed +representation. + +## 6. Compatibility rules + +* `wire_version` is part of the frozen contract. A decoder rejects a different + version rather than guessing. +* Adding a new `"type"` tag requires a spec change and an ADR; removing or + renaming a tag is a breaking change. +* The plugin envelope independently carries `wire_version`; do not assume the + C ABI major version and `wire_version` are the same number + (see `spec/plugin-abi.md` §2). +* All payloads are UTF-8 JSON. Endianness is irrelevant for the JSON layer, + but the C ABI structs that carry these bytes are target-dependent.