docs: formal specs, ADRs, STATUS matrix, contributor guide and honest README

Complete the documentation set for the M0.5 toolchain, grounded in the code:

- spec/references.md: import/reference/call edges, identities, loader rules,
  anchors, `ref` resolution/ambiguity, `check --refs` validation.
- spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal
  string rule, special floats, limits and exact float round-trip.
- spec/security.md: modes (strict refuses without OS isolation), capabilities,
  path rules, budgets, tracing, foreign process caveats, plugin trust.
- spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync,
  unsupported registry/git/environment work.
- spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact
  rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the
  implemented CLI plugin-invocation flow with its limitations.
- spec/math.md and spec/foreign.md: the implemented math subset and the
  foreign request/response/timeout/capability contract.
- docs/adr/0001..0007: accepted decisions for the span foundation, inert
  executables outside function bodies, ref ambiguity (S-07), strict refusal,
  AST normalization, CLI contract and BigInt/typed wire (S-08).
- docs/contributor-guide.md: repo map, workflow, fixture/golden updates,
  quality gate, ADR/commit policy.
- docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18
  Alpha checklist (conclusion: not yet Alpha-complete), including known
  divergences rather than overclaims.
- README.md rewritten from "design-only starter" to the working implementation
  with quickstart, implemented phases, deliberate limits and doc index.
- AGENTS.md: updated pointers/workflow only; architectural rules unchanged.
- Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside
  functions, unsupported math.simplify/differentiate).
This commit is contained in:
cflmy committed 2026-11-20 10:56:26 +08:00
1 parent fae0cd4b18
commit 0116160399
19 files changed
+3767 -30

No files matched your search

+217
View File
@@ -0,0 +1,217 @@
# Marqdo Implementation Status (M0.5)
Snapshot: **2026-11-20**, workspace version `0.1.0`, plugin ABI `1.0`,
`wire_version` 1, Markdown profile `marqdo.md.v1`.
This file maps the blueprint's phases (`docs/DEVELOPMENT_BLUEPRINT.md` §13) to
concrete evidence in the repository and lists honest open items. It is a status
report, not a promise. "Done" means implemented **for the documented subset**
and covered by tests; it does not mean the blueprint's full ambition is
complete.
Reproduce the evidence:
```bash
export PATH=/root/.cargo/bin:$PATH
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
cargo test --workspace
cargo test -p marqdo-cli
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
```
## 1. Phase matrix
| Phase | Status | Main evidence |
|---|---|---|
| **P0** Formal specs + parser goldens | **Done** | `spec/markdown-profile.md`, `spec/mapping.md`, `spec/expression.ebnf`, `spec/evaluation.md`; frozen profile test `crates/marqdo-markdown/src/profile.rs`; 150 AST fixtures + goldens in `tests/conformance/ast/`; 48 compile-error fixtures in `tests/negative/`; `inspect --ast`; `conformance_ast.rs`/`negative.rs` harnesses. |
| **P1** Interpreter, expressions, control flow | **Done** | `marqdo-expression`, `marqdo-ir`, `marqdo-frontend`, `marqdo-runtime`; factorial/countdown examples run without Python (`examples/factorial.mq.md`, `examples/hello.mq.md`); 36 execution goldens in `tests/conformance/semantics/`; runtime tests `crates/marqdo-runtime/tests/interpreter.rs`; prose invariance `crates/cli/tests/properties.rs`. |
| **P2** Modules, references, CLI experience | **Partial** | Imports/ModuleId/UnitId/anchor/cycle detection + `run/call/check/inspect/fmt` (`crates/cli`), multi-module example and `multi_module_imports_and_cycles` CLI test; `check --refs` reference validation (all loaded documents, on-demand static analysis of link targets) and `inspect --refs` static graph. Missing: cross-module link calls, rename/interface-lock tooling, canonical AST-preserving `fmt` (see §3). |
| **P3** Package environment | **Partial** | `marqdo.yaml`/`marqdo.lock` types+validation (`marqdo-package`), `init`, `add`/`remove` (manifest editing), deterministic `lock`, `sync --locked`; CLI test `lock_and_sync_locked_detect_dependency_tampering`; `examples/multi-module`. Missing: git resolution, transitive deps, registry, `tree/update/vendor`, `--offline`, environment materialization. |
| **P4** Security core and isolation | **Partial** | `CapabilityHost`, capabilities at execution points (console and math included), project-rooted FS with traversal rejection, budgets, effect trace, pure/project modes, **strict refuses without OS isolation** (`marqdo-sandbox`); CLI test `strict_mode_refuses_without_isolation`; host unit tests. Missing: OS isolation adapter, plugin worker, network enforcement. |
| **P5** Math and Foreign | **Partial (documented subset)** | `marqdo-math` exact-rational formula kernel for the grammar in `spec/math.md`; `math.parse/eval/substitute` host ops; `marqdo-foreign` Python/Node one-shot kernels with typed `foreign.call`; tests `crates/marqdo-math/tests/kernel.rs`, `crates/marqdo-foreign/tests/kernels.rs`; examples `math.mq.md`, `python-foreign.mq.md`. Missing: `simplify`/`differentiate`, matrices/series/symbolic algebra, foreign sessions, Python environment management. |
| **P6** Plugin ABI and developer experience | **Partial** | Frozen `include/marqdo_plugin_abi_v1.h`; C ABI mirrors/layout assertions, version negotiation, `plugin.yaml` validation, target triple + sha256 verification, wire envelopes, native loader (`marqdo-plugin-abi`); Rust SDK macro (`marqdo-plugin-sdk`); C demo (`examples/plugin-demo`); CLI `plugin inspect` and explicitly trusted in-process invocation (`--plugin`, `--trust-native-plugin`, `--allow-plugin`); tests `plugin_inspect_and_explicit_trusted_invocation`, `crates/marqdo-plugin-abi/tests/loader_unix.rs`. Missing: isolated worker transport, plugin capability bridge, installation, Rust plugin example crate. |
| **P7** Performance, release, CI | **Partial** | `.github/workflows/ci.yml` runs fmt/clippy/workspace tests plus CLI example smoke tests (including plugin inspect and invocation) on Linux x86_64. Missing: benchmarks, bytecode backend, cross-platform matrix, signed releases and supply-chain checks. Tree-walk interpreter only. |
## 2. Feature evidence (selected commands)
| Command | Verified behavior |
|---|---|
| `marqdo init demo` | creates `marqdo.yaml` (`>=0.1,<1`), `index.mq.md`, `.gitignore` (`.marqdo/`); refuses existing manifest (exit 1). |
| `marqdo check index.mq.md` | static only; 0 when no error diagnostics (warnings allowed); 1 on errors; JSON `command/success/errors/diagnostics`. |
| `marqdo run index.mq.md` | executes `main` through the Rust interpreter/host; program stdout on stdout; exit 1 on runtime/compile/strict errors. |
| `marqdo call file fn --arg n=5` | typed constant arguments; prints the non-`Unit` return value. |
| `marqdo inspect file --ast/--units/--signature/--ir/--refs/--platform --json` | static dashboards; `--platform` reports strict unavailable. |
| `marqdo fmt [--check]` | normalizes CRLF and trailing newlines only; exit 1 if `--check` detects a change. |
| `marqdo plugin inspect dir` | validates manifest + verifies sha256; never `dlopen`s (`loaded: false`). |
| `marqdo run doc --plugin dir --trust-native-plugin --allow-plugin demo.solver` | calls `solver.add(a=20,b=22)` and prints `42`; without trust/grant it exits 1. |
| `marqdo lock` / `sync --locked` | deterministic path-dependency lock; `--locked` fails on manifest/dependency drift. |
Test corpus at this snapshot (`tests/README.md`):
* 150 AST fixtures + goldens (`tests/conformance/ast/`);
* 36 semantic execution fixtures + goldens (`tests/conformance/semantics/`);
* 48 compile-time hard-error fixtures, 6 inert-executable warning fixtures,
10 runtime-error fixtures (`tests/negative/`);
* `crates/cli/tests/` harnesses: `conformance_ast`, `conformance_semantics`,
`negative`, `properties`, `integration_cli`;
* crate tests across all 15 crates, including real `python3`/`node` foreign
tests (skipped if unavailable) and a compiled C plugin test (skipped without
`cc`).
At this snapshot `cargo test -p marqdo-cli` passes all 21 harness tests
(2 CLI unit + AST/semantics/negative/properties/integration) and
`cargo test --workspace` passes. The fixture counts above match
`tests/README.md`; empty `tests/integration`, `tests/properties`,
`tests/security` and `tests/abi` directories are reserved for future
workstreams — the active corpus lives in `tests/conformance`, `tests/negative`
and `crates/cli/tests`.
## 3. Open items by phase
### P0
* Wire floats now round-trip exactly: `serde_json`'s `float_roundtrip`
feature is enabled and a Python `math.pi * 9` fixture pins the exact value
(`spec/value-wire.md` §3).
* Profile change safety relies on the frozen-options unit test; no CI gate
regenerates goldens (intentional).
### P1
* The blueprint asks for a draft Turing-completeness argument. The language
supports mutable locals, branch-first-match, per-item `while` loops and
recursion with bounded budgets; a formal encoding write-up is still missing.
* The default cooperative stack budget is 128 frames so very deep recursion
yields a structured `resource.exhausted` instead of exhausting the debug-build
native stack; `--stack` raises it deliberately. A true guard-page/iterative
interpreter hardening is still open.
### P2
* `fmt` is intentionally minimal (line endings/trailing newlines only), not the
blueprint's "canonical AST-equivalent formatting"; no semantic formatting,
AST-equivalence check or render-preserving rewrite exists.
* Reference validation is opt-in: `check --refs` (and `inspect --refs`) resolve
links; plain `check` does not fail on broken links because links are static
knowledge references rather than execution inputs.
* Link-form calls cannot cross modules; cross-module calls use imported alias
expressions. The blueprint's `> [f](./other.mq.md#f)(...)` example is not
supported.
* `ref("#...")` is module-local and does not accept a selector for units with
multiple resources (it errors with `ref.ambiguous`, ADR-0003).
* No rename tooling, interface lock/ABI diffing, or incremental cache.
* Duplicate/nested project boundaries are not diagnosed (nearest
`marqdo.yaml` wins).
### P3
* Git dependencies can be declared (`git` + `rev`, validated) but cannot be
resolved or locked: `lock` errors "git dependencies are not supported by this
build".
* No transitive dependency graph, conflict resolution, registry, publishing,
signing, `tree`, `update`, `vendor`, `--offline`, or `lock --locked`.
* No environment materialization (`.marqdo/env`, caches, kernels, sessions);
`sync` verifies but never installs or rewrites.
* `add`/`remove` are shallow manifest editors; adding a git dependency succeeds
at the manifest level even though `lock` will later refuse it.
* JSON errors from `add`/`remove`/`lock`/`sync` are plain stderr even with
`--json`.
### P4
* **Strict mode cannot run at all**: no OS isolation adapter (Linux
namespaces/seccomp/cgroups, Windows Job Object/appcontainer, macOS sandbox)
and `worker_processes: false`; `run --mode strict` exits 1 (ADR-0004).
* `console` capability is always on; `print` is not capability-checked.
* Network capability (`net_hosts`, `check_net`) exists but no host operation
performs network I/O and there is no `--allow-net` flag.
* Write-path symlink TOCTOU caveat (`spec/security.md` §2.1); symlink/race
hardening open.
* Budgets do not cover every allocation path; the interpreter's containers are
bounded only by fuel and available memory.
* Effect trace does not record argument values or paths (privacy-positive, but
less diagnostic detail); trace is opt-in and implicitly on for `--json`.
* `plugin_handle` wire values are structurally reconstructable, not
unforgeable capability tokens.
### P5
* Math is a deliberately limited calculator subset: no `simplify`,
`differentiate`, matrices, sums/integrals, trig/log functions, units,
assumptions or theorem proving. `math.simplify`/`math.differentiate` are
host ops that always return `math.unsupported`.
* Equation solving is affine-only, single target, target must occur exactly
once; other shapes are `math.unsupported`.
* Division by zero inside a formula maps to `runtime.type` (via
`MathErrorKind::Type`), not `runtime.division_by_zero`; documented divergence.
* Foreign kernels are one-shot only: no sessions, state bridge, Python/Node
environment management, or network policy for child processes.
* Foreign child processes are terminated only when the direct child is killed;
grandchildren are not tracked.
* Foreign per-call timeout (10 s) and output cap (1 MiB) are not CLI-configurable.
### P6
* No isolated plugin worker (length-prefixed IPC) or Wasm transport; only
explicitly trusted in-process native loading. A native plugin crash is a host
crash (`spec/plugin-abi.md` §1/§9).
* No plugin capability bridge from the CLI: plugins are loaded with no
capability handler, so every plugin `request_capability` is denied.
* Plugin calls are not statically validated; unknown namespaces are runtime
`symbol.unknown`. Positional plugin arguments are rejected.
* One namespace per plugin (first export prefix); multi-prefix manifests are not
reachable beyond the first prefix.
* No plugin installation/package management; `plugins:` in `marqdo.yaml` is
declarative only. No Rust plugin example crate (SDK has doc examples/tests).
* No hot unload/reload (ABI v1 design).
### P7
* Performance work is untouched: no benchmark suite, no bytecode backend or
differential tests against the tree-walker, no parse/compile caching.
* No CI workflows (`.github/` is empty of workflow files); no
Windows/macOS/Linux matrix, sanitizers/fuzz gates, signing, release
automation or supply-chain scanning.
* No published version, no compatibility matrix beyond the frozen ABI/format
documents.
## 4. Blueprint §18 "Alpha Definition of Done" mapping
| §18 item | Status | Evidence / gap |
|---|---|---|
| Run/call non-trivial recursive, branch, loop, call, parameter-inference programs without Python | **Met** | factorial/recursion (`examples/factorial.mq.md`, runtime + semantic goldens), countdown loop, multi-module alias call; interpreter is native Rust. |
| `.mq.md` readable under the chosen CommonMark/GFM/Math profile; standard AST behavior covered by goldens | **Met** | frozen `marqdo.md.v1`, 150 AST goldens, profile option test, `spec/markdown-profile.md`. |
| No heuristic downgrade of execution semantics; accurate source locations for invalid execution nodes | **Mostly met** | hard errors with spans for function-body executable shapes and negative corpus. Gaps: inert executable outside functions is a warning by design (ADR-0002); cross-module runtime error spans resolve against the entry file only; some lazy nested nodes are inert without a warning. |
| `check`/`inspect` side-effect free; imports, calls and static references verifiable across files | **Mostly met** | static load/analysis only; imports + cycle detection; cross-module calls validated; `check --refs` validates references across loaded documents and statically analyses link targets; `inspect --refs` prints the entry graph. |
| Minimal manifest, lock file, path/Git dependencies and environment sync | **Partial** | manifest/lock/path works, `sync --locked` verifies; git resolution, transitive deps, environment materialization and `--offline` missing. |
| All host side effects capability-authorized; untrusted code/native plugins cannot masquerade as strict sandbox | **Mostly met** | capability checks at execution points; strict refuses; native plugins require explicit trust and are documented as unsandboxed. Gaps: network enforcement not implemented (no network builtin is exposed), no worker isolation. |
| Formula computational subset explicit/testable; Foreign I/O, timeout and permissions explicit | **Met for the documented subset** | `spec/math.md`, `spec/foreign.md`; math kernel tests; foreign tests (timeout, protocol, output cap, env); permission gating tests/smoke. Simplification/differentiation explicitly unsupported. |
| Plugin C ABI v1, Rust SDK, C example, isolated worker basic path, ABI compat error tests | **Partial** | ABI/SDK/C example/loader/ABI-error tests exist; **isolated worker path missing**. |
| CI has at least one full research pipeline: Markdown function → Math → Foreign → Plugin → controlled output | **Partial** | CI now runs fmt/clippy/tests and CLI smoke tests; individual math/foreign/plugin paths are covered by examples and tests, but no single end-to-end pipeline document/test exists yet. |
| Examples, README, AGENTS and contributor docs allow a first change without legacy context | **Met** | `README.md`, `AGENTS.md`, this guide, `docs/STATUS.md`, specs, ADRs, `tests/README.md`, examples. |
**Conclusion: the Alpha definition of done is not fully met.** P0/P1 and the
documented P5 subset are solid; P2/P3/P4/P6 are functional but partial; P7 is
absent. The blocking items for an Alpha claim are the strict sandbox/worker
isolation story (P4/P6), git/environment package management (P3), and a CI
end-to-end pipeline (P7).
## 5. What is deliberately limited (do not overclaim)
* Only a single, frozen Markdown profile (`marqdo.md.v1`); changing parser
options/versions is a language change.
* Strict mode is a refusal, not an isolation implementation. `project` mode is
cooperative and in-process.
* Native plugins are trusted in-process code; there is no isolated plugin
worker, and a plugin crash takes down the host.
* Ints are arbitrary precision; finite floats are binary64 and wire round-trip
is exact (`float_roundtrip` enabled), but decimal-vs-binary formatting still
applies when displaying values.
* Math is a limited exact-rational calculator, not a CAS or a prover.
* Foreign kernels are optional, one-shot, unsandboxed child processes.
* `fmt` is minimal; `check` does not yet validate Markdown links; link-form
calls cannot cross modules.
* Package management is path-only in practice; `marqdo.yaml` capabilities are
requests, never grants.
@@ -0,0 +1,139 @@
# ADR-0001: Crate graph and the `marqdo-span` foundation
Status: **Accepted**
Date: 2026-11-20
## Context
`docs/DEVELOPMENT_BLUEPRINT.md` §11.1 fixes the dependency direction for the
workspace and states: "`markdown ->` zero other Marqdo crate; responsible for a
stable view of the standard AST." The implemented workspace instead has
fifteen crates, and `marqdo-markdown` depends on `marqdo-span`.
Why the divergence was necessary:
* Every layer needs identical byte spans: the Markdown AST
(`crates/marqdo-markdown/src/ast.rs`), expression AST
(`crates/marqdo-expression`), IR (`crates/marqdo-ir`), diagnostics and runtime
errors (`crates/marqdo-value`), effect traces (`crates/marqdo-host`), plugin
errors (`crates/marqdo-plugin-abi`) and the CLI renderer.
* A `Span` must be the same type across boundaries or every layer needs
conversion functions. Conversions are exactly where off-by-one bugs and
broken source locations appear, and they would make `Expr`/`Stmt`
serde output inconsistent between `inspect --ast` and `inspect --ir`.
* The blueprint's "zero Marqdo dependencies" rule exists to keep the parser
independent of the semantic layers. A tiny, effect-free span crate preserves
that property while satisfying the shared-data-contract requirement.
## Decision
1. Add `crates/marqdo-span` as a **leaf** crate with no Marqdo dependencies and
only `serde` as an external dependency. It defines:
* `Span { start: usize, end: usize }` — half-open byte range into the
original UTF-8 source, with `merge`, `shift`, `contains`, `len`;
* `LineCol { line, column }` — 1-based, byte columns;
* `LineIndex` — precomputed line starts for byte→line/column conversion.
2. Treat `Span` as part of the toolchain's shared data contract. Crates that
produce or consume source locations depend on `marqdo-span` directly; they
do **not** wrap it in local span types.
3. `marqdo-markdown` depends on `marqdo-span` (and `comrak`) but on **no
semantic crate**. It still owns only the canonical AST view, front matter
extraction and source maps; it performs no mapping, resolution or execution.
4. Keep the rest of the blueprint's direction:
`value -> span`, `expression -> span`, `ir -> span + expression`,
`markdown -> span + comrak`, `frontend -> markdown + expression + ir + value`,
`host -> value + math + foreign + sandbox`, `runtime -> ir + value + host +
expression`, `cli -> all` (composition root). There is no `runtime -> cli`
edge and no cycle; `cargo tree` is expected to stay acyclic.
5. Do not introduce the blueprint's separate `sema` or `environment` crates in
M0.5: semantic analysis lives in `marqdo-frontend`
(`analyze`, `calls`, `infer`, `signature`, `units`, `map`), and package/
environment formatting lives in `marqdo-package` plus the CLI. Splitting
them is a refactor, not a semantic change.
Actual dependency graph (workspace `Cargo.toml` plus crate manifests):
```text
span serde
value span, serde, serde_json, num-bigint, num-traits
expression span, serde, num-bigint, unicode-ident
markdown span, comrak, serde, serde_yaml, unicode-normalization
ir span, expression, serde
frontend span, markdown, expression, ir, value, sha2,
unicode-normalization, unicode-ident
math value, num-bigint, num-rational, num-traits, serde
foreign span, value, serde_json, thiserror
sandbox serde
plugin-abi span, value, serde, serde_json, serde_yaml, thiserror,
libloading, sha2, hex
plugin-sdk plugin-abi, serde, serde_json
host span, value, math, foreign, sandbox, serde, serde_json,
thiserror, glob
runtime span, value, ir, expression, host, num-bigint,
num-rational, num-traits, thiserror
package serde, serde_yaml
cli span, value, expression, markdown, ir, frontend, math,
foreign, plugin-abi, host, runtime, package, sandbox,
clap, serde, serde_json, walkdir, hex, sha2
```
## Alternatives
1. **Put `Span` in `marqdo-value`.** Then `marqdo-markdown` would depend on the
runtime value model, which is a semantic layer and pulls in `num-bigint` and
`serde_json`. Worse layering than a dedicated leaf.
2. **Duplicate `Span` in each crate and convert at boundaries.** Avoids a crate
but guarantees conversion code, mismatched serde shapes and duplicated
tests; rejected.
3. **Define `Span` in `marqdo-markdown` and let others depend on it.** Makes
the parser a de-facto foundation crate and lets non-Markdown layers depend
on `comrak`'s dependencies; rejected.
4. **Use a third-party range type.** Adds a dependency and gives no benefit
over ~170 lines; rejected.
5. **Keep the blueprint's literal "markdown has zero Marqdo deps".** Would
force either duplication or a span type in the parser that cannot be shared
with diagnostics; rejected in favor of a documented, narrow exception.
## Markdown AST shape
No syntax change. Every `MdNode` still has `span: Span` in the original file
(`crates/marqdo-markdown/src/ast.rs`), and the JSON inspection view is
`{"kind":"heading","level":2,"span":{"start":0,"end":7},...}`. Spans are
half-open byte offsets, BOM-rebased, and byte columns for diagnostics.
## Compatibility
* This is a **new repository**; there is no legacy compatibility burden.
* Public Rust API: moving `Span` later is a workspace-wide API change but has
no language/ABI/package-format impact. It would require updating every
`crates/*/Cargo.toml`.
* Stable observable contracts (AST JSON `span`, diagnostic JSON `span`, IR
`span`) are unchanged by the crate boundary. Changing span semantics
(e.g. line/column units) requires a spec change and ADR.
* The divergence from blueprint §11.1 is recorded here; future crates should
use `marqdo-span` rather than introducing another location type.
## Security
No new capability or effect. `marqdo-span` is `serde`-only, performs no I/O and
cannot execute anything. Byte offsets are derived from already-parsed source;
`LineIndex::line_col` clamps out-of-range offsets instead of panicking.
## Testing
* `crates/marqdo-span/src/lib.rs` tests multi-byte UTF-8 line/column mapping,
`merge`/`shift`/`contains`, and clamping.
* `crates/marqdo-markdown/src/profile.rs` tests BOM rebasing and that every AST
node carries a byte span; frontend tests assert diagnostics point at the
expected source line/column.
* `cargo test --workspace` must pass; `cargo tree` must remain acyclic (no
automated CI gate yet — this is a P7 item).
## Migration & rollout
Immediate: the crate exists and all layers already use it. Future work may
split the frontend into `sema`/`environment` crates; that must preserve the
graph above (`runtime` must never depend on `cli`, `markdown` must never depend
on semantic crates). If a second span representation is ever needed (for
example for generated code), add it as a conversion in `marqdo-span`, not as a
per-crate duplicate.
@@ -0,0 +1,140 @@
# ADR-0002: Executable nodes execute only inside level-2 function bodies
Status: **Accepted**
Date: 2026-11-20
## Context
The blueprint (§1.2 execution context, §3.2 control flow, unresolved issue
**S-03**) establishes that `Strong` (execute), `Emph` (return), ordered lists
(branch), bullet lists (loop) and block quotes (call) have program meaning
**inside a function body**, i.e. inside the range of a level-2 heading. It
leaves the module top level under-specified: if a document has `**x = 1**`
before any function, should that be a hard error, a silent no-op, or an
explicit top-level statement?
Two failure modes had to be avoided:
1. **Silent execution at module scope.** There is no function frame, no return
target and no entry point, so top-level effects would be surprising and
would make `check`/`inspect` harder to reason about.
2. **Hard error.** Ordinary Markdown documents routinely use `**bold**`,
`*italic*`, `1. ordered` and `> quotes` in prose. Treating every such node
outside a function as a fatal compile error would make the language hostile
to normal documentation and would break the "renders normally" goal.
The chosen baseline was: executable nodes only execute inside function bodies;
elsewhere they are ordinary document content, but this must not happen
silently.
## Decision
1. A **function body** is the child-block range of a `heading(level=2)` up to
the next heading of level ≤ 2 (`crates/marqdo-frontend/src/units.rs`,
`FunctionScope.body_start..body_end`). Executable roles are assigned only
inside those ranges by `crates/marqdo-frontend/src/map.rs`.
2. Outside every function body, `strong`, `emph`, `list` and `block_quote`
nodes are **inert document content**. They are not mapped to IR, are not
executed, and prose is not inspected for keywords.
3. The frontend emits a **warning** `markdown.inert_executable` for every
would-be execution node found outside a function body. The document still
compiles and runs; only `check`'s diagnostic output reflects the warning.
The warning is produced once per top-level node and has a hint to move the
construct under a `## function` heading.
* `block_quote` → "a block quote invocation ... is inert document content";
* `list` → "a list control structure ...";
* `paragraph` → the first `strong`/`emph` found in the paragraph
("an execution statement" / "a return statement").
* warnings are emitted only while scanning the document's direct top-level
blocks: headings, tables, code/math blocks and other non-paragraph blocks
produce none. A top-level paragraph is walked recursively, so a
`strong`/`emph` nested under a link or strikethrough inside that paragraph
can still produce the inert warning. Nodes inside a function body produce
no warning (they are mapped or rejected by the strict executable-shape
rules instead).
4. Warnings do not change exit codes: `check` exits 0 when there are no error
diagnostics even if warnings exist; `run`/`call` execute normally. JSON
`check` output sets `success = (errors == 0)` and includes the warning
entries in `diagnostics`.
Rationale for the nodes covered: these are exactly the block-level constructs
that `map.rs` would otherwise turn into statements if a function body existed.
Inline nodes inside a paragraph are covered by one paragraph-level warning
rather than one per node, which keeps normal prose readable.
## Alternatives
1. **Hard error for every executable-looking node outside a function.** Rejected
(see context); it forbids emphasis in ordinary Markdown and makes writing a
README a compile failure.
2. **Silent inert behaviour with no diagnostic.** Rejected: an author could
believe top-level `**x = 1**` had executed. The blueprint explicitly wants
invalid execution intent surfaced.
3. **Execute top-level nodes in an implicit module frame.** Rejected: no
defined entry semantics, no return target, would make imported modules
potential side-effect roots (violating "modules register definitions only").
4. **Compile-time error only when the payload parses as a statement.** Rejected:
this is the forbidden heuristic ("parses as code → execute") from blueprint
§1.3; role must come from node type and position, not from guessing.
5. **Warning per AST node.** Rejected as noisy for nested content; one warning
per top-level construct is enough to point at the problem.
## Markdown AST shape
No new node kinds. Relevant structure:
```text
document
heading(level=2) "f" <-- opens a function body
paragraph -> strong(text "x = 1") <-- executes (Stmt::Assign)
paragraph (before any heading)
strong(text "x = 1") <-- inert; emits markdown.inert_executable
list(ordered) <-- inert outside function; warning
block_quote <-- inert outside function; warning
```
A level-2 heading without a preceding level-1 heading is a module-root function
(`## main`) and its body executes normally. A `###` section inside the function
range stays inside the function body; headings never execute.
## Compatibility
* Warning code `markdown.inert_executable` is part of the CLI diagnostic
contract. Renaming it requires an ADR and a CLI version note.
* Making this a hard error later would be a breaking language change: documents
that currently compile (with warnings) would stop. If that is ever desired it
needs a migration plan (e.g. an opt-in strict lint).
* No package/ABI/IR shape change: inert nodes are simply absent from the IR.
* Prose invariance is preserved: editing narrative text outside functions
cannot change the IR.
## Security
No effect: inert nodes never reach the interpreter, host or kernels. The
warning reduces the chance of a false belief that a side effect happened,
which is a security-relevant honesty property. A document cannot cause an
execute-by-accident through top-level prose.
## Testing
* `crates/marqdo-frontend/src/units.rs::tests::
executable_outside_function_warns_but_does_not_execute` asserts the warning
and that the function index is still built.
* `units.rs` tests for function ranges, nested sections, root-level `## main`
and duplicate/empty headings.
* `crates/marqdo-frontend/src/analyze.rs` tests hard errors for invalid
executable shapes inside function bodies (the companion rule: when a node
*is* executable, its shape is strictly checked).
* Conformance/negative fixtures under `crates/cli/tests` and `tests/`
(work in progress by the fixtures workstream) are expected to cover the
warning and the inert behavior; `spec/mapping.md` §4 documents the contract.
## Migration & rollout
M0.5 ships with warnings. Rollout guidance:
* document authors move executable constructs under `## function` headings;
* tooling should show the warning in editors (the diagnostic already has a
precise span);
* if a future release makes inert execution a hard error, add a deprecation
window and a `check` flag so existing documents can be migrated.
@@ -0,0 +1,157 @@
# ADR-0003: `ref` resolution order and `ref.ambiguous` (S-07)
Status: **Accepted**
Date: 2026-11-20
## Context
`ref("#anchor")` is the only way to turn a heading range into an inert
resource value (`Formula` or `Code`) or a `FunctionRef`. The blueprint
(§5.2, unresolved issue **S-07**) is explicit that a heading range can contain
several formulas or code fences, and that the implementation must **never
silently pick the first one**:
> 一个范围存在多段公式时,默认对象选择必须是明确规则(例如禁止无选择器,要求
> `math.formulas(ref(...))` 列出候选;不选"第一条")。
The failing example is easy to write:
```markdown
## f
### 公式
$$a = 1$$
$$b = 2$$
**x = ref("#公式")**
```
If `ref` returned `a = 1`, the program would be non-deterministic with respect
to document structure (insert an earlier formula and the meaning changes) and
would hide a genuine authoring error.
At the same time the common case — one formula per heading — must stay simple;
requiring a selector for every reference would make `examples/math.mq.md`
awkward.
## Decision
`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref` implements this
algorithm:
1. The target must be a `Text` beginning with `#`; otherwise
`symbol.unknown` ("must be a `#heading` anchor"). `ref` is not a URL
resolver and does not cross modules.
2. The fragment is looked up in the **current module's anchor map**
(`Module::anchors`, GitHub-style slugs computed in
`crates/marqdo-frontend/src/units.rs`). `ref` does not fall back to
canonical unit names and does not search imported modules.
3. Collect the resources attributed to the resolved unit (innermost unit by
span, `crates/marqdo-frontend/src/resources.rs`).
* **Exactly 1** → return `Formula` or `Code` (with `unit` and `span`
provenance).
* **0** → if a function's unit equals the resolved unit, return
`FunctionRef { module, name }`; otherwise `symbol.unknown`.
* **≥ 2** → return **`ref.ambiguous`**:
"`#...` contains N resources; select one explicitly instead of relying on
the first". Never choose a resource.
4. Function references are callable through the normal call path; formulas and
code remain inert until `math.eval`/`foreign.*` is explicitly called (those
are separately capability-checked).
Resource attribution is deterministic: a resource belongs to its innermost
enclosing heading unit, resources keep document order, and a resource is a
Markdown `math`/`math_block` or `code_block` node (inline code is not a
resource).
The `ref.ambiguous` category is `ErrorCategory::RefAmbiguous` with the stable
code `ref.ambiguous` (`crates/marqdo-value/src/error.rs`).
## Alternatives
1. **First resource wins** (blueprint's rejected baseline). Rejected by S-07:
document edits would silently change program meaning, and there is no
diagnostic for the author.
2. **Always require a selector argument**
(`ref("#f", selector="second")`). Rejected for M0.5: no selector syntax or
addressing scheme is specified yet, and the single-resource case is by far
the common one. A future ADR can add an explicit selector while keeping the
ambiguity rule.
3. **Error on every heading with ≥ 2 resources at compile time**, even if never
referenced. Rejected: a document may legitimately contain several formulas
for prose and never `ref` them; only the reference is ambiguous.
4. **Return a list of all candidates.** Rejected for v1: it changes the return
type and forces every caller to handle a selector; a dedicated API
(`math.formulas(ref(...))`) would be a new spec, not a silent widening.
5. **Cross-module `ref`.** Rejected for v1: the wire/identity model and
`ModuleId` resolution live in the loader; `ref` is a pure runtime builtin
over the current module. Cross-module resources are an open item.
## Markdown AST shape
```text
heading(level=2) "f"
heading(level=3) "公式"
math_block "/$$ a = 1 $$/" -> Resource::Formula { unit: <#f/公式> }
math_block "/$$ b = 2 $$/" -> Resource::Formula { unit: <#f/公式> }
paragraph -> strong("x = ref(\"#公式\")")
```
Anchors are separate from unit paths: `#公式` is a GitHub slug for the heading
name, while the canonical `UnitId` is `<module>#f/公式`. Colliding slugs get
`-1`/`-2` suffixes; canonical ids never do.
## Compatibility
* `ref.ambiguous` is a stable runtime code; callers may branch on it.
* The single-resource and function-reference cases are frozen. Adding a
selector must be additive (new argument or new builtin) and must not change
the ≥ 2 error into a silent choice.
* `ref` target syntax (`#fragment`) is frozen; changing it to accept file paths
would need a new ADR.
* No package/ABI/wire change: formula/code values already carry
`unit`/`span`/`content_hash`.
Example behavior, verified by
`crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`:
```text
$ marqdo run index.mq.md # heading with two $$...$$ blocks and ref("#公式")
runtime error[ref.ambiguous]: `#公式` contains 2 resources; select one explicitly instead of relying on the first
```
## Security
* `ref` is a pure builtin: no host call, no capability required, no side
effect. It only returns inert values or a function reference.
* An inert `Formula`/`Code` does not execute. Evaluation/execution requires a
separate explicit call (`math.eval`, `foreign.run/call`), which is capability
checked at the host boundary.
* Refusing ambiguity prevents a class of "wrong formula executed" mistakes,
which matters for scientific documents where formulas are evidence.
## Testing
* `crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`
drives a real document through frontend + runtime and asserts the
`ref.ambiguous` code.
* `crates/marqdo-frontend/src/resources.rs` tests attribution to innermost
functions, multiple resources kept in one unit, and stable SHA-256 code
hashes.
* `crates/marqdo-markdown`/`frontend` tests cover anchors and slug collisions.
* `examples/math.mq.md` is the positive single-formula case; it runs under
`marqdo run` only via a function (the example itself defines callable
functions, no `main`).
* Missing anchor/not-a-function cases are covered by the `symbol.unknown`
paths; conformance fixtures are being added by the fixtures workstream.
## Migration & rollout
No migration needed for documents with exactly one resource per referenced
heading. Documents that relied on "first formula" behavior (there are none in
the examples) must split headings or wait for an explicit selector API. If a
selector is added later, keep `ref("#...")` behavior unchanged and introduce
the selector as a new argument/builtin, with tests for both the ambiguous and
the selected case.
@@ -0,0 +1,133 @@
# ADR-0004: Strict mode refuses to run without OS isolation
Status: **Accepted**
Date: 2026-11-20
## Context
Blueprint §9 defines three execution modes and one non-negotiable rule:
> `strict`: untrusted documents/code/plugins — independent restricted process,
> OS-level resource and syscall/file/network restrictions; if unavailable,
> **refuse to run**.
A cooperative in-process capability check cannot contain hostile native code
or an unrestricted child process. If `strict` silently degraded to `project`,
users would believe they had isolation they did not have. That is a security
bug, not a missing feature.
M0.5 implements capability checks, budgets and process supervision but **no OS
isolation adapter** (no Linux namespaces/seccomp/cgroups, no Windows Job
Object/appcontainer, no macOS sandbox), and the isolated plugin worker
transport is not implemented either.
## Decision
1. `crates/marqdo-sandbox` is honest about platform support:
* `PlatformCapabilities::detect()` always reports
`StrictIsolation::Unavailable { reason }` in this build, with a
platform-specific reason (Linux: "no OS isolation adapter
(namespaces/seccomp/cgroups) is implemented in this build"; macOS/Windows/
other targets have analogous messages);
* `worker_processes: false`.
2. `marqdo_sandbox::enforce(mode, platform)`:
* `pure` and `project` → `Ok(())`;
* `strict` → `Err(SandboxError::StrictUnavailable { reason })` unless
`platform.strict.is_available()`.
3. `run`/`call` call `enforce` **before** constructing the host, loading
plugins or executing anything. On failure they print
"refusing to run in strict mode: ..." (or a JSON `{"success":false,
"error":...}` envelope) and exit 1.
4. There is **no fallback**: the CLI never downgrades a requested strict run to
project mode. `SandboxMode::parse("strict")` succeeds (the mode name is
valid); only execution is refused.
5. `inspect --platform` exposes the detection report so users can see why:
`{"os":"linux","arch":"x86_64","strict":{"state":"unavailable","reason":...},
"worker_processes":false}`.
6. The rest of the system is labelled accordingly: `project` mode is
cooperative, in-process, and is not a sandbox; foreign kernels are
process-isolated but not confined; native plugins run in-process and require
explicit trust (ADR-0006/security spec).
## Alternatives
1. **Best-effort strict** (run with the same checks as project). Rejected: it
directly violates the blueprint's "cannot provide → refuse" rule and hides a
missing security boundary.
2. **Warn and continue.** Rejected for the same reason; a warning is not a
sandbox, and CI/agents may not read warnings.
3. **Implement a Linux namespaces/seccomp adapter now.** Out of scope for
M0.5: it needs platform-specific code, integration tests, resource limits,
CI on multiple kernels, and a way to supervise the worker protocol. This is
P4 work and belongs in its own ADR when implemented.
4. **Use Wasm/WASI for strict.** Attractive for plugins later, but it does not
sandbox arbitrary Python/Node processes or native `.so` files, and the
runtime is not implemented. Rejected for now.
5. **Claim strict via the process isolation of foreign kernels.** Rejected:
a child process inherits the user's privileges and filesystem/network
access; changing cwd and clearing env is hygiene, not confinement.
## Markdown AST shape
No language or AST change. A strict run is rejected before any document is
executed (after static loading/diagnostics); the CLI contract is:
```text
$ marqdo run index.mq.md --mode strict
error: refusing to run in strict mode: no OS isolation adapter
(namespaces/seccomp/cgroups) is implemented in this build. Strict mode
requires verified OS-level isolation and never falls back to an
unrestricted process.
$ echo $?
1
```
(Exact wording comes from
`crates/marqdo-sandbox/src/lib.rs::SandboxError::display`.)
## Compatibility
* Accepting `--mode strict` as a valid *name* is part of the CLI contract; the
refusal is a documented behavior, not a parse error. A future release that
implements isolation flips the detection report and starts executing strict
runs — that is the intended upgrade path.
* Exit code 1 (runtime/security refusal) is stable; usage errors are 2.
* `PlatformCapabilities` JSON shape (`state: available|unavailable`) is
versioned by documentation and should not change without an ADR.
* No package/ABI/wire impact.
## Security
* The guarantee "requested strict → refused rather than downgraded" is
enforced in code (`enforce`), not documentation.
* Users cannot accidentally believe untrusted docs are sandboxed by this
build; the platform report is the evidence.
* Project mode remains explicitly non-sandboxed. Native plugins (in-process)
and foreign child processes are recorded as trust decisions, not isolation.
* Secrets: strict refusal happens before plugin loading or foreign spawning,
so no untrusted code is touched by a refused command.
## Testing
* `crates/marqdo-sandbox/src/lib.rs::tests::
strict_refuses_when_isolation_is_unavailable` asserts the refusal and that
`pure`/`project` still succeed.
* `mode_names_round_trip` covers parsing/printing.
* CLI behavior (`--mode strict` exit 1, message) is covered by manual/CLI
smoke checks; a CLI integration test should be added with the conformance
harness.
* `inspect --platform` JSON is exercised manually and in the CLI help/example
documentation; it should get a golden test in the fixture workstream.
## Migration & rollout
When an OS isolation adapter lands:
1. implement a `StrictIsolation::Available { mechanism }` report and a real
supervisor/worker;
2. add integration tests for filesystem/network/process restrictions and
resource limits per platform, plus the platform capability matrix;
3. update `spec/security.md`, this ADR (or supersede it), `docs/STATUS.md` and
the CLI docs;
4. only then change `PlatformCapabilities::detect`; do not add a flag that
bypasses the refusal before that work is done.
+174
View File
@@ -0,0 +1,174 @@
# ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)
Status: **Accepted**
Date: 2026-11-20
## Context
The language derives program meaning from the **canonical AST**, never from
source spelling (`spec/markdown-profile.md`, `spec/mapping.md` §1). The parser
(`comrak =1.0.0`, frozen options) normalizes several syntaxes before Marqdo
sees them, and a few normalizations are lossy:
* `# H` and Setext `H\n===` both become a heading node; the source spelling is
lost unless recorded;
* GFM task list items (`- [x] text`) produce a task-specific item node;
* inline code spans are stored as a literal without the backtick delimiters,
but source spans include them;
* `~~strikethrough~~` is enabled by the profile and wraps arbitrary inlines.
The language needs a single, documented answer per question, because node type
and structural position decide executable roles.
## Decision
The profile (`marqdo.md.v1`) and the canonical AST
(`crates/marqdo-markdown/src/ast.rs`, `src/profile.rs`) freeze these rules:
1. **Setext headings are headings.** `# name` and a Setext level-1/2 underline
both produce `heading { level, setext }`; `setext` is inspection-only and
**never affects semantics**. `## name` and a Setext level-2 heading are both
functions. The profile keeps `parse.ignore_setext = false`.
2. **Task-list items are `item` nodes with a task flag.**
`item { task: Some(true|false) }` for GFM task items, `task: None`
otherwise. Node type stays `item`, so list structure (ordered vs bullet)
still determines branch/loop roles; the checkbox flag is inert data and does
not change execution. A task list item in an executable position still
requires its condition header like any other item.
3. **Code spans carry delimiter tick counts.** Inline `code { ticks }` records
the number of backticks, and the source span includes them; the literal is
the inner payload. This lets the frontend map expression payload spans back
to exact source bytes for diagnostics. "Ticks" is not user-visible program
meaning.
4. **Strikethrough is inert/lazy.** `strikethrough` is an inert wrapper: it is
never traversed for execution, its content is prose, and a `strong`/`emph`
inside it does not execute. It is not an error to write it; the language
simply does not use formatting as a control-flow mechanism here. (A code
span or math inside it is likewise inert.)
5. **Unmodelled nodes are preserved** as `other { tag }` with children and raw
literal. They are inert but never silently dropped from `inspect --ast`.
6. **Literal payloads are kept** on inert leaves: `text`, `code`, `code_block`,
`math`, `math_block`, `html_block`, `html_inline`, `front_matter`.
7. **Other frozen normalizations:**
* `**x**`/`__x__` and `*x*`/`_x_` are the same node kinds (`strong`/`emph`);
* ordered delimiters `1.`/`1)` and bullets `-`/`*`/`+` are the same
`list`/`item` structure; role depends on `ordered`, not marker;
* tight vs loose lists only affect rendering (`tight` is recorded, inert);
* `link`/`image` keep `url`/`title`; `table` keeps column alignments,
`table_row` keeps `header`; `heading` keeps `level`/`setext`;
* a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the
original file;
* front matter is recognized only at the document start (after BOM), with
`---` opening and `---`/`...` closing; a later `---` is a thematic break
or Setext underline.
These rules are part of profile id `marqdo.md.v1`. Changing them (or bumping
the comrak version/options) requires a profile id bump, updated goldens and this
ADR being superseded.
## Alternatives
1. **Distinguish Setext vs ATX headings semantically.** Rejected: identical AST
structure outside the marker must have identical meaning (`## H` and Setext
H2 are the same function). Keeping `setext` only for inspection avoids
silently diverging programs when a document changes style.
2. **Introduce a separate `task_item` node kind.** Rejected: it would require
every list-role check to special-case two item kinds and could be used to
hide a control structure behind a checkbox. Normalizing to `item { task }`
keeps list structure authoritative.
3. **Strip the tick information and store only the literal.** Rejected: payload
spans returned by the expression parser are relative to the concatenated
payload; without delimiter lengths, error spans cannot be mapped back to
file bytes precisely.
4. **Traverse strikethrough content for execution.** Rejected: it is a lazy
region by profile; executing inside it would make formatting semantics
surprising ("struck-through code runs"). It would also create a second way
to hide executable nodes.
5. **Drop unknown parser nodes.** Rejected: `inspect --ast` must show what the
parser produced; dropping syntax hides profile drift.
6. **Preserve raw source and re-parse with a custom parser.** Rejected by the
project's core rule: one standard Markdown parser, no second Markdown AST.
## Markdown AST shape
Frozen JSON shapes (`inspect --ast --json`):
```json
{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
"children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
```
```json
{"kind":"list","ordered":false,"tight":true,"span":{...},
"children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
```
```json
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
```
```json
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}
```
Attribute order in the JSON is fixed by the serializer; goldens are regenerated
only through the explicit update workflow.
### 5.1 Known interaction to keep in mind
Inside a function body, a `**strong**` (or an ordered/bullet list) nested under
`strikethrough` is inert and never executes; because the mapper does not
traverse strikethrough, it also produces no warning. Outside function bodies,
the `markdown.inert_executable` scan walks a top-level paragraph recursively,
so such a nested node may still produce the inert warning even though it would
not execute if the paragraph were inside a function. This is consistent with
"formatting never hides execution", but an author who intends execution must
not wrap it in strikethrough. Warnings for lazy-region content inside functions
are a possible future refinement; no behavior change is planned without an ADR.
## Compatibility
* Profile id `marqdo.md.v1` is frozen with comrak `=1.0.0` and the exact
extension flags; the unit test
`crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen`
asserts the option table (tables, tasklist, strikethrough, math dollars,
front matter; all other extensions disabled).
* Adding an AST field is a JSON output change for `inspect --ast`; it needs a
goldens update and a profile note. Changing a semantic rule (e.g. executing
inside strikethrough) is a language change and requires an ADR.
* No package/ABI/wire impact.
## Security
* The lazy-region rules ensure inline formatting inside headings, tables,
links, code, math, HTML and front matter never executes — a context-aware
visitor is used (`crates/marqdo-frontend/src/map.rs` only descends from
paragraphs into `strong`/`emph`, and ignores every other inline node).
* Strikethrough cannot be used to smuggle executable content; it is inert.
* Normalizing task items to `item` prevents a checkbox from hiding a
branch/loop header; the condition-header check still applies.
* No new capabilities or host calls.
## Testing
* `crates/marqdo-markdown/src/profile.rs`:
* `profile_options_are_frozen` — exact extension/parse option table;
* `setext_headings_are_heading_nodes` — Setext produces `heading { setext: true }`;
* `tables_and_task_lists_are_supported` — task items carry `task: Some(_)`;
* span/BOM tests.
* Frontend mapping tests use code spans as conditions and payloads, exercising
`ticks`/payload span mapping.
* `spec/markdown-profile.md` §5/§6 is the normative JSON shape; AST goldens
under `crates/cli/tests` (fixtures workstream) regenerate only with the
documented update command.
* Conformance fixtures are expected to include `**a**` vs `__a__`, Setext vs
ATX, task/list variants and strikethrough inertness.
## Migration & rollout
No migration for documents: Setext and ATX were always the same function under
this profile. When comrak is upgraded or the extension set changes, bump the
profile id, regenerate AST goldens, re-run all specs/ADRs that reference node
shapes, and document the change here (superseding this ADR if a rule changes).
Code-span `ticks` and `task` are additive inspection fields; keep them optional
in consumers that parse the JSON.
+225
View File
@@ -0,0 +1,225 @@
# ADR-0006: CLI output, exit-code and JSON contract
Status: **Accepted**
Date: 2026-11-20
## Context
CI, scripts and coding agents need a stable CLI contract, not just human
output. The blueprint (§12) requires: all commands available with `--json`,
non-zero error exit codes, program stdout kept separate from diagnostics, and
machine-readable error codes. The implemented CLI must be documented precisely
because scripts will depend on stream separation and exit status.
Implementation: `crates/cli/src/main.rs` (clap definitions, process exit),
`crates/cli/src/commands.rs` (behavior), `crates/cli/src/output.rs`
(diagnostic/JSON rendering), `crates/cli/src/args.rs` (`--arg` constants).
## Decision
### 1. Commands and flags (implemented surface)
| Command | Arguments / flags | What it does |
|---|---|---|
| `init [dir]` | — | create `marqdo.yaml`, `index.mq.md`, `.gitignore`; refuses an existing manifest |
| `run [file]` | `--mode`, `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin`, `--fuel`, `--stack`, `--max-output`, `--timeout`, `--plugin`, `--trust-native-plugin`, `--json`, `--trace` | execute `main` (or default `index.mq.md`) |
| `call <file> <function>` | `--arg NAME=VALUE` (repeatable) plus all `run` flags | execute one named function |
| `check [file]` | `--json` | static check; never executes |
| `inspect [file]` | `--ast`, `--units`, `--signature`, `--ir`, `--refs`, `--platform`, `--json` | static inspection; never executes |
| `fmt [file]` | `--check` | normalize CRLF and trailing newlines |
| `plugin inspect <dir-or-yaml>` | `--json` | validate manifest + verify artifact hash; never loads code |
| `add <name>` | `--path`, `--git`, `--rev`, `--json` | edit `marqdo.yaml` |
| `remove <name>` | `--json` | edit `marqdo.yaml` (and prune a stale lock entry) |
| `lock` | `--json` | write deterministic `marqdo.lock` |
| `sync` | `--locked`, `--json` | verify the lock is current |
There is no `test`, `tree`, `update`, `vendor`, `add` for plugins, `kernel`
command or `--project`/`--offline` flag in this build. `run` defaults the
file to `index.mq.md`; `check`/`inspect`/`fmt` do too. `call` requires an
explicit file path plus a function name.
`--mode` is only on `run`/`call` and accepts `pure|project|strict`
(default `project`). `--trust-native-plugin`/`--plugin` are only meaningful on
`run`/`call`. `inspect --platform` still requires a document argument (or
defaults to `index.mq.md`), because project loading happens first.
### 2. Exit codes
| Code | Meaning |
|---|---|
| `0` | success (for `check`: no **error** diagnostics; warnings do not fail) |
| `1` | a checked document has errors; a runtime/security failure; strict refusal; plugin load/inspect failure; lock/sync mismatch; `fmt --check` would reformat; `init` found an existing manifest; `add` duplicate; `remove` missing dependency; `sync` stale lock |
| `2` | usage error (clap), unreadable entry file, invalid `--mode` name, invalid capability combination (e.g. `--allow-read` in `pure`), invalid `--arg` value, unreadable manifest for project commands, unreadable file for `fmt` |
Codes are stable contract. Diagnostic **codes** (for example
`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`,
`permission.denied`, `ref.ambiguous`) are stable; message wording is not.
### 3. Stream separation
* **Program output** (`打印`/`print`) goes to stdout in text mode. In JSON mode
it is embedded as the `stdout` string; it is never interleaved with JSON.
* **Diagnostics and effect traces** go to stderr in text mode.
* `run` in text mode writes only program output to stdout (no implicit return
value). `call` in text mode additionally prints the returned value (using
`Value::display`) when it is not `Unit`.
* `check` text mode: diagnostics to stderr; `checked N module(s): ok` to stdout
on success; a `check failed: ...` summary to stderr on failure.
* `inspect` prints pretty JSON-ish structures to stdout; load errors to stderr.
* JSON mode prints exactly one pretty JSON object to stdout per invocation.
`add`/`remove`/`lock`/`sync` still print errors as plain `error: ...` on
stderr even with `--json` (JSON success envelope only) — an inconsistency to
fix before the contract is declared final.
### 4. JSON output shapes
`check`:
```json
{"command":"check","success":true,"errors":0,
"diagnostics":[{"file":"index.mq.md","module":"index.mq.md",
"line":3,"column":5,"severity":"error","code":"expr.parse",
"message":"...","hints":["..."]}]}
```
`inspect`:
```json
{"command":"inspect","module":"index.mq.md",
"ast":{...},"units":[...],"signatures":[...],"ir":[...],
"refs":[...],"platform":{...}}
```
Only the requested sections are present. With no section flags, a `summary`
with `modules`, `functions`, `units`, `resources` is emitted instead.
`run`/`call` (success):
```json
{"command":"run","success":true,
"value":{"type":"text","value":"..."},
"stdout":"program stdout text",
"trace":[{"op":"print","module":"index.mq.md",
"span":{"start":43,"end":57},"duration_micros":3,"result":"ok"}]}
```
Note: `call --json` also emits `"command":"run"` in this build (shared
implementation). Consumers should accept both names or key on the presence of
`diagnostics`/`value`; this is a known quirk.
`run`/`call` (failure): `{"command":"run","success":false,"stdout":...,
"error":{"code":"...","message":"...","span":{"line":..,"column":..,"file":..},
"frames":[...]},"trace":[...]}`. Runtime error spans are currently resolved
against the **entry file** only, even when the error came from an imported
module; frames carry function names, not module ids. This is a known precision
limitation.
`plugin inspect`:
```json
{"command":"plugin.inspect","success":true,"plugin":"demo.solver",
"version":"0.1.0","abi":"^1.0","abi_compatibility":"compatible",
"target":"x86_64-unknown-linux-gnu",
"artifact":{"file":"solver.so","path":"...","sha256":"...","verified":true},
"exports":[{"name":"solver.add","signature":"($a: Int, $b: Int) -> Int"}],
"loaded":false}
```
`load_project` failures with `--json` print `{"success":false,"error":"..."}`
without a `command` field.
There is no `json_schema_version` field yet. The blueprint asks for a
versioned JSON schema; until it exists, treat these keys as the stable subset
and ignore unknown keys. Adding keys is not considered breaking; renaming or
removing one is.
### 5. `--arg` values
`--arg NAME=VALUE` parses `VALUE` with the **expression grammar restricted to
constants** (`crates/cli/src/args.rs`): integers, floats, double-quoted text
(escapes supported), `true`/`false`/`None`, lists, records, and unary `-`/`+`
over constants. Calls and identifiers are rejected ("must be a literal
constant"). Text must be double-quoted; there is no bare-word fallback.
### 6. Determinism and safety of read-only commands
* `check`/`inspect` never execute project code (ADR context; see
`spec/security.md` §7).
* `plugin inspect` never loads plugin code.
* `fmt` only rewrites the file when not `--check`; exit 1 when `--check`
detects a difference.
## Alternatives
1. **Use exit code 1 for all failures.** Rejected: scripts need to distinguish
usage errors (2) from document/runtime failures (1); it also mirrors common
CLI conventions and clap's own usage exit code.
2. **Print JSON even for usage errors.** Rejected for now: clap owns usage
output before our code runs; wrapping it would mean intercepting clap
errors. Documented as a limitation.
3. **Interleave program output and JSON.** Rejected: it makes JSON
unparseable. Program stdout is embedded as a string instead.
4. **Return values from `run` on stdout in text mode.** Rejected: `run` is the
batch entry; implicit value printing would pollute program output. `call`
prints the return value exactly when non-`Unit`.
5. **Human-only messages with numeric exit codes only.** Rejected: diagnostics
and runtime errors have stable machine codes in both text and JSON.
## Markdown AST shape
The CLI itself does not change the AST. What it exposes:
* `inspect --ast` = canonical `MdNode` tree (see ADR-0005);
* `inspect --units` = `UnitInfo[]` (`marqdo-ir`);
* `inspect --signature` = `{name, local_name, signature: explicit|inferred,
params:[{name,type,has_default}]}`;
* `inspect --ir` = `Function[]` with `Stmt`/`Expr` and spans;
* `inspect --refs` = link edge array (`spec/references.md` §4);
* `inspect --platform` = `PlatformCapabilities` JSON.
## Compatibility
* Exit codes, stream separation and the documented JSON keys are the public
CLI contract. Changing them requires this ADR to be superseded and a
changelog entry.
* The old "design-only starter" README listed planned commands (`add`,
`lock && sync`, `plugin inspect`); all listed commands except a versioned
JSON schema are now implemented as described here.
* Diagnostic code renames are breaking for CI; message changes are not.
* No package/ABI/wire changes.
## Security
* `--json`/`--trace` enable effect tracing; traces contain op/module/span/
duration/result summary, never argument values (no secrets) — see
`spec/security.md` §4.
* JSON mode embeds program stdout, which is program-controlled data; consumers
must not treat it as trusted.
* Refusing to run with compile errors happens before host creation, so JSON
failure output cannot leak effects.
* Strict-mode refusal and capability denials produce stable codes for CI.
## Testing
* CLI behavior is currently covered by manual smoke checks and library-level
tests; the fixtures workstream is adding CLI integration tests under
`crates/cli/tests` (goldens/negative cases) and `tests/`.
* `crates/cli/src/args.rs` unit tests cover typed `--arg` parsing and rejection
of non-constant values.
* `crates/cli/src/output.rs` has no unit tests; diagnostic JSON is exercised
through check/run smoke tests.
* Add contract tests for: exit codes per failure class, stdout/stderr
separation, JSON key presence, `call --json` command naming, and runtime
error span from a non-entry module before calling the schema "final".
## Migration & rollout
1. Keep the current shapes stable while adding tests.
2. Add a `schema_version` to JSON envelopes and document the versioned schema;
then document that unknown keys may be added (forward compatible) while
existing keys are frozen.
3. Fix the `call --json` `"command":"run"` quirk and JSON errors in
`add/remove/lock/sync`; both are additive/corrective and should be
announced in the changelog.
4. If a command is renamed or an exit code changes, update this ADR, the
README quickstart, `docs/STATUS.md` and CI scripts in the same change.
@@ -0,0 +1,178 @@
# ADR-0007: Arbitrary-precision `Int` and the typed wire encoding (S-08)
Status: **Accepted**
Date: 2026-11-20
## Context
The blueprint (§3.1 and unresolved issue **S-08**) requires a numeric model
that does not silently lose precision at language, JSON/ABI and math
boundaries. It recommends arbitrary-precision integers for `Int` and type tags
plus string forms for large integers on the wire.
Three independent places could lose information:
1. **Language arithmetic.** A fixed-width `i64` would make `Int` silently
overflow or wrap. Factorials, powers and scientific constants routinely
exceed 64 bits. The expression grammar already has arbitrary-length integer
literals (`spec/expression.ebnf`).
2. **JSON / wire.** JSON has one number type (IEEE-754 double in most parsers).
Encoding `12345678901234567890` as a JSON number loses digits in
JavaScript/Go consumers; encoding `1` as a number makes `Int` and `Float`
indistinguishable.
3. **Math kernel.** The exact-rational engine needs an integer type that is
the *same* type as `Value::Int`, or every formula/binding crossing the host
boundary needs a lossy conversion.
The chosen dependency set (`Cargo.toml`) is:
```toml
num-bigint = { version = "0.4", features = ["serde"] }
num-rational = { version = "0.4", features = ["serde"] }
num-traits = "0.2"
```
## Decision
### 1. `Int` is arbitrary precision (`num-bigint::BigInt`)
* `Value::Int(BigInt)` (`crates/marqdo-value/src/value.rs`) is the only integer
representation in the runtime and the expression AST
(`crates/marqdo-expression/src/ast.rs`).
* Expression integer literals are arbitrary-precision decimal; there is no
fixed-width parsing step.
* Arithmetic (`crates/marqdo-runtime/src/value_ops.rs`):
* `+ - *` on two `Int` are exact `BigInt` operations;
* `/` **always** produces `Float` (exact ratio converted to binary64), so
there is no integer division surprise;
* `//` and `%` require `Int` operands; floor division is mathematical floor
and `a % b == a - b * (a // b)`, with the sign following the divisor;
* mixed `Int`/`Float` arithmetic produces `Float`; converting an `Int`
larger than 2^53 may lose precision (documented, not silently exported as
an `Int`);
* `Int`/`Float` comparisons are **exact**: the float is treated as the exact
binary rational it is (`BigRational::from_float`), so
`9007199254740993 > 9007199254740992.0` is `true`;
* `**` on `Int` is exact exponentiation, with a negative exponent rejected
as a type error and guards `MAX_EXACT_EXPONENT = 100_000` and
`MAX_POWER_BITS = 1_000_000` (`resource.exhausted`).
* There is no truthiness and no implicit string↔number conversion; `整数`/`int`
performs explicit conversions and rejects non-integral floats.
### 2. `num-bigint` 0.4 is pinned to match `num-rational` 0.4
`num-rational`'s generic `Ratio<T>` is instantiated as
`Ratio<num_bigint::BigInt>` in `crates/marqdo-math/src/ast.rs`
(`pub type Rational = Ratio<BigInt>`). The two crates must resolve to the same
`num-bigint` version, otherwise `BigInt` values from the value model cannot be
passed to `Ratio` without conversion (or a duplicate type). The workspace pins
both at the `0.4` line and the lockfile resolves one `num-bigint`. Upgrading
either crate requires checking that this shared-type invariant still holds and
running the math + runtime test suites.
### 3. The wire encoding never sends an integer as a JSON number
Canonical encoding (`crates/marqdo-value/src/wire.rs`; full contract in
`spec/value-wire.md`):
```json
{"type":"int","decimal":"12345678901234567890"}
```
* Every value is a JSON object with a `"type"` tag.
* `Int` uses a base-10 string. All digits, sign and magnitude survive any JSON
consumer including JavaScript.
* `Float` uses a JSON number only when finite; `NaN`, `Infinity`,
`-Infinity` use explicit `"special"` tags.
* JSON numbers appear only in non-value positions (`span.start/end`,
`plugin_handle.id`).
* Decoding is validated: unknown tags, missing fields, invalid decimals, wrong
types and limits are `WireError`s, never coercions.
* The same encoding is used for plugin ABI request/response values, foreign
kernel stdin `input`, foreign typed return values and `run --json` values.
### 4. Math results preserve exactness where possible
The math kernel keeps exact rationals and returns `Value::Int` exactly when the
exact result is an integer, otherwise `Value::Float`. Decimal literals are
exact rationals (`0.25` = `1/4`), and `\frac{1}{3} * 3` is exactly `1`. Constants
(`\pi`, `\e`) and non-perfect square roots introduce binary64 approximations and
the result is a `Float` (`spec/math.md` §3).
## Alternatives
1. **`i64`/`u64` integers with overflow errors.** Rejected: common programs
(factorials, powers) would need explicit bignum workarounds; the language
would inherit fixed-width semantics not implied by its arithmetic.
2. **`i128` integers.** Same problem, just a bigger ceiling; and the wire
precision problem remains for values beyond 2^53.
3. **Encode integers as JSON numbers and hope consumers use bignum JSON.**
Rejected: the JSON standard/ecosystem does not guarantee it; JavaScript's
`JSON.parse` loses digits; this is the exact S-08 failure.
4. **Encode every number as a string without type tags.** Rejected: loses the
`Int`/`Float` distinction and makes all consumers parse strings; type tags
are explicit and cheap.
5. **Use `f64` for all numbers.** Rejected: contradicts exact `Int`
semantics, the math kernel's exact rational requirement, and the
no-silent-precision-loss rule.
6. **Different integer crates in value vs math.** Rejected: forces lossy or
string conversions at every kernel boundary and duplicates tests.
## Markdown AST shape
No AST change. Numeric literals live inside executable payloads only (code
spans/plain text of `strong`/`emph`/conditions). A `math` node is inert LaTeX;
its numeric interpretation is the math kernel's job. The JSON inspection view
of IR/AST contains spans as JSON numbers, not user numeric values.
## Compatibility
* `Int` semantics (arbitrary precision, `/` → `Float`, exact comparisons,
`//`/`%` sign rules) are frozen language behavior; changing them requires an
ADR and conformance fixtures.
* `wire_version = 1` with typed tags and decimal-string integers is frozen.
Changing the integer representation is a wire-breaking change and needs a
new `wire_version` plus plugin ABI considerations.
* The dependency pin may move within `0.4` for security fixes, provided
`num-bigint` remains a single resolved version shared with `num-rational`.
* Package/lock formats are unaffected.
## Security
* Untrusted wire data cannot trigger huge allocations through the integer
parser without also counting against value depth/string limits: a decimal
string is bounded by `max_string_bytes` (16 MiB default) and `BigInt` parsing
is linear. The current decoder does not separately cap the *digit count* of
an `Int`; the envelope/string caps are the effective bound (open item).
* Arithmetic can grow memory; the runtime bounds it with fuel and the
`Int` power guards (`MAX_EXACT_EXPONENT`, `MAX_POWER_BITS`) and the math
kernel bounds exact numerator/denominator bits (`MAX_RESULT_BITS`).
* Exact `Int`/`Float` comparison prevents a class of "looks equal but is not"
authorization or verification bugs.
* No new capabilities; numeric parsing is pure.
## Testing
* `crates/marqdo-value/src/value.rs`: `big_integers_are_arbitrary_precision`,
`float_to_bigint_requires_integral_value`, display/format tests.
* `crates/marqdo-value/src/wire.rs`: `big_integers_keep_all_digits`,
`special_floats_are_tagged`, scalar round trips, untrusted-input limits.
* `crates/marqdo-runtime/src/value_ops.rs`: exact integer arithmetic,
`/`→Float, floor/mod sign rules, exact mixed comparisons, power guards,
division-by-zero.
* `crates/marqdo-math/tests/kernel.rs`: exact big integer arithmetic, exact
rational division, decimal literals as rationals, `pi`/`sqrt` approximating
to `Float`, `Int` vs `Float` result typing.
* `crates/marqdo-plugin-abi/src/wire.rs`: request round trips preserving tags
and refusing coercible-looking values beyond `i64`.
* `crates/marqdo-foreign/tests/kernels.rs`: typed foreign return values decode
exactly.
## Migration & rollout
No migration: the implementation has used `BigInt` and the typed wire format
since M0.5. For future dependency upgrades: run `cargo test --workspace`,
verify one resolved `num-bigint` (`cargo tree -i num-bigint`), and re-run wire
compatibility tests. For any change to the wire integer encoding, bump
`wire_version`, update `spec/value-wire.md`, the plugin ABI documentation and
the foreign protocol in one coordinated change.
+286
View File
@@ -0,0 +1,286 @@
# Marqdo Contributor Guide
This guide is for people changing the Marqdo CLI workspace. It assumes you have
read `AGENTS.md` (the non-negotiable architectural rules) and
`docs/DEVELOPMENT_BLUEPRINT.md` §0/§1/§11/§13/§14. Formal contracts live in
`spec/`; decisions live in `docs/adr/`.
## 1. Repository map
```text
marqdo-cli/
├── Cargo.toml workspace (15 crates, resolver 2)
├── AGENTS.md operator rules for contributors/agents
├── README.md user-facing quickstart and limits
├── docs/
│ ├── DEVELOPMENT_BLUEPRINT.md architecture baseline
│ ├── STATUS.md implementation matrix (P0–P7)
│ ├── contributor-guide.md this file
│ └── adr/0001..0007-*.md accepted decisions
├── spec/
│ ├── markdown-profile.md frozen parser profile `marqdo.md.v1`
│ ├── mapping.md AST → program roles
│ ├── expression.ebnf executable payload grammar
│ ├── evaluation.md values, numbers, control flow, builtins
│ ├── references.md imports/references/calls/ref()
│ ├── value-wire.md typed wire_version 1 values
│ ├── security.md modes, capabilities, budgets, trust
│ ├── package-format.md marqdo.yaml / marqdo.lock / init
│ ├── plugin-abi.md C ABI v1, SDK, host lifecycle
│ ├── math.md Formula subset and math.* semantics
│ └── foreign.md Python/Node kernel protocol
├── crates/
│ ├── marqdo-span/ Span, LineCol, LineIndex (leaf)
│ ├── marqdo-value/ Value, diagnostics, errors, wire encoding
│ ├── marqdo-expression/ lexer/parser/AST for executable payloads
│ ├── marqdo-markdown/ comrak profile, canonical AST, front matter
│ ├── marqdo-ir/ Module/Unit/Function/Stmt IR, builtin table
│ ├── marqdo-frontend/ units, map, signature, infer, calls, resources
│ ├── marqdo-math/ Formula parser/evaluator (exact rationals)
│ ├── marqdo-foreign/ Python/Node one-shot kernel supervisor
│ ├── marqdo-sandbox/ modes + platform capability detection
│ ├── marqdo-plugin-abi/ C ABI mirrors, manifest, artifact, loader, wire
│ ├── marqdo-plugin-sdk/ Rust plugin SDK and `mq_plugin_entry!`
│ ├── marqdo-host/ CapabilityHost, budgets, FS, math/foreign/plugin adapters
│ ├── marqdo-runtime/ tree-walk interpreter, value ops, builtins
│ ├── marqdo-package/ marqdo.yaml / marqdo.lock types + discovery
│ └── cli/ composition root: flags, commands, loader, output
├── include/marqdo_plugin_abi_v1.h frozen C ABI v1 header
├── examples/ hello, factorial, math, references, python-foreign,
│ multi-module/, plugin-demo/
├── tests/
│ ├── conformance/ast/ *.mq.md + *.ast.json AST goldens
│ ├── conformance/semantics/ *.mq.md + *.expected.json execution goldens
│ ├── negative/ compile-time + runtime hard-error fixtures
│ ├── README.md corpus layout and golden workflow
│ ├── integration/ properties/ security/ abi/ reserved dirs (see STATUS)
└── crates/cli/tests/ Rust harnesses for the corpus and CLI
```
## 2. Dependency direction (must stay acyclic)
```text
span -> serde
value -> span
expression -> span
markdown -> span, comrak (no semantic crate!)
ir -> span, expression
frontend -> markdown, expression, ir, value
host -> value, math, foreign, sandbox
runtime -> ir, value, host, expression
math -> value
foreign -> value
plugin-abi -> value
plugin-sdk -> plugin-abi
package -> serde only
sandbox -> serde only
cli -> everything (composition root)
```
Rules (from blueprint §11.1, adjusted by ADR-0001):
* `markdown` owns the only Markdown parser; it must not depend on semantic
crates, the host or the runtime. The single exception to "zero Marqdo deps"
is `marqdo-span`, which is a dep-free leaf shared by all layers.
* `frontend`/`ir`/`runtime` never depend on `cli`.
* `runtime` never parses Markdown and never performs I/O directly: all effects
go through `marqdo_host::Host`.
* `host` never depends on the CLI; kernels/plugins are adapted here.
* The CLI is the composition root: flag parsing, project loading, rendering and
process exit. It must not become a second interpreter.
* Check the graph with `cargo tree` (or `cargo tree -i <crate>`); `runtime ->
cli`, `markdown -> frontend`, and dependency cycles are review blockers.
## 3. Adding a feature or fixture (workflow)
The project uses a fixture-first workflow (blueprint §13/§14, AGENTS §
"Development workflow"). The expected sequence is:
1. **Find the rule.** Identify the affected spec section and blueprint section.
If the change alters observable language semantics, package/lock format,
ABI, authorization defaults or public CLI behavior, write/supersede an ADR
first (`docs/adr/`, template in blueprint §16.2).
2. **Write a failing fixture.** Add a small `.mq.md` positive and/or negative
case under the right directory:
* AST shape: `tests/conformance/ast/NNN-name.mq.md` + golden
`NNN-name.ast.json`;
* semantics: `tests/conformance/semantics/NNN-name.mq.md` +
`NNN-name.expected.json` (typed wire `args`/`value` + captured `stdout`);
* hard error: `tests/negative/NNN-name.mq.md` + `{"codes":[...]}`;
* inert-executable warning: `tests/negative/warnings/`;
* runtime error: `tests/negative/runtime/` with `{"function","args","codes"}`
(optional `grants`, `stack_limit`).
Run the relevant harness (see §6) and confirm it fails for the expected
reason.
3. **Update the spec.** `spec/*.md` is normative. If implementation and spec
disagree, fix whichever is wrong and note the tradeoff. Do not silently
invent behavior in code and leave the spec stale.
4. **Implement the smallest module.** Keep the layers separated: parser
(`marqdo-markdown`), semantic mapping (`marqdo-frontend`), IR
(`marqdo-ir`), interpreter (`marqdo-runtime`), effects (`marqdo-host`),
composition (`cli`). Never add a second Markdown parser, a regex-based
syntax decision, or a host bypass.
5. **Refresh goldens deliberately** (§4) and run the full gate (§6).
6. **Write the tradeoff/limitations** in `docs/STATUS.md` when a P-level claim
changes, and in the PR description (paths, commands run, skipped tests).
If a fixture reveals an implementation bug, fix the code, not the expected
behavior. Negative fixtures must assert hard errors — never weaken them to
warnings or prose fallbacks.
## 4. Golden update workflow
AST goldens (`tests/conformance/ast/*.ast.json`) are the serialized canonical
AST from `marqdo_markdown::parse(source).root`. Regenerate **only** for an
intentional, ADR-backed profile change:
```bash
UPDATE_GOLDENS=1 cargo test -p marqdo-cli --test conformance_ast
```
The update run rewrites every golden, prints the count and passes. Re-run
without the variable and review the diff: **a golden change is a language
change, not a bug fix**. A fixture without a golden and a golden without a
fixture are both errors.
Semantic goldens (`*.expected.json`): `UPDATE_SEMANTICS_GOLDENS=1` exists for
authoring convenience, but never use it blindly — verify every value against the
fixture source first. Invalid, inconsistent or unverified expected values defeat
the corpus.
Negative goldens are hand-written expected code lists (exact sorted multiset for
compile-time errors; warning suites; runtime suites). Do not auto-generate them.
Adding/removing a fixture and its golden must happen in the same change.
Fixtures are data: tests must not commit, copy or rewrite files outside a
throw-away temp directory. `tests/README.md` is the in-repo reference for corpus
counts, harness responsibilities and known observations.
## 5. Coding conventions
* Rust edition 2021, `rust-version = 1.82`, workspace version `0.1.0`.
* `unsafe` is forbidden in pure crates (`marqdo-math`, `marqdo-expression` use
`#![forbid(unsafe_code)]`); the ABI/loader crates contain the only
`unsafe` code and every block has a SAFETY comment explaining the contract.
* No panics on untrusted input: Markdown, expressions, LaTeX, wire data,
manifests and foreign output all produce structured errors. `unwrap()`/
`expect()` are acceptable only for genuinely impossible invariants or in
tests.
* Stable diagnostic/runtime **codes** are part of the contract; human messages
are not. Add new codes to `crates/marqdo-value/src/diagnostic.rs` and map
categories in `error.rs`.
* Values cross layers via `marqdo_value::Value` and the typed wire format
(`spec/value-wire.md`); never serialize `Int` as a JSON number.
* `check`/`inspect` must remain side-effect free: no execution, no foreign
process, no plugin `dlopen`, no file writes.
* Prose and executable nodes mix freely; prose never causes an action.
* Formatting: `rustfmt.toml` is authoritative; keep `cargo fmt --check` clean.
## 6. Build, test and quality gate
The workspace is offline-cached. Use the workspace cache as documented in
`tests/README.md` / AGENTS.md:
```bash
export PATH=/root/.cargo/bin:$PATH
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
```
Commands:
```bash
cargo build -p marqdo-cli # binary: target/debug/marqdo
cargo test --workspace # all crate unit + integration tests
cargo test -p marqdo-cli # conformance corpus + CLI integration tests
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
```
Harness map (`crates/cli/tests/`):
| Harness | Proves |
|---|---|
| `conformance_ast.rs` | frozen profile parses fixtures into checked-in AST (kinds, attributes, literals, spans, order) |
| `conformance_semantics.rs` | frontend + IR + runtime + host produce expected wire value/stdout |
| `negative.rs` | invalid executable shapes are hard errors; warning and runtime suites |
| `properties.rs` | prose-invariance (pure narrative edits do not change executable IR modulo spans) |
| `integration_cli.rs` | real CLI contract: init/check/run/call/inspect/fmt, multi-module, lock/sync, plugin trust and invocation |
Crate-level tests worth knowing:
* `marqdo-expression` — grammar/priority/nesting/limit tests;
* `marqdo-math/tests/kernel.rs` — exact arithmetic, bounds, equations, unsupported constructs;
* `marqdo-foreign/tests/kernels.rs` — real python3/node (skips if absent), timeout, output caps, env, wire protocol;
* `marqdo-plugin-abi/tests/loader_unix.rs` — compiles the C demo with `cc`, loads it, ABI-mismatch/negative cases (skips without `cc`);
* `marqdo-runtime/tests/interpreter.rs` — recursion, loops, returns, budgets, ambiguity;
* `marqdo-host` unit tests — capabilities, project FS, budgets.
Tests that need Python/Node/`cc` print a skip notice and pass when unavailable;
record such skips in the PR.
## 7. Running examples
From the repository root (after `cargo build -p marqdo-cli`):
```bash
B=target/debug/marqdo
$B run examples/hello.mq.md
$B call examples/factorial.mq.md 数学.阶乘 --arg n=5 # prints 120
$B check examples/references.mq.md
$B inspect examples/references.mq.md --refs
$B inspect examples/hello.mq.md --ast --json
$B run examples/math.mq.md # has no main: use call
$B call examples/math.mq.md 数学.计算圆面积
$B run examples/python-foreign.mq.md --allow-foreign # needs python3
$B run examples/multi-module/index.mq.md
(cd examples/multi-module && ../../$B lock && ../../$B sync --locked)
# Plugin demo (build the artifact + refresh its sha256 first)
./examples/plugin-demo/build.sh
$B plugin inspect examples/plugin-demo
$B run examples/plugin-demo/plugin-call.mq.md \
--plugin examples/plugin-demo --trust-native-plugin --allow-plugin demo.solver
```
Notes: `run` defaults to `index.mq.md` and executes `main`; `call` requires a
file and function; root-only examples (for example `math.mq.md`) define no
`main`. `--mode strict` is expected to refuse to run (ADR-0004).
## 8. Docs, ADRs and commit policy
* `spec/` = normative contracts; `docs/` = architecture/decisions/status;
`README.md` = user quickstart; `AGENTS.md` = contributor rules.
* Changes to language semantics, AST mapping, wire format, plugin ABI,
package/lock format, authorization defaults or public CLI behavior require an
**Accepted ADR** (or superseding one) plus spec updates and tests. Use the
blueprint §16.2 template: Context / Decision / Alternatives / Markdown AST
shape / Compatibility / Security / Testing / Migration.
* ADRs are immutable once Accepted: add a new ADR with
`Status: Superseded by ADR-NNNN` rather than rewriting history (typo fixes
aside).
* Docs must describe what the code does. When implementation and blueprint
disagree, document the implementation and flag the divergence explicitly as
a limitation/open item; never invent behavior.
* Do not edit generated/vendored state, and never modify tests to hide a
regression.
* Commit in reviewable units with a message stating the affected spec/ADR and
the commands run. Do not commit built plugin artifacts (`*.so` is
gitignored), the cargo cache, or temp projects.
* There is no git-commit automation in this workspace; contributors commit
manually after review.
## 9. Known gotchas
* `.cache/` is the offline cargo cache/target used in this environment; it is
not part of the deliverable.
* Foreign availability (`python3`/`node`) and a C compiler (`cc`) determine
which tests run; a skipped test is not a passed feature.
* The strict sandbox, plugin worker isolation and package registry are not
implemented. Do not write docs or tests that pretend they are — see
`docs/STATUS.md` and `spec/security.md`.
* Runtime error spans in the CLI currently resolve against the entry file only;
imported-module error locations are approximate (ADR-0006).
* The semantic corpus may pin current quirks (for example the external-link
call code, or a float's last digit). A fixture diff for such a quirk is a
deliberate change, not a free update.