docs: formal specs, ADRs, STATUS matrix, contributor guide and honest README
Complete the documentation set for the M0.5 toolchain, grounded in the code: - spec/references.md: import/reference/call edges, identities, loader rules, anchors, `ref` resolution/ambiguity, `check --refs` validation. - spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal string rule, special floats, limits and exact float round-trip. - spec/security.md: modes (strict refuses without OS isolation), capabilities, path rules, budgets, tracing, foreign process caveats, plugin trust. - spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync, unsupported registry/git/environment work. - spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the implemented CLI plugin-invocation flow with its limitations. - spec/math.md and spec/foreign.md: the implemented math subset and the foreign request/response/timeout/capability contract. - docs/adr/0001..0007: accepted decisions for the span foundation, inert executables outside function bodies, ref ambiguity (S-07), strict refusal, AST normalization, CLI contract and BigInt/typed wire (S-08). - docs/contributor-guide.md: repo map, workflow, fixture/golden updates, quality gate, ADR/commit policy. - docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18 Alpha checklist (conclusion: not yet Alpha-complete), including known divergences rather than overclaims. - README.md rewritten from "design-only starter" to the working implementation with quickstart, implemented phases, deliberate limits and doc index. - AGENTS.md: updated pointers/workflow only; architectural rules unchanged. - Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside functions, unsupported math.simplify/differentiate).
This commit is contained in:
1 parent
fae0cd4b18
commit
0116160399
19 files changed
+3767
-30
No files matched your search
@@ -1,6 +1,6 @@
|
||||
# Marqdo CLI — Agent / Contributor Operating Rules
|
||||
|
||||
**Start here:** read `docs/DEVELOPMENT_BLUEPRINT.md` before proposing code. It is the project architecture baseline. Then read the relevant `spec/` files and conformance fixtures. Do not assume any current implementation or legacy Marqdo syntax is authoritative.
|
||||
**Start here:** read `docs/DEVELOPMENT_BLUEPRINT.md` before proposing code. It is the project architecture baseline. Then read `docs/STATUS.md` (what is actually implemented), the relevant `spec/` files, `docs/adr/` for frozen decisions, `docs/contributor-guide.md` for the workflow, and `tests/README.md` plus the conformance fixtures. Do not assume any current implementation or legacy Marqdo syntax is authoritative.
|
||||
|
||||
## Non-negotiable architectural constraints
|
||||
|
||||
@@ -20,8 +20,8 @@
|
||||
1. Identify the affected rule and cite the section of `docs/DEVELOPMENT_BLUEPRINT.md`.
|
||||
2. Add a small `.mq.md` positive/negative conformance fixture and define the expected Markdown AST and execution IR.
|
||||
3. Implement only the smallest module that satisfies this fixture. Keep Markdown parser, semantic mapper, runtime and host separated.
|
||||
4. Run formatting, clippy, unit tests, gold tests and relevant security/ABI compatibility tests.
|
||||
5. Write the tradeoff and state of completion in the PR. Never say a feature is finished because a design example looks executable.
|
||||
4. Run formatting, clippy, unit tests, gold tests and relevant security/ABI compatibility tests (`cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`, `cargo test -p marqdo-cli`).
|
||||
5. Write the tradeoff and state of completion in the PR. Never say a feature is finished because a design example looks executable. Update `spec/` and `docs/STATUS.md` when behavior or status changes; the docs describe what the code does and flag blueprint divergences explicitly.
|
||||
|
||||
## Unresolved design topics
|
||||
|
||||
|
||||
@@ -1,39 +1,203 @@
|
||||
# Marqdo CLI — next-generation design starter
|
||||
# Marqdo CLI
|
||||
|
||||
> **Design-only starter package. This archive does not contain an executable CLI or a completed Rust implementation.**
|
||||
Marqdo is a **Markdown-native, independently interpreted programming language**:
|
||||
standard CommonMark/GFM structures carry program control (headings scope,
|
||||
`**strong**` executes, `*emph*` returns, ordered lists branch, bullet lists
|
||||
loop, block quotes call), prose and executable nodes mix in one document, and
|
||||
all side effects go through a capability-checked host boundary. The official
|
||||
implementation is a Rust workspace; Python/Node are optional foreign kernels,
|
||||
never the compiler target.
|
||||
|
||||
Marqdo aims to be a **Markdown-native, independently interpreted programming language**: Markdown AST structures carry program control semantics; readable prose can coexist with execution, while runtime effects are explicit and controlled. The new toolchain is independent of the original `cflmy/marqdo` repository.
|
||||
This repository is the new, independent Marqdo toolchain (M0.5). It is **not**
|
||||
a compatibility rewrite of `cflmy/marqdo`; the old syntax, package format and
|
||||
plugin ABI v2 are not claimed to work here.
|
||||
|
||||
## Read first
|
||||
> **Status: working implementation, not an Alpha release.** P0/P1 and the
|
||||
> documented Math/Foreign subset are implemented and tested; modules/package/
|
||||
> security/plugin support is functional but partial; strict isolation, worker
|
||||
> plugins, git packages and performance/release work are not done. See
|
||||
> [`docs/STATUS.md`](docs/STATUS.md) and "Deliberate limits" below.
|
||||
|
||||
- [`docs/DEVELOPMENT_BLUEPRINT.md`](docs/DEVELOPMENT_BLUEPRINT.md) — the comprehensive Chinese design and development specification: motivation, formal mapping, functions/parameters/calls/returns, execution, references, Math/Foreign kernels, environment and package manager, sandbox, plugin ABI/SDK, milestones, tests and related projects.
|
||||
- [`AGENTS.md`](AGENTS.md) — succinct onboarding instructions for coding agents and new contributors.
|
||||
## Build
|
||||
|
||||
## Planned CLI surface
|
||||
Requires Rust 1.82+.
|
||||
|
||||
```bash
|
||||
marqdo init
|
||||
marqdo run index.mq.md
|
||||
marqdo call index.mq.md main
|
||||
marqdo check index.mq.md
|
||||
marqdo inspect index.mq.md --ast
|
||||
marqdo add <package>
|
||||
marqdo lock && marqdo sync
|
||||
marqdo plugin inspect <artifact>
|
||||
cargo build -p marqdo-cli
|
||||
# binary: target/debug/marqdo (or $CARGO_TARGET_DIR/debug/marqdo)
|
||||
```
|
||||
|
||||
These commands are **planned APIs**, not yet available in this starter package. Begin by implementing P0 tests and a reference Markdown AST explorer; do not start with a monolithic CLI or port old parser heuristics.
|
||||
## Quickstart
|
||||
|
||||
## Design status
|
||||
```bash
|
||||
marqdo init demo
|
||||
cd demo
|
||||
marqdo check index.mq.md
|
||||
marqdo run index.mq.md # executes ## main
|
||||
marqdo inspect index.mq.md --ast
|
||||
marqdo inspect index.mq.md --signature
|
||||
marqdo call index.mq.md main
|
||||
```
|
||||
|
||||
The language direction and architecture principles have been selected. Detailed proposals still require ADRs and conformance testing before a stable version. See blueprint section 17 for unresolved items and section 18 for the Alpha definition of done.
|
||||
Example project from scratch (`index.mq.md`):
|
||||
|
||||
````markdown
|
||||
# 数学
|
||||
|
||||
## 阶乘
|
||||
|
||||
1. `n <= 1`
|
||||
|
||||
*1*
|
||||
|
||||
2. `n > 1`
|
||||
|
||||
**结果 = n * 阶乘(n=n-1)**
|
||||
|
||||
*结果*
|
||||
````
|
||||
|
||||
```bash
|
||||
marqdo check examples/factorial.mq.md
|
||||
marqdo call examples/factorial.mq.md 数学.阶乘 --arg n=5 # 120
|
||||
```
|
||||
|
||||
The parser is frozen: `marqdo.md.v1` = CommonMark 0.31.2 + GFM tables,
|
||||
strikethrough, task lists + `$...$`/`$$...$$` math (comrak `=1.0.0`). A `.mq.md`
|
||||
file still renders normally in any compatible Markdown reader; rendering never
|
||||
executes anything.
|
||||
|
||||
## What works (implemented and tested)
|
||||
|
||||
* **P0 contracts** — frozen Markdown profile, AST → program mapping, strict
|
||||
expression grammar, evaluation semantics; 150 AST goldens and 48 negative
|
||||
compile-error fixtures (`tests/`).
|
||||
* **P1 interpreter** — native Rust tree-walk interpreter: values, scopes,
|
||||
functions, static parameter inference, assignments, branches, loops,
|
||||
returns, recursion, structured errors, fuel/stack/output budgets.
|
||||
* **P2 references + CLI** — explicit YAML front-matter imports, `ModuleId`/
|
||||
`UnitId`/anchors, import cycle detection, multi-module calls,
|
||||
`run/call/check/inspect/fmt`, reference validation (`check --refs`) and the
|
||||
static link graph (`inspect --refs`).
|
||||
* **P3 project** — `marqdo.yaml` validation, deterministic `marqdo.lock` for
|
||||
path dependencies, `init`, `add`/`remove` (manifest editing), `lock`,
|
||||
`sync --locked`.
|
||||
* **P4 security core** — capability-checked host operations, project-relative
|
||||
FS globs with traversal rejection, budgets, effect trace, `pure`/`project`
|
||||
modes, and **`strict` mode refuses to run** because no OS isolation adapter
|
||||
exists.
|
||||
* **P5 Math/Foreign** — native `Formula` value from `$$...$$`; `math.parse/
|
||||
eval/substitute` over a documented exact-rational subset; one-shot Python/
|
||||
Node kernels via `foreign.run/call` with typed wire protocol, timeout and
|
||||
output limits.
|
||||
* **P6 plugin ABI** — frozen C ABI v1 header (`include/marqdo_plugin_abi_v1.h`),
|
||||
host bindings, Rust SDK (`mq_plugin_entry!`), `plugin.yaml` validation,
|
||||
target-triple + sha256 artifact verification, C demo, and explicitly trusted
|
||||
in-process plugin calls (`--plugin --trust-native-plugin --allow-plugin`).
|
||||
|
||||
Run everything:
|
||||
|
||||
```bash
|
||||
cargo test --workspace
|
||||
cargo test -p marqdo-cli # conformance corpus + CLI contract tests
|
||||
cargo fmt --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
## Deliberate limits (do not assume)
|
||||
|
||||
* **Strict sandbox is not implemented.** `--mode strict` exits 1; there is no
|
||||
OS isolation or isolated plugin worker. `project` mode is cooperative and
|
||||
in-process.
|
||||
* **Native plugins run in-process** and require an explicit unsafe trust flag.
|
||||
A plugin can crash or read the host; a C ABI is not a sandbox.
|
||||
* **Package management is path-only in practice.** Git dependencies can be
|
||||
declared but cannot be resolved/locked; there is no registry, transitive
|
||||
resolution, `--offline`, `tree`, `update` or environment materialization.
|
||||
Manifest `capabilities` are requests, never grants.
|
||||
* **Reference validation is opt-in**: `check --refs` reports broken links;
|
||||
plain `check` ignores them because links are static knowledge references.
|
||||
Link-form calls cannot cross modules; cross-module calls use imported alias
|
||||
expressions such as `geometry.圆面积(r=2)`.
|
||||
* **`ref("#anchor")` is module-local** and errors with `ref.ambiguous` when a
|
||||
heading range contains more than one formula/code block instead of picking
|
||||
the first.
|
||||
* **`fmt` is minimal** (line endings/trailing newlines), not AST-preserving
|
||||
reformatting.
|
||||
* **Math is a limited calculator**, not a CAS/prover: no `simplify`/
|
||||
`differentiate`/matrices/series; equation solving is affine-only.
|
||||
* **Foreign kernels are unsandboxed one-shot child processes** (no sessions,
|
||||
no state bridge); they run with the host user's filesystem/network access.
|
||||
* Finite floats are IEEE-754 binary64; the typed wire round-trip is exact
|
||||
(`serde_json/float_roundtrip`), tagged integers never lose digits, and
|
||||
display formatting is decimal rather than bit-identical.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```text
|
||||
crates/marqdo-span shared byte spans/source maps (leaf)
|
||||
crates/marqdo-value Value model, diagnostics, typed wire encoding
|
||||
crates/marqdo-expression strict expression/statement parser
|
||||
crates/marqdo-markdown frozen comrak profile, canonical AST, front matter
|
||||
crates/marqdo-ir module/unit/function/statement IR, builtin table
|
||||
crates/marqdo-frontend heading scopes, mapping, inference, static calls
|
||||
crates/marqdo-math exact-rational Formula subset
|
||||
crates/marqdo-foreign Python/Node one-shot kernel supervisor
|
||||
crates/marqdo-sandbox mode parsing + platform isolation detection
|
||||
crates/marqdo-plugin-abi C ABI v1 mirrors, manifest, artifacts, loader
|
||||
crates/marqdo-plugin-sdk Rust plugin SDK and entry macro
|
||||
crates/marqdo-host CapabilityHost, budgets, FS, math/foreign/plugin adapters
|
||||
crates/marqdo-runtime tree-walk interpreter
|
||||
crates/marqdo-package marqdo.yaml / marqdo.lock / root discovery
|
||||
crates/cli CLI composition root (binary `marqdo`)
|
||||
include/ frozen plugin C ABI header
|
||||
spec/ normative language/format/ABI specs
|
||||
docs/ blueprint, ADRs, status, contributor guide
|
||||
examples/ runnable documents + multi-module + plugin demo
|
||||
tests/ conformance/negative corpus (see tests/README.md)
|
||||
```
|
||||
|
||||
## Documentation index
|
||||
|
||||
* [`AGENTS.md`](AGENTS.md) — operating rules for contributors and coding agents.
|
||||
* [`docs/DEVELOPMENT_BLUEPRINT.md`](docs/DEVELOPMENT_BLUEPRINT.md) — architecture
|
||||
baseline (Chinese; the authoritative design document).
|
||||
* [`docs/contributor-guide.md`](docs/contributor-guide.md) — repo map, workflow,
|
||||
golden updates, quality gate.
|
||||
* [`docs/STATUS.md`](docs/STATUS.md) — P0–P7 implementation matrix and honest
|
||||
open items; blueprint §18 Alpha checklist.
|
||||
* [`docs/adr/`](docs/adr/) — accepted decisions (crate graph, inert executables,
|
||||
`ref` ambiguity, strict refusal, AST normalization, CLI contract, BigInt/wire).
|
||||
* Specs: [`markdown-profile.md`](spec/markdown-profile.md),
|
||||
[`mapping.md`](spec/mapping.md), [`expression.ebnf`](spec/expression.ebnf),
|
||||
[`evaluation.md`](spec/evaluation.md), [`references.md`](spec/references.md),
|
||||
[`value-wire.md`](spec/value-wire.md), [`security.md`](spec/security.md),
|
||||
[`package-format.md`](spec/package-format.md),
|
||||
[`plugin-abi.md`](spec/plugin-abi.md), [`math.md`](spec/math.md),
|
||||
[`foreign.md`](spec/foreign.md).
|
||||
* [`tests/README.md`](tests/README.md) — conformance corpus and `UPDATE_GOLDENS`
|
||||
workflow.
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
marqdo run examples/hello.mq.md
|
||||
marqdo call examples/factorial.mq.md 数学.阶乘 --arg n=5
|
||||
marqdo run examples/multi-module/index.mq.md
|
||||
marqdo run examples/python-foreign.mq.md --allow-foreign # needs python3 or node
|
||||
marqdo inspect examples/references.mq.md --refs
|
||||
marqdo inspect examples/hello.mq.md --platform # strict: unavailable
|
||||
|
||||
./examples/plugin-demo/build.sh
|
||||
marqdo plugin inspect examples/plugin-demo
|
||||
marqdo run examples/plugin-demo/plugin-call.mq.md \
|
||||
--plugin examples/plugin-demo --trust-native-plugin --allow-plugin demo.solver
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [CommonMark](https://spec.commonmark.org/0.31.2/)
|
||||
- [GFM](https://github.github.io/gfm/)
|
||||
- [Comrak](https://comrak.ee/)
|
||||
- [Cargo](https://doc.rust-lang.org/cargo/)
|
||||
- [uv](https://docs.astral.sh/uv/)
|
||||
- [Wasmtime Component Model](https://component-model.bytecodealliance.org/)
|
||||
- [Legacy Marqdo source](https://github.com/cflmy/marqdo)
|
||||
* [CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/) ·
|
||||
[GFM](https://github.github.io/gfm/) · [Comrak](https://comrak.ee/)
|
||||
* [Cargo](https://doc.rust-lang.org/cargo/) · [uv](https://docs.astral.sh/uv/)
|
||||
* [Wasmtime Component Model](https://component-model.bytecodealliance.org/)
|
||||
* Legacy Marqdo source (reference only): <https://github.com/cflmy/marqdo>
|
||||
+217
@@ -0,0 +1,217 @@
|
||||
# Marqdo Implementation Status (M0.5)
|
||||
|
||||
Snapshot: **2026-11-20**, workspace version `0.1.0`, plugin ABI `1.0`,
|
||||
`wire_version` 1, Markdown profile `marqdo.md.v1`.
|
||||
|
||||
This file maps the blueprint's phases (`docs/DEVELOPMENT_BLUEPRINT.md` §13) to
|
||||
concrete evidence in the repository and lists honest open items. It is a status
|
||||
report, not a promise. "Done" means implemented **for the documented subset**
|
||||
and covered by tests; it does not mean the blueprint's full ambition is
|
||||
complete.
|
||||
|
||||
Reproduce the evidence:
|
||||
|
||||
```bash
|
||||
export PATH=/root/.cargo/bin:$PATH
|
||||
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
|
||||
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
|
||||
cargo test --workspace
|
||||
cargo test -p marqdo-cli
|
||||
cargo fmt --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
## 1. Phase matrix
|
||||
|
||||
| Phase | Status | Main evidence |
|
||||
|---|---|---|
|
||||
| **P0** Formal specs + parser goldens | **Done** | `spec/markdown-profile.md`, `spec/mapping.md`, `spec/expression.ebnf`, `spec/evaluation.md`; frozen profile test `crates/marqdo-markdown/src/profile.rs`; 150 AST fixtures + goldens in `tests/conformance/ast/`; 48 compile-error fixtures in `tests/negative/`; `inspect --ast`; `conformance_ast.rs`/`negative.rs` harnesses. |
|
||||
| **P1** Interpreter, expressions, control flow | **Done** | `marqdo-expression`, `marqdo-ir`, `marqdo-frontend`, `marqdo-runtime`; factorial/countdown examples run without Python (`examples/factorial.mq.md`, `examples/hello.mq.md`); 36 execution goldens in `tests/conformance/semantics/`; runtime tests `crates/marqdo-runtime/tests/interpreter.rs`; prose invariance `crates/cli/tests/properties.rs`. |
|
||||
| **P2** Modules, references, CLI experience | **Partial** | Imports/ModuleId/UnitId/anchor/cycle detection + `run/call/check/inspect/fmt` (`crates/cli`), multi-module example and `multi_module_imports_and_cycles` CLI test; `check --refs` reference validation (all loaded documents, on-demand static analysis of link targets) and `inspect --refs` static graph. Missing: cross-module link calls, rename/interface-lock tooling, canonical AST-preserving `fmt` (see §3). |
|
||||
| **P3** Package environment | **Partial** | `marqdo.yaml`/`marqdo.lock` types+validation (`marqdo-package`), `init`, `add`/`remove` (manifest editing), deterministic `lock`, `sync --locked`; CLI test `lock_and_sync_locked_detect_dependency_tampering`; `examples/multi-module`. Missing: git resolution, transitive deps, registry, `tree/update/vendor`, `--offline`, environment materialization. |
|
||||
| **P4** Security core and isolation | **Partial** | `CapabilityHost`, capabilities at execution points (console and math included), project-rooted FS with traversal rejection, budgets, effect trace, pure/project modes, **strict refuses without OS isolation** (`marqdo-sandbox`); CLI test `strict_mode_refuses_without_isolation`; host unit tests. Missing: OS isolation adapter, plugin worker, network enforcement. |
|
||||
| **P5** Math and Foreign | **Partial (documented subset)** | `marqdo-math` exact-rational formula kernel for the grammar in `spec/math.md`; `math.parse/eval/substitute` host ops; `marqdo-foreign` Python/Node one-shot kernels with typed `foreign.call`; tests `crates/marqdo-math/tests/kernel.rs`, `crates/marqdo-foreign/tests/kernels.rs`; examples `math.mq.md`, `python-foreign.mq.md`. Missing: `simplify`/`differentiate`, matrices/series/symbolic algebra, foreign sessions, Python environment management. |
|
||||
| **P6** Plugin ABI and developer experience | **Partial** | Frozen `include/marqdo_plugin_abi_v1.h`; C ABI mirrors/layout assertions, version negotiation, `plugin.yaml` validation, target triple + sha256 verification, wire envelopes, native loader (`marqdo-plugin-abi`); Rust SDK macro (`marqdo-plugin-sdk`); C demo (`examples/plugin-demo`); CLI `plugin inspect` and explicitly trusted in-process invocation (`--plugin`, `--trust-native-plugin`, `--allow-plugin`); tests `plugin_inspect_and_explicit_trusted_invocation`, `crates/marqdo-plugin-abi/tests/loader_unix.rs`. Missing: isolated worker transport, plugin capability bridge, installation, Rust plugin example crate. |
|
||||
| **P7** Performance, release, CI | **Partial** | `.github/workflows/ci.yml` runs fmt/clippy/workspace tests plus CLI example smoke tests (including plugin inspect and invocation) on Linux x86_64. Missing: benchmarks, bytecode backend, cross-platform matrix, signed releases and supply-chain checks. Tree-walk interpreter only. |
|
||||
|
||||
## 2. Feature evidence (selected commands)
|
||||
|
||||
| Command | Verified behavior |
|
||||
|---|---|
|
||||
| `marqdo init demo` | creates `marqdo.yaml` (`>=0.1,<1`), `index.mq.md`, `.gitignore` (`.marqdo/`); refuses existing manifest (exit 1). |
|
||||
| `marqdo check index.mq.md` | static only; 0 when no error diagnostics (warnings allowed); 1 on errors; JSON `command/success/errors/diagnostics`. |
|
||||
| `marqdo run index.mq.md` | executes `main` through the Rust interpreter/host; program stdout on stdout; exit 1 on runtime/compile/strict errors. |
|
||||
| `marqdo call file fn --arg n=5` | typed constant arguments; prints the non-`Unit` return value. |
|
||||
| `marqdo inspect file --ast/--units/--signature/--ir/--refs/--platform --json` | static dashboards; `--platform` reports strict unavailable. |
|
||||
| `marqdo fmt [--check]` | normalizes CRLF and trailing newlines only; exit 1 if `--check` detects a change. |
|
||||
| `marqdo plugin inspect dir` | validates manifest + verifies sha256; never `dlopen`s (`loaded: false`). |
|
||||
| `marqdo run doc --plugin dir --trust-native-plugin --allow-plugin demo.solver` | calls `solver.add(a=20,b=22)` and prints `42`; without trust/grant it exits 1. |
|
||||
| `marqdo lock` / `sync --locked` | deterministic path-dependency lock; `--locked` fails on manifest/dependency drift. |
|
||||
|
||||
Test corpus at this snapshot (`tests/README.md`):
|
||||
|
||||
* 150 AST fixtures + goldens (`tests/conformance/ast/`);
|
||||
* 36 semantic execution fixtures + goldens (`tests/conformance/semantics/`);
|
||||
* 48 compile-time hard-error fixtures, 6 inert-executable warning fixtures,
|
||||
10 runtime-error fixtures (`tests/negative/`);
|
||||
* `crates/cli/tests/` harnesses: `conformance_ast`, `conformance_semantics`,
|
||||
`negative`, `properties`, `integration_cli`;
|
||||
* crate tests across all 15 crates, including real `python3`/`node` foreign
|
||||
tests (skipped if unavailable) and a compiled C plugin test (skipped without
|
||||
`cc`).
|
||||
|
||||
At this snapshot `cargo test -p marqdo-cli` passes all 21 harness tests
|
||||
(2 CLI unit + AST/semantics/negative/properties/integration) and
|
||||
`cargo test --workspace` passes. The fixture counts above match
|
||||
`tests/README.md`; empty `tests/integration`, `tests/properties`,
|
||||
`tests/security` and `tests/abi` directories are reserved for future
|
||||
workstreams — the active corpus lives in `tests/conformance`, `tests/negative`
|
||||
and `crates/cli/tests`.
|
||||
|
||||
## 3. Open items by phase
|
||||
|
||||
### P0
|
||||
|
||||
* Wire floats now round-trip exactly: `serde_json`'s `float_roundtrip`
|
||||
feature is enabled and a Python `math.pi * 9` fixture pins the exact value
|
||||
(`spec/value-wire.md` §3).
|
||||
* Profile change safety relies on the frozen-options unit test; no CI gate
|
||||
regenerates goldens (intentional).
|
||||
|
||||
### P1
|
||||
|
||||
* The blueprint asks for a draft Turing-completeness argument. The language
|
||||
supports mutable locals, branch-first-match, per-item `while` loops and
|
||||
recursion with bounded budgets; a formal encoding write-up is still missing.
|
||||
* The default cooperative stack budget is 128 frames so very deep recursion
|
||||
yields a structured `resource.exhausted` instead of exhausting the debug-build
|
||||
native stack; `--stack` raises it deliberately. A true guard-page/iterative
|
||||
interpreter hardening is still open.
|
||||
|
||||
### P2
|
||||
|
||||
* `fmt` is intentionally minimal (line endings/trailing newlines only), not the
|
||||
blueprint's "canonical AST-equivalent formatting"; no semantic formatting,
|
||||
AST-equivalence check or render-preserving rewrite exists.
|
||||
* Reference validation is opt-in: `check --refs` (and `inspect --refs`) resolve
|
||||
links; plain `check` does not fail on broken links because links are static
|
||||
knowledge references rather than execution inputs.
|
||||
* Link-form calls cannot cross modules; cross-module calls use imported alias
|
||||
expressions. The blueprint's `> [f](./other.mq.md#f)(...)` example is not
|
||||
supported.
|
||||
* `ref("#...")` is module-local and does not accept a selector for units with
|
||||
multiple resources (it errors with `ref.ambiguous`, ADR-0003).
|
||||
* No rename tooling, interface lock/ABI diffing, or incremental cache.
|
||||
* Duplicate/nested project boundaries are not diagnosed (nearest
|
||||
`marqdo.yaml` wins).
|
||||
|
||||
### P3
|
||||
|
||||
* Git dependencies can be declared (`git` + `rev`, validated) but cannot be
|
||||
resolved or locked: `lock` errors "git dependencies are not supported by this
|
||||
build".
|
||||
* No transitive dependency graph, conflict resolution, registry, publishing,
|
||||
signing, `tree`, `update`, `vendor`, `--offline`, or `lock --locked`.
|
||||
* No environment materialization (`.marqdo/env`, caches, kernels, sessions);
|
||||
`sync` verifies but never installs or rewrites.
|
||||
* `add`/`remove` are shallow manifest editors; adding a git dependency succeeds
|
||||
at the manifest level even though `lock` will later refuse it.
|
||||
* JSON errors from `add`/`remove`/`lock`/`sync` are plain stderr even with
|
||||
`--json`.
|
||||
|
||||
### P4
|
||||
|
||||
* **Strict mode cannot run at all**: no OS isolation adapter (Linux
|
||||
namespaces/seccomp/cgroups, Windows Job Object/appcontainer, macOS sandbox)
|
||||
and `worker_processes: false`; `run --mode strict` exits 1 (ADR-0004).
|
||||
* `console` capability is always on; `print` is not capability-checked.
|
||||
* Network capability (`net_hosts`, `check_net`) exists but no host operation
|
||||
performs network I/O and there is no `--allow-net` flag.
|
||||
* Write-path symlink TOCTOU caveat (`spec/security.md` §2.1); symlink/race
|
||||
hardening open.
|
||||
* Budgets do not cover every allocation path; the interpreter's containers are
|
||||
bounded only by fuel and available memory.
|
||||
* Effect trace does not record argument values or paths (privacy-positive, but
|
||||
less diagnostic detail); trace is opt-in and implicitly on for `--json`.
|
||||
* `plugin_handle` wire values are structurally reconstructable, not
|
||||
unforgeable capability tokens.
|
||||
|
||||
### P5
|
||||
|
||||
* Math is a deliberately limited calculator subset: no `simplify`,
|
||||
`differentiate`, matrices, sums/integrals, trig/log functions, units,
|
||||
assumptions or theorem proving. `math.simplify`/`math.differentiate` are
|
||||
host ops that always return `math.unsupported`.
|
||||
* Equation solving is affine-only, single target, target must occur exactly
|
||||
once; other shapes are `math.unsupported`.
|
||||
* Division by zero inside a formula maps to `runtime.type` (via
|
||||
`MathErrorKind::Type`), not `runtime.division_by_zero`; documented divergence.
|
||||
* Foreign kernels are one-shot only: no sessions, state bridge, Python/Node
|
||||
environment management, or network policy for child processes.
|
||||
* Foreign child processes are terminated only when the direct child is killed;
|
||||
grandchildren are not tracked.
|
||||
* Foreign per-call timeout (10 s) and output cap (1 MiB) are not CLI-configurable.
|
||||
|
||||
### P6
|
||||
|
||||
* No isolated plugin worker (length-prefixed IPC) or Wasm transport; only
|
||||
explicitly trusted in-process native loading. A native plugin crash is a host
|
||||
crash (`spec/plugin-abi.md` §1/§9).
|
||||
* No plugin capability bridge from the CLI: plugins are loaded with no
|
||||
capability handler, so every plugin `request_capability` is denied.
|
||||
* Plugin calls are not statically validated; unknown namespaces are runtime
|
||||
`symbol.unknown`. Positional plugin arguments are rejected.
|
||||
* One namespace per plugin (first export prefix); multi-prefix manifests are not
|
||||
reachable beyond the first prefix.
|
||||
* No plugin installation/package management; `plugins:` in `marqdo.yaml` is
|
||||
declarative only. No Rust plugin example crate (SDK has doc examples/tests).
|
||||
* No hot unload/reload (ABI v1 design).
|
||||
|
||||
### P7
|
||||
|
||||
* Performance work is untouched: no benchmark suite, no bytecode backend or
|
||||
differential tests against the tree-walker, no parse/compile caching.
|
||||
* No CI workflows (`.github/` is empty of workflow files); no
|
||||
Windows/macOS/Linux matrix, sanitizers/fuzz gates, signing, release
|
||||
automation or supply-chain scanning.
|
||||
* No published version, no compatibility matrix beyond the frozen ABI/format
|
||||
documents.
|
||||
|
||||
## 4. Blueprint §18 "Alpha Definition of Done" mapping
|
||||
|
||||
| §18 item | Status | Evidence / gap |
|
||||
|---|---|---|
|
||||
| Run/call non-trivial recursive, branch, loop, call, parameter-inference programs without Python | **Met** | factorial/recursion (`examples/factorial.mq.md`, runtime + semantic goldens), countdown loop, multi-module alias call; interpreter is native Rust. |
|
||||
| `.mq.md` readable under the chosen CommonMark/GFM/Math profile; standard AST behavior covered by goldens | **Met** | frozen `marqdo.md.v1`, 150 AST goldens, profile option test, `spec/markdown-profile.md`. |
|
||||
| No heuristic downgrade of execution semantics; accurate source locations for invalid execution nodes | **Mostly met** | hard errors with spans for function-body executable shapes and negative corpus. Gaps: inert executable outside functions is a warning by design (ADR-0002); cross-module runtime error spans resolve against the entry file only; some lazy nested nodes are inert without a warning. |
|
||||
| `check`/`inspect` side-effect free; imports, calls and static references verifiable across files | **Mostly met** | static load/analysis only; imports + cycle detection; cross-module calls validated; `check --refs` validates references across loaded documents and statically analyses link targets; `inspect --refs` prints the entry graph. |
|
||||
| Minimal manifest, lock file, path/Git dependencies and environment sync | **Partial** | manifest/lock/path works, `sync --locked` verifies; git resolution, transitive deps, environment materialization and `--offline` missing. |
|
||||
| All host side effects capability-authorized; untrusted code/native plugins cannot masquerade as strict sandbox | **Mostly met** | capability checks at execution points; strict refuses; native plugins require explicit trust and are documented as unsandboxed. Gaps: network enforcement not implemented (no network builtin is exposed), no worker isolation. |
|
||||
| Formula computational subset explicit/testable; Foreign I/O, timeout and permissions explicit | **Met for the documented subset** | `spec/math.md`, `spec/foreign.md`; math kernel tests; foreign tests (timeout, protocol, output cap, env); permission gating tests/smoke. Simplification/differentiation explicitly unsupported. |
|
||||
| Plugin C ABI v1, Rust SDK, C example, isolated worker basic path, ABI compat error tests | **Partial** | ABI/SDK/C example/loader/ABI-error tests exist; **isolated worker path missing**. |
|
||||
| CI has at least one full research pipeline: Markdown function → Math → Foreign → Plugin → controlled output | **Partial** | CI now runs fmt/clippy/tests and CLI smoke tests; individual math/foreign/plugin paths are covered by examples and tests, but no single end-to-end pipeline document/test exists yet. |
|
||||
| Examples, README, AGENTS and contributor docs allow a first change without legacy context | **Met** | `README.md`, `AGENTS.md`, this guide, `docs/STATUS.md`, specs, ADRs, `tests/README.md`, examples. |
|
||||
|
||||
**Conclusion: the Alpha definition of done is not fully met.** P0/P1 and the
|
||||
documented P5 subset are solid; P2/P3/P4/P6 are functional but partial; P7 is
|
||||
absent. The blocking items for an Alpha claim are the strict sandbox/worker
|
||||
isolation story (P4/P6), git/environment package management (P3), and a CI
|
||||
end-to-end pipeline (P7).
|
||||
|
||||
## 5. What is deliberately limited (do not overclaim)
|
||||
|
||||
* Only a single, frozen Markdown profile (`marqdo.md.v1`); changing parser
|
||||
options/versions is a language change.
|
||||
* Strict mode is a refusal, not an isolation implementation. `project` mode is
|
||||
cooperative and in-process.
|
||||
* Native plugins are trusted in-process code; there is no isolated plugin
|
||||
worker, and a plugin crash takes down the host.
|
||||
* Ints are arbitrary precision; finite floats are binary64 and wire round-trip
|
||||
is exact (`float_roundtrip` enabled), but decimal-vs-binary formatting still
|
||||
applies when displaying values.
|
||||
* Math is a limited exact-rational calculator, not a CAS or a prover.
|
||||
* Foreign kernels are optional, one-shot, unsandboxed child processes.
|
||||
* `fmt` is minimal; `check` does not yet validate Markdown links; link-form
|
||||
calls cannot cross modules.
|
||||
* Package management is path-only in practice; `marqdo.yaml` capabilities are
|
||||
requests, never grants.
|
||||
@@ -0,0 +1,139 @@
|
||||
# ADR-0001: Crate graph and the `marqdo-span` foundation
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
`docs/DEVELOPMENT_BLUEPRINT.md` §11.1 fixes the dependency direction for the
|
||||
workspace and states: "`markdown ->` zero other Marqdo crate; responsible for a
|
||||
stable view of the standard AST." The implemented workspace instead has
|
||||
fifteen crates, and `marqdo-markdown` depends on `marqdo-span`.
|
||||
|
||||
Why the divergence was necessary:
|
||||
|
||||
* Every layer needs identical byte spans: the Markdown AST
|
||||
(`crates/marqdo-markdown/src/ast.rs`), expression AST
|
||||
(`crates/marqdo-expression`), IR (`crates/marqdo-ir`), diagnostics and runtime
|
||||
errors (`crates/marqdo-value`), effect traces (`crates/marqdo-host`), plugin
|
||||
errors (`crates/marqdo-plugin-abi`) and the CLI renderer.
|
||||
* A `Span` must be the same type across boundaries or every layer needs
|
||||
conversion functions. Conversions are exactly where off-by-one bugs and
|
||||
broken source locations appear, and they would make `Expr`/`Stmt`
|
||||
serde output inconsistent between `inspect --ast` and `inspect --ir`.
|
||||
* The blueprint's "zero Marqdo dependencies" rule exists to keep the parser
|
||||
independent of the semantic layers. A tiny, effect-free span crate preserves
|
||||
that property while satisfying the shared-data-contract requirement.
|
||||
|
||||
## Decision
|
||||
|
||||
1. Add `crates/marqdo-span` as a **leaf** crate with no Marqdo dependencies and
|
||||
only `serde` as an external dependency. It defines:
|
||||
* `Span { start: usize, end: usize }` — half-open byte range into the
|
||||
original UTF-8 source, with `merge`, `shift`, `contains`, `len`;
|
||||
* `LineCol { line, column }` — 1-based, byte columns;
|
||||
* `LineIndex` — precomputed line starts for byte→line/column conversion.
|
||||
2. Treat `Span` as part of the toolchain's shared data contract. Crates that
|
||||
produce or consume source locations depend on `marqdo-span` directly; they
|
||||
do **not** wrap it in local span types.
|
||||
3. `marqdo-markdown` depends on `marqdo-span` (and `comrak`) but on **no
|
||||
semantic crate**. It still owns only the canonical AST view, front matter
|
||||
extraction and source maps; it performs no mapping, resolution or execution.
|
||||
4. Keep the rest of the blueprint's direction:
|
||||
`value -> span`, `expression -> span`, `ir -> span + expression`,
|
||||
`markdown -> span + comrak`, `frontend -> markdown + expression + ir + value`,
|
||||
`host -> value + math + foreign + sandbox`, `runtime -> ir + value + host +
|
||||
expression`, `cli -> all` (composition root). There is no `runtime -> cli`
|
||||
edge and no cycle; `cargo tree` is expected to stay acyclic.
|
||||
5. Do not introduce the blueprint's separate `sema` or `environment` crates in
|
||||
M0.5: semantic analysis lives in `marqdo-frontend`
|
||||
(`analyze`, `calls`, `infer`, `signature`, `units`, `map`), and package/
|
||||
environment formatting lives in `marqdo-package` plus the CLI. Splitting
|
||||
them is a refactor, not a semantic change.
|
||||
|
||||
Actual dependency graph (workspace `Cargo.toml` plus crate manifests):
|
||||
|
||||
```text
|
||||
span serde
|
||||
value span, serde, serde_json, num-bigint, num-traits
|
||||
expression span, serde, num-bigint, unicode-ident
|
||||
markdown span, comrak, serde, serde_yaml, unicode-normalization
|
||||
ir span, expression, serde
|
||||
frontend span, markdown, expression, ir, value, sha2,
|
||||
unicode-normalization, unicode-ident
|
||||
math value, num-bigint, num-rational, num-traits, serde
|
||||
foreign span, value, serde_json, thiserror
|
||||
sandbox serde
|
||||
plugin-abi span, value, serde, serde_json, serde_yaml, thiserror,
|
||||
libloading, sha2, hex
|
||||
plugin-sdk plugin-abi, serde, serde_json
|
||||
host span, value, math, foreign, sandbox, serde, serde_json,
|
||||
thiserror, glob
|
||||
runtime span, value, ir, expression, host, num-bigint,
|
||||
num-rational, num-traits, thiserror
|
||||
package serde, serde_yaml
|
||||
cli span, value, expression, markdown, ir, frontend, math,
|
||||
foreign, plugin-abi, host, runtime, package, sandbox,
|
||||
clap, serde, serde_json, walkdir, hex, sha2
|
||||
```
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Put `Span` in `marqdo-value`.** Then `marqdo-markdown` would depend on the
|
||||
runtime value model, which is a semantic layer and pulls in `num-bigint` and
|
||||
`serde_json`. Worse layering than a dedicated leaf.
|
||||
2. **Duplicate `Span` in each crate and convert at boundaries.** Avoids a crate
|
||||
but guarantees conversion code, mismatched serde shapes and duplicated
|
||||
tests; rejected.
|
||||
3. **Define `Span` in `marqdo-markdown` and let others depend on it.** Makes
|
||||
the parser a de-facto foundation crate and lets non-Markdown layers depend
|
||||
on `comrak`'s dependencies; rejected.
|
||||
4. **Use a third-party range type.** Adds a dependency and gives no benefit
|
||||
over ~170 lines; rejected.
|
||||
5. **Keep the blueprint's literal "markdown has zero Marqdo deps".** Would
|
||||
force either duplication or a span type in the parser that cannot be shared
|
||||
with diagnostics; rejected in favor of a documented, narrow exception.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No syntax change. Every `MdNode` still has `span: Span` in the original file
|
||||
(`crates/marqdo-markdown/src/ast.rs`), and the JSON inspection view is
|
||||
`{"kind":"heading","level":2,"span":{"start":0,"end":7},...}`. Spans are
|
||||
half-open byte offsets, BOM-rebased, and byte columns for diagnostics.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* This is a **new repository**; there is no legacy compatibility burden.
|
||||
* Public Rust API: moving `Span` later is a workspace-wide API change but has
|
||||
no language/ABI/package-format impact. It would require updating every
|
||||
`crates/*/Cargo.toml`.
|
||||
* Stable observable contracts (AST JSON `span`, diagnostic JSON `span`, IR
|
||||
`span`) are unchanged by the crate boundary. Changing span semantics
|
||||
(e.g. line/column units) requires a spec change and ADR.
|
||||
* The divergence from blueprint §11.1 is recorded here; future crates should
|
||||
use `marqdo-span` rather than introducing another location type.
|
||||
|
||||
## Security
|
||||
|
||||
No new capability or effect. `marqdo-span` is `serde`-only, performs no I/O and
|
||||
cannot execute anything. Byte offsets are derived from already-parsed source;
|
||||
`LineIndex::line_col` clamps out-of-range offsets instead of panicking.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-span/src/lib.rs` tests multi-byte UTF-8 line/column mapping,
|
||||
`merge`/`shift`/`contains`, and clamping.
|
||||
* `crates/marqdo-markdown/src/profile.rs` tests BOM rebasing and that every AST
|
||||
node carries a byte span; frontend tests assert diagnostics point at the
|
||||
expected source line/column.
|
||||
* `cargo test --workspace` must pass; `cargo tree` must remain acyclic (no
|
||||
automated CI gate yet — this is a P7 item).
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
Immediate: the crate exists and all layers already use it. Future work may
|
||||
split the frontend into `sema`/`environment` crates; that must preserve the
|
||||
graph above (`runtime` must never depend on `cli`, `markdown` must never depend
|
||||
on semantic crates). If a second span representation is ever needed (for
|
||||
example for generated code), add it as a conversion in `marqdo-span`, not as a
|
||||
per-crate duplicate.
|
||||
@@ -0,0 +1,140 @@
|
||||
# ADR-0002: Executable nodes execute only inside level-2 function bodies
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The blueprint (§1.2 execution context, §3.2 control flow, unresolved issue
|
||||
**S-03**) establishes that `Strong` (execute), `Emph` (return), ordered lists
|
||||
(branch), bullet lists (loop) and block quotes (call) have program meaning
|
||||
**inside a function body**, i.e. inside the range of a level-2 heading. It
|
||||
leaves the module top level under-specified: if a document has `**x = 1**`
|
||||
before any function, should that be a hard error, a silent no-op, or an
|
||||
explicit top-level statement?
|
||||
|
||||
Two failure modes had to be avoided:
|
||||
|
||||
1. **Silent execution at module scope.** There is no function frame, no return
|
||||
target and no entry point, so top-level effects would be surprising and
|
||||
would make `check`/`inspect` harder to reason about.
|
||||
2. **Hard error.** Ordinary Markdown documents routinely use `**bold**`,
|
||||
`*italic*`, `1. ordered` and `> quotes` in prose. Treating every such node
|
||||
outside a function as a fatal compile error would make the language hostile
|
||||
to normal documentation and would break the "renders normally" goal.
|
||||
|
||||
The chosen baseline was: executable nodes only execute inside function bodies;
|
||||
elsewhere they are ordinary document content, but this must not happen
|
||||
silently.
|
||||
|
||||
## Decision
|
||||
|
||||
1. A **function body** is the child-block range of a `heading(level=2)` up to
|
||||
the next heading of level ≤ 2 (`crates/marqdo-frontend/src/units.rs`,
|
||||
`FunctionScope.body_start..body_end`). Executable roles are assigned only
|
||||
inside those ranges by `crates/marqdo-frontend/src/map.rs`.
|
||||
2. Outside every function body, `strong`, `emph`, `list` and `block_quote`
|
||||
nodes are **inert document content**. They are not mapped to IR, are not
|
||||
executed, and prose is not inspected for keywords.
|
||||
3. The frontend emits a **warning** `markdown.inert_executable` for every
|
||||
would-be execution node found outside a function body. The document still
|
||||
compiles and runs; only `check`'s diagnostic output reflects the warning.
|
||||
The warning is produced once per top-level node and has a hint to move the
|
||||
construct under a `## function` heading.
|
||||
* `block_quote` → "a block quote invocation ... is inert document content";
|
||||
* `list` → "a list control structure ...";
|
||||
* `paragraph` → the first `strong`/`emph` found in the paragraph
|
||||
("an execution statement" / "a return statement").
|
||||
* warnings are emitted only while scanning the document's direct top-level
|
||||
blocks: headings, tables, code/math blocks and other non-paragraph blocks
|
||||
produce none. A top-level paragraph is walked recursively, so a
|
||||
`strong`/`emph` nested under a link or strikethrough inside that paragraph
|
||||
can still produce the inert warning. Nodes inside a function body produce
|
||||
no warning (they are mapped or rejected by the strict executable-shape
|
||||
rules instead).
|
||||
4. Warnings do not change exit codes: `check` exits 0 when there are no error
|
||||
diagnostics even if warnings exist; `run`/`call` execute normally. JSON
|
||||
`check` output sets `success = (errors == 0)` and includes the warning
|
||||
entries in `diagnostics`.
|
||||
|
||||
Rationale for the nodes covered: these are exactly the block-level constructs
|
||||
that `map.rs` would otherwise turn into statements if a function body existed.
|
||||
Inline nodes inside a paragraph are covered by one paragraph-level warning
|
||||
rather than one per node, which keeps normal prose readable.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Hard error for every executable-looking node outside a function.** Rejected
|
||||
(see context); it forbids emphasis in ordinary Markdown and makes writing a
|
||||
README a compile failure.
|
||||
2. **Silent inert behaviour with no diagnostic.** Rejected: an author could
|
||||
believe top-level `**x = 1**` had executed. The blueprint explicitly wants
|
||||
invalid execution intent surfaced.
|
||||
3. **Execute top-level nodes in an implicit module frame.** Rejected: no
|
||||
defined entry semantics, no return target, would make imported modules
|
||||
potential side-effect roots (violating "modules register definitions only").
|
||||
4. **Compile-time error only when the payload parses as a statement.** Rejected:
|
||||
this is the forbidden heuristic ("parses as code → execute") from blueprint
|
||||
§1.3; role must come from node type and position, not from guessing.
|
||||
5. **Warning per AST node.** Rejected as noisy for nested content; one warning
|
||||
per top-level construct is enough to point at the problem.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No new node kinds. Relevant structure:
|
||||
|
||||
```text
|
||||
document
|
||||
heading(level=2) "f" <-- opens a function body
|
||||
paragraph -> strong(text "x = 1") <-- executes (Stmt::Assign)
|
||||
paragraph (before any heading)
|
||||
strong(text "x = 1") <-- inert; emits markdown.inert_executable
|
||||
list(ordered) <-- inert outside function; warning
|
||||
block_quote <-- inert outside function; warning
|
||||
```
|
||||
|
||||
A level-2 heading without a preceding level-1 heading is a module-root function
|
||||
(`## main`) and its body executes normally. A `###` section inside the function
|
||||
range stays inside the function body; headings never execute.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Warning code `markdown.inert_executable` is part of the CLI diagnostic
|
||||
contract. Renaming it requires an ADR and a CLI version note.
|
||||
* Making this a hard error later would be a breaking language change: documents
|
||||
that currently compile (with warnings) would stop. If that is ever desired it
|
||||
needs a migration plan (e.g. an opt-in strict lint).
|
||||
* No package/ABI/IR shape change: inert nodes are simply absent from the IR.
|
||||
* Prose invariance is preserved: editing narrative text outside functions
|
||||
cannot change the IR.
|
||||
|
||||
## Security
|
||||
|
||||
No effect: inert nodes never reach the interpreter, host or kernels. The
|
||||
warning reduces the chance of a false belief that a side effect happened,
|
||||
which is a security-relevant honesty property. A document cannot cause an
|
||||
execute-by-accident through top-level prose.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-frontend/src/units.rs::tests::
|
||||
executable_outside_function_warns_but_does_not_execute` asserts the warning
|
||||
and that the function index is still built.
|
||||
* `units.rs` tests for function ranges, nested sections, root-level `## main`
|
||||
and duplicate/empty headings.
|
||||
* `crates/marqdo-frontend/src/analyze.rs` tests hard errors for invalid
|
||||
executable shapes inside function bodies (the companion rule: when a node
|
||||
*is* executable, its shape is strictly checked).
|
||||
* Conformance/negative fixtures under `crates/cli/tests` and `tests/`
|
||||
(work in progress by the fixtures workstream) are expected to cover the
|
||||
warning and the inert behavior; `spec/mapping.md` §4 documents the contract.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
M0.5 ships with warnings. Rollout guidance:
|
||||
|
||||
* document authors move executable constructs under `## function` headings;
|
||||
* tooling should show the warning in editors (the diagnostic already has a
|
||||
precise span);
|
||||
* if a future release makes inert execution a hard error, add a deprecation
|
||||
window and a `check` flag so existing documents can be migrated.
|
||||
@@ -0,0 +1,157 @@
|
||||
# ADR-0003: `ref` resolution order and `ref.ambiguous` (S-07)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
`ref("#anchor")` is the only way to turn a heading range into an inert
|
||||
resource value (`Formula` or `Code`) or a `FunctionRef`. The blueprint
|
||||
(§5.2, unresolved issue **S-07**) is explicit that a heading range can contain
|
||||
several formulas or code fences, and that the implementation must **never
|
||||
silently pick the first one**:
|
||||
|
||||
> 一个范围存在多段公式时,默认对象选择必须是明确规则(例如禁止无选择器,要求
|
||||
> `math.formulas(ref(...))` 列出候选;不选"第一条")。
|
||||
|
||||
The failing example is easy to write:
|
||||
|
||||
```markdown
|
||||
## f
|
||||
|
||||
### 公式
|
||||
|
||||
$$a = 1$$
|
||||
|
||||
$$b = 2$$
|
||||
|
||||
**x = ref("#公式")**
|
||||
```
|
||||
|
||||
If `ref` returned `a = 1`, the program would be non-deterministic with respect
|
||||
to document structure (insert an earlier formula and the meaning changes) and
|
||||
would hide a genuine authoring error.
|
||||
|
||||
At the same time the common case — one formula per heading — must stay simple;
|
||||
requiring a selector for every reference would make `examples/math.mq.md`
|
||||
awkward.
|
||||
|
||||
## Decision
|
||||
|
||||
`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref` implements this
|
||||
algorithm:
|
||||
|
||||
1. The target must be a `Text` beginning with `#`; otherwise
|
||||
`symbol.unknown` ("must be a `#heading` anchor"). `ref` is not a URL
|
||||
resolver and does not cross modules.
|
||||
2. The fragment is looked up in the **current module's anchor map**
|
||||
(`Module::anchors`, GitHub-style slugs computed in
|
||||
`crates/marqdo-frontend/src/units.rs`). `ref` does not fall back to
|
||||
canonical unit names and does not search imported modules.
|
||||
3. Collect the resources attributed to the resolved unit (innermost unit by
|
||||
span, `crates/marqdo-frontend/src/resources.rs`).
|
||||
* **Exactly 1** → return `Formula` or `Code` (with `unit` and `span`
|
||||
provenance).
|
||||
* **0** → if a function's unit equals the resolved unit, return
|
||||
`FunctionRef { module, name }`; otherwise `symbol.unknown`.
|
||||
* **≥ 2** → return **`ref.ambiguous`**:
|
||||
"`#...` contains N resources; select one explicitly instead of relying on
|
||||
the first". Never choose a resource.
|
||||
4. Function references are callable through the normal call path; formulas and
|
||||
code remain inert until `math.eval`/`foreign.*` is explicitly called (those
|
||||
are separately capability-checked).
|
||||
|
||||
Resource attribution is deterministic: a resource belongs to its innermost
|
||||
enclosing heading unit, resources keep document order, and a resource is a
|
||||
Markdown `math`/`math_block` or `code_block` node (inline code is not a
|
||||
resource).
|
||||
|
||||
The `ref.ambiguous` category is `ErrorCategory::RefAmbiguous` with the stable
|
||||
code `ref.ambiguous` (`crates/marqdo-value/src/error.rs`).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **First resource wins** (blueprint's rejected baseline). Rejected by S-07:
|
||||
document edits would silently change program meaning, and there is no
|
||||
diagnostic for the author.
|
||||
2. **Always require a selector argument**
|
||||
(`ref("#f", selector="second")`). Rejected for M0.5: no selector syntax or
|
||||
addressing scheme is specified yet, and the single-resource case is by far
|
||||
the common one. A future ADR can add an explicit selector while keeping the
|
||||
ambiguity rule.
|
||||
3. **Error on every heading with ≥ 2 resources at compile time**, even if never
|
||||
referenced. Rejected: a document may legitimately contain several formulas
|
||||
for prose and never `ref` them; only the reference is ambiguous.
|
||||
4. **Return a list of all candidates.** Rejected for v1: it changes the return
|
||||
type and forces every caller to handle a selector; a dedicated API
|
||||
(`math.formulas(ref(...))`) would be a new spec, not a silent widening.
|
||||
5. **Cross-module `ref`.** Rejected for v1: the wire/identity model and
|
||||
`ModuleId` resolution live in the loader; `ref` is a pure runtime builtin
|
||||
over the current module. Cross-module resources are an open item.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
```text
|
||||
heading(level=2) "f"
|
||||
heading(level=3) "公式"
|
||||
math_block "/$$ a = 1 $$/" -> Resource::Formula { unit: <#f/公式> }
|
||||
math_block "/$$ b = 2 $$/" -> Resource::Formula { unit: <#f/公式> }
|
||||
paragraph -> strong("x = ref(\"#公式\")")
|
||||
```
|
||||
|
||||
Anchors are separate from unit paths: `#公式` is a GitHub slug for the heading
|
||||
name, while the canonical `UnitId` is `<module>#f/公式`. Colliding slugs get
|
||||
`-1`/`-2` suffixes; canonical ids never do.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* `ref.ambiguous` is a stable runtime code; callers may branch on it.
|
||||
* The single-resource and function-reference cases are frozen. Adding a
|
||||
selector must be additive (new argument or new builtin) and must not change
|
||||
the ≥ 2 error into a silent choice.
|
||||
* `ref` target syntax (`#fragment`) is frozen; changing it to accept file paths
|
||||
would need a new ADR.
|
||||
* No package/ABI/wire change: formula/code values already carry
|
||||
`unit`/`span`/`content_hash`.
|
||||
|
||||
Example behavior, verified by
|
||||
`crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`:
|
||||
|
||||
```text
|
||||
$ marqdo run index.mq.md # heading with two $$...$$ blocks and ref("#公式")
|
||||
runtime error[ref.ambiguous]: `#公式` contains 2 resources; select one explicitly instead of relying on the first
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
* `ref` is a pure builtin: no host call, no capability required, no side
|
||||
effect. It only returns inert values or a function reference.
|
||||
* An inert `Formula`/`Code` does not execute. Evaluation/execution requires a
|
||||
separate explicit call (`math.eval`, `foreign.run/call`), which is capability
|
||||
checked at the host boundary.
|
||||
* Refusing ambiguity prevents a class of "wrong formula executed" mistakes,
|
||||
which matters for scientific documents where formulas are evidence.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`
|
||||
drives a real document through frontend + runtime and asserts the
|
||||
`ref.ambiguous` code.
|
||||
* `crates/marqdo-frontend/src/resources.rs` tests attribution to innermost
|
||||
functions, multiple resources kept in one unit, and stable SHA-256 code
|
||||
hashes.
|
||||
* `crates/marqdo-markdown`/`frontend` tests cover anchors and slug collisions.
|
||||
* `examples/math.mq.md` is the positive single-formula case; it runs under
|
||||
`marqdo run` only via a function (the example itself defines callable
|
||||
functions, no `main`).
|
||||
* Missing anchor/not-a-function cases are covered by the `symbol.unknown`
|
||||
paths; conformance fixtures are being added by the fixtures workstream.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration needed for documents with exactly one resource per referenced
|
||||
heading. Documents that relied on "first formula" behavior (there are none in
|
||||
the examples) must split headings or wait for an explicit selector API. If a
|
||||
selector is added later, keep `ref("#...")` behavior unchanged and introduce
|
||||
the selector as a new argument/builtin, with tests for both the ambiguous and
|
||||
the selected case.
|
||||
@@ -0,0 +1,133 @@
|
||||
# ADR-0004: Strict mode refuses to run without OS isolation
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
Blueprint §9 defines three execution modes and one non-negotiable rule:
|
||||
|
||||
> `strict`: untrusted documents/code/plugins — independent restricted process,
|
||||
> OS-level resource and syscall/file/network restrictions; if unavailable,
|
||||
> **refuse to run**.
|
||||
|
||||
A cooperative in-process capability check cannot contain hostile native code
|
||||
or an unrestricted child process. If `strict` silently degraded to `project`,
|
||||
users would believe they had isolation they did not have. That is a security
|
||||
bug, not a missing feature.
|
||||
|
||||
M0.5 implements capability checks, budgets and process supervision but **no OS
|
||||
isolation adapter** (no Linux namespaces/seccomp/cgroups, no Windows Job
|
||||
Object/appcontainer, no macOS sandbox), and the isolated plugin worker
|
||||
transport is not implemented either.
|
||||
|
||||
## Decision
|
||||
|
||||
1. `crates/marqdo-sandbox` is honest about platform support:
|
||||
* `PlatformCapabilities::detect()` always reports
|
||||
`StrictIsolation::Unavailable { reason }` in this build, with a
|
||||
platform-specific reason (Linux: "no OS isolation adapter
|
||||
(namespaces/seccomp/cgroups) is implemented in this build"; macOS/Windows/
|
||||
other targets have analogous messages);
|
||||
* `worker_processes: false`.
|
||||
2. `marqdo_sandbox::enforce(mode, platform)`:
|
||||
* `pure` and `project` → `Ok(())`;
|
||||
* `strict` → `Err(SandboxError::StrictUnavailable { reason })` unless
|
||||
`platform.strict.is_available()`.
|
||||
3. `run`/`call` call `enforce` **before** constructing the host, loading
|
||||
plugins or executing anything. On failure they print
|
||||
"refusing to run in strict mode: ..." (or a JSON `{"success":false,
|
||||
"error":...}` envelope) and exit 1.
|
||||
4. There is **no fallback**: the CLI never downgrades a requested strict run to
|
||||
project mode. `SandboxMode::parse("strict")` succeeds (the mode name is
|
||||
valid); only execution is refused.
|
||||
5. `inspect --platform` exposes the detection report so users can see why:
|
||||
`{"os":"linux","arch":"x86_64","strict":{"state":"unavailable","reason":...},
|
||||
"worker_processes":false}`.
|
||||
6. The rest of the system is labelled accordingly: `project` mode is
|
||||
cooperative, in-process, and is not a sandbox; foreign kernels are
|
||||
process-isolated but not confined; native plugins run in-process and require
|
||||
explicit trust (ADR-0006/security spec).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Best-effort strict** (run with the same checks as project). Rejected: it
|
||||
directly violates the blueprint's "cannot provide → refuse" rule and hides a
|
||||
missing security boundary.
|
||||
2. **Warn and continue.** Rejected for the same reason; a warning is not a
|
||||
sandbox, and CI/agents may not read warnings.
|
||||
3. **Implement a Linux namespaces/seccomp adapter now.** Out of scope for
|
||||
M0.5: it needs platform-specific code, integration tests, resource limits,
|
||||
CI on multiple kernels, and a way to supervise the worker protocol. This is
|
||||
P4 work and belongs in its own ADR when implemented.
|
||||
4. **Use Wasm/WASI for strict.** Attractive for plugins later, but it does not
|
||||
sandbox arbitrary Python/Node processes or native `.so` files, and the
|
||||
runtime is not implemented. Rejected for now.
|
||||
5. **Claim strict via the process isolation of foreign kernels.** Rejected:
|
||||
a child process inherits the user's privileges and filesystem/network
|
||||
access; changing cwd and clearing env is hygiene, not confinement.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No language or AST change. A strict run is rejected before any document is
|
||||
executed (after static loading/diagnostics); the CLI contract is:
|
||||
|
||||
```text
|
||||
$ marqdo run index.mq.md --mode strict
|
||||
error: refusing to run in strict mode: no OS isolation adapter
|
||||
(namespaces/seccomp/cgroups) is implemented in this build. Strict mode
|
||||
requires verified OS-level isolation and never falls back to an
|
||||
unrestricted process.
|
||||
$ echo $?
|
||||
1
|
||||
```
|
||||
|
||||
(Exact wording comes from
|
||||
`crates/marqdo-sandbox/src/lib.rs::SandboxError::display`.)
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Accepting `--mode strict` as a valid *name* is part of the CLI contract; the
|
||||
refusal is a documented behavior, not a parse error. A future release that
|
||||
implements isolation flips the detection report and starts executing strict
|
||||
runs — that is the intended upgrade path.
|
||||
* Exit code 1 (runtime/security refusal) is stable; usage errors are 2.
|
||||
* `PlatformCapabilities` JSON shape (`state: available|unavailable`) is
|
||||
versioned by documentation and should not change without an ADR.
|
||||
* No package/ABI/wire impact.
|
||||
|
||||
## Security
|
||||
|
||||
* The guarantee "requested strict → refused rather than downgraded" is
|
||||
enforced in code (`enforce`), not documentation.
|
||||
* Users cannot accidentally believe untrusted docs are sandboxed by this
|
||||
build; the platform report is the evidence.
|
||||
* Project mode remains explicitly non-sandboxed. Native plugins (in-process)
|
||||
and foreign child processes are recorded as trust decisions, not isolation.
|
||||
* Secrets: strict refusal happens before plugin loading or foreign spawning,
|
||||
so no untrusted code is touched by a refused command.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-sandbox/src/lib.rs::tests::
|
||||
strict_refuses_when_isolation_is_unavailable` asserts the refusal and that
|
||||
`pure`/`project` still succeed.
|
||||
* `mode_names_round_trip` covers parsing/printing.
|
||||
* CLI behavior (`--mode strict` exit 1, message) is covered by manual/CLI
|
||||
smoke checks; a CLI integration test should be added with the conformance
|
||||
harness.
|
||||
* `inspect --platform` JSON is exercised manually and in the CLI help/example
|
||||
documentation; it should get a golden test in the fixture workstream.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
When an OS isolation adapter lands:
|
||||
|
||||
1. implement a `StrictIsolation::Available { mechanism }` report and a real
|
||||
supervisor/worker;
|
||||
2. add integration tests for filesystem/network/process restrictions and
|
||||
resource limits per platform, plus the platform capability matrix;
|
||||
3. update `spec/security.md`, this ADR (or supersede it), `docs/STATUS.md` and
|
||||
the CLI docs;
|
||||
4. only then change `PlatformCapabilities::detect`; do not add a flag that
|
||||
bypasses the refusal before that work is done.
|
||||
@@ -0,0 +1,174 @@
|
||||
# ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The language derives program meaning from the **canonical AST**, never from
|
||||
source spelling (`spec/markdown-profile.md`, `spec/mapping.md` §1). The parser
|
||||
(`comrak =1.0.0`, frozen options) normalizes several syntaxes before Marqdo
|
||||
sees them, and a few normalizations are lossy:
|
||||
|
||||
* `# H` and Setext `H\n===` both become a heading node; the source spelling is
|
||||
lost unless recorded;
|
||||
* GFM task list items (`- [x] text`) produce a task-specific item node;
|
||||
* inline code spans are stored as a literal without the backtick delimiters,
|
||||
but source spans include them;
|
||||
* `~~strikethrough~~` is enabled by the profile and wraps arbitrary inlines.
|
||||
|
||||
The language needs a single, documented answer per question, because node type
|
||||
and structural position decide executable roles.
|
||||
|
||||
## Decision
|
||||
|
||||
The profile (`marqdo.md.v1`) and the canonical AST
|
||||
(`crates/marqdo-markdown/src/ast.rs`, `src/profile.rs`) freeze these rules:
|
||||
|
||||
1. **Setext headings are headings.** `# name` and a Setext level-1/2 underline
|
||||
both produce `heading { level, setext }`; `setext` is inspection-only and
|
||||
**never affects semantics**. `## name` and a Setext level-2 heading are both
|
||||
functions. The profile keeps `parse.ignore_setext = false`.
|
||||
2. **Task-list items are `item` nodes with a task flag.**
|
||||
`item { task: Some(true|false) }` for GFM task items, `task: None`
|
||||
otherwise. Node type stays `item`, so list structure (ordered vs bullet)
|
||||
still determines branch/loop roles; the checkbox flag is inert data and does
|
||||
not change execution. A task list item in an executable position still
|
||||
requires its condition header like any other item.
|
||||
3. **Code spans carry delimiter tick counts.** Inline `code { ticks }` records
|
||||
the number of backticks, and the source span includes them; the literal is
|
||||
the inner payload. This lets the frontend map expression payload spans back
|
||||
to exact source bytes for diagnostics. "Ticks" is not user-visible program
|
||||
meaning.
|
||||
4. **Strikethrough is inert/lazy.** `strikethrough` is an inert wrapper: it is
|
||||
never traversed for execution, its content is prose, and a `strong`/`emph`
|
||||
inside it does not execute. It is not an error to write it; the language
|
||||
simply does not use formatting as a control-flow mechanism here. (A code
|
||||
span or math inside it is likewise inert.)
|
||||
5. **Unmodelled nodes are preserved** as `other { tag }` with children and raw
|
||||
literal. They are inert but never silently dropped from `inspect --ast`.
|
||||
6. **Literal payloads are kept** on inert leaves: `text`, `code`, `code_block`,
|
||||
`math`, `math_block`, `html_block`, `html_inline`, `front_matter`.
|
||||
7. **Other frozen normalizations:**
|
||||
* `**x**`/`__x__` and `*x*`/`_x_` are the same node kinds (`strong`/`emph`);
|
||||
* ordered delimiters `1.`/`1)` and bullets `-`/`*`/`+` are the same
|
||||
`list`/`item` structure; role depends on `ordered`, not marker;
|
||||
* tight vs loose lists only affect rendering (`tight` is recorded, inert);
|
||||
* `link`/`image` keep `url`/`title`; `table` keeps column alignments,
|
||||
`table_row` keeps `header`; `heading` keeps `level`/`setext`;
|
||||
* a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the
|
||||
original file;
|
||||
* front matter is recognized only at the document start (after BOM), with
|
||||
`---` opening and `---`/`...` closing; a later `---` is a thematic break
|
||||
or Setext underline.
|
||||
|
||||
These rules are part of profile id `marqdo.md.v1`. Changing them (or bumping
|
||||
the comrak version/options) requires a profile id bump, updated goldens and this
|
||||
ADR being superseded.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Distinguish Setext vs ATX headings semantically.** Rejected: identical AST
|
||||
structure outside the marker must have identical meaning (`## H` and Setext
|
||||
H2 are the same function). Keeping `setext` only for inspection avoids
|
||||
silently diverging programs when a document changes style.
|
||||
2. **Introduce a separate `task_item` node kind.** Rejected: it would require
|
||||
every list-role check to special-case two item kinds and could be used to
|
||||
hide a control structure behind a checkbox. Normalizing to `item { task }`
|
||||
keeps list structure authoritative.
|
||||
3. **Strip the tick information and store only the literal.** Rejected: payload
|
||||
spans returned by the expression parser are relative to the concatenated
|
||||
payload; without delimiter lengths, error spans cannot be mapped back to
|
||||
file bytes precisely.
|
||||
4. **Traverse strikethrough content for execution.** Rejected: it is a lazy
|
||||
region by profile; executing inside it would make formatting semantics
|
||||
surprising ("struck-through code runs"). It would also create a second way
|
||||
to hide executable nodes.
|
||||
5. **Drop unknown parser nodes.** Rejected: `inspect --ast` must show what the
|
||||
parser produced; dropping syntax hides profile drift.
|
||||
6. **Preserve raw source and re-parse with a custom parser.** Rejected by the
|
||||
project's core rule: one standard Markdown parser, no second Markdown AST.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
Frozen JSON shapes (`inspect --ast --json`):
|
||||
|
||||
```json
|
||||
{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
|
||||
"children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"list","ordered":false,"tight":true,"span":{...},
|
||||
"children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}
|
||||
```
|
||||
|
||||
Attribute order in the JSON is fixed by the serializer; goldens are regenerated
|
||||
only through the explicit update workflow.
|
||||
|
||||
### 5.1 Known interaction to keep in mind
|
||||
|
||||
Inside a function body, a `**strong**` (or an ordered/bullet list) nested under
|
||||
`strikethrough` is inert and never executes; because the mapper does not
|
||||
traverse strikethrough, it also produces no warning. Outside function bodies,
|
||||
the `markdown.inert_executable` scan walks a top-level paragraph recursively,
|
||||
so such a nested node may still produce the inert warning even though it would
|
||||
not execute if the paragraph were inside a function. This is consistent with
|
||||
"formatting never hides execution", but an author who intends execution must
|
||||
not wrap it in strikethrough. Warnings for lazy-region content inside functions
|
||||
are a possible future refinement; no behavior change is planned without an ADR.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Profile id `marqdo.md.v1` is frozen with comrak `=1.0.0` and the exact
|
||||
extension flags; the unit test
|
||||
`crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen`
|
||||
asserts the option table (tables, tasklist, strikethrough, math dollars,
|
||||
front matter; all other extensions disabled).
|
||||
* Adding an AST field is a JSON output change for `inspect --ast`; it needs a
|
||||
goldens update and a profile note. Changing a semantic rule (e.g. executing
|
||||
inside strikethrough) is a language change and requires an ADR.
|
||||
* No package/ABI/wire impact.
|
||||
|
||||
## Security
|
||||
|
||||
* The lazy-region rules ensure inline formatting inside headings, tables,
|
||||
links, code, math, HTML and front matter never executes — a context-aware
|
||||
visitor is used (`crates/marqdo-frontend/src/map.rs` only descends from
|
||||
paragraphs into `strong`/`emph`, and ignores every other inline node).
|
||||
* Strikethrough cannot be used to smuggle executable content; it is inert.
|
||||
* Normalizing task items to `item` prevents a checkbox from hiding a
|
||||
branch/loop header; the condition-header check still applies.
|
||||
* No new capabilities or host calls.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-markdown/src/profile.rs`:
|
||||
* `profile_options_are_frozen` — exact extension/parse option table;
|
||||
* `setext_headings_are_heading_nodes` — Setext produces `heading { setext: true }`;
|
||||
* `tables_and_task_lists_are_supported` — task items carry `task: Some(_)`;
|
||||
* span/BOM tests.
|
||||
* Frontend mapping tests use code spans as conditions and payloads, exercising
|
||||
`ticks`/payload span mapping.
|
||||
* `spec/markdown-profile.md` §5/§6 is the normative JSON shape; AST goldens
|
||||
under `crates/cli/tests` (fixtures workstream) regenerate only with the
|
||||
documented update command.
|
||||
* Conformance fixtures are expected to include `**a**` vs `__a__`, Setext vs
|
||||
ATX, task/list variants and strikethrough inertness.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration for documents: Setext and ATX were always the same function under
|
||||
this profile. When comrak is upgraded or the extension set changes, bump the
|
||||
profile id, regenerate AST goldens, re-run all specs/ADRs that reference node
|
||||
shapes, and document the change here (superseding this ADR if a rule changes).
|
||||
Code-span `ticks` and `task` are additive inspection fields; keep them optional
|
||||
in consumers that parse the JSON.
|
||||
@@ -0,0 +1,225 @@
|
||||
# ADR-0006: CLI output, exit-code and JSON contract
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
CI, scripts and coding agents need a stable CLI contract, not just human
|
||||
output. The blueprint (§12) requires: all commands available with `--json`,
|
||||
non-zero error exit codes, program stdout kept separate from diagnostics, and
|
||||
machine-readable error codes. The implemented CLI must be documented precisely
|
||||
because scripts will depend on stream separation and exit status.
|
||||
|
||||
Implementation: `crates/cli/src/main.rs` (clap definitions, process exit),
|
||||
`crates/cli/src/commands.rs` (behavior), `crates/cli/src/output.rs`
|
||||
(diagnostic/JSON rendering), `crates/cli/src/args.rs` (`--arg` constants).
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Commands and flags (implemented surface)
|
||||
|
||||
| Command | Arguments / flags | What it does |
|
||||
|---|---|---|
|
||||
| `init [dir]` | — | create `marqdo.yaml`, `index.mq.md`, `.gitignore`; refuses an existing manifest |
|
||||
| `run [file]` | `--mode`, `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin`, `--fuel`, `--stack`, `--max-output`, `--timeout`, `--plugin`, `--trust-native-plugin`, `--json`, `--trace` | execute `main` (or default `index.mq.md`) |
|
||||
| `call <file> <function>` | `--arg NAME=VALUE` (repeatable) plus all `run` flags | execute one named function |
|
||||
| `check [file]` | `--json` | static check; never executes |
|
||||
| `inspect [file]` | `--ast`, `--units`, `--signature`, `--ir`, `--refs`, `--platform`, `--json` | static inspection; never executes |
|
||||
| `fmt [file]` | `--check` | normalize CRLF and trailing newlines |
|
||||
| `plugin inspect <dir-or-yaml>` | `--json` | validate manifest + verify artifact hash; never loads code |
|
||||
| `add <name>` | `--path`, `--git`, `--rev`, `--json` | edit `marqdo.yaml` |
|
||||
| `remove <name>` | `--json` | edit `marqdo.yaml` (and prune a stale lock entry) |
|
||||
| `lock` | `--json` | write deterministic `marqdo.lock` |
|
||||
| `sync` | `--locked`, `--json` | verify the lock is current |
|
||||
|
||||
There is no `test`, `tree`, `update`, `vendor`, `add` for plugins, `kernel`
|
||||
command or `--project`/`--offline` flag in this build. `run` defaults the
|
||||
file to `index.mq.md`; `check`/`inspect`/`fmt` do too. `call` requires an
|
||||
explicit file path plus a function name.
|
||||
|
||||
`--mode` is only on `run`/`call` and accepts `pure|project|strict`
|
||||
(default `project`). `--trust-native-plugin`/`--plugin` are only meaningful on
|
||||
`run`/`call`. `inspect --platform` still requires a document argument (or
|
||||
defaults to `index.mq.md`), because project loading happens first.
|
||||
|
||||
### 2. Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| `0` | success (for `check`: no **error** diagnostics; warnings do not fail) |
|
||||
| `1` | a checked document has errors; a runtime/security failure; strict refusal; plugin load/inspect failure; lock/sync mismatch; `fmt --check` would reformat; `init` found an existing manifest; `add` duplicate; `remove` missing dependency; `sync` stale lock |
|
||||
| `2` | usage error (clap), unreadable entry file, invalid `--mode` name, invalid capability combination (e.g. `--allow-read` in `pure`), invalid `--arg` value, unreadable manifest for project commands, unreadable file for `fmt` |
|
||||
|
||||
Codes are stable contract. Diagnostic **codes** (for example
|
||||
`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`,
|
||||
`permission.denied`, `ref.ambiguous`) are stable; message wording is not.
|
||||
|
||||
### 3. Stream separation
|
||||
|
||||
* **Program output** (`打印`/`print`) goes to stdout in text mode. In JSON mode
|
||||
it is embedded as the `stdout` string; it is never interleaved with JSON.
|
||||
* **Diagnostics and effect traces** go to stderr in text mode.
|
||||
* `run` in text mode writes only program output to stdout (no implicit return
|
||||
value). `call` in text mode additionally prints the returned value (using
|
||||
`Value::display`) when it is not `Unit`.
|
||||
* `check` text mode: diagnostics to stderr; `checked N module(s): ok` to stdout
|
||||
on success; a `check failed: ...` summary to stderr on failure.
|
||||
* `inspect` prints pretty JSON-ish structures to stdout; load errors to stderr.
|
||||
* JSON mode prints exactly one pretty JSON object to stdout per invocation.
|
||||
`add`/`remove`/`lock`/`sync` still print errors as plain `error: ...` on
|
||||
stderr even with `--json` (JSON success envelope only) — an inconsistency to
|
||||
fix before the contract is declared final.
|
||||
|
||||
### 4. JSON output shapes
|
||||
|
||||
`check`:
|
||||
|
||||
```json
|
||||
{"command":"check","success":true,"errors":0,
|
||||
"diagnostics":[{"file":"index.mq.md","module":"index.mq.md",
|
||||
"line":3,"column":5,"severity":"error","code":"expr.parse",
|
||||
"message":"...","hints":["..."]}]}
|
||||
```
|
||||
|
||||
`inspect`:
|
||||
|
||||
```json
|
||||
{"command":"inspect","module":"index.mq.md",
|
||||
"ast":{...},"units":[...],"signatures":[...],"ir":[...],
|
||||
"refs":[...],"platform":{...}}
|
||||
```
|
||||
|
||||
Only the requested sections are present. With no section flags, a `summary`
|
||||
with `modules`, `functions`, `units`, `resources` is emitted instead.
|
||||
|
||||
`run`/`call` (success):
|
||||
|
||||
```json
|
||||
{"command":"run","success":true,
|
||||
"value":{"type":"text","value":"..."},
|
||||
"stdout":"program stdout text",
|
||||
"trace":[{"op":"print","module":"index.mq.md",
|
||||
"span":{"start":43,"end":57},"duration_micros":3,"result":"ok"}]}
|
||||
```
|
||||
|
||||
Note: `call --json` also emits `"command":"run"` in this build (shared
|
||||
implementation). Consumers should accept both names or key on the presence of
|
||||
`diagnostics`/`value`; this is a known quirk.
|
||||
|
||||
`run`/`call` (failure): `{"command":"run","success":false,"stdout":...,
|
||||
"error":{"code":"...","message":"...","span":{"line":..,"column":..,"file":..},
|
||||
"frames":[...]},"trace":[...]}`. Runtime error spans are currently resolved
|
||||
against the **entry file** only, even when the error came from an imported
|
||||
module; frames carry function names, not module ids. This is a known precision
|
||||
limitation.
|
||||
|
||||
`plugin inspect`:
|
||||
|
||||
```json
|
||||
{"command":"plugin.inspect","success":true,"plugin":"demo.solver",
|
||||
"version":"0.1.0","abi":"^1.0","abi_compatibility":"compatible",
|
||||
"target":"x86_64-unknown-linux-gnu",
|
||||
"artifact":{"file":"solver.so","path":"...","sha256":"...","verified":true},
|
||||
"exports":[{"name":"solver.add","signature":"($a: Int, $b: Int) -> Int"}],
|
||||
"loaded":false}
|
||||
```
|
||||
|
||||
`load_project` failures with `--json` print `{"success":false,"error":"..."}`
|
||||
without a `command` field.
|
||||
|
||||
There is no `json_schema_version` field yet. The blueprint asks for a
|
||||
versioned JSON schema; until it exists, treat these keys as the stable subset
|
||||
and ignore unknown keys. Adding keys is not considered breaking; renaming or
|
||||
removing one is.
|
||||
|
||||
### 5. `--arg` values
|
||||
|
||||
`--arg NAME=VALUE` parses `VALUE` with the **expression grammar restricted to
|
||||
constants** (`crates/cli/src/args.rs`): integers, floats, double-quoted text
|
||||
(escapes supported), `true`/`false`/`None`, lists, records, and unary `-`/`+`
|
||||
over constants. Calls and identifiers are rejected ("must be a literal
|
||||
constant"). Text must be double-quoted; there is no bare-word fallback.
|
||||
|
||||
### 6. Determinism and safety of read-only commands
|
||||
|
||||
* `check`/`inspect` never execute project code (ADR context; see
|
||||
`spec/security.md` §7).
|
||||
* `plugin inspect` never loads plugin code.
|
||||
* `fmt` only rewrites the file when not `--check`; exit 1 when `--check`
|
||||
detects a difference.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Use exit code 1 for all failures.** Rejected: scripts need to distinguish
|
||||
usage errors (2) from document/runtime failures (1); it also mirrors common
|
||||
CLI conventions and clap's own usage exit code.
|
||||
2. **Print JSON even for usage errors.** Rejected for now: clap owns usage
|
||||
output before our code runs; wrapping it would mean intercepting clap
|
||||
errors. Documented as a limitation.
|
||||
3. **Interleave program output and JSON.** Rejected: it makes JSON
|
||||
unparseable. Program stdout is embedded as a string instead.
|
||||
4. **Return values from `run` on stdout in text mode.** Rejected: `run` is the
|
||||
batch entry; implicit value printing would pollute program output. `call`
|
||||
prints the return value exactly when non-`Unit`.
|
||||
5. **Human-only messages with numeric exit codes only.** Rejected: diagnostics
|
||||
and runtime errors have stable machine codes in both text and JSON.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
The CLI itself does not change the AST. What it exposes:
|
||||
|
||||
* `inspect --ast` = canonical `MdNode` tree (see ADR-0005);
|
||||
* `inspect --units` = `UnitInfo[]` (`marqdo-ir`);
|
||||
* `inspect --signature` = `{name, local_name, signature: explicit|inferred,
|
||||
params:[{name,type,has_default}]}`;
|
||||
* `inspect --ir` = `Function[]` with `Stmt`/`Expr` and spans;
|
||||
* `inspect --refs` = link edge array (`spec/references.md` §4);
|
||||
* `inspect --platform` = `PlatformCapabilities` JSON.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Exit codes, stream separation and the documented JSON keys are the public
|
||||
CLI contract. Changing them requires this ADR to be superseded and a
|
||||
changelog entry.
|
||||
* The old "design-only starter" README listed planned commands (`add`,
|
||||
`lock && sync`, `plugin inspect`); all listed commands except a versioned
|
||||
JSON schema are now implemented as described here.
|
||||
* Diagnostic code renames are breaking for CI; message changes are not.
|
||||
* No package/ABI/wire changes.
|
||||
|
||||
## Security
|
||||
|
||||
* `--json`/`--trace` enable effect tracing; traces contain op/module/span/
|
||||
duration/result summary, never argument values (no secrets) — see
|
||||
`spec/security.md` §4.
|
||||
* JSON mode embeds program stdout, which is program-controlled data; consumers
|
||||
must not treat it as trusted.
|
||||
* Refusing to run with compile errors happens before host creation, so JSON
|
||||
failure output cannot leak effects.
|
||||
* Strict-mode refusal and capability denials produce stable codes for CI.
|
||||
|
||||
## Testing
|
||||
|
||||
* CLI behavior is currently covered by manual smoke checks and library-level
|
||||
tests; the fixtures workstream is adding CLI integration tests under
|
||||
`crates/cli/tests` (goldens/negative cases) and `tests/`.
|
||||
* `crates/cli/src/args.rs` unit tests cover typed `--arg` parsing and rejection
|
||||
of non-constant values.
|
||||
* `crates/cli/src/output.rs` has no unit tests; diagnostic JSON is exercised
|
||||
through check/run smoke tests.
|
||||
* Add contract tests for: exit codes per failure class, stdout/stderr
|
||||
separation, JSON key presence, `call --json` command naming, and runtime
|
||||
error span from a non-entry module before calling the schema "final".
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
1. Keep the current shapes stable while adding tests.
|
||||
2. Add a `schema_version` to JSON envelopes and document the versioned schema;
|
||||
then document that unknown keys may be added (forward compatible) while
|
||||
existing keys are frozen.
|
||||
3. Fix the `call --json` `"command":"run"` quirk and JSON errors in
|
||||
`add/remove/lock/sync`; both are additive/corrective and should be
|
||||
announced in the changelog.
|
||||
4. If a command is renamed or an exit code changes, update this ADR, the
|
||||
README quickstart, `docs/STATUS.md` and CI scripts in the same change.
|
||||
@@ -0,0 +1,178 @@
|
||||
# ADR-0007: Arbitrary-precision `Int` and the typed wire encoding (S-08)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The blueprint (§3.1 and unresolved issue **S-08**) requires a numeric model
|
||||
that does not silently lose precision at language, JSON/ABI and math
|
||||
boundaries. It recommends arbitrary-precision integers for `Int` and type tags
|
||||
plus string forms for large integers on the wire.
|
||||
|
||||
Three independent places could lose information:
|
||||
|
||||
1. **Language arithmetic.** A fixed-width `i64` would make `Int` silently
|
||||
overflow or wrap. Factorials, powers and scientific constants routinely
|
||||
exceed 64 bits. The expression grammar already has arbitrary-length integer
|
||||
literals (`spec/expression.ebnf`).
|
||||
2. **JSON / wire.** JSON has one number type (IEEE-754 double in most parsers).
|
||||
Encoding `12345678901234567890` as a JSON number loses digits in
|
||||
JavaScript/Go consumers; encoding `1` as a number makes `Int` and `Float`
|
||||
indistinguishable.
|
||||
3. **Math kernel.** The exact-rational engine needs an integer type that is
|
||||
the *same* type as `Value::Int`, or every formula/binding crossing the host
|
||||
boundary needs a lossy conversion.
|
||||
|
||||
The chosen dependency set (`Cargo.toml`) is:
|
||||
|
||||
```toml
|
||||
num-bigint = { version = "0.4", features = ["serde"] }
|
||||
num-rational = { version = "0.4", features = ["serde"] }
|
||||
num-traits = "0.2"
|
||||
```
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. `Int` is arbitrary precision (`num-bigint::BigInt`)
|
||||
|
||||
* `Value::Int(BigInt)` (`crates/marqdo-value/src/value.rs`) is the only integer
|
||||
representation in the runtime and the expression AST
|
||||
(`crates/marqdo-expression/src/ast.rs`).
|
||||
* Expression integer literals are arbitrary-precision decimal; there is no
|
||||
fixed-width parsing step.
|
||||
* Arithmetic (`crates/marqdo-runtime/src/value_ops.rs`):
|
||||
* `+ - *` on two `Int` are exact `BigInt` operations;
|
||||
* `/` **always** produces `Float` (exact ratio converted to binary64), so
|
||||
there is no integer division surprise;
|
||||
* `//` and `%` require `Int` operands; floor division is mathematical floor
|
||||
and `a % b == a - b * (a // b)`, with the sign following the divisor;
|
||||
* mixed `Int`/`Float` arithmetic produces `Float`; converting an `Int`
|
||||
larger than 2^53 may lose precision (documented, not silently exported as
|
||||
an `Int`);
|
||||
* `Int`/`Float` comparisons are **exact**: the float is treated as the exact
|
||||
binary rational it is (`BigRational::from_float`), so
|
||||
`9007199254740993 > 9007199254740992.0` is `true`;
|
||||
* `**` on `Int` is exact exponentiation, with a negative exponent rejected
|
||||
as a type error and guards `MAX_EXACT_EXPONENT = 100_000` and
|
||||
`MAX_POWER_BITS = 1_000_000` (`resource.exhausted`).
|
||||
* There is no truthiness and no implicit string↔number conversion; `整数`/`int`
|
||||
performs explicit conversions and rejects non-integral floats.
|
||||
|
||||
### 2. `num-bigint` 0.4 is pinned to match `num-rational` 0.4
|
||||
|
||||
`num-rational`'s generic `Ratio<T>` is instantiated as
|
||||
`Ratio<num_bigint::BigInt>` in `crates/marqdo-math/src/ast.rs`
|
||||
(`pub type Rational = Ratio<BigInt>`). The two crates must resolve to the same
|
||||
`num-bigint` version, otherwise `BigInt` values from the value model cannot be
|
||||
passed to `Ratio` without conversion (or a duplicate type). The workspace pins
|
||||
both at the `0.4` line and the lockfile resolves one `num-bigint`. Upgrading
|
||||
either crate requires checking that this shared-type invariant still holds and
|
||||
running the math + runtime test suites.
|
||||
|
||||
### 3. The wire encoding never sends an integer as a JSON number
|
||||
|
||||
Canonical encoding (`crates/marqdo-value/src/wire.rs`; full contract in
|
||||
`spec/value-wire.md`):
|
||||
|
||||
```json
|
||||
{"type":"int","decimal":"12345678901234567890"}
|
||||
```
|
||||
|
||||
* Every value is a JSON object with a `"type"` tag.
|
||||
* `Int` uses a base-10 string. All digits, sign and magnitude survive any JSON
|
||||
consumer including JavaScript.
|
||||
* `Float` uses a JSON number only when finite; `NaN`, `Infinity`,
|
||||
`-Infinity` use explicit `"special"` tags.
|
||||
* JSON numbers appear only in non-value positions (`span.start/end`,
|
||||
`plugin_handle.id`).
|
||||
* Decoding is validated: unknown tags, missing fields, invalid decimals, wrong
|
||||
types and limits are `WireError`s, never coercions.
|
||||
* The same encoding is used for plugin ABI request/response values, foreign
|
||||
kernel stdin `input`, foreign typed return values and `run --json` values.
|
||||
|
||||
### 4. Math results preserve exactness where possible
|
||||
|
||||
The math kernel keeps exact rationals and returns `Value::Int` exactly when the
|
||||
exact result is an integer, otherwise `Value::Float`. Decimal literals are
|
||||
exact rationals (`0.25` = `1/4`), and `\frac{1}{3} * 3` is exactly `1`. Constants
|
||||
(`\pi`, `\e`) and non-perfect square roots introduce binary64 approximations and
|
||||
the result is a `Float` (`spec/math.md` §3).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **`i64`/`u64` integers with overflow errors.** Rejected: common programs
|
||||
(factorials, powers) would need explicit bignum workarounds; the language
|
||||
would inherit fixed-width semantics not implied by its arithmetic.
|
||||
2. **`i128` integers.** Same problem, just a bigger ceiling; and the wire
|
||||
precision problem remains for values beyond 2^53.
|
||||
3. **Encode integers as JSON numbers and hope consumers use bignum JSON.**
|
||||
Rejected: the JSON standard/ecosystem does not guarantee it; JavaScript's
|
||||
`JSON.parse` loses digits; this is the exact S-08 failure.
|
||||
4. **Encode every number as a string without type tags.** Rejected: loses the
|
||||
`Int`/`Float` distinction and makes all consumers parse strings; type tags
|
||||
are explicit and cheap.
|
||||
5. **Use `f64` for all numbers.** Rejected: contradicts exact `Int`
|
||||
semantics, the math kernel's exact rational requirement, and the
|
||||
no-silent-precision-loss rule.
|
||||
6. **Different integer crates in value vs math.** Rejected: forces lossy or
|
||||
string conversions at every kernel boundary and duplicates tests.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No AST change. Numeric literals live inside executable payloads only (code
|
||||
spans/plain text of `strong`/`emph`/conditions). A `math` node is inert LaTeX;
|
||||
its numeric interpretation is the math kernel's job. The JSON inspection view
|
||||
of IR/AST contains spans as JSON numbers, not user numeric values.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* `Int` semantics (arbitrary precision, `/` → `Float`, exact comparisons,
|
||||
`//`/`%` sign rules) are frozen language behavior; changing them requires an
|
||||
ADR and conformance fixtures.
|
||||
* `wire_version = 1` with typed tags and decimal-string integers is frozen.
|
||||
Changing the integer representation is a wire-breaking change and needs a
|
||||
new `wire_version` plus plugin ABI considerations.
|
||||
* The dependency pin may move within `0.4` for security fixes, provided
|
||||
`num-bigint` remains a single resolved version shared with `num-rational`.
|
||||
* Package/lock formats are unaffected.
|
||||
|
||||
## Security
|
||||
|
||||
* Untrusted wire data cannot trigger huge allocations through the integer
|
||||
parser without also counting against value depth/string limits: a decimal
|
||||
string is bounded by `max_string_bytes` (16 MiB default) and `BigInt` parsing
|
||||
is linear. The current decoder does not separately cap the *digit count* of
|
||||
an `Int`; the envelope/string caps are the effective bound (open item).
|
||||
* Arithmetic can grow memory; the runtime bounds it with fuel and the
|
||||
`Int` power guards (`MAX_EXACT_EXPONENT`, `MAX_POWER_BITS`) and the math
|
||||
kernel bounds exact numerator/denominator bits (`MAX_RESULT_BITS`).
|
||||
* Exact `Int`/`Float` comparison prevents a class of "looks equal but is not"
|
||||
authorization or verification bugs.
|
||||
* No new capabilities; numeric parsing is pure.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-value/src/value.rs`: `big_integers_are_arbitrary_precision`,
|
||||
`float_to_bigint_requires_integral_value`, display/format tests.
|
||||
* `crates/marqdo-value/src/wire.rs`: `big_integers_keep_all_digits`,
|
||||
`special_floats_are_tagged`, scalar round trips, untrusted-input limits.
|
||||
* `crates/marqdo-runtime/src/value_ops.rs`: exact integer arithmetic,
|
||||
`/`→Float, floor/mod sign rules, exact mixed comparisons, power guards,
|
||||
division-by-zero.
|
||||
* `crates/marqdo-math/tests/kernel.rs`: exact big integer arithmetic, exact
|
||||
rational division, decimal literals as rationals, `pi`/`sqrt` approximating
|
||||
to `Float`, `Int` vs `Float` result typing.
|
||||
* `crates/marqdo-plugin-abi/src/wire.rs`: request round trips preserving tags
|
||||
and refusing coercible-looking values beyond `i64`.
|
||||
* `crates/marqdo-foreign/tests/kernels.rs`: typed foreign return values decode
|
||||
exactly.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration: the implementation has used `BigInt` and the typed wire format
|
||||
since M0.5. For future dependency upgrades: run `cargo test --workspace`,
|
||||
verify one resolved `num-bigint` (`cargo tree -i num-bigint`), and re-run wire
|
||||
compatibility tests. For any change to the wire integer encoding, bump
|
||||
`wire_version`, update `spec/value-wire.md`, the plugin ABI documentation and
|
||||
the foreign protocol in one coordinated change.
|
||||
@@ -0,0 +1,286 @@
|
||||
# Marqdo Contributor Guide
|
||||
|
||||
This guide is for people changing the Marqdo CLI workspace. It assumes you have
|
||||
read `AGENTS.md` (the non-negotiable architectural rules) and
|
||||
`docs/DEVELOPMENT_BLUEPRINT.md` §0/§1/§11/§13/§14. Formal contracts live in
|
||||
`spec/`; decisions live in `docs/adr/`.
|
||||
|
||||
## 1. Repository map
|
||||
|
||||
```text
|
||||
marqdo-cli/
|
||||
├── Cargo.toml workspace (15 crates, resolver 2)
|
||||
├── AGENTS.md operator rules for contributors/agents
|
||||
├── README.md user-facing quickstart and limits
|
||||
├── docs/
|
||||
│ ├── DEVELOPMENT_BLUEPRINT.md architecture baseline
|
||||
│ ├── STATUS.md implementation matrix (P0–P7)
|
||||
│ ├── contributor-guide.md this file
|
||||
│ └── adr/0001..0007-*.md accepted decisions
|
||||
├── spec/
|
||||
│ ├── markdown-profile.md frozen parser profile `marqdo.md.v1`
|
||||
│ ├── mapping.md AST → program roles
|
||||
│ ├── expression.ebnf executable payload grammar
|
||||
│ ├── evaluation.md values, numbers, control flow, builtins
|
||||
│ ├── references.md imports/references/calls/ref()
|
||||
│ ├── value-wire.md typed wire_version 1 values
|
||||
│ ├── security.md modes, capabilities, budgets, trust
|
||||
│ ├── package-format.md marqdo.yaml / marqdo.lock / init
|
||||
│ ├── plugin-abi.md C ABI v1, SDK, host lifecycle
|
||||
│ ├── math.md Formula subset and math.* semantics
|
||||
│ └── foreign.md Python/Node kernel protocol
|
||||
├── crates/
|
||||
│ ├── marqdo-span/ Span, LineCol, LineIndex (leaf)
|
||||
│ ├── marqdo-value/ Value, diagnostics, errors, wire encoding
|
||||
│ ├── marqdo-expression/ lexer/parser/AST for executable payloads
|
||||
│ ├── marqdo-markdown/ comrak profile, canonical AST, front matter
|
||||
│ ├── marqdo-ir/ Module/Unit/Function/Stmt IR, builtin table
|
||||
│ ├── marqdo-frontend/ units, map, signature, infer, calls, resources
|
||||
│ ├── marqdo-math/ Formula parser/evaluator (exact rationals)
|
||||
│ ├── marqdo-foreign/ Python/Node one-shot kernel supervisor
|
||||
│ ├── marqdo-sandbox/ modes + platform capability detection
|
||||
│ ├── marqdo-plugin-abi/ C ABI mirrors, manifest, artifact, loader, wire
|
||||
│ ├── marqdo-plugin-sdk/ Rust plugin SDK and `mq_plugin_entry!`
|
||||
│ ├── marqdo-host/ CapabilityHost, budgets, FS, math/foreign/plugin adapters
|
||||
│ ├── marqdo-runtime/ tree-walk interpreter, value ops, builtins
|
||||
│ ├── marqdo-package/ marqdo.yaml / marqdo.lock types + discovery
|
||||
│ └── cli/ composition root: flags, commands, loader, output
|
||||
├── include/marqdo_plugin_abi_v1.h frozen C ABI v1 header
|
||||
├── examples/ hello, factorial, math, references, python-foreign,
|
||||
│ multi-module/, plugin-demo/
|
||||
├── tests/
|
||||
│ ├── conformance/ast/ *.mq.md + *.ast.json AST goldens
|
||||
│ ├── conformance/semantics/ *.mq.md + *.expected.json execution goldens
|
||||
│ ├── negative/ compile-time + runtime hard-error fixtures
|
||||
│ ├── README.md corpus layout and golden workflow
|
||||
│ ├── integration/ properties/ security/ abi/ reserved dirs (see STATUS)
|
||||
└── crates/cli/tests/ Rust harnesses for the corpus and CLI
|
||||
```
|
||||
|
||||
## 2. Dependency direction (must stay acyclic)
|
||||
|
||||
```text
|
||||
span -> serde
|
||||
value -> span
|
||||
expression -> span
|
||||
markdown -> span, comrak (no semantic crate!)
|
||||
ir -> span, expression
|
||||
frontend -> markdown, expression, ir, value
|
||||
host -> value, math, foreign, sandbox
|
||||
runtime -> ir, value, host, expression
|
||||
math -> value
|
||||
foreign -> value
|
||||
plugin-abi -> value
|
||||
plugin-sdk -> plugin-abi
|
||||
package -> serde only
|
||||
sandbox -> serde only
|
||||
cli -> everything (composition root)
|
||||
```
|
||||
|
||||
Rules (from blueprint §11.1, adjusted by ADR-0001):
|
||||
|
||||
* `markdown` owns the only Markdown parser; it must not depend on semantic
|
||||
crates, the host or the runtime. The single exception to "zero Marqdo deps"
|
||||
is `marqdo-span`, which is a dep-free leaf shared by all layers.
|
||||
* `frontend`/`ir`/`runtime` never depend on `cli`.
|
||||
* `runtime` never parses Markdown and never performs I/O directly: all effects
|
||||
go through `marqdo_host::Host`.
|
||||
* `host` never depends on the CLI; kernels/plugins are adapted here.
|
||||
* The CLI is the composition root: flag parsing, project loading, rendering and
|
||||
process exit. It must not become a second interpreter.
|
||||
* Check the graph with `cargo tree` (or `cargo tree -i <crate>`); `runtime ->
|
||||
cli`, `markdown -> frontend`, and dependency cycles are review blockers.
|
||||
|
||||
## 3. Adding a feature or fixture (workflow)
|
||||
|
||||
The project uses a fixture-first workflow (blueprint §13/§14, AGENTS §
|
||||
"Development workflow"). The expected sequence is:
|
||||
|
||||
1. **Find the rule.** Identify the affected spec section and blueprint section.
|
||||
If the change alters observable language semantics, package/lock format,
|
||||
ABI, authorization defaults or public CLI behavior, write/supersede an ADR
|
||||
first (`docs/adr/`, template in blueprint §16.2).
|
||||
2. **Write a failing fixture.** Add a small `.mq.md` positive and/or negative
|
||||
case under the right directory:
|
||||
* AST shape: `tests/conformance/ast/NNN-name.mq.md` + golden
|
||||
`NNN-name.ast.json`;
|
||||
* semantics: `tests/conformance/semantics/NNN-name.mq.md` +
|
||||
`NNN-name.expected.json` (typed wire `args`/`value` + captured `stdout`);
|
||||
* hard error: `tests/negative/NNN-name.mq.md` + `{"codes":[...]}`;
|
||||
* inert-executable warning: `tests/negative/warnings/`;
|
||||
* runtime error: `tests/negative/runtime/` with `{"function","args","codes"}`
|
||||
(optional `grants`, `stack_limit`).
|
||||
Run the relevant harness (see §6) and confirm it fails for the expected
|
||||
reason.
|
||||
3. **Update the spec.** `spec/*.md` is normative. If implementation and spec
|
||||
disagree, fix whichever is wrong and note the tradeoff. Do not silently
|
||||
invent behavior in code and leave the spec stale.
|
||||
4. **Implement the smallest module.** Keep the layers separated: parser
|
||||
(`marqdo-markdown`), semantic mapping (`marqdo-frontend`), IR
|
||||
(`marqdo-ir`), interpreter (`marqdo-runtime`), effects (`marqdo-host`),
|
||||
composition (`cli`). Never add a second Markdown parser, a regex-based
|
||||
syntax decision, or a host bypass.
|
||||
5. **Refresh goldens deliberately** (§4) and run the full gate (§6).
|
||||
6. **Write the tradeoff/limitations** in `docs/STATUS.md` when a P-level claim
|
||||
changes, and in the PR description (paths, commands run, skipped tests).
|
||||
|
||||
If a fixture reveals an implementation bug, fix the code, not the expected
|
||||
behavior. Negative fixtures must assert hard errors — never weaken them to
|
||||
warnings or prose fallbacks.
|
||||
|
||||
## 4. Golden update workflow
|
||||
|
||||
AST goldens (`tests/conformance/ast/*.ast.json`) are the serialized canonical
|
||||
AST from `marqdo_markdown::parse(source).root`. Regenerate **only** for an
|
||||
intentional, ADR-backed profile change:
|
||||
|
||||
```bash
|
||||
UPDATE_GOLDENS=1 cargo test -p marqdo-cli --test conformance_ast
|
||||
```
|
||||
|
||||
The update run rewrites every golden, prints the count and passes. Re-run
|
||||
without the variable and review the diff: **a golden change is a language
|
||||
change, not a bug fix**. A fixture without a golden and a golden without a
|
||||
fixture are both errors.
|
||||
|
||||
Semantic goldens (`*.expected.json`): `UPDATE_SEMANTICS_GOLDENS=1` exists for
|
||||
authoring convenience, but never use it blindly — verify every value against the
|
||||
fixture source first. Invalid, inconsistent or unverified expected values defeat
|
||||
the corpus.
|
||||
|
||||
Negative goldens are hand-written expected code lists (exact sorted multiset for
|
||||
compile-time errors; warning suites; runtime suites). Do not auto-generate them.
|
||||
|
||||
Adding/removing a fixture and its golden must happen in the same change.
|
||||
Fixtures are data: tests must not commit, copy or rewrite files outside a
|
||||
throw-away temp directory. `tests/README.md` is the in-repo reference for corpus
|
||||
counts, harness responsibilities and known observations.
|
||||
|
||||
## 5. Coding conventions
|
||||
|
||||
* Rust edition 2021, `rust-version = 1.82`, workspace version `0.1.0`.
|
||||
* `unsafe` is forbidden in pure crates (`marqdo-math`, `marqdo-expression` use
|
||||
`#![forbid(unsafe_code)]`); the ABI/loader crates contain the only
|
||||
`unsafe` code and every block has a SAFETY comment explaining the contract.
|
||||
* No panics on untrusted input: Markdown, expressions, LaTeX, wire data,
|
||||
manifests and foreign output all produce structured errors. `unwrap()`/
|
||||
`expect()` are acceptable only for genuinely impossible invariants or in
|
||||
tests.
|
||||
* Stable diagnostic/runtime **codes** are part of the contract; human messages
|
||||
are not. Add new codes to `crates/marqdo-value/src/diagnostic.rs` and map
|
||||
categories in `error.rs`.
|
||||
* Values cross layers via `marqdo_value::Value` and the typed wire format
|
||||
(`spec/value-wire.md`); never serialize `Int` as a JSON number.
|
||||
* `check`/`inspect` must remain side-effect free: no execution, no foreign
|
||||
process, no plugin `dlopen`, no file writes.
|
||||
* Prose and executable nodes mix freely; prose never causes an action.
|
||||
* Formatting: `rustfmt.toml` is authoritative; keep `cargo fmt --check` clean.
|
||||
|
||||
## 6. Build, test and quality gate
|
||||
|
||||
The workspace is offline-cached. Use the workspace cache as documented in
|
||||
`tests/README.md` / AGENTS.md:
|
||||
|
||||
```bash
|
||||
export PATH=/root/.cargo/bin:$PATH
|
||||
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
|
||||
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
|
||||
```
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
cargo build -p marqdo-cli # binary: target/debug/marqdo
|
||||
cargo test --workspace # all crate unit + integration tests
|
||||
cargo test -p marqdo-cli # conformance corpus + CLI integration tests
|
||||
cargo fmt --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
Harness map (`crates/cli/tests/`):
|
||||
|
||||
| Harness | Proves |
|
||||
|---|---|
|
||||
| `conformance_ast.rs` | frozen profile parses fixtures into checked-in AST (kinds, attributes, literals, spans, order) |
|
||||
| `conformance_semantics.rs` | frontend + IR + runtime + host produce expected wire value/stdout |
|
||||
| `negative.rs` | invalid executable shapes are hard errors; warning and runtime suites |
|
||||
| `properties.rs` | prose-invariance (pure narrative edits do not change executable IR modulo spans) |
|
||||
| `integration_cli.rs` | real CLI contract: init/check/run/call/inspect/fmt, multi-module, lock/sync, plugin trust and invocation |
|
||||
|
||||
Crate-level tests worth knowing:
|
||||
|
||||
* `marqdo-expression` — grammar/priority/nesting/limit tests;
|
||||
* `marqdo-math/tests/kernel.rs` — exact arithmetic, bounds, equations, unsupported constructs;
|
||||
* `marqdo-foreign/tests/kernels.rs` — real python3/node (skips if absent), timeout, output caps, env, wire protocol;
|
||||
* `marqdo-plugin-abi/tests/loader_unix.rs` — compiles the C demo with `cc`, loads it, ABI-mismatch/negative cases (skips without `cc`);
|
||||
* `marqdo-runtime/tests/interpreter.rs` — recursion, loops, returns, budgets, ambiguity;
|
||||
* `marqdo-host` unit tests — capabilities, project FS, budgets.
|
||||
|
||||
Tests that need Python/Node/`cc` print a skip notice and pass when unavailable;
|
||||
record such skips in the PR.
|
||||
|
||||
## 7. Running examples
|
||||
|
||||
From the repository root (after `cargo build -p marqdo-cli`):
|
||||
|
||||
```bash
|
||||
B=target/debug/marqdo
|
||||
$B run examples/hello.mq.md
|
||||
$B call examples/factorial.mq.md 数学.阶乘 --arg n=5 # prints 120
|
||||
$B check examples/references.mq.md
|
||||
$B inspect examples/references.mq.md --refs
|
||||
$B inspect examples/hello.mq.md --ast --json
|
||||
$B run examples/math.mq.md # has no main: use call
|
||||
$B call examples/math.mq.md 数学.计算圆面积
|
||||
$B run examples/python-foreign.mq.md --allow-foreign # needs python3
|
||||
$B run examples/multi-module/index.mq.md
|
||||
(cd examples/multi-module && ../../$B lock && ../../$B sync --locked)
|
||||
|
||||
# Plugin demo (build the artifact + refresh its sha256 first)
|
||||
./examples/plugin-demo/build.sh
|
||||
$B plugin inspect examples/plugin-demo
|
||||
$B run examples/plugin-demo/plugin-call.mq.md \
|
||||
--plugin examples/plugin-demo --trust-native-plugin --allow-plugin demo.solver
|
||||
```
|
||||
|
||||
Notes: `run` defaults to `index.mq.md` and executes `main`; `call` requires a
|
||||
file and function; root-only examples (for example `math.mq.md`) define no
|
||||
`main`. `--mode strict` is expected to refuse to run (ADR-0004).
|
||||
|
||||
## 8. Docs, ADRs and commit policy
|
||||
|
||||
* `spec/` = normative contracts; `docs/` = architecture/decisions/status;
|
||||
`README.md` = user quickstart; `AGENTS.md` = contributor rules.
|
||||
* Changes to language semantics, AST mapping, wire format, plugin ABI,
|
||||
package/lock format, authorization defaults or public CLI behavior require an
|
||||
**Accepted ADR** (or superseding one) plus spec updates and tests. Use the
|
||||
blueprint §16.2 template: Context / Decision / Alternatives / Markdown AST
|
||||
shape / Compatibility / Security / Testing / Migration.
|
||||
* ADRs are immutable once Accepted: add a new ADR with
|
||||
`Status: Superseded by ADR-NNNN` rather than rewriting history (typo fixes
|
||||
aside).
|
||||
* Docs must describe what the code does. When implementation and blueprint
|
||||
disagree, document the implementation and flag the divergence explicitly as
|
||||
a limitation/open item; never invent behavior.
|
||||
* Do not edit generated/vendored state, and never modify tests to hide a
|
||||
regression.
|
||||
* Commit in reviewable units with a message stating the affected spec/ADR and
|
||||
the commands run. Do not commit built plugin artifacts (`*.so` is
|
||||
gitignored), the cargo cache, or temp projects.
|
||||
* There is no git-commit automation in this workspace; contributors commit
|
||||
manually after review.
|
||||
|
||||
## 9. Known gotchas
|
||||
|
||||
* `.cache/` is the offline cargo cache/target used in this environment; it is
|
||||
not part of the deliverable.
|
||||
* Foreign availability (`python3`/`node`) and a C compiler (`cc`) determine
|
||||
which tests run; a skipped test is not a passed feature.
|
||||
* The strict sandbox, plugin worker isolation and package registry are not
|
||||
implemented. Do not write docs or tests that pretend they are — see
|
||||
`docs/STATUS.md` and `spec/security.md`.
|
||||
* Runtime error spans in the CLI currently resolve against the entry file only;
|
||||
imported-module error locations are approximate (ADR-0006).
|
||||
* The semantic corpus may pin current quirks (for example the external-link
|
||||
call code, or a float's last digit). A fixture diff for such a quirk is a
|
||||
deliberate change, not a free update.
|
||||
+2
-2
@@ -124,7 +124,7 @@ host boundary. Names are stable; adding one requires a spec update.
|
||||
| `整数` / `int` | `(x) -> Int` | from Int/Float(if integral)/Text(decimal) |
|
||||
| `小数` / `float` | `(x) -> Float` | from Int/Float/Text |
|
||||
| `表` / `table` | `(header: List, rows: List) -> Table` | validates row widths |
|
||||
| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address |
|
||||
| `math.parse/eval/simplify/differentiate` | see `spec/math.md` | P5 |
|
||||
| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address; module-local, refuses ambiguous resource units (`ref.ambiguous`) |
|
||||
| `math.parse/eval/substitute` | see `spec/math.md` | P5; `math.simplify`/`math.differentiate` are declared but return `math.unsupported` in this build |
|
||||
| `foreign.run/call` | see `spec/foreign.md` | P5, capability-gated |
|
||||
| `fs.read/write` | see `spec/security.md` | capability-gated |
|
||||
+221
@@ -0,0 +1,221 @@
|
||||
# Marqdo Foreign Kernels v1
|
||||
|
||||
Status: **one-shot kernels for Python and Node/JavaScript**, frozen protocol for
|
||||
M0.5. There are no sessions or persistent kernels.
|
||||
|
||||
Implementation: `crates/marqdo-foreign/src/lib.rs` (process supervision and
|
||||
protocol), `crates/marqdo-host/src/kernels.rs::foreign` (capability check and
|
||||
Marqdo mapping), runtime tests `crates/marqdo-foreign/tests/kernels.rs`,
|
||||
example `examples/python-foreign.mq.md`.
|
||||
|
||||
## 1. A code fence is a value, not a process
|
||||
|
||||
A `code_block` node becomes an inert `Code { language, source, content_hash,
|
||||
unit }` resource (`crates/marqdo-frontend/src/resources.rs`,
|
||||
`marqdo_value::CodeValue`). Reading, checking, rendering or importing a
|
||||
document never starts a foreign process. A process is created only when
|
||||
`foreign.run` or `foreign.call` is executed after the capability check.
|
||||
|
||||
Access is by `ref("#heading")` (exactly one code/formula resource in that
|
||||
heading range; `ref.ambiguous` otherwise — see `spec/references.md` §5):
|
||||
|
||||
````markdown
|
||||
## Python 校验
|
||||
|
||||
```python
|
||||
import json, math, sys
|
||||
envelope = json.load(sys.stdin)
|
||||
r = int(envelope["input"]["entries"]["r"]["decimal"])
|
||||
print(json.dumps({"type": "float", "value": math.pi * r ** 2}))
|
||||
```
|
||||
|
||||
## main
|
||||
|
||||
**结果 = foreign.call(ref("#python-校验"), {"r": 3})**
|
||||
|
||||
> 打印(结果)
|
||||
````
|
||||
|
||||
The anchor is the GitHub-style slug of the heading (`#python-校验`), not the
|
||||
heading text.
|
||||
|
||||
## 2. Supported languages
|
||||
|
||||
`normalize_language` is ASCII case-insensitive and trims surrounding
|
||||
whitespace:
|
||||
|
||||
| Canonical | Accepted aliases | Interpreter lookup |
|
||||
|---|---|---|
|
||||
| `python` | `python`, `python3`, `py` | `python3`, then `python` on `PATH` |
|
||||
| `javascript` | `javascript`, `js`, `node`, `nodejs` | `node` on `PATH` |
|
||||
|
||||
Any other language yields `ForeignError::KernelUnavailable` (the `Code` value is
|
||||
still valid data). `detect(language)` returns `KernelInfo { language, program,
|
||||
version }` by scanning `PATH` and running the interpreter with `--version`
|
||||
(5 s timeout, 64 KiB output cap, cleared environment except `PATH`); the
|
||||
`version` is the first non-empty line of stdout or stderr. `detect` and
|
||||
execution use the same `PATH`-only environment.
|
||||
|
||||
## 3. `RunRequest` / `RunResult`
|
||||
|
||||
```rust
|
||||
pub struct RunRequest {
|
||||
pub language: String,
|
||||
pub source: String,
|
||||
pub input: Option<serde_json::Value>, // an already-typed wire value
|
||||
pub timeout: std::time::Duration,
|
||||
pub max_output_bytes: usize,
|
||||
}
|
||||
|
||||
pub struct RunResult {
|
||||
pub status: RunStatus, // Success | Failed | TimedOut
|
||||
pub stdout: String, // lossy UTF-8 for `run`
|
||||
pub stderr: String, // preserved on failure
|
||||
pub exit_code: Option<i32>, // None on signal/timeout
|
||||
pub duration: std::time::Duration,
|
||||
pub value: Option<Value>, // only `call` on success
|
||||
}
|
||||
```
|
||||
|
||||
`RunRequest::new` defaults to a 30 s timeout and 1 MiB output cap **for direct
|
||||
library use**. The CLI host constructs requests with `StdHost` defaults:
|
||||
`foreign_timeout = 10 s`, `foreign_output_limit = 1 MiB` (stdout+stderr
|
||||
combined). The CLI has no per-kernel flag for these; `--timeout` only sets the
|
||||
interpreter wall-clock deadline.
|
||||
|
||||
`RunStatus::as_str()` yields `success`, `failed`, `timed_out`.
|
||||
|
||||
## 4. stdin envelope
|
||||
|
||||
The program is executed as a file; stdin receives exactly one JSON object
|
||||
(`crates/marqdo-foreign/src/lib.rs::request_envelope`):
|
||||
|
||||
```json
|
||||
{
|
||||
"wire_version": 1,
|
||||
"kind": "request",
|
||||
"input": <type-tagged wire value, or null>
|
||||
}
|
||||
```
|
||||
|
||||
`input` is **already encoded with the type-tagged value format**
|
||||
(`spec/value-wire.md`), not a plain JSON mirror of the Marqdo value. For
|
||||
example `{"r": 3}` becomes
|
||||
`{"type":"map","entries":{"r":{"type":"int","decimal":"3"}}}`. The example
|
||||
kernel reads `envelope["input"]["entries"]["r"]["decimal"]` accordingly. When
|
||||
`input` is absent, the field is JSON `null` and the write side closes stdin
|
||||
right after writing (EOF).
|
||||
|
||||
Marqdo values that cannot cross the wire (none in v1, but see
|
||||
`spec/value-wire.md` limits) are encoded before the process starts; the
|
||||
envelope may be size-limited by the host's output/wire budgets.
|
||||
|
||||
## 5. `run` versus `call`
|
||||
|
||||
| | `foreign.run(ref, input?)` | `foreign.call(ref, input)` |
|
||||
|---|---|---|
|
||||
| One-shot isolated execution | yes | yes |
|
||||
| stdin envelope | yes (input optional) | yes (input required) |
|
||||
| stdout contract | captured as data; any content | must be **exactly one** `wire_version` 1 typed value; no logs/trailing noise |
|
||||
| result | `Map` with `status`, `stdout`, `stderr`, `exit_code`, `duration_ms`, `value` (always `None` for `run`) | decoded `Value` |
|
||||
| non-zero exit | `status: "failed"` in the returned map | `Ok(RunResult { status: Failed })` from the library, but the host turns it into `foreign.failure` runtime error |
|
||||
| timeout | library returns `Ok` with `status: "timed_out"`; host returns the map | library returns `ForeignError::Timeout`; host maps to `resource.exhausted` |
|
||||
| stdout pollution / invalid wire | irrelevant | `ForeignError::Protocol` → `foreign.failure` |
|
||||
|
||||
`call` parses stdout with `serde_json` and then `marqdo_value::from_wire` with
|
||||
default limits; an unknown `type` tag, bad decimal, JSON syntax error, invalid
|
||||
UTF-8 or trailing bytes are all protocol errors. A successful call whose stdout
|
||||
has no typed value is a protocol error at the host boundary ("foreign call
|
||||
returned no typed value" / `foreign.failure`).
|
||||
|
||||
The host result map from `foreign.run` (`crates/marqdo-host/src/kernels.rs::
|
||||
foreign_result_map`) is:
|
||||
|
||||
```text
|
||||
{status: "success"|"failed"|"timed_out",
|
||||
stdout: Text, stderr: Text, exit_code: Int|None,
|
||||
duration_ms: Int, value: None}
|
||||
```
|
||||
|
||||
## 6. Process model and limits
|
||||
|
||||
`crates/marqdo-foreign/src/lib.rs`:
|
||||
|
||||
1. normalize the language (unknown → `KernelUnavailable` with install hint);
|
||||
2. find the interpreter on `PATH`;
|
||||
3. create a **private unique temp working directory** (mode 0700 on Unix) and
|
||||
write the program to a private temp file (`.py`/`.js`, mode 0600);
|
||||
4. spawn the interpreter **directly with `argv`** (no shell, no string
|
||||
interpolation), cwd = the fresh temp dir, environment cleared except
|
||||
`PATH`, stdin/stdout/stderr piped;
|
||||
5. a dedicated thread writes the stdin envelope and closes the pipe; reader
|
||||
threads capture stdout and stderr against a **shared** `max_output_bytes`
|
||||
budget; the supervisor polls `try_wait` and kills the child on the wall-clock
|
||||
deadline or output overflow;
|
||||
6. after the child is reaped, readers get a 2 s grace to finish; cleanup (Drop)
|
||||
removes the temp file and directory on success and error paths.
|
||||
|
||||
Bounds: wall-clock `timeout` (10 s in the CLI), combined output
|
||||
`max_output_bytes` (1 MiB in the CLI; stderr counts too), reader grace 2 s,
|
||||
detection timeout 5 s / 64 KiB. A kernel that ignores stdin and exits does not
|
||||
hang the host. `OutputTooLarge` drops the captured output and kills the child.
|
||||
Grandchild processes are not killed when the direct child is killed.
|
||||
|
||||
Environment: only `PATH` is inherited, so Marqdo secrets in other environment
|
||||
variables do not leak by default. This is hygiene, not a security boundary.
|
||||
|
||||
## 7. Capability gating
|
||||
|
||||
`foreign.run`/`foreign.call` are effectful builtins
|
||||
(`crates/marqdo-ir/src/builtins.rs`). The host adapter
|
||||
(`crates/marqdo-host/src/kernels.rs::foreign`) first checks
|
||||
`Capabilities::check_foreign`, which requires both `foreign` and `process`.
|
||||
|
||||
* default (`Capabilities::pure()`/`project()`): denied →
|
||||
`permission.denied` "foreign kernel execution was not granted";
|
||||
* `--allow-foreign` in `project` mode grants both flags;
|
||||
* `pure` mode rejects `--allow-foreign` at CLI parse time (exit 2);
|
||||
* `strict` mode refuses to run before any kernel is considered;
|
||||
* a manifest `capabilities.process`/foreign request never grants anything;
|
||||
* the check is repeated for every call (not cached at parse time).
|
||||
|
||||
If the first argument is not a `Code` value, the host returns `runtime.type`
|
||||
("foreign kernel expects a Code value from `ref("#代码块")`"). A code fence with
|
||||
an empty language is `kernel.unavailable` ("no kernel can be selected").
|
||||
Unsupported languages are `kernel.unavailable` with an install hint.
|
||||
|
||||
## 8. Error mapping
|
||||
|
||||
| Library error | Host runtime code |
|
||||
|---|---|
|
||||
| `KernelUnavailable { language, hint }` | `kernel.unavailable` (message includes hint) |
|
||||
| `Spawn { message }` | `host.error` |
|
||||
| `Io { message }` | `host.error` |
|
||||
| `Timeout { seconds }` | `resource.exhausted` |
|
||||
| `Protocol { message }` | `foreign.failure` |
|
||||
| `OutputTooLarge { limit }` | `resource.exhausted` |
|
||||
|
||||
`foreign.call` with a non-zero exit becomes `foreign.failure` with the stderr
|
||||
text preserved; a successful process whose stdout is not a typed value is also
|
||||
`foreign.failure`.
|
||||
|
||||
## 9. Security caveats
|
||||
|
||||
* **Process isolation is not a sandbox.** The child runs as the host user and
|
||||
can read/write any file the user can, open sockets, spawn processes and
|
||||
persist beyond the one-shot execution. Fresh cwd, cleared environment and a
|
||||
killed direct child are limited hygiene measures. Only `strict` mode
|
||||
(currently refused) would promise OS-level confinement.
|
||||
* There is no network policy for foreign kernels; `net` capability applies to
|
||||
host operations, not to what a child process does.
|
||||
* Output is captured, not inherited; the child cannot write to the CLI's
|
||||
stdout/stderr directly, but it can still affect the filesystem.
|
||||
* Kernel availability is environment-dependent (`python3`/`python`/`node` on
|
||||
`PATH`). Missing kernels must be fixable by the user (the error includes an
|
||||
install hint).
|
||||
* `foreign.run` returns raw stdout as data; consumers must not treat it as a
|
||||
typed value unless they parse it themselves. `foreign.call` is the typed
|
||||
path.
|
||||
* Sessions, shared state, persistent kernels and automatic environment
|
||||
management (uv/venv) are not implemented. Python/Node are optional foreign
|
||||
kernels, never the official execution backend.
|
||||
+227
@@ -0,0 +1,227 @@
|
||||
# Marqdo Math v1
|
||||
|
||||
Status: **frozen subset for M0.5.** The kernel computes numbers; it never proves
|
||||
theorems. `math.simplify` and `math.differentiate` are **not implemented** in
|
||||
this build.
|
||||
|
||||
Implementation: `crates/marqdo-math` (`parse`, `evaluate`, `substitute`,
|
||||
`MathError`), host adapter `crates/marqdo-host/src/kernels.rs::math`, builtin
|
||||
metadata `crates/marqdo-ir/src/builtins.rs`, tests
|
||||
`crates/marqdo-math/tests/kernel.rs`.
|
||||
|
||||
## 1. From Markdown to a value
|
||||
|
||||
A `math` / `math_block` node is an **inert document resource**, never an
|
||||
operation. The frontend attributes every formula to its innermost enclosing
|
||||
heading unit (`crates/marqdo-frontend/src/resources.rs`). `ref("#heading")`
|
||||
returns a `Formula` value only when that unit contains exactly one resource;
|
||||
with two or more formulas the runtime raises `ref.ambiguous` and refuses to
|
||||
pick one (`spec/references.md` §5, ADR-0003).
|
||||
|
||||
```markdown
|
||||
## 圆面积公式
|
||||
|
||||
$$
|
||||
A = \pi r^2
|
||||
$$
|
||||
|
||||
## 计算圆面积
|
||||
|
||||
**面积 = math.eval(ref("#圆面积公式"), {"r": 4}, target="A")**
|
||||
|
||||
*面积*
|
||||
```
|
||||
|
||||
Nothing is evaluated by reading or rendering the document. `math.*` must be
|
||||
called explicitly and the host checks the `math` capability at execution time
|
||||
(default: granted in `pure`/`project`; there is no CLI flag to revoke it today).
|
||||
|
||||
## 2. Grammar
|
||||
|
||||
The parser accepts exactly this LaTeX subset (whitespace insignificant; byte
|
||||
offsets are reported for parse errors):
|
||||
|
||||
```text
|
||||
formula = expr [ "=" expr ] ; single '=' only, never assignment
|
||||
expr = term { ("+" | "-") term } ;
|
||||
term = unary { ("*" | "/" | "·" | "×" | implicit) unary } ;
|
||||
unary = ("+" | "-") unary | power ;
|
||||
power = primary [ "^" unary ] ; right associative
|
||||
primary = number
|
||||
| identifier Unicode letter (plus digits/underscore)
|
||||
| "\pi" | "pi" | "\e" | "e" constants
|
||||
| "(" expr ")" | "\left(" expr "\right)"
|
||||
| "{" expr "}" grouping
|
||||
| "\frac{" expr "}{" expr "}" exact division
|
||||
| "\sqrt{" expr "}" real square root
|
||||
;
|
||||
number = digits [ "." digits ] | "." digits ; decimal, no exponent form
|
||||
```
|
||||
|
||||
Rules and edge cases:
|
||||
|
||||
* `\cdot` and `\times` are accepted as synonyms for `*`.
|
||||
* Implicit multiplication between adjacent factors: `2x`, `\pi r^2`, `(a)(b)`,
|
||||
`m a`. Concatenated letters form one identifier (`ma` is the variable `ma`;
|
||||
write `m a` or `m * a` for a product). Two adjacent number literals are a
|
||||
parse error ("ambiguous").
|
||||
* Unicode identifiers are supported (`面积`, `r`, `速度`). Plain multi-letter
|
||||
identifiers are variables, but names of common elementary functions (`sin`,
|
||||
`cos`, `log`, `exp`, ...) are reserved and rejected as unsupported so
|
||||
`sin(x)` is not silently read as a product.
|
||||
* Underscores are ordinary identifier characters: `x_1` is the variable `x_1`,
|
||||
not a subscript.
|
||||
* A single top-level `=` creates an `Equation { left, right }`; it is **not**
|
||||
assignment and never writes a Marqdo binding. A second `=` is a parse error.
|
||||
* `\frac{a}{b}` is exact division (`Div`); `1.5 + 2.5` is an exact rational
|
||||
computation, not binary floating point.
|
||||
* `\sqrt{x}` is real-valued: a negative radicand is a `Type` error.
|
||||
* Decimal literals have no exponent form (`1e3` is rejected); `2e` is
|
||||
`2 * e` (number times the constant), while `2e3` is rejected rather than
|
||||
reinterpreted.
|
||||
|
||||
### 2.1 Unsupported constructs
|
||||
|
||||
Everything outside the grammar is a structured `Unsupported` or `Parse` error:
|
||||
matrices, sums/integrals, function applications (`\sin`, `\cos`, ...),
|
||||
multi-letter commands, `\left[`/`\right]`, mismatched/missing braces,
|
||||
scientific notation, commas, arrays, subscripts, superscript commands other
|
||||
than the `^` operator, `\frac`/`\sqrt` without braced arguments, labels,
|
||||
alignment characters, line breaks. There is no `simplify`, `differentiate`,
|
||||
`integrate`, `matrix` or theorem/proof operation in this build.
|
||||
|
||||
## 3. Numeric semantics
|
||||
|
||||
* Exact rational arithmetic uses `num-rational::Ratio` over the
|
||||
`num-bigint::BigInt` type that also backs `Value::Int`; integers are
|
||||
arbitrary precision.
|
||||
* Decimal literals are exact rationals (`0.25` = `1/4`). `Int` bindings become
|
||||
exact integers. `Float` bindings are interpreted as the **exact binary
|
||||
rational value** of the `f64`; non-finite `Float` bindings are a `Type`
|
||||
error.
|
||||
* `+ - * / ^` are exact when both operands are exact. Division keeps a
|
||||
rational, so `\frac{1}{3} * 3` is exactly `1`.
|
||||
* The final value is a Marqdo `Int` exactly when the exact result is an
|
||||
integer, and a `Float` otherwise (`crates/marqdo-math/src/eval.rs`, the
|
||||
`Numeric` enum). Examples: `1.5 + 2.5` → `Int(4)`; `1 / 2` → `Float(0.5)`;
|
||||
`100 / 4` → `Int(25)`.
|
||||
* `\pi`, `\e` and a `\sqrt` of a non-perfect square introduce an `f64`
|
||||
approximation. Anything derived from them is a `Float`. Precision follows
|
||||
IEEE-754 binary64 (about 15–17 significant decimal digits); approximate
|
||||
results are never presented as exact.
|
||||
* `\sqrt` of a perfect square of an exact rational stays exact (`\sqrt{4}` →
|
||||
`Int(2)`); `\sqrt{1/4}` is `Float(0.5)` in this build (it becomes
|
||||
approximate via the root path, matching the documented test).
|
||||
* The exponent of `^` must evaluate to an integer. Magnitude is bounded by
|
||||
`MAX_EXPONENT`. Negative integer exponents are exact reciprocals (the result
|
||||
is therefore a `Float` unless it is integral). `0 ^ 0` is a `Type` error.
|
||||
* Exact intermediate/result numerators and denominators are bounded by
|
||||
`MAX_RESULT_BITS`; exceeding the bound returns a structured `Unsupported`
|
||||
error rather than hanging or exhausting memory.
|
||||
* Division by zero is a `Type` error inside the kernel. The host maps all
|
||||
`MathErrorKind::Type` to `runtime.type`; it does **not** produce
|
||||
the interpreter's `runtime.division_by_zero` code for a formula. This is a
|
||||
known divergence from the module doc comment in
|
||||
`crates/marqdo-math/src/lib.rs` and should be fixed or documented in an ADR.
|
||||
* Constants are reserved: a binding named `pi` or `e` never overrides the
|
||||
constant, and they cannot be solve targets (use `E`, `x`, `r`, ...).
|
||||
|
||||
### 3.1 Bounds
|
||||
|
||||
| Constant | Value | Location |
|
||||
|---|---|---|
|
||||
| `MAX_SOURCE_BYTES` | 1,000,000 | `crates/marqdo-math/src/parse.rs` |
|
||||
| `MAX_PARSE_DEPTH` | 256 | parse.rs |
|
||||
| `MAX_AST_NODES` | 512 | parse.rs |
|
||||
| `MAX_EXPONENT` | 10,000 | `crates/marqdo-math/src/eval.rs` |
|
||||
| `MAX_RESULT_BITS` | 1,000,000 | eval.rs |
|
||||
|
||||
The AST-node bound also bounds later recursion for long operator chains, so
|
||||
hostile input cannot overflow the stack during evaluation, rendering or
|
||||
substitution. No code path panics on untrusted LaTeX; all failures are
|
||||
`MathError`.
|
||||
|
||||
## 4. Equations and solving a target
|
||||
|
||||
`FormulaKind` is `Expression(expr)` or `Equation { left, right }`.
|
||||
|
||||
`evaluate(formula, bindings, target)`:
|
||||
|
||||
* for a plain expression, `target` must be `None`; supplying one is
|
||||
`Unsupported` ("a target variable is only meaningful for an equation");
|
||||
* for an equation, `target` is required (`Unsupported` otherwise); an empty
|
||||
target is `Unsupported`;
|
||||
* the target must occur **exactly once** and the equation must be **affine in
|
||||
the target**. Other variables are substituted from `bindings` first; a
|
||||
missing binding is `UndefinedVariable`.
|
||||
|
||||
Affine solving supported by `crates/marqdo-math/src/eval.rs`:
|
||||
|
||||
* `+`, `-`, multiplication by a target-free side, division where the target is
|
||||
in the numerator, and inversion of those operations in either direction:
|
||||
`2x = 4`, `4 = 2x`, `x / 2 = 3`, `3 = x / 2`, `x + 2 = 5`, `2x + 1 = 7`,
|
||||
`(x + 1) * 2 = 8`, `x = 2y + 1` (with `y` bound);
|
||||
* `F = m a` solved for `F`, `m` or `a`.
|
||||
|
||||
Rejected as `Unsupported`: target occurs zero times or more than once, target
|
||||
in a denominator, under a root, in an exponent, or in a non-affine product
|
||||
(e.g. `x * x`). Equations never mutate bindings and never create Marqdo
|
||||
variables.
|
||||
|
||||
## 5. Substitution
|
||||
|
||||
`substitute(formula, bindings)` replaces bound variables with exact numeric
|
||||
literals and keeps unbound variables symbolic. The returned `Formula.source` is
|
||||
a deterministic canonical rendering that `parse` accepts again; it is not
|
||||
byte-identical to the input formatting. Host-facing `math.substitute` wraps the
|
||||
result in a `Formula` value with `unit: None`, `display: false`, span of the
|
||||
call site.
|
||||
|
||||
## 6. Host-facing surface (`math.*`)
|
||||
|
||||
All operations are builtins in `crates/marqdo-ir/src/builtins.rs`, dispatched
|
||||
in `crates/marqdo-runtime/src/builtins_impl.rs`, and executed in the host
|
||||
adapter `crates/marqdo-host/src/kernels.rs`. `math` capability is checked
|
||||
before touching the kernel.
|
||||
|
||||
| Builtin | Parameters | Result |
|
||||
|---|---|---|
|
||||
| `math.parse` | `formula: Formula or Text` | `Formula` (source = input; `unit=None`, `display=false`); parse errors are `math.unsupported` |
|
||||
| `math.eval` | `formula: Formula or Text`, `bindings: Map`, optional `target: Text` | `Int` or `Float`; equation requires target |
|
||||
| `math.substitute` | `formula: Formula or Text`, `bindings: Map` | `Formula` with canonical source |
|
||||
| `math.simplify` | `formula` | **not implemented**: runtime error `math.unsupported` ("not part of the supported math subset in this build") |
|
||||
| `math.differentiate` | `formula`, `变量` | **not implemented**: `math.unsupported` |
|
||||
|
||||
Input typing:
|
||||
|
||||
* formula arguments accept a `Formula` value or `Text`; anything else is
|
||||
`runtime.type`;
|
||||
* `bindings` must be a `Map` whose values are `Int` or `Float`; any other
|
||||
value type is `runtime.type` with a message naming the binding;
|
||||
* `target` must be `Text` or `None`; other types are `runtime.type`;
|
||||
* builtin arity/parameter-name mistakes are catchable statically
|
||||
(`call.arity`, see `spec/mapping.md` §6);
|
||||
* the math kernel is a pure computation: no file, network, process or plugin
|
||||
effects, and nothing it returns is a capability.
|
||||
|
||||
### 6.1 Error mapping
|
||||
|
||||
Host mapping (`crates/marqdo-host/src/kernels.rs::math_error`):
|
||||
|
||||
| `MathErrorKind` | Runtime code |
|
||||
|---|---|
|
||||
| `Parse` | `math.unsupported` (message includes the byte offset when known) |
|
||||
| `Unsupported` | `math.unsupported` |
|
||||
| `UndefinedVariable` | `symbol.unknown` |
|
||||
| `Type` | `runtime.type` |
|
||||
|
||||
## 7. What this is not
|
||||
|
||||
* Not a computer-algebra system: no simplification, differentiation,
|
||||
integration, matrices, series, assumptions, units or symbolic rewriting.
|
||||
* Not a theorem prover: `F = m a` is an equation value, not a proof, and a
|
||||
numeric solve is not a certificate.
|
||||
* Not a LaTeX renderer: rendering is the Markdown reader's job; the kernel only
|
||||
receives the raw `latex` text of a `math` node.
|
||||
* Not error-free in the floating-point sense: approximation precision is
|
||||
documented, not guaranteed beyond binary64.
|
||||
@@ -0,0 +1,258 @@
|
||||
# Marqdo Project Package Format v1
|
||||
|
||||
Status: **v1 format as implemented in M0.5.** Git resolution, registries,
|
||||
transitive dependency solving and environment materialization are **not**
|
||||
implemented; see §8.
|
||||
|
||||
Implementation: `crates/marqdo-package/src/lib.rs` (manifest/lock types and
|
||||
validation), `crates/cli/src/commands.rs` (`init`, `add`, `remove`, `lock`,
|
||||
`sync`).
|
||||
|
||||
## 1. Project root
|
||||
|
||||
The project root is the nearest ancestor directory of the entry document that
|
||||
contains `marqdo.yaml` (`marqdo_package::discover_root`). If no manifest exists
|
||||
anywhere above the entry, the entry file's parent directory is used as the
|
||||
root (a document can be checked/run without a project).
|
||||
|
||||
`ModuleId`s are project-root-relative paths (`spec/references.md` §2.1), so the
|
||||
root choice affects import identity. `add`/`remove`/`lock`/`sync` discover the
|
||||
root from the **current working directory**, not from a document path. There is
|
||||
no `--project` flag in this build.
|
||||
|
||||
## 2. `marqdo.yaml`
|
||||
|
||||
```yaml
|
||||
project:
|
||||
name: example
|
||||
version: 0.1.0
|
||||
marqdo: ">=0.1,<1" # optional; informational in this build
|
||||
dependencies:
|
||||
geometry:
|
||||
path: ./packages/geometry # exactly one of path / git+rev
|
||||
quantum:
|
||||
git: https://github.com/example/quantum-mq
|
||||
rev: 0123456789abcdef0123456789abcdef01234567
|
||||
plugins:
|
||||
solver:
|
||||
version: "^1.0" # requirement only; not installed or authorized
|
||||
capabilities: # requests only; not grants
|
||||
fs:
|
||||
read: ["./data/**"]
|
||||
write: ["./output/**"]
|
||||
net:
|
||||
hosts: []
|
||||
process: false
|
||||
```
|
||||
|
||||
### 2.1 Fields
|
||||
|
||||
| Field | Type | Required | Semantics |
|
||||
|---|---|---|---|
|
||||
| `project.name` | string | yes | non-empty package name |
|
||||
| `project.version` | string | yes | non-empty version string (not semver-validated in the manifest) |
|
||||
| `project.marqdo` | string | no | required Marqdo runtime range; stored, not enforced by this build |
|
||||
| `dependencies.<name>.path` | path string | one of path/git | project-relative path to a dependency directory or `.mq.md` file |
|
||||
| `dependencies.<name>.git` | URL string | one of path/git | git source (must be pinned with `rev`) |
|
||||
| `dependencies.<name>.rev` | string | with `git` | exact revision; a bare `git` without `rev` is rejected |
|
||||
| `plugins.<name>.version` | string | per entry | non-empty plugin version requirement; declarative only |
|
||||
| `capabilities.fs.read` / `.write` | list of strings | no | requested project-relative globs |
|
||||
| `capabilities.net.hosts` | list of strings | no | requested hosts |
|
||||
| `capabilities.process` | bool | no | requested process execution |
|
||||
|
||||
Unknown fields are ignored (the types do not `deny_unknown_fields`). Duplicate
|
||||
YAML keys follow `serde_yaml` behavior (last one wins); there is no duplicate-key
|
||||
diagnostic.
|
||||
|
||||
### 2.2 Validation (`Manifest::validate`)
|
||||
|
||||
`marqdo.yaml` is loaded with `Manifest::from_yaml`/`Manifest::load`, which
|
||||
validates:
|
||||
|
||||
* `project.name` and `project.version` are non-empty after trimming;
|
||||
* for every dependency, exactly one source is present:
|
||||
* both `path` and `git` → error "declares both path and git";
|
||||
* neither → error "must declare a path or a pinned git source";
|
||||
* `git` without `rev` → error requiring an exact revision;
|
||||
* an absolute `path` → error "path must be project-relative";
|
||||
* for every plugin requirement, `version` is non-empty.
|
||||
|
||||
Note: a *git* dependency passes manifest validation when pinned, but the
|
||||
dependency is not resolvable or lockable in this build (§5).
|
||||
|
||||
### 2.3 `capabilities` are requests
|
||||
|
||||
The manifest is the project's **request** for permissions. It is never read
|
||||
when constructing the host capability set; only CLI flags/grant APIs create
|
||||
grants (`spec/security.md` §2). `net` has no implementing host op, and
|
||||
`process`/foreign execution still needs `--allow-foreign`.
|
||||
|
||||
## 3. `marqdo.lock`
|
||||
|
||||
`lock_version: 1`, packages sorted by name:
|
||||
|
||||
```yaml
|
||||
lock_version: 1
|
||||
packages:
|
||||
- name: geo
|
||||
source: path:./packages/geo
|
||||
sha256: 101a6f6db4cc48d8c138b4716028da9bc8934b5f02b2235e020fbcabba15b757
|
||||
```
|
||||
|
||||
| Field | Semantics |
|
||||
|---|---|
|
||||
| `lock_version` | must be `1`; any other value is rejected ("unsupported lock_version N (expected 1)") |
|
||||
| `packages[].name` | dependency name from the manifest |
|
||||
| `packages[].source` | `path:<manifest path as written>`; a git source would be `git:...@rev`, but git is not implemented in this build |
|
||||
| `packages[].sha256` | optional; lowercase hex SHA-256 of the dependency tree |
|
||||
|
||||
### 3.1 Deterministic serialization
|
||||
|
||||
`LockFile::to_yaml` sorts packages by name before serializing. The format
|
||||
contains no timestamps, no absolute paths and no machine-specific values, so
|
||||
repeated `marqdo lock` runs over unchanged dependencies are byte-identical.
|
||||
`LockFile::from_yaml` also sorts packages on load, so equality comparison is
|
||||
order-insensitive.
|
||||
|
||||
### 3.2 Tree hash
|
||||
|
||||
For a path dependency, `lock` hashes the dependency tree
|
||||
(`crates/cli/src/commands.rs::hash_tree`):
|
||||
|
||||
1. walk the dependency directory with `walkdir` (`sort_by_file_name`), keeping
|
||||
regular files only (symlinks are not followed and are skipped);
|
||||
2. sort files by their root-relative path (normalized to `/`);
|
||||
3. feed into SHA-256, for each file in order:
|
||||
`relative_path bytes`, a `0x00` byte, `file bytes`, a `0x00` byte;
|
||||
4. hex-encode the digest.
|
||||
|
||||
This is a deterministic content fingerprint, not a Merkle tree and not a
|
||||
cryptographic commitment to file modes, symlinks or empty directories. It
|
||||
detects content changes and additions/removals of regular files.
|
||||
|
||||
### 3.3 Lock generation rules
|
||||
|
||||
`marqdo lock` (run in a project):
|
||||
|
||||
1. discovers the project root from the cwd;
|
||||
2. loads and validates `marqdo.yaml`;
|
||||
3. for each manifest dependency, in map (sorted) order:
|
||||
* a git source → error "git dependencies are not supported by this build
|
||||
(path dependencies only)"; no lock is written;
|
||||
* no path source → error;
|
||||
* the dependency path must exist; the tree is hashed;
|
||||
4. writes `marqdo.lock` (exit 0, `locked N package(s)`).
|
||||
|
||||
What it does **not** do: resolve semver ranges, traverse a dependency's own
|
||||
`marqdo.yaml`/transitive dependencies, clone git repositories, download
|
||||
registries, verify package signatures, or install anything.
|
||||
|
||||
## 4. `sync`
|
||||
|
||||
`marqdo sync [--locked]` (run in a project):
|
||||
|
||||
1. discovers the root; loads the manifest and the existing `marqdo.lock`
|
||||
(missing lock → error "run `marqdo lock` first", exit 1);
|
||||
2. recomputes the expected lock file exactly as `lock` would (including tree
|
||||
hashes);
|
||||
3. with `--locked`, compares the expected lock with the on-disk lock; a
|
||||
difference (manifest change or dependency content change) prints
|
||||
"lockfile is out of date or dependency contents changed; run `marqdo lock`"
|
||||
and exits 1;
|
||||
4. without `--locked`, prints `synchronized N package(s)` and exits 0.
|
||||
|
||||
Current behavior: `sync` **never rewrites** the lock file and never materializes
|
||||
a `.marqdo/env` directory. It is a verification command only. There is no
|
||||
`--offline` flag and no `lock --locked` flag (the blueprint example
|
||||
`marqdo lock --locked` is not implemented; use `sync --locked`).
|
||||
|
||||
## 5. `init`
|
||||
|
||||
`marqdo init [dir]`:
|
||||
|
||||
* refuses to overwrite an existing `marqdo.yaml` (exit 1);
|
||||
* creates the directory if needed;
|
||||
* derives the project name from the directory's final path component (falling
|
||||
back to `example`);
|
||||
* writes `marqdo.yaml` from `Manifest::template`:
|
||||
|
||||
```yaml
|
||||
project:
|
||||
name: <name>
|
||||
version: 0.1.0
|
||||
marqdo: '>=0.1,<1'
|
||||
```
|
||||
|
||||
(empty dependencies/plugins/capabilities are omitted);
|
||||
* writes `index.mq.md`:
|
||||
|
||||
```markdown
|
||||
## main
|
||||
|
||||
**问候 = "你好,Marqdo"**
|
||||
|
||||
> 打印(问候)
|
||||
|
||||
*问候*
|
||||
```
|
||||
|
||||
* writes `.gitignore` containing `.marqdo/`;
|
||||
* prints the next-step hint `marqdo check index.mq.md && marqdo run index.mq.md`.
|
||||
|
||||
`init` does not create `marqdo.lock`, `.marqdo/`, a plugin directory or a git
|
||||
repository.
|
||||
|
||||
## 6. `add` / `remove`
|
||||
|
||||
Both commands edit **only** `marqdo.yaml` (and, for `remove`, prune a matching
|
||||
stale lock entry). They do not fetch, install, resolve or authorize anything.
|
||||
|
||||
* `marqdo add <name> --path <dir>` or
|
||||
`marqdo add <name> --git <url> --rev <sha>`:
|
||||
validate the dependency shape (same rules as §2.2), refuse an existing name
|
||||
(exit 1), insert it, write the manifest, and tell the user to run
|
||||
`marqdo lock`. Adding a git dependency is allowed at the manifest level even
|
||||
though `lock` will refuse it later.
|
||||
* `marqdo remove <name>`: remove the dependency from the manifest (exit 1 if
|
||||
absent), and drop a same-named entry from an existing lock file. It does not
|
||||
re-hash or re-lock the rest of the project.
|
||||
|
||||
This differs from the original task brief that listed `add`/`remove` as
|
||||
"not supported yet": the CLI does implement them as manifest editors. They
|
||||
are intentionally shallow and should not be confused with a dependency
|
||||
resolver.
|
||||
|
||||
## 7. JSON output
|
||||
|
||||
`check`, `inspect`, `run`, `call`, `plugin inspect`, `add`, `remove`, `lock`
|
||||
and `sync` accept `--json` and print a single JSON object on stdout. Relevant
|
||||
package shapes:
|
||||
|
||||
* `lock`: `{"command":"lock","success":true,"packages":[...]}`
|
||||
* `sync`: `{"command":"sync","success":true,"locked":<bool>,"packages":[...]}`
|
||||
* `add`: `{"command":"add","success":true,"dependency":"<name>","manifest":"<path>"}`
|
||||
* `remove`: `{"command":"remove","success":true,"dependency":"<name>"}`
|
||||
|
||||
Errors in these commands are printed as plain `error: ...` on stderr even when
|
||||
`--json` is passed (the JSON success envelope is not emitted; this is an
|
||||
inconsistency to fix).
|
||||
|
||||
## 8. Not supported in this build (explicit)
|
||||
|
||||
| Capability | Status |
|
||||
|---|---|
|
||||
| Git dependency resolution (clone/fetch/checkout) | Not implemented. Manifest accepts `git`+`rev`; `lock` refuses it. |
|
||||
| Registry / package publishing / SemVer resolution | Not implemented. |
|
||||
| Transitive dependency graph and conflict resolution | Not implemented; only direct manifest dependencies are hashed. |
|
||||
| `tree`, `update`, `vendor`, `package` commands | Not implemented. |
|
||||
| `--offline`, `--locked` on `lock` | Not implemented. |
|
||||
| Environment materialization (`.marqdo/env`, `cache`, `kernels`, `sessions`) | Not implemented; `init` only writes `.marqdo/` to `.gitignore`. |
|
||||
| Dependency signatures / trusted publishers / supply-chain scanning | Not implemented. |
|
||||
| Multiple/nested project boundary diagnostics | Not implemented; nearest `marqdo.yaml` wins. |
|
||||
| Lock file migration between `lock_version`s | Only version 1 is accepted. |
|
||||
| Plugin installation / plugin dependency management | Not implemented; `plugins:` in the manifest is declarative only. |
|
||||
|
||||
The lock/package format is deliberately small and deterministic so these can
|
||||
be added later without changing the identity model. Changing `lock_version`,
|
||||
`source` syntax or the tree-hash algorithm requires an ADR and a migration
|
||||
plan.
|
||||
@@ -0,0 +1,419 @@
|
||||
# Marqdo Plugin ABI v1
|
||||
|
||||
Status: **frozen ABI v1.0** (`MQ_ABI_MAJOR=1`, `MQ_ABI_MINOR=0`). The canonical
|
||||
C header is `include/marqdo_plugin_abi_v1.h`; the Rust mirrors are in
|
||||
`crates/marqdo-plugin-abi/src/abi.rs`; the plugin-side SDK is
|
||||
`crates/marqdo-plugin-sdk`.
|
||||
|
||||
> **A C ABI is not a sandbox.** A native plugin is loaded into the host
|
||||
> process. It can read host memory, call arbitrary syscalls, bypass the
|
||||
> capability model and crash the host. Loading requires an explicit trust
|
||||
> decision. The isolated worker transport described by the blueprint is **not
|
||||
> implemented** in this build.
|
||||
|
||||
## 1. Independent versions
|
||||
|
||||
Three versions exist and are never conflated:
|
||||
|
||||
| Version | Current | Where |
|
||||
|---|---|---|
|
||||
| Marqdo language/semantics | M0.5, `spec/*` | `docs/DEVELOPMENT_BLUEPRINT.md`, specs |
|
||||
| Plugin ABI major.minor | `1.0` | `MQ_ABI_MAJOR`/`MQ_ABI_MINOR`, `marqdo_plugin_abi::ABI_MAJOR/ABI_MINOR` |
|
||||
| Plugin package SemVer | e.g. `0.1.0` | `plugin.yaml` `plugin.version` |
|
||||
|
||||
The JSON wire protocol has its own `wire_version = 1`
|
||||
(`spec/value-wire.md`); it is not the ABI major version.
|
||||
|
||||
## 2. Stable-layout types
|
||||
|
||||
The header defines only fixed-width integers, `size_t`-based length+pointer
|
||||
buffers, opaque host-owned handles and `extern "C"` function pointers. Rust
|
||||
`String`/`Vec`/trait objects/`Result`/panics, C++ exceptions, Go objects and
|
||||
raw host pointers never cross the boundary.
|
||||
|
||||
On a 64-bit target (`include/marqdo_plugin_abi_v1.h` static assertions and
|
||||
`crates/marqdo-plugin-abi/src/abi.rs` compile-time assertions):
|
||||
|
||||
| Type | Size | Align |
|
||||
|---|---|---|
|
||||
| `MqBytes` | 16 | 8 |
|
||||
| `MqOwnedBytes` | 16 | 8 |
|
||||
| `MqPluginApi` | 32 | 8 |
|
||||
| `MqHostApi` | 56 | 8 |
|
||||
| `MqPluginInvoke` (fn ptr) | 8 | — |
|
||||
|
||||
On a 32-bit target the Rust side asserts `MqBytes`/`MqOwnedBytes` = 8,
|
||||
`MqPluginApi` = 20 and `MqHostApi` = 28. The layout uses `size_t`, so an
|
||||
artifact is only loadable by a host built for the **same target triple, pointer
|
||||
width and byte order**. `(NULL, 0)` is the only empty buffer; a non-empty
|
||||
buffer with a NULL pointer is a protocol error.
|
||||
|
||||
### 2.1 Ownership and lifetimes
|
||||
|
||||
| Buffer | Producer | Lifetime / free rule |
|
||||
|---|---|---|
|
||||
| `args_wire`, `export_name`, `signature_wire`, capability `request_wire`, `plugin_metadata_wire` (as seen by host) | caller passing it | borrowed for the duration of the call; consumer must copy and must not retain or free |
|
||||
| `result_wire`, `error_wire`, capability `reply_wire` | the producing side | must be allocated with `host->alloc(host->host_data, size)` and is released with `host->free(host->host_data, data)` by the consumer after copying; never `malloc`/`new`/GC memory |
|
||||
| `plugin_metadata_wire` (as returned by plugin) | plugin static storage | valid only during `mq_plugin_entry_v1`; host copies it immediately |
|
||||
|
||||
`MqCallContext` is opaque, host-owned and valid only for the invocation it was
|
||||
minted for. Plugins must not dereference, cache or copy it.
|
||||
|
||||
### 2.2 Size limits (enforced by host / SDK)
|
||||
|
||||
| Constant | Bytes |
|
||||
|---|---|
|
||||
| `MQ_MAX_EXPORT_NAME_BYTES` | 256 |
|
||||
| `MQ_MAX_SIGNATURE_WIRE_BYTES` | 4096 |
|
||||
| `MQ_MAX_METADATA_WIRE_BYTES` | 65536 |
|
||||
| `MQ_MAX_REQUEST_WIRE_BYTES` | 16 MiB (16,777,216) |
|
||||
| `MQ_MAX_RESULT_WIRE_BYTES` | 64 MiB (67,108,864) |
|
||||
| `MQ_MAX_ERROR_WIRE_BYTES` | 1 MiB (1,048,576) |
|
||||
| `MQ_MAX_CAPABILITY_REPLY_BYTES` | 16 MiB |
|
||||
|
||||
Wire JSON decoding additionally applies the default `WireLimits` from
|
||||
`spec/value-wire.md` (depth 64, 1,000,000 items, 16 MiB strings). Oversized
|
||||
payloads are rejected, never truncated and reparsed.
|
||||
|
||||
## 3. Exported entry point and tables
|
||||
|
||||
Required symbol: `mq_plugin_entry_v1` (`MQ_PLUGIN_ENTRY_SYMBOL`):
|
||||
|
||||
```c
|
||||
int32_t mq_plugin_entry_v1(const MqHostApi *host, MqPluginApi *plugin);
|
||||
```
|
||||
|
||||
Called exactly once, by the loading thread:
|
||||
|
||||
* the host supplies a valid `MqHostApi` table and a zeroed `MqPluginApi` whose
|
||||
`struct_size = sizeof(MqPluginApi)`;
|
||||
* the plugin must validate the host table, fill `plugin.abi_major`,
|
||||
`plugin.abi_minor`, `plugin.plugin_metadata_wire` and register every export
|
||||
via `host->register_function`;
|
||||
* it returns `MQ_OK` only after all exports are registered; on non-zero the
|
||||
host ignores the plugin table and fails the load.
|
||||
|
||||
`MQ_OK` and error codes (stable integers):
|
||||
|
||||
| Code | Name | Meaning |
|
||||
|---|---|---|
|
||||
| 0 | `MQ_OK` | success |
|
||||
| 1 | `MQ_ERR_INVALID_ARGUMENT` | NULL pointer, bad pointer/length pair |
|
||||
| 2 | `MQ_ERR_ABI_MISMATCH` | ABI major mismatch or incompatible `struct_size` |
|
||||
| 3 | `MQ_ERR_REGISTRATION` | an export could not be registered |
|
||||
| 4 | `MQ_ERR_UNSUPPORTED` | operation/type/capability unsupported |
|
||||
| 5 | `MQ_ERR_OUT_OF_MEMORY` | host allocation failed |
|
||||
| 6 | `MQ_ERR_BAD_WIRE` | payload is not valid `wire_version` 1 JSON |
|
||||
| 7 | `MQ_ERR_INTERNAL` | unexpected failure, including caught panic |
|
||||
| 8 | `MQ_ERR_CAPABILITY` | capability request denied/failed |
|
||||
|
||||
### 3.1 Version negotiation
|
||||
|
||||
Host side (`negotiate` in `crates/marqdo-plugin-abi/src/abi.rs`):
|
||||
|
||||
* `plugin.abi_major != 1` → load fails with `abi_major_mismatch`;
|
||||
* `plugin.abi_minor` greater than the host minor is accepted but **clamped** to
|
||||
the host minor (currently 0); only features defined at that minor may be
|
||||
used;
|
||||
* the plugin must reject a host table with `host == NULL`,
|
||||
`host->struct_size < sizeof(MqHostApi)`, `host->abi_major != 1` or NULL
|
||||
alloc/free/register_function/request_capability pointers.
|
||||
|
||||
`struct_size` is the extension mechanism: fields may only be appended in a
|
||||
later ABI minor, never reordered or resized; a consumer only reads a field when
|
||||
`struct_size` covers it.
|
||||
|
||||
### 3.2 Threading and lifetime
|
||||
|
||||
v1 is single-threaded and synchronous: entry is called once; invocations and
|
||||
host callbacks for one plugin instance run on that thread with at most one call
|
||||
in flight. A plugin must not start threads that call back into `MqHostApi`.
|
||||
v1 has no shutdown callback and no hot unload/reload; the `LoadedPlugin`
|
||||
wrapper unloads only when dropped, and the docs require one instance per
|
||||
artifact path (ABI v1 plugins may keep the host table in a process-global).
|
||||
No panic/exception/longjmp may cross the boundary; the SDK catches Rust panics
|
||||
and returns `MQ_ERR_INTERNAL` with a structured error envelope.
|
||||
|
||||
## 4. `plugin.yaml`
|
||||
|
||||
```yaml
|
||||
plugin:
|
||||
name: demo.solver
|
||||
version: 0.1.0
|
||||
abi: "^1.0"
|
||||
kind: native
|
||||
exports:
|
||||
- name: solver.add
|
||||
signature: "($a: Int, $b: Int) -> Int"
|
||||
requires:
|
||||
- math.compute
|
||||
artifacts:
|
||||
- target: x86_64-unknown-linux-gnu
|
||||
file: solver.so
|
||||
sha256: "<64 lowercase hex>"
|
||||
```
|
||||
|
||||
Validation (`crates/marqdo-plugin-abi/src/manifest.rs::PluginManifest::validate`):
|
||||
|
||||
* `name`: dotted segments, each starting with a lowercase ASCII letter and
|
||||
containing only lowercase letters, digits, `-`/`_`; ≤ 256 bytes; package
|
||||
names starting with `marqdo.` or `builtin.` are reserved;
|
||||
* `version`: semver-ish `major.minor.patch` with optional `-prerelease` and
|
||||
`+build`; numeric components have no leading zeros; not a SemVer range
|
||||
resolver;
|
||||
* `abi`: caret-style requirement `^1`, `^1.0`, `1` or `1.0`. Caret semantics:
|
||||
`^M.m` means `>=M.m, <M+1.0`; the `0.x` special case pins the minor. It must
|
||||
be compatible with the host's `1.0`;
|
||||
* `kind`: `native`, `worker` or `wasm`; only `native` is loadable in-process in
|
||||
this build;
|
||||
* `exports`: at least one, at most 256, unique, namespaced (must contain `.`),
|
||||
no reserved prefix (`marqdo.`, `builtin.`, `core.`, `std.`, `host.`),
|
||||
non-empty signature ≤ 4096 bytes, signature starts with `(` and contains
|
||||
`->`, no control characters;
|
||||
* `requires`: at most 64, unique names from the whitelist `log`,
|
||||
`math.compute`, `fs.read`, `fs.write`, `net.request`, `clock`, `random`,
|
||||
`artifact.record`; this is a declaration, not a grant;
|
||||
* `artifacts`: 1–32, one per target triple; target parses as a 3–4 component
|
||||
triple (lowercase, alphanumeric/underscore components); `file` is a
|
||||
non-empty relative path ≤ 256 bytes with no absolute/`..` escape; `sha256` is
|
||||
exactly 64 hex characters.
|
||||
|
||||
Manifest parsing/validation never executes plugin code and performs no
|
||||
filesystem access (artifact verification is a separate step).
|
||||
|
||||
## 5. Artifact selection and verification
|
||||
|
||||
* `HostTarget::current()` derives a conventional triple from
|
||||
`std::env::consts` (`x86_64-unknown-linux-gnu`, `aarch64-apple-darwin`,
|
||||
`x86_64-pc-windows-msvc`, ...).
|
||||
* `resolve_artifact` requires **exact triple equality**; pointer width and byte
|
||||
order follow from the triple. No fallback to another architecture.
|
||||
* `verify_artifact` resolves the path below the manifest directory and compares
|
||||
the file's SHA-256 with the manifest (case-insensitive hex). A missing file
|
||||
is `missing_artifact`; a digest mismatch is `hash_mismatch`.
|
||||
* Verification proves the artifact is the file described by the manifest; it
|
||||
proves nothing about trust or authorization.
|
||||
|
||||
## 6. Loader lifecycle (host side)
|
||||
|
||||
`crates/marqdo-plugin-abi/src/loader.rs::LoadedPlugin::open_with`:
|
||||
|
||||
1. `manifest.validate()`;
|
||||
2. require `kind == native` (otherwise `plugin_kind_not_loadable_in_process`);
|
||||
3. resolve artifact for the host target and verify its sha256;
|
||||
4. `libloading::Library::new` (a trust decision, not a sandbox);
|
||||
5. build the host API table and `HostState`, initialize `MqPluginApi`;
|
||||
6. call `mq_plugin_entry_v1` once; non-zero → structured load error;
|
||||
7. require `plugin_api.struct_size >= sizeof(MqPluginApi)`;
|
||||
8. negotiate ABI major/minor;
|
||||
9. copy `plugin_metadata_wire` and require metadata `name`/`version` to match
|
||||
the manifest exactly;
|
||||
10. take registered exports and cross-check against the manifest: duplicate
|
||||
registrations, missing declared exports, signature mismatches and
|
||||
undeclared exports all fail the load;
|
||||
11. keep the library mapped for the lifetime of `LoadedPlugin`.
|
||||
|
||||
Invocation (`LoadedPlugin::invoke`): validate the request envelope, check the
|
||||
request size, find the registered export, mint a `MqCallContext`, call the
|
||||
plugin's `MqPluginInvoke`, copy and free both output buffers, then enforce the
|
||||
envelope contract:
|
||||
|
||||
* success (`MQ_OK`): `error_wire` must be empty; `result_wire` must be a
|
||||
non-empty `wire_version` 1 success envelope; the typed value is optional
|
||||
(absent `value` means `Unit`);
|
||||
* failure: `error_wire` must hold a `wire_version` 1 error envelope; a missing
|
||||
or invalid error envelope still fails the call (structured
|
||||
`plugin_call_failed`).
|
||||
|
||||
Capability requests from the plugin go through
|
||||
`MqHostApi::request_capability` into the host's `CapabilityHandler`, on the
|
||||
same thread, one at a time. The handler receives the capability name and
|
||||
request bytes and returns reply bytes or an error. When no handler is
|
||||
installed (the CLI's case) **every request is denied** (`MQ_ERR_CAPABILITY`).
|
||||
|
||||
## 7. Wire envelopes
|
||||
|
||||
All payloads are UTF-8 JSON with `wire_version = 1`. Values inside `args` and
|
||||
`value` use the type-tagged encoding of `spec/value-wire.md`.
|
||||
|
||||
Request:
|
||||
|
||||
```json
|
||||
{
|
||||
"wire_version": 1,
|
||||
"kind": "request",
|
||||
"function": "solver.add",
|
||||
"args": {
|
||||
"a": {"type": "int", "decimal": "20"},
|
||||
"b": {"type": "int", "decimal": "22"}
|
||||
},
|
||||
"context": {"trace_id": "optional", "capability_handle": "opaque-and-scoped"}
|
||||
}
|
||||
```
|
||||
|
||||
Response (success):
|
||||
|
||||
```json
|
||||
{"wire_version":1,"kind":"response","value":{"type":"int","decimal":"42"}}
|
||||
```
|
||||
|
||||
A success response with neither `value` nor `error` means `Unit`. A response
|
||||
with both is rejected (`ambiguous_response`). Error:
|
||||
|
||||
```json
|
||||
{"wire_version":1,"kind":"response","error":{
|
||||
"kind":"invoke","code":"missing_argument",
|
||||
"message":"required argument `a` is missing","recoverable":true
|
||||
}}
|
||||
```
|
||||
|
||||
Envelope limits: at most `MAX_ARGS = 1024` request arguments; request ≤ 16 MiB,
|
||||
response ≤ 64 MiB, error ≤ 1 MiB. `context` carries no secrets and no
|
||||
reconstructable OS handles. `signature_wire` is
|
||||
`{"signature":"<signature string>"}`; `plugin_metadata_wire` is
|
||||
`{"name":"<package name>","version":"<semver>"}` (extra fields ignored).
|
||||
|
||||
## 8. Error model
|
||||
|
||||
Host-side errors are `PluginError { kind, code, message, detail?, recoverable }`
|
||||
with stable `kind` values: `manifest`, `abi`, `artifact`, `load`, `wire`,
|
||||
`capability`, `invoke`, `unsupported`. Examples: `abi_major_mismatch`,
|
||||
`no_matching_artifact`, `hash_mismatch`, `missing_entry_symbol`,
|
||||
`metadata_name_mismatch`, `export_signature_mismatch`,
|
||||
`unsupported_wire_version`. `message`/`detail` must never contain secrets.
|
||||
They are serializable and are used as the wire error body.
|
||||
|
||||
When a plugin is used from Marqdo code, host errors become structured
|
||||
`RuntimeError`s (`crates/marqdo-host/src/host.rs::map_plugin_error`):
|
||||
|
||||
| Plugin error kind | Runtime code |
|
||||
|---|---|
|
||||
| `abi` | `plugin.abi_mismatch` |
|
||||
| `capability` | `permission.denied` |
|
||||
| `wire` | `runtime.type` |
|
||||
| `unsupported` | `kernel.unavailable` |
|
||||
| `manifest`/`artifact`/`load`/`invoke` | `plugin.failure` |
|
||||
|
||||
## 9. Using a plugin from `.mq.md`
|
||||
|
||||
This build supports explicitly trusted **in-process native** plugins only.
|
||||
|
||||
```bash
|
||||
# 1. Build the C example and refresh its sha256 in plugin.yaml
|
||||
./examples/plugin-demo/build.sh
|
||||
|
||||
# 2. Inspect without loading
|
||||
marqdo plugin inspect examples/plugin-demo
|
||||
|
||||
# 3. Load with an explicit trust decision and a capability grant, then run
|
||||
marqdo run examples/plugin-demo/plugin-call.mq.md \
|
||||
--plugin examples/plugin-demo \
|
||||
--trust-native-plugin \
|
||||
--allow-plugin demo.solver
|
||||
```
|
||||
|
||||
Flags (`run` and `call`): `--plugin <manifest-or-dir>` (repeatable),
|
||||
`--trust-native-plugin`, `--allow-plugin <package-name>`. `--allow-plugin`
|
||||
matches the manifest `plugin.name` (for example `demo.solver`), not the call
|
||||
namespace (`solver`).
|
||||
|
||||
Document syntax (`examples/plugin-demo/plugin-call.mq.md`):
|
||||
|
||||
```markdown
|
||||
## main
|
||||
|
||||
**结果 = solver.add(a=20, b=22)**
|
||||
|
||||
> 打印(结果)
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
* the call namespace is derived from the **first** manifest export prefix
|
||||
(`solver.add` → namespace `solver`); a manifest whose exports use several
|
||||
prefixes registers only the first namespace, so exports under other prefixes
|
||||
are not reachable in this build;
|
||||
* calls accept **named arguments only**; positional arguments are a runtime
|
||||
type error ("plugin calls require named arguments");
|
||||
* each call becomes `HostOp::PluginInvoke` with a typed wire request; the
|
||||
plugin's typed response becomes a Marqdo value (`Unit` when absent);
|
||||
* the capability grant is re-checked on every call, not only at load;
|
||||
* plugin namespace calls are **not statically validated**: `check` passes a
|
||||
document that calls a namespace no plugin provides; at runtime the call fails
|
||||
with `symbol.unknown` ("no plugin namespace ... is loaded") unless
|
||||
`--plugin` loaded it;
|
||||
* plugin host-capability requests are denied because the CLI installs no
|
||||
capability handler (`LoadOptions::default()`);
|
||||
* `--mode strict` refuses to run before any plugin is loaded; `--mode pure`
|
||||
cannot grant `--allow-plugin`, so loading fails with `permission.denied`.
|
||||
|
||||
Trust boundary: `--trust-native-plugin` is an explicit unsafe decision; the
|
||||
process shares its address space with the plugin. In-process load is the only
|
||||
transport implemented — there is no worker process, Wasm component, hot
|
||||
unload, or plugin capability bridge. A native plugin crash is a host crash.
|
||||
|
||||
## 10. SDK usage (`crates/marqdo-plugin-sdk`)
|
||||
|
||||
The SDK hides the raw C layout and exports the entry symbol with
|
||||
`mq_plugin_entry!`. A complete Rust plugin crate:
|
||||
|
||||
```rust
|
||||
use marqdo_plugin_sdk::{mq_plugin_entry, CallContext, PluginError, Value};
|
||||
|
||||
fn add(_ctx: &mut CallContext, args: &[u8]) -> Result<Vec<u8>, PluginError> {
|
||||
let request = marqdo_plugin_sdk::wire::decode_request(args)?;
|
||||
let values = request.typed_args()?;
|
||||
let a = marqdo_plugin_sdk::wire::int_arg(&values, "a")?;
|
||||
let b = marqdo_plugin_sdk::wire::int_arg(&values, "b")?;
|
||||
marqdo_plugin_sdk::wire::success(Some(Value::Int((a + b).into())))
|
||||
}
|
||||
|
||||
mq_plugin_entry! {
|
||||
name: "demo.solver",
|
||||
version: "0.1.0",
|
||||
exports: {
|
||||
"solver.add" => "($a: Int, $b: Int) -> Int" => add,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The macro validates the host table, reports the plugin ABI version and metadata,
|
||||
registers every export and returns `MQ_OK` only if all registrations succeed.
|
||||
Handlers receive a `&mut CallContext`; use
|
||||
`CallContext::capability_request(name, request_bytes)` for an already-authorized
|
||||
host capability (the CLI grants none today), and
|
||||
`CallContext::log(level, message)` as a convenience. Panics are caught at the
|
||||
ABI boundary and converted to a structured error. The SDK depends only on
|
||||
`marqdo-plugin-abi`, never on host-only crates or the interpreter.
|
||||
|
||||
## 11. C demo (`examples/plugin-demo`)
|
||||
|
||||
`examples/plugin-demo/solver.c` is a dependency-free C implementation of one
|
||||
export (`solver.add`, `Int + Int`) using a minimal JSON scanner; it validates
|
||||
the host table, reports metadata and registers the export. `build.sh` compiles
|
||||
it (`cc -shared -fPIC -I include`) and rewrites the `sha256` in `plugin.yaml`
|
||||
(the checked-in manifest uses a zero placeholder until you build). The
|
||||
`examples/plugin-demo/README.md` walks through the security model, build,
|
||||
inspect and call flow.
|
||||
|
||||
The integration test `crates/marqdo-plugin-abi/tests/loader_unix.rs` compiles
|
||||
this exact `solver.c` with `cc`, loads it with `libloading`, round-trips a wire
|
||||
request, and rebuilds it with `-DMQ_SOLVER_REPORT_ABI_MAJOR=2` to prove that a
|
||||
wrong ABI major fails the load. It also covers a missing entry symbol, missing
|
||||
artifact, hash mismatch, malformed manifest and target mismatch. The tests skip
|
||||
when no C compiler is available.
|
||||
|
||||
## 12. Not implemented / explicit limitations
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| Isolated plugin worker transport (length-prefixed IPC) | Not implemented; `PlatformCapabilities::worker_processes = false`. |
|
||||
| Wasm Component + WIT | Not implemented. |
|
||||
| Hot unload/reload | Not supported by ABI v1 (no shutdown callback). |
|
||||
| Plugin capability bridge from the CLI | Not implemented; CLI supplies no handler, so all plugin host-capability requests are denied. |
|
||||
| Plugin installation by package manager | Not implemented; `marqdo.yaml` `plugins:` is declarative only. |
|
||||
| Static validation of plugin namespaces/exports/signatures against loaded plugins | Not implemented; errors are runtime-only. |
|
||||
| Multiple namespaces per plugin | Not implemented; single namespace from the first export prefix. |
|
||||
| Positional plugin arguments | Not supported; named only. |
|
||||
| Thread-safe / concurrent invocation | Not supported (single-threaded v1 contract). |
|
||||
| Old `marqdo_abi.h` v2 compatibility | Not implemented and not claimed; a separate adapter would be required. |
|
||||
@@ -0,0 +1,362 @@
|
||||
# Marqdo References, Imports and Calls v1
|
||||
|
||||
Status: **frozen for M0.5 except where marked "current limitation".** Language
|
||||
changes require an ADR plus updated tests.
|
||||
|
||||
Normative implementation:
|
||||
|
||||
| Concern | Code |
|
||||
|---|---|
|
||||
| Front matter parsing | `crates/marqdo-markdown/src/frontmatter.rs` |
|
||||
| Module loading, project root, cycles | `crates/cli/src/loader.rs` |
|
||||
| Module/Unit/Function identities | `crates/marqdo-ir/src/lib.rs` |
|
||||
| Heading scopes, anchors | `crates/marqdo-frontend/src/units.rs` |
|
||||
| Resource collection and attribution | `crates/marqdo-frontend/src/resources.rs` |
|
||||
| Link-call mapping | `crates/marqdo-frontend/src/map.rs` |
|
||||
| `ref(...)` runtime resolution | `crates/marqdo-runtime/src/builtins_impl.rs` |
|
||||
| Static link graph (`inspect --refs`) | `crates/cli/src/commands.rs` |
|
||||
| Cross-module call validation | `crates/marqdo-frontend/src/calls.rs` |
|
||||
|
||||
This file complements [`mapping.md`](mapping.md) (AST roles) and
|
||||
[`package-format.md`](package-format.md) (manifest/lock file).
|
||||
|
||||
## 1. Three edge kinds, never conflated
|
||||
|
||||
Blueprint §4.1 distinguishes three relations. The implementation keeps them
|
||||
separate and **only the call edge can execute code**.
|
||||
|
||||
| Edge | Written as | Resolved when | Executes? |
|
||||
|---|---|---|---|
|
||||
| **Import** | YAML front matter `import:` mapping | module load / static analysis | No. Imported modules only register definitions; their `main` never runs. |
|
||||
| **Reference** | Markdown `[label](url)` | `inspect --refs` (static graph) | No. Rendered/read as documentation only. |
|
||||
| **Call** | `> f(...)` block quote, link-form block quote `> [f](#f)(...)`, or a `CallExpr` in an executable payload | static validation, then execution | Yes, at runtime, through the interpreter. |
|
||||
|
||||
Consequences:
|
||||
|
||||
* A Markdown link never calls, imports or loads anything. `[计算作用力](#计算作用力)`
|
||||
is a reference; `> [计算作用力](#计算作用力)()` is a call.
|
||||
* `ref("#anchor")` is a **runtime builtin**, not an import; it only resolves an
|
||||
inert resource or function reference in the running module.
|
||||
* The reference graph is static metadata and creates no proof obligation.
|
||||
|
||||
## 2. Canonical identities
|
||||
|
||||
These types live in `crates/marqdo-ir/src/lib.rs`.
|
||||
|
||||
### 2.1 `ModuleId`
|
||||
|
||||
```rust
|
||||
pub struct ModuleId(pub String);
|
||||
```
|
||||
|
||||
* A canonical **project-root-relative path with `/` separators**, for example
|
||||
`index.mq.md`, `packages/geometry/index.mq.md`.
|
||||
* Computed in `crates/cli/src/loader.rs::Loader::module_id_for` by taking the
|
||||
canonicalized file path, stripping the project root, and joining the path
|
||||
components with `/`.
|
||||
* There is no "module name" identity separate from the path: the module id is
|
||||
the file location inside the project. Imported `@dependency` modules keep
|
||||
their project-root-relative path (for example
|
||||
`packages/geometry/index.mq.md`).
|
||||
* A file that canonicalizes outside the project root is rejected with
|
||||
"outside the project root" (entry: fatal load error; imported module:
|
||||
`module.unresolved` diagnostic).
|
||||
* Module ids use `BTreeMap` ordering, so program iteration and JSON output are
|
||||
deterministic.
|
||||
|
||||
### 2.2 `UnitId`
|
||||
|
||||
```rust
|
||||
pub struct UnitId { pub module: ModuleId, pub path: Vec<String> }
|
||||
```
|
||||
|
||||
* One addressable heading range. `path` is the canonical heading path from the
|
||||
outermost heading to this unit (root unit has an empty path).
|
||||
* Canonical heading names are NFC-normalized, trimmed, with internal
|
||||
whitespace collapsed to one space (`marqdo_markdown::normalize_name`).
|
||||
* `Display` renders `<module>#<path joined by '/'>`; the root unit renders as
|
||||
`<module>#`.
|
||||
* `UnitKind` is `root`, `namespace` (`#`), `function` (`##`) or `section`
|
||||
(`###`+). Sections are addressable but not callable.
|
||||
|
||||
### 2.3 `FunctionId`
|
||||
|
||||
```rust
|
||||
pub struct FunctionId { pub module: ModuleId, pub name: String }
|
||||
```
|
||||
|
||||
* `name` is the **namespace-qualified canonical function name**: `ns.fn` when
|
||||
the level-2 heading is inside a level-1 namespace, otherwise `fn`
|
||||
(`crates/marqdo-frontend/src/units.rs`). `Display` is `<module>::<name>`.
|
||||
* A function heading under a namespace yields both `FunctionId.name` (`ns.fn`)
|
||||
and `Function.local_name` (`fn`) plus `namespace: Some("ns")`.
|
||||
* Lookup (`Module::function`) first matches `id.name`, then a unique
|
||||
`local_name`; if several namespaces define the same local name the lookup
|
||||
returns `None` rather than guessing.
|
||||
* No overloads: duplicate canonical names in one scope are hard errors
|
||||
(`markdown.invalid_construct`, "duplicate heading name ... in the same
|
||||
scope") and are never auto-suffixed.
|
||||
|
||||
### 2.4 Anchors are a separate namespace, not identity
|
||||
|
||||
`UnitIndex.anchors: BTreeMap<String, UnitId>` maps GitHub-style heading slugs
|
||||
to units (`crates/marqdo-frontend/src/units.rs::github_slug`):
|
||||
|
||||
* digits/letters/`_` are kept and lowercased; spaces, `-` and tabs become `-`;
|
||||
all other punctuation is dropped;
|
||||
* collisions receive `-1`, `-2`, ... suffixes in document order (separate
|
||||
counters per base slug);
|
||||
* anchors exist for navigation and `ref("#...")`; they are **not** identity.
|
||||
Internal ids stay canonical, so a heading rename is a link-breaking change
|
||||
but never changes the canonical unit path.
|
||||
|
||||
## 3. Front matter imports
|
||||
|
||||
Imports exist only as an explicit YAML mapping in the document's front matter
|
||||
(recognized by the Markdown profile only at the start of the file):
|
||||
|
||||
```markdown
|
||||
---
|
||||
import:
|
||||
geometry: "@geometry"
|
||||
util: "./lib/util.mq.md"
|
||||
---
|
||||
|
||||
# main
|
||||
```
|
||||
|
||||
Parsing rules (`crates/marqdo-markdown/src/frontmatter.rs::imports`):
|
||||
|
||||
* the `import` key must be a mapping of **string alias → string target**;
|
||||
* aliases must be non-empty and unique within the document;
|
||||
* any other front matter key is metadata for tools and is ignored by the
|
||||
interpreter; invalid YAML is a `frontmatter.invalid` diagnostic, never
|
||||
silently dropped;
|
||||
* imports are never inferred: a link, a file name or a directory scan does not
|
||||
create an import;
|
||||
* the diagnostic span recorded for all imports of a file is the whole front
|
||||
matter span (`ImportDecl.span`).
|
||||
|
||||
### 3.1 Target forms
|
||||
|
||||
`Loader::resolve_import` recognizes exactly two forms:
|
||||
|
||||
| Target | Resolution |
|
||||
|---|---|
|
||||
| `@dependency` | Look up the alias after `@` in `marqdo.yaml` `dependencies`. The dependency must declare a **path** source. Resolve `project_root / path`; if it is a directory, append `index.mq.md`; canonicalize. A git-only dependency cannot be resolved in this build (`dependency.path` is `None` → unresolved). |
|
||||
| any other string (intended: `./file.mq.md`) | Join the string to the **importing file's directory**, then canonicalize. There is no extension inference, no search path and no fallback to a package directory. |
|
||||
|
||||
The resulting file is loaded as a module. If it canonicalizes outside the
|
||||
project root, the loader reports "outside the project root" and the import
|
||||
becomes a `module.unresolved` diagnostic.
|
||||
|
||||
### 3.2 Loader rules
|
||||
|
||||
`crates/cli/src/loader.rs` implements a static, non-executing load:
|
||||
|
||||
1. **Project root discovery** (`marqdo_package::discover_root`): walk up from
|
||||
the entry file (starting at its parent directory) to the nearest directory
|
||||
containing `marqdo.yaml`; if none is found, the entry file's parent
|
||||
directory is the root. There is no `--project` flag in this build and nested
|
||||
project boundaries are not diagnosed — the nearest manifest wins (open
|
||||
item).
|
||||
2. **Manifest** is optional at run time; if a `marqdo.yaml` exists in the root
|
||||
it is parsed and validated. An invalid manifest is a fatal load error
|
||||
(exit code 2), not a diagnostic list.
|
||||
3. **Entry module** is loaded first; `Program.main_module` is set to its
|
||||
module id.
|
||||
4. **Imports** are processed depth-first in front matter order. Each import
|
||||
target is resolved as in §3.1, then loaded recursively.
|
||||
5. **Cycle detection** uses the path stack: if the canonical target path is
|
||||
already being loaded, the importing file gets a `module.cycle` diagnostic
|
||||
("import ... creates an import cycle") and the import edge is skipped.
|
||||
Modules are otherwise loaded once; a second import of an already-loaded
|
||||
module reuses it.
|
||||
6. **Definitions only**: loading parses, indexes and analyzes source. No
|
||||
function is called, no `main` is run, no foreign/plugin code is touched.
|
||||
7. After loading, `calls::validate_program` validates cross-module and
|
||||
qualified-builtin calls and attaches diagnostics to the emitting file.
|
||||
8. Diagnostics are collected in deterministic (load) order; the CLI renders
|
||||
them on stderr (or as JSON).
|
||||
|
||||
Failure mapping: a failure to load the entry file or read the manifest is a
|
||||
**fatal load error** (`exit 2`, `error: ...` on stderr). A failure to resolve
|
||||
an imported module becomes a `module.unresolved` **diagnostic** on the
|
||||
importing file and `check`/`run` then exit 1 (run refuses to execute a project
|
||||
with compile errors).
|
||||
|
||||
## 4. Markdown reference edges (`inspect --refs`)
|
||||
|
||||
`marqdo inspect <file> --refs` walks every `link` node in the entry document
|
||||
and prints/writes an edge array (`crates/cli/src/commands.rs::refs_json`,
|
||||
`resolve_link`). Each edge has:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "references.mq.md",
|
||||
"url": "./math.mq.md#计算圆面积",
|
||||
"span": {"start": 54, "end": 95},
|
||||
"resolved": true,
|
||||
"target_module": "math.mq.md",
|
||||
"target_unit": "math.mq.md#数学/计算圆面积"
|
||||
}
|
||||
```
|
||||
|
||||
Resolution rules:
|
||||
|
||||
1. `#fragment` → look up the entry module's anchors, then fall back to a unit
|
||||
whose canonical **name** equals the fragment.
|
||||
2. `http://` / `https://` URL → counted as resolved with no local target.
|
||||
External links are never downloaded, imported or executed.
|
||||
3. anything else → split optional `#fragment`, canonicalize the path relative
|
||||
to the **entry file's directory**, require the target to be inside the
|
||||
project root, derive its `ModuleId`, and look up anchors/names in that
|
||||
module. If the target module is not imported, its source is **read and
|
||||
analyzed statically on demand** (`analyze_source`) so broken anchors can be
|
||||
detected without execution.
|
||||
4. `resolved: false` means the file or anchor could not be resolved.
|
||||
|
||||
Current limitations (documented, not hidden):
|
||||
|
||||
* `check --refs` validates every loaded document's link nodes and reports
|
||||
broken same-document anchors, missing files and missing target anchors as
|
||||
`ref.unresolved` errors, changing the command's exit status. Plain `check`
|
||||
(without `--refs`) does not treat reference problems as errors, because links
|
||||
are static knowledge references rather than execution inputs.
|
||||
* `inspect --refs` prints the edge graph for the entry document only; other
|
||||
loaded modules are still covered by `check --refs`.
|
||||
* The graph is not emitted for call edges or import edges; imports are visible
|
||||
through module counts/`inspect` summaries, calls through `inspect --ir`.
|
||||
|
||||
## 5. `ref("#anchor")` runtime semantics
|
||||
|
||||
`ref` is a **pure builtin** (`crates/marqdo-ir/src/builtins.rs`,
|
||||
`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref`). It never evaluates
|
||||
the referenced resource and requires no capability.
|
||||
|
||||
Signature: `ref(target: Text) -> Formula | Code | FunctionRef`.
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `target` must start with `#`; otherwise `symbol.unknown` with the message
|
||||
"ref target ... must be a `#heading` anchor". It is not a URL resolver.
|
||||
2. The fragment is looked up **only in the running module's anchor map**
|
||||
(`Module::anchors`). In this build `ref` does not fall back to canonical
|
||||
unit names and does not search imported modules; a fragment that is not a
|
||||
GitHub slug yields `symbol.unknown` ("no heading matches ...").
|
||||
3. The matching unit's resources (`Module::resources_of`, computed from the
|
||||
unit's heading range) are examined:
|
||||
* **exactly one resource** → `Formula { latex, display, unit }` or
|
||||
`Code { language, source, content_hash, unit }`. `unit` is the
|
||||
`UnitId::Display` string.
|
||||
* **zero resources** → if a function's unit equals the resolved unit, return
|
||||
a `FunctionRef`; otherwise `symbol.unknown`.
|
||||
* **more than one resource → `ref.ambiguous`** with the message
|
||||
"`#...` contains N resources; select one explicitly instead of relying on
|
||||
the first". The runtime **never** picks the first resource. This is
|
||||
blueprint issue S-07 and is frozen by ADR-0003.
|
||||
4. Resource attribution (`resources.rs`): a formula or code fence belongs to
|
||||
the **innermost enclosing heading unit** by source span; resources are kept
|
||||
in document order. Markdown `math`/`math_block` and `code_block` nodes
|
||||
become resources; inline `code` spans do not.
|
||||
|
||||
Example (from `examples/math.mq.md`):
|
||||
|
||||
```markdown
|
||||
## 圆面积公式
|
||||
|
||||
$$
|
||||
A = \pi r^2
|
||||
$$
|
||||
|
||||
## 计算圆面积
|
||||
|
||||
**面积 = math.eval(ref("#圆面积公式"), {"r": 4}, target="A")**
|
||||
|
||||
*面积*
|
||||
```
|
||||
|
||||
`ref("#圆面积公式")` resolves the single formula under that heading; the
|
||||
formula is still inert until `math.eval` consumes it.
|
||||
|
||||
### 5.1 Error model of `ref`
|
||||
|
||||
| Situation | Runtime code |
|
||||
|---|---|
|
||||
| target not starting with `#` | `symbol.unknown` |
|
||||
| no anchor/unit matches | `symbol.unknown` |
|
||||
| unit has no formula/code/function | `symbol.unknown` |
|
||||
| unit has ≥ 2 resources | `ref.ambiguous` |
|
||||
| module not present in the program | `symbol.unknown` |
|
||||
|
||||
`ref.unresolved` (`crates/marqdo-value/src/diagnostic.rs`) is a **compile-time**
|
||||
code used by link-form call mapping, not by the `ref` builtin.
|
||||
|
||||
## 6. Call edges
|
||||
|
||||
### 6.1 Expression calls
|
||||
|
||||
Inside a function body, `CallExpr` nodes call:
|
||||
|
||||
* a local function (same module, namespace-aware resolution: current namespace
|
||||
first, then canonical name, then a unique local name);
|
||||
* an imported function through its alias: `geometry.圆面积(r=2)`;
|
||||
* a builtin (`打印`, `math.eval`, ...) by canonical name or alias;
|
||||
* a `FunctionRef` value obtained earlier (for example from `ref`).
|
||||
|
||||
Static validation (`crates/marqdo-frontend/src/calls.rs`) checks arity,
|
||||
argument names and the named-only rule for inferred signatures. Type errors are
|
||||
runtime concerns in v1.
|
||||
|
||||
### 6.2 Block-quote calls
|
||||
|
||||
`> f(...)` is a call statement; the payload must parse as a complete
|
||||
`CallExpr` (see `mapping.md` §4.5). Its result is discarded.
|
||||
|
||||
Link form `> [label](#anchor)(args)` resolves the link target **within the
|
||||
current module** using the anchor map, then the canonical unit-name fallback,
|
||||
and requires the target to be a function unit. Notes:
|
||||
|
||||
* a zero-argument link call must still write `()`; proxy `> [f](#f)` is a hard
|
||||
error;
|
||||
* `http(s)` link targets used as calls are a hard `ref.unresolved` error;
|
||||
external links are never downloaded or called;
|
||||
* a relative-path link target such as `> [f](./other.mq.md#f)(x=1)` is
|
||||
**not supported** in this build: mapping reports `module.unresolved` with
|
||||
the hint to import the module in front matter and call through its alias.
|
||||
Cross-module calls therefore go through an alias expression
|
||||
(`geometry.f(...)`), not through a Markdown link. Blueprint §2.4's
|
||||
`> [计算面积](./math.mq.md#计算面积)(r=3)` example is a **known divergence**.
|
||||
|
||||
### 6.3 Imported alias calls
|
||||
|
||||
At runtime an alias call resolves via `Program::resolve_import` to the target
|
||||
module id; an unresolved alias is `symbol.unknown`. The static validator
|
||||
checks the target module exists and the function/arity match when the whole
|
||||
program is loaded, and emits `symbol.unknown` for an unknown function.
|
||||
|
||||
## 7. Diagnostics for references/imports
|
||||
|
||||
| Code | Severity | Where produced |
|
||||
|---|---|---|
|
||||
| `module.unresolved` | error | loader (bad import target), call mapping (relative-path link call) |
|
||||
| `module.cycle` | error | loader |
|
||||
| `ref.unresolved` | error | link-form call target mapping, external links used as calls, `check --refs` link validation |
|
||||
| `ref.ambiguous` | runtime error | `ref` builtin (≥ 2 resources) |
|
||||
| `symbol.unknown` | error / runtime error | static call validation; `ref` anchor miss at runtime |
|
||||
| `frontmatter.invalid` | error | analyser (bad YAML/import shape) |
|
||||
| `markdown.invalid_construct` | error | empty/duplicate heading names, invalid link-call shape |
|
||||
|
||||
## 8. Current limitations (open ADR candidates)
|
||||
|
||||
1. `ref("#...")` is module-local and has no cross-module form; there is no
|
||||
documented syntax for referencing a resource in an imported module.
|
||||
2. Ordinary Markdown link resolution is inspection-only; `check` does not
|
||||
validate links and there is no `--refs` flag on `check`.
|
||||
3. Link-form calls cannot cross modules (see §6.2).
|
||||
4. Headings inside container blocks or fenced code never create units, and a
|
||||
heading whose name is empty is a hard error rather than an ignored heading.
|
||||
5. Anchor lookup in `ref` does not fall back to canonical unit names, while
|
||||
link-form calls and `inspect --refs` do. This asymmetry is intentional for
|
||||
the frozen build but should be unified by a future ADR (it is easy to get
|
||||
surprising results with headings containing punctuation).
|
||||
@@ -0,0 +1,276 @@
|
||||
# Marqdo Security Model (implemented status)
|
||||
|
||||
Status: **describes the M0.5 implementation, including honest gaps.** This is
|
||||
not a security audit; it documents what the code actually enforces.
|
||||
|
||||
Normative implementation:
|
||||
|
||||
| Concern | Code |
|
||||
|---|---|
|
||||
| Sandbox modes, strict refusal | `crates/marqdo-sandbox/src/lib.rs` |
|
||||
| Capability sets and checks | `crates/marqdo-host/src/capability.rs` |
|
||||
| Host dispatch and per-op checks | `crates/marqdo-host/src/host.rs` |
|
||||
| Project-rooted filesystem | `crates/marqdo-host/src/fs.rs` |
|
||||
| Budgets | `crates/marqdo-host/src/budget.rs` |
|
||||
| Effect tracing | `crates/marqdo-host/src/trace.rs` |
|
||||
| Math/foreign adapters | `crates/marqdo-host/src/kernels.rs` |
|
||||
| Foreign child processes | `crates/marqdo-foreign/src/lib.rs` |
|
||||
| Plugin loading / in-process plugin calls | `crates/marqdo-host/src/host.rs` (`load_plugin`, `op_plugin_invoke`), `crates/marqdo-plugin-abi/src/loader.rs` |
|
||||
| CLI flags and capability construction | `crates/cli/src/main.rs`, `crates/cli/src/commands.rs` |
|
||||
|
||||
## 1. Modes
|
||||
|
||||
`SandboxMode` has three values (`pure`, `project`, `strict`), parsed by
|
||||
`SandboxMode::parse` and selectable on `run`/`call` with
|
||||
`--mode <pure|project|strict>` (default: `project`).
|
||||
|
||||
| Mode | Intended guarantee | Actual status in this build |
|
||||
|---|---|---|
|
||||
| `pure` | No external I/O; memory computation and budgets only. | **Enforced.** `Capabilities::pure()` = `console`, `math` only; every FS/foreign/plugin grant flag makes the CLI exit 2 with "pure mode prohibits external I/O grants". |
|
||||
| `project` | Operator-approved project resources under `CapabilityHost` + bounded resources. | **Enforced for the implemented operations.** Defaults equal `pure`; `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin` add grants. Sandboxing is cooperative (in-process checks), not OS isolation. |
|
||||
| `strict` | Untrusted documents/code/plugins under real OS isolation. | **Refused.** `PlatformCapabilities::detect()` always reports `strict: Unavailable` in this build because no OS isolation adapter (namespaces/seccomp/cgroups, Job Object, macOS sandbox) is implemented, and `worker_processes: false`. `marqdo_sandbox::enforce` returns `StrictUnavailable`; `run`/`call` print the reason and exit 1. There is **no fallback** to an unrestricted process. |
|
||||
|
||||
`marqdo inspect <file> --platform` prints the detection report, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"strict": {"state": "unavailable",
|
||||
"reason": "no OS isolation adapter (namespaces/seccomp/cgroups) is implemented in this build"},
|
||||
"worker_processes": false
|
||||
}
|
||||
```
|
||||
|
||||
Design rule (blueprint §9): a cooperative in-process check is not a sandbox.
|
||||
Strict mode will remain refused until a platform isolation adapter exists; see
|
||||
ADR-0004.
|
||||
|
||||
## 2. Capability grants
|
||||
|
||||
`Capabilities` (`crates/marqdo-host/src/capability.rs`) is the effective grant
|
||||
set for one run:
|
||||
|
||||
| Field | Default (`pure`/`project`) | Used by |
|
||||
|---|---|---|
|
||||
| `console: bool` | `true` | currently **not checked** by `op_print`; print is always allowed and only bounded by the output budget (open item) |
|
||||
| `math: bool` | `true` | `math.*` host ops |
|
||||
| `fs_read: Vec<String>` | empty | `fs.read` |
|
||||
| `fs_write: Vec<String>` | empty | `fs.write` |
|
||||
| `net_hosts: Vec<String>` | empty | `Capabilities::check_net` exists, but no host op or builtin performs network access in this build |
|
||||
| `process: bool` | `false` | required together with `foreign` for `foreign.run`/`foreign.call` |
|
||||
| `foreign: bool` | `false` | foreign kernels |
|
||||
| `plugins: Vec<String>` | empty | `plugin.invoke` and `StdHost::load_plugin` |
|
||||
|
||||
Requests versus grants:
|
||||
|
||||
* `marqdo.yaml` `capabilities:` is a **request**. It is parsed and validated
|
||||
(`crates/marqdo-package/src/lib.rs`) but is not consulted when building the
|
||||
host capability set: only CLI flags/grant APIs produce grants. Declaring a
|
||||
capability never grants it.
|
||||
* `marqdo.yaml` `plugins:` likewise declares requirements only; it does not
|
||||
install, load or authorize a plugin.
|
||||
* CLI grant flags (`crates/cli/src/main.rs::SandboxArgs`):
|
||||
* `--allow-read <GLOB>` and `--allow-write <GLOB>` (repeatable, project mode
|
||||
only);
|
||||
* `--allow-foreign` (project mode only; sets `foreign = true` and
|
||||
`process = true`);
|
||||
* `--allow-plugin <PACKAGE-NAME>` (repeatable, project mode only; matches the
|
||||
manifest `plugin.name`, not the call namespace);
|
||||
* `--plugin <PATH>` (repeatable, loads a plugin after the trust check);
|
||||
* `--trust-native-plugin` (explicit in-process native trust decision).
|
||||
* There is no network grant flag and no plugin-worker flag in this build.
|
||||
|
||||
### 2.1 Filesystem grants are project-relative globs
|
||||
|
||||
`fs.read`/`fs.write` paths go through `ProjectFs::resolve` **before** the
|
||||
capability check:
|
||||
|
||||
* empty paths, absolute paths, `..` components and root/prefix components are
|
||||
rejected with `permission.denied`;
|
||||
* `.` components are dropped; the normalized relative string uses `/`;
|
||||
* the capability patterns are matched with the `glob` crate against that
|
||||
normalized relative path; a leading `./` is stripped on both sides, so
|
||||
`--allow-read ./data/**` matches `data/a.csv` and does not match
|
||||
`secrets/key`; `--allow-write output/**` does not grant reads;
|
||||
* an invalid glob pattern is treated as non-matching (`Pattern::new(...).
|
||||
unwrap_or(false)`), so it silently grants nothing (there is no diagnostic).
|
||||
|
||||
Additional FS limits (`crates/marqdo-host/src/fs.rs`):
|
||||
|
||||
* reads are capped at `MAX_READ_BYTES = 8 MiB`, must be regular files, must be
|
||||
UTF-8, and the canonicalized path must stay inside the project root;
|
||||
* writes create parent directories; before writing, the **closest existing
|
||||
ancestor** is canonicalized and must stay inside the project root.
|
||||
|
||||
Known caveat: for writes, only the closest existing ancestor is checked and the
|
||||
final write uses the non-canonical path. A symlink created between the check
|
||||
and the write (TOCTOU) could escape the root. Reads canonicalize the final path
|
||||
before reading. Full symlink/race hardening is an open P4 item.
|
||||
|
||||
### 2.2 Checks happen at execution points, not only at parse time
|
||||
|
||||
`Host::invoke` is the only path to effects. `StdHost::dispatch` routes every
|
||||
`HostOp` and the checks run immediately before the effect:
|
||||
|
||||
* `fs.read` → `capabilities.check_read(path)` then `ProjectFs::read`;
|
||||
* `fs.write` → `check_write` then `ProjectFs::write`;
|
||||
* math ops → `capabilities.math` (in `crates/marqdo-host/src/kernels.rs`);
|
||||
* foreign ops → `capabilities.check_foreign` (requires `foreign && process`);
|
||||
* plugin invoke → the namespace must be loaded, then
|
||||
`capabilities.check_plugin(package)` on **every call**; loading also checks
|
||||
the same grant once;
|
||||
* print → output budget only.
|
||||
|
||||
Static analysis may warn, but it never replaces these runtime checks. There is
|
||||
no host bypass for tests: tests use the same `StdHost`/`Capabilities` types.
|
||||
|
||||
## 3. Resource budgets
|
||||
|
||||
`Budget` (`crates/marqdo-host/src/budget.rs`) is per run:
|
||||
|
||||
| Limit | Default | Enforcement |
|
||||
|---|---|---|
|
||||
| evaluation fuel | `DEFAULT_FUEL = 5_000_000` | 1 unit per statement, 1 per expression node, 1 per branch condition, 1 per loop iteration; exhaustion → `resource.exhausted` |
|
||||
| call stack depth | `DEFAULT_STACK_LIMIT = 256` | checked before every function call → `resource.exhausted` |
|
||||
| program output bytes | `DEFAULT_OUTPUT_BYTES = 1 MiB` | `print` output and `fs.read` contents are charged; overflow → `resource.exhausted` |
|
||||
| wall clock | none by default | `--timeout <seconds>` sets a deadline; checked between statements/iterations and before loop conditions → `resource.exhausted`. It does **not** preempt a running host call. |
|
||||
|
||||
CLI overrides (`run`/`call`): `--fuel`, `--stack`, `--max-output`,
|
||||
`--timeout`. Additional fixed limits:
|
||||
|
||||
* foreign kernel: `StdHost.foreign_timeout = 10s` and
|
||||
`foreign_output_limit = 1 MiB` (stdout+stderr combined) per call.
|
||||
These are not configurable from the CLI in this build;
|
||||
* plugin wire: request ≤ 16 MiB, result ≤ 64 MiB, error ≤ 1 MiB, metadata
|
||||
≤ 64 KiB (see `include/marqdo_plugin_abi_v1.h`);
|
||||
* math kernel: 1 MB source, 256 parse depth, 512 AST nodes, exponent ≤ 10,000,
|
||||
exact-result ≤ 1,000,000 bits (`crates/marqdo-math`).
|
||||
|
||||
These are cooperative budgets in the interpreter and the supervisor. Only
|
||||
foreign child processes are terminated by the host (kill on timeout/output
|
||||
overflow). In-process math and native plugins are not OS-limited.
|
||||
|
||||
## 4. Effect tracing
|
||||
|
||||
`EffectTrace` records one entry per `Host::invoke` when enabled
|
||||
(`--trace`, or implicitly `--json` for `run`/`call`). Entry fields:
|
||||
|
||||
```json
|
||||
{"op":"print","module":"hello.mq.md","span":{"start":43,"end":57},
|
||||
"duration_micros":3,"result":"ok"}
|
||||
```
|
||||
|
||||
* `op` is the stable `HostOp::name()` (`print`, `math.eval`, `foreign.call`,
|
||||
`fs.write`, `plugin.invoke`, ...);
|
||||
* `module` is the initiating module id; `span` is the call site when known;
|
||||
* `result` is `"ok"` or the failure code (for example
|
||||
`permission.denied`);
|
||||
* argument values and file contents are **not** recorded, so ordinary secrets
|
||||
do not leak into traces by design. Paths are also not recorded today.
|
||||
* In text mode trace entries are printed to stderr (`trace: ...`); in JSON
|
||||
mode they are a `trace` array next to the program stdout string. Program
|
||||
output is always separate from diagnostics/trace (stdout vs stderr).
|
||||
|
||||
## 5. Foreign kernels: process isolation, not a sandbox
|
||||
|
||||
`crates/marqdo-foreign/src/lib.rs` runs Python/Node one-shot:
|
||||
|
||||
* the program is written to a private temp file (mode 0600 on Unix), the child
|
||||
runs with a fresh private temp directory (mode 0700) as cwd, and both are
|
||||
removed on drop;
|
||||
* direct `argv` exec (no shell, no interpolation); stdin carries the versioned
|
||||
request envelope; stdout/stderr are always piped and captured;
|
||||
* the environment is cleared except `PATH`;
|
||||
* the wall-clock deadline is enforced by polling `try_wait` and killing the
|
||||
direct child; output flooding kills the child once the combined byte budget
|
||||
is exceeded; a 2-second reader grace prevents a grandchild holding the pipes
|
||||
from hanging the host;
|
||||
* a non-zero exit is data (`RunStatus::Failed`, stderr preserved); a timeout is
|
||||
`TimedOut` for `run` and an error for the typed `call`.
|
||||
|
||||
What this is **not**: an OS sandbox. The child runs with the host user's
|
||||
privileges, can open absolute paths, use the network and spawn further
|
||||
processes. Clearing the environment and changing cwd are hygiene, not
|
||||
confinement. Grandchildren are not killed when the direct child is killed.
|
||||
Strict mode refuses to run at all, so foreign execution is only reachable in
|
||||
`project` mode with `--allow-foreign`.
|
||||
|
||||
## 6. Plugins: in-process native trust boundary
|
||||
|
||||
* The only plugin transport implemented in this build is **in-process native
|
||||
loading** (`libloading`) after `manifest.validate()`, exact target-triple
|
||||
artifact resolution, sha256 verification, ABI negotiation and export
|
||||
metadata cross-checking (`crates/marqdo-plugin-abi/src/loader.rs`).
|
||||
* Loading requires **three** independent things:
|
||||
1. `--plugin <manifest-or-dir>` names the plugin;
|
||||
2. `--trust-native-plugin` is an explicit unsafe trust decision;
|
||||
3. the manifest package name is granted with `--allow-plugin <name>` (checked
|
||||
again on every invocation).
|
||||
Without trust, `StdHost::load_plugin` returns `permission.denied` with the
|
||||
hint that no isolated worker transport exists. There is no isolated worker,
|
||||
Wasm component or out-of-process plugin path in this build.
|
||||
* A native plugin runs in the host address space. It can read host memory,
|
||||
call arbitrary syscalls, bypass the capability model entirely and crash or
|
||||
corrupt the host process. **A C ABI is not a sandbox.** `--trust-native-plugin`
|
||||
must be treated as equivalent to running the plugin's code as the host user.
|
||||
* Plugin call sites are written as `<namespace>.<export>(...)` in a function
|
||||
body (for example `solver.add(a=20, b=22)`). The namespace is derived from
|
||||
the **first** manifest export prefix. Calls are made through
|
||||
`HostOp::PluginInvoke` with a typed `wire_version` 1 request; the plugin's
|
||||
typed response becomes a Marqdo value (`Unit` when absent). Named arguments
|
||||
are required in this build; positional plugin arguments are `runtime.type`.
|
||||
* Plugin errors map to structured runtime errors: ABI → `plugin.abi_mismatch`,
|
||||
capability → `permission.denied`, wire/value → `runtime.type`, unsupported →
|
||||
`kernel.unavailable`, everything else → `plugin.failure`. An unloaded
|
||||
namespace is `symbol.unknown`.
|
||||
* The plugin's host-capability requests are denied by default: the CLI loads
|
||||
plugins with `LoadOptions::default()`, i.e. **no capability handler**, so
|
||||
every `request_capability` call fails. The `requires:` list in `plugin.yaml`
|
||||
is only a declaration.
|
||||
* `marqdo plugin inspect` never loads code: it validates the manifest and
|
||||
verifies the artifact hash only.
|
||||
* Installation, inspection, loading and authorization are four separate
|
||||
decisions. Copying a plugin into the project or declaring it in
|
||||
`marqdo.yaml` grants nothing.
|
||||
|
||||
Known limitations: single namespace per plugin (derived from the first export),
|
||||
no hot unload/reload, no plugin capability bridge from this CLI, native crash
|
||||
is a host crash, `plugin.invoke` cannot be used in `pure`/`strict` mode.
|
||||
|
||||
## 7. `check` and `inspect` are side-effect free
|
||||
|
||||
Neither command executes project code. Specifically:
|
||||
|
||||
* `check` parses, indexes, maps to IR, runs static call validation and reports
|
||||
diagnostics. It never calls a function, never spawns a foreign kernel, never
|
||||
loads a native plugin and never writes files.
|
||||
* `inspect` with `--ast`, `--units`, `--signature`, `--ir`, `--refs` or
|
||||
`--platform` does the same static work and prints metadata. `--refs` may
|
||||
**read** other project source files on demand to resolve link targets, but
|
||||
only parses/analyzes them; nothing is executed or written.
|
||||
* `plugin inspect` reads `plugin.yaml`, validates it, resolves the artifact for
|
||||
the host target and computes/compares its sha256. It does not `dlopen`,
|
||||
instantiate or call the plugin (`"loaded": false` in JSON output).
|
||||
* `run`/`call` first perform the same static load; if any diagnostic has error
|
||||
severity, execution is refused before the host or any kernel is touched.
|
||||
* `fmt` only normalizes line endings/trailing newlines and writes the document
|
||||
when not in `--check` mode; it does not execute anything.
|
||||
|
||||
## 8. Open security items
|
||||
|
||||
1. Strict mode: no OS isolation adapter (P4); worker processes report
|
||||
`false`.
|
||||
2. `console` capability is not enforced; print is always allowed.
|
||||
3. Network capability exists in the model but no host operation uses it; there
|
||||
is no `--allow-net` flag.
|
||||
4. Write-path symlink TOCTOU (see §2.1).
|
||||
5. Plugin worker isolation and a plugin capability bridge are unimplemented;
|
||||
only trusted in-process native plugins are possible.
|
||||
6. `plugin_handle` wire values are structurally reconstructable (see
|
||||
`value-wire.md` §4.1); no unforgeable capability token.
|
||||
7. Budgets do not cover all allocation paths (for example list/table growth
|
||||
inside the interpreter is bounded only by fuel and memory).
|
||||
8. `--json` run output includes the full program stdout string; that is the
|
||||
program's own output, not a host secret, but it is embedded in the JSON
|
||||
document consumers must parse.
|
||||
@@ -0,0 +1,161 @@
|
||||
# Marqdo Value Wire Format v1
|
||||
|
||||
Status: **frozen for M0.5**. Changes require an ADR, a `wire_version` bump and
|
||||
cross-ABI compatibility tests.
|
||||
|
||||
Canonical implementation: `crates/marqdo-value/src/wire.rs`
|
||||
(`to_wire`, `from_wire`, `from_wire_with_limits`, `WireLimits`, `WireError`).
|
||||
The plugin envelope that *carries* these values is defined in
|
||||
`crates/marqdo-plugin-abi/src/wire.rs` and documented in
|
||||
[`plugin-abi.md`](plugin-abi.md); the foreign-kernel stdin envelope is
|
||||
documented in [`foreign.md`](foreign.md) and implemented in
|
||||
`crates/marqdo-foreign/src/lib.rs`.
|
||||
|
||||
## 1. Why a typed wire format
|
||||
|
||||
JSON has one numeric type. Marqdo has arbitrary-precision `Int`, IEEE-754
|
||||
`Float` (including `NaN` and both infinities) and 13 distinct runtime value
|
||||
types (`crates/marqdo-value/src/value.rs`). Encoding an `Int` as a JSON number
|
||||
would silently lose digits above 2^53 and would make `Int(1)` and
|
||||
`Float(1.0)` indistinguishable after a round trip. The wire format is
|
||||
therefore **fully type-tagged** and is not plain JSON.
|
||||
|
||||
The rules that follow from this:
|
||||
|
||||
1. Every value is a JSON **object** with a mandatory `"type"` string.
|
||||
2. **Integers never travel as JSON numbers.** `Int` is
|
||||
`{"type":"int","decimal":"<base-10 string>"}`, which preserves every digit,
|
||||
sign and leading-zero-free canonical form.
|
||||
3. `Float` uses a JSON number only for finite values; `NaN`, `Infinity` and
|
||||
`-Infinity` use an explicit `"special"` tag.
|
||||
4. Unknown tags, missing fields, wrong JSON types and invalid payloads are
|
||||
structured decode errors, never silent coercions.
|
||||
|
||||
JSON numbers still appear in two non-value positions: `plugin_handle.id` (a
|
||||
host generation counter) and `span.start` / `span.end` (byte offsets). Neither
|
||||
is a user arithmetic value.
|
||||
|
||||
## 2. Version and entry points
|
||||
|
||||
* `WIRE_VERSION: u32 = 1` (`marqdo_value::WIRE_VERSION`).
|
||||
* `to_wire(&Value) -> serde_json::Value` encodes.
|
||||
* `from_wire(&Json) -> Result<Value, WireError>` decodes with
|
||||
`WireLimits::default()`.
|
||||
* `from_wire_with_limits(&Json, WireLimits) -> Result<Value, WireError>`
|
||||
decodes with explicit limits.
|
||||
* Decoding is recursive and never panics on untrusted input; every failure is
|
||||
a `WireError`:
|
||||
* `UnknownType(String)` — missing/non-string/unknown `"type"`;
|
||||
* `MissingField(&'static str)` — required field absent;
|
||||
* `Invalid(String)` — wrong JSON type, invalid decimal, bad table shape,
|
||||
unknown float `special`, non-object value;
|
||||
* `LimitExceeded(&'static str)` — nesting depth, list/map length, string or
|
||||
map-key length.
|
||||
|
||||
`WireError` implements `Display` but has no stable numeric code of its own.
|
||||
Boundary crates wrap it in their own stable codes: the plugin layer uses
|
||||
`invalid_argument_wire` / `invalid_result_wire`
|
||||
(`crates/marqdo-plugin-abi/src/wire.rs`), and the foreign layer maps it to
|
||||
`Protocol` (`crates/marqdo-foreign/src/lib.rs`).
|
||||
|
||||
## 3. Value encodings
|
||||
|
||||
The table is normative and mirrors `encode`/`decode` in
|
||||
`crates/marqdo-value/src/wire.rs`.
|
||||
|
||||
| Marqdo type | Wire JSON | Notes |
|
||||
|---|---|---|
|
||||
| `Unit` | `{"type":"unit"}` | No result; distinct from `None`. |
|
||||
| `None` | `{"type":"none"}` | Explicit empty value. |
|
||||
| `Bool` | `{"type":"bool","value":true}` | `value` must be a JSON boolean. |
|
||||
| `Int` | `{"type":"int","decimal":"-123..."}` | Decimal string; decoded with `num_bigint::BigInt::from_str`. No `+`, no separators, no hex. `-0` decodes to `0` because `BigInt` has no signed zero. |
|
||||
| `Float` (finite) | `{"type":"float","value":1.5}` | JSON number decoded with `as_f64()`. `-0.0` round-trips as a JSON negative-zero number and decodes to `Float(-0.0)`. |
|
||||
| `Float` (`NaN`) | `{"type":"float","special":"nan"}` | Never a JSON number. |
|
||||
| `Float` (`+∞`) | `{"type":"float","special":"infinity"}` | |
|
||||
| `Float` (`−∞`) | `{"type":"float","special":"-infinity"}` | |
|
||||
| `Text` | `{"type":"text","value":"..."}` | UTF-8 JSON string. |
|
||||
| `List` | `{"type":"list","items":[<value>, ...]}` | Order preserved. |
|
||||
| `Map` | `{"type":"map","entries":{"key":<value>, ...}}` | Decoding rebuilds a `BTreeMap`, so key order is canonical (sorted) after a round trip. Duplicate keys cannot occur in a JSON object. |
|
||||
| `Table` | `{"type":"table","header":["a","b"],"rows":[[<value>,<value>], ...]}` | Header cells must be JSON strings; every row must have exactly `header.len()` cells or decoding fails. |
|
||||
| `Formula` | `{"type":"formula","latex":"...","display":false}` plus optional `"unit"` (string) and `"span":{"start":N,"end":M}` | `display` defaults to `false` on decode when absent. The formula is inert data; this encoding never evaluates it. |
|
||||
| `Code` | `{"type":"code","language":"python","source":"...","content_hash":"<sha256 hex>"}` plus optional `"unit"` and `"span"` | `language` and `content_hash` default to `""` on decode when absent; `source` is required. |
|
||||
| `FunctionRef` | `{"type":"function_ref","module":"index.mq.md","name":"main"}` | Both fields required strings. |
|
||||
| `PluginHandle` | `{"type":"plugin_handle","plugin":"demo.solver","export":"solver.add","id":1}` | `id` must be a non-negative JSON integer (`as_u64`). |
|
||||
|
||||
Optional fields (`unit`, `span`, `display`, `language`, `content_hash`) are
|
||||
emitted only when present on the value. Encoders always emit `display` for
|
||||
`Formula` and `language`/`content_hash`/`source` for `Code`.
|
||||
|
||||
### 3.1 Round-trip guarantees
|
||||
|
||||
For the values covered by `wire::tests::scalar_roundtrip`,
|
||||
`from_wire(to_wire(v)) == v`. Exact guarantees:
|
||||
|
||||
* all `Int` digits survive (`big_integers_keep_all_digits`);
|
||||
* `NaN`/`±Infinity` survive with their sign/tag (`special_floats_are_tagged`);
|
||||
* `Unit` and `None` never collapse into each other;
|
||||
* `Map` iteration order after decoding is sorted, which is the canonical order
|
||||
used by `Value::display`/`repr`.
|
||||
|
||||
Finite `Float` values are encoded as JSON numbers. The workspace enables
|
||||
`serde_json`'s `float_roundtrip` feature, so a finite binary64 value survives an
|
||||
encode/decode cycle exactly (verified by the Python `math.pi * 9` semantic
|
||||
fixture: `28.274333882308138` stays `28.274333882308138`). `NaN`/infinities are
|
||||
tagged and exact. Use `Text` when a decimal string must be exchanged verbatim
|
||||
rather than as a binary64 float.
|
||||
|
||||
## 4. Limits
|
||||
|
||||
`WireLimits::default()` (`crates/marqdo-value/src/wire.rs`):
|
||||
|
||||
| Limit | Default | Applies to |
|
||||
|---|---|---|
|
||||
| `max_depth` | 64 | Nesting of *values*: the top-level object is depth 0, every child decoded through `list.items`, `map.entries`, `table.rows` or `table.header` cells is depth+1. A value at depth > 64 is rejected with `LimitExceeded("nesting depth")`. |
|
||||
| `max_items` | 1,000,000 | `list.items` length and `map.entries` key count. |
|
||||
| `max_string_bytes` | 16 MiB | `text.value` byte length and map key byte length. |
|
||||
|
||||
### 4.1 Known gaps in the current decoder
|
||||
|
||||
The following are **not** bounded by `WireLimits` in
|
||||
`crates/marqdo-value/src/wire.rs` today and must be treated as open items
|
||||
(they are also bounded indirectly at the ABI layer by envelope size limits):
|
||||
|
||||
* `table.header` length, `table.rows` length and the number of columns are not
|
||||
checked against `max_items` (only row/header width equality is checked);
|
||||
* table header cell strings are not checked against `max_string_bytes`;
|
||||
* `formula.latex`, `code.source`, `function_ref.module`/`name`,
|
||||
`plugin_handle.plugin`/`export` are not checked against
|
||||
`max_string_bytes`;
|
||||
* a `plugin_handle` decoded from wire data is a structurally reconstructed
|
||||
`Value::PluginHandle`; this build does not carry an unforgeable capability
|
||||
token. Host code must therefore treat a plugin handle as opaque *intent* and
|
||||
re-authorize any operation, not as proof of prior authorization. The header
|
||||
comment "cannot be forged from wire data" is aspirational for this point and
|
||||
should be fixed or reworded in a future ADR.
|
||||
* JSON itself may contain duplicate object keys; `serde_json` keeps the last
|
||||
one, so `map.entries` with duplicate keys is not rejected as an error.
|
||||
|
||||
## 5. Where the format is used
|
||||
|
||||
| Boundary | Envelope | Value positions |
|
||||
|---|---|---|
|
||||
| Plugin ABI v1 request/response | `{"wire_version":1,"kind":"request"/"response",...}` (`crates/marqdo-plugin-abi/src/wire.rs`) | `request.args` is an object of name → typed value; `response.value` is one typed value. |
|
||||
| Plugin host capability request/reply | opaque bytes, shape per capability | defined by the capability, not by this file. |
|
||||
| Foreign kernel stdin | `{"wire_version":1,"kind":"request","input":<typed value or null>}` (`crates/marqdo-foreign/src/lib.rs`) | `input` is a `to_wire(value)` result, or JSON `null` when absent. |
|
||||
| Foreign typed return (`foreign.call`) | stdout must be exactly one typed value | decoded with `from_wire` and default limits. |
|
||||
|
||||
The `--json` output of `run`/`call` embeds `"value": to_wire(outcome.value)`
|
||||
(`crates/cli/src/commands.rs`), so CLI JSON consumers see the same typed
|
||||
representation.
|
||||
|
||||
## 6. Compatibility rules
|
||||
|
||||
* `wire_version` is part of the frozen contract. A decoder rejects a different
|
||||
version rather than guessing.
|
||||
* Adding a new `"type"` tag requires a spec change and an ADR; removing or
|
||||
renaming a tag is a breaking change.
|
||||
* The plugin envelope independently carries `wire_version`; do not assume the
|
||||
C ABI major version and `wire_version` are the same number
|
||||
(see `spec/plugin-abi.md` §2).
|
||||
* All payloads are UTF-8 JSON. Endianness is irrelevant for the JSON layer,
|
||||
but the C ABI structs that carry these bytes are target-dependent.
|
||||
Reference in new issue
Block a user