docs: formal specs, ADRs, STATUS matrix, contributor guide and honest README
Complete the documentation set for the M0.5 toolchain, grounded in the code: - spec/references.md: import/reference/call edges, identities, loader rules, anchors, `ref` resolution/ambiguity, `check --refs` validation. - spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal string rule, special floats, limits and exact float round-trip. - spec/security.md: modes (strict refuses without OS isolation), capabilities, path rules, budgets, tracing, foreign process caveats, plugin trust. - spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync, unsupported registry/git/environment work. - spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the implemented CLI plugin-invocation flow with its limitations. - spec/math.md and spec/foreign.md: the implemented math subset and the foreign request/response/timeout/capability contract. - docs/adr/0001..0007: accepted decisions for the span foundation, inert executables outside function bodies, ref ambiguity (S-07), strict refusal, AST normalization, CLI contract and BigInt/typed wire (S-08). - docs/contributor-guide.md: repo map, workflow, fixture/golden updates, quality gate, ADR/commit policy. - docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18 Alpha checklist (conclusion: not yet Alpha-complete), including known divergences rather than overclaims. - README.md rewritten from "design-only starter" to the working implementation with quickstart, implemented phases, deliberate limits and doc index. - AGENTS.md: updated pointers/workflow only; architectural rules unchanged. - Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside functions, unsupported math.simplify/differentiate).
This commit is contained in:
1 parent
fae0cd4b18
commit
0116160399
19 files changed
+3767
-30
No files matched your search
@@ -0,0 +1,139 @@
|
||||
# ADR-0001: Crate graph and the `marqdo-span` foundation
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
`docs/DEVELOPMENT_BLUEPRINT.md` §11.1 fixes the dependency direction for the
|
||||
workspace and states: "`markdown ->` zero other Marqdo crate; responsible for a
|
||||
stable view of the standard AST." The implemented workspace instead has
|
||||
fifteen crates, and `marqdo-markdown` depends on `marqdo-span`.
|
||||
|
||||
Why the divergence was necessary:
|
||||
|
||||
* Every layer needs identical byte spans: the Markdown AST
|
||||
(`crates/marqdo-markdown/src/ast.rs`), expression AST
|
||||
(`crates/marqdo-expression`), IR (`crates/marqdo-ir`), diagnostics and runtime
|
||||
errors (`crates/marqdo-value`), effect traces (`crates/marqdo-host`), plugin
|
||||
errors (`crates/marqdo-plugin-abi`) and the CLI renderer.
|
||||
* A `Span` must be the same type across boundaries or every layer needs
|
||||
conversion functions. Conversions are exactly where off-by-one bugs and
|
||||
broken source locations appear, and they would make `Expr`/`Stmt`
|
||||
serde output inconsistent between `inspect --ast` and `inspect --ir`.
|
||||
* The blueprint's "zero Marqdo dependencies" rule exists to keep the parser
|
||||
independent of the semantic layers. A tiny, effect-free span crate preserves
|
||||
that property while satisfying the shared-data-contract requirement.
|
||||
|
||||
## Decision
|
||||
|
||||
1. Add `crates/marqdo-span` as a **leaf** crate with no Marqdo dependencies and
|
||||
only `serde` as an external dependency. It defines:
|
||||
* `Span { start: usize, end: usize }` — half-open byte range into the
|
||||
original UTF-8 source, with `merge`, `shift`, `contains`, `len`;
|
||||
* `LineCol { line, column }` — 1-based, byte columns;
|
||||
* `LineIndex` — precomputed line starts for byte→line/column conversion.
|
||||
2. Treat `Span` as part of the toolchain's shared data contract. Crates that
|
||||
produce or consume source locations depend on `marqdo-span` directly; they
|
||||
do **not** wrap it in local span types.
|
||||
3. `marqdo-markdown` depends on `marqdo-span` (and `comrak`) but on **no
|
||||
semantic crate**. It still owns only the canonical AST view, front matter
|
||||
extraction and source maps; it performs no mapping, resolution or execution.
|
||||
4. Keep the rest of the blueprint's direction:
|
||||
`value -> span`, `expression -> span`, `ir -> span + expression`,
|
||||
`markdown -> span + comrak`, `frontend -> markdown + expression + ir + value`,
|
||||
`host -> value + math + foreign + sandbox`, `runtime -> ir + value + host +
|
||||
expression`, `cli -> all` (composition root). There is no `runtime -> cli`
|
||||
edge and no cycle; `cargo tree` is expected to stay acyclic.
|
||||
5. Do not introduce the blueprint's separate `sema` or `environment` crates in
|
||||
M0.5: semantic analysis lives in `marqdo-frontend`
|
||||
(`analyze`, `calls`, `infer`, `signature`, `units`, `map`), and package/
|
||||
environment formatting lives in `marqdo-package` plus the CLI. Splitting
|
||||
them is a refactor, not a semantic change.
|
||||
|
||||
Actual dependency graph (workspace `Cargo.toml` plus crate manifests):
|
||||
|
||||
```text
|
||||
span serde
|
||||
value span, serde, serde_json, num-bigint, num-traits
|
||||
expression span, serde, num-bigint, unicode-ident
|
||||
markdown span, comrak, serde, serde_yaml, unicode-normalization
|
||||
ir span, expression, serde
|
||||
frontend span, markdown, expression, ir, value, sha2,
|
||||
unicode-normalization, unicode-ident
|
||||
math value, num-bigint, num-rational, num-traits, serde
|
||||
foreign span, value, serde_json, thiserror
|
||||
sandbox serde
|
||||
plugin-abi span, value, serde, serde_json, serde_yaml, thiserror,
|
||||
libloading, sha2, hex
|
||||
plugin-sdk plugin-abi, serde, serde_json
|
||||
host span, value, math, foreign, sandbox, serde, serde_json,
|
||||
thiserror, glob
|
||||
runtime span, value, ir, expression, host, num-bigint,
|
||||
num-rational, num-traits, thiserror
|
||||
package serde, serde_yaml
|
||||
cli span, value, expression, markdown, ir, frontend, math,
|
||||
foreign, plugin-abi, host, runtime, package, sandbox,
|
||||
clap, serde, serde_json, walkdir, hex, sha2
|
||||
```
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Put `Span` in `marqdo-value`.** Then `marqdo-markdown` would depend on the
|
||||
runtime value model, which is a semantic layer and pulls in `num-bigint` and
|
||||
`serde_json`. Worse layering than a dedicated leaf.
|
||||
2. **Duplicate `Span` in each crate and convert at boundaries.** Avoids a crate
|
||||
but guarantees conversion code, mismatched serde shapes and duplicated
|
||||
tests; rejected.
|
||||
3. **Define `Span` in `marqdo-markdown` and let others depend on it.** Makes
|
||||
the parser a de-facto foundation crate and lets non-Markdown layers depend
|
||||
on `comrak`'s dependencies; rejected.
|
||||
4. **Use a third-party range type.** Adds a dependency and gives no benefit
|
||||
over ~170 lines; rejected.
|
||||
5. **Keep the blueprint's literal "markdown has zero Marqdo deps".** Would
|
||||
force either duplication or a span type in the parser that cannot be shared
|
||||
with diagnostics; rejected in favor of a documented, narrow exception.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No syntax change. Every `MdNode` still has `span: Span` in the original file
|
||||
(`crates/marqdo-markdown/src/ast.rs`), and the JSON inspection view is
|
||||
`{"kind":"heading","level":2,"span":{"start":0,"end":7},...}`. Spans are
|
||||
half-open byte offsets, BOM-rebased, and byte columns for diagnostics.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* This is a **new repository**; there is no legacy compatibility burden.
|
||||
* Public Rust API: moving `Span` later is a workspace-wide API change but has
|
||||
no language/ABI/package-format impact. It would require updating every
|
||||
`crates/*/Cargo.toml`.
|
||||
* Stable observable contracts (AST JSON `span`, diagnostic JSON `span`, IR
|
||||
`span`) are unchanged by the crate boundary. Changing span semantics
|
||||
(e.g. line/column units) requires a spec change and ADR.
|
||||
* The divergence from blueprint §11.1 is recorded here; future crates should
|
||||
use `marqdo-span` rather than introducing another location type.
|
||||
|
||||
## Security
|
||||
|
||||
No new capability or effect. `marqdo-span` is `serde`-only, performs no I/O and
|
||||
cannot execute anything. Byte offsets are derived from already-parsed source;
|
||||
`LineIndex::line_col` clamps out-of-range offsets instead of panicking.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-span/src/lib.rs` tests multi-byte UTF-8 line/column mapping,
|
||||
`merge`/`shift`/`contains`, and clamping.
|
||||
* `crates/marqdo-markdown/src/profile.rs` tests BOM rebasing and that every AST
|
||||
node carries a byte span; frontend tests assert diagnostics point at the
|
||||
expected source line/column.
|
||||
* `cargo test --workspace` must pass; `cargo tree` must remain acyclic (no
|
||||
automated CI gate yet — this is a P7 item).
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
Immediate: the crate exists and all layers already use it. Future work may
|
||||
split the frontend into `sema`/`environment` crates; that must preserve the
|
||||
graph above (`runtime` must never depend on `cli`, `markdown` must never depend
|
||||
on semantic crates). If a second span representation is ever needed (for
|
||||
example for generated code), add it as a conversion in `marqdo-span`, not as a
|
||||
per-crate duplicate.
|
||||
@@ -0,0 +1,140 @@
|
||||
# ADR-0002: Executable nodes execute only inside level-2 function bodies
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The blueprint (§1.2 execution context, §3.2 control flow, unresolved issue
|
||||
**S-03**) establishes that `Strong` (execute), `Emph` (return), ordered lists
|
||||
(branch), bullet lists (loop) and block quotes (call) have program meaning
|
||||
**inside a function body**, i.e. inside the range of a level-2 heading. It
|
||||
leaves the module top level under-specified: if a document has `**x = 1**`
|
||||
before any function, should that be a hard error, a silent no-op, or an
|
||||
explicit top-level statement?
|
||||
|
||||
Two failure modes had to be avoided:
|
||||
|
||||
1. **Silent execution at module scope.** There is no function frame, no return
|
||||
target and no entry point, so top-level effects would be surprising and
|
||||
would make `check`/`inspect` harder to reason about.
|
||||
2. **Hard error.** Ordinary Markdown documents routinely use `**bold**`,
|
||||
`*italic*`, `1. ordered` and `> quotes` in prose. Treating every such node
|
||||
outside a function as a fatal compile error would make the language hostile
|
||||
to normal documentation and would break the "renders normally" goal.
|
||||
|
||||
The chosen baseline was: executable nodes only execute inside function bodies;
|
||||
elsewhere they are ordinary document content, but this must not happen
|
||||
silently.
|
||||
|
||||
## Decision
|
||||
|
||||
1. A **function body** is the child-block range of a `heading(level=2)` up to
|
||||
the next heading of level ≤ 2 (`crates/marqdo-frontend/src/units.rs`,
|
||||
`FunctionScope.body_start..body_end`). Executable roles are assigned only
|
||||
inside those ranges by `crates/marqdo-frontend/src/map.rs`.
|
||||
2. Outside every function body, `strong`, `emph`, `list` and `block_quote`
|
||||
nodes are **inert document content**. They are not mapped to IR, are not
|
||||
executed, and prose is not inspected for keywords.
|
||||
3. The frontend emits a **warning** `markdown.inert_executable` for every
|
||||
would-be execution node found outside a function body. The document still
|
||||
compiles and runs; only `check`'s diagnostic output reflects the warning.
|
||||
The warning is produced once per top-level node and has a hint to move the
|
||||
construct under a `## function` heading.
|
||||
* `block_quote` → "a block quote invocation ... is inert document content";
|
||||
* `list` → "a list control structure ...";
|
||||
* `paragraph` → the first `strong`/`emph` found in the paragraph
|
||||
("an execution statement" / "a return statement").
|
||||
* warnings are emitted only while scanning the document's direct top-level
|
||||
blocks: headings, tables, code/math blocks and other non-paragraph blocks
|
||||
produce none. A top-level paragraph is walked recursively, so a
|
||||
`strong`/`emph` nested under a link or strikethrough inside that paragraph
|
||||
can still produce the inert warning. Nodes inside a function body produce
|
||||
no warning (they are mapped or rejected by the strict executable-shape
|
||||
rules instead).
|
||||
4. Warnings do not change exit codes: `check` exits 0 when there are no error
|
||||
diagnostics even if warnings exist; `run`/`call` execute normally. JSON
|
||||
`check` output sets `success = (errors == 0)` and includes the warning
|
||||
entries in `diagnostics`.
|
||||
|
||||
Rationale for the nodes covered: these are exactly the block-level constructs
|
||||
that `map.rs` would otherwise turn into statements if a function body existed.
|
||||
Inline nodes inside a paragraph are covered by one paragraph-level warning
|
||||
rather than one per node, which keeps normal prose readable.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Hard error for every executable-looking node outside a function.** Rejected
|
||||
(see context); it forbids emphasis in ordinary Markdown and makes writing a
|
||||
README a compile failure.
|
||||
2. **Silent inert behaviour with no diagnostic.** Rejected: an author could
|
||||
believe top-level `**x = 1**` had executed. The blueprint explicitly wants
|
||||
invalid execution intent surfaced.
|
||||
3. **Execute top-level nodes in an implicit module frame.** Rejected: no
|
||||
defined entry semantics, no return target, would make imported modules
|
||||
potential side-effect roots (violating "modules register definitions only").
|
||||
4. **Compile-time error only when the payload parses as a statement.** Rejected:
|
||||
this is the forbidden heuristic ("parses as code → execute") from blueprint
|
||||
§1.3; role must come from node type and position, not from guessing.
|
||||
5. **Warning per AST node.** Rejected as noisy for nested content; one warning
|
||||
per top-level construct is enough to point at the problem.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No new node kinds. Relevant structure:
|
||||
|
||||
```text
|
||||
document
|
||||
heading(level=2) "f" <-- opens a function body
|
||||
paragraph -> strong(text "x = 1") <-- executes (Stmt::Assign)
|
||||
paragraph (before any heading)
|
||||
strong(text "x = 1") <-- inert; emits markdown.inert_executable
|
||||
list(ordered) <-- inert outside function; warning
|
||||
block_quote <-- inert outside function; warning
|
||||
```
|
||||
|
||||
A level-2 heading without a preceding level-1 heading is a module-root function
|
||||
(`## main`) and its body executes normally. A `###` section inside the function
|
||||
range stays inside the function body; headings never execute.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Warning code `markdown.inert_executable` is part of the CLI diagnostic
|
||||
contract. Renaming it requires an ADR and a CLI version note.
|
||||
* Making this a hard error later would be a breaking language change: documents
|
||||
that currently compile (with warnings) would stop. If that is ever desired it
|
||||
needs a migration plan (e.g. an opt-in strict lint).
|
||||
* No package/ABI/IR shape change: inert nodes are simply absent from the IR.
|
||||
* Prose invariance is preserved: editing narrative text outside functions
|
||||
cannot change the IR.
|
||||
|
||||
## Security
|
||||
|
||||
No effect: inert nodes never reach the interpreter, host or kernels. The
|
||||
warning reduces the chance of a false belief that a side effect happened,
|
||||
which is a security-relevant honesty property. A document cannot cause an
|
||||
execute-by-accident through top-level prose.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-frontend/src/units.rs::tests::
|
||||
executable_outside_function_warns_but_does_not_execute` asserts the warning
|
||||
and that the function index is still built.
|
||||
* `units.rs` tests for function ranges, nested sections, root-level `## main`
|
||||
and duplicate/empty headings.
|
||||
* `crates/marqdo-frontend/src/analyze.rs` tests hard errors for invalid
|
||||
executable shapes inside function bodies (the companion rule: when a node
|
||||
*is* executable, its shape is strictly checked).
|
||||
* Conformance/negative fixtures under `crates/cli/tests` and `tests/`
|
||||
(work in progress by the fixtures workstream) are expected to cover the
|
||||
warning and the inert behavior; `spec/mapping.md` §4 documents the contract.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
M0.5 ships with warnings. Rollout guidance:
|
||||
|
||||
* document authors move executable constructs under `## function` headings;
|
||||
* tooling should show the warning in editors (the diagnostic already has a
|
||||
precise span);
|
||||
* if a future release makes inert execution a hard error, add a deprecation
|
||||
window and a `check` flag so existing documents can be migrated.
|
||||
@@ -0,0 +1,157 @@
|
||||
# ADR-0003: `ref` resolution order and `ref.ambiguous` (S-07)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
`ref("#anchor")` is the only way to turn a heading range into an inert
|
||||
resource value (`Formula` or `Code`) or a `FunctionRef`. The blueprint
|
||||
(§5.2, unresolved issue **S-07**) is explicit that a heading range can contain
|
||||
several formulas or code fences, and that the implementation must **never
|
||||
silently pick the first one**:
|
||||
|
||||
> 一个范围存在多段公式时,默认对象选择必须是明确规则(例如禁止无选择器,要求
|
||||
> `math.formulas(ref(...))` 列出候选;不选"第一条")。
|
||||
|
||||
The failing example is easy to write:
|
||||
|
||||
```markdown
|
||||
## f
|
||||
|
||||
### 公式
|
||||
|
||||
$$a = 1$$
|
||||
|
||||
$$b = 2$$
|
||||
|
||||
**x = ref("#公式")**
|
||||
```
|
||||
|
||||
If `ref` returned `a = 1`, the program would be non-deterministic with respect
|
||||
to document structure (insert an earlier formula and the meaning changes) and
|
||||
would hide a genuine authoring error.
|
||||
|
||||
At the same time the common case — one formula per heading — must stay simple;
|
||||
requiring a selector for every reference would make `examples/math.mq.md`
|
||||
awkward.
|
||||
|
||||
## Decision
|
||||
|
||||
`crates/marqdo-runtime/src/builtins_impl.rs::resolve_ref` implements this
|
||||
algorithm:
|
||||
|
||||
1. The target must be a `Text` beginning with `#`; otherwise
|
||||
`symbol.unknown` ("must be a `#heading` anchor"). `ref` is not a URL
|
||||
resolver and does not cross modules.
|
||||
2. The fragment is looked up in the **current module's anchor map**
|
||||
(`Module::anchors`, GitHub-style slugs computed in
|
||||
`crates/marqdo-frontend/src/units.rs`). `ref` does not fall back to
|
||||
canonical unit names and does not search imported modules.
|
||||
3. Collect the resources attributed to the resolved unit (innermost unit by
|
||||
span, `crates/marqdo-frontend/src/resources.rs`).
|
||||
* **Exactly 1** → return `Formula` or `Code` (with `unit` and `span`
|
||||
provenance).
|
||||
* **0** → if a function's unit equals the resolved unit, return
|
||||
`FunctionRef { module, name }`; otherwise `symbol.unknown`.
|
||||
* **≥ 2** → return **`ref.ambiguous`**:
|
||||
"`#...` contains N resources; select one explicitly instead of relying on
|
||||
the first". Never choose a resource.
|
||||
4. Function references are callable through the normal call path; formulas and
|
||||
code remain inert until `math.eval`/`foreign.*` is explicitly called (those
|
||||
are separately capability-checked).
|
||||
|
||||
Resource attribution is deterministic: a resource belongs to its innermost
|
||||
enclosing heading unit, resources keep document order, and a resource is a
|
||||
Markdown `math`/`math_block` or `code_block` node (inline code is not a
|
||||
resource).
|
||||
|
||||
The `ref.ambiguous` category is `ErrorCategory::RefAmbiguous` with the stable
|
||||
code `ref.ambiguous` (`crates/marqdo-value/src/error.rs`).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **First resource wins** (blueprint's rejected baseline). Rejected by S-07:
|
||||
document edits would silently change program meaning, and there is no
|
||||
diagnostic for the author.
|
||||
2. **Always require a selector argument**
|
||||
(`ref("#f", selector="second")`). Rejected for M0.5: no selector syntax or
|
||||
addressing scheme is specified yet, and the single-resource case is by far
|
||||
the common one. A future ADR can add an explicit selector while keeping the
|
||||
ambiguity rule.
|
||||
3. **Error on every heading with ≥ 2 resources at compile time**, even if never
|
||||
referenced. Rejected: a document may legitimately contain several formulas
|
||||
for prose and never `ref` them; only the reference is ambiguous.
|
||||
4. **Return a list of all candidates.** Rejected for v1: it changes the return
|
||||
type and forces every caller to handle a selector; a dedicated API
|
||||
(`math.formulas(ref(...))`) would be a new spec, not a silent widening.
|
||||
5. **Cross-module `ref`.** Rejected for v1: the wire/identity model and
|
||||
`ModuleId` resolution live in the loader; `ref` is a pure runtime builtin
|
||||
over the current module. Cross-module resources are an open item.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
```text
|
||||
heading(level=2) "f"
|
||||
heading(level=3) "公式"
|
||||
math_block "/$$ a = 1 $$/" -> Resource::Formula { unit: <#f/公式> }
|
||||
math_block "/$$ b = 2 $$/" -> Resource::Formula { unit: <#f/公式> }
|
||||
paragraph -> strong("x = ref(\"#公式\")")
|
||||
```
|
||||
|
||||
Anchors are separate from unit paths: `#公式` is a GitHub slug for the heading
|
||||
name, while the canonical `UnitId` is `<module>#f/公式`. Colliding slugs get
|
||||
`-1`/`-2` suffixes; canonical ids never do.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* `ref.ambiguous` is a stable runtime code; callers may branch on it.
|
||||
* The single-resource and function-reference cases are frozen. Adding a
|
||||
selector must be additive (new argument or new builtin) and must not change
|
||||
the ≥ 2 error into a silent choice.
|
||||
* `ref` target syntax (`#fragment`) is frozen; changing it to accept file paths
|
||||
would need a new ADR.
|
||||
* No package/ABI/wire change: formula/code values already carry
|
||||
`unit`/`span`/`content_hash`.
|
||||
|
||||
Example behavior, verified by
|
||||
`crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`:
|
||||
|
||||
```text
|
||||
$ marqdo run index.mq.md # heading with two $$...$$ blocks and ref("#公式")
|
||||
runtime error[ref.ambiguous]: `#公式` contains 2 resources; select one explicitly instead of relying on the first
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
* `ref` is a pure builtin: no host call, no capability required, no side
|
||||
effect. It only returns inert values or a function reference.
|
||||
* An inert `Formula`/`Code` does not execute. Evaluation/execution requires a
|
||||
separate explicit call (`math.eval`, `foreign.run/call`), which is capability
|
||||
checked at the host boundary.
|
||||
* Refusing ambiguity prevents a class of "wrong formula executed" mistakes,
|
||||
which matters for scientific documents where formulas are evidence.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-runtime/tests/interpreter.rs::ambiguous_ref_resource_is_an_error`
|
||||
drives a real document through frontend + runtime and asserts the
|
||||
`ref.ambiguous` code.
|
||||
* `crates/marqdo-frontend/src/resources.rs` tests attribution to innermost
|
||||
functions, multiple resources kept in one unit, and stable SHA-256 code
|
||||
hashes.
|
||||
* `crates/marqdo-markdown`/`frontend` tests cover anchors and slug collisions.
|
||||
* `examples/math.mq.md` is the positive single-formula case; it runs under
|
||||
`marqdo run` only via a function (the example itself defines callable
|
||||
functions, no `main`).
|
||||
* Missing anchor/not-a-function cases are covered by the `symbol.unknown`
|
||||
paths; conformance fixtures are being added by the fixtures workstream.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration needed for documents with exactly one resource per referenced
|
||||
heading. Documents that relied on "first formula" behavior (there are none in
|
||||
the examples) must split headings or wait for an explicit selector API. If a
|
||||
selector is added later, keep `ref("#...")` behavior unchanged and introduce
|
||||
the selector as a new argument/builtin, with tests for both the ambiguous and
|
||||
the selected case.
|
||||
@@ -0,0 +1,133 @@
|
||||
# ADR-0004: Strict mode refuses to run without OS isolation
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
Blueprint §9 defines three execution modes and one non-negotiable rule:
|
||||
|
||||
> `strict`: untrusted documents/code/plugins — independent restricted process,
|
||||
> OS-level resource and syscall/file/network restrictions; if unavailable,
|
||||
> **refuse to run**.
|
||||
|
||||
A cooperative in-process capability check cannot contain hostile native code
|
||||
or an unrestricted child process. If `strict` silently degraded to `project`,
|
||||
users would believe they had isolation they did not have. That is a security
|
||||
bug, not a missing feature.
|
||||
|
||||
M0.5 implements capability checks, budgets and process supervision but **no OS
|
||||
isolation adapter** (no Linux namespaces/seccomp/cgroups, no Windows Job
|
||||
Object/appcontainer, no macOS sandbox), and the isolated plugin worker
|
||||
transport is not implemented either.
|
||||
|
||||
## Decision
|
||||
|
||||
1. `crates/marqdo-sandbox` is honest about platform support:
|
||||
* `PlatformCapabilities::detect()` always reports
|
||||
`StrictIsolation::Unavailable { reason }` in this build, with a
|
||||
platform-specific reason (Linux: "no OS isolation adapter
|
||||
(namespaces/seccomp/cgroups) is implemented in this build"; macOS/Windows/
|
||||
other targets have analogous messages);
|
||||
* `worker_processes: false`.
|
||||
2. `marqdo_sandbox::enforce(mode, platform)`:
|
||||
* `pure` and `project` → `Ok(())`;
|
||||
* `strict` → `Err(SandboxError::StrictUnavailable { reason })` unless
|
||||
`platform.strict.is_available()`.
|
||||
3. `run`/`call` call `enforce` **before** constructing the host, loading
|
||||
plugins or executing anything. On failure they print
|
||||
"refusing to run in strict mode: ..." (or a JSON `{"success":false,
|
||||
"error":...}` envelope) and exit 1.
|
||||
4. There is **no fallback**: the CLI never downgrades a requested strict run to
|
||||
project mode. `SandboxMode::parse("strict")` succeeds (the mode name is
|
||||
valid); only execution is refused.
|
||||
5. `inspect --platform` exposes the detection report so users can see why:
|
||||
`{"os":"linux","arch":"x86_64","strict":{"state":"unavailable","reason":...},
|
||||
"worker_processes":false}`.
|
||||
6. The rest of the system is labelled accordingly: `project` mode is
|
||||
cooperative, in-process, and is not a sandbox; foreign kernels are
|
||||
process-isolated but not confined; native plugins run in-process and require
|
||||
explicit trust (ADR-0006/security spec).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Best-effort strict** (run with the same checks as project). Rejected: it
|
||||
directly violates the blueprint's "cannot provide → refuse" rule and hides a
|
||||
missing security boundary.
|
||||
2. **Warn and continue.** Rejected for the same reason; a warning is not a
|
||||
sandbox, and CI/agents may not read warnings.
|
||||
3. **Implement a Linux namespaces/seccomp adapter now.** Out of scope for
|
||||
M0.5: it needs platform-specific code, integration tests, resource limits,
|
||||
CI on multiple kernels, and a way to supervise the worker protocol. This is
|
||||
P4 work and belongs in its own ADR when implemented.
|
||||
4. **Use Wasm/WASI for strict.** Attractive for plugins later, but it does not
|
||||
sandbox arbitrary Python/Node processes or native `.so` files, and the
|
||||
runtime is not implemented. Rejected for now.
|
||||
5. **Claim strict via the process isolation of foreign kernels.** Rejected:
|
||||
a child process inherits the user's privileges and filesystem/network
|
||||
access; changing cwd and clearing env is hygiene, not confinement.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No language or AST change. A strict run is rejected before any document is
|
||||
executed (after static loading/diagnostics); the CLI contract is:
|
||||
|
||||
```text
|
||||
$ marqdo run index.mq.md --mode strict
|
||||
error: refusing to run in strict mode: no OS isolation adapter
|
||||
(namespaces/seccomp/cgroups) is implemented in this build. Strict mode
|
||||
requires verified OS-level isolation and never falls back to an
|
||||
unrestricted process.
|
||||
$ echo $?
|
||||
1
|
||||
```
|
||||
|
||||
(Exact wording comes from
|
||||
`crates/marqdo-sandbox/src/lib.rs::SandboxError::display`.)
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Accepting `--mode strict` as a valid *name* is part of the CLI contract; the
|
||||
refusal is a documented behavior, not a parse error. A future release that
|
||||
implements isolation flips the detection report and starts executing strict
|
||||
runs — that is the intended upgrade path.
|
||||
* Exit code 1 (runtime/security refusal) is stable; usage errors are 2.
|
||||
* `PlatformCapabilities` JSON shape (`state: available|unavailable`) is
|
||||
versioned by documentation and should not change without an ADR.
|
||||
* No package/ABI/wire impact.
|
||||
|
||||
## Security
|
||||
|
||||
* The guarantee "requested strict → refused rather than downgraded" is
|
||||
enforced in code (`enforce`), not documentation.
|
||||
* Users cannot accidentally believe untrusted docs are sandboxed by this
|
||||
build; the platform report is the evidence.
|
||||
* Project mode remains explicitly non-sandboxed. Native plugins (in-process)
|
||||
and foreign child processes are recorded as trust decisions, not isolation.
|
||||
* Secrets: strict refusal happens before plugin loading or foreign spawning,
|
||||
so no untrusted code is touched by a refused command.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-sandbox/src/lib.rs::tests::
|
||||
strict_refuses_when_isolation_is_unavailable` asserts the refusal and that
|
||||
`pure`/`project` still succeed.
|
||||
* `mode_names_round_trip` covers parsing/printing.
|
||||
* CLI behavior (`--mode strict` exit 1, message) is covered by manual/CLI
|
||||
smoke checks; a CLI integration test should be added with the conformance
|
||||
harness.
|
||||
* `inspect --platform` JSON is exercised manually and in the CLI help/example
|
||||
documentation; it should get a golden test in the fixture workstream.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
When an OS isolation adapter lands:
|
||||
|
||||
1. implement a `StrictIsolation::Available { mechanism }` report and a real
|
||||
supervisor/worker;
|
||||
2. add integration tests for filesystem/network/process restrictions and
|
||||
resource limits per platform, plus the platform capability matrix;
|
||||
3. update `spec/security.md`, this ADR (or supersede it), `docs/STATUS.md` and
|
||||
the CLI docs;
|
||||
4. only then change `PlatformCapabilities::detect`; do not add a flag that
|
||||
bypasses the refusal before that work is done.
|
||||
@@ -0,0 +1,174 @@
|
||||
# ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The language derives program meaning from the **canonical AST**, never from
|
||||
source spelling (`spec/markdown-profile.md`, `spec/mapping.md` §1). The parser
|
||||
(`comrak =1.0.0`, frozen options) normalizes several syntaxes before Marqdo
|
||||
sees them, and a few normalizations are lossy:
|
||||
|
||||
* `# H` and Setext `H\n===` both become a heading node; the source spelling is
|
||||
lost unless recorded;
|
||||
* GFM task list items (`- [x] text`) produce a task-specific item node;
|
||||
* inline code spans are stored as a literal without the backtick delimiters,
|
||||
but source spans include them;
|
||||
* `~~strikethrough~~` is enabled by the profile and wraps arbitrary inlines.
|
||||
|
||||
The language needs a single, documented answer per question, because node type
|
||||
and structural position decide executable roles.
|
||||
|
||||
## Decision
|
||||
|
||||
The profile (`marqdo.md.v1`) and the canonical AST
|
||||
(`crates/marqdo-markdown/src/ast.rs`, `src/profile.rs`) freeze these rules:
|
||||
|
||||
1. **Setext headings are headings.** `# name` and a Setext level-1/2 underline
|
||||
both produce `heading { level, setext }`; `setext` is inspection-only and
|
||||
**never affects semantics**. `## name` and a Setext level-2 heading are both
|
||||
functions. The profile keeps `parse.ignore_setext = false`.
|
||||
2. **Task-list items are `item` nodes with a task flag.**
|
||||
`item { task: Some(true|false) }` for GFM task items, `task: None`
|
||||
otherwise. Node type stays `item`, so list structure (ordered vs bullet)
|
||||
still determines branch/loop roles; the checkbox flag is inert data and does
|
||||
not change execution. A task list item in an executable position still
|
||||
requires its condition header like any other item.
|
||||
3. **Code spans carry delimiter tick counts.** Inline `code { ticks }` records
|
||||
the number of backticks, and the source span includes them; the literal is
|
||||
the inner payload. This lets the frontend map expression payload spans back
|
||||
to exact source bytes for diagnostics. "Ticks" is not user-visible program
|
||||
meaning.
|
||||
4. **Strikethrough is inert/lazy.** `strikethrough` is an inert wrapper: it is
|
||||
never traversed for execution, its content is prose, and a `strong`/`emph`
|
||||
inside it does not execute. It is not an error to write it; the language
|
||||
simply does not use formatting as a control-flow mechanism here. (A code
|
||||
span or math inside it is likewise inert.)
|
||||
5. **Unmodelled nodes are preserved** as `other { tag }` with children and raw
|
||||
literal. They are inert but never silently dropped from `inspect --ast`.
|
||||
6. **Literal payloads are kept** on inert leaves: `text`, `code`, `code_block`,
|
||||
`math`, `math_block`, `html_block`, `html_inline`, `front_matter`.
|
||||
7. **Other frozen normalizations:**
|
||||
* `**x**`/`__x__` and `*x*`/`_x_` are the same node kinds (`strong`/`emph`);
|
||||
* ordered delimiters `1.`/`1)` and bullets `-`/`*`/`+` are the same
|
||||
`list`/`item` structure; role depends on `ordered`, not marker;
|
||||
* tight vs loose lists only affect rendering (`tight` is recorded, inert);
|
||||
* `link`/`image` keep `url`/`title`; `table` keeps column alignments,
|
||||
`table_row` keeps `header`; `heading` keeps `level`/`setext`;
|
||||
* a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the
|
||||
original file;
|
||||
* front matter is recognized only at the document start (after BOM), with
|
||||
`---` opening and `---`/`...` closing; a later `---` is a thematic break
|
||||
or Setext underline.
|
||||
|
||||
These rules are part of profile id `marqdo.md.v1`. Changing them (or bumping
|
||||
the comrak version/options) requires a profile id bump, updated goldens and this
|
||||
ADR being superseded.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Distinguish Setext vs ATX headings semantically.** Rejected: identical AST
|
||||
structure outside the marker must have identical meaning (`## H` and Setext
|
||||
H2 are the same function). Keeping `setext` only for inspection avoids
|
||||
silently diverging programs when a document changes style.
|
||||
2. **Introduce a separate `task_item` node kind.** Rejected: it would require
|
||||
every list-role check to special-case two item kinds and could be used to
|
||||
hide a control structure behind a checkbox. Normalizing to `item { task }`
|
||||
keeps list structure authoritative.
|
||||
3. **Strip the tick information and store only the literal.** Rejected: payload
|
||||
spans returned by the expression parser are relative to the concatenated
|
||||
payload; without delimiter lengths, error spans cannot be mapped back to
|
||||
file bytes precisely.
|
||||
4. **Traverse strikethrough content for execution.** Rejected: it is a lazy
|
||||
region by profile; executing inside it would make formatting semantics
|
||||
surprising ("struck-through code runs"). It would also create a second way
|
||||
to hide executable nodes.
|
||||
5. **Drop unknown parser nodes.** Rejected: `inspect --ast` must show what the
|
||||
parser produced; dropping syntax hides profile drift.
|
||||
6. **Preserve raw source and re-parse with a custom parser.** Rejected by the
|
||||
project's core rule: one standard Markdown parser, no second Markdown AST.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
Frozen JSON shapes (`inspect --ast --json`):
|
||||
|
||||
```json
|
||||
{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
|
||||
"children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"list","ordered":false,"tight":true,"span":{...},
|
||||
"children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}
|
||||
```
|
||||
|
||||
Attribute order in the JSON is fixed by the serializer; goldens are regenerated
|
||||
only through the explicit update workflow.
|
||||
|
||||
### 5.1 Known interaction to keep in mind
|
||||
|
||||
Inside a function body, a `**strong**` (or an ordered/bullet list) nested under
|
||||
`strikethrough` is inert and never executes; because the mapper does not
|
||||
traverse strikethrough, it also produces no warning. Outside function bodies,
|
||||
the `markdown.inert_executable` scan walks a top-level paragraph recursively,
|
||||
so such a nested node may still produce the inert warning even though it would
|
||||
not execute if the paragraph were inside a function. This is consistent with
|
||||
"formatting never hides execution", but an author who intends execution must
|
||||
not wrap it in strikethrough. Warnings for lazy-region content inside functions
|
||||
are a possible future refinement; no behavior change is planned without an ADR.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Profile id `marqdo.md.v1` is frozen with comrak `=1.0.0` and the exact
|
||||
extension flags; the unit test
|
||||
`crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen`
|
||||
asserts the option table (tables, tasklist, strikethrough, math dollars,
|
||||
front matter; all other extensions disabled).
|
||||
* Adding an AST field is a JSON output change for `inspect --ast`; it needs a
|
||||
goldens update and a profile note. Changing a semantic rule (e.g. executing
|
||||
inside strikethrough) is a language change and requires an ADR.
|
||||
* No package/ABI/wire impact.
|
||||
|
||||
## Security
|
||||
|
||||
* The lazy-region rules ensure inline formatting inside headings, tables,
|
||||
links, code, math, HTML and front matter never executes — a context-aware
|
||||
visitor is used (`crates/marqdo-frontend/src/map.rs` only descends from
|
||||
paragraphs into `strong`/`emph`, and ignores every other inline node).
|
||||
* Strikethrough cannot be used to smuggle executable content; it is inert.
|
||||
* Normalizing task items to `item` prevents a checkbox from hiding a
|
||||
branch/loop header; the condition-header check still applies.
|
||||
* No new capabilities or host calls.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-markdown/src/profile.rs`:
|
||||
* `profile_options_are_frozen` — exact extension/parse option table;
|
||||
* `setext_headings_are_heading_nodes` — Setext produces `heading { setext: true }`;
|
||||
* `tables_and_task_lists_are_supported` — task items carry `task: Some(_)`;
|
||||
* span/BOM tests.
|
||||
* Frontend mapping tests use code spans as conditions and payloads, exercising
|
||||
`ticks`/payload span mapping.
|
||||
* `spec/markdown-profile.md` §5/§6 is the normative JSON shape; AST goldens
|
||||
under `crates/cli/tests` (fixtures workstream) regenerate only with the
|
||||
documented update command.
|
||||
* Conformance fixtures are expected to include `**a**` vs `__a__`, Setext vs
|
||||
ATX, task/list variants and strikethrough inertness.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration for documents: Setext and ATX were always the same function under
|
||||
this profile. When comrak is upgraded or the extension set changes, bump the
|
||||
profile id, regenerate AST goldens, re-run all specs/ADRs that reference node
|
||||
shapes, and document the change here (superseding this ADR if a rule changes).
|
||||
Code-span `ticks` and `task` are additive inspection fields; keep them optional
|
||||
in consumers that parse the JSON.
|
||||
@@ -0,0 +1,225 @@
|
||||
# ADR-0006: CLI output, exit-code and JSON contract
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
CI, scripts and coding agents need a stable CLI contract, not just human
|
||||
output. The blueprint (§12) requires: all commands available with `--json`,
|
||||
non-zero error exit codes, program stdout kept separate from diagnostics, and
|
||||
machine-readable error codes. The implemented CLI must be documented precisely
|
||||
because scripts will depend on stream separation and exit status.
|
||||
|
||||
Implementation: `crates/cli/src/main.rs` (clap definitions, process exit),
|
||||
`crates/cli/src/commands.rs` (behavior), `crates/cli/src/output.rs`
|
||||
(diagnostic/JSON rendering), `crates/cli/src/args.rs` (`--arg` constants).
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Commands and flags (implemented surface)
|
||||
|
||||
| Command | Arguments / flags | What it does |
|
||||
|---|---|---|
|
||||
| `init [dir]` | — | create `marqdo.yaml`, `index.mq.md`, `.gitignore`; refuses an existing manifest |
|
||||
| `run [file]` | `--mode`, `--allow-read`, `--allow-write`, `--allow-foreign`, `--allow-plugin`, `--fuel`, `--stack`, `--max-output`, `--timeout`, `--plugin`, `--trust-native-plugin`, `--json`, `--trace` | execute `main` (or default `index.mq.md`) |
|
||||
| `call <file> <function>` | `--arg NAME=VALUE` (repeatable) plus all `run` flags | execute one named function |
|
||||
| `check [file]` | `--json` | static check; never executes |
|
||||
| `inspect [file]` | `--ast`, `--units`, `--signature`, `--ir`, `--refs`, `--platform`, `--json` | static inspection; never executes |
|
||||
| `fmt [file]` | `--check` | normalize CRLF and trailing newlines |
|
||||
| `plugin inspect <dir-or-yaml>` | `--json` | validate manifest + verify artifact hash; never loads code |
|
||||
| `add <name>` | `--path`, `--git`, `--rev`, `--json` | edit `marqdo.yaml` |
|
||||
| `remove <name>` | `--json` | edit `marqdo.yaml` (and prune a stale lock entry) |
|
||||
| `lock` | `--json` | write deterministic `marqdo.lock` |
|
||||
| `sync` | `--locked`, `--json` | verify the lock is current |
|
||||
|
||||
There is no `test`, `tree`, `update`, `vendor`, `add` for plugins, `kernel`
|
||||
command or `--project`/`--offline` flag in this build. `run` defaults the
|
||||
file to `index.mq.md`; `check`/`inspect`/`fmt` do too. `call` requires an
|
||||
explicit file path plus a function name.
|
||||
|
||||
`--mode` is only on `run`/`call` and accepts `pure|project|strict`
|
||||
(default `project`). `--trust-native-plugin`/`--plugin` are only meaningful on
|
||||
`run`/`call`. `inspect --platform` still requires a document argument (or
|
||||
defaults to `index.mq.md`), because project loading happens first.
|
||||
|
||||
### 2. Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| `0` | success (for `check`: no **error** diagnostics; warnings do not fail) |
|
||||
| `1` | a checked document has errors; a runtime/security failure; strict refusal; plugin load/inspect failure; lock/sync mismatch; `fmt --check` would reformat; `init` found an existing manifest; `add` duplicate; `remove` missing dependency; `sync` stale lock |
|
||||
| `2` | usage error (clap), unreadable entry file, invalid `--mode` name, invalid capability combination (e.g. `--allow-read` in `pure`), invalid `--arg` value, unreadable manifest for project commands, unreadable file for `fmt` |
|
||||
|
||||
Codes are stable contract. Diagnostic **codes** (for example
|
||||
`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`,
|
||||
`permission.denied`, `ref.ambiguous`) are stable; message wording is not.
|
||||
|
||||
### 3. Stream separation
|
||||
|
||||
* **Program output** (`打印`/`print`) goes to stdout in text mode. In JSON mode
|
||||
it is embedded as the `stdout` string; it is never interleaved with JSON.
|
||||
* **Diagnostics and effect traces** go to stderr in text mode.
|
||||
* `run` in text mode writes only program output to stdout (no implicit return
|
||||
value). `call` in text mode additionally prints the returned value (using
|
||||
`Value::display`) when it is not `Unit`.
|
||||
* `check` text mode: diagnostics to stderr; `checked N module(s): ok` to stdout
|
||||
on success; a `check failed: ...` summary to stderr on failure.
|
||||
* `inspect` prints pretty JSON-ish structures to stdout; load errors to stderr.
|
||||
* JSON mode prints exactly one pretty JSON object to stdout per invocation.
|
||||
`add`/`remove`/`lock`/`sync` still print errors as plain `error: ...` on
|
||||
stderr even with `--json` (JSON success envelope only) — an inconsistency to
|
||||
fix before the contract is declared final.
|
||||
|
||||
### 4. JSON output shapes
|
||||
|
||||
`check`:
|
||||
|
||||
```json
|
||||
{"command":"check","success":true,"errors":0,
|
||||
"diagnostics":[{"file":"index.mq.md","module":"index.mq.md",
|
||||
"line":3,"column":5,"severity":"error","code":"expr.parse",
|
||||
"message":"...","hints":["..."]}]}
|
||||
```
|
||||
|
||||
`inspect`:
|
||||
|
||||
```json
|
||||
{"command":"inspect","module":"index.mq.md",
|
||||
"ast":{...},"units":[...],"signatures":[...],"ir":[...],
|
||||
"refs":[...],"platform":{...}}
|
||||
```
|
||||
|
||||
Only the requested sections are present. With no section flags, a `summary`
|
||||
with `modules`, `functions`, `units`, `resources` is emitted instead.
|
||||
|
||||
`run`/`call` (success):
|
||||
|
||||
```json
|
||||
{"command":"run","success":true,
|
||||
"value":{"type":"text","value":"..."},
|
||||
"stdout":"program stdout text",
|
||||
"trace":[{"op":"print","module":"index.mq.md",
|
||||
"span":{"start":43,"end":57},"duration_micros":3,"result":"ok"}]}
|
||||
```
|
||||
|
||||
Note: `call --json` also emits `"command":"run"` in this build (shared
|
||||
implementation). Consumers should accept both names or key on the presence of
|
||||
`diagnostics`/`value`; this is a known quirk.
|
||||
|
||||
`run`/`call` (failure): `{"command":"run","success":false,"stdout":...,
|
||||
"error":{"code":"...","message":"...","span":{"line":..,"column":..,"file":..},
|
||||
"frames":[...]},"trace":[...]}`. Runtime error spans are currently resolved
|
||||
against the **entry file** only, even when the error came from an imported
|
||||
module; frames carry function names, not module ids. This is a known precision
|
||||
limitation.
|
||||
|
||||
`plugin inspect`:
|
||||
|
||||
```json
|
||||
{"command":"plugin.inspect","success":true,"plugin":"demo.solver",
|
||||
"version":"0.1.0","abi":"^1.0","abi_compatibility":"compatible",
|
||||
"target":"x86_64-unknown-linux-gnu",
|
||||
"artifact":{"file":"solver.so","path":"...","sha256":"...","verified":true},
|
||||
"exports":[{"name":"solver.add","signature":"($a: Int, $b: Int) -> Int"}],
|
||||
"loaded":false}
|
||||
```
|
||||
|
||||
`load_project` failures with `--json` print `{"success":false,"error":"..."}`
|
||||
without a `command` field.
|
||||
|
||||
There is no `json_schema_version` field yet. The blueprint asks for a
|
||||
versioned JSON schema; until it exists, treat these keys as the stable subset
|
||||
and ignore unknown keys. Adding keys is not considered breaking; renaming or
|
||||
removing one is.
|
||||
|
||||
### 5. `--arg` values
|
||||
|
||||
`--arg NAME=VALUE` parses `VALUE` with the **expression grammar restricted to
|
||||
constants** (`crates/cli/src/args.rs`): integers, floats, double-quoted text
|
||||
(escapes supported), `true`/`false`/`None`, lists, records, and unary `-`/`+`
|
||||
over constants. Calls and identifiers are rejected ("must be a literal
|
||||
constant"). Text must be double-quoted; there is no bare-word fallback.
|
||||
|
||||
### 6. Determinism and safety of read-only commands
|
||||
|
||||
* `check`/`inspect` never execute project code (ADR context; see
|
||||
`spec/security.md` §7).
|
||||
* `plugin inspect` never loads plugin code.
|
||||
* `fmt` only rewrites the file when not `--check`; exit 1 when `--check`
|
||||
detects a difference.
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **Use exit code 1 for all failures.** Rejected: scripts need to distinguish
|
||||
usage errors (2) from document/runtime failures (1); it also mirrors common
|
||||
CLI conventions and clap's own usage exit code.
|
||||
2. **Print JSON even for usage errors.** Rejected for now: clap owns usage
|
||||
output before our code runs; wrapping it would mean intercepting clap
|
||||
errors. Documented as a limitation.
|
||||
3. **Interleave program output and JSON.** Rejected: it makes JSON
|
||||
unparseable. Program stdout is embedded as a string instead.
|
||||
4. **Return values from `run` on stdout in text mode.** Rejected: `run` is the
|
||||
batch entry; implicit value printing would pollute program output. `call`
|
||||
prints the return value exactly when non-`Unit`.
|
||||
5. **Human-only messages with numeric exit codes only.** Rejected: diagnostics
|
||||
and runtime errors have stable machine codes in both text and JSON.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
The CLI itself does not change the AST. What it exposes:
|
||||
|
||||
* `inspect --ast` = canonical `MdNode` tree (see ADR-0005);
|
||||
* `inspect --units` = `UnitInfo[]` (`marqdo-ir`);
|
||||
* `inspect --signature` = `{name, local_name, signature: explicit|inferred,
|
||||
params:[{name,type,has_default}]}`;
|
||||
* `inspect --ir` = `Function[]` with `Stmt`/`Expr` and spans;
|
||||
* `inspect --refs` = link edge array (`spec/references.md` §4);
|
||||
* `inspect --platform` = `PlatformCapabilities` JSON.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* Exit codes, stream separation and the documented JSON keys are the public
|
||||
CLI contract. Changing them requires this ADR to be superseded and a
|
||||
changelog entry.
|
||||
* The old "design-only starter" README listed planned commands (`add`,
|
||||
`lock && sync`, `plugin inspect`); all listed commands except a versioned
|
||||
JSON schema are now implemented as described here.
|
||||
* Diagnostic code renames are breaking for CI; message changes are not.
|
||||
* No package/ABI/wire changes.
|
||||
|
||||
## Security
|
||||
|
||||
* `--json`/`--trace` enable effect tracing; traces contain op/module/span/
|
||||
duration/result summary, never argument values (no secrets) — see
|
||||
`spec/security.md` §4.
|
||||
* JSON mode embeds program stdout, which is program-controlled data; consumers
|
||||
must not treat it as trusted.
|
||||
* Refusing to run with compile errors happens before host creation, so JSON
|
||||
failure output cannot leak effects.
|
||||
* Strict-mode refusal and capability denials produce stable codes for CI.
|
||||
|
||||
## Testing
|
||||
|
||||
* CLI behavior is currently covered by manual smoke checks and library-level
|
||||
tests; the fixtures workstream is adding CLI integration tests under
|
||||
`crates/cli/tests` (goldens/negative cases) and `tests/`.
|
||||
* `crates/cli/src/args.rs` unit tests cover typed `--arg` parsing and rejection
|
||||
of non-constant values.
|
||||
* `crates/cli/src/output.rs` has no unit tests; diagnostic JSON is exercised
|
||||
through check/run smoke tests.
|
||||
* Add contract tests for: exit codes per failure class, stdout/stderr
|
||||
separation, JSON key presence, `call --json` command naming, and runtime
|
||||
error span from a non-entry module before calling the schema "final".
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
1. Keep the current shapes stable while adding tests.
|
||||
2. Add a `schema_version` to JSON envelopes and document the versioned schema;
|
||||
then document that unknown keys may be added (forward compatible) while
|
||||
existing keys are frozen.
|
||||
3. Fix the `call --json` `"command":"run"` quirk and JSON errors in
|
||||
`add/remove/lock/sync`; both are additive/corrective and should be
|
||||
announced in the changelog.
|
||||
4. If a command is renamed or an exit code changes, update this ADR, the
|
||||
README quickstart, `docs/STATUS.md` and CI scripts in the same change.
|
||||
@@ -0,0 +1,178 @@
|
||||
# ADR-0007: Arbitrary-precision `Int` and the typed wire encoding (S-08)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
|
||||
## Context
|
||||
|
||||
The blueprint (§3.1 and unresolved issue **S-08**) requires a numeric model
|
||||
that does not silently lose precision at language, JSON/ABI and math
|
||||
boundaries. It recommends arbitrary-precision integers for `Int` and type tags
|
||||
plus string forms for large integers on the wire.
|
||||
|
||||
Three independent places could lose information:
|
||||
|
||||
1. **Language arithmetic.** A fixed-width `i64` would make `Int` silently
|
||||
overflow or wrap. Factorials, powers and scientific constants routinely
|
||||
exceed 64 bits. The expression grammar already has arbitrary-length integer
|
||||
literals (`spec/expression.ebnf`).
|
||||
2. **JSON / wire.** JSON has one number type (IEEE-754 double in most parsers).
|
||||
Encoding `12345678901234567890` as a JSON number loses digits in
|
||||
JavaScript/Go consumers; encoding `1` as a number makes `Int` and `Float`
|
||||
indistinguishable.
|
||||
3. **Math kernel.** The exact-rational engine needs an integer type that is
|
||||
the *same* type as `Value::Int`, or every formula/binding crossing the host
|
||||
boundary needs a lossy conversion.
|
||||
|
||||
The chosen dependency set (`Cargo.toml`) is:
|
||||
|
||||
```toml
|
||||
num-bigint = { version = "0.4", features = ["serde"] }
|
||||
num-rational = { version = "0.4", features = ["serde"] }
|
||||
num-traits = "0.2"
|
||||
```
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. `Int` is arbitrary precision (`num-bigint::BigInt`)
|
||||
|
||||
* `Value::Int(BigInt)` (`crates/marqdo-value/src/value.rs`) is the only integer
|
||||
representation in the runtime and the expression AST
|
||||
(`crates/marqdo-expression/src/ast.rs`).
|
||||
* Expression integer literals are arbitrary-precision decimal; there is no
|
||||
fixed-width parsing step.
|
||||
* Arithmetic (`crates/marqdo-runtime/src/value_ops.rs`):
|
||||
* `+ - *` on two `Int` are exact `BigInt` operations;
|
||||
* `/` **always** produces `Float` (exact ratio converted to binary64), so
|
||||
there is no integer division surprise;
|
||||
* `//` and `%` require `Int` operands; floor division is mathematical floor
|
||||
and `a % b == a - b * (a // b)`, with the sign following the divisor;
|
||||
* mixed `Int`/`Float` arithmetic produces `Float`; converting an `Int`
|
||||
larger than 2^53 may lose precision (documented, not silently exported as
|
||||
an `Int`);
|
||||
* `Int`/`Float` comparisons are **exact**: the float is treated as the exact
|
||||
binary rational it is (`BigRational::from_float`), so
|
||||
`9007199254740993 > 9007199254740992.0` is `true`;
|
||||
* `**` on `Int` is exact exponentiation, with a negative exponent rejected
|
||||
as a type error and guards `MAX_EXACT_EXPONENT = 100_000` and
|
||||
`MAX_POWER_BITS = 1_000_000` (`resource.exhausted`).
|
||||
* There is no truthiness and no implicit string↔number conversion; `整数`/`int`
|
||||
performs explicit conversions and rejects non-integral floats.
|
||||
|
||||
### 2. `num-bigint` 0.4 is pinned to match `num-rational` 0.4
|
||||
|
||||
`num-rational`'s generic `Ratio<T>` is instantiated as
|
||||
`Ratio<num_bigint::BigInt>` in `crates/marqdo-math/src/ast.rs`
|
||||
(`pub type Rational = Ratio<BigInt>`). The two crates must resolve to the same
|
||||
`num-bigint` version, otherwise `BigInt` values from the value model cannot be
|
||||
passed to `Ratio` without conversion (or a duplicate type). The workspace pins
|
||||
both at the `0.4` line and the lockfile resolves one `num-bigint`. Upgrading
|
||||
either crate requires checking that this shared-type invariant still holds and
|
||||
running the math + runtime test suites.
|
||||
|
||||
### 3. The wire encoding never sends an integer as a JSON number
|
||||
|
||||
Canonical encoding (`crates/marqdo-value/src/wire.rs`; full contract in
|
||||
`spec/value-wire.md`):
|
||||
|
||||
```json
|
||||
{"type":"int","decimal":"12345678901234567890"}
|
||||
```
|
||||
|
||||
* Every value is a JSON object with a `"type"` tag.
|
||||
* `Int` uses a base-10 string. All digits, sign and magnitude survive any JSON
|
||||
consumer including JavaScript.
|
||||
* `Float` uses a JSON number only when finite; `NaN`, `Infinity`,
|
||||
`-Infinity` use explicit `"special"` tags.
|
||||
* JSON numbers appear only in non-value positions (`span.start/end`,
|
||||
`plugin_handle.id`).
|
||||
* Decoding is validated: unknown tags, missing fields, invalid decimals, wrong
|
||||
types and limits are `WireError`s, never coercions.
|
||||
* The same encoding is used for plugin ABI request/response values, foreign
|
||||
kernel stdin `input`, foreign typed return values and `run --json` values.
|
||||
|
||||
### 4. Math results preserve exactness where possible
|
||||
|
||||
The math kernel keeps exact rationals and returns `Value::Int` exactly when the
|
||||
exact result is an integer, otherwise `Value::Float`. Decimal literals are
|
||||
exact rationals (`0.25` = `1/4`), and `\frac{1}{3} * 3` is exactly `1`. Constants
|
||||
(`\pi`, `\e`) and non-perfect square roots introduce binary64 approximations and
|
||||
the result is a `Float` (`spec/math.md` §3).
|
||||
|
||||
## Alternatives
|
||||
|
||||
1. **`i64`/`u64` integers with overflow errors.** Rejected: common programs
|
||||
(factorials, powers) would need explicit bignum workarounds; the language
|
||||
would inherit fixed-width semantics not implied by its arithmetic.
|
||||
2. **`i128` integers.** Same problem, just a bigger ceiling; and the wire
|
||||
precision problem remains for values beyond 2^53.
|
||||
3. **Encode integers as JSON numbers and hope consumers use bignum JSON.**
|
||||
Rejected: the JSON standard/ecosystem does not guarantee it; JavaScript's
|
||||
`JSON.parse` loses digits; this is the exact S-08 failure.
|
||||
4. **Encode every number as a string without type tags.** Rejected: loses the
|
||||
`Int`/`Float` distinction and makes all consumers parse strings; type tags
|
||||
are explicit and cheap.
|
||||
5. **Use `f64` for all numbers.** Rejected: contradicts exact `Int`
|
||||
semantics, the math kernel's exact rational requirement, and the
|
||||
no-silent-precision-loss rule.
|
||||
6. **Different integer crates in value vs math.** Rejected: forces lossy or
|
||||
string conversions at every kernel boundary and duplicates tests.
|
||||
|
||||
## Markdown AST shape
|
||||
|
||||
No AST change. Numeric literals live inside executable payloads only (code
|
||||
spans/plain text of `strong`/`emph`/conditions). A `math` node is inert LaTeX;
|
||||
its numeric interpretation is the math kernel's job. The JSON inspection view
|
||||
of IR/AST contains spans as JSON numbers, not user numeric values.
|
||||
|
||||
## Compatibility
|
||||
|
||||
* `Int` semantics (arbitrary precision, `/` → `Float`, exact comparisons,
|
||||
`//`/`%` sign rules) are frozen language behavior; changing them requires an
|
||||
ADR and conformance fixtures.
|
||||
* `wire_version = 1` with typed tags and decimal-string integers is frozen.
|
||||
Changing the integer representation is a wire-breaking change and needs a
|
||||
new `wire_version` plus plugin ABI considerations.
|
||||
* The dependency pin may move within `0.4` for security fixes, provided
|
||||
`num-bigint` remains a single resolved version shared with `num-rational`.
|
||||
* Package/lock formats are unaffected.
|
||||
|
||||
## Security
|
||||
|
||||
* Untrusted wire data cannot trigger huge allocations through the integer
|
||||
parser without also counting against value depth/string limits: a decimal
|
||||
string is bounded by `max_string_bytes` (16 MiB default) and `BigInt` parsing
|
||||
is linear. The current decoder does not separately cap the *digit count* of
|
||||
an `Int`; the envelope/string caps are the effective bound (open item).
|
||||
* Arithmetic can grow memory; the runtime bounds it with fuel and the
|
||||
`Int` power guards (`MAX_EXACT_EXPONENT`, `MAX_POWER_BITS`) and the math
|
||||
kernel bounds exact numerator/denominator bits (`MAX_RESULT_BITS`).
|
||||
* Exact `Int`/`Float` comparison prevents a class of "looks equal but is not"
|
||||
authorization or verification bugs.
|
||||
* No new capabilities; numeric parsing is pure.
|
||||
|
||||
## Testing
|
||||
|
||||
* `crates/marqdo-value/src/value.rs`: `big_integers_are_arbitrary_precision`,
|
||||
`float_to_bigint_requires_integral_value`, display/format tests.
|
||||
* `crates/marqdo-value/src/wire.rs`: `big_integers_keep_all_digits`,
|
||||
`special_floats_are_tagged`, scalar round trips, untrusted-input limits.
|
||||
* `crates/marqdo-runtime/src/value_ops.rs`: exact integer arithmetic,
|
||||
`/`→Float, floor/mod sign rules, exact mixed comparisons, power guards,
|
||||
division-by-zero.
|
||||
* `crates/marqdo-math/tests/kernel.rs`: exact big integer arithmetic, exact
|
||||
rational division, decimal literals as rationals, `pi`/`sqrt` approximating
|
||||
to `Float`, `Int` vs `Float` result typing.
|
||||
* `crates/marqdo-plugin-abi/src/wire.rs`: request round trips preserving tags
|
||||
and refusing coercible-looking values beyond `i64`.
|
||||
* `crates/marqdo-foreign/tests/kernels.rs`: typed foreign return values decode
|
||||
exactly.
|
||||
|
||||
## Migration & rollout
|
||||
|
||||
No migration: the implementation has used `BigInt` and the typed wire format
|
||||
since M0.5. For future dependency upgrades: run `cargo test --workspace`,
|
||||
verify one resolved `num-bigint` (`cargo tree -i num-bigint`), and re-run wire
|
||||
compatibility tests. For any change to the wire integer encoding, bump
|
||||
`wire_version`, update `spec/value-wire.md`, the plugin ABI documentation and
|
||||
the foreign protocol in one coordinated change.
|
||||
Reference in new issue
Block a user