Files
marqdo-cli/tests/README.md
T
cflmy 847c2c1580 fix: float wire precision, native stack safety, refs diagnostics and fixtures
Findings from the conformance corpus workstream, all fixed at the source:

- value/wire: enable serde_json's float_roundtrip so typed wire floats keep
  their exact digits across foreign/plugin boundaries (Python pi*9 now stays
  28.274333882308138).
- host: default interpreter stack budget lowered to 128 frames so deeply
  recursive documents hit a structured resource.exhausted instead of exhausting
  the native debug-build Rust stack; --stack remains available.
- frontend/map: calling an external HTTP(S) link is reported as ref.unresolved
  (consistent with other unresolvable call targets) instead of math.unsupported.
- frontend/analyze: invalid front matter emits a single frontmatter.invalid
  diagnostic (the YAML error is not repeated by the imports pass).
- spec/mapping: clarify that would-be execution nodes outside any function body
  are inert with a markdown.inert_executable warning (no "return outside a
  function" hard-error claim).
- tests: add the conformance corpus and harness (150 AST goldens, 36 semantic
  goldens, 48 hard-error + 6 warning + 10 runtime-error negatives, CLI/module/
  plugin integration tests, prose-invariance properties) and update the
  three reported observations to the fixed behaviour.
2026-11-20 10:54:16 +08:00

168 lines
7.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Marqdo conformance and integration test corpus
This directory holds the P0/P1 conformance corpus required by
`docs/DEVELOPMENT_BLUEPRINT.md` §13/§14. The Rust harnesses live in
`crates/cli/tests/` (they use the workspace libraries directly and the
`marqdo` binary via `CARGO_BIN_EXE_marqdo`); the fixtures are plain files here
so they can be rendered, diffed and reviewed like Markdown documents.
Everything here is data. Nothing in this directory modifies the repository, and
no test commits, copies or rewrites files outside a throw-away temp directory.
## Layout
```text
tests/
├── conformance/
│ ├── ast/ 150 × *.mq.md + sibling *.ast.json goldens
│ └── semantics/ 36 × *.mq.md + sibling *.expected.json goldens
├── negative/
│ ├── *.mq.md 48 × compile-time hard-error fixtures + *.expected.json
│ ├── warnings/ 6 × inert-executable warning fixtures + *.expected.json
│ └── runtime/ 10 × runtime-error fixtures + *.expected.json
└── README.md this file
```
Rust harnesses (all in `crates/cli/tests/`):
| Harness | What it proves |
|---|---|
| `conformance_ast.rs` | The frozen `marqdo.md.v1` profile parses each fixture into exactly the checked-in AST (node kinds, attributes, literals, byte spans, order). |
| `conformance_semantics.rs` | Each semantic fixture compiles through `marqdo_frontend`, is assembled into `marqdo_ir::Program` and executed with `marqdo_runtime` + `marqdo_host::StdHost`; returned wire value and program stdout match. |
| `negative.rs` | Invalid executable shapes are **hard errors** (never prose fallbacks), exact sorted diagnostic-code lists; warnings and runtime error codes are asserted in their own suites. |
| `properties.rs` | Blueprint §14.2 prose invariance: inserting/removing pure narration paragraphs never changes the executable function IR (modulo source spans) and diagnostics stay error-free. |
| `integration_cli.rs` | Real CLI contract: `init`, `check`, `run`, `call`, `inspect`, `fmt`, multi-module imports/cycles, `lock`/`sync --locked`, plugin inspection and explicitly trusted plugin invocation. |
## Running
Use the workspace cargo cache/target as documented in `AGENTS.md`:
```bash
export PATH=/root/.cargo/bin:$PATH
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
cargo test -p marqdo-cli # all conformance + integration harnesses
cargo test --workspace # whole workspace (includes crate unit tests)
```
Individual harnesses:
```bash
cargo test -p marqdo-cli --test conformance_ast
cargo test -p marqdo-cli --test conformance_semantics
cargo test -p marqdo-cli --test negative
cargo test -p marqdo-cli --test properties
cargo test -p marqdo-cli --test integration_cli
```
Fixture discovery is directory-based and sorted by file name, so the order is
deterministic on every platform. The AST and semantic suites fail loudly if the
corpus shrinks below 100 / 25 fixtures, and the negative suite fails below 30
compile-time hard-error fixtures.
## AST goldens and `UPDATE_GOLDENS`
`*.ast.json` files are the pretty-printed canonical AST view
(`marqdo_markdown::parse(source).root`), exactly as `marqdo inspect --ast --json`
would print it. They pin parser version, extension flags and span arithmetic.
Regenerate **only** for an intentional, ADR-backed profile change (the profile
is frozen in `spec/markdown-profile.md`):
```bash
UPDATE_GOLDENS=1 cargo test -p marqdo-cli --test conformance_ast
```
The update run rewrites every golden, prints how many it rewrote and **passes**.
Re-run without the variable to verify, and review the golden diff: a golden
change is a language change, not a bug fix.
A fixture without a golden is an error, and a golden without a fixture is an
error (stale corpus output), so add/remove both together.
## Semantic goldens
Each `*.expected.json` has the shape:
```json
{
"function": "<name passed to marqdo_runtime::run>",
"args": { "<name>": <wire value> },
"value": <wire value>,
"stdout": "..."
}
```
* `args` and `value` use the typed wire encoding from `marqdo_value::to_wire`
(`{"type":"int","decimal":"42"}`, `{"type":"text","value":"…"}`, …).
* `stdout` is the program output captured from the host (never diagnostics).
* Optional `"grants": ["foreign"]` enables `foreign.call`; that fixture is
skipped with a printed notice when `python3` is unavailable.
All values were produced by running the implementation and then checked by
hand against the fixture source. `UPDATE_SEMANTICS_GOLDENS=1` exists for
authoring convenience, but it must not be used blindly: rewriting a semantic
golden without verifying the value would defeat the purpose of the corpus.
## Negative fixtures
Compile-time (`tests/negative/*.mq.md`):
```json
{"codes": ["markdown.invalid_construct"]}
```
The expected list is the exact sorted multiset of **error** diagnostic codes
produced by `marqdo_frontend::analyze_source`. Warnings emitted by the same
document are ignored here. Every listed code is additionally asserted to have
`error` severity, and every fixture must produce at least one error — invalid
executable shapes never fall back to prose.
Warnings (`tests/negative/warnings/*.mq.md`):
```json
{"severity": "warning", "codes": ["markdown.inert_executable"]}
```
These documents must compile **without errors**; the sorted multiset of warning
codes must match exactly. `return`/`strong`/lists/block quotes outside a
function body are inert content (warning), not hard errors.
Runtime (`tests/negative/runtime/*.mq.md`):
```json
{"function": "main", "args": {}, "codes": ["runtime.division_by_zero"]}
```
These documents compile, then must fail at runtime with the expected stable
error code. Optional keys: `"grants": ["foreign"]` and `"stack_limit": 24`
(used for the recursion-depth fixture so the budget — not the host process
stack — is what fails).
## CLI integration notes
`integration_cli.rs` never leaves the repository dirty:
* temporary projects are created under `std::env::temp_dir()` and removed on
drop;
* the demo C plugin is built **in a temp copy** of `examples/plugin-demo`
(`cc -shared -fPIC -I <repo>/include`), and the `sha256` in the copied
`plugin.yaml` is refreshed there. The repo copy and its `.so` (gitignored) are
never touched;
* tests that need `python3` or `cc` print a skip notice and return when the tool
is unavailable.
## Notes discovered while authoring (all since fixed)
The fixtures initially pinned three observations that were reported and then
fixed in the implementation; the goldens below reflect the fixed behaviour:
* wire floats round-trip exactly (`serde_json/float_roundtrip` is enabled), so
Python `math.pi * 9` keeps `28.274333882308138`;
* the default interpreter stack budget is 128 frames so a deeply recursive
document returns a structured `resource.exhausted` instead of exhausting the
native debug-build stack (`--stack` can raise it deliberately);
* an external-link call form (`> [x](https://example.com)()`) is rejected with
`ref.unresolved`, the same code used for other unresolvable call targets.