Complete the documentation set for the M0.5 toolchain, grounded in the code: - spec/references.md: import/reference/call edges, identities, loader rules, anchors, `ref` resolution/ambiguity, `check --refs` validation. - spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal string rule, special floats, limits and exact float round-trip. - spec/security.md: modes (strict refuses without OS isolation), capabilities, path rules, budgets, tracing, foreign process caveats, plugin trust. - spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync, unsupported registry/git/environment work. - spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the implemented CLI plugin-invocation flow with its limitations. - spec/math.md and spec/foreign.md: the implemented math subset and the foreign request/response/timeout/capability contract. - docs/adr/0001..0007: accepted decisions for the span foundation, inert executables outside function bodies, ref ambiguity (S-07), strict refusal, AST normalization, CLI contract and BigInt/typed wire (S-08). - docs/contributor-guide.md: repo map, workflow, fixture/golden updates, quality gate, ADR/commit policy. - docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18 Alpha checklist (conclusion: not yet Alpha-complete), including known divergences rather than overclaims. - README.md rewritten from "design-only starter" to the working implementation with quickstart, implemented phases, deliberate limits and doc index. - AGENTS.md: updated pointers/workflow only; architectural rules unchanged. - Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside functions, unsupported math.simplify/differentiate).
8.6 KiB
ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)
Status: Accepted Date: 2026-11-20
Context
The language derives program meaning from the canonical AST, never from
source spelling (spec/markdown-profile.md, spec/mapping.md §1). The parser
(comrak =1.0.0, frozen options) normalizes several syntaxes before Marqdo
sees them, and a few normalizations are lossy:
# Hand SetextH\n===both become a heading node; the source spelling is lost unless recorded;- GFM task list items (
- [x] text) produce a task-specific item node; - inline code spans are stored as a literal without the backtick delimiters, but source spans include them;
~~strikethrough~~is enabled by the profile and wraps arbitrary inlines.
The language needs a single, documented answer per question, because node type and structural position decide executable roles.
Decision
The profile (marqdo.md.v1) and the canonical AST
(crates/marqdo-markdown/src/ast.rs, src/profile.rs) freeze these rules:
- Setext headings are headings.
# nameand a Setext level-1/2 underline both produceheading { level, setext };setextis inspection-only and never affects semantics.## nameand a Setext level-2 heading are both functions. The profile keepsparse.ignore_setext = false. - Task-list items are
itemnodes with a task flag.item { task: Some(true|false) }for GFM task items,task: Noneotherwise. Node type staysitem, so list structure (ordered vs bullet) still determines branch/loop roles; the checkbox flag is inert data and does not change execution. A task list item in an executable position still requires its condition header like any other item. - Code spans carry delimiter tick counts. Inline
code { ticks }records the number of backticks, and the source span includes them; the literal is the inner payload. This lets the frontend map expression payload spans back to exact source bytes for diagnostics. "Ticks" is not user-visible program meaning. - Strikethrough is inert/lazy.
strikethroughis an inert wrapper: it is never traversed for execution, its content is prose, and astrong/emphinside it does not execute. It is not an error to write it; the language simply does not use formatting as a control-flow mechanism here. (A code span or math inside it is likewise inert.) - Unmodelled nodes are preserved as
other { tag }with children and raw literal. They are inert but never silently dropped frominspect --ast. - Literal payloads are kept on inert leaves:
text,code,code_block,math,math_block,html_block,html_inline,front_matter. - Other frozen normalizations:
**x**/__x__and*x*/_x_are the same node kinds (strong/emph);- ordered delimiters
1./1)and bullets-/*/+are the samelist/itemstructure; role depends onordered, not marker; - tight vs loose lists only affect rendering (
tightis recorded, inert); link/imagekeepurl/title;tablekeeps column alignments,table_rowkeepsheader;headingkeepslevel/setext;- a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the original file;
- front matter is recognized only at the document start (after BOM), with
---opening and---/...closing; a later---is a thematic break or Setext underline.
These rules are part of profile id marqdo.md.v1. Changing them (or bumping
the comrak version/options) requires a profile id bump, updated goldens and this
ADR being superseded.
Alternatives
- Distinguish Setext vs ATX headings semantically. Rejected: identical AST
structure outside the marker must have identical meaning (
## Hand Setext H2 are the same function). Keepingsetextonly for inspection avoids silently diverging programs when a document changes style. - Introduce a separate
task_itemnode kind. Rejected: it would require every list-role check to special-case two item kinds and could be used to hide a control structure behind a checkbox. Normalizing toitem { task }keeps list structure authoritative. - Strip the tick information and store only the literal. Rejected: payload spans returned by the expression parser are relative to the concatenated payload; without delimiter lengths, error spans cannot be mapped back to file bytes precisely.
- Traverse strikethrough content for execution. Rejected: it is a lazy region by profile; executing inside it would make formatting semantics surprising ("struck-through code runs"). It would also create a second way to hide executable nodes.
- Drop unknown parser nodes. Rejected:
inspect --astmust show what the parser produced; dropping syntax hides profile drift. - Preserve raw source and re-parse with a custom parser. Rejected by the project's core rule: one standard Markdown parser, no second Markdown AST.
Markdown AST shape
Frozen JSON shapes (inspect --ast --json):
{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
"children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
{"kind":"list","ordered":false,"tight":true,"span":{...},
"children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}
Attribute order in the JSON is fixed by the serializer; goldens are regenerated only through the explicit update workflow.
5.1 Known interaction to keep in mind
Inside a function body, a **strong** (or an ordered/bullet list) nested under
strikethrough is inert and never executes; because the mapper does not
traverse strikethrough, it also produces no warning. Outside function bodies,
the markdown.inert_executable scan walks a top-level paragraph recursively,
so such a nested node may still produce the inert warning even though it would
not execute if the paragraph were inside a function. This is consistent with
"formatting never hides execution", but an author who intends execution must
not wrap it in strikethrough. Warnings for lazy-region content inside functions
are a possible future refinement; no behavior change is planned without an ADR.
Compatibility
- Profile id
marqdo.md.v1is frozen with comrak=1.0.0and the exact extension flags; the unit testcrates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozenasserts the option table (tables, tasklist, strikethrough, math dollars, front matter; all other extensions disabled). - Adding an AST field is a JSON output change for
inspect --ast; it needs a goldens update and a profile note. Changing a semantic rule (e.g. executing inside strikethrough) is a language change and requires an ADR. - No package/ABI/wire impact.
Security
- The lazy-region rules ensure inline formatting inside headings, tables,
links, code, math, HTML and front matter never executes — a context-aware
visitor is used (
crates/marqdo-frontend/src/map.rsonly descends from paragraphs intostrong/emph, and ignores every other inline node). - Strikethrough cannot be used to smuggle executable content; it is inert.
- Normalizing task items to
itemprevents a checkbox from hiding a branch/loop header; the condition-header check still applies. - No new capabilities or host calls.
Testing
crates/marqdo-markdown/src/profile.rs:profile_options_are_frozen— exact extension/parse option table;setext_headings_are_heading_nodes— Setext producesheading { setext: true };tables_and_task_lists_are_supported— task items carrytask: Some(_);- span/BOM tests.
- Frontend mapping tests use code spans as conditions and payloads, exercising
ticks/payload span mapping. spec/markdown-profile.md§5/§6 is the normative JSON shape; AST goldens undercrates/cli/tests(fixtures workstream) regenerate only with the documented update command.- Conformance fixtures are expected to include
**a**vs__a__, Setext vs ATX, task/list variants and strikethrough inertness.
Migration & rollout
No migration for documents: Setext and ATX were always the same function under
this profile. When comrak is upgraded or the extension set changes, bump the
profile id, regenerate AST goldens, re-run all specs/ADRs that reference node
shapes, and document the change here (superseding this ADR if a rule changes).
Code-span ticks and task are additive inspection fields; keep them optional
in consumers that parse the JSON.