Files
marqdo-cli/docs/adr/0005-ast-normalization.md
T
cflmy 49835424ee docs(spec): adopt RFC-LANG-0002 unified language semantics
Replace the Strong/Emph/List/Quote mapping in place with the single
@exec/@knowledge heading-mode model, unified payload carriers and keyword
control flow.

- add the authoritative RFC, ADR-0052 and a historical ADR index
- rewrite spec/mapping.md and spec/expression.ebnf as the normative
  mapping/statement grammar; update evaluation, references, math,
  foreign, package-format, plugin-abi, security and schema notes
- rewrite README, AGENTS, COMPATIBILITY and CHANGELOG for the breaking
  language change; update the architecture index/overview and active docs
- mark superseded planning docs, compute RFCs, evidence and syntax-role
  ADRs as historical/superseded without rewriting their decisions
- add migration guide, per-file inventory and dev-only audit scripts

The frontend/stdlib migration is completed in the following commits; this
commit contains documentation only.
2026-11-21 12:24:19 +08:00

9.2 KiB

ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)

Historical mapping note. The profile/AST normalization rules in this ADR remain current. Its role claims — that ordered/bullet list structure "determines branch/loop roles" and that strong/emph carry execute/return roles — were superseded by RFC-LANG-0002 / ADR-0052 and spec/mapping.md: lists and block quotes are presentation containers, and Strong, Emph and inline Code are interchangeable one-statement payload carriers. Strikethrough and the other inert regions stay inert.

Status: Accepted Date: 2026-11-20

Context

The language derives program meaning from the canonical AST, never from source spelling (spec/markdown-profile.md, spec/mapping.md §1). The parser (comrak =1.0.0, frozen options) normalizes several syntaxes before Marqdo sees them, and a few normalizations are lossy:

  • # H and Setext H\n=== both become a heading node; the source spelling is lost unless recorded;
  • GFM task list items (- [x] text) produce a task-specific item node;
  • inline code spans are stored as a literal without the backtick delimiters, but source spans include them;
  • ~~strikethrough~~ is enabled by the profile and wraps arbitrary inlines.

The language needs a single, documented answer per question, because node type and structural position decide executable roles.

Decision

The profile (marqdo.md.v1) and the canonical AST (crates/marqdo-markdown/src/ast.rs, src/profile.rs) freeze these rules:

  1. Setext headings are headings. # name and a Setext level-1/2 underline both produce heading { level, setext }; setext is inspection-only and never affects semantics. ## name and a Setext level-2 heading are both functions. The profile keeps parse.ignore_setext = false.
  2. Task-list items are item nodes with a task flag. item { task: Some(true|false) } for GFM task items, task: None otherwise. Node type stays item, so list structure (ordered vs bullet) still determines branch/loop roles; the checkbox flag is inert data and does not change execution. A task list item in an executable position still requires its condition header like any other item.
  3. Code spans carry delimiter tick counts. Inline code { ticks } records the number of backticks, and the source span includes them; the literal is the inner payload. This lets the frontend map expression payload spans back to exact source bytes for diagnostics. "Ticks" is not user-visible program meaning.
  4. Strikethrough is inert/lazy. strikethrough is an inert wrapper: it is never traversed for execution, its content is prose, and a strong/emph inside it does not execute. It is not an error to write it; the language simply does not use formatting as a control-flow mechanism here. (A code span or math inside it is likewise inert.)
  5. Unmodelled nodes are preserved as other { tag } with children and raw literal. They are inert but never silently dropped from inspect --ast.
  6. Literal payloads are kept on inert leaves: text, code, code_block, math, math_block, html_block, html_inline, front_matter.
  7. Other frozen normalizations:
    • **x**/__x__ and *x*/_x_ are the same node kinds (strong/emph);
    • ordered delimiters 1./1) and bullets -/*/+ are the same list/item structure; role depends on ordered, not marker;
    • tight vs loose lists only affect rendering (tight is recorded, inert);
    • link/image keep url/title; table keeps column alignments, table_row keeps header; heading keeps level/setext;
    • a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the original file;
    • front matter is recognized only at the document start (after BOM), with --- opening and ---/... closing; a later --- is a thematic break or Setext underline.

These rules are part of profile id marqdo.md.v1. Changing them (or bumping the comrak version/options) requires a profile id bump, updated goldens and this ADR being superseded.

Alternatives

  1. Distinguish Setext vs ATX headings semantically. Rejected: identical AST structure outside the marker must have identical meaning (## H and Setext H2 are the same function). Keeping setext only for inspection avoids silently diverging programs when a document changes style.
  2. Introduce a separate task_item node kind. Rejected: it would require every list-role check to special-case two item kinds and could be used to hide a control structure behind a checkbox. Normalizing to item { task } keeps list structure authoritative.
  3. Strip the tick information and store only the literal. Rejected: payload spans returned by the expression parser are relative to the concatenated payload; without delimiter lengths, error spans cannot be mapped back to file bytes precisely.
  4. Traverse strikethrough content for execution. Rejected: it is a lazy region by profile; executing inside it would make formatting semantics surprising ("struck-through code runs"). It would also create a second way to hide executable nodes.
  5. Drop unknown parser nodes. Rejected: inspect --ast must show what the parser produced; dropping syntax hides profile drift.
  6. Preserve raw source and re-parse with a custom parser. Rejected by the project's core rule: one standard Markdown parser, no second Markdown AST.

Markdown AST shape

Frozen JSON shapes (inspect --ast --json):

{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
 "children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
{"kind":"list","ordered":false,"tight":true,"span":{...},
 "children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}

Attribute order in the JSON is fixed by the serializer; goldens are regenerated only through the explicit update workflow.

5.1 Known interaction to keep in mind

Inside a function body, a **strong** (or an ordered/bullet list) nested under strikethrough is inert and never executes; because the mapper does not traverse strikethrough, it also produces no warning. Outside function bodies, the markdown.inert_executable scan walks a top-level paragraph recursively, so such a nested node may still produce the inert warning even though it would not execute if the paragraph were inside a function. This is consistent with "formatting never hides execution", but an author who intends execution must not wrap it in strikethrough. Warnings for lazy-region content inside functions are a possible future refinement; no behavior change is planned without an ADR.

Compatibility

  • Profile id marqdo.md.v1 is frozen with comrak =1.0.0 and the exact extension flags; the unit test crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen asserts the option table (tables, tasklist, strikethrough, math dollars, front matter; all other extensions disabled).
  • Adding an AST field is a JSON output change for inspect --ast; it needs a goldens update and a profile note. Changing a semantic rule (e.g. executing inside strikethrough) is a language change and requires an ADR.
  • No package/ABI/wire impact.

Security

  • The lazy-region rules ensure inline formatting inside headings, tables, links, code, math, HTML and front matter never executes — a context-aware visitor is used (crates/marqdo-frontend/src/map.rs only descends from paragraphs into strong/emph, and ignores every other inline node).
  • Strikethrough cannot be used to smuggle executable content; it is inert.
  • Normalizing task items to item prevents a checkbox from hiding a branch/loop header; the condition-header check still applies.
  • No new capabilities or host calls.

Testing

  • crates/marqdo-markdown/src/profile.rs:
    • profile_options_are_frozen — exact extension/parse option table;
    • setext_headings_are_heading_nodes — Setext produces heading { setext: true };
    • tables_and_task_lists_are_supported — task items carry task: Some(_);
    • span/BOM tests.
  • Frontend mapping tests use code spans as conditions and payloads, exercising ticks/payload span mapping.
  • spec/markdown-profile.md §5/§6 is the normative JSON shape; AST goldens under crates/cli/tests (fixtures workstream) regenerate only with the documented update command.
  • Conformance fixtures are expected to include **a** vs __a__, Setext vs ATX, task/list variants and strikethrough inertness.

Migration & rollout

No migration for documents: Setext and ATX were always the same function under this profile. When comrak is upgraded or the extension set changes, bump the profile id, regenerate AST goldens, re-run all specs/ADRs that reference node shapes, and document the change here (superseding this ADR if a rule changes). Code-span ticks and task are additive inspection fields; keep them optional in consumers that parse the JSON.