Files
marqdo-cli/docs/adr/0005-ast-normalization.md
cflmy 49835424ee docs(spec): adopt RFC-LANG-0002 unified language semantics
Replace the Strong/Emph/List/Quote mapping in place with the single
@exec/@knowledge heading-mode model, unified payload carriers and keyword
control flow.

- add the authoritative RFC, ADR-0052 and a historical ADR index
- rewrite spec/mapping.md and spec/expression.ebnf as the normative
  mapping/statement grammar; update evaluation, references, math,
  foreign, package-format, plugin-abi, security and schema notes
- rewrite README, AGENTS, COMPATIBILITY and CHANGELOG for the breaking
  language change; update the architecture index/overview and active docs
- mark superseded planning docs, compute RFCs, evidence and syntax-role
  ADRs as historical/superseded without rewriting their decisions
- add migration guide, per-file inventory and dev-only audit scripts

The frontend/stdlib migration is completed in the following commits; this
commit contains documentation only.
2026-11-21 12:24:19 +08:00

186 lines
9.2 KiB
Markdown

# ADR-0005: AST normalization (Setext headings, task items, code-span ticks, strikethrough)
> **Historical mapping note.** The profile/AST normalization rules in this ADR
> remain current. Its role claims — that ordered/bullet list structure
> "determines branch/loop roles" and that `strong`/`emph` carry execute/return
> roles — were superseded by
> [RFC-LANG-0002](../architecture/RFC-LANG-0002-unified-semantics.md) /
> [ADR-0052](0052-unified-markdown-semantics.md) and
> [`spec/mapping.md`](../../spec/mapping.md): lists and block quotes are
> presentation containers, and `Strong`, `Emph` and inline `Code` are
> interchangeable one-statement payload carriers. Strikethrough and the other
> inert regions stay inert.
Status: **Accepted**
Date: 2026-11-20
## Context
The language derives program meaning from the **canonical AST**, never from
source spelling (`spec/markdown-profile.md`, `spec/mapping.md` §1). The parser
(`comrak =1.0.0`, frozen options) normalizes several syntaxes before Marqdo
sees them, and a few normalizations are lossy:
* `# H` and Setext `H\n===` both become a heading node; the source spelling is
lost unless recorded;
* GFM task list items (`- [x] text`) produce a task-specific item node;
* inline code spans are stored as a literal without the backtick delimiters,
but source spans include them;
* `~~strikethrough~~` is enabled by the profile and wraps arbitrary inlines.
The language needs a single, documented answer per question, because node type
and structural position decide executable roles.
## Decision
The profile (`marqdo.md.v1`) and the canonical AST
(`crates/marqdo-markdown/src/ast.rs`, `src/profile.rs`) freeze these rules:
1. **Setext headings are headings.** `# name` and a Setext level-1/2 underline
both produce `heading { level, setext }`; `setext` is inspection-only and
**never affects semantics**. `## name` and a Setext level-2 heading are both
functions. The profile keeps `parse.ignore_setext = false`.
2. **Task-list items are `item` nodes with a task flag.**
`item { task: Some(true|false) }` for GFM task items, `task: None`
otherwise. Node type stays `item`, so list structure (ordered vs bullet)
still determines branch/loop roles; the checkbox flag is inert data and does
not change execution. A task list item in an executable position still
requires its condition header like any other item.
3. **Code spans carry delimiter tick counts.** Inline `code { ticks }` records
the number of backticks, and the source span includes them; the literal is
the inner payload. This lets the frontend map expression payload spans back
to exact source bytes for diagnostics. "Ticks" is not user-visible program
meaning.
4. **Strikethrough is inert/lazy.** `strikethrough` is an inert wrapper: it is
never traversed for execution, its content is prose, and a `strong`/`emph`
inside it does not execute. It is not an error to write it; the language
simply does not use formatting as a control-flow mechanism here. (A code
span or math inside it is likewise inert.)
5. **Unmodelled nodes are preserved** as `other { tag }` with children and raw
literal. They are inert but never silently dropped from `inspect --ast`.
6. **Literal payloads are kept** on inert leaves: `text`, `code`, `code_block`,
`math`, `math_block`, `html_block`, `html_inline`, `front_matter`.
7. **Other frozen normalizations:**
* `**x**`/`__x__` and `*x*`/`_x_` are the same node kinds (`strong`/`emph`);
* ordered delimiters `1.`/`1)` and bullets `-`/`*`/`+` are the same
`list`/`item` structure; role depends on `ordered`, not marker;
* tight vs loose lists only affect rendering (`tight` is recorded, inert);
* `link`/`image` keep `url`/`title`; `table` keeps column alignments,
`table_row` keeps `header`; `heading` keeps `level`/`setext`;
* a UTF-8 BOM at offset 0 is stripped and all spans are re-based onto the
original file;
* front matter is recognized only at the document start (after BOM), with
`---` opening and `---`/`...` closing; a later `---` is a thematic break
or Setext underline.
These rules are part of profile id `marqdo.md.v1`. Changing them (or bumping
the comrak version/options) requires a profile id bump, updated goldens and this
ADR being superseded.
## Alternatives
1. **Distinguish Setext vs ATX headings semantically.** Rejected: identical AST
structure outside the marker must have identical meaning (`## H` and Setext
H2 are the same function). Keeping `setext` only for inspection avoids
silently diverging programs when a document changes style.
2. **Introduce a separate `task_item` node kind.** Rejected: it would require
every list-role check to special-case two item kinds and could be used to
hide a control structure behind a checkbox. Normalizing to `item { task }`
keeps list structure authoritative.
3. **Strip the tick information and store only the literal.** Rejected: payload
spans returned by the expression parser are relative to the concatenated
payload; without delimiter lengths, error spans cannot be mapped back to
file bytes precisely.
4. **Traverse strikethrough content for execution.** Rejected: it is a lazy
region by profile; executing inside it would make formatting semantics
surprising ("struck-through code runs"). It would also create a second way
to hide executable nodes.
5. **Drop unknown parser nodes.** Rejected: `inspect --ast` must show what the
parser produced; dropping syntax hides profile drift.
6. **Preserve raw source and re-parse with a custom parser.** Rejected by the
project's core rule: one standard Markdown parser, no second Markdown AST.
## Markdown AST shape
Frozen JSON shapes (`inspect --ast --json`):
```json
{"kind":"heading","level":2,"setext":false,"span":{"start":0,"end":7},
"children":[{"kind":"text","span":{"start":3,"end":7},"literal":"阶乘"}]}
```
```json
{"kind":"list","ordered":false,"tight":true,"span":{...},
"children":[{"kind":"item","task":true,"span":{...},"children":[...]}]}
```
```json
{"kind":"code","ticks":2,"span":{"start":10,"end":16},"literal":"a * b"}
```
```json
{"kind":"strikethrough","span":{...},"children":[{"kind":"text",...}]}
```
Attribute order in the JSON is fixed by the serializer; goldens are regenerated
only through the explicit update workflow.
### 5.1 Known interaction to keep in mind
Inside a function body, a `**strong**` (or an ordered/bullet list) nested under
`strikethrough` is inert and never executes; because the mapper does not
traverse strikethrough, it also produces no warning. Outside function bodies,
the `markdown.inert_executable` scan walks a top-level paragraph recursively,
so such a nested node may still produce the inert warning even though it would
not execute if the paragraph were inside a function. This is consistent with
"formatting never hides execution", but an author who intends execution must
not wrap it in strikethrough. Warnings for lazy-region content inside functions
are a possible future refinement; no behavior change is planned without an ADR.
## Compatibility
* Profile id `marqdo.md.v1` is frozen with comrak `=1.0.0` and the exact
extension flags; the unit test
`crates/marqdo-markdown/src/profile.rs::tests::profile_options_are_frozen`
asserts the option table (tables, tasklist, strikethrough, math dollars,
front matter; all other extensions disabled).
* Adding an AST field is a JSON output change for `inspect --ast`; it needs a
goldens update and a profile note. Changing a semantic rule (e.g. executing
inside strikethrough) is a language change and requires an ADR.
* No package/ABI/wire impact.
## Security
* The lazy-region rules ensure inline formatting inside headings, tables,
links, code, math, HTML and front matter never executes — a context-aware
visitor is used (`crates/marqdo-frontend/src/map.rs` only descends from
paragraphs into `strong`/`emph`, and ignores every other inline node).
* Strikethrough cannot be used to smuggle executable content; it is inert.
* Normalizing task items to `item` prevents a checkbox from hiding a
branch/loop header; the condition-header check still applies.
* No new capabilities or host calls.
## Testing
* `crates/marqdo-markdown/src/profile.rs`:
* `profile_options_are_frozen` — exact extension/parse option table;
* `setext_headings_are_heading_nodes` — Setext produces `heading { setext: true }`;
* `tables_and_task_lists_are_supported` — task items carry `task: Some(_)`;
* span/BOM tests.
* Frontend mapping tests use code spans as conditions and payloads, exercising
`ticks`/payload span mapping.
* `spec/markdown-profile.md` §5/§6 is the normative JSON shape; AST goldens
under `crates/cli/tests` (fixtures workstream) regenerate only with the
documented update command.
* Conformance fixtures are expected to include `**a**` vs `__a__`, Setext vs
ATX, task/list variants and strikethrough inertness.
## Migration & rollout
No migration for documents: Setext and ATX were always the same function under
this profile. When comrak is upgraded or the extension set changes, bump the
profile id, regenerate AST goldens, re-run all specs/ADRs that reference node
shapes, and document the change here (superseding this ADR if a rule changes).
Code-span `ticks` and `task` are additive inspection fields; keep them optional
in consumers that parse the JSON.