spec(P0): freeze markdown profile, AST mapping, expression grammar and evaluation semantics

Add the formal specifications required before implementation:

- spec/markdown-profile.md: parser version, exact option table, byte spans,
  BOM policy, front matter rule, AST JSON view, structural equivalence.
- spec/mapping.md: normative AST-to-program mapping (heading scopes, lazy
  regions, strong/emph/list/blockquote roles), strict payload extraction and
  the hard-error checklist. Invalid executable shapes never degrade to prose.
- spec/expression.ebnf: the small independent expression/statement grammar
  (Unicode identifiers, arbitrary precision integers, precedence, named
  arguments, no comments, no truthiness).
- spec/evaluation.md: value model, exact integer semantics, scopes, order,
  parameter inference via definite-assignment data flow, budgets and the v1
  builtin table.
This commit is contained in:
cflmy committed 2026-11-20 10:26:15 +08:00
1 parent 10c0c0a013
commit 55a2b2d82d
4 files changed
+573

No files matched your search

+130
View File
@@ -0,0 +1,130 @@
# Marqdo Evaluation Semantics v1
Status: **frozen for M0.5**. Changes to observable behaviour require an ADR.
## 1. Values
| Type | Meaning |
|---|---|
| `Unit` | no result; a function that falls through returns `Unit` |
| `None` | explicit empty value; distinct from `Unit` |
| `Bool` | `true` / `false`; the only type usable in conditions |
| `Int` | arbitrary precision integer (`num-bigint`) |
| `Float` | IEEE-754 binary64 |
| `Text` | Unicode string |
| `List` | ordered values |
| `Map` | string-keyed record with deterministic (sorted) key order |
| `Table` | header names + rows of values (from GFM tables) |
| `Formula` | inert parsed LaTeX + provenance; math operations are explicit |
| `Code` | inert code fence + provenance; execution is explicit |
| `FunctionRef` | reference to a named function in a module |
| `PluginHandle` | opaque host-owned plugin export handle |
There is no truthiness: `if`-like conditions accept `Bool` only. There are no
implicit string↔number conversions; conversion builtins must be called.
## 2. Numeric semantics
* `Int` arithmetic (`+ - * // % **`) is exact. `/` always produces `Float`.
* Mixed `Int`/`Float` arithmetic yields `Float`; the conversion of an `Int`
larger than 2^53 may lose precision and this is documented, not silently
exported as a rounded `Int`.
* Comparisons between `Int` and `Float` are **exact** (the float is treated as
the exact binary rational it is); no lossy comparison is performed.
* `//` and `%` require `Int` operands. Floor division: `a // b` is the
mathematical floor; `a % b == a - b * (a // b)` (sign follows divisor).
* Division/modulo by zero is `runtime.division_by_zero`.
* `+` also concatenates two `Text` values. No other operator is overloaded.
* `**` on `Int` uses exact exponentiation; a negative exponent is a type error
(use `/` or `Float` operands for fractional results). Exponents above a
configurable budget are `resource.exhausted`.
* Floats may be `NaN`/`±Infinity` only as results of operations; they are
tagged explicitly on the wire. NaN compares unequal to everything.
## 3. Evaluation order and scopes
* Blocks execute in Markdown AST document order; executable inline nodes
execute in source order within a paragraph.
* A function call evaluates its arguments in source order, exactly once each,
then binds them to fresh parameter slots in an isolated frame. The callee
cannot see the caller's locals (no dynamic scoping); imported bindings are
read-only.
* Assignment binds/updates a local slot in the current frame. It never writes
an outer frame. There is no module-level assignment in v1: assignment
outside a function body is an error.
* `and`/`or` short-circuit and require `Bool` operands.
* `Return` (an `emph` node) unwinds through branch/loop bodies to the
enclosing function frame.
* `Branch` checks conditions in AST order and executes the first `true` branch
once. `Loop` re-evaluates its condition before each iteration.
* Imported modules only register definitions for the importing document; they
have no initialization side effects and their `main` is never started.
* Function calls recurse; recursion depth is bounded by the stack budget.
Mutual recursion is allowed.
## 4. Errors
All failures are structured. Compile-time problems produce diagnostics
(`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`, …). Runtime
problems produce `RuntimeError { category, span, message, frames, cause }`
with a stable category code. Runtime errors stop the current function and
propagate with a call stack. A host/foreign/plugin failure never panics the
interpreter; it becomes a structured error.
## 5. Parameter inference (static, name based)
Parameters are discovered by definite-assignment data-flow analysis over the
function's control-flow graph, never from prose:
1. Build the CFG of the function body in AST order, with branch and loop
edges. Loop bodies may execute zero times.
2. Compute the set of variables *definitely assigned* on every path from the
function entry.
3. Every read of a variable that is not definitely assigned at that point and
that has no assignment anywhere in the function is a **candidate
parameter** (sorted by canonical name).
4. If a variable is assigned on some paths but read before that assignment on
others, its identity is ambiguous: emit `symbol.uninitialized` (S-04)
instead of inventing a parameter.
5. Exclude builtins, imported aliases and reserved names from inference.
6. A read that is not explained by a parameter, a local, a builtin or an
import is `symbol.unknown` with a hint (typo protection).
Consequences: `**结果 = n * 阶乘(n=n-1)**` infers `n`; `**n = 3**` followed by
reads makes `n` a local. Inferred parameters are named arguments only; an
explicit signature paragraph allows positional arguments.
## 6. Resource budgets
Every run has explicit limits (all configurable, all reported as
`resource.exhausted`): evaluation fuel, call stack depth, total output bytes,
wall-clock time, and host process count. Loops consume fuel per iteration and
expressions per node so that a hostile document cannot hang the CLI. Strict
mode additionally enforces limits at the OS level where available; if the
platform cannot provide that, strict mode refuses to run (see `security.md`).
## 7. Determinism
For a fixed source, mode, capabilities and budget, evaluation is
deterministic. Stdout/stderr from host operations are separated: program
output goes to stdout, diagnostics/traces go to stderr, and `--json` output
never mixes with program output.
## 8. Builtins v1
Builtins are part of the language surface and are capability-checked at the
host boundary. Names are stable; adding one requires a spec update.
| Name | Signature | Notes |
|---|---|---|
| `打印` / `print` | `(x) -> Unit` | writes `Value::display()` + newline to stdout |
| `长度` / `len` | `(x) -> Int` | Text scalar count, List/Map/Table size |
| `类型` / `type` | `(x) -> Text` | value type name |
| `文本` / `str` | `(x) -> Text` | explicit conversion |
| `整数` / `int` | `(x) -> Int` | from Int/Float(if integral)/Text(decimal) |
| `小数` / `float` | `(x) -> Float` | from Int/Float/Text |
| `表` / `table` | `(header: List, rows: List) -> Table` | validates row widths |
| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address |
| `math.parse/eval/simplify/differentiate` | see `spec/math.md` | P5 |
| `foreign.run/call` | see `spec/foreign.md` | P5, capability-gated |
| `fs.read/write` | see `spec/security.md` | capability-gated |
+85
View File
@@ -0,0 +1,85 @@
(* Marqdo Expression Grammar v1
Status: frozen for M0.5. This is the only grammar for executable payloads.
It is a small independent language: it is NOT Markdown and NOT translated
to Python. Markdown AST decides where a payload exists; this grammar
decides what it means.
Terminals are described in prose where Unicode categories matter.
*)
program = [ statement ] , EOF ;
statement = assignment | expression ;
assignment = identifier , "=" , expression ;
(* "=" is a single "=" token; "==" is a distinct token. Assignment targets
are bare identifiers only: no member or index assignment in v1. *)
expression = or_expr ;
or_expr = and_expr , { "or" , and_expr } ;
and_expr = compare_expr , { "and" , compare_expr } ;
compare_expr = add_expr , [ compare_op , add_expr ] ;
(* non-associative: "a < b < c" is a parse error; write
"a < b and b < c" instead. *)
compare_op = "==" | "!=" | "<" | "<=" | ">" | ">=" ;
add_expr = mul_expr , { ( "+" | "-" ) , mul_expr } ;
mul_expr = unary_expr , { ( "*" | "/" | "//" | "%" ) , unary_expr } ;
unary_expr = ( "not" | "-" | "+" ) , unary_expr | power_expr ;
power_expr = postfix_expr , [ "**" , unary_expr ] ;
(* right associative: 2 ** 3 ** 2 == 2 ** (3 ** 2);
-x ** 2 == -(x ** 2); 2 ** -1 is valid *)
postfix_expr = primary , { call_suffix | member_suffix | index_suffix } ;
call_suffix = "(" , [ argument_list ] , ")" ;
argument_list = argument , { "," , argument } , [ "," ] ;
argument = [ identifier , "=" ] , expression ;
(* positional arguments must all precede named arguments;
duplicate argument names are errors *)
member_suffix = "." , identifier ;
index_suffix = "[" , expression , "]" ;
primary = int_literal
| float_literal
| text_literal
| "true" | "false" | "None"
| identifier
| "(" , expression , ")"
| list_literal
| record_literal ;
list_literal = "[" , [ expression , { "," , expression } , [ "," ] ] , "]" ;
record_literal = "{" , [ record_entry , { "," , record_entry } , [ "," ] ] , "}" ;
record_entry = ( identifier | text_literal ) , ":" , expression ;
(* Lexical rules *)
identifier = xid_start , { xid_continue } ;
(* Unicode XID per UAX #31: Chinese, Latin, etc. are valid.
Keywords are reserved and cannot be identifiers:
true, false, None, and, or, not. *)
int_literal = digit , { digit } ;
(* arbitrary precision decimal integer; no sign here (unary
minus handles negatives); no separators; no hex/octal. *)
float_literal = digit , { digit } , "." , digit , { digit } ,
[ ( "e" | "E" ) , [ "+" | "-" ] , digit , { digit } ] ;
(* requires digits on both sides of the dot; ".5" and "1."
are invalid. Floats are IEEE-754 binary64. *)
text_literal = '"' , { text_char | escape } , '"' ;
escape = "\\" , ( '"' | "\\" | "n" | "t" | "r" | "u{" , hexdigit , { hexdigit } , "}" ) ;
text_char = ? any Unicode scalar value except '"' and "\\" ? ;
(* Whitespace: any Unicode whitespace separates tokens. Newlines inside a
code-span payload are whitespace. There are no comments in v1. *)
(* Notes on the Markdown boundary *)
(* - A "**" inside a payload is the power operator only when it survived
Markdown parsing inside a code span or text that is not emphasis.
Authors should put multiplication/power expressions in code spans when
Markdown emphasis would otherwise consume the asterisks. *)
(* - Payload spans returned by the parser are relative to the concatenated
payload; the frontend maps them back through the text/code segments to
file spans. *)
(* - Maximum nesting depth is 128; exceeding it is an expr.parse error. *)
+235
View File
@@ -0,0 +1,235 @@
# Marqdo AST → Program Mapping v1
Status: **frozen for M0.5**. This file is the normative mapping from Markdown
AST structure to program meaning. It is derived directly from
`docs/DEVELOPMENT_BLUEPRINT.md` sections 1.2–1.4 and 3.2.
## 1. The two rules that override everything
1. **Node type and structural position decide the role.** A `strong` node in an
executable position is an execution statement and its payload must parse as
a `Statement`. It is never "prose that happens to look like code".
2. **Invalid executable shapes are hard errors.** If a node occupies an
executable role but its shape is invalid, compilation fails with a
`markdown.invalid_construct` diagnostic. There is no fallback to prose.
Prose never causes an action. Natural-language words (“if”, “return”, “若”,
“执行”) are never interpreted as keywords.
## 2. Document regions and heading scopes
A document root is a module. Heading nodes create addressable scopes:
| AST | Role | Requirements |
|---|---|---|
| `document` | module | exactly one root |
| `heading(level=1)` | namespace / object domain | non-empty canonical name |
| `heading(level=2)` | function | non-empty canonical name, unique in its scope |
| `heading(level>=3)` | section (source location) | does not create a function |
Scope rules:
* The range of a heading extends until the next heading of the same or a
lower level (i.e. same or smaller number), or the end of the document.
* A level-2 heading without a preceding level-1 heading belongs to the module
root. Nested headings inside a section remain part of the enclosing function.
* Headings inside container blocks (block quotes, list items) or inside fenced
code blocks never create functions.
* A function's canonical name is the normalised text of its heading children:
NFC, trimmed, internal whitespace collapsed to one space. Empty names are
errors. Duplicate canonical names in the same scope are errors. GitHub-style
anchor slugs are derived for navigation but are **not** the identity.
* `main` at module root is the default entry point for `marqdo run`; an
imported module's `main` is not run automatically.
## 3. Lazy (inert) regions
The following nodes are inert regardless of where they appear. Their inline
content is never executed:
* heading text (headings scope, they do not run)
* `link` labels and `image` alt text
* `table` cells (tables are document values)
* `code_block` and inline `code` outside designated payload positions
* `math` / `math_block`
* `html_block` / `html_inline`
* front matter
* text and other inline nodes inside prose paragraphs
Inline formatting inside those regions (e.g. `strong` inside a heading) does
not execute. The implementation must use a context-aware visitor, not a blind
whole-tree walk.
## 4. Executable roles in function bodies
Inside a function (`heading(level=2)` range) the following roles apply,
including inside branch and loop bodies:
| AST node | Role | Shape requirement | On invalid shape |
|---|---|---|---|
| `strong` | execution statement | strict payload parses as `Statement` | hard error |
| `emph` | return statement | strict payload parses as `Expression` | hard error |
| `list(ordered=true)` | branch | each item starts with a condition header | hard error |
| `list(ordered=false)` | loop | each item starts with a loop condition header | hard error |
| `block_quote` | call | one paragraph forming a complete call | hard error |
| `paragraph` | prose + inline execution | inline executable nodes run in order | n/a |
| everything else | inert document value | — | type/constraint errors only if used |
Outside functions (module top-level prose) the same executable nodes are inert
document content (S-03 in the blueprint: executable lists/strong/emph only run
inside a function body). The single exception is `main` itself, which is a
function.
### 4.1 Strict payload extraction
For `strong`/`emph`, the payload is the concatenation of the direct `text` and
`code` child literals, exactly as written. Nested `strong`, `emph`, `link`,
`strikethrough`, line breaks, math, HTML or any other inline node is an error:
one AST node must never be executed twice or hide control content.
`**x = 1**` and `**`x = 1`**` have the same payload `x = 1`. `**a * b**`
parses as text in most cases; when Markdown emphasis and the multiplication
operator conflict, the author should use a code span for the payload:
`` **`a * b`** ``. There is no text-based second parse.
### 4.2 Paragraph and document order
Executable inline nodes in a paragraph run in source order. Blocks run in AST
document order. A `return` short-circuits the remaining nodes of the enclosing
function, including nodes inside branches and loops.
### 4.3 Branch (`ordered list`)
Each item must:
1. contain at least one block;
2. have a `paragraph` as its **first** block;
3. have an inline `code` node as the **first inline node** of that paragraph;
its literal is the condition expression;
4. any remaining inline content in that first paragraph is prose.
The item's remaining blocks form the branch body. Items are tested in AST
order; the first condition evaluating to `true` is executed once. No match
means `Unit` and execution continues after the list. Conditions must be
`Bool`; they must be pure (side effects in conditions are a static error when
detectable and a runtime capability concern otherwise).
### 4.4 Loop (`bullet list`)
Each item is an independent `while` loop. Item structure matches the branch
header rule: the first paragraph's first inline node must be a `code` span
(condition), and the remaining blocks are the loop body. The condition is
re-evaluated before every iteration and must be `Bool`. Items execute in
order, so a bullet list is a sequence of loops, one per item.
Iteration over collections is expressed with stdlib `map`/`filter`/`fold`;
a dedicated `forEach` header is intentionally not part of v1.
### 4.5 Call (`block quote`)
A block quote is exactly one `paragraph` and nothing else. The paragraph is
parsed with the call grammar:
* plain payload: `打印("你好")`, `math.eval(ref("#公式"), {"r": r}, target="A")`
— the strict payload must parse as a complete `CallExpr`;
* link form: `[计算面积](./math.mq.md#计算面积)(r=3)` — the first inline node
must be a `link`, its destination identifies the target function, and the
remaining `text`/`code` payload must be a parenthesised argument list.
Anything else (a second paragraph, plain prose, a nested quote, an expression
that is not a call) is a hard error. Two calls require two block quote nodes.
`> [说明](#目标)` without an argument list is not accepted in v1.
The call result is discarded by the statement form; use `strong` to bind it
(`**面积 = 计算面积(r=3)**`).
## 5. Calls, returns and values
* A call is `函数(...)`; zero-argument calls still write `()`.
* Positional arguments are only allowed for functions with an explicit
signature. Named arguments can be given in any order for any function.
* Arguments are evaluated in source order exactly once before binding.
* `emph` returns the value of its expression from the enclosing function
(propagating through branch/loop bodies), never from a block.
* `*表达式*` used in a branch is a return from the whole function.
* A function that ends without a return evaluates to `Unit`. `None` is an
explicit empty value and is distinct from `Unit`.
* Return at module top level (outside any function) is an error.
## 6. Function signatures and parameter inference
* Optional explicit signature: the first `paragraph` immediately after the
function heading, if and only if it consists solely of one inline `code`
span, and that span parses as `(name: Type = default, ...)`.
The declaration is inert Markdown otherwise.
* Without a signature, parameters are inferred from **free reads of executable
expressions** using data-flow analysis over the function's control-flow
graph (see `evaluation.md` §5). Natural language, heading words, code spans
outside executable positions and prose never introduce parameters.
* Inferred parameters are identified by name. Functions with inferred
signatures accept **named arguments only**. Explicit signatures allow
positional arguments in declaration order.
* A read that is neither a parameter, a local assignment, a module constant,
an imported binding nor a builtin is a `symbol.unknown` error with a hint;
the CLI can list inferred signatures with `inspect --signature`.
## 7. Example: ordered list branch
Source:
```markdown
## 分类
1. `n < 0`
**结果 = -1**
2. `n == 0`
**结果 = 0**
3. `n > 0`
**结果 = 1**
*结果*
```
AST (abridged):
```text
heading(2) 分类
list(ordered) start=1
item
paragraph -> code("n < 0"), text("")
strong -> text("结果 = -1")
item
paragraph -> code("n == 0")
strong -> text("结果 = 0")
...
paragraph
emph -> text("结果")
```
IR:
```text
Function 分类(params=[n])
Branch [
(Eval(n < 0), [Assign(结果, -1)]),
(Eval(n == 0), [Assign(结果, 0)]),
(Eval(n > 0), [Assign(结果, 1)]),
]
Return(Ident(结果))
```
## 8. Hard-error checklist
The following are compile errors with source spans, never prose fallbacks:
* `strong`/`emph` in a function body whose payload does not parse;
* nested `strong`/`emph`/link inside an executable payload;
* ordered list item whose first inline node is not a code span condition;
* bullet list item without a condition header;
* non-`Bool` condition (detected statically when the type is known, else at
runtime as `runtime.type`);
* block quote that is not exactly one complete call;
* duplicate function names in a scope;
* empty heading names;
* return outside a function;
* executable node inside a lazy region being interpreted (must not happen).
+123
View File
@@ -0,0 +1,123 @@
# Marqdo Markdown Profile v1
Status: **frozen for M0.5**. Changes require an ADR, updated goldens and a
profile id bump.
Profile id: `marqdo.md.v1`
Parser: [comrak](https://comrak.ee/) `=1.0.0`, `default-features = false`
CommonMark baseline: 0.31.2
## 1. Why a profile
Markdown is the syntax of the language. Any parser behaviour that can change
the AST can change program meaning, so the parser version and every extension
flag are part of the language version, not an implementation detail. Marqdo
never maintains a second Markdown parser and never decides semantics from raw
text with regular expressions.
## 2. Frozen parser options
Enabled extensions:
| Option | Value | Consequence |
|---|---|---|
| `extension.table` | `true` | GFM tables produce `table`/`table_row`/`table_cell` nodes. |
| `extension.strikethrough` | `true` | `~~x~~` produces `strikethrough`. Inert unless it appears in an executable payload (where it is an error). |
| `extension.tasklist` | `true` | `- [x]`/`- [ ]` items carry a checkbox flag on the item node. |
| `extension.math_dollars` | `true` | `$...$` and inline `$$...$$` produce `math`; standalone `$$` fences with `math_dollars_block` produce `math_block`. |
| `extension.math_dollars_block` | `true` | Multi-line `$$` display math is a block node. |
| `extension.front_matter_delimiter` | `Some("---")` | YAML front matter is recognised only at the start of the document. |
Disabled (must stay `false`): `autolink`, `footnotes`, `inline_footnotes`,
`description_lists`, `multiline_block_quotes`, `alerts`, `ofm_callouts`,
`superscript`, `subscript`, `underline`, `highlight`, `insert`, `spoiler`,
`subtext`, `shortcodes`, `wikilinks_*`, `grid_table`, `greentext`,
`ofm_*`, `block_directive`, `header_attributes`, `fenced_code_attributes`,
`inline_code_attributes`, `link_attributes`, `cjk_friendly_emphasis`,
`phoenix_heex`, `tasklist_in_table`, `relaxed_tasklist_matching`,
`relaxed_autolinks`; parse options `smart`, `ignore_setext`,
`escaped_char_spans` stay `false`.
The unit test `profile::tests::profile_options_are_frozen` asserts this table.
## 3. Byte spans and source maps
Every AST node carries a half-open byte span `[start, end)` into the original
UTF-8 file. Spans are produced from the parser's line/column positions (byte
columns) with a line index, so multi-byte characters are handled correctly.
* `# 数学` has span covering exactly `# 数学` (8 bytes).
* A UTF-8 BOM is tolerated at offset 0. The BOM is stripped before parsing and
all spans are re-based onto the original file (the first character after the
BOM starts at byte 3 in that case).
* Spans of container nodes cover their full source region; inline spans cover
the inline source, not the surrounding markers.
* The source map is used for diagnostics and IDE navigation only. No semantics
is derived from the source text.
## 4. Front matter
YAML front matter is recognised only when the parser reports a `front_matter`
node, i.e. the document starts with `---` (after an optional BOM) and a closing
`---` or `...` line exists. A `---` later in the document is a thematic break
or a Setext heading underline, never front matter.
The parser node keeps the raw delimiters; Marqdo extracts the YAML body and
parses it exactly once with `serde_yaml`. Supported semantics:
* `import:` mapping from local alias to module address (see
`references.md`). Imports are explicit; nothing is inferred.
* Any other key is metadata for tools and is ignored by the interpreter.
* Invalid YAML is a diagnostic (`frontmatter.invalid`), never silently dropped.
* Front matter never executes.
## 5. AST view
`marqdo inspect --ast --json` prints this tree:
```json
{
"kind": "document",
"span": {"start": 0, "end": 20},
"children": [
{"kind": "heading", "level": 2, "setext": false, "span": {"start": 0, "end": 7},
"children": [{"kind": "text", "span": {"start": 3, "end": 7}, "literal": "阶乘"}]}
]
}
```
Rules:
* Node kinds mirror the profile; `task` list items are `item` nodes with
`"task": true|false` so that node type still determines list structure.
GFM task items replace the parser's item node; the profile normalises them
back to `item` plus the checkbox flag.
* Literal payloads are kept on inert leaves: `text`, `code`, `code_block`,
`math`, `math_block`, `html_block`, `html_inline`, `front_matter`, and the
`raw` payload of unmodelled nodes.
* `link`/`image` embed `url` and `title`; `heading` embeds `level` and
`setext`; `list` embeds `ordered`, optional `start` and `tight`; `table`
embeds column `alignments`; `table_row` embeds `header`.
* Unmodelled parser nodes are preserved as `{"kind":"other","tag":"..."}`
with their children and raw literal. They are inert for the language.
* Attribute order in JSON is fixed by the serializer and goldens are
regenerated only with an explicit update command.
## 6. Structural equivalence
Program semantics depend on node type and position, not on source spelling:
* `**x**` and `__x__` are both `strong` and behave identically.
* `*x*` and `_x_` are both `emph`.
* `## name` and a Setext level-2 heading are both `heading` with `level: 2`.
(`setext` is recorded for inspection only.)
* Ordered list delimiters `1.` / `1)` and bullet markers `-`/`*`/`+` produce
the same `list`/`item` structure; role does not depend on the marker.
* Tight versus loose lists (`tight`) only affects rendering; it does not
change program semantics.
## 7. Contract with readers
A `.mq.md` file is rendered normally by any CommonMark/GFM renderer (math
needs a math extension). Rendering never triggers execution. Code fences and
math remain inert `Code`/`Formula` values until an explicit call.