diff --git a/spec/evaluation.md b/spec/evaluation.md new file mode 100644 index 0000000..ae8e1d4 --- /dev/null +++ b/spec/evaluation.md @@ -0,0 +1,130 @@ +# Marqdo Evaluation Semantics v1 + +Status: **frozen for M0.5**. Changes to observable behaviour require an ADR. + +## 1. Values + +| Type | Meaning | +|---|---| +| `Unit` | no result; a function that falls through returns `Unit` | +| `None` | explicit empty value; distinct from `Unit` | +| `Bool` | `true` / `false`; the only type usable in conditions | +| `Int` | arbitrary precision integer (`num-bigint`) | +| `Float` | IEEE-754 binary64 | +| `Text` | Unicode string | +| `List` | ordered values | +| `Map` | string-keyed record with deterministic (sorted) key order | +| `Table` | header names + rows of values (from GFM tables) | +| `Formula` | inert parsed LaTeX + provenance; math operations are explicit | +| `Code` | inert code fence + provenance; execution is explicit | +| `FunctionRef` | reference to a named function in a module | +| `PluginHandle` | opaque host-owned plugin export handle | + +There is no truthiness: `if`-like conditions accept `Bool` only. There are no +implicit string↔number conversions; conversion builtins must be called. + +## 2. Numeric semantics + +* `Int` arithmetic (`+ - * // % **`) is exact. `/` always produces `Float`. +* Mixed `Int`/`Float` arithmetic yields `Float`; the conversion of an `Int` + larger than 2^53 may lose precision and this is documented, not silently + exported as a rounded `Int`. +* Comparisons between `Int` and `Float` are **exact** (the float is treated as + the exact binary rational it is); no lossy comparison is performed. +* `//` and `%` require `Int` operands. Floor division: `a // b` is the + mathematical floor; `a % b == a - b * (a // b)` (sign follows divisor). +* Division/modulo by zero is `runtime.division_by_zero`. +* `+` also concatenates two `Text` values. No other operator is overloaded. +* `**` on `Int` uses exact exponentiation; a negative exponent is a type error + (use `/` or `Float` operands for fractional results). Exponents above a + configurable budget are `resource.exhausted`. +* Floats may be `NaN`/`±Infinity` only as results of operations; they are + tagged explicitly on the wire. NaN compares unequal to everything. + +## 3. Evaluation order and scopes + +* Blocks execute in Markdown AST document order; executable inline nodes + execute in source order within a paragraph. +* A function call evaluates its arguments in source order, exactly once each, + then binds them to fresh parameter slots in an isolated frame. The callee + cannot see the caller's locals (no dynamic scoping); imported bindings are + read-only. +* Assignment binds/updates a local slot in the current frame. It never writes + an outer frame. There is no module-level assignment in v1: assignment + outside a function body is an error. +* `and`/`or` short-circuit and require `Bool` operands. +* `Return` (an `emph` node) unwinds through branch/loop bodies to the + enclosing function frame. +* `Branch` checks conditions in AST order and executes the first `true` branch + once. `Loop` re-evaluates its condition before each iteration. +* Imported modules only register definitions for the importing document; they + have no initialization side effects and their `main` is never started. +* Function calls recurse; recursion depth is bounded by the stack budget. + Mutual recursion is allowed. + +## 4. Errors + +All failures are structured. Compile-time problems produce diagnostics +(`markdown.invalid_construct`, `expr.parse`, `symbol.unknown`, …). Runtime +problems produce `RuntimeError { category, span, message, frames, cause }` +with a stable category code. Runtime errors stop the current function and +propagate with a call stack. A host/foreign/plugin failure never panics the +interpreter; it becomes a structured error. + +## 5. Parameter inference (static, name based) + +Parameters are discovered by definite-assignment data-flow analysis over the +function's control-flow graph, never from prose: + +1. Build the CFG of the function body in AST order, with branch and loop + edges. Loop bodies may execute zero times. +2. Compute the set of variables *definitely assigned* on every path from the + function entry. +3. Every read of a variable that is not definitely assigned at that point and + that has no assignment anywhere in the function is a **candidate + parameter** (sorted by canonical name). +4. If a variable is assigned on some paths but read before that assignment on + others, its identity is ambiguous: emit `symbol.uninitialized` (S-04) + instead of inventing a parameter. +5. Exclude builtins, imported aliases and reserved names from inference. +6. A read that is not explained by a parameter, a local, a builtin or an + import is `symbol.unknown` with a hint (typo protection). + +Consequences: `**结果 = n * 阶乘(n=n-1)**` infers `n`; `**n = 3**` followed by +reads makes `n` a local. Inferred parameters are named arguments only; an +explicit signature paragraph allows positional arguments. + +## 6. Resource budgets + +Every run has explicit limits (all configurable, all reported as +`resource.exhausted`): evaluation fuel, call stack depth, total output bytes, +wall-clock time, and host process count. Loops consume fuel per iteration and +expressions per node so that a hostile document cannot hang the CLI. Strict +mode additionally enforces limits at the OS level where available; if the +platform cannot provide that, strict mode refuses to run (see `security.md`). + +## 7. Determinism + +For a fixed source, mode, capabilities and budget, evaluation is +deterministic. Stdout/stderr from host operations are separated: program +output goes to stdout, diagnostics/traces go to stderr, and `--json` output +never mixes with program output. + +## 8. Builtins v1 + +Builtins are part of the language surface and are capability-checked at the +host boundary. Names are stable; adding one requires a spec update. + +| Name | Signature | Notes | +|---|---|---| +| `打印` / `print` | `(x) -> Unit` | writes `Value::display()` + newline to stdout | +| `长度` / `len` | `(x) -> Int` | Text scalar count, List/Map/Table size | +| `类型` / `type` | `(x) -> Text` | value type name | +| `文本` / `str` | `(x) -> Text` | explicit conversion | +| `整数` / `int` | `(x) -> Int` | from Int/Float(if integral)/Text(decimal) | +| `小数` / `float` | `(x) -> Float` | from Int/Float/Text | +| `表` / `table` | `(header: List, rows: List) -> Table` | validates row widths | +| `ref` | `(target: Text) -> FunctionRef/Formula/Code` | resolves a document address | +| `math.parse/eval/simplify/differentiate` | see `spec/math.md` | P5 | +| `foreign.run/call` | see `spec/foreign.md` | P5, capability-gated | +| `fs.read/write` | see `spec/security.md` | capability-gated | diff --git a/spec/expression.ebnf b/spec/expression.ebnf new file mode 100644 index 0000000..9cd7626 --- /dev/null +++ b/spec/expression.ebnf @@ -0,0 +1,85 @@ +(* Marqdo Expression Grammar v1 + Status: frozen for M0.5. This is the only grammar for executable payloads. + It is a small independent language: it is NOT Markdown and NOT translated + to Python. Markdown AST decides where a payload exists; this grammar + decides what it means. + + Terminals are described in prose where Unicode categories matter. +*) + +program = [ statement ] , EOF ; + +statement = assignment | expression ; +assignment = identifier , "=" , expression ; +(* "=" is a single "=" token; "==" is a distinct token. Assignment targets + are bare identifiers only: no member or index assignment in v1. *) + +expression = or_expr ; +or_expr = and_expr , { "or" , and_expr } ; +and_expr = compare_expr , { "and" , compare_expr } ; +compare_expr = add_expr , [ compare_op , add_expr ] ; + (* non-associative: "a < b < c" is a parse error; write + "a < b and b < c" instead. *) +compare_op = "==" | "!=" | "<" | "<=" | ">" | ">=" ; + +add_expr = mul_expr , { ( "+" | "-" ) , mul_expr } ; +mul_expr = unary_expr , { ( "*" | "/" | "//" | "%" ) , unary_expr } ; +unary_expr = ( "not" | "-" | "+" ) , unary_expr | power_expr ; +power_expr = postfix_expr , [ "**" , unary_expr ] ; + (* right associative: 2 ** 3 ** 2 == 2 ** (3 ** 2); + -x ** 2 == -(x ** 2); 2 ** -1 is valid *) + +postfix_expr = primary , { call_suffix | member_suffix | index_suffix } ; +call_suffix = "(" , [ argument_list ] , ")" ; +argument_list = argument , { "," , argument } , [ "," ] ; +argument = [ identifier , "=" ] , expression ; + (* positional arguments must all precede named arguments; + duplicate argument names are errors *) +member_suffix = "." , identifier ; +index_suffix = "[" , expression , "]" ; + +primary = int_literal + | float_literal + | text_literal + | "true" | "false" | "None" + | identifier + | "(" , expression , ")" + | list_literal + | record_literal ; + +list_literal = "[" , [ expression , { "," , expression } , [ "," ] ] , "]" ; +record_literal = "{" , [ record_entry , { "," , record_entry } , [ "," ] ] , "}" ; +record_entry = ( identifier | text_literal ) , ":" , expression ; + +(* Lexical rules *) + +identifier = xid_start , { xid_continue } ; + (* Unicode XID per UAX #31: Chinese, Latin, etc. are valid. + Keywords are reserved and cannot be identifiers: + true, false, None, and, or, not. *) + +int_literal = digit , { digit } ; + (* arbitrary precision decimal integer; no sign here (unary + minus handles negatives); no separators; no hex/octal. *) + +float_literal = digit , { digit } , "." , digit , { digit } , + [ ( "e" | "E" ) , [ "+" | "-" ] , digit , { digit } ] ; + (* requires digits on both sides of the dot; ".5" and "1." + are invalid. Floats are IEEE-754 binary64. *) + +text_literal = '"' , { text_char | escape } , '"' ; +escape = "\\" , ( '"' | "\\" | "n" | "t" | "r" | "u{" , hexdigit , { hexdigit } , "}" ) ; +text_char = ? any Unicode scalar value except '"' and "\\" ? ; + +(* Whitespace: any Unicode whitespace separates tokens. Newlines inside a + code-span payload are whitespace. There are no comments in v1. *) + +(* Notes on the Markdown boundary *) +(* - A "**" inside a payload is the power operator only when it survived + Markdown parsing inside a code span or text that is not emphasis. + Authors should put multiplication/power expressions in code spans when + Markdown emphasis would otherwise consume the asterisks. *) +(* - Payload spans returned by the parser are relative to the concatenated + payload; the frontend maps them back through the text/code segments to + file spans. *) +(* - Maximum nesting depth is 128; exceeding it is an expr.parse error. *) diff --git a/spec/mapping.md b/spec/mapping.md new file mode 100644 index 0000000..5342963 --- /dev/null +++ b/spec/mapping.md @@ -0,0 +1,235 @@ +# Marqdo AST → Program Mapping v1 + +Status: **frozen for M0.5**. This file is the normative mapping from Markdown +AST structure to program meaning. It is derived directly from +`docs/DEVELOPMENT_BLUEPRINT.md` sections 1.2–1.4 and 3.2. + +## 1. The two rules that override everything + +1. **Node type and structural position decide the role.** A `strong` node in an + executable position is an execution statement and its payload must parse as + a `Statement`. It is never "prose that happens to look like code". +2. **Invalid executable shapes are hard errors.** If a node occupies an + executable role but its shape is invalid, compilation fails with a + `markdown.invalid_construct` diagnostic. There is no fallback to prose. + +Prose never causes an action. Natural-language words (“if”, “return”, “若”, +“执行”) are never interpreted as keywords. + +## 2. Document regions and heading scopes + +A document root is a module. Heading nodes create addressable scopes: + +| AST | Role | Requirements | +|---|---|---| +| `document` | module | exactly one root | +| `heading(level=1)` | namespace / object domain | non-empty canonical name | +| `heading(level=2)` | function | non-empty canonical name, unique in its scope | +| `heading(level>=3)` | section (source location) | does not create a function | + +Scope rules: + +* The range of a heading extends until the next heading of the same or a + lower level (i.e. same or smaller number), or the end of the document. +* A level-2 heading without a preceding level-1 heading belongs to the module + root. Nested headings inside a section remain part of the enclosing function. +* Headings inside container blocks (block quotes, list items) or inside fenced + code blocks never create functions. +* A function's canonical name is the normalised text of its heading children: + NFC, trimmed, internal whitespace collapsed to one space. Empty names are + errors. Duplicate canonical names in the same scope are errors. GitHub-style + anchor slugs are derived for navigation but are **not** the identity. +* `main` at module root is the default entry point for `marqdo run`; an + imported module's `main` is not run automatically. + +## 3. Lazy (inert) regions + +The following nodes are inert regardless of where they appear. Their inline +content is never executed: + +* heading text (headings scope, they do not run) +* `link` labels and `image` alt text +* `table` cells (tables are document values) +* `code_block` and inline `code` outside designated payload positions +* `math` / `math_block` +* `html_block` / `html_inline` +* front matter +* text and other inline nodes inside prose paragraphs + +Inline formatting inside those regions (e.g. `strong` inside a heading) does +not execute. The implementation must use a context-aware visitor, not a blind +whole-tree walk. + +## 4. Executable roles in function bodies + +Inside a function (`heading(level=2)` range) the following roles apply, +including inside branch and loop bodies: + +| AST node | Role | Shape requirement | On invalid shape | +|---|---|---|---| +| `strong` | execution statement | strict payload parses as `Statement` | hard error | +| `emph` | return statement | strict payload parses as `Expression` | hard error | +| `list(ordered=true)` | branch | each item starts with a condition header | hard error | +| `list(ordered=false)` | loop | each item starts with a loop condition header | hard error | +| `block_quote` | call | one paragraph forming a complete call | hard error | +| `paragraph` | prose + inline execution | inline executable nodes run in order | n/a | +| everything else | inert document value | — | type/constraint errors only if used | + +Outside functions (module top-level prose) the same executable nodes are inert +document content (S-03 in the blueprint: executable lists/strong/emph only run +inside a function body). The single exception is `main` itself, which is a +function. + +### 4.1 Strict payload extraction + +For `strong`/`emph`, the payload is the concatenation of the direct `text` and +`code` child literals, exactly as written. Nested `strong`, `emph`, `link`, +`strikethrough`, line breaks, math, HTML or any other inline node is an error: +one AST node must never be executed twice or hide control content. + +`**x = 1**` and `**`x = 1`**` have the same payload `x = 1`. `**a * b**` +parses as text in most cases; when Markdown emphasis and the multiplication +operator conflict, the author should use a code span for the payload: +`` **`a * b`** ``. There is no text-based second parse. + +### 4.2 Paragraph and document order + +Executable inline nodes in a paragraph run in source order. Blocks run in AST +document order. A `return` short-circuits the remaining nodes of the enclosing +function, including nodes inside branches and loops. + +### 4.3 Branch (`ordered list`) + +Each item must: + +1. contain at least one block; +2. have a `paragraph` as its **first** block; +3. have an inline `code` node as the **first inline node** of that paragraph; + its literal is the condition expression; +4. any remaining inline content in that first paragraph is prose. + +The item's remaining blocks form the branch body. Items are tested in AST +order; the first condition evaluating to `true` is executed once. No match +means `Unit` and execution continues after the list. Conditions must be +`Bool`; they must be pure (side effects in conditions are a static error when +detectable and a runtime capability concern otherwise). + +### 4.4 Loop (`bullet list`) + +Each item is an independent `while` loop. Item structure matches the branch +header rule: the first paragraph's first inline node must be a `code` span +(condition), and the remaining blocks are the loop body. The condition is +re-evaluated before every iteration and must be `Bool`. Items execute in +order, so a bullet list is a sequence of loops, one per item. + +Iteration over collections is expressed with stdlib `map`/`filter`/`fold`; +a dedicated `forEach` header is intentionally not part of v1. + +### 4.5 Call (`block quote`) + +A block quote is exactly one `paragraph` and nothing else. The paragraph is +parsed with the call grammar: + +* plain payload: `打印("你好")`, `math.eval(ref("#公式"), {"r": r}, target="A")` + — the strict payload must parse as a complete `CallExpr`; +* link form: `[计算面积](./math.mq.md#计算面积)(r=3)` — the first inline node + must be a `link`, its destination identifies the target function, and the + remaining `text`/`code` payload must be a parenthesised argument list. + +Anything else (a second paragraph, plain prose, a nested quote, an expression +that is not a call) is a hard error. Two calls require two block quote nodes. +`> [说明](#目标)` without an argument list is not accepted in v1. + +The call result is discarded by the statement form; use `strong` to bind it +(`**面积 = 计算面积(r=3)**`). + +## 5. Calls, returns and values + +* A call is `函数(...)`; zero-argument calls still write `()`. +* Positional arguments are only allowed for functions with an explicit + signature. Named arguments can be given in any order for any function. +* Arguments are evaluated in source order exactly once before binding. +* `emph` returns the value of its expression from the enclosing function + (propagating through branch/loop bodies), never from a block. +* `*表达式*` used in a branch is a return from the whole function. +* A function that ends without a return evaluates to `Unit`. `None` is an + explicit empty value and is distinct from `Unit`. +* Return at module top level (outside any function) is an error. + +## 6. Function signatures and parameter inference + +* Optional explicit signature: the first `paragraph` immediately after the + function heading, if and only if it consists solely of one inline `code` + span, and that span parses as `(name: Type = default, ...)`. + The declaration is inert Markdown otherwise. +* Without a signature, parameters are inferred from **free reads of executable + expressions** using data-flow analysis over the function's control-flow + graph (see `evaluation.md` §5). Natural language, heading words, code spans + outside executable positions and prose never introduce parameters. +* Inferred parameters are identified by name. Functions with inferred + signatures accept **named arguments only**. Explicit signatures allow + positional arguments in declaration order. +* A read that is neither a parameter, a local assignment, a module constant, + an imported binding nor a builtin is a `symbol.unknown` error with a hint; + the CLI can list inferred signatures with `inspect --signature`. + +## 7. Example: ordered list branch + +Source: + +```markdown +## 分类 + +1. `n < 0` + **结果 = -1** +2. `n == 0` + **结果 = 0** +3. `n > 0` + **结果 = 1** + +*结果* +``` + +AST (abridged): + +```text +heading(2) 分类 + list(ordered) start=1 + item + paragraph -> code("n < 0"), text("") + strong -> text("结果 = -1") + item + paragraph -> code("n == 0") + strong -> text("结果 = 0") + ... + paragraph + emph -> text("结果") +``` + +IR: + +```text +Function 分类(params=[n]) + Branch [ + (Eval(n < 0), [Assign(结果, -1)]), + (Eval(n == 0), [Assign(结果, 0)]), + (Eval(n > 0), [Assign(结果, 1)]), + ] + Return(Ident(结果)) +``` + +## 8. Hard-error checklist + +The following are compile errors with source spans, never prose fallbacks: + +* `strong`/`emph` in a function body whose payload does not parse; +* nested `strong`/`emph`/link inside an executable payload; +* ordered list item whose first inline node is not a code span condition; +* bullet list item without a condition header; +* non-`Bool` condition (detected statically when the type is known, else at + runtime as `runtime.type`); +* block quote that is not exactly one complete call; +* duplicate function names in a scope; +* empty heading names; +* return outside a function; +* executable node inside a lazy region being interpreted (must not happen). diff --git a/spec/markdown-profile.md b/spec/markdown-profile.md new file mode 100644 index 0000000..76e0ac2 --- /dev/null +++ b/spec/markdown-profile.md @@ -0,0 +1,123 @@ +# Marqdo Markdown Profile v1 + +Status: **frozen for M0.5**. Changes require an ADR, updated goldens and a +profile id bump. + +Profile id: `marqdo.md.v1` +Parser: [comrak](https://comrak.ee/) `=1.0.0`, `default-features = false` +CommonMark baseline: 0.31.2 + +## 1. Why a profile + +Markdown is the syntax of the language. Any parser behaviour that can change +the AST can change program meaning, so the parser version and every extension +flag are part of the language version, not an implementation detail. Marqdo +never maintains a second Markdown parser and never decides semantics from raw +text with regular expressions. + +## 2. Frozen parser options + +Enabled extensions: + +| Option | Value | Consequence | +|---|---|---| +| `extension.table` | `true` | GFM tables produce `table`/`table_row`/`table_cell` nodes. | +| `extension.strikethrough` | `true` | `~~x~~` produces `strikethrough`. Inert unless it appears in an executable payload (where it is an error). | +| `extension.tasklist` | `true` | `- [x]`/`- [ ]` items carry a checkbox flag on the item node. | +| `extension.math_dollars` | `true` | `$...$` and inline `$$...$$` produce `math`; standalone `$$` fences with `math_dollars_block` produce `math_block`. | +| `extension.math_dollars_block` | `true` | Multi-line `$$` display math is a block node. | +| `extension.front_matter_delimiter` | `Some("---")` | YAML front matter is recognised only at the start of the document. | + +Disabled (must stay `false`): `autolink`, `footnotes`, `inline_footnotes`, +`description_lists`, `multiline_block_quotes`, `alerts`, `ofm_callouts`, +`superscript`, `subscript`, `underline`, `highlight`, `insert`, `spoiler`, +`subtext`, `shortcodes`, `wikilinks_*`, `grid_table`, `greentext`, +`ofm_*`, `block_directive`, `header_attributes`, `fenced_code_attributes`, +`inline_code_attributes`, `link_attributes`, `cjk_friendly_emphasis`, +`phoenix_heex`, `tasklist_in_table`, `relaxed_tasklist_matching`, +`relaxed_autolinks`; parse options `smart`, `ignore_setext`, +`escaped_char_spans` stay `false`. + +The unit test `profile::tests::profile_options_are_frozen` asserts this table. + +## 3. Byte spans and source maps + +Every AST node carries a half-open byte span `[start, end)` into the original +UTF-8 file. Spans are produced from the parser's line/column positions (byte +columns) with a line index, so multi-byte characters are handled correctly. + +* `# 数学` has span covering exactly `# 数学` (8 bytes). +* A UTF-8 BOM is tolerated at offset 0. The BOM is stripped before parsing and + all spans are re-based onto the original file (the first character after the + BOM starts at byte 3 in that case). +* Spans of container nodes cover their full source region; inline spans cover + the inline source, not the surrounding markers. +* The source map is used for diagnostics and IDE navigation only. No semantics + is derived from the source text. + +## 4. Front matter + +YAML front matter is recognised only when the parser reports a `front_matter` +node, i.e. the document starts with `---` (after an optional BOM) and a closing +`---` or `...` line exists. A `---` later in the document is a thematic break +or a Setext heading underline, never front matter. + +The parser node keeps the raw delimiters; Marqdo extracts the YAML body and +parses it exactly once with `serde_yaml`. Supported semantics: + +* `import:` mapping from local alias to module address (see + `references.md`). Imports are explicit; nothing is inferred. +* Any other key is metadata for tools and is ignored by the interpreter. +* Invalid YAML is a diagnostic (`frontmatter.invalid`), never silently dropped. +* Front matter never executes. + +## 5. AST view + +`marqdo inspect --ast --json` prints this tree: + +```json +{ + "kind": "document", + "span": {"start": 0, "end": 20}, + "children": [ + {"kind": "heading", "level": 2, "setext": false, "span": {"start": 0, "end": 7}, + "children": [{"kind": "text", "span": {"start": 3, "end": 7}, "literal": "阶乘"}]} + ] +} +``` + +Rules: + +* Node kinds mirror the profile; `task` list items are `item` nodes with + `"task": true|false` so that node type still determines list structure. + GFM task items replace the parser's item node; the profile normalises them + back to `item` plus the checkbox flag. +* Literal payloads are kept on inert leaves: `text`, `code`, `code_block`, + `math`, `math_block`, `html_block`, `html_inline`, `front_matter`, and the + `raw` payload of unmodelled nodes. +* `link`/`image` embed `url` and `title`; `heading` embeds `level` and + `setext`; `list` embeds `ordered`, optional `start` and `tight`; `table` + embeds column `alignments`; `table_row` embeds `header`. +* Unmodelled parser nodes are preserved as `{"kind":"other","tag":"..."}` + with their children and raw literal. They are inert for the language. +* Attribute order in JSON is fixed by the serializer and goldens are + regenerated only with an explicit update command. + +## 6. Structural equivalence + +Program semantics depend on node type and position, not on source spelling: + +* `**x**` and `__x__` are both `strong` and behave identically. +* `*x*` and `_x_` are both `emph`. +* `## name` and a Setext level-2 heading are both `heading` with `level: 2`. + (`setext` is recorded for inspection only.) +* Ordered list delimiters `1.` / `1)` and bullet markers `-`/`*`/`+` produce + the same `list`/`item` structure; role does not depend on the marker. +* Tight versus loose lists (`tight`) only affects rendering; it does not + change program semantics. + +## 7. Contract with readers + +A `.mq.md` file is rendered normally by any CommonMark/GFM renderer (math +needs a math extension). Rendering never triggers execution. Code fences and +math remain inert `Code`/`Formula` values until an explicit call.