feat(stdlib): add path and regex product modules

This commit is contained in:
Marqdo Agent committed 2026-11-21 08:32:32 +08:00
1 parent bde6f1d0ba
commit cffb6790be
10 files changed
+758 -8

No files matched your search

+45
View File
@@ -135,6 +135,8 @@ fn every_public_stdlib_module_has_a_product_test() {
"http.mq.md",
"io.mq.md",
"json.mq.md",
"path.mq.md",
"regex.mq.md",
"stream.mq.md",
"table.mq.md",
"text.mq.md",
@@ -231,6 +233,49 @@ fn csv_test_parses_and_formats_quoted_rows() {
assert_eq!(quoted[1]["value"], "b");
}
#[test]
fn path_test_joins_and_normalizes_project_paths() {
let report = run_product_test(
product_tests()
.iter()
.find(|file| file.name == "path.mq.md")
.expect("path test"),
);
let items = report["value"]["items"].as_array().expect("list");
assert_eq!(items[0]["value"], "a/b/c");
assert_eq!(items[1]["value"], "c.txt");
assert_eq!(items[2]["value"], "a/b");
assert_eq!(items[3]["value"], "txt");
assert_eq!(items[4]["value"], "a/c");
assert_eq!(items[5]["value"], "/a/c");
assert_eq!(items[6]["value"], "../a/b");
assert_eq!(items[7]["value"], true);
assert_eq!(items[8]["value"], false);
}
#[test]
fn regex_test_matches_anchors_classes_and_search() {
let report = run_product_test(
product_tests()
.iter()
.find(|file| file.name == "regex.mq.md")
.expect("regex test"),
);
let items = report["value"]["items"].as_array().expect("list");
assert_eq!(items[0]["value"], true);
assert_eq!(items[1]["value"], false);
assert_eq!(items[2]["value"], true);
assert_eq!(items[3]["value"], true);
assert_eq!(items[4]["value"], true);
assert_eq!(items[5]["value"], true);
assert_eq!(items[6]["value"], true);
assert_eq!(items[7]["value"], true);
assert_eq!(items[8]["value"], true);
assert_eq!(items[9]["value"], false);
assert_eq!(items[10]["value"], "regex.invalid");
assert_eq!(items[11]["value"], "regex.unsupported");
}
#[test]
fn format_test_pins_value_semantics_and_handle_observation() {
let report = run_product_test(
+7 -5
View File
@@ -58,7 +58,7 @@ byte-identical).
cargo fmt --all -- --check PASS
cargo clippy --workspace --all-targets -- -D warnings PASS
cargo test --workspace --locked PASS
469 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
471 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
```
The documentation and test migration is local work; re-run this block after it
@@ -201,10 +201,12 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
storage are unchanged. `std:fs/file` provides bounded `读文本`/`写文本`/`复制文件`
over the existing `resource.*` ABI plus `存在`/`元数据`/`列出目录`/`创建目录`/
`删除`/`重命名` over `resource.request` (ADR-0041); `data/json`/`data/csv`
provide recursive JSON and quoted single-row CSV product parsers. Append/seek/
watch, recursive walk, streaming JSON/CSV and a JSON depth bound remain open.
provide recursive JSON and quoted single-row CSV product parsers; `path` and a
`regex` subset are pure `.mq.md`. Append/seek/watch, recursive walk, streaming
JSON/CSV, a JSON depth bound, regex groups/alternation and `datetime`/`env`/
`process` remain open.
- Bundled standard library (CM-A10, ADR-0028): `stdlib/` ships
`format`/`text`/`io`/`fs`/`data`; explicit `std:` imports resolve only inside the bundle;
`format`/`text`/`io`/`fs`/`data`/`path`/`regex`; explicit `std:` imports resolve only inside the bundle;
a SHA-256 digest over `*.mq.md` is verified before loading; Analysis records
`project.stdlib`; a frozen-binary test asserts the CLI hash is unchanged after
running a `std:io` project and that tampering is rejected. `打印/print` is
@@ -230,7 +232,7 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
status/index links and SHA-pinned evidence; `crates/cli/tests/architecture_boundaries.rs`
pins the workspace graph, `marqdo-runtime -> marqdo-host`, the bundled stdlib
digest and the stdio/file/TCP endpoints; `crates/cli/tests/stdlib_product.rs`
runs every public stdlib module (`console/csv/demo/format/fs/http/io/json/stream/table/text/utf8`)
runs every public stdlib module (`console/csv/demo/format/fs/http/io/json/path/regex/stream/table/text/utf8`)
through `std:` imports over `stdlib/tests/*.mq.md`. `tools/pack-stdlib.py`
now sorts `.mq.md` by the same relative path key as the CLI. Source/test baseline
`0d40de7`; later docs-only commits keep code/test bytes identical. Evidence:
+4 -2
View File
@@ -18,7 +18,7 @@ platform".
| Dimension | Current state | Source |
|---|---|---|
| Implemented | Rust workspace `0.1.0`: Markdown/frontend/IR, tree-walk runtime, `Memory`, capability host, Tool/Analysis/Trace, Plugin ABI, Foreign kernels, bundled stdlib, CLI composition root | `Cargo.toml`, `crates/`, `stdlib/` |
| Tested locally | `cargo test --workspace --locked`: **469 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
| Tested locally | `cargo test --workspace --locked`: **471 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
| Bundled stdlib released | In-repo bundle `marqdo-stdlib@0.1.0`, SHA-256 digest verified before loading; it is **not** an external registry package | `stdlib/stdlib.json`, `stdlib/README.md` |
| Multi-platform verified | Linux local gate is the only executed lane here; Windows/macOS native lanes are **UNVERIFIED**, and the GitHub security lane remains a **VM-only** gate | `docs/STATUS.md` §1, §4 |
@@ -90,6 +90,8 @@ review RFC, not a new crate per provider.
| `resource.request` | `resource.request("fs.exists|metadata|list|mkdir|remove|rename", paths)`; project mode, grant-checked, recoverable fs Result | implemented `fs.*`; no seek/append/watch | `resource_streams`, `stdlib_product` |
| `std:data/json` | `json.解析(文本)` -> Result with nested Value; `json.格式化(值)` -> Text | implemented (recursive, recoverable errors) | `stdlib_product` |
| `std:data/csv` | `csv.解析行(文本)` -> Result with List<Text>; `csv.格式化行(列表)` -> Text | implemented (single row, quoted) | `stdlib_product` |
| `std:path` | `path.连接/文件名/目录名/扩展名/规范/是否绝对`; lexical only | implemented (no I/O) | `stdlib_product` |
| `std:regex` | `regex.匹配/搜索` with `^ $ . * + ? [...]` escapes | implemented subset; no groups/alternation | `stdlib_product` |
| file metadata / directory / rename / remove / mkdir | not part of the current Resource contract | **not implemented**; needs a separate Resource Operation ADR | — |
| TCP | `resource.open("tcp", ip:port, ["read","write"])`; exact IP grant, 5 s timeout | implemented; no DNS/TLS/server socket/UDP | `tcp_endpoint`, `http_over_tcp` |
| HTTP/1.1 | `stdlib/net/http.mq.md`: status/header/chunked-byte parse, request build | implemented over an already-authorized TCP stream; not a DNS/TLS client | `stdlib_http`, `http_over_tcp`, `stdlib_product` |
@@ -121,7 +123,7 @@ that a phase is complete.
| P1 | `crates/cli/src/commands.rs` and `crates/marqdo-host/src/host.rs` still mix multiple responsibilities | open refactor, behavior-preserving |
| P2 | HTTP parsing is Unicode-codepoint oriented; a general byte body is not the only path | open; convert before a general network release |
| P2 | No DNS, TLS, listener or UDP provider | open |
| P2 | No `std:path`, `std:regex`, `std:datetime`, `std:env`, `std:process`, secure random or compression | open |
| P2 | `std:path` and a regex subset are implemented; no `std:datetime`, `std:env`, `std:process`, secure random or compression | open |
| P2 | Memory limits: 16 MiB per region, 64 MiB per run | current budget, not a product limit |
| P3 | No concurrency tasks, atomics or cancellable long-running scheduling | open; separate ADR |
| P3 | No native optimizer/JIT or columnar numeric backend | open; benchmark-first |
@@ -0,0 +1,47 @@
# Evidence: `std:path` and `std:regex`
Date: 2026-11-21 (Asia/Shanghai)
Source baseline: `bde6f1d0ba6f3ab19829ea63e5e85f1df84979a4`
Repository: `marqdo-cli`
RFC item: P2 pure path/regex stdlib APIs
## Implemented
`stdlib/path.mq.md` (pure `.mq.md`):
- `连接(片段列表)` skips empty segments and inserts one `/`.
- `文件名`, `目录名`, `扩展名` split on the last `/` and `.`.
- `规范(路径)` lexically folds `.`, empty segments and `..`, keeps a leading
`/`; it never touches the filesystem.
- `是否绝对(路径)`.
`stdlib/regex.mq.md` (pure `.mq.md`):
- `匹配(模式, 文本)` requires a full match; `搜索(模式, 文本)` tries every
start position.
- Supports `^`, `$`, `.`, `*`, `+`, `?`, `[...]` ranges/negation and
backslash escapes. Recursive token matcher with backtracking.
- Invalid patterns are recoverable `regex.invalid`; groups/alternation are
`regex.unsupported`.
No Rust builtin or host operation was added.
## Tests
`stdlib/tests/path.mq.md` and `stdlib/tests/regex.mq.md` run through
`std:path`/`std:regex` and are pinned by
`crates/cli/tests/stdlib_product.rs` (10 product tests total).
```text
cargo fmt --all -- --check PASS
cargo clippy --workspace --all-targets -- -D warnings PASS
cargo test --workspace 471 passed, 0 failed, 2 ignored
cargo test -p marqdo-cli --test stdlib_product 10 passed
python3 tools/pack-stdlib.py PASS
```
## Not covered
No regex groups, alternation, named classes (`\d`), Unicode property classes,
incremental matching or a compiled-pattern cache. No path I/O or symlink
resolution. Those remain open product work.
+2
View File
@@ -15,6 +15,8 @@ module; a mismatch is a hard error.
- `data/json.mq.md` — recursive JSON parser and formatter (recoverable Result)
- `data/csv.mq.md` — quoted single-row CSV parser and formatter
- `data/table.mq.md` — value-returning table select/filter/sort/append/set/join/group/GFM
- `path.mq.md` — lexical join/basename/dirname/extension/normalize (no I/O)
- `regex.mq.md` — `^ $ . * + ? [...]` matcher and search (no groups)
- `net/http.mq.md` — HTTP/1.1 response parsing (headers/chunked) and request builder
- `tests/` — product self-tests for every public module
(`console`, `demo`, `format`, `http`, `io`, `stream`, `table`, `text`, `utf8`)
+180
View File
@@ -0,0 +1,180 @@
## 是否绝对
`(路径)`
**点 = text.codepoints(路径)**
1. `长度(点) == 0`
*false*
2. `点[0] == 47`
*true*
*false*
## 分割
`(文本, 分隔码点)`
**点 = text.codepoints(文本)**
**结果 = []**
**当前 = ""**
**下标 = 0**
- `下标 < 长度(点)`
1. `点[下标] == 分隔码点`
**结果 = list.append(结果, 当前)**
**当前 = ""**
2. `true`
**当前 = 当前 + text.from_codepoint(点[下标])**
**下标 = 下标 + 1**
*list.append(结果, 当前)*
## 连接
`(片段列表)`
空片段被跳过,片段之间只插入一个 `/`。
**结果 = ""**
**下标 = 0**
- `下标 < 长度(片段列表)`
**片段 = 文本(片段列表[下标])**
1. `片段 != ""`
1. `结果 == ""`
**结果 = 片段**
2. `true`
**结果 = 结果 + "/" + 片段**
2. `true`
**忽略 = 0**
**下标 = 下标 + 1**
*结果*
## 文件名
`(路径)`
**片段 = 分割(路径, 47)**
*片段[长度(片段) - 1]*
## 目录名
`(路径)`
**片段 = 分割(路径, 47)**
**结果 = []**
**下标 = 0**
- `下标 < 长度(片段) - 1`
**结果 = list.append(结果, 片段[下标])**
**下标 = 下标 + 1**
*连接(结果)*
## 扩展名
`(路径)`
**名字 = 文件名(路径)**
**点列表 = 分割(名字, 46)**
1. `长度(点列表) <= 1`
*""*
*点列表[长度(点列表) - 1]*
## 去尾
`(列表)`
**结果 = []**
**下标 = 0**
- `下标 < 长度(列表) - 1`
**结果 = list.append(结果, 列表[下标])**
**下标 = 下标 + 1**
*结果*
## 规范
`(路径)`
词法规范化:折叠 `.`、空片段和重复 `/`,用 `..` 消去前一片段;
绝对路径保留前导 `/`。不访问文件系统。
**片段 = 分割(路径, 47)**
**绝对 = 是否绝对(路径)**
**栈 = []**
**下标 = 0**
- `下标 < 长度(片段)`
**段 = 片段[下标]**
1. `段 == "" or 段 == "."`
**忽略 = 0**
2. `段 == ".."`
1. `长度(栈) > 0 and 栈[长度(栈) - 1] != ".."`
**栈 = 去尾(栈)**
2. `not 绝对`
**栈 = list.append(栈, 段)**
3. `true`
**忽略绝对根 = 0**
3. `true`
**栈 = list.append(栈, 段)**
**下标 = 下标 + 1**
1. `绝对`
*"/" + 连接(栈)*
*连接(栈)*
+434
View File
@@ -0,0 +1,434 @@
## 成功
`(值)`
*{ok: true, value: 值}*
## 失败
`(代码, 消息)`
*{ok: false, code: 代码, message: 消息}*
## 包含
`(列表, 值)`
**结果 = false**
**下标 = 0**
- `下标 < 长度(列表) and not 结果`
1. `列表[下标] == 值`
**结果 = true**
2. `true`
**忽略 = 0**
**下标 = 下标 + 1**
*结果*
## 去头
`(列表)`
**结果 = []**
**下标 = 1**
- `下标 < 长度(列表)`
**结果 = list.append(结果, 列表[下标])**
**下标 = 下标 + 1**
*结果*
## 设置最后
`(列表, 值)`
**结果 = []**
**下标 = 0**
- `下标 < 长度(列表)`
1. `下标 + 1 == 长度(列表)`
**结果 = list.append(结果, 值)**
2. `true`
**结果 = list.append(结果, 列表[下标])**
**下标 = 下标 + 1**
*结果*
## 片段
`(点, 起点, 终点)`
**结果 = ""**
**下标 = 起点**
- `下标 < 终点`
**结果 = 结果 + text.from_codepoint(点[下标])**
**下标 = 下标 + 1**
*结果*
## 是数字
`(码点)`
*码点 >= 48 and 码点 <= 57*
## 字面令牌
`(码点)`
包含数量词,默认恰好一次。
*{kind: "literal", value: 码点, neg: false, min: 1, max: 1}*
## 任意令牌
`()`
*{kind: "any", value: 0, neg: false, min: 1, max: 1}*
## 类令牌
`(集合, 否定)`
*{kind: "class", value: 集合, neg: 否定, min: 1, max: 1}*
## 锚点令牌
`(方向)`
*{kind: "anchor", value: 方向, neg: false, min: 1, max: 1}*
## 设置量词
`(令牌, 码点)`
1. `码点 == 42`
*{kind: 令牌.kind, value: 令牌.value, neg: 令牌.neg, min: 0, max: -1}*
2. `码点 == 43`
*{kind: 令牌.kind, value: 令牌.value, neg: 令牌.neg, min: 1, max: -1}*
*{kind: 令牌.kind, value: 令牌.value, neg: 令牌.neg, min: 0, max: 1}*
## 编译类
`(点, 位置)`
**当前 = 位置 + 1**
**否定 = false**
1. `当前 < 长度(点) and 点[当前] == 94`
**否定 = true**
**当前 = 当前 + 1**
2. `true`
**忽略否定 = 0**
**集合 = []**
**完成 = false**
**错误 = ""**
- `当前 < 长度(点) and not 完成 and 错误 == ""`
**码点 = 点[当前]**
1. `码点 == 93`
**完成 = true**
**当前 = 当前 + 1**
2. `码点 == 92`
1. `当前 + 1 >= 长度(点)`
**错误 = "regex.invalid"**
2. `true`
**集合 = list.append(集合, 点[当前 + 1])**
**当前 = 当前 + 2**
3. `当前 + 2 < 长度(点) and 点[当前 + 1] == 45 and 点[当前 + 2] != 93`
**开始 = 码点**
**结束 = 点[当前 + 2]**
1. `开始 > 结束`
**错误 = "regex.invalid"**
2. `true`
**值 = 开始**
- `值 <= 结束`
**集合 = list.append(集合, 值)**
**值 = 值 + 1**
**当前 = 当前 + 3**
4. `true`
**集合 = list.append(集合, 码点)**
**当前 = 当前 + 1**
1. `not 完成`
*失败("regex.invalid", "字符类未闭合")*
1. `错误 != ""`
*失败(错误, "字符类范围无效")*
*成功({set: 集合, neg: 否定, next: 当前})*
## 编译
`(点, 位置)`
**令牌表 = []**
**当前 = 位置**
**错误 = ""**
**代码 = ""**
- `当前 < 长度(点) and 错误 == ""`
**码点 = 点[当前]**
1. `码点 == 94`
**令牌表 = list.append(令牌表, 锚点令牌("start"))**
**当前 = 当前 + 1**
2. `码点 == 36`
**令牌表 = list.append(令牌表, 锚点令牌("end"))**
**当前 = 当前 + 1**
3. `码点 == 91`
**类结果 = 编译类(点, 当前)**
1. `not 类结果.ok`
**错误 = 类结果.code**
**代码 = 类结果.message**
2. `true`
**令牌表 = list.append(令牌表, 类令牌(类结果.value.set, 类结果.value.neg))**
**当前 = 类结果.value.next**
4. `码点 == 92`
1. `当前 + 1 >= 长度(点)`
**错误 = "regex.invalid"**
**代码 = "转义缺少字符"**
2. `true`
**令牌表 = list.append(令牌表, 字面令牌(点[当前 + 1]))**
**当前 = 当前 + 2**
5. `码点 == 46`
**令牌表 = list.append(令牌表, 任意令牌())**
**当前 = 当前 + 1**
6. `码点 == 40 or 码点 == 41 or 码点 == 124`
**错误 = "regex.unsupported"**
**代码 = "不支持分组或选择"**
7. `码点 == 42 or 码点 == 43 or 码点 == 63`
**错误 = "regex.invalid"**
**代码 = "量词没有前一项"**
8. `true`
**令牌表 = list.append(令牌表, 字面令牌(码点))**
**当前 = 当前 + 1**
- `错误 == "" and 当前 < 长度(点) and (点[当前] == 42 or 点[当前] == 43 or 点[当前] == 63)`
**令牌表 = 设置最后(令牌表, 设置量词(令牌表[长度(令牌表) - 1], 点[当前]))**
**当前 = 当前 + 1**
- `当前 < 长度(点) and (点[当前] == 42 or 点[当前] == 43 or 点[当前] == 63)`
**错误 = "regex.invalid"**
**代码 = "连续量词无效"**
1. `错误 != ""`
*失败(错误, 代码)*
*成功(令牌表)*
## 单字符匹配
`(令牌, 码点)`
**类别 = 令牌.kind**
1. `类别 == "any"`
*true*
2. `类别 == "literal"`
*令牌.value == 码点*
3. `类别 == "class"`
*包含(令牌.value, 码点) != 令牌.neg*
*false*
## 尝试匹配
`(令牌, 下标, 当前, 点, 位置, 次数, 要求结束)`
**下限 = 当前.min**
**上限 = 当前.max**
1. `次数 >= 下限 and 匹配序列(令牌, 下标 + 1, 点, 位置, 要求结束)`
*true*
1. `上限 >= 0 and 次数 >= 上限`
*false*
1. `位置 < 长度(点) and 单字符匹配(当前, 点[位置])`
*尝试匹配(令牌, 下标, 当前, 点, 位置 + 1, 次数 + 1, 要求结束)*
*false*
## 匹配序列
`(令牌, 下标, 点, 位置, 要求结束)`
1. `下标 >= 长度(令牌)`
1. `要求结束`
*位置 == 长度(点)*
*true*
**当前 = 令牌[下标]**
1. `当前.kind == "anchor"`
1. `当前.value == "start" and 位置 == 0`
*匹配序列(令牌, 下标 + 1, 点, 位置, 要求结束)*
2. `当前.value == "end" and 位置 == 长度(点)`
*匹配序列(令牌, 下标 + 1, 点, 位置, 要求结束)*
*false*
*尝试匹配(令牌, 下标, 当前, 点, 位置, 0, 要求结束)*
## 匹配
`(模式, 文本)`
编译后要求整个文本匹配。支持 `^`、`$`、`.`、`*`、`+`、`?`、
`[...]` 和 `\\` 转义;不支持分组和选择。
**编译结果 = 编译(text.codepoints(模式), 0)**
1. `not 编译结果.ok`
*编译结果*
*{ok: true, value: 匹配序列(编译结果.value, 0, text.codepoints(文本), 0, true)}*
## 搜索
`(模式, 文本)`
**编译结果 = 编译(text.codepoints(模式), 0)**
1. `not 编译结果.ok`
*编译结果*
**令牌 = 编译结果.value**
**点 = text.codepoints(文本)**
**下标 = 0**
**结果 = false**
- `下标 <= 长度(点) and not 结果`
1. `匹配序列(令牌, 0, 点, 下标, false)`
**结果 = true**
2. `true`
**忽略 = 0**
**下标 = 下标 + 1**
*成功(结果)*
+1 -1
View File
@@ -1,5 +1,5 @@
{
"name": "marqdo-stdlib",
"version": "0.1.0",
"digest": "sha256:dc0ab593afeaabde0c1f688b039f35d6efac28936412ba624a408cf0d587aeb8"
"digest": "sha256:b692e264f046098b83c8046ae9c19ba03c116a078eaac3bdcf207d121e794ffb"
}
+8
View File
@@ -0,0 +1,8 @@
---
import:
path: "std:path"
---
## main
*[path.连接(["a", "", "b", "c"]), path.文件名("a/b/c.txt"), path.目录名("a/b/c.txt"), path.扩展名("a/b/c.txt"), path.规范("./a//b/../c"), path.规范("/a/./b/../c"), path.规范("../a/b"), path.是否绝对("/a"), path.是否绝对("a")]*
+30
View File
@@ -0,0 +1,30 @@
---
import:
regex: "std:regex"
---
## main
**`星号 = regex.匹配("a*b", "aaab")`**
**`加号 = regex.匹配("a+b", "b")`**
**`锚点 = regex.匹配("^a.*z$", "abcz")`**
**`类 = regex.匹配("[a-c]+", "abc")`**
**`否定类 = regex.匹配("[^0-9]", "a")`**
**`任意 = regex.匹配("a.a", "aba")`**
**`可选 = regex.匹配("a?b", "b")`**
**`转义 = regex.匹配("\\.", ".")`**
**`搜索 = regex.搜索("b+c", "xxbbcyy")`**
**`无效 = regex.匹配("[a", "a")`**
**`不支持 = regex.匹配("(a)", "a")`**
*[星号.value, 加号.value, 锚点.value, 类.value, 否定类.value, 任意.value, 可选.value, 转义.value, 搜索.value, 无效.ok, 无效.code, 不支持.code]*