feat(stdlib): add JSON and CSV product parsers

This commit is contained in:
Marqdo Agent committed 2026-11-21 08:23:54 +08:00
1 parent fbe36bcc43
commit bde6f1d0ba
10 files changed
+942 -8

No files matched your search

+6 -5
View File
@@ -58,7 +58,7 @@ byte-identical).
cargo fmt --all -- --check PASS
cargo clippy --workspace --all-targets -- -D warnings PASS
cargo test --workspace --locked PASS
467 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
469 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
```
The documentation and test migration is local work; re-run this block after it
@@ -200,10 +200,11 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
`Value::Table`; inputs are never mutated; GFM AST laziness and physical T0
storage are unchanged. `std:fs/file` provides bounded `读文本`/`写文本`/`复制文件`
over the existing `resource.*` ABI plus `存在`/`元数据`/`列出目录`/`创建目录`/
`删除`/`重命名` over `resource.request` (ADR-0041). Append/seek/watch and a
recursive walk remain open.
`删除`/`重命名` over `resource.request` (ADR-0041); `data/json`/`data/csv`
provide recursive JSON and quoted single-row CSV product parsers. Append/seek/
watch, recursive walk, streaming JSON/CSV and a JSON depth bound remain open.
- Bundled standard library (CM-A10, ADR-0028): `stdlib/` ships
`format`/`text`/`io`/`fs`; explicit `std:` imports resolve only inside the bundle;
`format`/`text`/`io`/`fs`/`data`; explicit `std:` imports resolve only inside the bundle;
a SHA-256 digest over `*.mq.md` is verified before loading; Analysis records
`project.stdlib`; a frozen-binary test asserts the CLI hash is unchanged after
running a `std:io` project and that tampering is rejected. `打印/print` is
@@ -229,7 +230,7 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
status/index links and SHA-pinned evidence; `crates/cli/tests/architecture_boundaries.rs`
pins the workspace graph, `marqdo-runtime -> marqdo-host`, the bundled stdlib
digest and the stdio/file/TCP endpoints; `crates/cli/tests/stdlib_product.rs`
runs every public stdlib module (`console/demo/format/fs/http/io/stream/table/text/utf8`)
runs every public stdlib module (`console/csv/demo/format/fs/http/io/json/stream/table/text/utf8`)
through `std:` imports over `stdlib/tests/*.mq.md`. `tools/pack-stdlib.py`
now sorts `.mq.md` by the same relative path key as the CLI. Source/test baseline
`0d40de7`; later docs-only commits keep code/test bytes identical. Evidence:
+4 -2
View File
@@ -18,7 +18,7 @@ platform".
| Dimension | Current state | Source |
|---|---|---|
| Implemented | Rust workspace `0.1.0`: Markdown/frontend/IR, tree-walk runtime, `Memory`, capability host, Tool/Analysis/Trace, Plugin ABI, Foreign kernels, bundled stdlib, CLI composition root | `Cargo.toml`, `crates/`, `stdlib/` |
| Tested locally | `cargo test --workspace --locked`: **467 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
| Tested locally | `cargo test --workspace --locked`: **469 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
| Bundled stdlib released | In-repo bundle `marqdo-stdlib@0.1.0`, SHA-256 digest verified before loading; it is **not** an external registry package | `stdlib/stdlib.json`, `stdlib/README.md` |
| Multi-platform verified | Linux local gate is the only executed lane here; Windows/macOS native lanes are **UNVERIFIED**, and the GitHub security lane remains a **VM-only** gate | `docs/STATUS.md` §1, §4 |
@@ -88,6 +88,8 @@ review RFC, not a new crate per provider.
| project file write | `resource.open("file", path, ["write"])`; buffers until `close`, never flush-on-drop | implemented, project mode | `resource_streams`, `stdlib_product` |
| `std:fs/file` | `fs.读文本`, `fs.写文本`, `fs.复制文件` over `resource.open/read/write/close` and `Memory`; bounded at 8 MiB | implemented (read/write/copy) | `stdlib_product` |
| `resource.request` | `resource.request("fs.exists|metadata|list|mkdir|remove|rename", paths)`; project mode, grant-checked, recoverable fs Result | implemented `fs.*`; no seek/append/watch | `resource_streams`, `stdlib_product` |
| `std:data/json` | `json.解析(文本)` -> Result with nested Value; `json.格式化(值)` -> Text | implemented (recursive, recoverable errors) | `stdlib_product` |
| `std:data/csv` | `csv.解析行(文本)` -> Result with List<Text>; `csv.格式化行(列表)` -> Text | implemented (single row, quoted) | `stdlib_product` |
| file metadata / directory / rename / remove / mkdir | not part of the current Resource contract | **not implemented**; needs a separate Resource Operation ADR | — |
| TCP | `resource.open("tcp", ip:port, ["read","write"])`; exact IP grant, 5 s timeout | implemented; no DNS/TLS/server socket/UDP | `tcp_endpoint`, `http_over_tcp` |
| HTTP/1.1 | `stdlib/net/http.mq.md`: status/header/chunked-byte parse, request build | implemented over an already-authorized TCP stream; not a DNS/TLS client | `stdlib_http`, `http_over_tcp`, `stdlib_product` |
@@ -115,7 +117,7 @@ that a phase is complete.
| P0 | Documentation drift between `STATUS.md`, the architecture index and the evidence | being fixed by this migration and the docs gate |
| P1 | `std:fs/file` read/write/copy and `resource.request` fs metadata/directory operations are implemented; append/seek/watch/recursive walk remain open | open |
| P1 | No file append/seek/watch/recursive directory walk | open; future Resource Operation extension |
| P1 | JSON/CSV/UTF-8 decode remain mainly under `tests/conformance` | open as product stdlib API |
| P1 | JSON/CSV product parsers are implemented; no streaming/multi-line CSV or depth-bounded JSON walker | open |
| P1 | `crates/cli/src/commands.rs` and `crates/marqdo-host/src/host.rs` still mix multiple responsibilities | open refactor, behavior-preserving |
| P2 | HTTP parsing is Unicode-codepoint oriented; a general byte body is not the only path | open; convert before a general network release |
| P2 | No DNS, TLS, listener or UDP provider | open |
@@ -0,0 +1,49 @@
# Evidence: `std:data/json` and `std:data/csv`
Date: 2026-11-21 (Asia/Shanghai)
Source baseline: `fbe36bcc431ccadcb95b5aba74ecd7c39113f9ae`
Repository: `marqdo-cli`
RFC item: P1 JSON/CSV product stdlib APIs
## Implemented
`stdlib/data/json.mq.md` is a recursive JSON parser/formatter written only in
`.mq.md`:
- `解析(文本)` -> `{ok:true, value}` or `{ok:false, code, message}`;
rejects malformed numbers, strings, escapes, arrays, objects and trailing
input (`json.trailing`).
- `格式化(值)` -> canonical text: strings escaped, lists/objects comma-space
separated, `None` as `null`, Text/Int/Float/Bool by their value form.
`stdlib/data/csv.mq.md` is a quoted single-row CSV parser/formatter:
- `解析行(文本)` -> `{ok:true, value:List<Text>}`; `""` is a literal quote;
unterminated quotes and a quote inside an unquoted field are recoverable
`csv.unterminated_quote`/`csv.invalid_quote`.
- `格式化行(列表)` -> quotes fields containing comma/quote/CR/LF and doubles
inner quotes.
No Rust builtin, `HostOp` or parser change was added. Both modules are pure
`.mq.md` value functions.
## Tests
`stdlib/tests/json.mq.md` and `stdlib/tests/csv.mq.md` run through
`std:data/json` and `std:data/csv` and are pinned by
`crates/cli/tests/stdlib_product.rs` (8 product tests total).
```text
cargo fmt --all -- --check PASS
cargo clippy --workspace --all-targets -- -D warnings PASS
cargo test --workspace 469 passed, 0 failed, 2 ignored
cargo test -p marqdo-cli --test stdlib_product 8 passed
python3 tools/pack-stdlib.py PASS
```
## Not covered
No streaming/incremental JSON, no depth bound (deep nesting can exhaust the
interpreter stack), no multi-line CSV fields, no duplicate-key policy beyond
last-write-wins, and no `std:data/json` conversion to `Table`. Those remain
open product work.