feat(compress): add bounded gzip product codec
This commit is contained in:
1 parent
ff3f19a2ea
commit
efbb8007e1
16 files changed
+372
-9
No files matched your search
+4
-5
@@ -58,7 +58,7 @@ byte-identical).
|
||||
cargo fmt --all -- --check PASS
|
||||
cargo clippy --workspace --all-targets -- -D warnings PASS
|
||||
cargo test --workspace --locked PASS
|
||||
488 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
|
||||
490 passed, 0 failed, 2 ignored; includes 7/7 Linux security tests
|
||||
```
|
||||
|
||||
The documentation and test migration is local work; re-run this block after it
|
||||
@@ -209,10 +209,9 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
|
||||
runs an exact `--allow-process PROGRAM` grant with 5 s timeout and 1 MiB caps
|
||||
(ADR-0044, not an OS sandbox); `random.字节` reads explicitly granted
|
||||
`random.bytes` entropy (ADR-0046). File watch, binary offset reads, streaming
|
||||
JSON/CSV, a JSON depth bound, regex groups/alternation and compression remain
|
||||
open.
|
||||
JSON/CSV, a JSON depth bound and regex groups/alternation remain open.
|
||||
- Bundled standard library (CM-A10, ADR-0028): `stdlib/` ships
|
||||
`format`/`text`/`io`/`fs`/`data`/`path`/`regex`/`datetime`/`clock`/`env`/`process`/`random`; explicit `std:` imports resolve only inside the bundle;
|
||||
`format`/`text`/`io`/`fs`/`data`/`path`/`regex`/`datetime`/`clock`/`env`/`process`/`random`/`compress`; explicit `std:` imports resolve only inside the bundle;
|
||||
a SHA-256 digest over `*.mq.md` is verified before loading; Analysis records
|
||||
`project.stdlib`; a frozen-binary test asserts the CLI hash is unchanged after
|
||||
running a `std:io` project and that tampering is rejected. `打印/print` is
|
||||
@@ -238,7 +237,7 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
|
||||
status/index links and SHA-pinned evidence; `crates/cli/tests/architecture_boundaries.rs`
|
||||
pins the workspace graph, `marqdo-runtime -> marqdo-host`, the bundled stdlib
|
||||
digest and the stdio/file/TCP endpoints; `crates/cli/tests/stdlib_product.rs`
|
||||
runs every public stdlib module (`clock/console/csv/datetime/demo/env/format/fs/http/io/json/path/process/random/regex/stream/table/text/utf8`)
|
||||
runs every public stdlib module (`clock/compress/console/csv/datetime/demo/env/format/fs/http/io/json/path/process/random/regex/stream/table/text/utf8`)
|
||||
through `std:` imports over `stdlib/tests/*.mq.md`. `tools/pack-stdlib.py`
|
||||
now sorts `.mq.md` by the same relative path key as the CLI. Source/test baseline
|
||||
`0d40de7`; later docs-only commits keep code/test bytes identical. Evidence:
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# ADR-0049: gzip compression resource operations
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-21
|
||||
Relates to: ADR-0041
|
||||
Specs: `spec/evaluation.md`
|
||||
Evidence: `docs/evidence/2026-11-21-compress-product.md`
|
||||
|
||||
## Context
|
||||
|
||||
Compression is pure computation, but implementing gzip in `.mq.md` would either
|
||||
duplicate a large algorithm or under-deliver. Adding a new Rust builtin would
|
||||
expand the language surface; the existing non-streaming `resource.request`
|
||||
channel is a better fit for an explicitly bounded codec operation.
|
||||
|
||||
## Decision
|
||||
|
||||
Add two project-mode operations:
|
||||
|
||||
```text
|
||||
resource.request("compress.gzip", [text]) -> {ok:true, value:List<Int>}
|
||||
resource.request("compress.gunzip", [bytes]) -> {ok:true, value:Text}
|
||||
| {ok:false, code:"compress.invalid"}
|
||||
| {ok:false, code:"compress.not_utf8"}
|
||||
| {ok:false, code:"compress.too_large"}
|
||||
```
|
||||
|
||||
* `gzip` takes Text, caps input at 8 MiB, writes a deterministic gzip header
|
||||
(no filename/time) and returns a byte list.
|
||||
* `gunzip` takes a `List<Int>` of bytes, caps input and output at 8 MiB,
|
||||
returns Text on success, and maps bad gzip/UTF-8/oversize output to the
|
||||
recoverable codes above.
|
||||
* No new capability flag is added: compression is deterministic CPU work and
|
||||
does not disclose external state. It remains project-mode only because all
|
||||
`resource.request` operations do; `pure`/`strict` refuse it.
|
||||
* `flate2` (miniz_oxide backend) is the implementation. No new builtin,
|
||||
`HostOp`, parser node or wire value is added.
|
||||
* `stdlib/compress.mq.md` exposes `压缩`/`解压`.
|
||||
|
||||
## Consequences
|
||||
|
||||
* Callers can compress data without a second language surface, but must pass
|
||||
byte lists through ordinary `List<Int>` values.
|
||||
* Streaming gzip, raw deflate, zstd/brotli and binary `Memory` output remain
|
||||
open. A streaming codec would need a new Resource kind.
|
||||
@@ -18,7 +18,7 @@ platform".
|
||||
| Dimension | Current state | Source |
|
||||
|---|---|---|
|
||||
| Implemented | Rust workspace `0.1.0`: Markdown/frontend/IR, tree-walk runtime, `Memory`, capability host, Tool/Analysis/Trace, Plugin ABI, Foreign kernels, bundled stdlib, CLI composition root | `Cargo.toml`, `crates/`, `stdlib/` |
|
||||
| Tested locally | `cargo test --workspace --locked`: **488 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
|
||||
| Tested locally | `cargo test --workspace --locked`: **490 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline | `docs/STATUS.md` §3 |
|
||||
| Bundled stdlib released | In-repo bundle `marqdo-stdlib@0.1.0`, SHA-256 digest verified before loading; it is **not** an external registry package | `stdlib/stdlib.json`, `stdlib/README.md` |
|
||||
| Multi-platform verified | Linux local gate is the only executed lane here; Windows/macOS native lanes are **UNVERIFIED**, and the GitHub security lane remains a **VM-only** gate | `docs/STATUS.md` §1, §4 |
|
||||
|
||||
@@ -101,6 +101,7 @@ review RFC, not a new crate per provider.
|
||||
| `std:env` | `env.读取/存在` -> `resource.request("env.get", [name])`; exact `--allow-env` grant | implemented (project mode; no `env.list`) | `resource_streams`, `stdlib_product` |
|
||||
| `std:process` | `process.运行(程序, 参数列表)` -> `resource.request("process.run", ...)`; exact `--allow-process` grant, 5 s timeout, 1 MiB caps | implemented (project mode; not an OS sandbox) | `resource_streams`, `stdlib_product` |
|
||||
| `std:random` | `random.字节/整数` -> `resource.request("random.bytes", [count])`; exact `--allow-random` grant | implemented (project mode; byte-list RNG) | `resource_streams`, `stdlib_product` |
|
||||
| `std:compress` | `compress.压缩/解压` -> gzip byte list <-> Text; 8 MiB caps | implemented (project mode; no grant) | `resource_streams`, `stdlib_product` |
|
||||
| network endpoints | `tcp`/`udp`/`tcp-listener`/`tls` open + `resource.accept`; exact host grant, DNS after grant, TLS certificate validation | implemented (project mode) | `resource_streams` |
|
||||
| file metadata / directory / rename / remove / mkdir | not part of the current Resource contract | **not implemented**; needs a separate Resource Operation ADR | — |
|
||||
| TCP | `resource.open("tcp", ip:port, ["read","write"])`; exact IP grant, 5 s timeout | implemented; no DNS/TLS/server socket/UDP | `tcp_endpoint`, `http_over_tcp` |
|
||||
@@ -133,7 +134,7 @@ that a phase is complete.
|
||||
| P1 | CLI split into `commands/{sandbox,run,check,inspect,init,package,tool}.rs`; host split into `host/{resource_ops,providers,effects,dispatch}.rs` plus `process.rs`; `commands.rs` keeps shared project/runtime helpers and `host.rs` keeps the `Host` trait methods and capability checks | split; shared helpers and trait methods remain by design |
|
||||
| P2 | HTTP parsing is Unicode-codepoint oriented; a general byte body is not the only path | open; convert before a general network release |
|
||||
| P2 | DNS, UDP, TCP listener and TLS client endpoints are implemented; no TLS server/mTLS or custom-CA CLI | open |
|
||||
| P2 | `std:path`, a regex subset, `std:datetime`, `std:clock`, `std:env`, `std:process` and `std:random` are implemented; no compression | open |
|
||||
| P2 | `std:path`, regex subset, `std:datetime`, `std:clock`, `std:env`, `std:process`, `std:random` and gzip compression are implemented; no streaming/zstd/brotli | open |
|
||||
| P2 | Memory limits: 16 MiB per region, 64 MiB per run | current budget, not a product limit |
|
||||
| P3 | No concurrency tasks, atomics or cancellable long-running scheduling | open; separate ADR |
|
||||
| P3 | No native optimizer/JIT or columnar numeric backend | open; benchmark-first |
|
||||
@@ -171,5 +172,6 @@ that a phase is complete.
|
||||
- [ADR-0046: `random.bytes`](../adr/0046-random-resource-operation.md) — accepted explicitly granted OS entropy.
|
||||
- [ADR-0047: DNS/UDP/TCP listener](../adr/0047-net-dns-udp-listener.md) — accepted project-mode network endpoints.
|
||||
- [ADR-0048: TLS client endpoint](../adr/0048-tls-resource-endpoint.md) — accepted validated rustls client; system + embedder DER roots.
|
||||
- [ADR-0049: gzip compression](../adr/0049-compression-resource-operations.md) — accepted bounded gzip byte-list codec.
|
||||
- [Compute-machine RFC-0001](../MARQDO_COMPUTE_MACHINE_RFC_0001.md) and [core/stdlib RFC-0002](../MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md) — design input.
|
||||
- [Architecture audit and refactor RFC](Marqdo_CLI_Latest_Architecture_Audit_and_Refactor_RFC.md) — review input for this migration.
|
||||
@@ -53,6 +53,7 @@ The implemented first slices of these RFCs are recorded in `CURRENT.md` and
|
||||
| `random.bytes` resource operation | [0046](../adr/0046-random-resource-operation.md) | Accepted (explicit `--allow-random`, project only) |
|
||||
| DNS/UDP/TCP-listener endpoints | [0047](../adr/0047-net-dns-udp-listener.md) | Accepted (project only, exact host grant, no TLS) |
|
||||
| TLS client endpoint | [0048](../adr/0048-tls-resource-endpoint.md) | Accepted (rustls, validated, project only) |
|
||||
| gzip compression operations | [0049](../adr/0049-compression-resource-operations.md) | Accepted (bounded byte-list gzip) |
|
||||
| Table physical storage (T1/T2/T3), strict stdin/stderr/file/network resources | — | **Not decided**; do not implement by implication |
|
||||
|
||||
## Evidence and status
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# Evidence: `std:compress`
|
||||
|
||||
Date: 2026-11-21 (Asia/Shanghai)
|
||||
Source baseline: `ff3f19a2eaf9c2bd40a66335e682079b1d85dd9f`
|
||||
Repository: `marqdo-cli`
|
||||
Decision record: [ADR-0049](../adr/0049-compression-resource-operations.md)
|
||||
|
||||
## Implemented
|
||||
|
||||
`resource.request("compress.gzip", [text])` and
|
||||
`resource.request("compress.gunzip", [bytes])` use `flate2`/miniz_oxide:
|
||||
|
||||
- gzip: Text -> List<Int>, 8 MiB input cap, deterministic header.
|
||||
- gunzip: List<Int> -> Text, 8 MiB input/output caps; bad gzip/UTF-8/oversize
|
||||
output are recoverable `compress.invalid`/`compress.not_utf8`/
|
||||
`compress.too_large`.
|
||||
- Project-mode only; no new grant or builtin.
|
||||
|
||||
`stdlib/compress.mq.md` exposes `压缩`/`解压`.
|
||||
|
||||
## Tests
|
||||
|
||||
`stdlib/tests/compress.mq.md` round-trips `hello 世界`.
|
||||
`crates/marqdo-host/tests/resource_streams.rs` round-trips text and rejects
|
||||
bad gzip (33 host tests).
|
||||
|
||||
```text
|
||||
cargo fmt --all -- --check PASS
|
||||
cargo clippy --workspace --all-targets -- -D warnings PASS
|
||||
cargo test --workspace 490 passed, 0 failed, 2 ignored
|
||||
cargo test -p marqdo-host --test resource_streams 33 passed
|
||||
cargo test -p marqdo-cli --test stdlib_product 16 passed
|
||||
python3 tools/pack-stdlib.py PASS
|
||||
```
|
||||
|
||||
## Not covered
|
||||
|
||||
No streaming gzip, raw deflate, zstd/brotli, binary Memory output or
|
||||
decompression of untrusted data beyond the 8 MiB output cap.
|
||||
Reference in new issue
Block a user