feat(provider): add C ABI provider worker binding v1
- Add the separate marqdo-provider-worker binary: loads one trusted c-abi-1 shared library and serves the ADR-0035 framed-JSON resource protocol, with fail-closed startup and no in-process fallback (ADR-0038). - Reuse StdHost::register_worker_resource_provider; language still sees only resource.* and project-mode/name grants. - Compile a C test provider and run open/read/write/close through the helper; cover missing-library fail-closed. Trusted process boundary, not a sandbox; existing plugin-Worker ABI binding remains open. - Add ADR/spec/status/README/changelog/evidence; workspace member and tests. Gate: fmt, clippy -D warnings, 440 tests passed.
This commit is contained in:
1 parent
313ac035e8
commit
757c5ed15e
13 files changed
+562
-2
No files matched your search
+7
-1
@@ -136,6 +136,9 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
|
||||
PluginHandle `plugin/export` are read-only observation members (host id
|
||||
private); `stdlib/format.mq.md` reproduces builtin `打印`/`repr` for all 15
|
||||
value types, including Formula/Code/FunctionRef and Code UTF-8 byte length.
|
||||
- C ABI provider worker binding (CM-A09, ADR-0038): the separate
|
||||
`marqdo-provider-worker` helper loads a trusted C ABI library and serves the
|
||||
framed-JSON transport; process isolation but not a sandbox.
|
||||
- Provider package manifest v1 (CM-A09, ADR-0037): a local `marqdo.provider`
|
||||
manifest resolves a trusted C ABI library and registers it; no registry,
|
||||
signatures or CLI install yet.
|
||||
@@ -221,7 +224,7 @@ behavior. This table records only what is in this tree now.
|
||||
| P2 Marqdo-written allocator/data structures + minimal structure observation | **PASS (local pure lane); external stdlib repo publication pending** | `.mq.md`-only evidence: bump allocator with u32 codec, linked list, failed-allocation atomicity (48–50); CM-A04 structure primitives and a table filter (51–54, ADR-0023); a first-fit free-list allocator with block reuse, splitting, address-adjacent coalescing and `Result` maps (55, ADR-0024), a `Memory`-backed binary tree with recursive insert/in-order traversal (56), a `.mq.md` CSV row splitter (57), an open-addressing hash table with linear probing, update and `hash.full` (58), a `.mq.md` UTF-8 encoder over `Memory` (59), a validating UTF-8 decoder rejecting truncated, bad-continuation and surrogate/overlong input (60), a `Memory`-backed graph adjacency list (61), a recursive JSON parser for nested arrays/objects, strings with escapes, integers, booleans, null and error Results (62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (63), big-endian 16/32/64-bit plus LEB128 integer codecs (64), a `.mq.md` table sort rebuilding a new table from `table.row/columns/cell` (65), a table join (67), a quoted/escaped CSV row parser (68), table group (69), an Int matrix multiply (70), and a sparse-matrix multiply (73); the first-fit allocator also detects double free (`alloc.double_free`). Evidence: [`memory-p2-closure`](evidence/2026-11-20-memory-p2-closure.md), [`structure-api-p2`](evidence/2026-11-20-structure-api-p2.md), [`result-and-free-list-p2`](evidence/2026-11-20-result-and-free-list-p2.md). Local gate PASS: `crates/cli/tests/p2_closure.rs` runs 21 C01–C06/C11/C12 fixtures plus the stdlib table self-test in `--mode pure` (no foreign/plugin/import grants), scans sources for foreign/plugin/import shortcuts and compares value/stdout to goldens; evidence [`p2-closure-gate`](evidence/2026-11-20-p2-closure-gate.md). The bundled `stdlib/` tree, `std:` imports, digest verification, Analysis identity and frozen-CLI closure test are implemented (ADR-0028). Remaining outside C01–C12: publication as a separate external stdlib repository, and no remote CI URL. |
|
||||
| P3 Resource/Stream ABI | **PARTIAL** | ADR-0025/0026 CM-A06 slices: opaque `Resource`, `resource.open/read/write/close`, stdout write in pure/project, granted project-file read and buffered project-file write in project, EOF/closed/handle errors, wire/host rejection; strict permits only stdio/stdout and refuses file/network resources. Fixture 66 + CLI/host tests. CM-A07 **Compat-C** is implemented: when a program calls `打印/print`, the loader injects the bundled `stdlib/io.mq.md` and the runtime routes to `io.写值` (all 15 value types, UTF-8 via `Memory`, `resource.write(stdout)`); `HostOp::Print` and the legacy `Value::display()` print path are removed, and a missing bundle fails with `kernel.unavailable`. Strict permits only the existing stdio console channel. Providers are registered through ADR-0033 (in-process) and ADR-0035 (worker helper). Still missing: directories, C ABI provider binding, provider packages, strict provider support. |
|
||||
| P4 Table logical/physical migration | **PASS (local gate); T1/T2/T3 future** | ADR-0032/CM-A08: `Value::Table` semantics and GFM AST laziness unchanged; `stdlib/data/table.mq.md` provides select/filter/sort/append/set/join/group/`转GFM` as value-returning `.mq.md` functions with input immutability; zero Rust table algorithms were added. Cross-storage equivalence is pinned by fixture 72 (row Table vs `.mq.md` columnar store, identical headers/rows/cells). The optional T1/T2/T3 physical-storage/column-store migration was not required by the gate and is not implemented. |
|
||||
| P5 third-party Provider / TCP / HTTP | **PASS (local gate); external network/CI and plugin-Worker resource binding open** | C10: `stdlib/net/http.mq.md` + fixture 71 parse HTTP/1.1 status/headers/chunked bodies and build requests; `http_over_tcp.rs` runs a loopback HTTP exchange wholly in `.mq.md` over `resource.open("tcp",...)`. ADR-0034 IP-literal blocking TCP endpoint (exact IP grant, project-only, 5s timeouts) and ADR-0033 in-process `ResourceProvider` v1 (registration, protocol/kind checks, name grant, mock read/write provider) ADR-0035 host-managed worker provider transport (framed JSON helper, ping/open/read/write/close, trusted-only) and ADR-0036 trusted C ABI provider (shared-library symbols, version/load checks) and ADR-0037 local provider-package manifest installation are tested. External DNS/TLS/UDP, host HTTP, provider packages/signatures and the plugin-Worker resource binding remain out of scope. |
|
||||
| P5 third-party Provider / TCP / HTTP | **PASS (local gate); external network/CI and plugin-Worker resource binding open** | C10: `stdlib/net/http.mq.md` + fixture 71 parse HTTP/1.1 status/headers/chunked bodies and build requests; `http_over_tcp.rs` runs a loopback HTTP exchange wholly in `.mq.md` over `resource.open("tcp",...)`. ADR-0034 IP-literal blocking TCP endpoint (exact IP grant, project-only, 5s timeouts) and ADR-0033 in-process `ResourceProvider` v1 (registration, protocol/kind checks, name grant, mock read/write provider) ADR-0035 host-managed worker provider transport (framed JSON helper, ping/open/read/write/close, trusted-only) and ADR-0036 trusted C ABI provider (shared-library symbols, version/load checks) ADR-0037 local provider-package manifest installation and ADR-0038 C ABI provider worker binding (separate trusted helper process) are tested. External DNS/TLS/UDP, host HTTP, provider packages/signatures and the plugin-Worker resource binding remain out of scope. |
|
||||
| P6 performance / native backends | **BASELINE ONLY** | reproducible profile harness `tools/perf/bench.py` and [`p6-profile`](evidence/2026-11-20-p6-profile.md) for 7 workloads; no bytecode/JIT/columnar numeric backend or equivalence/optimization gate yet |
|
||||
|
||||
The P1 memory domain is pure computation: it needs no grant and is available in
|
||||
@@ -267,6 +270,9 @@ user-written allocator is explicitly **not** claimed.
|
||||
echo tests.
|
||||
- [`evidence/2026-11-20-table-cross-storage.md`](evidence/2026-11-20-table-cross-storage.md)
|
||||
— P4 row-store Table vs `.mq.md` columnar store equivalence for sort/filter.
|
||||
- [`evidence/2026-11-20-c-worker-binding.md`](evidence/2026-11-20-c-worker-binding.md)
|
||||
— ADR-0038 C ABI provider worker binding, compiled C adapter in the helper
|
||||
process and fail-closed missing-library case.
|
||||
- [`evidence/2026-11-20-provider-package.md`](evidence/2026-11-20-provider-package.md)
|
||||
— ADR-0037 local provider package manifest, C library resolution and
|
||||
registration test.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# ADR-0038: C ABI provider worker binding v1
|
||||
|
||||
Status: **Accepted (trusted helper; existing plugin-Worker ABI binding still open)**
|
||||
Date: 2026-11-20
|
||||
Relates to: `docs/MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md`
|
||||
(CM-A09, CM-A11, P5), ADR-0035, ADR-0036
|
||||
Bin: `crates/marqdo-provider-worker`
|
||||
Evidence: `docs/evidence/2026-11-20-c-worker-binding.md`
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0036 loads a trusted C ABI provider in the host process; ADR-0035 provides a
|
||||
framed-JSON worker transport. CM-A09 also asks for the Worker side of the C ABI:
|
||||
the provider library should be loadable in a separate process so a crash does
|
||||
not take down the interpreter, without inventing a new language surface. This
|
||||
ADR adds the concrete binding.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Worker binary
|
||||
|
||||
`marqdo-provider-worker <c-abi-library>`:
|
||||
|
||||
* loads exactly one `c-abi-1` shared library (ADR-0036 symbols);
|
||||
* verifies `marqdo_resource_provider_abi_version() == 1` at startup;
|
||||
* serves the ADR-0035 framed-JSON protocol (`ping/open/read/write/close/
|
||||
shutdown`) over stdin/stdout;
|
||||
* maps C provider failures to `{"ok":false,"code":"host.error","message":...}`
|
||||
and never falls back to in-process loading.
|
||||
|
||||
The binary is a separate artifact, not a new top-level `marqdo` CLI command.
|
||||
Registration is still host-side:
|
||||
`StdHost::register_worker_resource_provider(name, kinds, worker_bin, [library])`
|
||||
from ADR-0035.
|
||||
|
||||
### 2. Process boundary
|
||||
|
||||
* The C library runs in the worker process; a crash/exit is a structured host
|
||||
error and the host does not retry in-process.
|
||||
* The worker has the host user's OS privileges; this is a **trusted transport**,
|
||||
not a sandbox and not proof of untrusted-provider safety.
|
||||
* The language still sees only `resource.*`; provider tokens stay inside the
|
||||
worker and the host `ResourceHandle` remains the language value.
|
||||
* Project mode + `allow_resource_provider(name)` are still required; pure/strict
|
||||
refuse provider kinds (CM-A11).
|
||||
|
||||
### 3. What remains for CM-A09
|
||||
|
||||
* Binding the provider contract through the **existing plugin-Worker ABI**
|
||||
(manifest/artifact ownership and version negotiation) rather than the
|
||||
dedicated helper.
|
||||
* Provider registry/download/signatures and package installation commands.
|
||||
* Untrusted-provider isolation and strict-mode providers.
|
||||
* ABI version negotiation beyond v1.
|
||||
|
||||
## Consequences
|
||||
|
||||
* A trusted C adapter can be crash-isolated without changing the runtime,
|
||||
parser, `Value` or `HostOp`.
|
||||
* The worker helper is small and reusable by a future provider package manager.
|
||||
* The trust model is explicit: worker isolation ≠ sandbox; the provider library
|
||||
remains trusted code.
|
||||
|
||||
## Verification
|
||||
|
||||
`docs/evidence/2026-11-20-c-worker-binding.md` records a compiled C provider,
|
||||
the worker helper, a full open/read/write/close round-trip through the host
|
||||
worker transport, and fail-closed startup with a missing library.
|
||||
@@ -34,6 +34,7 @@ implementation status lives in `docs/STATUS.md`; frozen behavior is in
|
||||
| CM-A09 worker resource-provider transport v1 | [0035](../adr/0035-worker-resource-provider-transport.md) | Accepted (trusted helper) |
|
||||
| CM-A09 C ABI ResourceProvider v1 | [0036](../adr/0036-c-abi-resource-provider.md) | Accepted (trusted native adapter) |
|
||||
| CM-A09 provider package manifest v1 | [0037](../adr/0037-provider-package-manifest.md) | Accepted (host-side install) |
|
||||
| CM-A09 C ABI provider worker binding v1 | [0038](../adr/0038-c-abi-provider-worker-binding.md) | Accepted (trusted helper) |
|
||||
| Table physical storage (T1/T2/T3), C/Worker provider transport, strict file/network resources | — | Not decided; do not implement by implication |
|
||||
|
||||
## What a future phase must not do
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# RFC-CM CM-A09 evidence: C ABI provider worker binding v1
|
||||
|
||||
Date: 2026-11-20 (Asia/Shanghai)
|
||||
Repository: `marqdo-cli`
|
||||
Baseline: `313ac03` plus this step's changes.
|
||||
Decision record: [ADR-0038](../adr/0038-c-abi-provider-worker-binding.md)
|
||||
Related: [C ABI](2026-11-20-c-provider.md), [worker transport](2026-11-20-worker-provider.md).
|
||||
Scope: trusted helper binding; not a sandbox and not the plugin-Worker ABI.
|
||||
|
||||
## Artifact
|
||||
|
||||
`marqdo-provider-worker <library>` loads a `c-abi-1` shared library and serves
|
||||
the ADR-0035 framed-JSON protocol (ping/open/read/write/close/shutdown). The
|
||||
host registers it through the existing
|
||||
`StdHost::register_worker_resource_provider` API; no runtime/`HostOp`/parser
|
||||
change was required.
|
||||
|
||||
## Tests
|
||||
|
||||
`crates/marqdo-provider-worker/tests/worker_binding.rs`:
|
||||
|
||||
```text
|
||||
c_abi_provider_runs_inside_the_worker_helper
|
||||
cc -shared C provider; worker binary loaded;
|
||||
resource.open/read "worker"/"-abi" eof/write/close through the host.
|
||||
|
||||
worker_helper_without_c_provider_fails_closed
|
||||
missing library -> structured host error, no in-process fallback.
|
||||
```
|
||||
|
||||
The C provider state lives in the child; the host only sees the opaque
|
||||
`ResourceHandle` and `ResourceRead` bytes.
|
||||
|
||||
## Boundary
|
||||
|
||||
The worker is a trusted helper with host-user privileges. It isolates crashes
|
||||
and keeps the C library out of the interpreter process, but it is **not** an OS
|
||||
sandbox. Pure/strict still refuse provider kinds; strict remains fail-closed.
|
||||
|
||||
## Not done
|
||||
|
||||
* Binding through the existing plugin-Worker manifest/artifact ABI.
|
||||
* Provider registry/signatures/install commands.
|
||||
* Untrusted-provider isolation, strict providers, Windows worker packaging.
|
||||
* ABI version negotiation beyond rejecting non-v1 libraries.
|
||||
Reference in new issue
Block a user