feat(host): add provider package manifest v1

- Add a marqdo.provider schema v1 loader (schema/version/ABI/platform library)
  and StdHost::register_provider_package, reusing the C ABI adapter and shared
  kind/name checks (ADR-0037).
- Add a compiled-provider + provider.json installation test; language use still
  requires project mode and allow_resource_provider.
- Update ADR/spec/status/README/changelog and evidence; no registry, signature
  or CLI install is claimed.

Gate: fmt, clippy -D warnings, 437 tests passed.
This commit is contained in:
cflmy committed 2026-11-20 23:28:30 +08:00
1 parent 5e5c3a84d9
commit 974102dfc3
11 files changed
+274 -5

No files matched your search

+7 -1
View File
@@ -136,6 +136,9 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
PluginHandle `plugin/export` are read-only observation members (host id
private); `stdlib/format.mq.md` reproduces builtin `打印`/`repr` for all 15
value types, including Formula/Code/FunctionRef and Code UTF-8 byte length.
- Provider package manifest v1 (CM-A09, ADR-0037): a local `marqdo.provider`
manifest resolves a trusted C ABI library and registers it; no registry,
signatures or CLI install yet.
- C ABI ResourceProvider v1 (CM-A09, ADR-0036): trusted shared libraries expose
open/read/write/close through the frozen header; version/load failures are
fail-closed; project-mode + name grant only. Plugin-Worker binding remains open.
@@ -218,7 +221,7 @@ behavior. This table records only what is in this tree now.
| P2 Marqdo-written allocator/data structures + minimal structure observation | **PASS (local pure lane); external stdlib repo publication pending** | `.mq.md`-only evidence: bump allocator with u32 codec, linked list, failed-allocation atomicity (48–50); CM-A04 structure primitives and a table filter (51–54, ADR-0023); a first-fit free-list allocator with block reuse, splitting, address-adjacent coalescing and `Result` maps (55, ADR-0024), a `Memory`-backed binary tree with recursive insert/in-order traversal (56), a `.mq.md` CSV row splitter (57), an open-addressing hash table with linear probing, update and `hash.full` (58), a `.mq.md` UTF-8 encoder over `Memory` (59), a validating UTF-8 decoder rejecting truncated, bad-continuation and surrogate/overlong input (60), a `Memory`-backed graph adjacency list (61), a recursive JSON parser for nested arrays/objects, strings with escapes, integers, booleans, null and error Results (62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (63), big-endian 16/32/64-bit plus LEB128 integer codecs (64), a `.mq.md` table sort rebuilding a new table from `table.row/columns/cell` (65), a table join (67), a quoted/escaped CSV row parser (68), table group (69), an Int matrix multiply (70), and a sparse-matrix multiply (73); the first-fit allocator also detects double free (`alloc.double_free`). Evidence: [`memory-p2-closure`](evidence/2026-11-20-memory-p2-closure.md), [`structure-api-p2`](evidence/2026-11-20-structure-api-p2.md), [`result-and-free-list-p2`](evidence/2026-11-20-result-and-free-list-p2.md). Local gate PASS: `crates/cli/tests/p2_closure.rs` runs 21 C01–C06/C11/C12 fixtures plus the stdlib table self-test in `--mode pure` (no foreign/plugin/import grants), scans sources for foreign/plugin/import shortcuts and compares value/stdout to goldens; evidence [`p2-closure-gate`](evidence/2026-11-20-p2-closure-gate.md). The bundled `stdlib/` tree, `std:` imports, digest verification, Analysis identity and frozen-CLI closure test are implemented (ADR-0028). Remaining outside C01–C12: publication as a separate external stdlib repository, and no remote CI URL. |
| P3 Resource/Stream ABI | **PARTIAL** | ADR-0025/0026 CM-A06 slices: opaque `Resource`, `resource.open/read/write/close`, stdout write in pure/project, granted project-file read and buffered project-file write in project, EOF/closed/handle errors, wire/host rejection; strict permits only stdio/stdout and refuses file/network resources. Fixture 66 + CLI/host tests. CM-A07 **Compat-C** is implemented: when a program calls `打印/print`, the loader injects the bundled `stdlib/io.mq.md` and the runtime routes to `io.写值` (all 15 value types, UTF-8 via `Memory`, `resource.write(stdout)`); `HostOp::Print` and the legacy `Value::display()` print path are removed, and a missing bundle fails with `kernel.unavailable`. Strict permits only the existing stdio console channel. Providers are registered through ADR-0033 (in-process) and ADR-0035 (worker helper). Still missing: directories, C ABI provider binding, provider packages, strict provider support. |
| P4 Table logical/physical migration | **PASS (local gate); T1/T2/T3 future** | ADR-0032/CM-A08: `Value::Table` semantics and GFM AST laziness unchanged; `stdlib/data/table.mq.md` provides select/filter/sort/append/set/join/group/`转GFM` as value-returning `.mq.md` functions with input immutability; zero Rust table algorithms were added. Cross-storage equivalence is pinned by fixture 72 (row Table vs `.mq.md` columnar store, identical headers/rows/cells). The optional T1/T2/T3 physical-storage/column-store migration was not required by the gate and is not implemented. |
| P5 third-party Provider / TCP / HTTP | **PASS (local gate); external network/CI and plugin-Worker resource binding open** | C10: `stdlib/net/http.mq.md` + fixture 71 parse HTTP/1.1 status/headers/chunked bodies and build requests; `http_over_tcp.rs` runs a loopback HTTP exchange wholly in `.mq.md` over `resource.open("tcp",...)`. ADR-0034 IP-literal blocking TCP endpoint (exact IP grant, project-only, 5s timeouts) and ADR-0033 in-process `ResourceProvider` v1 (registration, protocol/kind checks, name grant, mock read/write provider) ADR-0035 host-managed worker provider transport (framed JSON helper, ping/open/read/write/close, trusted-only) and ADR-0036 trusted C ABI provider (shared-library symbols, version/load checks) are tested. External DNS/TLS/UDP, host HTTP, provider packages/signatures and the plugin-Worker resource binding remain out of scope. |
| P5 third-party Provider / TCP / HTTP | **PASS (local gate); external network/CI and plugin-Worker resource binding open** | C10: `stdlib/net/http.mq.md` + fixture 71 parse HTTP/1.1 status/headers/chunked bodies and build requests; `http_over_tcp.rs` runs a loopback HTTP exchange wholly in `.mq.md` over `resource.open("tcp",...)`. ADR-0034 IP-literal blocking TCP endpoint (exact IP grant, project-only, 5s timeouts) and ADR-0033 in-process `ResourceProvider` v1 (registration, protocol/kind checks, name grant, mock read/write provider) ADR-0035 host-managed worker provider transport (framed JSON helper, ping/open/read/write/close, trusted-only) and ADR-0036 trusted C ABI provider (shared-library symbols, version/load checks) and ADR-0037 local provider-package manifest installation are tested. External DNS/TLS/UDP, host HTTP, provider packages/signatures and the plugin-Worker resource binding remain out of scope. |
| P6 performance / native backends | **BASELINE ONLY** | reproducible profile harness `tools/perf/bench.py` and [`p6-profile`](evidence/2026-11-20-p6-profile.md) for 7 workloads; no bytecode/JIT/columnar numeric backend or equivalence/optimization gate yet |
The P1 memory domain is pure computation: it needs no grant and is available in
@@ -264,6 +267,9 @@ user-written allocator is explicitly **not** claimed.
echo tests.
- [`evidence/2026-11-20-table-cross-storage.md`](evidence/2026-11-20-table-cross-storage.md)
— P4 row-store Table vs `.mq.md` columnar store equivalence for sort/filter.
- [`evidence/2026-11-20-provider-package.md`](evidence/2026-11-20-provider-package.md)
— ADR-0037 local provider package manifest, C library resolution and
registration test.
- [`evidence/2026-11-20-c-provider.md`](evidence/2026-11-20-c-provider.md)
— ADR-0036 C ABI ResourceProvider v1, compiled C adapter round-trip and
fail-closed load/version checks.
@@ -0,0 +1,77 @@
# ADR-0037: Provider package manifest v1
Status: **Accepted (host-side installation metadata)**
Date: 2026-11-20
Relates to: `docs/MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md`
(CM-A09, CM-A10, P5), ADR-0036
Specs: `spec/evaluation.md`, `spec/security.md`
Evidence: `docs/evidence/2026-11-20-provider-package.md`
## Context
ADR-0036 lets a trusted C shared library implement resource operations, but the
host still needed a code-level path to the library. CM-A09 asks for provider
installation/discovery without changing the parser/runtime. This ADR defines a
small, versioned **package manifest** so a provider distribution can be
installed by a host embedding and reused by future external provider
repositories.
## Decision
### 1. Manifest format
```json
{
"schema": "marqdo.provider",
"schema_version": 1,
"name": "cabi",
"kinds": ["cabi.sensor"],
"abi": "c-abi-1",
"library": {
"linux": "libcabi_provider.so",
"macos": "libcabi_provider.dylib",
"default": "libcabi_provider.so"
}
}
```
* file name: `provider.json` by convention; the host API takes an explicit
manifest path.
* `schema`/`schema_version` must be exactly `marqdo.provider`/`1`.
* `abi` must be `c-abi-1` (ADR-0036); future ABIs need a new ADR.
* library paths are relative to the manifest directory (absolute paths are
accepted only for explicit local tests); platform key is `linux`, `macos`,
`windows`, or `default`.
* unknown fields are tolerated by serde, but unknown schema/ABI are rejected.
### 2. Host API and authorization
* `StdHost::register_provider_package(manifest_path)` reads/validates the
manifest, loads the C ABI library and registers it through the same
kind/name validation as ADR-0033/0036.
* Registration does not authorize language use: `resource.open` still requires
project mode and `Capabilities.allow_resource_provider(name)`. Pure/strict
refuse provider kinds.
* Language imports cannot read or install a provider manifest. Installation is
a host/package-manager action (INV-06).
### 3. Trust boundary
* The manifest points at trusted native code loaded in-process; it is not a
sandbox and does not add signatures. A future provider repository must add
signatures/verification and a Worker binding.
* No network/package registry fetch is implemented here; the host passes a
local path.
## Non-goals
* Provider registry download, signatures, dependency resolution or semver.
* CLI install/uninstall commands (host embedding only).
* Untrusted-provider isolation or strict-mode provider support.
* Automatic platform selection beyond the `library` map.
## Verification
`docs/evidence/2026-11-20-provider-package.md` records the manifest round-trip
test: compile a C provider, write `provider.json`, `register_provider_package`,
then `resource.open/read`.
+1
View File
@@ -33,6 +33,7 @@ implementation status lives in `docs/STATUS.md`; frozen behavior is in
| P5 blocking TCP resource endpoint v1 | [0034](../adr/0034-tcp-resource-endpoint.md) | Accepted (IP-literal, no DNS/TLS) |
| CM-A09 worker resource-provider transport v1 | [0035](../adr/0035-worker-resource-provider-transport.md) | Accepted (trusted helper) |
| CM-A09 C ABI ResourceProvider v1 | [0036](../adr/0036-c-abi-resource-provider.md) | Accepted (trusted native adapter) |
| CM-A09 provider package manifest v1 | [0037](../adr/0037-provider-package-manifest.md) | Accepted (host-side install) |
| Table physical storage (T1/T2/T3), C/Worker provider transport, strict file/network resources | — | Not decided; do not implement by implication |
## What a future phase must not do
@@ -0,0 +1,45 @@
# RFC-CM CM-A09 evidence: provider package manifest v1
Date: 2026-11-20 (Asia/Shanghai)
Repository: `marqdo-cli`
Baseline: `5e5c3a8` plus this step's changes.
Decision record: [ADR-0037](../adr/0037-provider-package-manifest.md)
Related: [C ABI provider](2026-11-20-c-provider.md), [STATUS §5](../STATUS.md).
Scope: local manifest installation; no registry/signature/Worker binding.
## Manifest
```json
{
"schema": "marqdo.provider",
"schema_version": 1,
"name": "cabi",
"kinds": ["cabi.sensor"],
"abi": "c-abi-1",
"library": {"linux": "libecho_provider.so", "default": "libecho_provider.so"}
}
```
`StdHost::register_provider_package` validates schema/version/ABI, resolves the
platform library relative to the manifest, loads it through the ADR-0036 C ABI
adapter and registers it through the shared kind/name checks. Language use still
requires project mode and `allow_resource_provider("cabi")`.
## Test
`crates/marqdo-host/tests/c_provider.rs::provider_package_manifest_installs_a_c_abi_library`:
1. compile the C provider with `cc -shared`;
2. write `provider.json` next to it;
3. `register_provider_package(&manifest)` succeeds;
4. `resource.open("cabi.sensor", ...)` and `resource.read` return `c-`.
Existing C tests cover version mismatch, missing library and pure/strict refusal.
No parser/`Value`/runtime/`HostOp` change was needed.
## Not done
* Provider registry/download, signatures, dependency resolution, semver.
* CLI install/uninstall commands; installation is host embedding API only.
* Untrusted-provider isolation, strict-mode providers, plugin-Worker binding.
* Windows library build lane.