Files
marqdo-cli/docs/architecture/CURRENT.md
T

19 KiB

Marqdo CLI — current architecture

Status: current architecture overview. This file summarises the tree as it is today. It is not a release promise, does not reopen a frozen ADR, and does not replace spec/ or a recorded test log.

  • Baseline: 9c1723d44f403d6b742136f7881515083b413cc2 (2026-11-21, RFC-LANG-0002 implementation and follow-up fixes; language code baseline).
  • Normative behavior: spec/; frozen decisions: docs/adr/; executed evidence: docs/STATUS.md and docs/evidence/.
  • This is the single current overview. Do not add a second running "current state" file; update this file, docs/STATUS.md and an evidence file together.

Language contract and implementation. The language contract is RFC-LANG-0002 (adopted by ADR-0052) and spec/mapping.md. The frontend, marqdo-expression, IR lowering, runtime and bundled stdlib/ implement it. The workspace gate is green at 9c1723d; rollout evidence and historical banners are indexed in ../migration/README.md.

Extensibility closure (RFC-EXT-0001). The current tree implements the single static callable binding (ADR-0054), official stdlib public aliases replacing print magic (ADR-0058), third-party resource providers through the normal CLI (ADR-0055), the capability-handler registry (ADR-0057), and explicit extension selection with lock v4 (ADR-0056). The remaining PR-08 action is a SHA-bound frozen-release rerun of the external conformance suite on the tagged commit; the suite PASSes against the release build of this tree. Normative boundaries: spec/callable-resolution.md, spec/extension-selection.md, spec/resource-provider-v2.md.

1. Four state dimensions

A claim is stated in exactly one of these dimensions. "Exists in the tree" is not the same as "tested here", "published in the bundled stdlib" or "verified on a platform".

Dimension Current state Source
Implemented Rust workspace 0.1.0: RFC-LANG-0002 Markdown/frontend/IR, v2 statement parser, tree-walk runtime, Memory, capability host, Tool/Analysis/Trace, Plugin ABI, Foreign kernels, migrated bundled stdlib, CLI composition root Cargo.toml, crates/, stdlib/
Tested locally cargo test --workspace --locked: 599 passed, 0 failed across 67 targets on Linux x86_64 in the RFC-EXT-0001 tree; fmt and clippy clean. Historical baseline: 507 passed at 9c1723d. docs/STATUS.md, docs/evidence/2026-11-22-ext-implementation.md
Bundled stdlib released In-repo bundle marqdo-stdlib@0.1.0, SHA-256 digest verified before loading; it is not an external registry package stdlib/stdlib.json, stdlib/README.md
Multi-platform verified Linux local gate is the only executed lane here; Windows/macOS native lanes are UNVERIFIED, and the GitHub security lane remains a VM-only gate docs/STATUS.md §1, §4

2. Responsibility layers

user .mq.md programs / agents / research workflows
                     |
stdlib/*.mq.md (versioned and digest-checked in this repository)
  core / collections / text / encoding / data / math
  io / console / stream / fs / net / process
  memory / alloc + portable pure algorithms
                     |
language semantics: Markdown -> AST -> frontend -> IR
                     |
execution: Value / MemoryStore / evaluator / budgets
  (computes; does not own stdout or file handles)
                     |
controlled platform API: open/read/write/close + non-streaming requests
  capability / budget / audit / trace
  stdio / file / tcp / OS adapters
  explicitly registered providers (in-process / worker / C ABI)
                     |
OS / external services / devices

CLI is only the composition root: parse arguments, load the project,
assemble Host + Runtime, route output, exit with a status.
`project_prepare::prepare` is the single finalisation step: it loads sources,
reads static extension descriptors (never `dlopen`/spawn), builds the unique
`CallableRegistry`, binds every call site once and hands the same
`PreparedProgram` to `check`/`run`/`call`/`inspect`/Analysis. The runtime
dispatches only by the resulting `CallBindingTable`; it never re-resolves a
name or guesses a plugin.
Callable Plugin / Foreign kernel / Tool Protocol are separate extension
categories; they do not enter the standard library or the language IR.

Layer rules:

  1. The language core does not know stdout, a project directory, a TCP timeout or a third-party plugin path.
  2. The platform layer owns I/O, clocks, entropy, sockets and file handles. It must not own JSON parsing, text formatting or memory allocation policy.
  3. The stdlib is .mq.md and ships in this repository. New ordinary algorithms go to stdlib/ + tests + docs; they do not become Rust builtins.
  4. The CLI selects inputs, resolves configuration, assembles the host and routes human/machine output. It is not a semantic policy centre.
  5. Callable Plugin and Resource Provider share one registry/authorization/audit mechanism, but keep different capability types.
  6. CLI policy/commands are commands/{sandbox,run,check,inspect,init,package,tool}.rs; host open-path/providers/effects/dispatch are host/{resource_ops,providers,effects,dispatch}.rs; process execution is process.rs. These are child modules, not new crates or a second backend.

3. Dependency direction and public contracts

Current Rust dependency edges (documented, not proposed):

marqdo-cli -> marqdo-runtime -> marqdo-host
marqdo-cli -> marqdo-package, marqdo-sandbox, marqdo-frontend, ...
marqdo-runtime -> marqdo-ir, marqdo-expression, marqdo-value
marqdo-host   -> marqdo-plugin-abi, marqdo-foreign, marqdo-sandbox, marqdo-math

marqdo-runtime -> marqdo-host is the current direction. Do not add a second execution backend or a marqdo-stdlib crate by implication. A future embedded runtime or a second backend needs the thin ExecutionPort contract described in the review RFC, not a new crate per provider.

Public contract API (.mq.md) State Tests
stdio stdin resource.open("stdio","stdin",["read"]), resource.read, resource.close; one run-scoped cursor implemented, read-only resource_streams, p2_closure, tests/conformance
stdio stdout resource.open("stdio","stdout",["write"]), resource.write, resource.close; separate output budget implemented stdlib_bundle, print_routing, resource_streams
stdio stderr resource.open("stdio","stderr",["write"]); separate from stdout and CLI diagnostics implemented resource_streams, stdlib_product
project file read resource.open("file", path, ["read"]); path/grant checked; bounded implemented, project mode resource_streams, stdlib_product
project file write resource.open("file", path, ["write"]); buffers until close, never flush-on-drop implemented, project mode resource_streams, stdlib_product
std:fs/file 读文本/写文本/复制文件/存在/元数据/列出目录/创建目录/删除/重命名/追加文本/读取片段/遍历 over resource.*; bounded at 8 MiB implemented; no watch/binary offset read stdlib_product
resource.request `resource.request("fs.exists metadata list
std:data/json json.解析/解析多个/格式化; nested Value, 128-depth bound, recoverable errors implemented (recursive, multi-value; no streaming) stdlib_product
std:data/csv csv.解析行/解析/格式化行/格式化; quoted single/multi-row, CRLF, quoted newlines implemented (no streaming) stdlib_product
std:path path.连接/文件名/目录名/扩展名/规范/是否绝对; lexical only implemented (no I/O) stdlib_product
std:regex pure 匹配/搜索 subset; host 扩展匹配/捕获 with groups/alternation/captures implemented both paths; pure path has no groups resource_streams, stdlib_product
std:datetime datetime.格式化(毫秒) / datetime.解析(文本); UTC ISO-8601, pre-1970 supported implemented (pure) stdlib_product
std:clock clock.现在() -> resource.request("clock.now", []) implemented (project mode; pure/strict denied) resource_streams, stdlib_product
std:env env.读取/存在 -> resource.request("env.get", [name]); exact --allow-env grant implemented (project mode; no env.list) resource_streams, stdlib_product
std:process process.运行(程序, 参数列表) -> resource.request("process.run", ...); exact --allow-process grant, 5 s timeout, 1 MiB caps implemented (project mode; not an OS sandbox) resource_streams, stdlib_product
std:random random.字节/整数 -> resource.request("random.bytes", [count]); exact --allow-random grant implemented (project mode; byte-list RNG) resource_streams, stdlib_product
std:compress compress.压缩/解压 -> gzip byte list <-> Text; 8 MiB caps implemented (project mode; no grant) resource_streams, stdlib_product
network endpoints tcp/udp/tcp-listener/tls open + resource.accept; exact host grant, DNS after grant, TLS certificate validation implemented (project mode) resource_streams
std:net/http byte-exact HTTP/1.1 parse/build; chunk sizes are byte counts; text wrapper compatible implemented (pure .mq.md) stdlib_http
HTTP/1.1 stdlib/net/http.mq.md: status/header/chunked-byte parse, request build implemented over an already-authorized TCP stream; not a DNS/TLS client stdlib_http, http_over_tcp, stdlib_product
print / official stdlib aliases 打印/print are verified stdlib.json public aliases bound to io.mq.md::写值; ordinary Marqdo calls, no print_override, no runtime branch, no HostOp::Print implemented (ADR-0058) stdlib_alias_routing, print_routing, stdlib_product
callable resolution one registry + one bind_program_calls pass; check/run/inspect/Analysis share CallSiteKey bindings; plugin exports reconcile exactly with static manifests; no runtime name fallback implemented (ADR-0054) callable_resolution, plugin_worker, analysis_trace
resource provider v2 CLI --provider/--allow-provider select and authorize a provider.json schema-2 c-abi-1 worker artifact; manifest/artifact/grants/duplicate kinds verified before spawn; sibling helper only implemented linux/x86_64; other platforms fail closed (ADR-0055) provider_cli
capability handlers trusted handler registry executes granted capabilities under budget/audit; missing handler is capability.unavailable; fs.read is the first real handler implemented (ADR-0057) capability_handlers
strict mode only the pure/Math execution path and the built-in stdout console channel; project/foreign/plugin/tool/trace/file/network grants are rejected implemented L1; L2 not implemented tests/security, resource_streams, stdlib_product

4. Extension path classification

Extension path Responsibility State
Tool Protocol external marqdo CLI tools, inspect/verify/IDE commands; versioned manifest and install/remove implemented v1; not a standard-I/O replacement inside a user program
Callable Plugin ABI native callable functions and controlled experimental acceleration implemented v1 (native + worker); static manifest is the contract and run reconciles exports exactly; not an ordinary JSON/CSV formatter
Resource Provider new resource kinds/devices/network/file endpoints v2 CLI entry implemented for trusted worker transport on linux/x86_64 (ADR-0055); v1 in-process/trusted C ABI remain embedder APIs; not one provider per high-level algorithm
Foreign kernel explicit cross-language compatibility for Python/Node one-shot execution implemented; registry accepts an explicitly declared unknown language; not a Marqdo implementation of common algorithms
Official stdlib public alias language-surface aliases such as 打印/print declared in verified bundle metadata implemented (ADR-0058); third parties cannot claim these names
Extension selection / lock v4 marqdo.yaml.extensions activation sources; lock v4 manifest+artifact identity; inspect --extensions implemented (ADR-0056); declaration is never a grant, pure/strict refuse third-party activation
Shared registry/policy/budget/audit/trace package/adapter identity, descriptor/version checks, per-request grants, timeouts, error classes and trace metadata housed inside marqdo-host; do not invent a shared binary ABI for function calls and resource streams

5. Gap and closure backlog

This is the honest remaining work, ordered by the review RFC. It is not a claim that a phase is complete.

Priority Gap State
P0 Language syntax migration: RFC-LANG-0002 single contract closed at 9c1723d; implementation, stdlib digest, examples and fixtures green; platform/strict evidence remains separate
P0 Stdlib release is a bundle in this repo; extraction is deferred (ADR-0039) accepted, keep in repo
P0 Documentation drift between STATUS.md, the architecture index and the evidence being fixed by this migration and the docs gate
P1 std:fs/file read/write/copy, metadata/directory, append/read_at/walk are implemented; watch and binary offset reads remain open open
P1 No file watch or binary read_at open; future Resource Operation extension
P1 JSON/CSV product parsers are implemented with depth bound and multi-value/multi-row; no incremental streaming parser open
P1 CLI split into commands/{sandbox,run,check,inspect,init,package,tool}.rs; host split into host/{resource_ops,providers,effects,dispatch}.rs plus process.rs; commands.rs keeps shared project/runtime helpers and host.rs keeps the Host trait methods and capability checks split; shared helpers and trait methods remain by design
P2 HTTP/1.1 framing is byte-exact; HTTP/2, trailers and transfer decoding remain open open
P2 DNS, UDP, TCP listener and TLS client endpoints are implemented; no TLS server/mTLS or custom-CA CLI open
P2 std:path, pure+host regex, std:datetime, std:clock, std:env, std:process, std:random and gzip compression are implemented; no streaming/zstd/brotli open
P2 Memory limits: 16 MiB per region, 64 MiB per run current budget, not a product limit
P3 No concurrency tasks, atomics or cancellable long-running scheduling open; separate ADR
P3 No native optimizer/JIT or columnar numeric backend open; benchmark-first
P3 Windows Native/Worker ABI is unsupported; macOS/Windows CI is unverified open
P0 RFC-EXT-0001 PR-04: marqdo.yaml.extensions + lock v4 extension selection, CLI composition-root wiring, inspect --extensions implemented/tested; see spec/extension-selection.md, ADR-0056, extension_selection tests
P0 RFC-EXT-0001 PR-08: frozen release binary rerun on the tagged commit + SHA-bound evidence manifest remaining release step (tools/ci/verify_extension_contracts.py)
P1 Provider worker transport only linux/x86_64; macOS/Windows provider lanes fail closed open; separate platform work

6. Documentation governance

  1. There is one current overview (this file), one executed-state file (docs/STATUS.md), frozen decisions in docs/adr/, and fact snapshots in docs/evidence/. Do not copy a running status into several planning docs.
  2. A design input (a proposed RFC or a historical planning doc) must say Proposed or Historical in its own header. “Proposed” never means “implemented”.
  3. Every status row in docs/STATUS.md must name an executed command, a log or a source path. Do not hand-edit a gate result without an evidence update.
  4. Before committing a documentation change, run: cargo test -p marqdo-cli --test docs_consistency and git diff --check. A broken relative link or a stale “current” claim fails the gate.
  5. When behavior or status changes, update code + tests + docs/STATUS.md + this file + an evidence file in the same change. A fixture/demo is not a published stdlib API.

7. References