Compare commits

..
67 Commits
Author SHA1 Message Date
cflmy 33a14d272c release: v1.5.2 — foreign v3 P0/P1 (stdin, plan API, snapshot, capture, persist) 2026-10-18 14:57:15 +08:00
cflmy 6ed70fd8b8 docs(design): defer G7 and G8 from the v3 target
Keep the cross-document scope and non-Python/Node adapter designs in the document for future work, but remove them from the current implementation target and priorities.
2026-10-18 14:51:12 +08:00
cflmy 115d1f6c0b feat(view): explicit persist and nested-plan boundary
Add per-cell Persist backed by lib/writeback and /api/session/cell/persist. Reject nested foreign.session plans with a top-level diagnostic and tests.
2026-10-18 14:47:54 +08:00
cflmy 7dc5e56850 feat(foreign): reject nested view plans
Enforce the v3 plan boundary: foreign.session must be a top-level binding. Nested plans are diagnosed instead of silently becoming stateless. Tests cover the AST helper and DocumentSession open.
2026-10-18 14:46:57 +08:00
cflmy 514caa55ae docs(design): mark G9 snapshot capture implemented
view capture and view output --snapshot are implemented with source-hash validation; remaining gaps are G3/G7/G8/G10.
2026-10-18 14:46:20 +08:00
cflmy 74908ebf04 feat(view): capture live snapshots for static export
Add view capture to record per-cell outputs and session summary with source hashes. view output --snapshot consumes the capture without re-executing and rejects source drift. Includes snapshot parsing tests.
2026-10-18 14:45:26 +08:00
cflmy 0f852593bc docs(design): record foreign v3 P0/P1 implementation status
Mark stdin/per-cell inputs/interrupt and plan/snapshot APIs as implemented; keep G3/G7/G8/G9/G10 as pending design items.
2026-10-18 14:44:29 +08:00
cflmy 34e2fb4f74 feat(foreign): snapshot export, import and replay
Export JSON-safe kernel values, cells and journal; restore values into a matching plan; replay cells with recorded stdin. Add /api/session/snapshot/{export,import,replay}.
2026-10-18 14:43:53 +08:00
cflmy 0e407b2bd3 feat(foreign): runtime plan updates and generation
Add explicit add/remove cell and set-value operations with a plan generation counter. Expose /api/session/plan/{assign,add,remove,value}.
2026-10-18 14:43:26 +08:00
cflmy e5e532d83f fix(foreign): keep session_list arity empty
ForeignSessionList accepts no id; only ForeignSessionInterrupt takes id.
2026-10-18 14:38:27 +08:00
cflmy 064522a338 feat(foreign): raw stdin, per-cell inputs and interrupt
Persistent Python/Node kernels receive raw program stdin per eval; foreign.session accepts an inputs table and view exposes per-cell stdin. Add session interrupt host API and view control.
2026-10-18 14:37:45 +08:00
cflmy c85f1a2829 feat(foreign): support raw stdin and per-cell inputs
Session eval now carries raw stdin, and the Python/Node workers temporarily replace program stdin during evaluation. foreign.session accepts an inputs table; view exposes per-cell stdin and Run All sends a per-cell input map. Stateless and persistent cells share the same input semantics.
2026-10-18 14:36:47 +08:00
cflmy a7266185b2 docs(design): propose foreign v3 capability plan
Document the remaining foreign runtime gaps: persistent raw stdin, per-cell input, dynamic plan boundaries, runtime plan updates, interrupt, state export/replay, cross-document scope, non-Python/Node adapters, live-to-static snapshots and explicit writeback policy.
2026-10-18 14:31:44 +08:00
cflmy 4d7c0bea1b fix: sync Cargo.lock with v1.5.1 version bump 2026-10-18 14:23:20 +08:00
cflmy 53758ebd1c debian: changelog entry for 1.5.1-1ppa1 2026-10-18 14:20:24 +08:00
cflmy dff0b700ae release: v1.5.1 — view static/live unified + foreign value passing 2026-10-18 14:05:19 +08:00
cflmy 4da07c1b26 fix(view): do not turn documentation tables into call chips
Keep descriptive table labels non-callable and suppress call annotations on table blocks. This removes the collapsed call {步骤...} chip while preserving the table and the return 950 chips at the actual foreign/writeback calls.
2026-10-18 13:59:24 +08:00
cflmy 78922102e4 fix(view): anchor imported-function returns at their call site
Track the caller line on CallFrame and use it for return annotations instead of the callee library line. Return 950 stays visible on the actual foreign/writeback calls; the Markdown table remains intact and no longer receives stray return chips.
2026-10-18 13:56:50 +08:00
cflmy 064bae2c65 docs(view): keep writeback table labels as inline code
Wrap foreign.run/session and writeback.record table labels in code spans so the prose table is not traced as executable calls; this removes the stray return 950 chips.
2026-10-18 13:52:51 +08:00
cflmy 92b91ffc81 fix(view): group and theme writeback blocks
Render contiguous writeback comment lines as one block, and map output-card/writeback tokens to the View dark theme. Static and live pages no longer show three dashed fragments or light output cards on dark.
2026-10-18 13:51:11 +08:00
cflmy ab6648ae0e docs(view): make the writeback page itself the runnable example
Move the foreign + writeback demo into the user persistence page. The page runs a Python code fence, records a keyed total slot, and the slot is visible in both source and static rendering. Remove the separate example document.
2026-10-18 13:48:05 +08:00
cflmy 5acf026e0c docs(view): add a runnable foreign writeback example
Move persistence guidance out of the walkthrough into separate static pages. Add examples/foreign-writeback/index.mq.md, which executes a foreign code fence and writes a keyed total slot into its own source.
2026-10-18 13:45:27 +08:00
cflmy 8dcd69412f docs(view): recommend static snapshots and writeback persistence
Add a persistence recommendation to the user-facing foreign walkthrough: static export for published snapshots, and lib/writeback for opt-in result slots in the .mq.md source. Show a readable code-fence based writeback example.
2026-10-18 13:40:44 +08:00
cflmy 2a13e52981 docs(view): show low-level session evaluation as code fences
Replace inline source strings with readable Python code fences bound as Marqdo code values. The low-level session evaluates those values; no long escaped source argument remains.
2026-10-18 13:35:51 +08:00
cflmy 9b269f6447 feat(foreign): pass Marqdo values into multi-kernel view sessions
foreign.session accepts a values table and initializes each kernel from ordinary Marqdo values. The static walkthrough now uses two kernels and a shared 订单 value instead of redefining orders inside Python. Static pages embed the session summary so kernel controls and Clean stay visible.
2026-10-18 13:32:51 +08:00
cflmy 15362cb048 fix(view): restore per-cell kernel switch and Clean
Code cards again show a kernel selector and a Clean button. Clean calls /api/session/cell/clear for live sessions and clears local output in static snapshots, without restarting kernels.
2026-10-18 13:27:15 +08:00
cflmy 05d48f68a4 docs(view): show kernel state and Marqdo value passing
Rewrite the static foreign walkthrough without stdin-driven examples. It now demonstrates session_set from an ordinary Marqdo table, persistent kernel state across evaluations, and a foreign.session view plan sharing state across cells. Static export executes those cells and embeds the real outputs.
2026-10-18 13:23:03 +08:00
cflmy e4387259ba fix(view): make static examples self-contained
Foreign cell examples read stdin with a deterministic fallback, so static exports show real output without tracebacks. Low-level session bindings named 会话 are no longer mistaken for a view plan; static export runs all cells again.
2026-10-18 13:19:41 +08:00
cflmy 518f8e386b fix(view): avoid stdin tracebacks in static snapshots
Static export skips cells that read stdin when the document has no preset stdin, and shows a short explanation instead of a traceback. Preset stdin still executes normally.
2026-10-18 13:14:17 +08:00
cflmy 9da12c9093 feat(view): add cell stdin for live and static runs
View gets a Cell stdin/input field wired to cell Run, Step and Run All. Stateless cells receive raw stdin; persistent kernels receive marqdo_input. Static export uses the document preset stdin and the same run_all path.
2026-10-18 13:10:08 +08:00
cflmy e0d34669e0 fix(view): use the same run path for static and live cells
Run All now records per-cell failures and continues; static export uses the same foreign_doc run_all and injects those records. A cell that succeeds in view therefore succeeds in the static snapshot, and failures match.
2026-10-18 13:03:46 +08:00
cflmy 0ba4d3d2d7 feat(view): render real cell outputs in static exports
Static export opens the foreign document session, runs cells once, and injects stdout/stderr/error into the code cards. Failures are shown as real errors; --no-exec still skips execution.
2026-10-18 13:00:41 +08:00
cflmy ad3ebf243a fix(view): keep static buttons with a no-live notice
Static pages now render the same toolbar and Execution Clear button as live view. Clicking any execution control shows a static-export notice instead of hiding controls or pretending to run.
2026-10-18 12:55:07 +08:00
cflmy 142d30ce08 fix(view): share the same shell between static and live output
Static exports now include the View shell and dark theme; interactive toolbar and Execution Clear stay hidden, while dynamic pages show them.
2026-10-18 12:52:48 +08:00
cflmy 4757b82edd fix(view): run stateless documents and clear Execution output
No-session view Run All now runs the document and renders stdout into Execution; Clear outputs clears Execution output, plots and plan cards as well as cell outputs.
2026-10-18 12:51:13 +08:00
cflmy bff2f53e52 fix(view): clear Execution output from the view surface
Add an Execution Clear button and clear the Execution out/plots/plan card in addition to code-cell outputs. The same clearOutputs path calls /api/session/clear, so ordinary stateless pages and foreign sessions are both wired.
2026-10-18 12:47:56 +08:00
cflmy fb28eee62f feat(foreign): own view sessions and clear outputs
foreign.session returns a handle from ordinary kernel/cell tables; view consumes only the 会话 binding. Sessions live in host foreign_doc. Without 会话, view cells run stateless one-shot. Add /api/session/clear and a Clear outputs button for every view page; clearing keeps persistent kernel state.
2026-10-18 12:42:15 +08:00
cflmy 01468ad976 test(view): add foreign and non-foreign preview fixtures
Add a pure Marqdo control page and make the Python view-cells example explicitly import lib/foreign.
2026-10-18 12:22:36 +08:00
cflmy f3791d11b4 refactor(view): evaluate cell plans through the interpreter
Delete the source-line table scanner. 内核表/会话表 are now evaluated as ordinary Marqdo bindings (with simple assignment dependencies) via Interpreter::capture_declarations. Every view page uses the same cell toolbar; run/watch semantics are unchanged; no new CLI or Markdown marker.
2026-10-18 12:17:42 +08:00
cflmy 7fef690167 refactor(view): unify cells into the view surface
Cell controls now appear only when a document declares 内核表/会话表. Remove Notebook branding and Jupyter In/Out prompts; ordinary view pages stay classic. There is no new CLI command or second interactive mode; run remains canonical execution and view is the only interactive surface.
2026-10-18 12:03:17 +08:00
cflmy 024cdee82a debian: add 1.5.0 PPA changelog 2026-10-18 03:36:53 +08:00
cflmy f666eaaf40 release: v1.5.0
Foreign v2 + document-native notebooks, view contrast/themes/resize, public OKF/EKC and Python examples. Bump CLI/wasm/ext to 1.5.0; no core surface delta.
2026-10-18 03:17:53 +08:00
cflmy e6f05c7748 docs(public): showcase foreign v2, OKF and EKC
Rewrite the public foreign reference, add bilingual complex Python notebook and OKF/EKC pages, and expand both homepages with executable foreign, notebook, knowledge and preflight examples.
2026-10-18 03:15:34 +08:00
cflmy 81f9ea0072 fix(view): map document tokens to the active theme
Legacy view tokens (--ink, --fill, --line, --muted) now resolve from the notebook theme. This fixes dark text and white annotation chips on the dark STRUCTURE surface, including function names, binding labels, tables, and the right rail.
2026-10-18 03:05:29 +08:00
cflmy dbf3f3a14b fix(view): raise code contrast and unify syntax colors
Remove the two conflicting highlight.js stylesheets and define one high-contrast token palette. Dark code foreground is #f0f0f0 on #1e1e1e (14.6:1); light is #24292f on #f6f8fa (13.8:1). Keywords, strings, and comments stay above readable contrast, and base view code blocks are resizable too.
2026-10-18 03:03:08 +08:00
cflmy 654e89d98c style(view): make cell resize affordance visible
Style the native vertical resize grip and add a hover hint on code and output panes.
2026-10-18 02:59:36 +08:00
cflmy be20629496 feat(view): resizable cells and light/dark themes
Add native vertical resize to code and output panes; add a VS Code Light token set with Jupyter light syntax highlighting, a toolbar theme toggle, and localStorage persistence. Keep the restored three-column layout.
2026-10-18 02:59:21 +08:00
cflmy 3223295841 fix(view): restore notebook left nav and right rail
Keep the original three-column layout (file navigation, main structure, Functions/Variables rail) and execution/source sections; restyle them with the same VS Code dark surface tokens so the notebook remains readable and consistent.
2026-10-18 02:56:34 +08:00
cflmy beb5bdbe8c style(view): align notebook with VS Code dark and Jupyter cells
Flat VS Code Dark surfaces and borders, Jupyter In[n]/Out[] cell structure, unified top cell toolbar, consistent typography and spacing, removed gradients, glows, heavy shadows, and multi-color pills.
2026-10-18 02:51:28 +08:00
cflmy b8cb022018 style(view): redesign notebook layout and controls
Single-column notebook shell, sticky glass toolbar, card cells with language/kernel/status pills, output panel, dark-mode support, responsive layout, and hidden duplicate Execution/Source sections.
2026-10-18 02:45:10 +08:00
cflmy b9e331cb31 test(notebook): add Python business-insights example
Five-cell Python notebook sharing one kernel: deterministic order generation, regional/product metrics, z-score anomaly detection, ASCII report + JSON, and an isolated second Python kernel that validates the report through an explicit file bridge.
2026-10-18 02:38:48 +08:00
cflmy 476c2b607b fix(view): use relative API URLs for path-prefix reverse proxy
Notebook and debugger controls now call api/... and open debug?path=... relative to the page, so the same HTML works under a path prefix such as /marqdo-view/<token>/. This enables same-origin DSH preview without publishing another Docker port.
2026-10-18 02:33:50 +08:00
cflmy d99b275336 fix(parse): do not over-promote branch-local variables to params
If/else arms that all bind a local (e.g. # llm key/url/mdl/nm) no longer leak it into constructor params. This fixes the missing `key` error in llm.create and restores the agent/llm gold tests. Adds a regression test.
2026-10-18 02:09:16 +08:00
cflmy 1415aee08f docs(notebook): record D1–D3 implementation status 2026-10-18 01:45:41 +08:00
cflmy 0f2c12b636 feat(notebook): link statement debugger from view
Expose /debug and /api/debug/* on the live view server; add a notebook Debug button so Marqdo statement stepping is reachable from the same page/session as cell stepping.
2026-10-18 01:44:05 +08:00
cflmy 9873df7099 feat(notebook): reload, journal and state panel polish
D3 (partial): reload re-parses the document while preserving kernels, marks changed cells dirty; Journal and Reload view actions; filtered Node variable names for the state panel; per-session timeout parsing before spawn.
2026-10-18 01:41:36 +08:00
cflmy 3001ed7ba5 feat(notebook): document session with shared kernels and view controls
D1/D2: 内核表/会话表 parsing, DocumentSession cell index, named/default kernels, shared state + isolation, run_cell/step/run_all/restart/assign, state bridge, journal, POST /api/session/*, and a live-view notebook bar with status badges, per-cell kernel selector, step/run-all and push/pull state form.
2026-10-18 01:39:36 +08:00
cflmy 050d71927f feat(foreign): persistent Python/Node kernels and state bridge
F3/F3.5: session_open/eval/close/list, session_set/get/names, Python and Node bootstraps, strict serial protocol, per-session timeout, crash isolation, drop cleanup, and node/bash gold tests (python conditional).
2026-10-18 01:32:48 +08:00
cflmy 7859170aec feat(foreign): JSON call and parallel fan-out
F2: call/call_many JSON bridge, run_many ordered parallel execution with per-item isolation and concurrency cap, plus node gold tests for call and call_many.
2026-10-18 01:29:15 +08:00
cflmy e32ff8f6b3 feat(foreign): structured invoke, presets, probe and config
F0/F1: safe execution core (exec gate, bounded IO, process-group timeout), structured invoke records, run options, 15 language presets, register/unset/probe/config, view stderr/exit/ms, and node/bash gold tests.
2026-10-18 01:26:53 +08:00
cflmy c21a28c3bc docs(foreign): design kernel runtime and document session
Record the Jupyter-beyond discussion, decisions, and the document-native notebook syntax (内核表/会话表). Split the design into F (foreign kernel runtime) and D (document session) layers, with roadmaps, state bridge, stepping, journal, and security rules.
2026-10-18 00:43:31 +08:00
cflmyandCursor 1e61dca9d8 release: Marqdo 1.4.6
CLI and wasm only. Bare self-increment now promotes a required parameter, matching design §8.2. Extension pack stays 1.4.5.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 23:58:09 +08:00
cflmyandCursor ba675a4767 fix(parse): promote a bare self-increment to a required parameter.
Design §8.2 treats the first bold assignment as a declaration, but merging only walked backtick mentions, so n = n + 1 never entered the signature.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 23:55:30 +08:00
cflmyandCursor bfd468d5e3 release: Marqdo 1.4.5
Streamable HTTP MCP, web invoke Bearer/401, openai/agent json.get hardening.
Bump CLI/Ext/wasm and debian changelog.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 17:36:07 +08:00
cflmyandCursor 8353ba38fe Add Streamable HTTP MCP and harden web /v1 invoke auth.
agent_mcp_serve gains http transport; invoke injects Bearer/X-Api-Key and maps
unauthorized to 401; openai/agent field access prefers json.get over footnotes.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 17:33:24 +08:00
cflmyandCursor b43a08b2e2 release: Marqdo 1.4.4
Soft LLM transport errors and table-first ext/ai; CLI + Ext + wasm SemVer bump.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 14:38:16 +08:00
cflmyandCursor 1908bf1a53 fix(ai): soft LLM transport errors and strip ext/ai json glue
Host/plugin HTTP failures return structured finish|status=error instead of
aborting the VM; agent/llm/openai/智能体 result maps use GFM tables and
table.put. SKILL gains positive copy-paste examples so agents stop mirroring
legacy json.set chains.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 14:27:45 +08:00
83 changed files with 14310 additions and 815 deletions

No files matched your search

+55 -3
View File
@@ -38,7 +38,7 @@ A `.mq.md` file must stay readable as Markdown. If a human cannot skim the data
Cells are expressions (`doc/design/table-cell-expressions.md`): bare words = text, `` `var` `` = variable, quote paths/`/` ratios.
### 2. Good patterns
### 2. Good patterns (copy these — not `json.set`)
**Wire / steps / commands** — `@` record table:
@@ -60,7 +60,53 @@ Cells are expressions (`doc/design/table-cell-expressions.md`): bare words = tex
| `wire` | `set_text` |
```
**Single dynamic key** — one `table.put` (or a helper), not a json chain:
**Agent / LLM result map** — one readable row, not a `json.parse` + `json.set` chain:
```markdown
`out` =
| status | task | result | llm_calls | tokens |
|--------|------|--------|-----------|--------|
| ok | `task` | `reply` | `llm_calls` | `tokens` |
*out*
```
**Soft error pack** (transport / gate failure) — still a table humans can skim:
```markdown
`pack` =
| finish | text | error | usage | tool_calls |
|--------|------|-------|-------|------------|
| error | "" | `msg` | `usage0` | `empty_tc` |
*pack*
```
**Optional / conditional key** — sparse `table.put` after the static table (one dynamic key only):
```markdown
`out` =
| status | task |
|--------|------|
| error | `task` |
1. `err`
**out = > table.put in=`out` at="error" value=`err`**
2. *
**_ = 1**
```
**Empty list / empty map** — prefer stdlib seeds over `json.parse text=[]` / `{}`:
```markdown
**empty_tc = > table.empty_list**
**empty_map = > table.empty_map**
```
**Single dynamic key from scratch** — one `table.put` (or a helper), not a json chain:
```markdown
**`set_text` = table.put in=None at="#log" value=msg**
@@ -73,14 +119,20 @@ Better still — call a named helper:
**browser.set_text sel="#log" text=msg**
```
### 3. Bad pattern (do not write)
**When `json.*` is OK:** `json.parse` / `json.stringify` / `json.quote` on **wire text**; `json.get` to read an opaque map. Never use `json.set` / `json.append` to *build* author-facing structures.
### 3. Bad pattern (do not write — and do not copy from old `ext/ai`)
```markdown
**`ret` = json.set map=None key="set_text" value=…**
**`ret` = json.set map=ret key="canvas" value=…**
**`cmds` = json.append list=cmds item=c0**
**`out` = json.parse text={"status":"ok"}**
**`out` = json.set map=out key="task" value=task**
```
Matching surrounding **legacy** `json.set` in `ext/ai/*.mq.md` is **not** compliance. Refactor the call site to tables / `table.put` instead of extending the glue.
## Writing application `.mq.md`
1. Frontmatter: title + imports (`table`, domain libs). **Do not import `json` unless parsing strings.**
+11 -1
View File
@@ -32,7 +32,7 @@ Canonical design (repo): `doc/design/markdown-mapping.md`, `doc/design/keywords.
8. Prefer ending side-effect-only function bodies with `*None*` / `*无*` / whole-line `**` / `****` when later lines must belong to an outer function. Lone `---` / `***` are Markdown thematic breaks (skipped), not function end.
9. **`ext/**` never calls `host_*`.** Agent/OKF helpers are plugin names (`agent_kb_*`, …) after `plugin.load`. Do **not** add agent/OKF domain code to `src/host/` (core bloat). See `doc/design/ext-agent.md` §4.
10. **Browser (route C/D/E/F):** client logic is Marqdo on WASM; official bridge is host glue and **may** implement lists, routing, storage, WebSocket, canvas, file read, observers — authors **must not hand-write business JS**. Use `web.client_embed` / `data-mq-source-url` auto-mount. Client effects: prefer **GFM tables + `lib/browser`** ([marqdo-dev](../marqdo-dev/SKILL.md)).
11. **Code-as-documentation / no bag glue:** Prefer **GFM tables** for maps, lists, wire, commands. Prefer **`table.put` / named helpers** over `json.set` / `json.append` chains. `lib/json` is for parse/stringify/quote only — not a dict builder. Unreadable json pipelines are a style bug. Read collections with **`[key](map)`**, not footnote indexes for new code.
11. **Code-as-documentation / no bag glue:** Prefer **GFM tables** for maps, lists, wire, commands. Prefer **`table.put` / named helpers** over `json.set` / `json.append` chains. `lib/json` is for parse/stringify/quote only — not a dict builder. Unreadable json pipelines are a style bug. Read collections with **`[key](map)`**, not footnote indexes for new code. **Positive copy-paste shapes** (result map, soft-error pack, sparse `table.put`): [marqdo-dev](../marqdo-dev/SKILL.md) §2 · [examples.md](examples.md) §6e. Do **not** extend legacy `json.set` in `ext/ai` — refactor to tables.
12. **One language per file:** Do not mix EN and ZH API names in the same `.mq.md` (imports, calls, params). Pick English *or* Chinese mirrors for the whole program.
13. **Sandbox trial habit (Agent / generated `.mq.md`):** Prefer `agent.trial` / `run sandbox=True` / `sandbox.open`→`run`→`diff`→`commit` before promoting to the host tree. Host files must stay unchanged until a successful commit. Caps live in `sandbox.profile` GFM tables — do not invent a second dialect.
@@ -203,6 +203,16 @@ Install: `marqdo ext add quantum` (ZH id: `量子`). Build: `cargo build --relea
Design: [ext-quantum.md](../../doc/design/ext-quantum.md) · Q7: [ext-quantum-q7.md](../../doc/design/ext-quantum-q7.md) · Q8 viz: [ext-quantum-viz-style.md](../../doc/design/ext-quantum-viz-style.md) · examples: [quantum-bell](../../examples/quantum-bell/) · [quantum-entanglement](../../examples/quantum-entanglement/).
## Foreign kernels (v1.5.0)
`lib/foreign` is now a full foreign-kernel surface, not only `run`:
- `invoke` returns a record (`ok`, `code`, `stdout`, `stderr`, `ms`, `timed_out`, `truncated`, `result`); pass `input` for JSON stdin and `output="json"` for a parsed result.
- `call` is JSON-in / JSON-out; `call_many` fans out one program over a list with ordered results.
- `session_open` starts a persistent Python/Node kernel; `session_eval` evaluates source; `session_set` / `session_get` bridge values explicitly as JSON. Close it with `session_close`.
- A view document opts into kernel state with `会话 = > foreign.session kernels=... cells=...`; the table names are free. `view` consumes only the `会话` handle and never scans source lines or reserved table names. Without `会话`, cells are stateless one-shot runs; no kernel is created. Clearing outputs calls `/api/session/clear` and keeps persistent kernel state. There is no separate notebook CLI mode or command, and no new Markdown marker exists.
- The same source remains the authority; state crossing kernels is explicit, never implicitly mirrored.
## AI authoring workflow
1. Decide language surface (English builtins + `lib/text.mq.md`, or Chinese + `lib/文本.mq.md`).
+50
View File
@@ -162,6 +162,56 @@ import browser:lib/browser.mq.md
Do **not** build effect maps with `json.set` chains. See [marqdo-dev](../marqdo-dev/SKILL.md).
## 6e. Agent / LLM result maps — GFM first (positive)
Return and soft-error packs are **tables**, not `json.set` pipelines. Full style contract: [marqdo-dev](../marqdo-dev/SKILL.md) §2.
```markdown
---
import table:lib/table.mq.md
---
## step_ok
+ `task`
+ `reply`
`out` =
| status | task | result |
|--------|------|--------|
| ok | `task` | `reply` |
*out*
## step_err
+ `task`
+ `err`
`out` =
| status | task | error |
|--------|------|-------|
| error | `task` | `err` |
*out*
## soft_http_pack
+ `msg`
+ `usage0`
**empty_tc = > table.empty_list**
`pack` =
| finish | text | error | usage | tool_calls |
|--------|------|-------|-------|------------|
| error | "" | `msg` | `usage0` | `empty_tc` |
*pack*
```
Do **not** mirror legacy `ext/ai/agent.mq.md` `json.parse`+`json.set` chains when adding new arms — convert the shape to a table.
## 7. Import text stdlib
```markdown
+12
View File
@@ -126,6 +126,18 @@ jobs:
## Marqdo ${{ github.ref_name }}
### 亮点 / Highlights
- **v1.5.2**:外联 v3 P0/P1——raw stdin + 每 cell 输入 · plan API · snapshot · `view capture` · Persist
- **v1.5.2**: Foreign v3 P0/P1 — raw stdin + per-cell inputs, plan API, snapshot export/import/replay, `view capture`, Persist
- **v1.5.2 ext**:扩展包同步 1.5.2 / Ext pack 1.5.2
- **v1.5.1**:view 静态/动态统一 + 外联值传递——共用执行路径 · 构建时运行注入 stdout · `foreign.session` 值传递 · cell stdin
- **v1.5.1**: View static/live unified + foreign value passing — shared execution path, build-time run with stdout injection, `foreign.session` value passing, cell stdin
- **v1.5.0**:外联 v2 + 文档原生 Notebook——invoke/JSON call/并行/持久 Python·Node/状态桥 · cell 单步 · 高对比 view
- **v1.5.0**: Foreign v2 + document-native notebooks — invoke, JSON calls, fan-out, persistent Python/Node, state bridge, cell stepping, high-contrast view
- **v1.5.0 ext**:扩展包同步 1.5.0 / Ext pack 1.5.0
- **v1.4.6**:裸自增升参——首次粗体赋值左值即声明;`n = n + 1` 无反引号也是必选形参
- **v1.4.6**: Bare self-increment is a required parameter — first bold assignment LHS is a declaration
- **v1.4.5**:Streamable HTTP MCP + web invoke Bearer/401 · openai/agent json.get 硬化
- **v1.4.5**: Streamable HTTP MCP + web invoke Bearer/401; openai/agent json.get hardening
- **v1.4.2**:Agent/LLM 优化 O1–O5——sandbox trial · promote proof · step_route/EFI/cost · lint · quarantine
- **v1.4.2**: Agent/LLM optimize O1–O5 — sandbox trial, promote proof, step_route/EFI/cost, lint, quarantine
- **v1.4.1**:view 标注连线——SVG mid→mid(underline 底中 → chip 顶中)· 帧 chip id 对齐 · L→R 密度
+179
View File
@@ -8,6 +8,185 @@
### Changed
## v1.5.2 — 2026-10-18
### Highlights
**外联 v3 P0/P1**:session eval 支持 raw stdin(Python/Node 持久内核);每 cell 独立 stdin 与 Interrupt;plan generation/add/remove/value API;snapshot export/import/replay;`view capture` / `view output --snapshot` 捕获实时快照;显式 Persist 按钮。CLI · wasm · ext **1.5.2**。
```bash
git checkout v1.5.2 && cargo build --release
marqdo ext add web # CDN → 1.5.2
```
**核心表面 Δ(Core surface Δ)**:**无增删**——外联 v3 扩展 `lib/foreign` 库 API;不改 MQ 关键词 / 表几何。
### Added
- **raw stdin + 每 cell 输入**:session eval 支持 raw stdin;`foreign.session inputs=` 每 cell 独立 stdin;view 增加每卡 stdin 字段与 Interrupt 按钮。
- **plan API**:plan generation/add/remove/value;运行时 plan 更新与 generation。
- **snapshot**:snapshot export/import/replay;`view capture` / `view output --snapshot` 捕获 live 快照用于静态导出。
- **显式 Persist**:Persist 按钮 + `/api/session/cell/persist`。
- **嵌套 plan 边界**:显式 persist 与嵌套 plan 边界;foreign 拒绝嵌套 view plans。
### Fixed
- **`session_list` arity**:保持空 arity。
### Changed
- CLI / wasm / `ext/VERSION` **同步 1.5.2**。
## v1.5.1 — 2026-10-18
### Highlights
**view 静态/动态统一 + 外联值传递**:静态导出与 live view 共用同一套 `foreign_doc` 执行路径;构建时实际运行代码单元并注入 stdout;`foreign.session` 可把 Marqdo 值传入多内核会话;cell stdin 支持 live/static/持久内核三种模式。CLI · wasm · ext **1.5.1**。
```bash
git checkout v1.5.1 && cargo build --release
marqdo ext add web # CDN → 1.5.1
```
**核心表面 Δ(Core surface Δ)**:**无增删**——view 执行路径统一 + 外联值传递;不改 MQ 关键词 / 表几何。
### Added
- **外联值传递**:`foreign.session kernels= cells=` 可把 Marqdo 值传入多内核 view 会话;`view` 只消费 `会话` 绑定,不再识别 `内核表` / `会话表` 保留名。
- **Cell stdin**:view 增加 `Cell stdin / input` 输入框;Run/Step/Run All 都会传给外联 cells;无状态 cell 使用 raw stdin,持久内核使用 `marqdo_input`;静态导出使用 frontmatter `stdin` 作为构建时输入。
- **静态导出真实执行**:构建时实际执行代码单元并把 stdout/stderr 注入代码卡;静态示例通过代码 fallback 或 frontmatter `stdin` 提供输入。
- **可运行外联写回示例**:新增 `examples/foreign-writeback/index.mq.md`,用 `foreign.run` + `writeback.record` 将结果写入 `.mq.md` 的 keyed 槽位。
- **写回块分组与主题**:view 对写回块按目标分组并统一主题样式。
### Fixed
- **code 卡控制恢复**:恢复每个代码卡的内核切换下拉与 Clean 按钮;Clean 清除单个 cell 输出/记录,不重启内核。
- **导入函数返回锚定**:导入函数的返回值锚定在调用位置。
- **静态导出自包含**:静态示例不再依赖空 stdin 或外部内核状态。
- **Execution 输出清除**:view 清除 Execution 输出从 view 表面。
- **文档表不再变 call chips**:view 不再把文档表转为 call chips。
- **写回表标签**:写回表标签保持为行内代码。
### Changed
- **静态/动态统一执行路径**:static 与 live cells 使用同一 `foreign_doc` `run_all` 路径;移除 `In []` / `Out []` 与 Notebook 品牌。
- **静态按钮行为**:静态页面执行按钮点击显示"静态页面无法真实执行",不伪造成功。
- **低层会话评估可见**:文档展示低层 session evaluation 为代码围栏。
- CLI / wasm / `ext/VERSION` **同步 1.5.1**。
## v1.5.0 — 2026-10-18
### Highlights
**外联 v2 / 文档原生 Notebook**:`lib/foreign` 从“运行一段代码”升级为完整外联内核:结构化 `invoke` 记录、JSON 进出的 `call` / `call_many`、并行扇出、Python/Node 持久内核、显式状态桥,以及用绑定表声明的 Notebook 单元。`view` 增加 cell 单步、Run All、内核切换、journal、状态面板、浅/深主题与可调高度代码区;高对比度 token 修复了代码和 STRUCTURE 文字与背景区分度不足的问题。CLI · wasm · ext **1.5.0**。
```bash
git checkout v1.5.0 && cargo build --release
```
**核心表面 Δ(Core surface Δ)**:**无增删**——外联 v2 只扩展 `lib/foreign` 的库 API;Notebook 的 `内核表` / `会话表` 是既有绑定表;没有新增 Markdown 标记。
### Added
- **用户静态站(外联 / OKF / EKC)**:新增中英双语 `features/19-foreign-view-cells` 与 `features/20-okf-ekc`,重写 `stdlib/foreign`,首页加入「一份文件三个面」、外联 Python 运行、Notebook 表、OKF/EKC 与 preflight 说明。
- **`lib/foreign` F0/F1**:`invoke` 结构化记录(`ok`/`code`/`stdout`/`stderr`/`ms`/`timed_out`/`truncated`/`result`);`run`/`run_lang` 新增 `env`/`cwd`/`timeout`/`args`/`max_output`;15 个语言预设;`probe`/`register`/`unset`/`load_config`;`.marqdo/foreign.toml`。
- **`lib/foreign` F2**:JSON 进 / JSON 出 `call`;并行 `run_many` / `call_many`(有序、单项隔离、并发上限);`invoke` 的 `input` / `output` / `result` / `result_error`。
- **`lib/foreign` F3/F3.5**:Python / Node 持久会话 `session_open` / `session_eval` / `session_close` / `session_list`;状态桥 `session_set` / `session_get` / `session_names`;会话超时即杀、崩溃隔离、drop 清理。
- **Notebook D1/D2**:`内核表` / `会话表` 文档原生语法;`DocumentSession`(cell index / 命名内核 / journal / value store);view Notebook 工具栏(Run All / Restart & Run All / Step / State / Journal / Reload)、状态徽标、内核下拉、Push/Pull 状态表单;`POST /api/session/*`。
- **Notebook D3**:`reload` 重新解析并保留内核,源码变更的 cell 标为 `dirty`;Journal/Reload/Debug 按钮;Debug 打开同一 view 服务的语句级单步页(`/debug` + `/api/debug/*`);状态桥 Push/Pull。
- **安全与健壮性**:所有外联入口检查 `exec` 能力;有界 stdout/stderr 读取与截断标记;stdin 写线程;Unix 进程组 / Windows `taskkill /T` 超时清理;shell-like argv 切分。
### Fixed
- **view 文档表面与标注 chip**:把旧 view 的 `--ink` / `--fill` / `--line` 等 token 映射到当前主题,修复 STRUCTURE 中 `main`、`内核表 =`、函数名和 annot-chip 深色文字/白色卡片与暗色背景不一致的问题。
- **view 代码对比度**:移除会互相覆盖的 highlight.js 深/浅两套外链样式,改用内置高对比 token 变量;深色代码正文 `#f0f0f0` on `#1e1e1e`(14.6:1),浅色 `#24292f` on `#f6f8fa`(13.8:1);关键词/字符串/注释均使用可读的对比色。
- **view 反向代理兼容**:Notebook / Debug 的 API 与 Debug URL 改为相对路径,可在 `/marqdo-view/...` 路径前缀下工作,便于 Docker 内通过 DSH 同源预览。
- **分支内共同局部变量**:if/else 各臂都赋值的局部变量不再被误提升为必选形参;修复 `# llm` 构造器无端要求 `key`,恢复 agent/llm/ext 全量测试。新增 `branch_else_common_local_is_not_a_param` 回归测试。
- **外联临时脚本路径**:入口目录为相对路径时,脚本参数现在使用绝对路径,避免 `tests/lib/tests/lib/...` 双拼接。
- **外联失败可观测**:解释器不存在时进入 `invoke` 记录,`run` 仍报错;view 显示 stderr / exit / ms。
### Changed
- **Notebook view 视觉重做**:采用 VS Code Dark 的扁平色板与 Jupyter 的 cell 组织:单一深色底、明确 surface/border 层级、`In [n]` / `Out [ ]` 提示、顶部 cell 工具栏、无渐变/无重阴影、暗色代码区与一致的字号/间距;保留并统一原始左侧文件导航、右侧 Functions/Variables 栏与 Execution/Source 区块;支持深色/浅色主题切换(localStorage 记忆)与代码区、输出区纵向拖拽调整高度。
- **`langs`** 现在返回「预设 ∪ 注册 ∪ 配置」的已知语言(原集合是子集)。
- 大输出从 `Err` 改为「截断 + `truncated=true`」;`subtask` 外联输出同步采用该语义。
## v1.4.6 — 2026-10-16
### Highlights
**裸自增升参**:粗体首次赋值的左值按设计 §8.2 视为声明。`**n = n + 1**` 在没有反引号提及时也会把 `n` 升成必选形参;缺参报 `missing argument for parameter`,不再是函数体内的 `undefined variable`。纯局部赋值(`**score = 1**`)与循环变量不升参。CLI · wasm **1.4.6**(扩展包仍为 **1.4.5**)。
```bash
git checkout v1.4.6 && cargo build --release
```
**核心表面 Δ(Core surface Δ)**:**无增删**——升参与 §8.2 对齐(赋值左值声明);不改 MQ 关键词 / 表几何。
### Fixed
- **升参**:`merge_inferred_params` 纳入首次粗体赋值左值。右值在绑定前读取同名(`n = n + 1`)时,该名成为必选推断形参。
### Changed
- CLI / wasm SemVer **1.4.6**。扩展包保持 **1.4.5**。
## v1.4.5 — 2026-10-16
### Highlights
**Streamable HTTP MCP + web `/v1` 鉴权硬化**:`agent_mcp_serve` 支持 `transport=http`(MCP Streamable HTTP 2025-03-26);web invoke 注入 `Authorization` / `X-Api-Key`,`ok=false`+`unauthorized` → HTTP 401;`ext/ai` openai/agent 字段访问改 `json.get`,消除 footnote 硬崩。CLI · Ext · wasm **1.4.5**。
```bash
git checkout v1.4.5 && cargo build --release
cargo build --release -p marqdo_plugin_agent
# HTTP MCP:
# QDAGENT_MCP_TRANSPORT=http marqdo run cli/求道-mcp.mq.md
marqdo ext add agent # CDN → 1.4.5
```
**核心表面 Δ(Core surface Δ)**:**新增** MCP Streamable HTTP 传输(`transport=http|streamable-http`);web invoke 契约扩展(header 注入 · 401);**无** MQ 关键词 / 表几何变更。
### Added
- **`agent_mcp_serve` Streamable HTTP**:`transport=http` · `host`/`port`/`path`;POST JSON-RPC + 可选 GET SSE;`Mcp-Session-Id`。facade:`ext/ai/agent` `## serve`。
### Fixed
- **web invoke**:`Authorization` / `X-Api-Key` 注入 payload;`jsonWrapInvokeValue` 对 `unauthorized` 返回 401;`{sse}` 自动 event-stream。
- **`ext/ai/llm/openai` · `agent` · `llm`**:字段访问改 `json.get` / `at`,避免 `[field](x)` 在非 map 上硬崩。
- **explore workbook**:子任务 `stream=False`(网关 SSE 524 更稳)。
### Changed
- CLI / Ext / wasm SemVer **1.4.5**。
- 设计:[doc/design/ext-web-artifact.md](doc/design/ext-web-artifact.md)(sse / bearer)。
## v1.4.4 — 2026-10-16
### Highlights
**LLM 传输软失败 + `ext/ai` 表优先重构**:上游连接拒绝/超时/HTTP≠200 时,Runtime 返回结构化 `finish|status=error`(不再硬中断 VM);`agent` / `智能体` / `llm/openai` 去掉 `json.set`/`json.append` 袋胶水。含 CLI host · `plugins/llm` · Ext pack **1.4.4**。
```bash
git checkout v1.4.4 && cargo build --release
cargo build --release -p marqdo_plugin_llm
marqdo run tests/ext/agent-llm-transport-soft.mq.md
marqdo ext add agent # CDN → 1.4.4
```
**核心表面 Δ(Core surface Δ)**:**无增删**——HTTP 宿主软失败契约 + Extension 风格重构;不改 MQ 关键词 / 表几何。
### Fixed
- **LLM / Agent 传输软失败**:`host_http_post` / SSE · `plugins/llm` · `ext/ai/llm/openai` 结构化 `finish=error`;`agent.step` / `plan` / `run` 收成 `status=error`。金样 `tests/ext/agent-llm-transport-soft.mq.md`。说明:[doc/research/agent-llm-transport-soft-2026-10.md](doc/research/agent-llm-transport-soft-2026-10.md)。
- **`agent.run`**:`cache == "soft-hit"` 引号比较(裸 `soft-hit` 曾被解析为减法)。
### Changed
- **`ext/ai` 去 json 胶水**:结果 map / 事件用 GFM 表 + `table.put` / `table.append` / `table.empty_*`。SKILL 正面示例:`.cursor/skills/marqdo-dev` §2 · `marqdo/examples.md` §6e。
- CLI / Ext / wasm SemVer **1.4.4**。
## Ext pack v1.4.3 — 2026-10-16
### Highlights
Generated
+2 -2
View File
@@ -572,7 +572,7 @@ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "marqdo"
version = "1.4.2"
version = "1.5.2"
dependencies = [
"anyhow",
"assert_cmd",
@@ -603,7 +603,7 @@ dependencies = [
[[package]]
name = "marqdo-wasm"
version = "1.4.2"
version = "1.5.2"
dependencies = [
"marqdo",
"serde_json",
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "marqdo"
version = "1.4.2"
version = "1.5.2"
edition = "2021"
license = "Apache-2.0"
description = "Marqdo — Markdown markers as an interpreted programming language"
@@ -19,7 +19,7 @@ cli = ["dep:clap"]
view = ["dep:tiny_http"]
net-host = ["dep:ureq", "dep:idna_adapter", "dep:zeroize", "dep:zip"]
plugin-host = ["dep:libloading"]
exec-host = []
exec-host = ["dep:libc"]
fs-host = []
tty = ["dep:libc"]
wasm-core = []
+11 -6
View File
@@ -149,9 +149,14 @@ marqdo catalog public -o .marqdo
---
## 现状(v1.4.2 · Ext pack **1.4.3**)
## 现状(v1.5.2)
- **Ext pack 1.4.3**:修复 `agent.run` cold EFI(`` `self`.preflight `` + miss resolve `llm_free`);`marqdo ext add agent` → CDN latest · [agent-run-efi-resolve-bugs-2026-10.md](doc/research/agent-run-efi-resolve-bugs-2026-10.md)
- **外联 v3 P0/P1(v1.5.2)**:raw stdin + 每 cell 独立输入 · plan API · snapshot export/import/replay · `view capture` · Persist 按钮 · 扩展包同步 **1.5.2**
- **view 静态/动态统一 + 外联值传递(v1.5.1)**:静态导出与 live view 共用 `foreign_doc` 执行路径 · 构建时运行注入 stdout · `foreign.session` 值传递 · cell stdin;扩展包同步 **1.5.0**
- **裸自增升参(v1.4.6)**:首次粗体赋值左值即声明;`**n = n + 1**` 无反引号也升为必选形参。扩展包仍为 **1.4.5**
- **Streamable HTTP MCP + web invoke 鉴权(v1.4.5)**:`agent_mcp_serve transport=http` · Bearer/`X-Api-Key` 注入 · `unauthorized`→401 · openai/agent `json.get` 硬化
- **LLM 传输软失败 + `ext/ai` 表优先(v1.4.4)**:上游不可达 → `status=error`(不 abort);`agent`/`llm` 去 `json.set` 胶水 · [agent-llm-transport-soft-2026-10.md](doc/research/agent-llm-transport-soft-2026-10.md)
- **Ext pack 1.4.3**:修复 `agent.run` cold EFI(`` `self`.preflight `` + miss resolve `llm_free`);已并入 **≥1.4.4** · [agent-run-efi-resolve-bugs-2026-10.md](doc/research/agent-run-efi-resolve-bugs-2026-10.md)
- **Agent/LLM 优化 O1–O5(v1.4.2)**:`ext/ai/sandbox` CoW · `agent.trial` / `sandbox=` · promote sandbox proof · `step_route` / EFI / cost · style lint · quarantine · [ext-ai-optimize.md](doc/roadmap/ext-ai-optimize.md)
- **view 标注连线(v1.4.1)**:underline→chip SVG mid→mid(底边中点→轨道→短竖线→顶边中点)· 帧 chip 共用 id · L→R 密度 · [view-prose-annot-gaps.md](doc/design/view-prose-annot-gaps.md)
- **叙述统一 / 公式 meta / 图片自写回(v1.4.0)**:`% marqdo: var=` 块公式绑定 · 臂/循环体叙述与 foreach 中英糖 · `![marqdo-out](…)` 图片自写回 · view 推断形参实值 · [fence-meta-and-control-prose.md](doc/design/fence-meta-and-control-prose.md) · [stdlib-writeback.md](doc/design/stdlib-writeback.md)
@@ -172,7 +177,7 @@ marqdo catalog public -o .marqdo
- **`web`**:**Artifact 模型(ADR [0007](doc/adr/0007-web-document-endpoint.md))** — `type: web|endpoint` Document/Endpoint;facade `page`/`route`/`serve`/`render`/`inspect`/`use`;Resource 在 `ext/data` · `ext/security` · `ext/net`;设计 [ext-web-artifact.md](doc/design/ext-web-artifact.md);原生 **Go `libweb`**([ADR 0004](doc/adr/0004-web-plugin-go.md));示例 [marqdo-blog](examples/marqdo-blog/) · [web-client-site](examples/web-client-site/) · [anlian-mq](examples/anlian-mq/)
- **`quantum`**:Q0–Q7 + Q8a/b 主题 SVG;示例 [quantum-entanglement](examples/quantum-entanglement/)
- **`linalg`**:L0–L6 + 公式文档面(中缀 / `declare` / 展示化简);示例 [linalg-transpose](examples/linalg-transpose/) · [linalg-svd](examples/linalg-svd/) · [linalg-least-squares](examples/linalg-least-squares/)
- **`agent`**:A1–A4 + **MCP Server/Client stdio** + Skill Compilation + Adaptive Routing + resume + **Engineering preflight**(`agent.preflight` / `eng_preflight`);宿主缺口见 [ext-hosting.md](doc/roadmap/ext-hosting.md) · [agent-framework-gaps-after-a4.md](doc/research/agent-framework-gaps-after-a4.md) · [engineering-knowledge.md](doc/design/engineering-knowledge.md)
- **`agent`**:A1–A4 + **MCP Server/Client**(stdio + **Streamable HTTP**)+ Skill Compilation + Adaptive Routing + resume + **Engineering preflight**(`agent.preflight` / `eng_preflight`);宿主缺口见 [ext-hosting.md](doc/roadmap/ext-hosting.md) · [agent-framework-gaps-after-a4.md](doc/research/agent-framework-gaps-after-a4.md) · [engineering-knowledge.md](doc/design/engineering-knowledge.md)
- **`llm`**:语义 `ask`/`stream`/`collect`(OpenAI/Ollama;可选原生 `plugins/llm`)
- 安装:`marqdo ext list` / `add …` / `remove`([ext-cli.md](doc/design/ext-cli.md);默认 `~/.marqdo/ext`)。原生插件先 `cargo build -p marqdo_plugin_*` 再 `ext add`
- **原生插件 ABI**:[`include/marqdo_abi.h`](include/marqdo_abi.h) · [ext-abi.md](doc/design/ext-abi.md);`plugins/{demo,agent,web,quantum,linalg,llm}`(`web` 为 Go;其余为 Rust)
@@ -181,7 +186,7 @@ marqdo catalog public -o .marqdo
- **浏览器 Marqdo(WASM)**:`marqdo wasm build` + 官方 bridge 自启(作者零业务 JS;桥内可含列表/路由/storage/ws/文件/Canvas/音频/Observer/拖放);`lib/browser` + GFM;`web.client_embed`;示例 [browser-hello](examples/browser-hello/) · [browser-app](examples/browser-app/) · [browser-media](examples/browser-media/) · [web-client-site](examples/web-client-site/)([ADR 0002](doc/adr/0002-browser-marqdo-wasm.md) · [D](doc/roadmap/browser-wasm-d.md) · [E](doc/roadmap/browser-wasm-e.md) · [F](doc/roadmap/browser-wasm-f.md))
- 选型:[ADR 0001 — Rust](doc/adr/0001-implementation-language.md) · [ADR 0002 — 浏览器 WASM](doc/adr/0002-browser-marqdo-wasm.md)(C0–C5 完结,见 [roadmap/browser-wasm.md](doc/roadmap/browser-wasm.md))· [ADR 0003 — 异步效应](doc/adr/0003-browser-async-effects.md)
### 如何使用最新 Marqdo(v1.4.2)
### 如何使用最新 Marqdo(v1.5.2)
**安装解释器(任选其一)**
@@ -196,7 +201,7 @@ marqdo catalog public -o .marqdo
```bash
# 1) 源码安装(跟 tag 或 main)
git clone https://github.com/cflmy/marqdo.git && cd marqdo
git checkout v1.4.2 # 或留在 main
git checkout v1.5.2 # 或留在 main
cargo build --release
export PATH="$PWD/target/release:$PATH"
marqdo version
@@ -235,7 +240,7 @@ marqdo wasm build
# → dist/wasm/ … 见 examples/browser-hello/
# 6) Releases:Windows exe/zip/vsix;Linux CLI zip + native `.so` zip
# https://github.com/cflmy/marqdo/releases/tag/v1.4.2
# https://github.com/cflmy/marqdo/releases/tag/v1.5.2
```
开发期也可用 `cargo run -- …` 代替已安装的 `marqdo`:
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "marqdo-wasm"
version = "1.4.2"
version = "1.5.2"
edition = "2021"
license = "Apache-2.0"
description = "Marqdo interpreter as WebAssembly (browser host bridge)"
+52
View File
@@ -1,3 +1,55 @@
marqdo (1.5.2-1ppa1) noble; urgency=medium
* Marqdo 1.5.2: Foreign v3 P0/P1 — raw stdin with per-cell inputs and
interrupt, plan generation/add/remove/value API, snapshot
export/import/replay, view capture for static export, explicit Persist
button, nested plan boundary.
-- cflmy <pingan@cflmy.cn> Sun, 18 Oct 2026 15:00:00 +0800
marqdo (1.5.1-1ppa1) noble; urgency=medium
* Marqdo 1.5.1: View static/live execution path unified; foreign value
passing into multi-kernel sessions; cell stdin for live/static/persistent
kernels; build-time execution with stdout injection in static exports;
runnable foreign writeback example; writeback block grouping.
-- cflmy <pingan@cflmy.cn> Sun, 18 Oct 2026 22:00:00 +0800
marqdo (1.5.0-1ppa1) noble; urgency=medium
* Marqdo 1.5.0: Foreign kernels v2 (structured invoke, JSON calls,
parallel fan-out, persistent Python/Node kernels, explicit state
bridge) and document-native notebooks in view; high-contrast
light/dark view themes, resizable code/output panes, OKF/EKC and
Python examples on the user site. Fixes branch-local variables
leaking into required constructor parameters.
-- cflmy <pingan@cflmy.cn> Sun, 18 Oct 2026 03:30:00 +0800
marqdo (1.4.6-1ppa1) noble; urgency=medium
* Marqdo 1.4.6: Promote a bare self-increment to a required parameter.
The first bold assignment LHS is a declaration (design §8.2); a missing
argument reports missing argument for parameter, not undefined variable.
-- cflmy <pingan@cflmy.cn> Fri, 16 Oct 2026 23:56:00 +0800
marqdo (1.4.5-1ppa1) noble; urgency=medium
* Marqdo 1.4.5: Streamable HTTP MCP (agent_mcp_serve transport=http);
web invoke Bearer/X-Api-Key inject + unauthorized→401; openai/agent
field access via json.get (no footnote hard-fail).
-- cflmy <pingan@cflmy.cn> Fri, 16 Oct 2026 17:40:00 +0800
marqdo (1.4.4-1ppa1) noble; urgency=medium
* Marqdo 1.4.4: Soft LLM transport errors (structured status=error);
ext/ai table-first refactor (no json.set glue).
-- cflmy <pingan@cflmy.cn> Fri, 16 Oct 2026 14:40:00 +0800
marqdo (1.4.2-1ppa1) noble; urgency=medium
* Marqdo 1.4.2: Agent/LLM optimize O1–O5 — sandbox trial opt-in,
+7 -1
View File
@@ -48,7 +48,13 @@
| [roadmap/stdlib-mid.md](roadmap/stdlib-mid.md) | **Mid M1–M6 路线(done)** |
| [roadmap/stdlib-mid2.md](roadmap/stdlib-mid2.md) | **Mid2 M7–M10 路线(规划)** |
| [design/stdlib-math.md](design/stdlib-math.md) | **数学库(数值/随机/作图/轻量求解)** |
| [design/stdlib-foreign.md](design/stdlib-foreign.md) | **外联胶水(\`\`\`lang 具名块)— 已落地** |
| [design/stdlib-foreign.md](design/stdlib-foreign.md) | **外联胶水(\`\`\`lang 具名块)— v1 已落地** |
| [design/foreign-kernel.md](design/foreign-kernel.md) | **外联内核:持久内核 / JSON 桥 / 状态桥 / 安全** |
| [design/notebook-session.md](design/notebook-session.md) | **文档会话:cell / kernel / step / journal / 状态面板** |
| [design/notebook-syntax.md](design/notebook-syntax.md) | **文档原生 notebook 语法:内核表 / 会话表** |
| [research/foreign-notebook-decisions.md](research/foreign-notebook-decisions.md) | **外联 / notebook 决策记录(逐问逐答)** |
| [roadmap/foreign-v2.md](roadmap/foreign-v2.md) | **外联内核 v2 路线(F0–F4)** |
| [roadmap/notebook-session.md](roadmap/notebook-session.md) | **文档会话路线(D0–D5)** |
| [design/ext-abi.md](design/ext-abi.md) | **原生插件 C ABI v1**(`lib/plugin`) |
| [design/ext-llm.md](design/ext-llm.md) | **官方扩展 `ext/llm`(OpenAI 兼容)** |
| [design/ext-agent.md](design/ext-agent.md) | **官方扩展 `ext/agent`(文档驱动:单步 / 多步工作簿 + 自写回;废弃 thin TOOL 循环)** |
+2 -2
View File
@@ -120,8 +120,8 @@ import data:ext/data/db.mq.md
| `path` | `路径` | URL |
| `method` | `方法` | GET/POST/… |
| `request` | `请求` | `json` / `form` / `none` |
| `response` | `响应` | `json` / `text` / `html` |
| `auth` | `鉴权` | 角色名或 `none` |
| `response` | `响应` | `json` / `text` / `html` / `sse`(`text/event-stream`) |
| `auth` | `鉴权` | 角色名或 `none`;`bearer` / `api_key` = 不要求 cookie,把 `Authorization` / `X-Api-Key` 注入 payload 由 `# main` 校验 |
正文在请求时执行;返回值按 `response` 序列化。Result 协议见 §8。
+449
View File
@@ -0,0 +1,449 @@
# 外联内核(Foreign Kernel)
| | |
|---|---|
| 状态 | **F0–F3.5 已实现** |
| 日期 | 2026-10-17 |
| 层 | Runtime(`src/foreign/` · `src/host/foreign*` · `lib/foreign.mq.md`) |
| 相关 | [foreign-notebook-decisions.md](../research/foreign-notebook-decisions.md) · [notebook-syntax.md](notebook-syntax.md) · [notebook-session.md](notebook-session.md) · [stdlib-foreign.md](stdlib-foreign.md) · [stdlib-subtask.md](stdlib-subtask.md) |
| 核心表面 Δ | 无 |
> 外联内核 = 一个长期存活的外部语言进程 + 生命周期 + 类型桥 + 记录。它既能被 `marqdo run` 里的显式调用使用,也能被 view 的 notebook 会话使用。
---
## 1. 定位与边界
### 1.1 负责
- 解释器发现、注册、配置、探测。
- 一次性执行:临时脚本、stdin/stdout/stderr、超时、输出上限。
- 持久内核:启动、求值、状态、关闭、列表。
- 状态桥:`session_set` / `session_get` / `session_names`。
- JSON 值桥:Marqdo `Value` ↔ JSON。
- 结构化记录:`ok/code/stdout/stderr/ms/timed_out/truncated/result/error`。
- 安全:`exec` 门禁、`cwd` 沙箱、进程组、并发/输出上限。
- 视图诊断:代码卡片 Run 的完整结果。
### 1.2 不负责
- 文档语法(`内核表` / `会话表`)→ [notebook-syntax.md](notebook-syntax.md)。
- 文档会话 / cell 顺序 / journal / view UI → [notebook-session.md](notebook-session.md)。
- Foreign → Marqdo 函数回调(v2 不做)。
- 依赖安装(pip/npm)。
- 远程 / 容器执行。
### 1.3 与相邻模块的区别
| 模块 | 生命周期 | 用途 |
|---|---|---|
| `foreign` | 一次性或持久 | 调外部语言;状态可选 |
| `subtask` | 一次性并发 | 后台任务、文件子任务、函数子任务 |
| `plugin` | 进程内 | 原生 C ABI 扩展 |
| `writeback` | 文档级 | 把输出写回 `.mq.md` |
`subtask` 不管理持久状态;`foreign` 不管理文档执行顺序。
---
## 1.4 实施状态(2026-10-17)
| 阶段 | 状态 | 关键文件 |
|---|---|---|
| F0 安全执行 / `invoke` 记录 | ✅ | `src/host/foreign.rs` |
| F1 预设 / probe / register / config | ✅ | `src/foreign/registry.rs` · `src/host/foreign.rs` |
| F2 JSON 桥 / run_many / call_many | ✅ | `src/host/foreign.rs` |
| F3 Python/Node 持久内核 | ✅ | `src/host/foreign_session.rs` |
| F3.5 session_set/get/names | ✅ | 同上 |
| F4 history / view 打磨 / 发布 | 部分 | — |
测试:`cargo test --lib foreign`(11+)、`cargo test --test gold lib_foreign`(12,python 条件跳过)、`tests/notebook/shared.mq.md` 手工 view 验证。
---
## 2. 语言与内核模型
### 2.1 语言预设
| lang | 别名 | 默认命令 | 扩展名 | 版本探测 | 会话 |
|---|---|---|---|---|---|
| `python` | `python3`, `py` | `python3`(Windows `python`) | `.py` | `--version` | ✅ |
| `node` | `javascript`, `js` | `node` | `.js` | `-v` | ✅ |
| `deno` | — | `deno run` | `.ts` | `--version` | ❌ |
| `bun` | — | `bun` | `.js` | `--version` | ❌ |
| `ruby` | — | `ruby` | `.rb` | `--version` | ❌ |
| `perl` | — | `perl` | `.pl` | `--version` | ❌ |
| `php` | — | `php -r {inline}` | `.php` | `--version` | ❌ |
| `lua` | — | `lua` | `.lua` | `-v` | ❌ |
| `r` | `rscript` | `Rscript` | `.R` | `--version` | ❌ |
| `julia` | — | `julia` | `.jl` | `--version` | ❌ |
| `go` | — | `go run {file}` | `.go` | `version` | ❌ |
| `bash` | `sh`, `shell` | `bash` | `.sh` | `--version` | ❌ |
| `powershell` | `pwsh`, `ps1` | `pwsh`(Windows `powershell`) | `.ps1` | `--version` | ❌ |
| `sqlite` | `sqlite3` | `sqlite3` | `.sql` | `--version` | ❌ |
| `tsx` | `typescript` | `tsx` | `.ts` | `--version` | ❌ |
说明:
- 预设只是默认值;注册 / 配置 / 环境变量整体覆盖。
- 会话能力首批只有 Python / Node;其它语言一次性执行。
- `php` 用 `{inline}`,围栏里直接写 PHP,不需要 `<?php`。
- `deno` 默认不给权限;需要时通过配置追加 `--allow-*`。
### 2.2 解析优先级
从高到低:
1. 调用级 `cmd`(整行 argv,shell-like 切分)+ `args`(追加)。
2. 进程内注册:`register` / `set_cmd`。
3. 环境变量 `MARQDO_FOREIGN_<LANG>`(整行或 JSON 数组)。
4. 项目配置 `.marqdo/foreign.toml` / `MARQDO_FOREIGN_CONFIG`。
5. 内置预设。
6. 都没有 → 带修复建议的诊断。
Python 额外发现顺序(在预设层内):
1. `VIRTUAL_ENV/bin/python`(Windows `Scripts/python.exe`);
2. `<cwd>/.venv` / `venv` / `env`;
3. 入口目录与 `fs_root` 向上查找;
4. PATH 的 `python3` / `python`。
`MARQDO_FOREIGN_NO_VENV=1` 关闭 venv 探测;显式 `cmd` / `set_cmd` / `MARQDO_FOREIGN_PYTHON` 优先。
相对路径:配置里的 `cmd` 相对配置目录;注册 / `set_cmd` / 环境变量里的 `cmd` 相对 `cwd`。
### 2.3 注册与模板
`register(lang, cmd?, args?, template?, ext?, env?, version_args?)`。
模板占位符:
| 占位符 | 展开 |
|---|---|
| `{file}` | 临时脚本绝对路径 |
| `{dir}` | 临时目录 |
| `{base}` | 脚本文件名 |
| `{inline}` | 源码作为一个 argv;不写临时文件 |
| `{lang}` | 语言 id |
| `{cwd}` | 实际工作目录 |
规则:
- `cmd` 与 `template` 至少一个;都给 → 报错。
- `{file}` 与 `{inline}` 不能同时出现。
- 都没有时自动追加 `{file}`。
- `{inline}` 超过 60 KiB → 报错并建议 `{file}`。
- 模板不做 shell 转义;需要 shell 时显式写 `bash -c`,路径由引擎生成(安全字符)。
配置示例:
```toml
[python]
cmd = ".venv/bin/python"
args = ["-u"]
[go]
template = ["go", "run", "{file}"]
[rust]
template = ["bash", "-c", "rustc -O {file} -o {dir}/main && {dir}/main"]
[php]
template = ["php", "-r", "{inline}"]
```
---
## 3. 一次性执行
### 3.1 API
| 英文 | 中文 | 形参 | 结果 |
|---|---|---|---|
| `run` | `运行` | `code`, 可选 `stdin` `env` `cwd` `timeout` `args` `max_output` | stdout 文本(v1 兼容) |
| `run_lang` | `按语言运行` | `lang`, `source`, 可选同上 | stdout 文本 |
| `invoke` | `执行` | 可选 `code` / `lang`+`source` / `stdin` / `input` / `env` / `cwd` / `timeout` / `args` / `output` / `max_output` / `cmd` / `keep` / `cache` | 记录 map |
| `call` | `调用` | 可选 `code` / `lang`+`source` / `input` / `stdin` / `env` / `cwd` / `timeout` / `args` | 解析后的 JSON 值 |
| `run_many` | `批量运行` | `runs`, 可选 `concurrency` | 记录列表 |
| `call_many` | `批量调用` | `inputs`, 可选 `code` / `lang`+`source` / `cmd` / `env` / `cwd` / `timeout` / `args` / `max_output` / `concurrency` | 结果值列表 |
### 3.2 记录结构
`invoke` / `run_many` / `session_eval` 共用:
| 键 | 类型 | 含义 |
|---|---|---|
| `ok` | Bool | exit=0 且未超时 |
| `code` | Int | 退出码;未启动 / kill → -1 |
| `stdout` | Text | 原始 stdout(可能截断) |
| `stderr` | Text | 原始 stderr(可能截断) |
| `ms` | Int | 墙钟毫秒 |
| `lang` | Text | 实际语言 |
| `cmd` | Text | 实际 argv |
| `timed_out` | Bool | 是否超时 |
| `truncated` | Bool | stdout/stderr 是否截断 |
| `result` | 任意 / None | 按 `output` 解析 |
| `result_error` | Text | json 解析失败原因 |
| `error` | Text | 启动 / 超时 / 策略原因 |
| `script` | Text | `keep=True` 的脚本路径 |
| `cached` | Bool | 缓存命中(P4) |
`session_eval` 额外:`id`、`alive`。
### 3.3 输出模式
| `output=` | `result` |
|---|---|
| `auto`(默认) | 尝试 JSON;失败 → None |
| `text` | None |
| `json` | 解析;失败 → `result_error` |
| `lines` | 行列表 |
| `bytes` | stdout 原始字节的 base64 |
`call` 固定 JSON:空 stdout → None;非空非法 JSON → 错误。
### 3.4 失败语义
- `run` / `call`:执行失败 → `Err`;错误包含语言、argv、退出码、stderr(截 4 KiB)、stdout 头。
- `invoke` / `run_many`:执行失败 → 记录;只有参数非法 / 策略拒绝 → `Err`。
- 输出过大:截断 + `truncated=true`,不再 OOM / 报错。
---
## 4. JSON 状态桥
### 4.1 值映射
| Marqdo | JSON |
|---|---|
| None | null |
| Bool | bool |
| Int | number |
| Num | number(NaN/Inf → null) |
| Text | string |
| List | array |
| Map | object |
| Formula | 矩阵 → 二维数组;其它 → 数字或字符串 |
| Code | 源码字符串 |
| Secret | JSON 中 `<secret>`;`env` 显式交付时取真实值 |
### 4.2 输入
- `input=` → JSON;写 stdin;同时提供 `MARQDO_INPUT`(≤8 KiB)与 `MARQDO_INPUT_FILE`(任意大小)。
- `stdin=` 文本与 `input=` 互斥。
- 都不给 → stdin EOF。
- 大输入走文件,避免环境变量大小限制。
### 4.3 输出
- `result` 按 `output` 解析。
- 原始 stdout/stderr 始终保留(截断到上限)。
- `call` 返回解析值;`invoke` 返回记录。
### 4.4 类型限制
v2 是**值拷贝**,不是对象引用:
- Python DataFrame / NumPy / Node 对象 → 需要先 `to_dict` / `to_json` / `JSON.stringify`。
- 大对象传输成本高;后续考虑 Arrow / Parquet / 文件句柄。
- Secret 不默认进入 JSON;需要 env 显式交付。
---
## 5. 持久内核与会话
### 5.1 API
| 英文 | 中文 | 形参 | 结果 |
|---|---|---|---|
| `session_open` | `会话开启` | `lang`, 可选 `cmd` `env` `cwd` `timeout` | 会话 id(Text) |
| `session_eval` | `会话求值` | `id`, 可选 `code` / `source` / `input` / `timeout` | 记录(含 id/alive) |
| `session_set` | `会话设值` | `id`, `name`, `value` | None |
| `session_get` | `会话取值` | `id`, `name` | JSON 值 / None |
| `session_names` | `会话变量` | `id` | 名称列表 |
| `session_close` | `会话关闭` | `id` | Bool |
| `session_list` | `会话表` | — | map 列表 |
### 5.2 支持语言
| 语言 | 方式 |
|---|---|
| python | `python3 -u -c <bootstrap>` |
| node | `node -e <bootstrap>` |
| 其它 | 报错:`lang=… has no session bootstrap; use run/invoke` |
### 5.3 线路协议
stdin 每行一个 JSON 请求;原始 stdout 每行一个 JSON 响应:
```json
{"id":1,"code":"x = 41","input":null}
{"id":2,"code":"x + 1","input":null}
```
```json
{"id":1,"ok":true,"stdout":"","stderr":"","result":null,"error":""}
{"id":2,"ok":true,"stdout":"","stderr":"","result":42,"error":""}
```
- 同一会话严格串行:等到匹配响应后才发下一条。
- 响应单行;JSON 转义换行。
- 崩溃 / EOF → `alive=false`,会话移除。
- 超时 → kill 会话(状态不可信)。
### 5.4 Python bootstrap
- 保留原始 `sys.stdout` 写协议;用户输出用 `redirect_stdout/stderr` 捕获。
- `ast.parse`:最后一条是表达式 → 先 exec 前面,再 eval 最后,作为 result。
- 异常 → `traceback.format_exc(limit=8)`,`ok=false`;会话继续。
- `input` → 全局 `marqdo_input`。
- 不可序列化 → `default=str`。
### 5.5 Node bootstrap
- 保存原始 `process.stdout.write` 写协议。
- eval 期间替换 `console.*` 与 `process.stdout/stderr.write`;结束后恢复。
- `vm.runInThisContext` 保留全局状态;`require` 挂到 `globalThis`。
- Promise → await;顶层 await 需用户包 `(async()=>{…})()`。
- 序列化失败 → `String(result)`。
### 5.6 状态桥
- `session_set`:值 → JSON → 请求 `{"set": name, "value": ...}` 或执行一段赋值。
- `session_get`:请求 `{"get": name}` → JSON → Marqdo Value。
- `session_names`:请求变量名列表。
- 与 `session_eval` 同一线路协议;严格串行。
- 名称必须是合法标识符;保留名(`marqdo_input` 等)另行文档。
### 5.7 生命周期
- id 为 `s1`、`s2`…。
- 每会话独立进程、cwd、env。
- `HostContext` drop → 全部 kill + wait。
- `subtask` 不继承会话。
- 不跨文档共享。
---
## 6. 环境契约
| 变量 | 值 |
|---|---|
| `MARQDO_FOREIGN` | `1` |
| `MARQDO_FOREIGN_LANG` | 语言 id |
| `MARQDO_FOREIGN_SCRIPT` | 脚本路径(文件模式) |
| `MARQDO_FOREIGN_DIR` | 脚本目录 |
| `MARQDO_FOREIGN_CWD` | 实际 cwd |
| `MARQDO_ENTRY` | 入口 `.mq.md` 路径 |
| `MARQDO_ENTRY_DIR` | 入口目录 |
| `MARQDO_INPUT` | input JSON(≤8 KiB) |
| `MARQDO_INPUT_FILE` | input JSON 文件(任意大小) |
语言默认:
- Python:`PYTHONUNBUFFERED=1`、`PYTHONIOENCODING=utf-8`、`PYTHONUTF8=1`;`PYTHONPATH` 前插 cwd。
- Node:`NODE_PATH` 前插 cwd(可选)。
`env` 是 map;值 `as_display()`;Secret 取真实值(显式交付)。不主动清空父环境。
---
## 7. 临时文件与 cwd
- 默认 `<cwd>/.marqdo/tmp/foreign-<pid>-<nanos>/`;不可写 → 系统临时目录。
- `MARQDO_FOREIGN_TMPDIR` 覆盖。
- 脚本 `main.<ext>`;Python 加 utf-8 声明。
- 默认 cwd = 入口目录;调用级 cwd 经 `resolve_path` + `fs_root` 校验。
- 成功/失败/超时都清理;`keep=True` 保留并回填 `script`。
- Node 相对 `require('./x')` 相对脚本目录;需要项目相对路径时用 `MARQDO_FOREIGN_CWD` 或 `{inline}` 模板。
---
## 8. 超时、IO、进程组
- `timeout`:秒,可小数;默认 10s;上限 3600s;`<=0` 报错。
- 超时 kill 进程组:
- Unix:`process_group(0)` + `kill(-pid, SIGKILL)`;
- Windows:`taskkill /PID <pid> /T /F`。
- `max_output`:每流默认 1 MiB;范围 1 KiB–64 MiB;保留前 N 字节,余下丢弃并继续排空;`truncated=true`。
- stdin 独立写线程,避免双向管道死锁;EPIPE 忽略。
- stdout/stderr 两个读线程;UTF-8 lossy。
- 记录原始字节(至上限)以支持 `bytes` 输出。
---
## 9. 安全
| 项 | 策略 |
|---|---|
| 能力门禁 | `run` / `invoke` / `probe` / `session_*` 全部检查 `allow_exec()`;沙箱 → 拒绝 |
| shell | 默认不经 shell;模板显式 `bash -c` 才走 |
| cwd | `resolve_path` + `fs_root`;越界拒绝 |
| 输出 | 有界读取 + 丢弃式排空 |
| 时间 | 上限 + 进程组 kill |
| 并发 | 默认 4,上限 32 |
| 会话 | 每文档独立;关闭/drop 清理;超时即杀 |
| Secret | JSON 不默认暴露;env 显式;面板打码 |
| wasm | `exec-host` 关闭时全部返回清晰错误;`langs` 返回 [] |
---
## 10. 兼容性
- `run` / `run_lang` / `set_cmd` / `langs` 既有调用可用。
- `run` 成功仍返回 stdout(去一个尾换行);非 0 仍报错。
- `set_cmd` 增加可选 `args`。
- `langs` 扩展为「预设 ∪ 注册 ∪ 配置」;原集合是子集。
- `subtask` 的 `spawn code=` / `lang=` 仍返回 stdout 文本。
- HostFn id 只追加;字节码参数 < 255。
- 无核心表面 Δ。
---
## 11. 测试矩阵
| 测试 | 依赖 | 断言 |
|---|---|---|
| 既有 `foreign-python` / `run-lang` / `外联-python` | python | 兼容 |
| `foreign-invoke` | node | ok/code/stdout/stderr/ms |
| `foreign-env-cwd` | node | env/cwd/MARQDO_* |
| `foreign-timeout` | node | timed_out, 无遗留进程 |
| `foreign-call-json` | node | 嵌套 list/map 往返 |
| `foreign-call-many` / `run-many` | node | 顺序/隔离/并发 |
| `foreign-session` | node(python 可选) | 跨 eval 状态、异常不杀 |
| `foreign-session-set-get` | node | 变量推拉 |
| `foreign-session-timeout` | node | alive=false |
| `foreign-probe` | node | path/version/ok |
| `foreign-register` | node | 模板 `{inline}` |
| `foreign-config` | node | `MARQDO_FOREIGN_CONFIG` |
| `foreign-caps` | 无 | exec=false 拒绝 |
| 单元测试 | 无 | shell 切分/模板/别名/JSON |
当前环境无 Python:node + bash 为主;python 条件跳过;Windows 用 `python_available()` / `bash_available()` 守卫。
---
## 12. 实现落点
| 文件 | 内容 |
|---|---|
| `src/foreign/registry.rs` | 预设/别名/模板/venv/PATH/配置/shell 切分(纯逻辑) |
| `src/host/foreign.rs` | Job/Execution/ForeignEnv;run/invoke/call/run_many/call_many;probe/register/unset/langs/load_config |
| `src/host/foreign_session.rs` | Python/Node bootstrap;会话生命周期;set/get/names |
| `src/host/dispatch.rs` | HostFn 追加 |
| `src/aliases.rs` | 参数别名 |
| `lib/foreign.mq.md` / `lib/外联.mq.md` | L1 API + 文档 |
| `src/view/mod.rs` / `html.rs` / `debug_page.rs` | Run 诊断;后续接文档会话 |
| `tests/gold.rs` / `tests/lib/foreign-*.mq.md` | 测试 |
---
## 13. 明确不做
- Foreign line-level step。
- Foreign → Marqdo 函数 callback。
- 自动镜像变量。
- 跨语言共享对象引用。
- 全进程快照。
- 自动安装依赖。
- 浏览器 wasm 外联。
+654
View File
@@ -0,0 +1,654 @@
# 外联运行时 v3 — 能力缺口与深化设计(Proposed)
| | |
|---|---|
| 状态 | **Proposed;尚未实现** |
| 日期 | 2026-10-18 |
| 层 | Runtime(foreign)· Document(`foreign.session`)· View(投影) |
| 相关 | [foreign-kernel.md](foreign-kernel.md) · [notebook-session.md](notebook-session.md) · [foreign-v2.md](../roadmap/foreign-v2.md) · [stdlib-writeback.md](stdlib-writeback.md) · [view-static-interactive.md](view-static-interactive.md) |
| 核心表面 Δ | **无增删**:不新增 Markdown 标记、不新增 notebook CLI |
> 本文只定义设计,不承诺已实现。当前基线是 v1.5.0 之后的本地工作:`foreign.session` 句柄、多内核、values 初始化、view 控制、静态执行与显式 writeback 已可用;本文处理仍然缺失的能力。
---
## 1. 设计约束
v3 必须继续满足以下约束,否则不进入实现:
1. **外联库拥有内核与状态**:`foreign` 管理内核、会话、cell 计划、输入、输出、journal 和生命周期。`view` 只消费句柄,不拥有第二套运行时。
2. **表与围栏是值**:`内核表`、`单元表`、`输入表`、`值表` 都是普通 Marqdo 值;没有保留表名,没有新增标记。
3. **单一执行语义**:`run` 是规范执行;`view` 是同一文档的交互投影;静态导出是同一会话的快照,不另写解释器。
4. **没有 Notebook 模式**:没有 `marqdo notebook`,没有 `In/Out`,没有“普通 view / notebook view”两套布局。
5. **显式 I/O**:Marqdo 值通过 `values` / `session_set` 传递;程序 stdin 是单独概念,不能伪装成 `marqdo_input`。
6. **显式持久化**:默认不改源码;writeback 必须由文档显式调用,或由用户显式选择策略。
7. **安全默认值**:执行受 capability 控制;输入/输出有界;不反序列化任意对象;跨文档共享必须显式开启。
---
## 2. 基线:现在已经有什么
| 能力 | 状态 |
|---|---|
| `foreign.run` / `run_lang` / `invoke` / `probe` | 已实现 |
| `foreign.call` / `call_many` / JSON 进出 / 并行 | 已实现 |
| `session_open` / `eval` / `set` / `get` / `names` / `close` / `list` | 已实现 |
| `foreign.session kernels= cells= values=` | 已实现 |
| 多内核 view 会话、cell 顺序、内核归属 | 已实现 |
| Marqdo 表通过 `values` 传入多个内核 | 已实现 |
| 无 `会话` 时无状态 cells | 已实现 |
| view 每 cell 内核选择 / Clean / Run | 已实现 |
| 静态导出执行 cells 并嵌入真实输出 | 已实现 |
| 静态页 session summary、内核归属、多内核展示 | 已实现 |
| 显式 `writeback.record` | 已实现(库能力) |
| 静态页按钮的“不可执行”提示 | 已实现 |
---
## 2.1 实施状态(2026-10-18)
| ID | 状态 | 备注 |
|---|---|---|
| G1/G2 | ✅ 已实现 | session eval raw stdin;`foreign.session inputs=`;per-cell UI;Python/Node 测试 |
| G5 | ✅ 基本实现 | `session_interrupt` + view Interrupt;Python SIGINT 保留会话;Node 进程可能被终止 |
| G4 | ✅ 已实现(API) | add/remove/set-value + generation;view 尚未提供 add/remove UI |
| G6 | ✅ 已实现(API) | export/import/replay JSON snapshot;view/CLI 尚未做文件封装 |
| G9 | ✅ 已实现(CLI/API 基础) | `view capture` + `view output --snapshot`;source hash 校验 |
| G3 | ✅ 边界已实现 | 顶层 plan;嵌套 plan 诊断 |
| G10 | ✅ 显式 persist | `/api/session/cell/persist` + view Persist;不自动写源码 |
| G7/G8 | 📌 已移出当前目标 | 设计保留,后续版本再开发 |
## 3. 缺口清单
| ID | 缺口 | 优先级 | 当前行为 | 目标 |
|---|---|---|---|---|
| G1 | 持久内核 raw stdin | P0 | 持久内核只有 `marqdo_input` | 程序 stdin 与控制协议分离 |
| G2 | 每 cell 独立 stdin | P0 | 页面级 Cell stdin,Run All 共用一个输入 | cell 级输入表 + UI |
| G3 | 动态 plan | P2 | 只读顶层 `会话` | 明确边界,或受控动态 |
| G4 | 运行时更新 plan | P1 | values 只在打开时初始化 | 显式 session 变更 API |
| G5 | kernel interrupt | P0 | 只能等超时/杀进程组 | 可中断当前 eval |
| G6 | 状态持久化 / replay | P1 | journal/dirty 只在内存 | 导出/恢复/replay |
| G7 | 跨文档内核 | P2 | 每文档独立 | 显式 scope/空间 |
| G8 | 非 Python/Node 持久内核 | P3 | Go 等一次性执行 | LangSpec 会话适配器 |
| G9 | live → static 输入捕获 | P0 | 静态构建不捕获临时 stdin | 显式 snapshot 捕获 |
| G10 | 自动 writeback | P2 | 必须显式调用 | 显式策略下的“运行后写回” |
---
## 4. 统一模型
### 4.1 会话句柄
`foreign.session` 返回的句柄是一个普通 Marqdo 值,建议稳定形状:
```text
{
kind: "foreign-session",
id: "<session-id>",
document: "<entry-source-hash>",
status: "created | ready | running | interrupted | dirty | closed",
kernels: [...],
cells: [...],
inputs: [...],
values: [...],
generation: 1
}
```
`view` 只读这些字段;任何执行操作都通过 `foreign` 的 host API。句柄不是第二套运行时。
### 4.2 三种输入
必须严格区分:
| 输入 | 目标 | 形态 | 当前 |
|---|---|---|---|
| `values` | 内核变量 | Marqdo 值 → JSON copy | 已实现 |
| `input` | `marqdo_input` | JSON 值 | 已实现(低层) |
| `stdin` | 程序 stdin | 原始文本/字节 | 仅无状态 cell 有 |
v3 不允许把 raw stdin 偷偷塞进 `marqdo_input`;两者语义不同。
### 4.3 每 cell 输入表
不新增标记,仍然使用普通表:
```mq.md
`单元输入` =
| 单元 | stdin | input |
|------|-------|-------|
| 计算 | hello | {rate: 0.08} |
| 复核 | | {check: true} |
```
`foreign.session` 增加可选参数:
```text
foreign.session kernels=... cells=... values=... inputs=`单元输入`
```
规则:
- `stdin` 只给该 cell 的程序 stdin;
- `input` 只给该 cell 的 `marqdo_input`;
- 空值表示不提供;
- 输入大小有上限,超限显式报错;
- 静态导出使用同一张表,因此 view 与静态快照可以一致。
### 4.4 执行流水线
```text
evaluate 会话 handle
→ open kernels
→ apply values
→ apply cell inputs
→ run cell
→ record {stdout, stderr, input_echo, result}
→ next cell
```
失败不终止整个 Run All;错误写入 cell record,journal 保留。
---
## 5. G1/G2:持久内核 stdin 与每 cell 输入
### 5.1 问题
当前持久内核的 worker 用 stdin 传输 JSON 控制协议,因此程序自己的 stdin 不可用;`marqdo_input` 只是 JSON 变量,不是 stdin。
### 5.2 方案 A(推荐):控制通道与程序 stdin 分离
- Unix:父进程创建 **socketpair**(或两个 pipe),worker 通过 fd 3/4 读写控制帧;程序 stdin 使用单独的 pipe。
- Windows:named pipe 或可继承的匿名 pipe 作为控制通道;程序 stdin 单独 pipe。
- worker 协议升级为 `protocol: 2`:
- 控制帧:`eval` / `set` / `get` / `interrupt` / `close`
- eval 帧可带 `stdin`(UTF-8)或 `stdin_b64`(二进制,后续)
- Python worker:每次 eval 前把 `sys.stdin` 替换为 `io.TextIOWrapper`/`StringIO`,eval 后恢复控制通道;用户代码读到的是真正的 stdin。
- Node worker:控制 readline 移到控制 pipe;每次 eval 把 `process.stdin` 替换为 `PassThrough`/`Readable`,eval 后恢复。
- 超时/中断关闭该次 stdin,不污染下一次 eval。
### 5.3 方案 B(降级):只支持 `input`
如果控制通道暂不实现,持久内核只支持 `input`,并显式拒绝 raw stdin:
```text
persistent kernel stdin is unavailable; use input= or stateless cells
```
不能静默把 stdin 变成 `marqdo_input`。
### 5.4 方案 C:stdin cell 降级为一次性
持久会话中遇到 raw stdin 时,用一次性进程执行该 cell;明确标记 `state: none`,不假装内核态连续。
### 5.5 UI 与静态
- `view`:每个代码卡提供 **Cell stdin** 小输入框,或者页面级输入表;推荐每 cell,因为它们经常不同。
- 静态:读取 `inputs` 表;没有输入时显示“requires stdin”说明,而不是 traceback。
- Run All 使用每 cell 的输入;不再把一个页面级 stdin 复制给所有 cell。
### 5.6 验收
- Python 与 Node 持久内核都能读取真实 stdin;
- 同一个 cell 连续 eval 不串输入;
- 超时/中断后下一次 eval 仍可读 stdin;
- 无状态 cell 行为不变;
- 静态快照使用相同输入表,结果与 live 一致(在相同输入下)。
---
## 6. G3:动态 plan 的边界
### 6.1 问题
`foreign.session` 目前只识别 entry 顶层的 `会话` 绑定。分支、函数、运行时循环中定义的 plan 不会被 view 发现。
### 6.2 设计决定(建议)
**v3 不支持任意控制流中的 plan。** 这是有意的语义边界:
- plan 是文档的声明,不是程序运行时的副作用;
- view 不应为了找 plan 而执行 `# main`;
- 避免 view 再次长出一套“运行程序再猜计划”的隐式逻辑。
允许的动态形式:
- 顶层赋值右侧的纯表达式,包括引入的纯函数;
- `foreign.session` 的 tables/values 可以由顶层表函数计算;
- 运行时变化通过 G4 的显式 session API 完成。
**不支持**:
- `##` 函数内部的 plan;
- `if`/`while` 分支里的 plan;
- 依赖 `# main` 副作用才能得到的 plan。
### 6.3 诊断
如果源码中出现非顶层 `foreign.session`,给出:
```text
foreign.session: plan must be a top-level binding (会话 = > foreign.session ...)
```
而不是静默降级为无状态。
---
## 7. G4:运行时更新 plan
### 7.1 目标
已打开的 view session 允许用户显式修改 cell 归属、增加/移除 cell、更新内核变量;所有修改都要可追踪。
### 7.2 API
```text
foreign.session_assign session=<handle> cell="计算" kernel="复核"
foreign.session_add_cell session=<handle> cell="新单元" kernel="分析"
foreign.session_remove_cell session=<handle> cell="旧单元"
foreign.session_set_value session=<handle> kernel="分析" name="rate" value=0.08
```
所有操作:
- 校验 cell/kernel 语言;
- 递增 `generation`;
- 写入 journal(kind=plan_change / value_change);
- 标记受影响 cell 为 `dirty`;
- 失败不改变 handle。
### 7.3 View
- 内核下拉复用 `session_assign`;
- 增加/删除 cell 放在显式菜单,不做隐式拖拽写入源码;
- UI 显示 generation / dirty 状态。
### 7.4 静态
静态快照不执行变更;它记录变更后的 plan,或在构建时执行一次并固化。
---
## 8. G5:kernel interrupt
### 8.1 API
```text
foreign.session_interrupt id=<session-id>
```
### 8.2 行为
- 控制通道发送 `interrupt`;
- worker 取消当前 eval,返回 `{ok:false, interrupted:true}`;
- 保留已完成的值和 journal;
- 状态从 `running` → `interrupted`;
- 下一次 eval 可以继续使用内核(除非进程已损坏);
- 超时继续作为最后一道防线。
### 8.3 View
- 工具栏增加 Interrupt;执行中的 cell 显示 interrupted;
- 不自动重跑;
- 静态导出没有 interrupt(快照不执行)。
### 8.4 跨平台
Unix:进程组信号 + 控制帧;Windows:控制帧 + `taskkill /T` fallback。两者都必须有测试。
---
## 9. G6:状态持久化与 replay
### 9.1 不做的事
- 不做 Python pickle / Node 堆快照;
- 不做任意对象反序列化;
- 不把内核内存默认写进 `.mq.md`。
### 9.2 显式快照
```text
foreign.session_export session=<handle> path=".marqdo/sessions/doc-1.json"
foreign.session_import path=".marqdo/sessions/doc-1.json"
```
快照只包含:
```text
{
version: 1,
document_hash: "...",
plan_hash: "...",
kernels: [{name, lang, cmd, values: {...}}],
cells: [{name, kernel, status, last_ok, last_error}],
journal: [...],
inputs: [...]
}
```
值只允许 JSON 可表达的 Marqdo 值;不可序列化的值显式失败。
### 9.3 replay
```text
foreign.session_replay path="..." verify=True
```
按 journal 顺序重放 cell + inputs,比较 stdout/status;差异写入 report。replay 不覆盖原 journal,除非显式 `append`。
### 9.4 存储
- 默认目录 `.marqdo/sessions/`(gitignore 可选);
- 用户可通过 path 放到受控目录;
- 不默认提交,不默认写源码。
---
## 10. G7:跨文档内核(Deferred / future)
> **状态:已移出 v3 当前实现目标。** 本节设计保留,作为后续版本(v4 或独立 session-scope 工作)的输入;当前不承诺 API 或兼容性。
### 10.1 原则
- 默认每文档独立;
- 跨文档必须显式 `scope=` / `space=`;
- 不自动按变量名或路径共享。
### 10.2 API
```text
foreign.session_open lang="python" scope="shared:analytics"
foreign.session_list scope="shared:analytics"
foreign.session_close id=... scope=...
```
- scope 注册表由 foreign runtime 管理;
- 同名冲突报错,不覆盖;
- view 只能打开当前文档声明允许的 scope;
- 生命周期由 owner/refcount 管理,最后一个引用关闭时清理。
### 10.3 安全
- 跨文档共享可能违反文档隔离,必须 capability + 显式配置;
- journal 标记 scope;
- 静态导出默认不打开跨文档 scope。
---
## 11. G8:非 Python/Node 持久内核(Deferred / future)
> **状态:已移出 v3 当前实现目标。** 本节设计保留,等待目标语言与 worker 适配器确定后再实现;当前不承诺 API。
### 11.1 模型
不是所有语言都需要持久内核。`LangSpec` 增加能力标志:
```text
session: none | protocol_v2
```
- `none`:只能 `run` / `invoke` 一次性;
- `protocol_v2`:实现控制通道与 eval 协议,可以 `session_open`。
### 11.2 适配器
Go 等语言通过一个薄 worker 实现:
```text
control frame: eval {code, stdin, input}
result frame: {ok, stdout, stderr, result, error}
```
不允许“看起来像持久内核、实际每次重启”的假实现。
### 11.3 失败语义
- `session_open` 对 `session: none` 的语言明确报错;
- `foreign.session` 的 plan 只能使用支持 session 的语言;
- 静态/无状态仍可使用一次性执行。
---
## 12. G9:live → static 输入捕获
### 12.1 问题
静态构建在构建环境执行,无法知道用户在 live view 中临时输入的 stdin;同一文档的结果可能不同。
### 12.2 设计
增加显式的“捕获快照”:
```text
view → Capture snapshot → .marqdo/snapshots/<doc>.view.json
```
快照内容:
- document source hash;
- session plan(kernels/cells/values);
- 每 cell 的 stdin/input;
- 每 cell 的 stdout/stderr/status;
- journal 摘要、语言/解释器版本;
- 捕获时间。
静态导出:
```text
marqdo view output --snapshot .marqdo/snapshots/doc.view.json
```
优先使用快照中的记录,校验 source hash,不重新执行;若 hash 不匹配,显式失败或 `--execute` 重跑。
### 12.3 安全/隐私
- 快照可能包含 secrets;默认不提交、不发布;
- 提供 `redact` 字段/正则;
- 静态页面只发布脱敏后的快照。
### 12.4 验收
- live 中成功的结果可以在静态页面原样显示;
- 不捕获内核内存,只捕获输入/输出/状态;
- 没有快照时回退到构建时执行,行为明确。
---
## 13. G10:自动 writeback
### 13.1 原则
**“自动”必须是用户显式选择的策略,不是默认副作用。**
### 13.2 API
```text
foreign.session persist=`写回策略`
```
策略表:
```mq.md
`写回策略` =
| 单元 | key |
|------|-----|
| 计算 | total |
| 复核 | check |
```
或 view 的 **Persist results** 按钮:将当前 cell 结果写到 `writeback.record key=...`。
### 13.3 语义
- 默认不写源码;
- 显式策略只在 **用户点击 Persist** 或 `marqdo run --writeback` 时执行;
- 静态构建默认不写回;`--writeback` 必须显式;
- writeback 只支持 JSON 可表达值;
- 写入位置/格式仍由 `writeback` 库决定。
### 13.4 与静态快照的关系
- 静态快照是只读发布;
- writeback 是源码内的活结果;
- 两者可以共存,但不能互相隐式触发。
---
## 14. API / 端点汇总
### 14.1 新增 foreign 函数
```text
foreign.session inputs=
foreign.session_assign
foreign.session_add_cell
foreign.session_remove_cell
foreign.session_set_value
foreign.session_interrupt
foreign.session_export
foreign.session_import
foreign.session_replay
foreign.session_open scope=
foreign.session_list scope=
```
### 14.2 View API
```text
POST /api/session/cell/run { cell, stdin }
POST /api/session/run_all { restart, inputs? }
POST /api/session/cell/clear
POST /api/session/plan/assign
POST /api/session/plan/add
POST /api/session/plan/remove
POST /api/session/plan/value
POST /api/session/interrupt
POST /api/session/snapshot/capture
```
所有端点只操作 foreign 句柄;view 不直接读写内核。
### 14.3 CLI
```text
marqdo view output --snapshot FILE
marqdo view output --execute
marqdo view output --no-exec
marqdo run --writeback
```
不新增 `marqdo notebook`。
---
## 15. 语义矩阵
| 能力 | run | view | static |
|---|---|---|---|
| `foreign.session` plan | 作为普通值执行 | 消费句柄 | 可执行或快照 |
| raw stdin | 支持(一次性/session) | 每 cell | inputs 表/快照 |
| interrupt | 无 UI | 可调用 | 不可用 |
| state export | 可调用 | 可显式导出 | 不导出或快照 |
| cross-doc(deferred) | 设计保留 | 设计保留 | 设计保留 |
| writeback | `--writeback` | 显式按钮 | 默认不写 |
---
## 16. 实施顺序
### P0(先做,直接影响真实使用)
1. **G1/G2 stdin**:控制通道 + 每 cell 输入表 + UI。
2. **G5 interrupt**:避免只能等超时。
3. **G9 snapshot capture**:解决 live/static 不一致。
### P1
4. **G4 runtime plan updates**。
5. **G6 export/replay**。
### P2
6. **G3 dynamic plan 边界**:按第 6 节,只做顶层纯表达式;任意控制流不承诺。
7. **G10 explicit writeback policy**。
### Deferred(不在当前目标)
- **G7 cross-document scope**:设计保留,后续版本再开发。
- **G8 non-Python/Node persistent adapters**:设计保留,等待目标语言与 worker。
---
## 17. 测试设计
### 单元
- `stdin` 协议:空输入、大输入、超时、中断、连续 eval;
- 每 cell 输入隔离;
- 返回值锚定在调用点(已修复的 CallFrame.call_line);
- 表格不产生 call chip(已修复);
- plan top-level 诊断;
- session generation / dirty;
- export/import/replay 校验。
### 集成
- Python 持久内核读 `sys.stdin`;
- Node 持久内核读 `process.stdin`;
- view Run All 使用每 cell 输入;
- interrupt 后下一次 eval 正常;
- static snapshot 与 live 结果一致;
- snapshot hash 不匹配显式失败;
- cross-doc scope 冲突/清理;
- Go 一次性 vs 持久能力诊断;
- writeback 只显式写入。
### 静态
- 输出无 traceback;
- 没有输入时显示说明而非伪造;
- 多内核 controls 显示;
- writeback slot 单块、暗色主题;
- 静态按钮提示。
---
## 18. 兼容与迁移
- 现有 `foreign.session kernels= cells= values=` 保持兼容;
- 新 `inputs=` 可选;
- `session_open` 旧协议继续可用;v2 控制通道通过 handshake 协商;
- 旧页面没有 `inputs` 时仍按空 stdin 执行/显示说明;
- 不修改已发布的 v1.5.0 tag;v3 以新版本发布。
---
## 19. 需要拍板的决策
1. **G1**:是否接受控制通道与程序 stdin 分离的协议改造?否则只能保留 `input`,不能宣称 raw stdin。
2. **G3**:是否接受“plan 必须是顶层声明”的边界?若不接受,需要定义 view 可安全调用的 plan 入口,风险更高。
3. **G6**:状态快照只允许 JSON 值,是否可接受?不做任意对象序列化。
4. **G9**:快照文件是否默认放在 `.marqdo/snapshots/` 且不提交?涉及 secrets。
5. **G10**:自动 writeback 是否只允许显式按钮/flag,而不是每次运行自动写源码?
---
## 20. 当前目标范围
v3 当前实现目标为:**G1、G2、G3、G4、G5、G6、G9、G10**。
**G7、G8 已移出当前目标**,但设计文档保留在本文件第 10、11 节,供后续开发;它们不计入 v3 完成条件,也不影响现有 API 的兼容性承诺。
## 21. 非目标
- 不新增 Markdown 标记;
- 不新增 notebook CLI/模式;
- 不把任意内核内存序列化;
- 不默认修改 `.mq.md`;
- 不自动镜像变量;
- 不把 view 变成第二套解释器;
- 不把“未实现”伪装成支持。
+531
View File
@@ -0,0 +1,531 @@
# 文档会话(View Cells)
| | |
|---|---|
| 状态 | **D1–D3 首版已实现;D4/D5 待做** |
| 日期 | 2026-10-17 |
| 层 | Runtime + Document(view 服务端 / 交互单元) |
| 相关 | [notebook-syntax.md](notebook-syntax.md) · [foreign-kernel.md](foreign-kernel.md) · [foreign-notebook-decisions.md](../research/foreign-notebook-decisions.md) · [view.md](view.md) · [view-debug.md](view-debug.md) · [stdlib-subtask.md](stdlib-subtask.md) |
| 核心表面 Δ | 无(读取既有 AST 与表格) |
> 文档会话是 view 背后的交互单元运行时:它记住哪些 cell、在哪个内核、按什么顺序执行、状态是什么、结果是否过期。它不是第二个 CLI 模式——`marqdo view` 是唯一的交互面,`marqdo run` 是规范执行。
---
## 1. 目标
1. **像 Jupyter 一样即时**:点一个 cell 就执行,状态保留。
2. **不像 Jupyter 一样隐藏**:执行顺序、内核归属、状态、过期情况都可查。
3. **不破坏 Marqdo 程序语义**:`marqdo run` 仍是规范执行;view 会话是交互镜头。
4. **Git 友好**:状态/输出不默认写进 `.mq.md`。
5. **多语言**:命名内核 + 显式状态桥。
6. **可复现**:Run All / journal / dirty / stale / replay。
---
## 1.1 实施状态(2026-10-18)
| 能力 | 状态 | 位置 |
|---|---|---|
| `foreign.session` 句柄(kernels/cells 表参数) | ✅ | `lib/foreign.mq.md` · `src/host/foreign_doc.rs` |
| cell plan 求值 | ✅ 解释器求值 `会话` 绑定;不扫描源码行、不认保留表名 | `src/interp.rs` · `src/host/foreign_doc.rs` |
| cell index(绑定名 / 行号 / source hash) | ✅ | 同上 |
| 命名内核 + 默认内核 + 语言校验 | ✅ | 同上 |
| 同内核共享 / 跨内核隔离 | ✅ | Node 金样例 + 单测 |
| `run_cell` / `step` / `run_all(restart)` | ✅ | `DocumentSession` |
| `restart_kernel` / `assign_kernel` | ✅ | 同上 |
| `kernel/interrupt` | 待做 | — |
| 状态桥 `push` / `pull` / `state_list` | ✅ | `DocumentSession` + `foreign_session` |
| journal(记录 + API + Journal 按钮) | ✅ | `DocumentSession::journal_json` · view |
| view 工具栏 / 状态徽标 / 内核下拉 / 状态表单 | ✅ | `src/view/html.rs` |
| `POST /api/session/*` | ✅(含 `reload`) | `src/view/notebook_api.rs` |
| cell step | ✅ | 同上 |
| Marqdo statement step | ✅(view 工具栏 Debug → 同一 view 服务的 `/debug`;`/api/debug/*`) | `src/debug` · `src/view/debug_api.rs` |
| dirty/stale 自动检测 | 部分(reload → dirty;assign → dirty;stale 传播待做) | `src/host/foreign_doc.rs` |
| writeback 集成 | 待做 | `writeback` |
| replay / 持久化 | 待做 | — |
| Python 会话 | ✅(本机无 python,CI/用户机条件测试) | `src/host/foreign_session.rs` |
| Node 会话 | ✅ | 同上 |
首版刻意不做:把状态写进 `.mq.md`、自动镜像变量、foreign 行级 debug、跨文档共享内核。
---
## 2. 两个执行面(不是两个 CLI 模式)
### 2.1 Program execution(规范)
- 入口:`marqdo run doc.mq.md` / CI / `view` 的 `Run document`。
- 执行:`# main` 程序语义;外联通过显式 `foreign.session_open/eval/call` 编排。
- 状态:一次运行内有效;运行结束进程退出(除非显式持久内核)。
- 结果:可复现;是文档的“真理”。
### 2.2 View cells(交互)
- 入口:`marqdo view doc.mq.md` → 文档单元(cell)控制。
- 执行:按 `会话表`(或自动顺序)逐 cell 执行。
- 状态:内核持久;可切换、重启、单步。
- 结果:默认临时显示;可显式 writeback。
- 不是“真理”:是探索界面;需要复现时用 Run All 或把编排写进 `# main`。
### 2.3 两者关系
```text
.mq.md
/ \
Program run View cells
(canonical) (interactive)
\ /
Document Session
├── cell index
├── kernel registry
├── journal
└── state panel
```
- 共享 cell index 与 kernel registry。
- `run` 不消费视图会话;`view` 只在文档绑定 `会话 = > foreign.session ...` 时提供持久 cell 控制;无 `会话` 时 cells 无状态一次性执行。
- 冲突时以 `run` 的程序语义为准;view cells 不改变文档语义。
---
## 3. 文档会话模型
```text
DocumentSession
├── doc_path / doc_hash
├── cells: Vec<Cell>
│ ├── id (name / anon@line)
│ ├── name (binding)
│ ├── lang
│ ├── line_start / line_end
│ ├── source_hash
│ ├── kernel (resolved)
│ └── status: never | clean | dirty | stale | error | running
├── kernels: KernelRegistry
│ ├── name → ForeignSession
│ └── default per language
├── value_store: HashMap<String, Value> // Marqdo 侧会话值
├── journal: Vec<ExecutionRecord>
├── created_ms / last_active_ms
└── surface: program | view
```
### 3.1 Cell
- 只有**绑定名**的代码围栏才是 cell:
```markdown
`加载` =
```python
...
```
```
- 未绑定围栏仍是叙述,不是 cell。
- `id`:优先绑定名;匿名围栏用 `anon@<line>`(不进入会话表)。
- 同名 cell 重复 → 诊断;会话表按名字引用。
- `source_hash` 用于 dirty 判断。
- `line_start/line_end` 用于定位;编辑后重建索引。
### 3.2 Cell index 构建
1. 读取/解析文档(现有 `parse_source` / `load_module`)。
2. 遍历 AST,找 `Stmt::Assign { name, value: Expr::Code(c) }`。
3. 记录 name、lang、source、span、source hash。
4. 通过解释器求值 `会话` 绑定(`foreign.session` 返回的句柄,含简单依赖);不扫描源码行、不认保留表名(见 [notebook-syntax.md](notebook-syntax.md))。
5. 解析 kernel 声明与 cell 归属;校验;生成诊断。
6. 生成执行顺序:
- 有会话表 → 行顺序;
- 无 → 文档顺序。
7. 未入会话的 cell → 状态 `unlisted`,UI 显示徽标。
### 3.3 Kernel registry
- key:`<document_id>:<kernel_name>`。
- 默认:`default:python`、`default:node`,按需启动。
- 命名:来自 `内核表`,首次执行时启动。
- 状态:`idle | starting | ready | busy | dead`。
- 同一 kernel 的 cell 严格串行;不同 kernel 可并行。
- 重启:kill + 清空状态 + 重新启动;cell 状态变 `never`。
- 关闭:kill + 移除。
### 3.4 Value store(Marqdo 侧会话值)
- 保存可从 Marqdo 推入内核的值(来自最近一次 Program run 的 bindings,或用户手动输入)。
- 也保存从内核取回的值(`session_get` / 状态面板 Pull)。
- 不是解释器 Env 的实时镜像;它是会话层的**值缓存**。
- 注入解释器:未来通过 `SessionEnv` 或显式变量赋值;v2 不做隐式注入。
- 用途:
- 状态面板显示;
- Push 到内核;
- 作为 `session_set` 的值来源;
- journal 记录。
### 3.5 Journal
每次执行追加一条:
```json
{
"seq": 3,
"cell": "汇总",
"kernel": "分析",
"lang": "python",
"ok": true,
"code": 0,
"ms": 42,
"source_hash": "…",
"stdout_hash": "…",
"result_hash": "…",
"started_ms": 1760000000000,
"error": ""
}
```
用途:
- 执行序号 / 顺序;
- dirty/stale 判断;
- replay;
- 视图显示;
- 调试(“为什么这个 cell 有值”)。
存储:内存 + 可选 `.marqdo/sessions/<doc-hash>.json`(默认 git 忽略)。
---
## 4. 执行语义
### 4.1 操作
| 操作 | 语义 |
|---|---|
| **Run cell** | 执行选中的 cell;必要时启动/复用 kernel |
| **Run above** | 按会话顺序执行到选中 cell(含) |
| **Run all** | 按会话顺序执行全部;可选 `restart` |
| **Step next cell** | 执行会话顺序中的下一个未执行/选中 cell |
| **Step next statement** | Marqdo 语句级单步(DebugController) |
| **Restart kernel** | kill + 重建;cell 状态 `never` |
| **Interrupt** | kill 当前 cell 进程组;kernel 标记 dead 或重启 |
| **Reset session** | 关闭所有 kernel,清 journal / value store |
### 4.2 执行一个 cell
```text
resolve cell -> resolve kernel
-> ensure kernel ready
-> [送入?] 显式 bridge 由文档代码/UI 负责
-> foreign.session_eval(kernel, source, input?)
-> record -> update cell status
-> [取回?] 由文档代码/UI 负责
-> journal append
-> view update
```
- 默认 cwd = 文档目录;env = kernel env。
- 输出:stdout / stderr / result / ms / truncated。
- 失败:cell 状态 `error`;journal 记录;不自动重试。
- 同一 kernel 串行;UI 禁用并发 Run。
### 4.3 Step 语义
| 类型 | 行为 |
|---|---|
| **Cell step** | 执行下一个会话 cell;状态保留 |
| **Marqdo statement step** | 执行下一条 Marqdo 语句;遇到 `foreign.session_eval` 时整个外联调用原子完成 |
| **Foreign line step** | 不做;未来 DAP |
- Cell step 是 view cells 的主单步。
- Statement step 复用 `DebugController`;需要把 debug session 与 DocumentSession 关联。
- 两者可以互相切换:cell step 到某个 Marqdo cell 后可 statement step;语句执行完回到 cell 边界。
### 4.4 中断
- `Interrupt`:向当前 kernel 进程组发 SIGINT/SIGKILL;记录 `interrupted`。
- 若 kernel 状态不确定 → 标记 dead;建议 restart。
- 不允许“中断后继续用同一 kernel”除非确认状态。
### 4.5 并发
- 同一 kernel:一次一个 cell。
- 不同 kernel:可并行(例如 Python 分析与 Node 前端)。
- `Run all` 默认按会话顺序串行;可配置“按 kernel 并行、按依赖串行”(后续)。
- 并发上限沿用 foreign(32)。
---
## 5. 状态桥(会话层)
### 5.1 显式 API
文档代码:
```markdown
把 `数值表` 送进 `分析` 内核:
**> foreign.session_set id="分析" name="xs" value=`数值表`**
从 `分析` 内核取回 `mean`:
**`均值` = > foreign.session_get id="分析" name="mean"**
```
### 5.2 状态面板
- 左侧:Marqdo 侧值(最近 Program run bindings + value store)。
- 右侧:每个 kernel 的变量(`session_names` + 按需 `session_get`)。
- 操作:
- Push:选 Marqdo 值 → kernel 变量;
- Pull:选 kernel 变量 → value store / 显示;
- Refresh;
- 复制名称/值。
- Secret 打码;大对象显示摘要(类型/长度/repr 截断)。
### 5.3 自动镜像
- v2 不做。
- 后续 `mirror`:`mq_` 前缀、白名单、按值拷贝、默认关。
- 原因见 [decisions](../research/foreign-notebook-decisions.md) §1-F。
### 5.4 类型
- JSON 拷贝;复杂对象需 `to_dict` / JSON。
- 大对象:v2 显示摘要;传输用文件/DB。
- 未来:Arrow / Parquet / 共享文件句柄。
---
## 6. Dirty / Stale / Run All
### 6.1 定义
| 状态 | 条件 |
|---|---|
| `never` | 未执行 |
| `clean` | 上次执行成功后源码未变 |
| `dirty` | 源码 hash 变了 |
| `stale` | 上游 cell 变了(依赖图;v2 先按会话顺序近似) |
| `error` | 上次执行失败 |
| `running` | 正在执行 |
| `unlisted` | 不在会话表 |
### 6.2 依赖
- v2:会话表行顺序 + 简单规则:
- 若上游 cell 的 source_hash 变了,下游标 `stale`;
- 若 kernel 重启,全部 `never`。
- v2.1:变量级依赖图(静态分析 + bridge 表),精确 stale。
- 不假装 Jupyter 的“隐藏顺序”:顺序就是表/文档顺序。
### 6.3 Run All
- 默认:按会话顺序执行;可选 `restart=true`。
- 规范执行:**restart + run all**;UI 明确按钮 `Restart & Run All`。
- 未重启时:显示“当前 kernel 有历史状态”警告。
- 执行后:生成 session summary(ok/fail/耗时/跳过的 cell)。
### 6.4 Replay
- 用 journal 的顺序和输入 hash,在新 kernel 上重放;
- 比较 stdout/result hash;
- 输出 divergence 报告。
- v2 可选;至少保留 journal 以便未来实现。
---
## 7. 输出与文档
### 7.1 默认
- View cells 输出只在 view 中显示;不写 `.mq.md`。
- 关闭/刷新后可通过 journal 恢复最近结果(若 session 持久化)。
- 不写 execution_count 到文档。
### 7.2 显式 writeback
- 复用 `lib/writeback` 的 `<!-- marqdo-out … -->`。
- cell 输出锚点:cell 名 + 行号;写入位置在围栏之后,或文档末尾的“输出”区。
- 可选:
- 单 cell 写回;
- 全部写回;
- 只写回文本/数值,不写复杂对象。
- Git 友好:输出是普通 Markdown 注释块;可 diff、可忽略。
### 7.3 不写回
- 避免 Jupyter 的 `.ipynb` 输出噪声。
- 需要分享结果时,导出 HTML / 静态 view,而不是改源文件。
---
## 8. View 交互
### 8.1 工具栏
```text
[Run All] [Restart & Run All] [Step Cell] [Step Statement]
Kernel: [分析 ▾] Status: ready [Restart] [Interrupt]
[State] [Journal] [Writeback ▾]
```
### 8.2 每个 code card
```text
`加载` python · kernel 分析 · run #3 · 42ms · clean
[Run] [Step] [Kernel ▾] [State] [Copy output]
```
- 状态徽标:`never` / `clean` / `dirty` / `stale` / `error` / `running` / `unlisted`。
- 输出区:stdout / stderr / result / 截断提示。
- 内核下拉:切换归属;不迁移状态;提示需要重跑。
- `Step` = 执行此 cell(cell step)。
### 8.3 状态面板
- Marqdo / kernel 分栏。
- Push/Pull。
- Secret 打码。
- 大对象摘要 + 复制 JSON。
### 8.4 Journal 面板
- 表格:seq / cell / kernel / ok / ms / 时间 / 错误。
- 点击跳转 cell。
- 导出 JSON。
### 8.5 键盘
- `Ctrl/Cmd + Enter`:Run cell。
- `Shift + Enter`:Run cell + 下一个。
- `Alt + Enter`:Run cell + 插入?不做。
- `F10`:Step statement(与 debug 一致)。
- `F5`:Run All。
---
## 9. 服务端 API
所有 API 以文档相对路径 + session id 定位;session 内存驻留,进程退出即失。
| 方法 | 路径 | 作用 |
|---|---|---|
| POST | `/api/session/open` | 打开文档会话,返回 cells / kernels / journal |
| POST | `/api/session/close` | 关闭会话,kill 所有 kernel |
| POST | `/api/session/reload` | 重新解析文档,保留 kernel(标记 dirty) |
| POST | `/api/session/cell/run` | 执行指定 cell |
| POST | `/api/session/cell/step` | 执行下一个 cell |
| POST | `/api/session/run_all` | 按会话顺序执行;`restart` 可选 |
| POST | `/api/session/kernel/restart` | 重启指定 kernel |
| POST | `/api/session/kernel/interrupt` | 中断当前 cell |
| POST | `/api/session/kernel/assign` | 改 cell 的 kernel 归属(会话内) |
| POST | `/api/session/state/list` | Marqdo / kernel 变量列表 |
| POST | `/api/session/state/push` | Marqdo 值 → kernel |
| POST | `/api/session/state/pull` | kernel 变量 → value store |
| POST | `/api/debug/*` + `/debug` | Marqdo 语句单步(Debug 按钮,复用 debugger) |
| POST | `/api/session/writeback` | 显式写回 cell 输出 |
| POST | `/api/session/journal` | 执行日志 |
| GET | `/api/session` | 当前会话摘要 |
响应统一:`{ok, session, ...}`;错误 `{ok:false, error, code?}`。
---
## 10. 生命周期与持久化
- Session key:`<root>/<doc_rel>`;每个文档一个。
- 打开:解析、建 cell index、按需启动 kernel。
- 空闲超时:默认 30 分钟关闭(可配置)。
- 文档关闭 / 服务退出:kill 所有 kernel。
- 文件变更:view 刷新时 `reload`;cell hash 变化 → dirty;kernel 保留。
- 持久化(可选):
- `.marqdo/sessions/<doc-hash>.json`:kernel 名称/语言、cell 归属、journal、value store 摘要;
- 默认 git 忽略;
- 不持久化进程状态 / 大对象。
- 不把 session 状态写进 `.mq.md`。
---
## 11. 安全
| 项 | 策略 |
|---|---|
| exec | 所有 kernel 启动/执行检查 `allow_exec`;沙箱拒绝 |
| 生命周期 | 每文档;关闭/drop kill;不跨文档 |
| cwd | 文档目录 / `fs_root`;越界拒绝 |
| 输出 | 有界;截断标记 |
| 超时 | cell 级;进程组 kill |
| 中断 | 杀进程组;kernel 标记 dead |
| Secret | 状态面板打码;JSON 不默认暴露 |
| 未信任文档 | 不自动打开会话;必须显式 Run |
| 并发 | 同 kernel 串行;不同 kernel 上限 32 |
| writeback | 显式;路径沙箱;不自动改文档 |
---
## 12. 测试
### 12.1 服务端
- cell index:绑定名/行号/lang/source hash;未绑定围栏排除。
- 会话解析:foreign.session 句柄/自动模式/重复/未知/语言不匹配。
- 会话:open/close/reload;kernel 启动/重启/中断。
- cell run:状态机 never→running→clean/error;dirty/stale。
- step:cell step 顺序;statement step 与 debug 联动。
- state:push/pull/list;Secret 打码;大对象摘要。
- journal:顺序、hash、失败。
- 并发:同 kernel 串行;不同 kernel 并行。
- 安全:exec=false 拒绝;cwd 越界;超时。
### 12.2 金样例
- `tests/notebook/kernels.mq.md`:foreign.session 句柄。
- `tests/notebook/shared-state.mq.md`:两个 Python cell 共享 `x`。
- `tests/notebook/step.mq.md`:cell step 顺序。
- `tests/notebook/state-bridge.mq.md`:Marqdo ↔ Python 推拉。
- `tests/notebook/invalid-*.mq.md`:诊断。
- 依赖:node 为主,python 条件跳过。
### 12.3 UI
- 手工冒烟:Run cell / Step / Run All / Restart / State Push/Pull / Journal。
- 不引入前端测试框架;用服务端 API + 静态检查。
---
## 13. 与 Jupyter 对照
| 维度 | Jupyter | Document Session |
|---|---|---|
| 源 | `.ipynb` | `.mq.md` |
| 执行顺序 | 隐藏的 `In[n]` | 会话表 / 文档顺序 / journal |
| 状态 | 隐藏 kernel | kernel + value store + 状态面板 |
| 多语言 | 一文档一 kernel | 命名内核 + 每 cell 归属 |
| 单步 | cell | cell + statement |
| 复现 | 手动 | Restart & Run All + journal + replay |
| Git | 输出/计数噪声 | 源码干净;输出显式 |
| 知识 | 无 | OKF / catalog |
| 调试 | 语言 pdb | Marqdo 断点 + cell 级;行级后续 |
---
## 14. 实现顺序(D 层)
| 阶段 | 内容 |
|---|---|
| D0 | 设计/语法/决策文档(本文件) |
| D1 | `src/host/foreign_doc.rs`:cell index、foreign.session 句柄、session 结构、诊断 |
| D2 | 服务端 API:open/close/cell run/step/run_all/restart/interrupt |
| D3 | view UI:工具栏、cell 徽标、kernel 下拉、状态面板、journal |
| D4 | statement step 接入;dirty/stale;writeback |
| D5 | replay / 持久化 / 并发优化 / 测试补全 |
依赖 F 层:F3 持久内核、F3.5 状态桥先完成,D2 才能工作。
---
## 15. 明确不做(v2)
- Foreign line-level step。
- 自动镜像变量。
- 跨文档共享 kernel。
- 全进程快照。
- 自动执行 cell。
- 把 session 状态写进 `.mq.md`。
- 前端 SPA 框架重写;沿用现有 HTML/JS。
+305
View File
@@ -0,0 +1,305 @@
# 文档原生 view cells 语法
| | |
|---|---|
| 状态 | **Decided;D1 解析已实现** |
| 日期 | 2026-10-17 |
| 层 | Document(读取约定) + Runtime(校验) |
| 相关 | [foreign-notebook-decisions.md](../research/foreign-notebook-decisions.md) · [notebook-session.md](notebook-session.md) · [foreign-kernel.md](foreign-kernel.md) · [core-surface.md](core-surface.md) · [layers.md](layers.md) |
| 核心表面 Δ | **无**——不新增标记;只用既有 `# main` / 赋值 / 表格 / 反引号 |
> 目标:让“这段代码在哪个内核里、按什么顺序执行、和谁共享状态”像目录一样可读,而不是像 IDE 配置一样可读。
---
## 1. 为什么不用围栏元数据
Jupyter 的 cell 元数据、VS Code 的 `kernel=`、YAML 的 `kernels:` 都属于**代码/编辑器思维**:
- 读者看到的是一行属性,不是一段知识;
- 属性藏在围栏 info string 或 JSON 里,Markdown 预览看不见意图;
- 一旦加 `kernel=`,它就是 Language 层语法,需要 ADR、core-surface 增行;
- 同一个文档的执行计划被拆散到每个围栏,无法一眼总览。
Marqdo 的选择:
> **执行计划本身是一张表。表是文档的一部分,人先读表,机器再读表。**
这不是新语法。GFM 表已经是集合,`` `名` = `` + 表已经是数据绑定;我们只约定两个**数据表的名字**。
---
## 2. 两个约定数据表
它们放在 `# main`(或唯一入口对象)的体内,和代码单元并列:
```markdown
# main
`内核表` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | python | | 读取与清洗数据 |
| 绘图 | python | | 独立环境 |
`会话表` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 加载 | 分析 | 读取原始数据 |
| 清洗 | 分析 | 处理缺失值 |
| 汇总 | 分析 | 按城市聚合 |
| 绘图 | 绘图 | 输出 SVG |
```
为什么是“绑定的表”,不是 `# 外联内核` 对象:
- `#` 是对象/函数;把元数据做成顶层对象会把文档拆成多个作用域,code fence 会被困在对象里。
- 绑定表是入口体内的**数据**,与 cell 同一作用域;不执行、不改变控制流。
- View / 文档会话读取 AST 中的绑定值,不需要执行文档。
- 解析器零改动。
### 2.1 `内核表` / `kernels`
| 列 | 必填 | 含义 |
|---|---|---|
| `名称` / `Name` | ✅ | 内核名,供 `会话表` 引用;建议用领域词(分析/绘图/前端) |
| `语言` / `Language` | ✅ | 预设 id 或已注册语言:`python` / `node` / … |
| `命令` / `Command` | 可选 | 覆盖解释器命令;空白 = 预设 / `set_cmd` / 配置 / 环境变量 |
| `说明` / `Notes` | 可选 | 给人读的一句话;不影响执行 |
规则:
- 名称在文件内唯一;重复 → 诊断。
- 语言不支持会话时,内核仍可声明,但只能执行一次性 cell;`session_*` 报错。
- `命令` 只写可执行文件;参数请用 `.marqdo/foreign.toml` 或 `session_open`。
- 没有 `内核表`:每种语言自动一个 `default` 内核。
- 声明了内核但 `会话表` 没引用:允许,只是没有被使用。
### 2.2 `会话表` / `session`
| 列 | 必填 | 含义 |
|---|---|---|
| `单元` / `Cell` | ✅ | 代码围栏的绑定名(`` `加载` = `` 后面的名字),表格里**不写反引号** |
| `内核` / `Kernel` | ✅ | `内核表` 中的名称,或语言 id(`python` / `node`)表示该语言默认内核 |
| `说明` / `Notes` | 可选 | 给人读的一句话 |
规则:
- **行顺序 = 执行顺序**。这是 `Run All`、`Step`、`Run Above` 的规范顺序。
- 只有列出的 cell 参与会话;未列出的代码围栏仍然显示,但标记为“未入会话”,Run All 不执行。
- 没有 `会话表`:所有绑定名的代码围栏按**文档出现顺序**自动入会话,每种语言用默认内核。
- `单元` 必须对应一个已绑定名的代码围栏;找不到 → 诊断。
- 同名 cell 重复出现 → 诊断;建议一个名字只绑一个围栏。
- `内核` 未声明且不是语言 id → 诊断,列出已声明内核。
- cell 的语言与内核语言不一致 → 诊断;别名等价(`python3`/`py` ↔ `python`)。
- 表可以出现在 `# main` 体内任意位置;读者可以从表一眼看到执行计划。
### 2.3 为什么放在 `# main` 体内
- `# main` 是入口;cell 绑定与元数据表同一作用域。
- `marqdo run` 会正常求值这两个表(得到 `List of Map`),没有副作用。
- 文档会话从 AST / 最近一次运行的 bindings 读取它们;不需要新的解析路径。
- 如果文档没有 `# main` 而是唯一 `#` 对象,同样适用。
---
## 3. 完整示例
````markdown
# main
这份文档演示一个最小 view cells 文档:两个 Python 单元共享 `分析` 内核,
一个绘图单元在隔离的 `绘图` 内核里运行。
`内核表` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | python | | 数据读取与聚合 |
| 绘图 | python | | 独立绘图环境 |
`会话表` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 加载 | 分析 | 读取原始数据 |
| 清洗 | 分析 | 处理缺失值 |
| 汇总 | 分析 | 按城市聚合 |
| 绘图 | 绘图 | 输出 SVG |
`加载` =
```python
import json
rows = [
{"city": "上海", "value": 3},
{"city": "北京", "value": 5},
]
print(json.dumps(rows, ensure_ascii=False))
```
`清洗` =
```python
rows = [r for r in rows if r["value"] > 0]
```
`汇总` =
```python
from collections import Counter
counts = Counter(r["city"] for r in rows)
print(dict(counts))
```
`绘图` =
```python
print("plot-ready")
```
`汇总` 和 `绘图` 分属两个内核;`绘图` 看不到 `rows`。
需要共享时,用 Marqdo 显式把值送过去:
**> foreign.session_set id="绘图" name="rows" value=`rows`**
````
读者即使不运行,也能从 `会话表` 读出:先加载、再清洗、再汇总,最后绘图;前三个共享 `分析`,绘图在 `绘图`。
---
## 4. 自动模式(没有表)
### 4.1 没有任何约定表
````markdown
# main
`分析` =
```python
x = 41
```
`结论` =
```python
print(x + 1)
```
````
解释:
- 两个 cell 都是 `python`;
- 自动使用 `default:python` 内核(一个);
- 按文档顺序执行;
- 共享 `x`。
现有文档不需要改一行就能在 view 里 cell-by-cell 运行。
### 4.2 只有内核表
- 所有 cell 按文档顺序;
- 每种语言解析到该语言的 `default` 内核;
- 命名内核只有被 `会话表` 引用时才使用。
### 4.3 只有会话表
- `内核` 列只能写语言 id;
- 每个语言 id 解析到该语言默认内核;
- 未列出的 cell 不入会话。
---
## 5. 可读性规则(语法评审标准)
新增/修改任何 cell 表约定,必须满足:
1. **先读人话**:去掉执行语义后,表格仍像目录/清单。
2. **不使用属性串**:不写 `kernel=…`、`class=…`、JSON/YAML 元数据。
3. **不隐藏意图**:执行顺序、内核归属必须在一处可见。
4. **不引入新标记**:只用 `#` / 表格 / 反引号 / 既有代码面。
5. **默认安全**:表格只声明,不自动执行;只有 Run / Step 触发。
6. **命名用领域词**:`分析`、`绘图`、`前端`,不用 `k1`、`default`、`stage2`(除非确实是默认)。
7. **说明列可选但推荐**:给未来的读者/LLM 一点上下文。
8. **错误可读**:诊断要指出“哪个单元、哪个内核、哪一行、怎么修”。
9. **中英对称**:中文用中文列名,英文用英文列名;语义一一对应。
10. **无表也能跑**:约定表是增强,不是前置条件。
---
## 6. 诊断规范
| 情况 | 诊断(中文示例) | 修复建议 |
|---|---|---|
| 内核名重复 | `内核表: 内核名「分析」重复(第 12、20 行)` | 改名或合并 |
| 会话引用未知内核 | `foreign.session: cell「汇总」引用了未声明的内核「分析」` | 在 kernels 表声明,或改用语言 id |
| 会话引用未知单元 | `会话表: 找不到名为「清洗」的代码单元` | 检查 `` `清洗` = `` 绑定 |
| 单元重复 | `会话表: 单元「汇总」重复出现` | 每个单元只列一次 |
| 语言不匹配 | `会话表: 单元「前端」(node) 不能使用 python 内核「分析」` | 换内核或换语言 |
| 未入会话 | (非错误)view 徽标:`未入会话` | 需要时加一行 |
| 内核表为空 | `内核表: 至少声明一个内核,或删除该表使用默认内核` | 删除表或补行 |
| 会话表为空 | `会话表: 至少列出一个单元,或删除该表使用自动顺序` | 删除表或补行 |
原则:**不静默忽略**。读者必须知道为什么某个 cell 没跑。
---
## 7. 与 program execution 的关系
- `内核表` / `会话表` 只是传递给 `foreign.session` 的普通表变量,不是保留名。
- `marqdo run` 仍然执行 `# main` 的程序语义;这两个表会被求值为普通数据。
- `run` 的外联编排用显式 API(`foreign.session_open` / `session_eval` / `call`);`foreign.session` 句柄服务 `view` 的 cells 与一键 Run All。
- 两者可以并存:
- 文档用表描述“交互时怎么跑”;
- `# main` 用显式调用描述“CI 时怎么跑”。
- 如果两者冲突,以 `run` 的程序语义为准;cell 表只影响 `view`。
这样避免把“执行计划”变成第二套程序语义。
---
## 8. 状态桥的语法边界
v2 **不**为状态交换新增表格/注释。原因:
- “什么时候送、什么时候取、送到哪个内核、赋值给哪个 Marqdo 变量”需要精确语义;
- 若做成隐式表格动作,容易产生隐藏状态与顺序歧义;
- v2 先用显式 API,把时序讲清楚:
```markdown
把 `数值表` 送进 `分析` 内核:
**> foreign.session_set id="分析" name="xs" value=`数值表`**
从 `分析` 内核取回 `mean`:
**`均值` = > foreign.session_get id="分析" name="mean"**
```
- view 提供状态面板与 Push/Pull 按钮,减少手写 API 的需要;
- 声明式 `共享表` 作为 v2.1 提案,等 session value store 与“每次单元执行前后”的时序稳定后再引入。
这不是降级:v2 的功能完整(可读可写、可推可拉),只是不用隐式语法承载时序。
---
## 9. 未来扩展(不承诺进 v2)
1. `共享表`:
`| 内核变量 | Marqdo 变量 | 方向 | 说明 |`,每次单元执行前送入、后取回。
2. 每个 cell 的“运行条件/依赖”列:`依赖` 列声明上游单元。
3. 内核别名:`内核表` 的 `名称` 可被 `use` 或短名引用。
4. 多文件会话:跨 `.mq.md` 的共享内核(默认仍隔离)。
5. 语言特定的“会话能力”列(是否支持持久、是否支持行级调试)。
任何一项都要先过 §5 的可读性规则与 `core-surface` 检查。
---
## 10. 实现落点
| 逻辑 | 位置 |
|---|---|
| 识别 `会话` 的 foreign.session 句柄 | `src/host/foreign_doc.rs` |
| 读取绑定值 | AST `Stmt::Assign` + `Expr::Table` / `Expr::List`;或最近一次 run 的 bindings |
| 校验名称/语言/重复/未入会话 | 同上;错误进入现有 Diagnostic |
| cell index(绑定名/行号/语言/source hash) | 解析后的 `Module` 遍历 |
| 只读展示 | view Structure:表渲染成普通表格 + 徽标 |
| 执行 | Document Session API(见 [notebook-session.md](notebook-session.md)) |
解析器本身**不需要改**。这是本设计最重要的约束:语法是既有标记的组合,不是新构造。
+2 -2
View File
@@ -2,9 +2,9 @@
| | |
|---|---|
| 状态 | **已落地(v1)** |
| 状态 | **v1 已落地**;v2 设计见 [foreign-kernel.md](foreign-kernel.md) · [notebook-session.md](notebook-session.md) · [notebook-syntax.md](notebook-syntax.md) |
| 日期 | 2026-08-06 |
| 相关 | [stdlib-modules.md](stdlib-modules.md) · [stdlib-math.md](stdlib-math.md)(公式同构绑定) |
| 相关 | [stdlib-modules.md](stdlib-modules.md) · [stdlib-math.md](stdlib-math.md)(公式同构绑定) · [foreign-kernel.md](foreign-kernel.md) · [notebook-session.md](notebook-session.md) · [roadmap/foreign-v2.md](../roadmap/foreign-v2.md) |
---
@@ -0,0 +1,58 @@
# 缺陷记录:LLM 传输失败硬中断 Agent Runtime
| | |
|---|---|
| 状态 | **Fixed(2026-10-16)** — 见 §3;金样 `tests/ext/agent-llm-transport-soft.mq.md` |
| 日期 | 2026-10-16 |
| 发现方 | qdagent(接通 `agent.run` live 路径时) |
| 影响面 | `host/net` · `plugins/llm` · `ext/ai/llm` · `ext/ai/llm/openai` · `ext/ai/agent` |
| 相关 | [agent-run-efi-resolve-bugs-2026-10.md](agent-run-efi-resolve-bugs-2026-10.md) · [ext-agent.md](../design/ext-agent.md) |
---
## 0. 一句话
上游 LLM **连接拒绝 / 超时 / 非 200** 时,Runtime 曾以 **硬错误** 终止整个 `marqdo run`;产品(qdagent ASK/BUILD)无法落到结构化 `status=error`。
---
## 1. 复现
```bash
OPENAI_API_KEY=offline OPENAI_BASE_URL=http://127.0.0.1:9 \
marqdo run tests/ext/agent-llm-transport-soft.mq.md
# 修复前:exit≠0,error: http POST …
# 修复后:exit 0,ask/step/run 均带 finish|status=error + error 文本
```
---
## 2. 根因
| 层 | 行为 |
|----|------|
| `host_http_post` / SSE | 传输失败 `Err(...)` → VM abort |
| `plugins/llm` `llm_chat_completions` | 同上 `Err` |
| `ext/ai/llm/openai` | HTTP≠200 → `sys.exit 1` |
| `agent.model_turn` / `step` / `plan` | 无软失败臂 |
---
## 3. 修复清单
| # | 改动 | 文件 |
|---|------|------|
| 1 | 传输失败 → `{status:0, body}` / SSE error event | `src/host/net.rs` |
| 2 | 插件返回 `{finish:error, error, …}` 而非 `Err` | `plugins/llm/src/lib.rs` |
| 3 | openai 非 200 → 结构化 pack,禁止 `sys.exit` | `ext/ai/llm/openai.mq.md` |
| 4 | `llm.ask` 透传 `error` / `finish` | `ext/ai/llm.mq.md` |
| 5 | `model_turn` / `step` / `plan` 收成 `status=error` | `ext/ai/agent.mq.md` |
| 6 | 金样 | `tests/ext/agent-llm-transport-soft.mq.md` |
**契约**:Agent Runtime 对 LLM 传输失败必须可捕获;`status=error` + `error` 字符串;进程退出码 0(程序正常返回错误图)。
---
## 4. 对 qdagent
`ensure_llm` 有 key 但上游不可达时,`ask`/`build` 可走既有 `mode=error` 臂写 workbook,不再整 CLI 崩。
+290
View File
@@ -0,0 +1,290 @@
# 外联内核 · 文档会话 · 决策记录
| | |
|---|---|
| 状态 | **Decided(由用户授权代为拍板)** |
| 日期 | 2026-10-17 |
| 范围 | 外联库(`lib/foreign`)· 外联内核 · 文档会话 · view notebook 交互 |
| 相关 | [foreign-kernel.md](../design/foreign-kernel.md) · [notebook-session.md](../design/notebook-session.md) · [notebook-syntax.md](../design/notebook-syntax.md) · [foreign-v2.md](../roadmap/foreign-v2.md) · [notebook-session.md](../roadmap/notebook-session.md) |
| 核心表面 Δ | **无**——不新增 Markdown 标记;只使用既有 `#` / `##` / 表格 / 反引号 |
> 本文件记录「为什么这样设计」。实现细节分别落在设计文档与路线图中;本文件不重复 API 表。
---
## 0. 背景与授权
用户的核心诉求:
1. Marqdo 的本质是 **代码即文档、文档即知识库**;`.mq.md` 天生兼容 Markdown / OKF 知识库。
2. 外联库的初衷是**对标 Jupyter**:让文档与代码并行有用;但 Jupyter 有 Git 难跟踪、隐藏状态、execution_count 噪声等问题,Marqdo 要解决并超越。
3. `view` 模式天然适合这种交互(已有执行按钮、结构视图、writeback、trace)。
4. 之前外联库功能太单一;接下来必须支持:
- 多个代码块共享外联状态;
- 单步执行;
- 切换哪些外联内核共享状态;
- Marqdo 的状态/变量被外联使用;
- 外联的状态/变量被 Marqdo 使用。
5. 用户要求:**安全第一、效果最好、功能最强悍、不降级**;充分测试;每份讨论落成详细文档;文档提交并推送后再开发。
6. 用户强调:**设计语法不要用代码思维**,语法要天然可读。
本文档 = 对该授权的决策记录。后续实现若与本文冲突,先改本文并说明理由。
---
## 1. 关键结论
### 结论 A:外联库不是“跑 Python 的函数”,而是多语言内核运行时
`lib/foreign` 的定位升级为:
> **文档运行时与外部语言运行时之间的内核/会话层**:启动内核、执行代码单元、持久化状态、双向类型桥、生命周期与安全。
它不等同于 `subtask`(一次性并发任务),也不等同于 `writeback`(输出持久化)。
### 结论 B:Jupyter 级体验需要两层,不是一层
| 层 | 名称 | 责任 |
|---|---|---|
| **F** | Foreign Kernel Runtime | 内核进程、语言发现/注册、JSON 桥、会话、安全、记录 |
| **D** | Document Session | 文档会话、cell index、kernel 分配、单步、状态面板、执行日志、view 交互 |
上一版 `foreign-v2` 只覆盖 F。共享状态 / 单步 / 切换内核属于 D。两层分开设计、分开验收。
### 结论 C:Hybrid 执行模型
- **Program mode(规范模式)**:`marqdo run` / CI / 可复现执行。文档是程序;外联通过显式 `foreign.session_*` / `call` 编排;结果可重复。
- **View cells(交互面)**:`view` 中按 cell 执行、单步、切换内核、查看状态。适合探索,不默认写入文档;不是第二个 CLI 模式。
- 两者共享同一个内核运行时与同一套 cell 索引;view cells 是 program execution 的**交互镜头**,不是替代品。
理由:如果让 cell 执行取代文档程序语义,Marqdo 就退化成 Jupyter;如果只有 Program mode,就失去 Jupyter 的即时反馈。Hybrid 同时保留两者。
### 结论 D:命名内核;同内核共享,跨内核/跨语言显式传值
- 一个文档可以有多个命名内核:`py:分析`、`py:绘图`、`js:前端`。
- 同一内核的 cell 共享全局状态。
- 不同内核(即使同语言)默认隔离。
- 不同语言不共享进程内存;只能通过 JSON / 文件 / DB 显式交换。
- 默认每个文档、每种语言一个 `default` 内核;命名内核按需开启。
- 跨文档默认隔离;全局内核以后再做。
### 结论 E:三种单步,分阶段实现
| 单步 | 含义 | 版本 |
|---|---|---|
| **Cell step** | 执行下一个代码单元,状态保留 | v2 必做 |
| **Marqdo statement step** | 逐条执行 Marqdo 语句,查看变量 | v2 复用现有 debugger |
| **Foreign line step** | 进入 Python/Node 逐行 | v2 不做;后续接 DAP/pdb/debugpy |
- 外联 cell 在 statement step 中是原子操作。
- “Step into foreign” 是独立项目,不能混入首版。
### 结论 F:显式状态桥优先,不做隐式自动镜像
- 默认通过 `session_set` / `session_get` / `call input` / `call result` 交换,JSON 拷贝。
- 不默认把 Marqdo 变量自动注入 Python,也不默认把 Python 变量自动写回 Marqdo。
- 原因:命名冲突、Secret 泄漏、类型丢失、隐式状态、不可复现、大对象拷贝。
- 后续可选 `mirror=True`:只同步 `mq_` 前缀、白名单类型、按值拷贝,且默认关闭。
### 结论 G:不新增围栏元数据;用文档原生的“内核表 / 会话表”
- 拒绝 ` ```python kernel=分析 `、`name=`、YAML kernel 配置、魔法注释。
- 使用既有绑定表 + GFM 表格(放在 `# main` 体内,与 cell 同作用域):
- `内核表` / `kernels`
- `会话表` / `session`
- 表格是**声明**,不是自动执行;只有 Run / Step 触发。
- 解析器不变,核心表面零增长;view/session 读取这两个约定绑定。
理由:`kernel=` 是 IDE 配置思维,破坏“代码即文档”;绑定表是文档目录/执行计划,人可读、机器可解析,且不把文档拆成多个对象作用域。
### 结论 H:输出默认临时,writeback 显式;状态不写进 `.mq.md`
- View cells 的输出默认只在 view 中显示,不修改文档。
- 需要持久化时显式 writeback(沿用 `<!-- marqdo-out … -->`),可 Git 跟踪、可忽略。
- 执行日志放 `.marqdo/sessions/…`(默认 git 忽略)。
- 不把 execution_count / kernel 状态 / base64 输出写进 `.mq.md`。
### 结论 I:首批会话语言 Python + Node
- Python:Jupyter 的主场景,优先支持。
- Node:v1 已有基础,且当前开发环境可用。
- 15 个语言预设继续用于**一次性执行**;R / Julia / Ruby 等的持久会话通过自定义 bootstrap 逐步加入,不阻塞首版。
### 结论 J:v2 不做“外联回调 Marqdo 函数”
- 状态/值双向:✅
- Marqdo 代码作为源码文本传给外联:✅(外联按自己的语言执行)
- 外联代码调用 Marqdo `##` 函数:❌ v2;后续做 callback RPC(控制通道 + 重入 + 死锁处理)。
- 原因:重入解释器、线程安全、死锁、安全面都远超首版范围;先把数据/状态桥做稳。
### 结论 K:安全与测试是一等公民
- 所有 spawn / probe / session 入口检查 `allow_exec`。
- 进程组超时清理;有界 IO;stdin 写线程;输出上限;并发上限。
- 每文档生命周期,关闭即 kill;跨文档不共享。
- 测试:node + bash 为最小环境主路径;python 条件覆盖;单元测试覆盖切分/模板/JSON;金样例覆盖记录/超时/会话/并行/状态桥。
---
## 2. 逐问逐答(保留原始讨论颗粒度)
### Q1:外联库是做什么的?
**答**:它是文档的多语言执行总线,负责:
1. 内核生命周期(启动/重启/中断/关闭/切换);
2. cell 执行(源码 → kernel → stdout/stderr/result);
3. 状态持久化(同一 kernel 的 cell 共享命名空间);
4. 类型桥(Marqdo Value ↔ JSON ↔ 语言对象);
5. 环境发现与配置(probe/register/venv/config);
6. 记录与安全(耗时、退出码、截断、cap、清理)。
它不负责:文档语法(Language)、UI 布局(Document)、知识目录(Catalog)。这些由相邻层提供。
### Q2:是否支持外联库共享状态?
**答**:支持,但严格分层:
- 同一 kernel:共享全局变量。
- 同一语言不同 kernel:隔离。
- 不同语言:不共享内存,只能显式传值。
- Marqdo ↔ kernel:显式桥。
- 跨文档:默认隔离。
### Q3:能否单步执行?
**答**:能,但要定义清楚:
- cell step:v2 做。
- Marqdo statement step:现有 debugger 已有,v2 接入 view。
- foreign line step:v2 不做;后续 DAP。
### Q4:能否切换哪几个外联库共享状态?
**答**:能,用命名内核:
- `py:分析` 下所有 cell 共享;
- `py:绘图` 隔离;
- 每个 cell 有 kernel 归属;
- 切换归属不迁移状态,需要重跑;
- 跨语言用显式桥。
### Q5:Marqdo 的代码/状态能否被外联使用?外联的状态/变量能否被 Marqdo 使用?
**答**:
- Marqdo 值 → 外联:`session_set` / `call input` / `session_eval input`;JSON 拷贝。
- 外联值 → Marqdo:`session_get` / `call` 返回值;赋值。
- Marqdo 代码 → 外联:作为源码文本传递;外联按自己的语言执行,不是 Marqdo 语义。
- 外联代码 → Marqdo 函数:v2 不做,后续 callback RPC。
- 自动镜像:v2 不做,后续可选。
### Q6:如何超越 Jupyter?
**答**:不是“功能更多”,而是把 Jupyter 的隐藏面变成显式面:
| Jupyter | Marqdo |
|---|---|
| `.ipynb` JSON 混入输出/execution_count | `.mq.md` 源码干净;输出可选 writeback |
| 隐藏 kernel 状态 | 内核 + 文档会话 + 状态面板 + journal |
| 一个 notebook 一个 kernel | 命名内核 + 每 cell 归属 + 显式桥 |
| 只有 cell step | cell step + statement step |
| 手动 Run All | restart & run all + dirty/stale + replay |
| 无知识目录 | OKF / catalog;cell 是可发现知识单元 |
---
## 3. 被拒绝的方案
| 方案 | 为什么拒绝 |
|---|---|
| 围栏元数据 ` ```python kernel=分析 ` | Language 层变更;需 ADR;把文档语法变成 IDE 配置;破坏可读性 |
| YAML `kernels:` 配置 | 代码思维;与“文档即知识”冲突;表格已能表达 |
| 魔法 HTML 注释 `<!-- kernel: 分析 -->` | 不可见、易漂移、读者不可见;违背可读性 |
| 自然语言 prose 解析(“把 xs 送给分析内核”) | 非确定性;误触发;违背“叙述默认安全” |
| 隐式自动镜像变量 | 命名冲突、Secret 泄漏、类型丢失、不可复现 |
| 一个文档一个 kernel | 无法做多语言/隔离实验,正是 Jupyter 的局限 |
| 把状态/输出写进 `.mq.md` | Git 噪声;正是要解决的 Jupyter 问题 |
| cell 自动执行 | 违背“未标记即叙述/未调用即数据”;破坏安全默认 |
| v2 做 foreign→Marqdo callback | 重入/死锁/安全复杂;应先稳定数据桥 |
---
## 4. 已锁定的实现边界
### v2 必做
- F0 安全执行内核 + `invoke` 记录 + `exec` 门禁 + 有界 IO + 进程组超时。
- F1 语言注册/发现:15 预设、`probe`、`register`、`unset`、配置、venv。
- F2 JSON 桥:`call` / `run_many` / `output` / `result`。
- F3 持久内核:Python / Node `session_open/eval/close/list`。
- F3.5 状态桥:`session_set/get/names`。
- D1 文档会话服务端:cell index、kernel registry、journal、open/close。
- D2 view notebook:内核选择、Run/Step/Run All/Restart、状态面板、dirty/stale。
- D3 Marqdo statement step 接入 view。
- D4 语法:`内核表` / `会话表` 解析与校验。
- D5 测试、文档、catalog、CHANGELOG。
### v2 不做(明确记录,避免“降级”误解)
- Foreign line-level step(接 DAP 是后续项目)。
- Foreign → Marqdo 函数 callback。
- 跨语言共享对象引用 / 共享内存。
- 全进程快照(只做变量级 JSON 快照)。
- 跨文档全局内核。
- 自动安装依赖。
- 浏览器 wasm 外联。
这些不是“砍掉”,而是**独立阶段**;v2 先把内核/会话/桥做稳。
---
## 5. 风险与对策
| 风险 | 对策 |
|---|---|
| 会话协议脆弱(原生 fd 输出、缓冲) | bootstrap 用原始 stdout;Python `-u`;Node 保存原始 write;超时 kill;文档标注限制 |
| 状态桥隐式化导致不可复现 | 默认显式;`mirror` 后续且默认关 |
| 多内核进程泄漏 | 每文档生命周期;Drop kill;进程组;session_close;测试 |
| 大对象 JSON 拷贝慢 | v2 明确拷贝语义;后续 Arrow/Parquet/文件句柄 |
| `内核表` / `会话表` 名称冲突 | 文档声明为保留绑定名;后续可用 frontmatter 指针重命名 |
| 表格行/单元不一致 | 明确诊断:未声明内核、重复单元、语言不匹配、未入会话 |
| 交互执行顺序混乱 | journal + dirty/stale + Run All canonical + 执行序号 |
| 安全(Secret/环境) | Secret 不进 JSON;env 显式交付;状态面板打码;exec cap |
| 测试环境无 Python | node/bash 主路径;python 条件测试;协议单测 |
---
## 6. 术语表(统一中英文)
| 中文 | English | 含义 |
|---|---|---|
| 外联内核 | foreign kernel | 外部语言进程 |
| 外联会话 | foreign session | 内核实例 + 命名空间 |
| 文档会话 | document session | Marqdo 状态 + 内核注册表 + journal |
| 代码单元 | cell | 绑定名的代码围栏 |
| 内核归属 | kernel assignment | cell → kernel |
| 状态桥 | state bridge | Marqdo 值 ↔ 内核变量 |
| 执行日志 | execution journal | 执行顺序/耗时/结果/hash |
| 新鲜 | clean | 源码未变且已成功执行 |
| 过期 | dirty | 源码变了,结果可能过期 |
| 失效 | stale | 上游依赖变了,需要重跑 |
| 规范执行 | canonical run | 重启后按会话表顺序 Run All |
---
## 7. 决策时间线
1. 用户提出外联库对标 Jupyter、共享状态、单步、切换、双向使用。
2. Agent 阅读 `src/host/foreign.rs`、`src/view`、`src/debug`、`src/host/subtask.rs`、`lib/foreign`、设计文档。
3. Agent 回答:需要 F + D 两层;单步分三种;共享分三层;显式桥。
4. 用户授权 Agent 拍板:安全、最强、不降级、充分测试、文档先行、提交推送后开发。
5. Agent 锁定本文件 §1 的 A–K 结论,进入文档编写与实现。
---
## 8. 后续变更流程
- 任何与本文冲突的实现,先改本文并写明「变更理由 / 影响 / 迁移」。
- 新增核心语法(任何新标记)必须走 `doc/adr/` + `core-surface.md`;本文所有语法均为既有标记的组合。
- 新增会话语言、跨语言数据平面、callback RPC,各自开独立设计文档,不塞进本文件。
+226
View File
@@ -0,0 +1,226 @@
# 外联内核 v2 路线图(F0–F4)
| | |
|---|---|
| 状态 | **F0–F3.5 已落地;F4 进行中** |
| 日期 | 2026-10-17 |
| 层 | Runtime |
| 相关 | [foreign-kernel.md](../design/foreign-kernel.md) · [notebook-session.md](../design/notebook-session.md) · [notebook-syntax.md](../design/notebook-syntax.md) · [foreign-notebook-decisions.md](../research/foreign-notebook-decisions.md) · [stdlib-foreign.md](../design/stdlib-foreign.md) · [notebook-session.md](notebook-session.md)(D 层路线) |
| 核心表面 Δ | 无 |
> 本路线图只覆盖 **F:外联内核运行时**。共享状态 / 单步 / 切换内核的文档侧在 [notebook-session.md](notebook-session.md)(D 层)。
---
## 0. 现状与目标
v1:`set_cmd` / `run` / `run_lang` / `langs`;临时文件 + 子进程 + stdout 文本;无状态、无结构化结果、无 exec 门禁、无有界 IO、无进程组清理。
v2 目标:安全、可发现、结构化、可持久、可并行,并成为 D 层文档会话的可靠底座。
---
## 1. 阶段总览
| 阶段 | 内容 | 估算 | 可独立发版 |
|---|---|---|---|
| **F0** | 执行内核与安全:Job/Execution、`exec` 门禁、有界 IO、进程组、`invoke` 记录、view 诊断 | 2–3 天 | 是 |
| **F1** | 语言注册与发现:registry、15 预设、probe、register/unset、配置、venv | 2–3 天 | 是 |
| **F2** | JSON 桥与并行:call/call_many、run_many、output/result | 2–3 天 | 是 |
| **F3** | 持久内核:Python/Node session_open/eval/close/list | 3–4 天 | 是 |
| **F3.5** | 状态桥:session_set/get/names | 1–2 天 | 是 |
| **F4** | 打磨、文档、catalog、CHANGELOG、1.5.0 | 1–2 天 | 1.5.0 |
D 层(文档会话)从 F3.5 完成后开始;不要把 D 层塞进 F 层。
---
## 2. F0 — 执行内核与安全
### 内容
- `src/host/foreign.rs` 重构:
- `Job`:lang/source/stdin/input/env/cwd/timeout/args/output/max_output/cmd/keep。
- `Execution`:stdout/stderr/raw bytes/exit/ms/timed_out/truncated/error。
- `ForeignEnv`:cwd/fs_root/specs/caps 快照(供并行)。
- `exec` 门禁:`run` / `invoke` / `run_many` / `probe` / `session_*` 统一检查。
- 有界 IO:读线程保留前 N 字节、继续排空;`truncated`。
- stdin 写线程;EPIPE 忽略。
- 超时:进程组 kill(Unix `process_group(0)` + `kill(-pid)`;Windows `taskkill /T /F`)。
- shell-like argv 切分(引号/转义)。
- `invoke` + 稳定记录。
- `run` / `run_lang` 增加 `env` / `cwd` / `timeout` / `args` / `max_output`。
- view `/api/foreign-run` 返回完整记录;JS 显示 stderr / exit / ms。
- `set_cmd` L1 补 `args`。
### 验收
- 既有 `lib_foreign_*` / `lib_subtask_foreign` 绿(python 可用时)。
- 新增 node/bash 金样例:非 0 退出、stderr、超时、env/cwd、截断。
- `exec=false` 拒绝。
- `cargo test` 绿;release build;wasm check。
### 主要文件
`src/host/foreign.rs` · `src/host/mod.rs` · `src/host/subtask.rs` · `src/view/mod.rs` · `src/view/html.rs` · `src/view/debug_page.rs` · `Cargo.toml` · `tests/gold.rs` · `tests/lib/foreign-*.mq.md`。
---
## 3. F1 — 语言注册与发现
### 内容
- `src/foreign/registry.rs`(纯逻辑):
- 15 预设、别名、扩展名、版本参数、会话能力。
- 模板占位符 `{file}` / `{dir}` / `{base}` / `{inline}` / `{lang}` / `{cwd}`。
- venv 探测、PATH 查找、shell-like 切分、配置解析。
- `HostContext`:
- `foreign_specs`、`foreign_config_loaded`。
- `probe` 缓存(可选)。
- API:`probe` / `langs` / `register` / `unset` / `load_config`。
- 配置:`.marqdo/foreign.toml` + `MARQDO_FOREIGN_CONFIG`;优先级见设计。
- `MARQDO_FOREIGN_<LANG>` 支持 JSON 数组与引号。
- `subtask` 继承注册/配置。
### 验收
- `probe` 对 node / bash 给出 path + version;缺失语言 `ok=false` 不 panic。
- 配置可把 `python` 指向 venv;模板可跑 `go run`(若本机有 go)。
- `langs` 含预设;`unset` 回退预设。
- 单元测试:切分、模板、别名、扩展名。
---
## 4. F2 — JSON 桥与并行
### 内容
- `call` / `call_many`;`invoke` 的 `input` / `output` / `result` / `result_error`。
- `Value ↔ JSON` 复用 `host::json`;`bytes` base64。
- `run_many` 并行引擎:`ForeignEnv` 快照、线程上限 32、结果同序、单项 panic 隔离。
- `call_many` 全跑完再报错(清理所有子进程)。
- 文档:Python / Node JSON 样板。
### 验收
- `call` 单列表 → map,mean 为 Num;嵌套 list/map 往返。
- `run_many` 顺序稳定;单项失败不影响其它项。
- 大输入(>8 KiB)走 `MARQDO_INPUT_FILE`。
- 并发上限与超时生效。
---
## 5. F3 — 持久内核
### 内容
- `src/host/foreign_session.rs`:
- Python / Node bootstrap(嵌入式常量)。
- 启动、reader/writer、严格串行、超时、崩溃、drop 清理。
- `session_open` / `session_eval` / `session_close` / `session_list`。
- 会话记录:与 invoke 同形 + `id` / `alive`。
- 会话环境:cwd / env / cmd;`MARQDO_*`。
- `HostContext` 字段(cfg exec-host)+ Drop。
### 验收
- Node:`let x=41` → `x+1` → 42;`console.log` 捕获;异常不杀会话。
- Python:同(有 python 时)。
- 超时 → kill + `alive=false`;close/drop 无遗留进程。
- 会话与 `subtask` 边界:不继承。
---
## 6. F3.5 — 状态桥
### 内容
- `session_set` / `session_get` / `session_names`。
- 值:Marqdo → JSON → kernel;kernel → JSON → Marqdo。
- 名称校验;Secret 规则;错误可读。
- 与 `session_eval` 同一线路,严格串行。
- L1 文档 + 金样例。
### 验收
- Node:`session_set xs=[1,2,3]`,`session_eval sum(xs)` → 6;`session_get` 取回。
- 复杂嵌套 map/list 往返。
- 不存在变量 → None + 可读诊断。
- 非 JSON 可序列化 → 错误说明。
---
## 7. F4 — 打磨与发布
- `history`(可选)/ `cache`(可选,默认关)。
- view 命令记忆 / Ctrl+Enter / 错误分块。
- `lib/foreign.mq.md` / `lib/外联.mq.md` 全量。
- `public/stdlib/*` 示例。
- catalog 重新生成;CHANGELOG;版本 1.5.0。
- 性能:空跑、100 次调用、会话 vs 一次性。
### 验收
- 新函数全部有中文/英文文档与示例。
- `marqdo catalog`、`public/` 构建通过。
- 手工 view Run 一个 node 卡片,显示 stdout/stderr/exit/ms。
---
## 8. 测试矩阵
> 当前环境:node / bash / sh / perl 可用,python3 不可用。新金样例以 node + bash 为主;python 用 `python_available()` 条件跳过。
| 测试 | 依赖 | 断言 |
|---|---|---|
| 既有 3 个 foreign 用例 | python | 兼容 |
| `foreign-invoke` | node | 记录字段 |
| `foreign-env-cwd` | node | env/cwd/MARQDO_* |
| `foreign-timeout` | node | timed_out |
| `foreign-call-json` | node | 类型往返 |
| `foreign-call-many` / `run-many` | node | 顺序/隔离 |
| `foreign-session` | node | 跨 eval |
| `foreign-session-set-get` | node | 状态桥 |
| `foreign-session-timeout` | node | alive=false |
| `foreign-probe` | node | path/version |
| `foreign-register` | node | 模板 |
| `foreign-config` | node | 配置 |
| `foreign-stdout-stderr` | bash | 成功也有 stderr |
| `foreign-caps` | 无 | 拒绝 |
| 单元测试 | 无 | 切分/模板/别名/JSON |
Windows:`python_available` / `bash_available` 守卫;进程组 `taskkill` 分支编译通过。
---
## 9. 兼容性
- `run` / `run_lang` / `set_cmd` / `langs` 兼容;只增可选参数。
- `run` 成功仍返回 stdout;非 0 仍报错。
- 大输出从“报错”改为“截断 + truncated”(更安全;CHANGELOG 写明)。
- `langs` 扩展为预设 ∪ 注册 ∪ 配置。
- HostFn id 末尾追加(196+)。
- 无核心表面 Δ。
---
## 10. 风险
| 风险 | 缓解 |
|---|---|
| 会话协议脆弱 | 原始 stdout;`-u`;超时 kill;测试;文档限制 |
| 并行进程过多 | 默认 4 / 上限 32;超时;输出上限 |
| 配置优先级难记 | 设计文档固定表;`probe` 显示实际 path |
| 模板 shell 注入 | 默认不 shell;路径引擎生成;文档警示 |
| 大输出截断改变 v1 行为 | CHANGELOG;`truncated` 可检测 |
| wasm 编译 | 新字段 cfg 隔离;registry 纯逻辑 |
---
## 11. 完成定义
- F0–F3.5 全部验收。
- `cargo test` 绿;release build;wasm check。
- 文档/CHANGELOG/catalog 更新。
- 提交并推送。
- 进入 D 层([notebook-session.md](notebook-session.md))。
+177
View File
@@ -0,0 +1,177 @@
# 文档会话路线图(D0–D5)
| | |
|---|---|
| 状态 | **D1–D3 已落地(首版);D4–D5 待做** |
| 日期 | 2026-10-17 |
| 层 | Runtime + Document(view 服务端 / 前端) |
| 相关 | [notebook-session.md](../design/notebook-session.md) · [notebook-syntax.md](../design/notebook-syntax.md) · [foreign-kernel.md](../design/foreign-kernel.md) · [foreign-v2.md](foreign-v2.md) · [foreign-notebook-decisions.md](../research/foreign-notebook-decisions.md) |
| 核心表面 Δ | 无 |
> F 层让“外部语言能持久运行”;D 层让“文档里的多个 cell 像 notebook 一样共享状态、单步、切换内核”。两层分开验收。
---
## 0. 依赖
- **F3**:Python / Node 持久内核。
- **F3.5**:`session_set/get/names`。
- 现有 `DebugController`(Marqdo 语句单步)。
- 现有 `writeback` / `annot` / view 结构。
没有 F3/F3.5,D2 只能做一次性 Run,不是共享状态。
---
## 1. 阶段总览
| 阶段 | 内容 | 估算 |
|---|---|---|
| **D0** | 设计/语法/决策文档 | 已完成 |
| **D1** | `src/notebook/`:cell index、表解析、session 结构、诊断 | 2–3 天 |
| **D2** | 服务端 API:open/close/cell run/step/run_all/restart/interrupt | 3–4 天 |
| **D3** | view UI:工具栏、cell 徽标、kernel 下拉、状态面板、journal | 3–4 天 |
| **D4** | statement step、dirty/stale、writeback | 2–3 天 |
| **D5** | replay、持久化、并发优化、测试补全、发布 | 2–3 天 |
---
## 2. D1 — Cell index 与表解析
### 内容
- 新模块 `src/notebook/`:
- `cell.rs`:从 AST 提取绑定名代码围栏;id/name/lang/line/source hash。
- `syntax.rs`:读取 `内核表` / `会话表`;中英列名。
- `session.rs`:`DocumentSession` 结构、kernel registry、value store、journal。
- `diag.rs`:重复/未知/语言不匹配/未入会话诊断。
- 自动模式:无表 → 文档顺序 + 默认内核。
- 解析器不改;只遍历 `Module.functions` + `Stmt::Assign { value: Expr::Code }`。
- 测试:单元 + `tests/notebook/*.mq.md`。
### 验收
- 两个 Python cell + 一个会话表 → cells 顺序、kernel 归属正确。
- 无表 → 自动顺序。
- 未知内核/未知单元/重复/语言不匹配 → 可读诊断。
- 未绑定围栏不进入 cell index。
---
## 3. D2 — 服务端 API
### 内容
- `src/view/notebook_api.rs`(或 `view/session_api.rs`):
- open / close / reload
- cell run / step / run_all
- kernel restart / interrupt / assign
- state list / push / pull
- journal
- Session 存储:`OnceLock<Mutex<HashMap<String, Arc<DocumentSession>>>>`(类似 debug sessions)。
- 每个文档一个 session;key = 相对路径。
- 执行线程:cell 执行不阻塞 HTTP;响应返回记录;SSE/EventBus 可选。
- 错误统一 JSON。
### 验收
- open 返回 cells/kernels/journal。
- cell run:never→running→clean/error;状态更新。
- step:按会话顺序执行下一个。
- run_all:按顺序;restart 选项。
- 同 kernel 串行;不同 kernel 可并行。
- close/drop kill 所有 kernel。
- API 单测/集成测试。
---
## 4. D3 — View UI
### 内容
- 工具栏:Run All / Restart & Run All / Step Cell / Step Statement / Kernel 状态 / Restart / Interrupt / State / Journal。
- Code card:
- 状态徽标(never/clean/dirty/stale/error/running/unlisted)
- kernel 下拉
- Run / Step
- 输出区(stdout/stderr/result/truncated)
- 执行序号 / 耗时
- 状态面板:
- Marqdo 值 / kernel 变量分栏
- Push / Pull / Refresh
- Secret 打码、大对象摘要
- Journal 面板:表格 + 跳转 + 导出。
- 键盘:Ctrl/Cmd+Enter、Shift+Enter、F5、F10。
- 静态导出的 Run 仍隐藏。
### 验收
- 手工冒烟:两个 Python cell 共享 `x`;切换 kernel 后隔离;Restart 清状态。
- 状态面板 Push/Pull。
- 错误 cell 红色徽标 + stderr。
- 页面刷新后 session 仍在(服务端内存)。
---
## 5. D4 — Statement step / dirty / writeback
### 内容
- Statement step:复用 `DebugController`,与 DocumentSession 关联;暂停点显示变量。
- dirty:reload 后 source hash 变化。
- stale:按会话顺序,上游变更 → 下游 stale(v2 近似)。
- writeback:单 cell / 全部;`<!-- marqdo-out -->`;不自动。
- 运行历史 / 执行序号。
### 验收
- Statement step 在 view 中可用;外联调用原子完成。
- 编辑 cell → dirty;Run All 后 clean。
- 上游变更 → 下游 stale。
- 显式 writeback 写入文档;不自动改文件。
---
## 6. D5 — Replay / 持久化 / 发布
### 内容
- Journal replay:新 kernel,按 journal 顺序/输入 hash 重放;比较 stdout/result hash;divergence 报告。
- 可选持久化 `.marqdo/sessions/<doc-hash>.json`(kernel/cell 归属/journal/value store 摘要;默认 git 忽略)。
- 并发优化:不同 kernel 并行;同 kernel 串行。
- 测试补全;文档;catalog;CHANGELOG。
- 版本:D 层随 1.5.x。
### 验收
- Replay 能复现 clean 会话;dirty 输入给出 divergence。
- 关闭/重启服务无进程泄漏。
- 文档完整;金样例绿。
---
## 7. 测试矩阵
| 测试 | 断言 |
|---|---|
| `notebook/kernels.mq.md` | 内核表/会话表解析 |
| `notebook/auto-session.mq.md` | 无表自动顺序 |
| `notebook/shared-state.mq.md` | 同 kernel 共享变量 |
| `notebook/isolated-kernels.mq.md` | 不同 kernel 隔离 |
| `notebook/step.mq.md` | cell step 顺序 |
| `notebook/state-bridge.mq.md` | Marqdo ↔ kernel 推拉 |
| `notebook/dirty.mq.md` | 编辑 → dirty/stale |
| `notebook/invalid-*.mq.md` | 诊断 |
| API 集成 | open/run/step/run_all/restart/interrupt/close |
| 安全 | exec=false、cwd 越界、超时、Secret 打码 |
依赖 node 为主;python 条件跳过。
---
## 8. 完成定义
- D1–D4 验收;D5 可选(replay/持久化)。
- 文档会话与 F 层联调通过。
- 手工 view 冒烟:共享状态 / 单步 / 切换 / 状态面板 / journal。
- 测试绿;文档更新;提交推送。
+1 -1
View File
@@ -1 +1 @@
1.4.3
1.5.2
+431 -300
View File
File diff suppressed because it is too large. Load diff
+42 -15
View File
@@ -5,6 +5,7 @@ description: >-
OpenAI-compatible and Ollama HTTP stay behind the semantic surface.
import sys:lib/sys.mq.md
import json:lib/json.mq.md
import table:lib/table.mq.md
import fs:lib/fs.mq.md
import text:lib/text.mq.md
import openai:ext/ai/llm/openai.mq.md
@@ -25,14 +26,14 @@ Collapse a stream event list to final answer text (`done.result` or `error.messa
**answer = ""**
- [ev](events)
**t = [type](ev)**
**t = > json.get value=`ev` key="type"**
1. `t` == "done"
**answer = [result](ev)**
**answer = > json.get value=`ev` key="result"**
2. `t` == "error"
**answer = [message](ev)**
**answer = > json.get value=`ev` key="message"**
3. `t` == "delta"
4. `t` == "text"
**chunk = [text](ev)**
**chunk = > json.get value=`ev` key="text"**
1. `chunk`
**answer = answer + chunk**
2. *
@@ -167,7 +168,8 @@ Module convenience: create a default handle and return **answer text**. Pass `pa
**m = > create model=`model` base_url=`base_url` api_key=`api_key`**
**r = > m.ask prompt=`p`**
*[text](r)*
**_ret = > json.get value=`r` key="text"**
*_ret*
# llm
+ `model`=None
@@ -283,9 +285,21 @@ Semantic ask → **LLMResult** (`text`, `model`, `finish`, `backend`, `usage`, `
**text = > json.get value=`pack` key="text"**
**usage = > json.get value=`pack` key="usage"**
**finish = > json.get value=`pack` key="finish"**
**err = > json.get value=`pack` key="error"**
**tc = > json.get value=`pack` key="tool_calls"**
1. not `tc`
**tc = > json.parse text=[]**
**tc = > table.empty_list**
2. *
**_ = 1**
1. not `text`
**text = ""**
2. *
**_ = 1**
1. not `finish`
1. `err`
**finish = "error"**
2. *
**finish = "stop"**
2. *
**_ = 1**
@@ -293,16 +307,27 @@ Hoist receiver fields before the result table — table cells can shadow `self`
column names (`model`, `usage`), so `[model](self)` / `[total_tokens](usage)`
inside the table is unsafe.
**mdl = [model](self)**
**b = [backend](self)**
**nm = [name](self)**
**tt = [total_tokens](usage)**
**mdl = > json.get value=`self` key="model"**
**b = > json.get value=`self` key="backend"**
**nm = > json.get value=`self` key="name"**
1. `usage`
**tt = > json.get value=`usage` key="total_tokens"**
2. *
**tt = 0**
1. not `tt`
**tt = 0**
2. *
**_ = 1**
1. `err`
**calls = 0**
2. *
**calls = 1**
`result` =
| text | model | finish | backend | usage | name | llm_calls | tokens | tool_calls |
|------|-------|--------|---------|-------|------|-----------|--------|------------|
| `text` | `mdl` | `finish` | `b` | `usage` | `nm` | 1 | `tt` | `tc` |
| text | model | finish | backend | usage | name | llm_calls | tokens | tool_calls | error |
|------|-------|--------|---------|-------|------|-----------|--------|------------|-------|
| `text` | `mdl` | `finish` | `b` | `usage` | `nm` | `calls` | `tt` | `tc` | `err` |
*result*
@@ -345,10 +370,12 @@ Internal / compatibility primitive. Prefer `ask` / `stream` in new code.
1. `stream`
**pack = > self.complete_result prompt=`prompt` stream=True echo=`echo`**
*[events](pack)*
**_ret = > json.get value=`pack` key="events"**
*_ret*
2. *
**pack = > self.complete_result prompt=`prompt` stream=False echo=`echo`**
*[text](pack)*
**_ret = > json.get value=`pack` key="text"**
*_ret*
## chat
+ `prompt`
+65 -26
View File
@@ -4,6 +4,7 @@ description: >-
OpenAI-compatible chat completions transport (HTTP + SSE).
Prefers optional plugins/llm ABI when present; falls back to lib/net.
Used by ext/ai/llm; authors should prefer llm.ask / llm.stream.
Result packs are GFM tables / table.put (code-as-documentation).
import net:lib/net.mq.md
import json:lib/json.mq.md
import table:lib/table.mq.md
@@ -53,21 +54,18 @@ Extract OpenAI-shaped `choices[0].message.tool_calls` as a list (or empty list).
**choices = > json.get value=`data` key="choices"**
1. not `choices`
**empty = > json.parse text=[]**
*empty*
*> table.empty_list*
2. *
**c0 = [1](choices)**
**msg = > json.get value=`c0` key="message"**
1. not `msg`
**empty = > json.parse text=[]**
*empty*
*> table.empty_list*
2. *
**tc = > json.get value=`msg` key="tool_calls"**
1. `tc`
*tc*
2. *
**empty = > json.parse text=[]**
*empty*
*> table.empty_list*
## chat_completions
+ `base_url`
@@ -79,7 +77,7 @@ Extract OpenAI-shaped `choices[0].message.tool_calls` as a list (or empty list).
+ `suffix`=/chat/completions
+ `bearer`="Bearer "
POST `/chat/completions` (or stream SSE). Prefers `plugins/llm` ABI (`llm_chat_completions`) when the native plugin is resolvable; otherwise `lib/net` HTTP/SSE. Non-stream → `{text,usage,finish,tool_calls}`; stream → `{events,finish}`.
POST `/chat/completions` (or stream SSE). Prefers `plugins/llm` ABI (`llm_chat_completions`) when the native plugin is resolvable; otherwise `lib/net` HTTP/SSE. Non-stream → `{text,usage,finish,tool_calls}`; stream → `{events,finish}`. Soft transport / HTTP failures return `finish=error` packs (never abort the VM).
**p = > plugin.native_path name="llm"**
1. `p`
@@ -106,40 +104,81 @@ POST `/chat/completions` (or stream SSE). Prefers `plugins/llm` ABI (`llm_chat_c
**req = > table.put in=`req` at="stream" value=True**
**body = > json.stringify value=`req`**
**resp = > net.http_post_sse url=`url` body=`body` headers=`headers` echo=`echo`**
1. [status](resp) == 200
**pack = > json.parse text={"finish":"stop"}**
**pack = > json.set map=`pack` key="events" value=[events](resp)**
**st = > json.get value=`resp` key="status"**
1. `st` == 200
**evs = > json.get value=`resp` key="events"**
`pack` =
| finish | events |
|--------|--------|
| stop | `evs` |
*pack*
2. *
> print text=ext/ai/llm/openai: HTTP error (stream)
> print text=[status](resp)
> sys.exit code=1
Soft transport / HTTP failure — structured pack, never abort the VM.
**msg = "HTTP " + `st`**
**evs = > json.get value=`resp` key="events"**
1. `evs`
**e0 = > at value=`evs` index=0**
**m2 = > json.get value=`e0` key="message"**
1. `m2`
**msg = `m2`**
`pack` =
| finish | text | error | events |
|--------|------|-------|--------|
| error | "" | `msg` | `evs` |
*pack*
2. *
**body = > json.stringify value=`req`**
**resp = > net.http_post url=`url` body=`body` headers=`headers`**
1. [status](resp) == 200
**data = > json.parse text=[body](resp)**
**msg = [message]([1]([choices](data)))**
**st = > json.get value=`resp` key="status"**
1. `st` == 200
**raw = > json.get value=`resp` key="body"**
**data = > json.parse text=`raw`**
**choices = > json.get value=`data` key="choices"**
**choice0 = > at value=`choices` index=0**
**msg = > json.get value=`choice0` key="message"**
**text = > json.get value=`msg` key="content"**
1. not `text`
**text = ""**
2. *
**_ = 1**
**usage = > usage_from data=`data`**
**fr = > json.get value=[1]([choices](data)) key="finish_reason"**
**fr = > json.get value=`choice0` key="finish_reason"**
1. not `fr`
**fr = "stop"**
2. *
**_ = 1**
**tc = > tool_calls_from data=`data`**
**pack = > json.parse text={"finish":"stop"}**
**pack = > json.set map=`pack` key="text" value=`text`**
**pack = > json.set map=`pack` key="usage" value=`usage`**
**pack = > json.set map=`pack` key="finish" value=`fr`**
**pack = > json.set map=`pack` key="tool_calls" value=`tc`**
`pack` =
| finish | text | usage | tool_calls |
|--------|------|-------|------------|
| `fr` | `text` | `usage` | `tc` |
*pack*
2. *
> print text=ext/ai/llm/openai: HTTP error
> print text=[status](resp)
> print text=[body](resp)
> sys.exit code=1
Soft transport / HTTP failure — structured pack, never abort the VM.
**raw = > json.get value=`resp` key="body"**
**msg = "HTTP " + `st`**
1. `raw`
**msg = `msg` + ": " + `raw`**
`usage0` =
| prompt_tokens | completion_tokens | total_tokens |
|---------------|-------------------|--------------|
| 0 | 0 | 0 |
**empty_tc = > table.empty_list**
`pack` =
| finish | text | error | usage | tool_calls |
|--------|------|-------|-------|------------|
| error | "" | `msg` | `usage0` | `empty_tc` |
*pack*
+28 -18
View File
@@ -88,13 +88,14 @@ import json:lib/json.mq.md
> 系统.退出 码=1
**id = > agent_alloc**
**h = > json.parse text={"_type":"智能体"}**
**h = > json.set map=`h` key="id" value=`id`**
**h = > json.set map=`h` key="model" value=`大模型`**
**h = > json.set map=`h` key="tools" value=`工具`**
`h` =
| _type | id | model | tools |
|-------|-----|-------|-------|
| 智能体 | `id` | `大模型` | `工具` |
1. `站立提示`
**h = > json.set map=`h` key="standing" value=`站立提示`**
**h = table.put in=`h` at="standing" value=`站立提示`**
2. *
**_ = 1**
@@ -138,8 +139,12 @@ O1 可选沙盒试跑。转发 `agent.trial`(表风格返回;宿主树在成
**model = > json.get value=`自` key="model"**
**tools = > json.get value=`自` key="tools"**
**user_turn = > json.parse text={"role":"user"}**
**user_turn = > json.set map=`user_turn` key="content" value=`任务`**
`user_turn` =
| role | content |
|------|---------|
| user | `任务` |
> agent_history_append id=`id` item=`user_turn`
1. `流式`
@@ -151,9 +156,11 @@ O1 可选沙盒试跑。转发 `agent.trial`(表风格返回;宿主树在成
**reply = > trim value=`reply`**
**decision = reply**
**tool_name = > 提取工具名 回复=`reply`**
**out = > json.parse text={"status":"ok"}**
**out = > json.set map=`out` key="task" value=`任务`**
**out = > json.set map=`out` key="decision" value=`decision`**
`out` =
| status | task | decision |
|--------|------|----------|
| ok | `任务` | `decision` |
1. `tool_name`
**tool_out = > 运行工具 工具=`tools` 名=`tool_name`**
@@ -161,8 +168,8 @@ O1 可选沙盒试跑。转发 `agent.trial`(表风格返回;宿主树在成
**deny = > split value=`tool_s` sep="Tool not allowed"**
**deny_n = > len value=`deny`**
1. `deny_n` > 1
**out = > json.set map=`out` key="status" value="error"**
**out = > json.set map=`out` key="error" value=`tool_s`**
**out = table.put in=`out` at="status" value="error"**
**out = table.put in=`out` at="error" value=`tool_s`**
**reply = tool_s**
2. *
**task_s = > json.stringify value=`任务`**
@@ -178,14 +185,17 @@ O1 可选沙盒试跑。转发 `agent.trial`(表风格返回;宿主树在成
**reply = > 大模型.流式结果 事件=`evs2`**
2. *
**reply = > model.运行 提示=`fp`**
**out = > json.set map=`out` key="tool" value=`tool_name`**
**out = > json.set map=`out` key="tool_result" value=`tool_s`**
**out = > json.set map=`out` key="result" value=`reply`**
**out = table.put in=`out` at="tool" value=`tool_name`**
**out = table.put in=`out` at="tool_result" value=`tool_s`**
**out = table.put in=`out` at="result" value=`reply`**
2. *
**out = > json.set map=`out` key="result" value=`reply`**
**out = table.put in=`out` at="result" value=`reply`**
**as_turn = > json.parse text={"role":"assistant"}**
**as_turn = > json.set map=`as_turn` key="content" value=`reply`**
`as_turn` =
| role | content |
|------|---------|
| assistant | `reply` |
> agent_history_append id=`id` item=`as_turn`
1. `沙盒`
+213 -6
View File
@@ -1,6 +1,7 @@
---
title: lib/foreign
description: Run bound code fences and inline sources via local interpreters
import table:lib/table.mq.md
---
Run bound code fences and inline sources via local interpreters.
@@ -8,29 +9,235 @@ Run bound code fences and inline sources via local interpreters.
## set_cmd
+ `lang`
+ `cmd`
+ `args`=None
Bind interpreter command cmd for language lang.
Bind interpreter command cmd for language lang; optional extra argv.
*> host_foreign_set_cmd lang=`lang` cmd=`cmd`*
*> host_foreign_set_cmd lang=`lang` cmd=`cmd` args=`args`*
## run
+ `code`
+ `stdin`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
+ `args`=None
+ `max_output`=None
Run a bound code fence value; optional stdin.
Run a bound code fence value; optional stdin, env map, cwd, timeout (seconds),
extra argv and output cap. Returns stdout text; non-zero exit is an error.
Caller: [foreign.run] code=`block`
*> host_foreign_run code=`code` stdin=`stdin`*
*> host_foreign_run code=`code` stdin=`stdin` env=`env` cwd=`cwd` timeout=`timeout` args=`args` max_output=`max_output`*
## run_lang
+ `lang`
+ `source`
+ `stdin`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
+ `args`=None
+ `max_output`=None
Run source text with language lang; optional stdin.
Run source text with language lang; same options as run.
*> host_foreign_run_lang lang=`lang` source=`source` stdin=`stdin`*
*> host_foreign_run_lang lang=`lang` source=`source` stdin=`stdin` env=`env` cwd=`cwd` timeout=`timeout` args=`args` max_output=`max_output`*
## invoke
+ `code`=None
+ `lang`=None
+ `source`=None
+ `stdin`=None
+ `input`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
+ `args`=None
+ `output`=auto
+ `max_output`=None
+ `cmd`=None
+ `keep`=False
Run code or lang+source and return a record map instead of raising on process
failure. Fields: ok, code, stdout, stderr, ms, lang, cmd, timed_out, truncated,
result, result_error, error, script, cached.
`input` is encoded as JSON on stdin (mutually exclusive with `stdin`). `output`
is auto | text | json | lines. `cmd` overrides the interpreter for this call.
*> host_foreign_invoke code=`code` lang=`lang` source=`source` stdin=`stdin` input=`input` env=`env` cwd=`cwd` timeout=`timeout` args=`args` output=`output` max_output=`max_output` cmd=`cmd` keep=`keep`*
## probe
+ `lang`=None
Probe one language, or every known language when lang is omitted. Returns
{lang, cmd, path, ok, version, error} (a list when lang is omitted).
*> host_foreign_probe lang=`lang`*
## register
+ `lang`
+ `cmd`=None
+ `args`=None
+ `template`=None
+ `ext`=None
+ `env`=None
+ `version_args`=None
Register a language invocation. Provide cmd (+args) or template (placeholders
{file}, {dir}, {base}, {inline}, {lang}, {cwd}); they are mutually exclusive.
*> host_foreign_register lang=`lang` cmd=`cmd` args=`args` template=`template` ext=`ext` env=`env` version_args=`version_args`*
## unset
+ `lang`
Remove a process-local registration. Presets/config/env still apply.
*> host_foreign_unset lang=`lang`*
## load_config
+ `path`=None
Load .marqdo/foreign.toml (or an explicit path); returns the number of language
tables read.
*> host_foreign_load_config path=`path`*
## call
+ `code`=None
+ `lang`=None
+ `source`=None
+ `input`=None
+ `stdin`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
+ `args`=None
JSON-in / JSON-out call: encode `input` on stdin, parse stdout JSON and return
the value (empty stdout -> None). Raises on process failure or invalid JSON.
*> host_foreign_call code=`code` lang=`lang` source=`source` input=`input` stdin=`stdin` env=`env` cwd=`cwd` timeout=`timeout` args=`args`*
## run_many
+ `runs`
+ `concurrency`=4
Run a list of code values or invoke-option maps in parallel; returns records in
input order. One failing item does not stop the others.
*> host_foreign_run_many runs=`runs` concurrency=`concurrency`*
## call_many
+ `inputs`
+ `code`=None
+ `lang`=None
+ `source`=None
+ `cmd`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
+ `args`=None
+ `max_output`=None
+ `concurrency`=4
Apply one code/function to each input in parallel; returns parsed JSON results
in input order. Raises with the failing index after all workers finish.
*> host_foreign_call_many inputs=`inputs` code=`code` lang=`lang` source=`source` cmd=`cmd` env=`env` cwd=`cwd` timeout=`timeout` args=`args` max_output=`max_output` concurrency=`concurrency`*
## session
+ `kernels`=None
+ `cells`=None
+ `values`=None
+ `inputs`=None
Declare a document view session. `kernels` and `cells` are ordinary tables:
kernels are `|名称|语言|命令|` (ZH or EN), cells are `|单元|内核|`.
Returns a foreign session handle consumed by `view`; without this handle the
view has no persistent kernel state.
**p = > table.empty_map**
**p = > table.put in=`p` at="kind" value="foreign-session"**
**p = > table.put in=`p` at="kernels" value=`kernels`**
**p = > table.put in=`p` at="cells" value=`cells`**
**p = > table.put in=`p` at="values" value=`values`**
**p = > table.put in=`p` at="inputs" value=`inputs`**
*p*
## session_open
+ `lang`
+ `cmd`=None
+ `env`=None
+ `cwd`=None
+ `timeout`=None
Start a persistent Python/Node kernel; returns a session id. `timeout` is the
default per-eval timeout.
*> host_foreign_session_open lang=`lang` cmd=`cmd` env=`env` cwd=`cwd` timeout=`timeout`*
## session_eval
+ `id`
+ `code`=None
+ `source`=None
+ `input`=None
+ `stdin`=None
+ `timeout`=None
Evaluate code (a code value) or source text in session id; returns a record
with id/alive. `input` is JSON-encoded for the kernel; `stdin` is raw program
stdin for this evaluation.
*> host_foreign_session_eval id=`id` code=`code` source=`source` input=`input` stdin=`stdin` timeout=`timeout`*
## session_set
+ `id`
+ `name`
+ `value`
Set one kernel variable from a Marqdo value (JSON copy).
*> host_foreign_session_set id=`id` name=`name` value=`value`*
## session_get
+ `id`
+ `name`
Read one kernel variable as a Marqdo value.
*> host_foreign_session_get id=`id` name=`name`*
## session_names
+ `id`
List kernel variable names.
*> host_foreign_session_names id=`id`*
## session_close
+ `id`
Kill and remove a session; returns whether it existed.
*> host_foreign_session_close id=`id`*
## session_interrupt
+ `id`
Interrupt the current evaluation of a persistent kernel. The session remains
usable for the next evaluation when the worker supports interruption.
*> host_foreign_session_interrupt id=`id`*
## session_list
List live sessions with id/lang/cmd/evals/alive/started_ms.
*> host_foreign_session_list*
## langs
+206 -6
View File
@@ -1,6 +1,7 @@
---
title: lib/外联
description: 通过本机解释器运行绑定的代码围栏与内联源码
import table:lib/table.mq.md
---
通过本机解释器运行绑定的代码围栏与内联源码。
@@ -8,29 +9,228 @@ description: 通过本机解释器运行绑定的代码围栏与内联源码
## 设置命令
+ `语言`
+ `命令`
+ `参数`=None
为语言绑定解释器命令。
为语言绑定解释器命令;可选附加参数。
*> host_foreign_set_cmd lang=`语言` cmd=`命令`*
*> host_foreign_set_cmd lang=`语言` cmd=`命令` args=`参数`*
## 运行
+ `代码`
+ `标准输入`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
+ `参数`=None
+ `最大输出`=None
运行已绑定的代码围栏值;可选标准输入。
运行已绑定的代码围栏值;可选标准输入、环境表、工作目录、超时(秒)、
附加参数与输出上限。返回 stdout 文本;非 0 退出即报错。
调用例:[外联.运行] 代码=`块`
*> host_foreign_run code=`代码` stdin=`标准输入`*
*> host_foreign_run code=`代码` stdin=`标准输入` env=`环境` cwd=`工作目录` timeout=`超时` args=`参数` max_output=`最大输出`*
## 按语言运行
+ `语言`
+ `源码`
+ `标准输入`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
+ `参数`=None
+ `最大输出`=None
用语言运行源码文本;可选标准输入。
用语言运行源码文本;选项同“运行”。
*> host_foreign_run_lang lang=`语言` source=`源码` stdin=`标准输入`*
*> host_foreign_run_lang lang=`语言` source=`源码` stdin=`标准输入` env=`环境` cwd=`工作目录` timeout=`超时` args=`参数` max_output=`最大输出`*
## 执行
+ `代码`=None
+ `语言`=None
+ `源码`=None
+ `标准输入`=None
+ `输入`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
+ `参数`=None
+ `输出`=auto
+ `最大输出`=None
+ `命令`=None
+ `保留`=False
运行代码或“语言+源码”,返回记录 map,而不是在进程失败时直接报错。
字段:ok、code、stdout、stderr、ms、lang、cmd、timed_out、truncated、
result、result_error、error、script、cached。
`输入` 会编码为 JSON 写入 stdin(与 `标准输入` 互斥)。`输出` 可取
auto | text | json | lines。`命令` 仅本次覆盖解释器。
*> host_foreign_invoke code=`代码` lang=`语言` source=`源码` stdin=`标准输入` input=`输入` env=`环境` cwd=`工作目录` timeout=`超时` args=`参数` output=`输出` max_output=`最大输出` cmd=`命令` keep=`保留`*
## 探测
+ `语言`=None
探测一个语言;省略语言时探测所有已知语言。返回
{lang, cmd, path, ok, version, error}(省略语言时是列表)。
*> host_foreign_probe lang=`语言`*
## 注册
+ `语言`
+ `命令`=None
+ `参数`=None
+ `模板`=None
+ `扩展名`=None
+ `环境`=None
+ `版本参数`=None
注册语言调用。提供命令(+参数)或模板(占位符
{file}、{dir}、{base}、{inline}、{lang}、{cwd});二者互斥。
*> host_foreign_register lang=`语言` cmd=`命令` args=`参数` template=`模板` ext=`扩展名` env=`环境` version_args=`版本参数`*
## 注销
+ `语言`
移除进程内注册;预设 / 配置 / 环境变量仍然生效。
*> host_foreign_unset lang=`语言`*
## 加载配置
+ `路径`=None
加载 .marqdo/foreign.toml(或显式路径);返回读取到的语言表数量。
*> host_foreign_load_config path=`路径`*
## 调用
+ `代码`=None
+ `语言`=None
+ `源码`=None
+ `输入`=None
+ `标准输入`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
+ `参数`=None
JSON 进 / JSON 出:把 `输入` 编码到 stdin,解析 stdout JSON 返回值
(stdout 为空 → None)。进程失败或 JSON 非法时报错。
*> host_foreign_call code=`代码` lang=`语言` source=`源码` input=`输入` stdin=`标准输入` env=`环境` cwd=`工作目录` timeout=`超时` args=`参数`*
## 批量运行
+ `任务表`
+ `并发`=4
并行运行一组代码值或 invoke 选项 map;按输入顺序返回记录。单项失败不影响其它项。
*> host_foreign_run_many runs=`任务表` concurrency=`并发`*
## 批量调用
+ `输入表`
+ `代码`=None
+ `语言`=None
+ `源码`=None
+ `命令`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
+ `参数`=None
+ `最大输出`=None
+ `并发`=4
把同一段代码并行作用到每个输入;按输入顺序返回解析后的 JSON 结果。
全部结束后若任一项失败,报错并指出下标。
*> host_foreign_call_many inputs=`输入表` code=`代码` lang=`语言` source=`源码` cmd=`命令` env=`环境` cwd=`工作目录` timeout=`超时` args=`参数` max_output=`最大输出` concurrency=`并发`*
## 会话
+ `内核表`=None
+ `单元表`=None
+ `值表`=None
+ `输入表`=None
声明文档视图会话。内核表和单元表都是普通表格;返回一个 foreign 会话句柄,
供 view 消费。没有这个句柄时,view 的 cells 是无状态的。
**p = > table.empty_map**
**p = > table.put in=`p` at="kind" value="foreign-session"**
**p = > table.put in=`p` at="kernels" value=`内核表`**
**p = > table.put in=`p` at="cells" value=`单元表`**
**p = > table.put in=`p` at="values" value=`值表`**
**p = > table.put in=`p` at="inputs" value=`输入表`**
*p*
## 会话开启
+ `语言`
+ `命令`=None
+ `环境`=None
+ `工作目录`=None
+ `超时`=None
启动一个持久的 Python / Node 内核;返回会话 id。`超时` 是每次求值的默认超时。
*> host_foreign_session_open lang=`语言` cmd=`命令` env=`环境` cwd=`工作目录` timeout=`超时`*
## 会话求值
+ `标识`
+ `代码`=None
+ `源码`=None
+ `输入`=None
+ `标准输入`=None
+ `超时`=None
在会话中求值代码(code 值)或源码文本;返回带 id/alive 的记录。
`输入` 会编码为 JSON 交给内核。
*> host_foreign_session_eval id=`标识` code=`代码` source=`源码` input=`输入` stdin=`标准输入` timeout=`超时`*
## 会话设值
+ `标识`
+ `名称`
+ `值`
把一个 Marqdo 值写入内核变量(JSON 拷贝)。
*> host_foreign_session_set id=`标识` name=`名称` value=`值`*
## 会话取值
+ `标识`
+ `名称`
把一个内核变量读成 Marqdo 值。
*> host_foreign_session_get id=`标识` name=`名称`*
## 会话中断
+ `标识`
中断持久内核当前求值;worker 支持时,会话仍可用于下一次求值。
*> host_foreign_session_interrupt id=`标识`*
## 会话变量
+ `标识`
列出内核变量名。
*> host_foreign_session_names id=`标识`*
## 会话关闭
+ `标识`
终止并移除会话;返回是否存在。
*> host_foreign_session_close id=`标识`*
## 会话表
列出活动会话的 id/lang/cmd/evals/alive/started_ms。
*> host_foreign_session_list*
## 语言表
+336 -41
View File
@@ -1,9 +1,14 @@
//! Minimal MCP Server (stdio JSON-RPC) — tools map to `lib.member` via host `call_lib_path`.
//! Minimal MCP Server — stdio JSON-RPC + Streamable HTTP (spec 2025-03-26).
use serde_json::{json, Map, Value};
use std::ffi::CStr;
use std::io::{BufRead, BufReader, Write};
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::os::raw::{c_char, c_int};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::thread;
use std::time::{SystemTime, UNIX_EPOCH};
use crate::{host_query_json_args, parse_args, set_err, set_out};
@@ -98,6 +103,40 @@ fn call_tool(tools: &[Value], name: &str, arguments: &Value) -> Result<Value, St
}))
}
/// Handle one JSON-RPC message. Returns `None` for notifications (no response body).
fn dispatch_rpc(name: &str, tools: &[Value], msg: &Value) -> Option<Value> {
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let method = msg.get("method").and_then(|v| v.as_str()).unwrap_or("");
let params = msg.get("params").cloned().unwrap_or(json!({}));
if matches!(id, Value::Null) && method.starts_with("notifications/") {
return None;
}
Some(match method {
"initialize" => rpc_result(
&id,
json!({
"protocolVersion": "2025-03-26",
"capabilities": { "tools": {} },
"serverInfo": { "name": name, "version": "0.1.0" }
}),
),
"tools/list" | "list_tools" => rpc_result(&id, tools_list(tools)),
"tools/call" | "call_tool" => {
let tname = params.get("name").and_then(|v| v.as_str()).unwrap_or("");
let args = params.get("arguments").cloned().unwrap_or(json!({}));
match call_tool(tools, tname, &args) {
Ok(r) => rpc_result(&id, r),
Err(e) => rpc_error(&id, -32000, &e),
}
}
"ping" => rpc_result(&id, json!({})),
"" => rpc_error(&id, -32600, "missing method"),
other => rpc_error(&id, -32601, &format!("method not found: {other}")),
})
}
/// Blocking stdio MCP loop. `tools` is a JSON array of `{name, fn, description}`.
pub fn serve_stdio(name: &str, tools: Vec<Value>) -> Result<Value, String> {
let stdin = std::io::stdin();
@@ -106,41 +145,274 @@ pub fn serve_stdio(name: &str, tools: Vec<Value>) -> Result<Value, String> {
let Some(msg) = read_line_message(&mut reader)? else {
break;
};
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let method = msg.get("method").and_then(|v| v.as_str()).unwrap_or("");
let params = msg.get("params").cloned().unwrap_or(json!({}));
if matches!(id, Value::Null) && method.starts_with("notifications/") {
continue;
if let Some(reply) = dispatch_rpc(name, &tools, &msg) {
write_message(&reply)?;
}
let reply_msg = match method {
"initialize" => rpc_result(
&id,
json!({
"protocolVersion": "2024-11-05",
"capabilities": { "tools": {} },
"serverInfo": { "name": name, "version": "0.1.0" }
}),
),
"tools/list" | "list_tools" => rpc_result(&id, tools_list(&tools)),
"tools/call" | "call_tool" => {
let tname = params.get("name").and_then(|v| v.as_str()).unwrap_or("");
let args = params.get("arguments").cloned().unwrap_or(json!({}));
match call_tool(&tools, tname, &args) {
Ok(r) => rpc_result(&id, r),
Err(e) => rpc_error(&id, -32000, &e),
}
}
"ping" => rpc_result(&id, json!({})),
"" => rpc_error(&id, -32600, "missing method"),
other => rpc_error(&id, -32601, &format!("method not found: {other}")),
};
write_message(&reply_msg)?;
}
Ok(json!({ "ok": true, "transport": "stdio" }))
}
fn new_session_id() -> String {
static SEQ: AtomicU64 = AtomicU64::new(1);
let n = SEQ.fetch_add(1, Ordering::Relaxed);
let ms = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis())
.unwrap_or(0);
format!("mcp-{ms}-{n}")
}
fn read_http_request(
reader: &mut BufReader<TcpStream>,
) -> Result<(String, String, Vec<(String, String)>, Vec<u8>), String> {
let mut request_line = String::new();
reader
.read_line(&mut request_line)
.map_err(|e| format!("read request: {e}"))?;
let parts: Vec<&str> = request_line.trim().split_whitespace().collect();
if parts.len() < 2 {
return Err("bad request line".into());
}
let method = parts[0].to_string();
let path = parts[1].to_string();
let mut headers = Vec::new();
let mut content_length = 0usize;
loop {
let mut line = String::new();
reader
.read_line(&mut line)
.map_err(|e| format!("read header: {e}"))?;
let line = line.trim_end_matches(['\r', '\n']);
if line.is_empty() {
break;
}
if let Some((k, v)) = line.split_once(':') {
let key = k.trim().to_string();
let val = v.trim().to_string();
if key.eq_ignore_ascii_case("content-length") {
content_length = val.parse().unwrap_or(0);
}
headers.push((key, val));
}
}
let mut body = vec![0u8; content_length];
if content_length > 0 {
reader
.read_exact(&mut body)
.map_err(|e| format!("read body: {e}"))?;
}
Ok((method, path, headers, body))
}
fn header_get<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> {
headers
.iter()
.find(|(k, _)| k.eq_ignore_ascii_case(name))
.map(|(_, v)| v.as_str())
}
fn write_http(
stream: &mut TcpStream,
status: u16,
reason: &str,
content_type: &str,
body: &[u8],
extra: &[(&str, &str)],
) -> Result<(), String> {
let mut head = format!(
"HTTP/1.1 {status} {reason}\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n",
body.len()
);
for (k, v) in extra {
head.push_str(k);
head.push_str(": ");
head.push_str(v);
head.push_str("\r\n");
}
head.push_str("\r\n");
stream
.write_all(head.as_bytes())
.map_err(|e| e.to_string())?;
stream.write_all(body).map_err(|e| e.to_string())?;
stream.flush().map_err(|e| e.to_string())?;
Ok(())
}
fn path_matches(req_path: &str, endpoint: &str) -> bool {
let base = req_path.split('?').next().unwrap_or(req_path);
base == endpoint || base.trim_end_matches('/') == endpoint.trim_end_matches('/')
}
fn handle_http_conn(
stream: TcpStream,
name: &str,
tools: &[Value],
endpoint: &str,
) -> Result<(), String> {
let mut reader = BufReader::new(stream);
let (method, path, headers, body) = read_http_request(&mut reader)?;
let mut stream = reader.into_inner();
if !path_matches(&path, endpoint) {
write_http(
&mut stream,
404,
"Not Found",
"application/json",
br#"{"error":"not_found"}"#,
&[],
)?;
return Ok(());
}
if method.eq_ignore_ascii_case("GET") {
// Optional SSE stream — empty keepalive then close (clients may 405).
let accept = header_get(&headers, "Accept").unwrap_or("");
if accept.contains("text/event-stream") {
let body = b": mcp streamable-http ready\n\n";
write_http(
&mut stream,
200,
"OK",
"text/event-stream",
body,
&[("Cache-Control", "no-cache")],
)?;
} else {
write_http(
&mut stream,
405,
"Method Not Allowed",
"application/json",
br#"{"error":"use_post_for_jsonrpc"}"#,
&[("Allow", "POST, GET")],
)?;
}
return Ok(());
}
if method.eq_ignore_ascii_case("DELETE") {
write_http(&mut stream, 200, "OK", "application/json", b"{}", &[])?;
return Ok(());
}
if !method.eq_ignore_ascii_case("POST") {
write_http(
&mut stream,
405,
"Method Not Allowed",
"application/json",
br#"{"error":"method_not_allowed"}"#,
&[("Allow", "POST, GET, DELETE")],
)?;
return Ok(());
}
let text = String::from_utf8_lossy(&body);
let msg: Value = serde_json::from_str(text.trim()).map_err(|e| format!("json-rpc: {e}"))?;
let is_init = msg.get("method").and_then(|v| v.as_str()) == Some("initialize");
let existing_sid = header_get(&headers, "Mcp-Session-Id")
.or_else(|| header_get(&headers, "MCP-Session-Id"))
.map(|s| s.to_string());
// Notifications → 202 Accepted, empty body.
if msg.get("id").is_none() || msg.get("id") == Some(&Value::Null) {
if let Some(m) = msg.get("method").and_then(|v| v.as_str()) {
if m.starts_with("notifications/") {
write_http(
&mut stream,
202,
"Accepted",
"application/json",
b"",
&[],
)?;
return Ok(());
}
}
}
let reply = match dispatch_rpc(name, tools, &msg) {
Some(r) => r,
None => {
write_http(
&mut stream,
202,
"Accepted",
"application/json",
b"",
&[],
)?;
return Ok(());
}
};
let body = serde_json::to_vec(&reply).map_err(|e| e.to_string())?;
let mut extras: Vec<(&str, String)> = Vec::new();
if is_init {
let sid = existing_sid.unwrap_or_else(new_session_id);
extras.push(("Mcp-Session-Id", sid));
} else if let Some(sid) = existing_sid {
extras.push(("Mcp-Session-Id", sid));
}
let extra_refs: Vec<(&str, &str)> = extras.iter().map(|(k, v)| (*k, v.as_str())).collect();
write_http(
&mut stream,
200,
"OK",
"application/json",
&body,
&extra_refs,
)?;
Ok(())
}
/// Streamable HTTP MCP (POST JSON-RPC + optional GET SSE). Spec 2025-03-26.
pub fn serve_http(
name: &str,
tools: Vec<Value>,
host: &str,
port: u16,
path: &str,
) -> Result<Value, String> {
let addr = format!("{host}:{port}");
let listener = TcpListener::bind(&addr).map_err(|e| format!("bind {addr}: {e}"))?;
let name = Arc::new(name.to_string());
let tools = Arc::new(tools);
let endpoint = Arc::new(if path.is_empty() {
"/mcp".to_string()
} else if path.starts_with('/') {
path.to_string()
} else {
format!("/{path}")
});
eprintln!(
"mcp streamable-http listening on http://{addr}{} (POST JSON-RPC)",
&*endpoint
);
for conn in listener.incoming() {
match conn {
Ok(stream) => {
let name = Arc::clone(&name);
let tools = Arc::clone(&tools);
let endpoint = Arc::clone(&endpoint);
thread::spawn(move || {
if let Err(e) = handle_http_conn(stream, &name, &tools, &endpoint) {
eprintln!("mcp http conn: {e}");
}
});
}
Err(e) => eprintln!("mcp http accept: {e}"),
}
}
Ok(json!({
"ok": true,
"transport": "http",
"host": host,
"port": port,
"path": *endpoint
}))
}
pub fn tools_from_value(v: &Value) -> Vec<Value> {
if let Some(arr) = v.as_array() {
return arr.clone();
@@ -180,13 +452,6 @@ pub unsafe extern "C" fn agent_mcp_serve(
.get("transport")
.and_then(|v| v.as_str())
.unwrap_or("stdio");
if transport != "stdio" {
set_err(
err_msg,
&format!("mcp serve: transport `{transport}` not implemented yet (use stdio)"),
);
return 1;
}
let name = args
.get("name")
.and_then(|v| v.as_str())
@@ -195,7 +460,38 @@ pub unsafe extern "C" fn agent_mcp_serve(
.get("tools")
.map(tools_from_value)
.unwrap_or_default();
match serve_stdio(name, tools) {
let result = match transport {
"stdio" => serve_stdio(name, tools),
"http" | "streamable-http" | "sse" => {
let host = args
.get("host")
.and_then(|v| v.as_str())
.unwrap_or("127.0.0.1");
let port = args
.get("port")
.and_then(|v| {
v.as_u64()
.or_else(|| v.as_i64().map(|n| n as u64))
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
})
.unwrap_or(7434) as u16;
let path = args
.get("path")
.and_then(|v| v.as_str())
.unwrap_or("/mcp");
serve_http(name, tools, host, port, path)
}
other => {
set_err(
err_msg,
&format!("mcp serve: transport `{other}` not supported (use stdio|http)"),
);
return 1;
}
};
match result {
Ok(v) => {
set_out(out_json, &v.to_string());
0
@@ -207,6 +503,5 @@ pub unsafe extern "C" fn agent_mcp_serve(
}
}
// Silence unused import if CStr unused in this module.
#[allow(dead_code)]
fn _unused(_: &CStr) {}
+42 -6
View File
@@ -245,7 +245,12 @@ fn chat_completions(args: &Value) -> Result<Value, String> {
let status = r.status();
if status != 200 {
let text = read_limited(r.into_reader())?;
return Err(format!("HTTP {status}: {text}"));
return Ok(json!({
"events": [{"type": "error", "message": format!("HTTP {status}: {text}")}],
"finish": "error",
"error": format!("HTTP {status}: {text}"),
"transport": "plugin",
}));
}
let events = parse_sse_events(BufReader::new(r.into_reader()), echo)?;
Ok(json!({
@@ -256,9 +261,19 @@ fn chat_completions(args: &Value) -> Result<Value, String> {
}
Err(ureq::Error::Status(code, r)) => {
let text = read_limited(r.into_reader()).unwrap_or_default();
Err(format!("HTTP {code}: {text}"))
Ok(json!({
"events": [{"type": "error", "message": format!("HTTP {code}: {text}")}],
"finish": "error",
"error": format!("HTTP {code}: {text}"),
"transport": "plugin",
}))
}
Err(e) => Err(format!("http POST sse {url}: {e}")),
Err(e) => Ok(json!({
"events": [{"type": "error", "message": format!("http POST sse {url}: {e}")}],
"finish": "error",
"error": format!("http POST sse {url}: {e}"),
"transport": "plugin",
})),
}
} else {
match req.send_string(&body) {
@@ -266,7 +281,14 @@ fn chat_completions(args: &Value) -> Result<Value, String> {
let status = r.status();
let text = read_limited(r.into_reader())?;
if status != 200 {
return Err(format!("HTTP {status}: {text}"));
return Ok(json!({
"text": "",
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
"finish": "error",
"error": format!("HTTP {status}: {text}"),
"tool_calls": [],
"transport": "plugin",
}));
}
let data: Value = serde_json::from_str(&text).map_err(|e| e.to_string())?;
let msg = data
@@ -299,9 +321,23 @@ fn chat_completions(args: &Value) -> Result<Value, String> {
}
Err(ureq::Error::Status(code, r)) => {
let text = read_limited(r.into_reader()).unwrap_or_default();
Err(format!("HTTP {code}: {text}"))
Ok(json!({
"text": "",
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
"finish": "error",
"error": format!("HTTP {code}: {text}"),
"tool_calls": [],
"transport": "plugin",
}))
}
Err(e) => Err(format!("http POST {url}: {e}")),
Err(e) => Ok(json!({
"text": "",
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
"finish": "error",
"error": format!("http POST {url}: {e}"),
"tool_calls": [],
"transport": "plugin",
})),
}
}
}
+64 -1
View File
@@ -3,6 +3,7 @@ package httpx
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
@@ -118,12 +119,74 @@ func jsonWrapInvokeValue(value any) (int, map[string]any, error) {
}
switch t := value.(type) {
case map[string]any:
return http.StatusOK, t, nil
code := http.StatusOK
if ok, has := t["ok"]; has {
if b, isBool := ok.(bool); isBool && !b {
if errMsg, _ := t["error"].(string); strings.EqualFold(errMsg, "unauthorized") ||
strings.Contains(strings.ToLower(errMsg), "unauthorized") {
code = http.StatusUnauthorized
}
}
}
return code, t, nil
default:
return http.StatusOK, map[string]any{"ok": true, "result": value}, nil
}
}
// writeSSE writes OpenAI-compatible (or generic) Server-Sent Events.
// value may be a string body, or a map with key "sse" / "body" / "chunks".
func writeSSE(w http.ResponseWriter, value any) {
w.Header().Set("Content-Type", "text/event-stream; charset=utf-8")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.WriteHeader(http.StatusOK)
flusher, _ := w.(http.Flusher)
writeChunk := func(s string) {
if s == "" {
return
}
if !strings.HasSuffix(s, "\n") {
s += "\n"
}
_, _ = io.WriteString(w, s)
if flusher != nil {
flusher.Flush()
}
}
switch t := value.(type) {
case string:
writeChunk(t)
case map[string]any:
if s, ok := t["sse"].(string); ok {
writeChunk(s)
return
}
if s, ok := t["body"].(string); ok {
writeChunk(s)
return
}
if chunks, ok := t["chunks"].([]any); ok {
for _, c := range chunks {
writeChunk(fmt.Sprint(c))
}
return
}
b, err := json.Marshal(t)
if err != nil {
writeChunk("data: {\"error\":\"sse_encode\"}\n\n")
return
}
writeChunk("data: " + string(b) + "\n\n")
writeChunk("data: [DONE]\n\n")
default:
writeChunk("data: " + fmt.Sprint(value) + "\n\n")
writeChunk("data: [DONE]\n\n")
}
}
func invokeArgsFromPayload(payload map[string]any) map[string]any {
args := map[string]any{}
for k, v := range payload {
+29 -12
View File
@@ -17,8 +17,8 @@ type artifactRoute struct {
Path string
File string
Request string // json | form | none | query
Response string // json | text | html
Auth string // role name or "" / "none"
Response string // json | text | html | sse
Auth string // role name or "" / "none" | "bearer"
}
func parseArtifactRoute(v any) artifactRoute {
@@ -57,16 +57,21 @@ func (st *state) mountArtifactRoutes(mux *http.ServeMux) {
func (st *state) handleArtifact(w http.ResponseWriter, r *http.Request, route artifactRoute) {
if route.Auth != "" && !strings.EqualFold(route.Auth, "none") {
cookie := r.Header.Get("Cookie")
user := session.UsernameFromCookie(cookie)
if user == "" {
jsonErr(w, http.StatusUnauthorized, "auth required")
return
}
role := session.RoleFromCookie(cookie)
if role != "" && role != route.Auth && role != "admin" && role != "superadmin" {
jsonErr(w, http.StatusForbidden, "forbidden")
return
if strings.EqualFold(route.Auth, "bearer") || strings.EqualFold(route.Auth, "api_key") {
// Bearer / API-key auth is enforced inside the Endpoint `# main`
// (payload carries authorization / api_key). Do not require cookies.
} else {
cookie := r.Header.Get("Cookie")
user := session.UsernameFromCookie(cookie)
if user == "" {
jsonErr(w, http.StatusUnauthorized, "auth required")
return
}
role := session.RoleFromCookie(cookie)
if role != "" && role != route.Auth && role != "admin" && role != "superadmin" {
jsonErr(w, http.StatusForbidden, "forbidden")
return
}
}
}
@@ -115,6 +120,16 @@ func (st *state) handleArtifact(w http.ResponseWriter, r *http.Request, route ar
payload[name] = v
}
}
if auth := strings.TrimSpace(r.Header.Get("Authorization")); auth != "" {
if _, exists := payload["authorization"]; !exists {
payload["authorization"] = auth
}
}
if key := strings.TrimSpace(r.Header.Get("X-Api-Key")); key != "" {
if _, exists := payload["api_key"]; !exists {
payload["api_key"] = key
}
}
value, err := plugin.RunArtifact(route.File, payload)
if err != nil {
@@ -131,6 +146,8 @@ func (st *state) handleArtifact(w http.ResponseWriter, r *http.Request, route ar
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, fmt.Sprint(value))
case "sse", "event-stream", "text/event-stream":
writeSSE(w, value)
default:
code, body, err := jsonWrapInvokeValue(value)
if err != nil {
@@ -102,6 +102,18 @@ func (st *state) handleInvoke(w http.ResponseWriter, r *http.Request, route invo
}
}
// Bind common auth / client headers for OpenAI-compatible and API-key surfaces.
if auth := strings.TrimSpace(r.Header.Get("Authorization")); auth != "" {
if _, exists := payload["authorization"]; !exists {
payload["authorization"] = auth
}
}
if key := strings.TrimSpace(r.Header.Get("X-Api-Key")); key != "" {
if _, exists := payload["api_key"]; !exists {
payload["api_key"] = key
}
}
args := invokeArgsFromPayload(payload)
value, err := plugin.CallLibPath(route.FnPath, args)
if err != nil {
@@ -114,10 +126,22 @@ func (st *state) handleInvoke(w http.ResponseWriter, r *http.Request, route invo
strings.Contains(msg, "no active host") {
code = http.StatusBadRequest
}
if strings.Contains(msg, "unauthorized") || strings.Contains(msg, "Unauthorized") {
code = http.StatusUnauthorized
}
jsonErr(w, code, msg)
return
}
// OpenAI-compatible handlers may return `{sse: "..."}` to force event-stream
// even when the route's declared return mode is json.
if m, ok := value.(map[string]any); ok {
if s, ok := m["sse"].(string); ok && strings.TrimSpace(s) != "" {
writeSSE(w, m)
return
}
}
switch route.ReturnMode {
case "text", "html", "text/html":
if value == nil {
@@ -135,6 +159,8 @@ func (st *state) handleInvoke(w http.ResponseWriter, r *http.Request, route invo
w.Header().Set("Content-Type", ct)
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, text)
case "sse", "event-stream", "text/event-stream":
writeSSE(w, value)
case "status":
if value == nil {
jsonErr(w, http.StatusInternalServerError, "null result")
+55
View File
@@ -37,6 +37,61 @@ print("hello-from-python")
Under Markdown this stays easy to read — which is the point: code as documentation, documentation as code.
One file, three surfaces.
| Surface | What it is |
|---|---|
| Document | ordinary Markdown: prose, tables, links and fenced code |
| Program | the same markers are executable: assignments, calls, returns |
| Knowledge | frontmatter, links and generated catalog entries an agent can query |
The static site you are reading is generated from these files, so the examples are not screenshots. Run the same source and you get the same page.
Foreign v2 is a good example. Probe Python, then run the bound fence:
**python = > foreign.probe lang="python"**
1. [ok](`python`)
**hello = > foreign.run code=`hi`**
> print text=`hello`
2. *
> print text="(python3 not found — install Python to run this cell)"
The same module also supports JSON calls, parallel fan-out, persistent kernels and an explicit state bridge. See [stdlib/foreign.mq.md](stdlib/foreign.mq.md) and the full Python walkthrough in [features/19-foreign-view-cells.mq.md](features/19-foreign-view-cells.mq.md).
A view document is still a document. Named kernels and a cell order are ordinary bound tables:
```text
`分析内核` =
| 名称 | 语言 | 说明 |
|------|------|------|
| 分析 | python | shared state |
| 复核 | python | isolated state |
`分析单元` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 准备数据 | 分析 | write |
| 计算指标 | 分析 | read |
| 独立复核 | 复核 | validate |
`会话` = > foreign.session kernels=`分析内核` cells=`分析单元`
```
`marqdo view` consumes the foreign-owned `会话` handle and turns it into Run All, Step, per-cell kernels, a journal and a state panel. Without `会话`, cells are stateless one-shot runs. There is no separate interactive mode or command; `marqdo run` still executes the file as one program.
Documentation becomes knowledge when it can be discovered, verified and reused. OKF supplies the portable Markdown + YAML shape; EKC projects the repository into catalog layers L0–L4, a graph and an engineering preflight. The source `.mq.md` remains the only authority; generated artifacts are marked and rebuildable.
| Loop | Question |
|---|---|
| OKF | How do people and agents share a readable knowledge page? |
| EKC | What existing capability, decision, constraint or failure should be reused before creating more? |
| Preflight | REUSE → ADAPT → CREATE |
Read [features/20-okf-ekc.mq.md](features/20-okf-ekc.mq.md) for the concepts and commands.
Welcome again — enjoy exploring Marqdo.
## greet
+55
View File
@@ -37,6 +37,61 @@ print("hello-from-python")
这些在 Markdown 里读起来仍然像文章——这就是 marqdo 想坚持的:代码即文档,文档即代码。
一份文件,三个面。
| 面 | 是什么 |
|---|---|
| 文档 | 普通 Markdown:叙述、表格、链接、围栏代码 |
| 程序 | 同一批标记可执行:赋值、调用、返回 |
| 知识 | frontmatter、链接与生成的 catalog 条目,Agent 可查询 |
你正在读的静态站就是由这些文件生成的,所以示例不是截图。运行同一份源,会得到同一页。
外联 v2 是个好例子。先探测 Python,再运行绑定围栏:
**python = > 外联.探测 语言="python"**
1. [ok](`python`)
**问候 = > 外联.运行 代码=`hi`**
> 打印 内容=`问候`
2. *
> 打印 内容="(未找到 python3 — 安装 Python 后可运行此单元)"
同一个模块还支持 JSON 调用、并行扇出、持久内核与显式状态桥。见 [stdlib/外联.mq.md](stdlib/外联.mq.md) 与完整 Python 示例 [features/19-外联视图单元.mq.md](features/19-外联视图单元.mq.md)。
view 仍然是一篇文档。命名内核与单元顺序就是普通绑定表:
```text
`分析内核` =
| 名称 | 语言 | 说明 |
|------|------|------|
| 分析 | python | 共享状态 |
| 复核 | python | 隔离状态 |
`分析单元` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 准备数据 | 分析 | 写入 |
| 计算指标 | 分析 | 读取 |
| 独立复核 | 复核 | 校验 |
`会话` = > foreign.session kernels=`分析内核` cells=`分析单元`
```
`marqdo view` 消费 foreign 拥有的 `会话` 句柄,变成 Run All、Step、每单元内核、journal 与状态面板。没有 `会话` 时,单元是无状态的一次性运行。没有单独的交互模式或命令;`marqdo run` 仍然把文件当单个程序运行。
文档能被发现、验证和复用时,才成为知识。OKF 提供可移植的 Markdown + YAML 形态;EKC 把仓库投影成 catalog L0–L4、图与工程 preflight。源 `.mq.md` 始终是唯一权威,生成物有标记、可重建。
| 循环 | 问题 |
|---|---|
| OKF | 人和 Agent 如何共享一页可读知识? |
| EKC | 创建之前,应该复用哪些已有能力、决策、约束与失败? |
| Preflight | REUSE → ADAPT → CREATE |
概念与命令见 [features/20-OKF与EKC.mq.md](features/20-OKF与EKC.mq.md)。
最后,再次欢迎你,尽情体验 marqdo 吧!
## 问候
+3
View File
@@ -11,6 +11,9 @@
| [`keywords/`](keywords/) | 关键字与 L0 内置 |
| [`stdlib/`](stdlib/) | **标准库**(导入方式 + text/表/文件/时间/系统/json/网络/数学/外联/插件/**自写回**/**子任务**;v0.1.2+ 内置于 CLI) |
| [`features/`](features/) | 特性迭代(含对象、官方 `ext/`) |
| [`features/19-foreign-view-cells.mq.md`](features/19-foreign-view-cells.mq.md) / [`19-外联视图单元.mq.md`](features/19-外联视图单元.mq.md) | 外联 v2 完整 Python 示例:probe / JSON call / 并行 / 持久内核 / 状态桥 / view 单元 |
| [`features/20-okf-ekc.mq.md`](features/20-okf-ekc.mq.md) / [`20-OKF与EKC.mq.md`](features/20-OKF与EKC.mq.md) | 代码即文档、文档即知识:OKF 形态与 EKC 工程知识编译器 |
| [`features/21-persistence.mq.md`](features/21-persistence.mq.md) / [`21-持久化.mq.md`](features/21-持久化.mq.md) | 外联结果持久化:静态快照与 writeback |
官方扩展(非 stdlib):`ext/llm`、`ext/agent`、`ext/web`、`ext/quantum`、`ext/linalg` — 见 `features/05-*.mq.md`、`06-*.mq.md`、`07-linalg.mq.md` / `07-线性代数.mq.md` 与 `doc/design/ext-*.md`。
+152
View File
@@ -0,0 +1,152 @@
---
title: Foreign kernels v2 — Python analysis in view
description: Kernel state, Marqdo-to-kernel values and view cells — without a stdin-only demo
import foreign:lib/foreign.mq.md
---
# main
This page shows the foreign runtime as a language feature: a persistent kernel, explicit state from Marqdo values, and view cells that share kernel state. It never asks the reader to type JSON into stdin.
---
Probe the interpreter first. The result is ordinary Marqdo data.
**python = > foreign.probe lang="python"**
> print text=[ok](`python`)
> print text=[version](`python`)
---
Marqdo values into a kernel.
The `订单` table is a normal Marqdo value. `foreign.session_set` copies it into the Python kernel as JSON; no shell and no stdin.
`订单` =
| @ | 区域 | 金额 |
|---|------|------|
| 1 | 华东 | 120 |
| 2 | 华东 | 300 |
| 3 | 华南 | 90 |
| 4 | 华南 | 140 |
| 5 | 华北 | 220 |
| 6 | 西部 | 80 |
**内核会话 = > foreign.session_open lang="python"**
**> foreign.session_set id=`内核会话` name="orders" value=`订单`**
`total_code` =
```python
round(sum(float(row["金额"]) for row in orders), 2)
```
**total = > foreign.session_eval id=`内核会话` code=`total_code`**
> print text=[result](`total`)
The low-level kernel session is managed entirely by `foreign`; the table crossed the boundary as an explicit JSON copy.
---
Kernel state persists.
Two evaluations share the same kernel. The second sees the first variable.
`set_values_code` =
```python
values = [3, 4, 5]
```
**> foreign.session_eval id=`内核会话` code=`set_values_code`**
`square_sum_code` =
```python
sum(value * value for value in values)
```
**square_sum = > foreign.session_eval id=`内核会话` code=`square_sum_code`**
> print text=[result](`square_sum`)
**> foreign.session_close id=`内核会话`**
This is kernel state, not a one-shot process. It is created, queried and closed through `foreign`.
---
View cells share state.
`foreign.session` declares a view session from ordinary tables. The table names are free; this document uses `分析内核` and `分析单元`.
`分析内核` =
| @ | 名称 | 语言 |
|---|------|------|
| 1 | 分析 | python |
| 2 | 复核 | python |
`分析单元` =
| @ | 单元 | 内核 |
|---|------|------|
| 1 | 计算 | 分析 |
| 2 | 复核 | 复核 |
`初始值` =
| 内核 | 名称 | 值 |
|------|------|----|
| 分析 | orders | `订单` |
| 复核 | orders | `订单` |
`会话` = > foreign.session kernels=`分析内核` cells=`分析单元` values=`初始值`
`计算` =
```python
import json
import statistics
by_region = {}
for row in orders:
by_region.setdefault(row["区域"], []).append(float(row["金额"]))
summary = []
for region, values in sorted(by_region.items()):
mean = statistics.mean(values)
sd = statistics.pstdev(values) or 1.0
summary.append({
"region": region,
"orders": len(values),
"total": round(sum(values), 2),
"mean": round(mean, 2),
"z_span": round((max(values) - mean) / sd, 3),
})
print(json.dumps({
"regions": summary,
"total": round(sum(float(row["金额"]) for row in orders), 2),
"count": len(orders),
}, ensure_ascii=False))
```
`复核` =
```python
import json
print(json.dumps({
"kernel": "复核",
"count": len(orders),
"total": round(sum(float(row["金额"]) for row in orders), 2),
}, ensure_ascii=False))
```
`查看` consumes the `会话` handle. Run All runs `计算` in the `分析` kernel and `复核` in the `复核` kernel; the static export runs the same cells and embeds their real outputs.
The important language point:
The `初始值` table passes the same Marqdo `订单` value into both kernels. Kernel state lives in `foreign`. View only renders the handle and its cells. `run` still executes the same document as a program.
There is no separate notebook CLI, no stdin requirement in this example, and no hidden language switch.
+152
View File
@@ -0,0 +1,152 @@
---
title: 外联内核 v2 — Python 分析(view 单元)
description: 内核态、Marqdo 值传入与 view 单元 —— 不依赖 stdin 的完整示例
导入 外联:lib/外联.mq.md
---
# main
这一页把外联运行时当作语言能力来展示:持久内核、从 Marqdo 值显式传入状态,以及共享内核状态的 view 单元。读者不需要向 stdin 输入 JSON。
---
先探测解释器。结果仍然是普通 Marqdo 数据。
**python = > 外联.探测 语言="python"**
> 打印 内容=[ok](`python`)
> 打印 内容=[version](`python`)
---
Marqdo 值进入内核。
`订单` 表是普通 Marqdo 值。`外联.会话设值` 把它作为 JSON 复制进 Python 内核;没有 shell,也没有 stdin。
`订单` =
| @ | 区域 | 金额 |
|---|------|------|
| 1 | 华东 | 120 |
| 2 | 华东 | 300 |
| 3 | 华南 | 90 |
| 4 | 华南 | 140 |
| 5 | 华北 | 220 |
| 6 | 西部 | 80 |
**内核会话 = > 外联.会话开启 语言="python"**
**> 外联.会话设值 标识=`内核会话` 名称="orders" 值=`订单`**
`合计代码` =
```python
round(sum(float(row["金额"]) for row in orders), 2)
```
**合计 = > 外联.会话求值 标识=`内核会话` 代码=`合计代码`**
> 打印 内容=[result](`合计`)
低层内核会话完全由 `外联` 管理;表格以显式 JSON 拷贝跨过边界。
---
内核态是持续的。
两次求值共享同一个内核。第二次能看到第一次创建的变量。
`设值代码` =
```python
values = [3, 4, 5]
```
**> 外联.会话求值 标识=`内核会话` 代码=`设值代码`**
`平方和代码` =
```python
sum(value * value for value in values)
```
**平方和 = > 外联.会话求值 标识=`内核会话` 代码=`平方和代码`**
> 打印 内容=[result](`平方和`)
**> 外联.会话关闭 标识=`内核会话`**
这是内核态,不是一次性进程。它由 `外联` 创建、查询和关闭。
---
view 单元共享状态。
`外联.会话` 从普通表格声明一个 view 会话。表名完全自由;本页用 `分析内核` 和 `分析单元`。
`分析内核` =
| @ | 名称 | 语言 |
|---|------|------|
| 1 | 分析 | python |
| 2 | 复核 | python |
`分析单元` =
| @ | 单元 | 内核 |
|---|------|------|
| 1 | 计算 | 分析 |
| 2 | 复核 | 复核 |
`初始值` =
| 内核 | 名称 | 值 |
|------|------|----|
| 分析 | orders | `订单` |
| 复核 | orders | `订单` |
`会话` = > 外联.会话 内核表=`分析内核` 单元表=`分析单元` 值表=`初始值`
`计算` =
```python
import json
import statistics
by_region = {}
for row in orders:
by_region.setdefault(row["区域"], []).append(float(row["金额"]))
summary = []
for region, values in sorted(by_region.items()):
mean = statistics.mean(values)
sd = statistics.pstdev(values) or 1.0
summary.append({
"region": region,
"orders": len(values),
"total": round(sum(values), 2),
"mean": round(mean, 2),
"z_span": round((max(values) - mean) / sd, 3),
})
print(json.dumps({
"regions": summary,
"total": round(sum(float(row["金额"]) for row in orders), 2),
"count": len(orders),
}, ensure_ascii=False))
```
`复核` =
```python
import json
print(json.dumps({
"kernel": "复核",
"count": len(orders),
"total": round(sum(float(row["金额"]) for row in orders), 2),
}, ensure_ascii=False))
```
`查看` 消费 `会话` 句柄。Run All 在 `分析` 内核执行 `计算`,在 `复核` 内核执行 `复核`;静态导出走同一批单元,并把真实输出嵌入页面。
语言上的重点:
`初始值` 表把同一个 Marqdo `订单` 值传给两个内核。内核态归 `外联` 管理。view 只渲染句柄和单元。`run` 仍然把同一份文档当程序执行。
没有单独的 notebook CLI,这个示例不依赖 stdin,也没有隐藏的语言切换。
+135
View File
@@ -0,0 +1,135 @@
---
title: 代码即文档,文档即知识 — OKF 与 EKC
description: Marqdo 如何以一份可执行源为唯一权威,把它投影成 OKF 风格知识与 Engineering Knowledge Compiler 制品
---
# main
Marqdo 的宪法是「代码即文档,文档即知识库」。`.mq.md` 不是程序截图:它同时是可读源码、可执行单元和知识页。
这个承诺只有在知识始终从源派生时才成立。OKF 提供可移植知识包的形态;EKC 提供工程化投影。二者都不成为第二份权威。
---
OKF 的形,Marqdo 的义。
OKF(Open Knowledge Format)用 Markdown + YAML frontmatter 表示知识:一页一概念、路径即身份、链接成图、强制面刻意保持很小。Marqdo 借用这个形态,但正文是可执行语法。
因此概念页可以直接指向证明它的文件:
```yaml
type: Capability
title: 持久外联内核
description: 在命名内核中运行 Python 或 Node 并保留状态
resource: public/features/19-外联视图单元.mq.md
status: stable
verified:
by: marqdo-knowledge/verify
tags:
- foreign
- view-cells
```
同一页可以在编辑器里读、从 catalog 链接、被 Agent 查询,也可以直接运行。未知字段保留;未知类型降级为可读正文。概念页不是源文件的复制品。
---
哪些是生成的,哪些不是。
| 层 | 作者 | 职责 |
|---|---|---|
| 源 | 人或编码 Agent | .mq.md;唯一语义权威 |
| 目录包 | marqdo catalog | 模块、符号、能力与图投影 |
| 任务知识 | plan 晋升 | 任务、技能与可执行资源包 |
| 静态站 | marqdo view output | 人读 HTML,永远不是权威 |
生成物带 GENERATED 标记,可以删除重建;没有它们源树仍能运行。Catalog 是索引,不是语言特性。
---
EKC:Engineering Knowledge Compiler。
EKC 把仓库投影成工程知识系统,并在 Agent 写新代码前提供 preflight:
```text
.mq.md ──► EKC ──► catalog L0–L4
├─ graph
├─ context pack
└─ Agent preflight
```
| 层 | 知识 | 例子 |
|---|---|---|
| L0 Repository | 语言、规模、根目录 | 仓库总览 |
| L1 Module | imports、exports、职责 | 外联内核模块 |
| L2 Symbol | 签名、调用、指纹 | Python 分析函数 |
| L3 Capability | 工程能做什么 | 运行持久外联内核 |
| L4 Knowledge | 决策、约束、失败 | 为什么状态桥必须显式 |
图记录 implements、uses、depends_on、tested_by、documented_by、constrained_by、decided_by、supersedes、conflicts_with、deprecated_by 等关系。Agent 先沿图发现已有能力,而不是重新造轮子。
---
Engineering First Information:先发现,再创建。
复用协议是硬顺序:
```text
REUSE → ADAPT → CREATE
```
preflight 可以在实现开始前返回已有能力、API、约束、决策与失败经验。知识图缺失时返回明确的 missing_knowledge,而不是静默编造。
五条原则:
| # | 原则 |
|---|---|
| 1 | Discover before Create |
| 2 | Reuse before Adapt |
| 3 | Adapt before Duplicate |
| 4 | Verify before Promote |
| 5 | Code changes update knowledge |
---
命令。
```bash
# 把仓库投影成 catalog + EKC 制品
marqdo catalog .
# 写代码前先问:已经有了什么?
marqdo knowledge preflight "增加一个持久外联内核"
marqdo find "外联 会话"
marqdo reuse "实现 JSON 并行扇出"
# 看一次修改会影响什么
marqdo impact public/features/19-外联视图单元.mq.md
# 校验知识投影
marqdo knowledge verify
```
`marqdo catalog` 保留人读的 `index.md` 与模块页;EKC 在 `.marqdo/` 下增加 L0–L4。两者都从同一棵源树生成。
---
每个声称的能力都必须有可执行资源。
一个知识页说“这个工程能分析订单”,就必须指向一个真正能跑的 `.mq.md` 资源。这是 OKF 形态的笔记与可执行知识系统的区别。
在外联示例里,页面本身就是证明:Python 调用、持久内核和状态桥都从读者正在看的同一份 Markdown 源运行。没有另一个会漂移的演示项目。
---
继续阅读。
OKF 设计:[OKF 设计](../../doc/design/okf.md)
Engineering Knowledge Compiler:[Engineering Knowledge Compiler](../../doc/design/engineering-knowledge.md)
Catalog CLI:[Catalog CLI](../../doc/design/catalog-cli.md)
OKF 与 Marqdo 调研:[OKF 与 Marqdo 调研](../../doc/research/okf-and-marqdo.md)
外联 view 单元示例:[外联 view 单元示例](19-外联视图单元.mq.md)
+135
View File
@@ -0,0 +1,135 @@
---
title: Code as documentation, documentation as knowledge — OKF and EKC
description: How Marqdo keeps one executable source of truth while projecting it into OKF-style knowledge and Engineering Knowledge Compiler artifacts
---
# main
Marqdo's constitution is code as documentation and documentation as knowledge. A `.mq.md` file is not a screenshot of a program: it is the readable source, the executable unit and the knowledge page at the same time.
That claim matters only if the knowledge stays derived from the source. OKF gives us the shape of a portable knowledge bundle; EKC gives us the engineering projection. Neither becomes a second source of truth.
---
OKF shape, Marqdo meaning.
OKF (Open Knowledge Format) describes knowledge as Markdown pages with YAML frontmatter: one concept per page, path as identity, links as graph, a deliberately small required surface. Marqdo borrows that shape, but its body is executable.
A concept page can therefore point at the exact file that proves it:
```yaml
type: Capability
title: Persistent foreign kernels
description: Run Python or Node in a named, stateful kernel
resource: public/features/19-foreign-view-cells.mq.md
status: stable
verified:
by: marqdo-knowledge/verify
tags:
- foreign
- view-cells
```
The same page can be opened in an editor, linked from a catalog, queried by an agent, and executed. Unknown fields are preserved; unknown types degrade to readable prose. The page is a concept, not a copy of one.
---
What is generated, and what is not.
| Layer | Author | Role |
|---|---|---|
| Source | person or coding agent | .mq.md; the only semantic authority |
| Catalog | marqdo catalog | module, symbol, capability and graph projection |
| Task knowledge | plan promotion | task, skill and executable resource bundle |
| Static site | marqdo view output | human-readable HTML, never an authority |
Generated artifacts are marked GENERATED and can be deleted and rebuilt. The source tree still runs without them. A catalog is an index, not a language feature.
---
EKC: the Engineering Knowledge Compiler.
EKC projects the repository into an engineering knowledge system, then exposes it through a preflight before an agent writes new code:
```text
.mq.md ──► EKC ──► catalog L0–L4
├─ graph
├─ context pack
└─ Agent preflight
```
| Layer | Knowledge | Example |
|---|---|---|
| L0 Repository | language, size, roots | repository summary |
| L1 Module | imports, exports, responsibility | a foreign kernel module |
| L2 Symbol | signature, calls, fingerprint | a Python analysis function |
| L3 Capability | what the project can do | run a persistent foreign kernel |
| L4 Knowledge | decisions, constraints, failures | why the bridge is explicit |
The graph records relations such as implements, uses, depends_on, tested_by, documented_by, constrained_by, decided_by, supersedes, conflicts_with and deprecated_by. That graph is how an agent discovers existing capability before inventing a duplicate.
---
Engineering First Information: discover before create.
The reuse protocol is a hard order:
```text
REUSE → ADAPT → CREATE
```
A preflight can return existing capabilities, APIs, constraints, decisions and failure notes before a new implementation starts. If the knowledge graph is missing, the result is an explicit missing_knowledge state, not a silent fabrication.
The five principles are:
| # | Principle |
|---|---|
| 1 | Discover before Create |
| 2 | Reuse before Adapt |
| 3 | Adapt before Duplicate |
| 4 | Verify before Promote |
| 5 | Code changes update knowledge |
---
The commands.
```bash
# Project the repository into catalog + EKC artifacts
marqdo catalog .
# Ask what already exists before writing code
marqdo knowledge preflight "add a persistent foreign kernel"
marqdo find "foreign session"
marqdo reuse "implement JSON fan-out"
# See what a change would touch
marqdo impact public/features/19-foreign-view-cells.mq.md
# Validate the knowledge projection
marqdo knowledge verify
```
`marqdo catalog` keeps the human-readable `index.md` and module pages; EKC adds L0–L4 files under `.marqdo/`. Both are generated from the same source tree.
---
Every claimed capability has an executable resource.
A knowledge page that says “this project can analyze orders” must point to a `.mq.md` resource that actually runs. That is the difference between an OKF-shaped note and an executable knowledge system.
In the foreign example, the page itself is the proof: the Python call, the persistent kernel and the state bridge all run from the same Markdown source that a reader is looking at. There is no separate demo project to drift out of sync.
---
Read more.
OKF design: [OKF design](../../doc/design/okf.md)
Engineering Knowledge Compiler: [Engineering Knowledge Compiler](../../doc/design/engineering-knowledge.md)
Catalog CLI: [Catalog CLI](../../doc/design/catalog-cli.md)
OKF and Marqdo research: [OKF and Marqdo research](../../doc/research/okf-and-marqdo.md)
Foreign view-cells example: [Foreign view-cells example](19-foreign-view-cells.mq.md)
+56
View File
@@ -0,0 +1,56 @@
---
title: Persisting foreign execution results
description: Static snapshots and writeback — how foreign results survive after the process exits
import foreign:lib/foreign.mq.md
import writeback:lib/writeback.mq.md
---
# main
Foreign code has two useful lives: as a published snapshot, and as a living document whose latest result is kept in source. They answer different questions and should not be confused.
---
Static snapshot: publish the result.
`marqdo view output` executes the document's cells and embeds their real output in the generated HTML. The `.mq.md` source is not modified. This is the default choice for user documentation, reviews and stable links: a reader sees the result without running anything, and the source stays clean.
The static snapshot is a point-in-time artifact. If an external service changes, rebuild it; do not treat it as a live dashboard.
---
Writeback: keep the result in the source.
For an exploratory or audit document, pair `foreign` with `lib/writeback`. `writeback.record` writes a keyed output slot next to the call. The slot is visible in the Markdown source, can be reviewed in a diff and can be committed with the document.
| Step | What it does |
|------|--------------|
| Foreign run / Foreign session | Execute the foreign code and return a Marqdo value |
| Writeback record | Persist that value in a named slot |
| git diff | Review the changed result like any other document change |
A real writeback in this page.
This page is itself the runnable example. The code fence below is ordinary Marqdo data; `foreign.run` executes it, then `writeback.record` writes the `total` slot into this source file.
`写回代码` =
```python
print(950)
```
**结果 = > foreign.run code=`写回代码`**
**> writeback.record value=`结果` key="total"**
<!-- marqdo-out total
950
-->
The generated slot is directly below that call. Re-running the page replaces only the `total` slot; it does not duplicate it.
Writeback is explicit. It is not hidden kernel serialization, does not persist arbitrary kernel memory, and does not touch a file unless the document calls it.
---
Choosing.
For published user docs, use the static snapshot. For a living report that must keep the latest run result in the `.mq.md` itself, use writeback. Both can coexist in one repository: static pages for readers, writeback examples for authors and audits.
+56
View File
@@ -0,0 +1,56 @@
---
title: 外联执行结果的持久化
description: 静态快照与自写回 —— 外联结果在进程退出后如何保留
导入 外联:lib/外联.mq.md
import writeback:lib/writeback.mq.md
---
# main
外联代码有两种有用的形态:发布快照,以及把最新结果保留在源码里的活文档。两者回答不同的问题,不应混为一谈。
---
静态快照:发布结果。
`marqdo view output` 执行文档单元,并把真实输出嵌入生成的 HTML。`.mq.md` 源码不会被修改。这是用户文档、评审和稳定链接的默认选择:读者不需要运行任何东西就能看到结果,源码保持干净。
静态快照是某个时间点的产物。外部服务变化后重新构建即可,不要把它当成实时看板。
---
自写回:把结果留在源码里。
对于探索性或审计型文档,把 `外联` 和 `lib/writeback` 搭配使用。`writeback.record` 会在调用点附近写入带 key 的输出槽位。槽位在 Markdown 源码中可见,可以在 diff 中审阅,也可以随文档提交。
| 步骤 | 作用 |
|------|------|
| 外联执行 / 外联会话 | 执行外联代码并返回 Marqdo 值 |
| 写回记录 | 把该值持久化到命名槽位 |
| git diff | 像审阅普通文档改动一样审阅结果 |
这一页本身就是完整示例。
这一页就是可运行的示例。下面的代码围栏是普通 Marqdo 值;`外联.运行` 执行它,然后 `writeback.record` 把 `total` 槽位写进本页源码。
`写回代码` =
```python
print(950)
```
**结果 = > 外联.运行 代码=`写回代码`**
**> writeback.record value=`结果` key="total"**
<!-- marqdo-out total
950
-->
生成的槽位就在这次调用下方。再次运行只替换 `total` 槽位,不会重复添加。
自写回是显式动作。它不是隐藏的内核序列化,不持久化任意内核内存,也不会在文档没有调用时修改文件。
---
如何选择。
面向用户的发布文档用静态快照;需要把最新执行结果留在 `.mq.md` 本身里的活文档用自写回。两者可以共存于同一个仓库:静态页面给读者,自写回示例给作者和审计。
+1 -1
View File
@@ -11,7 +11,7 @@ English-named files export English APIs; Chinese-named files export Chinese APIs
No extra allow flags: importing a module means you intend to use it. Under marqdo view or static export, exit soft-fails and long sleeps are clamped so the console stays up.
Modules: text (text / 文本), table (table / 表), files (fs / 文件), time (time / 时间), system (sys / 系统), json (json only), net (net / 网络), math (math / 数学), foreign (foreign / 外联), plugin (plugin / 插件), writeback (writeback / 自写回), subtask (subtask / 子任务).
Modules: text (text / 文本), table (table / 表), files (fs / 文件), time (time / 时间), system (sys / 系统), json (json only), net (net / 网络), math (math / 数学), foreign (foreign / 外联), plugin (plugin / 插件), writeback (writeback / 自写回), subtask (subtask / 子任务). Foreign v2 adds structured invoke records, JSON calls, parallel fan-out, persistent Python/Node kernels, an explicit state bridge and document-native view cells — see stdlib/foreign.mq.md and features/19-foreign-view-cells.mq.md.
From v0.1.2, official `lib/*.mq.md` is embedded in the `marqdo` binary (disk `lib/` or `MARQDO_LIB` still overrides). Standalone `.exe` works without a separate stdlib zip.
+1 -1
View File
@@ -11,7 +11,7 @@ description: 如何导入官方 lib 模块
导入即表示要用对应能力,无需再加允许旗标。marqdo view 或静态导出里,退出会软失败、长睡眠会被钳制,以免拖垮控制台。
模块:文本(text / 文本)、表(table / 表)、文件(fs / 文件)、时间(time / 时间)、系统(sys / 系统)、json(仅 json)、网络(net / 网络)、数学(math / 数学)、外联(foreign / 外联)、插件(plugin / 插件)、自写回(writeback / 自写回)、子任务(subtask / 子任务)。
模块:文本(text / 文本)、表(table / 表)、文件(fs / 文件)、时间(time / 时间)、系统(sys / 系统)、json(仅 json)、网络(net / 网络)、数学(math / 数学)、外联(foreign / 外联)、插件(plugin / 插件)、自写回(writeback / 自写回)、子任务(subtask / 子任务)。外联 v2 新增结构化执行记录、JSON 调用、并行扇出、Python/Node 持久内核、显式状态桥与文档原生 view 单元 — 见 stdlib/外联.mq.md 与 features/19-外联视图单元.mq.md。
自 v0.1.2 起,官方 `lib/*.mq.md` 内置于 `marqdo` 二进制(磁盘 `lib/` 或 `MARQDO_LIB` 仍可覆盖)。单独下载 `.exe` 即可导入标准库,无需再解压 stdlib 包。
+137 -9
View File
@@ -1,20 +1,148 @@
---
title: lib/foreign — external languages
description: Run local interpreters from Marqdo (English)
title: lib/foreign — foreign kernels v2
description: Structured execution, JSON calls, parallel fan-out, persistent kernels and an explicit state bridge
import foreign:lib/foreign.mq.md
stdin: 21
---
# main
Import lib/foreign.mq.md. Bind a code fence like a formula: empty-RHS assign then a ```lang block. The value type is code. Call run code=`name` to execute via your local interpreter. run_lang runs an inline source string.
`lib/foreign` runs code fences and inline sources with local interpreters. v2 keeps the Markdown surface unchanged while adding structured records, JSON in/out, parallel fan-out, persistent Python/Node kernels and an explicit state bridge.
`hi` =
```python
print("hello-from-python")
The same file is still a document: a bound fence is readable Markdown, and `foreign.*` turns it into an executable unit.
---
Probe before you run.
**node = > foreign.probe lang="node"**
> print text=[ok](`node`)
> print text=[version](`node`)
`probe` returns `{lang, cmd, path, ok, version, error}`. Omit `lang` to inspect every known language.
---
A structured record, not just stdout.
`marker` =
```node
console.log("hello from a foreign kernel");
```
Default command is python (Windows) or python3 (Unix). Override with set_cmd, env MARQDO_FOREIGN_PYTHON, or the command box in live view. Failures ask you to check that configuration.
**record = > foreign.invoke code=`marker`**
**`out` = [foreign.run] code=`hi`**
> print text=[ok](`record`)
> print text=[stdout](`record`)
> print text=[code](`record`)
> print text=`out`
`invoke` never throws on a process failure. Read `ok`, `code`, `stdout`, `stderr`, `ms`, `timed_out`, `truncated`, `result` and `error` from one record.
---
JSON in, JSON out.
`double` =
```node
let input = "";
process.stdin.on("data", chunk => { input += chunk; });
process.stdin.on("end", () => {
console.log(JSON.stringify(JSON.parse(input) * 2));
});
```
**single = > foreign.call code=`double` input=21**
> print text=`single`
The input is encoded as JSON on stdin; the last stdout value is parsed as JSON. Use `output="json"` with `invoke` when you also want the raw record.
---
Parallel fan-out.
`numbers` =
| value |
|-------|
| 1 |
| 2 |
| 3 |
| 4 |
**doubled = > foreign.call_many code=`double` inputs=`numbers` concurrency=4**
> print text=`doubled`
`call_many` applies one program to every input, runs independently, preserves input order, and returns parsed JSON results. `run_many` does the same for full invoke-option maps.
---
A persistent kernel.
**session = > foreign.session_open lang="node"**
**first = > foreign.session_eval id=`session` source="let total = 0; for (let i = 1; i <= 5; i += 1) total += i;"**
**second = > foreign.session_eval id=`session` source="total * 2"**
> print text=[result](`second`)
Variables created in the first evaluation survive into the second. The kernel is not a shell: use `session_eval`, `session_get`, `session_names`, then `session_close`.
---
The explicit state bridge.
**> foreign.session_set id=`session` name="rate" value=0.08**
**rate = > foreign.session_get id=`session` name="rate"**
> print text=`rate`
**> foreign.session_close id=`session`**
This is a JSON copy, not shared memory. The kernel receives a value; Marqdo receives a value. There is no implicit mirroring and no hidden global namespace.
---
Cells in view.
`foreign.session` owns the document's kernel state. The kernel and cell order are ordinary tables passed as arguments; the names are free:
```text
`分析内核` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | python | | shared analysis |
| 复核 | python | | isolated review |
`分析单元` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 准备数据 | 分析 | write state |
| 计算指标 | 分析 | read state |
| 独立复核 | 复核 | isolated kernel |
`会话` = > foreign.session kernels=`分析内核` cells=`分析单元`
```
`view` consumes the `会话` handle and adds Run All, Step, kernel switching, a journal and per-cell state. Without `会话`, view cells are stateless one-shot runs: same shell, no persistent kernel. There is no second CLI mode. See the complete Python walkthrough in [`features/19-foreign-view-cells.mq.md`](../features/19-foreign-view-cells.mq.md).
---
Safety defaults.
| Boundary | Default |
|---|---|
| Execution | requires the host execution capability |
| Shell | no shell; arguments are split explicitly |
| Output | bounded, with truncated |
| Timeout | process group killed on Unix / taskkill /T on Windows |
| CWD | file sandbox root; no implicit escape |
| Parallelism | capped fan-out; each item isolated |
Read the full API in [../../lib/foreign.mq.md](../../lib/foreign.mq.md) and the design in [../../doc/design/foreign-kernel.md](../../doc/design/foreign-kernel.md).
+137 -9
View File
@@ -1,20 +1,148 @@
---
title: lib/外联 — 外部语言
description: 从 Marqdo 调用本机解释器(中文)
title: lib/外联 — 外联内核 v2
description: 结构化执行、JSON 调用、并行扇出、持久内核与显式状态桥
导入 外联:lib/外联.mq.md
stdin: 21
---
# main
导入 lib/外联.mq.md。代码围栏与公式同构:空 RHS 赋值后跟 lang 代码块,得到 code 类型。用 运行 代码=`名` 调用本机解释器。
`lib/外联` 用本机解释器运行代码围栏和内联源码。v2 不改 Markdown 表面,新增结构化记录、JSON 进出、并行扇出、Python/Node 持久内核和显式状态桥。
`hi` =
```python
print("hello-from-python")
文件仍然是一篇文档:绑定的代码围栏是可读的 Markdown,`外联.*` 把它变成可执行单元。
---
先探测,再运行。
**node = > 外联.探测 语言="node"**
> 打印 内容=[ok](`node`)
> 打印 内容=[version](`node`)
`探测` 返回 `{lang, cmd, path, ok, version, error}`。省略语言时探测所有已知语言。
---
结构化记录,而不只是 stdout。
`标记` =
```node
console.log("来自外联内核的问候");
```
默认命令可用 设置命令 或 MARQDO_FOREIGN_PYTHON 覆盖;live view 里可改命令框再点 Run。
**记录 = > 外联.执行 代码=`标记`**
**`out` = [外联.运行] 代码=`hi`**
> 打印 内容=[ok](`记录`)
> 打印 内容=[stdout](`记录`)
> 打印 内容=[code](`记录`)
> print text=`out`
`执行` 在进程失败时不抛错,而是在一条记录里给出 `ok`、`code`、`stdout`、`stderr`、`ms`、`timed_out`、`truncated`、`result` 与 `error`。
---
JSON 进,JSON 出。
`双倍` =
```node
let input = "";
process.stdin.on("data", chunk => { input += chunk; });
process.stdin.on("end", () => {
console.log(JSON.stringify(JSON.parse(input) * 2));
});
```
**单个 = > 外联.调用 代码=`双倍` 输入=21**
> 打印 内容=`单个`
输入编码成 JSON 写到 stdin;stdout 的最后一个值再解析成 JSON。想同时保留原始记录时,用 `执行` 的 `输出="json"`。
---
并行扇出。
`数值表` =
| 数值 |
|------|
| 1 |
| 2 |
| 3 |
| 4 |
**翻倍 = > 外联.批量调用 代码=`双倍` 输入表=`数值表` 并发=4**
> 打印 内容=`翻倍`
`批量调用` 对每个输入独立运行同一程序,保持输入顺序并返回解析后的 JSON 列表;`批量运行` 对完整 invoke 选项表做同样的事。
---
持久内核。
**会话 = > 外联.会话开启 语言="node"**
**第一次 = > 外联.会话求值 标识=`会话` 源码="let total = 0; for (let i = 1; i <= 5; i += 1) total += i;"**
**第二次 = > 外联.会话求值 标识=`会话` 源码="total * 2"**
> 打印 内容=[result](`第二次`)
第一次求值创建的变量会保留到第二次。内核不是 shell:用 `会话求值`、`会话取值`、`会话变量`,最后 `会话关闭`。
---
显式状态桥。
**> 外联.会话设值 标识=`会话` 名称="rate" 值=0.08**
**利率 = > 外联.会话取值 标识=`会话` 名称="rate"**
> 打印 内容=`利率`
**> 外联.会话关闭 标识=`会话`**
这是 JSON 拷贝,不是共享内存:内核收到一个值,Marqdo 取回一个值。没有隐式镜像,也没有隐藏的全局命名空间。
---
view 中的单元。
`foreign.session` 拥有文档的内核状态。内核和单元顺序都是作为参数传入的普通表格,名字可以自由选择:
```text
`分析内核` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | python | | 共享分析 |
| 复核 | python | | 独立复核 |
`分析单元` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 准备数据 | 分析 | 写入状态 |
| 计算指标 | 分析 | 读取状态 |
| 独立复核 | 复核 | 隔离内核 |
`会话` = > foreign.session kernels=`分析内核` cells=`分析单元`
```
`view` 消费 `会话` 句柄,提供 Run All、Step、切换内核、journal 和单元状态。没有 `会话` 时,view 的 cells 是无状态的一次性运行:同一套外壳,但没有持久内核。没有第二个 CLI 模式。完整 Python 示例见 [`features/19-外联视图单元.mq.md`](../features/19-外联视图单元.mq.md)。
---
安全默认值。
| 边界 | 默认 |
|---|---|
| 执行 | 需要宿主执行能力 |
| Shell | 不经过 shell;参数显式切分 |
| 输出 | 有界,带 truncated |
| 超时 | Unix 杀进程组 / Windows taskkill /T |
| 工作目录 | 文件沙箱根目录;不隐式越界 |
| 并行 | 扇出有上限;每个任务隔离 |
完整 API 见 [../../lib/外联.mq.md](../../lib/外联.mq.md),设计见 [../../doc/design/foreign-kernel.md](../../doc/design/foreign-kernel.md)。
+61 -5
View File
@@ -124,11 +124,67 @@ pub fn canonical_param(builtin: &str, param: &str) -> String {
("host_json_stringify" | "stringify", "indent" | "缩进") => "indent".into(),
("host_plot", "steps" | "步数") => "steps".into(),
("host_plot", "derivative" | "导数") => "derivative".into(),
("host_foreign_run", "code" | "代码") => "code".into(),
("host_foreign_run" | "host_foreign_run_lang", "stdin" | "标准输入") => "stdin".into(),
("host_foreign_set_cmd" | "host_foreign_run_lang", "lang" | "语言") => "lang".into(),
("host_foreign_set_cmd", "cmd" | "命令") => "cmd".into(),
("host_foreign_run_lang", "source" | "源码") => "source".into(),
("host_foreign_run" | "host_foreign_invoke" | "host_foreign_call" | "host_foreign_call_many" | "host_foreign_session_eval", "code" | "代码") => "code".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke"
| "host_foreign_call",
"stdin" | "标准输入",
) => "stdin".into(),
(
"host_foreign_set_cmd"
| "host_foreign_run_lang"
| "host_foreign_invoke"
| "host_foreign_probe"
| "host_foreign_register"
| "host_foreign_unset"
| "host_foreign_call"
| "host_foreign_call_many"
| "host_foreign_session_open",
"lang" | "语言",
) => "lang".into(),
(
"host_foreign_set_cmd"
| "host_foreign_invoke"
| "host_foreign_register"
| "host_foreign_call_many"
| "host_foreign_session_open",
"cmd" | "命令",
) => "cmd".into(),
("host_foreign_run_lang" | "host_foreign_invoke" | "host_foreign_call" | "host_foreign_call_many" | "host_foreign_session_eval", "source" | "源码") => "source".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke" | "host_foreign_call" | "host_foreign_call_many" | "host_foreign_session_open" | "host_foreign_session_eval",
"env" | "环境",
) => "env".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke" | "host_foreign_call" | "host_foreign_call_many" | "host_foreign_session_open" | "host_foreign_session_eval",
"cwd" | "工作目录",
) => "cwd".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke" | "host_foreign_call" | "host_foreign_call_many" | "host_foreign_session_open" | "host_foreign_session_eval",
"timeout" | "超时",
) => "timeout".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke"
| "host_foreign_register" | "host_foreign_call" | "host_foreign_call_many",
"args" | "参数",
) => "args".into(),
(
"host_foreign_run" | "host_foreign_run_lang" | "host_foreign_invoke" | "host_foreign_call_many",
"max_output" | "最大输出",
) => "max_output".into(),
("host_foreign_invoke" | "host_foreign_call" | "host_foreign_session_eval", "input" | "输入") => "input".into(),
("host_foreign_session_eval" | "host_foreign_session_set" | "host_foreign_session_get" | "host_foreign_session_names" | "host_foreign_session_close", "id" | "标识") => "id".into(),
("host_foreign_session_set" | "host_foreign_session_get", "name" | "名称") => "name".into(),
("host_foreign_session_set", "value" | "值") => "value".into(),
("host_foreign_invoke", "output" | "输出") => "output".into(),
("host_foreign_invoke", "keep" | "保留") => "keep".into(),
("host_foreign_register", "template" | "模板") => "template".into(),
("host_foreign_register", "ext" | "扩展名") => "ext".into(),
("host_foreign_register", "version_args" | "版本参数") => "version_args".into(),
("host_foreign_load_config", "path" | "路径") => "path".into(),
("host_foreign_run_many", "runs" | "任务表") => "runs".into(),
("host_foreign_call_many", "inputs" | "输入表") => "inputs".into(),
("host_foreign_run_many" | "host_foreign_call_many", "concurrency" | "并发") => "concurrency".into(),
("host_ext_native_path", "name" | "名") => "name".into(),
_ => param.to_string(),
}
+3
View File
@@ -89,7 +89,10 @@ impl StmtEventKind {
pub struct CallFrame {
pub call_id: u32,
pub fn_name: String,
/// Definition line of the callee.
pub fn_line: u32,
/// Call site line in the executing document (0 for the entry call).
pub call_line: u32,
pub ordinal: u32,
pub args_brief: String,
}
+42 -23
View File
@@ -1,30 +1,49 @@
//! Foreign / code-block helpers (外联).
/// Default interpreter argv for a language tag (before env / set_cmd).
pub fn default_argv(lang: &str) -> Option<Vec<String>> {
match lang.to_ascii_lowercase().as_str() {
"python" | "python3" | "py" => Some(default_python_argv()),
"javascript" | "js" | "node" => Some(vec!["node".into()]),
_ => None,
}
}
pub mod registry;
pub fn default_python_argv() -> Vec<String> {
#[cfg(windows)]
{
vec!["python".into()]
}
#[cfg(not(windows))]
{
vec!["python3".into()]
}
}
pub use registry::{default_argv, default_cmd_display, default_python_argv};
/// Placeholder shown in the view command box.
pub fn default_cmd_display(lang: &str) -> String {
default_argv(lang)
.map(|a| a.join(" "))
.unwrap_or_else(|| lang.to_string())
/// Shell-like split: whitespace, single/double quotes, backslash escapes.
pub fn split_cmd_line(s: &str) -> Vec<String> {
let mut out = Vec::new();
let mut cur = String::new();
let mut quote: Option<char> = None;
let mut chars = s.chars().peekable();
while let Some(c) = chars.next() {
match quote {
Some(q) => {
if c == q {
quote = None;
} else if c == '\\' && q == '"' {
if let Some(n) = chars.next() {
cur.push(n);
}
} else {
cur.push(c);
}
}
None => {
if c.is_whitespace() {
if !cur.is_empty() {
out.push(std::mem::take(&mut cur));
}
} else if c == '\'' || c == '"' {
quote = Some(c);
} else if c == '\\' {
if let Some(n) = chars.next() {
cur.push(n);
}
} else {
cur.push(c);
}
}
}
}
if !cur.is_empty() {
out.push(cur);
}
out
}
/// Parse ```lang … opener → language id (first token).
+188
View File
@@ -0,0 +1,188 @@
//! Language presets and specs for `lib/foreign` (pure data; no process spawning).
//!
//! The host runner resolves a `LangSpec`, then builds the command. Keeping this
//! module free of `std::process` lets the browser/wasm build share the syntax and
//! preset tables.
use std::path::PathBuf;
/// One language's invocation recipe.
#[derive(Debug, Clone, Default)]
pub struct LangSpec {
pub lang: String,
/// Simple interpreter prefix; the script path is appended.
pub argv: Vec<String>,
/// Optional full argv template; supports `{file}`/`{inline}`/… placeholders.
pub template: Option<Vec<String>>,
pub ext: Option<String>,
pub env: Vec<(String, String)>,
/// Arguments appended to the executable for `probe` (e.g. `--version`).
pub version_args: Vec<String>,
/// Persistent session bootstrap available.
pub session: bool,
/// Directory for resolving a relative interpreter path (config file dir).
pub base_dir: Option<PathBuf>,
}
impl LangSpec {
fn simple(lang: &str, argv: &[&str], ext: &str, version: &[&str], session: bool) -> Self {
Self {
lang: lang.to_string(),
argv: argv.iter().map(|s| (*s).to_string()).collect(),
template: None,
ext: Some(ext.to_string()),
env: Vec::new(),
version_args: version.iter().map(|s| (*s).to_string()).collect(),
session,
base_dir: None,
}
}
fn templated(lang: &str, template: &[&str], ext: &str, version: &[&str]) -> Self {
Self {
lang: lang.to_string(),
argv: vec![template[0].to_string()],
template: Some(template.iter().map(|s| (*s).to_string()).collect()),
ext: Some(ext.to_string()),
env: Vec::new(),
version_args: version.iter().map(|s| (*s).to_string()).collect(),
session: false,
base_dir: None,
}
}
}
pub fn default_python_argv() -> Vec<String> {
#[cfg(windows)]
{
vec!["python".into()]
}
#[cfg(not(windows))]
{
vec!["python3".into()]
}
}
/// Canonical preset id for a language or alias. Unknown names pass through lowercased.
pub fn canonical(lang: &str) -> String {
let l = lang.to_ascii_lowercase();
match l.as_str() {
"python3" | "py" => "python".into(),
"javascript" | "js" => "node".into(),
"shell" | "sh" => "bash".into(),
"rscript" => "r".into(),
"pwsh" | "ps1" => "powershell".into(),
"sqlite3" => "sqlite".into(),
"typescript" => "tsx".into(),
_ => l,
}
}
/// Built-in preset for a language/alias.
pub fn preset_spec(lang: &str) -> Option<LangSpec> {
let id = canonical(lang);
Some(match id.as_str() {
"python" => LangSpec {
argv: default_python_argv(),
..LangSpec::simple("python", &["python3"], "py", &["--version"], true)
},
"node" => LangSpec::simple("node", &["node"], "js", &["-v"], true),
"deno" => LangSpec::simple("deno", &["deno", "run"], "ts", &["--version"], false),
"bun" => LangSpec::simple("bun", &["bun"], "js", &["--version"], false),
"ruby" => LangSpec::simple("ruby", &["ruby"], "rb", &["--version"], false),
"perl" => LangSpec::simple("perl", &["perl"], "pl", &["--version"], false),
"php" => LangSpec::templated("php", &["php", "-r", "{inline}"], "php", &["--version"]),
"lua" => LangSpec::simple("lua", &["lua"], "lua", &["-v"], false),
"r" => LangSpec::simple("r", &["Rscript"], "R", &["--version"], false),
"julia" => LangSpec::simple("julia", &["julia"], "jl", &["--version"], false),
"go" => LangSpec::templated("go", &["go", "run", "{file}"], "go", &["version"]),
"bash" => LangSpec::simple("bash", &["bash"], "sh", &["--version"], false),
"powershell" => {
#[cfg(windows)]
{
LangSpec::simple("powershell", &["powershell"], "ps1", &["--version"], false)
}
#[cfg(not(windows))]
{
LangSpec::simple("powershell", &["pwsh"], "ps1", &["--version"], false)
}
}
"sqlite" => LangSpec::simple("sqlite", &["sqlite3"], "sql", &["--version"], false),
"tsx" => LangSpec::simple("tsx", &["tsx"], "ts", &["--version"], false),
_ => return None,
})
}
/// Sorted preset ids (for `langs`).
pub fn preset_ids() -> Vec<String> {
[
"python", "node", "deno", "bun", "ruby", "perl", "php", "lua", "r", "julia", "go",
"bash", "powershell", "sqlite", "tsx",
]
.iter()
.map(|s| (*s).to_string())
.collect()
}
/// Display string for the command box (e.g. `python3`, `go run`, `php -r`).
pub fn default_cmd_display(lang: &str) -> String {
match preset_spec(lang) {
Some(spec) => {
if let Some(t) = &spec.template {
let mut parts = Vec::new();
for x in t {
if x.contains('{') {
break;
}
parts.push(x.clone());
}
if parts.is_empty() {
lang.to_string()
} else {
parts.join(" ")
}
} else {
spec.argv.join(" ")
}
}
None => lang.to_string(),
}
}
/// Default argv for v1 compatibility (simple presets only).
pub fn default_argv(lang: &str) -> Option<Vec<String>> {
preset_spec(lang).map(|s| s.argv)
}
/// Session support for a language (based on preset).
pub fn session_supported(lang: &str) -> bool {
preset_spec(lang).map(|s| s.session).unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn aliases_canonicalize() {
assert_eq!(canonical("py"), "python");
assert_eq!(canonical("javascript"), "node");
assert_eq!(canonical("sh"), "bash");
assert_eq!(canonical("unknown"), "unknown");
}
#[test]
fn presets_cover_docs() {
assert_eq!(preset_ids().len(), 15);
assert!(preset_spec("python").is_some());
assert!(preset_spec("node").is_some());
assert!(preset_spec("go").unwrap().template.is_some());
assert!(preset_spec("php").unwrap().template.is_some());
}
#[test]
fn display_uses_template_prefix() {
assert_eq!(default_cmd_display("go"), "go run");
assert_eq!(default_cmd_display("php"), "php -r");
}
}
+403 -3
View File
@@ -202,6 +202,22 @@ pub enum HostFn {
MetaGet = 193,
ModuleSource = 194,
ArtifactKind = 195,
ForeignInvoke = 196,
ForeignProbe = 197,
ForeignRegister = 198,
ForeignUnset = 199,
ForeignLoadConfig = 200,
ForeignCall = 201,
ForeignRunMany = 202,
ForeignCallMany = 203,
ForeignSessionOpen = 204,
ForeignSessionEval = 205,
ForeignSessionSet = 206,
ForeignSessionGet = 207,
ForeignSessionNames = 208,
ForeignSessionClose = 209,
ForeignSessionList = 210,
ForeignSessionInterrupt = 211,
}
@@ -394,6 +410,22 @@ impl HostFn {
193 => Self::MetaGet,
194 => Self::ModuleSource,
195 => Self::ArtifactKind,
196 => Self::ForeignInvoke,
197 => Self::ForeignProbe,
198 => Self::ForeignRegister,
199 => Self::ForeignUnset,
200 => Self::ForeignLoadConfig,
201 => Self::ForeignCall,
202 => Self::ForeignRunMany,
203 => Self::ForeignCallMany,
204 => Self::ForeignSessionOpen,
205 => Self::ForeignSessionEval,
206 => Self::ForeignSessionSet,
207 => Self::ForeignSessionGet,
208 => Self::ForeignSessionNames,
209 => Self::ForeignSessionClose,
210 => Self::ForeignSessionList,
211 => Self::ForeignSessionInterrupt,
_ => return None,
})
}
@@ -470,6 +502,22 @@ impl HostFn {
"host_foreign_set_cmd" => Self::ForeignSetCmd,
"host_foreign_run" => Self::ForeignRun,
"host_foreign_run_lang" => Self::ForeignRunLang,
"host_foreign_invoke" | "foreign_invoke" => Self::ForeignInvoke,
"host_foreign_probe" | "foreign_probe" => Self::ForeignProbe,
"host_foreign_register" | "foreign_register" => Self::ForeignRegister,
"host_foreign_unset" | "foreign_unset" => Self::ForeignUnset,
"host_foreign_load_config" | "foreign_load_config" => Self::ForeignLoadConfig,
"host_foreign_call" | "foreign_call" => Self::ForeignCall,
"host_foreign_run_many" | "foreign_run_many" => Self::ForeignRunMany,
"host_foreign_call_many" | "foreign_call_many" => Self::ForeignCallMany,
"host_foreign_session_open" | "foreign_session_open" => Self::ForeignSessionOpen,
"host_foreign_session_eval" | "foreign_session_eval" => Self::ForeignSessionEval,
"host_foreign_session_set" | "foreign_session_set" => Self::ForeignSessionSet,
"host_foreign_session_get" | "foreign_session_get" => Self::ForeignSessionGet,
"host_foreign_session_names" | "foreign_session_names" => Self::ForeignSessionNames,
"host_foreign_session_close" | "foreign_session_close" => Self::ForeignSessionClose,
"host_foreign_session_list" | "foreign_session_list" => Self::ForeignSessionList,
"host_foreign_session_interrupt" | "foreign_session_interrupt" => Self::ForeignSessionInterrupt,
"host_foreign_langs" => Self::ForeignLangs,
"host_dotenv_load" | "load_dotenv" => Self::DotenvLoad,
"host_http_request" | "http_request" => Self::HttpRequest,
@@ -666,6 +714,22 @@ impl HostFn {
Self::ForeignSetCmd => "host_foreign_set_cmd",
Self::ForeignRun => "host_foreign_run",
Self::ForeignRunLang => "host_foreign_run_lang",
Self::ForeignInvoke => "host_foreign_invoke",
Self::ForeignProbe => "host_foreign_probe",
Self::ForeignRegister => "host_foreign_register",
Self::ForeignUnset => "host_foreign_unset",
Self::ForeignLoadConfig => "host_foreign_load_config",
Self::ForeignCall => "host_foreign_call",
Self::ForeignRunMany => "host_foreign_run_many",
Self::ForeignCallMany => "host_foreign_call_many",
Self::ForeignSessionOpen => "host_foreign_session_open",
Self::ForeignSessionEval => "host_foreign_session_eval",
Self::ForeignSessionSet => "host_foreign_session_set",
Self::ForeignSessionGet => "host_foreign_session_get",
Self::ForeignSessionNames => "host_foreign_session_names",
Self::ForeignSessionClose => "host_foreign_session_close",
Self::ForeignSessionList => "host_foreign_session_list",
Self::ForeignSessionInterrupt => "host_foreign_session_interrupt",
Self::ForeignLangs => "host_foreign_langs",
Self::DotenvLoad => "host_dotenv_load",
Self::HttpRequest => "host_http_request",
@@ -832,6 +896,19 @@ impl HostFn {
Self::ForeignSetCmd => &["lang", "cmd"],
Self::ForeignRun => &["code"],
Self::ForeignRunLang => &["lang", "source"],
Self::ForeignInvoke | Self::ForeignCall => &[],
Self::ForeignProbe | Self::ForeignLoadConfig => &[],
Self::ForeignRegister | Self::ForeignUnset => &["lang"],
Self::ForeignRunMany => &["runs"],
Self::ForeignCallMany => &["inputs"],
Self::ForeignSessionOpen => &["lang"],
Self::ForeignSessionEval => &["id"],
Self::ForeignSessionSet => &["id", "name", "value"],
Self::ForeignSessionGet => &["id", "name"],
Self::ForeignSessionNames => &["id"],
Self::ForeignSessionClose => &["id"],
Self::ForeignSessionList => &[],
Self::ForeignSessionInterrupt => &["id"],
Self::ForeignLangs => &[],
Self::PluginLoad => &["path"],
Self::PluginUnload | Self::PluginList => &[],
@@ -935,7 +1012,58 @@ impl HostFn {
Self::PlotPoints => &["path", "grid"],
Self::PlotConic => &["b", "h", "k", "path", "grid"],
Self::ForeignSetCmd => &["args"],
Self::ForeignRun | Self::ForeignRunLang => &["stdin"],
Self::ForeignRun | Self::ForeignRunLang => {
&["stdin", "env", "cwd", "timeout", "args", "max_output"]
}
Self::ForeignInvoke => &[
"code",
"lang",
"source",
"stdin",
"input",
"env",
"cwd",
"timeout",
"args",
"output",
"max_output",
"cmd",
"keep",
],
Self::ForeignProbe => &["lang"],
Self::ForeignRegister => &[
"cmd",
"args",
"template",
"ext",
"env",
"version_args",
],
Self::ForeignUnset => &[],
Self::ForeignLoadConfig => &["path"],
Self::ForeignCall => &[
"code", "lang", "source", "input", "stdin", "env", "cwd", "timeout", "args",
],
Self::ForeignRunMany => &["concurrency"],
Self::ForeignCallMany => &[
"code",
"lang",
"source",
"cmd",
"env",
"cwd",
"timeout",
"args",
"max_output",
"concurrency",
],
Self::ForeignSessionOpen => &["cmd", "env", "cwd", "timeout"],
Self::ForeignSessionEval => &["code", "source", "input", "stdin", "timeout"],
Self::ForeignSessionSet
| Self::ForeignSessionGet
| Self::ForeignSessionNames
| Self::ForeignSessionClose
| Self::ForeignSessionList => &[],
Self::WritebackRecord | Self::WritebackGet | Self::WritebackClear => {
&["at_end", "key", "line"]
}
@@ -1184,7 +1312,12 @@ pub fn call_host(
HostFn::ForeignRun => {
#[cfg(feature = "exec-host")]
{
foreign::run(ctx, require(bound, "code")?, bound.get("stdin"))
foreign::run_with_options(
ctx,
require(bound, "code")?,
bound.get("stdin"),
bound,
)
}
#[cfg(not(feature = "exec-host"))]
{
@@ -1195,11 +1328,12 @@ pub fn call_host(
HostFn::ForeignRunLang => {
#[cfg(feature = "exec-host")]
{
foreign::run_lang(
foreign::run_lang_with_options(
ctx,
require(bound, "lang")?,
require(bound, "source")?,
bound.get("stdin"),
bound,
)
}
#[cfg(not(feature = "exec-host"))]
@@ -1208,6 +1342,272 @@ pub fn call_host(
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignInvoke => {
#[cfg(feature = "exec-host")]
{
foreign::invoke(
ctx,
foreign::InvokeArgs {
code: bound.get("code"),
lang: bound.get("lang"),
source: bound.get("source"),
stdin: bound.get("stdin"),
input: bound.get("input"),
env: bound.get("env"),
cwd: bound.get("cwd"),
timeout: bound.get("timeout"),
args: bound.get("args"),
output: bound.get("output"),
max_output: bound.get("max_output"),
cmd: bound.get("cmd"),
keep: bound.get("keep"),
},
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignProbe => {
#[cfg(feature = "exec-host")]
{
foreign::probe(ctx, bound.get("lang"))
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignRegister => {
#[cfg(feature = "exec-host")]
{
foreign::register(
ctx,
require(bound, "lang")?,
bound.get("cmd"),
bound.get("args"),
bound.get("template"),
bound.get("ext"),
bound.get("env"),
bound.get("version_args"),
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignUnset => {
#[cfg(feature = "exec-host")]
{
foreign::unset(ctx, require(bound, "lang")?)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignLoadConfig => {
#[cfg(feature = "exec-host")]
{
foreign::load_config(ctx, bound.get("path"))
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignCall => {
#[cfg(feature = "exec-host")]
{
foreign::call(
ctx,
foreign::InvokeArgs {
code: bound.get("code"),
lang: bound.get("lang"),
source: bound.get("source"),
stdin: bound.get("stdin"),
input: bound.get("input"),
env: bound.get("env"),
cwd: bound.get("cwd"),
timeout: bound.get("timeout"),
args: bound.get("args"),
output: None,
max_output: None,
cmd: None,
keep: None,
},
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignRunMany => {
#[cfg(feature = "exec-host")]
{
foreign::run_many(
ctx,
require(bound, "runs")?,
bound.get("concurrency"),
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignCallMany => {
#[cfg(feature = "exec-host")]
{
foreign::call_many(
ctx,
foreign::InvokeArgs {
code: bound.get("code"),
lang: bound.get("lang"),
source: bound.get("source"),
stdin: None,
input: None,
env: bound.get("env"),
cwd: bound.get("cwd"),
timeout: bound.get("timeout"),
args: bound.get("args"),
output: None,
max_output: bound.get("max_output"),
cmd: bound.get("cmd"),
keep: None,
},
require(bound, "inputs")?,
bound.get("concurrency"),
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionOpen => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::open(
ctx,
require(bound, "lang")?,
bound.get("cmd"),
bound.get("env"),
bound.get("cwd"),
bound.get("timeout"),
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionEval => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::eval_with_stdin(
ctx,
require(bound, "id")?,
bound.get("code"),
bound.get("source"),
bound.get("input"),
bound.get("stdin").and_then(|v| v.reveal_text()),
bound.get("timeout"),
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionSet => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::set(
ctx,
require(bound, "id")?,
require(bound, "name")?,
require(bound, "value")?,
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionGet => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::get(
ctx,
require(bound, "id")?,
require(bound, "name")?,
)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionNames => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::names(ctx, require(bound, "id")?)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionClose => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::close(ctx, require(bound, "id")?)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionInterrupt => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::interrupt(ctx, require(bound, "id")?)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = (ctx, bound);
Err("foreign unavailable in browser wasm".into())
}
}
HostFn::ForeignSessionList => {
#[cfg(feature = "exec-host")]
{
crate::host::foreign_session::list(ctx)
}
#[cfg(not(feature = "exec-host"))]
{
let _ = ctx;
Ok(Value::List(Vec::new()))
}
}
HostFn::ForeignLangs => {
#[cfg(feature = "exec-host")]
{
+1814 -225
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+791
View File
@@ -0,0 +1,791 @@
//! Persistent foreign kernels (Python / Node) and the session state bridge.
//!
//! Protocol: one JSON request per stdin line, one JSON response per original
//! stdout line. User stdout/stderr are captured inside the bootstrap.
use std::io::{BufRead, BufReader, Read, Write};
use std::process::{Child, Command, Stdio};
use std::sync::mpsc::{self, Receiver};
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use crate::host::foreign::{ensure_config, resolve_session_spec};
use crate::host::json::{json_to_value, value_to_json};
use crate::host::HostContext;
use crate::value::Value;
const SESSION_TIMEOUT_SECS: f64 = 10.0;
const MAX_SESSION_TIMEOUT_SECS: f64 = 3600.0;
const MAX_PROCESS_STDERR: usize = 1_048_576;
const PY_BOOTSTRAP: &str = r#"import sys, json, io, ast, traceback, contextlib
_out = sys.stdout
def _send(o):
_out.write(json.dumps(o, ensure_ascii=False, default=str) + "\n")
_out.flush()
_g = {"__name__": "__main__", "marqdo_input": None}
for _line in sys.stdin:
_line = _line.strip()
if not _line:
continue
try:
req = json.loads(_line)
except Exception:
_send({"id": None, "ok": False, "stdout": "", "stderr": "", "result": None, "error": "bad request"})
continue
rid = req.get("id")
op = req.get("op", "eval")
if op == "set":
_g[req.get("name")] = req.get("value")
_send({"id": rid, "ok": True, "stdout": "", "stderr": "", "result": None, "error": ""})
continue
if op == "get":
try:
_val = _g[req.get("name")]
except KeyError:
_val = None
_send({"id": rid, "ok": True, "stdout": "", "stderr": "", "result": _val, "error": ""})
continue
if op == "names":
_names = sorted(k for k in _g.keys() if not k.startswith("__"))
_send({"id": rid, "ok": True, "stdout": "", "stderr": "", "result": _names, "error": ""})
continue
_code = req.get("code", "")
_g["marqdo_input"] = req.get("input")
_old_stdin = sys.stdin
if req.get("stdin") is not None:
sys.stdin = io.StringIO(str(req.get("stdin")))
_out_buf = io.StringIO()
_err_buf = io.StringIO()
try:
_tree = ast.parse(_code, "<marqdo>")
_result = None
with contextlib.redirect_stdout(_out_buf), contextlib.redirect_stderr(_err_buf):
if _tree.body and isinstance(_tree.body[-1], ast.Expr):
_last = _tree.body.pop()
if _tree.body:
exec(compile(_tree, "<marqdo>", "exec"), _g)
_result = eval(compile(ast.Expression(_last.value), "<marqdo>", "eval"), _g)
else:
exec(compile(_tree, "<marqdo>", "exec"), _g)
_send({"id": rid, "ok": True, "stdout": _out_buf.getvalue(), "stderr": _err_buf.getvalue(), "result": _result, "error": ""})
except BaseException:
_send({"id": rid, "ok": False, "stdout": _out_buf.getvalue(), "stderr": _err_buf.getvalue(), "result": None, "error": traceback.format_exc(limit=8)})
finally:
sys.stdin = _old_stdin
"#;
const JS_BOOTSTRAP: &str = r#"const readline = require("readline");
const util = require("util");
const vm = require("vm");
const out = process.stdout.write.bind(process.stdout);
globalThis.require = require;
const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
function send(o) { out(JSON.stringify(o) + "\n"); }
function toJSON(v) { try { return JSON.parse(JSON.stringify(v)); } catch (e) { return String(v); } }
async function capture(fn) {
const logs = [], errs = [];
const ow = process.stdout.write.bind(process.stdout);
const ew = process.stderr.write.bind(process.stderr);
const ol = console.log, oi = console.info, owr = console.warn, oe = console.error;
console.log = console.info = (...a) => { logs.push(util.format(...a) + "\n"); };
console.warn = console.error = (...a) => { errs.push(util.format(...a) + "\n"); };
process.stdout.write = (c) => { logs.push(String(c)); return true; };
process.stderr.write = (c) => { errs.push(String(c)); return true; };
let result, error = "";
try { result = await fn(); } catch (e) { error = (e && e.stack) || String(e); }
process.stdout.write = ow; process.stderr.write = ew;
console.log = ol; console.info = oi; console.warn = owr; console.error = oe;
return { logs: logs.join(""), errs: errs.join(""), result: result === undefined ? null : result, error };
}
rl.on("line", async (line) => {
let req;
try { req = JSON.parse(line); } catch (e) { send({ id: null, ok: false, stdout: "", stderr: "", result: null, error: "bad request" }); return; }
const rid = req.id, op = req.op || "eval";
if (op === "set") { globalThis[req.name] = req.value; send({ id: rid, ok: true, stdout: "", stderr: "", result: null, error: "" }); return; }
if (op === "get") {
let result = null;
try { result = vm.runInThisContext(String(req.name), { filename: "<marqdo-get>" }); }
catch (e) { result = globalThis[req.name] === undefined ? null : globalThis[req.name]; }
send({ id: rid, ok: true, stdout: "", stderr: "", result: toJSON(result), error: "" });
return;
}
if (op === "names") {
const skip = new Set(["global","globalThis","process","Buffer","console","require","module","exports","__dirname","__filename","performance","crypto","navigator","fetch","URL","URLSearchParams","TextEncoder","TextDecoder","setTimeout","clearTimeout","setInterval","clearInterval","queueMicrotask","structuredClone","atob","btoa","marqdo_input"]);
const names = Object.keys(globalThis).filter(k => !k.startsWith("__") && !skip.has(k) && typeof globalThis[k] !== "function");
send({ id: rid, ok: true, stdout: "", stderr: "", result: names, error: "" });
return;
}
globalThis.marqdo_input = req.input === undefined ? null : req.input;
const oldStdin = process.stdin;
let stdinPatched = false;
if (req.stdin !== undefined && req.stdin !== null) {
try {
const { PassThrough } = require("stream");
const ps = new PassThrough();
ps.end(String(req.stdin));
Object.defineProperty(process, "stdin", { value: ps, configurable: true, writable: true });
stdinPatched = true;
} catch (e) {}
}
const cap = await capture(async () => vm.runInThisContext(req.code || "", { filename: "<marqdo>" }));
if (stdinPatched) {
try { Object.defineProperty(process, "stdin", { value: oldStdin, configurable: true, writable: true }); } catch (e) {}
}
send({ id: rid, ok: !cap.error, stdout: cap.logs, stderr: cap.errs, result: toJSON(cap.result), error: cap.error });
});
"#;
pub struct Session {
pub id: String,
pub lang: String,
pub cmd: String,
pub started_ms: i64,
pub timeout: Duration,
pub evals: u64,
pub alive: bool,
child: Child,
stdin: std::process::ChildStdin,
rx: Receiver<String>,
_reader: JoinHandle<()>,
stderr: Arc<Mutex<String>>,
_stderr_reader: JoinHandle<()>,
}
impl std::fmt::Debug for Session {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Session")
.field("id", &self.id)
.field("lang", &self.lang)
.field("evals", &self.evals)
.field("alive", &self.alive)
.finish()
}
}
impl Drop for Session {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
}
}
fn as_text<'a>(v: &'a Value, what: &str) -> Result<&'a str, String> {
match v {
Value::Text(s) => Ok(s.as_str()),
_ => Err(format!("{what} must be text")),
}
}
fn opt_text(v: Option<&Value>) -> Result<Option<String>, String> {
match v {
None | Some(Value::None) => Ok(None),
Some(Value::Text(s)) => Ok(Some(s.clone())),
Some(Value::Secret(s)) => Ok(Some(s.clone())),
Some(_) => Err("value must be text".into()),
}
}
fn parse_timeout(v: Option<&Value>, default: f64) -> Result<Duration, String> {
let secs = match v {
None | Some(Value::None) => default,
Some(Value::Int(n)) => *n as f64,
Some(Value::Num(n)) => *n,
Some(_) => return Err("timeout must be a number of seconds".into()),
};
if !secs.is_finite() || secs <= 0.0 {
return Err("timeout must be > 0 seconds".into());
}
Ok(Duration::from_secs_f64(secs.min(MAX_SESSION_TIMEOUT_SECS)))
}
fn parse_env(v: Option<&Value>) -> Result<Vec<(String, String)>, String> {
match v {
None | Some(Value::None) => Ok(Vec::new()),
Some(Value::Map(entries)) => Ok(entries
.iter()
.map(|(k, v)| {
let val = match v {
Value::Secret(s) => s.clone(),
other => other.as_display(),
};
(k.clone(), val)
})
.collect()),
Some(_) => Err("env must be a map".into()),
}
}
fn record(
id: Option<&str>,
lang: &str,
cmd: &str,
ok: bool,
stdout: String,
stderr: String,
result: Value,
error: String,
ms: u64,
timed_out: bool,
alive: bool,
) -> Value {
let mut fields = vec![
("ok".into(), Value::Bool(ok)),
("code".into(), Value::Int(if ok { 0 } else { -1 })),
("stdout".into(), Value::Text(stdout)),
("stderr".into(), Value::Text(stderr)),
("ms".into(), Value::Int(ms as i64)),
("lang".into(), Value::Text(lang.to_string())),
("cmd".into(), Value::Text(cmd.to_string())),
("timed_out".into(), Value::Bool(timed_out)),
("truncated".into(), Value::Bool(false)),
("result".into(), result),
("result_error".into(), Value::Text(String::new())),
("error".into(), Value::Text(error)),
("script".into(), Value::Text(String::new())),
("cached".into(), Value::Bool(false)),
("alive".into(), Value::Bool(alive)),
];
if let Some(id) = id {
fields.push(("id".into(), Value::Text(id.to_string())));
}
Value::Map(fields)
}
fn now_ms() -> i64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as i64)
.unwrap_or(0)
}
pub fn open(
ctx: &mut HostContext,
lang: &Value,
cmd: Option<&Value>,
env: Option<&Value>,
cwd: Option<&Value>,
timeout: Option<&Value>,
) -> Result<Value, String> {
ensure_config(ctx);
if !ctx.allow_exec() {
return Err("foreign session denied by host policy".into());
}
let requested = as_text(lang, "lang")?.to_ascii_lowercase();
let canonical = crate::foreign::registry::canonical(&requested);
if !crate::foreign::registry::session_supported(&canonical) {
return Err(format!(
"lang=`{canonical}` has no session bootstrap; use run/invoke"
));
}
let spec = resolve_session_spec(ctx, &canonical)?;
let mut argv = match opt_text(cmd)? {
Some(c) => {
let argv = crate::foreign::split_cmd_line(&c);
if argv.is_empty() {
return Err("session cmd override is empty".into());
}
argv
}
None => spec.argv.clone(),
};
if argv.is_empty() {
return Err(format!("no interpreter configured for lang=`{canonical}`"));
}
match canonical.as_str() {
"python" => {
argv.push("-u".into());
argv.push("-c".into());
argv.push(PY_BOOTSTRAP.into());
}
"node" => {
argv.push("-e".into());
argv.push(JS_BOOTSTRAP.into());
}
_ => return Err(format!("lang=`{canonical}` has no session bootstrap")),
}
let cwd_path = match opt_text(cwd)? {
Some(c) => ctx.resolve_path(&c)?,
None => ctx.cwd.clone(),
};
if !cwd_path.is_dir() {
return Err(format!(
"session cwd is not a directory: {}",
cwd_path.display()
));
}
let session_timeout = parse_timeout(timeout, SESSION_TIMEOUT_SECS)?;
let mut command = Command::new(&argv[0]);
for a in &argv[1..] {
command.arg(a);
}
command.current_dir(&cwd_path);
command.stdin(Stdio::piped());
command.stdout(Stdio::piped());
command.stderr(Stdio::piped());
command.env("MARQDO_FOREIGN", "1");
command.env("MARQDO_FOREIGN_LANG", &canonical);
command.env("MARQDO_FOREIGN_CWD", &cwd_path);
for (k, v) in &spec.env {
command.env(k, v);
}
for (k, v) in parse_env(env)? {
command.env(k, v);
}
command.env("PYTHONUNBUFFERED", "1");
command.env("PYTHONIOENCODING", "utf-8");
command.env("PYTHONUTF8", "1");
#[cfg(unix)]
{
use std::os::unix::process::CommandExt;
command.process_group(0);
}
let mut child = command.spawn().map_err(|e| {
format!(
"foreign session spawn failed (lang={canonical}, cmd={}): {e}",
argv[0]
)
})?;
let stdin = child
.stdin
.take()
.ok_or_else(|| "session stdin unavailable".to_string())?;
let stdout = child
.stdout
.take()
.ok_or_else(|| "session stdout unavailable".to_string())?;
let stderr_pipe = child.stderr.take();
let (tx, rx) = mpsc::channel::<String>();
let reader = thread::spawn(move || {
let mut reader = BufReader::new(stdout);
let mut line = String::new();
loop {
line.clear();
match reader.read_line(&mut line) {
Ok(0) => {
let _ = tx.send("__EOF__".into());
break;
}
Ok(_) => {
let _ = tx.send(line.trim_end_matches(['\r', '\n']).to_string());
}
Err(_) => {
let _ = tx.send("__EOF__".into());
break;
}
}
}
});
let stderr_buf = Arc::new(Mutex::new(String::new()));
let stderr_buf_t = Arc::clone(&stderr_buf);
let stderr_reader = thread::spawn(move || {
if let Some(mut r) = stderr_pipe {
let mut kept = Vec::new();
let mut buf = [0u8; 8192];
loop {
match r.read(&mut buf) {
Ok(0) => break,
Ok(n) => {
if kept.len() < MAX_PROCESS_STDERR {
let take = (MAX_PROCESS_STDERR - kept.len()).min(n);
kept.extend_from_slice(&buf[..take]);
}
}
Err(e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
Err(_) => break,
}
}
if let Ok(mut s) = stderr_buf_t.lock() {
s.push_str(&String::from_utf8_lossy(&kept));
}
}
});
let seq = ctx.foreign_session_seq + 1;
ctx.foreign_session_seq = seq;
let id = format!("s{seq}");
let started_ms = now_ms();
let cmd_display = argv[0].clone();
let session = Session {
id: id.clone(),
lang: canonical.clone(),
cmd: cmd_display.clone(),
started_ms,
timeout: session_timeout,
evals: 0,
alive: true,
child,
stdin,
rx,
_reader: reader,
stderr: stderr_buf,
_stderr_reader: stderr_reader,
};
ctx.foreign_sessions.insert(id.clone(), session);
let _ = timeout;
Ok(Value::Text(id))
}
fn session_dead_stderr(ctx: &HostContext, id: &str) -> String {
ctx.foreign_sessions
.get(id)
.and_then(|s| s.stderr.lock().ok().map(|v| v.clone()))
.unwrap_or_default()
}
fn remove_session(ctx: &mut HostContext, id: &str) {
ctx.foreign_sessions.remove(id);
}
pub fn eval(
ctx: &mut HostContext,
id: &Value,
code: Option<&Value>,
source: Option<&Value>,
input: Option<&Value>,
timeout: Option<&Value>,
) -> Result<Value, String> {
eval_with_stdin(ctx, id, code, source, input, None, timeout)
}
pub fn eval_with_stdin(
ctx: &mut HostContext,
id: &Value,
code: Option<&Value>,
source: Option<&Value>,
input: Option<&Value>,
stdin: Option<&str>,
timeout: Option<&Value>,
) -> Result<Value, String> {
ensure_config(ctx);
let id = as_text(id, "id")?.to_string();
if !ctx.foreign_sessions.contains_key(&id) {
return Err(format!("foreign session `{id}` not found"));
}
let session_lang = ctx.foreign_sessions[&id].lang.clone();
let code = match (present(code), present(source)) {
(Some(_), Some(_)) => {
return Err("session_eval: pass either code or source, not both".into())
}
(Some(c), None) => match c {
Value::Code(b) => {
if crate::foreign::registry::canonical(&b.lang) != session_lang {
return Err(format!(
"session_eval code language `{}` does not match session language `{}`",
b.lang, session_lang
));
}
b.source.clone()
}
_ => return Err("session_eval code must be a code value".into()),
},
(None, Some(s)) => as_text(s, "source")?.to_string(),
(None, None) => return Err("session_eval needs code or source".into()),
};
let input_json = match input {
None | Some(Value::None) => serde_json::Value::Null,
Some(v) => value_to_json(v)?,
};
let default_timeout = ctx.foreign_sessions[&id].timeout.as_secs_f64();
let dur = parse_timeout(timeout, default_timeout)?;
let request = serde_json::json!({
"id": 1,
"op": "eval",
"code": code,
"input": input_json,
"stdin": stdin,
});
let start = Instant::now();
let (lang, cmd) = {
let s = &ctx.foreign_sessions[&id];
(s.lang.clone(), s.cmd.clone())
};
let reply = match send_and_wait(ctx, &id, &request, dur) {
Ok(r) => r,
Err(e) => {
let timed_out = e.contains("timed out");
let stderr = session_dead_stderr(ctx, &id);
remove_session(ctx, &id);
return Ok(record(
Some(&id),
&lang,
&cmd,
false,
String::new(),
stderr,
Value::None,
e,
start.elapsed().as_millis() as u64,
timed_out,
false,
));
}
};
let ms = start.elapsed().as_millis() as u64;
let alive = ctx
.foreign_sessions
.get_mut(&id)
.map(|s| {
s.evals += 1;
s.alive
})
.unwrap_or(false);
let ok = reply
.get("ok")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let stdout = reply
.get("stdout")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
let stderr = reply
.get("stderr")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
let error = reply
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
let result = reply
.get("result")
.map(json_to_value)
.transpose()?
.unwrap_or(Value::None);
Ok(record(
Some(&id),
&lang,
&cmd,
ok,
stdout,
stderr,
result,
error,
ms,
false,
alive,
))
}
fn present(v: Option<&Value>) -> Option<&Value> {
match v {
None | Some(Value::None) => None,
Some(x) => Some(x),
}
}
fn send_and_wait(
ctx: &mut HostContext,
id: &str,
request: &serde_json::Value,
timeout: Duration,
) -> Result<serde_json::Value, String> {
let line = serde_json::to_string(request).map_err(|e| format!("session request: {e}"))?;
{
let session = ctx
.foreign_sessions
.get_mut(id)
.ok_or_else(|| format!("foreign session `{id}` not found"))?;
if let Ok(Some(_)) = session.child.try_wait() {
return Err("session process exited".into());
}
session
.stdin
.write_all(line.as_bytes())
.and_then(|_| session.stdin.write_all(b"\n"))
.and_then(|_| session.stdin.flush())
.map_err(|e| format!("session write: {e}"))?;
}
let deadline = Instant::now() + timeout;
loop {
let remaining = deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
kill_session(ctx, id);
return Err(format!(
"session timed out after {:.3}s",
timeout.as_secs_f64()
));
}
let session = ctx
.foreign_sessions
.get_mut(id)
.ok_or_else(|| format!("foreign session `{id}` not found"))?;
match session.rx.recv_timeout(remaining.min(Duration::from_millis(200))) {
Ok(line) => {
if line == "__EOF__" {
return Err("session process exited".into());
}
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&line) {
let rid = v.get("id").and_then(|x| x.as_u64()).unwrap_or(0);
if rid != 1 {
continue;
}
return Ok(v);
}
// Unexpected non-protocol output: keep waiting until the deadline.
continue;
}
Err(mpsc::RecvTimeoutError::Timeout) => {
if let Ok(Some(_)) = session.child.try_wait() {
return Err("session process exited".into());
}
continue;
}
Err(mpsc::RecvTimeoutError::Disconnected) => {
return Err("session reader disconnected".into())
}
}
}
}
fn kill_session(ctx: &mut HostContext, id: &str) {
if let Some(mut s) = ctx.foreign_sessions.remove(id) {
#[cfg(unix)]
unsafe {
libc::kill(-(s.child.id() as i32), libc::SIGKILL);
}
let _ = s.child.kill();
let _ = s.child.wait();
s.alive = false;
}
}
/// Interrupt the current evaluation without destroying the kernel. Python
/// reports KeyboardInterrupt through the normal result frame.
pub fn interrupt(ctx: &mut HostContext, id: &Value) -> Result<Value, String> {
let id = as_text(id, "id")?.to_string();
let session = ctx
.foreign_sessions
.get_mut(&id)
.ok_or_else(|| format!("foreign session `{id}` not found"))?;
#[cfg(unix)]
unsafe {
libc::kill(-(session.child.id() as i32), libc::SIGINT);
}
#[cfg(windows)]
{
let _ = Command::new("taskkill")
.args(["/PID", &session.child.id().to_string(), "/T", "/F"])
.output();
session.alive = false;
}
Ok(Value::Bool(true))
}
pub fn set(
ctx: &mut HostContext,
id: &Value,
name: &Value,
value: &Value,
) -> Result<Value, String> {
let id = as_text(id, "id")?.to_string();
let name = as_text(name, "name")?.to_string();
if !valid_name(&name) {
return Err("session_set: name must be an identifier".into());
}
let json = value_to_json(value)?;
let request = serde_json::json!({"id": 1, "op": "set", "name": name, "value": json});
let reply = send_and_wait(ctx, &id, &request, Duration::from_secs(10))?;
if reply.get("ok").and_then(|v| v.as_bool()).unwrap_or(false) {
Ok(Value::None)
} else {
Err(reply
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("session_set failed")
.to_string())
}
}
pub fn get(ctx: &mut HostContext, id: &Value, name: &Value) -> Result<Value, String> {
let id = as_text(id, "id")?.to_string();
let name = as_text(name, "name")?.to_string();
if !valid_name(&name) {
return Err("session_get: name must be an identifier".into());
}
let request = serde_json::json!({"id": 1, "op": "get", "name": name});
let reply = send_and_wait(ctx, &id, &request, Duration::from_secs(10))?;
if !reply.get("ok").and_then(|v| v.as_bool()).unwrap_or(false) {
return Err(reply
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("session_get failed")
.to_string());
}
match reply.get("result") {
Some(v) => json_to_value(v),
None => Ok(Value::None),
}
}
pub fn names(ctx: &mut HostContext, id: &Value) -> Result<Value, String> {
let id = as_text(id, "id")?.to_string();
let request = serde_json::json!({"id": 1, "op": "names"});
let reply = send_and_wait(ctx, &id, &request, Duration::from_secs(10))?;
if !reply.get("ok").and_then(|v| v.as_bool()).unwrap_or(false) {
return Err(reply
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("session_names failed")
.to_string());
}
match reply.get("result") {
Some(v) => json_to_value(v),
None => Ok(Value::List(Vec::new())),
}
}
pub fn close(ctx: &mut HostContext, id: &Value) -> Result<Value, String> {
let id = as_text(id, "id")?.to_string();
let existed = ctx.foreign_sessions.contains_key(&id);
kill_session(ctx, &id);
Ok(Value::Bool(existed))
}
pub fn list(ctx: &HostContext) -> Result<Value, String> {
let mut sessions: Vec<&Session> = ctx.foreign_sessions.values().collect();
sessions.sort_by(|a, b| a.id.cmp(&b.id));
let mut out = Vec::new();
for s in sessions {
out.push(Value::Map(vec![
("id".into(), Value::Text(s.id.clone())),
("lang".into(), Value::Text(s.lang.clone())),
("cmd".into(), Value::Text(s.cmd.clone())),
("evals".into(), Value::Int(s.evals as i64)),
("alive".into(), Value::Bool(s.alive)),
("started_ms".into(), Value::Int(s.started_ms)),
]));
}
Ok(Value::List(out))
}
pub fn shutdown(ctx: &mut HostContext) {
let ids: Vec<String> = ctx.foreign_sessions.keys().cloned().collect();
for id in ids {
kill_session(ctx, &id);
}
}
fn valid_name(name: &str) -> bool {
let mut chars = name.chars();
match chars.next() {
Some(c) if c == '_' || c.is_ascii_alphabetic() => {}
_ => return false,
}
chars.all(|c| c == '_' || c.is_ascii_alphanumeric())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn identifiers() {
assert!(valid_name("x"));
assert!(valid_name("_x1"));
assert!(!valid_name("1x"));
assert!(!valid_name("a-b"));
}
}
+26 -1
View File
@@ -14,6 +14,10 @@ mod encoding;
pub mod event_bus;
#[cfg(feature = "exec-host")]
pub mod foreign;
#[cfg(feature = "exec-host")]
pub mod foreign_doc;
#[cfg(feature = "exec-host")]
pub mod foreign_session;
mod fs;
mod hash;
mod html_ops;
@@ -27,7 +31,7 @@ mod secrets;
mod stats;
mod text_ops;
mod time;
mod toml_ops;
pub(crate) mod toml_ops;
mod url_ops;
mod uuid_ops;
#[cfg(feature = "plugin-host")]
@@ -93,6 +97,18 @@ pub struct HostContext {
pub plots: Vec<PlotArtifact>,
/// lang → interpreter argv (from `set_cmd`).
pub foreign_cmds: HashMap<String, Vec<String>>,
/// Process-local language registrations (`register` / `set_cmd`).
pub foreign_specs: HashMap<String, crate::foreign::registry::LangSpec>,
/// Project config specs (`.marqdo/foreign.toml`), lower priority than runtime.
pub foreign_config_specs: HashMap<String, crate::foreign::registry::LangSpec>,
/// Whether project config has been discovered/loaded.
pub foreign_config_loaded: bool,
/// Persistent foreign kernels (`session_open`).
#[cfg(feature = "exec-host")]
pub foreign_sessions: HashMap<String, foreign_session::Session>,
/// Monotonic session id counter.
#[cfg(feature = "exec-host")]
pub foreign_session_seq: u64,
/// Loaded native plugins (ABI v1/v2).
pub plugins: PluginState,
/// Entry `.mq.md` source text (for ABI host_query / context injection).
@@ -128,6 +144,13 @@ impl Default for HostContext {
rng: 0xC0FF_EE42_CAFE_BABE,
plots: Vec::new(),
foreign_cmds: HashMap::new(),
foreign_specs: HashMap::new(),
foreign_config_specs: HashMap::new(),
foreign_config_loaded: false,
#[cfg(feature = "exec-host")]
foreign_sessions: HashMap::new(),
#[cfg(feature = "exec-host")]
foreign_session_seq: 0,
plugins: PluginState::default(),
entry_source: None,
entry_path: None,
@@ -146,6 +169,8 @@ impl Default for HostContext {
impl Drop for HostContext {
fn drop(&mut self) {
subtask::kill_all(self);
#[cfg(feature = "exec-host")]
foreign_session::shutdown(self);
}
}
+17 -3
View File
@@ -152,8 +152,13 @@ pub fn http_post(
Some(v) => as_text(v, "content_type")?,
};
let hdrs = headers_from_value(headers)?;
let (status, resp) = http_exchange("POST", url, Some(body), Some(ct), &hdrs)?;
Ok(response_map(status, resp))
// Transport failures → status 0 + body message (soft). Callers that need
// hard errors can still treat status!=200 as failure; Agent/LLM surfaces
// map this into structured `{finish:error}` instead of aborting the VM.
match http_exchange("POST", url, Some(body), Some(ct), &hdrs) {
Ok((status, resp)) => Ok(response_map(status, resp)),
Err(e) => Ok(response_map(0, e)),
}
}
/// Generic request: method + url + optional body / content_type / headers.
@@ -847,7 +852,16 @@ pub fn http_post_sse(
),
]))
}
Err(e) => Err(format!("http POST sse {url}: {e}")),
Err(e) => Ok(Value::Map(vec![
("status".into(), Value::Int(0)),
(
"events".into(),
Value::List(vec![event_map(vec![
("type", Value::Text("error".into())),
("message", Value::Text(format!("http POST sse {url}: {e}"))),
])]),
),
])),
}
}
+13 -7
View File
@@ -120,8 +120,8 @@ fn republish_child_event(line: &str) {
pub(crate) struct ForeignTask {
child: Child,
temp_script: PathBuf,
stdout: JoinHandle<String>,
stderr: JoinHandle<String>,
stdout: JoinHandle<(String, bool)>,
stderr: JoinHandle<(String, bool)>,
}
enum FnState {
@@ -392,6 +392,9 @@ fn spawn_fn(
let cwd = ctx.cwd.clone();
let caps = ctx.caps.clone();
let foreign_cmds = ctx.foreign_cmds.clone();
let foreign_specs = ctx.foreign_specs.clone();
let foreign_config_specs = ctx.foreign_config_specs.clone();
let foreign_config_loaded = ctx.foreign_config_loaded;
let fs_root = ctx.fs_root.clone();
let argv = ctx.argv.clone();
@@ -407,6 +410,9 @@ fn spawn_fn(
host.fs_root = fs_root;
host.argv = argv;
host.foreign_cmds = foreign_cmds;
host.foreign_specs = foreign_specs;
host.foreign_config_specs = foreign_config_specs;
host.foreign_config_loaded = foreign_config_loaded;
host.set_entry_source(Some(&module_path), &source);
host.entry_metadata = module.metadata.clone();
let mut interp = Interpreter::with_capture(Some(&module_path), false).with_host(host);
@@ -587,12 +593,9 @@ fn join_foreign(mut task: ForeignTask) -> Result<Value, String> {
.child
.wait()
.map_err(|e| format!("subtask foreign join: {e}"))?;
let stdout = task.stdout.join().unwrap_or_default();
let stderr = task.stderr.join().unwrap_or_default();
let (stdout, out_trunc) = task.stdout.join().unwrap_or_default();
let (stderr, err_trunc) = task.stderr.join().unwrap_or_default();
let _ = std::fs::remove_file(&task.temp_script);
if stdout.len() > foreign::MAX_OUTPUT || stderr.len() > foreign::MAX_OUTPUT {
return Err("subtask foreign output too large".into());
}
if !status.success() {
let code = status.code().unwrap_or(1);
let mut msg = format!("subtask foreign failed (exit={code})");
@@ -609,6 +612,9 @@ fn join_foreign(mut task: ForeignTask) -> Result<Value, String> {
out.pop();
}
}
if out_trunc || err_trunc {
out.push_str("\n…(truncated)");
}
Ok(Value::Text(out))
}
+148
View File
@@ -208,6 +208,7 @@ impl Interpreter {
call_id,
fn_name: fun.name.clone(),
fn_line: fun.span.line,
call_line: self.current_span.line,
ordinal,
args_brief,
});
@@ -372,6 +373,98 @@ impl Interpreter {
result
}
/// Evaluate only the entry object's top-level assignments, without running
/// calls/returns/control flow. This is the view plan's language-native
/// projection: cell tables are ordinary Marqdo values, not source scans.
/// Evaluate selected entry bindings plus their assignment dependencies.
/// This is the view plan's language-native projection: cell tables are
/// ordinary Marqdo values, not source scans.
pub fn capture_declarations(
&mut self,
module: &Module,
names: &[&str],
) -> Result<std::collections::HashMap<String, Value>> {
let prev = self.site_module;
self.site_module = Some(module as *const Module);
let result = (|| {
let entry = find_top(module, "main").or_else(|| {
let entries: Vec<&Function> = module
.functions
.iter()
.filter(|f| f.is_object() && f.params.is_empty())
.collect();
if entries.len() == 1 {
Some(entries[0])
} else {
None
}
});
let entry = entry.ok_or_else(|| self.err("no `# main` object to inspect"))?;
let assignments: std::collections::HashMap<&str, &Expr> = entry
.body
.iter()
.filter_map(|stmt| match stmt {
Stmt::Assign { name, value, .. } => Some((name.as_str(), value)),
_ => None,
})
.collect();
let mut env = Env::with_metadata(module);
let mut done = std::collections::HashSet::new();
let mut visiting = std::collections::HashSet::new();
fn resolve(
this: &mut Interpreter,
module: &Module,
fun: &Function,
assignments: &std::collections::HashMap<&str, &Expr>,
name: &str,
env: &mut Env,
done: &mut std::collections::HashSet<String>,
visiting: &mut std::collections::HashSet<String>,
) -> Result<()> {
if done.contains(name) {
return Ok(());
}
if !visiting.insert(name.to_string()) {
return Err(this.err(format!("cyclic view-plan binding `{name}`")));
}
let Some(expr) = assignments.get(name) else {
visiting.remove(name);
return Ok(());
};
let mut deps = std::collections::HashSet::new();
collect_decl_reads(expr, &mut deps);
for dep in deps {
if assignments.contains_key(dep.as_str()) && env.get(&dep).is_none() {
resolve(this, module, fun, assignments, &dep, env, done, visiting)?;
}
}
let value = this.eval_expr(module, fun, env, expr)?;
env.set(name.to_string(), value);
done.insert(name.to_string());
visiting.remove(name);
Ok(())
}
for name in names {
resolve(
self,
module,
entry,
&assignments,
name,
&mut env,
&mut done,
&mut visiting,
)?;
}
self.entry_bindings = env.vars_clone();
Ok(self.entry_bindings.clone())
})();
self.site_module = prev;
result
}
pub fn run_module(&mut self, module: &Module) -> Result<Value> {
let prev = self.site_module;
self.site_module = Some(module as *const Module);
@@ -1557,6 +1650,61 @@ fn bind_args(
Ok(out)
}
fn collect_decl_reads(expr: &Expr, out: &mut std::collections::HashSet<String>) {
match expr {
Expr::Var(name) => {
out.insert(name.clone());
}
Expr::Interp(parts) => {
for part in parts {
match part {
InterpPart::Var(name) => {
out.insert(name.clone());
}
InterpPart::Index { base, .. } => {
out.insert(base.clone());
}
InterpPart::Lit(_) => {}
}
}
}
Expr::Index { base, label } => {
collect_decl_reads(base, out);
if let IndexKey::Var(name) = label {
out.insert(name.clone());
}
}
Expr::Binary { left, right, .. } => {
collect_decl_reads(left, out);
collect_decl_reads(right, out);
}
Expr::Unary { expr, .. } => collect_decl_reads(expr, out),
Expr::List(xs) => {
for x in xs {
collect_decl_reads(x, out);
}
}
Expr::Map(entries) => {
for (_, v) in entries {
collect_decl_reads(v, out);
}
}
Expr::Call(call) => {
if let Some(receiver) = &call.receiver {
out.insert(receiver.clone());
}
for arg in &call.args {
match arg {
Arg::Positional(e) | Arg::Named { value: e, .. } => {
collect_decl_reads(e, out);
}
}
}
}
Expr::Literal(_) | Expr::Formula(_) | Expr::Code(_) => {}
}
}
fn find_top<'a>(module: &'a Module, name: &str) -> Option<&'a Function> {
module.functions.iter().find(|f| f.name == name)
}
+23 -1
View File
@@ -16,7 +16,7 @@ use marqdo::knowledge::{
resolve, reuse_json, stale_json, verify, verify_json, write_knowledge, KnowledgeOptions,
LearnInput, ReuseBudget,
};
use marqdo::view::{serve, serve_debug, write_static, DebugOptions, OutputOptions, ViewOptions};
use marqdo::view::{serve, serve_debug, write_capture, write_static, CaptureOptions, DebugOptions, OutputOptions, ViewOptions};
use marqdo::{Backend, RunOptions};
#[derive(Parser, Debug)]
@@ -374,6 +374,17 @@ enum ViewAction {
#[arg(long)]
no_exec: bool,
#[arg(long, value_name = "FILE")]
snapshot: Option<PathBuf>,
},
/// Capture live cell results for `view output --snapshot`
Capture {
#[arg(value_name = "PATH")]
path: Option<PathBuf>,
#[arg(short = 'o', long = "out", required = true)]
out: PathBuf,
},
}
@@ -509,6 +520,7 @@ fn try_main(cli: Cli) -> Result<i32> {
path: out_path,
out,
no_exec,
snapshot,
}) => {
write_static(OutputOptions {
path: out_path
@@ -516,6 +528,16 @@ fn try_main(cli: Cli) -> Result<i32> {
.unwrap_or_else(|| PathBuf::from(".")),
out_dir: out,
no_exec,
snapshot,
})?;
Ok(0)
}
Some(ViewAction::Capture { path: out_path, out }) => {
write_capture(CaptureOptions {
path: out_path
.or(path)
.unwrap_or_else(|| PathBuf::from(".")),
out,
})?;
Ok(0)
}
+129 -23
View File
@@ -257,7 +257,7 @@ impl<'a> Cursor<'a> {
var_contracts: std::collections::HashMap::new(),
};
// Prose decls pending promotion (name → optional default text).
let mut prose_decls: Vec<(String, Option<String>)> = Vec::new();
let mut prose_decls: Vec<(String, Option<String>, u32)> = Vec::new();
// Parameter zone: explicit `+` lines (legacy / optional).
while self.skip_noise() {
@@ -335,8 +335,8 @@ impl<'a> Cursor<'a> {
for bit in bits {
match bit {
ProseBit::Decl { name, default, .. } => {
if !prose_decls.iter().any(|(n, _)| n == &name) {
prose_decls.push((name, default));
if !prose_decls.iter().any(|(n, _, _)| n == &name) {
prose_decls.push((name, default, span.line));
}
}
ProseBit::BoldCode { inner, .. } => {
@@ -1714,10 +1714,11 @@ fn stmt_from_italic_return(inner: &str, span: Span) -> Result<Stmt> {
Ok(Stmt::Return { value, span })
}
/// Merge prose decls into params: promote names that are read in the body (or have defaults).
/// Merge declarations into params: promote names that are read in the body (or have defaults).
///
/// Design §8.2: elevate only when an executable read cannot be satisfied by a prior
/// local assign that does not itself read the name (e.g. `score = > json.get …` is local;
/// Design §8.2: a declaration is a backtick mention or the first bold assignment
/// to that name. Elevate only when an executable read cannot be satisfied by a prior
/// local assign that does not itself read the name (e.g. `score = 1` is local;
/// `n = n + 1` still needs external `n`).
// ---------- 契约提取(T2.1 · 渐进式契约 / 标注类型) ----------
//
@@ -1881,36 +1882,62 @@ fn extract_function_contract(fun: &mut Function, lines: &[ClassifiedLine]) -> Re
fn merge_inferred_params(
fun: &mut Function,
prose_decls: &[(String, Option<String>)],
prose_decls: &[(String, Option<String>, u32)],
) -> Result<()> {
use std::collections::HashSet;
let mut needs_input = HashSet::new();
let mut bound = HashSet::new();
collect_names_needing_input(&fun.body, &mut bound, &mut needs_input);
let mut assign_decls = Vec::new();
collect_names_needing_input(
&fun.body,
&mut bound,
&mut needs_input,
&HashSet::new(),
&mut assign_decls,
);
for (name, default_txt) in prose_decls {
// Declaration order is source order: backtick mentions and first assignment
// left-hand sides. An assignment promotes only when its right-hand side (or a
// later read) still needs an external value.
let mut events: Vec<(u32, usize, String, Option<String>)> = Vec::new();
for (i, (name, default_txt, line)) in prose_decls.iter().enumerate() {
events.push((*line, i, name.clone(), default_txt.clone()));
}
let prose_n = events.len();
for (j, (name, line)) in assign_decls.iter().enumerate() {
if prose_decls.iter().any(|(n, _, _)| n == name) {
continue;
}
if !needs_input.contains(name) {
continue;
}
events.push((*line, prose_n + j, name.clone(), None));
}
events.sort_by_key(|e| (e.0, e.1));
for (_line, _ord, name, default_txt) in events {
if name == "self" || name == "自" {
continue;
}
if fun.params.iter().any(|p| &p.name == name) {
if fun.params.iter().any(|p| p.name == name) {
continue;
}
let has_default = default_txt.is_some();
// Elevate: defaulted always; or executable read that needs an external input.
if !has_default && !needs_input.contains(name) {
if !bound.contains(name) {
fun.dead_binds.push(name.clone());
if !has_default && !needs_input.contains(&name) {
if !bound.contains(&name) {
fun.dead_binds.push(name);
}
continue; // dead bind or pure local
}
let default = if let Some(txt) = default_txt {
Some(parse_expr_prefer_var(txt).or_else(|_| parse_value_or_interp(txt))?)
Some(parse_expr_prefer_var(&txt).or_else(|_| parse_value_or_interp(&txt))?)
} else {
None
};
fun.params.push(Param {
name: name.clone(),
name,
default,
inferred: true,
});
@@ -1921,21 +1948,30 @@ fn merge_inferred_params(
/// Walk body in order. `bound` = names already assigned locally without needing input.
/// On read of unbound name → needs input. On assign, RHS reads are checked first
/// (so `n = n + 1` marks `n` as needing input before binding).
/// `assign_decls` records the first assignment of each name, which §8.2 treats as
/// a declaration when no backtick mention came first. Loop items are not declarations.
fn collect_names_needing_input(
stmts: &[Stmt],
bound: &mut std::collections::HashSet<String>,
needs: &mut std::collections::HashSet<String>,
loop_items: &std::collections::HashSet<String>,
assign_decls: &mut Vec<(String, u32)>,
) {
for s in stmts {
match s {
Stmt::Assign { name, value, .. } => {
Stmt::Assign { name, value, span, .. } => {
let mut rhs_reads = std::collections::HashSet::new();
collect_reads_expr(value, &mut rhs_reads);
for r in &rhs_reads {
if !bound.contains(r) {
if !bound.contains(r) && !loop_items.contains(r) {
needs.insert(r.clone());
}
}
if !loop_items.contains(name)
&& !assign_decls.iter().any(|(n, _)| n == name)
{
assign_decls.push((name.clone(), span.line));
}
bound.insert(name.clone());
}
Stmt::Return { value, .. } => {
@@ -1966,6 +2002,8 @@ fn collect_names_needing_input(
}
}
Stmt::Branch { arms, .. } => {
let has_else = arms.iter().any(|a| a.condition.is_none());
let mut common: Option<std::collections::HashSet<String>> = None;
for a in arms {
if let Some(c) = &a.condition {
let mut reads = std::collections::HashSet::new();
@@ -1977,7 +2015,24 @@ fn collect_names_needing_input(
}
}
let mut arm_bound = bound.clone();
collect_names_needing_input(&a.body, &mut arm_bound, needs);
collect_names_needing_input(
&a.body,
&mut arm_bound,
needs,
loop_items,
assign_decls,
);
if has_else {
common = Some(match common {
None => arm_bound,
Some(prev) => prev.intersection(&arm_bound).cloned().collect(),
});
}
}
if let Some(c) = common {
for n in c {
bound.insert(n);
}
}
}
Stmt::While {
@@ -1986,19 +2041,24 @@ fn collect_names_needing_input(
let mut reads = std::collections::HashSet::new();
collect_reads_expr(condition, &mut reads);
for r in reads {
if !bound.contains(&r) {
if !bound.contains(&r) && !loop_items.contains(&r) {
needs.insert(r);
}
}
collect_names_needing_input(body, bound, needs);
collect_names_needing_input(body, bound, needs, loop_items, assign_decls);
}
Stmt::ForEach {
collection, body, ..
item,
collection,
body,
..
} => {
if !bound.contains(collection) {
if !bound.contains(collection) && !loop_items.contains(collection) {
needs.insert(collection.clone());
}
collect_names_needing_input(body, bound, needs);
let mut nested = loop_items.clone();
nested.insert(item.clone());
collect_names_needing_input(body, bound, needs, &nested, assign_decls);
}
}
}
@@ -2391,6 +2451,21 @@ Candidates may include lexical `score`.
assert!(!dump.contains("dead="), "{dump}");
}
#[test]
fn parse_bare_inc_is_required_param() {
// §8.2: the first bold assignment is a declaration. The RHS reads `n`
// before the bind, so `n` is a required parameter even without a backtick.
let src = "## inc\n\n**n = n + 1**\n*n*\n\n# main\n\n**print text=\"ok\"**\n";
let m = parse_source(src).unwrap();
let inc = m.functions.iter().find(|f| f.name == "inc").unwrap();
assert_eq!(
inc.params.iter().map(|p| p.name.as_str()).collect::<Vec<_>>(),
["n"]
);
assert!(inc.params[0].default.is_none());
assert!(inc.params[0].inferred);
}
#[test]
fn parse_dead_binds_visible_in_ast_dump() {
let src = include_str!("../../tests/markup-v03/dead-bind.mq.md");
@@ -2512,6 +2587,37 @@ Candidates may include lexical `score`.
other => panic!("expected code assign, got {other:?}"),
}
}
#[test]
fn branch_else_common_local_is_not_a_param() {
let src = r#"# main
## f
+ `a`=None
1. `a`
**k = a**
2. *
**k = None**
| x |
|---|
| `k` |
"#;
let m = parse_source(src).unwrap();
let f = m
.functions
.iter()
.flat_map(|outer| outer.children.iter())
.find(|f| f.name == "f")
.unwrap();
assert_eq!(
f.params.iter().map(|p| p.name.as_str()).collect::<Vec<_>>(),
["a"],
"{:?}",
f.params
);
}
}
/// 核心表面构造注册表(T1.3)——`(id, Markdown 外形)`。
+15 -2
View File
@@ -205,9 +205,22 @@ pub fn build_view_annots(module: &Module, source: &str, cap: &RunCapture) -> Vec
last_ret.insert((ev.line, ev.call_id, ev.loop_id, ev.iter), ev);
}
}
let call_sites: HashMap<u32, u32> = cap
.view_trace
.calls
.iter()
.map(|fr| (fr.call_id, fr.call_line))
.collect();
for ((line, _, _, _), ev) in last_ret {
// Return events from an imported library carry the callee's line
// number. Anchor the chip at the caller's line instead.
let anchor_line = call_sites
.get(&ev.call_id)
.copied()
.filter(|l| *l > 0)
.unwrap_or(line);
let text = lines
.get(line.saturating_sub(1) as usize)
.get(anchor_line.saturating_sub(1) as usize)
.copied()
.unwrap_or("");
let mut start = None;
@@ -222,7 +235,7 @@ pub fn build_view_annots(module: &Module, source: &str, cap: &RunCapture) -> Vec
out.push(ViewAnnot {
id: next_id("r"),
kind: AnnotKind::Return,
anchor_line: line,
anchor_line,
anchor_byte_start: start,
anchor_byte_end: end,
label: "return".into(),
+17 -8
View File
@@ -849,7 +849,7 @@ fn debug_js() -> &'static str {
}
renderBps();
if (session) {
post("/api/debug/breakpoints", {
post("api/debug/breakpoints", {
session: session,
breakpoints: bpList()
}).catch(function () {});
@@ -861,7 +861,7 @@ fn debug_js() -> &'static str {
startBtn.disabled = true;
setStatus("Starting…");
clearCurrent();
post("/api/debug/start", {
post("api/debug/start", {
path: path,
breakpoints: bpList(),
stdin: stdinText()
@@ -875,7 +875,7 @@ fn debug_js() -> &'static str {
if (!session) return;
setStatus("Continue…");
setPausedUi(false);
post("/api/debug/continue", {
post("api/debug/continue", {
session: session,
breakpoints: bpList()
}).then(applySnap).catch(function (e) { setStatus(String(e)); });
@@ -884,14 +884,14 @@ fn debug_js() -> &'static str {
if (!session) return;
setStatus("Step…");
setPausedUi(false);
post("/api/debug/step", {
post("api/debug/step", {
session: session,
breakpoints: bpList()
}).then(applySnap).catch(function (e) { setStatus(String(e)); });
});
stopBtn.addEventListener("click", function () {
if (!session) return;
post("/api/debug/stop", { session: session }).then(function () {
post("api/debug/stop", { session: session }).then(function () {
session = null;
clearCurrent();
setStatus("Stopped");
@@ -947,7 +947,7 @@ fn debug_js() -> &'static str {
out.hidden = false;
out.classList.remove("err");
out.textContent = "Running…";
fetch("/api/foreign-run", {
fetch("api/foreign-run", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
@@ -958,13 +958,22 @@ fn debug_js() -> &'static str {
})
.then(function (r) { return r.json(); })
.then(function (data) {
var parts = [];
if (data.stdout) parts.push(data.stdout);
if (data.stderr) parts.push("[stderr]\n" + data.stderr);
if (data.error) parts.push(data.error);
var meta = [];
if (typeof data.code === "number") meta.push("exit " + data.code);
if (typeof data.ms === "number") meta.push(data.ms + "ms");
if (data.timed_out) meta.push("timed out");
if (data.truncated) meta.push("truncated");
if (meta.length) parts.push("— " + meta.join(" · "));
if (data.ok) {
out.classList.remove("err");
out.textContent = data.stdout === "" ? "(no stdout)" : data.stdout;
} else {
out.classList.add("err");
out.textContent = data.error || "foreign run failed";
}
out.textContent = parts.length ? parts.join("\n\n") : "(no output)";
})
.catch(function (e) {
out.classList.add("err");
+18 -5
View File
@@ -101,11 +101,18 @@ pub fn render_module_document(module: &Module, source: &str, annots: &[ViewAnnot
}
LineKind::Writeback => {
flush_para(&mut para, &mut out);
let start_line = l.line_no;
let mut block = String::new();
while i < lines.len() && matches!(lines[i].kind, LineKind::Writeback) {
if !block.is_empty() {
block.push('\n');
}
block.push_str(lines[i].text.trim_end_matches(['\r', '\n']));
i += 1;
}
out.push_str(&crate::view::render::render_writeback_source_line(
l.line_no,
l.text.trim_end_matches(['\r', '\n']),
start_line, &block,
));
i += 1;
}
LineKind::Comment => {
let text = l.text.trim_end_matches(['\r', '\n']);
@@ -335,8 +342,14 @@ fn render_table_block(
html.push_str("</table>");
// Annots on first table line (bind of collection).
if let Some(list) = by_line.get(&first_line) {
if !list.is_empty() {
let owned: Vec<ViewAnnot> = list.iter().map(|a| (*a).clone()).collect();
// A table is already its own readable value; a call chip would collapse
// the whole collection into an ellipsis. Keep only non-call chips.
let owned: Vec<ViewAnnot> = list
.iter()
.filter(|a| a.kind != crate::view::annot::AnnotKind::Call)
.map(|a| (*a).clone())
.collect();
if !owned.is_empty() {
html.push_str(&render_annot_band(&owned, MAX_VISIBLE_ANNOTS));
}
}
+1298 -26
View File
File diff suppressed because it is too large. Load diff
+162 -3
View File
@@ -6,10 +6,11 @@ mod debug_api;
mod debug_page;
mod document;
mod html;
mod notebook_api;
mod output;
mod render;
pub use output::{write_static, OutputOptions};
pub use output::{write_capture, write_static, CaptureOptions, OutputOptions};
use std::fs;
use std::io::{self, Read};
@@ -173,6 +174,149 @@ fn handle(root: &ViewRoot, request: Request) -> Result<()> {
match path_part {
"/api/foreign-run" => return api_foreign_run(root, request),
"/api/run" => return api_run(root, request),
"/api/session/open" => {
return api_debug_post(request, |body| {
notebook_api::api_open(&root.root, body)
})
}
"/api/session/reload" => {
return api_debug_post(request, |body| {
notebook_api::api_reload(&root.root, body)
})
}
"/api/session/close" => {
return api_debug_post(request, |body| {
notebook_api::api_close(&root.root, body)
})
}
"/api/session/run_document" => {
return api_debug_post(request, |body| {
notebook_api::api_run_document(&root.root, body)
})
}
"/api/session/snapshot/export" => {
return api_debug_post(request, |body| {
notebook_api::api_export_snapshot(&root.root, body)
})
}
"/api/session/snapshot/import" => {
return api_debug_post(request, |body| {
notebook_api::api_import_snapshot(&root.root, body)
})
}
"/api/session/snapshot/replay" => {
return api_debug_post(request, |body| {
notebook_api::api_replay_snapshot(&root.root, body)
})
}
"/api/session/plan/assign" => {
return api_debug_post(request, |body| {
notebook_api::api_plan_assign(&root.root, body)
})
}
"/api/session/plan/add" => {
return api_debug_post(request, |body| {
notebook_api::api_plan_add(&root.root, body)
})
}
"/api/session/plan/remove" => {
return api_debug_post(request, |body| {
notebook_api::api_plan_remove(&root.root, body)
})
}
"/api/session/plan/value" => {
return api_debug_post(request, |body| {
notebook_api::api_plan_value(&root.root, body)
})
}
"/api/session/cell/persist" => {
return api_debug_post(request, |body| {
notebook_api::api_persist_cell(&root.root, body)
})
}
"/api/session/interrupt" => {
return api_debug_post(request, |body| {
notebook_api::api_interrupt(&root.root, body)
})
}
"/api/session/cell/clear" => {
return api_debug_post(request, |body| {
notebook_api::api_clear_cell(&root.root, body)
})
}
"/api/session/clear" => {
return api_debug_post(request, |body| {
notebook_api::api_clear(&root.root, body)
})
}
"/api/session/cell/run" => {
return api_debug_post(request, |body| {
notebook_api::api_cell_run(&root.root, body)
})
}
"/api/session/cell/step" => {
return api_debug_post(request, |body| {
notebook_api::api_cell_step(&root.root, body)
})
}
"/api/session/run_all" => {
return api_debug_post(request, |body| {
notebook_api::api_run_all(&root.root, body)
})
}
"/api/session/kernel/restart" => {
return api_debug_post(request, |body| {
notebook_api::api_kernel_restart(&root.root, body)
})
}
"/api/session/kernel/assign" => {
return api_debug_post(request, |body| {
notebook_api::api_assign(&root.root, body)
})
}
"/api/session/state/list" => {
return api_debug_post(request, |body| {
notebook_api::api_state_list(&root.root, body)
})
}
"/api/session/state/push" => {
return api_debug_post(request, |body| {
notebook_api::api_state_push(&root.root, body)
})
}
"/api/session/state/pull" => {
return api_debug_post(request, |body| {
notebook_api::api_state_pull(&root.root, body)
})
}
"/api/session/journal" => {
return api_debug_post(request, |body| {
notebook_api::api_journal(&root.root, body)
})
}
"/api/debug/start" => {
return api_debug_post(request, |body| {
debug_api::api_debug_start(&root.root, body)
})
}
"/api/debug/continue" => {
return api_debug_post(request, |body| {
debug_api::api_debug_action(body, crate::debug::DebugAction::Continue)
})
}
"/api/debug/step" => {
return api_debug_post(request, |body| {
debug_api::api_debug_action(body, crate::debug::DebugAction::Step)
})
}
"/api/debug/stop" => {
return api_debug_post(request, |body| debug_api::api_debug_stop(body))
}
"/api/debug/breakpoints" => {
return api_debug_post(request, |body| {
debug_api::api_debug_set_breakpoints(body)
})
}
_ => {
let resp =
Response::from_string("method not allowed").with_status_code(StatusCode(405));
@@ -211,6 +355,14 @@ fn handle(root: &ViewRoot, request: Request) -> Result<()> {
let body = page_file(&root.files, &rel, &vm, &LinkMode::Live);
respond_html(request, body)
}
"/debug" => {
let rel = query_param(query, "path").unwrap_or_default();
let resolved = resolve_rel(root, &rel)?;
let source = std::fs::read_to_string(&resolved).unwrap_or_default();
let vm = build_debug_model(&resolved, &rel, &source);
let body = page_debug(&root.files, &rel, &vm);
respond_html(request, body)
}
"/api/tree" => {
let json = files_json(&root.files);
respond_json(request, json)
@@ -595,6 +747,8 @@ fn api_foreign_run(root: &ViewRoot, mut request: Request) -> Result<()> {
.and_then(|x| x.as_str())
.unwrap_or("python");
let source = v.get("source").and_then(|x| x.as_str()).unwrap_or("");
let stdin = v.get("stdin").and_then(|x| x.as_str());
let timeout = v.get("timeout").and_then(|x| x.as_f64());
let cmd = v
.get("cmd")
.and_then(|x| x.as_str())
@@ -603,13 +757,18 @@ fn api_foreign_run(root: &ViewRoot, mut request: Request) -> Result<()> {
let json = if source.is_empty() {
serde_json::json!({ "ok": false, "error": "missing source" }).to_string()
} else {
match crate::host::foreign::run_with_cmd_override(
match crate::host::foreign::invoke_for_view(
&root.root,
lang,
source,
cmd,
stdin,
timeout,
) {
Ok(stdout) => serde_json::json!({ "ok": true, "stdout": stdout }).to_string(),
Ok(value) => match crate::host::json::value_to_json(&value) {
Ok(j) => j.to_string(),
Err(e) => serde_json::json!({ "ok": false, "error": e }).to_string(),
},
Err(error) => serde_json::json!({ "ok": false, "error": error }).to_string(),
}
};
+431
View File
@@ -0,0 +1,431 @@
//! Live-view notebook session APIs.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, OnceLock};
use crate::host::json::{json_to_value, value_to_json};
use crate::host::foreign_doc::DocumentSession;
use crate::value::Value;
fn sessions() -> &'static Mutex<HashMap<String, Arc<Mutex<DocumentSession>>>> {
static S: OnceLock<Mutex<HashMap<String, Arc<Mutex<DocumentSession>>>>> = OnceLock::new();
S.get_or_init(|| Mutex::new(HashMap::new()))
}
fn body_value(body: &str) -> serde_json::Value {
serde_json::from_str(body).unwrap_or(serde_json::Value::Null)
}
fn path_of(root: &Path, v: &serde_json::Value) -> Result<(String, PathBuf), String> {
let rel = v
.get("path")
.and_then(|p| p.as_str())
.unwrap_or("")
.trim()
.to_string();
if rel.is_empty() {
return Err("missing path".into());
}
let abs = root.join(rel.replace('/', std::path::MAIN_SEPARATOR_STR));
if !abs.is_file() {
return Err(format!("file not found: {rel}"));
}
Ok((rel, abs))
}
fn get_or_open(root: &Path, abs: &Path) -> Result<Arc<Mutex<DocumentSession>>, String> {
let key = abs.display().to_string();
let mut map = sessions().lock().map_err(|_| "session lock poisoned".to_string())?;
if let Some(s) = map.get(&key) {
return Ok(Arc::clone(s));
}
let session = DocumentSession::open(abs, root)?;
let arc = Arc::new(Mutex::new(session));
map.insert(key, Arc::clone(&arc));
Ok(arc)
}
fn json_value(v: &Value) -> serde_json::Value {
value_to_json(v).unwrap_or(serde_json::Value::Null)
}
fn ok_session(session: &DocumentSession, extra: serde_json::Value) -> serde_json::Value {
let mut out = serde_json::json!({
"ok": true,
"session": json_value(&session.summary()),
});
if let (Some(dst), Some(src)) = (out.as_object_mut(), extra.as_object()) {
for (k, v) in src {
dst.insert(k.clone(), v.clone());
}
}
out
}
fn err(error: String) -> serde_json::Value {
serde_json::json!({ "ok": false, "error": error })
}
pub fn api_run_document(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let (_rel, abs) = match path_of(root, &v) {
Ok(p) => p,
Err(e) => return err(e),
};
let mut opts = crate::RunOptions::default();
opts.fs_root = Some(root.to_path_buf());
opts.sleep_limit_ms = Some(30_000);
match crate::run_file_capture(&abs, &opts) {
Ok(cap) => serde_json::json!({
"ok": true,
"stdout": cap.stdout,
"value": json_value(&cap.value),
}),
Err(e) => serde_json::json!({ "ok": false, "error": format!("{e:#}") }),
}
}
pub fn api_open(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let (_rel, abs) = match path_of(root, &v) {
Ok(p) => p,
Err(e) => return err(e),
};
match get_or_open(root, &abs) {
Ok(s) => match s.lock() {
Ok(mut sess) => {
if let Err(e) = sess.reload(root) {
return err(e);
}
ok_session(&sess, serde_json::json!({}))
}
Err(_) => err("session lock poisoned".into()),
},
Err(e) => err(e),
}
}
pub fn api_reload(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let (_rel, abs) = match path_of(root, &v) {
Ok(p) => p,
Err(e) => return err(e),
};
match get_or_open(root, &abs) {
Ok(s) => match s.lock() {
Ok(mut sess) => match sess.reload(root) {
Ok(()) => ok_session(&sess, serde_json::json!({})),
Err(e) => err(e),
},
Err(_) => err("session lock poisoned".into()),
},
Err(e) => err(e),
}
}
pub fn api_close(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let (_rel, abs) = match path_of(root, &v) {
Ok(p) => p,
Err(e) => return err(e),
};
let key = abs.display().to_string();
let removed = sessions()
.lock()
.ok()
.and_then(|mut m| m.remove(&key));
if let Some(arc) = removed {
if let Ok(mut s) = arc.lock() {
s.close();
}
}
serde_json::json!({ "ok": true })
}
fn with_session<F>(root: &Path, body: &str, action: &str, f: F) -> serde_json::Value
where
F: FnOnce(&mut DocumentSession) -> Result<Value, String>,
{
let v = body_value(body);
let (_rel, abs) = match path_of(root, &v) {
Ok(p) => p,
Err(e) => return err(e),
};
let arc = match get_or_open(root, &abs) {
Ok(s) => s,
Err(e) => return err(e),
};
let mut sess = match arc.lock() {
Ok(s) => s,
Err(_) => return err("session lock poisoned".into()),
};
match f(&mut sess) {
Ok(value) => {
let mut extra = serde_json::Map::new();
extra.insert(action.into(), json_value(&value));
ok_session(&sess, serde_json::Value::Object(extra))
}
Err(e) => err(e),
}
}
pub fn api_persist_cell(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
let key = v.get("key").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() {
return err("missing cell".into());
}
with_session(root, body, "persisted", |s| s.persist_cell(cell, key))
}
pub fn api_interrupt(root: &Path, body: &str) -> serde_json::Value {
with_session(root, body, "interrupted", |s| s.interrupt())
}
pub fn api_export_snapshot(root: &Path, body: &str) -> serde_json::Value {
with_session(root, body, "snapshot", |s| s.export_snapshot())
}
pub fn api_import_snapshot(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let snapshot = v.get("snapshot").cloned().unwrap_or(serde_json::Value::Null);
let snapshot = match json_to_value(&snapshot) {
Ok(v) => v,
Err(e) => return err(e),
};
with_session(root, body, "restored", |s| s.import_snapshot(&snapshot))
}
pub fn api_replay_snapshot(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let snapshot = v.get("snapshot").cloned().unwrap_or(serde_json::Value::Null);
let snapshot = match json_to_value(&snapshot) {
Ok(v) => v,
Err(e) => return err(e),
};
with_session(root, body, "replay", |s| s.replay_snapshot(&snapshot))
}
pub fn api_clear_cell(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() {
return err("missing cell".into());
}
with_session(root, body, "cleared", |s| s.clear_cell(cell))
}
pub fn api_clear(root: &Path, body: &str) -> serde_json::Value {
with_session(root, body, "cleared", |s| Ok(s.clear_outputs()))
}
pub fn api_cell_run(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
let kernel = v.get("kernel").and_then(|x| x.as_str()).filter(|s| !s.is_empty());
let stdin = v.get("stdin").and_then(|x| x.as_str());
if cell.is_empty() {
return err("missing cell".into());
}
with_session(root, body, "record", |s| s.run_cell_with_stdin(cell, kernel, stdin))
}
pub fn api_cell_step(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let stdin = v.get("stdin").and_then(|x| x.as_str());
with_session(root, body, "record", |s| s.step_with_stdin(stdin))
}
pub fn api_run_all(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let restart = v.get("restart").and_then(|x| x.as_bool()).unwrap_or(false);
let stdin = v.get("stdin").and_then(|x| x.as_str());
let inputs: Option<HashMap<String, String>> = v.get("inputs").and_then(|x| x.as_object()).map(|m| {
m.iter()
.filter_map(|(k, val)| val.as_str().map(|s| (k.clone(), s.to_string())))
.collect()
});
with_session(root, body, "summary", |s| s.run_all_with_inputs(restart, stdin, inputs.as_ref()))
}
pub fn api_kernel_restart(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
if kernel.is_empty() {
return err("missing kernel".into());
}
with_session(root, body, "value", |s| s.restart_kernel(kernel))
}
pub fn api_plan_assign(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() || kernel.is_empty() {
return err("missing cell/kernel".into());
}
with_session(root, body, "value", |s| s.assign_kernel(cell, kernel))
}
pub fn api_plan_add(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
let lang = v.get("lang").and_then(|x| x.as_str()).unwrap_or("");
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
let source = v.get("source").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() || lang.is_empty() || kernel.is_empty() {
return err("missing cell/lang/kernel".into());
}
with_session(root, body, "value", |s| s.add_cell(cell, lang, source, kernel))
}
pub fn api_plan_remove(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() {
return err("missing cell".into());
}
with_session(root, body, "value", |s| s.remove_cell(cell))
}
pub fn api_plan_value(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
let name = v.get("name").and_then(|x| x.as_str()).unwrap_or("");
if kernel.is_empty() || name.is_empty() {
return err("missing kernel/name".into());
}
let value = match v.get("value") {
Some(v) => match json_to_value(v) {
Ok(v) => v,
Err(e) => return err(e),
},
None => Value::None,
};
with_session(root, body, "value", |s| s.set_value(kernel, name, value))
}
pub fn api_assign(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let cell = v.get("cell").and_then(|x| x.as_str()).unwrap_or("");
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
if cell.is_empty() || kernel.is_empty() {
return err("missing cell/kernel".into());
}
with_session(root, body, "value", |s| s.assign_kernel(cell, kernel))
}
pub fn api_state_list(root: &Path, body: &str) -> serde_json::Value {
with_session(root, body, "state", |s| s.state_list())
}
pub fn api_state_push(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
let name = v.get("name").and_then(|x| x.as_str()).unwrap_or("");
if kernel.is_empty() || name.is_empty() {
return err("missing kernel/name".into());
}
let value = match v.get("value") {
Some(v) => match json_to_value(v) {
Ok(v) => v,
Err(e) => return err(e),
},
None => Value::None,
};
with_session(root, body, "value", |s| s.push_value(kernel, name, value))
}
pub fn api_state_pull(root: &Path, body: &str) -> serde_json::Value {
let v = body_value(body);
let kernel = v.get("kernel").and_then(|x| x.as_str()).unwrap_or("");
let name = v.get("name").and_then(|x| x.as_str()).unwrap_or("");
if kernel.is_empty() || name.is_empty() {
return err("missing kernel/name".into());
}
with_session(root, body, "value", |s| s.pull_value(kernel, name))
}
pub fn api_journal(root: &Path, body: &str) -> serde_json::Value {
with_session(root, body, "journal", |s| Ok(s.journal_json()))
}
#[cfg(test)]
mod tests {
use super::*;
fn node_available() -> bool {
std::process::Command::new("node")
.args(["-e", "process.exit(0)"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
fn open_body() -> &'static str {
r#"{"path":"shared.mq.md"}"#
}
#[test]
fn api_shared_state_and_kernel_switch() {
if !node_available() {
return;
}
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/notebook");
let open = api_open(&root, open_body());
assert_eq!(open.get("ok").and_then(|v| v.as_bool()), Some(true), "{open:?}");
let run = api_cell_run(
&root,
r#"{"path":"shared.mq.md","cell":"设置"}"#,
);
assert_eq!(run.get("ok").and_then(|v| v.as_bool()), Some(true), "{run:?}");
let read = api_cell_run(
&root,
r#"{"path":"shared.mq.md","cell":"读取"}"#,
);
let stdout = read
.get("record")
.and_then(|r| r.get("stdout"))
.and_then(|v| v.as_str())
.unwrap_or("");
assert_eq!(stdout, "42\n", "{read:?}");
let assign = api_assign(
&root,
r#"{"path":"shared.mq.md","cell":"读取","kernel":"绘图"}"#,
);
assert_eq!(assign.get("ok").and_then(|v| v.as_bool()), Some(true), "{assign:?}");
let failed = api_cell_run(
&root,
r#"{"path":"shared.mq.md","cell":"读取"}"#,
);
assert_eq!(
failed
.get("record")
.and_then(|r| r.get("ok"))
.and_then(|v| v.as_bool()),
Some(false),
"{failed:?}"
);
let clear = api_clear(&root, r#"{"path":"shared.mq.md"}"#);
assert_eq!(clear.get("ok").and_then(|v| v.as_bool()), Some(true), "{clear:?}");
let cleared = clear.get("session").and_then(|s| s.get("cells")).and_then(|c| c.as_array()).cloned().unwrap_or_default();
assert!(cleared.iter().all(|c| c.get("runs").and_then(|v| v.as_u64()) == Some(0)));
let push = api_state_push(
&root,
r#"{"path":"shared.mq.md","kernel":"分析","name":"z","value":9}"#,
);
assert_eq!(push.get("ok").and_then(|v| v.as_bool()), Some(true), "{push:?}");
let pull = api_state_pull(
&root,
r#"{"path":"shared.mq.md","kernel":"分析","name":"z"}"#,
);
assert_eq!(
pull.get("value").and_then(|v| v.as_i64()),
Some(9),
"{pull:?}"
);
let _ = api_close(&root, open_body());
}
}
+328 -7
View File
@@ -1,20 +1,275 @@
//! Static HTML export: `marqdo view output`.
use std::collections::HashMap;
use std::fs;
use std::path::PathBuf;
use anyhow::{Context, Result};
use sha2::{Digest, Sha256};
use crate::view::html::{page_file, page_index, LinkMode};
use crate::host::foreign_doc::DocumentSession;
use crate::host::json::json_to_value;
use crate::value::Value;
use crate::view::html::{escape, page_file, page_index, LinkMode};
use crate::view::{build_file_view, build_root, ViewRoot};
pub struct OutputOptions {
pub path: PathBuf,
pub out_dir: PathBuf,
pub no_exec: bool,
pub snapshot: Option<PathBuf>,
}
pub struct CaptureOptions {
pub path: PathBuf,
pub out: PathBuf,
}
#[derive(Debug, Clone)]
struct StaticCellOutput {
ok: bool,
text: String,
}
fn source_hash(source: &str) -> String {
let mut h = Sha256::new();
h.update(source.as_bytes());
format!("{:x}", h.finalize())
}
fn value_text(v: &Value) -> Option<String> {
match v {
Value::Text(s) => Some(s.clone()),
Value::None => None,
_ => v.reveal_text().map(|s| s.to_string()),
}
}
fn value_bool(v: &Value) -> Option<bool> {
match v {
Value::Bool(b) => Some(*b),
_ => None,
}
}
fn record_field(record: &Value, key: &str) -> Option<Value> {
match record {
Value::Map(entries) => entries
.iter()
.find(|(k, _)| k == key)
.map(|(_, v)| v.clone()),
_ => None,
}
}
/// Run the document's cells at export time and keep their real outputs. This
/// is what makes a static page show foreign results instead of just source.
fn static_page_data(
root: &ViewRoot,
abs: &std::path::Path,
stdin: &str,
) -> (HashMap<String, StaticCellOutput>, Option<Value>) {
let mut out = HashMap::new();
let Ok(mut session) = DocumentSession::open(abs, &root.root) else {
return (out, None);
};
let page_stdin = if stdin.trim().is_empty() { None } else { Some(stdin) };
let Ok(summary) = session.run_all_with_stdin(false, page_stdin) else {
session.close();
return (out, None);
};
let records = match &summary {
Value::Map(entries) => entries
.iter()
.find(|(k, _)| k == "records")
.map(|(_, v)| v.clone()),
_ => None,
};
if let Some(Value::List(records)) = records {
for rec in records {
let Some(cell) = record_field(&rec, "cell").and_then(|v| value_text(&v)) else {
continue;
};
let ok = record_field(&rec, "ok")
.and_then(|v| value_bool(&v))
.unwrap_or(false);
let mut parts = Vec::new();
if let Some(stdout) = record_field(&rec, "stdout").and_then(|v| value_text(&v)) {
if !stdout.is_empty() {
parts.push(stdout);
}
}
if let Some(stderr) = record_field(&rec, "stderr").and_then(|v| value_text(&v)) {
if !stderr.is_empty() {
parts.push(format!("[stderr]\n{stderr}"));
}
}
if let Some(error) = record_field(&rec, "error").and_then(|v| value_text(&v)) {
if !error.is_empty() {
parts.push(error);
}
}
let has_body = !parts.is_empty();
let mut meta = Vec::new();
if let Some(code) = record_field(&rec, "code").and_then(|v| match v {
Value::Int(n) => Some(n),
_ => None,
}) {
if code != 0 {
meta.push(format!("exit {code}"));
}
}
if let Some(ms) = record_field(&rec, "ms").and_then(|v| match v {
Value::Int(n) => Some(n),
_ => None,
}) {
if ms > 0 {
meta.push(format!("{ms}ms"));
}
}
if !meta.is_empty() {
parts.push(format!("— {}", meta.join(" · ")));
}
let text = if !has_body {
"(no output)".to_string()
} else {
parts.join("\n\n")
};
out.insert(cell, StaticCellOutput { ok, text });
}
}
let view_summary = session.summary();
session.close();
(out, Some(view_summary))
}
/// Inject static cell outputs into code cards without running client JS.
fn inject_cell_outputs(html: &str, outputs: &HashMap<String, StaticCellOutput>) -> String {
let mut html = html.to_string();
for (name, out) in outputs {
let marker = format!(r#"data-code-name="{name}""#);
let Some(pos) = html.find(&marker) else {
continue;
};
let tail = &html[pos..];
let Some(out_pos) = tail.find(r#"<pre class="code-out" hidden></pre>"#) else {
continue;
};
let abs_pos = pos + out_pos;
let replacement = if out.ok {
format!(r#"<pre class="code-out">{}</pre>"#, escape(&out.text))
} else {
format!(r#"<pre class="code-out err">{}</pre>"#, escape(&out.text))
};
html.replace_range(
abs_pos..abs_pos + r#"<pre class="code-out" hidden></pre>"#.len(),
&replacement,
);
}
html
}
fn inject_static_session(html: &str, summary: Option<&Value>) -> String {
let Some(summary) = summary else {
return html.to_string();
};
let Ok(json) = crate::host::json::value_to_json(summary) else {
return html.to_string();
};
let Ok(json) = serde_json::to_string(&json) else {
return html.to_string();
};
let line = format!(
"window.MARQDO_VIEW_LIVE = false;\nwindow.MARQDO_STATIC_SESSION = {json};"
);
html.replace("window.MARQDO_VIEW_LIVE = false;", &line)
}
/// Capture live-cell results into a snapshot consumed by `view output --snapshot`.
pub fn write_capture(opts: CaptureOptions) -> Result<()> {
let root = build_root(&opts.path)?;
let mut pages = serde_json::Map::new();
for rel_pb in &root.files {
let rel = rel_pb.to_string_lossy().replace('\\', "/");
let abs = resolve_abs(&root, &rel)?;
let vm = build_file_view(&root, &abs, &rel, &[], false)?;
let (outputs, summary) = static_page_data(&root, &abs, &vm.preset_stdin);
let mut out_map = serde_json::Map::new();
for (name, output) in outputs {
out_map.insert(
name,
serde_json::json!({ "ok": output.ok, "text": output.text }),
);
}
let summary_json = summary
.as_ref()
.and_then(|v| crate::host::json::value_to_json(v).ok())
.unwrap_or(serde_json::Value::Null);
pages.insert(
rel.clone(),
serde_json::json!({
"source_hash": source_hash(&vm.source),
"summary": summary_json,
"outputs": out_map,
}),
);
}
let doc = serde_json::json!({ "version": 1, "pages": pages });
if let Some(parent) = opts.out.parent() {
fs::create_dir_all(parent)?;
}
fs::write(&opts.out, serde_json::to_string_pretty(&doc)?)
.with_context(|| format!("write {}", opts.out.display()))?;
eprintln!("marqdo view capture: {} page(s) → {}", root.files.len(), opts.out.display());
Ok(())
}
fn snapshot_page(
snapshot: &serde_json::Value,
rel: &str,
source: &str,
) -> Result<(HashMap<String, StaticCellOutput>, Option<Value>)> {
let page = snapshot
.get("pages")
.and_then(|p| p.get(rel))
.ok_or_else(|| anyhow::anyhow!("snapshot has no page `{rel}`"))?;
let expected = page
.get("source_hash")
.and_then(|v| v.as_str())
.unwrap_or("");
if expected != source_hash(source) {
anyhow::bail!("snapshot source hash mismatch for `{rel}`");
}
let mut outputs = HashMap::new();
if let Some(map) = page.get("outputs").and_then(|v| v.as_object()) {
for (name, value) in map {
outputs.insert(
name.clone(),
StaticCellOutput {
ok: value.get("ok").and_then(|v| v.as_bool()).unwrap_or(false),
text: value
.get("text")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string(),
},
);
}
}
let summary = page
.get("summary")
.filter(|v| !v.is_null())
.and_then(|v| json_to_value(v).ok());
Ok((outputs, summary))
}
pub fn write_static(opts: OutputOptions) -> Result<()> {
let snapshot = match &opts.snapshot {
Some(path) => Some(serde_json::from_str::<serde_json::Value>(
&fs::read_to_string(path).with_context(|| format!("read {}", path.display()))?,
)?),
None => None,
};
let root = build_root(&opts.path)?;
fs::create_dir_all(&opts.out_dir)
.with_context(|| format!("create {}", opts.out_dir.display()))?;
@@ -48,7 +303,20 @@ pub fn write_static(opts: OutputOptions) -> Result<()> {
let links = LinkMode::Static {
from: Some(rel.clone()),
};
let html = page_file(&root.files, &rel, &vm, &links);
let (cell_outputs, static_summary) = if let Some(snapshot) = &snapshot {
snapshot_page(snapshot, &rel, &vm.source)?
} else if opts.no_exec {
(HashMap::new(), None)
} else {
static_page_data(&root, &abs, &vm.preset_stdin)
};
let html = inject_static_session(
&inject_cell_outputs(
&page_file(&root.files, &rel, &vm, &links),
&cell_outputs,
),
static_summary.as_ref(),
);
let out_file = pages_root.join(format!("{rel}.html"));
if let Some(parent) = out_file.parent() {
fs::create_dir_all(parent)?;
@@ -57,11 +325,17 @@ pub fn write_static(opts: OutputOptions) -> Result<()> {
if first_page_html.is_none() {
// index.html = first file (same content, links rooted at OUT_DIR)
first_page_html = Some(page_file(
&root.files,
&rel,
&vm,
&LinkMode::Static { from: None },
first_page_html = Some(inject_static_session(
&inject_cell_outputs(
&page_file(
&root.files,
&rel,
&vm,
&LinkMode::Static { from: None },
),
&cell_outputs,
),
static_summary.as_ref(),
));
}
}
@@ -85,3 +359,50 @@ fn resolve_abs(root: &ViewRoot, rel: &str) -> Result<PathBuf> {
let abs = root.root.join(rel);
Ok(abs.canonicalize().unwrap_or(abs))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn snapshot_page_rejects_source_hash_mismatch() {
let snap = serde_json::json!({
"pages": {
"a.mq.md": {
"source_hash": "bad",
"summary": null,
"outputs": {}
}
}
});
assert!(snapshot_page(&snap, "a.mq.md", "# main\n").is_err());
}
#[test]
fn snapshot_page_reads_outputs_and_summary() {
let source = "# main\n";
let hash = source_hash(source);
let snap = serde_json::json!({
"pages": {
"a.mq.md": {
"source_hash": hash,
"summary": {"persistent": false, "cells": [], "kernels": []},
"outputs": {"a": {"ok": true, "text": "42"}}
}
}
});
let (outputs, summary) = snapshot_page(&snap, "a.mq.md", source).unwrap();
assert_eq!(outputs["a"].text, "42");
assert!(summary.is_some());
}
#[test]
fn injects_visible_static_cell_outputs() {
let html = r#"<div class="card" data-code-name="a"></div><pre class="code-out" hidden></pre>"#;
let mut outputs = HashMap::new();
outputs.insert("a".to_string(), StaticCellOutput { ok: true, text: "42".into() });
let html = inject_cell_outputs(html, &outputs);
assert!(html.contains(r#"<pre class="code-out">42</pre>"#), "{html}");
assert!(!html.contains("hidden"), "{html}");
}
}
+5 -1
View File
@@ -1254,11 +1254,13 @@ fn render_stmt(
let lang = escape(&c.lang);
let src = escape(&c.source);
format!(
r#"<div class="card code-card" data-lang="{lang}">
r#"<div class="card code-card" data-lang="{lang}" data-code-name="{name}">
<div class="code-head">
<span class="badge">code</span>
<code class="expr">`{name}` =</code>
<span class="code-lang">{lang}</span>
<button type="button" class="nb-clean" title="Clear this output">Clean</button>
<button type="button" class="nb-persist" title="Persist this result with writeback">Persist</button>
</div>
<pre class="code-body"><code class="language-{lang}">{src}</code></pre>
<div class="code-foot">
@@ -1537,6 +1539,8 @@ pub(crate) fn render_code_assign_card(name: &str, lang: &str, source: &str) -> S
<span class="badge">code</span>
<span class="code-name">{name}</span>
<span class="code-lang">{lang}</span>
<button type="button" class="nb-clean" title="Clear this output">Clean</button>
<button type="button" class="nb-persist" title="Persist this result with writeback">Persist</button>
</div>
<pre class="code-body"><code class="language-{lang}">{src}</code></pre>
<div class="code-foot">
+77
View File
@@ -0,0 +1,77 @@
---
title: agent / llm soft transport errors
description: >-
Regression (2026-10): unreachable OpenAI upstream must not abort the VM.
`llm.ask` / `agent.step` / `agent.run` return structured status=error.
import agent:ext/ai/agent.mq.md
import json:lib/json.mq.md
import llm:ext/ai/llm.mq.md
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
---
# main
> sys.env_set name="OPENAI_API_KEY" value="offline-transport-soft"
> sys.env_set name="OPENAI_BASE_URL" value="http://127.0.0.1:9"
**p = > plugin.native_path name="agent"**
1. `p`
> plugin.load path=`p`
2. *
> print text=no-agent-plugin
> sys.exit code=1
**model = > llm.llm**
**ask = > model.ask prompt="ping soft transport"**
**fin = > json.get value=`ask` key="finish"**
**aerr = > json.get value=`ask` key="error"**
1. `fin` == "error"
1. `aerr`
> print text=llm-ask-soft-ok
2. *
> print text=llm-ask-missing-error
> print text=`ask`
> sys.exit code=1
2. *
> print text=llm-ask-hard-or-ok-unexpected
> print text=`ask`
> sys.exit code=1
`tools` =
| 工具 |
|------|
**ag = > agent.agent model=`model` tools=`tools` standing="transport soft probe"**
**step = > `ag`.step task="unreachable upstream probe" writeback=False max_reads=1**
**sst = > json.get value=`step` key="status"**
**serr = > json.get value=`step` key="error"**
1. `sst` == "error"
1. `serr`
> print text=agent-step-soft-ok
2. *
> print text=agent-step-missing-error
> print text=`step`
> sys.exit code=1
2. *
> print text=agent-step-unexpected
> print text=`step`
> sys.exit code=1
**out = > `ag`.run task="unreachable upstream probe" efi=False learn=False verify=False writeback=False reuse=False near_match=False max_rounds=1**
**rst = > json.get value=`out` key="status"**
**rerr = > json.get value=`out` key="error"**
1. `rst` == "error"
1. `rerr`
> print text=agent-run-soft-ok
2. *
> print text=agent-run-missing-error
> print text=`out`
> sys.exit code=1
2. *
> print text=agent-run-unexpected
> print text=`out`
> sys.exit code=1
*"agent-llm-transport-soft-ok"*
+115
View File
@@ -3317,6 +3317,121 @@ fn lib_foreign_zh() {
assert_out("tests/lib/外联-python.mq.md", "中文外联");
}
fn node_available() -> bool {
Command::new("node")
.args(["-e", "process.exit(0)"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
#[test]
fn lib_foreign_invoke() {
if !node_available() {
eprintln!("skip lib_foreign_invoke: no node on PATH");
return;
}
assert_out("tests/lib/foreign-invoke.mq.md", "False\n3\nboom\nFalse");
}
#[test]
fn lib_foreign_json() {
if !node_available() {
eprintln!("skip lib_foreign_json: no node on PATH");
return;
}
assert_out("tests/lib/foreign-json.mq.md", "True\n4\n10");
}
#[test]
fn lib_foreign_call() {
if !node_available() {
eprintln!("skip lib_foreign_call: no node on PATH");
return;
}
assert_out(
"tests/lib/foreign-call.mq.md",
"2\n[2, 4, 6, 8]",
);
}
#[test]
fn lib_foreign_env_timeout() {
if !node_available() {
eprintln!("skip lib_foreign_env_timeout: no node on PATH");
return;
}
assert_out(
"tests/lib/foreign-env-timeout.mq.md",
"fast|node\nFalse\nTrue",
);
}
#[test]
fn lib_foreign_probe_register() {
if !node_available() {
eprintln!("skip lib_foreign_probe_register: no node on PATH");
return;
}
assert_out(
"tests/lib/foreign-probe-register.mq.md",
"True\nnode\ntiny",
);
}
#[test]
fn lib_foreign_config() {
if !node_available() {
eprintln!("skip lib_foreign_config: no node on PATH");
return;
}
let root = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
let output = Command::new(env!("CARGO_BIN_EXE_marqdo"))
.args(["run", "tests/lib/foreign-config.mq.md"])
.env(
"MARQDO_FOREIGN_CONFIG",
root.join("tests/lib/fixtures/foreign.toml"),
)
.current_dir(&root)
.output()
.expect("run foreign-config");
let code = output.status.code().unwrap_or(1);
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
assert_eq!(code, 0, "foreign-config stderr={stderr}");
assert_eq!(stdout.trim_end(), "1\ncfg", "stdout={stdout}");
}
#[test]
fn lib_foreign_session() {
if !node_available() {
eprintln!("skip lib_foreign_session: no node on PATH");
return;
}
assert_out("tests/lib/foreign-session.mq.md", "42\n41\n5\n1\nTrue");
}
#[test]
fn lib_foreign_session_timeout() {
if !node_available() {
eprintln!("skip lib_foreign_session_timeout: no node on PATH");
return;
}
assert_out(
"tests/lib/foreign-session-timeout.mq.md",
"False\nTrue\nFalse",
);
}
#[test]
fn lib_foreign_session_python() {
if !python_available() {
eprintln!("skip lib_foreign_session_python: no python on PATH");
return;
}
assert_out("tests/lib/foreign-session-python.mq.md", "42\n5");
}
#[test]
fn bytecode_lib_math_formula() {
assert_out_backend(
+2
View File
@@ -0,0 +1,2 @@
[tinycfg]
template = ["node", "-e", "{inline}"]
+30
View File
@@ -0,0 +1,30 @@
---
title: foreign call and call_many
import foreign:lib/foreign.mq.md
---
# main
`双倍` =
```node
let s = "";
process.stdin.on("data", d => { s += d; });
process.stdin.on("end", () => {
console.log(JSON.stringify(JSON.parse(s) * 2));
});
```
**单个 = > foreign.call code=`双倍` input=1**
> print text=`单个`
`数值表` =
| 数值 |
|------|
| 1 |
| 2 |
| 3 |
| 4 |
**多个 = > foreign.call_many code=`双倍` inputs=`数值表` concurrency=2**
> print text=`多个`
+12
View File
@@ -0,0 +1,12 @@
---
title: foreign config load
import foreign:lib/foreign.mq.md
---
# main
**载入 = > foreign.load_config**
> print text=`载入`
**运行 = > foreign.invoke lang="tinycfg" source="console.log('cfg')"**
> print text=[stdout](`运行`)
+32
View File
@@ -0,0 +1,32 @@
---
title: foreign env cwd timeout
import foreign:lib/foreign.mq.md
---
# main
`环境` =
| MODE | NOTE |
|------|------|
| fast | test |
`环境检查` =
```node
console.log(process.env.MODE + "|" + process.env.MARQDO_FOREIGN_LANG);
```
**检查 = > foreign.invoke code=`环境检查` env=`环境`**
**检查输出 = > trim [stdout](`检查`)**
> print text=`检查输出`
`慢` =
```node
setTimeout(() => console.log("late"), 5000);
```
**超时 = > foreign.invoke code=`慢` timeout=0.5**
> print text=[ok](`超时`)
> print text=[timed_out](`超时`)
+19
View File
@@ -0,0 +1,19 @@
---
title: foreign invoke record
import foreign:lib/foreign.mq.md
---
# main
`失败` =
```node
process.stderr.write("boom");
process.exit(3);
```
**记录 = > foreign.invoke code=`失败`**
> print text=[ok](`记录`)
> print text=[code](`记录`)
> print text=[stderr](`记录`)
> print text=[truncated](`记录`)
+32
View File
@@ -0,0 +1,32 @@
---
title: foreign invoke JSON input
import foreign:lib/foreign.mq.md
---
# main
`数值表` =
| 数值 |
|------|
| 1 |
| 2 |
| 3 |
| 4 |
`统计` =
```node
let s = "";
process.stdin.on("data", d => { s += d; });
process.stdin.on("end", () => {
const xs = JSON.parse(s);
console.log(JSON.stringify({ n: xs.length, sum: xs.reduce((a, b) => a + b, 0) }));
});
```
**记录 = > foreign.invoke code=`统计` input=`数值表` output="json"**
**统计结果 = [result](`记录`)**
> print text=[ok](`记录`)
> print text=[n](`统计结果`)
> print text=[sum](`统计结果`)
+24
View File
@@ -0,0 +1,24 @@
---
title: foreign probe and register
import foreign:lib/foreign.mq.md
---
# main
**探测 = > foreign.probe lang="node"**
> print text=[ok](`探测`)
> print text=[lang](`探测`)
`模板` =
| 步骤 |
|------|
| node |
| "-e" |
| {inline} |
**> foreign.register lang="tiny" template=`模板`**
**运行 = > foreign.invoke lang="tiny" source="console.log('tiny')"**
**运行输出 = > trim [stdout](`运行`)**
> print text=`运行输出`
**> foreign.unset lang="tiny"**
+18
View File
@@ -0,0 +1,18 @@
---
title: foreign python session
import foreign:lib/foreign.mq.md
---
# main
**会话 = > foreign.session_open lang="python"**
**设值 = > foreign.session_eval id=`会话` source="x = 41"**
**求值 = > foreign.session_eval id=`会话` source="x + 1"**
> print text=[result](`求值`)
**> foreign.session_set id=`会话` name="y" value=5**
**取值 = > foreign.session_get id=`会话` name="y"**
> print text=`取值`
**> foreign.session_close id=`会话`**
+13
View File
@@ -0,0 +1,13 @@
---
title: foreign session timeout
import foreign:lib/foreign.mq.md
---
# main
**会话 = > foreign.session_open lang="node"**
**慢 = > foreign.session_eval id=`会话` source="(async () => { await new Promise(r => setTimeout(r, 5000)); return 1; })()" timeout=0.3**
> print text=[ok](`慢`)
> print text=[timed_out](`慢`)
> print text=[alive](`慢`)
+27
View File
@@ -0,0 +1,27 @@
---
title: foreign persistent session
import foreign:lib/foreign.mq.md
---
# main
**会话 = > foreign.session_open lang="node"**
**设值 = > foreign.session_eval id=`会话` source="let x = 41"**
**求值 = > foreign.session_eval id=`会话` source="x + 1"**
> print text=[result](`求值`)
**取x = > foreign.session_get id=`会话` name="x"**
> print text=`取x`
**> foreign.session_set id=`会话` name="y" value=5**
**取值 = > foreign.session_get id=`会话` name="y"**
> print text=`取值`
**会话表 = > foreign.session_list**
**数量 = > len `会话表`**
> print text=`数量`
**关闭 = > foreign.session_close id=`会话`**
> print text=`关闭`
+31
View File
@@ -0,0 +1,31 @@
---
title: View without foreign
description: A pure Marqdo document — formula, table, plot and prose, no external kernel
import math:lib/math.mq.md
---
# main
This page has no foreign import and no ` ```lang ` code cell. It is still opened by the same `view` surface, with the same toolbar and layout. The cell list is simply empty.
The formula is ordinary Marqdo data:
$$
% marqdo: var=f
x^2 - 2
$$
**_plot = math.plot formula=`f` var="x" min=-4 max=4**
| region | orders | revenue |
|--------|--------|---------|
| 华东 | 45 | 56253 |
| 华南 | 45 | 67187 |
| 华北 | 45 | 65272 |
| 西部 | 45 | 66884 |
**total = 56253 + 67187 + 65272 + 66884**
> print text=`total`
This is the control page: no foreign, no cells, no separate mode. The same view shell is used.
+211
View File
@@ -0,0 +1,211 @@
---
title: Python 经营洞察 · View Cells
description: 多 cell 共享 Python 内核,做订单生成、区域/产品指标、异常检测、报告输出与独立复核。
import foreign:lib/foreign.mq.md
---
# main
`内核表` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | python | | 共享数据清洗、指标与异常检测 |
| 复核 | python | | 独立内核,只通过显式 JSON 文件复核 |
`会话表` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 准备数据 | 分析 | 生成可复现的订单数据 |
| 计算指标 | 分析 | 区域、产品、周度趋势 |
| 检测异常 | 分析 | 按区域金额做 z-score |
| 生成报告 | 分析 | ASCII 图 + JSON 报告 |
| 独立复核 | 复核 | 另一个内核读取报告做校验 |
`会话` = > foreign.session kernels=`内核表` cells=`会话表`
`准备数据` =
```python
import math
import random
from collections import defaultdict
from datetime import date, timedelta
SEED = 20261017
rng = random.Random(SEED)
START = date(2026, 1, 5)
REGIONS = ["华东", "华南", "华北", "西部"]
PRODUCTS = {"A": "智能传感器", "B": "边缘网关", "C": "数据订阅", "D": "运维服务"}
BASE_PRICE = {"A": 420.0, "B": 980.0, "C": 260.0, "D": 640.0}
def make_orders(n=180):
rows = []
for i in range(n):
week = i // 15 + 1
region = REGIONS[i % len(REGIONS)]
product = "ABCD"[(i * 7 + week) % 4]
seasonal = 1.0 + 0.12 * math.sin(week / 2.2)
qty = max(1, int(rng.gauss(3.0, 1.2)))
amount = round(BASE_PRICE[product] * qty * seasonal * rng.gauss(1.0, 0.08), 2)
rows.append({
"id": f"O{i + 1:04d}",
"week": week,
"region": region,
"product": product,
"qty": qty,
"amount": amount,
"day": (START + timedelta(days=i * 4)).isoformat(),
})
return rows
orders = make_orders()
assert len(orders) == 180
print(f"生成订单: {len(orders)} 条")
print(f"时间范围: {orders[0]['day']} → {orders[-1]['day']}")
print(f"种子: {SEED}")
print(f"样例: {orders[0]}")
```
`计算指标` =
```python
import statistics
by_region = defaultdict(list)
by_product = defaultdict(float)
by_week = defaultdict(float)
for o in orders:
by_region[o["region"]].append(o["amount"])
by_product[o["product"]] += o["amount"]
by_week[o["week"]] += o["amount"]
region_stats = []
for region, amounts in sorted(by_region.items()):
region_stats.append({
"region": region,
"orders": len(amounts),
"total": round(sum(amounts), 2),
"mean": round(statistics.mean(amounts), 2),
"median": round(statistics.median(amounts), 2),
})
region_stats.sort(key=lambda r: r["total"], reverse=True)
weekly_avg = [round(by_week[w] / 15, 2) for w in sorted(by_week)]
first3 = statistics.mean(weekly_avg[:3])
last3 = statistics.mean(weekly_avg[-3:])
trend_pct = round((last3 / first3 - 1) * 100, 2)
top_products = sorted(by_product.items(), key=lambda kv: kv[1], reverse=True)
total_revenue = round(sum(o["amount"] for o in orders), 2)
metrics = {
"order_count": len(orders),
"total_revenue": total_revenue,
"avg_order_value": round(total_revenue / len(orders), 2),
"trend_pct": trend_pct,
"region_stats": region_stats,
"top_products": [
{"code": code, "name": PRODUCTS[code], "revenue": round(value, 2)}
for code, value in top_products
],
}
print(f"总营收: ¥{total_revenue:,.2f}")
print(f"客单价: ¥{metrics['avg_order_value']:,.2f}")
print(f"周度趋势: {trend_pct:+.2f}%(首 3 周 vs 末 3 周)")
print("\n区域表现(按营收降序):")
for r in region_stats:
print(f" {r['region']} 订单 {r['orders']:>2} 营收 ¥{r['total']:>10,.2f} 中位数 ¥{r['median']:>8,.2f}")
print("\n产品营收:")
for item in metrics["top_products"]:
print(f" {item['code']} {item['name']} ¥{item['revenue']:,.2f}")
```
`检测异常` =
```python
by_region_amounts = defaultdict(list)
for o in orders:
by_region_amounts[o["region"]].append(o["amount"])
anomalies = []
for region, amounts in by_region_amounts.items():
mean = statistics.mean(amounts)
sd = statistics.pstdev(amounts)
if sd == 0:
continue
for o in orders:
if o["region"] != region:
continue
z = (o["amount"] - mean) / sd
if abs(z) >= 1.8:
anomalies.append({
"id": o["id"],
"region": region,
"product": PRODUCTS[o["product"]],
"amount": o["amount"],
"z": round(z, 2),
})
anomalies.sort(key=lambda a: abs(a["z"]), reverse=True)
print(f"异常订单: {len(anomalies)} / {len(orders)}(|z| ≥ 1.8)")
for a in anomalies[:6]:
print(f" {a['id']} {a['region']} {a['product']} ¥{a['amount']:,.2f} z={a['z']:+.2f}")
if len(anomalies) > 6:
print(f" … 其余 {len(anomalies) - 6} 条")
```
`生成报告` =
```python
import json
lines = []
lines.append("=" * 56)
lines.append("Marqdo Python 经营洞察 · 周报")
lines.append("=" * 56)
lines.append(f"订单数: {len(orders)} 总营收: ¥{total_revenue:,.2f} 趋势: {trend_pct:+.2f}%")
lines.append("")
max_total = max(r["total"] for r in region_stats)
for r in region_stats:
width = max(1, round(20 * r["total"] / max_total))
lines.append(f"{r['region']:<3} {r['total']:>10,.2f} {'█' * width}")
lines.append("")
lines.append(f"异常订单: {len(anomalies)} 最大异常: {anomalies[0]['id'] if anomalies else '无'}")
lines.append("=" * 56)
report = {
"title": "Marqdo Python 经营洞察",
"order_count": len(orders),
"total_revenue": total_revenue,
"trend_pct": trend_pct,
"regions": region_stats,
"top_products": metrics["top_products"],
"anomaly_count": len(anomalies),
}
report_path = "/tmp/marqdo-python-insights-report.json"
with open(report_path, "w", encoding="utf-8") as f:
json.dump(report, f, ensure_ascii=False, indent=2)
print("\n".join(lines))
print(f"\nJSON 报告已写入: {report_path}")
```
`独立复核` =
```python
import json
import os
path = "/tmp/marqdo-python-insights-report.json"
print("独立内核中 orders 可见:", "orders" in globals())
if not os.path.exists(path):
print("未找到报告,跳过复核")
else:
with open(path, encoding="utf-8") as f:
report = json.load(f)
region_sum = round(sum(r["total"] for r in report["regions"]), 2)
total_ok = abs(region_sum - report["total_revenue"]) < 0.01
count_ok = report["order_count"] == 180
regions_ok = len(report["regions"]) == 4
print(f"报告订单数: {report['order_count']}")
print(f"分区域合计: ¥{region_sum:,.2f} / 总额 ¥{report['total_revenue']:,.2f}")
print(f"区域数: {len(report['regions'])} 异常数: {report['anomaly_count']}")
print("复核通过:", total_ok and count_ok and regions_ok)
```
+38
View File
@@ -0,0 +1,38 @@
---
title: notebook shared state
import foreign:lib/foreign.mq.md
---
# main
`内核表` =
| 名称 | 语言 | 命令 | 说明 |
|------|------|------|------|
| 分析 | node | | 共享状态的内核 |
| 绘图 | node | | 隔离的内核 |
`会话表` =
| 单元 | 内核 | 说明 |
|------|------|------|
| 设置 | 分析 | 写入共享变量 |
| 读取 | 分析 | 读取共享变量 |
| 隔离 | 绘图 | 看不到 shared |
`会话` = > foreign.session kernels=`内核表` cells=`会话表`
`设置` =
```node
let shared = 41;
```
`读取` =
```node
console.log(shared + 1);
```
`隔离` =
```node
console.log(typeof shared === "undefined" ? "isolated" : "leaked");
```