Compare commits

..
11 Commits
Author SHA1 Message Date
cflmyandCursor 3ae8f2c843 release: ext pack v1.3.1 OIDC path redirect_uri
Bump ext/VERSION only; CLI remains 1.3.0.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-08 11:42:08 +08:00
cflmyandCursor e9c1a8a730 feat(web): resolve OIDC redirect_uri from request Host
Allow path-style redirect_uri (/oidc/callback) so apex/www and
proxied HTTPS share one config; persist the absolute URI for token exchange.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-08 11:35:12 +08:00
cflmyandCursor 5554b1642c feat(run): add --fs-root / MARQDO_FS_ROOT for subdirectory entries
marqdo run cli/… sandboxes to the entry directory by default; repo-root
sandbox+cwd lets qdagent keep thin CLI wrappers under cli/ without
escaping .marqdo or polluting cli/.marqdo.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 22:17:24 +08:00
cflmyandCursor fe15603a91 fix(llm): hoist ask result fields out of the return table
Table cells shadowed self/column names (model, usage), so [model](self)
and [total_tokens](usage) raised "footnote index needs list or map" and
blocked live agent.step / qdagent BUILD.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 22:08:48 +08:00
cflmyandCursor 1418742655 Skip docker/data/.cursor when compiling engineering knowledge.
Packaging trees and runtime vaults were inflating EKC with duplicate noise; dogfood agents should index product sources only.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 21:50:58 +08:00
cflmyandCursor ed31a2b31d release: v1.3.0 Engineering Knowledge Compiler (EKC)
Bump CLI/ext/wasm to 1.3.0; document L0–L4 catalog, reuse preflight, and
agent eng_* surfaces for AI-native engineering.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 16:46:30 +08:00
cflmyandCursor 56ab52356f feat(knowledge): land Engineering Knowledge Compiler (EKC) from 008
Give agents Engineering First Information before codegen: L0–L4 catalog,
reuse/preflight (REUSE→ADAPT→CREATE), graph, and CLI surfaces so large
repos stop reinventing existing capabilities.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 16:44:39 +08:00
cflmyandCursor 2e425c3010 chore: sync Cargo.lock to 1.2.0 for PPA vendor
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 11:59:48 +08:00
cflmyandCursor 46a0263d7f release: v1.2.0 Web Artifact (ADR 0007) Document/Endpoint/Resource
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 11:39:25 +08:00
cflmyandCursor ff252fd469 feat(web): land Artifact serve/endpoint path and retire compose golds
Wire data_source into Document render, register Endpoint handlers via
run_artifact, restore marqdo-blog as an Artifact site, and replace old
compose_* web golds with ADR 0007 fixtures and docs.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 11:36:45 +08:00
cflmyandCursor 67bac1bf02 feat(web): Document/Endpoint/Resource artifact rewrite (ADR 0007)
Replace the monolithic compose_*/configure author surface with file-level
web|endpoint artifacts, split ext/{web,data,security,net}, and lock
no-degrade rules so the model cannot regress to God-object APIs.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 11:06:38 +08:00
179 changed files with 14409 additions and 6966 deletions

No files matched your search

+26
View File
@@ -0,0 +1,26 @@
---
description: Marqdo Web Artifact — no degradation; Document/Endpoint/Resource only
globs: ext/web/**,ext/net/**,ext/data/**,ext/security/**,examples/**/*,tests/ext/web*,doc/design/ext-web*,doc/adr/0007*
alwaysApply: true
---
# Web Artifact — 不允许降级
Canonical: `doc/adr/0007-web-document-endpoint.md` · `doc/design/ext-web-artifact.md` · `doc/next/007.md` (superseded).
## Hard bans
1. **Do not** restore or reintroduce author-facing `compose_main` / `compose_intro` / `compose_*` fragmented table DSLs, god `# page` mutation piles, or `app.configure(...)` junk-drawer APIs.
2. **Do not** use `json.set` / `json.append` bag glue for pages, routes, or middleware. Tables = data. View schema is only `|type|slot|value|attrs|style|` (ZH aliases allowed).
3. **Do not** call `ensure_plugin` or `host_*` from author programs or from `ext/**` (plugin load is facade/capability only).
4. **Do not** register implicit user routes. Declared routes come from Artifact metadata or `web.route`. System routes (`/_part`, `/_form`, …) must appear in `web.inspect`.
5. **Do not** invent DSL, nested frontmatter under `##`, or new table dialects when the design is silent — **stop and ask**.
6. **Do not** mix EN/ZH API names in one `.mq.md`.
7. **Do not** silently delete capabilities. Removals require design text + tests updated in the same change.
8. **Do not** put DB/auth/cache/storage/websocket author APIs back into `ext/web/web.mq.md` — they live in `ext/data`, `ext/security`, `ext/net`.
## Required direction
- Document (`type: web` / `类型: 网页`), Endpoint (`type: endpoint` / `类型: 端点`), Resource modules.
- Facade only: `web.page` · `web.route` · `web.serve` · `web.render` · `web.inspect` (ZH mirrors).
- Code is documentation: the `.mq.md` is the page/API docs and the implementation.
+4 -2
View File
@@ -92,7 +92,7 @@ Better still — call a named helper:
7. **Do not use `##` inside `# main` for document sectioning** — those become nested functions. Use paragraphs / `---` breaks instead.
8. **Index:** prefer `[键](集合)` / `` [`名`](集合) `` over footnote `` m[^k] ``.
Browser client programs: prefer `import browser:lib/browser.mq.md` (no native plugin). Server sites: `ext/web` + GFM page/style tables.
Browser client programs: prefer `import browser:lib/browser.mq.md` (no native plugin). Server sites: Artifact Metadata (`type: web|endpoint`) + `ext/web` facade + `ext/data` / `ext/security` / `ext/net` — see [ext-web-artifact.md](../../doc/design/ext-web-artifact.md). **Never** restore `compose_*` / `app.configure`.
## Developing libraries (`lib/*`, `ext/*`)
@@ -121,7 +121,9 @@ Canonical thin-template + waves: [lib-code-as-docs.md](../../doc/design/lib-code
| Need | Place |
|------|--------|
| Browser effect maps (WASM client) | `lib/browser.mq.md` |
| Site HTML / HTTP / DB | `ext/web` classes + compose from tables |
| Site HTML / HTTP | `ext/web` Document/Endpoint + View tables |
| DB / cache / storage | `ext/data` |
| Auth / OIDC / RBAC | `ext/security` |
| List/map primitives | `lib/table.mq.md` |
| Parse JSON text | `lib/json.mq.md` only |
+36 -27
View File
@@ -216,7 +216,7 @@ Install: `marqdo ext add agent` (ZH: `智能体`). Native: `cargo build --releas
1. **Code is documentation** — the runbook `.mq.md` is the model’s ground truth, not a hidden system prompt bag.
2. **Call site is first-class** — `build_step_context` injects path/function/line so the model knows where it is in the program.
3. **Document is the knowledge base (OKF)** — writeback → promote/solidify → reuse / `llm_free`; MCP/corpus are evidence only.
3. **Document is the knowledge base (OKF)** — writeback → promote/solidify → reuse / `llm_free`; MCP/corpus are evidence only. **Engineering Knowledge(EKC)**:`marqdo knowledge|find|reuse|…` + `agent.preflight`(REUSE→ADAPT→CREATE)——见 [engineering-knowledge.md](../../doc/design/engineering-knowledge.md)。
4. **Skill Compilation (v2)** — successful runs become `.mq.md` episodes; after enough evidence, `maybe_learn` compiles an executable skill (`llm_free`). Prefer `## run` over bare `## plan` when you want learning.
5. **Adaptive Routing (P4)** — router is abstract (`marqdo` / `small-llm` / `llm` / `auto`); `jev` is a compat alias of `small-llm`. Decision via `router_model` (any llm handle). Policy Compilation writes `policies/router.mq.md`.
@@ -224,44 +224,53 @@ APIs: `# agent` / `# 智能体` → `## run` / `## route` / `## step` / `## plan
Examples: [agent-pong](../../examples/agent-pong/) · [agent-okf-flywheel](../../examples/agent-okf-flywheel/). Harness: `scripts/agent-harness.sh`. Design: [ext-agent.md](../../doc/design/ext-agent.md) · Wave B: [agent-framework-2026-09.md](../../doc/research/agent-framework-2026-09.md) · v2: [doc/next/003.md](../../doc/next/003.md) · P4: [doc/next/004.md](../../doc/next/004.md). Dev LLM: copy [.env.example](../../.env.example) → `.env` (gitignored).
## Official extension: `ext/web` (dynamic sites)
## Official extension: `ext/web` (Document / Endpoint / Resource)
Install: `marqdo ext add web` (ZH id: `网页`). Build native plugin first: `cargo build --release -p marqdo_plugin_web`.
Install: `marqdo ext add web` (ZH id: `网页`). Build native plugin: `scripts/build-web-plugin.sh` / `marqdo ext add web`.
Import **one language file** — never mix EN/ZH API names in the same `.mq.md`:
**Constitution (ADR 0007 — do not degrade):**
- EN: `import web:ext/web/web.mq.md`
1. **Document / Endpoint / Resource** — `.mq.md` with `type: web|endpoint` (ZH `类型: 网页|端点`) is the site surface; not a Flask-style class pile.
2. **Code is documentation** — the file is docs + route + handler.
3. **Table = data** — View schema only `|type|slot|value|attrs|style|`. No `compose_*` dialects, no `json.set` glue.
4. **Facade only** — `web.page` · `web.route` · `web.serve` · `web.render` · `web.inspect`. DB → `ext/data`; auth → `ext/security`; ws/http → `ext/net`.
5. **No author `ensure_plugin` / `host_*`**; import web = capability (plugin auto-load).
6. **Declared vs system routes**; `web.inspect` lists both. No silent user routes.
Import **one language** per file:
- EN: `import web:ext/web/web.mq.md` (+ `import data:ext/data/db.mq.md` as needed)
- ZH: `导入 网页:ext/web/网页.mq.md`
**Hard rules (web):**
```markdown
---
type: web
route: /
title: Hello
import web:ext/web/web.mq.md
---
- Authors use **GFM tables + `#` classes** — no `json.parse` / `json.set` glue, no hand-built part JSON.
- Table cells stay **literal strings**; path text like `` `posts`.`title` `` is resolved by web classes.
- **Quote path/MIME cells** that contain `/` (T5 cell expressions treat bare `/` as division): `"/static/logo.svg"`, `"image/png"`.
- **`ext/**` never calls `host_*`** — hot path is native `plugins/web` ABI (default **Go `libweb`**, build via `scripts/build-web-plugin.sh`).
- HTTPS: terminate at reverse proxy; set `cookie_secure=True` on auth (no in-process TLS).
# Hello
**Typical layout:**
```text
index.mq.md # entry + home page table + listen
pages/ # sub-pages
components/ # reusable |属性|值|样式| tables
styles/ # CSS modules
public/ # static + favicon.ico|png|svg
db/ # schema + open/init
data/ # sqlite runtime (gitignore)
Executable document as a page.
```
**Core objects (EN / ZH):** `# page`/`页面`, `# style`/`样式`, `# db`/`数据库`, `# form`/`表单`, `# app`/`应用`, `# auth`/`鉴权`, `# cache`/`缓存`, `# storage`/`存储`.
```markdown
---
type: endpoint
method: POST
path: /api/ping
response: json
---
**Typical flow:** `db.init` → `page.compose_*` → `app` + `static` / `listen`. Client: `marqdo wasm build -o static` + `web.client_embed source="/static/client.mq.md"` — **no author JS**; bridge may implement rich host effects ([browser-wasm-e.md](../../doc/roadmap/browser-wasm-e.md) · [browser-app](../../examples/browser-app/)).
*{"ok": true}*
```
**Shipped surface (W0–W7 + P3 + W8 + route D mount):** middleware + JSON API (`app.configure`); transactions, pagination, FTS search (`db.migrate`, `db.fts`, `db.search`); security (argon2, CSRF, SQLite sessions, login rate limit); SEO / RSS / Markdown (`page.meta`, `route_rss`, `lib/net.markdown_parse`); upload / download / gallery; sitemap / robots / error pages / redirects; RBAC (`app.gate`, user `role`); audit timestamps + FK in `db.init`; ETag on downloads; **W8** site icons (`app.icons` → `/favicon.ico`), head resource tables (`page.head`), image assembly (`make_images` / `page.images`); **表驱动引言** `page.compose_intro` / `页面.引言装配`(属性/值/样式表 → `.main-intro`,避免手写 HTML 字符串);**D** `client_embed` auto-mount + DOM effect helpers (`text_patch` / `dom_patch`).
Site entry may `# main` → `web.serve root="."`. Client WASM: `ext/web/client.mq.md` / `lib/browser` — no author JS.
Design: [ext-web.md](../../doc/design/ext-web.md) · assets: [web-assets-and-images.md](../../doc/design/web-assets-and-images.md) · capability matrix: [web-net-capabilities.md](../../doc/design/web-net-capabilities.md) · example: [examples/marqdo-blog/](../../examples/marqdo-blog/) · client: [examples/web-client-site/](../../examples/web-client-site/).
Design: [ext-web-artifact.md](../../doc/design/ext-web-artifact.md) · ADR [0007](../../doc/adr/0007-web-document-endpoint.md) · rule: `.cursor/rules/web-artifact-no-degrade.mdc` · example: [examples/marqdo-blog/](../../examples/marqdo-blog/).
Web patterns: [examples.md](examples.md) §13 · API index: [reference.md](reference.md).
Web patterns: [examples.md](examples.md) · API index: [reference.md](reference.md).
## Official extension: `ext/quantum` (circuits + Q7/Q8)
@@ -325,7 +334,7 @@ Examples: [linalg-svd](../../examples/linalg-svd/) · [linalg-least-squares](../
| Import `lib/text` then call bare `split` | `> text.split …` (qualified) |
| Import `lib/text` then call `拆分` | Match file language (`text.split`, not 文本) |
| `json.set` / `json.append` to build maps or lists | GFM tables; sparse `table.put`; named helpers (`browser.*`, `web.*`) |
| `json.set` to build page parts | GFM tables + `page.compose_*` / `web.page` methods |
| `json.set` / `compose_*` page parts | `type: web` Document + View table / `web.render` / `web.serve` |
| Import `json` for every browser handler | `import browser:lib/browser.mq.md` + tables ([marqdo-dev](../marqdo-dev/SKILL.md)) |
| Mix `web.page` and `网页.页面` in one file | One import language per `.mq.md` |
| Call `host_web_*` from `ext/web` | Use `# app` / `# db` methods; plugin ABI only |
+4 -2
View File
@@ -126,10 +126,12 @@ jobs:
## Marqdo ${{ github.ref_name }}
### 亮点 / Highlights
- **v1.3.0**:Engineering Knowledge Compiler——L0–L4 · `reuse`/`find`/`duplicate` · Agent preflight(REUSE→ADAPT→CREATE)
- **v1.3.0**: Engineering Knowledge Compiler — L0–L4 catalog, reuse/find/duplicate, Agent preflight (REUSE→ADAPT→CREATE)
- **v1.2.0**:Web Artifact(ADR 0007)——Document/Endpoint · `web.serve`/`render`/`inspect` · `ext/data|security|net` · 移除 `compose_*`/`configure`
- **v1.2.0**: Web Artifact (ADR 0007) — Document/Endpoint, web.serve/render/inspect, ext/data|security|net, compose_*/configure removed
- **v1.1.1**:AI 栈收口——llm `ask`/`stream`/`collect` · Metadata Binding · Skill Compilation + Adaptive Routing · MCP 客户端 + resume · 可选 `plugins/llm`
- **v1.1.1**: AI stack — llm ask/stream/collect, Metadata Binding, Skill Compilation + Adaptive Routing, MCP client + resume, optional plugins/llm
- **v1.1.0**:AI 原生闭环三件套——核心表面守卫 · 渐进式契约 + `marqdo check` · **MLSP for AI** · `run --json`
- **v1.1.0**: AI-native loop — core-surface guard, progressive contracts + `marqdo check`, **MLSP for AI**, `run --json`
- **v1.0.7**:WS/网络边界加固;`url_encode` `%20`
- **v1.0.3**:Ubuntu PPA + 扩展 CDN(ext.marqdo.com)
- **Linux 包必出**:`*-native-x86_64-unknown-linux-gnu.zip` + Linux CLI bundle
+87
View File
@@ -8,6 +8,93 @@
### Changed
## Ext pack v1.3.1 — 2026-10-08
### Highlights
**OIDC 路径型 `redirect_uri`(CLI 仍为 v1.3.0)**:`redirect_uri=/oidc/callback` 按请求 Host / `X-Forwarded-*` 拼绝对回调,写入会话供 token 交换;可选 `redirect_origins` allowlist。apex / www 多域名无需写死单一绝对 URI。
```bash
marqdo ext add web # CDN latest → 1.3.1
# 或:MARQDO_EXT_VERSION=1.3.1 marqdo ext add web
```
### Added
- **路径型 redirect_uri**:绝对 URL 行为不变;路径模式解析 public origin,session 键 `oidc_redirect_uri`。
- **`redirect_origins` / `回调来源`**:逗号分隔 origin allowlist(路径模式)。
- **ABI**:`web_app_oidc` 透传 `redirect_origins`;`ext/security/oidc.attach` 文档同步。
### Changed
- 扩展包 SemVer **1.3.1**(`ext/VERSION`);与 CLI **v1.3.0** 可独立安装。
## v1.3.0 — 2026-09-28
### Highlights
**Engineering Knowledge Compiler(EKC)**:把仓库投影为 AI 可查询的工程知识系统——L0–L4 Catalog、Capability / Decision / Constraint / Failure、知识图、Reuse Preflight(REUSE→ADAPT→CREATE)与 Context Pack。本版 CLI 与扩展包 **同步为 1.3.0**。
- **杀手命令**:`marqdo reuse "task"` / `find` / `duplicate` / `impact` / `conflicts` / `stale` / `verify` / `knowledge`
- **Agent**:`agent.preflight` · `agent_eng_*`;`plan … eng_preflight=True` 在造轮子前强制 EFI
- **投影**:`.marqdo/catalog|knowledge|graph|agent/` + `engineering.yaml`;`catalog` 默认挂钩 EKC
- **示例 / 金样**:`examples/engineering-knowledge/` · `tests/engineering/` · `tests/ext/eng-preflight.mq.md`
```bash
sudo add-apt-repository ppa:cflmy/marqdo && sudo apt update && sudo apt install marqdo
git checkout v1.3.0 && cargo build --release
marqdo knowledge . -o .marqdo
marqdo reuse "resolve configuration"
marqdo ext add agent
```
**核心表面 Δ(Core surface Δ)**:**无增删**——19 个核心构造不变。本版为 **Runtime / Tooling**(知识编译器 + CLI + agent 插件),不触碰语言标记。
### Added
- **`src/knowledge/` EKC**:L0–L4 IR、指纹、能力推导、图边、compile、find/reuse/preflight、impact/lifecycle/duplicate、metrics、policy、learn。
- **CLI**:`knowledge` · `find` · `reuse` · `impact` · `conflicts` · `stale` · `duplicate` · `verify`;`catalog`/`sync` 默认编译 EKC(`--no-knowledge` 可关)。
- **Agent**:`agent_eng_preflight` / `agent_eng_reuse` / `agent_eng_record`;`ext/ai` `## preflight` + `plan eng_preflight`。
- **规范**:`doc/design/engineering-knowledge.md`;提案叙事 `doc/next/008.md`(已实现标注)。
### Fixed
### Changed
- **OKF type 注册表**扩展 Capability/Decision/Constraint/Failure/Context Pack/Policy 等([okf.md](doc/design/okf.md))。
- **核心表面 Δ**:**无增删**。
## v1.2.0 — 2026-09-28
### Highlights
**Web Artifact 重写(ADR 0007)**:Document / Endpoint / Resource 一等公民;`type: web|endpoint`(ZH `类型: 网页|端点`);facade `web.page|route|serve|render|inspect|use`;DB/auth/ws 拆到 `ext/data` · `ext/security` · `ext/net`。本版 CLI 与扩展包 **同步为 1.2.0**。
- **Document**:文件级 Artifact + `data_source` / `data_order` / `data_where`;正文 Markdown → HTML;列表卡片 / 详情文章由 serve 时加载。
- **Endpoint**:`# main` 处理器;请求经 `run_artifact` Binding 注入;`response` 序列化 JSON/text/html。
- **破坏性**:删除作者面 `compose_*` / `app.configure` / 作者 `ensure_plugin`;旧 web gold 移除,新金样 `ext_web_*`。
- **示例**:`examples/marqdo-blog/` Artifact 站(pages/api/db + smoke);`web-client-site` serve 扫描。
```bash
sudo add-apt-repository ppa:cflmy/marqdo && sudo apt update && sudo apt install marqdo
git checkout v1.2.0 && cargo build --release
bash scripts/build-web-plugin.sh && marqdo ext add web
marqdo run examples/marqdo-blog/serve.mq.md
```
**核心表面 Δ(Core surface Δ)**:**无增删**——复用 Metadata `type`;装配语义在 ext + plugin。语言标记不变。
### Added
- **Web Artifact 模型(ADR 0007)**:`type: web|endpoint`(ZH `类型: 网页|端点`);核心 `src/artifact/`;`sys.artifact_kind`;无 `# main` 时 Result 摘要。
- **`ext/web` facade**:`page` · `route` · `serve` · `render` · `inspect` · `use`;子模块 page/route/dom/client/component。
- **`ext/data` · `ext/security` · `ext/net`**:从 monolithic web 拆出 Resource。
- **统一 View** `|type|slot|value|attrs|style|` + ABI `web_render_nodes` / `web_serve_root` / `web_inspect` / `web_app_use`。
- **规则**:`.cursor/rules/web-artifact-no-degrade.mdc`;设计 `doc/design/ext-web-artifact.md`。
### Fixed
- Document `data_source` / `data_where` now drive list cards and detail articles under `web.serve` (ADR 0007 flat metadata).
### Changed
- **破坏性**:删除作者面 `compose_*` / `app.configure` / 作者 `ensure_plugin`。旧 web gold 已移除;新金样 `web-artifact-*` / `ext_web_*`。
- **核心表面 Δ**:**无增删**(复用 Metadata `type`)。
- 修订 `ext-web.md` C1;`doc/next/007.md` superseded。
## v1.1.1 — 2026-09-24
### Highlights
Generated
+2 -2
View File
@@ -572,7 +572,7 @@ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "marqdo"
version = "1.1.1"
version = "1.3.0"
dependencies = [
"anyhow",
"assert_cmd",
@@ -603,7 +603,7 @@ dependencies = [
[[package]]
name = "marqdo-wasm"
version = "1.1.1"
version = "1.3.0"
dependencies = [
"marqdo",
"serde_json",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "marqdo"
version = "1.1.1"
version = "1.3.0"
edition = "2021"
license = "Apache-2.0"
description = "Marqdo — Markdown markers as an interpreted programming language"
+20 -10
View File
@@ -6,7 +6,7 @@ Marqdo 把 Markdown **标记当作编程语法**:同一个 `.mq.md` 文件,
欢迎访问 [marqdo 官方网站](https://www.marqdo.com/) 阅读更多特性与可执行文档。
语法宪法:[doc/design/markdown-mapping.md](doc/design/markdown-mapping.md) · 对象:[objects.md](doc/design/objects.md) · 调用实参:[call-arguments.md](doc/design/call-arguments.md) · 用户站:[user-site.md](doc/design/user-site.md) · 浏览:[view.md](doc/design/view.md) · 调试:[view-debug.md](doc/design/view-debug.md) · OKF 清单:[catalog-cli.md](doc/design/catalog-cli.md) · VS Code 扩展:[vscode-extension.md](doc/design/vscode-extension.md)(分支 **`vscode-extension`**;提交约定 [vscode-extension-commit.md](doc/design/vscode-extension-commit.md)) · **AI Skill**:[skills/marqdo/](skills/marqdo/)(说明 [ai-skill.md](doc/design/ai-skill.md)) · 官方扩展:[ext-llm.md](doc/design/ext-llm.md)(`ext/llm`)· [ext-agent.md](doc/design/ext-agent.md)(智能体开发框架)· [ext-cli.md](doc/design/ext-cli.md)(`marqdo ext list/add/remove`) · 原生插件 ABI:[ext-abi.md](doc/design/ext-abi.md)(`lib/plugin`) · 金样例:[tests/](tests/) · 用户文档:[public/](public/) · 变更:[CHANGELOG.md](CHANGELOG.md)
语法宪法:[doc/design/markdown-mapping.md](doc/design/markdown-mapping.md) · 对象:[objects.md](doc/design/objects.md) · 调用实参:[call-arguments.md](doc/design/call-arguments.md) · 用户站:[user-site.md](doc/design/user-site.md) · 浏览:[view.md](doc/design/view.md) · 调试:[view-debug.md](doc/design/view-debug.md) · OKF 清单:[catalog-cli.md](doc/design/catalog-cli.md) · Engineering Knowledge:[engineering-knowledge.md](doc/design/engineering-knowledge.md) · VS Code 扩展:[vscode-extension.md](doc/design/vscode-extension.md)(分支 **`vscode-extension`**;提交约定 [vscode-extension-commit.md](doc/design/vscode-extension-commit.md)) · **AI Skill**:[skills/marqdo/](skills/marqdo/)(说明 [ai-skill.md](doc/design/ai-skill.md)) · 官方扩展:[ext-llm.md](doc/design/ext-llm.md)(`ext/llm`)· [ext-agent.md](doc/design/ext-agent.md)(智能体开发框架)· [ext-cli.md](doc/design/ext-cli.md)(`marqdo ext list/add/remove`) · 原生插件 ABI:[ext-abi.md](doc/design/ext-abi.md)(`lib/plugin`) · 金样例:[tests/](tests/) · 用户文档:[public/](public/) · 变更:[CHANGELOG.md](CHANGELOG.md)
---
@@ -123,6 +123,13 @@ Marqdo 把 Markdown **标记当作编程语法**:同一个 `.mq.md` 文件,
```bash
marqdo catalog [PATH] -o OUT_DIR
marqdo sync [PATH] -o OUT_DIR # catalog 的别名
# Engineering Knowledge Compiler(L0–L4 能力目录 / 复用门闩)
marqdo knowledge [PATH] -o OUT_DIR
marqdo find "authentication"
marqdo reuse "implement oauth login"
marqdo duplicate [--fail]
marqdo verify
```
| 参数 | 默认 | 说明 |
@@ -130,7 +137,7 @@ marqdo sync [PATH] -o OUT_DIR # catalog 的别名
| `PATH` | `.` | 工程根或含 `.mq.md` 的目录 |
| `-o` / `--out` | `.marqdo` | 输出目录 |
产物示例:`catalog.yaml`、`index.md`、`modules/*.md`(`type: Marqdo Module` 等)。设计见 [generated-yaml-manifest.md](doc/design/generated-yaml-manifest.md) · 调研 [okf-and-marqdo.md](doc/research/okf-and-marqdo.md)。
`catalog` 成功后默认再跑 EKC(`--no-knowledge` 可关)。产物示例:`catalog.yaml`、`engineering.yaml`、`catalog/capabilities/`、`graph/graph.json`。设计见 [engineering-knowledge.md](doc/design/engineering-knowledge.md) · [catalog-cli.md](doc/design/catalog-cli.md) · 调研 [okf-and-marqdo.md](doc/research/okf-and-marqdo.md)。
智能体框架横向调研(LangGraph / CrewAI / 厂商 SDK 等 vs Marqdo 文档驱动优势):[agent-frameworks-and-marqdo.md](doc/research/agent-frameworks-and-marqdo.md) · 优化路线 [ext-agent-optimize.md](doc/roadmap/ext-agent-optimize.md)。
@@ -142,22 +149,25 @@ marqdo catalog public -o .marqdo
---
## 现状(v1.1.1)
## 现状(v1.3.0)
- **Engineering Knowledge(v1.3.0)**:EKC L0–L4 · `marqdo knowledge|find|reuse|duplicate|verify|…` · Agent `preflight` / REUSE→ADAPT→CREATE · [engineering-knowledge.md](doc/design/engineering-knowledge.md)
- **Web Artifact(v1.2.0 · ADR 0007)**:Document/Endpoint 一等公民(`type: web|endpoint`)· facade `page`/`route`/`serve`/`render`/`inspect`/`use` · Resource 拆到 `ext/data`/`security`/`net` · 统一 View 表 · 破坏性移除 `compose_*`/`configure`
- **AI 栈(v1.1.1)**:`ext/ai/llm` 语义 `ask`/`stream`/`collect` · Artifact Metadata Binding(`${env|arg|sys|secret}`)· Agent Skill Compilation + Adaptive Routing · MCP stdio 客户端 + resume 检查点 · 可选 `plugins/llm` Transport ABI
- **AI 原生闭环(v1.1.0)**:核心表面守卫(19 构造,单边改动 CI 必红)· **渐进式契约**(文档内嵌 `参数`/`返回`/`字段` 表 + 四边界校验 + `marqdo check` 防漂移)· **MLSP for AI**(`marqdo mlsp`:`locate`/`syntax`/`validate`/`repair_targets`/`repair_apply`/`schema`,AI 按需查语法而非背诵)· **结构化诊断**(`run --json`,错误即数据);验证见 [perf-validation-report-2026-09-23.md](doc/roadmap/perf-validation-report-2026-09-23.md)
- 映射与解释器:**v0.3 语法宪法**(`**` 代码 / `*` 返回 / 叙述声明);Phase I 树遍历 + 字节码后端;金样例在 `tests/` 与 `tests/markup-v03/`
- **对象**:`#` = 类型/构造,`##`+ = 函数/方法;见 [objects.md](doc/design/objects.md)
- **`marqdo view`**:文档浏览器(Structure + 函数大纲/搜索 + Execution + Variables 浮窗)
- **`marqdo debug`**:独立调试页(断点 / 单步 / locals;默认端口 7430;页面 favicon / 品牌使用官方 Logo)
- **`marqdo catalog` / `sync`**:OKF 风格 YAML + 模块概念页
- **`marqdo catalog` / `sync`**:OKF 风格 YAML + 模块概念页(默认挂钩 EKC)
- **`marqdo knowledge` / `find` / `reuse` / `impact` / `conflicts` / `stale` / `duplicate` / `verify`**:Engineering Knowledge Compiler([engineering-knowledge.md](doc/design/engineering-knowledge.md))
- **`marqdo version --check`**:与 GitHub 最新 release 对比
- 标准库:**内置于二进制**(v0.1.2+);磁盘 `lib/` 或 `MARQDO_LIB` 可覆盖。模块含文本、表、**浏览器效应**、文件、系统、时间、JSON、网络、数学、外联、插件、**自写回**、**子任务**;**中层 Mid M1–M6** + **Mid2 M7–M10**(datetime/url/toml/html/fs++/stats/log.fields 等,见 [stdlib-mid.md](doc/design/stdlib-mid.md) · [stdlib-mid2.md](doc/design/stdlib-mid2.md))
- **官方扩展库 `ext/`**(**非** stdlib,本版收口):
- **`web`**:W0–W7 + P3 + **W8**;**定制 C0–C4**;**宿主 H1–H2**(`proxy`/`invoke`,见 [ext-hosting.md](doc/roadmap/ext-hosting.md));**表驱动引言**(`compose_intro` / `引言装配`);原生层默认 **Go `libweb`**([ADR 0004](doc/adr/0004-web-plugin-go.md),`scripts/build-web-plugin.sh`);示例 [web-site](examples/web-site/) · [web-site-zh](examples/web-site-zh/) · [marqdo-blog](examples/marqdo-blog/) · [anlian-mq](examples/anlian-mq/);生产路径见 [web-asgi-servers-and-marqdo.md](doc/design/web-asgi-servers-and-marqdo.md)
- **`web`**:**Artifact 模型(ADR [0007](doc/adr/0007-web-document-endpoint.md))** — `type: web|endpoint` Document/Endpoint;facade `page`/`route`/`serve`/`render`/`inspect`/`use`;Resource 在 `ext/data` · `ext/security` · `ext/net`;设计 [ext-web-artifact.md](doc/design/ext-web-artifact.md);原生 **Go `libweb`**([ADR 0004](doc/adr/0004-web-plugin-go.md));示例 [marqdo-blog](examples/marqdo-blog/) · [web-client-site](examples/web-client-site/) · [anlian-mq](examples/anlian-mq/)
- **`quantum`**:Q0–Q7 + Q8a/b 主题 SVG;示例 [quantum-entanglement](examples/quantum-entanglement/)
- **`linalg`**:L0–L6 + 公式文档面(中缀 / `declare` / 展示化简);示例 [linalg-transpose](examples/linalg-transpose/) · [linalg-svd](examples/linalg-svd/) · [linalg-least-squares](examples/linalg-least-squares/)
- **`agent`**:A1–A4 + **MCP Server/Client stdio** + Skill Compilation + Adaptive Routing + resume;宿主缺口见 [ext-hosting.md](doc/roadmap/ext-hosting.md) · [agent-framework-gaps-after-a4.md](doc/research/agent-framework-gaps-after-a4.md)
- **`agent`**:A1–A4 + **MCP Server/Client stdio** + Skill Compilation + Adaptive Routing + resume + **Engineering preflight**(`agent.preflight` / `eng_preflight`);宿主缺口见 [ext-hosting.md](doc/roadmap/ext-hosting.md) · [agent-framework-gaps-after-a4.md](doc/research/agent-framework-gaps-after-a4.md) · [engineering-knowledge.md](doc/design/engineering-knowledge.md)
- **`llm`**:语义 `ask`/`stream`/`collect`(OpenAI/Ollama;可选原生 `plugins/llm`)
- 安装:`marqdo ext list` / `add …` / `remove`([ext-cli.md](doc/design/ext-cli.md);默认 `~/.marqdo/ext`)。原生插件先 `cargo build -p marqdo_plugin_*` 再 `ext add`
- **原生插件 ABI**:[`include/marqdo_abi.h`](include/marqdo_abi.h) · [ext-abi.md](doc/design/ext-abi.md);`plugins/{demo,agent,web,quantum,linalg,llm}`(`web` 为 Go;其余为 Rust)
@@ -166,7 +176,7 @@ marqdo catalog public -o .marqdo
- **浏览器 Marqdo(WASM)**:`marqdo wasm build` + 官方 bridge 自启(作者零业务 JS;桥内可含列表/路由/storage/ws/文件/Canvas/音频/Observer/拖放);`lib/browser` + GFM;`web.client_embed`;示例 [browser-hello](examples/browser-hello/) · [browser-app](examples/browser-app/) · [browser-media](examples/browser-media/) · [web-client-site](examples/web-client-site/)([ADR 0002](doc/adr/0002-browser-marqdo-wasm.md) · [D](doc/roadmap/browser-wasm-d.md) · [E](doc/roadmap/browser-wasm-e.md) · [F](doc/roadmap/browser-wasm-f.md))
- 选型:[ADR 0001 — Rust](doc/adr/0001-implementation-language.md) · [ADR 0002 — 浏览器 WASM](doc/adr/0002-browser-marqdo-wasm.md)(C0–C5 完结,见 [roadmap/browser-wasm.md](doc/roadmap/browser-wasm.md))· [ADR 0003 — 异步效应](doc/adr/0003-browser-async-effects.md)
### 如何使用最新 Marqdo(v1.1.1)
### 如何使用最新 Marqdo(v1.3.0)
**安装解释器(任选其一)**
@@ -181,7 +191,7 @@ marqdo catalog public -o .marqdo
```bash
# 1) 源码安装(跟 tag 或 main)
git clone https://github.com/cflmy/marqdo.git && cd marqdo
git checkout v1.1.1 # 或留在 main
git checkout v1.3.0 # 或留在 main
cargo build --release
export PATH="$PWD/target/release:$PATH"
marqdo version
@@ -211,7 +221,7 @@ marqdo ext add llm # 或:大模型(.mq.md + 可选 native llm)
# 4) 动态站示例(扩展装好后)
marqdo run examples/web-site-zh/index.mq.md # 表驱动引言 Demo
marqdo run examples/marqdo-blog/index.mq.md
marqdo run examples/marqdo-blog/serve.mq.md
marqdo run examples/anlian-mq/index.mq.md # W-G14 验收站
# 浏览器打开终端打印的 listen 地址;/favicon.ico 与 logo 装配见 W8
@@ -220,7 +230,7 @@ marqdo wasm build
# → dist/wasm/ … 见 examples/browser-hello/
# 6) Releases:Windows exe/zip/vsix;Linux CLI zip + native `.so` zip
# https://github.com/cflmy/marqdo/releases/tag/v1.1.1
# https://github.com/cflmy/marqdo/releases/tag/v1.3.0
```
开发期也可用 `cargo run -- …` 代替已安装的 `marqdo`:
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "marqdo-wasm"
version = "1.1.1"
version = "1.3.0"
edition = "2021"
license = "Apache-2.0"
description = "Marqdo interpreter as WebAssembly (browser host bridge)"
+16
View File
@@ -1,3 +1,19 @@
marqdo (1.3.0-1ppa1) noble; urgency=medium
* Marqdo 1.3.0: Engineering Knowledge Compiler (EKC) — L0–L4 catalog,
marqdo knowledge/find/reuse/duplicate/verify, Agent preflight
(REUSE→ADAPT→CREATE), graph + context packs.
-- cflmy <pingan@cflmy.cn> Mon, 28 Sep 2026 16:50:00 +0800
marqdo (1.2.0-1ppa1) noble; urgency=medium
* Marqdo 1.2.0: Web Artifact rewrite (ADR 0007) — Document/Endpoint/Resource,
type:web|endpoint, web.serve/render/inspect, ext/data|security|net split,
compose_* author surface removed; marqdo-blog Artifact example.
-- cflmy <pingan@cflmy.cn> Mon, 28 Sep 2026 11:39:04 +0800
marqdo (1.1.1-1ppa1) noble; urgency=medium
* Marqdo 1.1.1: AI stack closure — llm ask/stream/collect, Metadata Binding,
+1
View File
@@ -25,6 +25,7 @@
| [design/return-hr-and-code-surface.md](design/return-hr-and-code-surface.md) | 语句/返回/输出决议 |
| [design/generated-yaml-manifest.md](design/generated-yaml-manifest.md) | 生成式 OKF 清单(早期方向) |
| [design/okf.md](design/okf.md) | **Marqdo OKF 实现规范**(catalog v0 + 任务知识包) |
| [design/engineering-knowledge.md](design/engineering-knowledge.md) | **Engineering Knowledge Compiler(EKC)**:L0–L4、reuse、EFI |
| [design/catalog-cli.md](design/catalog-cli.md) | **`marqdo catalog` / `sync` OKF 清单命令** |
| [design/tech-stack.md](design/tech-stack.md) | 实现栈对比 |
| [adr/0001-implementation-language.md](adr/0001-implementation-language.md) | **Accepted:Rust 参考解释器;不用 Flex/Bison** |
+54
View File
@@ -0,0 +1,54 @@
# ADR 0007:Web Document / Endpoint / Resource(破坏性 · 代码即文档)
| | |
|---|---|
| Status | **Accepted** |
| Date | 2026-09-28 |
| Related | [ext-web-artifact.md](../design/ext-web-artifact.md) · [ext-web.md](../design/ext-web.md) · [binding.md](../design/binding.md) · ADR [0006](0006-artifact-metadata-binding.md) · [layers.md](../design/layers.md) · [doc/next/007.md](../next/007.md) |
## Context
- [`ext/web/web.mq.md`](../../ext/web/web.mq.md) 已膨胀为小型 Web Framework(Page God Object、`configure` 垃圾抽屉、碎片 Table DSL、隐式路由),与 Marqdo「代码即文档」冲突。
- [doc/next/007.md](../next/007.md) 提出 Document / Endpoint / Resource 三原语与拆分 `ext/web` · `ext/net` · `ext/data` · `ext/security`。
- 现行 [ext-web.md](../design/ext-web.md) **C1** 禁止为 web 改核心;Document/Endpoint 装载识别需要 Language/Runtime 参与(复用 Artifact Metadata `type`,**不**新增 CORE_CONSTRUCTS 标记)。
- Binding(ADR 0006)已提供文件级 Metadata;`type: prompt` 先例在扩展侧消费 metadata。
## Decision
1. **采纳** Document / Endpoint / Resource 为 Web 模型一等公民;正式规范见 [ext-web-artifact.md](../design/ext-web-artifact.md)。
2. **修订 ext-web C1**:允许核心在装载期识别 `type: web|endpoint`(及中文 `类型: 网页|端点`)并暴露 artifact 分类;**装配 / HTTP / DB 语义仍在 ext + plugin**。单元格求值、import 句法、CORE_CONSTRUCTS 计数不变。
3. **文件级 Artifact only**:不在 `##` 函数体嵌套 `---` frontmatter(与 Binding Phase 1 / 解析器一致)。
4. **破坏性重写作者面**:删除 `compose_*` 碎片 DSL、god `# page` 变异堆、`app.configure` 多参数抽屉、作者 `ensure_plugin`;examples 与 `tests/ext/web*` 同步重写。
5. **目录定稿**(007 §三十五 + §四十):
- `ext/web/` — page · route · component · dom · client · facade
- `ext/net/` — http · websocket · url
- `ext/data/` — db · cache · storage
- `ext/security/` — auth · oidc · rbac
6. **统一 View 表**:`| type | slot | value | attrs | style |`;Table = Data,禁止每函数私有方言。
7. **declared vs system routes**:用户 route 须声明;`/_part` · `/_form` 等为 system,经 `web.inspect` 可见。
8. **import = capability**:`import web:…` 由 facade 自动加载 native plugin;作者程序不写 `ensure_plugin` / `host_*`。
9. **Metadata v1 锁定(原 007 未决项)**:
- 发现:`web.serve root=` 扫描 `root/**/*.mq.md`,仅注册 `type`/`类型` ∈ {web,网页,endpoint,端点} 的文件;无 type 的库文件忽略。
- Data:仅 flat 键 `data_source` · `data_order` · `data_where`(text;where 为 `col=val` 逗号分隔)。嵌套 YAML `data:` 推迟。
- Result:`{ok: bool, value: any, error: text|None}`。
- Realtime 模块名:`ext/net/websocket.mq.md`(ZH:`实时.mq.md`)。
- ZH metadata:`类型`←type,`路由`←route,`方法`←method,`路径`←path,`请求`←request,`响应`←response,`鉴权`←auth,`数据源`←data_source,`排序`←data_order,`条件`←data_where。
## Consequences
### Positive
- `.mq.md` 同时是文档、路由、实现与鉴权声明。
- 与 LLM/Agent Artifact Metadata 统一心智。
- 职责边界清晰,可独立演化 data/security/net。
### Negative / Risks
- 破坏性:全部现有 web 作者程序与 smokes 必须改写。
- Endpoint 热路径仍经 plugin → `CallLibPath` / 文件装载回调;须保持 ABI 稳定演进。
- flat data 键表达力弱于嵌套 YAML;后续另 ADR。
### Neutral
- 不新增 Markdown 核心标记;`${…}` 仍仅 Metadata。
- Go `plugins/web` 可分波删除死 compose ABI;作者面先断。
+5
View File
@@ -23,15 +23,20 @@ YAML / OKF 风格清单是**从 `.mq.md` 派生的生成物**,不是手写配
marqdo catalog [PATH] -o OUT_DIR
# 别名(与早期草案一致)
marqdo sync [PATH] -o OUT_DIR
# 跳过 Engineering Knowledge 投影(仅旧式 catalog)
marqdo catalog [PATH] -o OUT_DIR --no-knowledge
```
| 参数 | 默认 | 说明 |
|------|------|------|
| `PATH` | `.` | 工程根或含 `.mq.md` 的目录(递归扫描) |
| `-o` / `--out` | `.marqdo` | 生成目录(写入前创建;**覆盖**同名生成文件;扫描时跳过该目录) |
| `--no-knowledge` | off | 不跑 EKC;默认在 catalog 成功后编译 L0–L4 / graph |
退出码:成功 0;无可扫描文件或 IO/解析失败非 0(单文件解析失败时写入 `diagnostics` 并继续其余文件,最终若有失败则非 0)。
Engineering Knowledge 命令见 [engineering-knowledge.md](engineering-knowledge.md)(`find` / `reuse` / `duplicate` / …)。
---
## 3. 生成物
+97
View File
@@ -0,0 +1,97 @@
# Engineering Knowledge Compiler(EKC)
| | |
|---|---|
| 状态 | **Accepted(v1 已落地)** |
| 日期 | 2026-09-28 |
| 提案 | [next/008.md](../next/008.md) |
| 相关 | [okf.md](okf.md) · [catalog-cli.md](catalog-cli.md) · [ext-agent.md](ext-agent.md) |
## 1. 定位
Marqdo 把仓库投影为 **可查询、可验证、可复用** 的工程知识系统。Source of Truth 仍是 `*.mq.md`;`.marqdo/**` 是 GENERATED 投影。
```text
.mq.md ──► EKC ──► Executable + Catalog L0–L4 + Graph + Context Pack
│
▼
Agent preflight (REUSE → ADAPT → CREATE)
```
**Engineering First Information (EFI)**:生成代码前先获得已有能力、约束、决策、失败经验与推荐复用。
## 2. 五层 Catalog
| 层 | 内容 | 落盘 |
|----|------|------|
| L0 Repository | 语言/规模 | `.marqdo/catalog/repository.mq.md` |
| L1 Module | imports/exports/responsibility | `.marqdo/catalog/modules/` |
| L2 Symbol | signature/calls/fingerprint | `.marqdo/catalog/symbols/` |
| L3 Capability | 工程能做什么 | `.marqdo/catalog/capabilities/` |
| L4 Knowledge | Decision/Constraint/Failure/… | `.marqdo/knowledge/{decisions,constraints,…}/` |
既有 `marqdo catalog` 根级 `catalog.yaml` / `index.md` / `modules/` / `concepts/` **保留**;`catalog` 成功后默认再跑 EKC(`--no-knowledge` 可关)。
## 3. 知识类型(OKF `type`)
| type | 含义 |
|------|------|
| `Capability` / `Function` | 能力与符号投影 |
| `Decision` | 为什么这样做 |
| `Constraint` | 必须/禁止 |
| `Failure` / `AntiPattern` | 负面知识 |
| `Pattern` / `Migration` / `Fact` | 模式、迁移、事实 |
| `Marqdo Context Pack` | Agent 上下文制品 |
| `Marqdo Policy` | 复用决议编译结果 |
| `Marqdo Engineering Knowledge` | 总览 / `engineering.yaml` |
未知 type 优雅降级(可读,不专属逻辑)。
## 4. 图边
`implements` · `uses` · `depends_on` · `tested_by` · `documented_by` · `constrained_by` · `decided_by` · `supersedes` · `conflicts_with` · `deprecated_by` · `derived_from` · `failed_by` · `related_to`
落盘:`.marqdo/graph/graph.json` · `edges.json` · `index.json`。
## 5. CLI
```bash
marqdo knowledge [PATH] -o .marqdo # compile(默认)
marqdo knowledge preflight "task"
marqdo knowledge record REUSE
marqdo knowledge learn --task ... --failure ...
marqdo find "authentication"
marqdo reuse "implement oauth login"
marqdo impact path/to/file.mq.md
marqdo conflicts | stale | duplicate [--fail] | verify
```
统一 `--json`。`duplicate` 默认阈值 `0.75`;`--fail` 作 CI Gate。
## 6. Reuse 协议
三态:**REUSE → ADAPT → CREATE**(硬顺序)。Reuse budget:`min_candidates` / `allow_create_after`。仅 `create_allowed: true` 时允许盲生成。
## 7. Context Pack
`.marqdo/agent/contexts/task-<id>.mq.md`:Task / Existing Capabilities / APIs / Constraints / Decisions / Failures / Recommended / Forbidden。
## 8. Agent
- Plugin:`agent_eng_preflight` / `agent_eng_reuse` / `agent_eng_record`
- Facade:`agent.preflight` · `agent.eng_reuse` · `agent.eng_record`
- `agent.plan … eng_preflight=True` 在 agent-kb 之前跑 EFI;缺知识图时 `missing_knowledge` 放行(兼容纯 agent-kb 测试)
## 9. 指标
`.marqdo/agent/episodes/reuse_metrics.json`:`reuse_ratio` / `adapt_ratio` / `novel_ratio` / `duplication_rate`。
## 10. 原则(五条)
1. Discover before Create
2. Reuse before Adapt
3. Adapt before Duplicate
4. Verify before Promote
5. Code changes update knowledge
**不做**:以向量库为知识模型;embedding 仅可选检索实现,默认关闭。Hybrid = lexical + symbol + capability + graph。
+4 -3
View File
@@ -4,7 +4,7 @@
|---|---|
| Status | **Accepted · redesign**(取代「thin TOOL: loop」方案) |
| Date | 2026-08-07 |
| Related | [ext-llm.md](ext-llm.md) · [ext-cli.md](ext-cli.md) · [ext-abi.md](ext-abi.md) · [**ext-agent-plan.md**](ext-agent-plan.md)(**多步锁定设计**) · [**ext-agent-parent.md**](ext-agent-parent.md)(**父 Plan-and-Move**) · [**okf.md**](okf.md)(**OKF / 任务知识包**) · [**调研:主流框架与 Marqdo 优势**](../research/agent-frameworks-and-marqdo.md) · [**缺口:A0–A4 之后**](../research/agent-framework-gaps-after-a4.md) · [module-namespace.md](module-namespace.md) · [objects.md](objects.md) · [stdlib-writeback.md](stdlib-writeback.md) · [stdlib-subtask.md](stdlib-subtask.md) · [markdown-mapping.md](markdown-mapping.md) |
| Related | [ext-llm.md](ext-llm.md) · [ext-cli.md](ext-cli.md) · [ext-abi.md](ext-abi.md) · [**ext-agent-plan.md**](ext-agent-plan.md)(**多步锁定设计**) · [**ext-agent-parent.md**](ext-agent-parent.md)(**父 Plan-and-Move**) · [**okf.md**](okf.md)(**OKF / 任务知识包**) · [**engineering-knowledge.md**](engineering-knowledge.md)(**EKC / EFI preflight**) · [**调研:主流框架与 Marqdo 优势**](../research/agent-frameworks-and-marqdo.md) · [**缺口:A0–A4 之后**](../research/agent-framework-gaps-after-a4.md) · [module-namespace.md](module-namespace.md) · [objects.md](objects.md) · [stdlib-writeback.md](stdlib-writeback.md) · [stdlib-subtask.md](stdlib-subtask.md) · [markdown-mapping.md](markdown-mapping.md) |
## 0. 为什么要改掉旧方案
@@ -167,8 +167,9 @@ ext/
**分层纪律(硬规则,勿破)**:
1. `ext/**/*.mq.md` **禁止**出现任何 `host_*` / `host_` 调用。
2. Agent 专属能力(含 OKF agent-kb:`agent_goal_sig` / `agent_kb_lookup` / `agent_kb_promote` / …)只进 **`plugins/agent` 注册名**,由 `ext/ai/agent` 在 `plugin.load` 后调用。
3. **禁止**向 `src/host/`(`HostFn` 表)添加 agent / OKF 领域原语,以免核心包体膨胀。通用 L0.5 仅保留 fs/json/subtask 等标准库底层。
2. Agent 专属能力(含 OKF agent-kb:`agent_goal_sig` / `agent_kb_lookup` / `agent_kb_promote` / …;以及 EKC:`agent_eng_preflight` / `agent_eng_reuse` / `agent_eng_record`)只进 **`plugins/agent` 注册名**,由 `ext/ai/agent` 在 `plugin.load` 后调用。
3. **禁止**向 `src/host/`(`HostFn` 表)添加 agent / OKF / EKC 领域原语,以免核心包体膨胀。通用 L0.5 仅保留 fs/json/subtask 等标准库底层。
4. **Engineering preflight**:`agent.preflight` / `plan … eng_preflight=True` 在 agent-kb 之前解析 REUSE/ADAPT/CREATE(见 [engineering-knowledge.md](engineering-knowledge.md))。缺 `.marqdo` 知识图时返回 `missing_knowledge` 并放行,以兼容纯任务包测试。
4. 进程退出用 `sys.exit` / `系统.退出`,不用裸 `exit`。
官方 `ext/ai/` 用 `lib/*` 包装或插件注册名;见 [module-namespace.md](module-namespace.md) §8 · [ext-abi.md](ext-abi.md)。
+197
View File
@@ -0,0 +1,197 @@
# 官方扩展:Web Artifact(Document / Endpoint / Resource)
| | |
|---|---|
| 状态 | **Accepted · 破坏性(取代 page/app God Object 作者面)** |
| 日期 | 2026-09-28 |
| ADR | [0007-web-document-endpoint.md](../adr/0007-web-document-endpoint.md) |
| 讨论稿 | [doc/next/007.md](../next/007.md)(superseded) |
| 安装 | `marqdo ext add web`(`web` / `网页`) |
| 相关 | [binding.md](binding.md) · [ext-web.md](ext-web.md) · [stdlib-i18n.md](stdlib-i18n.md) · [layers.md](layers.md) |
---
## 0. 一句话
> **Document describes the page. Table describes the data. Endpoint describes the interaction. Metadata describes the runtime. Marqdo executes all of them.**
---
## 1. 三原语
| Artifact | Metadata `type` / `类型` | 含义 |
|----------|--------------------------|------|
| **Document** | `web` / `网页` | Markdown 正文 + View → HTML;`route`/`路由` 注册 declared HTTP |
| **Endpoint** | `endpoint` / `端点` | 可执行正文为 handler;`path`+`method` 注册 API |
| **Resource** | (绑定/导入) | `ext/data` · `ext/security` · 未来 LLM 等,经 metadata / import 声明 |
核心装载识别 `type`/`类型`(见 `src/artifact/`);渲染与 listen 在 `ext/web` + `plugins/web`。
---
## 2. 目录
```text
ext/web/
web.mq.md · 网页.mq.md # facade: page route serve render inspect
page.mq.md · route.mq.md · component.mq.md · dom.mq.md · client.mq.md
ext/net/ http.mq.md · websocket.mq.md · url.mq.md
ext/data/ db.mq.md · cache.mq.md · storage.mq.md
ext/security/ auth.mq.md · oidc.mq.md · rbac.mq.md
```
作者入口通常:
```text
site/
index.mq.md # type: web · route: / · 或 # main → web.serve
pages/*.mq.md # type: web
api/*.mq.md # type: endpoint
data/ # sqlite runtime (gitignore)
```
---
## 3. Document(`type: web`)
```markdown
---
type: web
title: Marqdo Blog
route: /
method: GET
data_source: posts
data_order: -created_at
import web:ext/web/web.mq.md
import data:ext/data/db.mq.md
---
# Marqdo
Welcome to the executable web.
```
| 键 (EN) | 键 (ZH) | 说明 |
|---------|---------|------|
| `type` | `类型` | `web` / `网页` |
| `route` | `路由` | 路径,可含 `{param}` |
| `method` | `方法` | 默认 `GET` |
| `title` | `标题` | 文档标题 / `<title>` |
| `data_source` | `数据源` | 表名 |
| `data_order` | `排序` | 如 `-created_at` |
| `data_where` | `条件` | `col=val,col2=val2` |
正文 Markdown 经 `dom` 变为 DOM Artifact,再 `web.render`。可选 View 表:
```markdown
`nodes` =
| type | slot | value | attrs | style |
|------|------|-------|-------|-------|
| text | kicker | AI Native | | eyebrow |
| title | title | Marqdo | | title |
```
`page.render nodes=` 是**唯一**主装配路径。禁止 `compose_main` / `compose_intro` 等碎片 DSL。
---
## 4. Endpoint(`type: endpoint`)
```markdown
---
type: endpoint
method: POST
path: /api/posts
request: json
response: json
auth: member
import data:ext/data/db.mq.md
---
**post = > data.db.insert …**
*post*
```
| 键 (EN) | 键 (ZH) | 说明 |
|---------|---------|------|
| `type` | `类型` | `endpoint` / `端点` |
| `path` | `路径` | URL |
| `method` | `方法` | GET/POST/… |
| `request` | `请求` | `json` / `form` / `none` |
| `response` | `响应` | `json` / `text` / `html` |
| `auth` | `鉴权` | 角色名或 `none` |
正文在请求时执行;返回值按 `response` 序列化。Result 协议见 §8。
**Handler 约定(锁定)**:Endpoint 文件用 `# main` 作为请求处理器。请求体/query/path 参数经 `run_artifact` 以 Metadata Binding 覆盖注入(与 `--bind` 同源),正文用已提升的普通变量名。无 `# main` 时返回 Artifact Result 摘要(`ok`/`value`/`error`),适合作文档型 ping。
---
## 5. Facade API(EN / ZH)
| EN | ZH | 作用 |
|----|-----|------|
| `web.page` | `网页.页面` | 从 metadata/正文/nodes 构造 Document 句柄 |
| `web.route` | `网页.路由` | 注册 declared Document 或 Endpoint |
| `web.serve` | `网页.服务` | `root=` 扫描注册 + listen |
| `web.render` | `网页.渲染` | Document → HTML(离线/预览) |
| `web.inspect` | `网页.检视` | `{declared, system}` 路由表 |
Middleware 一等:`web.use` / `route.use`,表驱动;**禁止** `configure(cors, security, …)` 垃圾抽屉。
---
## 6. 发现算法(锁定)
`web.serve root=DIR host=… port=…`:
1. 递归枚举 `DIR` 下 `*.mq.md`(跳过 `data/`、隐藏目录、`node_modules`)。
2. 解析文件头 Metadata;无 `type`/`类型` 或非 web/endpoint → 跳过。
3. Document → declared route(缺 `route` 则相对 `root` 的路径映射,`index.mq.md` → `/`)。
4. Endpoint → declared path+method(`path`/`路径` 必需)。
5. System routes(`/_part/*`、`/_form/*`、静态等)由 runtime 挂载,仅出现在 `inspect.system`。
6. 调用 native listen。
---
## 7. Table 规则
- **View**:仅 `|type|slot|value|attrs|style|`(及列名 ZH 别名:`类型|槽|值|属性|样式`)。
- **Resource schema**:`|字段|类型|可空|` 等 — 专用且文档化,不是函数私有方言。
- **禁止** `|属性|值|样式|` / `|front|back|css|` / `|组件|样式|` 作为页面主装配(已删除)。
- Raw HTML/CSS = escape hatch(`dom.raw` / `page.css`),不是默认语言。
---
## 8. Result
所有易失败的 Resource / Endpoint 边界返回 map:
| 键 | 类型 | 说明 |
|----|------|------|
| `ok` | bool | 成功 |
| `value` | any | 成功载荷 |
| `error` | text \| None | 失败信息 |
---
## 9. 硬约束(取代旧 C1–C7 作者面)
| # | 约束 |
|---|------|
| A1 | Document/Endpoint 由核心识别 `type`;装配在 ext(ADR 0007) |
| A2 | 禁止 `json.set` 袋胶水;Table = Data |
| A3 | 中英分文件;同文件禁止混 API |
| A4 | 禁止作者 `ensure_plugin` / `host_*` |
| A5 | 禁止隐式用户 route;system 须 `inspect` 可见 |
| A6 | 禁止恢复 `compose_*` / `app.configure` / god page 变异堆 |
| A7 | 能力只许迁移/提升;删除须本设计明示 + tests 同步 |
| A8 | 设计未锁定处停工提问,禁止自行发明 DSL |
---
## 10. 与旧 ext-web.md
旧「页面表 + compose_* + app.route」作者面 **废止**。历史能力矩阵见 [web-net-capabilities.md](web-net-capabilities.md);迁移后以本文件与 ADR 0007 为准。
+17 -13
View File
@@ -2,18 +2,21 @@
| | |
|---|---|
| 状态 | **Accepted · W0–W7 + P3 + W8(站点图标/Head/图片装配)完结**(SQLite/Postgres 作者面:页面装配 / db CRUD+`where` / form / 嵌入主区 / `app.route` / 路由 `/_part` / admin UI / RBAC / 上传相册 / SEO·RSS·sitemap / **favicon·头资源·图片装配**;Postgres 等见 §9) |
| 日期 | 2026-08-11(W8:2026-08-29;请求上下文:2026-09-18) |
| 相关 | [markdown-mapping.md](markdown-mapping.md) · [module-namespace.md](module-namespace.md) · [objects.md](objects.md) · [ext-abi.md](ext-abi.md) · [ext-cli.md](ext-cli.md) · [stdlib-i18n.md](stdlib-i18n.md) · [web-assets-and-images.md](web-assets-and-images.md) · **[ext-web-request-context.md](ext-web-request-context.md)** |
| 状态 | **Superseded(作者面)· 历史能力档案** — 现行作者面见 **[ext-web-artifact.md](ext-web-artifact.md)** · ADR [0007](../adr/0007-web-document-endpoint.md) |
| 日期 | 2026-08-11(W8:2026-08-29;请求上下文:2026-09-18;Artifact 重构:2026-09-28) |
| 相关 | [ext-web-artifact.md](ext-web-artifact.md) · [markdown-mapping.md](markdown-mapping.md) · [module-namespace.md](module-namespace.md) · [objects.md](objects.md) · [ext-abi.md](ext-abi.md) · [ext-cli.md](ext-cli.md) · [stdlib-i18n.md](stdlib-i18n.md) · [web-assets-and-images.md](web-assets-and-images.md) · **[ext-web-request-context.md](ext-web-request-context.md)** |
| 安装(目标) | `marqdo ext add web`(中英:`web` / `网页`) |
| 本文目的 | 锁定**作者面**与**类 API**;指导实现 |
| 本文目的 | **历史**类 API 与能力矩阵;新站请勿按本文 `compose_*` / `app.configure` 写作 |
> **2026-09-28**:作者面迁移到 Document / Endpoint / Resource。C1 已由 ADR 0007 修订。下文保留作迁移对照,**不是**现行推荐 API。
---
## 0. 一句话
`ext/web` 用 **GFM 表格 + Marqdo 类方法** 描述动态网站:表仍是字典/列表嵌套;类方法把表装配成可渲染页面。
**不改 Marqdo 核心语法**;扩展库存在的理由,正是让「文档即代码」在网站场景下成立。
**现行**:`.mq.md` Artifact(`type: web|endpoint`)描述页面与交互;见 [ext-web-artifact.md](ext-web-artifact.md)。
**历史(废止)**:`ext/web` 曾用 GFM 表格 + `# page`/`# app` 类方法拼装站点。该模型因 God Object / 碎片 DSL / 隐式路由被破坏性替换。
---
@@ -41,13 +44,14 @@
| # | 约束 |
|---|------|
| C1 | **禁止**为 web 改动 Marqdo 核心:单元格求值、import、点号值读取等保持不变 |
| C2 | 作者面**禁止** `json.parse` / `json.set` 袋胶水、手写 part JSON、手写 assemble 袋 |
| C3 | 能力以 **`#` 类 + `##` 方法** 暴露;**英文库文件只用英文名,中文库文件只用中文名**(见 [stdlib-i18n.md](stdlib-i18n.md));同一 `.mq.md` 内禁止中英 API 混排 |
| C4 | 首页在 **`index.mq.md`**:页面表在此;可复用片段进 `components/`;主体可就地写 |
| C5 | **旧有网络扩展库全部废弃**:`plugins/web`、`ext/web/**`(含现网示例/gold 中依赖旧 API 者)评审通过后**删除或清空后重写**;禁止在旧代码上打补丁演进。**落实**:原生层 Go 全量重写见 [ext-web-go-rewrite.md](ext-web-go-rewrite.md);作者面 ABI 名保持兼容 |
| C6 | `# db` / `# 数据库` 必须提供完整**增删改查**;列表查询英文方法名为 **`select`**(中文 **`查询`**);后台 `/admin` 与表单提交走同一套写库 API |
| C7 | 用户输入经 `# form` / `# 表单`:字段表 + 校验表;**服务端校验必须在写库前执行**(见 §5.5) |
| C1 | **修订(ADR 0007)**:核心可识别 Artifact `type: web\|endpoint`(及 ZH);**不**新增 CORE_CONSTRUCTS 标记;单元格求值 / import 句法不变;装配语义仍在 ext + plugin。旧文「禁止为 web 改核心」作废 |
| C2 | 作者面**禁止** `json.parse` / `json.set` 袋胶水(现行见 ext-web-artifact A2) |
| C3 | 中英分文件;同文件禁止混 API(现行 A3) |
| C4 | **废止**「页面表 + compose」心智 → Document Artifact + `pages/` · `api/` |
| C5 | 破坏性重写作者面(ADR 0007);原生 Go 演进见 [ext-web-go-rewrite.md](ext-web-go-rewrite.md) |
| C6 | DB 迁至 **`ext/data`**;CRUD 仍须完整(`select` / `查询`) |
| C7 | 表单校验仍须在写库前执行;挂载经 declared Endpoint / system `/_form`(inspect 可见) |
| A1–A8 | **现行硬约束**见 [ext-web-artifact.md](ext-web-artifact.md) §9 |
---
+10 -5
View File
@@ -143,13 +143,18 @@ Marqdo 对齐的判断是:
| `type` | 状态 | 含义 | 主要生产者 |
|--------|------|------|------------|
| `Marqdo Catalog` | **v0 已实现** | 包总览(`catalog.yaml` / `index.md`) | `marqdo catalog` |
| `Marqdo Module` | **v0 已实现** | 一个 `.mq.md` 模块的投影 | `marqdo catalog` |
| `Marqdo Task` | **规范已定;实现待做** | 一项可匹配的任务意图 | `plan` 晋升 |
| `Marqdo Agent Skill` | **规范已定;实现待做** | 指向可执行工作簿的技能 | `plan` 晋升 |
| `Marqdo Catalog` | **已实现** | 包总览(`catalog.yaml` / `index.md`) | `marqdo catalog` |
| `Marqdo Module` | **已实现** | 一个 `.mq.md` 模块的投影 | `marqdo catalog` |
| `Marqdo Task` | **已实现** | 一项可匹配的任务意图 | `plan` 晋升 |
| `Marqdo Agent Skill` | **已实现** | 指向可执行工作簿的技能 | `plan` 晋升 |
| `Capability` / `Function` | **EKC** | 工程能力 / 符号投影 | `marqdo knowledge` |
| `Decision` / `Constraint` / `Failure` / `AntiPattern` / `Pattern` / `Migration` / `Fact` | **EKC** | L4 工程知识 | 源 frontmatter + learn |
| `Marqdo Context Pack` | **EKC** | Agent EFI 上下文制品 | `reuse` / `preflight` |
| `Marqdo Policy` | **EKC** | 复用决议编译 | `marqdo reuse` |
| `Marqdo Engineering Knowledge` | **EKC** | 工程知识总览 | `engineering.yaml` / `index.mq.md` |
| 其他 | 开放 | 第三方/未来扩展 | — |
**不**引入 Google OKF 的 `Metric` / BigQuery 表类型作为 Marqdo 核心词汇;需要时可由用户包自行使用,运行时降级。
完整 EKC 规范见 [engineering-knowledge.md](engineering-knowledge.md)。
---
+12 -5
View File
@@ -2,16 +2,23 @@
| | |
|---|---|
| 状态 | **Accepted · W0–W7 + P3 完结**(2026-08-28 复核) |
| 日期 | 2026-08-26(初稿)· 2026-08-28(完结复核) |
| 相关 | [ext-web.md](ext-web.md) · [ext-web-net.md](ext-web-net.md) · [ext-abi.md](ext-abi.md) · [stdlib-modules.md](stdlib-modules.md) |
| 目标 | 盘点 `ext/web` + `plugins/web` + `lib/net` 现状,对照主流语言网络栈,给出「开发一个完整 Web 项目」所需能力的差距清单与补强路线 |
| 状态 | **Historical capability matrix** — 作者面以 **[ext-web-artifact.md](ext-web-artifact.md)** / ADR [0007](../adr/0007-web-document-endpoint.md) 为准(2026-09-28) |
| 日期 | 2026-08-26(初稿)· 2026-08-28(完结复核)· 2026-09-28(Artifact 标注) |
| 相关 | [ext-web-artifact.md](ext-web-artifact.md) · [ext-web.md](ext-web.md) · [ext-web-net.md](ext-web-net.md) · [ext-abi.md](ext-abi.md) · [stdlib-modules.md](stdlib-modules.md) |
| 目标 | 盘点 `ext/web` + `plugins/web` + `lib/net` 现状;**下文 API 名多为 W0–W7 历史面**(`compose_*` / `configure` 已废止) |
> **2026-09-28**:作者面迁移到 Document / Endpoint / Resource。
> - Document/Endpoint:`type: web|endpoint` + `web.serve` / `web.render` / `web.inspect`
> - DB/cache/storage → `ext/data`;auth/oidc/rbac → `ext/security`;ws → `ext/net`
> - 统一 View:`|type|slot|value|attrs|style|`;中间件:`web.use`
> 本文件保留为**原生能力矩阵**对照,勿再按 `compose_*` / `app.configure` 写作。
---
## 0. 一句话结论
**`ext/web` + `plugins/web` 已完成 W0–W7 + P3 路线图**:中间件/JSON API、数据层(连接池/事务/分页/FTS/迁移/外键/审计时间戳)、安全硬化(argon2/CSRF/会话持久化/限速/RBAC)、内容站点标配(SEO/RSS/Markdown/分页 UI)、上传下载/相册/ETag、sitemap/robots/错误页/重定向、WebSocket 广播与访问日志。足以在 Marqdo 上实现博客/CMS/中小型 API 站点;**未内置**的仅剩标签页模板(D6,可用路由+`db.count` 自建)与应用层反垃圾。边界不变:**纯解析进 `lib/net`,HTTP 服务器与领域能力进 `plugins/web` + `ext/web` 作者面**。
**能力层(plugins/web)**:中间件/JSON API、数据层、安全、内容站点、上传下载、sitemap、WebSocket 等仍在原生实现中。
**作者面(2026-09-28)**:改为 Artifact Metadata + facade;`compose_*` / `configure` **已删除**。边界不变:**纯解析进 `lib/net`,HTTP 服务器进 `plugins/web`,作者 API 进 `ext/web` + `ext/data` + `ext/security` + `ext/net`**。
**宿主集成下一刀**(同域 LLM SSE 中继、HTTP→用户 `##`、与 MCP Server 协作)不在本 W 表内,见 [ext-hosting-gaps.md](../research/ext-hosting-gaps.md) · [ext-hosting-fill.md](ext-hosting-fill.md) · [ext-hosting.md](../roadmap/ext-hosting.md)。
+2182
View File
File diff suppressed because it is too large. Load diff
+2818
View File
File diff suppressed because it is too large. Load diff
+85 -78
View File
@@ -1,19 +1,22 @@
# 第 14 章:网络与 Web 扩展
Marqdo 的网络能力分两层:**标准库 `lib/net`**(HTTP 客户端、cookie / multipart 解析)与**官方扩展 `ext/web`**(网页应用、数据库、表单、登录鉴权、WebSocket)。分层的依据是「代码即文档」的可读性——基础、通用、低依赖的进标准库;领域性强、依赖重、只在 Web 场景才用的进扩展。
> **现行作者面(2026-09-28 · ADR 0007)**:Document / Endpoint / Resource —— 见 [ext-web-artifact.md](../design/ext-web-artifact.md)。
> 下文若仍出现 `compose_*` / `app.configure` / `web.db` 同文件混写,视为**历史示例**;DB → `ext/data`,鉴权 → `ext/security`,实时 → `ext/net`。
Marqdo 的网络能力分两层:**标准库 `lib/net`**(HTTP 客户端、cookie / multipart 解析)与**官方扩展 `ext/web`**(Document/Endpoint facade + 拆分后的 data/security/net)。分层的依据是「代码即文档」。
## 14.1 概览
| 能力 | 位置 | 典型函数 |
| 能力 | 位置 | 典型 API |
|------|------|----------|
| HTTP(S) 客户端 | `lib/net` | `http_get` / `http_post` / `http_request` |
| Cookie 解析 | `lib/net` | `cookie_parse` |
| multipart/form-data 解析 | `lib/net` | `multipart_parse` |
| 网页应用 / 路由 / 静态 | `ext/web` | `web.page` / `web.app` |
| SQLite 数据库 | `ext/web` | `web.db` |
| 表单 | `ext/web` | `web.form` |
| 登录鉴权(session) | `ext/web` | `web.auth` / `app.auth` |
| WebSocket 客户端 / 端点 | `ext/web` | `web.ws` / `app.route_ws` |
| Cookie / multipart | `lib/net` | `cookie_parse` / `multipart_parse` |
| Document / Endpoint | `ext/web` | `type: web\|endpoint` · `web.serve` · `web.render` · `web.inspect` |
| View 装配 | `ext/web` | `|type|slot|value|attrs|style|` → `web.render` |
| 数据库 | `ext/data` | `db` CRUD / txn |
| 表单 | `ext/data` | `form` |
| 鉴权 / RBAC / OIDC | `ext/security` | `auth` · `rbac` · `oidc` |
| WebSocket | `ext/net` | `websocket` |
```bash
# 运行使用 ext/web 的程序前,先安装扩展
@@ -67,107 +70,111 @@ import net:lib/net.mq.md
> print text=[name](`field`)
```
## 14.3 网页应用(`ext/web`)
## 14.3 网页应用(Artifact · ADR 0007)
`ext/web` 用类 + 方法装配页面。导入 `web:ext/web/web.mq.md`(中文 `网页:ext/web/网页.mq.md`)。
站点是一堆 `.mq.md`:**Document**(`type: web`)与 **Endpoint**(`type: endpoint`)。入口调用 `web.serve` 扫描目录并监听。完整设计见 [ext-web-artifact.md](../design/ext-web-artifact.md);示例 [marqdo-blog](../../examples/marqdo-blog/)。
### 页面、组件、主体
### Document 首页
```markdown
---
type: web
title: 我的站点
route: /
method: GET
data_source: posts
data_order: -created_at
import web:ext/web/web.mq.md
---
# 我的站点
欢迎。下方卡片列表由 `data_source` 在服务时从数据库加载。
```
### Endpoint
```markdown
---
type: endpoint
method: GET
path: /api/ping
request: none
response: json
---
# main
`首页` =
`out` =
| 组件 | 样式 |
|------|------|
| nav.`导航` | shell.`顶栏` |
| ok | service |
|----|---------|
| True | demo |
**page = > web.page title="我的站点" intro="<h1>你好</h1>"**
**page = > `page`.compose_components components=`首页`**
**html = > `page`.render**
*out*
```
### 服务入口
```markdown
---
import web:ext/web/web.mq.md
import posts:db/posts.mq.md
---
# main
**store = > posts.open url="sqlite:data/site.db"**
> web.serve root="." host="127.0.0.1" port=18081 db=`store` static_dir="public"
```
离线预览可用 View 表 + `web.render`:
```markdown
`nodes` =
| type | slot | value | attrs | style |
|------|------|-------|-------|-------|
| title | title | Marqdo | | title |
**html = > web.render nodes=`nodes` title="preview"**
> print text=`html`
```
### 应用 + 路由 + 监听
### 数据库(`ext/data`)
```markdown
**app = > web.app page=`page` host=127.0.0.1 port=18081**
**app = > `app`.route path="/about" page=`关于`**
> web_listen app=`app`
```
---
import data:ext/data/db.mq.md
---
`listen` 提供 `/`、路由页、表单端点 `/_form/{id}`、可选静态目录与 `/admin`。
# main
### 数据库(SQLite)
```markdown
**store = > web.db url="sqlite:site.db"**
**store = > data.db url="sqlite:site.db"**
> `store`.init name=articles fields=`字段表`
> `store`.insert table=articles rows=`数据`
**rows = > `store`.select table=articles limit=10**
- [行](rows)
> print text=[title](`行`)
```
## 14.4 登录鉴权(session / cookie)
## 14.4 鉴权(`ext/security`)
给 `/admin*` 加登录门禁:`app.auth` 挂用户表,未登录重定向到 `/admin/login`。
HTTP 门禁与登录走 `ext/security`(`auth` / `rbac` / `oidc`),再用 `web.use` 挂中间件表;勿再写 `app.configure`。独立脚本可用:
```markdown
`管理员` =
---
import rbac:ext/security/rbac.mq.md
---
| 行 | 用户名 | 密码 |
|----|--------|------|
| 1 | admin | secret |
| 2 | 站长 | pw123 |
# main
**app = > web.app page=`page` admin=True**
**app = > `app`.auth users=`管理员` session_ttl=3600**
> web_listen app=`app`
**gate = > rbac.rbac**
**r = > `gate`.can permissions="desk:access" needed="desk:access"**
> print text=[allowed](`r`)
```
`session_ttl` 是会话有效期(秒),过期后重新登录。
## 14.5 WebSocket(`ext/net`)
**独立鉴权工具** `web.auth`:登录、校验、登出(不依赖页面),返回 `{ok, session_id, username}`:
```markdown
**auth = > web.auth users=`管理员` session_ttl=3600**
**login = > `auth`.login username="admin" password="secret"**
1. [ok](`login`)
> print text=登录成功:[username](`login`)
2. *
> print text=登录失败
**sid = [session_id](login)**
**check = > `auth`.check session_id=`sid`**
> print text=[username](`check`)
> `auth`.logout session_id=`sid`
```
## 14.5 WebSocket
`app.route_ws` 注册端点,`web.ws.connect` 单次请求–响应:
```markdown
**app = > `app`.route_ws path="/live" echo=True**
> web_listen app=`app`
```
客户端(另开终端):
```markdown
**ws = > web.ws timeout_sec=30**
**echo = > `ws`.connect url="ws://127.0.0.1:18081/live" message="hi"**
1. [ok](`echo`)
> print text=[messages](`echo`)
2. *
> print text=连接失败:[error](`echo`)
```
实时能力在 `ext/net`(如 `websocket`),与 Document 站点并列导入,不再挂在 `web.app` 的 junk drawer 上。
## 14.6 字典操作:用表格与取元,不用 `json.get`/`json.set`
@@ -0,0 +1,20 @@
---
title: resolve_config
capability: config.resolve
status: stable
---
Use when: final normalized configuration is required.
# config
## resolve_config
+ `path`
`defaults` =
| host | port |
|------|------|
| localhost | 8080 |
* defaults
@@ -0,0 +1,21 @@
---
title: Engineering Knowledge demo
description: Tiny project demonstrating Capability Catalog + reuse + decisions.
---
# Engineering Knowledge demo
This folder is a miniature “large engineering” slice: config resolution already exists.
Before generating a new loader, run:
```bash
marqdo knowledge examples/engineering-knowledge -o examples/engineering-knowledge/.marqdo
marqdo reuse "load and resolve configuration" examples/engineering-knowledge -o examples/engineering-knowledge/.marqdo
marqdo duplicate examples/engineering-knowledge -o examples/engineering-knowledge/.marqdo
```
Expected: **REUSE config.resolve**, not a new `load_configuration`.
# main
* "see README via catalog/reuse CLI"
@@ -0,0 +1,18 @@
---
type: Decision
id: ADR-022
title: Cache is not persistence
status: accepted
applies_to:
- config.resolve
---
# Cache is not persistence
## Decision
Redis is only used for ephemeral cache.
## Reason
The application must remain correct when Redis is unavailable.
@@ -0,0 +1,12 @@
---
type: Constraint
id: C-014
title: Centralized config resolution
status: stable
applies_to:
- config.resolve
---
# Centralized config resolution
All configuration resolution must go through `resolve_config`.
@@ -0,0 +1,20 @@
---
type: Failure
id: F-008
title: Do not invent a second config loader
status: resolved
---
# Do not invent a second config loader
## Failed approach
Added `load_configuration` beside `resolve_config`.
## Failure
Divergent defaults and error handling.
## Correct approach
Call `resolve_config`.
+27 -42
View File
@@ -1,54 +1,39 @@
# examples/marqdo-blog
一个用 Marqdo 的 `ext/web` 网络扩展库搭建的博客系统示例:美观大气的现代化主题、
SQLite 驱动的文章/标签、动态路由、WebSocket 实时终端、以及受会话门禁保护的后台 CRUD。
Marqdo **Web Artifact** blog (ADR 0007).
## 运行
## Layout
```bash
cargo build -p marqdo_plugin_web
cargo run -- examples/marqdo-blog/index.mq.md
```text
index.mq.md # type: web · / · data_source: posts
pages/about.mq.md # 类型: 网页 · /about
pages/post.mq.md # type: web · /post/{slug} · data_where
api/ping.mq.md # type: endpoint · GET /api/ping
api/echo.mq.md # type: endpoint · POST /api/echo
db/posts.mq.md # ext/data schema + seed + open
public/ # static (mounted at /static)
serve.mq.md # web.serve + db open
smoke.sh # curl checks against a running serve
```
打开 http://127.0.0.1:18085/:
- `/` — 首页:文章卡片网格 + 实时 WebSocket 终端
- `/post/{slug}` — 动态路由渲染单篇文章(标题/日期/标签/正文)
- `/tags` — 标签归档;`/tag/{slug}` 查看某标签下的文章
- `/about` — 关于页
- `/admin` — 会话门禁后台,登录 `admin` / `marqdo`,可增删改文章与标签
- `/live` — WebSocket 回显端点
- `/static/*` — 静态资源(`public/` 下的 `live.js`、favicon、logo)
- `/favicon.ico` — 站点图标(`应用.图标` 表装配)
## 设计亮点(文档即代码)
- **样式即数据表格,装配即函数**(`styles/theme.mq.md`):
主题不再是一大段手写 CSS 字符串,而是由一张张 GFM 样式表组成 ——
`|选择器|属性|值|` 规则行(同一选择器多行自动合并成一条规则),
响应式用 `|媒体|选择器|属性|值|` 分组进 `@media` 块。
每个 `## 段`(基础/顶栏/侧栏/主体/卡片/页脚/表单/响应式)导出一张样式表,
`## 全局` 用 `网页.样式装配` 函数把每张表装配成 CSS 文本,再 `text.str_join` 拼成完整样式表,
最后经 `页面.样式` 注入页面。样式保持可读的数据表格,不写胶水代码。
- **图标 / 头资源 / 图片装配**(W8):`应用.图标` 表挂 `/favicon.ico` 与 SVG;
`页面.头装配` 写 apple-touch-icon;`页面.图片装配` 把 logo 表注入主区(`mq-images`)。
含 `/` 的路径与 MIME 单元格需加引号(如 `"/static/logo.svg"`、`"image/png"`)。
- **页面 = 组件装配 + 主体绑定**(`index.mq.md`):
导航/侧栏/页脚用 `|组件|样式|` 表装配,文章列表用 `|属性|值|` 绑定 SQLite 字段。
- **动态路由**:`/post/{slug}` 的 `slug` 注入查询条件,`页面.详情` 把首行渲染成文章。
- **数据模型**(`db/`):`schema.mq.md` 定义 `posts`/`tags`/`post_tags` 三张表,
`seed.mq.md` 提供种子数据,`index.mq.md` 幂等建表 + 灌数据。
## 冒烟测试
服务器启动后(端口 18085),运行:
## Run
```bash
bash scripts/build-web-plugin.sh
export MARQDO_EXT=$PWD/ext # if ~/.marqdo/ext is stale
export MARQDO_WEB_PLUGIN=$PWD/plugins/web/build/libweb.so
cd examples/marqdo-blog && marqdo run serve.mq.md
# other terminal:
bash examples/marqdo-blog/smoke.sh
```
覆盖:首页/卡片/主题 CSS/媒体查询、静态资源、动态路由详情、标签归档、鉴权门禁、
登录会话、后台表单、WebSocket 回显。全部通过时输出 `通过 N / N`。
| Path | Artifact |
|------|----------|
| `/` | Document home + post cards |
| `/about` | ZH Document |
| `/post/{slug}` | Document detail |
| `/api/ping` | Endpoint JSON |
| `/api/echo` | Endpoint JSON echo |
| `/static/*` | public assets |
可从仓库根目录或本目录运行:相对 `static_dir`(`public`)与相对数据库路径
(`data/marqdo-blog.db`)都按**入口脚本所在目录**解析,不依赖终端工作目录。
Design: [ext-web-artifact.md](../../doc/design/ext-web-artifact.md).
+25
View File
@@ -0,0 +1,25 @@
---
type: endpoint
method: POST
path: /api/echo
request: json
response: json
auth: none
---
# main
Echo request fields bound via Artifact Metadata (`arg` overlays).
**msg = message**
1. not `msg`
**msg = ""**
2. *
`out` =
| ok | message |
|----|---------|
| True | `msg` |
*out*
+19
View File
@@ -0,0 +1,19 @@
---
type: endpoint
method: GET
path: /api/ping
request: none
response: json
auth: none
description: Health ping — documentation is the API (ADR 0007).
---
# main
`out` =
| ok | service |
|----|---------|
| True | marqdo-blog |
*out*
-28
View File
@@ -1,28 +0,0 @@
---
title: db/index
description: 打开 sqlite、建三张表、幂等种子。
导入 网页:ext/web/网页.mq.md
import schema:schema.mq.md
import seed:seed.mq.md
---
## 打开
**store = > 网页.数据库 地址="sqlite:data/marqdo-blog.db"**
**字段 = > schema.posts**
**标签字段 = > schema.tags**
**关联字段 = > schema.post_tags**
> `store`.初始化 名=posts 字段=`字段`
> `store`.初始化 名=tags 字段=`标签字段`
> `store`.初始化 名=post_tags 字段=`关联字段`
**行 = > store.查询 表="posts" 上限=1**
1. `行`
*store*
2. *
**文章 = > seed.posts**
**标签 = > seed.tags**
**关联 = > seed.post_tags**
> `store`.插入 表=posts 行=`文章`
> `store`.插入 表=tags 行=`标签`
> `store`.插入 表=post_tags 行=`关联`
*store*
+44
View File
@@ -0,0 +1,44 @@
---
title: blog posts schema
import data:ext/data/db.mq.md
---
## schema
`fields` =
| 字段 | 类型 | 可空 |
|------|------|------|
| title | text | False |
| slug | text | False |
| summary | text | True |
| content | text | True |
| tag | text | True |
| created_at | text | True |
*fields*
## seed
`rows` =
| title | slug | summary | content | tag | created_at |
|-------|------|---------|---------|-----|------------|
| Hello Marqdo | hello | Welcome | # Hello\n\nExecutable documents. | intro | 2026-09-01 |
| Artifact Web | artifact-web | Document first | Pages are mq.md files. | design | 2026-09-28 |
*rows*
## open
+ `url`="sqlite:data/blog.db"
**store = > data.db url=`url`**
**fields = > schema**
> `store`.init name=posts fields=`fields`
**existing = > `store`.select table="posts" limit=1**
1. `existing`
*store*
2. *
**seed = > seed**
> `store`.insert table=posts rows=`seed`
*store*
+9 -125
View File
@@ -1,131 +1,15 @@
---
type: web
title: Marqdo 博客
description: 用 web 扩展库搭建的博客系统示例:首页列表、动态文章详情、标签归档、登录门禁后台。
route: /
method: GET
data_source: posts
data_order: -created_at
description: >-
Document Artifact 首页(ADR 0007):正文即文档,列表来自 data_source。
导入 网页:ext/web/网页.mq.md
import theme:styles/theme.mq.md
import nav:components/nav.mq.md
import side:components/side.mq.md
import foot:components/foot.mq.md
import db:db/index.mq.md
---
# main
# Marqdo 博客
`首页` =
| 组件 | 样式 |
|------|------|
| nav.`导航` | |
| side.`侧栏` | |
| foot.`页脚` | |
`列表` =
| 属性 | 值 | 样式 |
|------|-----|------|
| title | posts.title | |
| body | posts.summary | |
| meta | posts.created_at | |
| tag | posts.tag | |
| href | posts.slug | |
`详情绑定` =
| 属性 | 值 | 样式 |
|------|-----|------|
| title | posts.title | |
| meta | posts.created_at | |
| tag | posts.tag | |
| body | posts.content | |
`标签列表` =
| 属性 | 值 | 样式 |
|------|-----|------|
| title | tags.name | |
| meta | 点击查看该标签下的文章 | |
| href | tags.slug | |
`文章条件` =
| 字段 | 操作 | 值 |
|------|------|-----|
| slug | = | {slug} |
`标签条件` =
| 字段 | 操作 | 值 |
|------|------|-----|
| tag | = | {slug} |
`管理员` =
| 行 | 用户名 | 密码 |
|----|--------|------|
| 1 | admin | marqdo |
`站点图标` =
| 路径 | 关系 | 类型 | 尺寸 | 地址 |
|------|------|------|------|------|
| "public/favicon.png" | icon | "image/png" | 32x32 | "/favicon.ico" |
| "public/favicon.svg" | icon | "image/svg+xml" | any | "/static/favicon.svg" |
`头资源` =
| 关系 | 地址 | 类型 |
|------|------|------|
| apple-touch-icon | "/static/favicon.png" | "image/png" |
`品牌图` =
| 源 | 替代 | 类 | 链接 | 宽度 | 加载 |
|----|------|----|------|------|------|
| "/static/logo.svg" | Marqdo | brand-logo | "/" | 40 | eager |
**store = > db.打开**
**首页CSS = > theme.全局**
**page = > 网页.页面 标题="Marqdo 博客" 引言="<h1>Marqdo 博客</h1><p>用 web 扩展库搭建的博客:表格装配页面、SQLite 数据、动态路由、实时终端。</p><h2 id=\"live\">实时终端</h2><p id=\"live-status\" class=\"live-pill\">connecting…</p><input id=\"live-msg\" placeholder=\"输入一条消息…\"><button id=\"live-send\" disabled>发送</button><div id=\"live-out\"></div><script src=\"/static/live.js\"></script>"**
**page = > page.组件装配 组件=`首页`**
**page = > page.主体装配 主体=`列表`**
**page = > page.排序 排序="-created_at"**
**page = > page.样式 样式=`首页CSS`**
**page = > page.头装配 表=`头资源`**
**page = > page.图片装配 表=`品牌图`**
**about = > 网页.页面 标题="关于" 引言="<h1>关于本站</h1><p>一个用 Marqdo 的 web 扩展库(ext/web)写成的博客系统。</p><p>它演示了:GFM 表格装配导航与侧栏、SQLite 驱动的主体绑定、动态路由 /post/{slug}、标签归档、以及受 session 保护的后台 CRUD。</p>"**
**about = > about.组件装配 组件=`首页`**
**about = > about.样式 样式=`首页CSS`**
**post = > 网页.页面 标题="文章"**
**post = > post.组件装配 组件=`首页`**
**post = > post.主体装配 主体=`详情绑定`**
**post = > post.查询条件 条件=`文章条件`**
**post = > post.详情 详情=True**
**post = > post.样式 样式=`首页CSS`**
**tags = > 网页.页面 标题="标签归档" 引言="<h1>标签归档</h1><p>点击标签查看该主题下的文章。</p>"**
**tags = > tags.组件装配 组件=`首页`**
**tags = > tags.主体装配 主体=`标签列表`**
**tags = > tags.链接前缀 前缀="/tag/"**
**tags = > tags.样式 样式=`首页CSS`**
**tagged = > 网页.页面 标题="标签"**
**tagged = > tagged.组件装配 组件=`首页`**
**tagged = > tagged.主体装配 主体=`列表`**
**tagged = > tagged.查询条件 条件=`标签条件`**
**tagged = > tagged.排序 排序="-created_at"**
**tagged = > tagged.样式 样式=`首页CSS`**
**app = > 网页.应用 页面=page 数据库=store 后台=True 主机="127.0.0.1" 端口=18085**
**app = > app.路由 路径="/about" 页面=about**
**app = > app.路由 路径="/post/{slug}" 页面=post**
**app = > app.路由 路径="/tags" 页面=tags**
**app = > app.路由 路径="/tag/{slug}" 页面=tagged**
**app = > app.静态 目录="public" 挂载="/static"**
**app = > app.图标 表=`站点图标`**
**app = > app.路由实时 路径="/live" 回显=True**
**app = > app.鉴权 用户表=`管理员` 会话时长=3600**
> `app`.监听
可执行文档即网页。本站用 Artifact Metadata 声明路由;文章列表由 `data_source: posts` 在服务时加载。
+10
View File
@@ -0,0 +1,10 @@
---
类型: 网页
路由: /about
方法: GET
标题: 关于
---
# 关于 Marqdo
Marqdo 是可执行文档语言:文档即程序,程序即知识。
+13
View File
@@ -0,0 +1,13 @@
---
type: web
title: Post
route: /post/{slug}
method: GET
data_source: posts
data_where: slug={slug}
description: Dynamic Document route — body is the article shell; data from Resource.
---
# Post
Article page for `{slug}`. List and detail data are provided by `ext/data` at serve time when `data_source` is set on the Artifact.
+11
View File
@@ -0,0 +1,11 @@
---
title: marqdo-blog serve
description: Scan Artifact documents/endpoints and listen (ADR 0007).
import web:ext/web/web.mq.md
import posts:db/posts.mq.md
---
# main
**store = > posts.open url="sqlite:data/blog.db"**
> web.serve root="." host="127.0.0.1" port=18081 db=`store` static_dir="public"
+23 -53
View File
@@ -1,71 +1,41 @@
#!/usr/bin/env bash
# 冒烟测试:marqdo-blog 全功能验证。
# 用法:bash examples/marqdo-blog/smoke.sh
# Artifact blog smoke (ADR 0007). Usage: bash examples/marqdo-blog/smoke.sh
# Expects serve.mq.md already listening on BASE (default 18081).
set -uo pipefail
BASE="http://127.0.0.1:18085"
BASE="${BASE:-http://127.0.0.1:18081}"
PASS=0; FAIL=0
ok() { echo " ok: $1"; PASS=$((PASS+1)); }
bad() { echo " FAIL: $1"; FAIL=$((FAIL+1)); }
echo "== 首页 =="
echo "== Document home =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/")
[ "$code" = "200" ] && ok "GET / -> $code" || bad "GET / -> $code"
curl -s "$BASE/" | grep -q '<title>Marqdo 博客</title>' && ok "title" || bad "title"
curl -s "$BASE/" | grep -q 'class="card"' && ok "卡片渲染" || bad "卡片缺失"
curl -s "$BASE/" | grep -q 'aside.side' && ok "主题 CSS 注入" || bad "主题 CSS 缺失"
curl -s "$BASE/" | grep -q '@media (max-width: 860px)' && ok "响应式媒体查询" || bad "媒体查询缺失"
curl -s "$BASE/" | grep -q 'class="card"' && ok "data_source cards" || bad "cards missing"
curl -s "$BASE/" | grep -q 'Hello Marqdo' && ok "seed title" || bad "seed title missing"
echo "== 静态 =="
echo "== Static =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/static/live.js")
[ "$code" = "200" ] && ok "GET /static/live.js -> $code" || bad "static -> $code"
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/favicon.ico")
[ "$code" = "200" ] && ok "GET /favicon.ico -> $code" || bad "favicon -> $code"
curl -s "$BASE/" | grep -q 'rel="icon"' && ok "head icon link" || bad "head icon 缺失"
curl -s "$BASE/" | grep -q 'mq-images' && ok "图片装配 logo" || bad "mq-images 缺失"
echo "== 动态路由 =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/post/hello-marqdo")
[ "$code" = "200" ] && ok "GET /post/hello-marqdo -> $code" || bad "post detail -> $code"
curl -s "$BASE/post/hello-marqdo" | grep -q 'article-title' && ok "文章详情渲染" || bad "文章详情缺失"
echo "== Dynamic Document =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/post/hello")
[ "$code" = "200" ] && ok "GET /post/hello -> $code" || bad "post detail -> $code"
curl -s "$BASE/post/hello" | grep -q 'article-title' && ok "article title" || bad "article title missing"
echo "== 标签 =="
curl -s "$BASE/tags" | grep -q 'href="/tag/marqdo"' && ok "标签归档链接" || bad "标签链接缺失"
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/tag/web")
[ "$code" = "200" ] && ok "GET /tag/web -> $code" || bad "tag page -> $code"
echo "== ZH Document =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/about")
[ "$code" = "200" ] && ok "GET /about -> $code" || bad "about -> $code"
curl -s "$BASE/about" | grep -q '关于 Marqdo' && ok "about body" || bad "about body missing"
echo "== 鉴权门禁 =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/admin")
[ "$code" = "303" ] && ok "未登录 /admin -> $code (重定向登录)" || bad "未登录 /admin -> $code"
echo "== 登录 =="
rm -f /tmp/mq_smoke_cookies.txt /tmp/mq_smoke_headers.txt
curl -s -D /tmp/mq_smoke_headers.txt -o /dev/null -d "username=admin&password=marqdo" "$BASE/admin/login"
sid=$(grep -i "set-cookie" /tmp/mq_smoke_headers.txt | sed 's/.*marqdo_sid=\([^;]*\).*/\1/')
[ -n "$sid" ] && ok "登录获取会话 cookie" || bad "登录未返回 cookie"
code=$(curl -s -b "marqdo_sid=$sid" -o /dev/null -w "%{http_code}" "$BASE/admin")
[ "$code" = "200" ] && ok "带会话访问 /admin -> $code" || bad "带会话 /admin -> $code"
echo "== 发布文章 (CRUD) =="
code=$(curl -s -b "marqdo_sid=$sid" -o /dev/null -w "%{http_code}" "$BASE/admin/posts/new")
[ "$code" = "200" ] && ok "GET /admin/posts/new -> $code" || bad "new form -> $code"
echo "== WebSocket =="
wsok=$(python3 -c "
import socket, base64, os
s = socket.create_connection(('127.0.0.1', 18085), timeout=3)
key = base64.b64encode(os.urandom(16)).decode()
s.sendall((f'GET /live HTTP/1.1\r\nHost: x\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n').encode())
r = s.recv(1024).decode(errors='replace')
if '101' not in r.split('\r\n')[0]:
print('NO'); s.close(); raise SystemExit
mask = os.urandom(4); payload = b'ping'
m = bytes(b ^ mask[i%4] for i, b in enumerate(payload))
s.sendall(bytes([0x81, 0x80|len(payload)]) + mask + m)
d = s.recv(1024)
print('OK' if b'ping' in d else 'NO')
s.close()
" 2>/dev/null)
[ "$wsok" = "OK" ] && ok "WebSocket /live 回显" || bad "WebSocket 回显 $wsok"
echo "== Endpoint =="
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/api/ping")
[ "$code" = "200" ] && ok "GET /api/ping -> $code" || bad "ping -> $code"
curl -s "$BASE/api/ping" | grep -q 'marqdo-blog' && ok "ping json" || bad "ping json missing"
code=$(curl -s -o /tmp/mq_blog_echo.json -w "%{http_code}" \
-H 'Content-Type: application/json' -d '{"message":"hi"}' "$BASE/api/echo")
[ "$code" = "200" ] && ok "POST /api/echo -> $code" || bad "echo -> $code"
grep -q '"message":"hi"' /tmp/mq_blog_echo.json && ok "echo body" || bad "echo body missing"
echo
echo "通过 $PASS / $((PASS+FAIL))"
+9 -15
View File
@@ -1,20 +1,14 @@
# web-client-site(路线 D)
# examples/web-client-site
服务端 `ext/web` + 浏览器 WASM 会话;**作者零业务 JS**。
SSR **Document Artifact** + `web.client_embed` auto-mount(零作者 JS)。ADR 0007。
```bash
# 仓库根
cargo build -p marqdo --release
cargo build --release -p marqdo_plugin_web
./target/release/marqdo ext add web
# 构建 WASM + bridge 到本示例 static/
./target/release/marqdo wasm build -o examples/web-client-site/static
# 保留 client.mq.md(wasm build 只写入 wasm/bridge)
cd examples/web-client-site
marqdo run index.mq.md
# 打开 http://127.0.0.1:18090/
bash scripts/build-web-plugin.sh
export MARQDO_EXT=$PWD/ext # 若本机 ~/.marqdo/ext 未同步
marqdo wasm build -o examples/web-client-site/static
cargo run -- run examples/web-client-site/serve.mq.md
```
`index.mq.md` 用 `web.client_embed source="/static/client.mq.md"` 注入自启脚本;交互在 `static/client.mq.md`。
- `index.mq.md` — `type: web` 文档页
- `serve.mq.md` — `web.serve` + `static_dir`
- `static/client.mq.md` — `lib/browser` wire(不变)
+8 -13
View File
@@ -1,19 +1,14 @@
---
title: web-client-site
description: SSR shell + client_embed auto-mount; author zero JS (route D)
type: web
title: Marqdo client site
route: /
method: GET
description: SSR Document + client_embed auto-mount; author zero JS (ADR 0007).
import web:ext/web/web.mq.md
---
# main
# Marqdo client site
Server page with a counter UI. Client logic lives in `static/client.mq.md`
(loaded by official bridge via `client_embed`).
SSR shell with WASM session. Client logic lives in `static/client.mq.md`.
**`embed` = > web.client_embed bridge="/static/marqdo-bridge.js" wasm="/static/marqdo_wasm.wasm" source="/static/client.mq.md" boot=True**
**`intro` = "<h1>Marqdo client site</h1><p>SSR + WASM session. No author JS.</p><div id=\"count\">0</div><p><button id=\"bump\" type=\"button\" disabled>Bump</button> <button id=\"reset\" type=\"button\" disabled>Reset</button></p><pre id=\"log\">booting…</pre>" + embed**
**page = > web.page title="Marqdo client site" intro=intro**
**app = > web.app page=page host="127.0.0.1" port=18090**
**app = > app.static dir="static" mount="/static"**
> `app`.listen
To serve with static assets, run `serve.mq.md` in this directory.
+10
View File
@@ -0,0 +1,10 @@
---
title: web-client-site serve
import web:ext/web/web.mq.md
---
# main
**`embed` = > web.client_embed source="/static/client.mq.md"**
> print text=embed-ready
> web.serve root="." host="127.0.0.1" port=18090 static_dir="static"
+1 -1
View File
@@ -1 +1 @@
1.1.1
1.3.1
+76
View File
@@ -2104,6 +2104,51 @@ With `stream=True`, the model call uses SSE; `echo=True` prints delta text to st
*out*
## preflight
+ `task`
+ `root`="."
+ `marqdo_dir`=None
+ `min_candidates`=3
+ `allow_create_after`=2
+ `reuse_threshold`=0.72
+ `adapt_threshold`=0.45
Engineering First Information gate (008 / EKC). Reads compiled `.marqdo` knowledge.
Returns `{status,decision,create_allowed,existing,constraints,decisions,failures,recommended,context_path}`.
Hard rule: generate only when `create_allowed` is true (or decision is REUSE/ADAPT).
1. `marqdo_dir`
**pf = > agent_eng_preflight task=`task` root=`root` marqdo_dir=`marqdo_dir` min_candidates=`min_candidates` allow_create_after=`allow_create_after` reuse_threshold=`reuse_threshold` adapt_threshold=`adapt_threshold`**
2. *
**pf = > agent_eng_preflight task=`task` root=`root` min_candidates=`min_candidates` allow_create_after=`allow_create_after` reuse_threshold=`reuse_threshold` adapt_threshold=`adapt_threshold`**
*pf*
## eng_reuse
+ `task`
+ `root`="."
+ `marqdo_dir`=None
Alias of `preflight` focused on REUSE / ADAPT / CREATE payload.
1. `marqdo_dir`
*> agent_eng_reuse task=`task` root=`root` marqdo_dir=`marqdo_dir`*
2. *
*> agent_eng_reuse task=`task` root=`root`*
## eng_record
+ `decision`
+ `root`="."
+ `marqdo_dir`=None
+ `duplicated`=False
Record REUSE/ADAPT/CREATE into `.marqdo/agent/episodes/reuse_metrics.json`.
1. `marqdo_dir`
*> agent_eng_record decision=`decision` root=`root` marqdo_dir=`marqdo_dir` duplicated=`duplicated`*
2. *
*> agent_eng_record decision=`decision` root=`root` duplicated=`duplicated`*
## plan
+ `goal`
+ `workbook_dir`=None
@@ -2127,9 +2172,14 @@ With `stream=True`, the model call uses SSE; `echo=True` prints delta text to st
+ `trace`=False
+ `resume_id`=None
+ `resume_dir`=.marqdo/agent-resume
+ `eng_preflight`=False
+ `marqdo_dir`=.marqdo
+ `allow_create_after`=2
Multi-step with OKF agent-kb. Default workbook is `kb_dir/resources/<slug>.mq.md`. While task file count `< explore_n` and skill is not llm_free, force a new explore variant under `kb_dir/explore/<slug>/`. Code-first: llm_free hits skip parent LLM. File children return via `# main`; `plan` exposes that as `result`.
Engineering Knowledge preflight when `eng_preflight=True`: REUSE/ADAPT preferred; CREATE only if `create_allowed`. Production agents should enable this; tests that only exercise agent-kb may leave it false.
Reuse lookup: exact → alias → canonicalize → optional local n-gram `near` when `near_match=True` and score ≥ `near_threshold`. Non-hit path: optional `soft_match=True` parent REUSE/NEW over ranked `agent_kb_near_match` candidates; else decompose before first child spawn (`DECISION: RUN` / `CONTINUE`+patch / solidified `DONE`). Then `await` → revise loop. Process events (`decision` / `round` / `done` / tools) are always attached on the result map for audit and view process cards; `stream=True` additionally SSE-publishes them (and parent `delta`). `echo=True` prints `plan:decompose` / `plan:await` / deltas. `trace=True` writes events to writeback slot `trace`. Quiet child subtasks stay quiet.
**tools = > json.get value=`self` key="tools"**
@@ -2140,6 +2190,32 @@ Reuse lookup: exact → alias → canonicalize → optional local n-gram `near`
**explore_attempt = None**
**skel_kind = skeleton**
**events = > json.parse text=[]**
**eng = None**
1. `eng_preflight`
**eng = > agent_eng_preflight task=`goal` marqdo_dir=`marqdo_dir` allow_create_after=`allow_create_after`**
**eng_status = > json.get value=`eng` key="status"**
**eng_decision = > json.get value=`eng` key="decision"**
**eng_allowed = > json.get value=`eng` key="create_allowed"**
1. `eng_status` == missing_knowledge
**_ = 1**
2. `eng_decision` == REUSE
> agent_eng_record decision="REUSE" marqdo_dir=`marqdo_dir`
**_ = 1**
3. `eng_decision` == ADAPT
> agent_eng_record decision="ADAPT" marqdo_dir=`marqdo_dir`
**_ = 1**
4. `eng_allowed`
> agent_eng_record decision="CREATE" marqdo_dir=`marqdo_dir`
**_ = 1**
5. *
**out = > json.parse text={"status":"blocked","cache":"eng-block","eng":true}**
**out = > json.set map=`out` key="decision" value="BLOCKED"**
**out = > json.set map=`out` key="preflight" value=`eng`**
**out = > json.set map=`out` key="events" value=`events`**
*out*
2. *
**_ = 1**
**tf = > agent_kb_task_files kb_dir=`kb_dir` goal=`goal` tools=`tools`**
**nfiles = > json.get value=`tf` key="count"**
+10 -1
View File
@@ -289,11 +289,20 @@ Semantic ask → **LLMResult** (`text`, `model`, `finish`, `backend`, `usage`, `
2. *
**_ = 1**
Hoist receiver fields before the result table — table cells can shadow `self` /
column names (`model`, `usage`), so `[model](self)` / `[total_tokens](usage)`
inside the table is unsafe.
**mdl = [model](self)**
**b = [backend](self)**
**nm = [name](self)**
**tt = [total_tokens](usage)**
`result` =
| text | model | finish | backend | usage | name | llm_calls | tokens | tool_calls |
|------|-------|--------|---------|-------|------|-----------|--------|------------|
| `text` | [model](self) | `finish` | [backend](self) | `usage` | [name](self) | 1 | [total_tokens](usage) | `tc` |
| `text` | `mdl` | `finish` | `b` | `usage` | `nm` | 1 | `tt` | `tc` |
*result*
+8 -1
View File
@@ -240,11 +240,18 @@ import openai:ext/ai/llm/openai.mq.md
2. *
**_ = 1**
表单元格会遮蔽 `自` / 列名,先取出字段再组结果表。
**mdl = [model](自)**
**b = [backend](自)**
**nm = [name](自)**
**tt = [total_tokens](usage)**
`result` =
| text | model | finish | backend | usage | name | llm_calls | tokens | tool_calls |
|------|-------|--------|---------|-------|------|-----------|--------|------------|
| `text` | [model](自) | `finish` | [backend](自) | `usage` | [name](自) | 1 | [total_tokens](usage) | `tc` |
| `text` | `mdl` | `finish` | `b` | `usage` | `nm` | 1 | `tt` | `tc` |
*result*
+13 -2
View File
@@ -183,6 +183,15 @@ import json:lib/json.mq.md
*out*
## 工程预检
+ `任务`
+ `根目录`="."
+ `知识目录`=None
Engineering First Information 门闩。转发 `agent.preflight`。
*> agent.preflight task=`任务` root=`根目录` marqdo_dir=`知识目录`*
## 多步
+ `目标`
+ `工作簿目录`=None
@@ -204,14 +213,16 @@ import json:lib/json.mq.md
+ `流式`=False
+ `打印增量`=False
+ `轨迹`=False
+ `工程预检`=False
+ `工程知识目录`=.marqdo
多步工作簿默认写入 OKF `知识库目录/resources/<slug>.mq.md`;仅当传入 `工作簿目录` 时才用临时 runs。转发至英文 `agent.plan`。`流式` / `打印增量` / `轨迹` 对应 `stream` / `echo` / `trace`;`软命中` → `soft_match`;`近义命中` / `近义阈值` → `near_match` / `near_threshold`。
多步工作簿默认写入 OKF `知识库目录/resources/<slug>.mq.md`;仅当传入 `工作簿目录` 时才用临时 runs。转发至英文 `agent.plan`。`流式` / `打印增量` / `轨迹` 对应 `stream` / `echo` / `trace`;`软命中` → `soft_match`;`近义命中` / `近义阈值` → `near_match` / `near_threshold`。`工程预检` → `eng_preflight`。
**model = > json.get value=`self` key="model"**
**tools = > json.get value=`self` key="tools"**
**standing = > json.get value=`self` key="standing"**
**en = > agent.agent model=`model` tools=`tools` standing=`standing`**
*> en.plan goal=`目标` workbook_dir=`工作簿目录` workbook=`工作簿` max_rounds=`最大轮次` confirm=`确认分解` writeback=`写回` skeleton=`骨架` reuse=`复用` optimize=`优化` force=`强制` soft_match=`软命中` near_match=`近义命中` near_threshold=`近义阈值` promote=`晋升` kb_dir=`知识库目录` improve_every=`改进周期` explore_n=`探索次数` stream=`流式` echo=`打印增量` trace=`轨迹`*
*> en.plan goal=`目标` workbook_dir=`工作簿目录` workbook=`工作簿` max_rounds=`最大轮次` confirm=`确认分解` writeback=`写回` skeleton=`骨架` reuse=`复用` optimize=`优化` force=`强制` soft_match=`软命中` near_match=`近义命中` near_threshold=`近义阈值` promote=`晋升` kb_dir=`知识库目录` improve_every=`改进周期` explore_n=`探索次数` stream=`流式` echo=`打印增量` trace=`轨迹` eng_preflight=`工程预检` marqdo_dir=`工程知识目录`*
## 解析
+ `任务`
+60
View File
@@ -0,0 +1,60 @@
---
title: ext/data/cache
description: Cache Resource (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/data: native web plugin not found
> sys.exit code=1
****
# cache
+ `url`="memory:"
Key–value cache. Use `memory:` for in-process (tests / single process) or `redis://host:6379/0` for Redis.
> _load
*> web_cache_new url=`url`*
## get
+ `key`
**url = [url](self)**
*> web_cache_get url=`url` key=`key`*
## set
+ `key`
+ `value`
+ `ttl`=None
Optional `ttl` is seconds until expiry.
**url = [url](self)**
*> web_cache_set url=`url` key=`key` value=`value` ttl=`ttl`*
## del
+ `key`
**url = [url](self)**
*> web_cache_del url=`url` key=`key`*
## exists
+ `key`
**url = [url](self)**
*> web_cache_exists url=`url` key=`key`*
## ttl
+ `key`
**url = [url](self)**
*> web_cache_ttl url=`url` key=`key`*
+238
View File
@@ -0,0 +1,238 @@
---
title: ext/data/db
description: Database Resource — CRUD, txn, migrate, FTS (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/data: native web plugin not found
> sys.exit code=1
****
# db
+ `url`="sqlite:site.db"
Open a database handle. URL schemes: `sqlite:path` (default), `postgres://…` / `postgresql://…` (same CRUD methods).
> _load
*> web_db_new url=`url`*
## init
+ `name`
+ `fields`
Create a table from a schema table. Optional columns: `唯一`/`unique`, `索引`/`index` (creates UNIQUE / INDEX), `外键`/`fk`/`references` (e.g. `posts.id` or `posts(id)`). Columns named `created_at` / `updated_at` are filled automatically on insert/update when present.
**url = [url](self)**
*> web_db_init url=`url` name=`name` fields=`fields`*
## insert
+ `table`
+ `rows`
+ `txn`=None
**url = [url](self)**
*> web_db_insert url=`url` table=`table` rows=`rows` txn=`txn`*
## select
+ `table`
+ `where`=None
+ `limit`=200
+ `order`=None
+ `txn`=None
Simple filters: one-row map of column→value (AND `=`), or rows `|字段|操作|值|` (`=` `!=` `>` `>=` `<` `<=` `like` `in` `between` `is null`; add `|或|` = `是` to join a row with `OR`). `order` is a column name with optional `-` prefix for descending (`"created_at"`, `"-created_at"`), comma-separated for multiple keys. Pass `txn` to read inside an open transaction. For pages (with a total), use `paginate`.
**url = [url](self)**
**r = > web_db_select url=`url` table=`table` where=`where` limit=`limit` order=`order` offset=None txn=`txn`**
*[rows](r)*
## paginate
+ `table`
+ `where`=None
+ `limit`=200
+ `order`=None
+ `跳过`=0
+ `txn`=None
Like `select` but returns `{ rows, total }` — the total counts rows matching `where` regardless of `limit`/`跳过`, so you can render `上一页 / 下一页`. Set `跳过` to the number of rows to skip (e.g. page 2 with 10 per page ⇒ `跳过`=10).
**url = [url](self)**
*> web_db_select url=`url` table=`table` where=`where` limit=`limit` order=`order` offset=`跳过` txn=`txn`*
## get
+ `table`
+ `id`
+ `txn`=None
**url = [url](self)**
*> web_db_get url=`url` table=`table` id=`id` txn=`txn`*
## update
+ `table`
+ `id`
+ `row`
+ `txn`=None
**url = [url](self)**
*> web_db_update url=`url` table=`table` id=`id` row=`row` txn=`txn`*
## delete
+ `table`
+ `id`
+ `txn`=None
**url = [url](self)**
*> web_db_delete url=`url` table=`table` id=`id` txn=`txn`*
## exec
+ `sql`
+ `args`=None
+ `txn`=None
**url = [url](self)**
*> web_db_exec url=`url` sql=`sql` args=`args` txn=`txn`*
## query
+ `sql`
+ `args`=None
+ `txn`=None
Run bare SQL and return the result set — count / join / group / subqueries. Returns `{ rows, count }`.
**url = [url](self)**
*> web_db_query url=`url` sql=`sql` args=`args` txn=`txn`*
## count
+ `table`
+ `where`=None
+ `txn`=None
Count rows matching a `where` filter (same syntax as `select`). Returns a number.
**url = [url](self)**
**r = > web_db_count url=`url` table=`table` where=`where` txn=`txn`**
*[count](r)*
## migrate
+ `steps`
Apply versioned SQL migrations. `steps` is a `|version|sql|` / `|版本|SQL|` table. Applied versions are recorded in `_marqdo_migrations`. Re-running is a no-op for already-applied versions. SQLite only.
**url = [url](self)**
*> web_db_migrate url=`url` steps=`steps`*
## fts
+ `table`
+ `columns`
+ `name`=None
Create an FTS5 index on `table` for the listed content columns (CSV string or list). Default FTS name is `{table}_fts`. Keeps the index in sync via triggers. Requires integer `id` PK. SQLite only.
**url = [url](self)**
*> web_db_fts_create url=`url` table=`table` columns=`columns` name=`name`*
## search
+ `table`
+ `q`
+ `limit`=20
+ `name`=None
Full-text search (`MATCH`) against the FTS5 index from `fts`. Returns `{ rows, count }` with a `rank` column (bm25). SQLite only.
**url = [url](self)**
*> web_db_search url=`url` table=`table` q=`q` limit=`limit` name=`name`*
## 事务
Begin a transaction: borrows the pooled connection exclusively and returns a
`txn` handle. Write inside it, then `提交` (commit) or `回滚` (roll back).
Every statement runs on the same connection, so a batch is atomic.
**url = [url](self)**
*> web_db_begin url=`url`*
# txn
+ `txn`
+ `url`
A transaction handle from `db.事务`. All CRUD here runs on the transaction's
connection; finish with `提交` or `回滚`.
*self*
## insert
+ `table`
+ `rows`
**url = [url](self)**
**txn = [txn](self)**
*> web_db_insert url=`url` table=`table` rows=`rows` txn=`txn`*
## select
+ `table`
+ `where`=None
+ `limit`=200
+ `order`=None
Same filters as `db.select`; runs inside the transaction.
**url = [url](self)**
**txn = [txn](self)**
**r = > web_db_select url=`url` table=`table` where=`where` limit=`limit` order=`order` offset=None txn=`txn`**
*[rows](r)*
## get
+ `table`
+ `id`
**url = [url](self)**
**txn = [txn](self)**
*> web_db_get url=`url` table=`table` id=`id` txn=`txn`*
## update
+ `table`
+ `id`
+ `row`
**url = [url](self)**
**txn = [txn](self)**
*> web_db_update url=`url` table=`table` id=`id` row=`row` txn=`txn`*
## delete
+ `table`
+ `id`
**url = [url](self)**
**txn = [txn](self)**
*> web_db_delete url=`url` table=`table` id=`id` txn=`txn`*
## exec
+ `sql`
+ `args`=None
**url = [url](self)**
**txn = [txn](self)**
*> web_db_exec url=`url` sql=`sql` args=`args` txn=`txn`*
## 提交
Commit the transaction and return its connection to the pool.
**txn = [txn](self)**
*> web_db_commit txn=`txn`*
## 回滚
Roll the transaction back (undo every write) and return its connection.
**txn = [txn](self)**
*> web_db_rollback txn=`txn`*
+67
View File
@@ -0,0 +1,67 @@
---
title: ext/data/form
description: Form Resource — fields, validate, submit (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/data: native web plugin not found
> sys.exit code=1
****
# form
+ `table`=None
+ `action`=insert
+ `id`=None
Field table + rules table; submit writes through `# db`.
> _load
*> web_form_new table=`table` action=`action` id=`id`*
## fields
+ `fields`
*> web_form_fields form=`self` fields=`fields`*
## rules
+ `rules`
*> web_form_rules form=`self` rules=`rules`*
## labels
+ `submit`="Submit"
+ `cancel`="cancel"
+ `cancel_href`=""
Localized submit / cancel copy for rendered forms.
*> web_form_labels form=`self` submit=`submit` cancel=`cancel` cancel_href=`cancel_href`*
## validate
+ `rules`=None
+ `data`
*> web_form_validate form=`self` rules=`rules` data=`data`*
## render
+ `id`=form
+ `data`=None
+ `errors`=None
*> web_form_render form=`self` id=`id` data=`data` errors=`errors`*
## submit
+ `data`
+ `db`
**url = [url](db)**
*> web_form_submit form=`self` data=`data` url=`url`*
+86
View File
@@ -0,0 +1,86 @@
---
title: ext/data/storage
description: Object storage Resource (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/data: native web plugin not found
> sys.exit code=1
****
# storage
+ `url`="file:data/blobs"
Object storage. `file:dir` stores blobs on disk (offline / gold). `s3://bucket?endpoint=http://127.0.0.1:9000&access_key=…&secret_key=…` talks to MinIO / S3.
> _load
*> web_storage_new url=`url`*
## put
+ `key`
+ `body`=None
+ `path`=None
+ `content_type`="application/octet-stream"
Provide either `body` (text) or `path` (local file to upload).
**url = [url](self)**
*> web_storage_put url=`url` key=`key` body=`body` path=`path` content_type=`content_type`*
## get
+ `key`
**url = [url](self)**
*> web_storage_get url=`url` key=`key`*
## delete
+ `key`
**url = [url](self)**
*> web_storage_delete url=`url` key=`key`*
## list
+ `prefix`=""
**url = [url](self)**
*> web_storage_list url=`url` prefix=`prefix`*
# media
+ `storage`=None
Offline helpers for upload validation and saving into `# storage` (also used by HTTP `app.upload`).
> _load
*> web_media_new storage=`storage`*
## validate
+ `filename`
+ `content_type`="application/octet-stream"
+ `size`
+ `max_bytes`=5242880
+ `types`=None
*> web_upload_validate filename=`filename` content_type=`content_type` size=`size` max_bytes=`max_bytes` types=`types`*
## save
+ `path`
+ `key`=None
+ `content_type`="application/octet-stream"
+ `prefix`=uploads/
+ `storage`=None
**st = storage**
1. `st` == None
**st = [storage](self)**
2. *
*> web_upload_save storage=`st` path=`path` key=`key` content_type=`content_type` prefix=`prefix`*
+31
View File
@@ -0,0 +1,31 @@
---
title: ext/net/http
description: HTTP primitives surface for Web Runtime (ADR 0007). Prefer Document/Endpoint Artifacts over raw calls.
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/net: native web plugin not found
> sys.exit code=1
****
## result
+ `ok`
+ `value`=None
+ `error`=None
Unified Result map `{ok, value, error}`.
`out` =
| ok | value | error |
|----|-------|-------|
| `ok` | `value` | `error` |
*out*
+9
View File
@@ -0,0 +1,9 @@
---
title: ext/net/url
description: URL helpers (stdlib host wrappers). ADR 0007.
---
## encode
+ `text`
*> host_url_encode text=`text`*
+41
View File
@@ -0,0 +1,41 @@
---
title: ext/net/websocket
description: WebSocket client Resource (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/net: native web plugin not found
> sys.exit code=1
****
# ws
+ `timeout_sec`=30
WebSocket client helper.
> _load
`out` =
| timeout_sec | _type |
|-------------|-------|
| `timeout_sec` | ws |
*out*
## connect
+ `url`
+ `message`=""
+ `headers`=None
Single request–response: connect to `url`, send `message`, collect up to 256 server text replies (each up to 1 MiB), close. Transport errors and timeouts return `{ok:false, error}` instead of looking like an empty successful reply.
**timeout = [timeout_sec](self)**
*> web_ws_connect url=`url` message=`message` headers=`headers` timeout_sec=`timeout`*
+58
View File
@@ -0,0 +1,58 @@
---
title: ext/security/auth
description: Auth Resource (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/security: native web plugin not found
> sys.exit code=1
****
# auth
+ `users`
+ `session_ttl`=3600
Session/auth helper. Constructs a config object; `login` validates against the users table. To gate `/admin` on this app, use `app.auth users=…` instead.
> _load
*> web_auth_new users=`users` session_ttl=`session_ttl`*
## login
+ `username`
+ `password`
Validate credentials against the users table and create a session. Returns `{ok, session_id, username, role}`.
**users = [users](self)**
**ttl = [session_ttl](self)**
*> web_auth_login username=`username` password=`password` users=`users` session_ttl=`ttl`*
## check
+ `session_id`
Returns `{ok, username, role}` when the session is valid.
*> web_auth_check session_id=`session_id`*
## logout
+ `session_id`
Destroy the session.
*> web_auth_logout session_id=`session_id`*
## hash_password
+ `password`
Hash a plaintext password for storage in admin user tables (argon2id). Store the returned `hash` in the `password` column; login verifies automatically.
*> web_password_hash password=`password`*
+41
View File
@@ -0,0 +1,41 @@
---
title: ext/security/oidc
description: OIDC attach helper for app handles (ADR 0007). Prefer declared auth metadata on Endpoints.
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/security: native web plugin not found
> sys.exit code=1
****
## attach
+ `app`
+ `issuer`=None
+ `client_id`
+ `client_secret`
+ `redirect_uri`
+ `redirect_origins`=None
+ `scopes`="openid profile email"
+ `callback_path`=None
+ `authorize_url`=None
+ `token_url`=None
+ `userinfo_url`=None
`redirect_uri` may be an absolute URL (`https://qdqc.com/oidc/callback`) or a path
(`/oidc/callback`). Path mode resolves the absolute callback from the login request
Host (`X-Forwarded-Proto` / `X-Forwarded-Host` when behind a reverse proxy) and stores
it in the session so token exchange uses the same string.
Optional `redirect_origins` is a comma-separated allowlist
(`https://qdqc.com,https://www.qdqc.com`). When set, path-mode origins must match.
The IdP must still register every absolute callback URL that will be used.
> _load
*> web_app_oidc app=`app` issuer=`issuer` client_id=`client_id` client_secret=`client_secret` redirect_uri=`redirect_uri` redirect_origins=`redirect_origins` scopes=`scopes` callback_path=`callback_path` authorize_url=`authorize_url` token_url=`token_url` userinfo_url=`userinfo_url`*
+61
View File
@@ -0,0 +1,61 @@
---
title: ext/security/rbac
description: RBAC Resource (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
import table:lib/table.mq.md
---
## _load
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/security: native web plugin not found
> sys.exit code=1
****
# rbac
Offline helpers for role ⊥ permission. Prefer `app.enable_rbac` + gates for HTTP; use these for scripts/tests.
## can
+ `permissions`
+ `needed`
Return `{ok, allowed}` — whether held permission CSV includes any of `needed` (CSV). Held `*` allows all.
> _load
*> web_rbac_can permissions=`permissions` needed=`needed`*
## assign_role
+ `url`
+ `username`
+ `role`
Bind `username` (in `web_users`) to role name.
> _load
*> web_rbac_assign_role url=`url` username=`username` role=`role`*
## create_role
+ `url`
+ `name`
Create a non-system role; returns `{ok, id, name}`.
> _load
*> web_rbac_create_role url=`url` name=`name`*
## set_role_permissions
+ `url`
+ `role`
+ `permissions`
+ `grantable`=""
Replace permissions on a non-system role (CSV). `grantable` restricts grants (anti-escalation); empty = full catalog.
> _load
*> web_rbac_set_role_permissions url=`url` role=`role` permissions=`permissions` grantable=`grantable`*
+17 -28
View File
@@ -1,31 +1,20 @@
# ext/web
# ext/web — Document / Endpoint / Resource
Official dynamic website extension: **GFM tables + class methods**.
Design: [`doc/design/ext-web.md`](../../doc/design/ext-web.md).
Official dynamic site extension (**ADR 0007**). Design: [`doc/design/ext-web-artifact.md`](../../doc/design/ext-web-artifact.md).
| Surface | Import | Classes |
|---------|--------|---------|
| English | `> ext/web/web.mq.md` | `page` · `db` · `app` · `form` · `style` |
| Chinese | `> ext/web/网页.mq.md` | `页面` · `数据库` · `应用` · `表单` · `样式` |
```bash
cargo build -p marqdo_plugin_web
# optional install into ~/.marqdo/ext
marqdo ext add web
```
Canonical sample (design §4): [`examples/web-site/`](../../examples/web-site/).
```bash
cargo run -- run examples/web-site/index.mq.md
# http://127.0.0.1:18081/
# /_form/article · /admin · /admin/articles/new
```
Offline smokes: `tests/ext/web-smoke.mq.md`, `web-form-smoke.mq.md`, `web-admin-smoke.mq.md`.
| Feature | Notes |
| Surface | Import |
|---------|--------|
| Form §5.5 | field/rules tables · `validate` · `submit` · `mount_form` → `GET\|POST /_form/{id}` |
| Admin CRUD | `admin=True` · schema → same form path · new/edit/delete under `/admin/{table}` |
| Route | `` `app`.route path=/about page=`about` `` → `GET /about` |
| English | `import web:ext/web/web.mq.md` |
| Chinese | `导入 网页:ext/web/网页.mq.md` |
**Facade:** `page` · `route` · `serve` · `render` · `inspect` · `use`
**Related modules:** `ext/data` (db/cache/storage) · `ext/security` (auth/oidc/rbac) · `ext/net` (http/websocket/url)
```bash
bash scripts/build-web-plugin.sh
marqdo ext add web # optional install
marqdo run examples/marqdo-blog/serve.mq.md
```
Hard rules: no author `ensure_plugin` / `compose_*` / `configure` junk drawer; View schema `|type|slot|value|attrs|style|` only; declared vs system routes via `inspect`.
+18
View File
@@ -0,0 +1,18 @@
---
title: ext/web/_capability
description: Internal — load native web plugin once. Authors must not call this; import web = capability (ADR 0007).
import plugin:lib/plugin.mq.md
import sys:lib/sys.mq.md
---
## load
Load ABI v2 `web` plugin if present. Fail loud when missing.
**p = > plugin.native_path name="web"**
1. `p`
> plugin.load path=`p`
2. *
> print text=ext/web: native web plugin not found (build plugins/web or marqdo ext add web)
> sys.exit code=1
****
+50
View File
@@ -0,0 +1,50 @@
---
title: ext/web/client
description: Browser Runtime helpers (WASM bridge). Prefer lib/browser for effect tables (ADR 0007).
import cap:ext/web/_capability.mq.md
import table:lib/table.mq.md
---
## embed
+ `bridge`="/static/marqdo-bridge.js"
+ `wasm`="/static/marqdo_wasm.wasm"
+ `source`=""
+ `boot`=True
HTML snippet that loads the official browser Marqdo bridge (route C/D). With `boot` and non-empty `source`, auto-mounts. No author JS.
> cap.load
1. `boot`
1. `source` != ""
*"<script type=\"module\" src=\"" + bridge + "\" data-mq-wasm=\"" + wasm + "\" data-mq-source-url=\"" + source + "\"></script>"*
2. *
*"<script type=\"module\" src=\"" + bridge + "\"></script>"*
2. *
*"<script type=\"module\" src=\"" + bridge + "\" data-mq-no-boot=\"1\"></script>"*
## text_patch
+ `sel`
+ `text`
`set_text` effects map. Prefer `lib/browser` in client `.mq.md`.
**`m` = > table.put in=None at=sel value=text**
*> table.put in=None at="set_text" value=m*
## dom_patch
+ `sel`
+ `html`=""
+ `attrs`=None
+ `remove`=False
**`spec` = > table.put in=None at="html" value=html**
1. `attrs`
**`spec` = > table.put in=`spec` at="attrs" value=attrs**
2. *
**_ = 1**
1. `remove`
**`spec` = > table.put in=`spec` at="remove" value=True**
2. *
**_ = 1**
**`m` = > table.put in=None at=sel value=spec**
*> table.put in=None at="dom" value=m*
+20
View File
@@ -0,0 +1,20 @@
---
title: ext/web/component
description: Reusable Document fragments as View node tables (ADR 0007).
import table:lib/table.mq.md
---
## nodes
+ `table`
Identity helper — a component is a View table `|type|slot|value|attrs|style|`.
*table*
## merge
+ `a`
+ `b`
Concatenate two View node lists (list-of-maps or column tables). Prefer authoring one table.
*> table.concat a=`a` b=`b`*
+38
View File
@@ -0,0 +1,38 @@
---
title: ext/web/dom
description: DOM Artifact — Markdown/body to structure; raw HTML is escape hatch only (ADR 0007).
import cap:ext/web/_capability.mq.md
import table:lib/table.mq.md
---
## from_markdown
+ `text`
Turn Markdown text into a simple DOM map `{kind:"markdown", text}` for `page.render` / `web.render`.
> cap.load
**`out` = > table.put in=None at="kind" value="markdown"**
*> table.put in=`out` at="text" value=text*
## raw
+ `html`
Escape hatch — wrap a raw HTML string. Prefer Markdown / View nodes.
> cap.load
**`out` = > table.put in=None at="kind" value="raw"**
*> table.put in=`out` at="html" value=html*
## render
+ `dom`
Render a DOM Artifact (markdown or raw) to HTML via native plugin.
> cap.load
**kind = [kind](dom)**
1. `kind` == "raw"
**html = [html](dom)**
*html*
2. *
**text = [text](dom)**
*> web_dom_markdown text=`text`*
+54
View File
@@ -0,0 +1,54 @@
---
title: ext/web/page
description: Document → View. Sole compose path is render(nodes=) with unified View schema (ADR 0007).
import cap:ext/web/_capability.mq.md
import table:lib/table.mq.md
---
# page
+ `title`=""
+ `route`="/"
+ `method`="GET"
+ `body`=""
+ `nodes`=None
+ `data_source`=""
+ `data_order`=""
+ `data_where`=""
Web Document handle. Prefer constructing from Artifact Metadata via `web.page` facade.
View tables use only `|type|slot|value|attrs|style|` (ZH `|类型|槽|值|属性|样式|`).
> cap.load
`out` =
| _type | title | route | method | body | nodes | data_source | data_order | data_where |
|-------|-------|-------|--------|------|-------|-------------|------------|------------|
| page | `title` | `route` | `method` | `body` | `nodes` | `data_source` | `data_order` | `data_where` |
*out*
## render
+ `nodes`=None
+ `db`=None
Render Document to HTML. `nodes` is the unified View table; when omitted, uses `self.nodes` or Markdown `body`.
> cap.load
**n = nodes**
1. `n` == None
**n = [nodes](self)**
2. *
**_ = 1**
1. `n`
*> web_render_nodes page=`self` nodes=`n` db=`db`*
2. *
**body = [body](self)**
*> web_render_document page=`self` body=`body` db=`db`*
## css
+ `css`
Escape hatch — append raw CSS (not a second primary language).
> cap.load
*> web_page_css page=`self` css=`css`*
+60
View File
@@ -0,0 +1,60 @@
---
title: ext/web/route
description: Declared Endpoint / Document route registration (ADR 0007).
import cap:ext/web/_capability.mq.md
import table:lib/table.mq.md
---
# route
+ `app`
+ `path`
+ `method`="GET"
+ `page`=None
+ `fn`=""
+ `request`="json"
+ `response`="json"
+ `auth`=""
Declared route handle. Prefer `web.route` / Artifact `type: endpoint` discovery via `web.serve`.
> cap.load
`out` =
| _type | app | path | method | page | fn | request | response | auth |
|-------|-----|------|--------|------|----|---------|----------|------|
| route | `app` | `path` | `method` | `page` | `fn` | `request` | `response` | `auth` |
*out*
## use
+ `middleware`
Attach middleware table to this route (`|header|value|` or named capability rows).
> cap.load
*> web_route_use route=`self` middleware=`middleware`*
## register
+ `app`=None
Register this declared route on an app handle.
> cap.load
**a = app**
1. `a` == None
**a = [app](self)**
2. *
**_ = 1**
**page = [page](self)**
**fn = [fn](self)**
1. `page`
**path = [path](self)**
*> web_app_route app=`a` path=`path` page=`page`*
2. `fn`
**path = [path](self)**
**method = [method](self)**
**request = [request](self)**
**response = [response](self)**
*> web_app_invoke app=`a` path=`path` method=`method` fn=`fn` body=`request` return=`response`*
3. *
*> table.put in=None at="ok" value=False*
+94 -1091
View File
File diff suppressed because it is too large. Load diff
+47 -1113
View File
File diff suppressed because it is too large. Load diff
+6
View File
@@ -52,6 +52,12 @@ Lookup one metadata key; missing → None.
*> host_meta_get key=`key`*
## artifact_kind
Classify entry Artifact Metadata: `web` / `endpoint` / `prompt` / `other`, or None if no `type`/`类型`. ADR 0007.
*> host_artifact_kind*
## module_source
Entry `.mq.md` source text (for document-as-prompt). Empty string if unset.
+6
View File
@@ -52,6 +52,12 @@ description: 中文系统包装(加载环境 可选 路径=)
*> host_meta_get key=`键`*
## 工件种类
入口 Artifact 分类:`web` / `endpoint` / `prompt` / `other`;无 `type`/`类型` → None。ADR 0007。
*> host_artifact_kind*
## 模块源码
入口 `.mq.md` 全文(供 document-as-prompt)。未设置时为空文本。
+497
View File
@@ -0,0 +1,497 @@
//! Engineering Knowledge preflight / reuse (reads `.marqdo` projections).
//! Does not call host_* — consumes compiled knowledge graph artifacts.
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
use serde_json::{json, Value};
fn arg_text<'a>(args: &'a Value, key: &str) -> Result<&'a str, String> {
args.get(key)
.and_then(|v| v.as_str())
.ok_or_else(|| format!("missing string arg `{key}`"))
}
fn arg_f64(args: &Value, key: &str, default: f64) -> f64 {
match args.get(key) {
Some(Value::Number(n)) => n.as_f64().unwrap_or(default),
Some(Value::String(s)) => s.parse().unwrap_or(default),
_ => default,
}
}
fn arg_usize(args: &Value, key: &str, default: usize) -> usize {
match args.get(key) {
Some(Value::Number(n)) => n.as_u64().unwrap_or(default as u64) as usize,
Some(Value::String(s)) => s.parse().unwrap_or(default),
_ => default,
}
}
fn char_bigram_tf(text: &str) -> HashMap<String, f64> {
let chars: Vec<char> = text.chars().filter(|c| !c.is_whitespace()).collect();
let mut m = HashMap::new();
if chars.is_empty() {
return m;
}
if chars.len() == 1 {
m.insert(chars[0].to_string(), 1.0);
return m;
}
for w in chars.windows(2) {
let key: String = w.iter().collect();
*m.entry(key).or_insert(0.0) += 1.0;
}
m
}
fn cosine_tf(a: &HashMap<String, f64>, b: &HashMap<String, f64>) -> f64 {
let mut na = 0.0;
let mut nb = 0.0;
for v in a.values() {
na += v * v;
}
for v in b.values() {
nb += v * v;
}
if na == 0.0 || nb == 0.0 {
return 0.0;
}
let mut dot = 0.0;
for (k, va) in a {
if let Some(vb) = b.get(k) {
dot += va * vb;
}
}
dot / (na.sqrt() * nb.sqrt())
}
fn resolve_marqdo_root(start: &str) -> PathBuf {
let mut cur = PathBuf::from(start);
if cur.is_file() {
if let Some(p) = cur.parent() {
cur = p.to_path_buf();
}
}
for _ in 0..8 {
let cand = cur.join(".marqdo");
if cand.join("engineering.yaml").exists() || cand.join("graph/graph.json").exists() {
return cand;
}
if !cur.pop() {
break;
}
}
PathBuf::from(start).join(".marqdo")
}
#[derive(Clone)]
struct CapHit {
id: String,
name: String,
score: f64,
implemented_by: Vec<String>,
}
fn load_capabilities(marqdo: &Path) -> Vec<(String, String, Vec<String>, String)> {
// returns (id, name, implemented_by, material)
let mut out = Vec::new();
let eng = marqdo.join("engineering.yaml");
if let Ok(raw) = fs::read_to_string(&eng) {
let mut cur_id = String::new();
let mut cur_name = String::new();
let mut impls = Vec::new();
let mut in_caps = false;
let mut in_impl = false;
for line in raw.lines() {
let t = line.trim_end();
if t.trim() == "capabilities:" {
in_caps = true;
continue;
}
if in_caps && !line.starts_with(' ') && !line.starts_with('\t') && !t.trim().is_empty()
{
if t.trim() == "knowledge:" {
if !cur_id.is_empty() {
out.push((
cur_id.clone(),
cur_name.clone(),
impls.clone(),
format!("{cur_name} {cur_id}"),
));
}
break;
}
}
if !in_caps {
continue;
}
let trimmed = t.trim();
if let Some(rest) = trimmed.strip_prefix("- id:") {
if !cur_id.is_empty() {
out.push((
cur_id.clone(),
cur_name.clone(),
impls.clone(),
format!("{cur_name} {cur_id}"),
));
}
cur_id = rest.trim().trim_matches('"').to_string();
cur_name.clear();
impls.clear();
in_impl = false;
} else if let Some(rest) = trimmed.strip_prefix("name:") {
cur_name = rest.trim().trim_matches('"').to_string();
} else if trimmed == "implemented_by:" {
in_impl = true;
} else if in_impl {
if let Some(rest) = trimmed.strip_prefix("- ") {
impls.push(rest.trim().trim_matches('"').to_string());
} else if !trimmed.is_empty() && !line.starts_with(' ') {
in_impl = false;
}
}
}
if !cur_id.is_empty() {
out.push((
cur_id.clone(),
cur_name.clone(),
impls,
format!("{cur_name} {cur_id}"),
));
}
}
// Also scan catalog/capabilities/*.md
let cap_dir = marqdo.join("catalog/capabilities");
if let Ok(rd) = fs::read_dir(cap_dir) {
for ent in rd.flatten() {
let p = ent.path();
if !p
.file_name()
.and_then(|s| s.to_str())
.map(|s| s.ends_with(".md"))
.unwrap_or(false)
{
continue;
}
let Ok(src) = fs::read_to_string(&p) else {
continue;
};
let id = fm_field(&src, "id").unwrap_or_else(|| {
p.file_stem()
.and_then(|s| s.to_str())
.unwrap_or("cap")
.to_string()
});
let name = fm_field(&src, "name").unwrap_or_else(|| id.clone());
if out.iter().any(|(i, _, _, _)| i == &id) {
continue;
}
out.push((id.clone(), name.clone(), Vec::new(), format!("{name} {id} {src}")));
}
}
out
}
fn fm_field(source: &str, key: &str) -> Option<String> {
let prefix = format!("{key}:");
for line in source.lines() {
let t = line.trim();
if t == "---" {
continue;
}
if let Some(rest) = t.strip_prefix(&prefix) {
return Some(rest.trim().trim_matches('"').to_string());
}
}
None
}
fn load_knowledge_titles(marqdo: &Path, kind_subdir: &str) -> Vec<(String, String)> {
let dir = marqdo.join("knowledge").join(kind_subdir);
let mut out = Vec::new();
let Ok(rd) = fs::read_dir(dir) else {
return out;
};
for ent in rd.flatten() {
let p = ent.path();
let Ok(src) = fs::read_to_string(&p) else {
continue;
};
let id = fm_field(&src, "id").unwrap_or_else(|| {
p.file_stem()
.and_then(|s| s.to_str())
.unwrap_or("k")
.to_string()
});
let title = fm_field(&src, "title").unwrap_or_else(|| id.clone());
out.push((id, title));
}
out
}
fn rank_caps(task: &str, caps: &[(String, String, Vec<String>, String)]) -> Vec<CapHit> {
let qv = char_bigram_tf(task);
let mut hits = Vec::new();
for (id, name, impls, material) in caps {
let mut score = cosine_tf(&qv, &char_bigram_tf(material));
if name.to_ascii_lowercase().contains(&task.to_ascii_lowercase())
|| task
.to_ascii_lowercase()
.split_whitespace()
.any(|t| t.len() > 2 && name.to_ascii_lowercase().contains(t))
{
score = (score + 0.35).min(1.0);
}
if score > 0.12 {
hits.push(CapHit {
id: id.clone(),
name: name.clone(),
score,
implemented_by: impls.clone(),
});
}
}
hits.sort_by(|a, b| {
b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
});
hits
}
fn relevant_knowledge(task: &str, items: &[(String, String)], cap_names: &[String]) -> Vec<String> {
let task_l = task.to_ascii_lowercase();
items
.iter()
.filter(|(id, title)| {
let m = format!("{id} {title}").to_ascii_lowercase();
task_l.split_whitespace().any(|t| t.len() > 3 && m.contains(t))
|| cap_names.iter().any(|c| m.contains(&c.to_ascii_lowercase()))
})
.map(|(id, title)| format!("{title} ({id})"))
.collect()
}
/// `agent_eng_preflight` — Engineering First Information gate.
pub fn eng_preflight(args: &Value) -> Result<Value, String> {
let task = arg_text(args, "task").or_else(|_| arg_text(args, "goal"))?;
let root = args
.get("root")
.and_then(|v| v.as_str())
.unwrap_or(".");
let marqdo = args
.get("marqdo_dir")
.and_then(|v| v.as_str())
.map(PathBuf::from)
.unwrap_or_else(|| resolve_marqdo_root(root));
if !marqdo.join("engineering.yaml").exists() && !marqdo.join("graph/graph.json").exists() {
return Ok(json!({
"status": "missing_knowledge",
"create_allowed": false,
"decision": "CREATE",
"error": "run `marqdo knowledge` first",
"existing": [],
"recommended": [],
}));
}
let reuse_threshold = arg_f64(args, "reuse_threshold", 0.72);
let adapt_threshold = arg_f64(args, "adapt_threshold", 0.45);
let allow_create_after = arg_usize(args, "allow_create_after", 2);
let min_candidates = arg_usize(args, "min_candidates", 3);
let caps = load_capabilities(&marqdo);
let ranked = rank_caps(task, &caps);
let existing: Vec<String> = ranked.iter().take(8).map(|c| c.name.clone()).collect();
let cap_names = existing.clone();
let constraints = relevant_knowledge(
task,
&load_knowledge_titles(&marqdo, "constraints"),
&cap_names,
);
let decisions = relevant_knowledge(
task,
&load_knowledge_titles(&marqdo, "decisions"),
&cap_names,
);
let failures = relevant_knowledge(
task,
&load_knowledge_titles(&marqdo, "failures"),
&cap_names,
);
let best = ranked.first();
let best_score = best.map(|c| c.score).unwrap_or(0.0);
let analyzed = ranked.len().max(1).min(min_candidates.max(allow_create_after));
let (decision, create_allowed, recommended, reason) = if best_score >= reuse_threshold {
let b = best.unwrap();
(
"REUSE",
false,
vec![format!("REUSE {}", b.name)],
vec![
"score meets REUSE threshold".into(),
"same capability semantics".into(),
],
)
} else if best_score >= adapt_threshold {
let b = best.unwrap();
(
"ADAPT",
false,
vec![format!("ADAPT {}", b.name)],
vec!["near match — wrap/compose/extend".into()],
)
} else if analyzed >= allow_create_after {
(
"CREATE",
true,
vec!["CREATE new capability".into()],
vec!["no existing capability satisfies constraints".into()],
)
} else {
(
"CREATE",
false,
vec![],
vec![format!(
"insufficient candidates ({analyzed} < {allow_create_after})"
)],
)
};
// Write context pack
let contexts = marqdo.join("agent/contexts");
let _ = fs::create_dir_all(&contexts);
let mut h: u64 = 5381;
for b in task.as_bytes() {
h = h.wrapping_mul(33).wrapping_add(u64::from(*b));
}
let tid = format!("{h:016x}")[..12].to_string();
let ctx_path = contexts.join(format!("task-{tid}.mq.md"));
let pack = format!(
"---\ntype: Marqdo Context Pack\ntitle: Engineering Context\ndecision: {decision}\ncreate_allowed: {create_allowed}\n---\n\n# Engineering Context\n\n## Task\n\n{task}\n\n## Existing Capabilities\n\n{existing}\n\n## Recommended Reuse\n\n{recommended}\n",
existing = existing
.iter()
.map(|e| format!("- {e}"))
.collect::<Vec<_>>()
.join("\n"),
recommended = recommended
.iter()
.enumerate()
.map(|(i, r)| format!("{}. {r}", i + 1))
.collect::<Vec<_>>()
.join("\n"),
);
let _ = fs::write(&ctx_path, pack);
let status = if create_allowed {
"ready_create"
} else if decision == "REUSE" || decision == "ADAPT" {
"ready"
} else if decision == "CREATE" && !create_allowed {
"blocked"
} else {
"ready"
};
Ok(json!({
"status": status,
"intent": { "capability": best.map(|c| c.name.clone()).unwrap_or_else(|| task.to_string()) },
"existing": existing,
"constraints": constraints,
"decisions": decisions,
"failures": failures,
"recommended": recommended,
"create_allowed": create_allowed,
"decision": decision,
"confidence": best_score,
"reason": reason,
"context_path": ctx_path.display().to_string(),
"candidates": ranked.iter().take(8).map(|c| json!({
"id": c.id,
"name": c.name,
"score": c.score,
"implemented_by": c.implemented_by,
})).collect::<Vec<_>>(),
}))
}
/// `agent_eng_reuse` — alias of preflight focused on decision payload.
pub fn eng_reuse(args: &Value) -> Result<Value, String> {
let mut pf = eng_preflight(args)?;
if let Some(obj) = pf.as_object_mut() {
obj.insert(
"protocol".into(),
json!("REUSE|ADAPT|CREATE"),
);
}
Ok(pf)
}
/// `agent_eng_record` — append reuse metrics under `.marqdo/agent/episodes`.
pub fn eng_record(args: &Value) -> Result<Value, String> {
let decision = arg_text(args, "decision")?;
let root = args
.get("root")
.and_then(|v| v.as_str())
.unwrap_or(".");
let marqdo = args
.get("marqdo_dir")
.and_then(|v| v.as_str())
.map(PathBuf::from)
.unwrap_or_else(|| resolve_marqdo_root(root));
let duplicated = matches!(args.get("duplicated"), Some(Value::Bool(true)));
let path = marqdo.join("agent/episodes/reuse_metrics.json");
if let Some(p) = path.parent() {
let _ = fs::create_dir_all(p);
}
let mut m = if let Ok(raw) = fs::read_to_string(&path) {
serde_json::from_str::<Value>(&raw).unwrap_or(json!({}))
} else {
json!({})
};
let get_u = |v: &Value, k: &str| v.get(k).and_then(|x| x.as_u64()).unwrap_or(0);
let mut reused = get_u(&m, "reused");
let mut adapted = get_u(&m, "adapted");
let mut created = get_u(&m, "created");
let mut dups = get_u(&m, "duplicated");
match decision.to_ascii_uppercase().as_str() {
"REUSE" => reused += 1,
"ADAPT" => adapted += 1,
"CREATE" => {
created += 1;
if duplicated {
dups += 1;
}
}
_ => {}
}
m = json!({
"generated_functions": get_u(&m, "generated_functions") + 1,
"reused": reused,
"adapted": adapted,
"created": created,
"duplicated": dups,
});
let gen = m["generated_functions"].as_u64().unwrap_or(1) as f64;
m["reuse_ratio"] = json!(reused as f64 / gen);
m["adapt_ratio"] = json!(adapted as f64 / gen);
m["novel_ratio"] = json!(created as f64 / gen);
m["duplication_rate"] = json!(if created == 0 {
0.0
} else {
dups as f64 / created as f64
});
fs::write(&path, serde_json::to_string_pretty(&m).unwrap_or_default())
.map_err(|e| e.to_string())?;
Ok(m)
}
+25
View File
@@ -1,6 +1,7 @@
//! Agent plugin (C ABI v2): layout helpers + session bag + context via host_query + agent-kb + v2 memory + adaptive route.
mod corpus;
mod eng;
mod kb;
mod mcp_client;
mod mcp_server;
@@ -930,6 +931,9 @@ kb_ffi!(agent_resume_save, resume::resume_save, "agent_resume_save");
kb_ffi!(agent_resume_load, resume::resume_load, "agent_resume_load");
kb_ffi!(agent_resume_clear, resume::resume_clear, "agent_resume_clear");
kb_ffi!(agent_resume_list, resume::resume_list, "agent_resume_list");
kb_ffi!(agent_eng_preflight, eng::eng_preflight, "agent_eng_preflight");
kb_ffi!(agent_eng_reuse, eng::eng_reuse, "agent_eng_reuse");
kb_ffi!(agent_eng_record, eng::eng_record, "agent_eng_record");
#[no_mangle]
pub unsafe extern "C" fn marqdo_plugin_abi_version() -> u32 {
@@ -1155,6 +1159,27 @@ pub unsafe extern "C" fn marqdo_plugin_init(host: *const MarqdoHostApi) -> c_int
{
return 1;
}
if register(
host,
"agent_eng_preflight",
"task",
agent_eng_preflight,
) != 0
{
return 1;
}
if register(host, "agent_eng_reuse", "task", agent_eng_reuse) != 0 {
return 1;
}
if register(
host,
"agent_eng_record",
"decision",
agent_eng_record,
) != 0
{
return 1;
}
0
}
+326
View File
@@ -0,0 +1,326 @@
package main
/*
#cgo CFLAGS: -I${SRCDIR}/abi
#include "marqdo_abi_cgo.h"
extern int web_render_nodes(char *args_json, char **out_json, char **err_msg);
extern int web_render_document(char *args_json, char **out_json, char **err_msg);
extern int web_dom_markdown(char *args_json, char **out_json, char **err_msg);
extern int web_serve_root(char *args_json, char **out_json, char **err_msg);
extern int web_inspect(char *args_json, char **out_json, char **err_msg);
extern int web_app_use(char *args_json, char **out_json, char **err_msg);
extern int web_route_use(char *args_json, char **out_json, char **err_msg);
*/
import "C"
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"github.com/marqdo/marqdo/plugins/web/internal/app"
"github.com/marqdo/marqdo/plugins/web/internal/artifact"
"github.com/marqdo/marqdo/plugins/web/internal/httpx"
"github.com/marqdo/marqdo/plugins/web/internal/middleware"
"github.com/marqdo/marqdo/plugins/web/internal/page"
)
//export web_render_nodes
func web_render_nodes(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
frag, err := artifact.RenderNodes(args["nodes"])
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
pg := asPageMap(args["page"])
title := ""
if pg != nil {
if t, ok := pg["title"].(string); ok {
title = t
}
}
if t, ok := argStr(args, "title"); ok && t != "" {
title = t
}
return replyJSON(outJSON, errMsg, artifact.RenderHTMLPage(title, frag), nil)
}
//export web_render_document
func web_render_document(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
pg := asPageMap(args["page"])
title := ""
body := ""
if pg != nil {
if t, ok := pg["title"].(string); ok {
title = t
}
if b, ok := pg["body"].(string); ok {
body = b
}
}
if b, ok := argStr(args, "body"); ok && b != "" {
body = b
}
if t, ok := argStr(args, "title"); ok && t != "" {
title = t
}
return replyJSON(outJSON, errMsg, artifact.RenderDocument(title, body), nil)
}
//export web_dom_markdown
func web_dom_markdown(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
text, _ := argStr(args, "text")
return replyJSON(outJSON, errMsg, artifact.MarkdownHTML(text), nil)
}
//export web_serve_root
func web_serve_root(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
root, _ := argStr(args, "root")
if root == "" {
root = "."
}
abs, err := filepath.Abs(root)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
if st, err := os.Stat(abs); err != nil || !st.IsDir() {
return replyJSON(outJSON, errMsg, nil, fmt.Errorf("web.serve: root not a directory: %s", abs))
}
files, err := artifact.ScanRoot(abs)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
host, _ := argStr(args, "host")
if host == "" {
host = "127.0.0.1"
}
port := float64(18081)
if v, ok := args["port"]; ok && v != nil {
switch t := v.(type) {
case float64:
port = t
case string:
if n, e := strconv.Atoi(t); e == nil {
port = float64(n)
}
}
}
homeArgs := map[string]any{"title": "Marqdo"}
var homeMeta map[string]string
for _, f := range files {
if f.Kind == "web" && (f.Route == "/" || f.Rel == "index.mq.md") {
title := f.Meta["title"]
if title == "" {
title = f.Meta["标题"]
}
if title == "" {
title = "Marqdo"
}
homeArgs = map[string]any{
"title": title,
"intro": artifact.MarkdownHTML(f.Body),
}
homeMeta = f.Meta
break
}
}
home := page.New(homeArgs)
stampArtifactData(home, homeMeta)
a := app.New(map[string]any{
"page": home,
"db": args["db"],
"host": host,
"port": port,
})
artifactRoutes := map[string]any{}
for _, f := range files {
if f.Kind == "web" {
if f.Route == "/" {
continue
}
title := f.Meta["title"]
if title == "" {
title = f.Meta["标题"]
}
pg := page.New(map[string]any{
"title": title,
"intro": artifact.MarkdownHTML(f.Body),
})
stampArtifactData(pg, f.Meta)
out, err := app.Route(a, f.Route, pg)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
a = out
continue
}
// endpoint
req := f.Meta["request"]
if req == "" {
req = f.Meta["请求"]
}
if req == "" {
req = "json"
}
resp := f.Meta["response"]
if resp == "" {
resp = f.Meta["响应"]
}
if resp == "" {
resp = "json"
}
auth := f.Meta["auth"]
if auth == "" {
auth = f.Meta["鉴权"]
}
key := f.Method + " " + f.Route
artifactRoutes[key] = map[string]any{
"method": f.Method,
"path": f.Route,
"file": f.Path,
"request": req,
"response": resp,
"auth": auth,
"kind": "endpoint",
}
}
if len(artifactRoutes) > 0 {
a["artifact_routes"] = artifactRoutes
}
if mw := args["middleware"]; mw != nil {
out, err := middleware.Configure(a, map[string]any{"security": mw})
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
a = out
}
if sd, ok := argStr(args, "static_dir"); ok && sd != "" {
out, err := app.Static(a, sd, "/static")
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
a = out
}
a["inspect"] = artifact.InspectBag(a, files)
_, err = httpx.Listen(a, abs)
return replyJSON(outJSON, errMsg, map[string]any{"ok": true, "value": a["inspect"], "error": nil}, err)
}
//export web_inspect
func web_inspect(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
a := asPageMap(args["app"])
if a != nil {
if insp, ok := a["inspect"].(map[string]any); ok {
return replyJSON(outJSON, errMsg, insp, nil)
}
}
return replyJSON(outJSON, errMsg, artifact.InspectBag(a, nil), nil)
}
//export web_app_use
func web_app_use(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
a := asPageMap(args["app"])
out, err := middleware.Configure(a, map[string]any{"security": args["middleware"]})
return replyJSON(outJSON, errMsg, out, err)
}
//export web_route_use
func web_route_use(argsJSON *C.char, outJSON **C.char, errMsg **C.char) C.int {
args, err := parseArgs(argsJSON)
if err != nil {
return replyJSON(outJSON, errMsg, nil, err)
}
rt := asPageMap(args["route"])
if rt == nil {
rt = map[string]any{}
}
rt["middleware"] = args["middleware"]
return replyJSON(outJSON, errMsg, rt, nil)
}
// stampArtifactData applies flat Document metadata (ADR 0007) onto a page bag.
func stampArtifactData(pg map[string]any, meta map[string]string) {
if pg == nil || meta == nil {
return
}
src := meta["data_source"]
if src == "" {
src = meta["数据源"]
}
if src != "" {
pg["data_source"] = src
}
order := meta["data_order"]
if order == "" {
order = meta["排序"]
}
if order != "" {
pg["order"] = order
}
where := meta["data_where"]
if where == "" {
where = meta["条件"]
}
if where != "" {
// col=val,col2=val2 → query map (values may contain {param})
q := map[string]any{}
for _, part := range strings.Split(where, ",") {
part = strings.TrimSpace(part)
if part == "" {
continue
}
k, v, ok := strings.Cut(part, "=")
if !ok {
continue
}
q[strings.TrimSpace(k)] = strings.TrimSpace(v)
}
if len(q) > 0 {
pg["query"] = q
// Filtered Document routes render as article detail (007 /post/{slug}).
pg["detail"] = true
}
}
if d := meta["detail"]; d != "" {
pg["detail"] = d == "true" || d == "True" || d == "1"
}
if d := meta["详情"]; d != "" {
pg["detail"] = d == "true" || d == "True" || d == "1" || d == "是"
}
prefix := meta["link_prefix"]
if prefix == "" {
prefix = meta["链接前缀"]
}
if prefix != "" {
pg["link_prefix"] = prefix
} else if src != "" && where == "" {
// List pages default card links to /post/{slug}.
pg["link_prefix"] = "/post/"
}
}
+3
View File
@@ -256,6 +256,9 @@ func OIDC(appBag map[string]any, opts map[string]any) (map[string]any, error) {
"scopes": cfg.ScopesCSV(),
"callback_path": cfg.Callback(),
}
if cfg.RedirectOrigins != "" {
bag["redirect_origins"] = cfg.RedirectOrigins
}
if cfg.AuthorizeURL != "" {
bag["authorize_url"] = cfg.AuthorizeURL
}
+298
View File
@@ -0,0 +1,298 @@
// Package artifact implements Document/Endpoint View render, serve scan, and inspect (ADR 0007).
package artifact
import (
"fmt"
"html"
"os"
"path/filepath"
"strings"
"github.com/marqdo/marqdo/plugins/web/internal/render"
"github.com/marqdo/marqdo/plugins/web/internal/table"
)
// MetaType returns canonical kind: web | endpoint | "" from flat frontmatter map.
func MetaType(meta map[string]string) string {
if v := strings.TrimSpace(meta["type"]); v != "" {
return normalizeType(v)
}
if v := strings.TrimSpace(meta["类型"]); v != "" {
return normalizeType(v)
}
return ""
}
func normalizeType(v string) string {
switch v {
case "web", "网页":
return "web"
case "endpoint", "端点":
return "endpoint"
default:
return v
}
}
// ParseFrontmatter reads flat key: value pairs from a .mq.md file head.
func ParseFrontmatter(src string) (meta map[string]string, body string) {
meta = map[string]string{}
s := strings.TrimPrefix(src, "\ufeff")
if !strings.HasPrefix(s, "---\n") && !strings.HasPrefix(s, "---\r\n") {
return meta, src
}
rest := s[4:]
if strings.HasPrefix(rest, "\r\n") {
rest = rest[2:]
} else if strings.HasPrefix(rest, "\n") {
rest = rest[1:]
}
end := strings.Index(rest, "\n---\n")
if end < 0 {
end = strings.Index(rest, "\r\n---\r\n")
}
if end < 0 {
end = strings.Index(rest, "\n---\r\n")
}
if end < 0 {
return meta, src
}
fm := rest[:end]
body = rest[end:]
if i := strings.Index(body, "---"); i >= 0 {
body = body[i+3:]
body = strings.TrimPrefix(body, "\r")
body = strings.TrimPrefix(body, "\n")
}
for _, line := range strings.Split(fm, "\n") {
line = strings.TrimSpace(strings.TrimSuffix(line, "\r"))
if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, "import ") || strings.HasPrefix(line, "导入 ") {
continue
}
k, v, ok := strings.Cut(line, ":")
if !ok {
continue
}
k = strings.TrimSpace(k)
v = strings.TrimSpace(v)
if len(v) >= 2 {
if (v[0] == '"' && v[len(v)-1] == '"') || (v[0] == '\'' && v[len(v)-1] == '\'') {
v = v[1 : len(v)-1]
}
}
if k != "" {
meta[k] = v
}
}
return meta, body
}
// RenderNodes turns a unified View table into an HTML fragment.
// Columns: type|slot|value|attrs|style (ZH: 类型|槽|值|属性|样式).
func RenderNodes(nodes any) (string, error) {
raw := table.AsRows(nodes)
list, _ := raw.([]any)
var b strings.Builder
b.WriteString(`<div class="mq-view">`)
for _, item := range list {
row, ok := item.(map[string]any)
if !ok {
continue
}
ty := first(row, "type", "类型")
slot := first(row, "slot", "槽")
val := first(row, "value", "值")
attrs := first(row, "attrs", "属性")
style := first(row, "style", "样式")
class := "mq-node"
if style != "" {
class += " " + html.EscapeString(style)
}
if slot != "" {
class += " slot-" + html.EscapeString(slot)
}
switch strings.ToLower(ty) {
case "title", "标题":
b.WriteString(`<h1 class="` + class + `">` + inline(val) + `</h1>`)
case "link", "链接":
href := attrs
if href == "" {
href = "#"
}
b.WriteString(`<a class="` + class + `" href="` + html.EscapeString(href) + `">` + inline(val) + `</a>`)
case "html", "原文":
b.WriteString(`<div class="` + class + `">` + val + `</div>`)
default: // text
b.WriteString(`<p class="` + class + `">` + inline(val) + `</p>`)
}
}
b.WriteString(`</div>`)
return b.String(), nil
}
func inline(s string) string {
return html.EscapeString(s)
}
func first(row map[string]any, keys ...string) string {
for _, k := range keys {
if v, ok := row[k]; ok && v != nil {
return fmt.Sprint(v)
}
}
return ""
}
// RenderDocument builds a minimal HTML page from title + markdown body.
func RenderDocument(title, body string) string {
return RenderHTMLPage(title, render.MarkdownToHTML(body))
}
// RenderHTMLPage wraps an HTML fragment in a minimal document shell.
func RenderHTMLPage(title, innerHTML string) string {
t := html.EscapeString(title)
if t == "" {
t = "Marqdo"
}
return "<!DOCTYPE html><html><head><meta charset=\"utf-8\"><title>" + t +
"</title></head><body class=\"mq-document\"><main>" + innerHTML + "</main></body></html>"
}
// MarkdownHTML exposes render helper.
func MarkdownHTML(md string) string {
return render.MarkdownToHTML(md)
}
// ScanRoot finds Artifact .mq.md files under root.
func ScanRoot(root string) ([]ArtifactFile, error) {
var out []ArtifactFile
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
name := d.Name()
if d.IsDir() {
if name == "data" || name == ".git" || name == "node_modules" || strings.HasPrefix(name, ".") {
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(name, ".mq.md") {
return nil
}
raw, err := os.ReadFile(path)
if err != nil {
return err
}
meta, body := ParseFrontmatter(string(raw))
kind := MetaType(meta)
if kind != "web" && kind != "endpoint" {
return nil
}
rel, _ := filepath.Rel(root, path)
af := ArtifactFile{Path: path, Rel: rel, Kind: kind, Meta: meta, Body: body}
if kind == "web" {
af.Route = metaString(meta, "route", "路由")
if af.Route == "" {
af.Route = routeFromRel(rel)
}
af.Method = metaString(meta, "method", "方法")
if af.Method == "" {
af.Method = "GET"
}
} else {
af.Route = metaString(meta, "path", "路径")
af.Method = metaString(meta, "method", "方法")
if af.Method == "" {
af.Method = "GET"
}
if af.Route == "" {
return fmt.Errorf("endpoint %s: missing path/路径", rel)
}
}
out = append(out, af)
return nil
})
return out, err
}
// ArtifactFile is one discovered Document or Endpoint.
type ArtifactFile struct {
Path string
Rel string
Kind string
Route string
Method string
Meta map[string]string
Body string
}
func metaString(meta map[string]string, keys ...string) string {
for _, k := range keys {
if v := strings.TrimSpace(meta[k]); v != "" {
return v
}
}
return ""
}
func routeFromRel(rel string) string {
rel = filepath.ToSlash(rel)
if rel == "index.mq.md" {
return "/"
}
rel = strings.TrimSuffix(rel, ".mq.md")
rel = strings.TrimSuffix(rel, "/index")
if !strings.HasPrefix(rel, "/") {
rel = "/" + rel
}
return rel
}
// SystemRoutes lists runtime-owned routes for inspect.
func SystemRoutes() []map[string]any {
return []map[string]any{
{"method": "GET", "path": "/_part/*", "kind": "system"},
{"method": "GET|POST", "path": "/_form/*", "kind": "system"},
}
}
// InspectBag builds {declared, system} from an app bag and/or scanned files.
func InspectBag(app map[string]any, declared []ArtifactFile) map[string]any {
var dec []map[string]any
for _, a := range declared {
dec = append(dec, map[string]any{
"method": a.Method,
"path": a.Route,
"kind": a.Kind,
"file": a.Rel,
})
}
if app != nil {
if routes, ok := app["routes"].(map[string]any); ok {
for path := range routes {
dec = append(dec, map[string]any{"method": "GET", "path": path, "kind": "declared"})
}
}
if inv, ok := app["invoke_routes"].(map[string]any); ok {
for key := range inv {
dec = append(dec, map[string]any{"path": key, "kind": "endpoint"})
}
}
if art, ok := app["artifact_routes"].(map[string]any); ok {
for key, v := range art {
m, _ := v.(map[string]any)
dec = append(dec, map[string]any{
"path": key,
"kind": "endpoint",
"method": fmt.Sprint(m["method"]),
"file": fmt.Sprint(m["file"]),
})
}
}
}
return map[string]any{
"declared": dec,
"system": SystemRoutes(),
}
}
@@ -0,0 +1,55 @@
package artifact
import "testing"
func TestMetaType(t *testing.T) {
if MetaType(map[string]string{"type": "web"}) != "web" {
t.Fatal("en web")
}
if MetaType(map[string]string{"类型": "网页"}) != "web" {
t.Fatal("zh web")
}
if MetaType(map[string]string{"type": "endpoint"}) != "endpoint" {
t.Fatal("endpoint")
}
}
func TestParseFrontmatter(t *testing.T) {
src := "---\ntype: web\nroute: /x\n---\n\n# Hi\n"
meta, body := ParseFrontmatter(src)
if meta["type"] != "web" || meta["route"] != "/x" {
t.Fatalf("meta=%v", meta)
}
if body == "" || body[0] != '#' {
t.Fatalf("body=%q", body)
}
}
func TestRenderNodes(t *testing.T) {
nodes := map[string]any{
"type": []any{"title", "text"},
"slot": []any{"title", "lede"},
"value": []any{"Marqdo", "Hello"},
"attrs": []any{"", ""},
"style": []any{"title", "lede"},
}
html, err := RenderNodes(nodes)
if err != nil {
t.Fatal(err)
}
if !contains(html, "Marqdo") || !contains(html, "mq-view") {
t.Fatalf("html=%s", html)
}
}
func contains(s, sub string) bool {
return len(s) >= len(sub) && (s == sub || len(sub) == 0 ||
(func() bool {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return true
}
}
return false
})())
}
+18 -2
View File
@@ -117,11 +117,17 @@ func (st *state) handleOIDCStart(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
redirectURI, err := st.oidc.ResolveRedirectURI(r)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
session.SetValue(sid, "oidc_state", state)
session.SetValue(sid, "oidc_verifier", verifier)
session.SetValue(sid, "oidc_redirect_uri", redirectURI)
next := loginDest(r, st.oidcFallbackNext(r))
session.SetValue(sid, "oidc_next", next)
loc, err := st.oidc.AuthorizeRedirect(state, oidc.PKCEChallengeS256(verifier))
loc, err := st.oidc.AuthorizeRedirect(state, oidc.PKCEChallengeS256(verifier), redirectURI)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@@ -164,6 +170,15 @@ func (st *state) handleOIDCCallback(w http.ResponseWriter, r *http.Request) {
}
verifier, _ := session.GetValue(sid, "oidc_verifier")
verifierS, _ := verifier.(string)
redirVal, _ := session.GetValue(sid, "oidc_redirect_uri")
redirectURI, _ := redirVal.(string)
if redirectURI == "" && !st.oidc.IsPathRedirect() {
redirectURI = strings.TrimSpace(st.oidc.RedirectURI)
}
if redirectURI == "" {
http.Error(w, "oidc: missing redirect_uri in session", http.StatusBadRequest)
return
}
nextVal, _ := session.GetValue(sid, "oidc_next")
next, _ := nextVal.(string)
if next == "" {
@@ -171,9 +186,10 @@ func (st *state) handleOIDCCallback(w http.ResponseWriter, r *http.Request) {
}
session.DelKey(sid, "oidc_state")
session.DelKey(sid, "oidc_verifier")
session.DelKey(sid, "oidc_redirect_uri")
session.DelKey(sid, "oidc_next")
tok, err := st.oidc.ExchangeCode(oidcClient(), code, verifierS)
tok, err := st.oidc.ExchangeCode(oidcClient(), code, verifierS, redirectURI)
if err != nil {
http.Error(w, "oidc token exchange: "+err.Error(), http.StatusBadGateway)
return
+3
View File
@@ -122,6 +122,7 @@ func NewHandler(appBag map[string]any, entryDir string) (http.Handler, error) {
galleryRoutes: routeMapOf(appBag, "gallery_routes"),
proxyRoutes: routeMapOf(appBag, "proxy_routes"),
invokeRoutes: routeMapOf(appBag, "invoke_routes"),
artifactRoutes: routeMapOf(appBag, "artifact_routes"),
auth: authCfg,
oidc: oidc.ParseConfig(appBag),
tenant: tenant.FromBag(appBag),
@@ -238,6 +239,7 @@ func NewHandler(appBag map[string]any, entryDir string) (http.Handler, error) {
st.mountGalleryRoutes(mux)
st.mountProxyRoutes(mux)
st.mountInvokeRoutes(mux)
st.mountArtifactRoutes(mux)
st.mountAuthRoutes(mux)
mux.HandleFunc("/{path...}", st.handleNotFound)
@@ -307,6 +309,7 @@ type state struct {
galleryRoutes map[string]any
proxyRoutes map[string]any
invokeRoutes map[string]any
artifactRoutes map[string]any
auth authConfig
oidc oidc.Config
tenant tenant.Config
@@ -0,0 +1,142 @@
package httpx
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"github.com/marqdo/marqdo/plugins/web/internal/plugin"
"github.com/marqdo/marqdo/plugins/web/internal/session"
)
// artifactRoute is one Endpoint file registered by web.serve (ADR 0007).
type artifactRoute struct {
Method string
Path string
File string
Request string // json | form | none | query
Response string // json | text | html
Auth string // role name or "" / "none"
}
func parseArtifactRoute(v any) artifactRoute {
m, _ := v.(map[string]any)
return artifactRoute{
Method: strings.ToUpper(strOpt(m, "method", "GET")),
Path: strOpt(m, "path", ""),
File: strOpt(m, "file", ""),
Request: strings.ToLower(strOpt(m, "request", "json")),
Response: strings.ToLower(strOpt(m, "response", "json")),
Auth: strOpt(m, "auth", ""),
}
}
func (st *state) mountArtifactRoutes(mux *http.ServeMux) {
for _, key := range sortedKeys(st.artifactRoutes) {
cfg := parseArtifactRoute(st.artifactRoutes[key])
if cfg.File == "" || cfg.Path == "" {
continue
}
method := cfg.Method
if method == "" {
method = http.MethodGet
}
pattern := method + " " + goMuxPattern(cfg.Path)
routeCfg := cfg
mux.HandleFunc(pattern, func(w http.ResponseWriter, r *http.Request) {
if r.Method != method {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
st.handleArtifact(w, r, routeCfg)
})
}
}
func (st *state) handleArtifact(w http.ResponseWriter, r *http.Request, route artifactRoute) {
if route.Auth != "" && !strings.EqualFold(route.Auth, "none") {
cookie := r.Header.Get("Cookie")
user := session.UsernameFromCookie(cookie)
if user == "" {
jsonErr(w, http.StatusUnauthorized, "auth required")
return
}
role := session.RoleFromCookie(cookie)
if role != "" && role != route.Auth && role != "admin" && role != "superadmin" {
jsonErr(w, http.StatusForbidden, "forbidden")
return
}
}
payload := map[string]any{}
switch route.Request {
case "none":
payload = queryToMap(r.URL.RawQuery)
case "query":
payload = queryToMap(r.URL.RawQuery)
case "form":
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 8<<20))
if err != nil {
jsonErr(w, http.StatusBadRequest, fmt.Sprintf("body: %v", err))
return
}
payload = formToMap(body)
default: // json
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 8<<20))
if err != nil {
jsonErr(w, http.StatusBadRequest, fmt.Sprintf("body: %v", err))
return
}
q := queryToMap(r.URL.RawQuery)
if len(body) == 0 {
payload = q
} else {
var raw map[string]any
if err := json.Unmarshal(body, &raw); err != nil {
jsonErr(w, http.StatusBadRequest, fmt.Sprintf("json: %v", err))
return
}
if raw == nil {
payload = q
} else {
for k, v := range q {
if _, exists := raw[k]; !exists {
raw[k] = v
}
}
payload = raw
}
}
}
for _, name := range pathParamNames(route.Path) {
if v := r.PathValue(name); v != "" {
payload[name] = v
}
}
value, err := plugin.RunArtifact(route.File, payload)
if err != nil {
jsonErr(w, http.StatusInternalServerError, err.Error())
return
}
switch route.Response {
case "text":
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, fmt.Sprint(value))
case "html", "text/html":
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, fmt.Sprint(value))
default:
code, body, err := jsonWrapInvokeValue(value)
if err != nil {
jsonErr(w, code, err.Error())
return
}
writeJSON(w, code, body)
}
}
+139 -16
View File
@@ -20,8 +20,11 @@ type Config struct {
ClientID string
ClientSecret string
RedirectURI string
Scopes string
CallbackPath string
// RedirectOrigins is an optional comma-separated allowlist of origins
// (e.g. "https://qdqc.com,https://www.qdqc.com") used when RedirectURI is a path.
RedirectOrigins string
Scopes string
CallbackPath string
// AuthorizeURL / TokenURL / UserInfoURL override discovery when set.
AuthorizeURL string
TokenURL string
@@ -36,11 +39,35 @@ func (c Config) Enabled() bool {
(strings.TrimSpace(c.Issuer) != "" || strings.TrimSpace(c.AuthorizeURL) != "")
}
// IsPathRedirect reports whether RedirectURI is a path (resolved per request Host).
func (c Config) IsPathRedirect() bool {
s := strings.TrimSpace(c.RedirectURI)
if s == "" {
return false
}
if strings.HasPrefix(s, "/") {
return true
}
u, err := url.Parse(s)
if err != nil {
return false
}
// scheme-relative or bare path without host
return u.Scheme == "" && u.Host == "" && u.Path != ""
}
// Callback reports the local path (default /oidc/callback).
func (c Config) Callback() string {
p := strings.TrimSpace(c.CallbackPath)
if p == "" {
if u, err := url.Parse(c.RedirectURI); err == nil && u.Path != "" {
ru := strings.TrimSpace(c.RedirectURI)
if c.IsPathRedirect() {
if !strings.HasPrefix(ru, "/") {
ru = "/" + ru
}
return ru
}
if u, err := url.Parse(ru); err == nil && u.Path != "" {
return u.Path
}
return "/oidc/callback"
@@ -51,6 +78,91 @@ func (c Config) Callback() string {
return p
}
// PublicOrigin derives scheme://host from a login/callback request.
// Prefers X-Forwarded-Proto + X-Forwarded-Host (first value), else Host + TLS.
func PublicOrigin(r *http.Request) (string, error) {
if r == nil {
return "", fmt.Errorf("oidc: nil request")
}
scheme := strings.TrimSpace(r.Header.Get("X-Forwarded-Proto"))
if i := strings.IndexByte(scheme, ','); i >= 0 {
scheme = strings.TrimSpace(scheme[:i])
}
if scheme == "" {
if r.TLS != nil {
scheme = "https"
} else {
scheme = "http"
}
}
host := strings.TrimSpace(r.Header.Get("X-Forwarded-Host"))
if i := strings.IndexByte(host, ','); i >= 0 {
host = strings.TrimSpace(host[:i])
}
if host == "" {
host = strings.TrimSpace(r.Host)
}
if host == "" {
return "", fmt.Errorf("oidc: empty Host")
}
return scheme + "://" + host, nil
}
// ResolveRedirectURI returns the absolute redirect_uri for this request.
// Absolute RedirectURI is returned unchanged; path RedirectURI is joined to the request origin.
func (c Config) ResolveRedirectURI(r *http.Request) (string, error) {
ru := strings.TrimSpace(c.RedirectURI)
if ru == "" {
return "", fmt.Errorf("oidc: empty redirect_uri")
}
if !c.IsPathRedirect() {
return ru, nil
}
origin, err := PublicOrigin(r)
if err != nil {
return "", err
}
path := ru
if !strings.HasPrefix(path, "/") {
path = "/" + path
}
abs := origin + path
if err := c.checkOriginAllowlist(origin); err != nil {
return "", err
}
return abs, nil
}
// AllowedOrigins returns trimmed non-empty origins from RedirectOrigins.
func (c Config) AllowedOrigins() []string {
raw := strings.TrimSpace(c.RedirectOrigins)
if raw == "" {
return nil
}
var out []string
for _, p := range strings.Split(raw, ",") {
p = strings.TrimRight(strings.TrimSpace(p), "/")
if p != "" {
out = append(out, p)
}
}
return out
}
func (c Config) checkOriginAllowlist(origin string) error {
allowed := c.AllowedOrigins()
if len(allowed) == 0 {
return nil
}
origin = strings.TrimRight(strings.TrimSpace(origin), "/")
for _, a := range allowed {
if strings.EqualFold(a, origin) {
return nil
}
}
return fmt.Errorf("oidc: origin %q not in redirect_origins", origin)
}
type discoveryDoc struct {
AuthorizationEndpoint string `json:"authorization_endpoint"`
TokenEndpoint string `json:"token_endpoint"`
@@ -126,14 +238,19 @@ func PKCEChallengeS256(verifier string) string {
}
// AuthorizeRedirect builds the browser redirect to the IdP authorize endpoint.
func (c Config) AuthorizeRedirect(state, codeChallenge string) (string, error) {
// redirectURI must be the absolute URI used for this login (same value later for token exchange).
func (c Config) AuthorizeRedirect(state, codeChallenge, redirectURI string) (string, error) {
if c.AuthorizeURL == "" {
return "", fmt.Errorf("oidc: authorize URL not resolved")
}
redirectURI = strings.TrimSpace(redirectURI)
if redirectURI == "" {
return "", fmt.Errorf("oidc: empty redirect_uri")
}
q := url.Values{}
q.Set("response_type", "code")
q.Set("client_id", c.ClientID)
q.Set("redirect_uri", c.RedirectURI)
q.Set("redirect_uri", redirectURI)
q.Set("scope", c.ScopesCSV())
q.Set("state", state)
if codeChallenge != "" {
@@ -160,17 +277,22 @@ type TokenResponse struct {
}
// ExchangeCode swaps authorization code for tokens (client_secret_post).
func (c Config) ExchangeCode(client *http.Client, code, codeVerifier string) (*TokenResponse, error) {
// redirectURI must match the absolute URI sent in AuthorizeRedirect.
func (c Config) ExchangeCode(client *http.Client, code, codeVerifier, redirectURI string) (*TokenResponse, error) {
if c.TokenURL == "" {
return nil, fmt.Errorf("oidc: token URL not resolved")
}
redirectURI = strings.TrimSpace(redirectURI)
if redirectURI == "" {
return nil, fmt.Errorf("oidc: empty redirect_uri")
}
if client == nil {
client = &http.Client{Timeout: 15 * time.Second}
}
form := url.Values{}
form.Set("grant_type", "authorization_code")
form.Set("code", code)
form.Set("redirect_uri", c.RedirectURI)
form.Set("redirect_uri", redirectURI)
form.Set("client_id", c.ClientID)
form.Set("client_secret", c.ClientSecret)
if codeVerifier != "" {
@@ -313,15 +435,16 @@ func ParseConfig(m map[string]any) Config {
m = nested
}
c := Config{
Issuer: firstStr(m, "issuer", "发行方", "issuer_url"),
ClientID: firstStr(m, "client_id", "客户端编号", "客户端id", "clientId"),
ClientSecret: firstStr(m, "client_secret", "客户端密钥", "clientSecret"),
RedirectURI: firstStr(m, "redirect_uri", "回调", "回调地址", "redirect"),
Scopes: firstStr(m, "scopes", "范围", "scope"),
CallbackPath: firstStr(m, "callback_path", "回调路径"),
AuthorizeURL: firstStr(m, "authorize_url", "授权地址"),
TokenURL: firstStr(m, "token_url", "令牌地址"),
UserInfoURL: firstStr(m, "userinfo_url", "用户信息地址"),
Issuer: firstStr(m, "issuer", "发行方", "issuer_url"),
ClientID: firstStr(m, "client_id", "客户端编号", "客户端id", "clientId"),
ClientSecret: firstStr(m, "client_secret", "客户端密钥", "clientSecret"),
RedirectURI: firstStr(m, "redirect_uri", "回调", "回调地址", "redirect"),
RedirectOrigins: firstStr(m, "redirect_origins", "回调来源", "redirect_origin"),
Scopes: firstStr(m, "scopes", "范围", "scope"),
CallbackPath: firstStr(m, "callback_path", "回调路径"),
AuthorizeURL: firstStr(m, "authorize_url", "授权地址"),
TokenURL: firstStr(m, "token_url", "令牌地址"),
UserInfoURL: firstStr(m, "userinfo_url", "用户信息地址"),
}
return c
}
+138
View File
@@ -1,6 +1,10 @@
package oidc_test
import (
"crypto/tls"
"net/http"
"net/url"
"strings"
"testing"
"github.com/marqdo/marqdo/plugins/web/internal/oidc"
@@ -46,4 +50,138 @@ func TestPKCEAndParse(t *testing.T) {
if c.Callback() != "/oidc/callback" {
t.Fatalf("callback=%s", c.Callback())
}
if c.IsPathRedirect() {
t.Fatal("absolute redirect should not be path mode")
}
}
func TestPathRedirectParseAndCallback(t *testing.T) {
c := oidc.ParseConfig(map[string]any{
"issuer": "https://id.example.com",
"client_id": "app_x",
"client_secret": "secret",
"redirect_uri": "/oidc/callback",
"redirect_origins": "https://qdqc.com,https://www.qdqc.com",
})
if !c.Enabled() {
t.Fatal("expected enabled")
}
if !c.IsPathRedirect() {
t.Fatal("expected path mode")
}
if c.Callback() != "/oidc/callback" {
t.Fatalf("callback=%s", c.Callback())
}
origins := c.AllowedOrigins()
if len(origins) != 2 || origins[0] != "https://qdqc.com" {
t.Fatalf("origins=%v", origins)
}
}
func TestResolveRedirectURIAbsolute(t *testing.T) {
c := oidc.Config{RedirectURI: "https://qdqc.com/oidc/callback"}
r, _ := http.NewRequest(http.MethodGet, "http://www.qdqc.com/oidc/login", nil)
r.Host = "www.qdqc.com"
got, err := c.ResolveRedirectURI(r)
if err != nil {
t.Fatal(err)
}
if got != "https://qdqc.com/oidc/callback" {
t.Fatalf("got=%s", got)
}
}
func TestResolveRedirectURIPathFromHost(t *testing.T) {
c := oidc.Config{RedirectURI: "/oidc/callback"}
r, _ := http.NewRequest(http.MethodGet, "http://qdqc.com/oidc/login", nil)
r.Host = "qdqc.com"
got, err := c.ResolveRedirectURI(r)
if err != nil {
t.Fatal(err)
}
if got != "http://qdqc.com/oidc/callback" {
t.Fatalf("got=%s", got)
}
}
func TestResolveRedirectURIPathTLS(t *testing.T) {
c := oidc.Config{RedirectURI: "/oidc/callback"}
r, _ := http.NewRequest(http.MethodGet, "https://qdqc.com/oidc/login", nil)
r.Host = "qdqc.com"
r.TLS = &tls.ConnectionState{}
got, err := c.ResolveRedirectURI(r)
if err != nil {
t.Fatal(err)
}
if got != "https://qdqc.com/oidc/callback" {
t.Fatalf("got=%s", got)
}
}
func TestResolveRedirectURIXForwarded(t *testing.T) {
c := oidc.Config{RedirectURI: "/oidc/callback"}
r, _ := http.NewRequest(http.MethodGet, "http://127.0.0.1:18085/oidc/login", nil)
r.Host = "127.0.0.1:18085"
r.Header.Set("X-Forwarded-Proto", "https")
r.Header.Set("X-Forwarded-Host", "www.qdqc.com, other.example")
got, err := c.ResolveRedirectURI(r)
if err != nil {
t.Fatal(err)
}
if got != "https://www.qdqc.com/oidc/callback" {
t.Fatalf("got=%s", got)
}
}
func TestResolveRedirectURIAllowlistReject(t *testing.T) {
c := oidc.Config{
RedirectURI: "/oidc/callback",
RedirectOrigins: "https://qdqc.com,https://www.qdqc.com",
}
r, _ := http.NewRequest(http.MethodGet, "https://evil.example/oidc/login", nil)
r.Host = "evil.example"
r.Header.Set("X-Forwarded-Proto", "https")
_, err := c.ResolveRedirectURI(r)
if err == nil || !strings.Contains(err.Error(), "redirect_origins") {
t.Fatalf("expected allowlist error, got %v", err)
}
}
func TestResolveRedirectURIAllowlistOK(t *testing.T) {
c := oidc.Config{
RedirectURI: "/oidc/callback",
RedirectOrigins: "https://qdqc.com,https://www.qdqc.com",
}
r, _ := http.NewRequest(http.MethodGet, "https://www.qdqc.com/oidc/login", nil)
r.Host = "www.qdqc.com"
r.Header.Set("X-Forwarded-Proto", "https")
got, err := c.ResolveRedirectURI(r)
if err != nil {
t.Fatal(err)
}
if got != "https://www.qdqc.com/oidc/callback" {
t.Fatalf("got=%s", got)
}
}
func TestAuthorizeRedirectUsesExplicitURI(t *testing.T) {
c := oidc.Config{
ClientID: "app_x",
AuthorizeURL: "https://id.example.com/oauth2/authorize",
RedirectURI: "/oidc/callback",
}
loc, err := c.AuthorizeRedirect("st", "ch", "https://www.qdqc.com/oidc/callback")
if err != nil {
t.Fatal(err)
}
u, err := url.Parse(loc)
if err != nil {
t.Fatal(err)
}
if got := u.Query().Get("redirect_uri"); got != "https://www.qdqc.com/oidc/callback" {
t.Fatalf("redirect_uri=%s", got)
}
if u.Query().Get("state") != "st" {
t.Fatal("state")
}
}
+22
View File
@@ -49,3 +49,25 @@ func CallLibPath(path string, args map[string]any) (any, error) {
}
return v, nil
}
// RunArtifact loads and runs a Document/Endpoint `.mq.md` (ADR 0007).
// Request fields are passed as Metadata Binding overlays (`arg` / `--bind`).
func RunArtifact(path string, args map[string]any) (any, error) {
payload := map[string]any{"path": path}
if len(args) > 0 {
payload["args"] = args
}
raw, err := json.Marshal(payload)
if err != nil {
return nil, err
}
out, err := Query("run_artifact", string(raw))
if err != nil {
return nil, err
}
var v any
if err := json.Unmarshal([]byte(out), &v); err != nil {
return nil, err
}
return v, nil
}
@@ -0,0 +1,88 @@
package render
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/marqdo/marqdo/plugins/web/internal/db"
)
func TestResolveMainDataSource(t *testing.T) {
dir := t.TempDir()
url := "sqlite:" + filepath.Join(dir, "t.db")
fields := []any{
map[string]any{"字段": "title", "类型": "text", "可空": false},
map[string]any{"字段": "slug", "类型": "text", "可空": false},
map[string]any{"字段": "summary", "类型": "text", "可空": true},
map[string]any{"字段": "content", "类型": "text", "可空": true},
}
if _, err := db.Init(url, "posts", fields); err != nil {
t.Fatal(err)
}
rows := []any{
map[string]any{
"title": "Hello", "slug": "hello",
"summary": "sum", "content": "# Body",
},
}
if _, err := db.Insert(url, "posts", rows); err != nil {
t.Fatal(err)
}
page := map[string]any{
"intro": "<p>hi</p>",
"data_source": "posts",
"order": "-id",
"link_prefix": "/post/",
}
intro, items, _ := resolveMain(page, url)
if intro != "<p>hi</p>" {
t.Fatalf("intro=%q", intro)
}
if len(items) != 1 {
t.Fatalf("items=%d", len(items))
}
if items[0]["title"] != "Hello" || items[0]["href"] != "hello" {
t.Fatalf("item=%v", items[0])
}
html := RenderPage(page, url, "")
if !strings.Contains(html, `class="card"`) || !strings.Contains(html, "Hello") {
t.Fatalf("html missing card: %s", html[:min(400, len(html))])
}
_ = os.RemoveAll(dir)
}
func TestResolveMainDataWhereDetail(t *testing.T) {
dir := t.TempDir()
url := "sqlite:" + filepath.Join(dir, "t.db")
fields := []any{
map[string]any{"字段": "title", "类型": "text", "可空": false},
map[string]any{"字段": "slug", "类型": "text", "可空": false},
map[string]any{"字段": "content", "类型": "text", "可空": true},
}
if _, err := db.Init(url, "posts", fields); err != nil {
t.Fatal(err)
}
rows := []any{
map[string]any{"title": "A", "slug": "a", "content": "alpha"},
map[string]any{"title": "B", "slug": "b", "content": "beta"},
}
if _, err := db.Insert(url, "posts", rows); err != nil {
t.Fatal(err)
}
page := map[string]any{
"data_source": "posts",
"query": map[string]any{"slug": "{slug}"},
"params": map[string]any{"slug": "b"},
"detail": true,
}
_, items, _ := resolveMain(page, url)
if len(items) != 1 || items[0]["title"] != "B" {
t.Fatalf("items=%v", items)
}
html := RenderPage(page, url, "")
if !strings.Contains(html, "article-title") || !strings.Contains(html, "B") {
t.Fatalf("detail html: %s", html[:min(500, len(html))])
}
}
+73 -21
View File
@@ -420,22 +420,6 @@ func renderULWithMQ(links []navLink, class string, mqPairs [][2]string) string {
func resolveMain(page map[string]any, dbURL string) (intro string, items []map[string]any, total *int64) {
intro, _ = page["intro"].(string)
main, ok := page["main"]
if !ok {
return intro, nil, nil
}
bindsAny := table.AsBind(main)
arr, _ := bindsAny.([]any)
if len(arr) == 0 {
return intro, nil, nil
}
if dbURL == "" {
return intro, nil, nil
}
tn, ok := table.BindTableName(arr)
if !ok {
return intro, nil, nil
}
limit := int64(200)
offset := int64(0)
if p, ok := page["paginate"].(map[string]any); ok {
@@ -446,22 +430,90 @@ func resolveMain(page map[string]any, dbURL string) (intro string, items []map[s
offset = int64(v)
}
}
data := selectPageData(dbURL, tn, page, limit, offset)
// Legacy compose_main bind table (internal); prefer when present.
if main, ok := page["main"]; ok {
bindsAny := table.AsBind(main)
arr, _ := bindsAny.([]any)
if len(arr) > 0 && dbURL != "" {
if tn, ok := table.BindTableName(arr); ok {
data := selectPageData(dbURL, tn, page, limit, offset)
if t, ok := data["total"].(float64); ok {
ti := int64(t)
total = &ti
}
rawRows, _ := data["rows"].([]any)
projected := table.ProjectRows(arr, rawRows)
parr, _ := projected.([]any)
for _, v := range parr {
if m, ok := v.(map[string]any); ok {
items = append(items, m)
}
}
return intro, items, total
}
}
}
// Artifact Document: flat data_source / data_order / data_where (ADR 0007).
src, _ := page["data_source"].(string)
if src == "" || dbURL == "" {
return intro, nil, nil
}
data := selectPageData(dbURL, src, page, limit, offset)
if t, ok := data["total"].(float64); ok {
ti := int64(t)
total = &ti
}
rawRows, _ := data["rows"].([]any)
projected := table.ProjectRows(arr, rawRows)
parr, _ := projected.([]any)
for _, v := range parr {
if m, ok := v.(map[string]any); ok {
for _, v := range rawRows {
if m := projectArtifactRow(v); m != nil {
items = append(items, m)
}
}
return intro, items, total
}
// projectArtifactRow maps a DB row to card/article fields without compose binds.
func projectArtifactRow(row any) map[string]any {
obj, ok := row.(map[string]any)
if !ok || obj == nil {
return nil
}
m := map[string]any{}
if id, ok := obj["id"]; ok {
m["id"] = id
}
if t := text(obj["title"]); t != "" {
m["title"] = t
}
if s := text(obj["slug"]); s != "" {
m["href"] = s
} else if h := text(obj["href"]); h != "" {
m["href"] = h
}
// Detail pages prefer full content; list cards prefer summary.
if c := text(obj["content"]); c != "" {
m["body"] = c
} else if s := text(obj["summary"]); s != "" {
m["body"] = s
} else if b := text(obj["body"]); b != "" {
m["body"] = b
}
if tag := text(obj["tag"]); tag != "" {
m["tag"] = tag
}
if meta := text(obj["created_at"]); meta != "" {
m["meta"] = meta
} else if meta := text(obj["meta"]); meta != "" {
m["meta"] = meta
}
if len(m) == 0 {
return nil
}
return m
}
// listBind is one secondary page list (compose_list / 列表装配).
type listBind struct {
Items []map[string]any
+15 -11
View File
@@ -158,6 +158,13 @@ extern int web_cache_ttl(char *args_json, char **out_json, char **err_msg);
extern int web_app_proxy(char *args_json, char **out_json, char **err_msg);
extern int web_app_invoke(char *args_json, char **out_json, char **err_msg);
extern int web_api_key_check(char *args_json, char **out_json, char **err_msg);
extern int web_render_nodes(char *args_json, char **out_json, char **err_msg);
extern int web_render_document(char *args_json, char **out_json, char **err_msg);
extern int web_dom_markdown(char *args_json, char **out_json, char **err_msg);
extern int web_serve_root(char *args_json, char **out_json, char **err_msg);
extern int web_inspect(char *args_json, char **out_json, char **err_msg);
extern int web_app_use(char *args_json, char **out_json, char **err_msg);
extern int web_route_use(char *args_json, char **out_json, char **err_msg);
static int register_core(void) {
if (host_register((char *)"web_go_ready", (char *)"", web_go_ready) != 0) return 1;
@@ -169,16 +176,6 @@ static int register_core(void) {
if (host_register((char *)"web_page_detail", (char *)"page,detail", web_page_detail) != 0) return 1;
if (host_register((char *)"web_page_chrome", (char *)"page,nav_html,footer_html,body_class", web_page_chrome) != 0) return 1;
if (host_register((char *)"web_style", (char *)"name,table,strict", web_style) != 0) return 1;
if (host_register((char *)"web_compose_components", (char *)"page,components", web_compose_components) != 0) return 1;
if (host_register((char *)"web_compose_main", (char *)"page,main", web_compose_main) != 0) return 1;
if (host_register((char *)"web_compose_list", (char *)"page,main,query,order,target", web_compose_list) != 0) return 1;
if (host_register((char *)"web_compose_intro", (char *)"page,intro", web_compose_intro) != 0) return 1;
if (host_register((char *)"web_intro", (char *)"table", web_intro) != 0) return 1;
if (host_register((char *)"web_compose_form", (char *)"page,form,id,target", web_compose_form) != 0) return 1;
if (host_register((char *)"web_compose_auth_form", (char *)"page,action,submit,form_id,err_id,target,next,kind", web_compose_auth_form) != 0) return 1;
if (host_register((char *)"web_compose_nav_brand", (char *)"page,title,href,logo,logo_light,theme_key", web_compose_nav_brand) != 0) return 1;
if (host_register((char *)"web_compose_client", (char *)"page,source,bridge,wasm", web_compose_client) != 0) return 1;
if (host_register((char *)"web_compose_form_load", (char *)"page,table,id_param", web_compose_form_load) != 0) return 1;
if (host_register((char *)"web_render", (char *)"page", web_render) != 0) return 1;
if (host_register((char *)"web_db_new", (char *)"url", web_db_new) != 0) return 1;
if (host_register((char *)"web_db_init", (char *)"url,name,fields", web_db_init) != 0) return 1;
@@ -222,7 +219,7 @@ static int register_core(void) {
if (host_register((char *)"web_auth_logout", (char *)"session_id", web_auth_logout) != 0) return 1;
if (host_register((char *)"web_auth_new", (char *)"users,session_ttl", web_auth_new) != 0) return 1;
if (host_register((char *)"web_app_auth", (char *)"app,users,session_ttl,admin_prefix,login_redirect,logout_redirect,login_path,register,register_path,default_role,session_url", web_app_auth) != 0) return 1;
if (host_register((char *)"web_app_oidc", (char *)"app,issuer,client_id,client_secret,redirect_uri,scopes,callback_path,authorize_url,token_url,userinfo_url", web_app_oidc) != 0) return 1;
if (host_register((char *)"web_app_oidc", (char *)"app,issuer,client_id,client_secret,redirect_uri,redirect_origins,scopes,callback_path,authorize_url,token_url,userinfo_url", web_app_oidc) != 0) return 1;
if (host_register((char *)"web_app_gate", (char *)"app,path,roles,permissions,match,on_deny,exclude", web_app_gate) != 0) return 1;
if (host_register((char *)"web_app_rbac", (char *)"app,catalog,desk", web_app_rbac) != 0) return 1;
if (host_register((char *)"web_app_tenant", (char *)"app,mode,param,column,default_scope", web_app_tenant) != 0) return 1;
@@ -269,6 +266,13 @@ static int register_core(void) {
if (host_register((char *)"web_app_proxy", (char *)"app,path,upstream,stream,strip_prefix,methods,headers_from_env,timeout_ms", web_app_proxy) != 0) return 1;
if (host_register((char *)"web_app_invoke", (char *)"app,path,method,fn,body,return", web_app_invoke) != 0) return 1;
if (host_register((char *)"web_api_key_check", (char *)"key,authorization,pepper,keys", web_api_key_check) != 0) return 1;
if (host_register((char *)"web_render_nodes", (char *)"page,nodes,db,title", web_render_nodes) != 0) return 1;
if (host_register((char *)"web_render_document", (char *)"page,body,db,title", web_render_document) != 0) return 1;
if (host_register((char *)"web_dom_markdown", (char *)"text", web_dom_markdown) != 0) return 1;
if (host_register((char *)"web_serve_root", (char *)"root,host,port,db,static_dir,middleware", web_serve_root) != 0) return 1;
if (host_register((char *)"web_inspect", (char *)"app", web_inspect) != 0) return 1;
if (host_register((char *)"web_app_use", (char *)"app,middleware", web_app_use) != 0) return 1;
if (host_register((char *)"web_route_use", (char *)"route,middleware", web_route_use) != 0) return 1;
return 0;
}
*/
+5 -33
View File
@@ -164,44 +164,16 @@ APIs: `# agent` / `# 智能体` → `## step` / `## plan`. Context: standing + t
Examples: [agent-pong](../../examples/agent-pong/) · [agent-okf-flywheel](../../examples/agent-okf-flywheel/). Harness: `scripts/agent-harness.sh`. Design: [ext-agent.md](../../doc/design/ext-agent.md) · Wave B: [agent-framework-2026-09.md](../../doc/research/agent-framework-2026-09.md). Dev LLM: copy [.env.example](../../.env.example) → `.env` (gitignored).
## Official extension: `ext/web` (dynamic sites)
## Official extension: `ext/web` (Document / Endpoint / Resource)
Install: `marqdo ext add web` (ZH id: `网页`). Build native plugin first: `cargo build --release -p marqdo_plugin_web`.
Install: `marqdo ext add web` (ZH id: `网页`). See ADR 0007 · [ext-web-artifact.md](../../doc/design/ext-web-artifact.md).
Import **one language file** — never mix EN/ZH API names in the same `.mq.md`:
**Do not degrade:** Document/Endpoint metadata; facade `web.page|route|serve|render|inspect`; View `|type|slot|value|attrs|style|` only; no `compose_*` / `configure` / author `ensure_plugin`. DB → `ext/data`; auth → `ext/security`; net → `ext/net`.
- EN: `import web:ext/web/web.mq.md`
- ZH: `导入 网页:ext/web/网页.mq.md`
**Hard rules (web):**
- Authors use **GFM tables + `#` classes** — no `json.parse` / `json.set` glue, no hand-built part JSON.
- Table cells stay **literal strings**; path text like `` `posts`.`title` `` is resolved by web classes.
- **Quote path/MIME cells** that contain `/` (T5 cell expressions treat bare `/` as division): `"/static/logo.svg"`, `"image/png"`.
- **`ext/**` never calls `host_*`** — hot path is native `plugins/web` ABI.
- HTTPS: terminate at reverse proxy; set `cookie_secure=True` on auth (no in-process TLS).
**Typical layout:**
```text
index.mq.md # entry + home page table + listen
pages/ # sub-pages
components/ # reusable |属性|值|样式| tables
styles/ # CSS modules
public/ # static + favicon.ico|png|svg
db/ # schema + open/init
data/ # sqlite runtime (gitignore)
```
**Core objects (EN / ZH):** `# page`/`页面`, `# style`/`样式`, `# db`/`数据库`, `# form`/`表单`, `# app`/`应用`, `# auth`/`鉴权`, `# cache`/`缓存`, `# storage`/`存储`.
**Typical flow:** `db.init` → `page.compose_*` → `app` + `static` / `listen`. Client: `marqdo wasm build -o static` + `web.client_embed source="/static/client.mq.md"` — **no author JS**; bridge may implement rich host effects ([browser-wasm-e.md](../../doc/roadmap/browser-wasm-e.md) · [browser-app](../../examples/browser-app/)).
**Shipped surface (W0–W7 + P3 + W8 + route D mount):** middleware + JSON API (`app.configure`); transactions, pagination, FTS search (`db.migrate`, `db.fts`, `db.search`); security (argon2, CSRF, SQLite sessions, login rate limit); SEO / RSS / Markdown (`page.meta`, `route_rss`, `lib/net.markdown_parse`); upload / download / gallery; sitemap / robots / error pages / redirects; RBAC (`app.gate`, user `role`); audit timestamps + FK in `db.init`; ETag on downloads; **W8** site icons (`app.icons` → `/favicon.ico`), head resource tables (`page.head`), image assembly (`make_images` / `page.images`); **D** `client_embed` auto-mount + DOM effect helpers (`text_patch` / `dom_patch`).
Design: [ext-web.md](../../doc/design/ext-web.md) · assets: [web-assets-and-images.md](../../doc/design/web-assets-and-images.md) · capability matrix: [web-net-capabilities.md](../../doc/design/web-net-capabilities.md) · example: [examples/marqdo-blog/](../../examples/marqdo-blog/) · client: [examples/web-client-site/](../../examples/web-client-site/).
Web patterns: [examples.md](examples.md) §13 · API index: [reference.md](reference.md).
Design: [ext-web-artifact.md](../../doc/design/ext-web-artifact.md) · example: [examples/marqdo-blog/](../../examples/marqdo-blog/).
## Official extension: `ext/quantum` (circuits + Q7/Q8)
@@ -249,7 +221,7 @@ Design: [ext-quantum.md](../../doc/design/ext-quantum.md) · Q7: [ext-quantum-q7
| Import `lib/text` then call bare `split` | `> text.split …` (qualified) |
| Import `lib/text` then call `拆分` | Match file language (`text.split`, not 文本) |
| `json.set` / `json.append` to build maps or lists | GFM tables; sparse `table.put`; named helpers (`browser.*`, `web.*`) |
| `json.set` to build page parts | GFM tables + `page.compose_*` / `web.page` methods |
| `json.set` / `compose_*` page parts | `type: web` Document + View table / `web.render` / `web.serve` |
| Import `json` for every browser handler | `import browser:lib/browser.mq.md` + tables ([marqdo-dev](../marqdo-dev/SKILL.md)) |
| Mix `web.page` and `网页.页面` in one file | One import language per `.mq.md` |
| Call `host_web_*` from `ext/web` | Use `# app` / `# db` methods; plugin ABI only |
+22 -28
View File
@@ -259,44 +259,38 @@ if x > 0:
> print text=y-other
```
## 13. Dynamic site (ext/web, minimal)
## 13. Dynamic site (Web Artifact, minimal)
`index.mq.md`:
```markdown
---
type: web
title: Hello
route: /
method: GET
---
# Hello
Executable document as a page.
```
`serve.mq.md`:
```markdown
---
title: hello-site
import web:ext/web/web.mq.md
import db:db/index.mq.md
---
# main
`shell` =
| 组件 | 样式 |
|------|------|
| nav | |
`main` =
| 属性 | 值 | 样式 |
|------|-----|------|
| title | posts.title | |
| body | posts.summary | |
*store = > db.open*
*css = "body{font-family:sans-serif}"*
*page = > web.page title="Hello"*
*page = > page.compose_components components=`shell`*
*page = > page.compose_main main=`main`*
*page = > page.css css=`css`*
*app = > web.app page=`page` port=8080*
*app = > app.static prefix="/static" dir="static"*
> listen app=`app`
> web.serve root="." host="127.0.0.1" port=8080
```
- Import `ext/web/web.mq.md` **or** `ext/web/网页.mq.md` — not both; match API language to the file.
- Full blog/CMS patterns: `examples/marqdo-blog/` (routes, SEO, RSS, auth, upload, WS).
- Run after `marqdo ext add web` and building `marqdo_plugin_web`.
- Import `ext/web/web.mq.md` **or** `ext/web/网页.mq.md` — not both.
- Full Artifact blog: `examples/marqdo-blog/` (`pages/` · `api/` · `db/` · `data_source`).
- Design: `doc/design/ext-web-artifact.md` · ADR 0007. No `compose_*` / `configure`.
## 14. Quantum entanglement lab (ext/quantum Q7/Q8)
+13 -13
View File
@@ -60,24 +60,24 @@ Import one file; use **that** file’s function names.
Open the imported `.mq.md` under `lib/` or `ext/` to see exact `##` / `#` names and parameters. Gold tests: `tests/lib/`, `tests/structure/`, `tests/ext/`.
## ext/web (dynamic sites)
## ext/web (Artifact sites · ADR 0007)
Install: `marqdo ext add web` (`网页`). Native: `cargo build --release -p marqdo_plugin_web`.
Install: `marqdo ext add web` (`网页`). Native: `bash scripts/build-web-plugin.sh`.
| EN import | ZH import | Core types |
|-----------|-----------|------------|
| `ext/web/web.mq.md` | `ext/web/网页.mq.md` | `# page` `# style` `# db` `# form` `# app` `# auth` `# cache` `# storage` |
| EN import | ZH import | Surface |
|-----------|-----------|---------|
| `ext/web/web.mq.md` | `ext/web/网页.mq.md` | facade: `page` · `route` · `serve` · `render` · `inspect` · `use` |
| Area | EN methods (representative) | Notes |
|------|----------------------------|-------|
| Page | `compose_components`, `compose_main`, `render`, `meta`, `paginate` | GFM tables; cells are literal path strings |
| DB | `init`, `insert`, `select`, `paginate`, `get`, `update`, `delete`, `migrate`, `fts`, `search`, `count`, `事务`/`txn` | init supports `unique`/`index`/`fk`; auto `created_at`/`updated_at` |
| App | `route`, `static`, `configure`, `listen`, `auth`, `gate`, `upload`, `download`, `gallery`, `route_rss`, `redirect`, `error_page`, `sitemap`, `robots` | W0–W7 + P3 complete |
| Auth | `login`, `check`, `logout`, `hash_password` | argon2 + CSRF + SQLite sessions; `role` for RBAC |
| Kind | Metadata | Notes |
|------|----------|-------|
| Document | `type: web` / `类型: 网页` | `route` · `data_source` · `data_order` · `data_where` |
| Endpoint | `type: endpoint` / `类型: 端点` | `path` · `method` · `# main` handler · `request`/`response` |
| View | `|type|slot|value|attrs|style|` | `web.render nodes=` |
| Resource | `ext/data` · `ext/security` · `ext/net` | db/cache/storage · auth/rbac/oidc · websocket |
Drivers (Postgres / Redis / S3): `db url=postgres://…`, `cache url=memory:`, `storage url=file:…` — see `doc/design/ext-web-drivers.md`.
Do **not** use `compose_*` / `app.configure` / author `ensure_plugin`.
Example project: `examples/marqdo-blog/`. Capability matrix: `doc/design/web-net-capabilities.md`.
Example: `examples/marqdo-blog/`. Design: `doc/design/ext-web-artifact.md`. Historical matrix: `doc/design/web-net-capabilities.md`.
## ext/quantum (circuits + Q7/Q8)
+117
View File
@@ -0,0 +1,117 @@
//! Web / Endpoint Artifact classification from bound Metadata.
//!
//! Spec: `doc/design/ext-web-artifact.md` · ADR 0007.
//! Does not add CORE_CONSTRUCTS — reuses flat frontmatter `type` / `类型`.
use crate::value::Value;
/// Kind of executable document recognized for the Web model.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ArtifactKind {
/// `type: web` / `类型: 网页`
Web,
/// `type: endpoint` / `类型: 端点`
Endpoint,
/// `type: prompt` / `类型: 提示` (AI stack; classified for inspect)
Prompt,
/// `type: llm` / other known types
Other,
}
/// Read `type` or `类型` from bound metadata.
pub fn metadata_type(metadata: &[(String, Value)]) -> Option<String> {
for key in ["type", "类型"] {
if let Some(v) = meta_get(metadata, key) {
let s = v.as_display();
if !s.is_empty() {
return Some(s);
}
}
}
None
}
/// Classify artifact kind from metadata (None if no type key).
pub fn classify(metadata: &[(String, Value)]) -> Option<ArtifactKind> {
let t = metadata_type(metadata)?;
Some(match t.as_str() {
"web" | "网页" => ArtifactKind::Web,
"endpoint" | "端点" => ArtifactKind::Endpoint,
"prompt" | "提示" => ArtifactKind::Prompt,
_ => ArtifactKind::Other,
})
}
/// True when this module is a Web Document Artifact.
pub fn is_web_document(metadata: &[(String, Value)]) -> bool {
matches!(classify(metadata), Some(ArtifactKind::Web))
}
/// True when this module is an Endpoint Artifact.
pub fn is_endpoint(metadata: &[(String, Value)]) -> bool {
matches!(classify(metadata), Some(ArtifactKind::Endpoint))
}
/// Lookup a metadata string (EN key, then ZH aliases).
pub fn meta_text(metadata: &[(String, Value)], en: &str, zh: &str) -> Option<String> {
for key in [en, zh] {
if let Some(v) = meta_get(metadata, key) {
let s = v.as_display();
if !s.is_empty() {
return Some(s);
}
}
}
None
}
/// Route for a web document: `route` / `路由`, else None.
pub fn web_route(metadata: &[(String, Value)]) -> Option<String> {
meta_text(metadata, "route", "路由")
}
/// HTTP method defaulting to GET.
pub fn http_method(metadata: &[(String, Value)]) -> String {
meta_text(metadata, "method", "方法").unwrap_or_else(|| "GET".to_string())
}
/// Endpoint path: `path` / `路径`.
pub fn endpoint_path(metadata: &[(String, Value)]) -> Option<String> {
meta_text(metadata, "path", "路径")
}
fn meta_get<'a>(metadata: &'a [(String, Value)], key: &str) -> Option<&'a Value> {
metadata.iter().find(|(k, _)| k == key).map(|(_, v)| v)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::value::Value;
#[test]
fn classify_web_en_zh() {
let en = vec![("type".into(), Value::Text("web".into()))];
assert_eq!(classify(&en), Some(ArtifactKind::Web));
let zh = vec![("类型".into(), Value::Text("网页".into()))];
assert_eq!(classify(&zh), Some(ArtifactKind::Web));
}
#[test]
fn classify_endpoint() {
let m = vec![
("type".into(), Value::Text("endpoint".into())),
("path".into(), Value::Text("/api/x".into())),
("method".into(), Value::Text("POST".into())),
];
assert!(is_endpoint(&m));
assert_eq!(endpoint_path(&m).as_deref(), Some("/api/x"));
assert_eq!(http_method(&m), "POST");
}
#[test]
fn no_type() {
let m = vec![("title".into(), Value::Text("x".into()))];
assert_eq!(classify(&m), None);
}
}
+6 -1
View File
@@ -201,6 +201,7 @@ pub enum HostFn {
Meta = 192,
MetaGet = 193,
ModuleSource = 194,
ArtifactKind = 195,
}
@@ -392,6 +393,7 @@ impl HostFn {
192 => Self::Meta,
193 => Self::MetaGet,
194 => Self::ModuleSource,
195 => Self::ArtifactKind,
_ => return None,
})
}
@@ -586,6 +588,7 @@ impl HostFn {
"host_meta" | "meta" => Self::Meta,
"host_meta_get" | "meta_get" => Self::MetaGet,
"host_module_source" | "module_source" => Self::ModuleSource,
"host_artifact_kind" | "artifact_kind" => Self::ArtifactKind,
_ => return None,
})
}
@@ -767,6 +770,7 @@ impl HostFn {
Self::Meta => "host_meta",
Self::MetaGet => "host_meta_get",
Self::ModuleSource => "host_module_source",
Self::ArtifactKind => "host_artifact_kind",
Self::WritebackRecord => "host_writeback_record",
Self::WritebackGet => "host_writeback_get",
Self::WritebackClear => "host_writeback_clear",
@@ -790,7 +794,7 @@ impl HostFn {
&["path"]
}
Self::WriteText | Self::AppendText => &["path", "text"],
Self::NowUnix | Self::NowMs | Self::Args | Self::Cwd | Self::Meta | Self::ModuleSource => {
Self::NowUnix | Self::NowMs | Self::Args | Self::Cwd | Self::Meta | Self::ModuleSource | Self::ArtifactKind => {
&[]
}
Self::FormatTime => &["unix", "pattern"],
@@ -998,6 +1002,7 @@ pub fn call_host(
HostFn::Meta => sys::meta(ctx),
HostFn::MetaGet => sys::meta_get(ctx, require(bound, "key")?),
HostFn::ModuleSource => crate::host::agent_rt::module_source(ctx),
HostFn::ArtifactKind => sys::artifact_kind(ctx),
HostFn::Exit => sys::exit(ctx, require(bound, "code")?),
HostFn::Exec => sys::exec(
ctx,
+38
View File
@@ -417,6 +417,44 @@ unsafe extern "C" fn host_query(
});
Ok(serde_json::json!({ "ok": true }))
})(),
// ADR 0007: run a Document/Endpoint `.mq.md` for HTTP handlers.
"run_artifact" => (|| {
let raw = args_owned.as_deref().unwrap_or("{}");
let args: serde_json::Value = serde_json::from_str(if raw.trim().is_empty() {
"{}"
} else {
raw
})
.map_err(|e| format!("run_artifact args: {e}"))?;
let path = args
.get("path")
.and_then(|v| v.as_str())
.ok_or_else(|| "run_artifact requires `path`".to_string())?;
let p = std::path::PathBuf::from(path);
if !p.is_file() {
return Err(format!("run_artifact: not a file: {path}"));
}
let mut binds: Vec<(String, String)> = Vec::new();
if let Some(obj) = args.get("args").and_then(|v| v.as_object()) {
for (k, v) in obj {
let s = match v {
serde_json::Value::String(t) => t.clone(),
serde_json::Value::Null => continue,
other => other.to_string(),
};
binds.push((k.clone(), s));
}
}
let opts = crate::RunOptions {
binds,
allow_fs_write: ctx.caps.fs_write,
allow_exec: ctx.caps.exec,
allow_net: ctx.caps.net,
..crate::RunOptions::default()
};
let cap = crate::run_file_capture(&p, &opts).map_err(|e| e.to_string())?;
value_to_json(&cap.value)
})(),
other => Err(format!("host_query: unknown `{other}`")),
};
+11
View File
@@ -109,6 +109,17 @@ pub fn meta_get(ctx: &HostContext, key: &Value) -> Result<Value, String> {
.unwrap_or(Value::None))
}
/// Classify entry Artifact (`web` / `endpoint` / `prompt` / `other`); missing type → None.
pub fn artifact_kind(ctx: &HostContext) -> Result<Value, String> {
Ok(match crate::artifact::classify(&ctx.entry_metadata) {
None => Value::None,
Some(crate::artifact::ArtifactKind::Web) => Value::Text("web".into()),
Some(crate::artifact::ArtifactKind::Endpoint) => Value::Text("endpoint".into()),
Some(crate::artifact::ArtifactKind::Prompt) => Value::Text("prompt".into()),
Some(crate::artifact::ArtifactKind::Other) => Value::Text("other".into()),
})
}
pub fn exit(ctx: &HostContext, code: &Value) -> Result<Value, String> {
let c = as_i64(code, "code")?;
if ctx.soft_side_effects {
+41
View File
@@ -205,12 +205,53 @@ impl Interpreter {
true,
);
}
// Web Artifact entry (ADR 0007): `type: web|endpoint` may be pure document —
// no `# main`. Return a Result-shaped summary for run/check gold paths.
if let Some(kind) = crate::artifact::classify(&module.metadata) {
use crate::artifact::ArtifactKind;
if matches!(kind, ArtifactKind::Web | ArtifactKind::Endpoint) {
return Ok(Self::artifact_run_summary(module, kind));
}
}
bail!("no `# main` object to run");
})();
self.site_module = prev;
result
}
fn artifact_run_summary(module: &Module, kind: crate::artifact::ArtifactKind) -> Value {
use crate::artifact::{endpoint_path, http_method, web_route, ArtifactKind};
let kind_s = match kind {
ArtifactKind::Web => "web",
ArtifactKind::Endpoint => "endpoint",
ArtifactKind::Prompt => "prompt",
ArtifactKind::Other => "other",
};
let mut value = vec![
("kind".into(), Value::Text(kind_s.into())),
("method".into(), Value::Text(http_method(&module.metadata))),
];
match kind {
ArtifactKind::Web => {
if let Some(r) = web_route(&module.metadata) {
value.push(("route".into(), Value::Text(r)));
}
}
ArtifactKind::Endpoint => {
if let Some(r) = endpoint_path(&module.metadata) {
value.push(("path".into(), Value::Text(r)));
}
}
_ => {}
}
Value::Map(vec![
("ok".into(), Value::Bool(true)),
("value".into(), Value::Map(value)),
("error".into(), Value::None),
])
}
/// Resolve `lib.member` on the site (entry) module — used by ABI `call_lib_path`.
pub(crate) fn call_site_lib_path(
&mut self,
+85
View File
@@ -0,0 +1,85 @@
//! L3 Capability derivation from symbols + modules.
use crate::knowledge::fingerprint::tokenize_ident;
use crate::knowledge::ir::{CapabilityInfo, Evidence, ModuleInfo, SymbolInfo};
use crate::knowledge::util::slugify;
pub fn derive_capabilities(
symbols: &[SymbolInfo],
modules: &[ModuleInfo],
) -> Vec<CapabilityInfo> {
let mut caps: Vec<CapabilityInfo> = Vec::new();
for s in symbols {
if s.kind == "test" {
continue;
}
let (id, name, desc) = if let Some(ref c) = s.capability {
let id = slugify(c);
(id.clone(), c.clone(), s.use_when.clone().unwrap_or_else(|| c.clone()))
} else {
let tokens = tokenize_ident(&s.name);
let name = if tokens.is_empty() {
s.name.clone()
} else {
// module.verb_noun style
let mod_short = s
.module
.rsplit('/')
.next()
.unwrap_or(&s.module)
.replace(".mq.md", "");
format!("{}.{}", slugify(&mod_short), tokens.join("_"))
};
let id = slugify(&name);
let desc = s
.use_when
.clone()
.or_else(|| {
modules
.iter()
.find(|m| m.id == s.module)
.and_then(|m| m.responsibility.clone())
})
.unwrap_or_else(|| format!("Capability implemented by `{}`", s.name));
(id, name, desc)
};
if let Some(existing) = caps.iter_mut().find(|c| c.id == id) {
if !existing.implemented_by.contains(&s.id) {
existing.implemented_by.push(s.id.clone());
}
if !existing.modules.contains(&s.module) {
existing.modules.push(s.module.clone());
}
continue;
}
caps.push(CapabilityInfo {
id,
name,
description: desc,
implemented_by: vec![s.id.clone()],
modules: vec![s.module.clone()],
status: s.status.clone(),
use_when: s.use_when.clone(),
do_not_use_when: s.do_not_use_when.clone(),
related: s.related.clone(),
evidence: Evidence {
sources: vec![s.resource.clone()],
tests: s.tests.clone(),
..Evidence::default()
},
confidence: s.confidence.clone(),
introduced: None,
verified: None,
last_used: None,
last_tested: None,
possible_duplicates: Vec::new(),
});
}
// related edges are authored; graph builder adds RelatedTo edges
caps.sort_by(|a, b| a.id.cmp(&b.id));
caps
}
+595
View File
@@ -0,0 +1,595 @@
//! Compile KnowledgeGraph into `.marqdo/` projections.
use std::fs;
use std::path::Path;
use anyhow::{Context, Result};
use serde_json::json;
use crate::knowledge::extract::{extract_graph, ExtractOptions};
use crate::knowledge::ir::KnowledgeGraph;
use crate::knowledge::util::{module_stem, slugify, yaml_escape};
#[derive(Debug, Clone)]
pub struct KnowledgeOptions {
pub path: std::path::PathBuf,
pub out_dir: std::path::PathBuf,
}
pub fn compile(opts: KnowledgeOptions) -> Result<KnowledgeGraph> {
let extract_opts = ExtractOptions {
path: opts.path.clone(),
out_dir: opts.out_dir.clone(),
};
let graph = extract_graph(&extract_opts)?;
write_projections(&opts.out_dir, &graph)?;
eprintln!(
"marqdo knowledge: {} module(s), {} symbol(s), {} capability(ies), {} knowledge → {}",
graph.modules.len(),
graph.symbols.len(),
graph.capabilities.len(),
graph.knowledge.len(),
opts.out_dir.display()
);
Ok(graph)
}
pub fn load_graph(out_dir: &Path) -> Result<KnowledgeGraph> {
let graph_path = out_dir.join("graph").join("graph.json");
let edges_path = out_dir.join("graph").join("edges.json");
if !graph_path.exists() {
anyhow::bail!(
"no knowledge graph at {} — run `marqdo knowledge` first",
graph_path.display()
);
}
// Prefer re-extract for freshness when source path known; for query commands
// we rebuild from JSON snapshot fields + re-scan if catalog sibling exists.
// Snapshot load: reconstruct minimal IR from written pages + edges.json.
let edges_raw = fs::read_to_string(&edges_path).unwrap_or_else(|_| "{\"edges\":[]}".into());
let edges_v: serde_json::Value = serde_json::from_str(&edges_raw)?;
let graph_raw = fs::read_to_string(&graph_path)?;
let graph_v: serde_json::Value = serde_json::from_str(&graph_raw)?;
// Re-compile from recorded root if present in graph.json
if let Some(root) = graph_v
.pointer("/repository/root")
.and_then(|v| v.as_str())
{
let root_path = Path::new(root);
if root_path.exists() {
return extract_graph(&ExtractOptions {
path: root_path.to_path_buf(),
out_dir: out_dir.to_path_buf(),
});
}
}
// Fallback: empty-ish graph with edges only
let mut edges = Vec::new();
if let Some(arr) = edges_v.get("edges").and_then(|e| e.as_array()) {
for e in arr {
let kind = match e.get("kind").and_then(|k| k.as_str()).unwrap_or("") {
"implements" => crate::knowledge::ir::EdgeKind::Implements,
"uses" => crate::knowledge::ir::EdgeKind::Uses,
"depends_on" => crate::knowledge::ir::EdgeKind::DependsOn,
"tested_by" => crate::knowledge::ir::EdgeKind::TestedBy,
"documented_by" => crate::knowledge::ir::EdgeKind::DocumentedBy,
"constrained_by" => crate::knowledge::ir::EdgeKind::ConstrainedBy,
"decided_by" => crate::knowledge::ir::EdgeKind::DecidedBy,
"supersedes" => crate::knowledge::ir::EdgeKind::Supersedes,
"conflicts_with" => crate::knowledge::ir::EdgeKind::ConflictsWith,
"deprecated_by" => crate::knowledge::ir::EdgeKind::DeprecatedBy,
"derived_from" => crate::knowledge::ir::EdgeKind::DerivedFrom,
"failed_by" => crate::knowledge::ir::EdgeKind::FailedBy,
_ => crate::knowledge::ir::EdgeKind::RelatedTo,
};
edges.push(crate::knowledge::ir::Edge {
from: e.get("from").and_then(|v| v.as_str()).unwrap_or("").into(),
to: e.get("to").and_then(|v| v.as_str()).unwrap_or("").into(),
kind,
note: e.get("note").and_then(|v| v.as_str()).map(|s| s.to_string()),
});
}
}
Ok(KnowledgeGraph {
repository: crate::knowledge::ir::RepositoryInfo {
title: graph_v
.pointer("/repository/title")
.and_then(|v| v.as_str())
.unwrap_or("marqdo-project")
.into(),
root: graph_v
.pointer("/repository/root")
.and_then(|v| v.as_str())
.unwrap_or(".")
.into(),
languages: vec!["marqdo".into()],
module_count: 0,
symbol_count: 0,
capability_count: 0,
knowledge_count: 0,
},
modules: Vec::new(),
symbols: Vec::new(),
capabilities: Vec::new(),
knowledge: Vec::new(),
edges,
generated_by: graph_v
.pointer("/generated/by")
.and_then(|v| v.as_str())
.unwrap_or("marqdo")
.into(),
})
}
pub fn ensure_compiled(path: &Path, out_dir: &Path) -> Result<KnowledgeGraph> {
let graph_path = out_dir.join("graph").join("graph.json");
if graph_path.exists() {
// Always refresh from source when path given
return compile(KnowledgeOptions {
path: path.to_path_buf(),
out_dir: out_dir.to_path_buf(),
});
}
compile(KnowledgeOptions {
path: path.to_path_buf(),
out_dir: out_dir.to_path_buf(),
})
}
fn write_projections(out_dir: &Path, graph: &KnowledgeGraph) -> Result<()> {
let dirs = [
"catalog/modules",
"catalog/symbols",
"catalog/capabilities",
"knowledge/facts",
"knowledge/decisions",
"knowledge/constraints",
"knowledge/patterns",
"knowledge/failures",
"knowledge/migrations",
"graph",
"agent/policies",
"agent/skills",
"agent/contexts",
"agent/episodes",
"tests/reuse",
"tests/knowledge",
"tests/retrieval",
];
for d in dirs {
fs::create_dir_all(out_dir.join(d))
.with_context(|| format!("create {}/{}", out_dir.display(), d))?;
}
// L0 repository
fs::write(
out_dir.join("catalog/repository.mq.md"),
render_repository(graph),
)?;
// L1 modules
for m in &graph.modules {
let stem = module_stem(&m.resource);
fs::write(
out_dir.join("catalog/modules").join(format!("{stem}.md")),
render_module(m, graph),
)?;
}
// L2 symbols
for s in &graph.symbols {
let stem = slugify(&s.id.replace("::", "__").replace('/', "__"));
fs::write(
out_dir.join("catalog/symbols").join(format!("{stem}.md")),
render_symbol(s),
)?;
}
// L3 capabilities
for c in &graph.capabilities {
let stem = slugify(&c.id);
fs::write(
out_dir
.join("catalog/capabilities")
.join(format!("{stem}.md")),
render_capability(c),
)?;
}
// L4 knowledge
for k in &graph.knowledge {
let dir = out_dir.join("knowledge").join(k.kind.subdir());
let stem = slugify(&k.id.rsplit('/').next().unwrap_or(&k.id));
let path = dir.join(format!("{stem}.md"));
let body = if let Some(ref src) = k.source_text {
// Ensure generated marker without destroying authored content
if src.contains("GENERATED by marqdo") {
src.clone()
} else {
format!(
"<!-- projected by marqdo knowledge from {} -->\n{src}",
k.resource
)
}
} else {
render_knowledge(k)
};
fs::write(&path, body)?;
}
// Graph JSON
let graph_json = serde_json::to_string_pretty(&graph.to_graph_json())?;
fs::write(out_dir.join("graph/graph.json"), graph_json)?;
let edges_json = serde_json::to_string_pretty(&graph.to_edges_json())?;
fs::write(out_dir.join("graph/edges.json"), edges_json)?;
// Search index
let index = json!({
"type": "Marqdo Knowledge Index",
"generated": { "by": graph.generated_by },
"entries": graph.index_map(),
});
fs::write(
out_dir.join("graph/index.json"),
serde_json::to_string_pretty(&index)?,
)?;
// Root index.mq.md
fs::write(out_dir.join("index.mq.md"), render_index_mq(graph))?;
// Augment root catalog.yaml if present (append engineering summary)
write_engineering_summary(out_dir, graph)?;
Ok(())
}
fn write_engineering_summary(out_dir: &Path, graph: &KnowledgeGraph) -> Result<()> {
let path = out_dir.join("engineering.yaml");
let mut out = String::new();
out.push_str("# GENERATED by marqdo knowledge — do not edit by hand\n");
out.push_str("type: Marqdo Engineering Knowledge\n");
out.push_str(&format!("title: {}\n", yaml_escape(&graph.repository.title)));
out.push_str("generated:\n");
out.push_str(&format!(" by: {}\n", yaml_escape(&graph.generated_by)));
out.push_str("layers:\n");
out.push_str(&format!(" modules: {}\n", graph.modules.len()));
out.push_str(&format!(" symbols: {}\n", graph.symbols.len()));
out.push_str(&format!(" capabilities: {}\n", graph.capabilities.len()));
out.push_str(&format!(" knowledge: {}\n", graph.knowledge.len()));
out.push_str(&format!(" edges: {}\n", graph.edges.len()));
out.push_str("capabilities:\n");
if graph.capabilities.is_empty() {
out.push_str(" []\n");
} else {
for c in &graph.capabilities {
out.push_str(&format!(" - id: {}\n", yaml_escape(&c.id)));
out.push_str(&format!(" name: {}\n", yaml_escape(&c.name)));
out.push_str(&format!(" status: {}\n", c.status.as_str()));
if !c.implemented_by.is_empty() {
out.push_str(" implemented_by:\n");
for s in &c.implemented_by {
out.push_str(&format!(" - {}\n", yaml_escape(s)));
}
}
if !c.possible_duplicates.is_empty() {
out.push_str(" possible_duplicates:\n");
for s in &c.possible_duplicates {
out.push_str(&format!(" - {}\n", yaml_escape(s)));
}
}
}
}
out.push_str("knowledge:\n");
if graph.knowledge.is_empty() {
out.push_str(" []\n");
} else {
for k in &graph.knowledge {
out.push_str(&format!(" - id: {}\n", yaml_escape(&k.id)));
out.push_str(&format!(" type: {}\n", k.kind.as_str()));
out.push_str(&format!(" title: {}\n", yaml_escape(&k.title)));
out.push_str(&format!(" status: {}\n", k.status.as_str()));
}
}
fs::write(path, out)?;
Ok(())
}
fn render_repository(graph: &KnowledgeGraph) -> String {
let r = &graph.repository;
format!(
r#"---
type: Marqdo Repository
title: {title}
generated:
by: {by}
status: stable
---
# {title}
> GENERATED by Marqdo Engineering Knowledge Compiler. Do not edit by hand.
## Repository
| Field | Value |
|-------|-------|
| Languages | {langs} |
| Modules | {modules} |
| Symbols | {symbols} |
| Capabilities | {caps} |
| Knowledge | {know} |
| Edges | {edges} |
## Layers
- L0 Repository — this page
- L1 Modules — `catalog/modules/`
- L2 Symbols — `catalog/symbols/`
- L3 Capabilities — `catalog/capabilities/`
- L4 Engineering Knowledge — `knowledge/`
"#,
title = r.title,
by = graph.generated_by,
langs = r.languages.join(", "),
modules = r.module_count,
symbols = r.symbol_count,
caps = r.capability_count,
know = r.knowledge_count,
edges = graph.edges.len(),
)
}
fn render_module(m: &crate::knowledge::ir::ModuleInfo, graph: &KnowledgeGraph) -> String {
let mut exports = String::new();
if m.exports.is_empty() {
exports.push_str("_None_\n");
} else {
exports.push_str("| Symbol | Kind |\n|--------|------|\n");
for e in &m.exports {
let kind = graph
.symbols
.iter()
.find(|s| s.module == m.id && s.name == *e)
.map(|s| s.kind.as_str())
.unwrap_or("fn");
exports.push_str(&format!("| `{e}` | {kind} |\n"));
}
}
let resp = m.responsibility.as_deref().unwrap_or("_unspecified_");
format!(
r#"---
type: Marqdo Module
title: {title}
resource: {resource}
status: {status}
generated:
by: {by}
---
# {title}
> Auto-generated from `{resource}`. Do not edit by hand.
## Responsibility
{resp}
## Exports
{exports}
"#,
title = m.title,
resource = m.resource,
status = m.status.as_str(),
by = graph.generated_by,
resp = resp,
exports = exports,
)
}
fn render_symbol(s: &crate::knowledge::ir::SymbolInfo) -> String {
let params = if s.params.is_empty() {
"[]".into()
} else {
s.params
.iter()
.map(|p| format!(" - {p}"))
.collect::<Vec<_>>()
.join("\n")
};
let use_when = s.use_when.as_deref().unwrap_or("_unspecified_");
let do_not = s.do_not_use_when.as_deref().unwrap_or("_unspecified_");
format!(
r#"---
type: Function
name: {name}
module: {module}
symbol: {name}
resource: {resource}
status: {status}
visibility: {vis}
fingerprint: {fp}
capability: {cap}
generated:
by: marqdo/knowledge
---
# {name}
## Use when
{use_when}
## Do not use when
{do_not}
## Signature
params:
{params}
## Calls
{calls}
## Fingerprint
```
{fpt}
```
"#,
name = s.name,
module = s.module,
resource = s.resource,
status = s.status.as_str(),
vis = s.visibility,
fp = s.fingerprint,
cap = s.capability.as_deref().unwrap_or(""),
use_when = use_when,
do_not = do_not,
params = params,
calls = if s.calls.is_empty() {
"_None_".into()
} else {
s.calls
.iter()
.map(|c| format!("- `{c}`"))
.collect::<Vec<_>>()
.join("\n")
},
fpt = s.fingerprint_text,
)
}
fn render_capability(c: &crate::knowledge::ir::CapabilityInfo) -> String {
let use_when = c.use_when.as_deref().unwrap_or(&c.description);
let do_not = c.do_not_use_when.as_deref().unwrap_or("_unspecified_");
format!(
r#"---
type: Capability
id: {id}
name: {name}
status: {status}
generated:
by: marqdo/knowledge
---
# {name}
{desc}
## Use when
{use_when}
## Do not use when
{do_not}
## Implemented by
{impls}
## Possible duplicates
{dups}
"#,
id = c.id,
name = c.name,
status = c.status.as_str(),
desc = c.description,
use_when = use_when,
do_not = do_not,
impls = if c.implemented_by.is_empty() {
"_None_".into()
} else {
c.implemented_by
.iter()
.map(|s| format!("- `{s}`"))
.collect::<Vec<_>>()
.join("\n")
},
dups = if c.possible_duplicates.is_empty() {
"_None_".into()
} else {
c.possible_duplicates
.iter()
.map(|s| format!("- `{s}`"))
.collect::<Vec<_>>()
.join("\n")
},
)
}
fn render_knowledge(k: &crate::knowledge::ir::KnowledgeItem) -> String {
format!(
r#"---
type: {ty}
id: {id}
title: {title}
status: {status}
generated:
by: marqdo/knowledge
---
# {title}
{summary}
"#,
ty = k.kind.as_str(),
id = k.id,
title = k.title,
status = k.status.as_str(),
summary = k.body_summary,
)
}
fn render_index_mq(graph: &KnowledgeGraph) -> String {
format!(
r#"---
type: Marqdo Engineering Knowledge
title: Engineering Knowledge Index
generated:
by: {by}
---
# Engineering Knowledge Index
> GENERATED. Source of truth remains `*.mq.md`. This tree is a Knowledge Projection.
## Counts
| Layer | Count |
|-------|------:|
| Modules | {modules} |
| Symbols | {symbols} |
| Capabilities | {caps} |
| Knowledge | {know} |
| Edges | {edges} |
## Commands
```
marqdo knowledge
marqdo find "query"
marqdo reuse "task"
marqdo duplicate
marqdo impact PATH
marqdo conflicts
marqdo stale
marqdo verify
```
"#,
by = graph.generated_by,
modules = graph.modules.len(),
symbols = graph.symbols.len(),
caps = graph.capabilities.len(),
know = graph.knowledge.len(),
edges = graph.edges.len(),
)
}
+104
View File
@@ -0,0 +1,104 @@
//! Duplicate capability / symbol detection via fingerprints.
use crate::knowledge::fingerprint::symbol_similarity;
use crate::knowledge::ir::{CapabilityInfo, Edge, EdgeKind, SymbolInfo};
#[derive(Debug, Clone)]
pub struct DuplicatePair {
pub left: String,
pub right: String,
pub similarity: f64,
pub left_resource: String,
pub right_resource: String,
}
pub fn find_duplicates(symbols: &[SymbolInfo], threshold: f64) -> Vec<DuplicatePair> {
let mut pairs = Vec::new();
for i in 0..symbols.len() {
if symbols[i].kind == "test" {
continue;
}
for j in (i + 1)..symbols.len() {
if symbols[j].kind == "test" {
continue;
}
if symbols[i].module == symbols[j].module && symbols[i].name == symbols[j].name {
continue;
}
let sim = symbol_similarity(&symbols[i], &symbols[j]);
if sim >= threshold {
pairs.push(DuplicatePair {
left: symbols[i].id.clone(),
right: symbols[j].id.clone(),
similarity: sim,
left_resource: symbols[i].resource.clone(),
right_resource: symbols[j].resource.clone(),
});
}
}
}
pairs.sort_by(|a, b| {
b.similarity
.partial_cmp(&a.similarity)
.unwrap_or(std::cmp::Ordering::Equal)
});
pairs
}
/// During extract: add conflicts_with-style related edges for near-duplicates.
pub fn annotate_duplicates(
symbols: &[SymbolInfo],
edges: &mut Vec<Edge>,
_capabilities: &[CapabilityInfo],
) {
for pair in find_duplicates(symbols, 0.75) {
edges.push(Edge {
from: format!("symbol:{}", pair.left),
to: format!("symbol:{}", pair.right),
kind: EdgeKind::ConflictsWith,
note: Some(format!("possible_duplicate:{:.2}", pair.similarity)),
});
}
}
pub fn apply_duplicate_flags(
mut capabilities: Vec<CapabilityInfo>,
symbols: &[SymbolInfo],
) -> Vec<CapabilityInfo> {
let pairs = find_duplicates(symbols, 0.75);
for pair in &pairs {
for c in &mut capabilities {
if c.implemented_by.iter().any(|s| s == &pair.left) {
if !c.possible_duplicates.contains(&pair.right) {
c.possible_duplicates.push(pair.right.clone());
}
}
if c.implemented_by.iter().any(|s| s == &pair.right) {
if !c.possible_duplicates.contains(&pair.left) {
c.possible_duplicates.push(pair.left.clone());
}
}
}
}
capabilities
}
pub fn format_duplicates_text(pairs: &[DuplicatePair]) -> String {
let mut out = String::from("Possible duplicate capabilities\n\n");
if pairs.is_empty() {
out.push_str("_None above threshold._\n");
return out;
}
for (i, p) in pairs.iter().enumerate() {
out.push_str(&format!(
"{}. {}::{}\n {}::{}\n similarity: {:.2}\n\n",
i + 1,
p.left_resource,
p.left.rsplit("::").next().unwrap_or(&p.left),
p.right_resource,
p.right.rsplit("::").next().unwrap_or(&p.right),
p.similarity
));
}
out
}
+528
View File
@@ -0,0 +1,528 @@
//! Extract Engineering Knowledge IR from a Marqdo source tree.
use std::fs;
use std::path::{Path, PathBuf};
use anyhow::{Context, Result};
use crate::ast::Function;
use crate::knowledge::fingerprint::{collect_calls, fingerprint_hash, fingerprint_text, ops_hint};
use crate::knowledge::ir::{
Confidence, Evidence, KnowledgeGraph, KnowledgeItem, KnowledgeKind, Lifecycle, ModuleInfo,
RepositoryInfo, SymbolInfo,
};
use crate::knowledge::util::{
extract_fm_field, extract_fm_list, extract_nested_by, extract_section, first_paragraph,
module_stem, slugify,
};
use crate::lex::{classify_source, LineKind};
use crate::parse::parse_source;
pub struct ExtractOptions {
pub path: PathBuf,
pub out_dir: PathBuf,
}
pub fn extract_graph(opts: &ExtractOptions) -> Result<KnowledgeGraph> {
let root = if opts.path.is_file() {
opts.path
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| PathBuf::from("."))
} else {
opts.path.clone()
};
let root = root.canonicalize().unwrap_or(root);
let out_canon = opts.out_dir.canonicalize().unwrap_or(opts.out_dir.clone());
let version = env!("CARGO_PKG_VERSION");
let mut files = Vec::new();
if opts.path.is_file() {
files.push(opts.path.canonicalize().unwrap_or(opts.path.clone()));
} else {
collect_mq_md(&root, &root, &out_canon, &mut files)?;
files.sort();
}
let mut modules = Vec::new();
let mut symbols = Vec::new();
let mut knowledge = Vec::new();
let mut had_err = false;
for abs in &files {
let rel = abs
.strip_prefix(&root)
.unwrap_or(abs)
.to_string_lossy()
.replace('\\', "/");
if rel.contains("/.marqdo/") || rel.starts_with(".marqdo/") {
continue;
}
match inspect_source(abs, &rel) {
Ok((m, syms, ks)) => {
modules.push(m);
symbols.extend(syms);
knowledge.extend(ks);
}
Err(e) => {
eprintln!("knowledge: skip {rel}: {e:#}");
had_err = true;
}
}
}
// Also scan authored knowledge markdown (non-.mq.md) under knowledge/ or doc/adr/
collect_authored_knowledge(&root, &out_canon, &mut knowledge)?;
// Wire called_by
let mut call_index: std::collections::HashMap<String, Vec<String>> =
std::collections::HashMap::new();
for s in &symbols {
for c in &s.calls {
let short = c.rsplit('.').next().unwrap_or(c).to_string();
call_index.entry(short).or_default().push(s.id.clone());
call_index.entry(c.clone()).or_default().push(s.id.clone());
}
}
for s in &mut symbols {
let mut cb = call_index.remove(&s.name).unwrap_or_default();
cb.extend(call_index.remove(&s.id).unwrap_or_default());
cb.sort();
cb.dedup();
s.called_by = cb;
}
// Detect tests: paths containing /tests/ or name starting with test_
for s in &mut symbols {
if s.resource.contains("/tests/") || s.resource.starts_with("tests/") {
// module-level test file — mark symbols as tests for same-named exports later
}
if s.name.starts_with("test_") || s.name.starts_with("测试") {
s.kind = "test".into();
}
}
let test_syms: Vec<(String, String)> = symbols
.iter()
.filter(|s| s.kind == "test" || s.resource.contains("/tests/") || s.resource.starts_with("tests/"))
.map(|s| (s.name.clone(), s.id.clone()))
.collect();
for s in &mut symbols {
for (tname, tid) in &test_syms {
if tname.contains(&s.name) || s.tests.iter().any(|t| t == tid) {
if !s.tests.contains(tid) && tid != &s.id {
s.tests.push(tid.clone());
}
}
}
}
let capabilities = crate::knowledge::capability::derive_capabilities(&symbols, &modules);
let mut edges = crate::knowledge::graph::build_edges(&modules, &symbols, &capabilities, &knowledge);
crate::knowledge::duplicate::annotate_duplicates(&symbols, &mut edges, &capabilities);
// Re-derive capability duplicate links after annotate (annotate mutates via returned list)
let capabilities =
crate::knowledge::duplicate::apply_duplicate_flags(capabilities, &symbols);
let repository = RepositoryInfo {
title: root
.file_name()
.and_then(|s| s.to_str())
.unwrap_or("marqdo-project")
.to_string(),
root: root.to_string_lossy().to_string(),
languages: vec!["marqdo".into()],
module_count: modules.len(),
symbol_count: symbols.len(),
capability_count: capabilities.len(),
knowledge_count: knowledge.len(),
};
if had_err {
eprintln!("knowledge: extraction completed with some skipped files");
}
Ok(KnowledgeGraph {
repository,
modules,
symbols,
capabilities,
knowledge,
edges,
generated_by: format!("marqdo/{version}"),
})
}
fn collect_mq_md(
root: &Path,
dir: &Path,
out_canon: &Path,
out: &mut Vec<PathBuf>,
) -> Result<()> {
for entry in fs::read_dir(dir)? {
let entry = entry?;
let p = entry.path();
if p.is_dir() {
let name = p.file_name().and_then(|s| s.to_str()).unwrap_or("");
if name == "target"
|| name == ".git"
|| name == "node_modules"
|| name == "docker"
|| name == "data"
|| name == ".cursor"
{
continue;
}
if let Ok(canon) = p.canonicalize() {
if canon == *out_canon {
continue;
}
}
collect_mq_md(root, &p, out_canon, out)?;
} else if p
.file_name()
.and_then(|s| s.to_str())
.map(|s| s.ends_with(".mq.md"))
.unwrap_or(false)
{
out.push(p);
}
}
Ok(())
}
fn collect_authored_knowledge(
root: &Path,
out_canon: &Path,
out: &mut Vec<KnowledgeItem>,
) -> Result<()> {
walk_knowledge_md(root, root, out_canon, out)
}
fn walk_knowledge_md(
root: &Path,
dir: &Path,
out_canon: &Path,
out: &mut Vec<KnowledgeItem>,
) -> Result<()> {
for entry in fs::read_dir(dir)? {
let entry = entry?;
let p = entry.path();
if p.is_dir() {
let name = p.file_name().and_then(|s| s.to_str()).unwrap_or("");
if name == "target"
|| name == ".git"
|| name == ".marqdo"
|| name == "docker"
|| name == "data"
|| name == ".cursor"
{
continue;
}
if let Ok(canon) = p.canonicalize() {
if canon == *out_canon {
continue;
}
}
walk_knowledge_md(root, &p, out_canon, out)?;
continue;
}
let Some(name) = p.file_name().and_then(|s| s.to_str()) else {
continue;
};
// Authored knowledge pages: .md with engineering type, not .mq.md (those go through inspect)
if !name.ends_with(".md") || name.ends_with(".mq.md") {
continue;
}
let source = fs::read_to_string(&p)?;
let Some(ty) = extract_fm_field(&source, "type") else {
continue;
};
let kind = KnowledgeKind::parse(&ty);
match kind {
KnowledgeKind::Decision
| KnowledgeKind::Constraint
| KnowledgeKind::Failure
| KnowledgeKind::AntiPattern
| KnowledgeKind::Pattern
| KnowledgeKind::Migration
| KnowledgeKind::Fact => {}
_ => continue,
}
let rel = p
.strip_prefix(root)
.unwrap_or(&p)
.to_string_lossy()
.replace('\\', "/");
out.push(knowledge_from_source(&source, &rel, kind));
}
Ok(())
}
fn knowledge_from_source(source: &str, rel: &str, kind: KnowledgeKind) -> KnowledgeItem {
let title = extract_fm_field(source, "title")
.or_else(|| {
source.lines().find_map(|l| {
l.trim()
.strip_prefix("# ")
.map(|s| s.trim().to_string())
})
})
.unwrap_or_else(|| rel.to_string());
let id = extract_fm_field(source, "id").unwrap_or_else(|| {
format!(
"{}/{}",
kind.subdir(),
slugify(&title)
)
});
let status = Lifecycle::parse(
&extract_fm_field(source, "status").unwrap_or_else(|| "stable".into()),
);
let applies_to = extract_fm_list(source, "applies_to");
let supersedes = extract_fm_list(source, "supersedes");
let conflicts_with = extract_fm_list(source, "conflicts_with");
let related = extract_fm_list(source, "related");
let stale_after = extract_fm_field(source, "stale_after");
let summary = first_paragraph(source);
let mut evidence = Evidence::default();
evidence.sources.push(rel.to_string());
KnowledgeItem {
id,
kind,
title,
resource: rel.to_string(),
status,
body_summary: summary,
applies_to,
supersedes,
conflicts_with,
related,
stale_after,
evidence,
confidence: Confidence {
level: "high".into(),
implementation: "authored".into(),
tests: "unknown".into(),
runtime: "unknown".into(),
},
source_text: Some(source.to_string()),
}
}
fn inspect_source(
abs: &Path,
rel: &str,
) -> Result<(ModuleInfo, Vec<SymbolInfo>, Vec<KnowledgeItem>)> {
let source = fs::read_to_string(abs).with_context(|| format!("read {}", abs.display()))?;
let imports = extract_frontmatter_imports(&source);
let title = extract_fm_field(&source, "title")
.unwrap_or_else(|| rel.trim_end_matches(".mq.md").to_string());
let verified_by = extract_nested_by(&source, "verified");
let sources = extract_fm_list(&source, "sources");
let status = Lifecycle::parse(
&extract_fm_field(&source, "status").unwrap_or_else(|| "stable".into()),
);
let responsibility = first_paragraph(&source);
let responsibility = if responsibility.is_empty() {
None
} else {
Some(responsibility)
};
let mut knowledge = Vec::new();
if let Some(ty) = extract_fm_field(&source, "type") {
let kind = KnowledgeKind::parse(&ty);
match kind {
KnowledgeKind::Decision
| KnowledgeKind::Constraint
| KnowledgeKind::Failure
| KnowledgeKind::AntiPattern
| KnowledgeKind::Pattern
| KnowledgeKind::Migration
| KnowledgeKind::Fact => {
knowledge.push(knowledge_from_source(&source, rel, kind));
}
_ => {}
}
}
let module_id = rel.trim_end_matches(".mq.md").to_string();
let mut symbols = Vec::new();
match parse_source(&source) {
Ok(module) => {
for f in &module.functions {
push_function_symbols(f, &module_id, rel, &source, &mut symbols);
}
}
Err(_) => {
for line in classify_source(&source) {
if line.kind != LineKind::Code {
continue;
}
let t = line.text.trim();
if let Some(rest) = t.strip_prefix("# ") {
if !rest.starts_with('#') {
let name = rest
.split('=')
.next()
.unwrap_or(rest)
.trim()
.to_string();
let id = format!("{module_id}::{name}");
let fp = fingerprint_text(&name, &[], &[], &[]);
symbols.push(SymbolInfo {
id,
name,
module: module_id.clone(),
resource: rel.to_string(),
kind: "fn".into(),
visibility: "public".into(),
params: Vec::new(),
calls: Vec::new(),
called_by: Vec::new(),
tests: Vec::new(),
fingerprint: fingerprint_hash(&fp),
fingerprint_text: fp,
use_when: None,
do_not_use_when: None,
related: Vec::new(),
reuse: extract_fm_list(&source, "reuse"),
capability: extract_fm_field(&source, "capability"),
status: status.clone(),
evidence: Evidence {
sources: vec![rel.to_string()],
..Evidence::default()
},
confidence: Confidence::default(),
});
}
}
}
}
}
let exports: Vec<String> = symbols.iter().map(|s| s.name.clone()).collect();
let module = ModuleInfo {
id: module_id,
resource: rel.to_string(),
title,
imports,
exports,
responsibility,
verified_by,
sources,
status,
};
Ok((module, symbols, knowledge))
}
fn push_function_symbols(
f: &Function,
module_id: &str,
rel: &str,
source: &str,
out: &mut Vec<SymbolInfo>,
) {
let name = f.name.clone();
let id = format!("{module_id}::{name}");
let params: Vec<String> = f.params.iter().map(|p| p.name.clone()).collect();
let calls = collect_calls(f);
let ops = ops_hint(f);
let fp = fingerprint_text(&name, &params, &calls, &ops);
let kind = if f.is_object() {
"object".to_string()
} else {
"fn".to_string()
};
// Prefer function-local sections if present in source near heading — fall back to file-level
let use_when = extract_section(source, "Use when").or_else(|| prose_field(source, "Use when"));
let do_not = extract_section(source, "Do not use when")
.or_else(|| prose_field(source, "Do not use when"));
let related = extract_fm_list(source, "related");
let reuse = extract_fm_list(source, "reuse");
let capability = extract_fm_field(source, "capability");
let status = Lifecycle::parse(
&extract_fm_field(source, "status").unwrap_or_else(|| "stable".into()),
);
out.push(SymbolInfo {
id,
name,
module: module_id.to_string(),
resource: rel.to_string(),
kind,
visibility: "public".into(),
params,
calls,
called_by: Vec::new(),
tests: Vec::new(),
fingerprint: fingerprint_hash(&fp),
fingerprint_text: fp,
use_when,
do_not_use_when: do_not,
related,
reuse,
capability,
status,
evidence: Evidence {
sources: vec![rel.to_string()],
..Evidence::default()
},
confidence: Confidence {
level: "high".into(),
implementation: "verified".into(),
tests: "unknown".into(),
runtime: "unknown".into(),
},
});
for child in &f.children {
push_function_symbols(child, module_id, rel, source, out);
}
}
fn prose_field(source: &str, label: &str) -> Option<String> {
let prefix = format!("{label}:");
for line in source.lines() {
let t = line.trim();
if let Some(rest) = t.strip_prefix(&prefix) {
let v = rest.trim();
if !v.is_empty() {
return Some(v.to_string());
}
}
}
None
}
fn extract_frontmatter_imports(source: &str) -> Vec<String> {
let source = source.strip_prefix('\u{feff}').unwrap_or(source);
let mut imports = Vec::new();
let lines: Vec<&str> = source.lines().collect();
if lines.first().map(|l| l.trim()) != Some("---") {
return imports;
}
for line in lines.iter().skip(1) {
let t = line.trim();
if t == "---" {
break;
}
if let Ok(Some(crate::parse::ImportLine::File(imp))) =
crate::parse::parse_import_line(t)
{
imports.push(format!("import {}:{}", imp.bind, imp.path));
} else if let Ok(Some(crate::parse::ImportLine::Member(u))) =
crate::parse::parse_import_line(t)
{
imports.push(format!("import {}:{}", u.bind, u.path.join(".")));
}
}
imports
}
#[allow(dead_code)]
pub fn stem_of(resource: &str) -> String {
module_stem(resource)
}
+231
View File
@@ -0,0 +1,231 @@
//! Hybrid knowledge find (lexical + symbol + capability + graph).
use crate::knowledge::fingerprint::text_similarity;
use crate::knowledge::ir::{KnowledgeGraph, KnowledgeKind};
use serde_json::{json, Value};
#[derive(Debug, Clone)]
pub struct FindHit {
pub id: String,
pub kind: String,
pub title: String,
pub score: f64,
pub resource: Option<String>,
pub snippet: String,
}
pub fn find(graph: &KnowledgeGraph, query: &str, limit: usize) -> Vec<FindHit> {
let q = query.trim();
if q.is_empty() {
return Vec::new();
}
let mut hits = Vec::new();
for c in &graph.capabilities {
let material = format!(
"{} {} {} {} {}",
c.name,
c.description,
c.use_when.as_deref().unwrap_or(""),
c.do_not_use_when.as_deref().unwrap_or(""),
c.implemented_by.join(" ")
);
let mut score = text_similarity(q, &material);
if c.name.to_ascii_lowercase().contains(&q.to_ascii_lowercase())
|| q.to_ascii_lowercase()
.split_whitespace()
.any(|t| c.name.to_ascii_lowercase().contains(t))
{
score = (score + 0.35).min(1.0);
}
if score > 0.15 {
hits.push(FindHit {
id: format!("capability:{}", c.id),
kind: "Capability".into(),
title: c.name.clone(),
score,
resource: c
.implemented_by
.first()
.and_then(|sid| graph.symbol_by_id(sid))
.map(|s| s.resource.clone()),
snippet: c.description.clone(),
});
}
}
for s in &graph.symbols {
if s.kind == "test" {
continue;
}
let material = format!(
"{} {} {} {} {}",
s.name,
s.fingerprint_text,
s.use_when.as_deref().unwrap_or(""),
s.module,
s.calls.join(" ")
);
let mut score = text_similarity(q, &material);
if s.name.to_ascii_lowercase().contains(&q.to_ascii_lowercase()) {
score = (score + 0.4).min(1.0);
}
if score > 0.18 {
hits.push(FindHit {
id: format!("symbol:{}", s.id),
kind: "Function".into(),
title: s.name.clone(),
score,
resource: Some(s.resource.clone()),
snippet: s.fingerprint_text.clone(),
});
}
}
for k in &graph.knowledge {
let material = format!("{} {} {}", k.title, k.body_summary, k.kind.as_str());
let mut score = text_similarity(q, &material);
if k.title.to_ascii_lowercase().contains(&q.to_ascii_lowercase()) {
score = (score + 0.35).min(1.0);
}
// Boost engineering knowledge types for governance queries
if matches!(
k.kind,
KnowledgeKind::Constraint
| KnowledgeKind::Decision
| KnowledgeKind::Failure
| KnowledgeKind::AntiPattern
) && score > 0.1
{
score = (score + 0.1).min(1.0);
}
if score > 0.15 {
hits.push(FindHit {
id: format!("knowledge:{}", k.id),
kind: k.kind.as_str().to_string(),
title: k.title.clone(),
score,
resource: Some(k.resource.clone()),
snippet: k.body_summary.clone(),
});
}
}
// Graph expansion: boost neighbors of top hits
hits.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
let top_ids: Vec<String> = hits.iter().take(5).map(|h| h.id.clone()).collect();
for tid in &top_ids {
for e in graph.neighbors(tid, None) {
let other = if e.from == *tid { &e.to } else { &e.from };
if let Some(h) = hits.iter_mut().find(|h| h.id == *other) {
h.score = (h.score + 0.05).min(1.0);
}
}
}
hits.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
hits.truncate(limit);
hits
}
pub fn format_find_text(query: &str, hits: &[FindHit], graph: &KnowledgeGraph) -> String {
let mut out = format!("Engineering First Information\n\nQuery: {query}\n\n");
if hits.is_empty() {
out.push_str("No matching capabilities, symbols, or knowledge.\n");
return out;
}
let caps: Vec<_> = hits.iter().filter(|h| h.kind == "Capability").collect();
let impls: Vec<_> = hits.iter().filter(|h| h.kind == "Function").collect();
let constraints: Vec<_> = hits.iter().filter(|h| h.kind == "Constraint").collect();
let decisions: Vec<_> = hits.iter().filter(|h| h.kind == "Decision").collect();
let failures: Vec<_> = hits
.iter()
.filter(|h| h.kind == "Failure" || h.kind == "AntiPattern")
.collect();
let related: Vec<_> = hits
.iter()
.filter(|h| {
h.kind != "Capability"
&& h.kind != "Function"
&& h.kind != "Constraint"
&& h.kind != "Decision"
&& h.kind != "Failure"
&& h.kind != "AntiPattern"
})
.collect();
if !caps.is_empty() {
out.push_str("Capability\n");
for h in &caps {
out.push_str(&format!(" {} ({:.2})\n", h.title, h.score));
}
out.push('\n');
}
if !impls.is_empty() {
out.push_str("Implementation\n");
for h in &impls {
out.push_str(&format!(
" {}\n",
h.resource.as_deref().unwrap_or(&h.title)
));
}
out.push('\n');
}
if !constraints.is_empty() {
out.push_str("Constraints\n");
for h in &constraints {
out.push_str(&format!(" {}\n", h.title));
}
out.push('\n');
}
if !decisions.is_empty() {
out.push_str("Decisions\n");
for h in &decisions {
out.push_str(&format!(" {}\n", h.title));
}
out.push('\n');
}
if !failures.is_empty() {
out.push_str("Failures\n");
for h in &failures {
out.push_str(&format!(" {}\n", h.title));
}
out.push('\n');
}
if !related.is_empty() {
out.push_str("Related\n");
for h in &related {
out.push_str(&format!(" {} ({})\n", h.title, h.kind));
}
out.push('\n');
}
// Graph-related capabilities for top cap
if let Some(top) = caps.first() {
let id = top.id.strip_prefix("capability:").unwrap_or(&top.id);
if let Some(c) = graph.capability_by_id(id) {
if !c.related.is_empty() {
out.push_str("Related capabilities\n");
for r in &c.related {
out.push_str(&format!(" {r}\n"));
}
}
}
}
out
}
pub fn find_json(query: &str, hits: &[FindHit]) -> Value {
json!({
"query": query,
"hits": hits.iter().map(|h| json!({
"id": h.id,
"kind": h.kind,
"title": h.title,
"score": h.score,
"resource": h.resource,
"snippet": h.snippet,
})).collect::<Vec<_>>(),
})
}
+250
View File
@@ -0,0 +1,250 @@
//! Symbol fingerprints + lexical similarity (char-bigram cosine).
use std::collections::HashMap;
use sha2::{Digest, Sha256};
use crate::ast::{Expr, Function, Stmt};
use crate::knowledge::ir::SymbolInfo;
pub fn char_bigram_tf(text: &str) -> HashMap<String, f64> {
let chars: Vec<char> = text.chars().filter(|c| !c.is_whitespace()).collect();
let mut m = HashMap::new();
if chars.is_empty() {
return m;
}
if chars.len() == 1 {
m.insert(chars[0].to_string(), 1.0);
return m;
}
for w in chars.windows(2) {
let key: String = w.iter().collect();
*m.entry(key).or_insert(0.0) += 1.0;
}
m
}
pub fn cosine_tf(a: &HashMap<String, f64>, b: &HashMap<String, f64>) -> f64 {
let mut na = 0.0;
let mut nb = 0.0;
for v in a.values() {
na += v * v;
}
for v in b.values() {
nb += v * v;
}
if na == 0.0 || nb == 0.0 {
return 0.0;
}
let mut dot = 0.0;
for (k, va) in a {
if let Some(vb) = b.get(k) {
dot += va * vb;
}
}
dot / (na.sqrt() * nb.sqrt())
}
pub fn text_similarity(a: &str, b: &str) -> f64 {
cosine_tf(&char_bigram_tf(a), &char_bigram_tf(b))
}
pub fn tokenize_ident(name: &str) -> Vec<String> {
let mut out = Vec::new();
let mut cur = String::new();
for c in name.chars() {
if c == '_' || c == '-' || c == '.' || c == '/' {
if !cur.is_empty() {
out.push(cur.to_ascii_lowercase());
cur.clear();
}
} else if c.is_uppercase() && !cur.is_empty() && cur.chars().last().is_some_and(|x| x.is_lowercase())
{
out.push(cur.to_ascii_lowercase());
cur.clear();
cur.push(c);
} else {
cur.push(c);
}
}
if !cur.is_empty() {
out.push(cur.to_ascii_lowercase());
}
out
}
fn collect_calls_from_expr(expr: &Expr, out: &mut Vec<String>) {
match expr {
Expr::Call(c) => {
let name = if let Some(path) = &c.path {
path.join(".")
} else if let Some(recv) = &c.receiver {
format!("{recv}.{}", c.callee)
} else {
c.callee.clone()
};
out.push(name);
for a in &c.args {
match a {
crate::ast::Arg::Positional(e) | crate::ast::Arg::Named { value: e, .. } => {
collect_calls_from_expr(e, out);
}
}
}
}
Expr::Unary { expr, .. } => collect_calls_from_expr(expr, out),
Expr::Binary { left, right, .. } => {
collect_calls_from_expr(left, out);
collect_calls_from_expr(right, out);
}
Expr::List(xs) => {
for e in xs {
collect_calls_from_expr(e, out);
}
}
Expr::Map(pairs) => {
for (_, e) in pairs {
collect_calls_from_expr(e, out);
}
}
Expr::Index { base, .. } => collect_calls_from_expr(base, out),
Expr::Interp(parts) => {
for p in parts {
if let crate::ast::InterpPart::Index { .. } = p {
// no call
}
}
}
_ => {}
}
}
fn collect_calls_from_stmts(stmts: &[Stmt], out: &mut Vec<String>) {
for s in stmts {
match s {
Stmt::Assign { value, .. } | Stmt::Return { value, .. } | Stmt::Expr { value, .. } => {
collect_calls_from_expr(value, out);
}
Stmt::Call { call, .. } => {
let name = if let Some(path) = &call.path {
path.join(".")
} else if let Some(recv) = &call.receiver {
format!("{recv}.{}", call.callee)
} else {
call.callee.clone()
};
out.push(name);
for a in &call.args {
match a {
crate::ast::Arg::Positional(e) | crate::ast::Arg::Named { value: e, .. } => {
collect_calls_from_expr(e, out);
}
}
}
}
Stmt::Branch { arms, .. } => {
for arm in arms {
if let Some(c) = &arm.condition {
collect_calls_from_expr(c, out);
}
collect_calls_from_stmts(&arm.body, out);
}
}
Stmt::While {
condition, body, ..
} => {
collect_calls_from_expr(condition, out);
collect_calls_from_stmts(body, out);
}
Stmt::ForEach { body, .. } => collect_calls_from_stmts(body, out),
}
}
}
pub fn collect_calls(f: &Function) -> Vec<String> {
let mut out = Vec::new();
collect_calls_from_stmts(&f.body, &mut out);
for child in &f.children {
out.extend(collect_calls(child));
}
out.sort();
out.dedup();
out
}
pub fn ops_hint(f: &Function) -> Vec<String> {
let mut ops = Vec::new();
let name = f.name.to_ascii_lowercase();
for verb in [
"parse", "merge", "validate", "normalize", "resolve", "load", "save", "read", "write",
"encode", "decode", "verify", "auth", "cache", "retry", "serialize", "render", "query",
"create", "update", "delete", "check", "format", "convert",
] {
if name.contains(verb) {
ops.push(verb.to_string());
}
}
for c in collect_calls(f) {
let low = c.to_ascii_lowercase();
for verb in ["parse", "merge", "validate", "json", "http", "hash", "auth"] {
if low.contains(verb) && !ops.iter().any(|o| o == verb) {
ops.push(verb.to_string());
}
}
}
// Body-size / structure hints keep near-duplicates discoverable
let body_n = f.body.len();
if body_n > 0 {
ops.push(format!("stmts:{body_n}"));
}
if !f.params.is_empty() {
ops.push(format!("arity:{}", f.params.len()));
}
ops
}
pub fn fingerprint_text(name: &str, params: &[String], calls: &[String], ops: &[String]) -> String {
let mut parts = Vec::new();
parts.push(format!("name:{}", tokenize_ident(name).join("_")));
if !params.is_empty() {
parts.push(format!("params:{}", params.join(",")));
}
if !calls.is_empty() {
let mut c = calls.to_vec();
c.sort();
parts.push(format!("calls:{}", c.join(",")));
}
if !ops.is_empty() {
parts.push(format!("ops:{}", ops.join(",")));
}
parts.join("|")
}
pub fn fingerprint_hash(text: &str) -> String {
let mut h = Sha256::new();
h.update(text.as_bytes());
format!("{:x}", h.finalize())[..16].to_string()
}
pub fn symbol_similarity(a: &SymbolInfo, b: &SymbolInfo) -> f64 {
let mut score = text_similarity(&a.fingerprint_text, &b.fingerprint_text);
let name_sim = text_similarity(&a.name, &b.name);
// Token overlap on snake_case names (resolve_config vs normalize_settings share little;
// merge_config vs merge_settings share "merge")
let ta = tokenize_ident(&a.name);
let tb = tokenize_ident(&b.name);
let mut overlap = 0.0;
if !ta.is_empty() && !tb.is_empty() {
let shared = ta.iter().filter(|t| tb.contains(t)).count() as f64;
overlap = shared / (ta.len().max(tb.len()) as f64);
}
let param_bonus = if !a.params.is_empty() && a.params == b.params {
0.25
} else if a.params.len() == b.params.len() && !a.params.is_empty() {
0.1
} else {
0.0
};
score = (score * 0.25) + (name_sim * 0.15) + (overlap * 0.45) + param_bonus;
score.min(1.0)
}
+170
View File
@@ -0,0 +1,170 @@
//! Knowledge graph edge construction.
use crate::knowledge::ir::{
CapabilityInfo, Edge, EdgeKind, KnowledgeItem, KnowledgeKind, ModuleInfo, SymbolInfo,
};
pub fn build_edges(
modules: &[ModuleInfo],
symbols: &[SymbolInfo],
capabilities: &[CapabilityInfo],
knowledge: &[KnowledgeItem],
) -> Vec<Edge> {
let mut edges = Vec::new();
// Module depends_on
for m in modules {
for imp in &m.imports {
let path = imp
.split(':')
.nth(1)
.unwrap_or(imp)
.trim()
.to_string();
if let Some(dep) = modules.iter().find(|o| {
o.resource == path
|| o.resource.ends_with(&path)
|| o.resource.ends_with(&format!("/{path}"))
|| o.id == path.trim_end_matches(".mq.md")
}) {
edges.push(Edge {
from: format!("module:{}", m.id),
to: format!("module:{}", dep.id),
kind: EdgeKind::DependsOn,
note: Some(imp.clone()),
});
}
}
}
// Symbol uses (calls)
for s in symbols {
for c in &s.calls {
let short = c.rsplit('.').next().unwrap_or(c);
if let Some(target) = symbols.iter().find(|t| t.name == short || t.name == *c || t.id == *c)
{
edges.push(Edge {
from: format!("symbol:{}", s.id),
to: format!("symbol:{}", target.id),
kind: EdgeKind::Uses,
note: None,
});
}
}
for t in &s.tests {
edges.push(Edge {
from: format!("symbol:{}", s.id),
to: format!("symbol:{t}"),
kind: EdgeKind::TestedBy,
note: None,
});
}
for r in &s.related {
edges.push(Edge {
from: format!("symbol:{}", s.id),
to: r.clone(),
kind: EdgeKind::RelatedTo,
note: None,
});
}
for r in &s.reuse {
edges.push(Edge {
from: format!("symbol:{}", s.id),
to: r.clone(),
kind: EdgeKind::DerivedFrom,
note: Some("reuse".into()),
});
}
}
// Capability implements
for c in capabilities {
for sid in &c.implemented_by {
edges.push(Edge {
from: format!("capability:{}", c.id),
to: format!("symbol:{sid}"),
kind: EdgeKind::Implements,
note: None,
});
}
for r in &c.related {
edges.push(Edge {
from: format!("capability:{}", c.id),
to: r.clone(),
kind: EdgeKind::RelatedTo,
note: None,
});
}
}
// Knowledge edges
for k in knowledge {
let kid = format!("knowledge:{}", k.id);
for a in &k.applies_to {
let kind = match k.kind {
KnowledgeKind::Constraint => EdgeKind::ConstrainedBy,
KnowledgeKind::Decision => EdgeKind::DecidedBy,
KnowledgeKind::Failure | KnowledgeKind::AntiPattern => EdgeKind::FailedBy,
_ => EdgeKind::DocumentedBy,
};
// Edge direction: subject --constrained_by--> knowledge (from subject to knowledge)
// Or capability constrained_by knowledge: from applies_to to knowledge
edges.push(Edge {
from: normalize_ref(a, capabilities, symbols),
to: kid.clone(),
kind,
note: Some(k.title.clone()),
});
}
for s in &k.supersedes {
edges.push(Edge {
from: kid.clone(),
to: format!("knowledge:{s}"),
kind: EdgeKind::Supersedes,
note: None,
});
}
for s in &k.conflicts_with {
edges.push(Edge {
from: kid.clone(),
to: format!("knowledge:{s}"),
kind: EdgeKind::ConflictsWith,
note: None,
});
}
for r in &k.related {
edges.push(Edge {
from: kid.clone(),
to: normalize_ref(r, capabilities, symbols),
kind: EdgeKind::RelatedTo,
note: None,
});
}
if k.status == crate::knowledge::ir::Lifecycle::Deprecated {
// no target
}
}
edges.sort_by(|a, b| {
(&a.from, a.kind.as_str(), &a.to).cmp(&(&b.from, b.kind.as_str(), &b.to))
});
edges.dedup_by(|a, b| a.from == b.from && a.to == b.to && a.kind == b.kind);
edges
}
fn normalize_ref(r: &str, capabilities: &[CapabilityInfo], symbols: &[SymbolInfo]) -> String {
if r.starts_with("capability:")
|| r.starts_with("symbol:")
|| r.starts_with("module:")
|| r.starts_with("knowledge:")
{
return r.to_string();
}
if let Some(c) = capabilities.iter().find(|c| c.id == r || c.name == r) {
return format!("capability:{}", c.id);
}
if let Some(s) = symbols.iter().find(|s| s.id == r || s.name == r) {
return format!("symbol:{}", s.id);
}
r.to_string()
}
+179
View File
@@ -0,0 +1,179 @@
//! Knowledge impact analysis for changed paths.
use serde_json::{json, Value};
use crate::knowledge::ir::{EdgeKind, KnowledgeGraph};
#[derive(Debug, Clone)]
pub struct ImpactReport {
pub changed: Vec<String>,
pub capabilities: Vec<String>,
pub decisions: Vec<String>,
pub constraints: Vec<String>,
pub tests: Vec<String>,
pub documentation: Vec<String>,
pub stale_knowledge: Vec<String>,
pub symbols: Vec<String>,
}
pub fn impact(graph: &KnowledgeGraph, paths: &[String]) -> ImpactReport {
let mut report = ImpactReport {
changed: paths.to_vec(),
capabilities: Vec::new(),
decisions: Vec::new(),
constraints: Vec::new(),
tests: Vec::new(),
documentation: Vec::new(),
stale_knowledge: Vec::new(),
symbols: Vec::new(),
};
for path in paths {
let norm = path.replace('\\', "/");
for s in &graph.symbols {
if s.resource == norm
|| s.resource.ends_with(&norm)
|| norm.ends_with(&s.resource)
{
report.symbols.push(s.id.clone());
if let Some(ref cap) = s.capability {
report.capabilities.push(cap.clone());
}
for c in &graph.capabilities {
if c.implemented_by.iter().any(|id| id == &s.id) {
report.capabilities.push(c.name.clone());
}
}
report.tests.extend(s.tests.clone());
}
}
for m in &graph.modules {
if m.resource == norm || m.resource.ends_with(&norm) || norm.ends_with(&m.resource)
{
for s in graph.symbols.iter().filter(|s| s.module == m.id) {
report.symbols.push(s.id.clone());
}
}
}
for k in &graph.knowledge {
if k.resource == norm || k.resource.ends_with(&norm) {
match k.kind {
crate::knowledge::ir::KnowledgeKind::Decision => {
report.decisions.push(k.id.clone())
}
crate::knowledge::ir::KnowledgeKind::Constraint => {
report.constraints.push(k.id.clone())
}
_ => report.documentation.push(k.resource.clone()),
}
}
// applies_to matches changed symbols/modules
for a in &k.applies_to {
if report.symbols.iter().any(|s| s.contains(a) || a.contains(s))
|| report.capabilities.iter().any(|c| c == a || a.contains(c))
{
report.stale_knowledge.push(k.id.clone());
match k.kind {
crate::knowledge::ir::KnowledgeKind::Decision => {
report.decisions.push(k.id.clone())
}
crate::knowledge::ir::KnowledgeKind::Constraint => {
report.constraints.push(k.id.clone())
}
_ => {}
}
}
}
}
}
// Graph walk: from changed symbols follow edges
let seed: Vec<String> = report
.symbols
.iter()
.map(|s| format!("symbol:{s}"))
.collect();
for sid in &seed {
for e in graph.neighbors(sid, None) {
let other = if &e.from == sid { &e.to } else { &e.from };
if other.starts_with("capability:") {
report
.capabilities
.push(other.trim_start_matches("capability:").to_string());
} else if other.starts_with("knowledge:") {
report
.stale_knowledge
.push(other.trim_start_matches("knowledge:").to_string());
} else if e.kind == EdgeKind::TestedBy {
report.tests.push(other.clone());
}
}
}
report.capabilities.sort();
report.capabilities.dedup();
report.decisions.sort();
report.decisions.dedup();
report.constraints.sort();
report.constraints.dedup();
report.tests.sort();
report.tests.dedup();
report.documentation.sort();
report.documentation.dedup();
report.stale_knowledge.sort();
report.stale_knowledge.dedup();
report.symbols.sort();
report.symbols.dedup();
report
}
pub fn format_impact_text(r: &ImpactReport) -> String {
let mut out = String::new();
out.push_str("Changed:\n");
for c in &r.changed {
out.push_str(&format!(" {c}\n"));
}
out.push_str("\nAffected capabilities:\n");
for c in &r.capabilities {
out.push_str(&format!(" {c}\n"));
}
if r.capabilities.is_empty() {
out.push_str(" (none)\n");
}
out.push_str("\nAffected decisions:\n");
for d in &r.decisions {
out.push_str(&format!(" {d}\n"));
}
if r.decisions.is_empty() {
out.push_str(" (none)\n");
}
out.push_str("\nAffected constraints:\n");
for c in &r.constraints {
out.push_str(&format!(" {c}\n"));
}
out.push_str("\nAffected tests:\n");
for t in &r.tests {
out.push_str(&format!(" {t}\n"));
}
out.push_str("\nPotential stale knowledge:\n");
for s in &r.stale_knowledge {
out.push_str(&format!(" {s}\n"));
}
if r.stale_knowledge.is_empty() {
out.push_str(" (none)\n");
}
out
}
pub fn impact_json(r: &ImpactReport) -> Value {
json!({
"changed": r.changed,
"capabilities": r.capabilities,
"decisions": r.decisions,
"constraints": r.constraints,
"tests": r.tests,
"documentation": r.documentation,
"stale_knowledge": r.stale_knowledge,
"symbols": r.symbols,
})
}
+471
View File
@@ -0,0 +1,471 @@
//! Engineering Knowledge IR (L0–L4) — projections of `.mq.md`, not a second source of truth.
use serde_json::{json, Value};
use std::collections::BTreeMap;
/// Lifecycle: draft → verified → stable → deprecated → superseded.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Lifecycle {
Draft,
Verified,
Stable,
Deprecated,
Superseded,
}
impl Lifecycle {
pub fn as_str(&self) -> &'static str {
match self {
Self::Draft => "draft",
Self::Verified => "verified",
Self::Stable => "stable",
Self::Deprecated => "deprecated",
Self::Superseded => "superseded",
}
}
pub fn parse(s: &str) -> Self {
match s.trim().to_ascii_lowercase().as_str() {
"verified" => Self::Verified,
"stable" => Self::Stable,
"deprecated" => Self::Deprecated,
"superseded" => Self::Superseded,
_ => Self::Draft,
}
}
}
#[derive(Debug, Clone, Default)]
pub struct Evidence {
pub sources: Vec<String>,
pub tests: Vec<String>,
pub executions: Vec<String>,
pub commits: Vec<String>,
}
impl Evidence {
pub fn to_json(&self) -> Value {
json!({
"sources": self.sources,
"tests": self.tests,
"executions": self.executions,
"commits": self.commits,
})
}
}
/// Graded confidence with explanation (not a bare float).
#[derive(Debug, Clone)]
pub struct Confidence {
pub level: String, // high | medium | low
pub implementation: String,
pub tests: String,
pub runtime: String,
}
impl Default for Confidence {
fn default() -> Self {
Self {
level: "medium".into(),
implementation: "extracted".into(),
tests: "unknown".into(),
runtime: "unknown".into(),
}
}
}
impl Confidence {
pub fn to_json(&self) -> Value {
json!({
"level": self.level,
"evidence": {
"implementation": self.implementation,
"tests": self.tests,
"runtime": self.runtime,
}
})
}
}
#[derive(Debug, Clone)]
pub struct RepositoryInfo {
pub title: String,
pub root: String,
pub languages: Vec<String>,
pub module_count: usize,
pub symbol_count: usize,
pub capability_count: usize,
pub knowledge_count: usize,
}
#[derive(Debug, Clone)]
pub struct ModuleInfo {
pub id: String,
pub resource: String,
pub title: String,
pub imports: Vec<String>,
pub exports: Vec<String>,
pub responsibility: Option<String>,
pub verified_by: Option<String>,
pub sources: Vec<String>,
pub status: Lifecycle,
}
#[derive(Debug, Clone)]
pub struct SymbolInfo {
pub id: String,
pub name: String,
pub module: String,
pub resource: String,
pub kind: String, // fn | object | type
pub visibility: String,
pub params: Vec<String>,
pub calls: Vec<String>,
pub called_by: Vec<String>,
pub tests: Vec<String>,
pub fingerprint: String,
pub fingerprint_text: String,
pub use_when: Option<String>,
pub do_not_use_when: Option<String>,
pub related: Vec<String>,
pub reuse: Vec<String>,
pub capability: Option<String>,
pub status: Lifecycle,
pub evidence: Evidence,
pub confidence: Confidence,
}
#[derive(Debug, Clone)]
pub struct CapabilityInfo {
pub id: String,
pub name: String,
pub description: String,
pub implemented_by: Vec<String>,
pub modules: Vec<String>,
pub status: Lifecycle,
pub use_when: Option<String>,
pub do_not_use_when: Option<String>,
pub related: Vec<String>,
pub evidence: Evidence,
pub confidence: Confidence,
pub introduced: Option<String>,
pub verified: Option<String>,
pub last_used: Option<String>,
pub last_tested: Option<String>,
pub possible_duplicates: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum KnowledgeKind {
Fact,
Decision,
Constraint,
Pattern,
AntiPattern,
Failure,
Migration,
Function,
Capability,
Other(String),
}
impl KnowledgeKind {
pub fn as_str(&self) -> &str {
match self {
Self::Fact => "Fact",
Self::Decision => "Decision",
Self::Constraint => "Constraint",
Self::Pattern => "Pattern",
Self::AntiPattern => "AntiPattern",
Self::Failure => "Failure",
Self::Migration => "Migration",
Self::Function => "Function",
Self::Capability => "Capability",
Self::Other(s) => s.as_str(),
}
}
pub fn parse(s: &str) -> Self {
let t = s.trim();
let bare = t.strip_prefix("Marqdo ").unwrap_or(t);
match bare {
"Fact" => Self::Fact,
"Decision" | "ADR" => Self::Decision,
"Constraint" => Self::Constraint,
"Pattern" => Self::Pattern,
"AntiPattern" | "Anti-Pattern" => Self::AntiPattern,
"Failure" => Self::Failure,
"Migration" => Self::Migration,
"Function" => Self::Function,
"Capability" => Self::Capability,
other => Self::Other(other.to_string()),
}
}
pub fn subdir(&self) -> &'static str {
match self {
Self::Fact => "facts",
Self::Decision => "decisions",
Self::Constraint => "constraints",
Self::Pattern | Self::AntiPattern => "patterns",
Self::Failure => "failures",
Self::Migration => "migrations",
Self::Function | Self::Capability | Self::Other(_) => "facts",
}
}
}
#[derive(Debug, Clone)]
pub struct KnowledgeItem {
pub id: String,
pub kind: KnowledgeKind,
pub title: String,
pub resource: String,
pub status: Lifecycle,
pub body_summary: String,
pub applies_to: Vec<String>,
pub supersedes: Vec<String>,
pub conflicts_with: Vec<String>,
pub related: Vec<String>,
pub stale_after: Option<String>,
pub evidence: Evidence,
pub confidence: Confidence,
/// Raw source text (copied into knowledge/ when authored under src).
pub source_text: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub enum EdgeKind {
Implements,
Uses,
DependsOn,
TestedBy,
DocumentedBy,
ConstrainedBy,
DecidedBy,
Supersedes,
ConflictsWith,
DeprecatedBy,
DerivedFrom,
FailedBy,
RelatedTo,
}
impl EdgeKind {
pub fn as_str(&self) -> &'static str {
match self {
Self::Implements => "implements",
Self::Uses => "uses",
Self::DependsOn => "depends_on",
Self::TestedBy => "tested_by",
Self::DocumentedBy => "documented_by",
Self::ConstrainedBy => "constrained_by",
Self::DecidedBy => "decided_by",
Self::Supersedes => "supersedes",
Self::ConflictsWith => "conflicts_with",
Self::DeprecatedBy => "deprecated_by",
Self::DerivedFrom => "derived_from",
Self::FailedBy => "failed_by",
Self::RelatedTo => "related_to",
}
}
}
#[derive(Debug, Clone)]
pub struct Edge {
pub from: String,
pub to: String,
pub kind: EdgeKind,
pub note: Option<String>,
}
impl Edge {
pub fn to_json(&self) -> Value {
let mut m = json!({
"from": self.from,
"to": self.to,
"kind": self.kind.as_str(),
});
if let Some(ref n) = self.note {
m["note"] = json!(n);
}
m
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ReuseDecision {
Reuse,
Adapt,
Create,
}
impl ReuseDecision {
pub fn as_str(&self) -> &'static str {
match self {
Self::Reuse => "REUSE",
Self::Adapt => "ADAPT",
Self::Create => "CREATE",
}
}
}
#[derive(Debug, Clone)]
pub struct ReuseCandidate {
pub id: String,
pub kind: String,
pub title: String,
pub score: f64,
pub resource: Option<String>,
pub reason: Vec<String>,
}
#[derive(Debug, Clone)]
pub struct ReuseResult {
pub decision: ReuseDecision,
pub confidence: f64,
pub capability: Option<String>,
pub symbol: Option<String>,
pub source: Option<String>,
pub reason: Vec<String>,
pub candidates: Vec<ReuseCandidate>,
pub constraints: Vec<String>,
pub decisions: Vec<String>,
pub failures: Vec<String>,
pub create_allowed: bool,
pub recommended: Vec<String>,
}
#[derive(Debug, Clone)]
pub struct KnowledgeGraph {
pub repository: RepositoryInfo,
pub modules: Vec<ModuleInfo>,
pub symbols: Vec<SymbolInfo>,
pub capabilities: Vec<CapabilityInfo>,
pub knowledge: Vec<KnowledgeItem>,
pub edges: Vec<Edge>,
pub generated_by: String,
}
impl KnowledgeGraph {
pub fn symbol_by_id(&self, id: &str) -> Option<&SymbolInfo> {
self.symbols.iter().find(|s| s.id == id || s.name == id)
}
pub fn capability_by_id(&self, id: &str) -> Option<&CapabilityInfo> {
self.capabilities
.iter()
.find(|c| c.id == id || c.name == id)
}
pub fn neighbors(&self, id: &str, kind: Option<EdgeKind>) -> Vec<&Edge> {
self.edges
.iter()
.filter(|e| {
(e.from == id || e.to == id)
&& kind.as_ref().map(|k| &e.kind == k).unwrap_or(true)
})
.collect()
}
pub fn to_graph_json(&self) -> Value {
let mut nodes = Vec::new();
nodes.push(json!({
"id": "repo",
"kind": "repository",
"title": self.repository.title,
}));
for m in &self.modules {
nodes.push(json!({
"id": format!("module:{}", m.id),
"kind": "module",
"title": m.title,
"resource": m.resource,
}));
}
for s in &self.symbols {
nodes.push(json!({
"id": format!("symbol:{}", s.id),
"kind": "symbol",
"title": s.name,
"resource": s.resource,
"fingerprint": s.fingerprint,
}));
}
for c in &self.capabilities {
nodes.push(json!({
"id": format!("capability:{}", c.id),
"kind": "capability",
"title": c.name,
"status": c.status.as_str(),
}));
}
for k in &self.knowledge {
nodes.push(json!({
"id": format!("knowledge:{}", k.id),
"kind": k.kind.as_str(),
"title": k.title,
"status": k.status.as_str(),
}));
}
json!({
"type": "Marqdo Knowledge Graph",
"generated": { "by": self.generated_by },
"repository": {
"title": self.repository.title,
"root": self.repository.root,
"modules": self.repository.module_count,
"symbols": self.repository.symbol_count,
"capabilities": self.repository.capability_count,
"knowledge": self.repository.knowledge_count,
},
"nodes": nodes,
"edge_count": self.edges.len(),
})
}
pub fn to_edges_json(&self) -> Value {
json!({
"type": "Marqdo Knowledge Edges",
"generated": { "by": self.generated_by },
"edges": self.edges.iter().map(|e| e.to_json()).collect::<Vec<_>>(),
})
}
pub fn index_map(&self) -> BTreeMap<String, Value> {
let mut m = BTreeMap::new();
for s in &self.symbols {
m.insert(
format!("symbol:{}", s.id),
json!({
"name": s.name,
"module": s.module,
"resource": s.resource,
"capability": s.capability,
"fingerprint": s.fingerprint_text,
}),
);
}
for c in &self.capabilities {
m.insert(
format!("capability:{}", c.id),
json!({
"name": c.name,
"description": c.description,
"implemented_by": c.implemented_by,
}),
);
}
for k in &self.knowledge {
m.insert(
format!("knowledge:{}", k.id),
json!({
"kind": k.kind.as_str(),
"title": k.title,
"summary": k.body_summary,
}),
);
}
m
}
}
+130
View File
@@ -0,0 +1,130 @@
//! Knowledge compilation from agent episodes / failure writebacks.
use std::fs;
use std::path::Path;
use anyhow::Result;
use crate::knowledge::util::slugify;
#[derive(Debug, Clone)]
pub struct LearnInput {
pub task: String,
pub failed_approach: Option<String>,
pub failure: Option<String>,
pub correct_approach: Option<String>,
pub decision_title: Option<String>,
pub decision_body: Option<String>,
pub constraint_title: Option<String>,
pub constraint_body: Option<String>,
}
pub fn learn(out_dir: &Path, input: &LearnInput) -> Result<Vec<std::path::PathBuf>> {
let mut written = Vec::new();
if input.failed_approach.is_some() || input.failure.is_some() {
let dir = out_dir.join("knowledge/failures");
fs::create_dir_all(&dir)?;
let stem = slugify(&format!("fail-{}", &input.task));
let path = dir.join(format!("{stem}.md"));
let body = format!(
r#"---
type: Failure
task: {task}
status: resolved
generated:
by: marqdo/knowledge-learn
---
# Do not repeat: {task}
## Failed approach
{failed}
## Failure
{failure}
## Correct approach
{correct}
## Introduced by
agent episode learn
"#,
task = input.task,
failed = input.failed_approach.as_deref().unwrap_or("_unspecified_"),
failure = input.failure.as_deref().unwrap_or("_unspecified_"),
correct = input.correct_approach.as_deref().unwrap_or("_unspecified_"),
);
fs::write(&path, body)?;
written.push(path);
}
if let (Some(title), Some(body)) = (&input.decision_title, &input.decision_body) {
let dir = out_dir.join("knowledge/decisions");
fs::create_dir_all(&dir)?;
let stem = slugify(title);
let path = dir.join(format!("{stem}.md"));
let page = format!(
r#"---
type: Decision
title: {title}
status: accepted
generated:
by: marqdo/knowledge-learn
---
# {title}
## Decision
{body}
"#,
title = title,
body = body,
);
fs::write(&path, page)?;
written.push(path);
}
if let (Some(title), Some(body)) = (&input.constraint_title, &input.constraint_body) {
let dir = out_dir.join("knowledge/constraints");
fs::create_dir_all(&dir)?;
let stem = slugify(title);
let path = dir.join(format!("{stem}.md"));
let page = format!(
r#"---
type: Constraint
title: {title}
status: stable
generated:
by: marqdo/knowledge-learn
---
# {title}
{body}
"#,
title = title,
body = body,
);
fs::write(&path, page)?;
written.push(path);
}
// Episode receipt
let ep = out_dir.join("agent/episodes");
fs::create_dir_all(&ep)?;
let ep_path = ep.join(format!("{}.json", slugify(&input.task)));
let receipt = serde_json::json!({
"task": input.task,
"learned": written.iter().map(|p| p.display().to_string()).collect::<Vec<_>>(),
});
fs::write(&ep_path, serde_json::to_string_pretty(&receipt)?)?;
written.push(ep_path);
Ok(written)
}
+254
View File
@@ -0,0 +1,254 @@
//! Lifecycle: stale, conflicts, verify.
use std::fs;
use std::path::Path;
use std::time::SystemTime;
use serde_json::{json, Value};
use crate::knowledge::ir::{EdgeKind, KnowledgeGraph, Lifecycle};
#[derive(Debug, Clone)]
pub struct StaleItem {
pub id: String,
pub reason: String,
}
#[derive(Debug, Clone)]
pub struct ConflictItem {
pub left: String,
pub right: String,
pub note: Option<String>,
}
#[derive(Debug, Clone)]
pub struct VerifyIssue {
pub severity: String, // error | warning
pub message: String,
}
pub fn find_stale(graph: &KnowledgeGraph, root: &Path) -> Vec<StaleItem> {
let mut out = Vec::new();
let now = chrono::Utc::now().date_naive();
for k in &graph.knowledge {
if let Some(ref sa) = k.stale_after {
if let Ok(d) = chrono::NaiveDate::parse_from_str(sa, "%Y-%m-%d") {
if d < now {
out.push(StaleItem {
id: k.id.clone(),
reason: format!("stale_after {sa} passed"),
});
}
}
}
if matches!(k.status, Lifecycle::Deprecated | Lifecycle::Superseded) {
out.push(StaleItem {
id: k.id.clone(),
reason: format!("status={}", k.status.as_str()),
});
}
// Source newer than projection evidence — mtime drift
let src = root.join(&k.resource);
if src.exists() {
if let (Ok(meta), Ok(proj_meta)) = (
fs::metadata(&src),
fs::metadata(root.join(".marqdo/graph/graph.json"))
.or_else(|_| fs::metadata(root.join("graph/graph.json"))),
) {
if let (Ok(st), Ok(pt)) = (meta.modified(), proj_meta.modified()) {
if st > pt {
out.push(StaleItem {
id: k.id.clone(),
reason: "source newer than knowledge graph — recompile".into(),
});
}
}
}
}
}
for c in &graph.capabilities {
if matches!(c.status, Lifecycle::Deprecated | Lifecycle::Superseded) {
out.push(StaleItem {
id: c.id.clone(),
reason: format!("capability status={}", c.status.as_str()),
});
}
}
// Dedup
out.sort_by(|a, b| a.id.cmp(&b.id));
out.dedup_by(|a, b| a.id == b.id && a.reason == b.reason);
out
}
pub fn find_conflicts(graph: &KnowledgeGraph) -> Vec<ConflictItem> {
let mut out = Vec::new();
for e in &graph.edges {
if e.kind == EdgeKind::ConflictsWith {
out.push(ConflictItem {
left: e.from.clone(),
right: e.to.clone(),
note: e.note.clone(),
});
}
}
// Knowledge items that claim supersession without edge already covered
for k in &graph.knowledge {
for c in &k.conflicts_with {
out.push(ConflictItem {
left: format!("knowledge:{}", k.id),
right: format!("knowledge:{c}"),
note: Some("authored conflicts_with".into()),
});
}
}
out.sort_by(|a, b| (&a.left, &a.right).cmp(&(&b.left, &b.right)));
out.dedup_by(|a, b| a.left == b.left && a.right == b.right);
out
}
pub fn verify(graph: &KnowledgeGraph) -> Vec<VerifyIssue> {
let mut issues = Vec::new();
// Symbols referenced by capabilities must exist
for c in &graph.capabilities {
for sid in &c.implemented_by {
if graph.symbol_by_id(sid).is_none() {
issues.push(VerifyIssue {
severity: "error".into(),
message: format!(
"capability `{}` implements missing symbol `{sid}`",
c.id
),
});
}
}
}
// Edges must reference known-ish nodes (soft: warn)
let mut node_ids = std::collections::HashSet::new();
node_ids.insert("repo".into());
for m in &graph.modules {
node_ids.insert(format!("module:{}", m.id));
}
for s in &graph.symbols {
node_ids.insert(format!("symbol:{}", s.id));
}
for c in &graph.capabilities {
node_ids.insert(format!("capability:{}", c.id));
}
for k in &graph.knowledge {
node_ids.insert(format!("knowledge:{}", k.id));
}
for e in &graph.edges {
if !node_ids.contains(&e.from) && !e.from.contains('/') {
// allow raw applies_to strings
if !e.from.starts_with("knowledge:")
&& !e.from.starts_with("symbol:")
&& !e.from.starts_with("capability:")
&& !e.from.starts_with("module:")
{
continue;
}
issues.push(VerifyIssue {
severity: "warning".into(),
message: format!("edge from unknown node `{}`", e.from),
});
}
}
// Duplicate high-similarity without documentation
for c in &graph.capabilities {
if !c.possible_duplicates.is_empty() {
issues.push(VerifyIssue {
severity: "warning".into(),
message: format!(
"capability `{}` has possible_duplicates: {}",
c.id,
c.possible_duplicates.join(", ")
),
});
}
}
// Supersedes targets should exist
for k in &graph.knowledge {
for s in &k.supersedes {
if !graph.knowledge.iter().any(|o| o.id == *s || o.id.ends_with(s)) {
issues.push(VerifyIssue {
severity: "warning".into(),
message: format!("knowledge `{}` supersedes unknown `{s}`", k.id),
});
}
}
}
issues
}
pub fn format_stale_text(items: &[StaleItem]) -> String {
let mut out = String::from("Stale knowledge\n\n");
if items.is_empty() {
out.push_str("_None._\n");
return out;
}
for i in items {
out.push_str(&format!("- {} — {}\n", i.id, i.reason));
}
out
}
pub fn format_conflicts_text(items: &[ConflictItem]) -> String {
let mut out = String::from("Knowledge conflicts\n\n");
if items.is_empty() {
out.push_str("_None._\n");
return out;
}
for i in items {
out.push_str(&format!(
"- {} ↔ {}{}\n",
i.left,
i.right,
i.note
.as_ref()
.map(|n| format!(" ({n})"))
.unwrap_or_default()
));
}
out
}
pub fn format_verify_text(issues: &[VerifyIssue]) -> String {
let mut out = String::from("Knowledge verify\n\n");
if issues.is_empty() {
out.push_str("OK — code and knowledge projections are consistent.\n");
return out;
}
for i in issues {
out.push_str(&format!("[{}] {}\n", i.severity, i.message));
}
out
}
pub fn stale_json(items: &[StaleItem]) -> Value {
json!({ "stale": items.iter().map(|i| json!({"id": i.id, "reason": i.reason})).collect::<Vec<_>>() })
}
pub fn conflicts_json(items: &[ConflictItem]) -> Value {
json!({ "conflicts": items.iter().map(|i| json!({"left": i.left, "right": i.right, "note": i.note})).collect::<Vec<_>>() })
}
pub fn verify_json(issues: &[VerifyIssue]) -> Value {
json!({
"ok": issues.iter().all(|i| i.severity != "error"),
"issues": issues.iter().map(|i| json!({"severity": i.severity, "message": i.message})).collect::<Vec<_>>(),
})
}
#[allow(dead_code)]
fn _system_time_now() -> SystemTime {
SystemTime::now()
}
Loaded 100 of 179 files, more files were not shown because too many files have changed in this diff. Show more