Add local n-gram near match for OKF agent-kb reuse.

Sparse bigram cosine closes near-duplicate goal fragmentation; soft_match parents now judge ranked scored candidates instead of a flat task list.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
cflmyandCursor committed 2026-08-11 10:07:03 +08:00
1 parent f240bdf156
commit ab8ea8438f
9 files changed
+427 -88

No files matched your search

+1 -1
View File
@@ -83,7 +83,7 @@ cargo build -p marqdo_plugin_agent
```
- Demo (ABI v1): `demo_add`, `demo_echo` — [`tests/lib/plugin-demo.mq.md`](../../tests/lib/plugin-demo.mq.md).
- Agent (ABI v2): layout + session bag + context queries + **OKF agent-kb** (`agent_goal_sig`, `agent_goal_slug`, `agent_kb_lookup`, `agent_kb_canonicalize`, `agent_kb_list_tasks`, `agent_kb_add_alias`, `agent_kb_promote`, `agent_kb_record_hit`, `agent_workbook_solidify`, `agent_kb_task_files`) — [`plugins/agent`](../../plugins/agent); used by [`ext/ai/agent.mq.md`](../../ext/ai/agent.mq.md). Lookup: `plugin.native_path name=agent` or [ext-cli.md](ext-cli.md) install / `MARQDO_AGENT_PLUGIN`.
- Agent (ABI v2): layout + session bag + context queries + **OKF agent-kb** (`agent_goal_sig`, `agent_goal_slug`, `agent_kb_lookup`, `agent_kb_near_match`, `agent_kb_canonicalize`, `agent_kb_list_tasks`, `agent_kb_add_alias`, `agent_kb_promote`, `agent_kb_record_hit`, `agent_workbook_solidify`, `agent_kb_task_files`) — [`plugins/agent`](../../plugins/agent); used by [`ext/ai/agent.mq.md`](../../ext/ai/agent.mq.md). Lookup: `plugin.native_path name=agent` or [ext-cli.md](ext-cli.md) install / `MARQDO_AGENT_PLUGIN`.
## Layering
+7 -4
View File
@@ -347,8 +347,9 @@ Prefer spawning `resource` over re-planning.
1. **精确**:slug 试探 + frontmatter `sig` → 扫目录 `sig:` → 旧 hex 文件名。
2. **别名(Accepted)**:精确 miss 后,扫 Task FM `aliases:`(YAML 列表或 `[a, b]`);`normalize_goal(query)` 与某条 alias **精确相等**则命中同一 Skill/resource。lookup 返回 `match: exact|alias`;`plan` 对 alias 写 `cache=soft-hit`。
3. **规范句(Accepted)**:仍 miss 时对 goal 做轻量 `agent_kb_canonicalize`(剥站立前缀、去尾标点),再精确/别名各试一次;命中返回 `match: canonical`,`cache=soft-hit`。
4. **父裁决 soft_match(Accepted,默认关)**:`plan soft_match=True` 且仍 miss 时,把 `agent_kb_list_tasks` 候选交给父模型:`DECISION: REUSE` + `SLUG:` 或 `DECISION: NEW`;REUSE 后可 `agent_kb_add_alias` 把当前 goal 写入别名。
5. **不做**默认 embedding / 编辑距离主路径。向量层见 [okf-near-match.md](../roadmap/okf-near-match.md) E。
4. **近义(Accepted,默认开)**:仍 miss 且 `near_match=True` 时,对 Task `title|description|aliases` 做本地**字符 bigram 词袋 + 余弦**;`best.score >= near_threshold`(默认 `0.78`)则 `match: near` 并带 `score`,`cache=soft-hit`,并可 `agent_kb_add_alias`。这是稀疏词法近邻,**不是** embedding 模型。
5. **父裁决 soft_match(Accepted,默认关)**:仍 miss 且 `plan soft_match=True` 时,把 `agent_kb_near_match` 排序候选(含 score;空则退回 `list_tasks`)交给父模型:`DECISION: REUSE` + `SLUG:` 或 `DECISION: NEW`;REUSE 后可 `agent_kb_add_alias`。
6. **不做**默认 dense embedding / 远程向量库。远期 dense 仍见 [okf-near-match.md](../roadmap/okf-near-match.md) E。
`agent_kb_promote` 可选 `aliases=`(字符串或列表)写入 Task;再晋升时合并既有 aliases。
@@ -359,7 +360,9 @@ Prefer spawning `resource` over re-planning.
| `reuse` / `复用` | `True` | 允许知识包快路径 |
| `optimize` / `优化` | `False` | 跳过知识包,强制再规划 |
| `force` / `强制` | `False` | 忽略知识包 |
| `soft_match` / `软命中` | `False` | miss 后父裁决 REUSE/NEW(须显式打开) |
| `soft_match` / `软命中` | `False` | miss 后父裁决 REUSE/NEW(须显式打开;候选为 near 排序列表) |
| `near_match` / `近义命中` | `True` | lookup 第四趟本地 n-gram;`False` 关闭 |
| `near_threshold` / `近义阈值` | `0.78` | near 自动命中最低余弦分 |
| `promote` / `晋升` | `True` | DONE 后固化并写入/更新概念与 resource |
| `kb_dir` / `知识库目录` | `.marqdo/agent-kb` | bundle 根 |
| `workbook_dir` / `工作簿目录` | `None` | 默认直接写 `kb_dir/resources/<slug>.mq.md`;显式传入则用该目录下的临时 `workbook-<slug>-<ts>.mq.md` |
@@ -435,7 +438,7 @@ agent-kb 运行时实现归属 **agent 插件**(`agent_goal_sig` / `agent_kb_l
| **O2** | 本文锁定的任务知识包:`Marqdo Task` / `Agent Skill`;`plan` reuse/promote | **done**(host kb_* + plan 快路径) |
| **O3** | catalog 增强:`verified`/`sources` 可选;模块页可点击依赖;扫入 `agent-kb` 概念供人浏览 | **done** |
| **O4** | `log.md` 策展、可选 `kb query`、Attested-style「再现跑通」收据(非 BigQuery) | 远期 |
| **O5** | 相近任务软命中:aliases + canonicalize + `soft_match` 父裁决已落地;向量仍见 [okf-near-match.md](../roadmap/okf-near-match.md) E | **A/B/C done** |
| **O5** | 相近任务软命中:aliases + canonicalize + 本地 n-gram `near` + `soft_match` 父裁决已落地;dense embedding 仍见 [okf-near-match.md](../roadmap/okf-near-match.md) E | **A/B/C/D-near done** |
O2 设计落盘后,智能体侧实现文档可薄化为指向本文 §7,避免两套真理。
+39 -68
View File
@@ -1,8 +1,8 @@
# OKF 相近任务命中(未来规划)
# OKF 相近任务命中
| | |
|---|---|
| 状态 | **A 规范句 + B aliases + C 父裁决(soft_match)已落地**;E 向量仍待 |
| 状态 | **A 规范句 + B aliases + C 父裁决(soft_match)+ D 本地 n-gram `near` 已落地**;E dense embedding 仍非目标 |
| 日期 | 2026-08-11 |
| 相关 | [okf.md](../design/okf.md) §7 · [ext-agent-plan.md](../design/ext-agent-plan.md) §4.2 · [ext-agent.md](../design/ext-agent.md) |
| 触发观察 | `man-test`:相近 goal 生成两份独立 resource |
@@ -11,127 +11,98 @@
## 1. 问题
v1 agent-kb **默认只做精确命中**(仍是默认快路径):
v1 agent-kb **精确命中仍是默认快路径**:
1. `normalize_goal`:去首尾空白、合并连续空白(**不做**近义改写)。
2. `sig` = 规范化字符串的稳定短哈希;`slug` = 可读路径名。
3. `agent_kb_lookup`:精确 → aliases → **canonicalize 后再精确/别名**;仍 miss 且 `plan soft_match=True` 时父裁决 REUSE/NEW。
3. `agent_kb_lookup`:精确 → aliases → canonicalize → **本地 n-gram `near`(默认开)**;仍 miss 且 `plan soft_match=True` 时父裁决 REUSE/NEW(候选为排序 near 列表)。
历史痛点(已由 A/B/C 缓解):
历史痛点(已由 A/B/C/D 缓解):
| goal(调用方输入) | 旧行为 |
|--------------------|--------|
| `你是一个智能体,帮我规划明天的行程` | 独立 slug,与下方不命中 |
| `帮我规划明天的行程` | 另一份 resource |
| `帮我规划明天行程` | 又一份(近句碎片) |
调用方直觉:「这不是同一件事吗?」——现可用 canonicalize(剥站立前缀)或显式 `soft_match` 父裁决。
调用方直觉:「这不是同一件事吗?」——现可用 canonicalize、默认 `near_match`,或显式 `soft_match` 父裁决。
---
## 2. 为何 v1 坚持精确指纹
## 2. 为何精确指纹仍优先
| 理由 | 说明 |
|------|------|
| 假阳性代价高 | 错复用旧工作簿比多建一个文件更糟(静默答错题) |
| 可审计 | `sig` 可复算;命中不依赖黑盒相似度阈值 |
| 体量假设 | 单仓 agent-kb 通常不大;调用方可固定规范 goal |
| 假阳性代价高 | 错复用旧工作簿比多建一个文件更糟 |
| 可审计 | `sig` 可复算;`near` 留下 `match`/`score` |
| 哲学 | 多轮 = 子文件接力 + OKF 索引,不是聊天式模糊检索 |
**结论:** 精确命中应长期保留为**默认快路径**;相近命中只能是**显式可选层**,且必须可关闭、可解释。
**结论:** 精确命中长期保留为**默认快路径**;`near` 是可关闭的第四趟(`near_match=False`)。
---
## 3. 目标
在 **不破坏** 精确 `sig` 契约的前提下:
1. **已落地**:A 规范句、B aliases、C `soft_match` 父裁决(默认关)。
2. **仍待**:E 向量(仅有证据再开)。
非目标(仍不做为默认):
- 把 agent-kb 做成通用向量知识库 / RAG 产品。
- 用不可审计的静默相似度覆盖精确 `sig`。
- 单文件对话日志 / 状态表来「记住」同主题(与 [ext-agent-plan.md](../design/ext-agent-plan.md) §4.2 冲突)。
---
## 4. 候选方案(不必上向量库)
相近命中 **不等于** 必须上向量检索。按与 Marqdo 契合度排序:
## 3. 已落地层
### A. 规范化后再精确命中 — **已落地**
| | |
|--|--|
| 做法 | `agent_kb_canonicalize`:剥站立前缀、去尾 `?`/`?`/`。`,再 `normalize_goal`;lookup 第三趟 |
| 落地 | 插件 + `match: canonical`;`cache=soft-hit` |
`agent_kb_canonicalize`:剥站立前缀、去尾标点;`match: canonical`。
### B. 别名 / 标签表 — **已落地**
| | |
|--|--|
| 做法 | Task FM `aliases:`;lookup 第二趟精确匹配 |
| 落地 | `match: alias`;promote / `agent_kb_add_alias` |
Task FM `aliases:`;`match: alias`。
### C. 父智能体软裁决(miss 后)— **已落地(默认关)**
### C. 父智能体软裁决 — **已落地(默认关)**
`plan soft_match=True`:`agent_kb_near_match` 排序候选(含 `score`;空则退回 `list_tasks`)→ 父 `DECISION: REUSE` + `SLUG:` 或 `DECISION: NEW`。
### D. 本地字符 bigram + 余弦 — **已落地(默认开)**
| | |
|--|--|
| 做法 | `plan soft_match=True`:`agent_kb_list_tasks` → 父 `DECISION: REUSE` + `SLUG:` 或 `DECISION: NEW` |
| 落地 | REUSE → 现有 hit 分支;可选 append alias |
| 做法 | 稀疏词法向量化(**不是** embedding 模型):canonicalize 后对 goal 与 Task `title\|description\|aliases` 建字符 bigram 词袋,算余弦 |
| API | `agent_kb_near_match`;`agent_kb_lookup` 第四趟 |
| 参数 | `near_match`/`近义命中` 默认 `True`;`near_threshold`/`近义阈值` 默认 `0.78` |
| 痕迹 | `match: near` + `score`;`cache=soft-hit`;可写 alias |
### D. 字面近似(弱,仅拼写容错)
术语:这是 **sparse lexical near-match**,勿宣传为向量库 / embedding 检索。
| | |
|--|--|
| 做法 | 编辑距离、共用字符 n-gram、去标点后再比 |
| 优点 | 零模型、便宜 |
| 风险 | 中文近义(「行程 / 一日游」)基本无效;易假阳性 |
| 建议 | 最多作 D 的辅助,不单独作为语义层 |
### E. Dense embedding(可选远期)
### E. 向量检索(可选远期)
| | |
|--|--|
| 做法 | 对 task `description` / 规范句做 embedding,ANN Top-K,再经阈值或父模型确认 |
| 优点 | 开放域措辞更稳 |
| 风险 | 依赖外部分数与阈值;运维与可审计性差;偏离「Markdown 即索引」 |
| 建议 | **仅当** A–C 在真实包体量下不够时再评估;默认关闭 |
远程 / 本地 dense 向量与 ANN **仍非本仓默认目标**;仅当 D 在真实包体量下证据不足时再评估。
---
## 5. 演进顺序
## 4. 演进顺序
```text
精确 sig(永不删除)
→ A 规范句 + B aliases(已落地)
→ C soft_match 父裁决(已落地,默认关)
→ E 向量(仅有证据再开)
→ D 本地 n-gram near(已落地,默认开)
→ E dense embedding(仅有证据再开)
```
验收原则(仍适用):
验收原则:
1. 默认行为与精确路径一致(`soft_match` 默认 False;canonicalize 无模型、低风险)。
2. 软命中留下 `cache=soft-hit` / `match` 痕迹。
3. 假阳性可被 `force=True` / 显式 `workbook=` 覆盖。
1. `near_match=False` 时与无 near 行为一致;`force`/`optimize` 仍跳过 reuse。
2. 软/近义命中留下 `cache=soft-hit` / `match` / 可选 `score`。
3. 假阳性可被 `force=True` / 显式 `workbook=` / 调高 `near_threshold` 覆盖。
4. 实现仍在 **agent 插件 + `ext/ai`**,不进核心 `src/host`。
---
## 6. 调用方建议
## 5. 调用方建议
- 固定规范 goal 字符串;站立语可由 canonicalize 剥掉,但仍宜少拼。
- 续跑同一主题时传 `workbook=` 或已知 resource 路径。
- 近义但字面差大时显式 `soft_match=True`(多一次父 LLM)。
- 策展:合并重复 task,次要措辞写入 `aliases`。
- 固定规范 goal;站立语可由 canonicalize 剥掉。
- 近句碎片依赖默认 `near_match`;字面差更大时开 `soft_match=True`。
- 策展:合并重复 task,次要措辞写入 `aliases`(O4)。
---
## 7. 与现有路线图的关系
## 6. 与现有路线图的关系
| 条目 | 关系 |
|------|------|
| [okf.md](../design/okf.md) O2 / O5 | 精确 reuse + A/B/C 软命中;E 向量仍待 |
| [okf.md](../design/okf.md) O2 / O5 | 精确 reuse + A/B/C/D;E dense 仍待 |
| [okf.md](../design/okf.md) O4 | `kb query` / 策展可继续扩 aliases |
| [ext-agent-plan.md](../design/ext-agent-plan.md) | 多轮仍只走子文件接力;软命中只影响「选哪个子文件」 |
+26 -9
View File
@@ -936,19 +936,24 @@ Parse `SLUG: …` / `标识:…` after a soft-match REUSE decision.
+ `goal`
+ `tasks`
Short parent prompt: REUSE+SLUG or NEW only.
Short parent prompt: REUSE+SLUG or NEW only. Candidates may include lexical `score` (prefer higher).
*`esc` = > json.parse text={"a":"\n\n--- goal ---\n","b":"\n\n--- candidate tasks (slug | title) ---\n","c":"\n\n--- how to act ---\nExact reuse miss. Reply with ONE protocol only:\nDECISION: REUSE\nSLUG: <exact-slug-from-list>\nor\nDECISION: NEW\nNo other prose.\n","nl":"\n","sep":" | "} *
*`esc` = > json.parse text={"a":"\n\n--- goal ---\n","b":"\n\n--- candidate tasks (slug | title | score) ---\n","c":"\n\n--- how to act ---\nExact reuse miss; candidates are ranked by local n-gram score when present. Prefer higher scores. Reply with ONE protocol only:\nDECISION: REUSE\nSLUG: <exact-slug-from-list>\nor\nDECISION: NEW\nNo other prose.\n","nl":"\n","sep":" | ","sp":"score="} *
*`a` = > json.get value=`esc` key=a *
*`b` = > json.get value=`esc` key=b *
*`c` = > json.get value=`esc` key=c *
*`nl` = > json.get value=`esc` key=nl *
*`sep` = > json.get value=`esc` key=sep *
*`sp` = > json.get value=`esc` key=sp *
*`lines` = > json.parse text=[] *
- [`t`](`tasks`)
*`slug` = > json.get value=`t` key=slug *
*`title` = > json.get value=`t` key=title *
*`row` = `slug` + `sep` + `title` *
*`score` = > json.get value=`t` key=score *
1. `score`
*`row` = `slug` + `sep` + `title` + `sep` + `sp` + `score` *
2. *
*`row` = `slug` + `sep` + `title` *
*`lines` = > json.append list=`lines` item=`row` *
*`catalog` = > join value=`lines` sep=`nl` *
*`p` = Soft-match: same task intent? *
@@ -1408,6 +1413,8 @@ With `stream=True`, the model call uses SSE; `echo=True` prints delta text to st
+ `optimize`=False
+ `force`=False
+ `soft_match`=False
+ `near_match`=True
+ `near_threshold`=0.78
+ `promote`=True
+ `kb_dir`=.marqdo/agent-kb
+ `improve_every`=3
@@ -1418,7 +1425,7 @@ With `stream=True`, the model call uses SSE; `echo=True` prints delta text to st
Multi-step with OKF agent-kb. Default workbook is `kb_dir/resources/<slug>.mq.md`. While task file count `< explore_n` and skill is not llm_free, force a new explore variant under `kb_dir/explore/<slug>/`. Code-first: llm_free hits skip parent LLM. File children return via `# main`; `plan` exposes that as `result`.
Non-hit path: optional `soft_match=True` parent REUSE/NEW; else **decompose** before first child spawn (`DECISION: RUN` / `CONTINUE`+patch / solidified `DONE`). Then `await` → revise loop. With `stream=True`, emit parent `delta` / `decision` / `tool_start`/`tool_end` / `round` / `done` on `events`. `echo=True` prints `plan:decompose` / `plan:await` / deltas. `trace=True` writes events to writeback slot `trace`. Quiet child subtasks stay quiet.
Reuse lookup: exact → alias → canonicalize → optional local n-gram `near` when `near_match=True` and score ≥ `near_threshold`. Non-hit path: optional `soft_match=True` parent REUSE/NEW over ranked `agent_kb_near_match` candidates; else **decompose** before first child spawn (`DECISION: RUN` / `CONTINUE`+patch / solidified `DONE`). Then `await` → revise loop. With `stream=True`, emit parent `delta` / `decision` / `tool_start`/`tool_end` / `round` / `done` on `events`. `echo=True` prints `plan:decompose` / `plan:await` / deltas. `trace=True` writes events to writeback slot `trace`. Quiet child subtasks stay quiet.
*`tools` = > json.get value=`self` key=tools *
*`cache` = miss*
@@ -1437,7 +1444,7 @@ Non-hit path: optional `soft_match=True` parent REUSE/NEW; else **decompose** be
2. `optimize`
*`_` = 1*
3. `reuse`
*`hit` = > agent_kb_lookup kb_dir=`kb_dir` goal=`goal` tools=`tools` *
*`hit` = > agent_kb_lookup kb_dir=`kb_dir` goal=`goal` tools=`tools` near_match=`near_match` near_threshold=`near_threshold` *
1. `hit`
*`match_kind` = > json.get value=`hit` key=match *
1. `match_kind` == alias
@@ -1446,9 +1453,13 @@ Non-hit path: optional `soft_match=True` parent REUSE/NEW; else **decompose** be
*`cache_label` = soft-hit*
*`hit_slug` = > json.get value=`hit` key=slug *
> agent_kb_add_alias kb_dir=`kb_dir` slug=`hit_slug` alias=`goal`
3. `match_kind` == soft
3. `match_kind` == near
*`cache_label` = soft-hit*
4. *
*`hit_slug` = > json.get value=`hit` key=slug *
> agent_kb_add_alias kb_dir=`kb_dir` slug=`hit_slug` alias=`goal`
4. `match_kind` == soft
*`cache_label` = soft-hit*
5. *
*`cache_label` = hit*
*`lf` = > json.get value=`hit` key=llm_free *
1. `lf`
@@ -1533,9 +1544,15 @@ Non-hit path: optional `soft_match=True` parent REUSE/NEW; else **decompose** be
*`path` = None*
2. *
1. `soft_match`
*`listed` = > agent_kb_list_tasks kb_dir=`kb_dir` *
*`tasks` = > json.get value=`listed` key=tasks *
*`near` = > agent_kb_near_match kb_dir=`kb_dir` goal=`goal` *
*`tasks` = > json.get value=`near` key=candidates *
*`tn` = > len value=`tasks` *
1. `tn` < 1
*`listed` = > agent_kb_list_tasks kb_dir=`kb_dir` *
*`tasks` = > json.get value=`listed` key=tasks *
*`tn` = > len value=`tasks` *
2. *
*`_` = 1*
1. `tn` > 0
*`soft_prompt` = > build_soft_match_prompt goal=`goal` tasks=`tasks` *
*`model` = > json.get value=`self` key=model *
+4 -2
View File
@@ -216,6 +216,8 @@ description: 文档驱动智能体 — 单步 / 多步;工具经 lib/子任务
+ `优化`=False
+ `强制`=False
+ `软命中`=False
+ `近义命中`=True
+ `近义阈值`=0.78
+ `晋升`=True
+ `知识库目录`=.marqdo/agent-kb
+ `改进周期`=3
@@ -224,10 +226,10 @@ description: 文档驱动智能体 — 单步 / 多步;工具经 lib/子任务
+ `打印增量`=False
+ `轨迹`=False
多步工作簿默认写入 OKF `知识库目录/resources/<slug>.mq.md`;仅当传入 `工作簿目录` 时才用临时 runs。转发至英文 `agent.plan`。`流式` / `打印增量` / `轨迹` 对应 `stream` / `echo` / `trace`;`软命中` → `soft_match`。
多步工作簿默认写入 OKF `知识库目录/resources/<slug>.mq.md`;仅当传入 `工作簿目录` 时才用临时 runs。转发至英文 `agent.plan`。`流式` / `打印增量` / `轨迹` 对应 `stream` / `echo` / `trace`;`软命中` → `soft_match`;`近义命中` / `近义阈值` → `near_match` / `near_threshold`。
*`model` = > json.get value=`self` key=model *
*`tools` = > json.get value=`self` key=tools *
*`standing` = > json.get value=`self` key=standing *
*`en` = > agent.agent model=`model` tools=`tools` standing=`standing` *
**> `en`.plan goal=`目标` workbook_dir=`工作簿目录` workbook=`工作簿` max_rounds=`最大轮次` confirm=`确认分解` writeback=`写回` skeleton=`骨架` reuse=`复用` optimize=`优化` force=`强制` soft_match=`软命中` promote=`晋升` kb_dir=`知识库目录` improve_every=`改进周期` explore_n=`探索次数` stream=`流式` echo=`打印增量` trace=`轨迹`**
**> `en`.plan goal=`目标` workbook_dir=`工作簿目录` workbook=`工作簿` max_rounds=`最大轮次` confirm=`确认分解` writeback=`写回` skeleton=`骨架` reuse=`复用` optimize=`优化` force=`强制` soft_match=`软命中` near_match=`近义命中` near_threshold=`近义阈值` promote=`晋升` kb_dir=`知识库目录` improve_every=`改进周期` explore_n=`探索次数` stream=`流式` echo=`打印增量` trace=`轨迹`**
+248 -4
View File
@@ -1,5 +1,6 @@
//! OKF agent-kb helpers (plugin-side). No dependency on marqdo `src/host`.
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
@@ -606,6 +607,158 @@ pub fn kb_canonicalize(args: &Value) -> Result<Value, String> {
Ok(Value::String(canonicalize_goal_str(goal)))
}
fn arg_bool(args: &Value, key: &str, default: bool) -> bool {
match args.get(key) {
None | Some(Value::Null) => default,
Some(Value::Bool(b)) => *b,
Some(Value::Number(n)) => n.as_i64().unwrap_or(0) != 0,
Some(Value::String(s)) => matches!(
s.trim(),
"1" | "true" | "True" | "TRUE" | "yes" | "Yes" | "on" | "ON"
),
_ => default,
}
}
fn arg_f64(args: &Value, key: &str, default: f64) -> f64 {
match args.get(key) {
None | Some(Value::Null) => default,
Some(Value::Number(n)) => n.as_f64().unwrap_or(default),
Some(Value::String(s)) => s.parse().unwrap_or(default),
_ => default,
}
}
/// Character bigram bag (whitespace stripped). Sparse lexical vectorization — not an embedding model.
pub fn char_bigram_tf(text: &str) -> HashMap<String, f64> {
let chars: Vec<char> = text.chars().filter(|c| !c.is_whitespace()).collect();
let mut m = HashMap::new();
if chars.is_empty() {
return m;
}
if chars.len() == 1 {
m.insert(chars[0].to_string(), 1.0);
return m;
}
for w in chars.windows(2) {
let key = format!("{}{}", w[0], w[1]);
*m.entry(key).or_insert(0.0) += 1.0;
}
m
}
pub fn cosine_tf(a: &HashMap<String, f64>, b: &HashMap<String, f64>) -> f64 {
let mut na = 0.0;
let mut nb = 0.0;
for v in a.values() {
na += v * v;
}
for v in b.values() {
nb += v * v;
}
if na == 0.0 || nb == 0.0 {
return 0.0;
}
let mut dot = 0.0;
if a.len() <= b.len() {
for (k, va) in a {
if let Some(vb) = b.get(k) {
dot += va * vb;
}
}
} else {
for (k, vb) in b {
if let Some(va) = a.get(k) {
dot += va * vb;
}
}
}
dot / (na.sqrt() * nb.sqrt())
}
#[cfg(test)]
fn near_score(query: &str, candidate: &str) -> f64 {
let q = canonicalize_goal_str(query);
let c = canonicalize_goal_str(candidate);
if q.is_empty() || c.is_empty() {
return 0.0;
}
cosine_tf(&char_bigram_tf(&q), &char_bigram_tf(&c))
}
fn task_near_material(src: &str, slug: &str) -> String {
let title = extract_fm_field(src, "title").unwrap_or_else(|| slug.to_string());
let desc = extract_fm_field(src, "description").unwrap_or_default();
let aliases = extract_fm_aliases(src);
let mut parts = vec![title, desc];
parts.extend(aliases);
parts.join(" ")
}
#[derive(Clone)]
struct NearCand {
slug: String,
title: String,
score: f64,
}
fn rank_near_tasks(root: &Path, goal: &str, limit: usize) -> Vec<NearCand> {
let tasks_dir = root.join("concepts/tasks");
let skills_dir = root.join("concepts/skills");
let res_dir = root.join("resources");
if !tasks_dir.is_dir() {
return Vec::new();
}
let q = canonicalize_goal_str(goal);
let qv = char_bigram_tf(&q);
if qv.is_empty() {
return Vec::new();
}
let mut out = Vec::new();
let Ok(rd) = fs::read_dir(&tasks_dir) else {
return Vec::new();
};
for ent in rd.flatten() {
let p = ent.path();
if p.extension().and_then(|e| e.to_str()) != Some("md") {
continue;
}
let Ok(src) = fs::read_to_string(&p) else {
continue;
};
let slug = p
.file_stem()
.map(|s| s.to_string_lossy().into_owned())
.unwrap_or_default();
if slug.is_empty() {
continue;
}
if !skills_dir.join(format!("{slug}.md")).is_file()
|| !res_dir.join(format!("{slug}.mq.md")).is_file()
{
continue;
}
let title = extract_fm_field(&src, "title").unwrap_or_else(|| slug.clone());
let material = task_near_material(&src, &slug);
let score = cosine_tf(&qv, &char_bigram_tf(&canonicalize_goal_str(&material)));
if score > 0.0 {
out.push(NearCand {
slug,
title,
score,
});
}
}
out.sort_by(|a, b| {
b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| a.slug.cmp(&b.slug))
});
out.truncate(limit.max(1));
out
}
fn lookup_hit_payload(
kb: &str,
root: &Path,
@@ -613,6 +766,7 @@ fn lookup_hit_payload(
slug: &str,
match_kind: &str,
query_sig: &str,
score: Option<f64>,
) -> Result<Value, String> {
let skill_src =
fs::read_to_string(skill_path).map_err(|e| format!("kb_lookup read skill: {e}"))?;
@@ -645,7 +799,7 @@ fn lookup_hit_payload(
}
}
Ok(json!({
let mut out = json!({
"hit": true,
"match": match_kind,
"sig": skill_sig,
@@ -658,7 +812,13 @@ fn lookup_hit_payload(
"resource": kb_rel(kb, &format!("resources/{slug}.mq.md")),
"task": kb_rel(kb, &format!("concepts/tasks/{slug}.md")),
"skill": kb_rel(kb, &format!("concepts/skills/{slug}.md")),
}))
});
if let Some(s) = score {
if let Some(obj) = out.as_object_mut() {
obj.insert("score".into(), json!((s * 1000.0).round() / 1000.0));
}
}
Ok(out)
}
fn resolve_skill_paths_by_slug(
@@ -699,7 +859,7 @@ pub fn kb_lookup(args: &Value) -> Result<Value, String> {
.and_then(|v| v.as_str())
.map(|g| goal_sig_hex(&material_for(g, tools)))
.unwrap_or_default();
return lookup_hit_payload(kb, &root, &skill_path, &slug, "soft", &query_sig);
return lookup_hit_payload(kb, &root, &skill_path, &slug, "soft", &query_sig, None);
}
let goal = arg_text(args, "goal")?;
@@ -707,6 +867,7 @@ pub fn kb_lookup(args: &Value) -> Result<Value, String> {
let query_sig = goal_sig_hex(&material_for(goal, tools));
let slug_hint = goal_slug_str(goal);
let mut match_kind = "exact";
let mut near_score_hit: Option<f64> = None;
let resolved = resolve_skill_paths(&root, &query_sig, &slug_hint)
.or_else(|| {
match_kind = "alias";
@@ -722,12 +883,72 @@ pub fn kb_lookup(args: &Value) -> Result<Value, String> {
let slug2 = goal_slug_str(&canon);
resolve_skill_paths(&root, &sig2, &slug2)
.or_else(|| resolve_skill_paths_by_alias(&root, &canon))
})
.or_else(|| {
if !arg_bool(args, "near_match", false) {
return None;
}
let threshold = arg_f64(args, "near_threshold", 0.78);
let ranked = rank_near_tasks(&root, goal, 8);
let best = ranked.first()?;
if best.score < threshold {
return None;
}
match_kind = "near";
near_score_hit = Some(best.score);
resolve_skill_paths_by_slug(&root, &best.slug)
});
let Some((_task, skill_path, _res, slug)) = resolved else {
return Ok(Value::Null);
};
lookup_hit_payload(kb, &root, &skill_path, &slug, match_kind, &query_sig)
lookup_hit_payload(
kb,
&root,
&skill_path,
&slug,
match_kind,
&query_sig,
near_score_hit,
)
}
pub fn kb_near_match(args: &Value) -> Result<Value, String> {
let kb = arg_text(args, "kb_dir")?;
let goal = arg_text(args, "goal")?;
let limit = match args.get("limit") {
None | Some(Value::Null) => 8usize,
Some(Value::Number(n)) => n.as_u64().unwrap_or(8) as usize,
Some(Value::String(s)) => s.parse().unwrap_or(8),
_ => 8,
}
.clamp(1, 40);
let root = resolve_path(kb);
let ranked = rank_near_tasks(&root, goal, limit);
let candidates: Vec<Value> = ranked
.iter()
.map(|c| {
json!({
"slug": c.slug,
"title": c.title,
"score": (c.score * 1000.0).round() / 1000.0,
})
})
.collect();
let best = ranked.first().map(|c| {
json!({
"slug": c.slug,
"title": c.title,
"score": (c.score * 1000.0).round() / 1000.0,
})
});
let score = ranked.first().map(|c| (c.score * 1000.0).round() / 1000.0);
Ok(json!({
"candidates": candidates,
"count": ranked.len() as i64,
"best": best,
"score": score,
}))
}
pub fn kb_list_tasks(args: &Value) -> Result<Value, String> {
@@ -1102,4 +1323,27 @@ mod tests {
let c2 = canonicalize_goal_str("你是一个智能体,帮我规划明天的行程?");
assert_eq!(c2, "帮我规划明天的行程");
}
#[test]
fn near_score_trip_near_dupes_high() {
let a = "帮我规划明天行程";
let b = "帮我规划明天的行程";
let s = near_score(a, b);
assert!(s >= 0.78, "expected high near score, got {s}");
let c = "帮我完成明天的行程规划";
let s2 = near_score(b, c);
let unrelated = near_score(b, "明天做什么好");
assert!(
s2 > unrelated,
"paraphrase should beat unrelated: paraphrase={s2} unrelated={unrelated}"
);
}
#[test]
fn near_score_unrelated_low() {
let a = "帮我规划明天的行程";
let b = "明天做什么好";
let s = near_score(a, b);
assert!(s < 0.78, "expected below default threshold, got {s}");
}
}
+4
View File
@@ -879,6 +879,7 @@ kb_ffi!(agent_goal_sig, kb::goal_sig, "agent_goal_sig");
kb_ffi!(agent_goal_slug, kb::goal_slug, "agent_goal_slug");
kb_ffi!(agent_kb_canonicalize, kb::kb_canonicalize, "agent_kb_canonicalize");
kb_ffi!(agent_kb_lookup, kb::kb_lookup, "agent_kb_lookup");
kb_ffi!(agent_kb_near_match, kb::kb_near_match, "agent_kb_near_match");
kb_ffi!(agent_kb_list_tasks, kb::kb_list_tasks, "agent_kb_list_tasks");
kb_ffi!(agent_kb_add_alias, kb::kb_add_alias, "agent_kb_add_alias");
kb_ffi!(agent_kb_promote, kb::kb_promote, "agent_kb_promote");
@@ -971,6 +972,9 @@ pub unsafe extern "C" fn marqdo_plugin_init(host: *const MarqdoHostApi) -> c_int
if register(host, "agent_kb_lookup", "kb_dir,goal", agent_kb_lookup) != 0 {
return 1;
}
if register(host, "agent_kb_near_match", "kb_dir,goal", agent_kb_near_match) != 0 {
return 1;
}
if register(host, "agent_kb_list_tasks", "kb_dir", agent_kb_list_tasks) != 0 {
return 1;
}
+77
View File
@@ -0,0 +1,77 @@
---
title: OKF agent-kb near_match (offline)
description: Promote trip resource; near-dupe goals hit match=near; unrelated misses.
> ext/ai/agent.mq.md
> lib/fs.mq.md
> lib/json.mq.md
> lib/plugin.mq.md
> lib/sys.mq.md
---
# main
*`p` = > plugin.native_path name=agent *
1. `p`
> plugin.load path=`p`
2. *
> print text=no-agent-plugin
> sys.exit code=1
*`kb` = .marqdo/agent-kb-near-test *
*`wb` = .marqdo/agent-runs/near-trip.mq.md *
*`goal` = 帮我规划明天的行程 *
*`body` = "---\ntitle: trip\n---\n\n# main\n\n**trip-ok**\n" *
> fs.write_text path=`wb` text=`body`
*`prom` = > agent_kb_promote kb_dir=`kb` goal=`goal` workbook=`wb` *
*`okp` = > json.get value=`prom` key=promoted *
1. `okp`
> print text=promoted
2. *
> print text=promote-fail
*`slug` = > json.get value=`prom` key=slug *
*`g1` = 帮我规划明天行程 *
*`hit1` = > agent_kb_lookup kb_dir=`kb` goal=`g1` near_match=True near_threshold=0.78 *
1. `hit1`
> print text=near-ok
2. *
> print text=near-miss
*`mk1` = > json.get value=`hit1` key=match *
> print text=`mk1`
*`slug1` = > json.get value=`hit1` key=slug *
1. `slug` == `slug1`
> print text=same-slug
2. *
> print text=slug-drift
*`off` = > agent_kb_lookup kb_dir=`kb` goal=`g1` near_match=False *
1. `off`
> print text=off-hit
2. *
> print text=off-miss
*`g2` = 明天做什么好 *
*`hit2` = > agent_kb_lookup kb_dir=`kb` goal=`g2` near_match=True near_threshold=0.78 *
1. `hit2`
> print text=neg-hit
2. *
> print text=neg-miss
*`ranked` = > agent_kb_near_match kb_dir=`kb` goal=`g1` *
*`best` = > json.get value=`ranked` key=best *
*`bs` = > json.get value=`best` key=slug *
1. `bs` == `slug`
> print text=rank-ok
2. *
> print text=rank-bad
*`tasks` = > json.get value=`ranked` key=candidates *
*`prompt` = > agent.build_soft_match_prompt goal=`g1` tasks=`tasks` *
*`has` = > split value=`prompt` sep=score= *
*`hn` = > len value=`has` *
1. `hn` > 1
> print text=prompt-score-ok
2. *
> print text=prompt-score-bad
+21
View File
@@ -1432,6 +1432,27 @@ list-ok",
);
}
#[test]
fn ext_agent_kb_near() {
let status = Command::new("cargo")
.args(["build", "-p", "marqdo_plugin_agent"])
.current_dir(env!("CARGO_MANIFEST_DIR"))
.status()
.expect("build agent plugin");
assert!(status.success(), "failed to build marqdo_plugin_agent");
assert_out(
"tests/ext/agent-kb-near.mq.md",
"promoted
near-ok
near
same-slug
off-miss
neg-miss
rank-ok
prompt-score-ok",
);
}
#[test]
fn ext_agent_soft_match() {
assert_out(