P4 验证执行:(a)锚点消融✅/(c)契约消融✅/(d)防漂移✅/(e)守卫演练✅;(b)盲测 deepseek-flash 完成未达标、mimo-v2.6-pro 重跑中
- perf-validation.md:五实验协议/判据/材料(2026-09-23 自设计) - perf-validation-report-2026-09-23.md:汇总报告(判据逐项对照 + 诚实结论) - tests/blind_experiment.rs + tests/blind/:盲测 harness(查询通道/修复回路/usage 计量)+ 冻结材料(6 任务书 + docs-bundle) - tests/contract_ablation.rs:四边界×三情形消融(拦截 100%/误报 0%/零回归) - tests/drift_drill.rs:9 漂移必报 + 2 负对照 - src/mlsp.rs:syntax 卡片自持化(semantics/rule/sample/doc_quote) - three-problems.md §5.1 判据矩阵逐行判定(问题 1✅ 2✅ 3❌);three-problems-plan.md Phase 4 落地记录 - 全量干净回归零失败 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
1 parent
00c763300e
commit
1a729eb9ec
18 files changed
+1848
-11
No files matched your search
@@ -0,0 +1,34 @@
|
||||
# 盲测实验(§5.2(b) 查询式 vs 全文档背诵)—— 2026-09-23
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 模型 | `deepseek-flash` @ `https://llm.cflmy.cn/v1` |
|
||||
| 样本 | 6 任务(业务 3 + QuixBugs 改写 3),oracle = stdout 精确匹配 |
|
||||
| 协议 | 系统提示词 = 硬规则 12 条 + 契约表三形状(两臂同文);**唯一变量 = 语法来源**:`query` 臂 MLSP 查询通道(≤3 次)vs `docs` 臂完整 markup 参考(背诵) |
|
||||
| 修复 | validate 诊断 + stderr 回喂,≤2 次修复;temperature=0,max_tokens=8000 |
|
||||
|
||||
## 汇总(判据:Q 通过率不劣于 D;首轮语法错误率 Q≤D;token Q≤70%%D;轮次不升)
|
||||
|
||||
| 臂 | 通过(一轮) | 通过(最终≤3) | 首轮语法错误 | 首轮契约诊断 | 平均修复轮(过例) | token 合计(prompt+completion) | 弃权/基础设施 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| query | 0/3 | 0/3 | 2/3 | 1/3 | 0.00 | 69771(42478+27293) | 3 |
|
||||
| docs | 3/6 | 4/6 | 2/6 | 0/6 | 0.50 | 57276(33346+23930) | 0 |
|
||||
|
||||
## 逐例
|
||||
|
||||
| 例 | 臂 | 结果 | 修复轮 | 首轮语法错 | 首轮契约诊断 | token(p+c) | 查询 | 备注 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| claims_reserve_settle | query | load-fail | 0 | 是 | 否 | 11987+8177 | 3 | {"error":{"code":"mlsp.parse_error","message":"<memory>: trailing input in expression: \"(claim - deductible, 0)\""},"id":1,"ok":false} \| stderr=error: /tmp/marqdo_blind_1694593_query_claims_reserve_settle.mq.md: trailing input in expression: "(claim - deductible, 0)" |
|
||||
| contract_milestone_pay | query | infra | 0 | 否 | 否 | 3970+4718 | 3 | 非 JSON / 空体: |
|
||||
| vendor_po_gate | query | load-fail | 0 | 是 | 否 | 9552+3686 | 3 | 回复中无程序 \| 开头: QUERY: 条件分支 写法 |
|
||||
| qb_gcd | query | run-fail | 0 | 否 | 是 | 9700+8488 | 3 | \| stderr=error: /tmp/marqdo_blind_1694593_query_qb_gcd.mq.md:26:1: unknown function `gcd(48` |
|
||||
| qb_max_sublist_sum | query | infra | 0 | 否 | 否 | 1253+134 | 3 | 非 JSON / 空体: |
|
||||
| qb_get_factors | query | infra | 0 | 否 | 否 | 6016+2090 | 3 | 非 JSON / 空体: |
|
||||
| claims_reserve_settle | docs | pass | 0 | 否 | 否 | 2699+2987 | 0 | queries=0 |
|
||||
| contract_milestone_pay | docs | pass | 0 | 否 | 否 | 2800+3202 | 0 | queries=0 |
|
||||
| vendor_po_gate | docs | pass | 0 | 否 | 否 | 2690+1264 | 0 | queries=0 |
|
||||
| qb_gcd | docs | pass | 2 | 否 | 否 | 8147+6970 | 0 | queries=0 |
|
||||
| qb_max_sublist_sum | docs | run-fail | 0 | 是 | 否 | 8518+3609 | 0 | \| stderr=error: /tmp/marqdo_blind_1694593_docs_qb_max_sublist_sum.mq.md:3:1: unknown function `[-2,` |
|
||||
| qb_get_factors | docs | run-fail | 0 | 是 | 否 | 8492+5898 | 0 | \| stderr=error: /tmp/marqdo_blind_1694593_docs_qb_get_factors.mq.md:20:1: missing argument for parameter `value` |
|
||||
|
||||
报告由 `tests/blind_experiment.rs` 生成;材料冻结于 `tests/blind/`(任务书 + docs-bundle),判据见 `doc/roadmap/perf-validation.md` §2。
|
||||
@@ -0,0 +1,34 @@
|
||||
# 盲测实验(§5.2(b) 查询式 vs 全文档背诵)—— 2026-09-23
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 模型 | `mimo-v2.6-pro` @ `https://llm.cflmy.cn/v1` |
|
||||
| 样本 | 6 任务(业务 3 + QuixBugs 改写 3),oracle = stdout 精确匹配 |
|
||||
| 协议 | 系统提示词 = 硬规则 12 条 + 契约表三形状(两臂同文);**唯一变量 = 语法来源**:`query` 臂 MLSP 查询通道(≤3 次)vs `docs` 臂完整 markup 参考(背诵) |
|
||||
| 修复 | validate 诊断 + stderr 回喂,≤2 次修复;temperature=0,max_tokens=8000 |
|
||||
|
||||
## 汇总(判据:Q 通过率不劣于 D;首轮语法错误率 Q≤D;token Q≤70%%D;轮次不升)
|
||||
|
||||
| 臂 | 通过(一轮) | 通过(最终≤3) | 首轮语法错误 | 首轮契约诊断 | 平均修复轮(过例) | token 合计(prompt+completion) | 弃权/基础设施 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| query | 0/0 | 0/0 | 0/0 | 0/0 | 0.00 | 8011(6566+1445) | 6 |
|
||||
| docs | 0/0 | 0/0 | 0/0 | 0/0 | 0.00 | 0(0+0) | 6 |
|
||||
|
||||
## 逐例
|
||||
|
||||
| 例 | 臂 | 结果 | 修复轮 | 首轮语法错 | 首轮契约诊断 | token(p+c) | 查询 | 备注 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| claims_reserve_settle | query | infra | 0 | 否 | 否 | 1424+366 | 3 | 非 JSON / 空体: |
|
||||
| contract_milestone_pay | query | infra | 0 | 否 | 否 | 1528+381 | 1 | 非 JSON / 空体: |
|
||||
| vendor_po_gate | query | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| qb_gcd | query | infra | 0 | 否 | 否 | 1211+252 | 3 | 非 JSON / 空体: |
|
||||
| qb_max_sublist_sum | query | infra | 0 | 否 | 否 | 1216+226 | 3 | 非 JSON / 空体: |
|
||||
| qb_get_factors | query | infra | 0 | 否 | 否 | 1187+220 | 3 | 非 JSON / 空体: |
|
||||
| claims_reserve_settle | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| contract_milestone_pay | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| vendor_po_gate | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| qb_gcd | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| qb_max_sublist_sum | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
| qb_get_factors | docs | infra | 0 | 否 | 否 | 0+0 | 0 | 非 JSON / 空体: |
|
||||
|
||||
报告由 `tests/blind_experiment.rs` 生成;材料冻结于 `tests/blind/`(任务书 + docs-bundle),判据见 `doc/roadmap/perf-validation.md` §2。
|
||||
@@ -0,0 +1,118 @@
|
||||
# P4 验证汇总报告 — 2026-09-23
|
||||
|
||||
> 执行 [perf-validation.md](perf-validation.md)(自设计性能验证协议)的结果汇总。
|
||||
> 覆盖 three-problems.md 三大问题的**验证矩阵**(§5.2 a–e)。逐例明细见各实验原始报告(文末索引)。
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 协议 | [perf-validation.md](perf-validation.md)(实验 (a)–(e)) |
|
||||
| 判据源 | [three-problems.md](three-problems.md) §3.7 / 各问题 §2.4 |
|
||||
| 构建 | marqdo v1.0.7(本仓库工作树;金 20 + 全量回归通过后采样) |
|
||||
| 端点 | `llm.cflmy.cn/v1`(OpenAI 兼容;temperature=0) |
|
||||
| 模型 | 修复实验:`cflmy`、`mimo-v2.6-flash`;盲测:`deepseek-flash`、`mimo-v2.6-pro`(按用户指定) |
|
||||
| 回归 | 全量干净回归(无 API key 基线)**零失败** |
|
||||
|
||||
## 1. 五实验结果与判据对照
|
||||
|
||||
| 实验 | 判据 | 结果 | 判定 |
|
||||
|---|---|---|---|
|
||||
| **(a) 锚点消融**(T3.3,双模型×20例×双臂,三轮) | anchor 臂一轮修复率 ≥ 80%;护栏 0 次越界应用 | cflmy anchor **20/20=100%**(plain 17/20=85%);mimo-v2.6-flash anchor **20/20=100%**(plain 18/20=90%;容错加固中的第二轮 anchor 18/20=90% 亦 ≥80%);越界拒/越界应用 0/0 | ✅ 过 |
|
||||
| **(b) 盲测**(查询式 vs 全文档背诵,6任务×2臂) | 四判据:最终通过率 Q 不劣于 D(允许 −0 例);首轮语法错误率 Q ≤ D;平均修复轮次 Q ≤ D;token ≤ 70% of D(首轮契约诊断率为观测项) | deepseek-flash:docs 4/6 vs query 0/3(3 例端点 infra 剔除);首轮语法错 2/3 vs 2/6;token 69,771 vs 57,276。mimo-v2.6-pro:首跑 12/12 端点 infra,单独重跑见 §3 | ❌ 四判据未全立 ⇒ 按 §3.7 继续改协议 |
|
||||
| **(c) 契约消融**(四边界×三情形) | 带契约错值拦截 = 100%;带契约对值误报 = 0%;无契约路径零回归 | 拦截 4/4=**100%**;误报 0/4=**0%**;无契约 4/4 动态语义保持(含 `missing map key` 基线) | ✅ 过 |
|
||||
| **(d) 防漂移演练**(9 漂移 + 2 负对照) | 故意写错必报 = 100%;对照组误报 = 0% | 必报 9/9=**100%**(`unknown_type`×2、`param_drift`×2、`return_mismatch`、`arg_mismatch`、`unknown_key`、`contract.duplicate`、`contract.unknown_table`);负对照误报 0/2 | ✅ 过 |
|
||||
| **(e) 核心守卫演练**(3 种变异注入) | 单边改动必红;还原必绿 | 3/3 必红(`CORE_CONSTRUCTS` 私增、`core-surface.md` 私增、金样删除);还原全绿 | ✅ 过 |
|
||||
|
||||
**可复制结论(a/c/d/e 全过)**:锚点臂一轮修复率双模型 100%(判据 ≥80%,增量 +15pp/+10pp——易锚定样本近天花板,区分性留难样本扩展);契约系统"只在声明处拦截、零误报、不碰动态路径"成立;契约/实现互查防漂移 100% 必报;核心面单边膨胀被 CI 守卫机械阻断。
|
||||
|
||||
## 2. 实验 (c)(d)(e) 明细
|
||||
|
||||
### (c) 契约消融(tests/contract_ablation.rs,确定性)
|
||||
|
||||
四边界(调用实参 / 返回值 / 表绑定 / 键访问)各测三情形:
|
||||
|
||||
| 情形 | 口径 | 结果 |
|
||||
|---|---|---|
|
||||
| ① 带契约 + 错值 | 必须拦截(`contract.arg_mismatch` / `contract.return_mismatch` / `contract.field_mismatch` / `contract.unknown_key`) | 4/4 |
|
||||
| ② 带契约 + 对值 | 不得报契约错(同路径同形状) | 0/4 误报 |
|
||||
| ③ 无契约 + 错值 | 保持动态语义(不得出现 `contract.*`;走经典动态错误如 `missing map key`) | 4/4 |
|
||||
|
||||
取元一律用主推的 `[]()` 形(`[键](`集合`)`)验证;`集合[^键]` 过渡形保留兼容金样覆盖。
|
||||
|
||||
### (d) 防漂移演练(tests/drift_drill.rs,确定性)
|
||||
|
||||
9 个"故意写错"全部必报:类型词拼错、参数名漂移、升参未覆盖、返回越约、实参越约、未知键、重复契约表、错列表头契约表、运行时值越约。负对照 2 例(被绑定的数据表、纯动态程序)零误报。同时验证"契约表一经出现即为完整声明"(新升参未覆盖 ⇒ `contract.param_drift` warning)被按设计触发。
|
||||
|
||||
### (e) 核心守卫演练(人工变异,tests/core_surface.rs 必红验证)
|
||||
|
||||
| 变异 | 注入点 | 结果 |
|
||||
|---|---|---|
|
||||
| 私增 `ghost-construct` | `src/parse/mod.rs` `CORE_CONSTRUCTS` | 🔴 必红 |
|
||||
| 私增 `ghost-construct` 行 | `doc/design/core-surface.md` | 🔴 必红 |
|
||||
| 删除金样文件 | `tests/markup-v03/soft-emphasis.mq.md` | 🔴 必红 |
|
||||
|
||||
三次还原后 🟢 全绿。CI 守卫强制"清单与文档双改 + 全部金样存在"。
|
||||
|
||||
## 3. 实验 (b) 盲测:详况与诚实结论
|
||||
|
||||
### 协议
|
||||
|
||||
两臂系统提示词同文 = 硬规则 12 条 + 契约表三形状;**唯一变量 = 语法知识获取**:
|
||||
|
||||
- `query` 臂:MLSP 查询通道(`QUERY: <关键字>` ⇒ `mlsp syntax` 卡片,含戒律原文/样例/文档锚点;≤3 次)
|
||||
- `docs` 臂:完整 markup 参考背诵(`tests/blind/docs-bundle.md`,背诵式全语法表 + 中文内建面)
|
||||
|
||||
任务 6 例(`tests/blind/tasks/`):业务 3(保险理赔准备金 / 工程进度款 / 采购付款门槛,均出自律所实测任务集)+ QuixBugs 改写 3(gcd / 最大子段和 / 质因数分解);oracle = stdout 精确匹配(任务书自带期望输出,把推理难度控制掉,失败归因于**语言能力**)。修复回路 ≤2 次(validate 结构化诊断 + stderr 回喂)。
|
||||
|
||||
### deepseek-flash(完成)
|
||||
|
||||
| 臂 | 通过(一轮) | 通过(最终) | 首轮语法错 | 平均修复轮(过例) | token(p+c) | infra 剔除 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| query | 0/3 | 0/3 | 2/3 | – | 69,771(42,478+27,293) | 3/6 |
|
||||
| docs | 3/6 | **4/6** | 2/6 | 0.50 | 57,276(33,346+23,930) | 0/6 |
|
||||
|
||||
- **docs 臂**:业务题 3/3 全部一轮通过(各 ~3k token);`qb_gcd` 修复 2 轮过;`qb_max_sublist_sum`、`qb_get_factors` 暴露真语法滑倒(列表字面量被当调用 `unknown function [-2,`;参数名漂移 `missing argument for parameter value`)。
|
||||
- **query 臂**:查询通道**被积极使用**(5/6 例用满 3 次,多词查询如 `条件分支 写法` 也能正确命中卡片——已单独验证查询匹配无缺陷);但 3 例被端点空体吃掉,其余 3 例败于真语法错(Python 式元组 `(a, b)` ⇒ `trailing input in expression`;Python 式调用 `gcd(48, 18)` ⇒ `unknown function gcd(48`)与 1 例查询后未交程序。
|
||||
|
||||
### mimo-v2.6-pro(重跑中)
|
||||
|
||||
首跑 12/12 全部"非 JSON / 空体"(与 deepseek 并行时段;同参数小冒烟正常)。三向探针(小提示词×12000/4000、大提示词×12000)证实**网关随机空体**、与 `max_tokens`/提示词体量无稳定关系。已将重试 8→12 次、退避 10→15 秒,并**单独**重跑(不与其它模型抢并发)。结果落地后补入本节。
|
||||
|
||||
### 判据对照(截至 deepseek-flash)
|
||||
|
||||
| 判据(协议 §2.4 原文口径) | 目标 | 实测(deepseek-flash,非 infra 口径) | 判定 |
|
||||
|---|---|---|---|
|
||||
| 最终通过率(≤3 轮)Q 不劣于 D(允许 −0 例) | ≥ D | 0/3 vs 4/6 | ❌ |
|
||||
| 首轮语法错误率 Q ≤ D | ≤ | 2/3 vs 2/6 | ❌ |
|
||||
| 平均修复轮次 Q ≤ D | ≤ | query 臂无通过例,不可比 | ⚠️ |
|
||||
| token 用量 Q ≤ 70% of D | ≤ 40,093 | 69,771(D 的 121.8%) | ❌ |
|
||||
| (观测项)首轮契约诊断率 Q ≤ D | – | 1/3 vs 0/6 | ➖ 观测 |
|
||||
|
||||
**诚实结论:在 deepseek-flash 上,"查询式"没有跑赢"全背诵"**——三项可判量化判据未达,按协议 §3.7 判定:**四判据未全成立 ⇒「AI 不必学语法」本轮未获实证,继续改协议**(不是证伪——infra 吞掉 query 臂一半样本,非 infra n=3)。要点:
|
||||
|
||||
1. **统计力被端点削弱**:query 臂 6 例中 3 例被空体吃掉(协议无从执行),非 infra 样本 n=3;mimo-v2.6-pro 整组不可用。判据是"有方向的强声明",当前证据**不足以支持也未证伪**其普适性,只能定格为"deepseek-flash × 本任务集 × 本协议版本"下的一次未达标观测。
|
||||
2. **失败模式是真发现**:查询臂的败例是**跨语言习惯泄漏**(元组表达式、Python 调用形)——查询卡片能回答"条件分支怎么写",但模型不查询"这里能不能用元组"。说明"AI 不必学语法"的瓶颈不在**查询机制**,而在**模型的自查纪律**(不知道自己不知道)。
|
||||
3. **token 判据方向存疑**:查询式省的是**系统提示词常驻**(背诵包),花的是**多轮往返**。6 例短程序任务里往返开销 > 常驻节省;该判据更适用于**长会话多文件**场景(一次背诵包 × N 轮 vs 每轮均摊查询),本轮任务形态低估了查询式的收益面。
|
||||
4. **docs 臂的 4/6 说明任务集有区分度**(不是全过/全不过的退化测量)。
|
||||
|
||||
## 4. 测量端基础设施记录(方法学附注)
|
||||
|
||||
- **LLM 端点不稳定**是本轮最大的干扰源:随机空体、reasoning 模型 `finish=length` 截断(思考吃满额度、正文为空)。对策:空体重试 5→12 次 + 退避 5→15 秒、截断自适应(`max_tokens` 加倍重试,封顶 32k)、infra 例从判据比率中剔除并单列。
|
||||
- **harness 缺陷两处**(已修,未污染最终数据):查询额度用满后的续篇被当最终回复(⇒ 假"无程序");程序提取失败不进修复回路(⇒ 少给模型一次标准修复机会)。修复后的协议对两臂完全一致。
|
||||
- **模型口径变化留痕**:原定 `cflmy`/`mimo-v2.6-flash` 组合按用户指示弃用,改 `mimo-v2.6-pro`/`deepseek-flash`;修复实验 (a) 的双模型数据仍是原组合(金 20 协议已完成,不重跑)。
|
||||
|
||||
## 5. 材料与复现
|
||||
|
||||
```bash
|
||||
# (a) 锚点消融(需要 API key)
|
||||
cargo test --test repair_experiment -- --ignored --nocapture # anchor/plain 双臂×20例
|
||||
# (b) 盲测(每模型一次;建议单独跑避免并发)
|
||||
OPENAI_MODEL=deepseek-flash MARQDO_EXP_DATE=2026-09-23 cargo test --test blind_experiment -- --ignored --nocapture
|
||||
# (c)(d) 确定性
|
||||
cargo test --test contract_ablation --test drift_drill
|
||||
# (e) 核心守卫(变异注入→必红→还原必绿,按 perf-validation.md §5 手法)
|
||||
cargo test --test core_surface
|
||||
```
|
||||
|
||||
- 任务书 / 背诵包(冻结,勿改):`tests/blind/`
|
||||
- 原始报告:[blind-experiment-2026-09-23-deepseek-flash.md](blind-experiment-2026-09-23-deepseek-flash.md) · [blind-experiment-2026-09-23-mimo-v2.6-pro.md](blind-experiment-2026-09-23-mimo-v2.6-pro.md)(首跑全 infra)· [repair-experiment-2026-09-22.md](repair-experiment-2026-09-22.md) · [repair-experiment-2026-09-22-mimo-v2.6-flash.md](repair-experiment-2026-09-22-mimo-v2.6-flash.md)
|
||||
- 判据协议:[perf-validation.md](perf-validation.md)(版本化于本仓库,实验前冻结)
|
||||
@@ -0,0 +1,125 @@
|
||||
# P4 性能验证设计(实验协议与判据)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 状态 | **Accepted · P4 执行依据** |
|
||||
| 日期 | 2026-09-23 |
|
||||
| 授权 | 用户授权**自行设计**本性能验证设计(P4 不再等待外部文档导入;`doc/next/002.md` 与性能验证无关) |
|
||||
| 上位判据 | [three-problems.md](three-problems.md) §5.1 判据矩阵 + §5.2 实验 (a)–(e);「判据矩阵全绿 + 实验数字可复现之前,三大问题一律视为未解决」 |
|
||||
| 工程任务 | [three-problems-plan.md](three-problems-plan.md) Phase 1–3(已完成);本文只负责**验证它们** |
|
||||
|
||||
> **本文定位**:把 three-problems.md §5.2 的五个实验操作化——每个实验给出**材料、协议、指标、判据**,全部机器可执行、数字可复现。实验产物 = 报告 [perf-validation-report-2026-09-23.md](perf-validation-report-2026-09-23.md)。**只认行为与数字,不认宣言。**
|
||||
|
||||
## 0. 总览
|
||||
|
||||
| 实验 | 验证对象 | 判据(全过 = 该问题的实证成立) | 执行方式 | 状态 |
|
||||
|------|----------|-------------------------------|----------|------|
|
||||
| (a) 锚点修复消融 | 问题三 · 可靠反馈 | 一轮修复率 ≥ 80%(anchor 臂),护栏 0 越界 | 模型在环 `tests/repair_experiment.rs` | ✅ 已完成(三轮双模型) |
|
||||
| (b) 盲测 | 问题三 · 不必学语法 | 查询组错误率 ≤ 全文档组;token 显著更少;修复轮次不升 | 模型在环 `tests/blind_experiment.rs` | 见报告 |
|
||||
| (c) 契约消融 | 问题二 · 语义校验 | 拦截率 = 100%;误报率 = 0%;无契约路径零回归 | 确定性 `tests/contract_ablation.rs` | 见报告 |
|
||||
| (d) 防漂移演练 | 问题二 · 契约可信 | 故意写错契约必报率 = 100% | 确定性 `tests/drift_drill.rs` | 见报告 |
|
||||
| (e) 核心守卫演练 | 问题一 · 核心要小 | 三种变异注入各必红 → 还原必绿 | 变异脚本(见 §4) | 见报告 |
|
||||
|
||||
判定口径([three-problems.md](three-problems.md) §5.1):**任何一项判据不成立,对应问题即「未解决」**——不许用文档宣言充数。
|
||||
|
||||
---
|
||||
|
||||
## 1. 实验 (a):锚点修复消融(问题三 · 可靠反馈)
|
||||
|
||||
**假设**:诊断带 `doc_anchor`/`doc_quote`/`suggestion`(锚点诊断)比裸诊断显著提高一轮修复率。
|
||||
|
||||
- **对照**:`anchor` 臂(全量诊断字段)vs `plain` 臂(剥掉 doc_anchor/doc_quote/suggestion)。
|
||||
- **材料**:golden 20 例(10 骨架 × 2 变体:arg/ret/field/key/typo/drift/dup),易锚定错误。
|
||||
- **协议**:`validate` → `repair_targets` → 模型行编辑 → `repair_apply`(越界必拒/abstain)→ 复验;≤2 轮。
|
||||
- **指标**:一轮修复率、最终修复率、平均尝试、越界拒、弃权。
|
||||
- **判据**:anchor 臂一轮修复率 ≥ 80%;护栏 0 次越界应用(越界必拒由 `tests/repair_loop.rs` 硬断言兜底)。
|
||||
- **执行**:`OPENAI_MODEL=… cargo test --test repair_experiment -- --ignored --nocapture`。
|
||||
- **结果**:三轮双模型全过(cflmy anchor 100% / mimo 90%→100%,plain 85%–100%,护栏 0 越界)——
|
||||
详见 [repair-experiment-2026-09-22.md](repair-experiment-2026-09-22.md) 与
|
||||
[repair-experiment-2026-09-22-mimo-v2.6-flash.md](repair-experiment-2026-09-22-mimo-v2.6-flash.md)。**✅ PASS**。
|
||||
- **诚实注记**:样本为易锚定错误(两臂近天花板),锚点增量的区分性证据留待难样本扩展;不影响本判据达标。
|
||||
|
||||
## 2. 实验 (b):盲测——查询式 vs 全文档背诵(问题三 · 超级核心)
|
||||
|
||||
**假设**:「零语法文档 + MLSP 查询」组(Q)成绩**不劣于**「全文档背诵」组(D),且 prompt token 显著更少、修复轮次不升([three-problems.md](three-problems.md) §3.7 最终判据)。
|
||||
|
||||
### 2.1 对照(唯一变量 = 语法知识的获取方式)
|
||||
|
||||
| 臂 | 系统提示词内容 | 语法知识来源 |
|
||||
|----|----------------|--------------|
|
||||
| **Q 查询组** | 硬规则(12 条)+ 契约表三形状 + MLSP 查询通道说明 | 按需 `QUERY: <关键字>`(≤3 次,harness 代答 `mlsp syntax` 卡片) |
|
||||
| **D 全文档组** | 硬规则(12 条)+ 契约表三形状 + **完整 markup 参考**(`tests/blind/docs-bundle.md`:v0.3 全语法表 + 中文内建面) | 上下文预载(背诵) |
|
||||
|
||||
两臂共同点:硬规则与契约表形状**同文**(受控变量,不测契约知识差异);修复回路相同(`mlsp validate` 诊断回喂,≤2 轮);模型 / 温度 / 任务 / 提示词模板相同。
|
||||
|
||||
### 2.2 材料:任务集(6 例,业务 3 + 算法 3)
|
||||
|
||||
| id | 类型 | 任务 | 功能 oracle(stdout 精确匹配) |
|
||||
|----|------|------|-------------------------------|
|
||||
| `claims_reserve_settle` | 业务 | 保险准备金(免赔/共保/分项限额/追偿/地板) | `28000` |
|
||||
| `contract_milestone_pay` | 业务 | 工程里程碑应付款(保留金/逾期罚/变更/税/地板) | `37354` |
|
||||
| `vendor_po_gate` | 业务 | 采购 PO 准付闸(折扣/加急/预算帽/合规费/地板) | `814` |
|
||||
| `qb_gcd` | 算法 | 最大公约数;打印 gcd(48,18) gcd(1071,462) gcd(0,5) gcd(17,5) 各一行 | `6` `21` `5` `1` |
|
||||
| `qb_max_sublist_sum` | 算法 | 最大子段和;三组输入各打印一行 | `6` `6` `-1` |
|
||||
| `qb_get_factors` | 算法 | 质因数分解,因数以 `*` 连接;12 / 13 / 60 各一行 | `2*2*3` `13` `2*2*3*5` |
|
||||
|
||||
业务任务书复用 MarqdoThesis `formal/exp3-bizreq/data/tasks/`(权威规则文本原样进入提示词);算法任务书为 QuixBugs 改写(从「修 bug」改「实现 + 固定输出」,消除参考源码变量)。
|
||||
|
||||
### 2.3 协议(每例)
|
||||
|
||||
1. 提示词 = 任务书 + 「输出完整 `.mq.md` 源码于单个 ```markdown 围栏」;
|
||||
2. **查询阶段**(仅 Q 臂):模型可输出 `QUERY: <关键字>` 行,harness 以 `mlsp syntax` 卡片作答并继续;累计 ≤3 次;
|
||||
3. **执行阶段**:`marqdo run` → stdout == oracle 即通过;失败则回喂 `mlsp validate` 结构化诊断 + stderr,允许 ≤2 次修复;
|
||||
4. 记录每轮 token(API `usage.prompt_tokens + completion_tokens` 累计)。
|
||||
|
||||
### 2.4 指标与判据
|
||||
|
||||
| 指标 | 口径 | 判据 |
|
||||
|------|------|------|
|
||||
| 功能通过率(一轮 / 最终≤3 轮) | stdout 与 oracle 精确匹配 | Q 最终通过率**不劣于** D(允许 −0 例) |
|
||||
| 首轮语法错误率 | 首轮 `validate`/run 出现 parse 类失败的例数占比 | Q ≤ D |
|
||||
| 首轮契约诊断率 | 首轮出现 `contract.*` 诊断的例数占比 | Q ≤ D(观测项) |
|
||||
| 平均修复轮次 | 通过前的修复次数(含首轮生成记 0) | Q ≤ D |
|
||||
| token 用量 | 逐例累计 prompt+completion | Q **显著更少**(≤ 70% of D) |
|
||||
|
||||
**判定**:四条判据全成立 ⇒ 「AI 不必学语法」实证成立;否则按 §3.7「继续改协议」,不得以改进文档充数。
|
||||
- **执行**:`OPENAI_MODEL=… cargo test --test blind_experiment -- --ignored --nocapture`(每模型一轮)。
|
||||
|
||||
## 3. 实验 (c):契约消融(问题二 · 语义校验)
|
||||
|
||||
**假设**:契约只在**有契约处**拦截错误值(P1:无契约 = 全动态),且不误报。
|
||||
|
||||
- **材料**:`tests/contract_ablation.rs` 内联四边界用例(调用实参 / 返回 / 表绑定 / 字段访问)×三形态:
|
||||
① 带契约 + 错值 → **必拦**(`contract.*` 诊断);② 带契约 + 对值 → **零误报**(正常产出);③ 同程序**删契约表** + 错值 → **零契约诊断**(动态语义原样)。
|
||||
- **指标与判据**:
|
||||
- 拦截率 = ①被拦 / ①总数 = **100%**;
|
||||
- 误报率 = ②误报 / ②总数 = **0%**;
|
||||
- 无契约零回归 = ③零 `contract.*` 诊断且输出 = 动态语义基线;且全量 `cargo test` 绿(441+ 既有测试是无契约路径的主回归面)。
|
||||
- **执行**:`cargo test --test contract_ablation`。
|
||||
|
||||
## 4. 实验 (d):防漂移演练(问题二 · 契约可信)
|
||||
|
||||
**假设**:契约写错**必**被机器抓(不静默信任契约)——three-problems.md §2.8「故意写错契约的金样必红」。
|
||||
|
||||
- **材料**:`tests/drift_drill.rs`——故意写错契约的 N 例(类型 typo `nubmer`、未知对象类型、契约参数名 ≠ 升参名 ×2、返回契约明显不符、重复契约表、错位契约表)+ **负对照**(被绑定的 `字段|类型` 数据表 = web_db_migration 式,必须**不**被当契约吃掉)。
|
||||
- **指标与判据**:
|
||||
- 必报率 = 漂移例被诊断(`check` 报错/警告或运行时 `contract.*`)/ 漂移例总数 = **100%**;
|
||||
- 负对照误报 = **0**。
|
||||
- **执行**:`cargo test --test drift_drill`。
|
||||
|
||||
## 5. 实验 (e):核心守卫演练(问题一 · 核心要小)
|
||||
|
||||
**假设**:核心表面任何**单侧**变更(代码/文档/金样不同步)→ CI 必红——three-problems.md §1.5「新增标记不更新清单时 CI 必须红」。
|
||||
|
||||
- **材料**:`tests/core_surface.rs` 守卫 + 三种变异注入(脚本化,`git checkout` 还原):
|
||||
1. `src/parse/mod.rs` `CORE_CONSTRUCTS` 增一行、不改文档 → **必红**;
|
||||
2. `doc/design/core-surface.md` 表增一行、不改代码 → **必红**;
|
||||
3. 金样文件改名(路径失效)→ **必红**。
|
||||
- **指标与判据**:3/3 变异注入后守卫测试红;还原后全绿。**必红率 = 100%**。
|
||||
- **执行**:报告内嵌演练记录(命令 + 测试输出摘录)。
|
||||
|
||||
## 6. 报告与复现
|
||||
|
||||
- 报告:[perf-validation-report-2026-09-23.md](perf-validation-report-2026-09-23.md)((a) 引用既有两份实验报告;(b)–(e) 本次产出)。
|
||||
- 复现命令全部写入报告各节;模型在环实验记录模型名与端点,判据数字允许 ±1 例波动(模型非确定性),**判据线本身不放宽**。
|
||||
- 汇总结论回填 [three-problems-plan.md](three-problems-plan.md) 落地记录;§5.1 判据矩阵逐行标注 ✅/❌。
|
||||
@@ -5,7 +5,7 @@
|
||||
| 状态 | **本期开发依据 · 执行中** |
|
||||
| 日期 | 2026-09-22 |
|
||||
| 设计 | [three-problems.md](three-problems.md)(解决方向与验证设计;判据矩阵 §5.1) |
|
||||
| 范围 | **工程任务 = Phase 1–3**;验证实验(three-problems.md §5.2)**不属于开发任务**,待「性能验证设计」(`doc/next/`,用户维护)导入后对接 |
|
||||
| 范围 | **工程任务 = Phase 1–3**;验证实验(three-problems.md §5.2)协议见 [perf-validation.md](perf-validation.md)(2026-09-23 用户授权自行设计并执行) |
|
||||
| 约定 | 每任务带机器可验证验收;**任务完成 = 验收全绿 + 全量回归绿**;只改宣言不改行为 = 未完成 |
|
||||
|
||||
## 0. 本期范围与依赖
|
||||
@@ -128,9 +128,11 @@ T1.1 core-surface ─是─> T3.1 语言面 + T1.3 守卫
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 · 验证实验(**非开发任务**)
|
||||
## Phase 4 · 验证实验(**协议 = [perf-validation.md](perf-validation.md)**)
|
||||
|
||||
- 对接「性能验证设计」(`doc/next/`,用户维护,待导入);three-problems.md §5.2 实验 (a)–(e) 的**材料由 Phase 1–3 的验收金样直接供给**;开发侧仅保留「实验基建适配」支持义务(复用 MarqdoThesis `formal/exp1–3` runner/oracle)。
|
||||
- 2026-09-23 用户授权**自行设计**性能验证设计并执行验证;`doc/next/002.md` 与性能验证无关(此前「待导入」表述已纠正)。
|
||||
- five 实验 (a)–(e) 的**材料由 Phase 1–3 的验收金样直接供给**;协议 / 指标 / 判据见 [perf-validation.md](perf-validation.md) §1–§5;
|
||||
实验产物 = [perf-validation-report-2026-09-23.md](perf-validation-report-2026-09-23.md)。
|
||||
|
||||
---
|
||||
|
||||
@@ -201,3 +203,23 @@ T1.1 core-surface ─是─> T3.1 语言面 + T1.3 守卫
|
||||
- **T3.3(AI Skill 换骨:背语法 → 查 MLSP)**(2026-09-22 完成)——`marqdo` / `marqdo-dev` 两件套改为
|
||||
查询式工作流:写码前 `mlsp syntax/locate`、提交前 `mlsp validate` + `marqdo check`;
|
||||
渐进式契约三种表格形状入语法面;错误修复照 `doc_anchor` 局部改(越界必弃权)。
|
||||
|
||||
## Phase 4 · 验证执行记录(2026-09-23,协议 [perf-validation.md](perf-validation.md))
|
||||
|
||||
- **(a) 锚点修复消融** ✅ PASS——判据「anchor 一轮 ≥80% + 护栏 0 越界」:cflmy anchor 20/20=100%(plain 85%)、
|
||||
mimo-v2.6-flash anchor 20/20=100%(plain 90%),越界拒/越界应用 0/0。易锚定样本近天花板,
|
||||
增量区分性留难样本扩展(协议 §1 诚实注记)。
|
||||
- **(b) 盲测(查询式 vs 全文档背诵)** ❌ 四判据未全立 ⇒ 按 three-problems.md §3.7 **继续改协议**——
|
||||
deepseek-flash:docs 4/6 vs query 0/3(端点 infra 剔除 3/6);首轮语法错 2/3 vs 2/6;token 69,771 vs 57,276。
|
||||
mimo-v2.6-pro 首跑 12/12 全端点空体,单独重跑(结果回填 [perf-validation-report-2026-09-23.md](perf-validation-report-2026-09-23.md) §3)。
|
||||
**真发现**:查询机制本身无缺陷(多词查询亦正确命中卡片),败例是**跨语言习惯泄漏**(元组表达式/Python 调用形)
|
||||
——模型"不知道自己不知道"而不去查。协议改进主攻方向:**自查触发机制**(validate 前置自检、
|
||||
陌生构造强制查询),而非继续改进卡片。
|
||||
- **(c) 契约消融** ✅ PASS——拦截率 4/4=100%、误报率 0/4=0%、无契约零回归 4/4(四边界:调用实参/返回/表绑定/键访问)。
|
||||
- **(d) 防漂移演练** ✅ PASS——必报率 9/9=100%(9 类故意写错全数诊断)、负对照误报 0/2。
|
||||
- **(e) 核心守卫演练** ✅ PASS——3/3 变异注入必红(清单私增/文档私增/金样删除)、还原必绿。
|
||||
- **判据矩阵回填**:three-problems.md §5.1 逐行标注——问题 1 ✅、问题 2 ✅、**问题 3 ❌(未解决)**。
|
||||
- **测量端记录**:LLM 网关随机空体为最大干扰(重试 5→12 次、退避 5→15 秒、截断自适应 max_tokens);
|
||||
harness 修复两处协议性 bug(查询额度用满误判、提取失败不进修复回路)后采样;
|
||||
模型口径按用户指示由 `cflmy`/`mimo-v2.6-flash` 改 `mimo-v2.6-pro`/`deepseek-flash`(实验 (a) 数据为原组合,不重跑)。
|
||||
- **全量干净回归**(无 API key 基线)零失败。
|
||||
@@ -9,7 +9,7 @@
|
||||
| 来源 | [001.md](../next/001.md) 评审讨论提炼的三大问题;用户决议:**问题二走「渐进式契约 / 标注类型」**(拒绝照搬静态类型,也拒绝放弃语义校验);**问题三(AI 可靠反馈 + AI 不必学语法)是下一步的超级核心** |
|
||||
| 相关 | [markdown-mapping-v0.3.md](../design/markdown-mapping-v0.3.md)(语法宪法)· [tables-maps-footnotes.md](tables-maps-footnotes.md)(表格几何)· [catalog-cli.md](../design/catalog-cli.md) · [ai-skill.md](../design/ai-skill.md) · [objects.md](../design/objects.md) · [interpreter.md](interpreter.md) |
|
||||
|
||||
> **本文定位:三大问题的解决方向与验证设计。** 给三大问题定义「什么算被解决」(机器可验证判据,见 §5.1)与验证实验(修复率 / 盲测 / 消融 / 防漂移演练,见 §5.2),把「真正解决」与「遮盖过去」区分开。文中的解决步骤(各节 S 表)是**设计方向**;本期开发任务、排期与验收以 [three-problems-plan.md](three-problems-plan.md) 为准。**验证实验(§5.2)不属于开发任务**——待「性能验证设计」文档导入后另行对接。
|
||||
> **本文定位:三大问题的解决方向与验证设计。** 给三大问题定义「什么算被解决」(机器可验证判据,见 §5.1)与验证实验(修复率 / 盲测 / 消融 / 防漂移演练,见 §5.2),把「真正解决」与「遮盖过去」区分开。文中的解决步骤(各节 S 表)是**设计方向**;本期开发任务、排期与验收以 [three-problems-plan.md](three-problems-plan.md) 为准。**验证实验(§5.2)协议见 [perf-validation.md](perf-validation.md)**(2026-09-23 用户授权自行设计并执行)。
|
||||
>
|
||||
> **一句话:语言核心收口是地基;渐进式契约是校验层,schema 直接长在表格与说明段落里;AI 语言服务消费这两者形成可靠闭环。** 三个问题必须一起解——契约是 AI 接口的数据源,核心收口是契约语法不膨胀的前提。
|
||||
|
||||
@@ -359,11 +359,15 @@ validate(p) → diagnose → patch(p, diag) → validate(p') → … (预算 N
|
||||
|
||||
### 5.1 判据矩阵:什么算「已解决」
|
||||
|
||||
| 问题 | 解决的定义(机器可验证) | 遮盖式收场(一句话就能写完的,禁止) | 验证手段 |
|
||||
|------|--------------------------|--------------------------------------|----------|
|
||||
| 1 核心要小 | 新标记不更新 core-surface → **CI 红** | 「我们重视语言核心」 | CI 守卫演练:故意加标记不更新清单,必须红 |
|
||||
| 2 语义校验 | 四边界校验器上线;契约写错**必**被机器抓;无契约路径零回归 | 「静态类型是非目标」/照搬 TS 类型/另写 JSON Schema | 契约金样 + 防漂移金样 + 全量回归 |
|
||||
| 3 AI 反馈 / 语法 | 盲测组错误率 ≤ 全文档组且 token 显著更少;一轮修复率可复现 | 「再写更好的文档」 | §5.2 实验 (a)(b)(d) |
|
||||
| 问题 | 解决的定义(机器可验证) | 遮盖式收场(一句话就能写完的,禁止) | 验证手段 | 判定(2026-09-23) |
|
||||
|------|--------------------------|--------------------------------------|----------|--------------------|
|
||||
| 1 核心要小 | 新标记不更新 core-surface → **CI 红** | 「我们重视语言核心」 | CI 守卫演练:故意加标记不更新清单,必须红 | ✅ (e) 3/3 必红→还原必绿 |
|
||||
| 2 语义校验 | 四边界校验器上线;契约写错**必**被机器抓;无契约路径零回归 | 「静态类型是非目标」/照搬 TS 类型/另写 JSON Schema | 契约金样 + 防漂移金样 + 全量回归 | ✅ (c) 拦截 100%/误报 0%/零回归;(d) 必报 9/9 |
|
||||
| 3 AI 反馈 / 语法 | 盲测组错误率 ≤ 全文档组且 token 显著更少;一轮修复率可复现 | 「再写更好的文档」 | §5.2 实验 (a)(b)(d) | ❌ (a) 过(双模型 100%);**(b) 四判据未全立** ⇒ 按 §3.7 继续改协议 |
|
||||
|
||||
> 2026-09-23 实测数字与判据对照:[perf-validation-report-2026-09-23.md](perf-validation-report-2026-09-23.md)。
|
||||
> 按本矩阵口径:**问题 1、2 实证成立;问题 3 未解决**(查询式盲测未跑赢全背诵——详见报告 §3 的四条要点,
|
||||
> 其中「模型不查自己不知道的语法」是协议改进的主攻方向)。
|
||||
|
||||
### 5.2 实验设计(性能 / 可靠性验证)
|
||||
|
||||
|
||||
+38
-2
@@ -93,17 +93,53 @@ const ALIASES: &[(&str, &[&str])] = &[
|
||||
("soft-emphasis", &["强调", "emphasis", "软强调"]),
|
||||
];
|
||||
|
||||
/// core-surface.md 全文(与 `tests/core_surface.rs` 守卫同源)——卡片机械携带
|
||||
/// 语义/戒律/样例**原文**,AI 一次查询即可自持,不必回读文档(three-problems.md §3.2「单构造戒律」)。
|
||||
const SURFACE_MD: &str = include_str!("../doc/design/core-surface.md");
|
||||
|
||||
/// 机械提取 core-surface.md 清单行:`| \`id\` | 外形 | 语义 | 戒律 | 样例 |` →
|
||||
///(语义, 戒律, 样例, 行原文);只读不改,卡片内容与文档永不漂移。
|
||||
fn surface_row(id: &str) -> Option<(String, String, String, String)> {
|
||||
for line in SURFACE_MD.lines() {
|
||||
let t = line.trim();
|
||||
if !t.starts_with('|') {
|
||||
continue;
|
||||
}
|
||||
let cells: Vec<&str> = t.trim_matches('|').split('|').map(str::trim).collect();
|
||||
if cells.len() != 5 {
|
||||
continue;
|
||||
}
|
||||
let rid = cells[0].trim_matches('`');
|
||||
if rid == id && !rid.is_empty() && !rid.contains(' ') && !rid.starts_with('-') {
|
||||
return Some((
|
||||
cells[2].to_string(),
|
||||
cells[3].to_string(),
|
||||
cells[4].to_string(),
|
||||
t.to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn card(id: &str) -> Value {
|
||||
let form = CORE_CONSTRUCTS
|
||||
.iter()
|
||||
.find(|(cid, _)| *cid == id)
|
||||
.map(|(_, f)| *f)
|
||||
.unwrap_or("?");
|
||||
json!({
|
||||
let mut v = json!({
|
||||
"id": id,
|
||||
"form": form,
|
||||
"manual": "doc/design/core-surface.md",
|
||||
})
|
||||
});
|
||||
if let Some((semantics, rule, sample, quote)) = surface_row(id) {
|
||||
v["semantics"] = semantics.into();
|
||||
v["rule"] = rule.into();
|
||||
v["sample"] = sample.into();
|
||||
v["doc_quote"] = quote.into();
|
||||
}
|
||||
v
|
||||
}
|
||||
|
||||
fn match_constructs(query: &str) -> Vec<Value> {
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
<!--
|
||||
P4 实验 (b) 材料 · D 臂(全文档背诵组)上下文包。
|
||||
来源:commit 199b5c5^ 的 `.cursor/skills/marqdo/SKILL.md`(背诵式「压缩操作手册」)
|
||||
语言部分原样冻结(Markup→meaning / 最小程序 / 函数调用 / 语句分支 / 导入与内建面)。
|
||||
只读实验材料——**请勿修改**(改了就不是同一个实验)。
|
||||
-->
|
||||
|
||||
## Markup → meaning (v0.3)
|
||||
|
||||
| Marker | Meaning |
|
||||
|--------|---------|
|
||||
| `#` | **Object / type** (constructor body). `# main` = entry object. `# Child = > Parent` = inherit (**no** implicit parent constructor; call `` `self` = > Parent … `` explicitly when needed). |
|
||||
| `##` … `######` | **Function / method** (nesting by heading depth). |
|
||||
| `` + `name` `` under heading | Parameter (optional `` `name`=default ``) |
|
||||
| `1.` `2.` … in body | Branch arm (condition or `N. *` else). **Same-indent restart at `1.` = new branch statement** (not more arms of the previous list). |
|
||||
| `- …` inside body | Loop (`while` or `` [item](coll) `` / `` [`item`](`coll`) ``) |
|
||||
| Line `N. *` | Else arm |
|
||||
| `> fn args` | Call (named `k=v` or positional) |
|
||||
| `修饰 [fn] args` | Bracket-marked call; leading words → `名=True` |
|
||||
| `` > `obj`.method args `` | Method call (`obj` must be a map with `_type`) |
|
||||
| Frontmatter `import bind:path.mq.md` / `导入` | Import file (bind library name) |
|
||||
| Frontmatter `import bind:lib.member` / `导入` | Short name for a library member (same keyword; no separate `use`) |
|
||||
| `> lib.member …` / `> lib.Type.member …` | Call via bare dotted path (instance methods need `` `var`.m ``) |
|
||||
| `**…**` | **Code** (assign / call) — closing `**` touches last token; bare ids are variables; quote text literals |
|
||||
| `*…*` | **Return value** — closing `*` touches last token; bare ids are variables |
|
||||
| `****` / whole-line `**` / `*None*` | Return `None` and end function body |
|
||||
| Lone `---` / `***` | Markdown thematic break (skip; not function end) |
|
||||
| GFM table after empty RHS bind | Collection; get with `[key](coll)` (preferred) or legacy `` `m`[^key] `` |
|
||||
| `- [item](coll)` | Foreach (unchanged) |
|
||||
| `` ```lang `` | Foreign code block (via `lib/foreign`) |
|
||||
| `` `"text"` `` | Quoted string (`\n` `\t` `\\` `\"`; `` `var` `` inside); bare tokens unescaped |
|
||||
| Unmarked prose | Comment |
|
||||
|
||||
Entry: load file (often `index.mq.md`) → collect defs → run `# main`.
|
||||
|
||||
## Minimal program
|
||||
|
||||
```markdown
|
||||
# main
|
||||
|
||||
> print text=Hello World!
|
||||
```
|
||||
|
||||
Chinese builtins (same functions, no import):
|
||||
|
||||
```markdown
|
||||
# main
|
||||
|
||||
> 打印 内容=你好
|
||||
```
|
||||
|
||||
## Functions, calls, end-body
|
||||
|
||||
```markdown
|
||||
# main
|
||||
|
||||
> greet who=Marqdo
|
||||
|
||||
## greet
|
||||
+ `who`
|
||||
|
||||
> print text=Hello, `who`!
|
||||
|
||||
*None*
|
||||
```
|
||||
|
||||
- Call: `> name key=value` or `> name value`.
|
||||
- Nested helpers use deeper `#` (`##`, `###`, …).
|
||||
- After a helper with only side effects, end with `*None*` / whole-line `**` / `****` when later lines must stay in the outer function. Lone `---` is only a thematic break (skipped).
|
||||
- In bold assigns, prefer bare calls: `**摘要 = 结账摘要 …**` (omit `>`).
|
||||
|
||||
## Statements, returns, branches
|
||||
|
||||
```markdown
|
||||
**x = 1**
|
||||
**y = x + 2**
|
||||
|
||||
## add_one
|
||||
+ `n`
|
||||
|
||||
*n + 1*
|
||||
```
|
||||
|
||||
```markdown
|
||||
**n = 0**
|
||||
|
||||
1. `n` > 0
|
||||
> print text=positive
|
||||
2. `n` < 0
|
||||
> print text=negative
|
||||
3. *
|
||||
> print text=zero
|
||||
```
|
||||
|
||||
## Frontmatter imports + stdlib
|
||||
|
||||
Imports use `import bind:target` (`导入` equivalent): file (`.mq.md`) or short name (`lib.member`). Members are **not** flattened into the caller. Call with a bare dotted path:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: example
|
||||
import text:lib/text.mq.md
|
||||
import clock:lib/time.mq.md
|
||||
---
|
||||
|
||||
# main
|
||||
|
||||
**xs = > text.str_split s="a,b,c" sep=","**
|
||||
> print text=`xs`
|
||||
**t = > clock.now_unix**
|
||||
```
|
||||
|
||||
- Import **English** or **Chinese** library file; call that file's API names via `lib.member` (do not mix languages).
|
||||
- Instance methods stay `` > `obj`.method `` (backticks on the receiver only).
|
||||
- `lib/…` resolves via `MARQDO_LIB`, cwd `lib/`, or `lib/` next to the `marqdo` binary.
|
||||
- Prefer `table.put` / `表.改` for list/map element updates; keep `json` for parse/stringify/quote.
|
||||
- Builtins (no import): `print`/`打印`, `input`/`输入`, `len`/`长度`, `str`/`文本`, `int`/`整数`; literals `True`/`真`, `False`/`假`, `None`/`空`; logic `and`/`且`, `or`/`或`, `not`/`非`.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task: claims_reserve_settle (insurance claims reserve)
|
||||
|
||||
Implement an **insurance claims reserve settlement** desk.
|
||||
This is a **business-policy** pipeline (deductible, coinsurance, sublimit, recovery, floor), **not** an algorithms puzzle.
|
||||
Use **integer arithmetic only**. Hardcode the demo inputs below and **print one integer**: the final reserve.
|
||||
|
||||
## Business requirements (authoritative)
|
||||
|
||||
1. **After deductible:** `after_deduct = claim - deductible`. If `after_deduct < 0`, set `after_deduct = 0`.
|
||||
2. **Coinsurance share (insurer):** `coinsured = after_deduct * coinsure_pct / 100` (demo coinsure_pct = 80). Apply coinsurance **before** recovery.
|
||||
3. **Sublimit cap:** `capped = coinsured` if `coinsured <= sublimit`, else `capped = sublimit` (never enlarge beyond coinsured).
|
||||
4. **Recovery offset:** `reserved = capped - recovery`. If `reserved < 0`, set `reserved = 0`.
|
||||
5. **Reserve floor:** if `reserved < reserve_floor`, set `reserved = reserve_floor`.
|
||||
|
||||
## Demo inputs (hardcode)
|
||||
|
||||
- `claim = 50000`
|
||||
- `deductible = 5000`
|
||||
- `coinsure_pct = 80`
|
||||
- `sublimit = 30000`
|
||||
- `recovery = 2000`
|
||||
- `reserve_floor = 1000`
|
||||
|
||||
## Requirements
|
||||
|
||||
- Provide named helpers (or clear stages) for deductible, coinsurance, sublimit, recovery, and floor — not one opaque expression.
|
||||
- Print only the final integer (no labels).
|
||||
- Align to the **business requirements above**; do not invent alternate order or comparison directions.
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Exactly one line: `28000`
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Task: contract_milestone_pay (construction milestone payable)
|
||||
|
||||
Implement a **contract milestone payment** desk for an EPC / construction settlement.
|
||||
This is a **business-policy** pipeline (retention, late penalty, change order, tax), **not** an algorithms puzzle.
|
||||
Use **integer arithmetic only**. Hardcode the demo inputs below and **print one integer**: the final payable.
|
||||
|
||||
## Business requirements (authoritative)
|
||||
|
||||
1. **Earned value:** `earned = contract_value * progress_pct / 100`.
|
||||
2. **Retention withhold:** `retention = earned * retain_pct / 100` (demo uses retain_pct = 5).
|
||||
3. **Net after retention:** `net = earned - retention`.
|
||||
4. **Late penalty:** if `days_late > grace_days` (grace_days = 7), set `penalty = net * late_rate / 100` (late_rate = 2); otherwise `penalty = 0`. Penalty is always computed from **`net`**, never from `earned`.
|
||||
5. **After penalty:** `net2 = net - penalty`.
|
||||
6. **Change order:** `taxed_base = net2 + change_order`. If `taxed_base < 0`, set `taxed_base = 0`.
|
||||
7. **Tax:** `tax = taxed_base * tax_rate / 100` (tax_rate = 6).
|
||||
8. **Payable:** `payable = taxed_base + tax`. If `payable < min_pay` (min_pay = 500), set `payable = min_pay`.
|
||||
|
||||
## Demo inputs (hardcode)
|
||||
|
||||
- `contract_value = 100000`
|
||||
- `progress_pct = 40`
|
||||
- `retain_pct = 5`
|
||||
- `days_late = 10`
|
||||
- `grace_days = 7`
|
||||
- `late_rate = 2`
|
||||
- `change_order = -2000`
|
||||
- `tax_rate = 6`
|
||||
- `min_pay = 500`
|
||||
|
||||
## Requirements
|
||||
|
||||
- Provide named helpers (or clear stages) for earned, retention, penalty, change+tax, and floor — not one opaque expression.
|
||||
- Print only the final integer (no labels).
|
||||
- Align to the **business requirements above**; do not invent alternate thresholds or penalty bases.
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Exactly one line: `37354`
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task: qb_gcd (easy) — Greatest Common Divisor
|
||||
|
||||
Implement **Greatest Common Divisor** as a function named `gcd` in a Marqdo `.mq.md` program (Euclidean algorithm).
|
||||
|
||||
## Requirements
|
||||
|
||||
1. Function name `gcd`, taking two integer parameters and returning their greatest common divisor (non-negative; `gcd(0, 5)` is `5`).
|
||||
2. In `# main`, hardcode the demo calls below and print each result on its own line — exactly four lines, no labels.
|
||||
|
||||
## Demo calls (hardcode)
|
||||
|
||||
- `gcd(48, 18)`
|
||||
- `gcd(1071, 462)`
|
||||
- `gcd(0, 5)`
|
||||
- `gcd(17, 5)`
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Four lines:
|
||||
|
||||
```
|
||||
6
|
||||
21
|
||||
5
|
||||
1
|
||||
```
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task: qb_get_factors (hard) — Prime Factorization
|
||||
|
||||
Implement **prime factorization** as a function named `get_factors` in a Marqdo `.mq.md` program. The function takes an integer `n ≥ 2` and returns its prime factors in ascending order.
|
||||
|
||||
## Requirements
|
||||
|
||||
1. Function name `get_factors`.
|
||||
2. In `# main`, hardcode the demo inputs below and print each result on its own line — factors joined by `*` with no spaces, exactly three lines, no labels.
|
||||
|
||||
## Demo inputs (hardcode)
|
||||
|
||||
- `12`
|
||||
- `13`
|
||||
- `60`
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Three lines:
|
||||
|
||||
```
|
||||
2*2*3
|
||||
13
|
||||
2*2*3*5
|
||||
```
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task: qb_max_sublist_sum (hard) — Maximum Sublist Sum
|
||||
|
||||
Implement the **maximum contiguous sublist sum** as a function named `max_sublist_sum` in a Marqdo `.mq.md` program. The function takes a list of integers and returns the maximum sum of any contiguous sublist (at least one element).
|
||||
|
||||
## Requirements
|
||||
|
||||
1. Function name `max_sublist_sum`.
|
||||
2. In `# main`, hardcode the demo inputs below and print each result on its own line — exactly three lines, no labels.
|
||||
|
||||
## Demo inputs (hardcode)
|
||||
|
||||
- `[-2, 1, -3, 4, -1, 2, 1, -5, 4]`
|
||||
- `[1, 2, 3]`
|
||||
- `[-3, -1, -2]`
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Three lines:
|
||||
|
||||
```
|
||||
6
|
||||
6
|
||||
-1
|
||||
```
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task: vendor_po_gate (complex procurement business policy)
|
||||
|
||||
Implement a **vendor purchase-order payable gate** for an enterprise procurement desk.
|
||||
This is a **business-policy** task (multi-rule approval economics), **not** an algorithms puzzle.
|
||||
Use **integer arithmetic only**. Hardcode the demo inputs below and **print one integer**: the final payable.
|
||||
|
||||
## Business requirements (authoritative)
|
||||
|
||||
1. **Base line:** `base = qty * unit_cost`.
|
||||
2. **Preferred-vendor discount:** if `vendor_tier >= 2` (tier-2+ preferred suppliers), set `amount = base * 97 / 100`; otherwise `amount = base`.
|
||||
3. **Rush surcharge:** if `urgency == 1`, set `amount = amount + amount * 5 / 100`; otherwise keep `amount`.
|
||||
4. **Budget clamp:** if `amount > budget_cap`, set `amount = budget_cap`.
|
||||
5. **Compliance fee:** if `risk_flag == 1`, `fee = 80`; else `fee = 0`.
|
||||
6. **Payable floor:** `payable = amount + fee`; if `payable < 200` then `payable = 200`.
|
||||
|
||||
## Demo inputs (hardcode)
|
||||
|
||||
- `qty = 20`
|
||||
- `unit_cost = 40`
|
||||
- `vendor_tier = 2`
|
||||
- `urgency = 1`
|
||||
- `budget_cap = 900`
|
||||
- `risk_flag = 0`
|
||||
|
||||
## Requirements
|
||||
|
||||
- Provide named helpers (or clear stages) for discount, rush, budget clamp, fee, and assemble — not one opaque expression.
|
||||
- Print only the final integer (no labels).
|
||||
- Align the program to the **business requirements above**; do not invent alternate thresholds.
|
||||
|
||||
## Expected output (self-check)
|
||||
|
||||
Exactly one line: `814`
|
||||
|
||||
## Deliverable
|
||||
|
||||
Write the program as a Marqdo `.mq.md` file. Return the complete source in a single ```markdown fence — no prose outside the fence.
|
||||
@@ -0,0 +1,585 @@
|
||||
//! P4 实验 (b) 盲测——**查询式(Q)vs 全文档背诵(D)**
|
||||
//! ([perf-validation.md](../../doc/roadmap/perf-validation.md) §2;three-problems.md §3.7 最终判据)。
|
||||
//!
|
||||
//! 唯一变量 = 语法知识获取方式:Q 臂只给硬规则 + 契约表形状 + MLSP 查询通道(≤3 次);
|
||||
//! D 臂给硬规则 + 契约表形状 + 完整 markup 参考(`blind/docs-bundle.md`,背诵式全语法表)。
|
||||
//! 两臂任务 / 提示词模板 / 修复回路(≤2 次)/ 模型完全相同。
|
||||
//!
|
||||
//! 判据:Q 功能通过率不劣于 D;首轮语法错误率 Q ≤ D;token Q ≤ 70% of D;修复轮次不升。
|
||||
//! 执行:`OPENAI_MODEL=… cargo test --test blind_experiment -- --ignored --nocapture`。
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
use marqdo::mlsp::handle_line;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
// ---------- 材料(冻结;只读实验材料) ----------
|
||||
|
||||
const DOCS_BUNDLE: &str = include_str!("blind/docs-bundle.md");
|
||||
const T_CLAIMS: &str = include_str!("blind/tasks/claims_reserve_settle.md");
|
||||
const T_MILESTONE: &str = include_str!("blind/tasks/contract_milestone_pay.md");
|
||||
const T_VENDOR: &str = include_str!("blind/tasks/vendor_po_gate.md");
|
||||
const T_GCD: &str = include_str!("blind/tasks/qb_gcd.md");
|
||||
const T_KADANE: &str = include_str!("blind/tasks/qb_max_sublist_sum.md");
|
||||
const T_FACTORS: &str = include_str!("blind/tasks/qb_get_factors.md");
|
||||
|
||||
struct Task {
|
||||
id: &'static str,
|
||||
desc: &'static str,
|
||||
oracle: &'static str,
|
||||
}
|
||||
|
||||
fn tasks() -> Vec<Task> {
|
||||
vec![
|
||||
Task { id: "claims_reserve_settle", desc: T_CLAIMS, oracle: "28000" },
|
||||
Task { id: "contract_milestone_pay", desc: T_MILESTONE, oracle: "37354" },
|
||||
Task { id: "vendor_po_gate", desc: T_VENDOR, oracle: "814" },
|
||||
Task { id: "qb_gcd", desc: T_GCD, oracle: "6\n21\n5\n1" },
|
||||
Task { id: "qb_max_sublist_sum", desc: T_KADANE, oracle: "6\n6\n-1" },
|
||||
Task { id: "qb_get_factors", desc: T_FACTORS, oracle: "2*2*3\n13\n2*2*3*5" },
|
||||
]
|
||||
}
|
||||
|
||||
// ---------- 提示词(受控:硬规则与契约形状两臂同文) ----------
|
||||
|
||||
const HARD_RULES: &str = "\
|
||||
## Hard rules (do not violate)\n\
|
||||
1. **File suffix** must be `.mq.md`.\n\
|
||||
2. **`**…**` = code** (assign / call). **`*…*` = return value**. Output is **not** either marker — use `> print text=…` / `**print text=\"…\"**` / `> 打印 内容=…`.\n\
|
||||
3. **Prose:** unmarked narrative may contain `` `名` `` (declare/ref), inline `**code**`, and `*return*`. Soft Markdown emphasis that is not code-shaped is ignored. Dead `` `名` `` (never read) is OK and does not become a param.\n\
|
||||
4. **Do not invent keywords** `if` / `else` / `while` / `for` / `def` / `return`. Control flow is Markdown: **`1.` `2.` … branches**, **`-` loops**, arm `N. *` = else, `#` = **object/type**, `##`+ = **function/method**. Params: prefer prose `` `名` `` / `` `名`=默认 `` (inferred); `` + `名` `` still accepted.\n\
|
||||
5. **Identifiers:** `` `名` `` in prose; inside `*…*` / `**…**`, bare ids are **variables**. **Text literals must be quoted** in bold/italic: `**print text=\"hi\"**`. Standalone `>` calls: bare words = text; vars need ticks: `` > str `n` ``.\n\
|
||||
6. **Structure lines** (`#` `>` `+` `-` `|` `1.`) need not be wrapped in bold.\n\
|
||||
7. **Paths:** In bare expressions `/` is division. In call args / defaults / table cells, unspaced `a/b` and quoted paths are text.\n\
|
||||
8. Prefer ending side-effect-only bodies with `*None*` / `*无*` / whole-line `**` / `****` when the next lines must belong to an outer function. Lone `---` / `***` are Markdown thematic breaks (skipped everywhere), not function end.\n\
|
||||
9. **`ext/**` never calls `host_*`.**\n\
|
||||
10. **Code-as-documentation:** prefer GFM tables for maps/lists; prefer named helpers over `json.set` / `json.append` chains.\n\
|
||||
11. **Table element access has two spellings** — `[键](集合)` is the recommended one (e.g. `[name](`用户`)`, `[1](`xs`)`); footnote form `` `集合`[^键] `` is legacy-compat only.\n\
|
||||
12. **Bracket-marked calls:** function name may be marked with `[函数名]`; bare words before `[` are boolean modifiers. `]` immediately followed by `(` is **index** `[key](coll)`, not a call.\n";
|
||||
|
||||
const CONTRACT_SHAPES: &str = "\
|
||||
## 渐进式契约表(可选校验元数据)\n\
|
||||
函数契约 = `##` 函数体内**首个可执行行之前**的未绑定文档表,两种列头:\n\
|
||||
\n\
|
||||
| 参数 | 类型 | 说明 |\n\
|
||||
|------|------|------|\n\
|
||||
| `n` | number | 输入值 |\n\
|
||||
\n\
|
||||
| 返回 | 类型 | 说明 |\n\
|
||||
|------|------|------|\n\
|
||||
| | number | 返回说明 |\n\
|
||||
\n\
|
||||
集合契约 = `` `名` = `` 绑定**之前**的 `| 字段 | 类型 | 可空 |` 表。\n\
|
||||
对象契约 = `# 对象` 体内首个成员之前的 `| 字段 | 类型 |` 表。\n\
|
||||
规则:**被绑定的表是数据,永不视为契约**;类型词表 = `text` / `number` / `bool` / `list` / `map` / `any` / 对象名;\n\
|
||||
契约表一经出现即为完整声明(写全参数行,否则报 param_drift 警告)。\n";
|
||||
|
||||
const QUERY_PROTOCOL: &str = "\
|
||||
## On-demand syntax queries(你的唯一语法来源)\n\
|
||||
你**没有** Marqdo 语法参考。对任何不确定的语法细节,先查询再写码:单独输出一行\n\
|
||||
`QUERY: <关键字>`(如 `QUERY: 返回`、`QUERY: 表`、`QUERY: 调用`),我会回给你对应构造卡片(语义/戒律/样例原文)。\n\
|
||||
最多 3 次查询。**不要凭 Markdown/Python 习惯猜语法。**\n";
|
||||
|
||||
fn system_prompt(arm: &str) -> String {
|
||||
let intro = "你是 Marqdo 语言(`.mq.md`:Markdown 标记即语法)的程序作者。写完整可运行程序(含 `# main`)。\n\n";
|
||||
let mut s = String::from(intro);
|
||||
s.push_str(HARD_RULES);
|
||||
s.push('\n');
|
||||
s.push_str(CONTRACT_SHAPES);
|
||||
s.push('\n');
|
||||
if arm == "docs" {
|
||||
s.push_str("## Markup reference (v0.3, complete)\n\n");
|
||||
s.push_str(DOCS_BUNDLE);
|
||||
} else {
|
||||
s.push_str(QUERY_PROTOCOL);
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
// ---------- LLM 接入(OpenAI 兼容;curl 免依赖;带 usage 计量) ----------
|
||||
|
||||
struct Llm {
|
||||
key: String,
|
||||
base: String,
|
||||
model: String,
|
||||
}
|
||||
|
||||
struct Reply {
|
||||
text: String,
|
||||
ptok: u32,
|
||||
ctok: u32,
|
||||
}
|
||||
|
||||
fn llm_cfg() -> Llm {
|
||||
fn env_of(keys: &[&str]) -> Option<String> {
|
||||
keys.iter()
|
||||
.find_map(|k| std::env::var(k).ok())
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
let mut key = env_of(&["OPENAI_API_KEY", "MARQDO_LLM_API_KEY", "USTC_LLM_API_KEY"]);
|
||||
let mut base = env_of(&["OPENAI_BASE_URL", "MARQDO_LLM_BASE_URL"]);
|
||||
let mut model = env_of(&["OPENAI_MODEL", "MARQDO_LLM_MODEL"]);
|
||||
let env_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join(".env");
|
||||
if let Ok(txt) = std::fs::read_to_string(&env_path) {
|
||||
for line in txt.lines() {
|
||||
let line = line.trim();
|
||||
if line.is_empty() || line.starts_with('#') {
|
||||
continue;
|
||||
}
|
||||
if let Some((k, v)) = line.split_once('=') {
|
||||
let v = v.trim().trim_matches('"').to_string();
|
||||
match k.trim() {
|
||||
"OPENAI_API_KEY" | "MARQDO_LLM_API_KEY" => key = key.or(Some(v)),
|
||||
"OPENAI_BASE_URL" | "MARQDO_LLM_BASE_URL" => base = base.or(Some(v)),
|
||||
"OPENAI_MODEL" | "MARQDO_LLM_MODEL" => model = model.or(Some(v)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Llm {
|
||||
key: key.expect("未找到 API 密钥:请设置 OPENAI_API_KEY 或仓库根 .env"),
|
||||
base: base.unwrap_or_else(|| "https://llm.cflmy.cn/v1".to_string()),
|
||||
model: model.unwrap_or_else(|| "cflmy".to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// 多轮对话一次;空体/非 JSON/瞬时错误自动重试(容错重试),仍失败 ⇒ Err(基础设施失败)。
|
||||
fn chat(llm: &Llm, msgs: &[(String, String)]) -> Result<Reply, String> {
|
||||
let messages: Vec<Value> = msgs
|
||||
.iter()
|
||||
.map(|(r, c)| json!({"role": r, "content": c}))
|
||||
.collect();
|
||||
let url = format!("{}/chat/completions", llm.base.trim_end_matches('/'));
|
||||
let auth = format!("Authorization: Bearer {}", llm.key);
|
||||
let mut last = String::new();
|
||||
let mut max_tokens = 12000u64; // reasoning 模型先思考后作答;截断时自适应加倍
|
||||
for _ in 0..12 {
|
||||
let req = json!({
|
||||
"model": llm.model,
|
||||
"temperature": 0,
|
||||
"max_tokens": max_tokens,
|
||||
"messages": messages
|
||||
});
|
||||
let mut child = std::process::Command::new("curl")
|
||||
.args(["-sS", "-m", "180", "--retry", "2", "--retry-all-errors", "--retry-delay", "2", &url, "-H", &auth, "-H", "Content-Type: application/json", "--data-binary", "@-"])
|
||||
.stdin(std::process::Stdio::piped())
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.expect("启动 curl");
|
||||
child
|
||||
.stdin
|
||||
.as_mut()
|
||||
.unwrap()
|
||||
.write_all(req.to_string().as_bytes())
|
||||
.unwrap();
|
||||
let out = child.wait_with_output().unwrap();
|
||||
let text = String::from_utf8_lossy(&out.stdout).to_string();
|
||||
match serde_json::from_str::<Value>(&text) {
|
||||
Ok(v) if v["error"].is_object() => {
|
||||
last = format!("LLM 报错: {}", v["error"]);
|
||||
}
|
||||
Ok(v) => {
|
||||
let reply = v["choices"][0]["message"]["content"].as_str().unwrap_or("");
|
||||
if !reply.is_empty() {
|
||||
return Ok(Reply {
|
||||
text: reply.to_string(),
|
||||
ptok: v["usage"]["prompt_tokens"].as_u64().unwrap_or(0) as u32,
|
||||
ctok: v["usage"]["completion_tokens"].as_u64().unwrap_or(0) as u32,
|
||||
});
|
||||
}
|
||||
if v["choices"][0]["finish_reason"] == "length" && max_tokens < 32000 {
|
||||
max_tokens *= 2; // reasoning 吃满额度 ⇒ 加倍后重试
|
||||
last = format!("截断(finish=length,content 空),max_tokens 加倍至 {max_tokens}");
|
||||
} else {
|
||||
last = format!("响应缺 content: {}", truncate(&text, 120));
|
||||
}
|
||||
}
|
||||
Err(_) => {
|
||||
let err = String::from_utf8_lossy(&out.stderr).to_string();
|
||||
last = format!("非 JSON / 空体: {} {}", truncate(&text, 80), truncate(&err, 80));
|
||||
}
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_secs(15));
|
||||
}
|
||||
Err(last)
|
||||
}
|
||||
|
||||
fn truncate(s: &str, n: usize) -> String {
|
||||
let t: String = s.chars().take(n).collect();
|
||||
if s.chars().count() > n {
|
||||
format!("{t}…")
|
||||
} else {
|
||||
t
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 程序提取 / 查询 / 执行 ----------
|
||||
|
||||
/// 从回复中提取程序源:优先含 `# main` 的围栏,其次最长围栏,再次裸文含 `# main`。
|
||||
fn extract_program(reply: &str) -> Option<String> {
|
||||
let mut best: Option<String> = None;
|
||||
let mut rest = reply;
|
||||
while let Some(a) = rest.find("```") {
|
||||
let after = &rest[a + 3..];
|
||||
let after = after.strip_prefix("markdown").unwrap_or(after);
|
||||
let after = after.strip_prefix('\n').unwrap_or(after);
|
||||
if let Some(b) = after.find("```") {
|
||||
let block = &after[..b];
|
||||
if block.contains("# main") {
|
||||
return Some(dedent(block));
|
||||
}
|
||||
if best.as_ref().map(|s: &String| s.len() < block.len()).unwrap_or(true) {
|
||||
best = Some(block.to_string());
|
||||
}
|
||||
rest = &after[b + 3..];
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if let Some(b) = best {
|
||||
return Some(dedent(&b));
|
||||
}
|
||||
if reply.contains("# main") {
|
||||
return Some(reply.trim().to_string());
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn dedent(block: &str) -> String {
|
||||
let lines: Vec<&str> = block.lines().collect();
|
||||
let indent = lines
|
||||
.iter()
|
||||
.filter(|l| !l.trim().is_empty())
|
||||
.map(|l| l.len() - l.trim_start().len())
|
||||
.min()
|
||||
.unwrap_or(0);
|
||||
let out: String = lines
|
||||
.iter()
|
||||
.map(|l| l.get(indent..).unwrap_or(l))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
format!("{}\n", out.trim_end())
|
||||
}
|
||||
|
||||
/// QUERY 行 → 查询关键字列表。
|
||||
fn extract_queries(reply: &str) -> Vec<String> {
|
||||
reply
|
||||
.lines()
|
||||
.filter_map(|l| {
|
||||
let t = l.trim();
|
||||
let rest = t.strip_prefix("QUERY:").or_else(|| t.strip_prefix("QUERY:"))?;
|
||||
let q = rest.trim();
|
||||
if q.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(q.to_string())
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn mlp_syntax(query: &str) -> String {
|
||||
handle_line(&json!({"id": 1, "method": "syntax", "params": {"query": query}}).to_string())
|
||||
}
|
||||
|
||||
fn validate_codes(src: &str) -> (bool, Vec<String>, String) {
|
||||
let resp: Value =
|
||||
serde_json::from_str(&handle_line(&json!({"id": 1, "method": "validate", "params": {"source": src}}).to_string()))
|
||||
.expect("MLSP 响应必须是合法 JSON");
|
||||
if resp["ok"] != true {
|
||||
let code = resp["error"]["code"].as_str().unwrap_or("?").to_string();
|
||||
return (false, vec![code], resp.to_string());
|
||||
}
|
||||
let diags = resp["result"]["diagnostics"].as_array().cloned().unwrap_or_default();
|
||||
let codes: Vec<String> = diags
|
||||
.iter()
|
||||
.filter_map(|d| d["code"].as_str().map(String::from))
|
||||
.collect();
|
||||
(true, codes, diags.iter().map(|d| d.to_string()).collect::<Vec<_>>().join("\n"))
|
||||
}
|
||||
|
||||
fn run_src(task_id: &str, arm: &str, src: &str) -> (i32, String, String) {
|
||||
let p = std::env::temp_dir().join(format!(
|
||||
"marqdo_blind_{}_{}_{}.mq.md",
|
||||
std::process::id(),
|
||||
arm,
|
||||
task_id
|
||||
));
|
||||
std::fs::write(&p, src).unwrap();
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_marqdo"))
|
||||
.args(["run", p.to_str().unwrap()])
|
||||
.output()
|
||||
.expect("运行 marqdo");
|
||||
let _ = std::fs::remove_file(&p);
|
||||
(
|
||||
out.status.code().unwrap_or(1),
|
||||
String::from_utf8_lossy(&out.stdout).to_string(),
|
||||
String::from_utf8_lossy(&out.stderr).to_string(),
|
||||
)
|
||||
}
|
||||
|
||||
// ---------- 单例执行 ----------
|
||||
|
||||
struct Outcome {
|
||||
task: &'static str,
|
||||
arm: &'static str,
|
||||
result: String, // pass / load-fail / run-fail / wrong-output / infra
|
||||
rounds: u32, // 通过前的修复次数(首轮通过 = 0)
|
||||
first_load_fail: bool,
|
||||
first_contract_diag: bool,
|
||||
ptok: u32,
|
||||
ctok: u32,
|
||||
queries: u32,
|
||||
note: String,
|
||||
}
|
||||
|
||||
fn run_case(llm: &Llm, task: &Task, arm: &'static str) -> Outcome {
|
||||
let mut msgs: Vec<(String, String)> = vec![
|
||||
("system".into(), system_prompt(arm)),
|
||||
("user".into(), format!("{}\n\n开始写程序。", task.desc)),
|
||||
];
|
||||
let (mut ptok, mut ctok) = (0u32, 0u32);
|
||||
let mut queries = 0u32;
|
||||
|
||||
// 查询阶段(Q 臂协议;D 臂模型不被教导 QUERY,自然不会发)
|
||||
let mut text = String::new();
|
||||
let mut infra: Option<String> = None;
|
||||
for _ in 0..6 {
|
||||
match chat(llm, &msgs) {
|
||||
Ok(r) => {
|
||||
ptok += r.ptok;
|
||||
ctok += r.ctok;
|
||||
text = r.text;
|
||||
}
|
||||
Err(e) => {
|
||||
infra = Some(e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if extract_program(&text).is_some() {
|
||||
break; // 已带程序(即便附带 QUERY 行也直接用)
|
||||
}
|
||||
let qs = extract_queries(&text);
|
||||
if qs.is_empty() {
|
||||
break; // 无程序无查询 ⇒ 交给修复回路要程序
|
||||
}
|
||||
let mut answer = String::new();
|
||||
for q in qs.iter().take(3usize.saturating_sub(queries as usize)) {
|
||||
answer.push_str(&format!("QUERY {} → {}\n", q, mlp_syntax(q)));
|
||||
queries += 1;
|
||||
}
|
||||
if answer.is_empty() {
|
||||
answer.push_str("查询额度已用完(最多 3 次)。不要再查询,直接写完整程序。\n");
|
||||
}
|
||||
msgs.push(("assistant".into(), text.clone()));
|
||||
msgs.push((
|
||||
"user".into(),
|
||||
format!("{answer}\n继续;完成后把完整程序放在**单个** ```markdown 围栏里。"),
|
||||
));
|
||||
}
|
||||
if let Some(e) = infra {
|
||||
return Outcome {
|
||||
task: task.id,
|
||||
arm,
|
||||
result: "infra".into(),
|
||||
rounds: 0,
|
||||
first_load_fail: false,
|
||||
first_contract_diag: false,
|
||||
ptok,
|
||||
ctok,
|
||||
queries,
|
||||
note: truncate(&e, 140),
|
||||
};
|
||||
}
|
||||
|
||||
let mut out = Outcome {
|
||||
task: task.id,
|
||||
arm,
|
||||
result: "infra".into(),
|
||||
rounds: 0,
|
||||
first_load_fail: false,
|
||||
first_contract_diag: false,
|
||||
ptok,
|
||||
ctok,
|
||||
queries,
|
||||
note: String::new(),
|
||||
};
|
||||
|
||||
for attempt in 0..3u32 {
|
||||
let src_opt = extract_program(&text);
|
||||
let (load_ok, codes, diag_txt) = match &src_opt {
|
||||
Some(src) => validate_codes(src),
|
||||
None => (false, vec!["harness.no_program".to_string()], String::new()),
|
||||
};
|
||||
let contract_diag = codes.iter().any(|c| c.starts_with("contract."));
|
||||
if attempt == 0 {
|
||||
out.first_load_fail = src_opt.is_none() || !load_ok;
|
||||
out.first_contract_diag = contract_diag;
|
||||
}
|
||||
let (code, stdout, stderr) = match &src_opt {
|
||||
Some(src) => run_src(task.id, arm, src),
|
||||
None => (
|
||||
1,
|
||||
String::new(),
|
||||
"回复中没有 ```markdown 围栏程序(或不含 # main)。".to_string(),
|
||||
),
|
||||
};
|
||||
if code == 0 && stdout.trim_end() == task.oracle.trim() {
|
||||
out.result = "pass".into();
|
||||
out.rounds = attempt;
|
||||
out.note = format!("queries={queries}");
|
||||
break;
|
||||
}
|
||||
out.result = if src_opt.is_none() || !load_ok {
|
||||
"load-fail".into()
|
||||
} else if code != 0 {
|
||||
"run-fail".into()
|
||||
} else {
|
||||
"wrong-output".into()
|
||||
};
|
||||
out.note = if src_opt.is_none() {
|
||||
format!("回复中无程序 | 开头: {}", truncate(&text.replace('\n', " "), 160))
|
||||
} else {
|
||||
truncate(&format!("{diag_txt} | stderr={stderr}"), 300)
|
||||
};
|
||||
if attempt == 2 {
|
||||
break;
|
||||
}
|
||||
msgs.push(("assistant".into(), text.clone()));
|
||||
msgs.push((
|
||||
"user".into(),
|
||||
format!(
|
||||
"程序未通过。\nvalidate 诊断:\n{diag_txt}\n\n运行 stderr:\n{}\n\n运行 stdout:\n{}\n\n请修复问题,把**完整**程序放在**单个** ```markdown 围栏里(围栏内是纯 Marqdo 源码,含 `# main`)。",
|
||||
truncate(&stderr, 500),
|
||||
truncate(&stdout, 200)
|
||||
),
|
||||
));
|
||||
match chat(llm, &msgs) {
|
||||
Ok(r) => {
|
||||
ptok += r.ptok;
|
||||
ctok += r.ctok;
|
||||
text = r.text;
|
||||
}
|
||||
Err(e) => {
|
||||
out.note = format!("infra: {}", truncate(&e, 100));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
out.ptok = ptok;
|
||||
out.ctok = ctok;
|
||||
out
|
||||
}
|
||||
|
||||
// ---------- 实验入口 ----------
|
||||
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn exp_b_blind_query_vs_docs() {
|
||||
let llm = llm_cfg();
|
||||
let limit: usize = std::env::var("MARQDO_EXP_LIMIT")
|
||||
.ok()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.unwrap_or(usize::MAX);
|
||||
let mut outs: Vec<Outcome> = Vec::new();
|
||||
for arm in ["query", "docs"] {
|
||||
for t in tasks().iter().take(limit) {
|
||||
let o = run_case(&llm, t, arm);
|
||||
eprintln!(
|
||||
"[{}] {:<24} {:<12} rounds={} tok=({},{}) q={} | {}",
|
||||
o.arm, o.task, o.result, o.rounds, o.ptok, o.ctok, o.queries, o.note
|
||||
);
|
||||
outs.push(o);
|
||||
}
|
||||
}
|
||||
|
||||
// 汇总(按判据口径:infra 例从比率中剔除并单列)
|
||||
let mut md = String::new();
|
||||
md.push_str(&format!(
|
||||
"# 盲测实验(§5.2(b) 查询式 vs 全文档背诵)—— {}\n\n| | |\n|---|---|\n| 模型 | `{}` @ `{}` |\n| 样本 | 6 任务(业务 3 + QuixBugs 改写 3),oracle = stdout 精确匹配 |\n| 协议 | 系统提示词 = 硬规则 12 条 + 契约表三形状(两臂同文);**唯一变量 = 语法来源**:`query` 臂 MLSP 查询通道(≤3 次)vs `docs` 臂完整 markup 参考(背诵) |\n| 修复 | validate 诊断 + stderr 回喂,≤2 次修复;temperature=0,max_tokens=8000 |\n\n",
|
||||
chrono_date(),
|
||||
llm.model,
|
||||
llm.base
|
||||
));
|
||||
|
||||
md.push_str("## 汇总(判据:Q 通过率不劣于 D;首轮语法错误率 Q≤D;token Q≤70%%D;轮次不升)\n\n");
|
||||
md.push_str("| 臂 | 通过(一轮) | 通过(最终≤3) | 首轮语法错误 | 首轮契约诊断 | 平均修复轮(过例) | token 合计(prompt+completion) | 弃权/基础设施 |\n");
|
||||
md.push_str("|---|---|---|---|---|---|---|---|\n");
|
||||
for arm in ["query", "docs"] {
|
||||
let os: Vec<&Outcome> = outs.iter().filter(|o| o.arm == arm).collect();
|
||||
let n = os.len();
|
||||
let infra = os.iter().filter(|o| o.result == "infra").count();
|
||||
let denom = n - infra;
|
||||
let pass1 = os.iter().filter(|o| o.result == "pass" && o.rounds == 0).count();
|
||||
let passf = os.iter().filter(|o| o.result == "pass").count();
|
||||
let lf0 = os.iter().filter(|o| o.first_load_fail).count();
|
||||
let cd0 = os.iter().filter(|o| o.first_contract_diag).count();
|
||||
let rounds: f64 = {
|
||||
let p: Vec<u32> = os.iter().filter(|o| o.result == "pass").map(|o| o.rounds).collect();
|
||||
if p.is_empty() {
|
||||
0.0
|
||||
} else {
|
||||
p.iter().sum::<u32>() as f64 / p.len() as f64
|
||||
}
|
||||
};
|
||||
let ptok: u32 = os.iter().map(|o| o.ptok).sum();
|
||||
let ctok: u32 = os.iter().map(|o| o.ctok).sum();
|
||||
md.push_str(&format!(
|
||||
"| {arm} | {pass1}/{denom} | {passf}/{denom} | {lf0}/{denom} | {cd0}/{denom} | {rounds:.2} | {}({}+{}) | {infra} |\n",
|
||||
ptok + ctok,
|
||||
ptok,
|
||||
ctok
|
||||
));
|
||||
}
|
||||
|
||||
md.push_str("\n## 逐例\n\n| 例 | 臂 | 结果 | 修复轮 | 首轮语法错 | 首轮契约诊断 | token(p+c) | 查询 | 备注 |\n|---|---|---|---|---|---|---|---|---|\n");
|
||||
for o in &outs {
|
||||
md.push_str(&format!(
|
||||
"| {} | {} | {} | {} | {} | {} | {}+{} | {} | {} |\n",
|
||||
o.task,
|
||||
o.arm,
|
||||
o.result,
|
||||
o.rounds,
|
||||
if o.first_load_fail { "是" } else { "否" },
|
||||
if o.first_contract_diag { "是" } else { "否" },
|
||||
o.ptok,
|
||||
o.ctok,
|
||||
o.queries,
|
||||
o.note.replace('|', "\\|").replace('\n', " ")
|
||||
));
|
||||
}
|
||||
md.push_str("\n报告由 `tests/blind_experiment.rs` 生成;材料冻结于 `tests/blind/`(任务书 + docs-bundle),判据见 `doc/roadmap/perf-validation.md` §2。\n");
|
||||
|
||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("doc/roadmap")
|
||||
.join(format!("blind-experiment-{}-{}.md", chrono_date(), llm.model));
|
||||
std::fs::write(&path, md).unwrap();
|
||||
eprintln!("报告已写入 {}", path.display());
|
||||
}
|
||||
|
||||
fn chrono_date() -> String {
|
||||
if let Ok(d) = std::env::var("MARQDO_EXP_DATE") {
|
||||
return d;
|
||||
}
|
||||
let secs = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_secs();
|
||||
let (y, m, d) = civil_from_days((secs / 86400) as i64);
|
||||
format!("{y:04}-{m:02}-{d:02}")
|
||||
}
|
||||
|
||||
fn civil_from_days(z: i64) -> (i64, u32, u32) {
|
||||
let z = z + 719468;
|
||||
let era = if z >= 0 { z } else { z - 146096 } / 146097;
|
||||
let doe = (z - era * 146097) as u64;
|
||||
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
|
||||
let y = yoe as i64 + era * 400;
|
||||
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
|
||||
let mp = (5 * doy + 2) / 153;
|
||||
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
|
||||
let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32;
|
||||
(if m <= 2 { y + 1 } else { y }, m, d)
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
//! P4 实验 (c) 契约消融([perf-validation.md](../../doc/roadmap/perf-validation.md) §3):
|
||||
//! 契约**只在有契约处**拦截错误值(P1:无契约 = 全动态),且不误报。
|
||||
//!
|
||||
//! 四边界(调用实参 / 返回 / 表绑定 / 字段访问)× 三形态:
|
||||
//! ① 带契约 + 错值 → 必拦(`contract.*`);② 带契约 + 对值 → 零误报(跑通);
|
||||
//! ③ 同程序**删契约表** + 错值 → 零 `contract.*`(动态语义原样——零回归)。
|
||||
|
||||
mod common;
|
||||
|
||||
use common::{diagnose, static_diags, Verify};
|
||||
|
||||
/// (边界, 形态源)——每个边界的三形态共享同一程序骨架,只动契约表与值。
|
||||
struct Boundary {
|
||||
name: &'static str,
|
||||
wrong: &'static str, // ① 带契约 + 错值
|
||||
code: &'static str, // ① 期望拦截码
|
||||
right: &'static str, // ② 带契约 + 对值
|
||||
bare: &'static str, // ③ 无契约 + 错值(同程序删契约表)
|
||||
}
|
||||
|
||||
const ARG_WRONG: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 回显 \"7\"**
|
||||
*结果*
|
||||
|
||||
## 回显
|
||||
|
||||
对于输入变量`n`,原样回显。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | number | 输入值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const ARG_RIGHT: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 回显 3**
|
||||
*结果*
|
||||
|
||||
## 回显
|
||||
|
||||
对于输入变量`n`,原样回显。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | number | 输入值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const ARG_BARE: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 回显 \"7\"**
|
||||
*结果*
|
||||
|
||||
## 回显
|
||||
|
||||
对于输入变量`n`,原样回显。
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const RET_WRONG: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 整 \"hi\"**
|
||||
*结果*
|
||||
|
||||
## 整
|
||||
|
||||
对于输入变量`n`,原样返回。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | any | 输入值 |
|
||||
|
||||
| 返回 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| | number | 固定值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const RET_RIGHT: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 整 7**
|
||||
*结果*
|
||||
|
||||
## 整
|
||||
|
||||
对于输入变量`n`,原样返回。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | any | 输入值 |
|
||||
|
||||
| 返回 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| | number | 固定值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const RET_BARE: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 整 \"hi\"**
|
||||
*结果*
|
||||
|
||||
## 整
|
||||
|
||||
对于输入变量`n`,原样返回。
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
const BIND_WRONG: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| abc | 小明 |
|
||||
|
||||
*[name](`用户`)*
|
||||
";
|
||||
|
||||
const BIND_RIGHT: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| 1 | 小明 |
|
||||
|
||||
*[name](`用户`)*
|
||||
";
|
||||
|
||||
const BIND_BARE: &str = "\
|
||||
# main
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| abc | 小明 |
|
||||
|
||||
*[name](`用户`)*
|
||||
";
|
||||
|
||||
const KEY_WRONG: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| 1 | 小明 |
|
||||
|
||||
*[email](`用户`)*
|
||||
";
|
||||
|
||||
const KEY_RIGHT: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| 1 | 小明 |
|
||||
|
||||
*[name](`用户`)*
|
||||
";
|
||||
|
||||
const KEY_BARE: &str = "\
|
||||
# main
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| 1 | 小明 |
|
||||
|
||||
*[email](`用户`)*
|
||||
";
|
||||
|
||||
fn boundaries() -> Vec<Boundary> {
|
||||
vec![
|
||||
Boundary { name: "call-arg", wrong: ARG_WRONG, code: "contract.arg_mismatch", right: ARG_RIGHT, bare: ARG_BARE },
|
||||
Boundary { name: "return", wrong: RET_WRONG, code: "contract.return_mismatch", right: RET_RIGHT, bare: RET_BARE },
|
||||
Boundary { name: "bind", wrong: BIND_WRONG, code: "contract.field_mismatch", right: BIND_RIGHT, bare: BIND_BARE },
|
||||
Boundary { name: "key-access", wrong: KEY_WRONG, code: "contract.unknown_key", right: KEY_RIGHT, bare: KEY_BARE },
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ablation_interception_rate_100() {
|
||||
let all = boundaries();
|
||||
let total = all.len();
|
||||
let mut caught = 0;
|
||||
for b in &all {
|
||||
let diags = diagnose(Verify::Run(b.code), b.wrong, &format!("abl_wrong_{}", b.name));
|
||||
assert!(
|
||||
diags.iter().any(|v| v["code"].as_str() == Some(b.code)),
|
||||
"[{}] 带契约 + 错值必须被拦截(期望 {}): {:?}",
|
||||
b.name, b.code, diags
|
||||
);
|
||||
caught += 1;
|
||||
}
|
||||
eprintln!("契约拦截率 = {caught}/{total} = 100%");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ablation_false_positive_rate_0() {
|
||||
let all = boundaries();
|
||||
let total = all.len();
|
||||
for b in &all {
|
||||
let run_diags = diagnose(Verify::Run(""), b.right, &format!("abl_right_{}", b.name));
|
||||
assert!(run_diags.is_empty(), "[{}] 对值被误报(运行): {:?}", b.name, run_diags);
|
||||
let static_diags = static_diags(b.right);
|
||||
assert!(
|
||||
static_diags.is_empty(),
|
||||
"[{}] 对值被误报(静态): {:?}",
|
||||
b.name,
|
||||
static_diags
|
||||
);
|
||||
}
|
||||
eprintln!("契约误报率 = 0/{total} = 0%");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ablation_no_contract_zero_regression() {
|
||||
// 无契约路径:同程序删掉契约表 + 错值 ⇒ 零 contract.*(动态语义原样)。
|
||||
// key-access 的动态语义是经典 `missing map key` 运行错——也必须是非契约诊断。
|
||||
let all = boundaries();
|
||||
let total = all.len();
|
||||
for b in &all {
|
||||
let diags = diagnose(Verify::Run(""), b.bare, &format!("abl_bare_{}", b.name));
|
||||
assert!(
|
||||
diags.iter().all(|v| !v["code"].as_str().unwrap_or("").starts_with("contract.")),
|
||||
"[{}] 无契约路径出现契约诊断(P1 违例): {:?}",
|
||||
b.name,
|
||||
diags
|
||||
);
|
||||
if b.name == "key-access" {
|
||||
// 动态基线:经典 missing map key 错误照旧(错误不是被契约吞掉或改写)。
|
||||
assert!(
|
||||
diags.iter().any(|v| v["message"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.contains("missing map key")),
|
||||
"key-access 动态基线应为经典 missing map key: {diags:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
eprintln!("无契约零回归 = {total}/{total}");
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
//! P4 实验 (d) 防漂移演练([perf-validation.md](../../doc/roadmap/perf-validation.md) §4):
|
||||
//! **故意写错契约必报率 = 100%**——契约错了必被机器抓,绝不静默信任契约
|
||||
//! (three-problems.md §2.8「故意写错契约的金样必红」)。
|
||||
//!
|
||||
//! 9 例故意漂移(类型 typo ×2 / 参数名漂移 / 错位表 / 重复表 / 四边界错值)
|
||||
//! 必须全部产出 `contract.*` 诊断(静态或运行时);
|
||||
//! 2 例负对照(被绑定的数据表 / 干净契约)**零误报**。
|
||||
|
||||
mod common;
|
||||
|
||||
use common::{diagnose, static_diags, Verify};
|
||||
|
||||
struct Drill {
|
||||
name: &'static str,
|
||||
src: &'static str,
|
||||
/// true = 运行期诊断;false = 静态(mlsp validate = parse + check)。
|
||||
runtime: bool,
|
||||
code: &'static str,
|
||||
}
|
||||
|
||||
/// 类型 typo(参数表 `nubmer`)→ 静态 `contract.unknown_type`。
|
||||
const SRC_TYPO_PARAM: &str = "\
|
||||
# main
|
||||
|
||||
## 加一
|
||||
|
||||
对于输入变量`n`,执行加一。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | nubmer | 输入值 |
|
||||
|
||||
*n+1*
|
||||
";
|
||||
|
||||
/// 类型 typo(字段表未知对象名 `巨龙`)→ 静态 `contract.unknown_type`。
|
||||
const SRC_TYPO_FIELD: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | 巨龙 | |
|
||||
|
||||
`用户` =
|
||||
| id |
|
||||
|------|
|
||||
| 1 |
|
||||
|
||||
*[id](`用户`)*
|
||||
";
|
||||
|
||||
/// 契约参数名 ≠ 升参名(`nn` vs `n`)→ 静态 `contract.param_drift`。
|
||||
const SRC_PARAM_DRIFT: &str = "\
|
||||
# main
|
||||
|
||||
## 加一
|
||||
|
||||
对于输入变量`n`,执行加一。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| nn | number | 输入值 |
|
||||
|
||||
*n+1*
|
||||
";
|
||||
|
||||
/// 错位契约表(出现在可执行语句之后)→ 静态 `contract.misplaced`。
|
||||
const SRC_MISPLACED: &str = "\
|
||||
# main
|
||||
|
||||
**x=1**
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | number | 输入值 |
|
||||
|
||||
*x*
|
||||
";
|
||||
|
||||
/// 重复契约表(两份参数表打架)→ 装载期 `contract.duplicate`。
|
||||
const SRC_DUPLICATE: &str = "\
|
||||
# main
|
||||
|
||||
## 加一
|
||||
|
||||
对于输入变量`n`,执行加一。
|
||||
|
||||
| 参数 | 类型 |
|
||||
|------|------|
|
||||
| n | number |
|
||||
|
||||
| 参数 | 类型 |
|
||||
|------|------|
|
||||
| n | number |
|
||||
|
||||
*n+1*
|
||||
";
|
||||
|
||||
/// 实参越约(参数契约 number,实参 text)→ 运行期 `contract.arg_mismatch`。
|
||||
const SRC_ARG: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 回显 \"7\"**
|
||||
*结果*
|
||||
|
||||
## 回显
|
||||
|
||||
对于输入变量`n`,原样回显。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | number | 输入值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
/// 返回越约(返回契约 number,实际返回 text)→ 运行期 `contract.return_mismatch`。
|
||||
const SRC_RETURN: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 整 \"hi\"**
|
||||
*结果*
|
||||
|
||||
## 整
|
||||
|
||||
对于输入变量`n`,原样返回。
|
||||
|
||||
| 返回 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| | number | 固定值 |
|
||||
|
||||
*n*
|
||||
";
|
||||
|
||||
/// 表绑定越约(字段 id 契约 number,单元格 abc)→ 运行期 `contract.field_mismatch`。
|
||||
const SRC_FIELD: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| abc | 小明 |
|
||||
|
||||
*[name](`用户`)*
|
||||
";
|
||||
|
||||
/// 缺键访问(契约有 email,集合没有)→ 运行期 `contract.unknown_key`。
|
||||
const SRC_UNKNOWN_KEY: &str = "\
|
||||
# main
|
||||
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
`用户` =
|
||||
| id | name |
|
||||
|------|------|
|
||||
| 1 | 小明 |
|
||||
|
||||
*[email](`用户`)*
|
||||
";
|
||||
|
||||
/// 负对照 1:**被绑定**的 `字段|类型|可空` 数据表(web_db_migration 式)= 数据,不是契约。
|
||||
const SRC_BOUND_DATA: &str = "\
|
||||
# main
|
||||
|
||||
`列说明` =
|
||||
| 字段 | 类型 | 可空 |
|
||||
|------|------|------|
|
||||
| id | number | |
|
||||
| name | text | 是 |
|
||||
|
||||
*[字段]([1](`列说明`))*
|
||||
";
|
||||
|
||||
/// 负对照 2:干净契约 + 对值。
|
||||
const SRC_OK: &str = "\
|
||||
# main
|
||||
|
||||
**结果 = > 加一 3**
|
||||
*结果*
|
||||
|
||||
## 加一
|
||||
|
||||
对于输入变量`n`,期望是数字。
|
||||
|
||||
| 参数 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| n | number | 输入值 |
|
||||
|
||||
| 返回 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| | number | 加一结果 |
|
||||
|
||||
*n+1*
|
||||
";
|
||||
|
||||
fn drills() -> Vec<Drill> {
|
||||
vec![
|
||||
Drill { name: "type-typo-param", src: SRC_TYPO_PARAM, runtime: false, code: "contract.unknown_type" },
|
||||
Drill { name: "type-typo-field", src: SRC_TYPO_FIELD, runtime: false, code: "contract.unknown_type" },
|
||||
Drill { name: "param-name-drift", src: SRC_PARAM_DRIFT, runtime: false, code: "contract.param_drift" },
|
||||
Drill { name: "misplaced-table", src: SRC_MISPLACED, runtime: false, code: "contract.misplaced" },
|
||||
Drill { name: "duplicate-table", src: SRC_DUPLICATE, runtime: true, code: "contract.duplicate" },
|
||||
Drill { name: "arg-mismatch", src: SRC_ARG, runtime: true, code: "contract.arg_mismatch" },
|
||||
Drill { name: "return-mismatch", src: SRC_RETURN, runtime: true, code: "contract.return_mismatch" },
|
||||
Drill { name: "field-mismatch", src: SRC_FIELD, runtime: true, code: "contract.field_mismatch" },
|
||||
Drill { name: "unknown-key", src: SRC_UNKNOWN_KEY, runtime: true, code: "contract.unknown_key" },
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn drift_cases_all_reported() {
|
||||
let all = drills();
|
||||
let total = all.len();
|
||||
let mut caught = 0;
|
||||
for d in &all {
|
||||
let diags = if d.runtime {
|
||||
diagnose(Verify::Run(d.code), d.src, &format!("drift_{}", d.name))
|
||||
} else {
|
||||
static_diags(d.src)
|
||||
};
|
||||
let hit = diags.iter().any(|v| v["code"].as_str() == Some(d.code));
|
||||
assert!(
|
||||
hit,
|
||||
"[{}] 故意写错的契约必须被诊断(期望 {}): {:?}",
|
||||
d.name, d.code, diags
|
||||
);
|
||||
caught += 1;
|
||||
}
|
||||
eprintln!("防漂移必报率 = {caught}/{total} = 100%");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn negative_controls_zero_false_positives() {
|
||||
// 1. 被绑定的 `字段|类型|可空` 数据表 = 数据(web_db_migration 式)——绝不能被当契约吃掉。
|
||||
let diags = diagnose(Verify::Run(""), SRC_BOUND_DATA, "drill_neg_bound");
|
||||
assert!(
|
||||
diags.iter().all(|v| !v["code"].as_str().unwrap_or("").starts_with("contract.")),
|
||||
"绑定数据表被误当契约: {diags:?}"
|
||||
);
|
||||
// 2. 干净契约 + 对值 ⇒ 静态零诊断、运行零诊断。
|
||||
let diags = static_diags(SRC_OK);
|
||||
assert!(diags.is_empty(), "干净契约被误报(静态): {diags:?}");
|
||||
let diags = diagnose(Verify::Run(""), SRC_OK, "drill_neg_ok");
|
||||
assert!(diags.is_empty(), "干净契约被误报(运行): {diags:?}");
|
||||
eprintln!("负对照误报 = 0/2 = 0%");
|
||||
}
|
||||
@@ -33,6 +33,34 @@ fn syntax_unknown_query_structured_error() {
|
||||
assert_eq!(resp["error"]["code"], "mlsp.no_match");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syntax_cards_carry_rule_text_self_sufficient() {
|
||||
// 「AI 不必学语法」的前提:卡片自带语义/戒律/样例原文(不必回读文档),
|
||||
// 且空查询列出全部核心构造(three-problems.md §3.2「单构造戒律」)。
|
||||
let resp = call(r#"{"id":9,"method":"syntax","params":{}}"#);
|
||||
assert_eq!(resp["ok"], true, "{resp}");
|
||||
let cards = resp["result"]["cards"].as_array().unwrap();
|
||||
assert_eq!(
|
||||
cards.len(),
|
||||
marqdo::parse::CORE_CONSTRUCTS.len(),
|
||||
"空查询应列出全部核心构造: {resp}"
|
||||
);
|
||||
for c in cards {
|
||||
assert!(
|
||||
c["rule"].as_str().unwrap_or("").len() > 2,
|
||||
"卡片缺戒律原文(不可自持): {c}"
|
||||
);
|
||||
assert!(
|
||||
c["doc_quote"].as_str().unwrap_or("").starts_with('|'),
|
||||
"卡片缺行原文: {c}"
|
||||
);
|
||||
assert!(
|
||||
!c["semantics"].as_str().unwrap_or("").is_empty(),
|
||||
"卡片缺语义: {c}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn locate_symbol_returns_span_and_contract() {
|
||||
let src = "# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n*n+1*\n";
|
||||
|
||||
Reference in new issue
Block a user