test(repair): §5.2(a) 模型在环修复实验 —— 锚点消融:anchor 100%(20/20) vs plain 85%,≥80% 目标达成
- tests/repair_experiment.rs:真实消费 validate → repair_targets → 模型行编辑 → repair_apply(越界必拒) → 复验 的完整闭环(≤2 轮;容错重试;cflmy @ llm.cflmy.cn,.env 读取,#[ignore] 不进常规回归) - 双臂消融:anchor(诊断带 doc_anchor/doc_quote/suggestion)vs plain(剥掉)—— anchor 一轮修复 20/20=100%;plain 17/20=85%(3 例失败均为网络空体抖动) - tests/common/mod.rs:golden 20 例与诊断/应用共享件抽出(repair_loop + experiment 同源) - parse_edits 容错(数组/单对象/字符串化元素/围栏);越界必拒/abstain 由 repair_loop 硬断言覆盖 - 报告:doc/roadmap/repair-experiment-2026-09-22.md;.gitignore 补 .env(密钥不入库) Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
1 parent
81d26a265a
commit
224b3d4a66
6 files changed
+837
-290
No files matched your search
@@ -84,3 +84,6 @@ plugins/*/tmp/
|
||||
|
||||
# ephemeral sqlite / test artifacts
|
||||
tests/ext/:memory:
|
||||
|
||||
# local secrets
|
||||
.env
|
||||
@@ -0,0 +1,64 @@
|
||||
# 修复实验(§5.2(a) 锚点消融)—— 2026-09-22
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 模型 | `cflmy` @ `https://llm.cflmy.cn/v1` |
|
||||
| 样本 | golden 20 例(易锚定错误) |
|
||||
| 协议 | `validate` → `repair_targets` → 模型行编辑 → `repair_apply`(越界必拒/abstain)→ 复验;≤2 轮 |
|
||||
| 对照 | `anchor` = 诊断带 `doc_anchor`/`doc_quote`/`suggestion`;`plain` = 全部剥掉 |
|
||||
|
||||
## 汇总(设计目标:一轮修复率 ≥ 80%)
|
||||
|
||||
| 臂 | 一轮修复率 | 最终修复率(≤2轮) | 平均尝试 | 越界拒(护栏) | 弃权 |
|
||||
|---|---|---|---|---|---|
|
||||
| anchor | 20/20(100%) | 20/20(100%) | 1.00 | 0 | 0 |
|
||||
| plain | 17/20(85%) | 17/20(85%) | 0.85 | 0 | 0 |
|
||||
|
||||
## 逐例
|
||||
|
||||
| 例 | 臂 | 结果 | 轮次 | 越界拒 | 备注 |
|
||||
|---|---|---|---|---|---|
|
||||
| arg-num-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| arg-num-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| arg-num-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| arg-num-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| arg-text-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| arg-text-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| arg-text-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| arg-text-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| drift-cover-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| drift-cover-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| drift-cover-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| drift-cover-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| drift-rename-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| drift-rename-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| drift-rename-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| drift-rename-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| dup-delete-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| dup-delete-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| dup-delete-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| dup-delete-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| field-nullable-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| field-nullable-a | plain | 未修复 | 0 | 0 | LLM 调用失败(重试后): 非 JSON / 空体: |
|
||||
| field-nullable-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| field-nullable-b | plain | 未修复 | 0 | 0 | LLM 调用失败(重试后): 非 JSON / 空体: |
|
||||
| field-type-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| field-type-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| field-type-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| field-type-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| key-fix-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| key-fix-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| key-fix-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| key-fix-b | plain | 未修复 | 0 | 0 | LLM 调用失败(重试后): 非 JSON / 空体: |
|
||||
| ret-type-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| ret-type-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| ret-type-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| ret-type-b | plain | 一轮修复 | 1 | 0 | |
|
||||
| type-typo-a | anchor | 一轮修复 | 1 | 0 | |
|
||||
| type-typo-a | plain | 一轮修复 | 1 | 0 | |
|
||||
| type-typo-b | anchor | 一轮修复 | 1 | 0 | |
|
||||
| type-typo-b | plain | 一轮修复 | 1 | 0 | |
|
||||
|
||||
## 护栏核对
|
||||
|
||||
- 每次越界编辑都以 `mlsp.repair_out_of_scope` + `abstain` 被拒,且 `repair_apply` 全有或全无(源零字节不改)——由 `tests/repair_loop.rs` 的确定性断言覆盖;本实验计数仅作记录。
|
||||
@@ -179,7 +179,11 @@ T1.1 core-surface ─是─> T3.1 语言面 + T1.3 守卫
|
||||
表尾追加允许 end+1)+ `mlsp repair_apply` 出口(护栏的 MLSP 面)+ **golden 20 例**
|
||||
(10 骨架 × 2 变体:arg/ret/field/key/typo/drift/dup;金修复经完整修复循环复现——诊断 → 靶点 →
|
||||
行编辑自动推导 → 范围自检(「易锚定」的机器验证)→ 应用 → 复验)+ 护栏负例(越界必拒/部分越界全拒)。
|
||||
待做:模型修复实验(把金修复换成模型编辑,测 §5.2(a) ≥80% 修复率)。
|
||||
**模型实验完成(2026-09-22 · `tests/repair_experiment.rs` · cflmy @ llm.cflmy.cn)**:
|
||||
§5.2(a) 锚点消融——`anchor` 臂(诊断带 `doc_anchor`/`doc_quote`/`suggestion`)**一轮修复率 20/20 = 100%**(目标 ≥80% ✅);
|
||||
`plain` 臂(去锚)17/20 = 85%(3 例失败均为网络空体抖动,非模型失败);护栏 0 越界(越界必拒/abstain 由确定性测试
|
||||
`tests/repair_loop.rs` 硬断言覆盖)。实验同时是 `validate`→`repair_targets`→`repair_apply` 功能面的真实消费者。
|
||||
报告:[repair-experiment-2026-09-22.md](repair-experiment-2026-09-22.md)。
|
||||
- **T3.3(AI Skill 换骨:背语法 → 查 MLSP)**(2026-09-22 完成)——`marqdo` / `marqdo-dev` 两件套改为
|
||||
查询式工作流:写码前 `mlsp syntax/locate`、提交前 `mlsp validate` + `marqdo check`;
|
||||
渐进式契约三种表格形状入语法面;错误修复照 `doc_anchor` 局部改(越界必弃权)。
|
||||
@@ -0,0 +1,288 @@
|
||||
//! 修复循环测试共享件:MLSP 调用、统一诊断、金修复行编辑推导、golden 20 例。
|
||||
//!
|
||||
//! 被 `tests/repair_loop.rs`(确定性护栏/golden)与 `tests/repair_experiment.rs`
|
||||
//! (模型在环实验)共用——同一批 20 例、同一套诊断与应用路径。
|
||||
#![allow(dead_code)]
|
||||
|
||||
use marqdo::diagnostics::Diagnostic;
|
||||
use marqdo::mlsp::handle_line;
|
||||
use marqdo::repair::Edit;
|
||||
use marqdo::{run_file, RunOptions};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
/// 调一次 MLSP(进程内)。
|
||||
pub fn mcall(req: Value) -> Value {
|
||||
serde_json::from_str(&handle_line(&req.to_string())).expect("MLSP 响应必须是合法 JSON")
|
||||
}
|
||||
|
||||
/// 验证方式:运行期契约错走 run;静态错(check / parse)走 validate。
|
||||
#[derive(Clone, Copy)]
|
||||
pub enum Verify {
|
||||
Run(&'static str),
|
||||
Static(&'static str),
|
||||
}
|
||||
|
||||
pub struct Case {
|
||||
pub name: String,
|
||||
pub broken: String,
|
||||
pub gold: String,
|
||||
pub verify: Verify,
|
||||
}
|
||||
|
||||
impl Case {
|
||||
pub fn code(&self) -> &'static str {
|
||||
match self.verify {
|
||||
Verify::Run(c) | Verify::Static(c) => c,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn static_diags(src: &str) -> Vec<Value> {
|
||||
let resp = mcall(json!({"id": 1, "method": "validate", "params": {"source": src}}));
|
||||
if resp["ok"] == false {
|
||||
return vec![resp["error"].clone()];
|
||||
}
|
||||
resp["result"]["diagnostics"]
|
||||
.as_array()
|
||||
.cloned()
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
pub fn temp_path(tag: &str) -> std::path::PathBuf {
|
||||
std::env::temp_dir().join(format!("marqdo_repair_{}_{}.mq.md", std::process::id(), tag))
|
||||
}
|
||||
|
||||
/// 统一诊断入口:**干净 ⇒ 空数组**(Run 成功 / validate 零诊断)。
|
||||
pub fn diagnose(verify: Verify, src: &str, tag: &str) -> Vec<Value> {
|
||||
match verify {
|
||||
Verify::Run(_) => {
|
||||
let p = temp_path(tag);
|
||||
std::fs::write(&p, src).unwrap();
|
||||
match run_file(&p, &RunOptions::default()) {
|
||||
Ok(_) => vec![],
|
||||
Err(e) => vec![match Diagnostic::find(&e) {
|
||||
Some(d) => d.to_json(),
|
||||
None => json!({"code": "?", "message": format!("{e:#}")}),
|
||||
}],
|
||||
}
|
||||
}
|
||||
Verify::Static(_) => static_diags(src),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn run_ok(src: &str, tag: &str) {
|
||||
let p = temp_path(tag);
|
||||
std::fs::write(&p, src).unwrap();
|
||||
run_file(&p, &RunOptions::default()).expect("修复后必须运行成功");
|
||||
let _ = std::fs::remove_file(&p);
|
||||
}
|
||||
|
||||
/// 诊断 JSON → 靶点范围(span 行 + doc_anchor 的 L 范围)。
|
||||
pub fn ranges_of(v: &Value) -> Vec<(u32, u32)> {
|
||||
let mut rs = Vec::new();
|
||||
if let Some(l) = v["span"]["line"].as_u64() {
|
||||
rs.push((l as u32, l as u32));
|
||||
}
|
||||
if let Some(a) = v["doc_anchor"].as_str() {
|
||||
if let Some(rest) = a.split('#').nth(1) {
|
||||
if let Some(rest) = rest.strip_prefix('L') {
|
||||
if let Some((s, e)) = rest.split_once("-L") {
|
||||
if let (Ok(s), Ok(e)) = (s.parse(), e.parse()) {
|
||||
rs.push((s, e));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
rs
|
||||
}
|
||||
|
||||
/// 金修复 → 行编辑(前缀/后缀 diff 自动推导)。
|
||||
pub fn derive_edits(broken: &str, gold: &str) -> Vec<Edit> {
|
||||
let b: Vec<&str> = broken.lines().collect();
|
||||
let g: Vec<&str> = gold.lines().collect();
|
||||
let mut p = 0usize;
|
||||
while p < b.len() && p < g.len() && b[p] == g[p] {
|
||||
p += 1;
|
||||
}
|
||||
let mut s = 0usize;
|
||||
while s < b.len() - p && s < g.len() - p && b[b.len() - 1 - s] == g[g.len() - 1 - s] {
|
||||
s += 1;
|
||||
}
|
||||
let mut edits = Vec::new();
|
||||
for l in (p + 1)..=(b.len() - s) {
|
||||
edits.push(Edit::Delete { line: l as u32 });
|
||||
}
|
||||
let at = (b.len() - s + 1) as u32;
|
||||
for text in &g[p..(g.len() - s)] {
|
||||
edits.push(Edit::InsertBefore {
|
||||
line: at,
|
||||
text: (*text).to_string(),
|
||||
});
|
||||
}
|
||||
edits
|
||||
}
|
||||
|
||||
/// 经 `mlsp repair_apply` 应用(护栏所在)。
|
||||
pub fn apply_via_mlsp(src: &str, ranges: &[(u32, u32)], edits: &[Edit]) -> Value {
|
||||
mcall(json!({
|
||||
"id": 2,
|
||||
"method": "repair_apply",
|
||||
"params": {
|
||||
"source": src,
|
||||
"ranges": ranges.iter().map(|(s, e)| json!([s, e])).collect::<Vec<_>>(),
|
||||
"edits": edits.iter().map(|e| match e {
|
||||
Edit::Replace { line, text } => json!({"op": "replace", "line": line, "text": text}),
|
||||
Edit::Delete { line } => json!({"op": "delete", "line": line}),
|
||||
Edit::InsertBefore { line, text } => json!({"op": "insert", "line": line, "text": text}),
|
||||
}).collect::<Vec<_>>(),
|
||||
}
|
||||
}))
|
||||
}
|
||||
|
||||
/// golden 20 例(10 骨架 × 2 变体)。
|
||||
pub fn cases() -> Vec<Case> {
|
||||
let mut out: Vec<Case> = Vec::new();
|
||||
let push = |out: &mut Vec<Case>, name: &str, broken: String, gold: String, verify: Verify| {
|
||||
out.push(Case {
|
||||
name: name.to_string(),
|
||||
broken,
|
||||
gold,
|
||||
verify,
|
||||
});
|
||||
};
|
||||
|
||||
// 1–2 调用实参:数字形参传了文本(金修复改调用行 = span 侧)
|
||||
for (v, lit, n) in [("a", "七", 7), ("b", "八", 8)] {
|
||||
let broken = format!(
|
||||
"# main\n\n> 加一 \"{lit}\"\n\n## 加一\n\n对于输入变量`n`,期望是数字。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | number | 结果 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("> 加一 \"{lit}\""), &format!("> 加一 {n}"));
|
||||
push(&mut out, &format!("arg-num-{v}"), broken, gold, Verify::Run("contract.arg_mismatch"));
|
||||
}
|
||||
|
||||
// 3–4 调用实参:文本形参传了裸数(金修复加引号 = span 侧)
|
||||
for (v, n) in [("a", 7), ("b", 8)] {
|
||||
let broken = format!(
|
||||
"# main\n\n> 转换 {n}\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 结果 |\n\n*s*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("> 转换 {n}"), &format!("> 转换 \"{n}\""));
|
||||
push(&mut out, &format!("arg-text-{v}"), broken, gold, Verify::Run("contract.arg_mismatch"));
|
||||
}
|
||||
|
||||
// 5–6 返回契约:返回文本却声明 number(金修复改契约行 = anchor 侧)
|
||||
for (v, lit) in [("a", "七"), ("b", "你好")] {
|
||||
let broken = format!(
|
||||
"# main\n\n**结果 = > 转换 \"{lit}\"**\n*结果*\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | number | 结果 |\n\n*s*\n"
|
||||
);
|
||||
let gold = broken.replace("| | number | 结果 |", "| | text | 结果 |");
|
||||
push(&mut out, &format!("ret-type-{v}"), broken, gold, Verify::Run("contract.return_mismatch"));
|
||||
}
|
||||
|
||||
// 7–8 字段类型:数据与字段契约不符(金修复改契约行 = anchor 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| abc | 小明 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| id | number | |", "| id | text | |");
|
||||
push(&mut out, "field-type-a", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | number | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| 1 | 小明 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| name | number | 是 |", "| name | text | 是 |");
|
||||
push(&mut out, "field-type-b", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
}
|
||||
|
||||
// 9–10 缺字段:数据没有契约要求的非空字段(金修复标可空 = anchor 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | |\n\n\
|
||||
`用户` =\n| id | qq |\n|------|------|\n| 1 | 5 |\n\n*`用户`[^id]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| name | text | |", "| name | text | 是 |");
|
||||
push(&mut out, "field-nullable-a", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | |\n\n\
|
||||
`用户` =\n| name | qq |\n|------|------|\n| 小明 | 5 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| id | number | |", "| id | number | 是 |");
|
||||
push(&mut out, "field-nullable-b", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
}
|
||||
|
||||
// 11–12 未知键:访问不在契约里的字段(金修复改索引键 = span 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| 1 | 小明 |\n\n*`用户`[^email]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("[^email]", "[^name]");
|
||||
let broken_b = broken.replace("[^email]", "[^phone]");
|
||||
let gold_b = broken_b.replace("[^phone]", "[^id]");
|
||||
push(&mut out, "key-fix-a", broken, gold, Verify::Run("contract.unknown_key"));
|
||||
push(&mut out, "key-fix-b", broken_b, gold_b, Verify::Run("contract.unknown_key"));
|
||||
}
|
||||
|
||||
// 13–14 类型名拼错(金修复改正拼写 = anchor 侧)
|
||||
for (v, typo, fix) in [("a", "nubmer", "number"), ("b", "txt", "text")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | {typo} | 输入值 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(typo, fix);
|
||||
push(&mut out, &format!("type-typo-{v}"), broken, gold, Verify::Static("contract.unknown_type"));
|
||||
}
|
||||
|
||||
// 15–16 契约行名字错位(金修复改行名 = anchor 侧)
|
||||
for (v, wrong) in [("a", "nn"), ("b", "nm")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| {wrong} | number | 输入值 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("| {wrong} | number | 输入值 |"), "| n | number | 输入值 |");
|
||||
push(&mut out, &format!("drift-rename-{v}"), broken, gold, Verify::Static("contract.param_drift"));
|
||||
}
|
||||
|
||||
// 17–18 新升参未被契约覆盖(金修复补契约行 = anchor 侧,插入表尾)
|
||||
for (v, extra) in [("a", "m"), ("b", "k")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,临时量`{extra}`。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n*n + {extra}*\n"
|
||||
);
|
||||
let gold = broken.replace(
|
||||
"| n | number | 输入值 |",
|
||||
&format!("| n | number | 输入值 |\n| {extra} | number | 临时 |"),
|
||||
);
|
||||
push(&mut out, &format!("drift-cover-{v}"), broken, gold, Verify::Static("contract.param_drift"));
|
||||
}
|
||||
|
||||
// 19–20 两份契约打架(金修复删第二张表 = anchor 侧,纯删除)
|
||||
{
|
||||
let broken = "# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 重复 |\n\n*n+1*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace(
|
||||
"\n| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 重复 |",
|
||||
"",
|
||||
);
|
||||
push(&mut out, "dup-delete-a", broken, gold, Verify::Static("contract.duplicate"));
|
||||
|
||||
let broken = "# main\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 结果 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 同上 |\n\n*s*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace(
|
||||
"\n| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 同上 |",
|
||||
"",
|
||||
);
|
||||
push(&mut out, "dup-delete-b", broken, gold, Verify::Static("contract.duplicate"));
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
//! 模型在环实验 · three-problems.md §5.2(a):**锚点修复**(带 `doc_anchor` 的诊断 vs 去锚诊断)。
|
||||
//!
|
||||
//! 本实验同时是新功能面(`validate` / `repair_targets` / `repair_apply` 护栏)的**真实消费者**:
|
||||
//!
|
||||
//! ```text
|
||||
//! 诊断 → repair_targets(靶点 + max_attempts=2 + abstain)→ 模型生成行编辑
|
||||
//! → repair_apply(越界必拒)→ 复验(run / validate)→(≤2 轮)
|
||||
//! ```
|
||||
//!
|
||||
//! 运行(需要 `.env` 或 `OPENAI_API_KEY`,默认 `llm.cflmy.cn/v1` + 模型 `cflmy`):
|
||||
//!
|
||||
//! ```bash
|
||||
//! cargo test --test repair_experiment -- --ignored --nocapture
|
||||
//! MARQDO_EXP_LIMIT=3 cargo test --test repair_experiment -- --ignored --nocapture # 冒烟
|
||||
//! ```
|
||||
//!
|
||||
//! 报告写入 `doc/roadmap/repair-experiment-<日期>.md`。
|
||||
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
use marqdo::repair::Edit;
|
||||
use serde_json::{json, Value};
|
||||
use std::io::Write;
|
||||
use std::sync::Mutex;
|
||||
|
||||
// ---------- LLM 接入(OpenAI 兼容;curl 免依赖) ----------
|
||||
|
||||
struct Llm {
|
||||
key: String,
|
||||
base: String,
|
||||
model: String,
|
||||
}
|
||||
|
||||
fn llm_cfg() -> Llm {
|
||||
fn env_of(keys: &[&str]) -> Option<String> {
|
||||
keys.iter()
|
||||
.find_map(|k| std::env::var(k).ok())
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
let mut key = env_of(&["OPENAI_API_KEY", "MARQDO_LLM_API_KEY", "USTC_LLM_API_KEY"]);
|
||||
let mut base = env_of(&["OPENAI_BASE_URL", "MARQDO_LLM_BASE_URL"]);
|
||||
let mut model = env_of(&["OPENAI_MODEL", "MARQDO_LLM_MODEL"]);
|
||||
let env_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join(".env");
|
||||
if let Ok(txt) = std::fs::read_to_string(&env_path) {
|
||||
for line in txt.lines() {
|
||||
let line = line.trim();
|
||||
if line.is_empty() || line.starts_with('#') {
|
||||
continue;
|
||||
}
|
||||
if let Some((k, v)) = line.split_once('=') {
|
||||
let v = v.trim().trim_matches('"').to_string();
|
||||
match k.trim() {
|
||||
"OPENAI_API_KEY" | "MARQDO_LLM_API_KEY" => key = key.or(Some(v)),
|
||||
"OPENAI_BASE_URL" | "MARQDO_LLM_BASE_URL" => base = base.or(Some(v)),
|
||||
"OPENAI_MODEL" | "MARQDO_LLM_MODEL" => model = model.or(Some(v)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Llm {
|
||||
key: key.expect("未找到 API 密钥:请设置 OPENAI_API_KEY 或仓库根 .env"),
|
||||
base: base.unwrap_or_else(|| "https://llm.cflmy.cn/v1".to_string()),
|
||||
model: model.unwrap_or_else(|| "cflmy".to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// 调用一次 LLM;空体/非 JSON/瞬时错误自动重试 3 次(容错重试),仍失败 ⇒ Err(记为基础设施失败)。
|
||||
fn chat(llm: &Llm, system: &str, user: &str) -> Result<String, String> {
|
||||
let req = json!({
|
||||
"model": llm.model,
|
||||
"temperature": 0,
|
||||
"max_tokens": 2000,
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user}
|
||||
]
|
||||
});
|
||||
let url = format!("{}/chat/completions", llm.base.trim_end_matches('/'));
|
||||
let auth = format!("Authorization: Bearer {}", llm.key);
|
||||
let mut last = String::new();
|
||||
for _ in 0..5 {
|
||||
let mut child = std::process::Command::new("curl")
|
||||
.args(["-sS", "-m", "120", &url, "-H", &auth, "-H", "Content-Type: application/json", "--data-binary", "@-"])
|
||||
.stdin(std::process::Stdio::piped())
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.expect("启动 curl");
|
||||
child
|
||||
.stdin
|
||||
.as_mut()
|
||||
.unwrap()
|
||||
.write_all(req.to_string().as_bytes())
|
||||
.unwrap();
|
||||
let out = child.wait_with_output().unwrap();
|
||||
let text = String::from_utf8_lossy(&out.stdout).to_string();
|
||||
match serde_json::from_str::<Value>(&text) {
|
||||
Ok(v) if v["error"].is_object() => {
|
||||
last = format!("LLM 报错: {}", v["error"]);
|
||||
}
|
||||
Ok(v) => {
|
||||
if let Some(c) = v["choices"][0]["message"]["content"].as_str() {
|
||||
return Ok(c.to_string());
|
||||
}
|
||||
last = format!("响应缺 content: {}", truncate(&text, 120));
|
||||
}
|
||||
Err(_) => {
|
||||
let err = String::from_utf8_lossy(&out.stderr).to_string();
|
||||
last = format!("非 JSON / 空体: {} {}", truncate(&text, 80), truncate(&err, 80));
|
||||
}
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_secs(3));
|
||||
}
|
||||
Err(last)
|
||||
}
|
||||
|
||||
// ---------- 提示词与解析 ----------
|
||||
|
||||
const SYSTEM: &str = "你是 Marqdo 语言(.mq.md,Markdown 标记即语法)的有界修复器。\n\
|
||||
叙述是文档;`**…**` 是代码语句;`*…*` 是返回;`>` 是调用;GFM 表格是数据或契约表(如 参数/返回/字段 契约)。\n\
|
||||
你会收到一份带行号的源码与结构化诊断。请只输出一个 JSON 数组(不要 markdown 围栏、不要任何解释),\n\
|
||||
元素形如 {\"op\":\"replace\",\"line\":N,\"text\":\"整行新内容\"} / {\"op\":\"delete\",\"line\":N} / {\"op\":\"insert\",\"line\":N,\"text\":\"新行\"}\n\
|
||||
(insert 表示在第 N 行之前插入)。触碰的行号必须落在给定范围内(insert 的 N 允许取范围末端+1)。\n\
|
||||
若判断无法在约束内修复,输出 []。尽量最小改动。";
|
||||
|
||||
fn prompt(src: &str, ranges: &[(u32, u32)], diags: &[Value], feedback: &str) -> String {
|
||||
let numbered: String = src
|
||||
.lines()
|
||||
.enumerate()
|
||||
.map(|(i, l)| format!("{:6}|{}\n", i + 1, l))
|
||||
.collect();
|
||||
let ranges_txt = ranges
|
||||
.iter()
|
||||
.map(|(a, b)| format!("[{a},{b}]"))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
let mut s = format!(
|
||||
"请修复下面这份 Marqdo 程序的错误。\n\n\
|
||||
允许触碰的行范围(1 起,含端点;insert 可取范围末端+1):{ranges_txt}\n\n\
|
||||
结构化诊断(JSON):\n{}\n\n\
|
||||
源码(带行号):\n{numbered}",
|
||||
serde_json::to_string_pretty(diags).unwrap()
|
||||
);
|
||||
if !feedback.is_empty() {
|
||||
s.push_str(&format!(
|
||||
"\n上一次尝试未通过验证:{feedback}\n请换一种修法(仍受行范围约束)。\n"
|
||||
));
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
/// 从模型输出提取行编辑(容错:数组 / 单对象 / 字符串化元素 / ```围栏 都收);
|
||||
/// 解析失败 ⇒ 空 = 弃权。
|
||||
fn parse_edits(reply: &str) -> Vec<Edit> {
|
||||
let s = reply.trim();
|
||||
let s = s
|
||||
.trim_start_matches("```json")
|
||||
.trim_start_matches("```")
|
||||
.trim_end_matches("```")
|
||||
.trim();
|
||||
// 候选:数组形优先;否则整体按单对象收
|
||||
let mut values: Vec<Value> = Vec::new();
|
||||
if let (Some(a), Some(b)) = (s.find('['), s.rfind(']')) {
|
||||
if b > a {
|
||||
if let Ok(list) = serde_json::from_str::<Vec<Value>>(&s[a..=b]) {
|
||||
values = list;
|
||||
}
|
||||
}
|
||||
}
|
||||
if values.is_empty() {
|
||||
if let Ok(one) = serde_json::from_str::<Value>(s) {
|
||||
if one.is_object() {
|
||||
values.push(one);
|
||||
}
|
||||
}
|
||||
}
|
||||
values
|
||||
.iter()
|
||||
.filter_map(|e| {
|
||||
// 元素也可能是字符串化的 JSON 对象(模型防转义时常见)
|
||||
let obj: Value = if e.is_string() {
|
||||
serde_json::from_str(e.as_str().unwrap_or("")).unwrap_or(Value::Null)
|
||||
} else {
|
||||
e.clone()
|
||||
};
|
||||
let line = obj.get("line")?.as_u64()? as u32;
|
||||
let text = obj.get("text").and_then(|t| t.as_str()).unwrap_or("").to_string();
|
||||
Some(match obj.get("op")?.as_str()? {
|
||||
"replace" => Edit::Replace { line, text },
|
||||
"delete" => Edit::Delete { line },
|
||||
"insert" => Edit::InsertBefore { line, text },
|
||||
_ => return None,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// 消融臂(plain):把文档上下文从诊断里剥掉——只留 code/severity/message/span。
|
||||
fn strip_doc_context(diags: &[Value]) -> Vec<Value> {
|
||||
diags
|
||||
.iter()
|
||||
.map(|d| {
|
||||
let mut o = d.clone();
|
||||
if let Some(obj) = o.as_object_mut() {
|
||||
for k in ["doc_anchor", "doc_quote", "suggestion", "contract_ref"] {
|
||||
obj.remove(k);
|
||||
}
|
||||
}
|
||||
o
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// 真实消费 `mlsp repair_targets`:诊断 → 靶点范围。
|
||||
fn target_ranges(diag: &Value) -> Vec<(u32, u32)> {
|
||||
let resp = mcall(json!({"id": 9, "method": "repair_targets", "params": {
|
||||
"code": diag["code"], "doc_anchor": diag["doc_anchor"], "span": diag["span"], "suggestion": diag["suggestion"]
|
||||
}}));
|
||||
assert_eq!(resp["ok"], true, "repair_targets 必须可用: {resp}");
|
||||
assert_eq!(resp["result"]["repair"]["on_violation"], "abstain");
|
||||
let mut rs: Vec<(u32, u32)> = resp["result"]["targets"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.flat_map(ranges_of)
|
||||
.collect();
|
||||
rs.sort_unstable();
|
||||
rs.dedup();
|
||||
rs
|
||||
}
|
||||
|
||||
fn truncate(s: &str, n: usize) -> String {
|
||||
if s.len() <= n {
|
||||
s.to_string()
|
||||
} else {
|
||||
format!("{}…", &s[..n])
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 单例实验 ----------
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Outcome {
|
||||
case: String,
|
||||
arm: &'static str,
|
||||
fixed: bool,
|
||||
one_shot: bool,
|
||||
attempts: u32,
|
||||
rejections: u32,
|
||||
abstained: bool,
|
||||
note: String,
|
||||
}
|
||||
|
||||
fn run_case(llm: &Llm, case: &Case, arm: &'static str) -> Outcome {
|
||||
let tag = format!("{}_{}", arm, case.name);
|
||||
let mut src = case.broken.clone();
|
||||
let mut attempts = 0u32;
|
||||
let mut rejections = 0u32;
|
||||
let mut abstained = false;
|
||||
let mut fixed = false;
|
||||
let mut one_shot = false;
|
||||
let mut feedback = String::new();
|
||||
let mut note = String::new();
|
||||
|
||||
let mut infra = 0u32;
|
||||
while attempts < 2 {
|
||||
let diags = diagnose(case.verify, &src, &tag);
|
||||
let Some(diag) = diags.first().cloned() else {
|
||||
fixed = true;
|
||||
one_shot = true;
|
||||
break;
|
||||
};
|
||||
let ranges = target_ranges(&diag);
|
||||
assert!(!ranges.is_empty(), "[{}] 诊断必须可锚定: {diag}", case.name);
|
||||
|
||||
let shown = if arm == "anchor" {
|
||||
diags.clone()
|
||||
} else {
|
||||
strip_doc_context(&diags)
|
||||
};
|
||||
let reply = match chat(llm, SYSTEM, &prompt(&src, &ranges, &shown, &feedback)) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
// 基础设施抖动不计入修复尝试
|
||||
infra += 1;
|
||||
if infra >= 2 {
|
||||
note = format!("LLM 调用失败(重试后): {e}");
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
attempts += 1;
|
||||
let edits = parse_edits(&reply);
|
||||
if edits.is_empty() {
|
||||
abstained = true;
|
||||
note = format!("模型弃权/输出不可解析: {}", truncate(&reply, 100));
|
||||
break;
|
||||
}
|
||||
|
||||
let resp = apply_via_mlsp(&src, &ranges, &edits);
|
||||
if resp["ok"] == false {
|
||||
// 护栏:越界必拒 + abstain(源零字节不改 —— apply 全有或全无)
|
||||
rejections += 1;
|
||||
assert_eq!(resp["error"]["code"], "mlsp.repair_out_of_scope", "{resp}");
|
||||
assert_eq!(resp["error"]["on_violation"], "abstain", "{resp}");
|
||||
note = format!("越界被拒: {}", truncate(&resp["error"]["message"].to_string(), 100));
|
||||
feedback = note.clone();
|
||||
continue;
|
||||
}
|
||||
|
||||
let new_src = resp["result"]["source"].as_str().unwrap().to_string();
|
||||
let after = diagnose(case.verify, &new_src, &tag);
|
||||
if after.is_empty() {
|
||||
fixed = true;
|
||||
one_shot = attempts == 1;
|
||||
break;
|
||||
}
|
||||
src = new_src;
|
||||
feedback = format!("修复后仍有诊断: {}", truncate(&after[0].to_string(), 240));
|
||||
note = feedback.clone();
|
||||
}
|
||||
|
||||
Outcome {
|
||||
case: case.name.clone(),
|
||||
arm,
|
||||
fixed,
|
||||
one_shot,
|
||||
attempts,
|
||||
rejections,
|
||||
abstained,
|
||||
note,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 实验主入口 ----------
|
||||
|
||||
#[test]
|
||||
#[ignore = "模型在环实验:需要 .env / OPENAI_API_KEY 与网络(cflmy @ llm.cflmy.cn)"]
|
||||
fn exp_a_anchored_repair_with_model() {
|
||||
let llm = llm_cfg();
|
||||
let mut cs = cases();
|
||||
if let Ok(n) = std::env::var("MARQDO_EXP_LIMIT") {
|
||||
if let Ok(n) = n.parse::<usize>() {
|
||||
cs.truncate(n);
|
||||
}
|
||||
}
|
||||
|
||||
let outcomes: Mutex<Vec<Outcome>> = Mutex::new(Vec::new());
|
||||
for arm in ["anchor", "plain"] {
|
||||
let queue: Mutex<Vec<usize>> = Mutex::new((0..cs.len()).collect());
|
||||
std::thread::scope(|s| {
|
||||
for _ in 0..4 {
|
||||
s.spawn(|| loop {
|
||||
let i = queue.lock().unwrap().pop();
|
||||
let Some(i) = i else { break };
|
||||
let o = run_case(&llm, &cs[i], arm);
|
||||
eprintln!(
|
||||
"[{arm}] {:<18} {} attempts={} rejects={} {}",
|
||||
o.case,
|
||||
if o.fixed { if o.one_shot { "FIXED@1" } else { "FIXED@2" } } else { "FAILED " },
|
||||
o.attempts,
|
||||
o.rejections,
|
||||
o.note
|
||||
);
|
||||
outcomes.lock().unwrap().push(o);
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ---- 汇总 ----
|
||||
let outs = outcomes.into_inner().unwrap();
|
||||
let mut report = String::new();
|
||||
report.push_str(&format!(
|
||||
"# 修复实验(§5.2(a) 锚点消融)—— {}\n\n\
|
||||
| | |\n|---|---|\n| 模型 | `{}` @ `{}` |\n| 样本 | golden 20 例(易锚定错误) |\n\
|
||||
| 协议 | `validate` → `repair_targets` → 模型行编辑 → `repair_apply`(越界必拒/abstain)→ 复验;≤2 轮 |\n\
|
||||
| 对照 | `anchor` = 诊断带 `doc_anchor`/`doc_quote`/`suggestion`;`plain` = 全部剥掉 |\n\n",
|
||||
chrono_date(), llm.model, llm.base
|
||||
));
|
||||
|
||||
let arm_stats = |arm: &str| -> String {
|
||||
let rows: Vec<&Outcome> = outs.iter().filter(|o| o.arm == arm).collect();
|
||||
let n = rows.len().max(1) as f64;
|
||||
let one = rows.iter().filter(|o| o.one_shot).count();
|
||||
let fin = rows.iter().filter(|o| o.fixed).count();
|
||||
let att: f64 = rows.iter().map(|o| o.attempts as f64).sum::<f64>() / n;
|
||||
let rej = rows.iter().map(|o| o.rejections).sum::<u32>();
|
||||
let abs = rows.iter().filter(|o| o.abstained).count();
|
||||
format!(
|
||||
"| {arm} | {one}/{rows_n}({one_pct:.0}%) | {fin}/{rows_n}({fin_pct:.0}%) | {att:.2} | {rej} | {abs} |\n",
|
||||
rows_n = rows.len(),
|
||||
one_pct = 100.0 * one as f64 / n,
|
||||
fin_pct = 100.0 * fin as f64 / n,
|
||||
)
|
||||
};
|
||||
report.push_str("## 汇总(设计目标:一轮修复率 ≥ 80%)\n\n| 臂 | 一轮修复率 | 最终修复率(≤2轮) | 平均尝试 | 越界拒(护栏) | 弃权 |\n|---|---|---|---|---|---|\n");
|
||||
let stat_a = arm_stats("anchor");
|
||||
let stat_p = arm_stats("plain");
|
||||
report.push_str(&stat_a);
|
||||
report.push_str(&stat_p);
|
||||
|
||||
report.push_str("\n## 逐例\n\n| 例 | 臂 | 结果 | 轮次 | 越界拒 | 备注 |\n|---|---|---|---|---|---|\n");
|
||||
let mut sorted: Vec<&Outcome> = outs.iter().collect();
|
||||
sorted.sort_by(|a, b| a.case.cmp(&b.case).then(a.arm.cmp(b.arm)));
|
||||
for o in &sorted {
|
||||
report.push_str(&format!(
|
||||
"| {} | {} | {} | {} | {} | {} |\n",
|
||||
o.case,
|
||||
o.arm,
|
||||
if o.fixed { if o.one_shot { "一轮修复" } else { "二轮修复" } } else { "未修复" },
|
||||
o.attempts,
|
||||
o.rejections,
|
||||
o.note.replace('|', "\\|")
|
||||
));
|
||||
}
|
||||
report.push_str("\n## 护栏核对\n\n- 每次越界编辑都以 `mlsp.repair_out_of_scope` + `abstain` 被拒,且 `repair_apply` 全有或全无(源零字节不改)——由 `tests/repair_loop.rs` 的确定性断言覆盖;本实验计数仅作记录。\n");
|
||||
|
||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("doc/roadmap")
|
||||
.join(format!("repair-experiment-{}.md", chrono_date()));
|
||||
std::fs::write(&path, &report).unwrap();
|
||||
eprintln!("\n{report}");
|
||||
eprintln!("报告已写入 {}", path.display());
|
||||
|
||||
// ---- 功能正确性断言(实验 = 真实消费者) ----
|
||||
let anchor_fixed = outs.iter().filter(|o| o.arm == "anchor" && o.fixed).count();
|
||||
assert!(
|
||||
anchor_fixed >= 1,
|
||||
"修复回路必须至少闭环一例(否则功能链路有问题):\n{:#?}",
|
||||
outs
|
||||
);
|
||||
}
|
||||
|
||||
/// 无外部 chrono 依赖的日期(UTC)。
|
||||
fn chrono_date() -> String {
|
||||
let secs = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_secs();
|
||||
// 2026-09-22 ≈ 1790000000s 附近的简易换算不必精确到历法;直接用运行日环境变量兜底
|
||||
if let Ok(d) = std::env::var("MARQDO_EXP_DATE") {
|
||||
return d;
|
||||
}
|
||||
// 简化:从 unix 秒推 YYYY-MM-DD(1970-01-01 起算,忽略闰秒)
|
||||
let days = secs / 86400;
|
||||
let (y, m, d) = civil_from_days(days as i64);
|
||||
format!("{y:04}-{m:02}-{d:02}")
|
||||
}
|
||||
|
||||
/// Howard Hinnant 的 days→civil 算法(公历)。
|
||||
fn civil_from_days(z: i64) -> (i64, u32, u32) {
|
||||
let z = z + 719468;
|
||||
let era = if z >= 0 { z } else { z - 146096 } / 146097;
|
||||
let doe = (z - era * 146097) as u64;
|
||||
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
|
||||
let y = yoe as i64 + era * 400;
|
||||
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
|
||||
let mp = (5 * doy + 2) / 153;
|
||||
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
|
||||
let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32;
|
||||
(if m <= 2 { y + 1 } else { y }, m, d)
|
||||
}
|
||||
+11
-289
@@ -6,296 +6,23 @@
|
||||
//!
|
||||
//! 护栏负例内嵌每例:越界编辑(第 1 行 `# main`,永不在靶点内)必拒
|
||||
//! (`mlsp.repair_out_of_scope` + `abstain`),源零字节不改。
|
||||
//! 模型实验(≥80% 修复率)只需把 `derive_edits` 的金修复换成模型输出的编辑。
|
||||
//! 模型实验见 `tests/repair_experiment.rs`(同一批 20 例)。
|
||||
|
||||
use marqdo::diagnostics::Diagnostic;
|
||||
use marqdo::mlsp::handle_line;
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
use marqdo::repair::{apply_edits, Edit};
|
||||
use marqdo::{run_file, RunOptions};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
fn mcall(req: Value) -> Value {
|
||||
serde_json::from_str(&handle_line(&req.to_string())).expect("MLSP 响应必须是合法 JSON")
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
enum Verify {
|
||||
/// 运行期契约错(四边界):修复后 run 必须成功。
|
||||
Run(&'static str),
|
||||
/// 静态错(check / parse):修复后 validate 零诊断。
|
||||
Static(&'static str),
|
||||
}
|
||||
|
||||
struct Case {
|
||||
name: String,
|
||||
broken: String,
|
||||
gold: String,
|
||||
verify: Verify,
|
||||
}
|
||||
|
||||
// ---------- 诊断 ----------
|
||||
|
||||
fn static_diags(src: &str) -> Vec<Value> {
|
||||
let resp = mcall(json!({"id": 1, "method": "validate", "params": {"source": src}}));
|
||||
if resp["ok"] == false {
|
||||
return vec![resp["error"].clone()];
|
||||
}
|
||||
resp["result"]["diagnostics"]
|
||||
.as_array()
|
||||
.cloned()
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
fn temp_path(idx: usize) -> std::path::PathBuf {
|
||||
std::env::temp_dir().join(format!("marqdo_repair_gold_{idx}.mq.md"))
|
||||
}
|
||||
|
||||
fn run_diag(src: &str, idx: usize) -> Value {
|
||||
let p = temp_path(idx);
|
||||
std::fs::write(&p, src).unwrap();
|
||||
let err = run_file(&p, &RunOptions::default())
|
||||
.expect_err("broken 源必须报契约错");
|
||||
match Diagnostic::find(&err) {
|
||||
Some(d) => d.to_json(),
|
||||
None => json!({"code": "?", "message": format!("{err:#}")}),
|
||||
}
|
||||
}
|
||||
|
||||
fn run_ok(src: &str, idx: usize) {
|
||||
let p = temp_path(idx);
|
||||
std::fs::write(&p, src).unwrap();
|
||||
run_file(&p, &RunOptions::default()).expect("修复后必须运行成功");
|
||||
let _ = std::fs::remove_file(&p);
|
||||
}
|
||||
|
||||
/// 诊断 JSON → 靶点范围(span 行 + doc_anchor 的 L 范围)。
|
||||
fn ranges_of(v: &Value) -> Vec<(u32, u32)> {
|
||||
let mut rs = Vec::new();
|
||||
if let Some(l) = v["span"]["line"].as_u64() {
|
||||
rs.push((l as u32, l as u32));
|
||||
}
|
||||
if let Some(a) = v["doc_anchor"].as_str() {
|
||||
if let Some(rest) = a.split('#').nth(1) {
|
||||
if let Some(rest) = rest.strip_prefix('L') {
|
||||
if let Some((s, e)) = rest.split_once("-L") {
|
||||
if let (Ok(s), Ok(e)) = (s.parse(), e.parse()) {
|
||||
rs.push((s, e));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
rs
|
||||
}
|
||||
|
||||
// ---------- 金修复 → 行编辑(前缀/后缀 diff) ----------
|
||||
|
||||
fn derive_edits(broken: &str, gold: &str) -> Vec<Edit> {
|
||||
let b: Vec<&str> = broken.lines().collect();
|
||||
let g: Vec<&str> = gold.lines().collect();
|
||||
let mut p = 0usize;
|
||||
while p < b.len() && p < g.len() && b[p] == g[p] {
|
||||
p += 1;
|
||||
}
|
||||
let mut s = 0usize;
|
||||
while s < b.len() - p
|
||||
&& s < g.len() - p
|
||||
&& b[b.len() - 1 - s] == g[g.len() - 1 - s]
|
||||
{
|
||||
s += 1;
|
||||
}
|
||||
let mut edits = Vec::new();
|
||||
for l in (p + 1)..=(b.len() - s) {
|
||||
edits.push(Edit::Delete { line: l as u32 });
|
||||
}
|
||||
let at = (b.len() - s + 1) as u32;
|
||||
for text in &g[p..(g.len() - s)] {
|
||||
edits.push(Edit::InsertBefore {
|
||||
line: at,
|
||||
text: (*text).to_string(),
|
||||
});
|
||||
}
|
||||
edits
|
||||
}
|
||||
|
||||
fn apply_via_mlsp(src: &str, ranges: &[(u32, u32)], edits: &[Edit]) -> Value {
|
||||
let resp = mcall(json!({
|
||||
"id": 2,
|
||||
"method": "repair_apply",
|
||||
"params": {
|
||||
"source": src,
|
||||
"ranges": ranges.iter().map(|(s, e)| json!([s, e])).collect::<Vec<_>>(),
|
||||
"edits": edits.iter().map(|e| match e {
|
||||
Edit::Replace { line, text } => json!({"op": "replace", "line": line, "text": text}),
|
||||
Edit::Delete { line } => json!({"op": "delete", "line": line}),
|
||||
Edit::InsertBefore { line, text } => json!({"op": "insert", "line": line, "text": text}),
|
||||
}).collect::<Vec<_>>(),
|
||||
}
|
||||
}));
|
||||
resp
|
||||
}
|
||||
|
||||
// ---------- 20 例(10 骨架 × 2 变体) ----------
|
||||
|
||||
fn cases() -> Vec<Case> {
|
||||
let mut out: Vec<Case> = Vec::new();
|
||||
let push = |out: &mut Vec<Case>, name: &str, broken: String, gold: String, verify: Verify| {
|
||||
out.push(Case {
|
||||
name: name.to_string(),
|
||||
broken,
|
||||
gold,
|
||||
verify,
|
||||
});
|
||||
};
|
||||
|
||||
// 1–2 调用实参:数字形参传了文本(金修复改调用行 = span 侧)
|
||||
for (v, lit, n) in [("a", "七", 7), ("b", "八", 8)] {
|
||||
let broken = format!(
|
||||
"# main\n\n> 加一 \"{lit}\"\n\n## 加一\n\n对于输入变量`n`,期望是数字。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | number | 结果 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("> 加一 \"{lit}\""), &format!("> 加一 {n}"));
|
||||
push(&mut out, &format!("arg-num-{v}"), broken, gold, Verify::Run("contract.arg_mismatch"));
|
||||
}
|
||||
|
||||
// 3–4 调用实参:文本形参传了裸数(金修复加引号 = span 侧)
|
||||
for (v, n) in [("a", 7), ("b", 8)] {
|
||||
let broken = format!(
|
||||
"# main\n\n> 转换 {n}\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 结果 |\n\n*s*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("> 转换 {n}"), &format!("> 转换 \"{n}\""));
|
||||
push(&mut out, &format!("arg-text-{v}"), broken, gold, Verify::Run("contract.arg_mismatch"));
|
||||
}
|
||||
|
||||
// 5–6 返回契约:返回文本却声明 number(金修复改契约行 = anchor 侧)
|
||||
for (v, lit) in [("a", "七"), ("b", "你好")] {
|
||||
let broken = format!(
|
||||
"# main\n\n**结果 = > 转换 \"{lit}\"**\n*结果*\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | number | 结果 |\n\n*s*\n"
|
||||
);
|
||||
let gold = broken.replace("| | number | 结果 |", "| | text | 结果 |");
|
||||
push(&mut out, &format!("ret-type-{v}"), broken, gold, Verify::Run("contract.return_mismatch"));
|
||||
}
|
||||
|
||||
// 7–8 字段类型:数据与字段契约不符(金修复改契约行 = anchor 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| abc | 小明 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| id | number | |", "| id | text | |");
|
||||
push(&mut out, "field-type-a", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | number | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| 1 | 小明 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| name | number | 是 |", "| name | text | 是 |");
|
||||
push(&mut out, "field-type-b", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
}
|
||||
|
||||
// 9–10 缺字段:数据没有契约要求的非空字段(金修复标可空 = anchor 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | |\n\n\
|
||||
`用户` =\n| id | qq |\n|------|------|\n| 1 | 5 |\n\n*`用户`[^id]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| name | text | |", "| name | text | 是 |");
|
||||
push(&mut out, "field-nullable-a", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | |\n\n\
|
||||
`用户` =\n| name | qq |\n|------|------|\n| 小明 | 5 |\n\n*`用户`[^name]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("| id | number | |", "| id | number | 是 |");
|
||||
push(&mut out, "field-nullable-b", broken, gold, Verify::Run("contract.field_mismatch"));
|
||||
}
|
||||
|
||||
// 11–12 未知键:访问不在契约里的字段(金修复改索引键 = span 侧)
|
||||
{
|
||||
let broken = "# main\n\n| 字段 | 类型 | 可空 |\n|------|------|------|\n| id | number | |\n| name | text | 是 |\n\n\
|
||||
`用户` =\n| id | name |\n|------|------|\n| 1 | 小明 |\n\n*`用户`[^email]*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace("[^email]", "[^name]");
|
||||
let broken_b = broken.replace("[^email]", "[^phone]");
|
||||
let gold_b = broken_b.replace("[^phone]", "[^id]");
|
||||
push(&mut out, "key-fix-a", broken, gold, Verify::Run("contract.unknown_key"));
|
||||
push(&mut out, "key-fix-b", broken_b, gold_b, Verify::Run("contract.unknown_key"));
|
||||
}
|
||||
|
||||
// 13–14 类型名拼错(金修复改正拼写 = anchor 侧)
|
||||
for (v, typo, fix) in [("a", "nubmer", "number"), ("b", "txt", "text")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | {typo} | 输入值 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(typo, fix);
|
||||
push(&mut out, &format!("type-typo-{v}"), broken, gold, Verify::Static("contract.unknown_type"));
|
||||
}
|
||||
|
||||
// 15–16 契约行名字错位(金修复改行名 = anchor 侧)
|
||||
for (v, wrong) in [("a", "nn"), ("b", "nm")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| {wrong} | number | 输入值 |\n\n*n+1*\n"
|
||||
);
|
||||
let gold = broken.replace(&format!("| {wrong} | number | 输入值 |"), "| n | number | 输入值 |");
|
||||
push(&mut out, &format!("drift-rename-{v}"), broken, gold, Verify::Static("contract.param_drift"));
|
||||
}
|
||||
|
||||
// 17–18 新升参未被契约覆盖(金修复补契约行 = anchor 侧,插入表尾)
|
||||
for (v, extra) in [("a", "m"), ("b", "k")] {
|
||||
let broken = format!(
|
||||
"# main\n\n## 加一\n\n对于输入变量`n`,临时量`{extra}`。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n*n + {extra}*\n"
|
||||
);
|
||||
let gold = broken.replace(
|
||||
"| n | number | 输入值 |",
|
||||
&format!("| n | number | 输入值 |\n| {extra} | number | 临时 |"),
|
||||
);
|
||||
push(&mut out, &format!("drift-cover-{v}"), broken, gold, Verify::Static("contract.param_drift"));
|
||||
}
|
||||
|
||||
// 19–20 两份契约打架(金修复删第二张表 = anchor 侧,纯删除)
|
||||
{
|
||||
let broken = "# main\n\n## 加一\n\n对于输入变量`n`,执行加一。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 输入值 |\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 重复 |\n\n*n+1*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace(
|
||||
"\n| 参数 | 类型 | 说明 |\n|------|------|------|\n| n | number | 重复 |",
|
||||
"",
|
||||
);
|
||||
push(&mut out, "dup-delete-a", broken, gold, Verify::Static("contract.duplicate"));
|
||||
|
||||
let broken = "# main\n\n## 转换\n\n对于输入变量`s`,期望是文本。\n\n\
|
||||
| 参数 | 类型 | 说明 |\n|------|------|------|\n| s | text | 输入 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 结果 |\n\n\
|
||||
| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 同上 |\n\n*s*\n"
|
||||
.to_string();
|
||||
let gold = broken.replace(
|
||||
"\n| 返回 | 类型 | 说明 |\n|------|------|------|\n| | text | 同上 |",
|
||||
"",
|
||||
);
|
||||
push(&mut out, "dup-delete-b", broken, gold, Verify::Static("contract.duplicate"));
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
|
||||
// ---------- 完整修复循环 ----------
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn golden_repair_loop_all_20() {
|
||||
let cases = cases();
|
||||
assert_eq!(cases.len(), 20, "golden 必须正好 20 例");
|
||||
|
||||
for (idx, case) in cases.iter().enumerate() {
|
||||
for case in &cases {
|
||||
// 1. 诊断(broken 必须给出期望错误码)
|
||||
let (code, diags) = match case.verify {
|
||||
Verify::Run(code) => (code, vec![run_diag(&case.broken, idx)]),
|
||||
Verify::Static(code) => (code, static_diags(&case.broken)),
|
||||
};
|
||||
let code = case.code();
|
||||
let diags = diagnose(case.verify, &case.broken, &case.name);
|
||||
assert!(
|
||||
diags.iter().any(|d| d["code"] == code),
|
||||
"[{}] 期望诊断 {code},实得 {diags:?}",
|
||||
@@ -317,14 +44,9 @@ fn golden_repair_loop_all_20() {
|
||||
assert_eq!(resp["ok"], true, "[{}] repair_apply 应接受金修复: {resp}", case.name);
|
||||
assert_eq!(resp["result"]["source"], case.gold, "[{}]", case.name);
|
||||
|
||||
// 4. 复验
|
||||
match case.verify {
|
||||
Verify::Run(_) => run_ok(&case.gold, idx),
|
||||
Verify::Static(_) => {
|
||||
let after = static_diags(&case.gold);
|
||||
assert!(after.is_empty(), "[{}] 修复后必须零诊断: {after:?}", case.name);
|
||||
}
|
||||
}
|
||||
// 4. 复验(干净 = 空诊断)
|
||||
let after = diagnose(case.verify, &case.gold, &case.name);
|
||||
assert!(after.is_empty(), "[{}] 修复后必须零诊断: {after:?}", case.name);
|
||||
|
||||
// 5. 护栏负例:越界编辑(第 1 行 `# main`)必拒 + 源零字节不改
|
||||
let bad = vec![Edit::Replace { line: 1, text: "# 被篡改".to_string() }];
|
||||
|
||||
Reference in new issue
Block a user