test(workbench): harden F-matrix evidence and demote Verified claim

Add dual-endpoint LLM isolation, usage revision replace, cancel/F08/F11
asserts, and mark the design status Partial until remaining F gaps close.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
cflmyandCursor committed 2026-10-25 11:01:02 +08:00
1 parent 03396afd02
commit 090b026547
6 files changed
+370 -29

No files matched your search

@@ -1,6 +1,6 @@
# qdagent 知识工作台所需的多模型运行与 View 嵌入支持
日期:2026-10-25。状态:Verified(M00–M06 切片,以 `tests/workbench_contracts` + MLSP capabilities 证据为准;原生非兼容适配器仍 unavailable)。
日期:2026-10-25。状态:Partial / in progress(M00–M06 脚手架与合同门禁已落地;F 矩阵仍有缺口,不得整文宣称 Verified;原生非兼容适配器仍 unavailable)。
源码基线:Marqdo `dc853fb`(2.1.2),qdagent `643dd31`,已同步远端。应用侧配套:[qdagent 多提供商与知识工作台](../../../qdagent/doc/design/multi-provider-and-knowledge-workbench-2026-10-25.md)。跨仓相对链接以兄弟仓库工作区布局为准。
@@ -407,28 +407,28 @@ M01 支持多个兼容提供商不等待原生协议全做完;但原生适配
```bash
cargo test --offline --locked --test workbench_contracts
cargo test --offline --locked -p marqdo_plugin_llm
cargo test --offline --locked --test executable_knowledge f1_agent_eng_bind_gate_mq_offline -- --exact
```
已交付并通过自动化子集:
已交付并通过自动化子集(**非** F01–F16 全绿):
| 切片 | 证据 |
|---|---|
| M00 | fixtures + JSON Schema 消费;`capabilities`;`force_unloaded` / `force_*_off` 诚实降级(F16) |
| M01 | `messages[]` + strict selection;`RunContextV1`;workbook skeleton 禁 `llm.llm`;子任务转发 model/base_url |
| M02 | usage null≠0;event_sink;`MARQDO_CANCEL`/`_TOKEN`;web `sse-live` 断连写 cancel |
| M03 | `view.document` 不经 `build_file_view`;heading outline;path escape 拒绝;F08 无副作用标记 |
| M04 | `view.reference_resolve` stale/version_unavailable;`artifact.inspect` version_mismatch;embed `sse_accepts_packet_strict` |
| M05 | extract `:B` locator;analysis_cache 按 project_root 隔离(F14) |
| M06 | CI `executable-knowledge.yml` 预构建 agent+llm 并跑 `workbench_contracts` |
| M00 | fixtures + JSON Schema 消费;`capabilities`;`force_unloaded` / `force_*_off`(F16) |
| M01 | strict selection;`RunContextV1` 并行不串扰;llm plugin messages/roles;workbook 禁 `llm.llm` |
| M02 | usage null≠0;revision 替换(F05);cancel token(F06/F07);event_sink;web `sse-live` 断连 cancel |
| M03 | `view.document`;CJK outline/nodes;path escape;静态 a11y 子集;F08 无副作用标记 |
| M04 | stale / version_unavailable / **ambiguous**(08-same-name);`artifact.version_mismatch`;embed strict SSE |
| M05 | extract `:B`;analysis_cache 隔离(F14 子集) |
| M06 | CI 预构建 agent+llm + `workbench_contracts` |
已知限制(不得宣称已完成):
已知限制(**发版阻断直至补齐或书面豁免**):
- `anthropic-native` / `gemini-native` 仍 `available:false`。
- PDF/DOCX/OCR extract 仍显式 unsupported。
- 历史正文快照存储仍由应用负责;框架对缺快照返回 `version_unavailable`。
- `v3_corpus_1000_compile_smoke` 耗时门限随机器变化;release/debug 分剖面记录,不删 corpus。
- 未调用付费供应商;流式/取消端到端依赖假 HTTP / 本地 cancel token。
- 双真实 HTTP 端点联调、Agent resume 全路径、Go `sse-live` 集成测试仍薄。
- DocumentView 六类步骤完整树 / 历史正文快照仍未闭合(缺快照 → `version_unavailable`)。
- 完整浏览器 F12 留给 qdagent;原生适配器 / PDF/OCR 仍 unavailable。
- 未调用付费供应商。
### 12.2 历史基线记录(设计起草时)
+163
View File
@@ -461,3 +461,166 @@ marqdo_plugin!(marqdo_plugin_sdk::ABI_VERSION, [
("llm_ping", "", llm_ping),
("llm_chat_completions", "base_url,api_key,model,prompt,messages,selection,strict_selection,auth_mode,stream,echo,suffix,bearer", llm_chat_completions),
]);
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn f03_messages_preserve_roles_and_reject_image() {
let msgs = build_messages(&json!({
"messages": [
{"role": "system", "content": "rules"},
{"role": "user", "content": "你好"},
{"role": "assistant", "content": "先前"},
{"role": "user", "content": "最新用户"}
]
}))
.unwrap();
let arr = msgs.as_array().unwrap();
assert_eq!(arr.len(), 4);
assert_eq!(arr[3]["role"], "user");
assert_eq!(arr[3]["content"], "最新用户");
let err = build_messages(&json!({
"messages": [{"role": "user", "content": [{"type": "image", "url": "x"}]}]
}))
.unwrap_err();
assert!(err.contains("unsupported_input"), "{err}");
}
#[test]
fn f01_strict_selection_does_not_invent_env_model() {
std::env::remove_var("MARQDO_CANCEL");
std::env::remove_var("MARQDO_CANCEL_TOKEN");
let err = chat_completions(&json!({
"strict_selection": true,
"base_url": "http://127.0.0.1:9",
"api_key": "k",
"prompt": "hi"
// missing selection → must fail before any HTTP
}))
.unwrap_err();
assert!(err.contains("selection_missing"), "{err}");
}
#[test]
fn f06_cancel_short_circuits_before_http() {
// Prefer token file over process-global MARQDO_CANCEL to avoid parallel test races.
let tok = std::env::temp_dir().join(format!("llm-cancel-{}", std::process::id()));
std::fs::write(&tok, "cancelled").unwrap();
std::env::set_var("MARQDO_CANCEL_TOKEN", &tok);
let out = chat_completions(&json!({
"base_url": "http://127.0.0.1:9",
"api_key": "k",
"model": "m",
"prompt": "hi"
}));
std::env::remove_var("MARQDO_CANCEL_TOKEN");
let _ = std::fs::remove_file(&tok);
let out = out.unwrap();
assert_eq!(out["finish"], "cancelled");
}
/// F01: two local endpoints receive distinct models/auth — no cross-talk.
#[test]
fn f01_dual_endpoint_isolation() {
use std::io::{Read, Write};
use std::net::TcpListener;
use std::sync::{Arc, Mutex};
use std::thread;
fn spawn_mock(tag: &'static str, seen: Arc<Mutex<Vec<String>>>) -> String {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
thread::spawn(move || {
if let Ok((mut stream, _)) = listener.accept() {
let mut raw = Vec::new();
let mut buf = [0u8; 4096];
loop {
let n = stream.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
raw.extend_from_slice(&buf[..n]);
if let Some(pos) = raw.windows(4).position(|w| w == b"\r\n\r\n") {
let header = String::from_utf8_lossy(&raw[..pos]);
let cl = header
.lines()
.find_map(|l| l.strip_prefix("Content-Length:"))
.and_then(|v| v.trim().parse::<usize>().ok())
.unwrap_or(0);
let body_start = pos + 4;
while raw.len() < body_start + cl {
let n = stream.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
raw.extend_from_slice(&buf[..n]);
}
break;
}
}
let req = String::from_utf8_lossy(&raw).to_string();
seen.lock().unwrap().push(format!("{tag}:{req}"));
let body = r#"{"choices":[{"message":{"content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}}"#;
let resp = format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
body.len(),
body
);
let _ = stream.write_all(resp.as_bytes());
}
});
format!("http://{addr}")
}
let seen_a = Arc::new(Mutex::new(Vec::new()));
let seen_b = Arc::new(Mutex::new(Vec::new()));
let url_a = spawn_mock("A", seen_a.clone());
let url_b = spawn_mock("B", seen_b.clone());
let out_a = chat_completions(&json!({
"base_url": url_a,
"api_key": "key-a",
"model": "model-a",
"auth_mode": "api_key",
"messages": [{"role":"user","content":"from-a"}],
"selection": {
"selection_id": "sel-a",
"provider_id": "p-a",
"model_id": "m-a",
"upstream_model_id": "model-a",
"strict": true
},
"strict_selection": true
}))
.unwrap();
let out_b = chat_completions(&json!({
"base_url": url_b,
"api_key": "key-b",
"model": "model-b",
"auth_mode": "api_key",
"messages": [{"role":"user","content":"from-b"}],
"selection": {
"selection_id": "sel-b",
"provider_id": "p-b",
"model_id": "m-b",
"upstream_model_id": "model-b",
"strict": true
},
"strict_selection": true
}))
.unwrap();
assert_eq!(out_a["finish"], "stop", "{out_a}");
assert_eq!(out_b["finish"], "stop", "{out_b}");
let a = seen_a.lock().unwrap().join("\n");
let b = seen_b.lock().unwrap().join("\n");
assert!(a.contains("model-a"), "A saw wrong model: {a}");
assert!(a.contains("key-a") || a.contains("Authorization"), "{a}");
assert!(!a.contains("model-b"), "A must not see B's model: {a}");
assert!(b.contains("model-b"), "B saw wrong model: {b}");
assert!(!b.contains("model-a"), "B must not see A's model: {b}");
}
}
+4 -1
View File
@@ -22,4 +22,7 @@ pub use event_sink::{bind_event_sink, clear_event_sink, emit_event, emit_message
pub use locator::{resolve_reference, KnowledgeLocator};
pub use run_context::{cancel_requested, RunContext};
pub use selection::{parse_model_selection, ModelSelection};
pub use usage::{model_call_usage_from_provider, runtime_event, token_or_null};
pub use usage::{
model_call_usage_from_provider, model_call_usage_from_provider_rev, replace_usage_if_newer,
runtime_event, token_or_null,
};
+31 -1
View File
@@ -18,6 +18,19 @@ pub fn model_call_usage_from_provider(
selection: &Value,
provider_usage: &Value,
status: &str,
) -> Value {
model_call_usage_from_provider_rev(call_id, attempt_id, run_id, selection, provider_usage, status, 1)
}
/// Same as [`model_call_usage_from_provider`] with an explicit revision (F05).
pub fn model_call_usage_from_provider_rev(
call_id: &str,
attempt_id: &str,
run_id: &str,
selection: &Value,
provider_usage: &Value,
status: &str,
revision: u64,
) -> Value {
let has_usage = provider_usage.is_object();
let input = provider_usage
@@ -97,7 +110,7 @@ pub fn model_call_usage_from_provider(
"total_tokens": token_or_null(total),
"cached_input_tokens": token_or_null(cached),
"reasoning_output_tokens": token_or_null(reasoning),
"revision": 1,
"revision": revision.max(1),
"provider_request_id": provider_usage.get("provider_request_id").cloned().unwrap_or(Value::Null),
"diagnostics": diagnostics,
// Legacy fields kept but marked — apps must not treat token_cost as money.
@@ -110,6 +123,23 @@ pub fn model_call_usage_from_provider(
})
}
/// Replace prior usage for the same (call_id, attempt_id) when revision is newer (F05).
/// Returns the winning snapshot; does not sum cumulative stream counters.
pub fn replace_usage_if_newer(prior: &Value, next: &Value) -> Value {
let same_call = prior.get("call_id") == next.get("call_id")
&& prior.get("attempt_id") == next.get("attempt_id");
if !same_call {
return next.clone();
}
let pr = prior.get("revision").and_then(|v| v.as_u64()).unwrap_or(0);
let nr = next.get("revision").and_then(|v| v.as_u64()).unwrap_or(0);
if nr >= pr {
next.clone()
} else {
prior.clone()
}
}
/// RuntimeEventV1 envelope helper.
pub fn runtime_event(
event_id: &str,
@@ -0,0 +1,12 @@
---
title: F08 side-effect probe
status: draft
---
# F08
This document would write a marker if executed. View must never run it.
## mark
> file.write path=".workbench-f08-marker" text="executed"
+145 -12
View File
@@ -308,13 +308,33 @@ fn m01_run_context_spawn_child_inherits_selection() {
}
#[test]
fn m01_messages_shape_rejects_unsupported_part() {
// Plugin validation is mirrored by selection/message contract: unsupported part types error.
let bad = serde_json::json!([
{"role": "user", "content": [{"type": "image", "url": "x"}]}
]);
let ty = bad[0]["content"][0]["type"].as_str().unwrap();
assert_ne!(ty, "text");
fn m01_f01_parallel_selections_do_not_cross() {
let a = marqdo::workbench::RunContext::from_value(&serde_json::json!({
"run_id": "run-a",
"selection_context": {
"upstream_model_id": "model-a",
"base_url": "http://127.0.0.1:7001"
}
}))
.unwrap();
let b = marqdo::workbench::RunContext::from_value(&serde_json::json!({
"run_id": "run-b",
"selection_context": {
"upstream_model_id": "model-b",
"base_url": "http://127.0.0.1:7002"
}
}))
.unwrap();
let ea = a.selection_env();
let eb = b.selection_env();
let model_a = ea.iter().find(|(k, _)| k == "OPENAI_MODEL").map(|(_, v)| v.as_str());
let model_b = eb.iter().find(|(k, _)| k == "OPENAI_MODEL").map(|(_, v)| v.as_str());
let base_a = ea.iter().find(|(k, _)| k == "OPENAI_BASE_URL").map(|(_, v)| v.as_str());
let base_b = eb.iter().find(|(k, _)| k == "OPENAI_BASE_URL").map(|(_, v)| v.as_str());
assert_eq!(model_a, Some("model-a"));
assert_eq!(model_b, Some("model-b"));
assert_ne!(base_a, base_b);
assert_ne!(a.run_id, b.run_id);
}
#[test]
@@ -339,6 +359,63 @@ fn m02_usage_revision_and_cancel_token() {
assert!(!marqdo::workbench::cancel_requested(Some("/no/such/cancel/token")));
}
#[test]
fn m02_f05_usage_revision_replaces_same_attempt_not_sum() {
let sel = serde_json::json!({"selection_id": "s"});
let partial = marqdo::workbench::model_call_usage_from_provider_rev(
"c1",
"a1",
"r1",
&sel,
&serde_json::json!({"prompt_tokens": 10, "completion_tokens": 1, "total_tokens": 11}),
"started",
1,
);
let final_u = marqdo::workbench::model_call_usage_from_provider_rev(
"c1",
"a1",
"r1",
&sel,
&serde_json::json!({"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}),
"succeeded",
2,
);
let won = marqdo::workbench::replace_usage_if_newer(&partial, &final_u);
assert_eq!(won["revision"], 2);
assert_eq!(won["total_tokens"], 15);
// Older revision must not overwrite.
let kept = marqdo::workbench::replace_usage_if_newer(&final_u, &partial);
assert_eq!(kept["revision"], 2);
assert_eq!(kept["total_tokens"], 15);
// New attempt is a distinct identity (caller must not merge).
let retry = marqdo::workbench::model_call_usage_from_provider_rev(
"c1",
"a2",
"r1",
&sel,
&serde_json::json!({"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4}),
"succeeded",
1,
);
assert_ne!(retry["attempt_id"], final_u["attempt_id"]);
}
#[test]
fn m02_f07_cancel_does_not_undo_external_write() {
let marker = std::env::temp_dir().join(format!("mq-f07-{}", std::process::id()));
std::fs::write(&marker, "already-written").unwrap();
let tok = std::env::temp_dir().join(format!("mq-f07-cancel-{}", std::process::id()));
std::fs::write(&tok, "cancelled").unwrap();
assert!(marqdo::workbench::cancel_requested(Some(tok.to_str().unwrap())));
// Cancel must not claim or perform undo of prior external writes.
assert_eq!(
std::fs::read_to_string(&marker).unwrap(),
"already-written"
);
let _ = std::fs::remove_file(&marker);
let _ = std::fs::remove_file(&tok);
}
#[test]
fn m02_event_sink_emits_runtime_event() {
let path = std::env::temp_dir().join(format!("mq-evt-{}.ndjson", std::process::id()));
@@ -391,18 +468,74 @@ fn m05_project_cache_isolation_f14() {
#[test]
fn m03_f08_side_effect_marker_not_created_by_view() {
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/executable_knowledge/baseline/02-extract");
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/workbench/fixtures");
let marker = root.join(".workbench-f08-marker");
let _ = std::fs::remove_file(&marker);
let resp = mlsp(
"view.document",
serde_json::json!({
"project": root.to_string_lossy(),
"document": "main.mq.md",
"mode": "read"
"document": "f08_write_side_effect.mq.md",
"mode": "read",
"include": ["fragment", "diagnostics"]
}),
);
assert_eq!(resp["ok"], true, "{resp}");
assert!(!marker.exists(), "view.document must not create side-effect markers");
assert_eq!(resp["result"]["execution"]["state"], "not_run");
assert!(
!marker.exists(),
"view.document must not execute file.write side effects"
);
}
#[test]
fn m03_f09_outline_nodes_chinese_and_static_a11y_subset() {
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/executable_knowledge/baseline/02-extract");
let resp = mlsp(
"view.document",
serde_json::json!({
"project": root.to_string_lossy(),
"document": "main.mq.md",
"mode": "read",
"include": ["fragment", "outline", "nodes", "refs"]
}),
);
assert_eq!(resp["ok"], true, "{resp}");
let doc = &resp["result"];
let outline = doc["outline"].as_array().unwrap();
assert!(outline.iter().any(|o| o["kind"] == "heading"));
let nodes = doc["nodes"].as_array().unwrap();
assert!(!nodes.is_empty(), "expected semantic nodes");
// Chinese / CJK labels should round-trip in outline or nodes when present in source.
let blob = format!("{outline:?}{nodes:?}");
assert!(
blob.chars().any(|c| c >= '\u{4e00}'),
"expected CJK in outline/nodes for 02-extract: {blob}"
);
let fragment = doc["render"]["fragment"].as_str().unwrap_or("");
// F12 static HTML subset (full browser → qdagent): semantic article, no script, mq namespace.
assert!(fragment.starts_with("<article") || fragment.contains("mq-document"));
assert!(!fragment.contains("<script"));
assert!(!fragment.contains("javascript:"));
}
#[test]
fn m04_f11_ambiguous_same_name() {
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/executable_knowledge/baseline/08-same-name");
assert!(root.is_dir(), "missing 08-same-name fixture");
// a.mq.md and b.mq.md both define 商品小计 — omit document_id so both stay in scope.
let resp = mlsp(
"view.reference_resolve",
serde_json::json!({
"project": root.to_string_lossy(),
"project_id": root.to_string_lossy(),
"artifact_id": "商品小计"
}),
);
assert_eq!(resp["ok"], true, "{resp}");
assert_eq!(resp["result"]["resolution"], "ambiguous", "{resp}");
let cands = resp["result"]["candidates"].as_array().unwrap();
assert!(cands.len() >= 2, "{resp}");
}