diff --git a/README.md b/README.md index 2b1231d..8a38b73 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,25 @@ rules are [`RFC-LANG-0002`](docs/architecture/RFC-LANG-0002-unified-semantics.md * **Provider package manifest v1 (RFC-CM CM-A09)** — local `marqdo.provider` manifest installs a trusted C ABI provider; no registry/signature/CLI install (ADR-0037). +* **Third-party Resource Provider v2 through the normal CLI (RFC-EXT-0001 + PR-03)** — a `provider.json` v2 bundle with a targeted, sha256-verified + `c-abi-1` artifact is selected with repeatable `--provider` and authorized by + the exact `--allow-provider` name on `run`/`call`; the existing + `marqdo-provider-worker` transport is spawned only after every manifest, + artifact, grant and duplicate-kind check passes. `check`/`inspect` never load + providers, `pure`/`strict` refuse them, and the worker transport is + linux/x86_64-only in this release (ADR-0055, + [`spec/resource-provider-v2.md`](spec/resource-provider-v2.md)). +* **Single callable binding through the frozen composition root + (RFC-EXT-0001 PR-01/02)** — one `CallableRegistry` and one binding pass feed + `check`/`run`/`call`/`inspect`/Analysis; the runtime dispatches by + `CallSiteKey` and never re-resolves a name or guesses a plugin. Static plugin + descriptors must reconcile one-to-one with the loaded exports (ADR-0054, + [`spec/callable-resolution.md`](spec/callable-resolution.md)). +* **Official stdlib public aliases (RFC-EXT-0001 PR-06)** — `打印`/`print` are + verified `stdlib.json` metadata bound to `io.mq.md::写值`; the + `print_override`/AST-scan/runtime special case is removed with byte-identical + output (ADR-0058). * **C ABI ResourceProvider v1 (RFC-CM CM-A09)** — trusted native shared libraries expose resource open/read/write/close through a frozen header; project + name grant; plugin-Worker binding remains open (ADR-0036). @@ -358,6 +377,115 @@ marqdo benchmark run . marqdo tool remove benchmark ``` +### Extending Marqdo without modifying or rebuilding the CLI + +RFC-EXT-0001 defines five extension kinds. Each one is discovered only through +an explicit manifest/flag, is versioned, and is authorized separately from +being visible. A physically separate repository exercising all five against a +frozen binary is `marqdo-extension-conformance` +(`https://git.cflmy.cn/cflmy/marqdo-extension-conformance`). + +| Kind | Static discovery | Execution | Grant | +|---|---|---|---| +| `.mq.md` package | `marqdo.yaml` dependency + `marqdo.lock` | Marqdo interpreter | project mode | +| Callable plugin | `plugin.yaml` ABI v1 exports | worker (default) or trusted native | `--plugin` + `--allow-plugin NAME` | +| Foreign kernel | `kernel.json` + pinned executable digest | one-shot child process | `--kernel` + `--allow-foreign` | +| Resource provider | `provider.json` schema v2 | `marqdo-provider-worker` + C ABI | `--provider` + `--allow-provider NAME` | +| CLI Tool | `tool.json` install record | independent CLI process | tool installation, no project grants | + +```bash +# 1. Pure Markdown library from a pinned Git dependency +marqdo lock && marqdo sync +marqdo check consumer/index.mq.md +marqdo run consumer/index.mq.md + +# 2. Callable ABI plugin (worker transport by default) +marqdo run plugin/demo.mq.md \ + --plugin plugin/build --allow-plugin demo.ext + +# 3. Foreign kernel with a language label the CLI has never heard of +marqdo run kernel/demo.mq.md \ + --kernel kernel/build/kernel.json --allow-foreign + +# 4. Resource provider v2 (see the next section) +marqdo run provider/demo.mq.md \ + --provider provider/build/provider.json --allow-provider demo.sensor + +# 5. Independent CLI Tool consuming only public Analysis v2 +marqdo tool install tool/tool.json +marqdo inspect-ext tool/demo.mq.md +``` + +#### One callable resolution, visible in `check` and `run` + +Calls are bound statically once per prepared program (ADR-0054, normative +details in [`spec/callable-resolution.md`](spec/callable-resolution.md)): + +* `f(...)`: local callable value → official alias/reserved builtin → current + namespace → canonical name → unique local name; +* `alias.f(...)`: explicit import alias → builtin dotted name → exact plugin + export; +* `some_value.f(...)`: dynamic language value only. + +`check` and `run` consume the same binding; an unknown export, wrong arguments +or a duplicate export fails before execution. An import alias that collides +with a builtin/plugin dotted namespace (for example importing a module as +`math`) is rejected with `ext.import_namespace_conflict`; rename it. A plugin +manifest is the static contract: `run` verifies the loaded artifact and its +exports/signatures match the manifest exactly +(`ext.plugin_binding_mismatch` otherwise) and never synthesises a signature. + +#### Official stdlib aliases, no print magic + +`打印`/`print` are declared in the verified `stdlib/stdlib.json` +(`public_aliases`, covered by `aliases_digest`) and bound as ordinary +`Marqdo` calls to `io.mq.md::写值` (ADR-0058). There is no +`Program.print_override`, no AST-wide print scan and no runtime print branch; +output bytes, permissions and Trace behavior are unchanged, and a program that +never uses the alias does not load the stdlib module graph. Third-party +packages cannot claim `打印`/`print` or `std.*` names. + +### Third-party resource providers (`provider.json` v2) + +An external repository adds a new resource kind (for example `demo.sensor`) +with a `provider.json` schema-version-2 bundle and a targeted `c-abi-1` shared +library. No CLI rebuild and no core change are needed: select the manifest with +repeatable `--provider` and authorize it by its exact manifest `name` with +`--allow-provider` (also repeatable). A `--provider` without a matching +`--allow-provider` is `permission.denied` and the helper is never started; +wildcards (`*`) are not accepted. + +```bash +marqdo run index.mq.md \ + --provider vendor/demo-sensor/provider.json \ + --allow-provider demo.sensor + +marqdo call index.mq.md main \ + --provider vendor/demo-sensor/provider.json \ + --allow-provider demo.sensor +``` + +The document still uses the ordinary resource surface, e.g. +`resource.open("demo.sensor", "unit-1", ["read"])`; provider kinds do not become +new callables. + +* `check`/`inspect` do not accept these options and never load a provider. +* `--mode pure` and `--mode strict` reject `--provider`/`--allow-provider` in + preflight; strict L1 stays fail-closed and does not silently switch to + `project`. +* Everything is verified before any worker starts: `schema_version: 2`, + `abi: c-abi-1`, `transport: worker`, strict SemVer, non-reserved kinds, the + current target artifact's bundle containment and sha256, exact grants and + duplicate provider names/kinds. +* The helper `marqdo-provider-worker` must ship next to the `marqdo` executable + for the same target (no `PATH` fallback); a missing helper is + `ext.worker_unavailable`. The worker transport is linux/x86_64-only in this + release; other platforms fail with `ext.platform_unsupported`. +* The helper is a trusted process, not an OS sandbox: it runs with the user's + privileges. `sha256` is an integrity check, not publisher authentication. + See [`spec/resource-provider-v2.md`](spec/resource-provider-v2.md) and + [ADR-0055](docs/adr/0055-resource-provider-cli-v2.md). + ## References * [CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/) · diff --git a/crates/cli/src/analysis.rs b/crates/cli/src/analysis.rs index a1cfac6..08ed436 100644 --- a/crates/cli/src/analysis.rs +++ b/crates/cli/src/analysis.rs @@ -536,7 +536,7 @@ fn expr_calls( ) { match expr { Expr::Call { callee, args, span } => { - let (state, target) = resolve_callee(project, module, callee); + let (state, target) = resolve_callee(project, module, function, *span); calls.push(json!({ "function": function.id.to_string(), "span": span_json(*span), @@ -572,42 +572,34 @@ fn expr_calls( } } +/// Project the authoritative call binding into the Analysis `state`/`target` +/// fields. +/// +/// Analysis is a read-only projection of the same prepared binding the runtime +/// consumes (RFC-EXT-0001 §4/§9): there is no second name-resolution algorithm +/// here. `state` keeps its v1 vocabulary (`function`, `builtin`, `plugin`, +/// `unresolved`); official stdlib aliases such as `打印` therefore appear as +/// `function` targets because that is the implementation that actually runs. fn resolve_callee( project: &LoadedProject, - module: &Module, - callee: &Expr, + _module: &Module, + function: &Function, + span: Span, ) -> (&'static str, Value) { - match callee { - Expr::Ident(ident) => { - if marqdo_ir::builtins::find(&ident.name).is_some() { - ("builtin", json!(ident.name)) - } else if let Some(function) = module.function(&ident.name) { - ("function", json!(function.id.to_string())) - } else { - ("unresolved", Value::Null) - } + let key = marqdo_ir::call_binding::CallSiteKey::new(function.id.clone(), span); + match project.program.call_bindings.get(&key) { + Some(marqdo_ir::call_binding::BoundTarget::Builtin { canonical_name }) => { + ("builtin", json!(canonical_name)) } - Expr::Member { base, name, .. } => { - if let Expr::Ident(alias) = base.as_ref() { - let qualified = format!("{}.{}", alias.name, name); - if marqdo_ir::builtins::find(&qualified).is_some() { - return ("builtin", json!(qualified)); - } - if let Some(target_module) = project.program.resolve_import(&module.id, &alias.name) - { - if let Some(target) = project.program.module(target_module) { - if let Some(function) = target.function(name) { - return ("function", json!(function.id.to_string())); - } - } - } - if project.program.external_callable(&qualified).is_some() { - return ("plugin", json!(qualified)); - } - } + Some(marqdo_ir::call_binding::BoundTarget::Marqdo { function }) => { + ("function", json!(function.to_string())) + } + Some(marqdo_ir::call_binding::BoundTarget::AbiPlugin { export, .. }) => { + ("plugin", json!(export)) + } + Some(marqdo_ir::call_binding::BoundTarget::DynamicValue) | None => { ("unresolved", Value::Null) } - _ => ("unresolved", Value::Null), } } diff --git a/crates/cli/src/commands.rs b/crates/cli/src/commands.rs index 77a6dac..f0003da 100644 --- a/crates/cli/src/commands.rs +++ b/crates/cli/src/commands.rs @@ -27,148 +27,42 @@ pub use sandbox::SandboxSettings; pub use tool::{fmt, plugin_inspect}; use crate::args; -use crate::loader::{self, FileDiagnostic, LoadedProject}; +use crate::extension_descriptors::{self, StaticPluginSet}; +use crate::loader::{FileDiagnostic, LoadedProject}; use crate::output; +use crate::project_prepare::{self, PrepareOptions, PreparedProgram}; -fn load_project(file: Option<&Path>, json: bool) -> Result { - let file = file.unwrap_or_else(|| Path::new("index.mq.md")); - match loader::load(file) { - Ok(project) => Ok(project), +/// Prepare one document through the single composition-root step. +/// +/// `check`/`run`/`call`/`inspect` must all use this function so they share the +/// same call bindings and diagnostics (RFC-EXT-0001 §4.4, §18.2). +pub(super) fn prepare_project( + file: Option<&Path>, + json: bool, + plugins: &[PathBuf], +) -> Result { + let options = PrepareOptions { + plugins: plugins.to_vec(), + }; + match project_prepare::prepare(file, &options) { + Ok(prepared) => Ok(prepared), Err(error) => { if json { - output::print_json(&json!({"success": false, "error": error})); + output::print_json(&json!({ + "success": false, + "error": { + "code": error.code, + "message": error.message, + } + })); } else { - eprintln!("error: {error}"); - } - if error.contains("marqdo.lock") || error.contains("lockfile") { - Err(1) - } else { - Err(2) + eprintln!("error[{}]: {}", error.code, error.message); } + Err(error.exit_code) } } } -/// Add plugin export descriptors from manifests without loading a library. -fn add_static_plugin_descriptors( - project: &mut LoadedProject, - paths: &[PathBuf], -) -> Result<(), String> { - for path in paths { - let manifest_path = if path.is_dir() { - path.join("plugin.yaml") - } else { - path.clone() - }; - let manifest = marqdo_plugin_abi::PluginManifest::load(&manifest_path) - .map_err(|error| format!("{}: {}", error.code, error.message))?; - manifest - .validate() - .map_err(|error| format!("{}: {}", error.code, error.message))?; - let digest = manifest - .resolve_artifact(&marqdo_plugin_abi::HostTarget::current()) - .map_err(|error| format!("{}: {}", error.code, error.message))? - .sha256 - .clone(); - for export in &manifest.plugin.exports { - if project - .program - .external_callables - .iter() - .any(|callable| callable.public_name == export.name) - { - return Err(format!( - "plugin export `{}` is declared more than once", - export.name - )); - } - let signature = plugin_signature(&export.signature).map_err(|error| { - format!( - "plugin export `{}` has an invalid signature: {error}", - export.name - ) - })?; - project - .program - .external_callables - .push(marqdo_ir::callable::CallableDescriptor { - id: marqdo_ir::callable::CallableId(format!( - "plugin:{}:{}", - manifest.plugin.name, export.name - )), - public_name: export.name.clone(), - owner_package: manifest.plugin.name.clone(), - signature, - effects_requested: manifest.plugin.requires.clone(), - origin: marqdo_ir::callable::CallableOrigin::AbiPlugin { - package: manifest.plugin.name.clone(), - export: export.name.clone(), - }, - implementation_digest: Some(digest.clone()), - source_span: None, - }); - } - } - if paths.is_empty() { - return Ok(()); - } - let existing: Vec<(Option, String)> = project - .diagnostics - .iter() - .map(|file| { - ( - file.diagnostic.span, - format!("{}:{}", file.module_id, file.diagnostic.message), - ) - }) - .collect(); - for (module_id, diagnostic) in marqdo_frontend::calls::validate_program(&project.program) { - let key = format!("{}:{}", module_id, diagnostic.message); - if existing.contains(&(diagnostic.span, key)) { - continue; - } - if let Some(file) = project.files.get(&module_id) { - project.diagnostics.push(FileDiagnostic { - module_id, - path: file.path.clone(), - source: file.source.clone(), - diagnostic, - }); - } - } - Ok(()) -} - -/// Convert a plugin manifest signature (`($a: Int) -> Int`) to the language -/// parameter list, dropping the display-only return annotation and `$` sigils. -fn plugin_signature(raw: &str) -> Result { - let before_return = raw.split("->").next().unwrap_or(raw).trim(); - let open = before_return - .find('(') - .ok_or_else(|| "missing `(`".to_string())?; - let close = before_return - .rfind(')') - .ok_or_else(|| "missing `)`".to_string())?; - if close < open { - return Err("malformed parameter list".to_string()); - } - let params = before_return[open + 1..close].replace('$', ""); - let parsed = - marqdo_frontend::signature::parse(&format!("({params})"), marqdo_span::Span::default()); - if parsed.has_errors() { - return Err(parsed - .diagnostics - .iter() - .map(|diagnostic| diagnostic.message.clone()) - .collect::>() - .join("; ")); - } - Ok(marqdo_ir::callable::CallableSignature { - params: parsed.params, - kind: marqdo_ir::SignatureKind::Explicit, - }) -} - /// `marqdo check`. fn validate_refs(project: &LoadedProject) -> Vec { let mut diagnostics = Vec::new(); diff --git a/crates/cli/src/commands/check.rs b/crates/cli/src/commands/check.rs index b4dc8f9..eb3af82 100644 --- a/crates/cli/src/commands/check.rs +++ b/crates/cli/src/commands/check.rs @@ -9,18 +9,11 @@ pub fn check( plugins: &[PathBuf], emit: Option<&Path>, ) -> i32 { - let mut project = match load_project(file, json) { - Ok(project) => project, + let prepared = match prepare_project(file, json, plugins) { + Ok(prepared) => prepared, Err(code) => return code, }; - if let Err(error) = add_static_plugin_descriptors(&mut project, plugins) { - if json { - output::print_json(&json!({"success": false, "error": error})); - } else { - eprintln!("error: {error}"); - } - return 2; - } + let mut project = prepared.loaded; if refs { let ref_diagnostics = validate_refs(&project); project.diagnostics.extend(ref_diagnostics); diff --git a/crates/cli/src/commands/inspect.rs b/crates/cli/src/commands/inspect.rs index e9ee715..d1665ce 100644 --- a/crates/cli/src/commands/inspect.rs +++ b/crates/cli/src/commands/inspect.rs @@ -20,10 +20,11 @@ pub struct InspectFlags { } pub fn inspect(file: Option<&Path>, flags: &InspectFlags, json: bool) -> i32 { - let project = match load_project(file, json) { - Ok(project) => project, + let prepared = match prepare_project(file, json, &[]) { + Ok(prepared) => prepared, Err(code) => return code, }; + let project = prepared.loaded; let entry = project.entry_file(); let module = &entry.analysis.module; let selected_any = diff --git a/crates/cli/src/commands/run.rs b/crates/cli/src/commands/run.rs index 8b46fde..4a2871f 100644 --- a/crates/cli/src/commands/run.rs +++ b/crates/cli/src/commands/run.rs @@ -4,6 +4,7 @@ //! runtime helpers, but keeps the run session out of the command table. use super::*; +use crate::provider_manifest; use marqdo_sandbox::StrictChildState; pub fn execute( @@ -15,18 +16,12 @@ pub fn execute( trace: bool, ) -> i32 { let command_name = if function.is_some() { "call" } else { "run" }; - let mut project = match load_project(file, json) { - Ok(project) => project, + let prepared = match prepare_project(file, json, &settings.plugins) { + Ok(prepared) => prepared, Err(code) => return code, }; - if let Err(error) = add_static_plugin_descriptors(&mut project, &settings.plugins) { - if json { - output::print_json(&json!({"success": false, "error": error})); - } else { - eprintln!("error: {error}"); - } - return 2; - } + let project = prepared.loaded; + let static_plugins = prepared.plugins; if project.has_errors() { if json { output::print_json(&json!({ @@ -136,10 +131,80 @@ pub fn execute( return 2; } } - for plugin_path in &settings.plugins { - if let Err(message) = - load_plugin_into_host(&mut host, plugin_path, settings.trust_native_plugin) + // T6: verify every selected artifact/path/hash before starting a worker or + // loading native code. Static path/signature checks happened in `prepare`. + // + // Provider selection is host assembly, not callable semantics: `check` + // never sees it. Parse and verify every `--provider` manifest, resolve the + // current target artifact, check the exact grant and reject duplicate + // names/kinds *before* any provider or plugin worker process is spawned + // (RFC-EXT-0001 §5.4.6/§18.2 T6–T7). + let providers = match provider_manifest::read_provider_manifests(&settings.providers) { + Ok(providers) => providers, + Err(error) => { + emit_prepare_error(json, command_name, &error); + return error.exit_code; + } + }; + // `--provider` alone never authorizes; the exact `--allow-provider NAME` + // is required before the helper process may be started. An ungranted + // provider fails closed instead of being silently ignored. + for provider in &providers { + if !settings + .allow_provider + .iter() + .any(|grant| grant == &provider.name) { + let error = extension_descriptors::PrepareError::load( + "permission.denied", + format!( + "resource provider `{}` was selected with --provider but is not granted; \ + add --allow-provider {}", + provider.name, provider.name + ), + ); + emit_prepare_error(json, command_name, &error); + return error.exit_code; + } + } + // A grant that matches no selected provider is a deterministic option + // error: providers are never discovered from PATH, the working directory or + // the network. + for grant in &settings.allow_provider { + if !providers.iter().any(|provider| &provider.name == grant) { + let error = extension_descriptors::PrepareError::option( + "ext.provider_grant_unmatched", + format!( + "--allow-provider `{grant}` matches no selected --provider manifest; \ + providers are never discovered automatically" + ), + ); + emit_prepare_error(json, command_name, &error); + return error.exit_code; + } + } + for plugin in &static_plugins.plugins { + if let Err(error) = extension_descriptors::verify_bundle_regular_file( + &plugin.base_dir, + &plugin.artifact_relative, + ) { + emit_prepare_error(json, command_name, &error); + return error.exit_code; + } + let artifact = plugin.artifact_path(); + if let Err(error) = extension_descriptors::verify_sha256(&artifact, &plugin.artifact_sha256) + { + emit_prepare_error(json, command_name, &error); + return error.exit_code; + } + } + // T7: start only selected, granted and verified extensions. + for plugin in &static_plugins.plugins { + if let Err(message) = load_plugin_into_host( + &mut host, + &plugin.manifest_path, + settings.trust_native_plugin, + ) { if json { output::print_json(&json!({"success": false, "error": message})); } else { @@ -148,36 +213,24 @@ pub fn execute( return 1; } } - // Expose the complete export names loaded from trusted manifests so the - // runtime resolves exactly those names instead of probing namespaces. - for (name, _signature, digest, package) in host.plugin_exports() { - if project - .program - .external_callables - .iter() - .any(|callable| callable.public_name == name) - { - continue; + // T8: exact one-to-one reconciliation with the static descriptors. Never + // synthesise a missing signature (`params: Vec::new()`) or guess an export. + if let Err(message) = reconcile_plugin_exports(&static_plugins, &host) { + emit_error(json, command_name, "ext.plugin_binding_mismatch", &message); + return 1; + } + // T7b: start only granted and verified provider helpers. Every manifest, + // artifact, target, grant and duplicate-kind check above already passed, so + // no worker is spawned for a provider that failed verification. + for provider in &providers { + if let Err(error) = provider_manifest::load_resource_provider_into_host( + &mut host, + provider, + &settings.allow_provider, + ) { + emit_prepare_error(json, command_name, &error); + return error.exit_code; } - project - .program - .external_callables - .push(marqdo_ir::callable::CallableDescriptor { - id: marqdo_ir::callable::CallableId(format!("plugin:{package}:{name}")), - public_name: name.clone(), - owner_package: package.clone(), - signature: marqdo_ir::callable::CallableSignature { - params: Vec::new(), - kind: marqdo_ir::SignatureKind::Explicit, - }, - effects_requested: vec!["host".to_string()], - origin: marqdo_ir::callable::CallableOrigin::AbiPlugin { - package, - export: name, - }, - implementation_digest: Some(digest), - source_span: None, - }); } let function_name = function.unwrap_or("main").to_string(); let entry_module = &project.program.modules[&project.entry]; @@ -292,6 +345,80 @@ pub fn execute( } } +fn emit_prepare_error( + json: bool, + command_name: &str, + error: &crate::extension_descriptors::PrepareError, +) { + if json { + output::print_json(&json!({ + "command": command_name, + "success": false, + "error": {"code": error.code, "message": error.message}, + })); + } else { + eprintln!("error[{}]: {}", error.code, error.message); + } +} + +/// Compare the Host's actual plugin exports with the static manifest set. +/// +/// Names, raw signatures, owners and verified artifact digests must match +/// exactly; extra, missing or altered exports fail closed with +/// `ext.plugin_binding_mismatch` and the main function never runs. +fn reconcile_plugin_exports( + static_plugins: &StaticPluginSet, + host: &StdHost, +) -> Result<(), String> { + let mut expected: std::collections::BTreeMap = + std::collections::BTreeMap::new(); + for plugin in &static_plugins.plugins { + for export in &plugin.exports { + expected.insert( + export.name.clone(), + ( + export.signature.clone(), + plugin.artifact_sha256.clone(), + plugin.package.clone(), + ), + ); + } + } + let loaded = host.plugin_exports(); + if loaded.len() != expected.len() { + return Err(format!( + "static plugin descriptors declare {} export(s) but the runtime loaded {}; \ + refusing to execute", + expected.len(), + loaded.len() + )); + } + for (name, signature, digest, package) in loaded { + let Some((expected_signature, expected_digest, expected_package)) = expected.get(&name) + else { + return Err(format!( + "runtime export `{name}` was not declared by any selected plugin manifest" + )); + }; + if &signature != expected_signature { + return Err(format!( + "export `{name}` signature mismatch: manifest `{expected_signature}`, runtime `{signature}`" + )); + } + if &digest != expected_digest { + return Err(format!( + "export `{name}` artifact digest mismatch: manifest `{expected_digest}`, runtime `{digest}`" + )); + } + if &package != expected_package { + return Err(format!( + "export `{name}` owner mismatch: manifest `{expected_package}`, runtime `{package}`" + )); + } + } + Ok(()) +} + /// Read piped stdin once at the composition boundary. /// /// A terminal stdin is not read, so `marqdo run` never blocks waiting for an @@ -310,3 +437,87 @@ fn read_stdin() -> Vec { } bytes } + +#[cfg(test)] +mod tests { + use super::*; + + fn empty_host() -> StdHost { + StdHost::new( + SandboxMode::Project, + marqdo_host::Capabilities::pure(), + std::env::temp_dir(), + ) + } + + fn static_plugin_with_export() -> StaticPluginSet { + let descriptor = marqdo_ir::callable::CallableDescriptor { + id: marqdo_ir::callable::CallableId("plugin:demo.ext:solver.add".into()), + public_name: "solver.add".into(), + owner_package: "demo.ext".into(), + signature: marqdo_ir::callable::CallableSignature { + params: vec![ + marqdo_ir::Param { + name: "a".into(), + span: Span::default(), + type_annotation: None, + default: None, + }, + marqdo_ir::Param { + name: "b".into(), + span: Span::default(), + type_annotation: None, + default: None, + }, + ], + kind: marqdo_ir::SignatureKind::Explicit, + }, + effects_requested: vec!["math.compute".into()], + origin: marqdo_ir::callable::CallableOrigin::AbiPlugin { + package: "demo.ext".into(), + export: "solver.add".into(), + }, + implementation_digest: Some("a".repeat(64)), + source_span: None, + }; + StaticPluginSet { + plugins: vec![crate::extension_descriptors::StaticPlugin { + manifest_path: PathBuf::from("/fixtures/demo/plugin.yaml"), + base_dir: PathBuf::from("/fixtures/demo"), + package: "demo.ext".into(), + version: "0.1.0".into(), + kind: marqdo_plugin_abi::PluginKind::Worker, + manifest_sha256: format!("sha256:{}", "b".repeat(64)), + artifact_target: marqdo_plugin_abi::HostTarget::current() + .triple() + .to_string(), + artifact_sha256: "a".repeat(64), + artifact_relative: "build/plugin.so".into(), + exports: vec![crate::extension_descriptors::StaticPluginExport { + name: "solver.add".into(), + signature: "($a: Int, $b: Int) -> Int".into(), + descriptor, + }], + requires: Vec::new(), + }], + } + } + + #[test] + fn reconcile_accepts_matching_empty_sets() { + assert!( + reconcile_plugin_exports(&StaticPluginSet::default(), &empty_host()).is_ok(), + "no selected plugin and no loaded plugin must reconcile" + ); + } + + #[test] + fn reconcile_rejects_missing_runtime_exports() { + let error = reconcile_plugin_exports(&static_plugin_with_export(), &empty_host()) + .expect_err("a declared export that the runtime did not load must fail closed"); + assert!( + error.contains("declare 1 export(s)") || error.contains("not declared"), + "{error}" + ); + } +} diff --git a/crates/cli/src/commands/sandbox.rs b/crates/cli/src/commands/sandbox.rs index 20ae4af..382032f 100644 --- a/crates/cli/src/commands/sandbox.rs +++ b/crates/cli/src/commands/sandbox.rs @@ -43,6 +43,10 @@ pub struct SandboxSettings { pub timeout: Option, /// Explicitly trusted plugin manifests to load. pub plugins: Vec, + /// Explicitly selected resource provider v2 manifests (RFC-EXT-0001 §5.4). + pub providers: Vec, + /// Exact resource provider name grants (declaration is not a grant). + pub allow_provider: Vec, /// Explicit trust decision for in-process native plugin loading. pub trust_native_plugin: bool, /// Optional Trace v2 output path. @@ -68,8 +72,8 @@ impl SandboxSettings { /// Reject strict-mode requests that this L1 isolation level does not /// support. Strict currently covers pure Marqdo and native Math only; - /// granting Foreign, ABI plugins, tools or trace files would require - /// extending the supervisor contract first. + /// granting Foreign, ABI plugins, resource providers, tools or trace files + /// would require extending the supervisor contract first. pub fn strict_preflight(&self) -> Result<(), String> { let requested = self.allow_foreign || !self.allow_read.is_empty() @@ -80,6 +84,8 @@ impl SandboxSettings { || self.allow_random || !self.allow_plugin.is_empty() || !self.plugins.is_empty() + || !self.providers.is_empty() + || !self.allow_provider.is_empty() || self.trust_native_plugin || !self.kernels.is_empty() || self.trace_file.is_some(); @@ -87,8 +93,9 @@ impl SandboxSettings { return Err( "strict mode currently supports pure Marqdo and native Math only; \ --allow-read/--allow-write/--allow-env/--allow-process/--allow-random/\ - --allow-foreign/--allow-plugin/--plugin/--trust-native-plugin/--kernel/--trace-file are - rejected. Use --mode project for those capabilities." + --allow-foreign/--allow-plugin/--plugin/--provider/--allow-provider/\ + --trust-native-plugin/--kernel/--trace-file are rejected. \ + Use --mode project for those capabilities." .to_string(), ); } @@ -108,9 +115,13 @@ impl SandboxSettings { || self.allow_random || self.allow_foreign || !self.allow_plugin.is_empty() + || !self.providers.is_empty() + || !self.allow_provider.is_empty() { return Err( - "pure mode prohibits external I/O grants; use --mode project".to_string(), + "pure mode prohibits external I/O grants and extension providers; \ + use --mode project" + .to_string(), ); } Ok(Capabilities::pure()) @@ -141,6 +152,10 @@ impl SandboxSettings { for plugin in &self.allow_plugin { capabilities = capabilities.allow_plugin(plugin.clone()); } + for provider in &self.allow_provider { + validate_provider_grant(provider)?; + capabilities = capabilities.allow_resource_provider(provider.clone()); + } Ok(capabilities) } SandboxMode::Strict => { @@ -150,3 +165,28 @@ impl SandboxSettings { } } } + +/// Reject non-exact `--allow-provider` grants. +/// +/// Provider grants are exact stable names (RFC-EXT-0001 §5.2): there is no `*` +/// wildcard, no prefix matching and no implicit discovery, so a grant either +/// names a selected manifest or the run fails closed. +fn validate_provider_grant(name: &str) -> Result<(), String> { + if name.is_empty() { + return Err("--allow-provider requires a non-empty provider name".to_string()); + } + if name + .chars() + .any(|character| matches!(character, '*' | '?' | '[' | ']')) + { + return Err(format!( + "--allow-provider `{name}`: wildcards are not supported; grants are exact provider names" + )); + } + if name.chars().any(char::is_control) { + return Err(format!( + "--allow-provider `{name}`: provider names must not contain control characters" + )); + } + Ok(()) +} diff --git a/crates/cli/src/extension_descriptors.rs b/crates/cli/src/extension_descriptors.rs new file mode 100644 index 0000000..286cdb8 --- /dev/null +++ b/crates/cli/src/extension_descriptors.rs @@ -0,0 +1,414 @@ +//! Static extension descriptors for the CLI composition root +//! (RFC-EXT-0001 §4.4/§6, ADR-0054/0056). +//! +//! This module only *reads* manifests. It never `dlopen`s a library, never +//! spawns a worker, never opens a network connection and never executes a +//! side-effect script. The descriptors produced here feed the single +//! [`marqdo_ir::callable::CallableRegistry`] and binding pass; `run` later +//! verifies the real artifacts against these recorded identities. + +use std::collections::BTreeSet; +use std::path::{Component, Path, PathBuf}; + +use marqdo_ir::callable::{CallableDescriptor, CallableId, CallableOrigin, CallableSignature}; +use marqdo_ir::SignatureKind; +use marqdo_plugin_abi::{HostTarget, PluginKind, PluginManifest}; +use marqdo_span::Span; +use sha2::{Digest, Sha256}; + +/// A structured prepare-stage failure. +#[derive(Debug, Clone)] +pub struct PrepareError { + /// Stable diagnostic code (`ext.*` unless reusing an existing event code). + pub code: String, + /// Human-readable message; never contains secrets or binary content. + pub message: String, + /// CLI exit code (2 for manifest/option errors, 1 for load/verify errors). + pub exit_code: i32, +} + +impl PrepareError { + /// Construct a manifest/option error (exit code 2). + pub fn option(code: impl Into, message: impl Into) -> Self { + Self { + code: code.into(), + message: message.into(), + exit_code: 2, + } + } + + /// Construct a loading/verification error (exit code 1). + pub fn load(code: impl Into, message: impl Into) -> Self { + Self { + code: code.into(), + message: message.into(), + exit_code: 1, + } + } +} + +/// One statically declared export. +#[derive(Debug, Clone)] +pub struct StaticPluginExport { + /// Complete dotted export name. + pub name: String, + /// Raw manifest signature string, compared exactly at run time. + pub signature: String, + /// Parsed static descriptor used by the binding pass. + pub descriptor: CallableDescriptor, +} + +/// One statically read plugin package. +/// +/// The identity fields are recorded for `inspect --extensions`, evidence and +/// reconciliation; they are not consulted for name resolution (the binding +/// table is). +#[allow(dead_code)] +#[derive(Debug, Clone)] +pub struct StaticPlugin { + /// Canonical manifest path. + pub manifest_path: PathBuf, + /// Package directory (artifact root). + pub base_dir: PathBuf, + /// Declared package name. + pub package: String, + /// Declared package version. + pub version: String, + /// Deployment kind (`native`, `worker`, `wasm`). + pub kind: PluginKind, + /// `sha256:` of the manifest file bytes. + pub manifest_sha256: String, + /// Target triple selected for this host. + pub artifact_target: String, + /// Declared (not yet verified) artifact sha256, lowercase hex. + pub artifact_sha256: String, + /// Bundle-relative artifact path. + pub artifact_relative: String, + /// Exports in manifest order. + pub exports: Vec, + /// Declared capability requests (metadata only; never grants). + pub requires: Vec, +} + +impl StaticPlugin { + /// Absolute artifact path inside the package bundle. + pub fn artifact_path(&self) -> PathBuf { + self.base_dir.join(&self.artifact_relative) + } +} + +/// Static plugin selection read from explicit CLI `--plugin` flags. +#[derive(Debug, Clone, Default)] +pub struct StaticPluginSet { + /// Packages in deterministic manifest order. + pub plugins: Vec, +} + +impl StaticPluginSet { + /// All export descriptors in deterministic order. + pub fn descriptors(&self) -> Vec { + let mut descriptors = Vec::new(); + for plugin in &self.plugins { + for export in &plugin.exports { + descriptors.push(export.descriptor.clone()); + } + } + descriptors + } +} + +/// Read and validate all explicitly selected `--plugin` manifests. +/// +/// No binary is opened or executed. Duplicate package identities, duplicate +/// full export names, reserved names and malformed signatures are rejected +/// here, before any binding is produced. +pub fn read_plugin_descriptors(paths: &[PathBuf]) -> Result { + let mut set = StaticPluginSet::default(); + let mut export_names: BTreeSet = BTreeSet::new(); + let mut packages: BTreeSet<(String, String)> = BTreeSet::new(); + for path in paths { + let manifest_path = if path.is_dir() { + path.join("plugin.yaml") + } else { + path.clone() + }; + let manifest_path = manifest_path.canonicalize().map_err(|error| { + PrepareError::option( + "ext.manifest_unreadable", + format!( + "cannot open plugin manifest `{}`: {error}", + manifest_path.display() + ), + ) + })?; + let base_dir = manifest_path + .parent() + .map(Path::to_path_buf) + .unwrap_or_else(|| PathBuf::from(".")); + let base_dir = base_dir.canonicalize().unwrap_or(base_dir); + let manifest_bytes = std::fs::read(&manifest_path).map_err(|error| { + PrepareError::option( + "ext.manifest_unreadable", + format!( + "cannot read plugin manifest `{}`: {error}", + manifest_path.display() + ), + ) + })?; + let manifest = PluginManifest::load(&manifest_path) + .map_err(|error| PrepareError::option(error.code.clone(), error.message.clone()))?; + manifest + .validate() + .map_err(|error| PrepareError::option(error.code.clone(), error.message.clone()))?; + let host = HostTarget::current(); + let artifact = manifest.resolve_artifact(&host).map_err(|error| { + PrepareError::option("ext.no_matching_artifact", error.message.clone()) + })?; + let artifact_relative = artifact.file.clone(); + ensure_bundle_relative(&artifact_relative).map_err(|error| { + PrepareError::option( + "ext.artifact_integrity", + format!( + "plugin `{}` artifact path `{artifact_relative}` is invalid: {error}", + manifest.plugin.name + ), + ) + })?; + let manifest_sha256 = format!("sha256:{}", hex(&Sha256::digest(&manifest_bytes))); + let identity = (manifest.plugin.name.clone(), manifest_sha256.clone()); + if !packages.insert(identity.clone()) { + // The exact same package declaration twice is idempotent. + continue; + } + if set + .plugins + .iter() + .any(|plugin| plugin.package == identity.0) + { + return Err(PrepareError::option( + "ext.duplicate_callable", + format!( + "plugin package `{}` is selected with two different manifests; \ + explicit aliases are not implemented, so this is rejected", + identity.0 + ), + )); + } + let mut exports = Vec::new(); + for export in &manifest.plugin.exports { + if !export_names.insert(export.name.clone()) { + return Err(PrepareError::option( + "ext.duplicate_callable", + format!( + "plugin export `{}` is declared by more than one selected package", + export.name + ), + )); + } + let signature = plugin_signature(&export.signature).map_err(|error| { + PrepareError::option( + "ext.plugin_invalid_signature", + format!( + "plugin export `{}` has an invalid signature: {error}", + export.name + ), + ) + })?; + let descriptor = CallableDescriptor { + id: CallableId(format!("plugin:{}:{}", manifest.plugin.name, export.name)), + public_name: export.name.clone(), + owner_package: manifest.plugin.name.clone(), + signature, + effects_requested: manifest.plugin.requires.clone(), + origin: CallableOrigin::AbiPlugin { + package: manifest.plugin.name.clone(), + export: export.name.clone(), + }, + implementation_digest: Some(artifact.sha256.clone()), + source_span: None, + }; + exports.push(StaticPluginExport { + name: export.name.clone(), + signature: export.signature.clone(), + descriptor, + }); + } + set.plugins.push(StaticPlugin { + manifest_path, + base_dir, + package: manifest.plugin.name.clone(), + version: manifest.plugin.version.clone(), + kind: manifest.plugin.kind, + manifest_sha256, + artifact_target: host.triple().to_string(), + artifact_sha256: artifact.sha256.clone(), + artifact_relative, + exports, + requires: manifest.plugin.requires.clone(), + }); + } + set.plugins.sort_by(|left, right| { + (&left.package, &left.manifest_sha256).cmp(&(&right.package, &right.manifest_sha256)) + }); + Ok(set) +} + +/// Convert a plugin manifest signature (`($a: Int, $b: Int) -> Int`) to the +/// language parameter list. +/// +/// ABI v1 transports named arguments as a map, so positional arguments are a +/// static-validated convenience: the binding pass normalises them to the +/// manifest order. Default values are rejected until the ABI wire semantics for +/// them are versioned (RFC-EXT-0001 §4.5.3); silently injecting defaults at run +/// time is forbidden. +pub fn plugin_signature(raw: &str) -> Result { + let before_return = raw.split("->").next().unwrap_or(raw).trim(); + let open = before_return + .find('(') + .ok_or_else(|| "missing `(`".to_string())?; + let close = before_return + .rfind(')') + .ok_or_else(|| "missing `)`".to_string())?; + if close < open { + return Err("malformed parameter list".to_string()); + } + let params = before_return[open + 1..close].replace('$', ""); + let parsed = marqdo_frontend::signature::parse(&format!("({params})"), Span::default()); + if parsed.has_errors() { + return Err(parsed + .diagnostics + .iter() + .map(|diagnostic| diagnostic.message.clone()) + .collect::>() + .join("; ")); + } + if parsed.params.iter().any(|param| param.default.is_some()) { + return Err( + "ABI v1 plugin signatures must not declare default values; make parameters required" + .to_string(), + ); + } + Ok(CallableSignature { + params: parsed.params, + kind: SignatureKind::Explicit, + }) +} + +/// Reject absolute paths, `..` components and empty paths. +pub fn ensure_bundle_relative(path: &str) -> Result<(), String> { + let mut normalised = PathBuf::new(); + for component in Path::new(path).components() { + match component { + Component::Normal(part) => normalised.push(part), + Component::CurDir => {} + Component::ParentDir | Component::RootDir | Component::Prefix(_) => { + return Err("must be a relative path inside the package".to_string()) + } + } + } + if normalised.as_os_str().is_empty() { + return Err("must not be empty".to_string()); + } + Ok(()) +} + +/// Verify a bundle file is a regular file below `bundle_root`, travelling +/// through symlinks only if the final canonical target stays inside the bundle. +pub fn verify_bundle_regular_file( + bundle_root: &Path, + relative: &str, +) -> Result { + ensure_bundle_relative(relative).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!("artifact path `{relative}` is invalid: {error}"), + ) + })?; + let root = bundle_root.canonicalize().map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!( + "cannot resolve package root `{}`: {error}", + bundle_root.display() + ), + ) + })?; + let candidate = root.join(relative); + let canonical = candidate.canonicalize().map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!("cannot open artifact `{}`: {error}", candidate.display()), + ) + })?; + if !canonical.starts_with(&root) { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "artifact `{relative}` resolves outside its package bundle (`{}`)", + canonical.display() + ), + )); + } + let metadata = std::fs::metadata(&canonical).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!("cannot stat artifact `{}`: {error}", canonical.display()), + ) + })?; + if !metadata.is_file() { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!("artifact `{}` is not a regular file", canonical.display()), + )); + } + Ok(canonical) +} + +/// Verify a file's lowercase sha256 against a declared value. +pub fn verify_sha256(path: &Path, declared: &str) -> Result<(), PrepareError> { + let bytes = std::fs::read(path).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!("cannot read artifact `{}`: {error}", path.display()), + ) + })?; + let actual = hex(&Sha256::digest(&bytes)); + if actual != declared.to_ascii_lowercase() { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "artifact `{}` sha256 mismatch: declared {declared}, actual {actual}", + path.display() + ), + )); + } + Ok(()) +} + +fn hex(bytes: &[u8]) -> String { + let mut out = String::with_capacity(bytes.len() * 2); + for byte in bytes { + out.push_str(&format!("{byte:02x}")); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn plugin_signature_rejects_defaults_and_parses_positional() { + assert!(plugin_signature("($a: Int, $b: Int) -> Int").is_ok()); + let error = plugin_signature("($a: Int = 1) -> Int").unwrap_err(); + assert!(error.contains("default"), "{error}"); + } + + #[test] + fn bundle_relative_paths_reject_traversal() { + assert!(ensure_bundle_relative("lib/demo.so").is_ok()); + assert!(ensure_bundle_relative("../demo.so").is_err()); + assert!(ensure_bundle_relative("/etc/passwd").is_err()); + assert!(ensure_bundle_relative("").is_err()); + } +} diff --git a/crates/cli/src/loader.rs b/crates/cli/src/loader.rs index 6a13a88..2072935 100644 --- a/crates/cli/src/loader.rs +++ b/crates/cli/src/loader.rs @@ -14,7 +14,8 @@ use std::collections::BTreeMap; use std::path::{Path, PathBuf}; use marqdo_expression::Expr; -use marqdo_frontend::{analyze_source, AnalysisResult}; +use marqdo_frontend::AnalysisResult; +use marqdo_ir::callable::OfficialAlias; use marqdo_ir::{ModuleId, PlaceTarget, Program, ResolvedImport, Stmt}; use marqdo_package::{ discover_root, LockFile, Manifest, PackageResolver, ResolvedPackage, LOCK_FILE, MANIFEST_FILE, @@ -157,6 +158,11 @@ pub struct LoadedProject { pub diagnostics: Vec, /// Verified bundled stdlib identity, when one was available. pub stdlib: Option, + /// Official stdlib public aliases referenced by this program (ADR-0058). + /// + /// These are ordinary `Marqdo` bindings; the runtime has no print special + /// case and no `Program.print_override`. + pub official_aliases: Vec, } impl LoadedProject { @@ -280,32 +286,11 @@ pub fn load(entry: &Path) -> Result { let entry_id = loader.load_module(&entry_module.package, &entry_module.relative_path)?; loader.program.main_module = Some(entry_id.clone()); - // Compat-B (ADR-0030): only programs that actually call 打印/print need the - // stdlib print module; injecting it for every program would change the - // module graph, analysis output and startup cost for no benefit. - if program_uses_print(&loader.program) { - if let Some(stdlib) = loader.stdlib.clone() { - let package = stdlib.package(); - let io = loader.load_module(&package, "io.mq.md")?; - let Some(module) = loader.program.module(&io) else { - return Err("stdlib print module was not loaded".to_string()); - }; - let Some(function) = module.function("写值") else { - return Err("stdlib print module has no `写值` function".to_string()); - }; - loader.program.print_override = Some(function.id.clone()); - } - } - for (module_id, diagnostic) in marqdo_frontend::calls::validate_program(&loader.program) { - if let Some(file) = loader.files.get(&module_id) { - loader.diagnostics.push(FileDiagnostic { - module_id, - path: file.path.clone(), - source: file.source.clone(), - diagnostic, - }); - } - } + // Official stdlib public aliases (ADR-0058): the verified metadata names + // the target module/function; only aliases the program actually references + // load their module, and the runtime binds them as ordinary Marqdo calls. + let official_aliases = loader.load_official_aliases()?; + let stdlib_info = loader.stdlib.as_ref().map(|bundle| StdlibInfo { version: bundle.version().to_string(), digest: bundle.digest().to_string(), @@ -328,6 +313,7 @@ pub fn load(entry: &Path) -> Result { packages: loader.packages, diagnostics: loader.diagnostics, stdlib: stdlib_info, + official_aliases, }) } @@ -378,7 +364,7 @@ impl Loader { .or_insert_with(|| package_root.clone()); let source = std::fs::read_to_string(&path) .map_err(|error| format!("cannot read `{}`: {error}", path.display()))?; - let analysis = analyze_source(&module_id.to_string(), &source); + let analysis = marqdo_frontend::analyze_source_for_program(&module_id.to_string(), &source); let imports: Vec<(String, String, marqdo_span::Span)> = analysis .document .frontmatter() @@ -516,57 +502,170 @@ impl Loader { } Ok(module_id) } -} -/// Whether the loaded program calls the legacy `打印`/`print` builtin. -fn program_uses_print(program: &Program) -> bool { - program.modules.values().any(|module| { - module - .functions - .iter() - .any(|function| stmts_use_print(&function.body)) - }) -} - -fn stmts_use_print(stmts: &[Stmt]) -> bool { - stmts.iter().any(|stmt| match stmt { - Stmt::Assign { target, value, .. } => { - let target_uses = match target { - PlaceTarget::Local { .. } => false, - PlaceTarget::Index { base, index, .. } => { - expr_uses_print(base) || expr_uses_print(index) - } + /// Resolve the official stdlib public aliases referenced by the program. + /// + /// Only the verified `stdlib.json` metadata can declare an official alias; + /// a third-party package cannot claim `打印`/`print` or `std.*`. Modules are + /// loaded only when the program references one of the alias names, so an + /// ordinary program does not grow its module graph or startup cost. + fn load_official_aliases(&mut self) -> Result, String> { + let Some(stdlib) = self.stdlib.clone() else { + return Ok(Vec::new()); + }; + if stdlib.public_aliases().is_empty() { + return Ok(Vec::new()); + } + let referenced = referenced_identifiers(&self.program); + let mut aliases = Vec::new(); + for alias in stdlib.public_aliases() { + if !alias + .public_names + .iter() + .any(|name| referenced.contains(name)) + { + continue; + } + let resolved = stdlib.resolve(&alias.target_module)?; + let module_id = self.load_module(&resolved.package, &resolved.relative_path)?; + let Some(module) = self.program.module(&module_id) else { + return Err(format!( + "stdlib public alias target module `{}` was not loaded", + alias.target_module + )); }; - target_uses || expr_uses_print(value) + let Some(function) = module.function(&alias.target_function) else { + return Err(format!( + "stdlib public alias `{}` target `{}::{}` does not exist", + alias.public_names.join("/"), + alias.target_module, + alias.target_function + )); + }; + if let Some(expected) = &alias.expected_signature { + let parsed = + marqdo_frontend::signature::parse(expected, marqdo_span::Span::default()); + if parsed.has_errors() { + return Err(format!( + "stdlib public alias `{}` has an invalid expected_signature `{expected}`", + alias.public_names.join("/") + )); + } + let expected_names: Vec<&str> = parsed + .params + .iter() + .map(|param| param.name.as_str()) + .collect(); + let actual_names: Vec<&str> = function + .params + .iter() + .map(|param| param.name.as_str()) + .collect(); + if expected_names != actual_names + || function.signature != marqdo_ir::SignatureKind::Explicit + { + return Err(format!( + "stdlib public alias `{}` expected signature `{expected}` does not match \ + target `{}::{}` (params {:?}, kind {:?})", + alias.public_names.join("/"), + alias.target_module, + alias.target_function, + actual_names, + function.signature + )); + } + } + aliases.push(OfficialAlias { + public_names: alias.public_names.clone(), + function: function.id.clone(), + signature: marqdo_ir::callable::CallableSignature { + params: function.params.clone(), + kind: function.signature, + }, + owner_package: module_id.instance.clone(), + effects_requested: alias.effects.clone(), + }); } - Stmt::Eval { expr, .. } | Stmt::Return { value: expr, .. } => expr_uses_print(expr), - Stmt::Branch { arms, .. } => arms - .iter() - .any(|arm| expr_uses_print(&arm.condition) || stmts_use_print(&arm.body)), - Stmt::Loop { - condition, body, .. - } => expr_uses_print(condition) || stmts_use_print(body), - }) + Ok(aliases) + } +} + +/// Collect every identifier referenced by a program. +/// +/// This is a conservative "is this alias used at all?" scan; it only decides +/// whether an official alias target module should be loaded. It does not decide +/// call targets (that is the single binding pass). +fn referenced_identifiers(program: &Program) -> std::collections::BTreeSet { + let mut names = std::collections::BTreeSet::new(); + for module in program.modules.values() { + for function in &module.functions { + for param in &function.params { + if let Some(default) = ¶m.default { + collect_identifiers_expr(default, &mut names); + } + } + collect_identifiers_stmts(&function.body, &mut names); + } + } + names +} + +fn collect_identifiers_stmts(stmts: &[Stmt], names: &mut std::collections::BTreeSet) { + for stmt in stmts { + match stmt { + Stmt::Assign { target, value, .. } => { + if let PlaceTarget::Index { base, index, .. } = target { + collect_identifiers_expr(base, names); + collect_identifiers_expr(index, names); + } + collect_identifiers_expr(value, names); + } + Stmt::Eval { expr, .. } | Stmt::Return { value: expr, .. } => { + collect_identifiers_expr(expr, names) + } + Stmt::Branch { arms, .. } => { + for arm in arms { + collect_identifiers_expr(&arm.condition, names); + collect_identifiers_stmts(&arm.body, names); + } + } + Stmt::Loop { + condition, body, .. + } => { + collect_identifiers_expr(condition, names); + collect_identifiers_stmts(body, names); + } + } + } } -fn expr_uses_print(expr: &Expr) -> bool { +fn collect_identifiers_expr(expr: &Expr, names: &mut std::collections::BTreeSet) { match expr { + Expr::Ident(ident) => { + names.insert(ident.name.clone()); + } + Expr::List(items) => items + .iter() + .for_each(|item| collect_identifiers_expr(item, names)), + Expr::Record(entries) => entries + .iter() + .for_each(|entry| collect_identifiers_expr(&entry.value, names)), + Expr::Unary { operand, .. } => collect_identifiers_expr(operand, names), + Expr::Binary { left, right, .. } => { + collect_identifiers_expr(left, names); + collect_identifiers_expr(right, names); + } + Expr::Member { base, .. } => collect_identifiers_expr(base, names), + Expr::Index { base, index, .. } => { + collect_identifiers_expr(base, names); + collect_identifiers_expr(index, names); + } Expr::Call { callee, args, .. } => { - matches!(callee.as_ref(), Expr::Ident(ident) if matches!(ident.name.as_str(), "打印" | "print")) - || expr_uses_print(callee) - || args.iter().any(|arg| expr_uses_print(&arg.value)) + collect_identifiers_expr(callee, names); + for arg in args { + collect_identifiers_expr(&arg.value, names); + } } - Expr::List(items) => items.iter().any(expr_uses_print), - Expr::Record(entries) => entries.iter().any(|entry| expr_uses_print(&entry.value)), - Expr::Unary { operand, .. } => expr_uses_print(operand), - Expr::Binary { left, right, .. } => expr_uses_print(left) || expr_uses_print(right), - Expr::Member { base, .. } => expr_uses_print(base), - Expr::Index { base, index, .. } => expr_uses_print(base) || expr_uses_print(index), - Expr::Int(_) - | Expr::Float(_) - | Expr::Text(_) - | Expr::Bool(_) - | Expr::None - | Expr::Ident(_) => false, + Expr::Int(_) | Expr::Float(_) | Expr::Text(_) | Expr::Bool(_) | Expr::None => {} } } diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 4a25c35..108953e 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -7,8 +7,11 @@ mod analysis; mod args; mod commands; +mod extension_descriptors; mod loader; mod output; +mod project_prepare; +mod provider_manifest; mod stdlib; mod tool; mod trace; @@ -128,6 +131,16 @@ struct SandboxArgs { /// Explicitly trusted plugin manifest/directory to load (repeatable). #[arg(long = "plugin", value_name = "PATH")] plugin: Vec, + /// Explicitly selected resource provider v2 manifest (repeatable). + /// + /// Selecting a provider never authorizes it; `--allow-provider NAME` is + /// required as well. `check`/`inspect` do not load providers. + #[arg(long = "provider", value_name = "PATH")] + provider: Vec, + /// Grant one selected resource provider by its exact manifest name + /// (repeatable; wildcards are not supported). + #[arg(long = "allow-provider", value_name = "NAME")] + allow_provider: Vec, /// Write a versioned Trace v2 file for this run. #[arg(long = "trace-file", value_name = "PATH")] trace_file: Option, @@ -164,6 +177,8 @@ impl SandboxArgs { max_region_memory: self.max_region_memory, timeout: self.timeout, plugins: self.plugin.clone(), + providers: self.provider.clone(), + allow_provider: self.allow_provider.clone(), trust_native_plugin: self.trust_native_plugin, trace_file: self.trace_file.clone(), trace_max_events: self.trace_max_events, diff --git a/crates/cli/src/project_prepare.rs b/crates/cli/src/project_prepare.rs new file mode 100644 index 0000000..a7c5edd --- /dev/null +++ b/crates/cli/src/project_prepare.rs @@ -0,0 +1,138 @@ +//! The single `prepare` composition-root step (RFC-EXT-0001 §4.4, §18.2). +//! +//! Order of operations (T1–T5 of the RFC): +//! +//! 1. load the Markdown/package graph (read only, no execution); +//! 2. read static extension descriptors (never `dlopen`/spawn); +//! 3. build the unique [`CallableRegistry`] and reject static conflicts; +//! 4. run the single program-wide call binding pass; +//! 5. produce a [`PreparedProgram`] consumed by `check`, `run`, `call`, +//! `inspect` and Analysis. +//! +//! `run` may only assemble the Host and start workers after this returns. + +use std::path::PathBuf; + +use marqdo_ir::callable::CallableRegistry; + +use crate::extension_descriptors::{read_plugin_descriptors, PrepareError, StaticPluginSet}; +use crate::loader::{self, FileDiagnostic, LoadedProject}; + +/// Options that influence static preparation. +#[derive(Debug, Clone, Default)] +pub struct PrepareOptions { + /// Explicit `--plugin` manifests (repeatable, no implicit discovery). + pub plugins: Vec, +} + +/// A statically prepared program: everything `check`/`run`/`inspect` share. +pub struct PreparedProgram { + /// Loaded project graph with authoritative diagnostics. + pub loaded: LoadedProject, + /// Static plugin descriptors (identity only; no binary loaded). + pub plugins: StaticPluginSet, + /// Unique callable registry used by the binding pass. + /// + /// Kept on the prepared value so later consumers (`inspect --extensions`, + /// evidence) can report the same static table without rebuilding it. + #[allow(dead_code)] + pub registry: CallableRegistry, +} + +/// Prepare one document without executing any project or extension code. +pub fn prepare( + file: Option<&std::path::Path>, + options: &PrepareOptions, +) -> Result { + // T1: static module/package loading. No call diagnostics are produced by + // the loader; the single binding pass below owns them. + let mut loaded = loader::load(file.unwrap_or_else(|| std::path::Path::new("index.mq.md"))) + .map_err(|message| default_load_error(&message))?; + + // T2: extension descriptors are read from manifests only. + let plugins = read_plugin_descriptors(&options.plugins)?; + + // T3: unique static table. Duplicate exports/builtin shadowing are rejected + // before any call site can be bound. + for descriptor in plugins.descriptors() { + if loaded + .program + .external_callable(&descriptor.public_name) + .is_some() + { + return Err(PrepareError::option( + "ext.duplicate_callable", + format!( + "plugin export `{}` is declared more than once", + descriptor.public_name + ), + )); + } + if marqdo_ir::builtins::find(&descriptor.public_name).is_some() { + return Err(PrepareError::option( + "ext.duplicate_callable", + format!( + "plugin export `{}` conflicts with a builtin; plugins may not shadow builtins", + descriptor.public_name + ), + )); + } + loaded.program.external_callables.push(descriptor); + } + let mut registry = CallableRegistry::try_from_program(&loaded.program).map_err(|conflict| { + PrepareError::option( + "ext.duplicate_callable", + format!("static callable table conflict: {conflict}"), + ) + })?; + + // Official stdlib public aliases (ADR-0058) are ordinary Marqdo bindings. + for alias in &loaded.official_aliases { + registry + .insert_official_alias(alias.clone()) + .map_err(|error| { + PrepareError::option( + "ext.duplicate_callable", + format!("stdlib public alias: {error}"), + ) + })?; + } + + // T4: exactly one call binding pass for the whole program. + let bound = marqdo_frontend::calls::bind_program_calls(&loaded.program, ®istry); + for (module_id, diagnostic) in bound.diagnostics { + let Some(file) = loaded.files.get(&module_id) else { + continue; + }; + let (path, source) = (file.path.clone(), file.source.clone()); + let duplicate = loaded.diagnostics.iter().any(|existing| { + existing.module_id == module_id + && existing.diagnostic.code == diagnostic.code + && existing.diagnostic.span == diagnostic.span + && existing.diagnostic.message == diagnostic.message + }); + if !duplicate { + loaded.diagnostics.push(FileDiagnostic { + module_id, + path, + source, + diagnostic, + }); + } + } + loaded.program.set_call_bindings(bound.bindings); + + Ok(PreparedProgram { + loaded, + plugins, + registry, + }) +} + +fn default_load_error(message: &str) -> PrepareError { + if message.contains("marqdo.lock") || message.contains("lockfile") { + PrepareError::load("project.lock", message) + } else { + PrepareError::option("project.load", message) + } +} diff --git a/crates/cli/src/provider_manifest.rs b/crates/cli/src/provider_manifest.rs new file mode 100644 index 0000000..ad13005 --- /dev/null +++ b/crates/cli/src/provider_manifest.rs @@ -0,0 +1,952 @@ +//! Strict third-party Resource Provider v2 manifests for the CLI composition +//! root (RFC-EXT-0001 §5.3, ADR-0055). +//! +//! This module *reads* `provider.json` schema version 2 and verifies the +//! selected artifact. It never `dlopen`s a library, never spawns a helper and +//! never authorizes a provider: parsing a manifest is not a grant. The v1 +//! `marqdo.provider` manifest stays a host-embedding path +//! (`StdHost::register_provider_package`, ADR-0037) and is rejected here with +//! `ext.unsupported_provider_schema`; there is no silent migration. +//! +//! The CLI only accepts the `c-abi-1` + `worker` transport combination and +//! resolves the helper as a sibling of the running `marqdo` executable; PATH +//! lookup is never used. Integrity hashes are content-integrity checks, not +//! publisher authentication. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; + +use marqdo_host::StdHost; +use marqdo_plugin_abi::{HostTarget, TargetTriple}; +use serde::Deserialize; +use sha2::{Digest, Sha256}; + +use crate::extension_descriptors::{ + ensure_bundle_relative, verify_bundle_regular_file, verify_sha256, PrepareError, +}; + +/// Provider manifest file name used by the CLI (`--provider` accepts this file). +pub const PROVIDER_MANIFEST_FILE: &str = "provider.json"; + +/// Provider manifest schema name. +pub const PROVIDER_SCHEMA: &str = "marqdo.provider"; +/// Provider manifest schema version accepted by the CLI. +pub const PROVIDER_SCHEMA_VERSION: u64 = 2; +/// Provider ABI accepted by the CLI (ADR-0036/0038). +pub const PROVIDER_ABI: &str = "c-abi-1"; +/// Provider transport accepted by the CLI (ADR-0035/0038). +pub const PROVIDER_TRANSPORT: &str = "worker"; +/// Helper executable shipped next to the `marqdo` binary (ADR-0038). +pub const PROVIDER_WORKER_BINARY: &str = "marqdo-provider-worker"; + +/// Upper bounds for a single provider manifest; these bound parser work only. +pub const MAX_PROVIDER_NAME_BYTES: usize = 128; +pub const MAX_PROVIDER_KINDS: usize = 64; +pub const MAX_PROVIDER_ARTIFACTS: usize = 32; +pub const MAX_PROVIDER_PERMISSIONS: usize = 64; +/// Maximum selected artifact size (256 MiB). +pub const MAX_PROVIDER_ARTIFACT_BYTES: u64 = 256 * 1024 * 1024; +/// Maximum manifest size (1 MiB); a manifest is metadata, not an artifact. +pub const MAX_PROVIDER_MANIFEST_BYTES: u64 = 1024 * 1024; + +/// Core resource kinds owned by the host; third parties may never override +/// them (RFC-EXT-0001 §5.3.1). `stdio`/`file`/`tcp`/`tls`/`udp`/`tcp-listener` +/// are dispatched by `marqdo-host` before the provider registry is consulted. +pub const RESERVED_RESOURCE_KINDS: &[&str] = + &["stdio", "file", "tcp", "tls", "udp", "tcp-listener"]; + +/// Raw `provider.json` schema version 2 document. +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ProviderManifest { + schema: String, + schema_version: u64, + name: String, + version: String, + abi: String, + transport: String, + kinds: Vec, + artifacts: Vec, + #[serde(default)] + permissions_requested: Vec, + /// Free-form, non-executable metadata. Ignored for execution and never used + /// to select artifacts or grant permissions. + #[serde(rename = "metadata", default)] + _metadata: Option, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct ManifestArtifact { + target: String, + path: String, + sha256: String, +} + +/// A manifest whose selected artifact has been resolved and verified. +/// +/// Producing one requires reading the manifest, checking the schema rules, +/// selecting the artifact for the current host target and verifying the +/// content hash. It still does not grant access and no worker has been started. +#[derive(Debug, Clone)] +pub struct VerifiedProvider { + /// Stable grant name (`--allow-provider NAME` must match exactly). + pub name: String, + /// Strict SemVer version declared by the manifest. + pub version: String, + /// Non-reserved resource kinds served by this provider. + pub kinds: Vec, + /// Canonical manifest path. + pub manifest_path: PathBuf, + /// Canonical bundle directory (manifest parent). + pub base_dir: PathBuf, + /// `sha256:` of the manifest bytes at verification time. + pub manifest_digest: String, + /// Selected transport (`worker` in v2). + pub transport: String, + /// Canonical path of the selected, hash-verified artifact. + pub selected_artifact_path: PathBuf, + /// Bundle-relative artifact path exactly as declared. + pub selected_artifact_relative: String, + /// Lowercase hex sha256 declared by the manifest. + pub selected_artifact_sha256: String, + /// Target triple the selected artifact was built for. + pub selected_target: String, +} + +impl VerifiedProvider { + /// Re-verify the manifest bytes and the selected artifact immediately + /// before the provider worker is spawned. + /// + /// `run` must never trust a hash it checked earlier: another process can + /// replace the manifest or the artifact between selection and use. + pub fn reverify(&self) -> Result<(), PrepareError> { + let bytes = std::fs::read(&self.manifest_path).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!( + "provider manifest `{}` could not be re-read before use: {error}", + self.manifest_path.display() + ), + ) + })?; + if bytes.len() as u64 > MAX_PROVIDER_MANIFEST_BYTES { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "provider manifest `{}` grew beyond the {MAX_PROVIDER_MANIFEST_BYTES}-byte limit before use", + self.manifest_path.display() + ), + )); + } + let digest = manifest_digest(&bytes); + if digest != self.manifest_digest { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "provider manifest `{}` changed after verification (expected {}, actual {}); refusing to start its worker", + self.manifest_path.display(), + self.manifest_digest, + digest + ), + )); + } + let artifact = + verify_bundle_regular_file(&self.base_dir, &self.selected_artifact_relative)?; + if artifact != self.selected_artifact_path { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "provider `{}` artifact `{}` now resolves to `{}` instead of the verified `{}`", + self.name, + self.selected_artifact_relative, + artifact.display(), + self.selected_artifact_path.display() + ), + )); + } + check_artifact_size(&artifact)?; + verify_sha256(&artifact, &self.selected_artifact_sha256)?; + Ok(()) + } +} + +/// Read, validate and verify every explicitly selected `--provider` manifest. +/// +/// No worker is spawned and no artifact is executed. Duplicate provider names +/// and duplicate resource kinds across providers are rejected deterministically +/// (the returned list is sorted by provider name) so a later registration can +/// never overwrite an earlier one. +pub fn read_provider_manifests(paths: &[PathBuf]) -> Result, PrepareError> { + let mut manifest_paths: BTreeSet = BTreeSet::new(); + for path in paths { + if path.is_dir() { + return Err(PrepareError::option( + "ext.manifest_unreadable", + format!( + "--provider `{}` must name a `{PROVIDER_MANIFEST_FILE}` file, not a directory", + path.display() + ), + )); + } + let canonical = path.canonicalize().map_err(|error| { + PrepareError::option( + "ext.manifest_unreadable", + format!( + "cannot open provider manifest `{}`: {error}", + path.display() + ), + ) + })?; + manifest_paths.insert(canonical); + } + + let mut providers = Vec::with_capacity(manifest_paths.len()); + for path in manifest_paths { + providers.push(read_provider_manifest(&path)?); + } + + // Deterministic conflict resolution independent of CLI argument order. + providers.sort_by(|left, right| { + (&left.name, &left.manifest_digest).cmp(&(&right.name, &right.manifest_digest)) + }); + let mut unique: Vec = Vec::with_capacity(providers.len()); + for provider in providers { + if let Some(existing) = unique.iter().find(|item| item.name == provider.name) { + if existing.manifest_digest != provider.manifest_digest { + return Err(PrepareError::option( + "ext.duplicate_provider", + format!( + "provider name `{}` is selected with two different manifests (`{}` and `{}`); \ + explicit aliases are not implemented, so this is rejected", + provider.name, + existing.manifest_path.display(), + provider.manifest_path.display() + ), + )); + } + // The exact same declaration selected twice is idempotent. + continue; + } + unique.push(provider); + } + + let mut claimed_kinds: BTreeMap = BTreeMap::new(); + for provider in &unique { + for kind in &provider.kinds { + if let Some(owner) = claimed_kinds.get(kind) { + return Err(PrepareError::option( + "ext.duplicate_provider_kind", + format!( + "resource kind `{kind}` is declared by both provider `{owner}` and provider `{}`; \ + duplicate kinds are rejected before any worker starts", + provider.name + ), + )); + } + claimed_kinds.insert(kind.clone(), provider.name.clone()); + } + } + + Ok(unique) +} + +fn read_provider_manifest(canonical_path: &Path) -> Result { + let metadata = std::fs::metadata(canonical_path).map_err(|error| { + PrepareError::option( + "ext.manifest_unreadable", + format!( + "cannot stat provider manifest `{}`: {error}", + canonical_path.display() + ), + ) + })?; + if metadata.len() > MAX_PROVIDER_MANIFEST_BYTES { + return Err(PrepareError::option( + "ext.manifest_invalid", + format!( + "provider manifest `{}` is {} bytes, larger than the {MAX_PROVIDER_MANIFEST_BYTES}-byte limit", + canonical_path.display(), + metadata.len() + ), + )); + } + let bytes = std::fs::read(canonical_path).map_err(|error| { + PrepareError::option( + "ext.manifest_unreadable", + format!( + "cannot read provider manifest `{}`: {error}", + canonical_path.display() + ), + ) + })?; + let digest = manifest_digest(&bytes); + let value: serde_json::Value = serde_json::from_slice(&bytes).map_err(|error| { + PrepareError::option( + "ext.manifest_invalid", + format!( + "provider manifest `{}` is not valid UTF-8 JSON: {error}", + canonical_path.display() + ), + ) + })?; + + // Schema/version gate runs before structural parsing so a v1 (or future) + // manifest gets the stable `ext.unsupported_provider_schema` code instead + // of a generic field error. + let schema = value.get("schema").and_then(serde_json::Value::as_str); + if schema != Some(PROVIDER_SCHEMA) { + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider manifest `{}` does not declare `schema: \"{PROVIDER_SCHEMA}\"`; \ + the CLI accepts schema version {PROVIDER_SCHEMA_VERSION} only", + canonical_path.display() + ), + )); + } + let schema_version = value + .get("schema_version") + .and_then(serde_json::Value::as_u64); + if schema_version != Some(PROVIDER_SCHEMA_VERSION) { + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider manifest `{}` declares schema_version {}; the CLI accepts {PROVIDER_SCHEMA_VERSION} only \ + (v1 and future versions are rejected, there is no silent migration)", + canonical_path.display(), + schema_version + .map(|version| version.to_string()) + .unwrap_or_else(|| "".to_string()) + ), + )); + } + + let manifest: ProviderManifest = serde_json::from_value(value).map_err(|error| { + PrepareError::option( + "ext.manifest_invalid", + format!( + "provider manifest `{}` is not a valid {PROVIDER_SCHEMA} v{PROVIDER_SCHEMA_VERSION} document: {error}", + canonical_path.display() + ), + ) + })?; + if manifest.schema != PROVIDER_SCHEMA || manifest.schema_version != PROVIDER_SCHEMA_VERSION { + // Unreachable after the Value-level gate; kept fail-closed. + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider manifest `{}` uses an unsupported schema", + canonical_path.display() + ), + )); + } + validate_manifest(&manifest, canonical_path)?; + + let base_dir = canonical_path + .parent() + .map(Path::to_path_buf) + .unwrap_or_else(|| PathBuf::from(".")); + let selected_target = HostTarget::current().triple().to_string(); + let artifact = manifest + .artifacts + .iter() + .find(|artifact| artifact.target == selected_target) + .ok_or_else(|| { + PrepareError::option( + "ext.no_matching_artifact", + format!( + "provider `{}` has no artifact for the current target `{selected_target}`; \ + other platforms are never substituted", + manifest.name + ), + ) + })?; + + let artifact_relative = normalise_artifact_path(&manifest.name, &artifact.path)?; + let artifact_path = verify_bundle_regular_file(&base_dir, &artifact_relative)?; + check_artifact_size(&artifact_path)?; + verify_sha256(&artifact_path, &artifact.sha256)?; + + Ok(VerifiedProvider { + name: manifest.name, + version: manifest.version, + kinds: manifest.kinds, + manifest_path: canonical_path.to_path_buf(), + base_dir, + manifest_digest: digest, + transport: manifest.transport, + selected_artifact_path: artifact_path, + selected_artifact_relative: artifact_relative, + selected_artifact_sha256: artifact.sha256.to_ascii_lowercase(), + selected_target, + }) +} + +fn validate_manifest( + manifest: &ProviderManifest, + manifest_path: &Path, +) -> Result<(), PrepareError> { + let invalid = |message: String| { + PrepareError::option( + "ext.manifest_invalid", + format!("provider manifest `{}`: {message}", manifest_path.display()), + ) + }; + + validate_provider_name(&manifest.name) + .map_err(|error| invalid(format!("invalid provider name: {error}")))?; + if RESERVED_RESOURCE_KINDS.contains(&manifest.name.as_str()) { + return Err(invalid(format!( + "provider name `{}` is a reserved core resource kind", + manifest.name + ))); + } + semver::Version::parse(&manifest.version).map_err(|error| { + invalid(format!( + "version `{}` is not strict SemVer: {error}", + manifest.version + )) + })?; + if manifest.abi != PROVIDER_ABI { + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider `{}` declares abi `{}`; schema v{PROVIDER_SCHEMA_VERSION} accepts `{PROVIDER_ABI}` only", + manifest.name, manifest.abi + ), + )); + } + if manifest.transport != PROVIDER_TRANSPORT { + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider `{}` declares transport `{}`; schema v{PROVIDER_SCHEMA_VERSION} accepts `{PROVIDER_TRANSPORT}` only", + manifest.name, manifest.transport + ), + )); + } + if manifest.kinds.is_empty() { + return Err(invalid( + "kinds must declare at least one resource kind".to_string(), + )); + } + if manifest.kinds.len() > MAX_PROVIDER_KINDS { + return Err(invalid(format!( + "kinds declares more than {MAX_PROVIDER_KINDS} entries" + ))); + } + let mut kinds = BTreeSet::new(); + for kind in &manifest.kinds { + validate_provider_name(kind) + .map_err(|error| invalid(format!("invalid resource kind `{kind}`: {error}")))?; + if RESERVED_RESOURCE_KINDS.contains(&kind.as_str()) { + return Err(invalid(format!( + "resource kind `{kind}` is reserved by the host and cannot be overridden by a provider" + ))); + } + if !kinds.insert(kind.clone()) { + return Err(invalid(format!( + "resource kind `{kind}` is declared more than once" + ))); + } + } + if manifest.artifacts.is_empty() { + return Err(invalid( + "artifacts must declare at least one target artifact".to_string(), + )); + } + if manifest.artifacts.len() > MAX_PROVIDER_ARTIFACTS { + return Err(invalid(format!( + "artifacts declares more than {MAX_PROVIDER_ARTIFACTS} entries" + ))); + } + let mut targets = BTreeSet::new(); + for artifact in &manifest.artifacts { + let triple = TargetTriple::parse(&artifact.target).map_err(|error| { + invalid(format!( + "artifact target `{}` is not a target triple: {error}", + artifact.target + )) + })?; + if !targets.insert(triple.to_string()) { + return Err(invalid(format!( + "artifact target `{triple}` is declared more than once" + ))); + } + if !is_lowercase_sha256(&artifact.sha256) { + return Err(invalid(format!( + "artifact for target `{:?}` must declare a 64-character lowercase hex sha256", + artifact.target + ))); + } + } + if manifest.permissions_requested.len() > MAX_PROVIDER_PERMISSIONS { + return Err(invalid(format!( + "permissions_requested declares more than {MAX_PROVIDER_PERMISSIONS} entries" + ))); + } + let mut permissions = BTreeSet::new(); + for permission in &manifest.permissions_requested { + if !is_known_capability(permission) && !is_vendor_capability(permission) { + return Err(invalid(format!( + "permissions_requested entry `{permission}` is not a known capability or a dotted vendor capability" + ))); + } + if !permissions.insert(permission.clone()) { + return Err(invalid(format!( + "permission `{permission}` is declared more than once" + ))); + } + } + // `metadata` is intentionally not validated: it has no execution meaning. + Ok(()) +} + +/// Validate a dotted provider/kind name using the plugin name grammar +/// (ASCII lowercase segment start, then lowercase/digits/`-`/`_`). +pub fn validate_provider_name(name: &str) -> Result<(), String> { + if name.is_empty() { + return Err("must not be empty".to_string()); + } + if name.len() > MAX_PROVIDER_NAME_BYTES { + return Err(format!( + "must not be longer than {MAX_PROVIDER_NAME_BYTES} bytes" + )); + } + for segment in name.split('.') { + if segment.is_empty() { + return Err(format!("`{name}` contains an empty segment")); + } + let mut bytes = segment.bytes(); + let first = bytes.next().unwrap_or(b'a'); + if !first.is_ascii_lowercase() { + return Err(format!( + "`{name}` must start every `.`-separated segment with a lowercase ASCII letter" + )); + } + if !bytes.all(|byte| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-' || byte == b'_' + }) { + return Err(format!( + "`{name}` may only contain lowercase ASCII letters, digits, `-` and `_`" + )); + } + } + Ok(()) +} + +/// Reject absolute paths and `..` traversal in declared artifact paths. +fn normalise_artifact_path(provider: &str, path: &str) -> Result { + ensure_bundle_relative(path).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!("provider `{provider}` artifact path `{path}` is invalid: {error}"), + ) + })?; + Ok(path.to_string()) +} + +fn check_artifact_size(path: &Path) -> Result<(), PrepareError> { + let metadata = std::fs::metadata(path).map_err(|error| { + PrepareError::load( + "ext.artifact_integrity", + format!( + "cannot stat provider artifact `{}`: {error}", + path.display() + ), + ) + })?; + if metadata.len() > MAX_PROVIDER_ARTIFACT_BYTES { + return Err(PrepareError::load( + "ext.artifact_integrity", + format!( + "provider artifact `{}` is {} bytes, larger than the {MAX_PROVIDER_ARTIFACT_BYTES}-byte limit", + path.display(), + metadata.len() + ), + )); + } + Ok(()) +} + +fn is_lowercase_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn is_known_capability(name: &str) -> bool { + marqdo_plugin_abi::KNOWN_CAPABILITIES.contains(&name) +} + +/// A dotted vendor capability proposal (validation only, never a grant). +fn is_vendor_capability(name: &str) -> bool { + let segments: Vec<&str> = name.split('.').collect(); + if segments.len() < 3 { + return false; + } + if matches!( + segments[0], + "console" | "math" | "fs" | "net" | "process" | "log" | "clock" | "random" + ) { + return false; + } + segments.iter().all(|segment| { + !segment.is_empty() + && segment.bytes().enumerate().all(|(index, byte)| { + byte.is_ascii_lowercase() || byte.is_ascii_digit() || (byte == b'-' && index > 0) + }) + }) +} + +fn manifest_digest(bytes: &[u8]) -> String { + let mut hasher = Sha256::new(); + hasher.update(bytes); + let digest = hasher.finalize(); + let mut hex = String::with_capacity(digest.len() * 2); + for byte in digest { + hex.push_str(&format!("{byte:02x}")); + } + format!("sha256:{hex}") +} + +/// Whether the provider `worker` transport is supported on this build. +/// +/// RFC-EXT-0001 §5.6 makes Linux x86_64 the mandatory lane. Other targets stay +/// compilable but must fail closed with `ext.platform_unsupported` instead of +/// pretending a helper works. +pub fn provider_worker_platform_supported() -> bool { + cfg!(all(target_os = "linux", target_arch = "x86_64")) +} + +/// Resolve the provider helper shipped next to the running `marqdo` executable. +/// +/// Resolution is *only* the sibling file: there is no PATH lookup and no +/// directory search (RFC-EXT-0001 §5.4.5). A missing helper is +/// `ext.worker_unavailable`, never a fallback to in-process native loading. +pub fn provider_worker_executable() -> Result { + if !provider_worker_platform_supported() { + return Err(PrepareError::load( + "ext.platform_unsupported", + format!( + "the resource provider worker transport is supported on linux/x86_64 only in this release; \ + this build targets {}/{}", + std::env::consts::OS, + std::env::consts::ARCH + ), + )); + } + let executable = std::env::current_exe().map_err(|error| { + PrepareError::load( + "ext.worker_unavailable", + format!( + "cannot locate the marqdo executable to find `{PROVIDER_WORKER_BINARY}`: {error}" + ), + ) + })?; + let directory = executable.parent().ok_or_else(|| { + PrepareError::load( + "ext.worker_unavailable", + format!( + "the marqdo executable `{}` has no parent directory", + executable.display() + ), + ) + })?; + let helper = directory.join(format!( + "{PROVIDER_WORKER_BINARY}{}", + std::env::consts::EXE_SUFFIX + )); + match std::fs::metadata(&helper) { + Ok(metadata) if metadata.is_file() => Ok(helper), + _ => Err(PrepareError::load( + "ext.worker_unavailable", + format!( + "the resource provider worker helper `{}` is missing; it must be shipped next to the \ + marqdo executable (PATH lookup is not allowed)", + helper.display() + ), + )), + } +} + +/// Check the exact `--allow-provider` grant, re-verify the manifest and +/// artifact, then register the existing worker resource-provider transport. +/// +/// The grant check is repeated here (it was already checked during pre-spawn +/// verification) so no caller can register an ungranted provider. The provider +/// is always registered through +/// [`StdHost::register_worker_resource_provider`]; this path never falls back +/// to the trusted in-process C ABI or a native library. +pub fn load_resource_provider_into_host( + host: &mut StdHost, + provider: &VerifiedProvider, + allow_provider: &[String], +) -> Result<(), PrepareError> { + if !allow_provider.iter().any(|grant| grant == &provider.name) { + return Err(PrepareError::load( + "permission.denied", + format!( + "resource provider `{}` was selected but not granted; add --allow-provider {}", + provider.name, provider.name + ), + )); + } + if provider.transport != PROVIDER_TRANSPORT { + return Err(PrepareError::option( + "ext.unsupported_provider_schema", + format!( + "provider `{}` transport `{}` cannot be loaded by the CLI", + provider.name, provider.transport + ), + )); + } + // Re-verify immediately before use; never trust an earlier hash. + provider.reverify()?; + let worker = provider_worker_executable()?; + let kinds: Vec<&str> = provider.kinds.iter().map(String::as_str).collect(); + let args = vec![provider + .selected_artifact_path + .to_string_lossy() + .into_owned()]; + host.register_worker_resource_provider(&provider.name, &kinds, &worker, &args) + .map_err(|error| { + PrepareError::load( + "ext.worker_unavailable", + format!( + "cannot start the worker for resource provider `{}` v{} (target {}) using `{}`: {}", + provider.name, + provider.version, + provider.selected_target, + worker.display(), + error.message + ), + ) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn tempdir(tag: &str) -> PathBuf { + let path = std::env::temp_dir().join(format!( + "marqdo-provider-manifest-{}-{tag}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|duration| duration.as_nanos()) + .unwrap_or(0) + )); + std::fs::create_dir_all(&path).unwrap(); + path + } + + fn write_manifest(dir: &Path, manifest: &str) -> PathBuf { + let path = dir.join(PROVIDER_MANIFEST_FILE); + std::fs::write(&path, manifest).unwrap(); + path + } + + fn artifact(dir: &Path, name: &str, bytes: &[u8]) -> (String, String) { + let path = dir.join(name); + std::fs::write(&path, bytes).unwrap(); + let mut hasher = Sha256::new(); + hasher.update(bytes); + let mut hex = String::new(); + for byte in hasher.finalize() { + hex.push_str(&format!("{byte:02x}")); + } + (name.to_string(), hex) + } + + fn manifest_text(name: &str, kinds: &[&str], path: &str, sha: &str) -> String { + let kinds = kinds + .iter() + .map(|kind| format!("\"{kind}\"")) + .collect::>() + .join(", "); + format!( + r#"{{ + "schema": "marqdo.provider", + "schema_version": 2, + "name": "{name}", + "version": "0.1.0", + "abi": "c-abi-1", + "transport": "worker", + "kinds": [{kinds}], + "artifacts": [ + {{"target": "{}", "path": "{path}", "sha256": "{sha}"}} + ], + "permissions_requested": [] +}}"#, + HostTarget::current().triple() + ) + } + + #[test] + fn valid_v2_manifest_verifies_artifact() { + let dir = tempdir("valid"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let manifest = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &sha), + ); + let providers = read_provider_manifests(&[manifest]).unwrap(); + assert_eq!(providers.len(), 1); + assert_eq!(providers[0].name, "demo.sensor"); + assert_eq!( + providers[0].selected_target, + HostTarget::current().triple().to_string() + ); + assert!(providers[0].manifest_digest.starts_with("sha256:")); + providers[0].reverify().unwrap(); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn v1_manifest_is_rejected_without_migration() { + let dir = tempdir("v1"); + let manifest = write_manifest( + &dir, + r#"{"schema":"marqdo.provider","schema_version":1,"name":"cabi","kinds":["cabi.sensor"],"abi":"c-abi-1","library":{"linux":"libcabi.so"}}"#, + ); + let error = read_provider_manifests(&[manifest]).unwrap_err(); + assert_eq!(error.code, "ext.unsupported_provider_schema"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn missing_target_is_no_matching_artifact() { + let dir = tempdir("target"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let manifest = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &sha).replace( + &HostTarget::current().triple().to_string(), + "aarch64-unknown-linux-gnu", + ), + ); + let error = read_provider_manifests(&[manifest]).unwrap_err(); + assert_eq!(error.code, "ext.no_matching_artifact"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn reserved_kind_is_rejected() { + let dir = tempdir("reserved"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let manifest = write_manifest(&dir, &manifest_text("demo.sensor", &["file"], &path, &sha)); + let error = read_provider_manifests(&[manifest]).unwrap_err(); + assert_eq!(error.code, "ext.manifest_invalid"); + assert!(error.message.contains("reserved"), "{}", error.message); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn traversal_and_digest_mismatch_are_artifact_integrity() { + let dir = tempdir("integrity"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let traversal = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], "../libdemo.so", &sha), + ); + let error = read_provider_manifests(&[traversal]).unwrap_err(); + assert_eq!(error.code, "ext.artifact_integrity"); + let mismatch = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &"0".repeat(64)), + ); + let error = read_provider_manifests(&[mismatch]).unwrap_err(); + assert_eq!(error.code, "ext.artifact_integrity"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn duplicate_kinds_across_providers_are_rejected() { + let dir = tempdir("duplicate-kind"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let first = write_manifest( + &dir, + &manifest_text("demo.one", &["demo.sensor"], &path, &sha), + ); + let second_dir = dir.join("second"); + std::fs::create_dir_all(&second_dir).unwrap(); + std::fs::copy(dir.join(&path), second_dir.join(&path)).unwrap(); + let second = write_manifest( + &second_dir, + &manifest_text("demo.two", &["demo.sensor"], &path, &sha), + ); + let error = read_provider_manifests(&[first, second]).unwrap_err(); + assert_eq!(error.code, "ext.duplicate_provider_kind"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn duplicate_provider_name_with_different_manifests_is_rejected() { + let dir = tempdir("duplicate-name"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let first = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &sha), + ); + let second_dir = dir.join("second"); + std::fs::create_dir_all(&second_dir).unwrap(); + std::fs::copy(dir.join(&path), second_dir.join(&path)).unwrap(); + // Same name, different version => different manifest digest. + let second = write_manifest( + &second_dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &sha) + .replace("\"version\": \"0.1.0\"", "\"version\": \"0.2.0\""), + ); + let error = read_provider_manifests(&[first.clone(), second]).unwrap_err(); + assert_eq!(error.code, "ext.duplicate_provider"); + // Selecting the exact same manifest twice is idempotent. + let providers = read_provider_manifests(&[first.clone(), first]).unwrap(); + assert_eq!(providers.len(), 1); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn unknown_fields_and_bad_names_are_manifest_invalid() { + let dir = tempdir("invalid"); + let (path, sha) = artifact(&dir, "libdemo.so", b"provider-bytes"); + let unknown = write_manifest( + &dir, + &manifest_text("demo.sensor", &["demo.sensor"], &path, &sha).replace( + "\"permissions_requested\": []", + "\"permissions_requested\": [], \"exec\": true", + ), + ); + let error = read_provider_manifests(&[unknown]).unwrap_err(); + assert_eq!(error.code, "ext.manifest_invalid"); + let bad_name = write_manifest( + &dir, + &manifest_text("Demo.Sensor", &["demo.sensor"], &path, &sha), + ); + let error = read_provider_manifests(&[bad_name]).unwrap_err(); + assert_eq!(error.code, "ext.manifest_invalid"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn worker_helper_resolution_only_uses_the_executable_sibling() { + // The real helper is installed next to the CLI binary by the + // integration fixture; this test only asserts the platform/absence + // contract and that a path is never derived from PATH. + if provider_worker_platform_supported() { + let helper = provider_worker_executable(); + // In a unit-test binary `current_exe()` is the test binary, so the + // sibling helper is normally absent: the error must be structured. + match helper { + Ok(path) => assert!(path.is_file()), + Err(error) => assert_eq!(error.code, "ext.worker_unavailable"), + } + } else { + let error = provider_worker_executable().unwrap_err(); + assert_eq!(error.code, "ext.platform_unsupported"); + } + } +} diff --git a/crates/cli/src/stdlib.rs b/crates/cli/src/stdlib.rs index 3b397db..fc476c1 100644 --- a/crates/cli/src/stdlib.rs +++ b/crates/cli/src/stdlib.rs @@ -13,6 +13,25 @@ use sha2::{Digest, Sha256}; /// Environment override used by tests and relocatable installations. pub const STDLIB_DIR_ENV: &str = "MARQDO_STDLIB_DIR"; +/// One official bundled-stdlib public alias (ADR-0058). +/// +/// The alias table lives in the verified `stdlib.json` metadata, not in Rust +/// source: `打印`/`print` are ordinary `Marqdo` bindings to `io.mq.md::写值` +/// and the runtime no longer special-cases print names. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PublicAlias { + /// Public names that resolve to the target function. + pub public_names: Vec, + /// Package-relative target module (for example `io.mq.md`). + pub target_module: String, + /// Canonical/local target function name. + pub target_function: String, + /// Declared parameter names, used to verify the target signature. + pub expected_signature: Option, + /// Conservative effect request summary. + pub effects: Vec, +} + /// Verified bundled standard library. #[derive(Debug, Clone)] pub struct StdlibBundle { @@ -20,6 +39,7 @@ pub struct StdlibBundle { name: String, version: String, digest: String, + aliases: Vec, } impl StdlibBundle { @@ -81,14 +101,42 @@ impl StdlibBundle { "stdlib digest mismatch: declared {declared}, actual {actual}" )); } + let aliases = parse_public_aliases(&manifest, &manifest_path)?; + let declared_alias_digest = manifest + .get("aliases_digest") + .and_then(serde_json::Value::as_str); + let actual_alias_digest = aliases_digest(&aliases); + match (declared_alias_digest, aliases.is_empty()) { + (Some(declared), _) => { + if declared != actual_alias_digest { + return Err(format!( + "stdlib public-alias metadata digest mismatch: declared {declared}, \ + actual {actual_alias_digest}" + )); + } + } + (None, false) => { + return Err(format!( + "`{}` declares public_aliases without aliases_digest", + manifest_path.display() + )); + } + (None, true) => {} + } Ok(Self { root, name, version, digest: actual, + aliases, }) } + /// Verified official public aliases. + pub fn public_aliases(&self) -> &[PublicAlias] { + &self.aliases + } + /// Bundle root. pub fn root(&self) -> &Path { &self.root @@ -227,6 +275,167 @@ fn hex(bytes: &[u8]) -> String { out } +fn parse_public_aliases( + manifest: &serde_json::Value, + manifest_path: &Path, +) -> Result, String> { + let Some(entries) = manifest.get("public_aliases") else { + return Ok(Vec::new()); + }; + let entries = entries.as_array().ok_or_else(|| { + format!( + "`{}` public_aliases must be an array", + manifest_path.display() + ) + })?; + let mut aliases = Vec::new(); + let mut seen_names = std::collections::BTreeSet::new(); + for entry in entries { + let public_names = entry + .get("public_names") + .and_then(serde_json::Value::as_array) + .ok_or_else(|| { + format!( + "`{}` public alias has no public_names array", + manifest_path.display() + ) + })? + .iter() + .map(|name| { + name.as_str().map(ToString::to_string).ok_or_else(|| { + format!( + "`{}` public alias name must be a string", + manifest_path.display() + ) + }) + }) + .collect::, _>>()?; + if public_names.is_empty() { + return Err(format!( + "`{}` public alias declares no names", + manifest_path.display() + )); + } + for name in &public_names { + if name.is_empty() { + return Err(format!( + "`{}` public alias name must not be empty", + manifest_path.display() + )); + } + if !seen_names.insert(name.clone()) { + return Err(format!( + "`{}` public alias name `{name}` is declared more than once", + manifest_path.display() + )); + } + } + let target_module = entry + .get("target_module") + .and_then(serde_json::Value::as_str) + .ok_or_else(|| { + format!( + "`{}` public alias has no target_module", + manifest_path.display() + ) + })? + .to_string(); + validate_bundle_relative(&target_module).map_err(|error| { + format!( + "`{}` public alias target_module `{target_module}` is invalid: {error}", + manifest_path.display() + ) + })?; + let target_function = entry + .get("target_function") + .and_then(serde_json::Value::as_str) + .ok_or_else(|| { + format!( + "`{}` public alias has no target_function", + manifest_path.display() + ) + })? + .to_string(); + if target_function.is_empty() { + return Err(format!( + "`{}` public alias target_function must not be empty", + manifest_path.display() + )); + } + let expected_signature = entry + .get("expected_signature") + .and_then(serde_json::Value::as_str) + .map(ToString::to_string); + let effects = entry + .get("effects") + .and_then(serde_json::Value::as_array) + .map(|effects| { + effects + .iter() + .filter_map(serde_json::Value::as_str) + .map(ToString::to_string) + .collect() + }) + .unwrap_or_default(); + aliases.push(PublicAlias { + public_names, + target_module, + target_function, + expected_signature, + effects, + }); + } + aliases.sort_by(|left, right| { + (&left.target_module, &left.target_function) + .cmp(&(&right.target_module, &right.target_function)) + }); + Ok(aliases) +} + +fn validate_bundle_relative(path: &str) -> Result<(), String> { + let mut normalised = PathBuf::new(); + for component in Path::new(path).components() { + match component { + Component::Normal(part) => normalised.push(part), + Component::CurDir => {} + Component::ParentDir | Component::RootDir | Component::Prefix(_) => { + return Err("must be a relative path inside the bundle".to_string()) + } + } + } + if normalised.as_os_str().is_empty() { + return Err("must not be empty".to_string()); + } + Ok(()) +} + +/// Canonical digest over the official public-alias table. +/// +/// Format (identical to `tools/pack-stdlib.py`): for every alias sorted by +/// `(target_module, target_function)`: module, NUL, function, NUL, then every +/// public name sorted, NUL. A fixed binary format avoids JSON formatting drift. +pub fn aliases_digest(aliases: &[PublicAlias]) -> String { + let mut sorted = aliases.to_vec(); + sorted.sort_by(|left, right| { + (&left.target_module, &left.target_function) + .cmp(&(&right.target_module, &right.target_function)) + }); + let mut hasher = Sha256::new(); + for alias in &sorted { + hasher.update(alias.target_module.as_bytes()); + hasher.update([0]); + hasher.update(alias.target_function.as_bytes()); + hasher.update([0]); + let mut names = alias.public_names.clone(); + names.sort(); + for name in names { + hasher.update(name.as_bytes()); + hasher.update([0]); + } + } + format!("sha256:{}", hex(&hasher.finalize())) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/cli/tests/architecture_boundaries.rs b/crates/cli/tests/architecture_boundaries.rs index 94231a4..5e9cabf 100644 --- a/crates/cli/tests/architecture_boundaries.rs +++ b/crates/cli/tests/architecture_boundaries.rs @@ -272,3 +272,125 @@ fn walk_rs(root: &Path) -> Vec { } files } + +/// RFC-EXT-0001 §4/§14: the runtime must dispatch by the prepared binding and +/// never re-resolve a name or special-case a print function. +#[test] +fn callable_binding_is_the_only_static_resolution_path() { + let root = repo_root(); + let loader = read(&root.join("crates/cli/src/loader.rs")); + let interpreter = read(&root.join("crates/marqdo-runtime/src/interpreter.rs")); + let run = read(&root.join("crates/cli/src/commands/run.rs")); + let frontend = read(&root.join("crates/marqdo-frontend/src/calls.rs")); + + assert!( + !loader.contains("program_uses_print"), + "loader must not scan for per-function names (ADR-0058)" + ); + assert!( + !interpreter.contains("print_override"), + "the runtime must not special-case print (ADR-0054/0058)" + ); + assert!( + !interpreter.contains("external_callables.is_empty"), + "the runtime must not fall back to plugin guessing (ADR-0054)" + ); + assert!( + interpreter.contains("call_bindings.get"), + "the runtime must dispatch by the prepared call binding table" + ); + assert!( + !interpreter.contains("builtins::find(&qualified)"), + "a qualified name must not be re-resolved at run time" + ); + assert!( + !run.contains("external_callables") + || !run.contains("external_callables") + || !run.contains("push(marqdo_ir::callable::CallableDescriptor"), + "run must not synthesise plugin descriptors (ADR-0054/§4.5)" + ); + assert!( + frontend.contains("bind_program_calls"), + "the frontend must expose the single binding entry point" + ); +} + +/// RFC-EXT-0001 §14.2/S4: no business-domain builtin or host-op branch may be +/// added for an ecosystem feature. These must be implemented as `.mq.md`, +/// plugin exports or provider streams. +#[test] +fn no_business_domain_builtins_or_host_ops() { + let root = repo_root(); + let builtins = read(&root.join("crates/marqdo-ir/src/builtins.rs")); + let host = read(&root.join("crates/marqdo-host/src/host.rs")); + for forbidden in ["quantum.", "ai.chat", "vendor.", "demo.sensor", "marqdo."] { + assert!( + !builtins.contains(forbidden), + "business builtin namespace `{forbidden}` must not exist" + ); + } + assert!( + !host.contains("\"demo.sensor\""), + "provider kinds must be data, not host-op string branches" + ); +} + +/// RFC-EXT-0001 §8: official stdlib public aliases are verified metadata, not +/// hard-coded runtime names. Recompute the canonical metadata digest. +#[test] +fn official_stdlib_public_aliases_are_verified_metadata() { + let root = repo_root().join("stdlib"); + let manifest: serde_json::Value = + serde_json::from_slice(&std::fs::read(root.join("stdlib.json")).unwrap()).unwrap(); + let aliases = manifest["public_aliases"] + .as_array() + .expect("stdlib.json must declare public_aliases"); + assert!(!aliases.is_empty()); + let mut hasher = Sha256::new(); + let mut entries: Vec<&serde_json::Value> = aliases.iter().collect(); + entries.sort_by_key(|alias| { + ( + alias["target_module"].as_str().unwrap_or("").to_string(), + alias["target_function"].as_str().unwrap_or("").to_string(), + ) + }); + for alias in &entries { + hasher.update(alias["target_module"].as_str().unwrap().as_bytes()); + hasher.update([0]); + hasher.update(alias["target_function"].as_str().unwrap().as_bytes()); + hasher.update([0]); + let mut names: Vec<&str> = alias["public_names"] + .as_array() + .unwrap() + .iter() + .filter_map(serde_json::Value::as_str) + .collect(); + names.sort(); + for name in names { + hasher.update(name.as_bytes()); + hasher.update([0]); + } + } + let actual = format!("sha256:{}", hex(&hasher.finalize())); + assert_eq!( + manifest["aliases_digest"], actual, + "stdlib.json aliases_digest must cover the public alias table" + ); + for alias in aliases { + let target_module = alias["target_module"].as_str().unwrap(); + let target_function = alias["target_function"].as_str().unwrap(); + let module = read(&root.join(target_module)); + assert!( + module.contains(&format!("## {target_function}")), + "public alias target `{target_module}::{target_function}` must exist" + ); + } +} + +fn hex(bytes: &[u8]) -> String { + let mut out = String::with_capacity(bytes.len() * 2); + for byte in bytes { + out.push_str(&format!("{byte:02x}")); + } + out +} diff --git a/crates/cli/tests/callable_resolution.rs b/crates/cli/tests/callable_resolution.rs new file mode 100644 index 0000000..ec98fcf --- /dev/null +++ b/crates/cli/tests/callable_resolution.rs @@ -0,0 +1,653 @@ +//! Black-box callable resolution / binding conformance (RFC-EXT-0001 §4.6). +//! +//! Every test drives the real `marqdo` binary through the public CLI. The +//! point is to prove that `check` and `run` share one static resolution and +//! that the runtime never guesses a plugin from a dotted string prefix. +#![cfg(unix)] + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; +use std::sync::atomic::{AtomicU32, Ordering}; + +use marqdo_plugin_abi::{sha256_file, HostTarget}; + +static COUNTER: AtomicU32 = AtomicU32::new(0); + +struct TempDir(PathBuf); + +impl TempDir { + fn new(tag: &str) -> Self { + let id = COUNTER.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "marqdo-callable-test-{}-{id}-{tag}", + std::process::id() + )); + std::fs::create_dir_all(&path).unwrap(); + Self(path) + } + + fn path(&self) -> &Path { + &self.0 + } +} + +impl Drop for TempDir { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .parent() + .unwrap() + .to_path_buf() +} + +fn cli(args: &[&str], dir: &Path) -> Output { + Command::new(env!("CARGO_BIN_EXE_marqdo")) + .args(args) + .current_dir(dir) + .output() + .unwrap() +} + +fn write(path: &Path, text: &str) { + std::fs::write(path, text).unwrap(); +} + +fn library_name() -> &'static str { + if cfg!(target_os = "macos") { + "solver.dylib" + } else { + "solver.so" + } +} + +fn shared_flags() -> &'static [&'static str] { + if cfg!(target_os = "macos") { + &["-dynamiclib", "-fPIC"] + } else { + &["-shared", "-fPIC"] + } +} + +fn have_cc() -> bool { + Command::new("cc") + .arg("--version") + .output() + .map(|output| output.status.success()) + .unwrap_or(false) +} + +fn compile_plugin(dir: &Path, extra: &[&str]) -> PathBuf { + let output = dir.join(library_name()); + let status = Command::new("cc") + .args(["-std=c11", "-O2"]) + .args(shared_flags()) + .arg("-I") + .arg(repo().join("include")) + .args(extra) + .arg(repo().join("examples/plugin-demo/solver.c")) + .arg("-o") + .arg(&output) + .status() + .expect("cc"); + assert!(status.success(), "cc failed"); + output +} + +fn write_worker_manifest(dir: &Path, artifact: &Path, exports: &str) { + let sha = sha256_file(artifact).unwrap(); + write( + &dir.join("plugin.yaml"), + &format!( + r#"plugin: + name: demo.solver + version: 0.1.0 + abi: "^1.0" + kind: worker + exports: +{exports} requires: + - math.compute + artifacts: + - target: {} + file: {} + sha256: "{sha}" +"#, + HostTarget::current().triple(), + artifact.file_name().unwrap().to_string_lossy() + ), + ); +} + +fn add_export(name: &str, signature: &str) -> String { + format!(" - name: {name}\n signature: \"{signature}\"\n") +} + +fn stderr(output: &Output) -> String { + String::from_utf8_lossy(&output.stderr).to_string() +} + +fn stdout(output: &Output) -> String { + String::from_utf8_lossy(&output.stdout).to_string() +} + +// CALL-01 +#[test] +fn import_alias_conflicting_with_builtin_namespace_is_rejected() { + let dir = TempDir::new("call01"); + write( + &dir.path().join("math.mq.md"), + "## @exec eval\n\n`params(a: Int)`\n\n`return a + 100`\n", + ); + write( + &dir.path().join("index.mq.md"), + "---\nimport:\n math: \"./math.mq.md\"\n---\n\n## @exec main\n\n`return math.eval(a=1)`\n", + ); + let check = cli(&["check", "index.mq.md"], dir.path()); + assert_eq!(check.status.code(), Some(1), "{}", stderr(&check)); + assert!( + stderr(&check).contains("ext.import_namespace_conflict"), + "{}", + stderr(&check) + ); + let run = cli(&["run", "index.mq.md"], dir.path()); + assert_eq!(run.status.code(), Some(1)); + assert!(stdout(&run).is_empty()); + assert!( + stderr(&run).contains("ext.import_namespace_conflict"), + "{}", + stderr(&run) + ); +} + +// CALL-02 +#[test] +fn dotted_builtin_call_binds_without_an_import() { + let output = cli(&["check", "examples/math.mq.md"], &repo()); + assert_eq!(output.status.code(), Some(0), "{}", stderr(&output)); + let run = cli(&["call", "examples/math.mq.md", "计算圆面积"], &repo()); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); +} + +// CALL-03 +#[test] +fn renamed_import_alias_binds_the_module_function() { + let dir = TempDir::new("call03"); + write( + &dir.path().join("math.mq.md"), + "## @exec eval\n\n`params(a: Int)`\n\n`return a + 100`\n", + ); + write( + &dir.path().join("index.mq.md"), + "---\nimport:\n mymath: \"./math.mq.md\"\n---\n\n## @exec main\n\n`return mymath.eval(a=1)`\n", + ); + let check = cli(&["check", "index.mq.md"], dir.path()); + assert_eq!(check.status.code(), Some(0), "{}", stderr(&check)); + let run = cli(&["run", "index.mq.md", "--json"], dir.path()); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); + let json: serde_json::Value = serde_json::from_str(&stdout(&run)).unwrap(); + assert_eq!( + json["value"], + serde_json::json!({"type": "int", "decimal": "101"}) + ); +} + +// CALL-04 / CALL-05 +#[test] +fn unknown_plugin_export_is_a_static_symbol_error() { + let dir = TempDir::new("call04"); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`return solver.add(a=1, b=2)`\n", + ); + let check = cli(&["check", "index.mq.md"], dir.path()); + assert_eq!(check.status.code(), Some(1)); + assert!( + stderr(&check).contains("symbol.unknown"), + "{}", + stderr(&check) + ); + let run = cli(&["run", "index.mq.md"], dir.path()); + assert_eq!(run.status.code(), Some(1)); + assert!(stderr(&run).contains("symbol.unknown"), "{}", stderr(&run)); + assert!(stdout(&run).is_empty()); + // CALL-05: the diagnostic does not change when an unrelated plugin is + // loaded, because the static export table is still the only source. + if !have_cc() { + return; + } + let plugin_dir = TempDir::new("call05-plugin"); + let artifact = compile_plugin(plugin_dir.path(), &[]); + write_worker_manifest( + plugin_dir.path(), + &artifact, + &add_export("solver.add", "($a: Int, $b: Int) -> Int"), + ); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`return solver.subtract(a=1, b=2)`\n", + ); + let with_plugin = cli( + &[ + "run", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--allow-plugin", + "demo.solver", + ], + dir.path(), + ); + assert_eq!(with_plugin.status.code(), Some(1)); + assert!( + stderr(&with_plugin).contains("symbol.unknown"), + "{}", + stderr(&with_plugin) + ); +} + +// CALL-06 +#[test] +fn plugin_with_two_prefixes_binds_both_exports() { + if !have_cc() { + eprintln!("skipping: cc is unavailable"); + return; + } + let dir = TempDir::new("call06"); + let plugin_dir = TempDir::new("call06-plugin"); + let artifact = compile_plugin(plugin_dir.path(), &["-DMQ_SOLVER_MULTI_EXPORT"]); + write_worker_manifest( + plugin_dir.path(), + &artifact, + &(add_export("solver.add", "($a: Int, $b: Int) -> Int") + + &add_export("solver.neg", "($a: Int) -> Int")), + ); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`和 = solver.add(20, b=22)`\n\n`相反数 = solver.neg(7)`\n\n`打印(和)`\n\n`打印(相反数)`\n", + ); + let output = cli( + &[ + "run", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--allow-plugin", + "demo.solver", + ], + dir.path(), + ); + assert_eq!(output.status.code(), Some(0), "{}", stderr(&output)); + assert_eq!(stdout(&output), "42\n-7\n"); +} + +// CALL-07 +#[test] +fn duplicate_complete_export_across_plugins_is_rejected_at_prepare() { + if !have_cc() { + eprintln!("skipping: cc is unavailable"); + return; + } + let dir = TempDir::new("call07"); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`return solver.add(a=1, b=2)`\n", + ); + let mut plugins = Vec::new(); + for tag in ["a", "b"] { + let plugin_dir = TempDir::new(tag); + let artifact = compile_plugin(plugin_dir.path(), &[]); + let sha = sha256_file(&artifact).unwrap(); + write( + &plugin_dir.path().join("plugin.yaml"), + &format!( + r#"plugin: + name: demo.{tag}.solver + version: 0.1.0 + abi: "^1.0" + kind: worker + exports: + - name: solver.add + signature: "($a: Int, $b: Int) -> Int" + artifacts: + - target: {} + file: {} + sha256: "{sha}" +"#, + HostTarget::current().triple(), + artifact.file_name().unwrap().to_string_lossy() + ), + ); + plugins.push(plugin_dir); + } + let check = cli( + &[ + "check", + "index.mq.md", + "--plugin", + plugins[0].path().to_str().unwrap(), + "--plugin", + plugins[1].path().to_str().unwrap(), + ], + dir.path(), + ); + assert_eq!(check.status.code(), Some(2), "{}", stderr(&check)); + assert!( + stderr(&check).contains("ext.duplicate_callable"), + "{}", + stderr(&check) + ); +} + +// CALL-08 +#[test] +fn bad_plugin_arguments_fail_in_check_and_run_identically() { + if !have_cc() { + eprintln!("skipping: cc is unavailable"); + return; + } + let dir = TempDir::new("call08"); + let plugin_dir = TempDir::new("call08-plugin"); + let artifact = compile_plugin(plugin_dir.path(), &[]); + write_worker_manifest( + plugin_dir.path(), + &artifact, + &add_export("solver.add", "($a: Int, $b: Int) -> Int"), + ); + for (tag, call) in [ + ("missing", "solver.add(a=1)"), + ("unknown", "solver.add(a=1, b=2, c=3)"), + ("duplicate", "solver.add(a=1, a=2, b=3)"), + ("too-many-positional", "solver.add(1, 2, 3)"), + ] { + write( + &dir.path().join("index.mq.md"), + &format!("## @exec main\n\n`return {call}`\n"), + ); + let check = cli( + &[ + "check", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + ], + dir.path(), + ); + assert_eq!(check.status.code(), Some(1), "{tag}: {}", stderr(&check)); + let expected = if tag == "duplicate" { + // The expression grammar rejects duplicate named arguments before + // call binding; either way the failure is static. + "sem.syntax" + } else { + "call.arity" + }; + assert!( + stderr(&check).contains(expected), + "{tag}: {}", + stderr(&check) + ); + let run = cli( + &[ + "run", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--allow-plugin", + "demo.solver", + ], + dir.path(), + ); + assert_eq!(run.status.code(), Some(1), "{tag}: {}", stderr(&run)); + assert!(stdout(&run).is_empty(), "{tag}"); + // `check` appends a summary line and `run` a refusal line, but the + // structured call.arity diagnostic itself must be identical. + let arity = |text: &str| { + text.lines() + .filter(|line| line.contains("call.arity") || line.contains("sem.syntax")) + .map(ToString::to_string) + .collect::>() + }; + assert!(!arity(&stderr(&check)).is_empty(), "{tag}"); + assert_eq!( + arity(&stderr(&check)), + arity(&stderr(&run)), + "{tag}: check != run" + ); + } +} + +// CALL-09 +#[test] +fn changed_artifact_digest_is_rejected_before_main_runs() { + if !have_cc() { + eprintln!("skipping: cc is unavailable"); + return; + } + let dir = TempDir::new("call09"); + let plugin_dir = TempDir::new("call09-plugin"); + let artifact = compile_plugin(plugin_dir.path(), &[]); + write_worker_manifest( + plugin_dir.path(), + &artifact, + &add_export("solver.add", "($a: Int, $b: Int) -> Int"), + ); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`打印(\"must-not-run\")`\n\n`return solver.add(a=1, b=2)`\n", + ); + // Static check succeeds: it never opens the artifact. + let check = cli( + &[ + "check", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + ], + dir.path(), + ); + assert_eq!(check.status.code(), Some(0), "{}", stderr(&check)); + // Corrupt the artifact bytes; run must reject before the entry function. + let mut bytes = std::fs::read(&artifact).unwrap(); + bytes.push(0); + std::fs::write(&artifact, bytes).unwrap(); + let run = cli( + &[ + "run", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--allow-plugin", + "demo.solver", + ], + dir.path(), + ); + assert_eq!(run.status.code(), Some(1), "{}", stderr(&run)); + assert!( + stderr(&run).contains("ext.artifact_integrity"), + "{}", + stderr(&run) + ); + assert!(stdout(&run).is_empty(), "{}", stdout(&run)); +} + +// CALL-11 +#[test] +fn local_function_reference_is_a_dynamic_language_call() { + let dir = TempDir::new("call11"); + write( + &dir.path().join("index.mq.md"), + "## @exec 平方\n\n`params(n: Int)`\n\n`return n * n`\n\n## @exec 主\n\n`f = ref(\"#平方\")`\n\n`return f(n=4)`\n", + ); + let run = cli(&["call", "index.mq.md", "主", "--json"], dir.path()); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); + let json: serde_json::Value = serde_json::from_str(&stdout(&run)).unwrap(); + assert_eq!( + json["value"], + serde_json::json!({"type": "int", "decimal": "16"}) + ); +} + +// CALL-12 +#[test] +fn calling_a_text_value_is_not_callable_and_never_reaches_the_host() { + let dir = TempDir::new("call12"); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`f = \"text\"`\n\n`return f()`\n", + ); + let check = cli(&["check", "index.mq.md"], dir.path()); + assert_eq!(check.status.code(), Some(0), "{}", stderr(&check)); + let run = cli(&["run", "index.mq.md"], dir.path()); + assert_eq!(run.status.code(), Some(1), "{}", stderr(&run)); + assert!( + stderr(&run).contains("call.not_callable") || stderr(&run).contains("not_callable"), + "{}", + stderr(&run) + ); +} + +// CALL-13 +#[test] +fn explicit_import_aliases_bind_same_named_functions_independently() { + let dir = TempDir::new("call13"); + write( + &dir.path().join("a.mq.md"), + "## @exec value\n\n`params()`\n\n`return 10`\n", + ); + write( + &dir.path().join("b.mq.md"), + "## @exec value\n\n`params()`\n\n`return 20`\n", + ); + write( + &dir.path().join("index.mq.md"), + "---\nimport:\n aa: \"./a.mq.md\"\n bb: \"./b.mq.md\"\n---\n\n## @exec main\n\n`return aa.value() + bb.value()`\n", + ); + let run = cli(&["run", "index.mq.md", "--json"], dir.path()); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); + let json: serde_json::Value = serde_json::from_str(&stdout(&run)).unwrap(); + assert_eq!( + json["value"], + serde_json::json!({"type": "int", "decimal": "30"}) + ); +} + +// CALL-14 +#[test] +fn check_never_executes_a_plugin_artifact_or_side_effect_script() { + let dir = TempDir::new("call14"); + let plugin_dir = TempDir::new("call14-plugin"); + let marker = plugin_dir.path().join("executed.marker"); + let artifact = plugin_dir.path().join(library_name()); + write( + &artifact, + &format!("#!/bin/sh\ntouch '{}'\nexit 0\n", marker.display()), + ); + let sha = sha256_file(&artifact).unwrap(); + write( + &plugin_dir.path().join("plugin.yaml"), + &format!( + r#"plugin: + name: demo.script + version: 0.1.0 + abi: "^1.0" + kind: worker + exports: + - name: demo.run + signature: "() -> Int" + artifacts: + - target: {} + file: {} + sha256: "{sha}" +"#, + HostTarget::current().triple(), + artifact.file_name().unwrap().to_string_lossy() + ), + ); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`return demo.run()`\n", + ); + let check = cli( + &[ + "check", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + ], + dir.path(), + ); + assert_eq!(check.status.code(), Some(0), "{}", stderr(&check)); + assert!(!marker.exists(), "check executed the artifact"); +} + +// CALL-15 +#[test] +fn check_emit_and_run_share_the_same_static_binding_source() { + if !have_cc() { + eprintln!("skipping: cc is unavailable"); + return; + } + let dir = TempDir::new("call15"); + let plugin_dir = TempDir::new("call15-plugin"); + let artifact = compile_plugin(plugin_dir.path(), &[]); + write_worker_manifest( + plugin_dir.path(), + &artifact, + &add_export("solver.add", "($a: Int, $b: Int) -> Int"), + ); + write( + &dir.path().join("index.mq.md"), + "## @exec main\n\n`return solver.add(a=1, b=2)`\n", + ); + let analysis_path = dir.path().join("analysis.json"); + let check = cli( + &[ + "check", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--emit", + analysis_path.to_str().unwrap(), + ], + dir.path(), + ); + assert_eq!(check.status.code(), Some(0), "{}", stderr(&check)); + let analysis: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&analysis_path).unwrap()).unwrap(); + let functions = analysis["modules"] + .as_array() + .and_then(|modules| modules.first()) + .and_then(|module| module["functions"].as_array()) + .unwrap_or_else(|| panic!("analysis has no module functions: {analysis}")); + assert!(!functions.is_empty()); + let run = cli( + &[ + "run", + "index.mq.md", + "--plugin", + plugin_dir.path().to_str().unwrap(), + "--allow-plugin", + "demo.solver", + "--json", + ], + dir.path(), + ); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); + // Without the descriptor the same call is statically unknown: run/check + // consume the same prepared binding, not a runtime name probe. + let without = cli(&["check", "index.mq.md"], dir.path()); + assert_eq!(without.status.code(), Some(1)); + assert!(stderr(&without).contains("symbol.unknown")); +} + +// CALL-16 +#[test] +fn print_alias_keeps_the_existing_output_bytes() { + let run = cli(&["run", "examples/hello.mq.md"], &repo()); + assert_eq!(run.status.code(), Some(0), "{}", stderr(&run)); + assert_eq!(stdout(&run), "你好,Marqdo\n"); +} diff --git a/crates/cli/tests/conformance_semantics.rs b/crates/cli/tests/conformance_semantics.rs index 16daef3..6ef7a19 100644 --- a/crates/cli/tests/conformance_semantics.rs +++ b/crates/cli/tests/conformance_semantics.rs @@ -166,8 +166,8 @@ fn semantic_goldens_match_end_to_end_execution() { let mut program = marqdo_ir::Program::default(); program.insert(analysis.module.clone()); program.main_module = Some(module_id.clone()); - install_stdlib_print(&mut program); install_entry_stdlib_imports(&mut program, &module_id, &analysis); + bind_calls_with_stdlib_aliases(&mut program); let mut capabilities = Capabilities::pure(); if grant_foreign { @@ -284,26 +284,88 @@ fn normalise_relative(path: &Path) -> String { .join("/") } -/// Load the bundled stdlib print modules for direct-runtime fixtures. +/// Bind direct-runtime fixtures exactly like the CLI composition root. /// -/// The CLI loader normally injects these for Compat-B. The semantic harness -/// builds a Program directly, so it mirrors the same resolution here; without -/// it, fixtures that call `打印` would hit the Compat-C "no bundle" error. -fn install_stdlib_print(program: &mut Program) { +/// The semantic harness builds a `Program` directly, so it mirrors the +/// production path: load the official stdlib public-alias targets, build the +/// unique registry, run the single binding pass and install its table. There is +/// no `print_override` and no runtime name guessing. +fn bind_calls_with_stdlib_aliases(program: &mut Program) { let root = repo_root().join("stdlib"); - if !root.join("stdlib.json").is_file() { - return; - } let mut loaded = std::collections::BTreeSet::new(); - let io = match load_stdlib_module(&root, "io.mq.md", program, &mut loaded) { - Ok(id) => id, - Err(error) => panic!("cannot load bundled stdlib: {error}"), - }; - let function = program - .module(&io) - .and_then(|module| module.function("写值")) - .unwrap_or_else(|| panic!("stdlib io has no 写值 function")); - program.print_override = Some(function.id.clone()); + let mut official = Vec::new(); + if root.join("stdlib.json").is_file() { + let text = fs::read_to_string(root.join("stdlib.json")) + .unwrap_or_else(|error| panic!("cannot read stdlib.json: {error}")); + let manifest: serde_json::Value = serde_json::from_str(&text) + .unwrap_or_else(|error| panic!("invalid stdlib.json: {error}")); + if let Some(aliases) = manifest + .get("public_aliases") + .and_then(serde_json::Value::as_array) + { + for entry in aliases { + let public_names: Vec = entry + .get("public_names") + .and_then(serde_json::Value::as_array) + .unwrap_or_else(|| panic!("public alias has no public_names")) + .iter() + .filter_map(serde_json::Value::as_str) + .map(ToString::to_string) + .collect(); + let target_module = entry + .get("target_module") + .and_then(serde_json::Value::as_str) + .unwrap_or_else(|| panic!("public alias has no target_module")); + let target_function = entry + .get("target_function") + .and_then(serde_json::Value::as_str) + .unwrap_or_else(|| panic!("public alias has no target_function")); + let module_id = load_stdlib_module(&root, target_module, program, &mut loaded) + .unwrap_or_else(|error| panic!("cannot load stdlib alias module: {error}")); + let function = program + .module(&module_id) + .and_then(|module| module.function(target_function)) + .unwrap_or_else(|| { + panic!("stdlib alias target `{target_function}` is missing") + }); + let effects = entry + .get("effects") + .and_then(serde_json::Value::as_array) + .map(|effects| { + effects + .iter() + .filter_map(serde_json::Value::as_str) + .map(ToString::to_string) + .collect() + }) + .unwrap_or_default(); + official.push(marqdo_ir::callable::OfficialAlias { + public_names, + function: function.id.clone(), + signature: marqdo_ir::callable::CallableSignature { + params: function.params.clone(), + kind: function.signature, + }, + owner_package: module_id.instance.clone(), + effects_requested: effects, + }); + } + } + } + let mut registry = program.callable_registry(); + for alias in official { + registry + .insert_official_alias(alias) + .unwrap_or_else(|error| panic!("cannot register stdlib alias: {error}")); + } + let bound = marqdo_frontend::calls::bind_program_calls(program, ®istry); + if !bound.diagnostics.is_empty() { + panic!( + "fixture failed static call binding: {:#?}", + bound.diagnostics + ); + } + program.set_call_bindings(bound.bindings); } fn load_stdlib_module( diff --git a/crates/cli/tests/negative.rs b/crates/cli/tests/negative.rs index e9f483f..d532d4e 100644 --- a/crates/cli/tests/negative.rs +++ b/crates/cli/tests/negative.rs @@ -224,6 +224,7 @@ fn runtime_negative_goldens() { let mut program = marqdo_ir::Program::default(); program.insert(analysis.module.clone()); program.main_module = Some(module_id.clone()); + bind_program_calls(&mut program); let mut host = StdHost::new( SandboxMode::Project, Capabilities::pure(), @@ -284,3 +285,15 @@ fn runtime_negative_goldens() { } let _ = fs::remove_dir_all(&project_root); } + +/// Bind direct-runtime fixtures exactly like the CLI composition root. +fn bind_program_calls(program: &mut marqdo_ir::Program) { + let registry = program.callable_registry(); + let bound = marqdo_frontend::calls::bind_program_calls(program, ®istry); + assert!( + bound.diagnostics.is_empty(), + "fixture failed static call binding: {:#?}", + bound.diagnostics + ); + program.set_call_bindings(bound.bindings); +} diff --git a/crates/cli/tests/properties.rs b/crates/cli/tests/properties.rs index e9ad322..0c221a5 100644 --- a/crates/cli/tests/properties.rs +++ b/crates/cli/tests/properties.rs @@ -395,6 +395,14 @@ fn narration_removal_preserves_execution() { let mut program = marqdo_ir::Program::default(); program.insert(analysis.module.clone()); program.main_module = Some(module_id.clone()); + let registry = program.callable_registry(); + let bound = marqdo_frontend::calls::bind_program_calls(&program, ®istry); + assert!( + bound.diagnostics.is_empty(), + "property fixture failed binding: {:#?}", + bound.diagnostics + ); + program.set_call_bindings(bound.bindings); let mut host = StdHost::new( SandboxMode::Project, Capabilities::pure(), diff --git a/crates/cli/tests/provider_cli.rs b/crates/cli/tests/provider_cli.rs new file mode 100644 index 0000000..f3db429 --- /dev/null +++ b/crates/cli/tests/provider_cli.rs @@ -0,0 +1,1050 @@ +//! PR-03 black-box CLI tests for third-party Resource Provider v2 +//! (RFC-EXT-0001 §5, ADR-0055). +//! +//! These tests drive the real `marqdo` binary (`CARGO_BIN_EXE_marqdo`) against +//! a real `provider.json` v2 bundle, a real `c-abi-1` C sensor library compiled +//! with `cc`, and the real `marqdo-provider-worker` helper. No host API is +//! called directly: this is the same path a user gets from the public CLI. +//! +//! The helper is a separate release artifact that cargo does not build for +//! `-p marqdo-cli`; [`worker_helper`] builds it with cargo if it is missing and +//! copies it next to the CLI executable, which is exactly where production +//! resolution expects it (a sibling of the running `marqdo`, never PATH). +//! +//! Linux x86_64 is the mandatory provider-worker lane (RFC-EXT-0001 §5.6). +//! Other platforms must keep this file compiling but run no provider +//! assertions; the CLI itself fails closed with `ext.platform_unsupported`. +#![cfg(all(target_os = "linux", target_arch = "x86_64"))] + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; +use std::sync::atomic::{AtomicU32, Ordering}; +use std::sync::OnceLock; + +use marqdo_plugin_abi::HostTarget; +use sha2::{Digest, Sha256}; + +const MARKER_ENV: &str = "MARQDO_TEST_PROVIDER_MARKER"; + +/// C `c-abi-1` provider: `demo.sensor` yields `hello` (5 bytes), +/// `demo.actuator` yields `xy` (2 bytes). It appends `worker-started` when the +/// worker loads the library, so tests can prove a worker was (or was not) +/// spawned for an unauthorized/invalid provider. +const SENSOR_C: &str = r#" +#include +#include +#include +#include +#include + +typedef struct { + uint64_t token; + size_t pos; + const char *data; + size_t len; + int closed; +} Stream; + +static Stream streams[64]; +static int stream_count = 0; + +static void mark(const char *what) { + const char *path = getenv("MARQDO_TEST_PROVIDER_MARKER"); + if (!path || !*path) return; + FILE *file = fopen(path, "a"); + if (!file) return; + fprintf(file, "%s\n", what); + fclose(file); +} + +uint32_t marqdo_resource_provider_abi_version(void) { + mark("worker-started"); + return 1u; +} + +int32_t marqdo_resource_open(const char *kind, const char *descriptor, + const char *rights_csv, uint64_t *out_token) { + (void)descriptor; + (void)rights_csv; + if (stream_count >= 64) return 1; + Stream *stream = &streams[stream_count]; + memset(stream, 0, sizeof(*stream)); + if (strcmp(kind, "demo.sensor") == 0) { + stream->data = "hello"; + stream->len = 5; + } else if (strcmp(kind, "demo.actuator") == 0) { + stream->data = "xy"; + stream->len = 2; + } else { + return 1; + } + stream->token = (uint64_t)(stream_count + 1); + stream_count += 1; + mark("open"); + *out_token = stream->token; + return 0; +} + +int32_t marqdo_resource_read(uint64_t token, uint8_t *buffer, uint32_t capacity, + uint32_t *out_read, int32_t *out_eof) { + for (int i = 0; i < stream_count; i++) { + Stream *stream = &streams[i]; + if (stream->token != token || stream->closed) continue; + size_t remaining = stream->len - stream->pos; + size_t count = remaining < capacity ? remaining : capacity; + memcpy(buffer, stream->data + stream->pos, count); + stream->pos += count; + *out_read = (uint32_t)count; + *out_eof = stream->pos >= stream->len ? 1 : 0; + return 0; + } + return 1; +} + +int32_t marqdo_resource_write(uint64_t token, const uint8_t *bytes, + uint32_t length, uint32_t *out_written) { + (void)token; + (void)bytes; + *out_written = length; + return 0; +} + +int32_t marqdo_resource_close(uint64_t token) { + for (int i = 0; i < stream_count; i++) { + if (streams[i].token == token) { + streams[i].closed = 1; + mark("close"); + return 0; + } + } + return 1; +} + +int32_t marqdo_resource_last_error(char *buffer, uint32_t capacity) { + if (buffer && capacity) buffer[0] = 0; + return 0; +} +"#; + +const READ_DOC: &str = r#"## @exec main + +`内存 = memory.new(1, 1)` + +`流 = resource.open("demo.sensor", "unit-1", ["read"])` + +`读取 = resource.read(流, 内存, 0, 5)` + +`关闭 = resource.close(流)` + +`return {count: 读取.value.count, eof: 读取.value.eof, closed: 关闭.ok, data: 内存[0]}` +"#; + +const TWO_KINDS_DOC: &str = r#"## @exec main + +`内存 = memory.new(1, 1)` + +`传感器 = resource.open("demo.sensor", "unit-1", ["read"])` + +`执行器 = resource.open("demo.actuator", "unit-2", ["read"])` + +`第一次 = resource.read(传感器, 内存, 0, 5)` + +`第二次 = resource.read(执行器, 内存, 0, 2)` + +`关闭传感器 = resource.close(传感器)` + +`关闭执行器 = resource.close(执行器)` + +`return [第一次.value.count, 第二次.value.count, 关闭传感器.ok, 关闭执行器.ok]` +"#; + +const TRIVIAL_DOC: &str = "## @exec main\n\n`return 1`\n"; + +fn cli_binary() -> PathBuf { + PathBuf::from(env!("CARGO_BIN_EXE_marqdo")) +} + +fn profile_dir() -> PathBuf { + cli_binary() + .parent() + .expect("the CLI binary has a parent directory") + .to_path_buf() +} + +fn repo_root() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("crates/cli has a parent") + .parent() + .expect("crates has a parent") + .to_path_buf() +} + +fn worker_file_name() -> String { + format!("marqdo-provider-worker{}", std::env::consts::EXE_SUFFIX) +} + +/// Build `marqdo-provider-worker` if needed and install it next to the CLI. +/// +/// Production resolution is only "a sibling of the current `marqdo` binary"; +/// the test therefore performs the same installation a release tarball would +/// and never weakens the resolver with an environment override or PATH lookup. +fn worker_helper() -> PathBuf { + static INSTALLED: OnceLock> = OnceLock::new(); + match INSTALLED.get_or_init(install_worker_helper) { + Ok(path) => path.clone(), + Err(error) => panic!("cannot build the provider worker helper: {error}"), + } +} + +fn install_worker_helper() -> Result { + let sibling = profile_dir().join(worker_file_name()); + if sibling.is_file() { + return Ok(sibling); + } + let repo = repo_root(); + let target_dir = repo.join("target").join("provider-worker-helper"); + let (profile, release_flag) = match profile_dir().file_name().and_then(|name| name.to_str()) { + Some("release") => ("release", Some("--release")), + _ => ("debug", None), + }; + let mut last_error = String::new(); + // A dedicated target directory avoids contending with the cargo + // invocation that is already running this test binary. + for offline in [true, false] { + let mut command = Command::new( + std::env::var_os("CARGO").unwrap_or_else(|| std::ffi::OsString::from("cargo")), + ); + command + .current_dir(&repo) + .args([ + "build", + "--package", + "marqdo-provider-worker", + "--bin", + "marqdo-provider-worker", + "--message-format=json", + "--target-dir", + ]) + .arg(&target_dir); + if let Some(flag) = release_flag { + command.arg(flag); + } + if offline { + command.arg("--offline"); + } + let output = command + .output() + .map_err(|error| format!("cannot run cargo: {error}"))?; + if !output.status.success() { + last_error = String::from_utf8_lossy(&output.stderr) + .lines() + .rev() + .take(6) + .collect::>() + .into_iter() + .rev() + .collect::>() + .join("\n"); + continue; + } + let mut executable: Option = None; + for line in output.stdout.split(|byte| *byte == b'\n') { + let Ok(message) = serde_json::from_slice::(line) else { + continue; + }; + let Some(path) = message.get("executable").and_then(|value| value.as_str()) else { + continue; + }; + if Path::new(path) + .file_name() + .map(|name| name.to_string_lossy() == worker_file_name()) + .unwrap_or(false) + { + executable = Some(PathBuf::from(path)); + } + } + let built = executable.unwrap_or_else(|| target_dir.join(profile).join(worker_file_name())); + if !built.is_file() { + last_error = format!("cargo succeeded but `{}` is missing", built.display()); + continue; + } + fs::copy(&built, &sibling).map_err(|error| { + format!( + "cannot install `{}` as the CLI sibling `{}`: {error}", + built.display(), + sibling.display() + ) + })?; + return Ok(sibling); + } + Err(last_error) +} + +/// One provider bundle under a throw-away directory. +struct Fixture { + dir: PathBuf, + lib_relative: String, + lib_sha256: String, +} + +impl Fixture { + fn new(tag: &str) -> Self { + static COUNTER: AtomicU32 = AtomicU32::new(0); + let path = std::env::temp_dir().join(format!( + "marqdo-provider-cli-{}-{}-{tag}", + std::process::id(), + COUNTER.fetch_add(1, Ordering::Relaxed) + )); + let _ = fs::remove_dir_all(&path); + fs::create_dir_all(&path).expect("create fixture directory"); + let lib_relative = "lib/libdemo_sensor.so".to_string(); + let source = path.join("sensor.c"); + fs::write(&source, SENSOR_C).expect("write sensor.c"); + let library = path.join(&lib_relative); + fs::create_dir_all(library.parent().expect("lib dir")).expect("create lib dir"); + let status = Command::new("cc") + .args(["-shared", "-fPIC", "-O2", "-o"]) + .arg(&library) + .arg(&source) + .status() + .expect("run cc"); + assert!(status.success(), "cc could not build the test provider"); + let sha256 = sha256_hex(&fs::read(&library).expect("read provider library")); + Fixture { + dir: path, + lib_relative, + lib_sha256: sha256, + } + } + + fn path(&self) -> &Path { + &self.dir + } + + fn write(&self, relative: &str, contents: &str) -> PathBuf { + let path = self.dir.join(relative); + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).expect("create parent directory"); + } + fs::write(&path, contents).expect("write fixture file"); + path + } + + fn project(&self, document: &str) -> PathBuf { + self.write("index.mq.md", document) + } + + fn manifest(&self, name: &str, kinds: &[&str]) -> String { + self.manifest_with( + name, + kinds, + &self.lib_relative, + &self.lib_sha256, + &HostTarget::current().triple().to_string(), + ) + } + + fn manifest_with( + &self, + name: &str, + kinds: &[&str], + artifact_relative: &str, + sha256: &str, + target: &str, + ) -> String { + serde_json::json!({ + "schema": "marqdo.provider", + "schema_version": 2, + "name": name, + "version": "0.1.0", + "abi": "c-abi-1", + "transport": "worker", + "kinds": kinds, + "artifacts": [{ + "target": target, + "path": artifact_relative, + "sha256": sha256, + }], + "permissions_requested": [], + }) + .to_string() + } +} + +impl Drop for Fixture { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.dir); + } +} + +fn sha256_hex(bytes: &[u8]) -> String { + let mut hasher = Sha256::new(); + hasher.update(bytes); + let mut hex = String::with_capacity(64); + for byte in hasher.finalize() { + hex.push_str(&format!("{byte:02x}")); + } + hex +} + +fn marker_path(dir: &Path) -> PathBuf { + dir.join("provider-marker.txt") +} + +fn marker_text(path: &Path) -> String { + fs::read_to_string(path).unwrap_or_default() +} + +fn run_cli(dir: &Path, args: &[&str], marker: &Path) -> Output { + Command::new(cli_binary()) + .args(args) + .current_dir(dir) + .env(MARKER_ENV, marker) + .output() + .unwrap_or_else(|error| panic!("cannot run marqdo {args:?}: {error}")) +} + +fn json_of(output: &Output) -> serde_json::Value { + serde_json::from_slice(&output.stdout).unwrap_or_else(|error| { + panic!( + "expected one JSON document, got {:?} (stderr {:?}): {error}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ) + }) +} + +fn error_code(report: &serde_json::Value) -> String { + report + .get("error") + .and_then(|error| error.get("code")) + .or_else(|| report.get("code")) + .and_then(|code| code.as_str()) + .unwrap_or("") + .to_string() +} + +fn map_field<'a>(report: &'a serde_json::Value, name: &str) -> &'a serde_json::Value { + &report["value"]["entries"][name] +} + +fn cc_available() -> bool { + Command::new("cc") + .arg("--version") + .output() + .map(|output| output.status.success()) + .unwrap_or(false) +} + +/// Every fixture test needs `cc`; this mirrors the existing CLI/worker tests. +macro_rules! require_cc { + () => { + if !cc_available() { + eprintln!("UNSUPPORTED: `cc` is unavailable, cannot build the c-abi-1 test provider"); + return; + } + }; +} + +#[test] +fn prov_01_authorized_v2_worker_reads_five_bytes() { + require_cc!(); + let helper = worker_helper(); + assert!(helper.is_file(), "worker helper must be a real executable"); + let fixture = Fixture::new("prov-01"); + fixture.project(READ_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.sensor", &["demo.sensor"]), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 manifest path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + let report = json_of(&output); + assert_eq!( + output.status.code(), + Some(0), + "run failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(report["success"].as_bool().unwrap_or(false)); + assert_eq!(map_field(&report, "count")["decimal"], "5"); + assert!(map_field(&report, "eof")["value"] + .as_bool() + .unwrap_or(false)); + assert!(map_field(&report, "closed")["value"] + .as_bool() + .unwrap_or(false)); + // `hello` is in the Memory region the provider filled on the first byte. + assert_eq!(map_field(&report, "data")["decimal"], "104"); + assert_eq!(report["stdout"], ""); + // The Host recorded the real resource effect (not a synthetic success). + let trace = report["trace"].as_array().expect("trace array"); + assert!( + trace + .iter() + .any(|entry| entry["op"] == "resource.open" && entry["result"] == "ok"), + "missing resource.open trace: {trace:?}" + ); + let marks = marker_text(&marker); + assert_eq!( + marks + .lines() + .filter(|line| *line == "worker-started") + .count(), + 1, + "exactly one worker must start: {marks:?}" + ); + assert!(marks.lines().any(|line| line == "open"), "{marks:?}"); + assert_eq!( + marks.lines().filter(|line| *line == "close").count(), + 1, + "close must happen exactly once: {marks:?}" + ); +} + +#[test] +fn prov_02_without_grant_is_permission_denied_and_never_starts_the_worker() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-02"); + fixture.project(READ_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.sensor", &["demo.sensor"]), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 manifest path"), + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(1)); + let report = json_of(&output); + assert!(!report["success"].as_bool().unwrap_or(true)); + assert_eq!(error_code(&report), "permission.denied"); + assert!( + !marker.exists(), + "an unauthorized provider worker was started: {:?}", + marker_text(&marker) + ); +} + +#[test] +fn prov_03_grant_without_manifest_is_never_discovered() { + require_cc!(); + let fixture = Fixture::new("prov-03"); + fixture.project(TRIVIAL_DOC); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(2)); + let report = json_of(&output); + assert_eq!(error_code(&report), "ext.provider_grant_unmatched"); + assert!( + !marker.exists(), + "a provider was discovered without --provider" + ); +} + +#[test] +fn prov_04_duplicate_kind_across_providers_is_rejected_before_spawn() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-04"); + fixture.project(TRIVIAL_DOC); + let first = fixture.write( + "first.json", + &fixture.manifest("demo.one", &["demo.sensor"]), + ); + let second = fixture.write( + "second.json", + &fixture.manifest("demo.two", &["demo.sensor"]), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + first.to_str().expect("utf-8 path"), + "--provider", + second.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.one", + "--allow-provider", + "demo.two", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(2)); + let report = json_of(&output); + assert_eq!(error_code(&report), "ext.duplicate_provider_kind"); + assert!( + !marker.exists(), + "a duplicate kind must fail before any worker starts" + ); +} + +#[test] +fn prov_05_one_provider_serves_two_non_reserved_kinds() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-05"); + fixture.project(TWO_KINDS_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.multi", &["demo.sensor", "demo.actuator"]), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 manifest path"), + "--allow-provider", + "demo.multi", + "--json", + ], + &marker, + ); + assert_eq!( + output.status.code(), + Some(0), + "run failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + let report = json_of(&output); + let items = report["value"]["items"].as_array().expect("list result"); + assert_eq!(items[0]["decimal"], "5"); + assert_eq!(items[1]["decimal"], "2"); + assert!(items[2]["value"].as_bool().unwrap_or(false)); + assert!(items[3]["value"].as_bool().unwrap_or(false)); + let marks = marker_text(&marker); + assert_eq!( + marks + .lines() + .filter(|line| *line == "worker-started") + .count(), + 1, + "one provider (one worker) serves both kinds: {marks:?}" + ); + assert_eq!( + marks.lines().filter(|line| *line == "open").count(), + 2, + "{marks:?}" + ); + assert_eq!( + marks.lines().filter(|line| *line == "close").count(), + 2, + "{marks:?}" + ); +} + +#[test] +fn prov_06_reserved_core_kinds_are_rejected() { + require_cc!(); + let _ = worker_helper(); + for kind in ["stdio", "file", "tcp", "tls", "udp", "tcp-listener"] { + let fixture = Fixture::new("prov-06"); + fixture.project(TRIVIAL_DOC); + let manifest = fixture.write("provider.json", &fixture.manifest("demo.reserved", &[kind])); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 manifest path"), + "--allow-provider", + "demo.reserved", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(2), "kind {kind}"); + let report = json_of(&output); + assert_eq!(error_code(&report), "ext.manifest_invalid", "kind {kind}"); + assert!( + !marker.exists(), + "reserved kind {kind} must not start a worker" + ); + } +} + +#[test] +fn prov_07_digest_symlink_and_traversal_fail_before_spawn() { + require_cc!(); + let _ = worker_helper(); + + // (a) content digest mismatch. + let fixture = Fixture::new("prov-07-digest"); + fixture.project(TRIVIAL_DOC); + let wrong_sha = "0".repeat(64); + let digest_manifest = fixture.write( + "provider.json", + &fixture.manifest_with( + "demo.sensor", + &["demo.sensor"], + &fixture.lib_relative, + &wrong_sha, + &HostTarget::current().triple().to_string(), + ), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + digest_manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(1)); + assert_eq!(error_code(&json_of(&output)), "ext.artifact_integrity"); + assert!(!marker.exists(), "digest mismatch must not start a worker"); + + // (b) artifact symlink escaping the *bundle root* (the manifest parent): + // the hash matches the real library, so only the bundle-containment check + // can reject it. + let fixture = Fixture::new("prov-07-symlink"); + fixture.project(TRIVIAL_DOC); + let bundle = fixture.path().join("bundle"); + fs::create_dir_all(bundle.join("lib")).expect("create bundle dir"); + let outside = fixture.path().join("outside"); + fs::create_dir_all(&outside).expect("create outside dir"); + let outside_library = outside.join("libdemo_sensor.so"); + fs::copy(fixture.path().join(&fixture.lib_relative), &outside_library) + .expect("copy library outside the bundle"); + std::os::unix::fs::symlink(&outside_library, bundle.join(&fixture.lib_relative)) + .expect("create escaping symlink"); + let symlink_manifest = bundle.join("provider.json"); + fs::write( + &symlink_manifest, + fixture.manifest("demo.sensor", &["demo.sensor"]), + ) + .expect("write bundle manifest"); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + symlink_manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(1)); + assert_eq!(error_code(&json_of(&output)), "ext.artifact_integrity"); + assert!(!marker.exists(), "symlink escape must not start a worker"); + + // (c) `..` path traversal declared in the manifest. + let fixture = Fixture::new("prov-07-traversal"); + fixture.project(TRIVIAL_DOC); + let traversal_manifest = fixture.write( + "provider.json", + &fixture.manifest_with( + "demo.sensor", + &["demo.sensor"], + "../lib/libdemo_sensor.so", + &fixture.lib_sha256, + &HostTarget::current().triple().to_string(), + ), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + traversal_manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(1)); + assert_eq!(error_code(&json_of(&output)), "ext.artifact_integrity"); + assert!(!marker.exists(), "path traversal must not start a worker"); +} + +#[test] +fn prov_08_missing_current_target_is_no_matching_artifact() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-08"); + fixture.project(TRIVIAL_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest_with( + "demo.sensor", + &["demo.sensor"], + &fixture.lib_relative, + &fixture.lib_sha256, + "aarch64-unknown-linux-gnu", + ), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(2)); + assert_eq!(error_code(&json_of(&output)), "ext.no_matching_artifact"); + assert!(!marker.exists()); +} + +#[test] +fn prov_09_v1_manifest_is_unsupported_without_migration() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-09"); + fixture.project(TRIVIAL_DOC); + let v1 = serde_json::json!({ + "schema": "marqdo.provider", + "schema_version": 1, + "name": "demo.sensor", + "kinds": ["demo.sensor"], + "abi": "c-abi-1", + "library": { "linux": fixture.lib_relative }, + }) + .to_string(); + let manifest = fixture.write("provider.json", &v1); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(output.status.code(), Some(2)); + let report = json_of(&output); + assert_eq!(error_code(&report), "ext.unsupported_provider_schema"); + assert!( + !marker.exists(), + "the v1 host-embedding manifest path is not reachable from the CLI" + ); +} + +#[test] +fn prov_10_pure_and_strict_preflight_refuse_provider_flags() { + require_cc!(); + let _ = worker_helper(); + let fixture = Fixture::new("prov-10"); + fixture.project(TRIVIAL_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.sensor", &["demo.sensor"]), + ); + let marker = marker_path(fixture.path()); + let manifest_arg = manifest.to_str().expect("utf-8 manifest path"); + + let pure = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--mode", + "pure", + "--provider", + manifest_arg, + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(pure.status.code(), Some(2)); + assert_eq!(error_code(&json_of(&pure)), "cli.invalid_grants"); + assert!( + !marker.exists(), + "pure mode must not start a provider worker" + ); + + let strict = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--mode", + "strict", + "--provider", + manifest_arg, + "--allow-provider", + "demo.sensor", + "--json", + ], + &marker, + ); + assert_eq!(strict.status.code(), Some(2)); + assert_eq!(error_code(&json_of(&strict)), "strict.unsupported_grant"); + assert!( + !marker.exists(), + "strict L1 must not start a provider worker" + ); + + // The wildcard grant is rejected too: grants are exact names. + let wildcard = run_cli( + fixture.path(), + &[ + "run", + "index.mq.md", + "--provider", + manifest_arg, + "--allow-provider", + "*", + "--json", + ], + &marker, + ); + assert_eq!(wildcard.status.code(), Some(2)); + assert_eq!(error_code(&json_of(&wildcard)), "cli.invalid_grants"); +} + +#[test] +fn prov_13_missing_sibling_helper_is_worker_unavailable_without_path_fallback() { + require_cc!(); + // Install the helper next to the real CLI binary and put that directory on + // PATH: a PATH-based resolver would now succeed, which must not happen. + let helper = worker_helper(); + let helper_dir = helper.parent().expect("helper directory").to_path_buf(); + let fixture = Fixture::new("prov-13"); + fixture.project(READ_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.sensor", &["demo.sensor"]), + ); + let copied_dir = fixture.path().join("release"); + fs::create_dir_all(&copied_dir).expect("create release dir"); + let copied_cli = copied_dir.join("marqdo"); + fs::copy(cli_binary(), &copied_cli).expect("copy the CLI binary"); + let marker = marker_path(fixture.path()); + let path = format!( + "{}:{}", + helper_dir.display(), + std::env::var("PATH").unwrap_or_default() + ); + let output = Command::new(&copied_cli) + .args([ + "run", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 path"), + "--allow-provider", + "demo.sensor", + "--json", + ]) + .current_dir(fixture.path()) + .env("PATH", path) + .env(MARKER_ENV, &marker) + .output() + .expect("run the relocated CLI"); + assert_eq!( + output.status.code(), + Some(1), + "stderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + let report = json_of(&output); + assert_eq!(error_code(&report), "ext.worker_unavailable"); + let message = report["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("marqdo-provider-worker") && message.contains("missing"), + "{message}" + ); + assert!( + !marker.exists(), + "the helper on PATH must never be used instead of the sibling" + ); +} + +#[test] +fn check_never_accepts_or_loads_provider_manifests() { + require_cc!(); + let fixture = Fixture::new("check"); + fixture.project(TRIVIAL_DOC); + let manifest = fixture.write( + "provider.json", + &fixture.manifest("demo.sensor", &["demo.sensor"]), + ); + let marker = marker_path(fixture.path()); + let output = run_cli( + fixture.path(), + &[ + "check", + "index.mq.md", + "--provider", + manifest.to_str().expect("utf-8 path"), + ], + &marker, + ); + // `check` has no --provider option (clap usage error, exit 2) and never + // starts a provider worker. + assert_eq!(output.status.code(), Some(2)); + assert!( + String::from_utf8_lossy(&output.stderr).contains("unexpected argument"), + "stderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(!marker.exists()); +} diff --git a/crates/cli/tests/stdlib_alias_routing.rs b/crates/cli/tests/stdlib_alias_routing.rs new file mode 100644 index 0000000..5747ec5 --- /dev/null +++ b/crates/cli/tests/stdlib_alias_routing.rs @@ -0,0 +1,260 @@ +//! Official stdlib public alias routing (RFC-EXT-0001 §8, ADR-0058). +//! +//! `打印`/`print` are ordinary `Marqdo` bindings to `io.mq.md::写值` declared +//! in the verified `stdlib.json` metadata. There is no `print_override`, no +//! runtime name special case and no hard-coded per-function scan. +#![cfg(unix)] + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; +use std::sync::atomic::{AtomicU32, Ordering}; + +static COUNTER: AtomicU32 = AtomicU32::new(0); + +struct TempDir(PathBuf); + +impl TempDir { + fn new(tag: &str) -> Self { + let id = COUNTER.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "marqdo-stdlib-alias-test-{}-{id}-{tag}", + std::process::id() + )); + std::fs::create_dir_all(&path).unwrap(); + Self(path) + } + + fn path(&self) -> &Path { + &self.0 + } +} + +impl Drop for TempDir { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .parent() + .unwrap() + .to_path_buf() +} + +fn cli(args: &[&str], dir: &Path, env: &[(&str, &Path)]) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_marqdo")); + command.args(args).current_dir(dir); + for (key, value) in env { + command.env(key, value); + } + command.output().unwrap() +} + +fn stderr(output: &Output) -> String { + String::from_utf8_lossy(&output.stderr).to_string() +} + +fn stdout(output: &Output) -> String { + String::from_utf8_lossy(&output.stdout).to_string() +} + +fn copy_stdlib(target: &Path) { + let source = repo().join("stdlib"); + std::fs::create_dir_all(target).unwrap(); + for entry in std::fs::read_dir(&source).unwrap() { + let entry = entry.unwrap(); + let path = entry.path(); + if path.is_dir() { + copy_dir(&path, &target.join(entry.file_name())); + } else { + std::fs::copy(&path, target.join(entry.file_name())).unwrap(); + } + } +} + +fn copy_dir(source: &Path, target: &Path) { + std::fs::create_dir_all(target).unwrap(); + for entry in std::fs::read_dir(source).unwrap() { + let entry = entry.unwrap(); + let path = entry.path(); + if path.is_dir() { + copy_dir(&path, &target.join(entry.file_name())); + } else { + std::fs::copy(&path, target.join(entry.file_name())).unwrap(); + } + } +} + +#[test] +fn print_and_its_alias_are_bound_to_the_official_module() { + let dir = TempDir::new("print"); + std::fs::write( + dir.path().join("index.mq.md"), + "## @exec main\n\n`打印(1)`\n\n`print(\"two\")`\n\n`return None`\n", + ) + .unwrap(); + let output = cli(&["run", "index.mq.md"], dir.path(), &[]); + assert_eq!(output.status.code(), Some(0), "{}", stderr(&output)); + assert_eq!(stdout(&output), "1\ntwo\n"); +} + +#[test] +fn program_without_print_does_not_load_the_stdlib_module_graph() { + let dir = TempDir::new("no-print"); + std::fs::write( + dir.path().join("index.mq.md"), + "## @exec main\n\n`params()`\n\n`return 1 + 2`\n", + ) + .unwrap(); + let analysis_path = dir.path().join("factorial.json"); + let output = cli( + &[ + "check", + "index.mq.md", + "--emit", + analysis_path.to_str().unwrap(), + ], + dir.path(), + &[], + ); + assert_eq!(output.status.code(), Some(0), "{}", stderr(&output)); + let analysis: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&analysis_path).unwrap()).unwrap(); + let modules = analysis["modules"].as_array().unwrap(); + assert!( + modules + .iter() + .all(|module| !module["id"].as_str().unwrap_or("").starts_with("std:")), + "a program without print must not load the stdlib: {analysis}" + ); +} + +#[test] +fn print_program_loads_exactly_the_declared_alias_target() { + let dir = TempDir::new("with-print"); + std::fs::write( + dir.path().join("index.mq.md"), + "## @exec main\n\n`打印(\"x\")`\n\n`return None`\n", + ) + .unwrap(); + let analysis_path = dir.path().join("hello.json"); + let output = cli( + &[ + "check", + "index.mq.md", + "--emit", + analysis_path.to_str().unwrap(), + ], + dir.path(), + &[], + ); + assert_eq!(output.status.code(), Some(0), "{}", stderr(&output)); + let analysis: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&analysis_path).unwrap()).unwrap(); + let ids: Vec<&str> = analysis["modules"] + .as_array() + .unwrap() + .iter() + .filter_map(|module| module["id"].as_str()) + .collect(); + assert!( + ids.iter().any(|id| id.ends_with("::io.mq.md")), + "print must load io.mq.md: {ids:?}" + ); + assert!( + ids.iter().any(|id| id.ends_with("::format.mq.md")), + "io.mq.md imports format.mq.md: {ids:?}" + ); +} + +#[test] +fn tampered_public_alias_metadata_digest_is_rejected() { + let dir = TempDir::new("alias-digest"); + let stdlib = dir.path().join("stdlib"); + copy_stdlib(&stdlib); + let manifest_path = stdlib.join("stdlib.json"); + let text = std::fs::read_to_string(&manifest_path).unwrap(); + let mut manifest: serde_json::Value = serde_json::from_str(&text).unwrap(); + manifest["aliases_digest"] = serde_json::json!("sha256:".to_string() + &"0".repeat(64)); + std::fs::write( + &manifest_path, + serde_json::to_string_pretty(&manifest).unwrap() + "\n", + ) + .unwrap(); + let doc = dir.path().join("index.mq.md"); + std::fs::write(&doc, "## @exec main\n\n`打印(1)`\n\n`return None`\n").unwrap(); + let output = cli( + &["run", doc.to_str().unwrap()], + dir.path(), + &[("MARQDO_STDLIB_DIR", &stdlib)], + ); + assert!(!output.status.success(), "{}", stderr(&output)); + assert!( + stderr(&output).contains("public-alias metadata digest mismatch"), + "{}", + stderr(&output) + ); +} + +#[test] +fn tampered_alias_target_signature_is_rejected() { + let dir = TempDir::new("alias-target"); + let stdlib = dir.path().join("stdlib"); + copy_stdlib(&stdlib); + let manifest_path = stdlib.join("stdlib.json"); + let text = std::fs::read_to_string(&manifest_path).unwrap(); + let mut manifest: serde_json::Value = serde_json::from_str(&text).unwrap(); + manifest["public_aliases"][0]["expected_signature"] = serde_json::json!("params(值, 额外)"); + // Recompute the declared metadata digest so only the target check can fail. + let canonical = manifest["public_aliases"][0].clone(); + let module = canonical["target_module"].as_str().unwrap(); + let function = canonical["target_function"].as_str().unwrap(); + let mut names: Vec<&str> = canonical["public_names"] + .as_array() + .unwrap() + .iter() + .filter_map(serde_json::Value::as_str) + .collect(); + names.sort(); + use sha2::{Digest, Sha256}; + let mut hasher = Sha256::new(); + hasher.update(module.as_bytes()); + hasher.update([0]); + hasher.update(function.as_bytes()); + hasher.update([0]); + for name in names { + hasher.update(name.as_bytes()); + hasher.update([0]); + } + let digest = format!("sha256:{}", hex(&hasher.finalize())); + manifest["aliases_digest"] = serde_json::json!(digest); + std::fs::write( + &manifest_path, + serde_json::to_string_pretty(&manifest).unwrap() + "\n", + ) + .unwrap(); + let doc = dir.path().join("index.mq.md"); + std::fs::write(&doc, "## @exec main\n\n`打印(1)`\n\n`return None`\n").unwrap(); + let output = cli( + &["run", doc.to_str().unwrap()], + dir.path(), + &[("MARQDO_STDLIB_DIR", &stdlib)], + ); + assert!(!output.status.success(), "{}", stderr(&output)); + assert!( + stderr(&output).contains("expected signature") || stderr(&output).contains("alias"), + "{}", + stderr(&output) + ); +} + +fn hex(bytes: &[u8]) -> String { + let mut out = String::with_capacity(bytes.len() * 2); + for byte in bytes { + out.push_str(&format!("{byte:02x}")); + } + out +} diff --git a/crates/marqdo-frontend/src/analyze.rs b/crates/marqdo-frontend/src/analyze.rs index ceed252..1db046b 100644 --- a/crates/marqdo-frontend/src/analyze.rs +++ b/crates/marqdo-frontend/src/analyze.rs @@ -39,7 +39,30 @@ impl AnalysisResult { /// Analyze one Markdown source as a Marqdo module. /// /// `module_id` is the canonical project-relative identity (e.g. `index.mq.md`). +/// +/// This standalone entry point also runs the module-local call check so that a +/// single document can be validated without a program graph. The CLI +/// composition root uses [`analyze_source_for_program`] instead: it defers all +/// call resolution to the single program-wide binding pass (ADR-0054). pub fn analyze_source(module_id: &str, source: &str) -> AnalysisResult { + analyze_source_inner(module_id, source, true) +} + +/// Analyze one Markdown source for the composition root. +/// +/// Identical to [`analyze_source`] except that it does not run the provisional +/// module-local call check; the caller binds calls exactly once over the whole +/// program (including extension descriptors) and therefore owns the only +/// authoritative call diagnostics. +pub fn analyze_source_for_program(module_id: &str, source: &str) -> AnalysisResult { + analyze_source_inner(module_id, source, false) +} + +fn analyze_source_inner( + module_id: &str, + source: &str, + validate_local_calls: bool, +) -> AnalysisResult { let id = ModuleId::from(module_id); let document = marqdo_markdown::parse(source); let units = units::build_units(&document, &id); @@ -161,7 +184,9 @@ pub fn analyze_source(module_id: &str, source: &str) -> AnalysisResult { resources, anchors: units.anchors.clone(), }; - diagnostics.extend(calls::validate_module(&module)); + if validate_local_calls { + diagnostics.extend(calls::validate_module(&module)); + } // One stable order for every consumer: source position first, then the // stable code, as required by `spec/mapping.md` §8. Diagnostics without a diff --git a/crates/marqdo-frontend/src/calls.rs b/crates/marqdo-frontend/src/calls.rs index 4252849..ee25cb1 100644 --- a/crates/marqdo-frontend/src/calls.rs +++ b/crates/marqdo-frontend/src/calls.rs @@ -1,33 +1,51 @@ -//! Static call validation. +//! Static call resolution and binding (RFC-EXT-0001 §4, ADR-0054). //! -//! Calls are resolved after all functions have signatures (including inferred -//! ones), so arity/name mistakes are compile errors with source spans instead -//! of surprises at runtime. Type errors stay runtime concerns: v1 has no static -//! type inference beyond literal checks. +//! This module owns the *single* resolution algorithm used by `check`, `run`, +//! `inspect` and Analysis: it walks every statically visible call site exactly +//! once, resolves it against [`marqdo_ir::callable::CallableRegistry`] and +//! records a [`marqdo_ir::call_binding::BoundTarget`] in one +//! [`marqdo_ir::call_binding::CallBindingTable`]. The runtime consumes that +//! table and never re-resolves a name. +//! +//! Resolution priority is explicit and order-independent: +//! +//! | call form | rule | +//! |---|---| +//! | `f(...)` | local variable (dynamic) → official alias/plugin-fixed builtin → module function (current namespace first, then canonical, then unique local) | +//! | `alias.f(...)` | explicit import alias → builtin dotted prefix → exact plugin export | +//! | `some_value.f(...)` | dynamic language value only | +//! +//! Import aliases that collide with a builtin dotted namespace or a visible +//! plugin namespace are rejected with `ext.import_namespace_conflict`; declared +//! plugin exports that collide with each other or with builtins are rejected +//! when the registry is built. The previous "try builtin, then guess plugin at +//! runtime" behaviour is gone. +//! +//! `validate_module` keeps the standalone frontend API (a single document with +//! no program graph) for AST/conformance tests: unresolved imports are +//! *deferred* there because only the composition root can bind a whole program. use std::collections::{BTreeMap, BTreeSet}; use marqdo_expression::{Arg, Expr, Ident}; -use marqdo_ir::{builtins, Module, ModuleId, Param, PlaceTarget, Program, SignatureKind, Stmt}; +use marqdo_ir::builtins; +use marqdo_ir::call_binding::{BoundTarget, CallBindingTable, CallSiteKey}; +use marqdo_ir::callable::{CallableDescriptor, CallableRegistry}; +use marqdo_ir::{ + Function, FunctionId, Module, ModuleId, PlaceTarget, Program, SignatureKind, Stmt, +}; use marqdo_span::Span; use marqdo_value::{codes, Diagnostic}; -/// Signature information needed to validate a call. -#[derive(Debug, Clone)] -pub struct CallableSignature { - /// Parameter list. - pub params: Vec, - /// Explicit signatures allow positional arguments; inferred do not. - pub kind: SignatureKind, -} - -/// Build the callable signature table for a module. -pub fn signature_table(module: &Module) -> BTreeMap { +/// Build the callable signature table for one module (diagnostics helper). +pub fn signature_table( + module: &Module, +) -> BTreeMap { let mut table = BTreeMap::new(); for function in &module.functions { table.insert( function.id.name.clone(), - CallableSignature { + marqdo_ir::callable::CallableSignature { params: function.params.clone(), kind: function.signature, }, @@ -36,10 +54,107 @@ pub fn signature_table(module: &Module) -> BTreeMap { table } -/// Validate every call in every function body of the module. +/// Result of binding a whole program. +#[derive(Debug, Clone, Default)] +pub struct BoundProgram { + /// One deterministic binding per static call site. + pub bindings: CallBindingTable, + /// Call-resolution diagnostics, sorted later by the caller. + pub diagnostics: Vec<(ModuleId, Diagnostic)>, +} + +/// Validate every call in a single module (standalone frontend entry). +/// +/// Imports and plugin exports cannot be resolved without a program graph, so +/// those call sites are deferred to the composition-root binding pass instead +/// of producing false errors or a second priority order. pub fn validate_module(module: &Module) -> Vec { - let table = signature_table(module); + let mut program = Program::default(); + program.modules.insert(module.id.clone(), module.clone()); + program.main_module = Some(module.id.clone()); + let registry = CallableRegistry::try_from_program(&program) + .expect("a single module cannot contain duplicate external declarations"); + bind_with(module, &program, ®istry, true) + .diagnostics + .into_iter() + .map(|(_, diagnostic)| diagnostic) + .collect() +} + +/// Validate every call in a loaded program (tests and standalone frontend API). +/// +/// The CLI composition root uses [`bind_program_calls`] and keeps the binding +/// table; this wrapper keeps the historical API for frontend tests. +pub fn validate_program(program: &Program) -> Vec<(ModuleId, Diagnostic)> { + let registry = match CallableRegistry::try_from_program(program) { + Ok(registry) => registry, + Err(_) => CallableRegistry::from_program(program), + }; + bind_program_calls_with(program, ®istry, false).diagnostics +} + +/// Resolve every call site in `program` against `registry` exactly once. +/// +/// This is the only binding entry point used by the composition root. The +/// returned table must be installed on [`Program::set_call_bindings`] before +/// execution. +pub fn bind_program_calls(program: &Program, registry: &CallableRegistry) -> BoundProgram { + bind_program_calls_with(program, registry, false) +} + +fn bind_program_calls_with( + program: &Program, + registry: &CallableRegistry, + standalone: bool, +) -> BoundProgram { + let mut result = BoundProgram::default(); + for module in program.modules.values() { + let mut bound = bind_with(module, program, registry, standalone); + // Conflict diagnostics are produced once per module import, before + // any call site; keep them in front of call diagnostics. + result.diagnostics.append(&mut bound.diagnostics); + for (key, target) in bound.bindings.iter() { + result.bindings.insert(key.clone(), target.clone()); + } + } + result +} + +fn bind_with( + module: &Module, + program: &Program, + registry: &CallableRegistry, + standalone: bool, +) -> BoundProgram { + let mut conflicted = BTreeSet::new(); let mut diagnostics = Vec::new(); + // Reject import aliases that collide with a fixed builtin dotted namespace + // or a visible plugin namespace. `alias.f(...)` must have one target. + for import in &module.imports { + let builtin_conflict = registry.has_builtin_dotted_prefix(&import.alias); + let plugin_conflict = registry.has_external_dotted_prefix(&import.alias); + if builtin_conflict || plugin_conflict { + conflicted.insert(import.alias.clone()); + let owner = if builtin_conflict { + "a builtin dotted namespace" + } else { + "a declared plugin export namespace" + }; + diagnostics.push(( + module.id.clone(), + Diagnostic::error( + codes::EXT_IMPORT_NAMESPACE_CONFLICT, + Some(import.span), + format!("import alias `{}` conflicts with {owner}", import.alias), + ) + .with_hint(format!( + "rename the import alias, for example `{}_lib`, so `{}.…` has one meaning", + import.alias, import.alias + )), + )); + } + } + let mut bound = BoundProgram::default(); for function in &module.functions { let mut locals: BTreeSet = function .params @@ -47,73 +162,50 @@ pub fn validate_module(module: &Module) -> Vec { .map(|param| param.name.clone()) .collect(); collect_assigned(&function.body, &mut locals); - let mut check = CallChecker { - current_namespace: function.namespace.clone(), + let mut binder = Binder { + module, + program, + registry, + function_id: function.id.clone(), + namespace: function.namespace.clone(), locals: &locals, - table: &table, - imports: module - .imports - .iter() - .map(|import| import.alias.clone()) - .collect(), - module_id: module.id.clone(), - program: None, - cross_only: false, + standalone, + conflicted: &conflicted, + bindings: CallBindingTable::default(), diagnostics: Vec::new(), }; - check.stmts(&function.body); - diagnostics.extend(check.diagnostics); - } - diagnostics -} - -/// Validate cross-module and qualified-builtin calls once a whole program is -/// loaded. Local calls are already validated by [`validate_module`]. -pub fn validate_program(program: &Program) -> Vec<(ModuleId, Diagnostic)> { - let mut results = Vec::new(); - for module in program.modules.values() { - let table = signature_table(module); - for function in &module.functions { - let mut locals: BTreeSet = function - .params - .iter() - .map(|param| param.name.clone()) - .collect(); - collect_assigned(&function.body, &mut locals); - let mut check = CallChecker { - current_namespace: function.namespace.clone(), - locals: &locals, - table: &table, - imports: module - .imports - .iter() - .map(|import| import.alias.clone()) - .collect(), - module_id: module.id.clone(), - program: Some(program), - cross_only: true, - diagnostics: Vec::new(), - }; - check.stmts(&function.body); - results.extend( - check - .diagnostics - .into_iter() - .map(|diagnostic| (module.id.clone(), diagnostic)), - ); + for param in &function.params { + // Signature defaults are compile-time expressions evaluated by the + // runtime in the callee's own frame; bind their calls with the + // callee identity so the runtime lookup matches. + if let Some(default) = ¶m.default { + binder.expr(default); + } } + binder.stmts(&function.body); + for (key, target) in binder.bindings.iter() { + bound.bindings.insert(key.clone(), target.clone()); + } + diagnostics.extend( + binder + .diagnostics + .into_iter() + .map(|diagnostic| (module.id.clone(), diagnostic)), + ); + } + BoundProgram { + bindings: bound.bindings, + diagnostics, } - results } fn collect_assigned(stmts: &[Stmt], out: &mut BTreeSet) { for stmt in stmts { match stmt { - Stmt::Assign { - target: PlaceTarget::Local { name, .. }, - .. - } => { - out.insert(name.clone()); + Stmt::Assign { target, .. } => { + if let PlaceTarget::Local { name, .. } = target { + out.insert(name.clone()); + } } Stmt::Branch { arms, .. } => { for arm in arms { @@ -121,23 +213,25 @@ fn collect_assigned(stmts: &[Stmt], out: &mut BTreeSet) { } } Stmt::Loop { body, .. } => collect_assigned(body, out), - _ => {} + Stmt::Eval { .. } | Stmt::Return { .. } => {} } } } -struct CallChecker<'a> { - current_namespace: Option, +struct Binder<'a> { + module: &'a Module, + program: &'a Program, + registry: &'a CallableRegistry, + function_id: FunctionId, + namespace: Option, locals: &'a BTreeSet, - table: &'a BTreeMap, - imports: BTreeSet, - module_id: ModuleId, - program: Option<&'a Program>, - cross_only: bool, + standalone: bool, + conflicted: &'a BTreeSet, + bindings: CallBindingTable, diagnostics: Vec, } -impl<'a> CallChecker<'a> { +impl<'a> Binder<'a> { fn stmts(&mut self, stmts: &[Stmt]) { for stmt in stmts { match stmt { @@ -192,205 +286,309 @@ impl<'a> CallChecker<'a> { } } + fn bind(&mut self, span: Span, target: BoundTarget) { + let key = CallSiteKey::new(self.function_id.clone(), span); + self.bindings.insert(key, target); + } + + fn diagnostic(&mut self, code: &str, span: Span, message: String, hint: Option) { + let mut diagnostic = Diagnostic::error(code, Some(span), message); + if let Some(hint) = hint { + diagnostic = diagnostic.with_hint(hint); + } + self.diagnostics.push(diagnostic); + } + fn call(&mut self, callee: &Expr, args: &[Arg], span: Span) { match callee { - Expr::Ident(Ident { name, .. }) => { - if self.cross_only { + Expr::Ident(Ident { + name, + span: name_span, + }) => { + if self.locals.contains(name) { + self.bind(span, BoundTarget::DynamicValue); + return; + } + if let Some(target) = self.registry.alias_target(name).cloned() { + if let Some(descriptor) = self.registry.builtin(name) { + self.validate_signature( + &descriptor.signature, + args, + name, + span, + *name_span, + ); + } + self.bind(span, BoundTarget::Marqdo { function: target }); return; } if let Some(builtin) = builtins::find(name) { if let Err(message) = builtins::validate_args(builtin, args) { - self.diagnostics.push( - Diagnostic::error(codes::CALL_ARITY, Some(span), message) - .with_hint("check the builtin parameter list"), + self.diagnostic( + codes::CALL_ARITY, + span, + message, + Some("check the builtin parameter list".to_string()), ); } + self.bind( + span, + BoundTarget::Builtin { + canonical_name: builtin.name.to_string(), + }, + ); return; } - let name_span = match callee { - Expr::Ident(Ident { span, .. }) => *span, - _ => span, - }; - if let Some(signature) = self.resolve(name) { - self.validate_against(name, signature, args, span, name_span); - return; + match self.resolve_module_function(name) { + Resolution::One(function) => { + if let Some(target) = self.program.function(&function) { + let signature = marqdo_ir::callable::CallableSignature { + params: target.params.clone(), + kind: target.signature, + }; + self.validate_signature(&signature, args, name, span, *name_span); + } + self.bind(span, BoundTarget::Marqdo { function }); + } + Resolution::Ambiguous => { + self.diagnostic( + codes::SYMBOL_UNKNOWN, + *name_span, + format!("ambiguous function name `{name}`"), + Some( + "qualify it with its namespace, for example `namespace.函数(...)`" + .to_string(), + ), + ); + } + Resolution::None => { + if self.standalone + && self + .module + .imports + .iter() + .any(|import| import.alias == *name) + { + // A local may hold import-related values in the + // standalone frontend API; defer. + return; + } + self.diagnostic( + codes::SYMBOL_UNKNOWN, + *name_span, + format!("unknown function `{name}`"), + Some( + "define it as a level-2 heading, call a builtin, or load the \ + plugin that declares it" + .to_string(), + ), + ); + } } - if self.locals.contains(name) || self.imports.contains(name) { - // A local may hold a FunctionRef value; resolved at runtime. - return; - } - self.diagnostics.push( - Diagnostic::error( - codes::SYMBOL_UNKNOWN, - Some(name_span), - format!("unknown function `{name}`"), - ) - .with_hint("define it as a level-2 heading or call a builtin"), - ); } Expr::Member { base, name, .. } => { - if let Expr::Ident(Ident { name: alias, .. }) = base.as_ref() { - let qualified = format!("{}.{name}", alias); - if let Some(builtin) = builtins::find(&qualified) { - if let Err(message) = builtins::validate_args(builtin, args) { - self.diagnostics.push( - Diagnostic::error(codes::CALL_ARITY, Some(span), message) - .with_hint("check the builtin parameter list"), + if let Expr::Ident(Ident { + name: alias, + span: alias_span, + }) = base.as_ref() + { + if self.locals.contains(alias) { + self.bind(span, BoundTarget::DynamicValue); + return; + } + if self.conflicted.contains(alias) { + // The import conflict diagnostic already tells the user + // that `alias.…` has no single meaning; do not cascade. + return; + } + if let Some(target_module) = + self.program.resolve_import(&self.module.id, alias).cloned() + { + if let Some(function) = self.registry.function_id(&target_module, name) { + if let Some(target) = self.program.function(&function) { + let signature = marqdo_ir::callable::CallableSignature { + params: target.params.clone(), + kind: target.signature, + }; + self.validate_signature( + &signature, + args, + &format!("{alias}.{name}"), + span, + span, + ); + } + self.bind(span, BoundTarget::Marqdo { function }); + } else { + self.diagnostic( + codes::SYMBOL_UNKNOWN, + span, + format!("module `{alias}` has no function `{name}`"), + Some(format!( + "check `inspect --signature` on the imported module `{alias}`" + )), ); } return; } - if self.imports.contains(alias) { - if let Some(program) = self.program { - if let Some(target_module) = - program.resolve_import(&self.module_id, alias) - { - if let Some(target) = program.module(target_module) { - if let Some(function) = target.function(name) { - let signature = CallableSignature { - params: function.params.clone(), - kind: function.signature, - }; - self.validate_against( - &qualified, &signature, args, span, span, - ); - return; - } - self.diagnostics.push( - Diagnostic::error( - codes::SYMBOL_UNKNOWN, - Some(span), - format!("module `{alias}` has no function `{name}`"), - ) - .with_hint( - "check `inspect --signature` on the imported module", - ), - ); - return; + let qualified = format!("{alias}.{name}"); + if let Some(builtin) = builtins::find(&qualified) { + if let Err(message) = builtins::validate_args(builtin, args) { + self.diagnostic( + codes::CALL_ARITY, + span, + message, + Some("check the builtin parameter list".to_string()), + ); + } + self.bind( + span, + BoundTarget::Builtin { + canonical_name: builtin.name.to_string(), + }, + ); + return; + } + if let Some(descriptor) = self.registry.external(&qualified) { + self.validate_signature( + &descriptor.signature, + args, + &qualified, + span, + span, + ); + let param_names = descriptor + .signature + .params + .iter() + .map(|param| param.name.clone()) + .collect(); + let target = match &descriptor.origin { + marqdo_ir::callable::CallableOrigin::AbiPlugin { package, export } => { + BoundTarget::AbiPlugin { + package: package.clone(), + export: export.clone(), + digest: descriptor + .implementation_digest + .clone() + .unwrap_or_default(), + param_names, } } - } + _ => BoundTarget::DynamicValue, + }; + self.bind(span, target); + return; } - // A declared ABI export is a static descriptor: validate - // its signature here, without loading the shared library. - if let Some(program) = self.program { - if let Some(descriptor) = program.external_callable(&qualified) { - if let Err(message) = - descriptor.signature.validate_args(args, &qualified) - { - self.diagnostics.push( - Diagnostic::error(codes::CALL_ARITY, Some(span), message) - .with_hint("check the plugin manifest export signature"), - ); - } - return; - } + if self.standalone { + // Without a program graph an import or plugin export + // cannot be distinguished statically; defer instead of + // emitting a false error. + return; } + self.diagnostic( + codes::SYMBOL_UNKNOWN, + *alias_span, + format!("unknown function or plugin export `{qualified}`"), + Some( + "declare the export in a plugin manifest and pass `--plugin` for \ + `run`, or fix the module import alias" + .to_string(), + ), + ); + return; } + self.bind(span, BoundTarget::DynamicValue); self.expr(base); } - other => self.expr(other), - } - } - - fn resolve(&self, name: &str) -> Option<&'a CallableSignature> { - if let Some(namespace) = &self.current_namespace { - if let Some(found) = self.table.get(&format!("{namespace}.{name}")) { - return Some(found); + other => { + self.bind(span, BoundTarget::DynamicValue); + self.expr(other); } } - if let Some(found) = self.table.get(name) { - return Some(found); - } - // Unique local name across namespaces. - let mut matches = self - .table - .iter() - .filter(|(key, _)| key.ends_with(&format!(".{name}"))) - .map(|(_, signature)| signature); - let first = matches.next()?; - if matches.next().is_some() { - None - } else { - Some(first) - } } - fn validate_against( + fn validate_signature( &mut self, - name: &str, - signature: &CallableSignature, + signature: &marqdo_ir::callable::CallableSignature, args: &[Arg], + display: &str, call_span: Span, name_span: Span, ) { - if signature.kind == SignatureKind::Inferred { - if let Some(positional) = args.iter().find(|arg| arg.name.is_none()) { - self.diagnostics.push( - Diagnostic::error( - codes::CALL_ARITY, - Some(positional.span), - format!( - "`{name}` has an inferred signature and only accepts named arguments" - ), - ) - .with_hint("pass arguments as `name = value`"), - ); + if let Err(message) = signature.validate_args(args, display) { + // Positional overflow and missing arguments point at the call or + // the function name respectively; keep messages deterministic. + let span = if message.contains("missing required") { + name_span + } else { + call_span + }; + self.diagnostic( + codes::CALL_ARITY, + span, + message, + Some("check `inspect --signature` or the plugin manifest signature".to_string()), + ); + } + } + + fn resolve_module_function(&self, name: &str) -> Resolution { + if let Some(namespace) = &self.namespace { + let canonical = format!("{namespace}.{name}"); + if let Some(function) = self + .module + .functions + .iter() + .find(|function| function.id.name == canonical) + { + return Resolution::One(function.id.clone()); } } - let mut supplied: BTreeSet<&str> = BTreeSet::new(); - let mut positional = 0usize; - for arg in args { - match &arg.name { - Some(argument_name) => { - if !signature - .params - .iter() - .any(|param| param.name == *argument_name) - { - self.diagnostics.push( - Diagnostic::error( - codes::CALL_ARITY, - Some(arg.span), - format!("`{name}` has no parameter named `{argument_name}`"), - ) - .with_hint("check `inspect --signature`"), - ); - } - supplied.insert(argument_name.as_str()); - } - None => { - if let Some(param) = signature.params.get(positional) { - supplied.insert(param.name.as_str()); - positional += 1; - } else { - self.diagnostics.push(Diagnostic::error( - codes::CALL_ARITY, - Some(call_span), - format!( - "`{name}` accepts at most {} positional argument(s)", - signature.params.len() - ), - )); - } - } - } + if let Some(function) = self + .module + .functions + .iter() + .find(|function| function.id.name == name) + { + return Resolution::One(function.id.clone()); } - for param in &signature.params { - if param.default.is_none() && !supplied.contains(param.name.as_str()) { - self.diagnostics.push( - Diagnostic::error( - codes::CALL_ARITY, - Some(name_span), - format!("`{name}` is missing required argument `{}`", param.name), - ) - .with_hint("pass it by name, e.g. `x = value`"), - ); - } + let mut matches = self + .module + .functions + .iter() + .filter(|function| function.local_name == name); + match (matches.next(), matches.next()) { + (Some(function), None) => Resolution::One(function.id.clone()), + (None, _) => Resolution::None, + (Some(_), Some(_)) => Resolution::Ambiguous, } } } +enum Resolution { + One(FunctionId), + Ambiguous, + None, +} + +/// Whether a descriptor is a statically declared ABI plugin export. +pub fn is_abi_export(descriptor: &CallableDescriptor) -> bool { + matches!( + descriptor.origin, + marqdo_ir::callable::CallableOrigin::AbiPlugin { .. } + ) +} + +/// Whether a function signature requires explicit named/positional checking. +pub fn _signature_kind(signature: &marqdo_ir::callable::CallableSignature) -> SignatureKind { + signature.kind +} + +/// Helper used by tests and by `inspect` to render a function body trace. +#[allow(dead_code)] +fn _unused(_: &Function, _: &Program) {} + #[cfg(test)] mod tests { use super::*; @@ -494,11 +692,36 @@ mod tests { package: "demo".into(), export: "solver.add".into(), }, - implementation_digest: None, + implementation_digest: Some("a".repeat(64)), source_span: None, }); - let diagnostics = validate_program(&program); - assert!(diagnostics.iter().any(|(_, d)| d.code == codes::CALL_ARITY)); + let registry = CallableRegistry::try_from_program(&program).unwrap(); + let bound = bind_program_calls(&program, ®istry); + assert!(bound + .diagnostics + .iter() + .any(|(_, d)| d.code == codes::CALL_ARITY)); + // A valid positional/named mix binds to the exact export identity. + let module = make_module(vec![function( + "main", + &[], + SignatureKind::Explicit, + vec![return_call("solver.add(1, b=2)")], + )]); + program.modules.insert(module.id.clone(), module); + let registry = CallableRegistry::try_from_program(&program).unwrap(); + let bound = bind_program_calls(&program, ®istry); + assert!(bound.diagnostics.is_empty(), "{:?}", bound.diagnostics); + let target = bound.bindings.iter().next().unwrap().1; + match target { + BoundTarget::AbiPlugin { + package, export, .. + } => { + assert_eq!(package, "demo"); + assert_eq!(export, "solver.add"); + } + other => panic!("unexpected target {other:?}"), + } } #[test] @@ -556,4 +779,31 @@ mod tests { .iter() .any(|d| d.code == codes::CALL_ARITY)); } + + #[test] + fn import_alias_conflicting_with_builtin_namespace_is_rejected() { + let module = Module { + id: ModuleId::from("index.mq.md"), + units: vec![], + functions: vec![function( + "main", + &[], + SignatureKind::Explicit, + vec![return_call("math.eval(x=1)")], + )], + imports: vec![marqdo_ir::ImportDecl { + alias: "math".into(), + target: "./math.mq.md".into(), + span: Span::new(0, 1), + }], + resources: vec![], + anchors: std::collections::BTreeMap::new(), + }; + let diagnostics = validate_module(&module); + assert!(diagnostics + .iter() + .any(|d| d.code == codes::EXT_IMPORT_NAMESPACE_CONFLICT)); + // Standalone mode defers unresolved imports, but the composition root + // run is covered by the CLI black-box CALL-01 test. + } } diff --git a/crates/marqdo-frontend/src/lib.rs b/crates/marqdo-frontend/src/lib.rs index beeeef8..2e5abb9 100644 --- a/crates/marqdo-frontend/src/lib.rs +++ b/crates/marqdo-frontend/src/lib.rs @@ -14,5 +14,5 @@ pub mod resources; pub mod signature; pub mod units; -pub use analyze::{analyze_source, AnalysisResult}; +pub use analyze::{analyze_source, analyze_source_for_program, AnalysisResult}; pub use units::{build_units, github_slug, FunctionScope, HeadingInfo, UnitIndex}; diff --git a/crates/marqdo-host/src/capability.rs b/crates/marqdo-host/src/capability.rs index d9fb976..1079dbb 100644 --- a/crates/marqdo-host/src/capability.rs +++ b/crates/marqdo-host/src/capability.rs @@ -1,12 +1,19 @@ -//! Capability model. +//! Capability model and the trusted capability-handler registry. //! //! A project manifest can *request* capabilities, but requests are not grants. //! Effective permissions are the host default for the sandbox mode intersected //! with what the operator/CLI explicitly granted. Every host operation checks //! against this set at the point of execution. +//! +//! Executable vendor/core capabilities additionally need an explicitly +//! registered trusted [`CapabilityHandler`] (ADR-0057, RFC-EXT-0001 §7.2). +//! Being authorized is never the same as being executable: a request without a +//! handler fails with `capability.unavailable` instead of returning an empty +//! successful reply. -use std::collections::BTreeSet; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::{Arc, Mutex}; +use std::time::Instant; use glob::Pattern; use marqdo_plugin_abi::abi::MAX_CAPABILITY_REPLY_BYTES; @@ -14,8 +21,33 @@ use marqdo_plugin_abi::PluginError; use marqdo_span::Span; use marqdo_value::{ErrorCategory, RuntimeError}; +use crate::budget::Budget; use crate::fs::ProjectFs; +/// Core capability names understood by [`CapabilityBroker::decide`]. +/// +/// This is the closed set of host primitives (ADR-0011). A trusted embedder +/// may install a [`CapabilityHandler`] for one of these names; until it does, +/// the capability is authorized by policy but *unavailable* at execution. +pub const CORE_CAPABILITIES: [&str; 10] = [ + "console", + "math.compute", + "fs.read", + "fs.write", + "net.request", + "process", + "clock", + "random", + "log", + "artifact.record", +]; + +/// Capability namespace prefixes reserved for the host itself. +/// +/// Handler registration rejects these prefixes so a vendor package can never +/// shadow a host-internal operation. +pub const RESERVED_HANDLER_PREFIXES: [&str; 2] = ["marqdo.", "host."]; + /// One audited capability request from a plugin. #[derive(Debug, Clone, PartialEq, Eq)] pub struct CapabilityAudit { @@ -27,19 +59,143 @@ pub struct CapabilityAudit { pub decision: &'static str, /// Human-readable reason. pub reason: String, + /// Wall-clock duration of the decision/execution in microseconds. + pub duration_micros: u64, } -/// Broker for plugin host-capability requests (ADR-0011). +/// One capability request handed to an installed [`CapabilityHandler`]. +#[derive(Debug, Clone, Copy)] +pub struct CapabilityRequest<'a> { + /// Trusted package identity supplied by the host, never by plugin wire data. + pub package_instance: &'a str, + /// Capability being requested. + pub capability: &'a str, + /// Raw request payload (JSON for the predefined capabilities). + pub payload: &'a [u8], + /// Run/session trace id used to correlate audit entries with a run. + pub trace_id: &'a str, +} + +/// Trusted, explicitly registered executor for one capability. /// -/// Core capabilities are checked against the effective grant set. Vendor -/// capabilities are accepted only when a trusted host policy registered the -/// exact name; a declaration in a manifest is never enough. -#[derive(Debug, Clone)] +/// RFC-EXT-0001 §7.2 names the policy parameter `EffectiveGrant` and the budget +/// `ResourceBudget`; this build deliberately reuses the existing +/// [`Capabilities`] grant set and [`Budget`] instead of introducing a second +/// permission system (the RFC explicitly allows that substitution). +/// +/// A handler is only reachable after [`CapabilityBroker::decide`] authorizes +/// the request, but it must still verify its own scope (project-relative paths, +/// globs, per-file limits) and must charge work to `budget` where applicable. +/// Registration is host-only: there is no wire path from a plugin to this +/// registry. +pub trait CapabilityHandler: Send { + /// Stable capability name this handler serves. + fn name(&self) -> &str; + /// Execute one already-authorized request. + fn execute( + &mut self, + request: &CapabilityRequest<'_>, + capabilities: &Capabilities, + budget: &mut Budget, + ) -> Result, PluginError>; +} + +/// Built-in handler for granted `fs.read` requests. +/// +/// Reads go through [`ProjectFs`] (root confinement, traversal rejection, +/// regular-file/UTF-8/8 MiB limits) and the grant is checked **again** against +/// the normalized project-relative path at the execution point, so a raw +/// payload cannot win the glob check with a path the filesystem later +/// normalizes differently. +pub struct FsReadHandler { + fs: ProjectFs, +} + +impl FsReadHandler { + /// Install the built-in handler over one project filesystem view. + pub fn new(fs: ProjectFs) -> Self { + Self { fs } + } +} + +impl CapabilityHandler for FsReadHandler { + fn name(&self) -> &str { + "fs.read" + } + + fn execute( + &mut self, + request: &CapabilityRequest<'_>, + capabilities: &Capabilities, + _budget: &mut Budget, + ) -> Result, PluginError> { + // Reply-size limits are enforced by the broker, so this handler keeps + // the historical per-read behavior and only needs the grant re-check. + let raw_path = request_path(request.payload)?; + let (_, relative) = self.fs.resolve(&raw_path, None).map_err(map_runtime)?; + capabilities + .check_read(&relative, None) + .map_err(map_runtime)?; + let contents = self.fs.read(&raw_path, None).map_err(map_runtime)?; + Ok(contents.into_bytes()) + } +} + +struct HandlerEntry { + handler: Box, + /// Exact trusted package binding. `None` means the handler is authorized + /// by the capability grant alone (for example project-scoped `fs.read`). + package: Option, +} + +/// Broker for plugin host-capability requests (ADR-0011, ADR-0057). +/// +/// The broker keeps two independent sets: trusted vendor *policy* entries and +/// installed *handlers*. Execution is +/// +/// ```text +/// decide (grant + policy + payload shape) +/// -> handler installed? (else capability.unavailable) +/// -> package-bound handler owns the claimed package? (else permission.denied) +/// -> handler executes under the broker budget and reply limit +/// -> audit { package, capability, granted|denied|unavailable, reason, duration } +/// ``` +/// +/// A capability that is merely authorized never executes: there is no +/// empty-success path. pub struct CapabilityBroker { capabilities: Capabilities, vendor_policy: BTreeSet, + handlers: BTreeMap, + reply_limit: usize, + budget: Budget, + trace_id: String, audit: Arc>>, - fs: Option, +} + +impl std::fmt::Debug for CapabilityBroker { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let bound: Vec<(&str, &str)> = self + .handlers + .iter() + .filter_map(|(name, entry)| { + entry + .package + .as_deref() + .map(|package| (name.as_str(), package)) + }) + .collect(); + formatter + .debug_struct("CapabilityBroker") + .field("capabilities", &self.capabilities) + .field("vendor_policy", &self.vendor_policy) + .field("handlers", &self.handlers.keys().collect::>()) + .field("package_bound_handlers", &bound) + .field("reply_limit", &self.reply_limit) + .field("budget", &self.budget) + .field("trace_id", &self.trace_id) + .finish() + } } impl CapabilityBroker { @@ -53,19 +209,116 @@ impl CapabilityBroker { Self { capabilities, vendor_policy: BTreeSet::new(), + handlers: BTreeMap::new(), + reply_limit: MAX_CAPABILITY_REPLY_BYTES, + budget: Budget::default(), + trace_id: String::new(), audit, - fs: None, } } - /// Attach the project filesystem used to execute granted `fs.read` - /// requests. Without it, `fs.read` is authorized but unavailable. + /// Attach the project filesystem and install the trusted built-in + /// `fs.read` handler. + /// + /// Without an attached filesystem, granted `fs.read` requests stay + /// authorized but have no handler, so execution returns + /// `capability.unavailable` (never an empty success). + /// + /// # Panics + /// + /// Panics if a handler for `fs.read` was already registered on this broker, + /// which would make the built-in handler ambiguous. pub fn with_fs(mut self, fs: ProjectFs) -> Self { - self.fs = Some(fs); + self.register_handler(FsReadHandler::new(fs)) + .expect("a fresh broker has no fs.read handler to conflict with"); self } + /// Override the maximum reply size a handler may return. + pub fn with_reply_limit(mut self, bytes: usize) -> Self { + self.reply_limit = bytes; + self + } + + /// Attach the per-run budget that handlers charge work to. + pub fn with_budget(mut self, budget: Budget) -> Self { + self.budget = budget; + self + } + + /// Set the run/session trace id surfaced to handlers. + pub fn with_trace_id(mut self, trace_id: impl Into) -> Self { + self.trace_id = trace_id.into(); + self + } + + /// Remaining broker budget (diagnostics and tests). + pub fn budget(&self) -> &Budget { + &self.budget + } + + /// Install a trusted handler reachable by any package that holds a grant. + /// + /// Registration is a host-only operation; a plugin manifest cannot reach + /// this API. Duplicate, reserved and empty/invalid names are rejected. + pub fn register_handler(&mut self, handler: H) -> Result<(), PluginError> + where + H: CapabilityHandler + 'static, + { + self.install_handler(handler, None) + } + + /// Install a trusted handler bound to one exact trusted package identity. + /// + /// A request that claims any other package is `permission.denied`, even + /// when the capability grant and vendor policy match. + pub fn register_package_handler( + &mut self, + package: impl Into, + handler: H, + ) -> Result<(), PluginError> + where + H: CapabilityHandler + 'static, + { + self.install_handler(handler, Some(package.into())) + } + + fn install_handler(&mut self, handler: H, package: Option) -> Result<(), PluginError> + where + H: CapabilityHandler + 'static, + { + let name = handler.name().to_string(); + validate_handler_name(&name)?; + if self.handlers.contains_key(&name) { + return Err(PluginError::capability( + "capability.handler_conflict", + format!("a capability handler is already registered for `{name}`"), + )); + } + self.handlers.insert( + name, + HandlerEntry { + handler: Box::new(handler), + package, + }, + ); + Ok(()) + } + + /// Names with an installed trusted handler, in deterministic order. + pub fn handler_names(&self) -> Vec<&str> { + self.handlers.keys().map(String::as_str).collect() + } + + /// Whether a trusted executing handler is installed for `capability`. + pub fn has_handler(&self, capability: &str) -> bool { + self.handlers.contains_key(capability) + } + /// Register a trusted vendor capability policy entry. + /// + /// This is policy only: execution additionally requires an installed + /// handler. Registering a vendor name does not install anything. pub fn allow_vendor(&mut self, capability: impl Into) { self.vendor_policy.insert(capability.into()); } @@ -76,103 +329,132 @@ impl CapabilityBroker { } /// Authorize one request, returning reply bytes on success. + /// + /// This is the grant decision only; it does not prove that an executor is + /// installed. Call [`Self::execute`] to actually run a capability. pub fn authorize( &mut self, package: &str, capability: &str, request: &[u8], ) -> Result, PluginError> { + let started = Instant::now(); let result = self.decide(package, capability, request); let (decision, reason) = match &result { Ok(()) => ("granted", String::new()), Err(error) => ("denied", format!("{}: {}", error.code, error.message)), }; - if let Ok(mut audit) = self.audit.lock() { - audit.push(CapabilityAudit { - package: package.to_string(), - capability: capability.to_string(), - decision, - reason, - }); - } + self.record(package, capability, decision, &reason, started); result.map(|()| Vec::new()) } - /// Authorize and execute one capability request. + /// Authorize and execute one capability request through its trusted + /// handler. /// - /// Only `fs.read` has a real implementation in this build. Other - /// capabilities return `capability.unavailable` instead of pretending an - /// empty reply means success (ADR-0011, spec §12.2). + /// The chain is fixed: `decide` → handler installed → package binding → + /// handler execution under the broker budget and reply limit → audit. + /// Anything without an installed handler is `capability.unavailable`; it + /// is never reported as an empty success (ADR-0011, ADR-0057). pub fn execute( &mut self, package: &str, capability: &str, request: &[u8], ) -> Result, PluginError> { + let started = Instant::now(); + + // 1. Authorization: effective grants, vendor policy, payload shape. if let Err(error) = self.decide(package, capability, request) { let reason = format!("{}: {}", error.code, error.message); - self.record(package, capability, "denied", &reason); + self.record(package, capability, "denied", &reason, started); return Err(error); } - let result = match capability { - "fs.read" => { - let path = request_path(request)?; - let Some(fs) = &self.fs else { - return self.unavailable( - package, - capability, - "no project filesystem is attached", - ); - }; - match fs.read(&path, None) { - Ok(contents) => { - if contents.len() > MAX_CAPABILITY_REPLY_BYTES { - return self.unavailable( - package, - capability, - "the file exceeds the capability reply limit", - ); - } - Ok(contents.into_bytes()) - } - Err(error) => { - let error = map_runtime(error); - let reason = format!("{}: {}", error.code, error.message); - self.record(package, capability, "denied", &reason); - Err(error) - } - } + + // 2. A trusted executor must exist; a package-bound executor must own + // the trusted package identity the host is claiming. + let bound_package = match self.handlers.get(capability) { + Some(entry) => entry.package.clone(), + None => { + let message = "this capability has no executing handler in this build"; + self.record(package, capability, "unavailable", message, started); + return Err(PluginError::capability( + "capability.unavailable", + message.to_string(), + )); } - _ => self.unavailable( - package, - capability, - "this capability has no executing handler in this build", - ), }; - if result.is_ok() { - self.record(package, capability, "granted", ""); + if let Some(bound) = bound_package { + if bound != package { + let error = plugin_denied(&format!( + "capability `{capability}` is bound to another trusted package" + )); + let reason = format!("{}: {}", error.code, error.message); + self.record(package, capability, "denied", &reason, started); + return Err(error); + } + } + + // 3. Execute under the broker budget and reply limit. + let broker_request = CapabilityRequest { + package_instance: package, + capability, + payload: request, + trace_id: &self.trace_id, + }; + let reply_limit = self.reply_limit; + let outcome = self + .handlers + .get_mut(capability) + .expect("handler presence was checked above") + .handler + .execute(&broker_request, &self.capabilities, &mut self.budget); + match outcome { + Ok(contents) => { + if contents.len() > reply_limit { + let error = PluginError::capability( + "resource.exhausted", + format!( + "capability reply is {} bytes; the limit is {reply_limit} bytes", + contents.len() + ), + ); + let reason = format!("{}: {}", error.code, error.message); + self.record(package, capability, "denied", &reason, started); + return Err(error); + } + self.record(package, capability, "granted", "", started); + Ok(contents) + } + Err(error) => { + let reason = format!("{}: {}", error.code, error.message); + let decision = if error.code == "capability.unavailable" { + "unavailable" + } else { + "denied" + }; + self.record(package, capability, decision, &reason, started); + Err(error) + } } - result } - fn unavailable( + fn record( &mut self, package: &str, capability: &str, - message: &str, - ) -> Result { - let error = PluginError::capability("capability.unavailable", message.to_string()); - self.record(package, capability, "unavailable", message); - Err(error) - } - - fn record(&mut self, package: &str, capability: &str, decision: &'static str, reason: &str) { + decision: &'static str, + reason: &str, + started: Instant, + ) { + let elapsed = started.elapsed(); + let duration_micros = u64::try_from(elapsed.as_micros()).unwrap_or(u64::MAX); if let Ok(mut audit) = self.audit.lock() { audit.push(CapabilityAudit { package: package.to_string(), capability: capability.to_string(), decision, reason: reason.to_string(), + duration_micros, }); } } @@ -214,17 +496,14 @@ impl CapabilityBroker { .then_some(()) .ok_or_else(|| plugin_denied("process capability was not granted")), other => { - if is_vendor_capability(other) - && (self.vendor_policy.contains(other) || self.vendor_policy.contains("*")) - { - Ok(()) - } else if is_vendor_capability(other) { - Err(PluginError::capability( - "capability.unavailable", - format!( - "vendor capability `{other}` has no trusted handler for package `{package}`" - ), - )) + if is_vendor_capability(other) { + if self.vendor_policy.contains(other) || self.vendor_policy.contains("*") { + Ok(()) + } else { + Err(plugin_denied(&format!( + "vendor capability `{other}` has no trusted policy entry for package `{package}`" + ))) + } } else { Err(PluginError::capability( "capability.unknown", @@ -304,6 +583,38 @@ pub fn is_vendor_capability(name: &str) -> bool { }) } +/// Validate a trusted handler name before registration. +/// +/// Registration uses [`CapabilityHandler::name`] as the registry key, so there +/// is no second name argument that could disagree with the handler's declared +/// capability (the RFC's "declared protocol mismatch" rejection). +fn validate_handler_name(name: &str) -> Result<(), PluginError> { + if name.is_empty() || name.trim() != name { + return Err(PluginError::capability( + "capability.handler_invalid", + "capability handler names must be non-empty and must not contain surrounding whitespace", + )); + } + if RESERVED_HANDLER_PREFIXES + .iter() + .any(|prefix| name.starts_with(prefix)) + { + return Err(PluginError::capability( + "capability.handler_reserved", + format!("capability handler name `{name}` uses a reserved host namespace"), + )); + } + if !CORE_CAPABILITIES.contains(&name) && !is_vendor_capability(name) { + return Err(PluginError::capability( + "capability.handler_invalid", + format!( + "`{name}` is not a registrable capability name: expected a core capability or a vendor-namespaced name" + ), + )); + } + Ok(()) +} + /// Effective capability grants for one run. #[derive(Debug, Clone, Default)] pub struct Capabilities { @@ -594,20 +905,56 @@ mod tests { .unwrap_err(); assert_eq!(denied.code, "permission.denied"); assert!(broker.authorize("demo.solver", "fs.read", b"{}").is_err()); - let unavailable = broker + // Declaring a vendor capability is not a grant: without a trusted + // policy entry authorization is denied (RFC-EXT-0001 CAP-01). + let vendor_denied = broker .authorize("demo.solver", "vendor.device.read", b"{}") .unwrap_err(); - assert_eq!(unavailable.code, "capability.unavailable"); + assert_eq!(vendor_denied.code, "permission.denied"); broker.allow_vendor("vendor.device.read"); assert!(broker .authorize("demo.solver", "vendor.device.read", b"{}") .is_ok()); + // Policy is not a handler: authorize() succeeds but execution still + // needs an executor and reports `capability.unavailable`. + assert!(!broker.has_handler("vendor.device.read")); + assert_eq!( + broker + .execute("demo.solver", "vendor.device.read", b"{}") + .unwrap_err() + .code, + "capability.unavailable" + ); assert!(!is_vendor_capability("vendor")); assert!(!is_vendor_capability("fs.read.extra")); let audit = broker.audit(); let audit = audit.lock().unwrap(); - assert_eq!(audit.len(), 5); + assert_eq!(audit.len(), 6); assert!(audit.iter().any(|entry| entry.decision == "denied")); + assert!(audit.iter().any(|entry| entry.decision == "unavailable")); + assert!(audit.iter().all(|entry| entry.duration_micros <= 5_000_000)); + } + + #[test] + fn every_core_capability_is_understood_by_decide() { + for name in CORE_CAPABILITIES { + let broker = CapabilityBroker::new(Capabilities::project()); + let payload: &[u8] = match name { + "fs.read" => br#"{"path":"data/a.csv"}"#, + "fs.write" => br#"{"path":"data/a.csv","contents":"x"}"#, + "net.request" => br#"{"host":"127.0.0.1"}"#, + _ => b"{}", + }; + let error = broker + .decide("demo.solver", name, payload) + .err() + .map(|error| error.code); + assert_ne!( + error.as_deref(), + Some("capability.unknown"), + "core capability `{name}` must be handled by decide" + ); + } } #[test] @@ -638,6 +985,9 @@ mod execute_tests { .allow_read("data/**") .allow_write("data/**"); let mut broker = CapabilityBroker::new(capabilities).with_fs(fs); + assert!(broker.has_handler("fs.read")); + assert_eq!(broker.handler_names(), vec!["fs.read"]); + assert_eq!(broker.reply_limit, MAX_CAPABILITY_REPLY_BYTES); let contents = broker .execute("demo.solver", "fs.read", br#"{"path":"data/allowed.txt"}"#) diff --git a/crates/marqdo-host/src/lib.rs b/crates/marqdo-host/src/lib.rs index 27ad071..915c84b 100644 --- a/crates/marqdo-host/src/lib.rs +++ b/crates/marqdo-host/src/lib.rs @@ -19,7 +19,10 @@ pub mod resource; pub mod trace; pub use budget::{Budget, DEFAULT_FUEL, DEFAULT_OUTPUT_BYTES, DEFAULT_STACK_LIMIT}; -pub use capability::Capabilities; +pub use capability::{ + Capabilities, CapabilityAudit, CapabilityBroker, CapabilityHandler, CapabilityRequest, + FsReadHandler, CORE_CAPABILITIES, RESERVED_HANDLER_PREFIXES, +}; pub use context::ExecutionContext; pub use fs::ProjectFs; pub use handles::{contains_plugin_handle, PluginHandleTable}; diff --git a/crates/marqdo-host/tests/capability_handlers.rs b/crates/marqdo-host/tests/capability_handlers.rs new file mode 100644 index 0000000..7af8aef --- /dev/null +++ b/crates/marqdo-host/tests/capability_handlers.rs @@ -0,0 +1,581 @@ +//! Trusted capability-handler registry (RFC-EXT-0001 §7, ADR-0057). +//! +//! These are end-to-end tests over the public `marqdo-host` API. They cover the +//! CAP-01..CAP-12 checks listed in RFC-EXT-0001 §7.5: policy vs handler, +//! package binding, registration validation, real `fs.read` scope enforcement, +//! reply budgets, forged/replayed handles, audit hygiene, resource-stream +//! regression, project-vs-strict labeling and the "no handler is never a +//! silent OK" rule. + +use std::sync::atomic::{AtomicU32, Ordering}; +use std::time::{Duration, Instant}; + +use marqdo_host::{ + contains_plugin_handle, Budget, Capabilities, CapabilityBroker, CapabilityHandler, + CapabilityRequest, PluginHandleTable, ProjectFs, ResourceTable, SandboxMode, StdHost, + CORE_CAPABILITIES, +}; +use marqdo_plugin_abi::PluginError; +use marqdo_value::{RuntimeError, Value}; + +static COUNTER: AtomicU32 = AtomicU32::new(0); + +struct TempDir(std::path::PathBuf); + +impl TempDir { + fn new() -> Self { + let id = COUNTER.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "marqdo-capability-handlers-{}-{id}", + std::process::id() + )); + std::fs::create_dir_all(&path).unwrap(); + Self(path) + } + + fn path(&self) -> &std::path::Path { + &self.0 + } +} + +impl Drop for TempDir { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +/// A trusted test handler that returns canned bytes. +struct FixedHandler { + name: &'static str, + reply: Vec, +} + +impl CapabilityHandler for FixedHandler { + fn name(&self) -> &str { + self.name + } + + fn execute( + &mut self, + _request: &CapabilityRequest<'_>, + _capabilities: &Capabilities, + _budget: &mut Budget, + ) -> Result, PluginError> { + Ok(self.reply.clone()) + } +} + +/// A trusted test handler that proves the request identity and budget reach it. +struct EchoHandler { + name: &'static str, + fuel: u64, +} + +impl CapabilityHandler for EchoHandler { + fn name(&self) -> &str { + self.name + } + + fn execute( + &mut self, + request: &CapabilityRequest<'_>, + _capabilities: &Capabilities, + budget: &mut Budget, + ) -> Result, PluginError> { + budget + .consume_fuel(self.fuel, None) + .map_err(map_runtime_error)?; + Ok(format!( + "{}|{}|{}", + request.package_instance, request.capability, request.trace_id + ) + .into_bytes()) + } +} + +fn map_runtime_error(error: RuntimeError) -> PluginError { + PluginError::capability(error.code(), error.message) +} + +fn audit_entries(broker: &CapabilityBroker) -> Vec { + let audit = broker.audit(); + let guard = audit.lock().expect("audit lock"); + guard.clone() +} + +/// CAP-01: a declared vendor capability without a trusted policy entry is +/// denied, even when a handler happens to be installed. +#[test] +fn cap_01_vendor_capability_without_policy_is_denied() { + let mut broker = CapabilityBroker::new(Capabilities::project()); + broker + .register_handler(FixedHandler { + name: "vendor.device.read", + reply: b"device".to_vec(), + }) + .unwrap(); + + let error = broker + .execute("demo.solver", "vendor.device.read", b"{}") + .unwrap_err(); + assert_eq!(error.code, "permission.denied", "{error:?}"); + assert!(broker + .authorize("demo.solver", "vendor.device.read", b"{}") + .is_err()); + + let audit = audit_entries(&broker); + assert_eq!(audit.len(), 2); + assert!(audit.iter().all(|entry| entry.decision == "denied")); + assert!(audit + .iter() + .all(|entry| entry.capability == "vendor.device.read")); +} + +/// CAP-02: authorized but no handler installed is `capability.unavailable`; +/// `authorize` alone does not execute and cannot fake an empty success. +#[test] +fn cap_02_authorized_capability_without_handler_is_unavailable() { + let mut broker = CapabilityBroker::new(Capabilities::project().allow_write("out/**")); + broker.allow_vendor("vendor.device.read"); + + let unavailable = broker + .execute("demo.solver", "vendor.device.read", b"{}") + .unwrap_err(); + assert_eq!( + unavailable.code, "capability.unavailable", + "{unavailable:?}" + ); + + // Policy is a grant, not an executor: authorize() succeeds, execute() still + // refuses because no trusted handler was registered. + assert!(broker + .authorize( + "demo.solver", + "fs.write", + br#"{"path":"out/result.txt","contents":"x"}"#, + ) + .is_ok()); + let write = broker + .execute( + "demo.solver", + "fs.write", + br#"{"path":"out/result.txt","contents":"x"}"#, + ) + .unwrap_err(); + assert_eq!(write.code, "capability.unavailable", "{write:?}"); + + let audit = audit_entries(&broker); + assert!(audit + .iter() + .any(|entry| entry.decision == "unavailable" && entry.capability == "fs.write")); + assert!(audit + .iter() + .any(|entry| entry.decision == "granted" && entry.capability == "fs.write")); +} + +/// CAP-03: a handler bound to one trusted package rejects cross-package +/// impersonation, while the bound package still executes and the request +/// identity/trace id reach the handler. +#[test] +fn cap_03_package_bound_handler_rejects_impersonation() { + let mut broker = CapabilityBroker::new(Capabilities::project()).with_trace_id("trace-9"); + broker.allow_vendor("vendor.device.read"); + broker + .register_package_handler( + "demo.solver", + EchoHandler { + name: "vendor.device.read", + fuel: 0, + }, + ) + .unwrap(); + + let spoofed = broker + .execute("attacker.solver", "vendor.device.read", b"{}") + .unwrap_err(); + assert_eq!(spoofed.code, "permission.denied", "{spoofed:?}"); + + let reply = broker + .execute("demo.solver", "vendor.device.read", b"{}") + .unwrap(); + assert_eq!( + String::from_utf8(reply).unwrap(), + "demo.solver|vendor.device.read|trace-9" + ); + + let audit = audit_entries(&broker); + assert!(audit + .iter() + .any(|entry| entry.decision == "denied" && entry.package == "attacker.solver")); + assert!(audit + .iter() + .any(|entry| entry.decision == "granted" && entry.package == "demo.solver")); +} + +/// CAP-04: duplicate, reserved, empty and malformed handler names are rejected +/// and never installed. +#[test] +fn cap_04_registration_rejects_duplicate_reserved_and_empty_names() { + let mut broker = CapabilityBroker::new(Capabilities::project()); + let rejected = |broker: &mut CapabilityBroker, name: &'static str, expected: &str| { + let error = broker + .register_handler(FixedHandler { + name, + reply: Vec::new(), + }) + .unwrap_err(); + assert_eq!(error.code, expected, "name `{name}`: {error:?}"); + assert!(!broker.has_handler(name)); + }; + + rejected(&mut broker, "", "capability.handler_invalid"); + rejected(&mut broker, "host.terminal", "capability.handler_reserved"); + rejected( + &mut broker, + "marqdo.internal.read", + "capability.handler_reserved", + ); + rejected( + &mut broker, + "not a capability", + "capability.handler_invalid", + ); + rejected(&mut broker, "vendor..read", "capability.handler_invalid"); + + broker + .register_handler(FixedHandler { + name: "vendor.device.read", + reply: b"device".to_vec(), + }) + .unwrap(); + let duplicate = broker + .register_handler(FixedHandler { + name: "vendor.device.read", + reply: b"other".to_vec(), + }) + .unwrap_err(); + assert_eq!( + duplicate.code, "capability.handler_conflict", + "{duplicate:?}" + ); + + // The trusted built-in fs.read handler cannot be replaced either. + let dir = TempDir::new(); + let mut with_fs = + CapabilityBroker::new(Capabilities::project()).with_fs(ProjectFs::new(dir.path())); + assert!(with_fs.has_handler("fs.read")); + let conflict = with_fs + .register_handler(FixedHandler { + name: "fs.read", + reply: Vec::new(), + }) + .unwrap_err(); + assert_eq!(conflict.code, "capability.handler_conflict", "{conflict:?}"); +} + +/// CAP-05: the real `fs.read` handler enforces ProjectFs root/scope limits; a +/// permissive glob grant cannot be used to read outside the project root or to +/// traverse with `..`. +#[test] +fn cap_05_fs_read_handler_enforces_scope() { + let dir = TempDir::new(); + std::fs::create_dir_all(dir.path().join("data")).unwrap(); + std::fs::write(dir.path().join("data/allowed.txt"), "allowed").unwrap(); + std::fs::create_dir_all(dir.path().join("private")).unwrap(); + std::fs::write(dir.path().join("private/secret.txt"), "secret").unwrap(); + + let mut broker = CapabilityBroker::new(Capabilities::project().allow_read("data/**")) + .with_fs(ProjectFs::new(dir.path())); + let allowed = broker + .execute("demo.solver", "fs.read", br#"{"path":"data/allowed.txt"}"#) + .unwrap(); + assert_eq!(String::from_utf8(allowed).unwrap(), "allowed"); + + for out_of_scope in [ + br#"{"path":"private/secret.txt"}"#.as_slice(), + br#"{"path":"../escape.txt"}"#.as_slice(), + br#"{"path":"/etc/passwd"}"#.as_slice(), + ] { + let error = broker + .execute("demo.solver", "fs.read", out_of_scope) + .unwrap_err(); + assert_eq!( + error.code, "permission.denied", + "{out_of_scope:?}: {error:?}" + ); + } + + // Even a `**` grant only authorizes project-relative paths: traversal and + // absolute paths are rejected by ProjectFs::resolve at the handler. + let mut permissive = CapabilityBroker::new(Capabilities::project().allow_read("**")) + .with_fs(ProjectFs::new(dir.path())); + for rejected in [ + br#"{"path":"data/../private/secret.txt"}"#.as_slice(), + br#"{"path":"/etc/passwd"}"#.as_slice(), + ] { + let error = permissive + .execute("demo.solver", "fs.read", rejected) + .unwrap_err(); + assert_eq!(error.code, "permission.denied", "{rejected:?}: {error:?}"); + } +} + +/// CAP-06: a handler reply over the broker budget is a resource limit, and the +/// handler receives (and charges) the broker budget. +#[test] +fn cap_06_handler_reply_over_budget_is_resource_limit() { + let mut broker = CapabilityBroker::new(Capabilities::project()) + .with_reply_limit(64) + .with_budget(Budget { + fuel: 10, + ..Budget::default() + }); + broker.allow_vendor("vendor.device.big"); + broker.allow_vendor("vendor.device.fuel"); + broker + .register_handler(FixedHandler { + name: "vendor.device.big", + reply: vec![b'x'; 65], + }) + .unwrap(); + broker + .register_handler(EchoHandler { + name: "vendor.device.fuel", + fuel: 4, + }) + .unwrap(); + + let oversized = broker + .execute("demo.solver", "vendor.device.big", b"{}") + .unwrap_err(); + assert_eq!(oversized.code, "resource.exhausted", "{oversized:?}"); + + let reply = broker + .execute("demo.solver", "vendor.device.fuel", b"{}") + .unwrap(); + assert_eq!( + String::from_utf8(reply).unwrap(), + "demo.solver|vendor.device.fuel|" + ); + assert_eq!(broker.budget().remaining_fuel(), 6); + + let audit = audit_entries(&broker); + assert!(audit.iter().all(|entry| entry.duration_micros <= 5_000_000)); + assert!(audit + .iter() + .any(|entry| entry.capability == "vendor.device.big" && entry.decision == "denied")); +} + +/// CAP-07: forged plugin handles are rejected by the host table, and a handle +/// hidden inside an untrusted extension value is detected. +#[test] +fn cap_07_forged_handles_are_rejected() { + let mut table = PluginHandleTable::default(); + let handle = table.mint("demo.capability", "capability.read", "demo.capability", 1); + table + .validate(&handle, "demo.capability", 1) + .expect("host-minted handle is valid"); + + let mut forged = handle.clone(); + forged.id = handle.id.wrapping_add(1000); + let error = table + .validate(&forged, "demo.capability", 1) + .expect_err("forged id must be rejected"); + assert!( + error.message.contains("plugin_handle.untrusted"), + "{error:?}" + ); + + let wrong_owner = table + .validate(&handle, "other.package", 1) + .expect_err("cross-owner handle must be rejected"); + assert!( + wrong_owner.message.contains("plugin_handle.untrusted"), + "{wrong_owner:?}" + ); + + let untrusted_value = Value::Map( + [("handle".to_string(), Value::PluginHandle(handle.clone()))] + .into_iter() + .collect(), + ); + assert!(contains_plugin_handle(&untrusted_value)); +} + +/// CAP-08: a handle from an old run or a cleared (unloaded) run is rejected on +/// replay. +#[test] +fn cap_08_replayed_handle_after_run_end_is_rejected() { + let mut table = PluginHandleTable::default(); + let handle = table.mint("demo.capability", "capability.read", "demo.capability", 7); + table + .validate(&handle, "demo.capability", 7) + .expect("current run accepts its own handle"); + + let new_run = table + .validate(&handle, "demo.capability", 8) + .expect_err("a new run must not accept the old handle"); + assert!( + new_run.message.contains("plugin_handle.untrusted"), + "{new_run:?}" + ); + + table.clear_run(7); + assert!(table.is_empty()); + let replayed = table + .validate(&handle, "demo.capability", 7) + .expect_err("a cleared run must reject the replayed handle"); + assert!( + replayed.message.contains("plugin_handle.untrusted"), + "{replayed:?}" + ); +} + +/// CAP-09: the capability audit records decisions and durations but never file +/// contents or request secrets. +#[test] +fn cap_09_audit_contains_no_file_contents_or_secrets() { + const SECRET: &str = "TOP-SECRET-PAYLOAD"; + const API_KEY: &str = "sk-live-not-audited"; + let dir = TempDir::new(); + std::fs::create_dir_all(dir.path().join("data")).unwrap(); + std::fs::write(dir.path().join("data/secret.txt"), SECRET).unwrap(); + + let mut broker = CapabilityBroker::new(Capabilities::project().allow_read("data/**")) + .with_fs(ProjectFs::new(dir.path())); + let request = format!(r#"{{"path":"data/secret.txt","api_key":"{API_KEY}"}}"#); + let contents = broker + .execute("demo.solver", "fs.read", request.as_bytes()) + .unwrap(); + assert_eq!(String::from_utf8(contents).unwrap(), SECRET); + + let denied = broker + .execute("demo.solver", "fs.read", br#"{"path":"private/key.txt"}"#) + .unwrap_err(); + assert_eq!(denied.code, "permission.denied"); + + let audit_debug = format!("{:?}", audit_entries(&broker)); + assert!(!audit_debug.contains(SECRET), "{audit_debug}"); + assert!(!audit_debug.contains(API_KEY), "{audit_debug}"); + + let audit = audit_entries(&broker); + assert!(audit + .iter() + .any(|entry| entry.decision == "granted" && entry.capability == "fs.read")); + assert!(audit + .iter() + .any(|entry| entry.decision == "denied" && entry.capability == "fs.read")); + assert!(audit.iter().all(|entry| entry.duration_micros <= 5_000_000)); +} + +/// CAP-10: low-level resource open/read/write/close stays on a sane performance +/// path. This is a catastrophic-regression guard with a deliberately generous +/// bound, not a benchmark (CI timeouts remain authoritative). +#[test] +fn cap_10_resource_stream_operations_do_not_regress_catastrophically() { + const CHUNK: usize = 256 * 1024; + const ITERATIONS: usize = 32; + let payload = vec![b'x'; CHUNK]; + let mut table = ResourceTable::default(); + + let started = Instant::now(); + for _ in 0..ITERATIONS { + let reader = table.open_file("data/blob.bin", payload.clone()); + let read = table.read(reader, payload.len(), None).unwrap(); + assert_eq!(read.bytes.len(), CHUNK); + assert!(read.eof); + table.close(reader, None).unwrap(); + + let writer = table.open_file_writer("out/blob.bin"); + assert_eq!(table.write(writer, &payload, None).unwrap(), CHUNK); + table.close(writer, None).unwrap(); + } + let elapsed = started.elapsed(); + // 32 x (256 KiB read + 256 KiB write) is ~16 MiB of in-memory copying. + assert!( + elapsed < Duration::from_secs(5), + "resource open/read/write/close took {elapsed:?} for {ITERATIONS} iterations" + ); +} + +/// CAP-11: the default `project` mode is not an OS-strict mode and grants +/// nothing beyond `pure` by default. +#[test] +fn cap_11_default_project_mode_is_not_os_strict() { + let dir = TempDir::new(); + let host = StdHost::new(SandboxMode::Project, Capabilities::project(), dir.path()); + assert_eq!(host.mode, SandboxMode::Project); + assert_ne!(host.mode, SandboxMode::Strict); + assert_eq!(host.mode.as_str(), "project"); + + // Project defaults equal pure defaults: no implicit fs/process/plugin grant. + assert!(host.capabilities.fs_read.is_empty()); + assert!(host.capabilities.fs_write.is_empty()); + assert!(!host.capabilities.process); + assert!(!host.capabilities.foreign); + assert!(host.capabilities.plugins.is_empty()); + + // Strict isolation availability is a property of `strict`; project mode is + // cooperative in-process checks, whether or not the platform could isolate. + assert_eq!( + SandboxMode::parse(host.mode.as_str()), + Some(SandboxMode::Project) + ); +} + +/// CAP-12: core capabilities without an executing handler are always +/// `capability.unavailable`, never a silent empty success. +#[test] +fn cap_12_core_capabilities_without_handler_are_unavailable() { + let dir = TempDir::new(); + std::fs::create_dir_all(dir.path().join("data")).unwrap(); + std::fs::write(dir.path().join("data/ok.txt"), "ok").unwrap(); + + let mut capabilities = Capabilities::project() + .allow_read("data/**") + .allow_write("data/**") + .allow_net("127.0.0.1") + .allow_process("/bin/true"); + capabilities.process = true; + let mut broker = CapabilityBroker::new(capabilities).with_fs(ProjectFs::new(dir.path())); + + // The one migrated capability has a real handler. + let read = broker + .execute("demo.solver", "fs.read", br#"{"path":"data/ok.txt"}"#) + .unwrap(); + assert_eq!(String::from_utf8(read).unwrap(), "ok"); + + // Every other core capability is authorized by its grant but has no + // executor in this build. + for name in CORE_CAPABILITIES { + if name == "fs.read" { + continue; + } + let payload: &[u8] = match name { + "fs.write" => br#"{"path":"data/x","contents":"x"}"#, + "net.request" => br#"{"host":"127.0.0.1"}"#, + _ => b"{}", + }; + let error = broker.execute("demo.solver", name, payload).unwrap_err(); + assert_eq!( + error.code, "capability.unavailable", + "core capability `{name}` must not silently succeed: {error:?}" + ); + assert!(!broker.has_handler(name)); + } + + let unknown = broker + .execute("demo.solver", "teleport", b"{}") + .unwrap_err(); + assert_eq!(unknown.code, "capability.unknown", "{unknown:?}"); + + let audit = audit_entries(&broker); + assert!( + audit + .iter() + .filter(|entry| entry.decision == "unavailable") + .count() + >= CORE_CAPABILITIES.len() - 1, + "{audit:?}" + ); +} diff --git a/crates/marqdo-ir/src/call_binding.rs b/crates/marqdo-ir/src/call_binding.rs new file mode 100644 index 0000000..b55c0fd --- /dev/null +++ b/crates/marqdo-ir/src/call_binding.rs @@ -0,0 +1,201 @@ +//! Resolved static call bindings (RFC-EXT-0001 §4.2, ADR-0054). +//! +//! The frontend resolves every statically visible call site exactly once and +//! records the decision here. The runtime only dispatches by this table: it +//! never re-resolves a qualified name and never guesses a plugin from a string +//! prefix. The table is internal IR (not part of Analysis/Trace v2 wire +//! contracts) and is deliberately skipped when a [`crate::Program`] is +//! serialized. +//! +//! Identity rules: +//! +//! * a [`CallSiteKey`] is `(module, function, byte span)` and is therefore +//! independent of disk paths, registration order and hash-map iteration; +//! * a bound target names the *resolved implementation*, including the plugin +//! package, complete export name and verified artifact digest for ABI calls; +//! * [`BoundTarget::DynamicValue`] is the only non-static path and may only +//! call language values (for example a `FunctionRef`); it is never a back +//! door to an unknown ABI export. + +use std::collections::BTreeMap; +use std::fmt; + +use serde::{Deserialize, Serialize}; + +use crate::{FunctionId, ModuleId}; + +/// Stable identity of one static call site. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +pub struct CallSiteKey { + /// Module containing the call. + pub module: ModuleId, + /// Function containing the call. + pub function: FunctionId, + /// First byte offset of the call expression. + pub span_start: usize, + /// End byte offset (exclusive) of the call expression. + pub span_end: usize, +} + +impl CallSiteKey { + /// Construct a key from a function identity and an expression span. + pub fn new(function: FunctionId, span: marqdo_span::Span) -> Self { + Self { + module: function.module.clone(), + function, + span_start: span.start, + span_end: span.end, + } + } +} + +impl fmt::Display for CallSiteKey { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{}@{}..{}", + self.function, self.span_start, self.span_end + ) + } +} + +/// Resolved implementation target for a call site. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum BoundTarget { + /// A fixed language builtin, by canonical builtin name. + Builtin { + /// Canonical name (for example `math.eval`), not a user alias. + canonical_name: String, + }, + /// A function defined by a Markdown heading. + Marqdo { + /// Exact function identity. + function: FunctionId, + }, + /// A statically declared ABI plugin export. + AbiPlugin { + /// Plugin package that owns the export. + package: String, + /// Complete dotted export name. + export: String, + /// Verified artifact digest from the selected manifest artifact. + digest: String, + /// Declared parameter names in manifest order; used to normalise + /// positional arguments deterministically for the ABI v1 map wire. + param_names: Vec, + }, + /// The callee is a language value resolved at runtime (local variable or + /// an expression result). Only language callables such as `FunctionRef` + /// are valid; this path cannot invoke an ABI plugin. + DynamicValue, +} + +impl BoundTarget { + /// Whether this binding names a statically known implementation. + pub fn is_static(&self) -> bool { + !matches!(self, BoundTarget::DynamicValue) + } +} + +/// One deterministic binding table for a prepared program. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct CallBindingTable { + sites: BTreeMap, +} + +impl CallBindingTable { + /// Insert a binding. Returns `false` when the site was already bound to a + /// different target; re-binding the same target is idempotent. + pub fn insert(&mut self, key: CallSiteKey, target: BoundTarget) -> bool { + match self.sites.get(&key) { + Some(existing) => existing == &target, + None => { + self.sites.insert(key, target); + true + } + } + } + + /// Look up the bound target for a call site. + pub fn get(&self, key: &CallSiteKey) -> Option<&BoundTarget> { + self.sites.get(key) + } + + /// Number of bound call sites. + pub fn len(&self) -> usize { + self.sites.len() + } + + /// Whether no call site is bound. + pub fn is_empty(&self) -> bool { + self.sites.is_empty() + } + + /// Iterate bindings deterministically (call site order). + pub fn iter(&self) -> impl Iterator { + self.sites.iter() + } + + /// Number of statically bound (non-dynamic) call sites. + pub fn static_count(&self) -> usize { + self.sites + .values() + .filter(|target| target.is_static()) + .count() + } + + /// Number of dynamically resolved call sites. + pub fn dynamic_count(&self) -> usize { + self.sites.len() - self.static_count() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use marqdo_span::Span; + + fn function(name: &str) -> FunctionId { + FunctionId { + module: ModuleId::root_default("index.mq.md"), + name: name.to_string(), + } + } + + fn key(name: &str, span: Span) -> CallSiteKey { + CallSiteKey::new(function(name), span) + } + + #[test] + fn keys_are_deterministic_and_span_based() { + let left = key("f", Span::new(10, 20)); + let right = key("f", Span::new(10, 20)); + let other = key("f", Span::new(21, 30)); + assert_eq!(left, right); + assert_ne!(left, other); + assert_eq!(left.to_string(), "root:default::index.mq.md::f@10..20"); + } + + #[test] + fn insert_is_idempotent_but_rejects_conflicting_targets() { + let mut table = CallBindingTable::default(); + let site = key("f", Span::new(1, 2)); + assert!(table.insert( + site.clone(), + BoundTarget::Builtin { + canonical_name: "长度".into() + } + )); + assert!(table.insert( + site.clone(), + BoundTarget::Builtin { + canonical_name: "长度".into() + } + )); + assert!(!table.insert(site, BoundTarget::DynamicValue)); + assert_eq!(table.len(), 1); + assert_eq!(table.static_count(), 1); + assert_eq!(table.dynamic_count(), 0); + } +} diff --git a/crates/marqdo-ir/src/callable.rs b/crates/marqdo-ir/src/callable.rs index 824a4c9..64c0051 100644 --- a/crates/marqdo-ir/src/callable.rs +++ b/crates/marqdo-ir/src/callable.rs @@ -10,7 +10,7 @@ use std::fmt; use serde::{Deserialize, Serialize}; -use crate::{builtins, Function, Module, ModuleId, Param, Program, SignatureKind}; +use crate::{builtins, Function, FunctionId, Module, ModuleId, Param, Program, SignatureKind}; use marqdo_span::Span; /// Globally unique callable identity. @@ -138,6 +138,42 @@ pub struct CallableRegistry { builtins: BTreeMap, functions: BTreeMap>, external: BTreeMap, + /// Official stdlib public aliases (for example `打印`/`print`) mapped to + /// the bundled Markdown function that implements them. Only the verified + /// stdlib loader may populate this table; third-party packages cannot. + official_aliases: BTreeMap, +} + +/// One official bundled-stdlib public alias. +#[derive(Debug, Clone, PartialEq)] +pub struct OfficialAlias { + /// Public names that resolve to the same target (for example both + /// `打印` and `print`). + pub public_names: Vec, + /// Bundled Marqdo function implementing the alias. + pub function: FunctionId, + /// Effective signature; the loader must have checked it against the + /// verified stdlib metadata before registration. + pub signature: CallableSignature, + /// Owner package instance (the verified stdlib instance). + pub owner_package: String, + /// Conservative effect request summary for Trace/Analysis. + pub effects_requested: Vec, +} + +/// A deterministic static-table conflict. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CallableConflict { + /// Colliding public name. + pub public_name: String, + /// Human-readable reason; never used as a machine code. + pub reason: String, +} + +impl fmt::Display for CallableConflict { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "`{}`: {}", self.public_name, self.reason) + } } impl CallableRegistry { @@ -150,21 +186,46 @@ impl CallableRegistry { } /// Build a registry for all modules and declared external descriptors. - pub fn from_program(program: &Program) -> Self { + /// + /// Duplicate external names are rejected rather than silently keeping the + /// first entry: a static table must never depend on registration order. + pub fn try_from_program(program: &Program) -> Result { let mut registry = Self::default(); registry.add_builtins(); for module in program.modules.values() { registry.add_module(module); } for descriptor in &program.external_callables { - // Conflicting declarations are reported by the loader; keep the - // first deterministic entry here rather than overwriting silently. registry - .external - .entry(descriptor.public_name.clone()) - .or_insert_with(|| descriptor.clone()); + .insert_external(descriptor.clone()) + .map_err(|reason| CallableConflict { + public_name: descriptor.public_name.clone(), + reason, + })?; } - registry + Ok(registry) + } + + /// Build a registry for all modules and declared external descriptors. + /// + /// Prefer [`CallableRegistry::try_from_program`] on the CLI composition + /// path; this compatibility wrapper keeps the first declaration for + /// already-validated callers (tests and embedders). + pub fn from_program(program: &Program) -> Self { + Self::try_from_program(program).unwrap_or_else(|_| { + let mut registry = Self::default(); + registry.add_builtins(); + for module in program.modules.values() { + registry.add_module(module); + } + for descriptor in &program.external_callables { + registry + .external + .entry(descriptor.public_name.clone()) + .or_insert_with(|| descriptor.clone()); + } + registry + }) } /// Look up a builtin by canonical name or alias. @@ -188,6 +249,85 @@ impl CallableRegistry { self.external.values() } + /// Register one official bundled-stdlib public alias. + /// + /// The loader calls this only after verifying the bundled stdlib digest and + /// the metadata digest, so a third-party package can never claim `打印`, + /// `print` or any other official public name through this path. + pub fn insert_official_alias(&mut self, alias: OfficialAlias) -> Result<(), String> { + if alias.public_names.is_empty() { + return Err("official alias must declare at least one public name".to_string()); + } + if alias.function.module.instance.is_empty() || alias.function.name.is_empty() { + return Err("official alias target function identity must not be empty".to_string()); + } + for name in &alias.public_names { + if name.is_empty() { + return Err("official alias public name must not be empty".to_string()); + } + if self.external.contains_key(name) { + return Err(format!( + "official alias `{name}` conflicts with a declared plugin export" + )); + } + let descriptor = CallableDescriptor { + id: CallableId(format!("marqdo:{}", alias.function)), + public_name: name.clone(), + owner_package: alias.owner_package.clone(), + signature: alias.signature.clone(), + effects_requested: alias.effects_requested.clone(), + origin: CallableOrigin::Marqdo, + implementation_digest: None, + source_span: None, + }; + // Official aliases intentionally replace the reserved placeholder + // builtin (for example `打印`); no third-party path can do this. + self.builtins.insert(name.clone(), descriptor); + self.official_aliases + .insert(name.clone(), alias.function.clone()); + } + Ok(()) + } + + /// Target Marqdo function of an official alias, if `name` is one. + pub fn alias_target(&self, name: &str) -> Option<&FunctionId> { + self.official_aliases.get(name) + } + + /// Iterate official aliases deterministically (public name, target). + pub fn official_aliases(&self) -> impl Iterator { + self.official_aliases.iter() + } + + /// Whether a dotted builtin namespace starts with `prefix.`. + pub fn has_builtin_dotted_prefix(&self, prefix: &str) -> bool { + self.builtins.keys().any(|name| { + name.strip_prefix(prefix) + .is_some_and(|rest| rest.starts_with('.')) + }) + } + + /// Whether a declared plugin export starts with `prefix.`. + pub fn has_external_dotted_prefix(&self, prefix: &str) -> bool { + self.external.keys().any(|name| { + name.strip_prefix(prefix) + .is_some_and(|rest| rest.starts_with('.')) + }) + } + + /// Resolve a module function to its exact identity. + pub fn function_id(&self, module: &ModuleId, name: &str) -> Option { + let descriptor = self.function(module, name)?; + if matches!(descriptor.origin, CallableOrigin::Marqdo) { + Some(FunctionId { + module: module.clone(), + name: descriptor.public_name.clone(), + }) + } else { + None + } + } + /// Insert an external descriptor, rejecting duplicate public names. pub fn insert_external(&mut self, descriptor: CallableDescriptor) -> Result<(), String> { if self.builtins.contains_key(&descriptor.public_name) { diff --git a/crates/marqdo-ir/src/lib.rs b/crates/marqdo-ir/src/lib.rs index e6073fc..cc3b784 100644 --- a/crates/marqdo-ir/src/lib.rs +++ b/crates/marqdo-ir/src/lib.rs @@ -17,6 +17,7 @@ use marqdo_span::Span; use serde::{Deserialize, Serialize}; pub mod builtins; +pub mod call_binding; pub mod callable; /// Canonical logical identity of a module: package instance plus a path @@ -476,10 +477,13 @@ pub struct Program { /// metadata only: `check` never loads a library. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub external_callables: Vec, - /// Compat-B/C `打印`/`print` destination (ADR-0030/0031). When set, the - /// runtime calls this stdlib function; there is no legacy HostOp path. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub print_override: Option, + /// Resolved static call bindings (RFC-EXT-0001 §4.2, ADR-0054). + /// + /// Internal IR: deliberately skipped on the wire so Analysis/Trace v2 + /// consumers are unaffected. A prepared program always carries a complete + /// table; the runtime never re-resolves names. + #[serde(skip)] + pub call_bindings: call_binding::CallBindingTable, } impl Program { @@ -517,6 +521,19 @@ impl Program { pub fn callable_registry(&self) -> callable::CallableRegistry { callable::CallableRegistry::from_program(self) } + + /// Resolved static call binding for one call site, if any. + pub fn call_binding( + &self, + key: &call_binding::CallSiteKey, + ) -> Option<&call_binding::BoundTarget> { + self.call_bindings.get(key) + } + + /// Install the prepared binding table. + pub fn set_call_bindings(&mut self, bindings: call_binding::CallBindingTable) { + self.call_bindings = bindings; + } } #[cfg(test)] diff --git a/crates/marqdo-runtime/src/interpreter.rs b/crates/marqdo-runtime/src/interpreter.rs index bc5172b..cdcb8f3 100644 --- a/crates/marqdo-runtime/src/interpreter.rs +++ b/crates/marqdo-runtime/src/interpreter.rs @@ -54,7 +54,6 @@ pub struct RunOutcome { /// Execution frame for one function call. struct Frame { function: FunctionId, - namespace: Option, locals: BTreeMap, } @@ -179,7 +178,6 @@ impl<'a> Interpreter<'a> { }; let mut frame = Frame { function: function.id.clone(), - namespace: function.namespace.clone(), locals, }; self.depth += 1; @@ -277,7 +275,6 @@ impl<'a> Interpreter<'a> { // an empty frame so they cannot reach caller state. let mut empty = Frame { function: function.id.clone(), - namespace: function.namespace.clone(), locals: BTreeMap::new(), }; let value = self.eval_expr(default, &mut empty)?; @@ -616,113 +613,96 @@ impl<'a> Interpreter<'a> { span: Span, frame: &mut Frame, ) -> Result { - let caller_module = frame.function.module.clone(); - match callee { - Expr::Ident(ident) => { - if let Some(value) = frame.locals.get(&ident.name).cloned() { - let args = self.eval_args(args, frame)?; - return self.call_value(program, value, args, Some(span), Some(&caller_module)); - } - if matches!(ident.name.as_str(), "打印" | "print") { - if let Some(target) = program.print_override.clone() { - let args = self.eval_args(args, frame)?; - let Some(function) = program.function(&target) else { - return Err(RuntimeError::new( - ErrorCategory::UnknownSymbol, - Some(span), - format!("print override `{target}` does not exist"), - )); - }; - return self.call_function( - program, - function, - args, - Some(span), - Some(&caller_module), - ); - } - } - if let Some(builtin) = builtins::find(&ident.name) { - let args = self.eval_args(args, frame)?; - return self.call_builtin(builtin, args, Some(span), frame); - } - let args = self.eval_args(args, frame)?; - let (module, namespace) = (frame.function.module.clone(), frame.namespace.clone()); - let Some(function) = - resolve_function(program, &module, namespace.as_deref(), &ident.name) - else { + // ADR-0054 / RFC-EXT-0001 §4: the frontend resolved this call site + // exactly once. Dispatch by the recorded identity; never re-resolve a + // name, never infer a plugin from a dotted prefix. + let key = marqdo_ir::call_binding::CallSiteKey::new(frame.function.clone(), span); + let Some(target) = program.call_bindings.get(&key).cloned() else { + return Err(RuntimeError::new( + ErrorCategory::Internal, + Some(span), + format!( + "call site `{key}` has no static binding; run a prepared program (ADR-0054)" + ), + )); + }; + match target { + marqdo_ir::call_binding::BoundTarget::Builtin { canonical_name } => { + let Some(builtin) = builtins::find(&canonical_name) else { return Err(RuntimeError::new( - ErrorCategory::UnknownSymbol, + ErrorCategory::Internal, Some(span), - format!("unknown function `{}`", ident.name), + format!("bound builtin `{canonical_name}` is not registered"), )); }; - self.call_function(program, function, args, Some(span), Some(&caller_module)) + let args = self.eval_args(args, frame)?; + self.call_builtin(builtin, args, Some(span), frame) } - Expr::Member { base, name, .. } => { - if let Expr::Ident(alias) = base.as_ref() { - if let Some(target_module) = - program.resolve_import(&frame.function.module, &alias.name) - { - let args = self.eval_args(args, frame)?; - let Some(function) = resolve_function(program, target_module, None, name) - else { - return Err(RuntimeError::new( - ErrorCategory::UnknownSymbol, - Some(span), - format!("unknown function `{name}` in module `{target_module}`"), - )); - }; - return self.call_function( - program, - function, - args, - Some(span), - Some(&caller_module), - ); - } - let qualified = format!("{}.{name}", alias.name); - if let Some(builtin) = builtins::find(&qualified) { - let args = self.eval_args(args, frame)?; - return self.call_builtin(builtin, args, Some(span), frame); - } - if !frame.locals.contains_key(&alias.name) { - // A declared external descriptor is the only static - // source of plugin exports. Calls without a descriptor - // are still accepted for compatibility only when no - // plugin descriptors exist at all; this fallback is - // removed once the CLI always supplies descriptors. - let qualified = format!("{}.{}", alias.name, name); - let declared = program.external_callable(&qualified).is_some(); - if !declared && !program.external_callables.is_empty() { - return Err(RuntimeError::new( - ErrorCategory::UnknownSymbol, - Some(span), - format!("unknown plugin export `{qualified}`"), - )); - } - // Plugin calls take named arguments only in this build. - let evaluated = self.eval_args(args, frame)?; - let mut named = std::collections::BTreeMap::new(); - for argument in evaluated { - let Some(argument_name) = argument.name else { + marqdo_ir::call_binding::BoundTarget::Marqdo { function } => { + let Some(target_function) = program.function(&function) else { + return Err(RuntimeError::new( + ErrorCategory::Internal, + Some(span), + format!("bound function `{function}` does not exist"), + )); + }; + let args = self.eval_args(args, frame)?; + self.call_function( + program, + target_function, + args, + Some(span), + Some(&frame.function.module.clone()), + ) + } + marqdo_ir::call_binding::BoundTarget::AbiPlugin { + package: _, + export, + digest: _, + param_names, + } => { + // ABI v1 transports named arguments as a map. Normalise + // positional arguments deterministically against the static + // manifest order; static validation already rejected unknown, + // duplicate or missing arguments. + let evaluated = self.eval_args(args, frame)?; + let mut named = std::collections::BTreeMap::new(); + let mut position = 0usize; + for argument in evaluated { + let name = match argument.name { + Some(name) => name, + None => { + let Some(name) = param_names.get(position).cloned() else { return Err(RuntimeError::new( - ErrorCategory::Type, - Some(argument.span), - "plugin calls require named arguments", + ErrorCategory::Internal, + Some(span), + format!( + "positional argument {} was not normalised for `{export}`", + position + 1 + ), )); }; - named.insert(argument_name, argument.value); + position += 1; + name } - let module = frame.function.module.clone(); - return self.host_invoke( - HostOp::PluginInvoke, - vec![Value::Text(qualified), Value::Map(named)], + }; + if named.insert(name.clone(), argument.value).is_some() { + return Err(RuntimeError::new( + ErrorCategory::Arity, Some(span), - &module, - ); + format!("duplicate argument `{name}` for `{export}`"), + )); } } + let module = frame.function.module.clone(); + self.host_invoke( + HostOp::PluginInvoke, + vec![Value::Text(export), Value::Map(named)], + Some(span), + &module, + ) + } + marqdo_ir::call_binding::BoundTarget::DynamicValue => { let callee_value = self.eval_expr(callee, frame)?; let args = self.eval_args(args, frame)?; self.call_value( @@ -730,18 +710,7 @@ impl<'a> Interpreter<'a> { callee_value, args, Some(span), - Some(&caller_module), - ) - } - other => { - let callee_value = self.eval_expr(other, frame)?; - let args = self.eval_args(args, frame)?; - self.call_value( - program, - callee_value, - args, - Some(span), - Some(&caller_module), + Some(&frame.function.module.clone()), ) } } @@ -935,21 +904,6 @@ impl<'a> Interpreter<'a> { } } -fn resolve_function<'p>( - program: &'p Program, - module: &ModuleId, - namespace: Option<&str>, - name: &str, -) -> Option<&'p Function> { - let module = program.module(module)?; - if let Some(namespace) = namespace { - if let Some(found) = module.namespaced_function(namespace, name) { - return Some(found); - } - } - module.function(name) -} - /// Bind already-evaluated arguments to a builtin parameter list. fn bind_builtin_args( builtin: &'static builtins::Builtin, diff --git a/crates/marqdo-runtime/tests/interpreter.rs b/crates/marqdo-runtime/tests/interpreter.rs index 124dd2e..ba838da 100644 --- a/crates/marqdo-runtime/tests/interpreter.rs +++ b/crates/marqdo-runtime/tests/interpreter.rs @@ -19,6 +19,14 @@ fn build(source: &str) -> Program { ..Program::default() }; program.insert(result.module); + let registry = program.callable_registry(); + let bound = marqdo_frontend::calls::bind_program_calls(&program, ®istry); + assert!( + bound.diagnostics.is_empty(), + "unexpected binding diagnostics: {:#?}", + bound.diagnostics + ); + program.set_call_bindings(bound.bindings); program } diff --git a/crates/marqdo-value/src/diagnostic.rs b/crates/marqdo-value/src/diagnostic.rs index 4c36cc0..95804f3 100644 --- a/crates/marqdo-value/src/diagnostic.rs +++ b/crates/marqdo-value/src/diagnostic.rs @@ -81,6 +81,34 @@ pub mod codes { pub const PROJECT_LOCK: &str = "project.lock"; /// Dependency resolution failed. pub const PROJECT_DEPENDENCY: &str = "project.dependency"; + /// A statically visible callable export is declared more than once + /// (RFC-EXT-0001 §16.1). + pub const EXT_DUPLICATE_CALLABLE: &str = "ext.duplicate_callable"; + /// An import alias collides with a builtin or plugin dotted namespace. + pub const EXT_IMPORT_NAMESPACE_CONFLICT: &str = "ext.import_namespace_conflict"; + /// A loaded plugin's exports/signature/owner/artifact digest do not exactly + /// match the statically prepared descriptors. + pub const EXT_PLUGIN_BINDING_MISMATCH: &str = "ext.plugin_binding_mismatch"; + /// Two resource providers try to register the same non-reserved kind. + pub const EXT_DUPLICATE_PROVIDER_KIND: &str = "ext.duplicate_provider_kind"; + /// A legacy/future provider manifest is passed to the v2-only CLI entry. + pub const EXT_UNSUPPORTED_PROVIDER_SCHEMA: &str = "ext.unsupported_provider_schema"; + /// No provider artifact matches the current host target triple. + pub const EXT_NO_MATCHING_ARTIFACT: &str = "ext.no_matching_artifact"; + /// A provider artifact differs from its declared sha256, escapes its + /// bundle, is not a regular file or is an unallowed symlink. + pub const EXT_ARTIFACT_INTEGRITY: &str = "ext.artifact_integrity"; + /// The matching provider worker/helper program is missing or cannot start. + pub const EXT_WORKER_UNAVAILABLE: &str = "ext.worker_unavailable"; + /// The current platform explicitly does not support the requested + /// extension transport. + pub const EXT_PLATFORM_UNSUPPORTED: &str = "ext.platform_unsupported"; + /// A locked extension identity does not match the actual manifest or + /// artifact on disk. + pub const EXT_EXTENSION_LOCK_MISMATCH: &str = "ext.extension_lock_mismatch"; + /// An extension is explicitly selected but is not part of the project lock + /// (development-only use; release mode rejects it). + pub const EXT_UNLOCKED_EXTENSION: &str = "ext.unlocked_extension"; /// Internal invariant violation; never expected in normal operation. pub const INTERNAL: &str = "internal.error"; } diff --git a/docs/adr/0054-callable-binding-single-source.md b/docs/adr/0054-callable-binding-single-source.md new file mode 100644 index 0000000..2b22ea3 --- /dev/null +++ b/docs/adr/0054-callable-binding-single-source.md @@ -0,0 +1,125 @@ +# ADR-0054: One static callable binding, one runtime dispatch + +- Status: Accepted (2026-11-21) +- Supersedes the resolution model of ADR-0009/ADR-0010 where they conflict +- Implements: RFC-EXT-0001 §4 (PR-01/PR-02) + +## Context + +The tree had four independent ways to decide what a call means: + +1. `marqdo-frontend/src/calls.rs` checked a Member call's builtin name before + its import alias; +2. `marqdo-runtime/src/interpreter.rs` checked the import alias before the + builtin for the same expression; +3. the runtime treated an undeclared `alias.name` as a plugin invocation when + `program.external_callables` was empty ("try it at run time"); +4. `marqdo-cli` synthesised an empty-parameter descriptor (`params: Vec::new()`) + for exports that were loaded but not statically described. + +Consequences observed while building RFC-EXT-0001's fixtures: + +* `check` could accept a call that `run` dispatched to a different + implementation (or vice versa); +* an unknown symbol failed only at run time, with a Host side effect already + attempted; +* an import alias that happened to equal a builtin dotted namespace (`math`) + silently changed the meaning of `math.eval(...)` depending on the pipeline; +* plugin arity/name errors could pass static checking and only fail (or + succeed with different arguments) at the ABI boundary. + +The language contract (RFC-LANG-0002) already says calls are expressions and +that `check` and `run` must agree; the implementation had no such guarantee. + +## Decision + +1. **One registry.** `marqdo_ir::callable::CallableRegistry` is the only static + table. It contains builtins (including official bundled-stdlib public + aliases), module functions keyed by canonical name plus unique local name, + and exact plugin exports (`namespace.function`). Duplicate declarations are + rejected when the registry is built; order never decides the answer. +2. **One binding pass.** `marqdo_frontend::calls::bind_program_calls(program, + registry)` walks every statically visible call expression exactly once and + produces `CallBindingTable` plus diagnostics. It replaces the split + `validate_module`/`validate_program` priority logic on the CLI path (the + standalone frontend keeps a wrapper for single-document tests). The result + is installed on `Program::call_bindings`. +3. **Deterministic identity.** A call site is + `(module id, function id, byte span)` (`CallSiteKey`), never a disk path, + registration order or hash-map iteration. A bound target is one of: + * `Builtin { canonical_name }` — fixed core function; + * `Marqdo { function }` — exact `FunctionId` (module functions and official + stdlib aliases such as `打印`/`print` → `io.mq.md::写值`); + * `AbiPlugin { package, export, digest, param_names }` — complete export + name plus the statically verified artifact digest and manifest argument + order; + * `DynamicValue` — a language value (for example a `FunctionRef`); this is + the **only** non-static path and cannot reach an ABI export by name. +4. **Runtime dispatches by identity.** `eval_call` looks up the site's binding + and calls the recorded target. It never calls `builtins::find(qualified)`, + never inspects `external_callables.is_empty()` and never guesses a plugin. + A missing binding is an internal error (`internal.error`), not a reason to + re-resolve. +5. **Prepare finalises once.** `marqdo-cli` has one composition step + (`project_prepare::prepare`): load sources → read extension descriptors + (no `dlopen`/spawn) → build the registry → bind calls once. `check`, + `run`/`call`, `inspect` and Analysis consume that one prepared program. + Static plugin descriptors are inserted before binding, not appended after a + first validation. +6. **Exact run-time reconciliation.** `run` verifies each selected plugin + artifact, loads it, then compares the Host's actual exports (name, raw + signature, artifact digest, owner package) with the static descriptors + one-to-one. Any extra, missing or altered export fails with + `ext.plugin_binding_mismatch` before the entry function runs. No descriptor + is synthesised at run time. +7. **Print is not special.** `Program.print_override` and the runtime + `打印`/`print` branch are removed. The builtin placeholder remains only for + embedders without the official bundle and fails with + `kernel.unavailable`; the official bundle declares aliases in verified + metadata (ADR-0058). + +## Priority matrix (normative) + +See `spec/callable-resolution.md` for the complete matrix and diagnostics. In +short: + +* `f(...)`: local variable → official alias/reserved builtin → current + namespace → canonical name → unique local name → otherwise + `symbol.unknown` (ambiguity is an error, never a first-match); +* `alias.f(...)`: explicit import alias → builtin dotted namespace → exact + plugin export → otherwise `symbol.unknown`; +* `some_value.f(...)`: dynamic language dispatch only. + +Import aliases that collide with a builtin or plugin dotted namespace are +rejected at prepare with `ext.import_namespace_conflict`; duplicate plugin +exports are rejected with `ext.duplicate_callable`. + +## Consequences + +* `check` and `run` now share binding identity; CALL-01…CALL-16 in + `crates/cli/tests/callable_resolution.rs` pin the matrix. +* 0.x breaking change: a project importing a module under an alias that equals + a builtin namespace (for example `math`) must rename the alias. There is no + `--legacy` switch and no dual semantics. +* Analysis v2's `calls[].state/target` are now projected from the same binding + (official aliases appear as `function` targets). Analysis v2 schema and + fields are otherwise unchanged; private `CallBindingTable` is not serialized. +* Runtime no longer performs per-call name resolution, so the dynamic path is + limited to language values. A text or map value called as a function is + `call.not_callable` and never becomes a Host plugin request. +* Plugin manifests must be complete before binding: a manifest with a missing + or malformed signature is a prepare error; ABI v1 default values are + rejected until the wire semantics are versioned. + +## Rejected alternatives + +* **Pick a fixed precedence and keep two implementations.** Rejected: the RFC + requires one static answer, and duplicated precedence is exactly the defect. +* **Build the registry lazily in the runtime.** Rejected: `check` would still + not prove the binding, and runtime loading would change static resolution. +* **Keep the empty-`external_callables` fallback for "developer convenience".** + Rejected: it turns a compile error into a Host call attempt and breaks + `check`'s no-execution guarantee. +* **Expose `CallableId`/`CallBindingTable` in Analysis v2.** Rejected: + private IR must not become a public wire contract without a version bump. +EOF diff --git a/docs/adr/0055-resource-provider-cli-v2.md b/docs/adr/0055-resource-provider-cli-v2.md new file mode 100644 index 0000000..6e32663 --- /dev/null +++ b/docs/adr/0055-resource-provider-cli-v2.md @@ -0,0 +1,196 @@ +# ADR-0055: Resource Provider v2 through the normal CLI + +Status: **Accepted** +Date: 2026-11-21 +Relates to: RFC-EXT-0001 §5/§16/§18, ADR-0033, ADR-0034, ADR-0035, ADR-0036, +ADR-0037, ADR-0038, ADR-0021 (strict L1), ADR-0052 (single Markdown semantics) +Specs: `spec/resource-provider-v2.md` +Implementation: `crates/cli/src/provider_manifest.rs`, +`crates/cli/src/commands/sandbox.rs`, `crates/cli/src/commands/run.rs`, +`crates/cli/src/main.rs`, `crates/cli/tests/provider_cli.rs` + +## Context + +The host already implements versioned resource-provider registration +(`register_resource_provider`, `register_worker_resource_provider`, +`register_c_resource_provider`, `register_provider_package`) and +`resource.open` dispatches unknown kinds through that registry with a provider +name grant (ADR-0033/0035/0036/0037/0038). What was missing was a **public +composition-root entry point**: `main.rs`/`SandboxSettings`/`run.rs` had no +`--provider`/`--allow-provider`, so an external repository could not use a new +resource kind on a frozen `marqdo` binary without patching and rebuilding the +CLI (RFC-EXT-0001 EXT-07). + +The existing `marqdo.provider` manifest (`schema_version: 1`, ADR-0037) is a +host-embedding installation format: `name/kinds/abi/library`, no `version`, no +targeted artifact digest and no transport. It cannot support portable, +integrity-checked CLI loading. RFC-EXT-0001 §5.3 asks for a v2 document rather +than silently overloading v1. It also requires that selecting an extension is +not authorizing it, that `pure`/`strict` keep failing closed, and that no new +`HostOp`/builtin/business branch is introduced. + +This ADR records the PR-03 decision on the current branch. + +## Decision + +### 1. CLI surface and ordering + +`--provider PATH` (repeatable, a `provider.json` file) and +`--allow-provider NAME` (repeatable, exact name) are added to `SandboxArgs`, so +they are shared by `run` and `call` only. `check`/`inspect` do not declare them +and never read a provider manifest. `SandboxSettings` carries +`providers: Vec` and `allow_provider: Vec`; the `project` +capability set calls the existing `Capabilities::allow_resource_provider(name)` +for each grant. + +Run ordering follows RFC-EXT-0001 §18.2: `prepare` (no provider work) → +diagnostics → mode/strict preflight → capabilities → host assembly and budgets +→ kernels → **all provider manifests/targets/artifacts/grants/duplicate kinds +verified and plugin artifacts verified** → plugin workers loaded and exports +reconciled → provider helpers spawned → entry-function check → execute. No +worker (plugin or provider) starts before every T6/T6b verification passes, and +the main function never runs after a provider failure. + +`pure` and `strict` reject both flags in preflight (`cli.invalid_grants` / +`strict.unsupported_grant`, exit 2) without silently switching to `project`. +`--allow-provider` rejects wildcards (`*`, `?`, `[`, `]`) and empty names; a +grant that matches no selected manifest is `ext.provider_grant_unmatched` +(exit 2). A `--provider` selected without its exact grant is +`permission.denied` (exit 1) before the helper is spawned. + +### 2. Strict v2 manifest parser + +New `crates/cli/src/provider_manifest.rs` owns the document, separate from the +plugin `plugin.yaml` parser: + +* accepts exactly `schema = "marqdo.provider"`, `schema_version = 2`, + `abi = "c-abi-1"`, `transport = "worker"`; other versions/ABIs/transports are + `ext.unsupported_provider_schema` (no silent migration, no v1 fallback from + the CLI); +* strict dotted lowercase name/kind grammar, strict SemVer, non-empty unique + non-reserved kinds, bounded counts, known/vendor `permissions_requested` + declarations only, optional non-executable `metadata`; +* unknown top-level/artifact fields are rejected (`ext.manifest_invalid`); +* artifacts need unique parseable targets and 64-lowercase-hex sha256; + the current `HostTarget` artifact is selected (`ext.no_matching_artifact` + when absent), resolved inside the manifest bundle (absolute/`..`/symlink + escape/non-regular/size limit/sha mismatch → `ext.artifact_integrity`); +* duplicate names with different manifest digests (`ext.duplicate_provider`) + and duplicate kinds across providers (`ext.duplicate_provider_kind`) are + deterministic because paths are canonicalised, sorted and de-duplicated + before conflict detection. + +`VerifiedProvider` is the static `{name, kinds, manifest_path, base_dir, +manifest_digest, transport, selected_artifact_path/relative/sha256/target, +version}` record. It is still declaration/verification only: no `dlopen`, no +spawn, no grant. + +### 3. Activation reuses the existing host API + +`load_resource_provider_into_host(host, verified, allow_provider)` repeats the +exact grant check, re-verifies the manifest digest and the artifact +(regular file, containment, size, sha256) immediately before use, resolves the +helper, and calls **the existing** +`StdHost::register_worker_resource_provider(name, kinds, helper, [artifact])`. +It never falls back to `register_c_resource_provider`, +`register_provider_package`, in-process native loading or another provider. +`resource.open` still re-checks the provider name grant in the Host. Provider +registration keeps using `register_provider_checked` for name/kind/duplicate +validation, so no second provider registry is introduced. + +### 4. Helper resolution and platform claim + +The helper is resolved only as a sibling of the running `marqdo` executable: +`/marqdo-provider-worker[.exe]`. `PATH`, the current directory, +environment overrides and downloads are not consulted. Missing/non-regular +helper, spawn failure or failed `ping` → `ext.worker_unavailable` (exit 1). + +The `worker` transport is claimed only on **linux/x86_64**; every other target +returns `ext.platform_unsupported` (exit 1) instead of pretending success. +Release artifacts must ship the helper next to the CLI for the same target +triple. macOS/Windows stay buildable and explicitly unsupported until they have +a native fixture and release evidence (RFC-EXT-0001 §5.6). + +### 5. No new host op or language surface + +There is no new `HostOp`, builtin, `resource_request` string branch, `Value` +variant or Markdown syntax. A provider kind is still an argument to +`resource.open("", descriptor, rights)`; the only change is the CLI +composition root that selects, verifies and authorizes the existing host +transport. Provider tokens stay inside the helper; the language sees only the +opaque Host `Resource` handle. Worker errors are structured `RuntimeError`s +surfaced as CLI errors — the CLI never panics on a helper crash/timeout. + +### 6. Tests + +`crates/cli/tests/provider_cli.rs` drives the real CLI binary with a real C +`c-abi-1` sensor library and the real `marqdo-provider-worker` helper built by +cargo and installed next to the CLI (the same layout a release tarball uses; +production resolution is not weakened for tests). Coverage: PROV-01/02/03/04/ +05/06/07/08/09/10/13, plus `check` refusing the flag. Marker files written by +the C library's ABI-version hook prove whether a worker actually started, so +"no worker on unauthorized/invalid input" is evidence, not an assertion about +code paths. + +## Consequences + +* A third party can add a new resource kind to a frozen `marqdo` binary through + `provider.json` v2 + `--provider` + `--allow-provider`, without a core + rebuild, new builtin or new CLI command. +* v1 `marqdo.provider` remains a host-embedding format; CLI users get a clear + `ext.unsupported_provider_schema` instead of a silently unverified load. +* Integrity hashing raises the cost of accidental corruption/tampering but is + not a trust root; a compromised or malicious helper still runs with the + user's privileges. Documentation states this explicitly. +* `pure`/`strict` semantics and the existing resource/provider host contracts + are unchanged (strict still L1-only and fail-closed). +* The provider helper becomes a release artifact that must be packaged next to + the CLI; a missing helper is a first-class, deterministic error. + +## Alternatives considered + +1. **Reuse `plugin.yaml`/plugin ABI directly for providers.** Rejected: a + resource provider's contract is open/read/write/close with provider-private + tokens and stream kinds, not callable exports; sharing the manifest/ABI + parser would conflate two independent protocols (RFC-EXT-0001 §1.3). +2. **Extend v1 in place with optional `version`/`artifacts`.** Rejected: + v1 is host-embedding installation metadata and is already accepted by + `register_provider_package`; silently changing its meaning or accepting a + half-populated v1 would break the frozen ADR-0037 contract. v2 is explicit. +3. **Allow in-process C ABI loading when the worker helper is missing.** + Rejected: it would silently lower the process boundary and RFC-EXT-0001 + §5.2/§5.4 forbids falling back to Native. `ext.worker_unavailable` is the + correct fail-closed result. +4. **PATH lookup for `marqdo-provider-worker`.** Rejected: `PATH` is + attacker-controlled state and would make the executed artifact + non-deterministic; the sibling rule binds the helper to the release. +5. **Register providers lazily at the first `resource.open`.** Rejected for + this slice: authorization and integrity must be decided before the main + function executes, and the existing host API is synchronous spawn-on-register. +6. **Accept a `*` provider grant for developer convenience.** Rejected: grants + are exact names (RFC-EXT-0001 §5.2); wildcards would let a typo authorize an + arbitrary selected provider. + +## Deviations recorded + +* `ext.manifest_invalid`, `ext.duplicate_provider` and + `ext.provider_grant_unmatched` are **additive** `ext.*` codes beyond the list + frozen in RFC-EXT-0001 §18.1. The frozen list has no generic malformed-v2 + code, no duplicate-provider-name code and no unmatched-grant code; reusing + `ext.unsupported_provider_schema`/`ext.duplicate_provider_kind` for these + would report the wrong reason. They follow the same `ext.` naming and exit + code contract and are frozen by this ADR. +* `ext.artifact_integrity` failures exit 1 (load/verify), while static + manifest-selection failures exit 2, matching RFC-EXT-0001's exit-code + guidance; plugin artifact checking already uses the same split. +* Only linux/x86_64 is claimed for the provider `worker` transport in this + release (RFC-EXT-0001 §5.6 permits this explicit platform limit; Windows and + macOS return `ext.platform_unsupported`). + +## Verification + +`crates/cli/tests/provider_cli.rs` (12 real tests, linux/x86_64) plus +`crates/cli/src/provider_manifest.rs` unit tests pass; `spec/security.md`, +`spec/resource-provider-v2.md` and the README provider section document the +same contract. Evidence is recorded in the PR-03 task report (commands, exit +codes, per-PROV matrix). diff --git a/docs/adr/0057-capability-handler-registration.md b/docs/adr/0057-capability-handler-registration.md new file mode 100644 index 0000000..a78c5fc --- /dev/null +++ b/docs/adr/0057-capability-handler-registration.md @@ -0,0 +1,229 @@ +# ADR-0057: Trusted capability-handler registration + +Status: **Accepted** +Date: 2026-11-21 +Implements: RFC-EXT-0001 §7 (PR-05) +Extends: ADR-0011 (capability broker) +Scope: `crates/marqdo-host/src/capability.rs` (host-internal registry) + +## Context + +ADR-0011 introduced `CapabilityBroker` with two separate ideas but only one +implementation path: + +* `decide()` answered "is this capability granted?" from `Capabilities` and a + trusted vendor-policy allow-list; and +* `execute()` special-cased the single real core implementation (`fs.read`) + inline, returning `capability.unavailable` for everything else. + +That shape is why adding a capability looked like a one-line allow-list edit, +and it made "declared in policy" and "can actually execute" the same object. +RFC-EXT-0001 §7.1/§7.2 requires the chain: + +```text +manifest requests capability + -> operator/CLI grants it + -> broker asks: is a trusted handler installed? + -> handler re-checks owner/grant/scope/budget + -> handler executes + -> audit { identity, effect, decision, duration, trace_id } +``` + +A name in an allow-list must never be enough to execute, and a capability with +no executor must never look like an empty success. + +## Decision + +### 1. Two independent registries + +`CapabilityBroker` keeps: + +* `vendor_policy: BTreeSet` — names the trusted host *policy* accepts + (from `allow_vendor`); and +* `handlers: BTreeMap` — names that have an explicitly + registered trusted executor. + +`HandlerEntry` additionally carries an optional exact trusted package binding +(`None` = authorized by the capability grant alone). + +The two sets cannot substitute for each other: + +* vendor capability without a policy entry -> `permission.denied` + (declaration is not a grant; CAP-01); +* vendor policy entry without a handler -> `capability.unavailable` (CAP-02); +* core capability granted by `Capabilities` but without a handler -> + `capability.unavailable` (CAP-12). + +### 2. Handler interface + +```rust +pub struct CapabilityRequest<'a> { + pub package_instance: &'a str, // trusted identity from the host, not wire data + pub capability: &'a str, + pub payload: &'a [u8], + pub trace_id: &'a str, +} + +pub trait CapabilityHandler: Send { + fn name(&self) -> &str; + fn execute( + &mut self, + request: &CapabilityRequest<'_>, + capabilities: &Capabilities, + budget: &mut Budget, + ) -> Result, PluginError>; +} +``` + +The registry key is `handler.name()`, so the handler's declared capability can +never disagree with the name it is registered under (the RFC's "declared +protocol mismatch" rejection cannot occur by construction). + +Registration is host-only; there is no wire path from a plugin manifest to this +API. `register_handler` installs a handler usable by any package holding a +grant; `register_package_handler(package, handler)` binds it to one exact +trusted package identity and denies every other claimed package (CAP-03, the +"trust the package the host loaded, not a self-reported name" rule of §7.4). + +### 3. Registration validation + +`register_handler` / `register_package_handler` reject: + +| condition | stable code | +|---|---| +| empty name, surrounding whitespace, or not a core/vendor name | `capability.handler_invalid` | +| name under a reserved host prefix (`marqdo.`, `host.`) | `capability.handler_reserved` | +| name already registered (including the built-in `fs.read`) | `capability.handler_conflict` | + +A rejected handler is never installed. The component that constructs the +broker is the trust boundary: only trusted host/embedder code can register, so +"untrusted source" is rejected by the absence of an API rather than a flag. + +### 4. Fixed execution chain and audit + +`execute()` is one function with one order: + +1. `decide()` — `Capabilities`, vendor policy, request payload shape; +2. handler installed? — otherwise audit `unavailable`, return + `capability.unavailable`; +3. package-bound handler owns the claimed package? — otherwise audit `denied`, + return `permission.denied`; +4. handler executes under the broker `Budget` and the broker reply limit; +5. audit one entry with `package`, `capability`, `decision` + (`granted`/`denied`/`unavailable`), reason and `duration_micros`. + +`authorize()` remains the grant decision only (`granted`/`denied`); it never +executes and therefore cannot demonstrate an executor. This keeps the +authorization and execution questions separate and auditable. + +The reply is bounded by `reply_limit` (default +`MAX_CAPABILITY_REPLY_BYTES`) and oversize replies are `resource.exhausted` +(CAP-06). Handlers also receive `&mut Budget`, so an embedder can wire the +per-run budget and handlers can charge fuel/output (tested). Response payloads +are never written to the audit log, so file contents and request secrets stay +out of it (CAP-09). + +### 5. `fs.read` becomes the first standard handler + +The former inline `fs.read` match arm moved into `FsReadHandler`, installed by +`CapabilityBroker::with_fs(fs)`. It: + +* resolves the path with `ProjectFs::resolve` (absolute paths, `..`, empty + paths rejected) **before** I/O; +* re-checks `Capabilities::check_read` against the normalized + project-relative path at the execution point, so the grant is enforced on the + same path the filesystem uses; +* reads through `ProjectFs` (regular file, UTF-8, project root, 8 MiB limit). + +`with_fs` still installs exactly one handler; `capability_broker.rs` and +`host.rs` call sites are unchanged. Every other core capability and every +vendor capability without a registered handler keeps returning +`capability.unavailable`; nothing returns an empty `Ok` as a stand-in for +"implemented". + +## RFC deviations (explicit) + +1. **`EffectiveGrant` -> `Capabilities`; `ResourceBudget` -> `Budget`.** The + RFC (§7.2) says these are design-interface names and the implementation must + reuse the existing grant and limit structures instead of building a second + permission system; that is exactly what happened. +2. **`CapabilityError` -> `PluginError`.** The capability bridge across the + plugin/worker ABI already uses `marqdo_plugin_abi::PluginError`; reusing it + avoids a second error taxonomy. Codes remain the stable contract. +3. **`trace_id` is plumbed through `CapabilityRequest` and settable with + `with_trace_id`, but the host wiring is a follow-up.** `host.rs` does not set + it yet, so audit entries from the current CLI plugin path have an empty + trace id. Wiring it requires the run/session id from `ExecutionContext`, + which is outside this PR's write scope. +4. **`resource.exhausted` preserves the broker code but not the runtime + category.** `PluginError` kind `Capability` maps to + `RuntimeError` category `PermissionDenied` in `host.rs::map_plugin_error`. + The stable code is `resource.exhausted` (what CAP-06 asserts at the broker + boundary); teaching the host mapping about a resource-limit plugin code is a + follow-up because `host.rs` is outside this PR's scope. +5. **`CapabilityBroker` is no longer `Clone`** because it owns `Box`. No in-tree caller cloned it; a future embedder that + needs a clone should share the broker behind the same lock instead, which is + also what the one-call-at-a-time ABI requires. + +## Alternatives + +1. Keep the closed match arm and add capabilities to it. Rejected: every + vendor operation would need a core release and the broker would still have + no separation between "granted" and "executable". +2. Treat a vendor policy entry as proof of an executor. Rejected: that is the + exact "declaration becomes privilege" failure the RFC forbids; CAP-02 locks + it out. +3. Register handlers by `(name, package)` strings only and trust the caller's + package argument. Rejected: a wire-supplied package name must not authorize + anything; package-bound registration pins the identity the host selected at + load time. +4. Give every core capability an empty handler so `execute()` always returns + `Ok`. Rejected: it converts "not implemented" into a silent success, the + failure mode RFC §7.1 calls out. + +## Compatibility + +* Existing `Capabilities`, `with_fs`, `with_audit`, `allow_vendor`, + `authorize`, `execute` and `capability_audit()` call sites are unchanged; + `host.rs` needed no edit. +* One observable change: a vendor capability without a trusted policy entry is + now `permission.denied` instead of `capability.unavailable`. This matches + ADR-0011 ("a missing grant is `permission.denied`") and RFC CAP-01. +* `CapabilityAudit` gained `duration_micros`; all fields remain public and the + existing tests that read `package`/`capability`/`decision` compile unchanged. +* `CapabilityBroker` lost `Clone` (see deviation 5). +* No ABI, wire, manifest or CLI change. + +## Security + +* Registration is a trusted-host operation with no manifest/wire entry point; + a plugin cannot install, replace or enumerate handlers. +* Package-bound handlers reject cross-package impersonation even when policy + and grant match. +* `fs.read` keeps `ProjectFs` root confinement, glob scope and size limits; the + grant is re-checked on the normalized path at the execution point. +* Audit records decisions and durations but never payload contents. +* This remains a cooperative in-process API contract, not an OS sandbox: a + native plugin can still issue direct syscalls. The registry narrows what the + host will *do on request*; it does not confine plugin memory or syscalls. + +## Testing + +`crates/marqdo-host/tests/capability_handlers.rs` covers CAP-01..CAP-12: +policy-vs-handler separation, package binding, registration rejection, real +`fs.read` scope enforcement, reply budget and handler budget charging, forged +and replayed handles, audit hygiene, resource-stream regression smoke, project +mode not being OS-strict, and "no handler is never a silent success". +Unit tests in `capability.rs` keep the `CORE_CAPABILITIES` list in sync with +`decide()` and assert that the built-in `fs.read` handler is installed by +`with_fs`. + +## Migration & rollout + +`with_fs` installs the migrated `fs.read` handler automatically, so existing +plugin loads keep working. Future core capabilities migrate by adding a handler +and registering it in the trusted composition root; until then they remain +`capability.unavailable`. Vendor operations that need a non-streaming API +should first use the existing `resource.open/read/write/close` stream path +(RFC-EXT-0001 §7.3) instead of registering a new host handler. diff --git a/docs/adr/0058-stdlib-public-alias.md b/docs/adr/0058-stdlib-public-alias.md new file mode 100644 index 0000000..5474cb2 --- /dev/null +++ b/docs/adr/0058-stdlib-public-alias.md @@ -0,0 +1,105 @@ +# ADR-0058: Official stdlib public aliases replace print magic + +- Status: Accepted (2026-11-21) +- Implements: RFC-EXT-0001 §8 (PR-06) +- Related: ADR-0028/0030/0031/0039, ADR-0054 + +## Context + +`打印`/`print` were implemented twice: + +1. `crates/cli/src/loader.rs::program_uses_print` walked the entire AST, and if + any function called `打印`/`print` it injected the bundled + `stdlib/io.mq.md` module and set `Program.print_override` to its `写值` + function; +2. `crates/marqdo-runtime/src/interpreter.rs` contained a `打印`/`print` + special case that redirected to the override, while the builtin table also + contained a placeholder `打印` entry. + +Copying this pattern to the next official high-level function (formatting, +console input, …) would add one AST scanner and one runtime branch per name. +The RFC forbids that ("为标准库每个高级函数复制 print_override 模式"), while +also requiring byte-identical output and no new Host op. + +## Decision + +1. **Verified metadata.** The bundled `stdlib/stdlib.json` declares + + ```json + "public_aliases": [ + { + "public_names": ["print", "打印"], + "target_module": "io.mq.md", + "target_function": "写值", + "expected_signature": "params(值)", + "effects": ["host"] + } + ], + "aliases_digest": "sha256:<64 lowercase hex>" + ``` + + `aliases_digest` is the SHA-256 of the canonical binary encoding: for each + alias sorted by `(target_module, target_function)`, the module bytes, NUL, + the function bytes, NUL, then every public name in sorted order followed by + NUL. The same encoding is implemented by `tools/pack-stdlib.py` and + `crates/cli/src/stdlib.rs::aliases_digest`; `stdlib.json` cannot be edited + without updating the digest, and the loader verifies both digests before any + module loads. Metadata is integrity-checked, not signed: a SHA-256 proves + content equality, not publisher identity. +2. **Lazy, generic loading.** The loader reads the metadata, decides whether the + program references any public alias name (a generic identifier scan, not a + per-function hard code), and only then loads the declared target module. + The target signature is checked against `expected_signature`; a mismatch is + a load error. A program without `打印`/`print` keeps its previous module + graph and startup cost. +3. **Ordinary Marqdo binding.** For each used alias the loader produces an + `OfficialAlias { public_names, function, signature, owner_package, + effects_requested }`; `project_prepare` registers it in the + `CallableRegistry`. The binding pass targets + `Marqdo { function: ::io.mq.md::写值 }`. The runtime calls + an ordinary Markdown function; it has no print branch. +4. **Reserved names stay reserved.** Official aliases may replace the builtin + placeholder for the same public name. Third-party registry insertion rejects + builtin *and* official-alias names (`ext.duplicate_callable`), so no plugin + or imported module can claim `打印`/`print`, `std.*` or `marqdo.*`. + `plugin.yaml` still rejects reserved export prefixes at manifest validation. +5. **No Host print operation.** Output continues to flow through + `io.写值` → `format.显示` → UTF-8 bytes → `Memory` → + `resource.write(stdout)`. There is no `HostOp::Print` and no second + formatter. Unbundled embedders that call the placeholder builtin receive + `kernel.unavailable`. +6. **Removed code.** `program_uses_print`, `stmts_use_print`, + `expr_uses_print`, `Program.print_override`, the runtime redirect branch and + the whole-AST print scanner are deleted. A structural test + (`architecture_boundaries.rs::callable_binding_is_the_only_static_resolution_path`) + fails if they reappear. + +## Consequences + +* `打印` and `print` are now ordinary calls in Analysis/Trace: the call target + is the stdlib function, `state` is `function`, and no synthetic `builtin` + target is reported for the alias. +* Output must stay byte-identical. `crates/cli/tests/stdlib_alias_routing.rs` + plus the existing print/format/Trace goldens pin this; a diff is a blocking + regression, not an acceptable "new design". +* A tampered alias table (metadata digest or expected signature) is rejected + before module loading; `stdlib.json` and the `.mq.md` tree remain separately + digest-pinned, and `tools/pack-stdlib.py` refuses to build an inconsistent + bundle. +* The generic metadata path is the only place a future official high-level name + needs to be declared locally. Any *new language syntax* still requires a + language RFC; aliases cannot change `@exec`/`@knowledge` semantics. + +## Rejected alternatives + +* **Keep `print_override` and add a second alias for another function.** + Rejected: it multiplies hard-coded scans/branches and violates RFC-EXT-0001 + §8 and §14.10. +* **Move the print formatter into the Host as a `Print` op.** Rejected: it + would reintroduce an un-audited output path and duplicate the stdlib + formatter; output is a resource write. +* **Let third-party packages register `print`/`打印` aliases.** Rejected: + official public names are a language surface, not an extension registry. +* **Put the alias table in Rust `match` arms instead of metadata.** Rejected: + the table must be versioned with the verified bundle so a stale/mixed bundle + cannot run a different function. diff --git a/docs/architecture/CURRENT.md b/docs/architecture/CURRENT.md index 7af5b83..2b48482 100644 --- a/docs/architecture/CURRENT.md +++ b/docs/architecture/CURRENT.md @@ -15,6 +15,15 @@ replace `spec/` or a recorded test log. > bundled `stdlib/` implement it. The workspace gate is green at `9c1723d`; > rollout evidence and historical banners are indexed in > [`../migration/README.md`](../migration/README.md). +> +> **Extensibility closure (RFC-EXT-0001).** The current tree implements the +> single static callable binding (ADR-0054), official stdlib public aliases +> replacing print magic (ADR-0058), third-party resource providers through the +> normal CLI (ADR-0055), and the capability-handler registry (ADR-0057). The +> extension selection/lock v4 composition and the frozen external-binary +> evidence are tracked as remaining PR-04/PR-08 work, not claimed here. The +> normative resolution contract is +> [`spec/callable-resolution.md`](../../spec/callable-resolution.md). ## 1. Four state dimensions @@ -25,7 +34,7 @@ platform". | Dimension | Current state | Source | |---|---|---| | Implemented | Rust workspace `0.1.0`: RFC-LANG-0002 Markdown/frontend/IR, v2 statement parser, tree-walk runtime, `Memory`, capability host, Tool/Analysis/Trace, Plugin ABI, Foreign kernels, migrated bundled stdlib, CLI composition root | `Cargo.toml`, `crates/`, `stdlib/` | -| Tested locally | `cargo test --workspace --locked`: **507 passed, 0 failed, 2 ignored** on Linux x86_64 at the baseline; fmt and clippy clean | `docs/STATUS.md` §0, `docs/evidence/2026-11-21-semantics-v2-green.md` | +| Tested locally | `cargo test --workspace --locked`: **584 passed, 0 failed** across 66 targets on Linux x86_64 in the RFC-EXT-0001 tree; fmt and clippy clean. Historical baseline: 507 passed at `9c1723d`. | `docs/STATUS.md`, `docs/evidence/2026-11-22-ext-implementation.md` | | Bundled stdlib released | In-repo bundle `marqdo-stdlib@0.1.0`, SHA-256 digest verified before loading; it is **not** an external registry package | `stdlib/stdlib.json`, `stdlib/README.md` | | Multi-platform verified | Linux local gate is the only executed lane here; Windows/macOS native lanes are **UNVERIFIED**, and the GitHub security lane remains a **VM-only** gate | `docs/STATUS.md` §1, §4 | @@ -53,6 +62,12 @@ OS / external services / devices CLI is only the composition root: parse arguments, load the project, assemble Host + Runtime, route output, exit with a status. +`project_prepare::prepare` is the single finalisation step: it loads sources, +reads static extension descriptors (never `dlopen`/spawn), builds the unique +`CallableRegistry`, binds every call site once and hands the same +`PreparedProgram` to `check`/`run`/`call`/`inspect`/Analysis. The runtime +dispatches only by the resulting `CallBindingTable`; it never re-resolves a +name or guesses a plugin. Callable Plugin / Foreign kernel / Tool Protocol are separate extension categories; they do not enter the standard library or the language IR. ``` @@ -112,7 +127,10 @@ review RFC, not a new crate per provider. | network endpoints | `tcp`/`udp`/`tcp-listener`/`tls` open + `resource.accept`; exact host grant, DNS after grant, TLS certificate validation | implemented (project mode) | `resource_streams` | | `std:net/http` | byte-exact HTTP/1.1 parse/build; chunk sizes are byte counts; text wrapper compatible | implemented (pure `.mq.md`) | `stdlib_http` | | HTTP/1.1 | `stdlib/net/http.mq.md`: status/header/chunked-byte parse, request build | implemented over an already-authorized TCP stream; not a DNS/TLS client | `stdlib_http`, `http_over_tcp`, `stdlib_product` | -| print | bundled `io.mq.md`/`console.mq.md` through `resource.write`; legacy `HostOp::Print` removed | implemented (Compat-C) | `print_routing`, `stdlib_product` | +| print / official stdlib aliases | `打印`/`print` are verified `stdlib.json` public aliases bound to `io.mq.md::写值`; ordinary Marqdo calls, no `print_override`, no runtime branch, no `HostOp::Print` | implemented (ADR-0058) | `stdlib_alias_routing`, `print_routing`, `stdlib_product` | +| callable resolution | one registry + one `bind_program_calls` pass; `check`/`run`/`inspect`/Analysis share `CallSiteKey` bindings; plugin exports reconcile exactly with static manifests; no runtime name fallback | implemented (ADR-0054) | `callable_resolution`, `plugin_worker`, `analysis_trace` | +| resource provider v2 CLI | `--provider`/`--allow-provider` select and authorize a `provider.json` schema-2 `c-abi-1` worker artifact; manifest/artifact/grants/duplicate kinds verified before spawn; sibling helper only | implemented linux/x86_64; other platforms fail closed (ADR-0055) | `provider_cli` | +| capability handlers | trusted handler registry executes granted capabilities under budget/audit; missing handler is `capability.unavailable`; `fs.read` is the first real handler | implemented (ADR-0057) | `capability_handlers` | | strict mode | only the pure/Math execution path and the built-in stdout console channel; project/foreign/plugin/tool/trace/file/network grants are rejected | implemented L1; L2 not implemented | `tests/security`, `resource_streams`, `stdlib_product` | ## 4. Extension path classification @@ -120,9 +138,10 @@ review RFC, not a new crate per provider. | Extension path | Responsibility | State | |---|---|---| | Tool Protocol | external `marqdo` CLI tools, inspect/verify/IDE commands; versioned manifest and install/remove | implemented v1; not a standard-I/O replacement inside a user program | -| Callable Plugin ABI | native callable functions and controlled experimental acceleration | implemented v1 (native + worker); not an ordinary JSON/CSV formatter | -| Resource Provider | new resource kinds/devices/network/file endpoints | implemented v1 in-process, worker and trusted C ABI; not one provider per high-level algorithm | -| Foreign kernel | explicit cross-language compatibility for Python/Node one-shot execution | implemented; not a Marqdo implementation of common algorithms | +| Callable Plugin ABI | native callable functions and controlled experimental acceleration | implemented v1 (native + worker); static manifest is the contract and run reconciles exports exactly; not an ordinary JSON/CSV formatter | +| Resource Provider | new resource kinds/devices/network/file endpoints | v2 CLI entry implemented for trusted worker transport on linux/x86_64 (ADR-0055); v1 in-process/trusted C ABI remain embedder APIs; not one provider per high-level algorithm | +| Foreign kernel | explicit cross-language compatibility for Python/Node one-shot execution | implemented; registry accepts an explicitly declared unknown language; not a Marqdo implementation of common algorithms | +| Official stdlib public alias | language-surface aliases such as `打印`/`print` declared in verified bundle metadata | implemented (ADR-0058); third parties cannot claim these names | | Shared registry/policy/budget/audit/trace | package/adapter identity, descriptor/version checks, per-request grants, timeouts, error classes and trace metadata | housed inside `marqdo-host`; do not invent a shared binary ABI for function calls and resource streams | ## 5. Gap and closure backlog @@ -146,6 +165,9 @@ that a phase is complete. | P3 | No concurrency tasks, atomics or cancellable long-running scheduling | open; separate ADR | | P3 | No native optimizer/JIT or columnar numeric backend | open; benchmark-first | | P3 | Windows Native/Worker ABI is unsupported; macOS/Windows CI is unverified | open | +| P0 | RFC-EXT-0001 PR-04: `marqdo.yaml.extensions` + lock v4 extension selection, CLI composition-root wiring, `inspect --extensions` | **open / in progress** | +| P0 | RFC-EXT-0001 PR-08: frozen release binary, external conformance rerun, SHA-bound evidence manifest, final docs/status convergence | **open / in progress** | +| P1 | Provider worker transport only linux/x86_64; macOS/Windows provider lanes fail closed | open; separate platform work | ## 6. Documentation governance @@ -184,3 +206,9 @@ that a phase is complete. - [ADR-0052: byte-exact HTTP framing](../adr/0051-byte-exact-http-framing.md) — accepted byte-list HTTP/1.1 parser/builder. - [Compute-machine RFC-0001](../MARQDO_COMPUTE_MACHINE_RFC_0001.md) and [core/stdlib RFC-0002](../MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md) — design input. - [Architecture audit and refactor RFC](Marqdo_CLI_Latest_Architecture_Audit_and_Refactor_RFC.md) — review input for this migration. +- [RFC-EXT-0001](RFC-EXT-0001.md) — extensibility closure task specification (implementation status tracked here and in STATUS). +- [ADR-0054: single callable binding](../adr/0054-callable-binding-single-source.md) — one registry/binding pass, runtime dispatch by identity. +- [ADR-0055: resource provider CLI v2](../adr/0055-resource-provider-cli-v2.md) — provider v2 manifest/worker CLI closure and trust boundary. +- [ADR-0057: capability handler registration](../adr/0057-capability-handler-registration.md) — declared capability vs installed handler vs audit. +- [ADR-0058: stdlib public aliases](../adr/0058-stdlib-public-alias.md) — verified metadata replaces print magic. +- [`spec/callable-resolution.md`](../../spec/callable-resolution.md) and [`spec/resource-provider-v2.md`](../../spec/resource-provider-v2.md) — normative extension boundaries. diff --git a/docs/ci.md b/docs/ci.md index b06ee4e..65be162 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -72,3 +72,45 @@ cargo build --release --locked -p marqdo-cli The ignored `.secrets/dev.env` may contain a Woodpecker API token for local queries/triggers. The token is not needed by the pipeline itself and must never be committed. + +## External extension evidence (RFC-EXT-0001 PR-07/PR-08) + +A release candidate must also prove that a **frozen** CLI binary can use +extensions from physically separate repositories without modifying or +rebuilding this workspace. This is a new necessary condition; it does **not** +replace the security/platform/performance gates above. + +Required inputs: + +* a release build of this commit: `cargo build --release --locked -p marqdo-cli`; +* the provider helper shipped next to it for the same target: + `marqdo-provider-worker` (the provider transport resolves only that sibling; + no `PATH` fallback); +* a clean checkout of the external fixture repository + `https://git.cflmy.cn/cflmy/marqdo-extension-conformance`. + +Generation and validation: + +```bash +CLI="$(realpath target/release/marqdo)" +python3 tools/ci/verify_extension_contracts.py \ + --cli "$CLI" \ + --fixtures /path/to/marqdo-extension-conformance +python3 tools/ci/verify_extension_contracts.py \ + --validate docs/evidence/extension/<40-hex-source-SHA>/manifest.json +``` + +The generator hashes the CLI before and after the external suite, records the +source SHA and fixture repo SHA, and refuses to write evidence for a changed +binary, a dirty fixture checkout, a `FAIL`, or an unexplained +`UNSUPPORTED`/`SKIPPED` result. The external suite itself +(`tests/run_all.py` in that repository) uses only `$MARQDO_BIN` (absolute), +never Cargo, and covers: pinned-Git `.mq.md` package, worker ABI plugin, +unknown foreign kernel, provider v2 worker, and an independent Analysis/Trace +consumer. + +The schema is [`evidence/extension/schema.json`](evidence/extension/schema.json); +the generator is `tools/ci/verify_extension_contracts.py`. A manifest whose +`tested_sha` is not the release SHA is invalid. Platform-specific results must +be reported as `UNSUPPORTED` (for example provider worker on non-linux/x86_64); +a Linux PASS never substitutes for a macOS/Windows lane. diff --git a/docs/evidence/2026-11-22-ext-implementation.md b/docs/evidence/2026-11-22-ext-implementation.md new file mode 100644 index 0000000..9ee66f2 --- /dev/null +++ b/docs/evidence/2026-11-22-ext-implementation.md @@ -0,0 +1,195 @@ +# RFC-EXT-0001 implementation evidence (P0 core + PR-03/05/06/07 first pass) + +Status: **partial, honest record.** PR-00/01/02/03/05/06 are implemented and +tested on Linux x86_64 in this tree; PR-04 (extension selection/lock v4) and the +final frozen-release evidence are still in progress. This document is not a +release claim. The authoritative executed status is regenerated in +`docs/STATUS.md` and `docs/evidence/extension//manifest.json`. + +## 1. Identity and environment + +| Field | Value | +|---|---| +| Trees | `/root/work/marqdo-cli` (branch `feat/extensible-ecosystem`) and separate `/root/work/marqdo-extension-conformance` | +| Baseline | `595279fe38048b52b96d9ff1e7d57a867ec5c4c6`; green baseline in `docs/evidence/ext-baseline-595279f.md` | +| Rust | `rustc 1.100.0`, `cargo 1.100.0` | +| OS / arch | Linux x86_64 | +| Commands | logged under `.cache/ext/` (gitignored) | + +No commit SHA is claimed for the implementation yet: work is uncommitted on the +feature branch, so this file cannot be a same-SHA release artifact. + +## 2. Executed workspace gate (current tree) + +```text +cargo fmt --all -- --check PASS (exit 0) +cargo clippy --workspace --all-targets -- -D warnings PASS (exit 0) +cargo test --workspace --locked PASS (exit 0) + 66 test targets: 584 passed, 0 failed, 0 ignored +``` + +Targeted new suites: + +```text +cargo test -p marqdo-cli --test callable_resolution 14 passed +cargo test -p marqdo-cli --test stdlib_alias_routing 5 passed +cargo test -p marqdo-cli --test provider_cli 12 passed +cargo test -p marqdo-cli --test analysis_trace 7 passed +cargo test -p marqdo-cli --test plugin_worker 4 passed +cargo test -p marqdo-cli --test print_routing 2 passed +cargo test -p marqdo-cli --test architecture_boundaries 9 passed +cargo test -p marqdo-host (includes capability_handlers 12) +``` + +## 3. What was implemented (source paths) + +### PR-01/02 — one static binding, bound runtime dispatch + +* `crates/marqdo-ir/src/call_binding.rs` — `CallSiteKey`, `BoundTarget` + (`Builtin`/`Marqdo`/`AbiPlugin`/`DynamicValue`), deterministic + `CallBindingTable`, unit tests. +* `crates/marqdo-ir/src/callable.rs` — `try_from_program` (rejects duplicate + exports instead of first-wins), `OfficialAlias` registration, dotted-prefix + conflict helpers, `function_id`. +* `crates/marqdo-frontend/src/calls.rs` — the single `bind_program_calls` pass, + RFC priority matrix, import namespace conflicts, exact plugin signature + validation; `validate_module` is a standalone-frontend wrapper. +* `crates/marqdo-frontend/src/analyze.rs` — `analyze_source_for_program` defers + call checking to the composition root; standalone `analyze_source` keeps its + single-document checks for the conformance corpora. +* `crates/marqdo-runtime/src/interpreter.rs` — dispatch by `CallSiteKey`; the + unknown-plugin fallback, `print_override` branch and dead `resolve_function` + are removed; bound ABI calls normalise positional arguments to manifest order. +* `crates/cli/src/extension_descriptors.rs` — static plugin manifest reading + (no `dlopen`/spawn), signature parsing (defaults rejected), bundle-relative + path checks, artifact sha helpers. +* `crates/cli/src/project_prepare.rs` — one `prepare()`: load → descriptors → + registry → bind once → `PreparedProgram`; diagnostics are appended once. +* `crates/cli/src/commands/{check,run,inspect}.rs` + `loader.rs` — all consume + `prepare_project`; `run` verifies artifacts before load and reconciles the + Host exports one-to-one (`ext.plugin_binding_mismatch` on any difference). +* `crates/cli/src/analysis.rs` — Analysis `calls[].state/target` are projected + from the same binding table (no second resolver). +* `crates/cli/tests/callable_resolution.rs` — CALL-01/02/03/04/05/06/07/08/09/ + 11/12/13/14/15/16 as black-box CLI tests (CALL-10 has unit coverage in + `commands/run.rs` for missing/extra runtime exports). + +Normative docs: `spec/callable-resolution.md`, `docs/adr/0054-*`, +`spec/plugin-abi.md` §12, `spec/analysis-v2.md` calls note. + +### PR-03 — resource provider v2 through the normal CLI + +* `crates/cli/src/main.rs` — repeatable `--provider PATH` and + `--allow-provider NAME` on `run`/`call` only. +* `crates/cli/src/commands/sandbox.rs` — grants map to + `allow_resource_provider`; `pure`/`strict` preflight refuse the flags; + wildcards rejected. +* `crates/cli/src/provider_manifest.rs` — strict schema v2 parser/verifier, + target selection, bundle containment, sha256, reserved-kind rejection, + duplicate kind/provider checks, sibling `marqdo-provider-worker` resolution + (no `PATH` fallback; `ext.worker_unavailable`/`ext.platform_unsupported`). +* `crates/cli/src/commands/run.rs` — verify all manifests/artifacts/grants and + duplicate kinds before spawning; spawn only granted+verified providers; no + native fallback; no new `HostOp`/`resource_request` branch. +* Tests: `crates/cli/tests/provider_cli.rs` (real C sensor + real worker + helper), 12 tests; `spec/resource-provider-v2.md`, ADR-0055, README section. + +Provider evidence recorded by the PR-03 implementer: + +| ID | Result | Evidence summary | +|---|---|---| +| PROV-01 | PASS | exit 0, `count=5`, `eof=true`, one worker/open/close marker | +| PROV-02 | PASS | exit 1 `permission.denied`; helper never started | +| PROV-03 | PASS | exit 2 `ext.provider_grant_unmatched`; no worker | +| PROV-04 | PASS | exit 2 `ext.duplicate_provider_kind`; no worker | +| PROV-05 | PASS | one provider, two kinds, counts `[5,2]`, both closed | +| PROV-06 | PASS | `stdio/file/tcp/tls/udp/tcp-listener` rejected `ext.manifest_invalid` | +| PROV-07 | PASS | sha mismatch / symlink escape / `..` → `ext.artifact_integrity` before spawn | +| PROV-08 | PASS | `ext.no_matching_artifact` | +| PROV-09 | PASS | v1 → `ext.unsupported_provider_schema`, no migration | +| PROV-10 | PASS | pure `cli.invalid_grants`, strict `strict.unsupported_grant`, `*` rejected | +| PROV-13 | PASS | sibling helper missing → `ext.worker_unavailable`, PATH helper unused | + +Platform: provider worker transport is claimed **linux/x86_64 only**; macOS and +Windows are explicitly `UNSUPPORTED`/fail-closed (`ext.platform_unsupported`), +not claimed supported. PROV-11/12/14/15 belong to host-protocol and PR-07 +external closure and are not claimed by the PR-03 CLI matrix. + +### PR-05 — capability handler registry + +`crates/marqdo-host/src/capability.rs` now has a trusted handler directory, +`fs.read` as a real handler, decide → handler-installed → execute under budget → +audit with duration; missing handler stays `capability.unavailable`, duplicate/ +reserved/empty registrations are rejected. `crates/marqdo-host/tests/ +capability_handlers.rs` has 12 CAP tests (CAP-01…CAP-12); ADR-0057 and +`spec/security.md` updated. `cargo test -p marqdo-host` PASS. + +### PR-06 — official stdlib public aliases + +`stdlib/stdlib.json` declares `public_aliases` (`print`/`打印` → +`io.mq.md::写值`) and `aliases_digest`; `crates/cli/src/stdlib.rs` verifies the +metadata digest and expected target signature; `loader.rs` loads the target +module only when the alias name is referenced and produces `OfficialAlias` +records; `project_prepare` registers them and the binding pass targets the +ordinary `Marqdo` function. `program_uses_print`, `Program.print_override`, the +runtime print branch and the AST scanner are deleted; the placeholder builtin +still fails with `kernel.unavailable` for unbundled embedders. Tests: +`crates/cli/tests/stdlib_alias_routing.rs` (5) + existing print/format/Trace +goldens; the stdlib fixture aliases that collided with builtin namespaces +(`table`/`text`/`fs`) were renamed and the bundle digest regenerated. +`tools/pack-stdlib.py` verifies `aliases_digest` before packaging. +ADR-0058 documents the decision. + +### PR-07 (first pass) — independent external repository + +`/root/work/marqdo-extension-conformance` is a separate Git repository (own +history, remote `https://git.cflmy.cn/cflmy/marqdo-extension-conformance`, +pushed; commit `b729afc` at the time of this record). It contains: + +* `pure-lib/` + `consumer/` with a pinned full-SHA Git dependency; +* `plugin/` — real C ABI v1 worker plugin, three exports under two prefixes; +* `kernel/` — `mocklang` provider with a language the CLI does not know; +* `provider/` — `provider.json` v2 + real C ABI sensor library; +* `tool/` — stdlib-only Python consumer of Analysis v2 / Trace v2; +* `tests/run_all.py`, `tests/verify_frozen_cli.py`, `CONTRACT.md`, + `.github/workflows/conformance.yml`. + +Provisional run against the current development binary (debug build, not a +release candidate): + +```text +[PASS] plugin_worker, callable_registry, foreign_kernel, + resource_provider_cli, package_pinned_git, + tool_analysis_consumer, negative_security + (all categories PASS; frozen CLI sha256 recorded in evidence/results.json) +``` + +This is **not** final PR-07/PR-08 evidence: the binary was not a `--release` +candidate, and `tools/ci/verify_extension_contracts.py` has not yet emitted a +`docs/evidence/extension//manifest.json`. The external suite will be rerun +against the frozen release binary after PR-04 lands. + +## 4. Known gaps / deviations (not hidden) + +1. **PR-04 not landed yet.** `marqdo.yaml.extensions` and lock v4 are still in + progress; the CLI composition root for extension selection is not wired. +2. **Provider platform claims are Linux x86_64 only**; macOS/Windows are + explicitly unsupported for the worker transport. The RFC's mac/Windows + provider lanes are not claimed. +3. **Additive provider error codes** `ext.manifest_invalid`, + `ext.duplicate_provider`, `ext.provider_grant_unmatched` were added because + the RFC's frozen code list has no precise code for malformed v2, duplicate + provider names, or an unmatched `--allow-provider`; documented in + `spec/resource-provider-v2.md` and ADR-0055. +4. **`--allow-provider` without a selected manifest** aborts with + `permission.denied` before spawning rather than silently ignoring the grant + (needed to satisfy PROV-02 with the existing spawn-on-register Host API). +5. **Analysis v2 `calls[].state/target` semantics changed for official aliases** + (print now projects as a `function` target). This is a projection of + existing fields, documented in `spec/analysis-v2.md`/ADR-0054/ADR-0058; + there is no v3 bump and the private binding table is not exposed. +6. **CALL-10's host-side mismatches** are unit-tested for missing/extra exports; + a live TOCTOU signature/digest mismatch is covered by artifact-tamper tests + plus exact reconciliation, not by swapping a manifest mid-run. +7. Full `--release` build, frozen external rerun, SHA-bound evidence manifest, + docs/STATUS/CURRENT/COMPATIBILITY/CHANGELOG finalisation remain for PR-08. diff --git a/docs/evidence/ext-baseline-595279f.md b/docs/evidence/ext-baseline-595279f.md new file mode 100644 index 0000000..a55a43d --- /dev/null +++ b/docs/evidence/ext-baseline-595279f.md @@ -0,0 +1,82 @@ +# RFC-EXT-0001 baseline audit — `595279f` + +Status: **baseline evidence recorded before implementation.** This is not a +release claim and none of the new RFC-EXT-0001 behaviour existed at this SHA. + +## 1. Identity and environment + +| Field | Value | +|---|---| +| Repository | `cflmy/marqdo-cli` (private remote `https://git.cflmy.cn/cflmy/marqdo-cli`) | +| Baseline commit | `595279fe38048b52b96d9ff1e7d57a867ec5c4c6` (`master`, 2026-11-21) | +| Working branch | `feat/extensible-ecosystem` | +| Rust | `rustc 1.100.0 (0ef1bc386 2026-11-09)` | +| Cargo | `cargo 1.100.0 (3acc79c0c 2026-11-05)` | +| OS / arch | Linux `x86_64` (Ubuntu container) | +| Local homes | `RUSTUP_HOME=.rustup-home`, `CARGO_HOME=.cargo-home` | +| Command log | `.cache/ext/baseline.log` (local, gitignored) | + +## 2. Executed gate (same SHA, before any RFC-EXT-0001 edit) + +```text +cargo fmt --all -- --check exit 0 PASS +cargo clippy --workspace --all-targets -- -D warnings exit 0 PASS +cargo test --workspace --locked exit 0 PASS + 62 test targets: 521 passed, 0 failed, 2 ignored +``` + +The RFC §2.2 static inventory commands were also run and their findings are +recorded in §3. A green baseline is only the starting point, not evidence for +any new feature. + +## 3. Source facts at the baseline (RFC §2.1 refresh) + +Each row was checked against the actual tree, not against the RFC prose. None of +the rows was `already_closed` at `595279f`. + +| ID | Observed at `595279f` | Impact | RFC treatment | +|---|---|---|---| +| EXT-01 | `marqdo-frontend/src/units.rs` `SemanticMode` is `Exec`/`Knowledge` only; unknown `@xxx` is `sem.invalid_mode_marker` | third parties cannot add an execution mode (intended closure) | keep; metadata is inert Markdown | +| EXT-02 | `frontend/src/calls.rs` checks Member builtin before import; `runtime/src/interpreter.rs` checks import before builtin | check and run can bind different targets | PR-01/02 single binding | +| EXT-03 | `ir/src/callable.rs` has `CallableRegistry` but calls still use `builtins::find` and runtime fallbacks | registry was not the single truth | PR-01/02 | +| EXT-04 | interpreter tolerated undeclared `alias.name` as a plugin call when `external_callables` was empty | static-unknown symbol failed only at run time | PR-02 remove fallback | +| EXT-05 | `cli/src/commands/run.rs` synthesised `params: Vec::new()` descriptors for loaded exports | manifest/runtime signature could differ silently | PR-02 exact reconciliation | +| EXT-06 | `loader.rs` validated the program before the CLI appended static plugin descriptors, then validated again | duplicated diagnostics and ordering hazards | PR-01 single `prepare` finalize | +| EXT-07 | `host/src/host/providers.rs` has Rust/C/Worker provider registration; CLI had no `--provider` | providers were only reachable by trusted embedders | PR-03 | +| EXT-08 | `host/src/capability.rs::execute` implemented only `fs.read`; other names returned unavailable | declaring a capability did not mean executing it | PR-05 handler registry | +| EXT-09 | `host/src/host.rs::resource_request` has per-string built-in operations | adding a non-streaming operation required core edits | PR-03/05; keep core operations | +| EXT-10 | `cli/src/loader.rs::program_uses_print` injected `stdlib/io.mq.md` and set `Program.print_override`; runtime special-cased `打印/print` | migrating another stdlib function by copying this pattern would hard-code every name | PR-06 public aliases | +| EXT-11 | `foreign/src/lib.rs::KernelRegistry` supports unknown languages; old `normalize_language` only knows Python/Node | registry is the execution path, legacy helper is display-only | keep registry as the only query | +| EXT-12 | Plugin ABI v1 Native/Worker implemented; Wasm unsupported; Tool Protocol tested | do not rebuild plugin/tool platform | keep + add identity reconciliation | +| EXT-13 | `package/src/lib.rs` supports path + pinned `git` + package instances; `stdlib/` is a verified bundle | real `.mq.md` source decoupling exists | keep + external black-box test | +| EXT-14 | strict L1 permits only limited pure computation and stdout and rejects Worker/Foreign/Provider | project mode is not an OS sandbox | keep fail-closed; strict L2 is a separate RFC | + +## 4. Issues reproduced while recording this baseline + +They were reproduced while writing the RFC acceptance fixtures; all are fixed by +the PR series recorded later in `docs/STATUS.md` and +`docs/evidence/extension//manifest.json`: + +1. **Duplicate resolution (EXT-02/03/04).** A fixture with an import alias that + matched a builtin dotted namespace (`import: {math: ./math.mq.md}` plus + `math.eval(...)`) had no deterministic binding rule; the frontend checked the + builtin while the runtime checked the import first. +2. **Runtime plugin guessing (EXT-04).** `solver.add(...)` without `--plugin` + reached the runtime and attempted `PluginInvoke` instead of failing during + `check` with `symbol.unknown`. +3. **Synthesised plugin signatures (EXT-05).** `run` could execute a plugin + export whose static descriptor had an empty parameter list, so an arity or + position/name mismatch was only discovered (or not) at the ABI boundary. +4. **Print magic (EXT-10).** `打印`/`print` were routed by a whole-AST + `program_uses_print` scan plus a runtime name special case; there was no + generic de-magicking path for a second official stdlib function. +5. **Provider CLI gap (EXT-07).** There was no `--provider`/`--allow-provider`; + an external `provider.json` could not be used through the shipped binary. + +## 5. Not claimed by this baseline + +* No `--provider`/`--allow-provider` existed; strict L2 remains unimplemented. +* No frozen-binary external repository evidence existed. +* No `docs/evidence/extension//manifest.json` existed. +* G3 remote security and G4 performance status is unchanged by this document; + see `docs/STATUS.md` and `docs/releasing.md`. diff --git a/docs/evidence/extension/README.md b/docs/evidence/extension/README.md new file mode 100644 index 0000000..2ab38e3 --- /dev/null +++ b/docs/evidence/extension/README.md @@ -0,0 +1,32 @@ +# External-extension evidence + +RFC-EXT-0001 §10.2/§11.3 requires SHA-bound proof that an **unchanged** CLI +binary can use extensions from physically separate Git repositories. This +directory holds that evidence. + +* `schema.json` — JSON Schema for the evidence manifest. +* `<40-hex-source-SHA>/manifest.json` — generated by + `tools/ci/verify_extension_contracts.py`; never hand-written. + +Generation (from a clean fixture checkout and a release build of this repo): + +```bash +cargo build --release --locked -p marqdo-cli +CLI="$(realpath target/release/marqdo)" +python3 tools/ci/verify_extension_contracts.py \ + --cli "$CLI" \ + --fixtures /path/to/marqdo-extension-conformance +python3 tools/ci/verify_extension_contracts.py \ + --validate docs/evidence/extension//manifest.json +``` + +The generator refuses to write evidence when: + +* the CLI binary hash changes between the start and end of the suite; +* the fixture repository has uncommitted changes; +* a required result is missing or `FAIL`; +* `UNSUPPORTED`/`SKIPPED` is recorded without `--allow-unverified` and a report + entry explaining the platform limitation. + +A local or container run is **not** a substitute for the release gates in +`docs/releasing.md` (G3/G4), and no tag is created by this evidence. diff --git a/docs/evidence/extension/schema.json b/docs/evidence/extension/schema.json new file mode 100644 index 0000000..5509eb8 --- /dev/null +++ b/docs/evidence/extension/schema.json @@ -0,0 +1,63 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://marqdo.dev/schemas/extension-evidence-v1.json", + "title": "Marqdo external-extension evidence manifest", + "description": "SHA-bound evidence that independent repositories can use the frozen CLI without modifying or recompiling marqdo-cli (RFC-EXT-0001 §11.3). Generated by tools/ci/verify_extension_contracts.py; never hand-edited.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "schema_version", + "tested_sha", + "cli_binary_sha256", + "fixture_repo_sha", + "os", + "arch", + "results", + "commands", + "failures", + "unverified" + ], + "properties": { + "schema": { "const": "marqdo.extension-evidence" }, + "schema_version": { "const": 1 }, + "tested_sha": { + "type": "string", + "pattern": "^[0-9a-f]{40}$", + "description": "Exact marqdo-cli source commit the frozen binary was built from." + }, + "cli_binary_sha256": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "fixture_repo_sha": { + "type": "string", + "pattern": "^[0-9a-f]{40}$" + }, + "os": { "type": "string", "enum": ["linux", "macos", "windows"] }, + "arch": { "type": "string" }, + "results": { + "type": "object", + "additionalProperties": { + "type": "string", + "enum": ["PASS", "FAIL", "UNSUPPORTED", "SKIPPED"] + }, + "required": [ + "callable_registry", + "plugin_worker", + "foreign_kernel", + "resource_provider_cli", + "package_pinned_git", + "tool_analysis_consumer", + "negative_security" + ] + }, + "commands": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "failures": { "type": "array", "items": { "type": "string" } }, + "unverified": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/docs/releasing.md b/docs/releasing.md index 5266e5d..c432ba3 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -129,3 +129,33 @@ self-hosted G-C URL. See delete the failing test. - If the security gate regresses: set strict back to `Unavailable`, publish a security note, and do not claim isolation until the lane is green again. + +## 7. Extension-ecosystem gate (RFC-EXT-0001) + +Before a tag, in addition to §3 and §4: + +1. `cargo build --release --locked -p marqdo-cli` and build + `-p marqdo-provider-worker`; ship both binaries in the release archive for + the same target. A provider request without the sibling helper must fail + with `ext.worker_unavailable` — never fall back to `PATH`. +2. Run the external frozen-binary suite from + `https://git.cflmy.cn/cflmy/marqdo-extension-conformance` against the exact + release binaries and generate + `docs/evidence/extension/<40-hex-source-SHA>/manifest.json` with + `tools/ci/verify_extension_contracts.py`. The manifest must validate against + `docs/evidence/extension/schema.json`, its `tested_sha` must equal the + tagged commit, and every mandatory result must be `PASS` on the claimed + platform. `UNSUPPORTED` is allowed only for a platform explicitly excluded + from the support matrix in `COMPATIBILITY.md`. +3. `check`/`run`/`inspect`/Analysis share one prepared call binding + (`spec/callable-resolution.md`); `check` never loads an extension binary; + plugin exports reconcile exactly with static manifests; official stdlib + aliases are verified metadata with byte-identical print output. +4. Existing gates remain mandatory and unchanged: G3 strict isolation evidence, + G4 measured performance on the fixed node, native platform lanes, schema + consumers and `tools/ci/release_evidence.py`. The extension evidence is an + additional gate; it never substitutes for them, and local/container smoke is + not a release claim. +5. If any extension gate is missing or mismatched, the release decision is + **DO NOT RELEASE**. Tagging and asset publication still require explicit + owner approval. diff --git a/spec/analysis-v2.md b/spec/analysis-v2.md index 3ed6dd6..cbdd4a2 100644 --- a/spec/analysis-v2.md +++ b/spec/analysis-v2.md @@ -95,6 +95,13 @@ a separate statement. `imports`, `references` and `calls` remain static facts; a Markdown reference is not a call. `effects` remains `["unknown"]` (a conservative summary, not a purity proof). +`calls[]` is projected from the single prepared call binding +(`spec/callable-resolution.md` §6): `state` remains +`function`/`builtin`/`plugin`/`unresolved`, and official stdlib aliases such as +`打印` are reported as `function` targets because that is the implementation +that runs. The private `CallBindingTable` is not serialized; Analysis v2's +schema and version are unchanged. + An `assign` node keeps the v1 string `target` for a local rebinding. Addressable memory adds one additive target form (ADR-0022); consumers that only understand local assignments can ignore an object target: diff --git a/spec/callable-resolution.md b/spec/callable-resolution.md new file mode 100644 index 0000000..a287018 --- /dev/null +++ b/spec/callable-resolution.md @@ -0,0 +1,281 @@ +# Marqdo callable resolution and binding + +Status: **normative for the current implementation** (ADR-0054/ADR-0058, +RFC-EXT-0001 §4). This document describes the single static resolution used by +`check`, `run`, `call`, `inspect` and Analysis. There is no second priority +order and no runtime name guessing. + +## 1. Identity + +A call is an expression (`Expr::Call`). Each statically visible call site has a +deterministic key: + +```text +CallSiteKey = (module id, function id, span.start, span.end) +``` + +The key is independent of the checkout path, package cache location, +registration order and map iteration. For a signature-default expression the +owning function is the function that declares the default, because the runtime +evaluates defaults in that function's empty frame. + +A binding is one of: + +| Bound target | Meaning | +|---|---| +| `Builtin { canonical_name }` | fixed language builtin, addressed by its canonical name (never by a user alias) | +| `Marqdo { function }` | exact `FunctionId` of a Markdown-defined function, including an official bundled-stdlib public alias | +| `AbiPlugin { package, export, digest, param_names }` | exact plugin export; `digest` is the selected, verified artifact SHA-256 and `param_names` is the manifest order used to normalise positional arguments | +| `DynamicValue` | the callee is a language value (for example a `FunctionRef`); resolved at run time from the value, never by name | + +`CallBindingTable` is private IR. It is not serialized into Analysis or Trace. +The public `calls[].state/target` projection is derived from it (see §6). + +## 2. Resolution matrix + +### 2.1 Unqualified `f(...)` + +In this exact order; the first applicable step decides: + +1. **Local variable.** If `f` is a parameter or has been assigned anywhere in + the function body, the site binds `DynamicValue`. At run time the value must + be callable (`FunctionRef` in this release); a `Text`/`Map`/etc. value is + `call.not_callable`. This is a language variable rule, not an escape hatch to + a plugin. +2. **Official alias or reserved builtin.** The registry's builtin table contains + fixed builtins and, when the verified bundle is present and the program uses + them, official stdlib aliases. An official alias wins over the placeholder + builtin of the same name. Third parties cannot insert names here. +3. **Current namespace function.** If the enclosing level-1 namespace `N` + declares `N.f`, that function is chosen. +4. **Canonical function name.** A function whose canonical name is exactly `f`. +5. **Unique local name.** A function whose `local_name` is `f` in exactly one + namespace. If two namespaces define the same local name and none of steps + 1–4 applies, the call is an error (`symbol.unknown`, message reports the + ambiguity); it is never "first in document order". +6. Otherwise `symbol.unknown`. + +### 2.2 Qualified `alias.f(...)` + +1. **Local variable named `alias`.** If `alias` is a local value, the site binds + `DynamicValue`; `alias.f` is ordinary language member access. This cannot + become a plugin call. +2. **Explicit import.** If the module declares `import: { alias: }`, + the call binds only a function of that imported module (canonical name first, + then unique local name). A missing function is `symbol.unknown`; it never + falls back to a builtin or plugin with the same dotted spelling. +3. **Builtin dotted namespace.** Otherwise, an exact builtin + `alias.f` (for example `math.eval`) binds `Builtin`. +4. **Exact plugin export.** Otherwise, if a selected static plugin declares the + complete name `alias.f`, the site binds that export. The manifest signature + is validated here; no library is loaded. +5. Otherwise `symbol.unknown`. + +`alias` in step 1/2 must be exactly an explicit import or a local value; there +is no implicit namespace search across packages. + +### 2.3 Other callees + +`some_value.f(...)`, `f_list[i](...)`, `(expr)(...)` and any other non-Ident, +non-member-of-Ident callee bind `DynamicValue`. Only language callables can be +invoked at run time. + +### 2.4 Import namespace conflicts + +Before any call is bound, each module's import aliases are checked against: + +* builtin dotted namespaces (`alias.` must not prefix any builtin name), and +* visible plugin dotted namespaces (`alias.` must not prefix any selected + plugin export). + +A collision is `ext.import_namespace_conflict` at the front-matter import span, +with a rename hint. Example: importing `./math.mq.md` as `math` conflicts with +the builtin namespace `math.`; rename the alias to `mymath`. This is a +deliberate 0.x breaking change with no compatibility switch (ADR-0054). + +### 2.5 Duplicate callables + +The static table rejects: + +* the same complete plugin export declared by two selected packages + (`ext.duplicate_callable` at prepare); +* a plugin export that equals a builtin name or an official alias + (`ext.duplicate_callable`); +* two different manifests for the same plugin package identity in one selection + (`ext.duplicate_callable`); the exact same package identity is idempotent; +* a malformed/ambiguous static table entry (`ext.duplicate_callable`). + +Order of `--plugin` flags never decides which declaration wins. + +## 3. Argument checking + +### 3.1 Markdown functions and builtins + +`CallableSignature::validate_args` enforces: + +* named arguments must exist exactly once; unknown names are `call.arity`; +* positional arguments require an explicit language signature and map to the + declared parameter order; overflow is `call.arity`; +* required parameters without defaults must be supplied; missing is + `call.arity`; +* inferred signatures accept named arguments only (`call.arity` otherwise). + +Messages are deterministic and identical between `check` and `run`. + +### 3.2 ABI plugin exports + +* The static manifest signature is authoritative. Unknown, duplicate, missing + or excess arguments are rejected at prepare/bind time (`call.arity`); the + plugin is never spawned for a statically invalid call. +* ABI v1 transports arguments as a named map. Positional arguments are + statically validated and deterministically normalised to the manifest's + declared parameter order at run time. Example: `solver.add(1, b=2)` becomes + `{a:1,b:2}`. +* Manifest signature default values are rejected in ABI v1 until the wire + semantics are versioned; no default is injected at run time. +* A plugin that is declared but not selected (`--plugin` missing) is + `symbol.unknown`; a selected plugin without an exact export descriptor is + `symbol.unknown`. There is no runtime fallback. + +## 4. Run-time reconciliation + +`run`/`call` performs, in order: + +1. static prepare (sources, manifests, signatures, binding) — no binary loaded; +2. verify each selected artifact path, target and SHA-256; +3. verify the `--allow-plugin` grant; +4. load exactly the selected artifacts; +5. compare the Host's actual exports with the static descriptors one-to-one + (`name`, raw signature, artifact digest, owner package). + +A mismatch in step 5 is `ext.plugin_binding_mismatch`, reported before the +entry function runs. `run` must never synthesise a missing descriptor or an +empty signature. + +## 5. `check` versus `run` + +`check` and `run` consume the same `PreparedProgram` binding table. Therefore: + +* the same call site has the same `BoundTarget` in both; +* a statically invalid call fails `check` and `run` with the same diagnostic + code and message; `run` refuses before any Host side effect; +* `check --plugin` reads manifests and signatures only: it never `dlopen`s a + library, spawns a worker or executes a side-effect script; a missing plugin + binary is not an error for `check`; +* `--emit` Analysis is derived from the same binding (no second resolver). + +## 6. Analysis projection + +`calls[]` keeps the v2 fields `{function, span, callee, state, target}`. +`state` is one of `function`, `builtin`, `plugin`, `unresolved`; it is now +derived from the binding table: + +| Bound target | `state` | `target` | +|---|---|---| +| `Builtin` | `builtin` | canonical builtin name | +| `Marqdo` | `function` | exact `FunctionId` string | +| `AbiPlugin` | `plugin` | complete export name | +| `DynamicValue` / unbound | `unresolved` | `null` | + +Official stdlib aliases (for example `打印`) therefore project as `function` +targets, because that is the implementation that runs. This is a projection of +the existing fields, not a new public schema. The private `CallBindingTable` is +never exposed; a future explicit `resolved_call_target` field would require an +Analysis version bump. + +## 7. Diagnostics + +| Code | Stage | Trigger | +|---|---|---| +| `symbol.unknown` | prepare/bind | no target for the call; ambiguity; unknown imported function; undeclared plugin export | +| `call.arity` | prepare/bind | bad argument name/number/duplication, invalid positional use | +| `ext.import_namespace_conflict` | prepare | import alias collides with a builtin/plugin dotted namespace | +| `ext.duplicate_callable` | prepare | duplicate complete export, builtin shadowing, conflicting package manifests | +| `ext.plugin_binding_mismatch` | run init | loaded exports/signatures/owner/digest differ from static descriptors | +| `ext.no_matching_artifact` | manifest read | plugin/provider has no artifact for the current target | +| `ext.artifact_integrity` | manifest read / run init | sha256 mismatch, path traversal, non-regular file, symlink escape | +| `call.not_callable` | run | dynamic value is not callable | +| `internal.error` | run | a call site has no binding (a prepared program must always be complete) | + +Existing codes are reused rather than duplicated; `ext.capability_handler_unavailable` +is internal-only — the public wire code remains `capability.unavailable`. + +## 8. Examples + +### 8.1 Imported function, no conflict + +```markdown +--- +import: + mymath: "./math.mq.md" +--- + +## @exec main + +`return mymath.eval(a=1)` +``` + +Binds `Marqdo { function: ::math.mq.md::eval }` in both +`check` and `run`. + +### 8.2 Builtin without an import + +```markdown +## @exec main + +`return math.eval(ref("#公式"), {"r": 2}, target="A")` +``` + +Binds `Builtin { canonical_name: "math.eval" }`. + +### 8.3 Import namespace conflict + +```markdown +--- +import: + math: "./math.mq.md" +--- + +## @exec main + +`return math.eval(a=1)` +``` + +`check` fails with `ext.import_namespace_conflict`; `run` refuses before +executing anything. Rename the alias. + +### 8.4 Plugin export + +```markdown +## @exec main + +`return solver.add(1, b=2)` +``` + +* without `--plugin solver/plugin.yaml` or with a manifest that does not + declare `solver.add`: `symbol.unknown` (no Host fallback); +* with the manifest: binds `AbiPlugin { package: "demo.solver", export: + "solver.add", digest: "", param_names: ["a","b"] }`; +* with a changed artifact byte: `ext.artifact_integrity` before the entry + function; +* with a missing `b`: `call.arity` at prepare. + +### 8.5 Dynamic `FunctionRef` + +```markdown +## @exec 平方 + +`params(n: Int)` + +`return n * n` + +## @exec main + +`f = ref("#平方")` + +`return f(n=4)` +``` + +The call site binds `DynamicValue`; the runtime calls the `FunctionRef` value. +A `Text` value in the same position fails `call.not_callable` and never reaches +the Host. diff --git a/spec/plugin-abi.md b/spec/plugin-abi.md index ab8ba68..48ab825 100644 --- a/spec/plugin-abi.md +++ b/spec/plugin-abi.md @@ -459,3 +459,33 @@ when no C compiler is available. | Positional plugin arguments | Not supported; named only. | | Thread-safe / concurrent invocation | Not supported (single-threaded v1 contract). | | Old `marqdo_abi.h` v2 compatibility | Not implemented and not claimed; a separate adapter would be required. | + +## 12. Static descriptor / run-time reconciliation (RFC-EXT-0001 §4.5) + +The CLI never derives a callable signature from a loaded library. The +**manifest is the static contract** and the runtime is checked against it: + +1. `check --plugin ` reads and validates the manifest, including every + export name and signature; it does not load the artifact and does not need + the artifact file to exist. +2. `run`/`call` first verifies the selected artifact path, host target and + SHA-256, then checks the plugin grant, then loads exactly the selected + artifacts. +3. After loading, the Host's actual exports are compared one-to-one with the + static descriptors: complete name, raw signature string, owner package and + verified artifact digest. Extra, missing, renamed or differently-signed + exports fail with `ext.plugin_binding_mismatch` before the entry function + runs. +4. The binding pass validates call arguments against the manifest signature. + ABI v1 transports a named map, so positional arguments are statically + validated and deterministically normalised to manifest order + (`solver.add(1, b=2)` → `{a:1,b:2}`). Unknown, duplicate, missing or excess + arguments are `call.arity` before any process is started. +5. Manifest signatures with default values are rejected in ABI v1 until the + default-value wire semantics are versioned. The host never injects a default. +6. Duplicate complete export names across selected packages, or exports that + shadow a builtin/official name, are `ext.duplicate_callable` at prepare. + Selection order never decides which declaration wins. + +None of this changes ABI v1's C layout or wire format; it closes the static +side of the contract described by `spec/callable-resolution.md`. diff --git a/spec/resource-provider-v2.md b/spec/resource-provider-v2.md new file mode 100644 index 0000000..ddb9881 --- /dev/null +++ b/spec/resource-provider-v2.md @@ -0,0 +1,287 @@ +# Marqdo Resource Provider v2 (`provider.json`) + +Status: **frozen provider manifest schema version 2** for the `marqdo` CLI +composition root. Implementation: `crates/cli/src/provider_manifest.rs`; +prerequisite host transports: ADR-0033/0035/0036/0037/0038; CLI decision: +ADR-0055. The worker helper is `marqdo-provider-worker` +(`crates/marqdo-provider-worker`, ADR-0038). + +> **A trusted helper is not a sandbox.** The provider `worker` transport runs a +> separate process with the invoking user's OS privileges. It isolates a crash +> or protocol failure from the interpreter; it does **not** restrict file +> system, network, process or memory access. `sha256` values are content +> *integrity* checks, not publisher authentication. `strict` L1 refuses every +> provider (fail-closed) until a real isolation adapter exists. + +## 1. CLI surface + +```text +marqdo run FILE [--provider PATH/provider.json ...] [--allow-provider NAME ...] +marqdo call FILE FUNCTION [--arg NAME=VALUE ...] \ + [--provider PATH/provider.json ...] [--allow-provider NAME ...] +marqdo check FILE # no --provider/--allow-provider options exist +``` + +* `--provider PATH` is repeatable. `PATH` names a `provider.json` file + (directories are rejected). Only explicitly passed manifests are considered; + there is no directory scan, package-manager discovery, registry download or + network fetch. +* `--allow-provider NAME` is repeatable and matches the manifest `name` + **exactly**. There is no `*`/glob wildcard, prefix match or implicit + authorization. Selecting a provider with `--provider` never grants it. +* `check`/`inspect` do not accept these options and never read a provider + manifest, start a helper or call provider code. Provider kinds are not new + callables; they are values passed to `resource.open(kind, ...)`. +* `--mode pure` and `--mode strict` reject `--provider`/`--allow-provider` in + preflight (exit 2, no helper spawn, no silent downgrade to `project`). + `pure` reports `cli.invalid_grants`; `strict` reports + `strict.unsupported_grant`. + +### 1.1 Ordering + +`run`/`call` follow RFC-EXT-0001 §18.2: + +1. T1–T5: load the Markdown graph and prepare call bindings (unchanged; + providers are not part of `prepare`). +2. T6: parse all `--provider` manifests, validate schema/name/version/ABI/ + transport/kinds/artifact declarations, select and verify the current-target + artifact (regular file, inside the bundle, size bound, sha256), reject + duplicate provider names and duplicate kinds across providers, and check + the exact `--allow-provider` grants. +3. T6b: verify all selected plugin manifests/artifacts (existing behavior). +4. T7: start plugin workers and reconcile exports, then spawn one + `marqdo-provider-worker` per granted+verified provider. +5. T9: execute the entry function; every `resource.open` re-checks the provider + name grant in the Host. + +No worker process (plugin or provider) is started before every T6/T6b check +passes. A manifest, target, integrity, grant or duplicate-kind failure leaves +the main function unexecuted. + +An `--allow-provider NAME` that matches no selected manifest is a deterministic +option error (`ext.provider_grant_unmatched`, exit 2) rather than silent +discovery fan-out. + +## 2. Manifest file + +* File name convention: `provider.json`. It must be a UTF-8 JSON object of at + most 1 MiB and is read from an explicit path, never from `PATH`. +* The CLI accepts **schema version 2 only**. A `schema_version: 1` manifest is + rejected with `ext.unsupported_provider_schema`; there is no silent + migration and no v1/v2 auto-detection. (v1 remains usable only through the + host embedding API `StdHost::register_provider_package`, ADR-0037.) +* Unknown top-level fields are rejected inside schema v2. Extensible, + non-executable data must live in the `metadata` object, which has no + execution meaning. A future schema version must be a new document, not an + in-place extension of v2. + +```json +{ + "schema": "marqdo.provider", + "schema_version": 2, + "name": "demo.sensor", + "version": "0.1.0", + "abi": "c-abi-1", + "transport": "worker", + "kinds": ["demo.sensor"], + "artifacts": [ + { + "target": "x86_64-unknown-linux-gnu", + "path": "lib/libdemo_sensor.so", + "sha256": "<64-lowercase-hex-sha256>" + } + ], + "permissions_requested": [] +} +``` + +### 2.1 Fields + +| Field | Required | Rules | +|---|---|---| +| `schema` | yes | Exactly `"marqdo.provider"`. | +| `schema_version` | yes | Exactly `2` (integer). | +| `name` | yes | Stable grant name. Non-empty dotted name: each `.`-separated segment starts with a lowercase ASCII letter and continues with lowercase ASCII letters, digits, `-` or `_`; at most 128 bytes. A name equal to a reserved core resource kind is rejected. | +| `version` | yes | Strict SemVer (`semver::Version`, three numeric components). | +| `abi` | yes | Exactly `"c-abi-1"` (ABI v1, ADR-0036/0038). Anything else is `ext.unsupported_provider_schema`. | +| `transport` | yes | Exactly `"worker"` (framed-JSON helper, ADR-0035/0038). Anything else is `ext.unsupported_provider_schema`. | +| `kinds` | yes | Non-empty, at most 64 unique dotted names with the same grammar as `name`. Core kinds are reserved (see §2.2). | +| `artifacts` | yes | Non-empty, at most 32 entries with unique target triples. Each entry has `target`, `path`, `sha256`; unknown entry fields are rejected. | +| `permissions_requested` | no | Declarations only, at most 64 unique strings: a known host capability (`marqdo_plugin_abi::KNOWN_CAPABILITIES`) or a dotted vendor capability (at least three segments, no core namespace prefix). Declarations never grant anything. | +| `metadata` | no | Arbitrary JSON. Ignored for execution; never used to select artifacts or grant permissions. | + +`artifacts[].target` must parse as a target triple. `artifacts[].path` is +bundle-relative: absolute paths, `..` components and other escapes are +rejected. `artifacts[].sha256` must be 64 lowercase hexadecimal characters. +`artifacts[].path` is resolved against the manifest's own directory (the +bundle root); a symlink is followed only when the canonical target stays inside +that root, and the final target must be a regular file no larger than 256 MiB. + +### 2.2 Reserved core kinds + +`stdio`, `file`, `tcp`, `tls`, `udp` and `tcp-listener` are implemented by +`marqdo-host` before the provider registry is consulted. A provider may not +declare them (nor use one as its `name`); registration can never shadow them. + +Reserved core kinds are rejected with `ext.manifest_invalid`. + +### 2.3 Artifact selection + +The only artifact used is the one whose `target` equals +`marqdo_plugin_abi::HostTarget::current().triple()`. A missing current target +is `ext.no_matching_artifact` (exit 2); another platform's artifact or a +`default` fallback is never substituted. + +## 3. Identity and integrity + +* The manifest content digest is `sha256:` over the manifest bytes; the + selected artifact digest is the declared lowercase `sha256`. Both are + re-verified immediately before the helper is spawned, and again when the + registration helper is called, so a file replaced between preparation and + activation fails closed. +* These hashes prove the manifest/artifact pair is byte-identical to what was + selected. They do **not** prove who published it, that it is trustworthy or + that it may be granted. Publisher trust needs a signature/trust-root layer + that is out of scope for schema v2. +* No version of this schema stores a secret, environment value or shell + fragment. Manifests are never concatenated into a shell command. + +### 3.1 Duplicate identity + +* The same provider `name` selected with two different manifest digests is + `ext.duplicate_provider`; the exact same manifest selected twice is + idempotent. +* One non-reserved `kind` declared by two providers is + `ext.duplicate_provider_kind`. Registration never overwrites a previous + provider ("last wins" is forbidden). +* Error selection is deterministic: manifests are canonicalised, de-duplicated + and sorted before conflicts are reported, independent of CLI argument order. +* One provider may serve several non-reserved kinds; all of them are + registered together and must be usable. + +## 4. Worker helper resolution + +The CLI resolves the helper **only** as a sibling of the running `marqdo` +executable: + +```text +/marqdo-provider-worker[.exe] +``` + +* No `PATH` lookup, no current-directory search, no environment override and no + download fallback. A helper found on `PATH` is ignored. +* A missing sibling (or a non-regular file) is `ext.worker_unavailable` + (exit 1). A helper that fails to start or fails the protocol ping is also + `ext.worker_unavailable`; there is no fallback to in-process/native loading. +* Release archives must ship the `marqdo` binary and the + `marqdo-provider-worker` helper built for the same target triple. +* The `worker` transport is supported only on **linux/x86_64** in this release. + Other platforms compile but fail closed with `ext.platform_unsupported` + (exit 1). Windows and macOS are explicitly unsupported until they have a + native fixture and release evidence; Linux success is not a platform claim. + +## 5. Worker protocol v1 and token lifecycle + +The helper is `marqdo-provider-worker `. It loads +exactly one `c-abi-1` shared library, verifies +`marqdo_resource_provider_abi_version() == 1`, and speaks the ADR-0035 framed +protocol over stdin/stdout: 4-byte big-endian length + JSON, 1 MiB maximum per +frame. + +```text +{"op":"ping"} -> {"ok":true} (handshake) +{"op":"open","kind","descriptor","rights"} -> {"ok":true,"token":u64} +{"op":"read","token","max_bytes"} -> {"ok":true,"bytes":[u8...],"eof":bool} +{"op":"write","token","bytes":[u8...]} -> {"ok":true,"count":u64} +{"op":"close","token"} -> {"ok":true} +{"op":"shutdown"} -> process exit +{"ok":false,"code","message"} -> structured failure +``` + +Token lifecycle: + +1. `ping` must succeed before the host registers the provider. A failed + handshake aborts the run (`ext.worker_unavailable`). +2. `open` creates a provider-private `u64` token in the helper. The language + never sees it: the Host issues the opaque language `Resource` handle that + binds run id + provider name + token. Provider tokens cannot be forged into + language values or wire messages. +3. `read`/`write`/`close` are dispatched by the Host with the token it stored. + A read returns at most `max_bytes`; bytes cross the boundary by copy. +4. `close` closes exactly one token. Repeated or unknown tokens must fail with + a structured error; the Host never panics. +5. When the run ends (normal, error or timeout) every open token is closed and + the helper receives `shutdown`; the Host kills and reaps the child on drop. + A helper crash/protocol violation becomes a structured `host.error` for the + offending operation (CLI exit 1) and is never retried against another + provider or an in-process transport. +6. Provider error codes map to existing categories: `permission.denied`, + `resource.closed`, `resource.exhausted`, `resource.unsupported`, otherwise + `host.error`. + +## 6. Authorization + +* Registration, manifest parsing and artifact verification are not + authorization. Every `resource.open("", ...)` still requires: + project mode, the provider name in `Capabilities.resource_providers` + (populated only by exact `--allow-provider NAME`), and the Host's provider + name check. `permissions_requested` declarations do not affect this. +* `pure` and `strict` never reach provider kinds. `project` is the only mode + that can use a third-party provider. +* Unauthorized selection fails closed before the worker is spawned: + `--provider` without a matching `--allow-provider` is `permission.denied` + (exit 1) and the helper is never started. + +## 7. Error codes and exit codes + +Manifest/CLI option failures exit 2; load/verification/spawn failures exit 1; +`permission.denied` exits 1. With `--json` the CLI emits exactly one JSON +document with a `code` (nested under `error.code` for these structured +errors). + +| Code | Stage | Trigger | +|---|---|---| +| `ext.manifest_unreadable` | manifest read (2) | missing/unreadable path, directory instead of `provider.json` | +| `ext.unsupported_provider_schema` | manifest read (2) | `schema` mismatch, `schema_version != 2`, unsupported `abi`/`transport` | +| `ext.manifest_invalid` | manifest read (2) | invalid UTF-8/JSON, unknown field, bad name/kind/version/permission, reserved kind, duplicate kind/target inside one manifest | +| `ext.no_matching_artifact` | manifest read (2) | no artifact for the current host target | +| `ext.duplicate_provider` | manifest read (2) | same provider name with two different manifest digests | +| `ext.duplicate_provider_kind` | manifest read (2) | two providers declare the same non-reserved kind | +| `ext.provider_grant_unmatched` | grant check (2) | `--allow-provider` names no selected manifest | +| `ext.artifact_integrity` | artifact verify / pre-spawn re-verify (1) | sha256 mismatch, absolute/`..`/symlink escape, non-regular file, size limit, manifest changed after verification | +| `permission.denied` | grant check (1) | `--provider` selected without its exact `--allow-provider` grant | +| `ext.worker_unavailable` | spawn (1) | sibling helper missing/not a regular file, spawn failure, ping failure | +| `ext.platform_unsupported` | spawn (1) | provider `worker` transport on a non-linux/x86_64 build | +| `cli.invalid_grants` | CLI preflight (2) | wildcard/empty `--allow-provider`, provider flags in `pure` mode | +| `strict.unsupported_grant` | strict preflight (2) | provider flags in `strict` mode | + +`ext.manifest_invalid`, `ext.duplicate_provider` and +`ext.provider_grant_unmatched` are additive codes: RFC-EXT-0001 §18.1 froze the +provider-specific codes it could name, but a generic malformed-v2-manifest +code, a duplicate-provider-name code and an unmatched-grant code are required +to keep failure reasons distinct. They follow the same `ext.` naming and exit +code contract and are stable. + +## 8. Test matrix (PR-03) + +`crates/cli/tests/provider_cli.rs` builds a real C `c-abi-1` sensor provider, +runs the real `marqdo-provider-worker` next to the real CLI binary, and covers: + +| ID | Evidence | +|---|---| +| PROV-01 | granted v2 worker + `resource.open/read/close("demo.sensor")` returns count 5, EOF, one close, `resource.open` in the host trace | +| PROV-02 | `--provider` without grant → `permission.denied`; marker proves the helper was never started | +| PROV-03 | `--allow-provider` without `--provider` → `ext.provider_grant_unmatched`, no worker | +| PROV-04 | same kind from two providers → `ext.duplicate_provider_kind` before any worker | +| PROV-05 | one provider with two non-reserved kinds → both open/read/close, one helper | +| PROV-06 | `stdio`/`file`/`tcp`/`tls`/`udp`/`tcp-listener` kinds rejected, no worker | +| PROV-07 | digest mismatch, symlink escape, `..` traversal → `ext.artifact_integrity`, no worker | +| PROV-08 | no current-target artifact → `ext.no_matching_artifact` | +| PROV-09 | `schema_version: 1` → `ext.unsupported_provider_schema` | +| PROV-10 | `pure`/`strict` + provider flags → preflight refusal, no worker; wildcard grant rejected | +| PROV-13 | relocated CLI without sibling helper (helper present on `PATH`) → `ext.worker_unavailable`, PATH is never used | + +PROV-11 (worker crash/timeout/size-limit), PROV-12 (closed/stale/forged token), +PROV-14 (concurrent lifecycle) and PROV-15 (frozen-binary external repository) +are covered by the host worker/protocol tests or belong to the PR-07 external +conformance repository; they are not part of this CLI slice's manifest matrix. diff --git a/spec/security.md b/spec/security.md index c9c3036..051b37a 100644 --- a/spec/security.md +++ b/spec/security.md @@ -9,6 +9,7 @@ Normative implementation: |---|---| | Sandbox modes, strict refusal | `crates/marqdo-sandbox/src/lib.rs` | | Capability sets and checks | `crates/marqdo-host/src/capability.rs` | +| Trusted capability handlers | `crates/marqdo-host/src/capability.rs` (`CapabilityBroker`, `CapabilityHandler`, `FsReadHandler`; ADR-0057) | | Host dispatch and per-op checks | `crates/marqdo-host/src/host.rs` | | Project-rooted filesystem | `crates/marqdo-host/src/fs.rs` | | Budgets | `crates/marqdo-host/src/budget.rs` | @@ -175,18 +176,72 @@ before reading. Full symlink/race hardening is an open P4 item. * plugin invoke → the **complete export name** must be registered, then `capabilities.check_plugin(package)` on **every call**; loading also checks the same grant once; -* plugin host-capability requests → `CapabilityBroker` checks the package and - effective grants and **executes** granted `fs.read` through `ProjectFs`, - returning the real file bytes. Other core capabilities (`fs.write`, - `net.request`, `process`, console/math/log/...), and vendor capabilities - without a trusted executing handler, return `capability.unavailable`; they - never pretend an empty reply is success. Unknown/vendor names without a - trusted policy entry become `capability.unavailable`/`permission.denied`; +* plugin host-capability requests → `CapabilityBroker` runs a fixed chain: + `decide` (package, effective grants, trusted vendor policy, payload shape) → + a trusted `CapabilityHandler` must be installed for the exact capability name + → a package-bound handler must own the trusted package identity → the handler + executes under the broker budget and reply limit → one audit entry. Granted + `fs.read` executes through `FsReadHandler`, which resolves the path with + `ProjectFs` and re-checks the read grant on the normalized project-relative + path before returning the real file bytes. Every capability without an + installed handler — other core capabilities (`fs.write`, `net.request`, + `process`, console/math/log/clock/random/artifact.record) and vendor + capabilities without a trusted executor — returns `capability.unavailable`; + it never pretends an empty reply is success. A vendor name with no trusted + policy entry is `permission.denied` (ADR-0011, ADR-0057); * print → output budget only. Static analysis may warn, but it never replaces these runtime checks. There is no host bypass for tests: tests use the same `StdHost`/`Capabilities` types. +### 2.3 Executable capabilities need an installed trusted handler + +`CapabilityBroker` (`crates/marqdo-host/src/capability.rs`, ADR-0057) separates +*being authorized* from *being executable*: + +| Question | Registry | Failure | +|---|---|---| +| Is this capability granted for this trusted package? | `Capabilities` / `vendor_policy` | `permission.denied` | +| Is a trusted executor installed for this exact name? | `handlers: BTreeMap` | `capability.unavailable` | +| Does a package-bound handler own the claimed package? | `HandlerEntry.package` | `permission.denied` | + +Execution order is fixed: `decide` → handler installed → package binding → +handler execution under the broker `Budget` and reply limit → audit. A name in +`vendor_policy` is policy only; it does not install anything, and it cannot +turn into an executor. A core capability granted by `Capabilities` but missing +a handler is `capability.unavailable`, never an empty `Ok`. + +* **Registration is host-only.** `register_handler(handler)` / + `register_package_handler(package, handler)` are called by the trusted host / + embedder composition root; no manifest, plugin or wire message can reach the + registry. The registry key is `handler.name()`, so a handler cannot be + registered under a name it does not declare. +* **Rejected names** (stable codes): empty/whitespace/malformed names are + `capability.handler_invalid`; names under the reserved host prefixes + `marqdo.` / `host.` are `capability.handler_reserved`; an already registered + name (including the built-in `fs.read`) is `capability.handler_conflict`. A + rejected handler is not installed. +* **`fs.read`** is the first migrated standard handler (`FsReadHandler`), + installed by `CapabilityBroker::with_fs(fs)`. It keeps the `ProjectFs` root, + traversal/absolute-path rejection, regular-file/UTF-8/8 MiB limits and the + glob grant, and it re-checks the grant against the normalized path at the + execution point. The broker reply limit (`MAX_CAPABILITY_REPLY_BYTES` by + default) bounds the returned bytes; an oversize reply is + `resource.exhausted`. +* **Audit**: every decision records package, capability, + `granted`/`denied`/`unavailable`, a reason and `duration_micros`. Payloads + and file contents are never recorded, so ordinary secrets do not appear in + the audit log. +* **Budgets**: handlers receive `&mut Budget` and can charge fuel/output; the + broker also enforces the reply byte limit. Native handlers still run + in-process and are trusted code, not a sandbox. + +Known gaps in this build: the broker `trace_id` is settable +(`with_trace_id`) but `host.rs` does not supply it yet; and a broker-level +`resource.exhausted` reply keeps its stable code but the host plugin-error +mapping currently collapses `PluginError` kind `capability` to the +`permission.denied` runtime category (changing that mapping is a follow-up). + ## 3. Resource budgets `Budget` (`crates/marqdo-host/src/budget.rs`) is per run: @@ -338,11 +393,15 @@ Neither command executes project code. Specifically: 4. Write-path symlink TOCTOU (see §2.1). 5. Plugin **worker crash isolation** and the worker capability bridge are implemented (not usable in strict L1): both native and worker plugins can - reach the broker's - `fs.read` execution when granted. Other core capabilities and vendor - handlers remain unavailable. Broker execution is a cooperative API - contract, not an OS sandbox; a worker and a native plugin can still issue - direct syscalls. + reach the broker's `fs.read` handler when granted (ADR-0057). Only + `fs.read` has a registered handler; every other core capability and every + vendor capability without an explicitly registered trusted handler remains + `capability.unavailable`, and there is no API to register one from a plugin, + manifest or CLI flag. Broker execution is a cooperative API contract, not an + OS sandbox; a worker and a native plugin can still issue direct syscalls. + Follow-ups: wire the run `trace_id` into the broker audit and teach the host + plugin-error mapping to report `resource.exhausted` as a resource category + rather than `permission.denied`. 6. `plugin_handle` wire values are structurally reconstructable, but host authority comes only from the `PluginHandleTable` owner/run/generation check; untrusted plugin/foreign output containing a handle is rejected diff --git a/stdlib/stdlib.json b/stdlib/stdlib.json index a2e8d2a..9095a1d 100644 --- a/stdlib/stdlib.json +++ b/stdlib/stdlib.json @@ -1,5 +1,20 @@ { "name": "marqdo-stdlib", "version": "0.1.0", - "digest": "sha256:b94f1a48ca6619689584aca3af7befac64ac9fdb7ea7c0a9ccb491df8df226b8" + "digest": "sha256:5407d3665ba2fe105820d0f1af5efaafa10c277a69eb819c3225719813c8c713", + "public_aliases": [ + { + "public_names": [ + "print", + "打印" + ], + "target_module": "io.mq.md", + "target_function": "写值", + "expected_signature": "params(值)", + "effects": [ + "host" + ] + } + ], + "aliases_digest": "sha256:5f513c0fdbb3c1e9829be3c9fa23a2d1e89cb8510073fb5eb1e8e2f5265a83bb" } diff --git a/stdlib/tests/fs.mq.md b/stdlib/tests/fs.mq.md index 48e0c89..2b26bd4 100644 --- a/stdlib/tests/fs.mq.md +++ b/stdlib/tests/fs.mq.md @@ -1,38 +1,38 @@ --- import: - fs: "std:fs/file" + fs_lib: "std:fs/file" --- # @exec fs ## main -**创建 = fs.创建目录("out")** +**创建 = fs_lib.创建目录("out")** -**写入 = fs.写文本("out/fs-a.txt", "你好\n")** +**写入 = fs_lib.写文本("out/fs-a.txt", "你好\n")** -**存在 = fs.存在("out/fs-a.txt")** +**存在 = fs_lib.存在("out/fs-a.txt")** -**元数据 = fs.元数据("out/fs-a.txt")** +**元数据 = fs_lib.元数据("out/fs-a.txt")** -**列表 = fs.列出目录("out")** +**列表 = fs_lib.列出目录("out")** -**是文件 = fs.是文件("out/fs-a.txt")** +**是文件 = fs_lib.是文件("out/fs-a.txt")** -**是目录 = fs.是目录("out")** +**是目录 = fs_lib.是目录("out")** -**重命名 = fs.重命名("out/fs-a.txt", "out/fs-b.txt")** +**重命名 = fs_lib.重命名("out/fs-a.txt", "out/fs-b.txt")** -**读取 = fs.读文本("out/fs-b.txt")** +**读取 = fs_lib.读文本("out/fs-b.txt")** -**追加 = fs.追加文本("out/fs-b.txt", "世界\n")** +**追加 = fs_lib.追加文本("out/fs-b.txt", "世界\n")** -**片段 = fs.读取片段("out/fs-b.txt", 0, 100)** +**片段 = fs_lib.读取片段("out/fs-b.txt", 0, 100)** -**遍历 = fs.遍历("out")** +**遍历 = fs_lib.遍历("out")** -**删除 = fs.删除("out/fs-b.txt")** +**删除 = fs_lib.删除("out/fs-b.txt")** -**删除目录 = fs.删除("out")** +**删除目录 = fs_lib.删除("out")** `return [创建.ok, 写入.value.count, 存在.value, 元数据.value.kind, 元数据.value.size, 列表.value, 是文件.value, 是目录.value, 重命名.ok, 读取.value, 追加.value, 片段.value, 遍历.value, 删除.ok, 删除目录.ok]` diff --git a/stdlib/tests/table.mq.md b/stdlib/tests/table.mq.md index da8be64..ae648b5 100644 --- a/stdlib/tests/table.mq.md +++ b/stdlib/tests/table.mq.md @@ -1,6 +1,6 @@ --- import: - table: "std:data/table" + table_lib: "std:data/table" --- # @exec table @@ -9,22 +9,22 @@ import: **数据 = 表(["名称", "分数"], [["甲", 87], ["乙", 92], ["丙", 71]])** -**选择 = table.选择列(数据, 1)** +**选择 = table_lib.选择列(数据, 1)** -**过滤 = table.过滤数值大于(数据, 1, 80)** +**过滤 = table_lib.过滤数值大于(数据, 1, 80)** -**排序 = table.排序(数据, 1)** +**排序 = table_lib.排序(数据, 1)** -**追加 = table.追加行(数据, ["丁", 65])** +**追加 = table_lib.追加行(数据, ["丁", 65])** -**设置 = table.设置单元格(数据, 0, 1, 99)** +**设置 = table_lib.设置单元格(数据, 0, 1, 99)** **左 = 表(["名称", "分数"], [["甲", 87], ["乙", 92]])** **右 = 表(["名称", "等级"], [["乙", "A"], ["甲", "B"]])** -**连接 = table.连接(左, 右, 0, 0)** +**连接 = table_lib.连接(左, 右, 0, 0)** -**分组 = table.分组(表(["类别", "值"], [["a", 1], ["b", 2], ["a", 3]]), 0, 1)** +**分组 = table_lib.分组(表(["类别", "值"], [["a", 1], ["b", 2], ["a", 3]]), 0, 1)** -`return [选择, 过滤, 排序, 追加, 设置, 连接, 分组, 数据, table.转GFM(数据)]` +`return [选择, 过滤, 排序, 追加, 设置, 连接, 分组, 数据, table_lib.转GFM(数据)]` diff --git a/stdlib/tests/text.mq.md b/stdlib/tests/text.mq.md index 989c81d..51c51b7 100644 --- a/stdlib/tests/text.mq.md +++ b/stdlib/tests/text.mq.md @@ -1,6 +1,6 @@ --- import: - text: "std:text" + text_lib: "std:text" --- # @exec text @@ -9,6 +9,6 @@ import: **内存 = memory.new(1, 1)** -**长度 = text.utf8编码("é", 内存, 0)** +**长度 = text_lib.utf8编码("é", 内存, 0)** `return [长度, 内存[0], 内存[1]]` diff --git a/tools/ci/verify_extension_contracts.py b/tools/ci/verify_extension_contracts.py new file mode 100755 index 0000000..6d13fd0 --- /dev/null +++ b/tools/ci/verify_extension_contracts.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +"""Frozen-CLI external-extension evidence generator and validator. + +RFC-EXT-0001 §10.2/§11.3: a release candidate must prove that independent +repositories work against an *unchanged* CLI binary. This script: + +1. hashes the CLI binary before and after the external suite and refuses to + emit PASS if the hash moved; +2. records the exact marqdo-cli source SHA and the external fixture repo SHA; +3. runs the fixture repository's own black-box suite with ``MARQDO_BIN`` set to + the frozen binary (the suite itself must not run cargo or read the CLI + source tree); +4. validates/normalises the per-case results into the repository evidence + schema (``docs/evidence/extension/schema.json``); +5. writes ``docs/evidence/extension//manifest.json``. + +Usage: + + python3 tools/ci/verify_extension_contracts.py \ + --cli target/release/marqdo \ + --fixtures ../marqdo-extension-conformance \ + --out docs/evidence/extension//manifest.json + + python3 tools/ci/verify_extension_contracts.py --validate manifest.json + +Only Python's standard library is used. No network access is performed by this +script; the external suite documents its own offline cache behaviour. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import platform +import subprocess +import sys +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +SCHEMA = REPO / "docs" / "evidence" / "extension" / "schema.json" + +REQUIRED_RESULTS = [ + "callable_registry", + "plugin_worker", + "foreign_kernel", + "resource_provider_cli", + "package_pinned_git", + "tool_analysis_consumer", + "negative_security", +] + +ALLOWED = {"PASS", "FAIL", "UNSUPPORTED", "SKIPPED"} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def git_sha(directory: Path) -> str: + output = subprocess.run( + ["git", "-C", str(directory), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + return output.stdout.strip() + + +def os_name() -> str: + name = platform.system().lower() + if name.startswith("linux"): + return "linux" + if name == "darwin": + return "macos" + if name.startswith("windows"): + return "windows" + raise SystemExit(f"unsupported OS for extension evidence: {name}") + + +def validate_manifest(manifest: dict) -> list[str]: + problems: list[str] = [] + if manifest.get("schema") != "marqdo.extension-evidence": + problems.append("schema must be marqdo.extension-evidence") + if manifest.get("schema_version") != 1: + problems.append("schema_version must be 1") + for field in ["tested_sha", "fixture_repo_sha"]: + value = manifest.get(field, "") + if not (isinstance(value, str) and len(value) == 40 and all(c in "0123456789abcdef" for c in value)): + problems.append(f"{field} must be a 40-character lowercase git SHA") + cli = manifest.get("cli_binary_sha256", "") + if not (isinstance(cli, str) and len(cli) == 64 and all(c in "0123456789abcdef" for c in cli)): + problems.append("cli_binary_sha256 must be a 64-character lowercase sha256") + if manifest.get("os") not in {"linux", "macos", "windows"}: + problems.append("os must be linux, macos or windows") + results = manifest.get("results") + if not isinstance(results, dict): + problems.append("results must be an object") + else: + for name in REQUIRED_RESULTS: + if results.get(name) not in ALLOWED: + problems.append(f"results.{name} must be one of {sorted(ALLOWED)}") + for key, value in results.items(): + if key not in REQUIRED_RESULTS: + problems.append(f"unknown result key {key}") + if value not in ALLOWED: + problems.append(f"result {key}={value} is not allowed") + commands = manifest.get("commands") + if not isinstance(commands, list) or not commands or not all(isinstance(c, str) for c in commands): + problems.append("commands must be a non-empty array of strings") + for field in ["failures", "unverified"]: + if not isinstance(manifest.get(field), list): + problems.append(f"{field} must be an array") + if not problems: + for name, value in (manifest.get("results") or {}).items(): + if value in {"FAIL"}: + problems.append(f"result {name} is FAIL") + return problems + + +def run_suite(fixtures: Path, cli: Path) -> tuple[bool, dict, list[str]]: + """Run the external suite and return (ok, results, commands).""" + results: dict[str, str] = {} + commands: list[str] = [] + env = dict(os.environ) + env["MARQDO_BIN"] = str(cli) + env["MARQDO_FROZEN_CLI"] = str(cli) + runner = fixtures / "tests" / "run_all.py" + if not runner.is_file(): + return False, results, commands + commands.append(f"python3 {runner} (cwd={fixtures})") + completed = subprocess.run( + [sys.executable, str(runner)], + cwd=fixtures, + env=env, + capture_output=True, + text=True, + ) + print(completed.stdout, end="") + print(completed.stderr, end="", file=sys.stderr) + results_path = fixtures / "evidence" / "results.json" + if results_path.is_file(): + with results_path.open("r", encoding="utf-8") as handle: + payload = json.load(handle) + for key, value in (payload.get("results") or {}).items(): + results[key] = str(value).upper() + return completed.returncode == 0, results, commands + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cli", type=Path, help="Absolute/relative frozen marqdo binary") + parser.add_argument("--fixtures", type=Path, help="External fixture repository root") + parser.add_argument("--source-repo", type=Path, default=REPO, help="marqdo-cli checkout") + parser.add_argument("--out", type=Path, help="Evidence manifest output path") + parser.add_argument("--validate", type=Path, help="Only validate an existing manifest") + parser.add_argument("--allow-unverified", action="store_true", help="Record UNSUPPORTED/SKIPPED without failing") + args = parser.parse_args() + + if args.validate: + manifest = json.loads(args.validate.read_text(encoding="utf-8")) + problems = validate_manifest(manifest) + if problems: + for problem in problems: + print(f"INVALID: {problem}", file=sys.stderr) + return 1 + print(f"valid extension evidence: {args.validate}") + return 0 + + if not args.cli or not args.fixtures: + parser.error("--cli and --fixtures are required unless --validate is used") + cli = args.cli.resolve() + fixtures = args.fixtures.resolve() + if not cli.is_file(): + print(f"error: CLI binary not found: {cli}", file=sys.stderr) + return 2 + if not (fixtures / ".git").exists(): + print(f"error: fixture path is not a git repository: {fixtures}", file=sys.stderr) + return 2 + if subprocess.run( + ["git", "-C", str(fixtures), "status", "--porcelain"], + capture_output=True, + text=True, + check=True, + ).stdout.strip(): + print("error: fixture repository has uncommitted changes", file=sys.stderr) + return 2 + + cli_before = sha256_file(cli) + tested_sha = git_sha(args.source_repo.resolve()) + fixture_sha = git_sha(fixtures) + ok, results, commands = run_suite(fixtures, cli) + cli_after = sha256_file(cli) + if cli_before != cli_after: + print("error: frozen CLI hash changed during the external suite", file=sys.stderr) + return 1 + + failures = [name for name, value in results.items() if value == "FAIL"] + unverified = [ + name for name, value in results.items() if value in {"UNSUPPORTED", "SKIPPED"} + ] + missing = [name for name in REQUIRED_RESULTS if name not in results] + if missing: + failures.extend(f"missing:{name}" for name in missing) + manifest = { + "schema": "marqdo.extension-evidence", + "schema_version": 1, + "tested_sha": tested_sha, + "cli_binary_sha256": cli_before, + "fixture_repo_sha": fixture_sha, + "os": os_name(), + "arch": platform.machine(), + "results": {name: results.get(name, "SKIPPED") for name in REQUIRED_RESULTS}, + "commands": commands, + "failures": failures, + "unverified": unverified, + } + problems = validate_manifest(manifest) + if unverified and not args.allow_unverified: + problems.append( + "UNSUPPORTED/SKIPPED results require --allow-unverified and an explicit report entry" + ) + if not ok and not failures: + failures.append("external suite exited non-zero without structured results") + problems.append("external suite failed") + if problems: + for problem in problems: + print(f"FAIL: {problem}", file=sys.stderr) + print(json.dumps(manifest, indent=2)) + return 1 + out = args.out or (REPO / "docs" / "evidence" / "extension" / tested_sha / "manifest.json") + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(f"wrote extension evidence: {out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/pack-stdlib.py b/tools/pack-stdlib.py index 5d420c5..dbe455b 100755 --- a/tools/pack-stdlib.py +++ b/tools/pack-stdlib.py @@ -18,6 +18,22 @@ STDLIB = ROOT / "stdlib" DIST = ROOT / "dist" +def aliases_digest(manifest: dict) -> str: + digest = hashlib.sha256() + for alias in sorted( + manifest.get("public_aliases", []), + key=lambda a: (a["target_module"], a["target_function"]), + ): + digest.update(alias["target_module"].encode()) + digest.update(b"\0") + digest.update(alias["target_function"].encode()) + digest.update(b"\0") + for name in sorted(alias["public_names"]): + digest.update(name.encode()) + digest.update(b"\0") + return f"sha256:{digest.hexdigest()}" + + def tree_digest() -> str: digest = hashlib.sha256() for path in sorted( @@ -37,6 +53,9 @@ def main() -> int: digest = tree_digest() if manifest["digest"] != f"sha256:{digest}": raise SystemExit("stdlib.json digest does not match the tree") + alias_digest = aliases_digest(manifest) + if manifest.get("aliases_digest") != alias_digest: + raise SystemExit("stdlib.json aliases_digest does not match public_aliases") files = sorted( [p for p in STDLIB.rglob("*") if p.is_file() and p.name != "stdlib.json"] @@ -65,6 +84,7 @@ def main() -> int: "name": manifest["name"], "version": version, "tree_digest": f"sha256:{digest}", + "aliases_digest": alias_digest, "archive": archive.name, "archive_sha256": f"sha256:{archive_sha}", "files": [p.relative_to(STDLIB).as_posix() for p in files],