feat(stdlib): ship a verified bundled stdlib with std: imports
- Move the Compat-A print prototype into stdlib/ as the official offline bundle with stdlib.json (version 0.1.0 + sha256 content digest). - Add std:<relative> import resolution in the loader with bundle path confinement, extension inference and fail-closed digest verification. - Record the verified version/digest/instance_id in Analysis v2 project.stdlib; stdlib modules get a std:name@version#digest instance id. - Add CLI tests: std:io execution, digest-tamper rejection, missing manifest, Analysis identity, no foreign/plugin calls, and frozen binary hash equality. - Update ADR-0028, specs, status, changelog and evidence. Gate: fmt, clippy -D warnings, 415 tests passed.
This commit is contained in:
1 parent
1a978c7a42
commit
1377af1601
28 files changed
+879
-23
No files matched your search
+6
-1
@@ -108,7 +108,12 @@ currently `0.1.0`; no release tag has been cut. Dates use ISO 8601.
|
||||
an `fs_write` grant, buffers in the host, flushes only on `close`, never
|
||||
flushes on drop, has an 8 MiB ceiling, and is refused in pure/strict. A
|
||||
`.mq.md` chunked copy loop reproduces a file without a Rust `copy_file`.
|
||||
- CM-A07 **Compat-A** prototype (ADR-0027): `examples/stdlib/` provides
|
||||
- CM-A10 bundled stdlib (ADR-0028): a fixed `stdlib/` tree with
|
||||
`format/text/io`, explicit `std:<path>` imports, SHA-256 digest verification
|
||||
before module loading, a `std:marqdo-stdlib@<version>#<digest>` package
|
||||
instance id, `project.stdlib` in Analysis v2, and test coverage for
|
||||
`std:` resolution, tamper rejection and frozen-binary hash equality.
|
||||
- CM-A07 **Compat-A** prototype (ADR-0027): `stdlib/` provides
|
||||
`format.显示`, UTF-8 encoding and `io.写值` (`resource.write(stdout)`), and
|
||||
its output is comparison-tested byte-for-byte against builtin `打印` for
|
||||
Unit/None/Bool/Int/Float/Text/List/Map/Table. Builtin `打印` remains the
|
||||
|
||||
Generated
+1
@@ -343,6 +343,7 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"tempfile",
|
||||
"walkdir",
|
||||
]
|
||||
|
||||
|
||||
@@ -99,10 +99,11 @@ executes anything.
|
||||
`{ok:false,code,message}` as an ordinary Map, with a `.mq.md` first-fit
|
||||
free-list allocator demonstrating reuse, splitting and `alloc.no_space`
|
||||
recovery (ADR-0024). No `Result` type or exception syntax.
|
||||
* **Stdlib print prototype (RFC-CM CM-A07 Compat-A)** — ordinary imported
|
||||
`.mq.md` `format.显示` + UTF-8 + `resource.write(stdout)` matches builtin
|
||||
`打印` for the covered value set; the builtin is still the default and no
|
||||
alias is routed (ADR-0027).
|
||||
* **Bundled stdlib + print prototype (RFC-CM CM-A07/A10)** — offline
|
||||
`stdlib/` with explicit `std:io`/`std:format`/`std:text` imports, SHA-256
|
||||
digest verification, Analysis identity and a frozen-CLI closure test; the
|
||||
`.mq.md` print chain matches builtin `打印` for the covered values, while
|
||||
`打印` remains the default builtin (ADR-0027/0028).
|
||||
* **Resource streams (RFC-CM CM-A06)** — opaque host-issued `Resource` handles;
|
||||
`resource.open/read/write/close` transfer bytes through `Memory`; stdout
|
||||
write in pure/project, granted project-file read, and buffered project-file
|
||||
|
||||
@@ -33,3 +33,6 @@ semver = { workspace = true }
|
||||
[[bin]]
|
||||
name = "marqdo"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = { workspace = true }
|
||||
@@ -61,6 +61,11 @@ pub fn build(project: &LoadedProject) -> Value {
|
||||
"root_id": project.identity.root_id(),
|
||||
"identity_kind": project.identity.kind(),
|
||||
"lock_digest": lock_digest(project),
|
||||
"stdlib": project.stdlib.as_ref().map(|stdlib| json!({
|
||||
"version": stdlib.version,
|
||||
"digest": stdlib.digest,
|
||||
"instance_id": stdlib.instance_id,
|
||||
})),
|
||||
},
|
||||
"modules": modules,
|
||||
"diagnostics": diagnostics_json(project),
|
||||
|
||||
@@ -20,6 +20,8 @@ use marqdo_package::{
|
||||
};
|
||||
use marqdo_value::{codes, Diagnostic};
|
||||
|
||||
use crate::stdlib::StdlibBundle;
|
||||
|
||||
/// One analysed source file.
|
||||
pub struct LoadedFile {
|
||||
/// Canonical module identity (package instance + package-relative path).
|
||||
@@ -125,6 +127,17 @@ pub struct LocatedSource<'a> {
|
||||
pub column: usize,
|
||||
}
|
||||
|
||||
/// Identity of the verified bundled stdlib used by a load, if any.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct StdlibInfo {
|
||||
/// Declared version.
|
||||
pub version: String,
|
||||
/// Verified content digest.
|
||||
pub digest: String,
|
||||
/// Package instance id.
|
||||
pub instance_id: String,
|
||||
}
|
||||
|
||||
/// A fully loaded project ready for checking or running.
|
||||
pub struct LoadedProject {
|
||||
/// Project root.
|
||||
@@ -141,6 +154,8 @@ pub struct LoadedProject {
|
||||
pub packages: BTreeMap<String, PathBuf>,
|
||||
/// All diagnostics in deterministic order.
|
||||
pub diagnostics: Vec<FileDiagnostic>,
|
||||
/// Verified bundled stdlib identity, when one was available.
|
||||
pub stdlib: Option<StdlibInfo>,
|
||||
}
|
||||
|
||||
impl LoadedProject {
|
||||
@@ -205,6 +220,7 @@ struct Loader {
|
||||
program: Program,
|
||||
diagnostics: Vec<FileDiagnostic>,
|
||||
stack: Vec<PathBuf>,
|
||||
stdlib: Option<StdlibBundle>,
|
||||
}
|
||||
|
||||
/// Load the entry file and its explicitly imported modules.
|
||||
@@ -250,6 +266,7 @@ pub fn load(entry: &Path) -> Result<LoadedProject, String> {
|
||||
};
|
||||
let resolver = PackageResolver::new(root.clone(), manifest.clone(), lock);
|
||||
let entry_module = resolver.resolve_entry(&entry)?;
|
||||
let stdlib = StdlibBundle::discover().map_err(|error| format!("stdlib: {error}"))?;
|
||||
let mut loader = Loader {
|
||||
resolver,
|
||||
packages: BTreeMap::new(),
|
||||
@@ -257,6 +274,7 @@ pub fn load(entry: &Path) -> Result<LoadedProject, String> {
|
||||
program: Program::default(),
|
||||
diagnostics: Vec::new(),
|
||||
stack: Vec::new(),
|
||||
stdlib,
|
||||
};
|
||||
let entry_id = loader.load_module(&entry_module.package, &entry_module.relative_path)?;
|
||||
loader.program.main_module = Some(entry_id.clone());
|
||||
@@ -270,6 +288,11 @@ pub fn load(entry: &Path) -> Result<LoadedProject, String> {
|
||||
});
|
||||
}
|
||||
}
|
||||
let stdlib_info = loader.stdlib.as_ref().map(|bundle| StdlibInfo {
|
||||
version: bundle.version().to_string(),
|
||||
digest: bundle.digest().to_string(),
|
||||
instance_id: bundle.instance_id(),
|
||||
});
|
||||
let identity = match &manifest {
|
||||
Some(_) => ProjectIdentity::Manifest {
|
||||
instance: entry_id.instance.clone(),
|
||||
@@ -286,6 +309,7 @@ pub fn load(entry: &Path) -> Result<LoadedProject, String> {
|
||||
files: loader.files,
|
||||
packages: loader.packages,
|
||||
diagnostics: loader.diagnostics,
|
||||
stdlib: stdlib_info,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -360,7 +384,16 @@ impl Loader {
|
||||
.unwrap_or_default();
|
||||
|
||||
for (alias, target, span) in imports {
|
||||
let resolved = if let Some(alias_name) = target.strip_prefix('@') {
|
||||
let resolved = if let Some(std_target) = target.strip_prefix("std:") {
|
||||
match &self.stdlib {
|
||||
Some(stdlib) => stdlib.resolve(std_target),
|
||||
None => Err(format!(
|
||||
"no bundled stdlib is available for `{target}`; install the official \
|
||||
CLI bundle or set {STDLIB_DIR_ENV}",
|
||||
STDLIB_DIR_ENV = crate::stdlib::STDLIB_DIR_ENV
|
||||
)),
|
||||
}
|
||||
} else if let Some(alias_name) = target.strip_prefix('@') {
|
||||
self.resolver.resolve_alias_from(package, alias_name)
|
||||
} else {
|
||||
self.resolver.resolve_relative(package, &path, &target)
|
||||
|
||||
@@ -9,6 +9,7 @@ mod args;
|
||||
mod commands;
|
||||
mod loader;
|
||||
mod output;
|
||||
mod stdlib;
|
||||
mod tool;
|
||||
mod trace;
|
||||
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
//! Bundled standard library discovery, digest verification and `std:` imports.
|
||||
//!
|
||||
//! The official CLI ships a fixed `marqdo-stdlib` tree (ADR-0026/CM-A10). This
|
||||
//! module finds that tree, verifies its declared content digest before any
|
||||
//! module is loaded, and resolves explicit `std:<relative>` imports to it.
|
||||
//! Nothing is fetched implicitly and there is no second module search path.
|
||||
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
|
||||
use marqdo_package::{PackageOrigin, ResolvedModule, ResolvedPackage};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
/// Environment override used by tests and relocatable installations.
|
||||
pub const STDLIB_DIR_ENV: &str = "MARQDO_STDLIB_DIR";
|
||||
|
||||
/// Verified bundled standard library.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct StdlibBundle {
|
||||
root: PathBuf,
|
||||
name: String,
|
||||
version: String,
|
||||
digest: String,
|
||||
}
|
||||
|
||||
impl StdlibBundle {
|
||||
/// Discover the bundle.
|
||||
///
|
||||
/// Order: `MARQDO_STDLIB_DIR` (explicit, fail if invalid), a `stdlib`
|
||||
/// directory next to the executable, then the development tree.
|
||||
pub fn discover() -> Result<Option<Self>, String> {
|
||||
if let Some(dir) = std::env::var_os(STDLIB_DIR_ENV) {
|
||||
return Self::load(PathBuf::from(dir)).map(Some);
|
||||
}
|
||||
if let Ok(exe) = std::env::current_exe() {
|
||||
if let Some(parent) = exe.parent() {
|
||||
let candidate = parent.join("stdlib");
|
||||
if candidate.is_dir() {
|
||||
return Self::load(candidate).map(Some);
|
||||
}
|
||||
}
|
||||
}
|
||||
let dev = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("..")
|
||||
.join("..")
|
||||
.join("stdlib");
|
||||
if dev.is_dir() {
|
||||
return Self::load(dev).map(Some);
|
||||
}
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// Load and verify one bundle directory.
|
||||
pub fn load(root: impl Into<PathBuf>) -> Result<Self, String> {
|
||||
let root = root.into();
|
||||
let root = root.canonicalize().map_err(|error| {
|
||||
format!("cannot open stdlib directory `{}`: {error}", root.display())
|
||||
})?;
|
||||
let manifest_path = root.join("stdlib.json");
|
||||
let text = std::fs::read_to_string(&manifest_path)
|
||||
.map_err(|error| format!("cannot read `{}`: {error}", manifest_path.display()))?;
|
||||
let manifest: serde_json::Value = serde_json::from_str(&text)
|
||||
.map_err(|error| format!("invalid `{}`: {error}", manifest_path.display()))?;
|
||||
let name = manifest
|
||||
.get("name")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.ok_or_else(|| format!("`{}` has no name", manifest_path.display()))?
|
||||
.to_string();
|
||||
let version = manifest
|
||||
.get("version")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.ok_or_else(|| format!("`{}` has no version", manifest_path.display()))?
|
||||
.to_string();
|
||||
let declared = manifest
|
||||
.get("digest")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.ok_or_else(|| format!("`{}` has no digest", manifest_path.display()))?
|
||||
.to_string();
|
||||
let actual = tree_digest(&root)?;
|
||||
if declared != actual {
|
||||
return Err(format!(
|
||||
"stdlib digest mismatch: declared {declared}, actual {actual}"
|
||||
));
|
||||
}
|
||||
Ok(Self {
|
||||
root,
|
||||
name,
|
||||
version,
|
||||
digest: actual,
|
||||
})
|
||||
}
|
||||
|
||||
/// Declared stdlib version.
|
||||
pub fn version(&self) -> &str {
|
||||
&self.version
|
||||
}
|
||||
|
||||
/// Verified content digest (`sha256:<hex>`).
|
||||
pub fn digest(&self) -> &str {
|
||||
&self.digest
|
||||
}
|
||||
|
||||
/// Logical package identity used for module ids.
|
||||
pub fn instance_id(&self) -> String {
|
||||
format!(
|
||||
"std:{}@{}#{}",
|
||||
self.name,
|
||||
self.version,
|
||||
self.digest.trim_start_matches("sha256:")
|
||||
)
|
||||
}
|
||||
|
||||
/// Package view consumed by the loader.
|
||||
pub fn package(&self) -> ResolvedPackage {
|
||||
ResolvedPackage {
|
||||
instance_id: self.instance_id(),
|
||||
name: self.name.clone(),
|
||||
version: self.version.clone(),
|
||||
origin: PackageOrigin::Path {
|
||||
declared: PathBuf::from("stdlib"),
|
||||
},
|
||||
revision: None,
|
||||
content_digest: Some(self.digest.clone()),
|
||||
module_root: self.root.clone(),
|
||||
entry_module: "index.mq.md".to_string(),
|
||||
manifest: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a `std:<relative>` target inside the bundle.
|
||||
pub fn resolve(&self, target: &str) -> Result<ResolvedModule, String> {
|
||||
let mut normalised = PathBuf::new();
|
||||
for component in Path::new(target).components() {
|
||||
match component {
|
||||
Component::Normal(part) => normalised.push(part),
|
||||
Component::CurDir => {}
|
||||
Component::ParentDir | Component::RootDir | Component::Prefix(_) => {
|
||||
return Err(format!(
|
||||
"stdlib import `{target}` must be a relative path inside the stdlib"
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
if normalised.as_os_str().is_empty() {
|
||||
return Err("stdlib import must name a module".to_string());
|
||||
}
|
||||
if normalised.extension().is_none() {
|
||||
normalised.set_extension("mq.md");
|
||||
}
|
||||
let candidate = self.root.join(&normalised);
|
||||
let canonical = candidate.canonicalize().map_err(|error| {
|
||||
format!(
|
||||
"cannot open stdlib module `{target}` at `{}`: {error}",
|
||||
candidate.display()
|
||||
)
|
||||
})?;
|
||||
let relative = canonical
|
||||
.strip_prefix(&self.root)
|
||||
.map_err(|_| format!("stdlib import `{target}` escapes the stdlib root"))?;
|
||||
let relative_path = relative
|
||||
.components()
|
||||
.map(|component| component.as_os_str().to_string_lossy().to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("/");
|
||||
Ok(ResolvedModule {
|
||||
package: self.package(),
|
||||
relative_path,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Content digest over all `.mq.md` files: sorted path, NUL, bytes, NUL.
|
||||
pub fn tree_digest(root: &Path) -> Result<String, String> {
|
||||
let mut files = Vec::new();
|
||||
collect_markdown(root, root, &mut files)?;
|
||||
files.sort();
|
||||
let mut hasher = Sha256::new();
|
||||
for (relative, path) in files {
|
||||
let bytes = std::fs::read(&path)
|
||||
.map_err(|error| format!("cannot read `{}`: {error}", path.display()))?;
|
||||
hasher.update(relative.as_bytes());
|
||||
hasher.update([0]);
|
||||
hasher.update(&bytes);
|
||||
hasher.update([0]);
|
||||
}
|
||||
Ok(format!("sha256:{}", hex(&hasher.finalize())))
|
||||
}
|
||||
|
||||
fn collect_markdown(
|
||||
root: &Path,
|
||||
dir: &Path,
|
||||
out: &mut Vec<(String, PathBuf)>,
|
||||
) -> Result<(), String> {
|
||||
let entries = std::fs::read_dir(dir)
|
||||
.map_err(|error| format!("cannot read `{}`: {error}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|error| format!("cannot read `{}`: {error}", dir.display()))?;
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
collect_markdown(root, &path, out)?;
|
||||
} else if path
|
||||
.file_name()
|
||||
.is_some_and(|name| name.to_string_lossy().ends_with(".mq.md"))
|
||||
{
|
||||
let relative = path
|
||||
.strip_prefix(root)
|
||||
.map_err(|_| "stdlib file escaped its root".to_string())?
|
||||
.components()
|
||||
.map(|component| component.as_os_str().to_string_lossy().to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("/");
|
||||
out.push((relative, path));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn hex(bytes: &[u8]) -> String {
|
||||
let mut out = String::with_capacity(bytes.len() * 2);
|
||||
for byte in bytes {
|
||||
out.push_str(&format!("{byte:02x}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn dev_bundle() -> PathBuf {
|
||||
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("..")
|
||||
.join("..")
|
||||
.join("stdlib")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dev_bundle_verifies_and_resolves() {
|
||||
let bundle = StdlibBundle::load(dev_bundle()).unwrap();
|
||||
assert!(bundle.digest.starts_with("sha256:"));
|
||||
let io = bundle.resolve("io").unwrap();
|
||||
assert_eq!(io.relative_path, "io.mq.md");
|
||||
assert!(io.package.instance_id.starts_with("std:marqdo-stdlib@"));
|
||||
assert!(bundle.resolve("../escape").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn digest_tamper_is_rejected() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let source = dev_bundle();
|
||||
for entry in std::fs::read_dir(&source).unwrap() {
|
||||
let entry = entry.unwrap();
|
||||
if entry.path().is_file() {
|
||||
std::fs::copy(entry.path(), dir.path().join(entry.file_name())).unwrap();
|
||||
}
|
||||
}
|
||||
std::fs::write(dir.path().join("io.mq.md"), "## tampered\n").unwrap();
|
||||
let error = StdlibBundle::load(dir.path()).unwrap_err();
|
||||
assert!(error.contains("digest mismatch"), "{error}");
|
||||
}
|
||||
}
|
||||
@@ -26,7 +26,7 @@ fn stderr(output: &Output) -> String {
|
||||
|
||||
#[test]
|
||||
fn mqmd_stdlib_print_output_matches_builtin_print() {
|
||||
let output = cli(&["run", "examples/stdlib/demo.mq.md"]);
|
||||
let output = cli(&["run", "stdlib/demo.mq.md"]);
|
||||
assert_eq!(output.status.code(), Some(0), "{}", stderr(&output));
|
||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||
let lines: Vec<&str> = stdout.lines().collect();
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
//! Bundled stdlib (`std:`) and frozen-CLI closure tests (CM-A10).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
|
||||
fn repo_root() -> PathBuf {
|
||||
Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.parent()
|
||||
.unwrap()
|
||||
.parent()
|
||||
.unwrap()
|
||||
.to_path_buf()
|
||||
}
|
||||
|
||||
fn binary() -> &'static str {
|
||||
env!("CARGO_BIN_EXE_marqdo")
|
||||
}
|
||||
|
||||
fn stdlib_dir() -> PathBuf {
|
||||
repo_root().join("stdlib")
|
||||
}
|
||||
|
||||
fn run(dir: &Path, stdlib: &Path, args: &[&str]) -> std::process::Output {
|
||||
Command::new(binary())
|
||||
.args(args)
|
||||
.current_dir(dir)
|
||||
.env("MARQDO_STDLIB_DIR", stdlib)
|
||||
.output()
|
||||
.expect("run marqdo")
|
||||
}
|
||||
|
||||
struct TempDir(PathBuf);
|
||||
|
||||
impl TempDir {
|
||||
fn new(tag: &str) -> Self {
|
||||
let path = std::env::temp_dir().join(format!("marqdo-stdlib-{}-{tag}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&path);
|
||||
std::fs::create_dir_all(&path).unwrap();
|
||||
Self(path)
|
||||
}
|
||||
fn path(&self) -> &Path {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.0);
|
||||
}
|
||||
}
|
||||
|
||||
fn copy_stdlib(source: &Path, target: &Path) {
|
||||
std::fs::create_dir_all(target).unwrap();
|
||||
for entry in std::fs::read_dir(source).unwrap() {
|
||||
let entry = entry.unwrap();
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
copy_stdlib(&path, &target.join(entry.file_name()));
|
||||
} else {
|
||||
std::fs::copy(&path, target.join(entry.file_name())).unwrap();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn write_project(dir: &Path, program: &str) {
|
||||
std::fs::write(dir.join("index.mq.md"), program).unwrap();
|
||||
}
|
||||
|
||||
const IMPORT_IO: &str = r#"---
|
||||
import:
|
||||
io: "std:io"
|
||||
---
|
||||
|
||||
## main
|
||||
|
||||
> io.写值(["bundle", 1, 2.5])
|
||||
|
||||
*None*
|
||||
"#;
|
||||
|
||||
#[test]
|
||||
fn std_import_resolves_the_verified_bundle() {
|
||||
let dir = TempDir::new("resolve");
|
||||
write_project(dir.path(), IMPORT_IO);
|
||||
let output = run(dir.path(), &stdlib_dir(), &["run", "index.mq.md"]);
|
||||
assert_eq!(
|
||||
output.status.code(),
|
||||
Some(0),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
assert_eq!(
|
||||
String::from_utf8_lossy(&output.stdout),
|
||||
"[\"bundle\", 1, 2.5]\n"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn analysis_records_the_verified_stdlib_digest() {
|
||||
let dir = TempDir::new("analysis");
|
||||
write_project(dir.path(), IMPORT_IO);
|
||||
let target = dir.path().join("analysis.json");
|
||||
let output = run(
|
||||
dir.path(),
|
||||
&stdlib_dir(),
|
||||
&["check", "index.mq.md", "--emit", target.to_str().unwrap()],
|
||||
);
|
||||
assert_eq!(
|
||||
output.status.code(),
|
||||
Some(0),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
let analysis: serde_json::Value =
|
||||
serde_json::from_slice(&std::fs::read(&target).unwrap()).unwrap();
|
||||
let manifest: serde_json::Value =
|
||||
serde_json::from_slice(&std::fs::read(stdlib_dir().join("stdlib.json")).unwrap()).unwrap();
|
||||
assert_eq!(
|
||||
analysis["project"]["stdlib"]["version"],
|
||||
manifest["version"]
|
||||
);
|
||||
assert_eq!(analysis["project"]["stdlib"]["digest"], manifest["digest"]);
|
||||
assert!(analysis["project"]["stdlib"]["instance_id"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.starts_with("std:marqdo-stdlib@"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn frozen_binary_hash_is_unchanged_after_stdlib_execution() {
|
||||
let before = std::fs::read(binary()).unwrap();
|
||||
let before_hash = marqdo_value_sha(&before);
|
||||
let dir = TempDir::new("frozen");
|
||||
write_project(dir.path(), IMPORT_IO);
|
||||
let output = run(dir.path(), &stdlib_dir(), &["run", "index.mq.md"]);
|
||||
assert_eq!(output.status.code(), Some(0));
|
||||
let after = std::fs::read(binary()).unwrap();
|
||||
assert_eq!(before_hash, marqdo_value_sha(&after));
|
||||
}
|
||||
|
||||
fn marqdo_value_sha(bytes: &[u8]) -> String {
|
||||
use sha2::{Digest, Sha256};
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(bytes);
|
||||
hasher
|
||||
.finalize()
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tampered_bundle_is_rejected_before_module_loading() {
|
||||
let dir = TempDir::new("tamper");
|
||||
let bundle = dir.path().join("stdlib");
|
||||
copy_stdlib(&stdlib_dir(), &bundle);
|
||||
std::fs::write(bundle.join("io.mq.md"), "## tampered\n").unwrap();
|
||||
write_project(dir.path(), IMPORT_IO);
|
||||
let output = run(dir.path(), &bundle, &["run", "index.mq.md"]);
|
||||
assert!(!output.status.success());
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
assert!(stderr.contains("stdlib digest mismatch"), "{stderr}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bundle_uses_no_foreign_or_plugin_calls() {
|
||||
fn visit(dir: &Path, offenders: &mut Vec<String>) {
|
||||
for entry in std::fs::read_dir(dir).unwrap() {
|
||||
let entry = entry.unwrap();
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
visit(&path, offenders);
|
||||
} else if path.to_string_lossy().ends_with(".mq.md") {
|
||||
let source = std::fs::read_to_string(&path).unwrap();
|
||||
if source.contains("foreign.") || source.contains("plugin.") {
|
||||
offenders.push(path.display().to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut offenders = Vec::new();
|
||||
visit(&stdlib_dir(), &mut offenders);
|
||||
assert!(
|
||||
offenders.is_empty(),
|
||||
"stdlib bundle must not depend on foreign/plugin calls: {offenders:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_bundle_manifest_is_a_hard_error() {
|
||||
let dir = TempDir::new("missing");
|
||||
let empty = dir.path().join("empty-stdlib");
|
||||
std::fs::create_dir_all(&empty).unwrap();
|
||||
write_project(dir.path(), IMPORT_IO);
|
||||
let output = run(dir.path(), &empty, &["run", "index.mq.md"]);
|
||||
assert!(!output.status.success());
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
assert!(stderr.contains("stdlib"), "{stderr}");
|
||||
}
|
||||
+11
-2
@@ -123,6 +123,12 @@ This is still a **local VM lane**, not a self-hosted GitHub Actions security URL
|
||||
(`format.显示` → UTF-8 encode → `Memory` → `resource.write(stdout)`) matches
|
||||
builtin `打印` for the covered value set; builtin remains default and no
|
||||
alias is routed.
|
||||
- Bundled standard library (CM-A10, ADR-0028): `stdlib/` ships
|
||||
`format`/`text`/`io`; explicit `std:` imports resolve only inside the bundle;
|
||||
a SHA-256 digest over `*.mq.md` is verified before loading; Analysis records
|
||||
`project.stdlib`; a frozen-binary test asserts the CLI hash is unchanged after
|
||||
running a `std:io` project and that tampering is rejected. `打印` is still
|
||||
the builtin default.
|
||||
- Resource streams (RFC-CM CM-A06, ADR-0025): opaque host-issued `Resource`
|
||||
handles with `resource.open/read/write/close`; bytes move only through
|
||||
`Memory`; stdout write is capability/output-budget checked and project-file
|
||||
@@ -166,8 +172,8 @@ behavior. This table records only what is in this tree now.
|
||||
|---|---|---|
|
||||
| P0 freeze (ADRs, spec deltas, baseline) | **P1 + partial P2 slice done** | ADR-0022 freezes CM-A01/CM-A02/CM-A03; ADR-0023 freezes CM-A04 (minimal List/Map/Text/Table observation and construction); ADR-0024 freezes CM-A05 (recoverable `Result` map convention); `spec/expression.ebnf`, `spec/evaluation.md`, `spec/value-wire.md`, `spec/analysis-v2.md`, `spec/security.md` updated; existing AST/value goldens unchanged |
|
||||
| P1 Memory + place assignment | **Implemented locally** | `Value::Memory`, run-scoped `MemoryStore` (64 KiB pages, 64 MiB total / 16 MiB region), `memory.new/size/grow`, `m[i]` read, `m[i] = byte`; semantic fixtures 45–47, runtime negatives 13–18, `examples/memory-buffer.mq.md`; evidence [`evidence/2026-11-20-memory-p1.md`](evidence/2026-11-20-memory-p1.md) |
|
||||
| P2 Marqdo-written allocator/data structures + minimal structure observation | **PARTIAL, gate NOT passed** | `.mq.md`-only evidence: bump allocator with u32 codec, linked list, failed-allocation atomicity (48–50); CM-A04 structure primitives and a table filter (51–54, ADR-0023); a first-fit free-list allocator with block reuse, splitting, address-adjacent coalescing and `Result` maps (55, ADR-0024), a `Memory`-backed binary tree with recursive insert/in-order traversal (56), a `.mq.md` CSV row splitter (57), an open-addressing hash table with linear probing, update and `hash.full` (58), a `.mq.md` UTF-8 encoder over `Memory` (59), a validating UTF-8 decoder rejecting truncated, bad-continuation and surrogate/overlong input (60), a `Memory`-backed graph adjacency list (61), a recursive JSON parser for nested arrays/objects, strings with escapes, integers, booleans, null and error Results (62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (63), big-endian 16/32/64-bit plus LEB128 integer codecs (64), a `.mq.md` table sort rebuilding a new table from `table.row/columns/cell` (65), a table join (67), a quoted/escaped CSV row parser (68), table group (69), and an Int matrix multiply (70); the first-fit allocator also detects double free (`alloc.double_free`). Evidence: [`memory-p2-closure`](evidence/2026-11-20-memory-p2-closure.md), [`structure-api-p2`](evidence/2026-11-20-structure-api-p2.md), [`result-and-free-list-p2`](evidence/2026-11-20-result-and-free-list-p2.md). P2 remaining: a separate stdlib repository/package and the frozen-CLI closure lane. C01–C06/C11/C12 now each have `.mq.md` evidence, including JSON floats and a JSON formatter. |
|
||||
| P3 Resource/Stream ABI | **PARTIAL** | ADR-0025/0026 CM-A06 slices: opaque `Resource`, `resource.open/read/write/close`, stdout write in pure/project, granted project-file read and buffered project-file write in project, EOF/closed/handle errors, wire/host rejection; strict refuses all opens. Fixture 66 + CLI/host tests. CM-A07 is **Compat-A only**: `examples/stdlib` provides an explicitly imported `.mq.md` `format/utf8/io` chain whose output matches builtin `打印` for Unit/None/Bool/Int/Float/Text/List/Map/Table; builtin `打印` remains default, no alias routing, and Formula/Code/FunctionRef/PluginHandle formatting is still unobservable. Still missing: Compat-B/C, directories, network, generic provider registry. |
|
||||
| P2 Marqdo-written allocator/data structures + minimal structure observation | **PARTIAL, gate NOT passed** | `.mq.md`-only evidence: bump allocator with u32 codec, linked list, failed-allocation atomicity (48–50); CM-A04 structure primitives and a table filter (51–54, ADR-0023); a first-fit free-list allocator with block reuse, splitting, address-adjacent coalescing and `Result` maps (55, ADR-0024), a `Memory`-backed binary tree with recursive insert/in-order traversal (56), a `.mq.md` CSV row splitter (57), an open-addressing hash table with linear probing, update and `hash.full` (58), a `.mq.md` UTF-8 encoder over `Memory` (59), a validating UTF-8 decoder rejecting truncated, bad-continuation and surrogate/overlong input (60), a `Memory`-backed graph adjacency list (61), a recursive JSON parser for nested arrays/objects, strings with escapes, integers, booleans, null and error Results (62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (63), big-endian 16/32/64-bit plus LEB128 integer codecs (64), a `.mq.md` table sort rebuilding a new table from `table.row/columns/cell` (65), a table join (67), a quoted/escaped CSV row parser (68), table group (69), and an Int matrix multiply (70); the first-fit allocator also detects double free (`alloc.double_free`). Evidence: [`memory-p2-closure`](evidence/2026-11-20-memory-p2-closure.md), [`structure-api-p2`](evidence/2026-11-20-structure-api-p2.md), [`result-and-free-list-p2`](evidence/2026-11-20-result-and-free-list-p2.md). P2 remaining: publication as a separate external stdlib repository. The bundled `stdlib/` tree, `std:` imports, digest verification, Analysis identity and frozen-CLI closure test are implemented (ADR-0028). C01–C06/C11/C12 each have `.mq.md` evidence, including JSON floats and a JSON formatter. |
|
||||
| P3 Resource/Stream ABI | **PARTIAL** | ADR-0025/0026 CM-A06 slices: opaque `Resource`, `resource.open/read/write/close`, stdout write in pure/project, granted project-file read and buffered project-file write in project, EOF/closed/handle errors, wire/host rejection; strict refuses all opens. Fixture 66 + CLI/host tests. CM-A07 is **Compat-A only**: the bundled `stdlib/` provides an explicitly imported `.mq.md` `format/utf8/io` chain (`std:` imports) whose output matches builtin `打印` for Unit/None/Bool/Int/Float/Text/List/Map/Table; builtin `打印` remains default, no alias routing, and Formula/Code/FunctionRef/PluginHandle formatting is still unobservable. Still missing: Compat-B/C, directories, network, generic provider registry. |
|
||||
| P4 Table logical/physical migration | **NOT IMPLEMENTED** | `Value::Table` remains `Vec<Vec<Value>>`; no stdlib table algorithms |
|
||||
| P5 third-party Provider / TCP / HTTP | **NOT IMPLEMENTED** | `fs.write`, `net.request`, `process`, `clock`, `random` remain unavailable broker services (see §4) |
|
||||
| P6 performance / native backends | **NOT IMPLEMENTED** | no bytecode/JIT/GPU/QPU work |
|
||||
@@ -193,6 +199,9 @@ user-written allocator is explicitly **not** claimed.
|
||||
- [`evidence/2026-11-20-stdlib-print-compat-a.md`](evidence/2026-11-20-stdlib-print-compat-a.md)
|
||||
— CM-A07 Compat-A `.mq.md` print prototype, side-by-side output comparison,
|
||||
covered/uncovered value types and the Compat-B blocker.
|
||||
- [`evidence/2026-11-20-bundled-stdlib-cm-a10.md`](evidence/2026-11-20-bundled-stdlib-cm-a10.md)
|
||||
— CM-A10 bundled stdlib, `std:` imports, digest verification, Analysis
|
||||
identity and frozen-binary closure test.
|
||||
- [`evidence/RELEASE_CANDIDATE_0.1.0_8af7262.md`](evidence/RELEASE_CANDIDATE_0.1.0_8af7262.md)
|
||||
— G0..G12 matrix and RC decision.
|
||||
- [`security-gate.md`](security-gate.md) — mechanism, attack matrix and strict
|
||||
|
||||
@@ -40,13 +40,13 @@ claim CM-A07 complete.
|
||||
|
||||
### 2. Compat-A implementation
|
||||
|
||||
Add a normal `.mq.md` module package under `examples/stdlib/`:
|
||||
Add a normal `.mq.md` module package (later moved to the bundled `stdlib/` tree by ADR-0028):
|
||||
|
||||
```text
|
||||
examples/stdlib/format.mq.md # 显示 / 列表显示 / 映射显示 / 表格显示 / 转义文本
|
||||
examples/stdlib/text.mq.md # utf8编码 (Memory bytes)
|
||||
examples/stdlib/io.mq.md # 写文本 / 写值 (resource.open stdout + write/close)
|
||||
examples/stdlib/demo.mq.md # side-by-side comparison with 打印
|
||||
stdlib/format.mq.md # 显示 / 列表显示 / 映射显示 / 表格显示 / 转义文本
|
||||
stdlib/text.mq.md # utf8编码 (Memory bytes)
|
||||
stdlib/io.mq.md # 写文本 / 写值 (resource.open stdout + write/close)
|
||||
stdlib/demo.mq.md # side-by-side comparison with 打印
|
||||
```
|
||||
|
||||
`io.写值(值)` = `format.显示(值) + "\n"` → `text.utf8编码` → `Memory` →
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# ADR-0028: Bundled standard library, `std:` imports and digest verification (CM-A10)
|
||||
|
||||
Status: **Accepted**
|
||||
Date: 2026-11-20
|
||||
Relates to: `docs/MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md`
|
||||
(CM-A10, CM-A07, P2/P3), ADR-0027
|
||||
Specs: `spec/analysis-v2.md`, `spec/references.md`
|
||||
Evidence: `docs/evidence/2026-11-20-bundled-stdlib-cm-a10.md`
|
||||
|
||||
## Context
|
||||
|
||||
CM-A07 Compat-A put a real `.mq.md` print chain under `examples/stdlib/`, but it
|
||||
was reachable only by relative path from a demo in the same directory. A
|
||||
standard library that ordinary projects can use offline needs:
|
||||
|
||||
* a fixed bundle shipped with the CLI;
|
||||
* an explicit import spelling that cannot be shadowed by a project file;
|
||||
* a content identity so a different bundle with the same version cannot
|
||||
masquerade;
|
||||
* no network, Git or package installation required at run time.
|
||||
|
||||
The RFC requires the official distribution to carry a tested stdlib version
|
||||
offline, while allowing independent releases. This ADR freezes the **bundled
|
||||
first slice**; it does not create the external `cflmy/marqdo-stdlib`
|
||||
repository or a public publishing workflow.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Bundle location
|
||||
|
||||
The official bundle is the `stdlib/` tree shipped next to the CLI (or vendored
|
||||
at the repository root during development). Discovery order:
|
||||
|
||||
1. `MARQDO_STDLIB_DIR` — explicit override; if set and invalid, loading fails
|
||||
hard (no silent fallback);
|
||||
2. `<executable-dir>/stdlib` if it exists;
|
||||
3. the development tree `<repo>/stdlib` for source builds.
|
||||
|
||||
If no bundle is found, a project that uses `std:` fails with a clear error; a
|
||||
project that does not use `std:` is unaffected. No network, Git or package
|
||||
manager is consulted for a bundled stdlib.
|
||||
|
||||
### 2. Explicit `std:` imports only
|
||||
|
||||
Front-matter imports may name a stdlib module as:
|
||||
|
||||
```yaml
|
||||
import:
|
||||
io: "std:io"
|
||||
fmt: "std:format"
|
||||
parser: "std:data/json"
|
||||
```
|
||||
|
||||
The target after `std:` is a **package-relative** path inside the bundle.
|
||||
`..`, absolute paths and `std:` targets that escape the bundle are rejected;
|
||||
`.mq.md` is appended when the target has no extension. `std:` is resolved
|
||||
before relative/alias imports and never searches the project or dependencies.
|
||||
There is no implicit `std` namespace, no global name, and no auto-import.
|
||||
|
||||
Module ids use a stable package instance:
|
||||
|
||||
```text
|
||||
std:marqdo-stdlib@<version>#<sha256-hex>
|
||||
```
|
||||
|
||||
so the same logical module from a different bundle is a different instance.
|
||||
|
||||
### 3. Content digest
|
||||
|
||||
`stdlib/stdlib.json` carries the name, version and digest. The digest is:
|
||||
|
||||
```text
|
||||
sha256( for each sorted *.mq.md relative path:
|
||||
path_utf8, NUL, file_bytes, NUL )
|
||||
```
|
||||
|
||||
It excludes `stdlib.json`, README files and non-`.mq.md` files. The loader
|
||||
recomputes it before loading any stdlib module and fails closed on mismatch or
|
||||
a missing manifest. A bundle is therefore identified by **name + version +
|
||||
exact markdown bytes**, not by a path or a version string alone.
|
||||
|
||||
### 4. Visibility and analysis
|
||||
|
||||
Analysis v2 records the verified bundle identity under `project.stdlib`:
|
||||
|
||||
```json
|
||||
{"version":"0.1.0","digest":"sha256:<hex>",
|
||||
"instance_id":"std:marqdo-stdlib@0.1.0#<hex>"}
|
||||
```
|
||||
|
||||
It is `null` when no bundle was available. A project that imports `std:` always
|
||||
has a non-null identity. Trace/diagnostics keep the real stdlib source spans;
|
||||
the bundle is normal Markdown, not a special-cased module type.
|
||||
|
||||
### 5. Frozen-CLI closure lane
|
||||
|
||||
The first closure test builds the CLI once, hashes the executable, runs a
|
||||
project importing `std:io`, then re-hashes the executable and asserts equality.
|
||||
It also asserts:
|
||||
|
||||
* the project uses only `.mq.md` and standard imports, no `foreign`/`plugin`
|
||||
calls;
|
||||
* a tampered bundle fails the digest check before any module executes;
|
||||
* Analysis records the verified digest.
|
||||
|
||||
This is a local gate, not a replacement for a remote CI lane.
|
||||
|
||||
### 6. Versioning
|
||||
|
||||
The bundle version is `0.1.0` for this slice. A CLI may load a bundle whose
|
||||
version is declared compatible by this ADR; the RFC's full SemVer/digest
|
||||
contract and independent release workflow remain CM-A10 follow-up. A future
|
||||
core feature that the bundle requires must be reflected in both the bundle
|
||||
version and the CLI's compatibility table before release.
|
||||
|
||||
## Non-goals
|
||||
|
||||
* Creating/publishing the external `cflmy/marqdo-stdlib` repository or a
|
||||
package registry entry.
|
||||
* Automatic download/update, Git fetching, or manifest-driven stdlib
|
||||
installation.
|
||||
* A second plugin/provider system or a second module search path.
|
||||
* Compat-B/C name routing of `打印` (ADR-0027) — the bundle does not change
|
||||
builtin resolution.
|
||||
* Windows/macOS packaging layout; only the source-tree/executable-adjacent
|
||||
layout is exercised locally.
|
||||
|
||||
## Consequences
|
||||
|
||||
* Any project can `import: {io: "std:io"}` and use the print prototype
|
||||
offline; `打印` remains the default builtin.
|
||||
* A changed stdlib file without a digest update is rejected before execution.
|
||||
* The bundle is still an in-repo vendored package. Independent repository
|
||||
release and CI signing are explicitly tracked as remaining CM-A10 work.
|
||||
* Analysis artifacts and a frozen binary can be tied to the exact stdlib
|
||||
digest, which is the prerequisite for Compat-B and reproducible releases.
|
||||
|
||||
## Verification
|
||||
|
||||
`docs/evidence/2026-11-20-bundled-stdlib-cm-a10.md` records the `std:` import
|
||||
test, digest-tamper rejection, Analysis identity assertion, no-foreign policy
|
||||
scan and frozen binary hash equality.
|
||||
@@ -6,7 +6,7 @@ normative RFC text stays at its reviewable path under `docs/`.
|
||||
| Document | Status | Implemented part |
|
||||
|---|---|---|
|
||||
| [RFC-CM-0001: compute machine, addressable memory and system boundary](../MARQDO_COMPUTE_MACHINE_RFC_0001.md) | Draft / design input | R1 (P1): `Memory` value, `memory.new/size/grow`, byte read/write, Memory-index assignment, budgets, wire/host rejection. Implemented per [ADR-0022](../adr/0022-addressable-memory-and-place-assignment.md); see [STATUS §5](../STATUS.md). |
|
||||
| [RFC-CM-0002: minimal core, stdlib self-hosting and system capability migration](../MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md) | Draft / design input | P0/P1, a **partial P2**, and a **partial P3** first slice (ADR-0025/0026: stdout write, granted project-file read and buffered project-file write via `resource.*`; strict refuses all opens; ADR-0027 Compat-A `.mq.md` print prototype, builtin still default). P2: fixtures 48–50 (`.mq.md` bump allocator, u32 codec, linked list/traversal, failed-allocation atomicity), CM-A04 structure primitives for List/Map/Text/Table (ADR-0023, fixtures 51–54), and a `.mq.md` first-fit free-list allocator with block splitting, address-adjacent coalescing and `Result` maps (ADR-0024, fixture 55), plus a `Memory`-backed binary tree with recursive insert/in-order traversal (fixture 56), a `.mq.md` CSV row splitter (fixture 57), an open-addressing hash table with linear probing (fixture 58), a `.mq.md` UTF-8 encoder and validating decoder (`utf8.truncated`/`invalid_continuation`/`invalid_scalar`) (fixtures 59–60), a `Memory`-backed graph adjacency list (fixture 61), a recursive JSON parser/formatter for arrays, objects, strings with escapes, ints/floats/booleans/null and error Results (fixture 62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (fixture 63), big-endian 16/32/64-bit plus LEB128 integer codecs (fixture 64), a `.mq.md` table sort rebuilding the table without mutating the input (fixture 65), a table join built from `table.row/columns/cell` (fixture 67), and a quoted/escaped CSV row parser (fixture 68), table group (fixture 69) and an Int matrix multiply (fixture 70). No separate stdlib repo, network/generic providers, P4 Table migration, P5 providers or P6 performance work. |
|
||||
| [RFC-CM-0002: minimal core, stdlib self-hosting and system capability migration](../MARQDO_COMPUTE_CORE_STDLIB_MIGRATION_RFC_0002.md) | Draft / design input | P0/P1, a **partial P2**, and a **partial P3** first slice (ADR-0025/0026: stdout write, granted project-file read and buffered project-file write via `resource.*`; strict refuses all opens; ADR-0027 Compat-A `.mq.md` print prototype, builtin still default). P2: fixtures 48–50 (`.mq.md` bump allocator, u32 codec, linked list/traversal, failed-allocation atomicity), CM-A04 structure primitives for List/Map/Text/Table (ADR-0023, fixtures 51–54), and a `.mq.md` first-fit free-list allocator with block splitting, address-adjacent coalescing and `Result` maps (ADR-0024, fixture 55), plus a `Memory`-backed binary tree with recursive insert/in-order traversal (fixture 56), a `.mq.md` CSV row splitter (fixture 57), an open-addressing hash table with linear probing (fixture 58), a `.mq.md` UTF-8 encoder and validating decoder (`utf8.truncated`/`invalid_continuation`/`invalid_scalar`) (fixtures 59–60), a `Memory`-backed graph adjacency list (fixture 61), a recursive JSON parser/formatter for arrays, objects, strings with escapes, ints/floats/booleans/null and error Results (fixture 62), a `Memory`-backed stack bytecode VM with PUSH/ADD/MUL/JMP/JZ/HALT (fixture 63), big-endian 16/32/64-bit plus LEB128 integer codecs (fixture 64), a `.mq.md` table sort rebuilding the table without mutating the input (fixture 65), a table join built from `table.row/columns/cell` (fixture 67), and a quoted/escaped CSV row parser (fixture 68), table group (fixture 69) and an Int matrix multiply (fixture 70). No separate external stdlib repo, network/generic providers, P4 Table migration, P5 providers or P6 performance work. |
|
||||
|
||||
The RFCs are architecture constraints, not evidence that a feature exists. The
|
||||
implementation status lives in `docs/STATUS.md`; frozen behavior is in
|
||||
@@ -24,7 +24,8 @@ implementation status lives in `docs/STATUS.md`; frozen behavior is in
|
||||
| CM-A06 minimal Resource/Stream ABI | [0025](../adr/0025-minimal-resource-stream-abi.md) | Accepted (first slice) |
|
||||
| CM-A06 buffered project-file write | [0026](../adr/0026-buffered-project-file-write-streams.md) | Accepted |
|
||||
| CM-A07 print migration staging / Compat-A | [0027](../adr/0027-print-migration-compat-a.md) | Accepted (Compat-A only) |
|
||||
| CM-A07 Compat-B/C, stdlib packaging, Table storage, generic providers, strict resources | — | Not decided; do not implement by implication |
|
||||
| CM-A10 bundled stdlib / `std:` imports / digest | [0028](../adr/0028-bundled-stdlib-and-std-imports.md) | Accepted (first slice) |
|
||||
| CM-A07 Compat-B/C, Table storage, generic providers, strict resources | — | Not decided; do not implement by implication |
|
||||
|
||||
## What a future phase must not do
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# RFC-CM CM-A10 evidence: bundled stdlib, `std:` imports and digest
|
||||
|
||||
Date: 2026-11-20 (Asia/Shanghai)
|
||||
Repository: `marqdo-cli`
|
||||
Baseline: `92bfbf2` plus this step's changes.
|
||||
Decision record: [ADR-0028](../adr/0028-bundled-stdlib-and-std-imports.md)
|
||||
Related: [Compat-A](2026-11-20-stdlib-print-compat-a.md), [STATUS §5](../STATUS.md).
|
||||
Scope: bundled first slice; **no external repository publication**.
|
||||
|
||||
## Environment and gate
|
||||
|
||||
Ubuntu 26.04 development container, Linux x86_64, Rust 1.100.0.
|
||||
|
||||
```text
|
||||
cargo fmt --all -- --check PASS
|
||||
cargo clippy --workspace --all-targets -- -D warnings PASS
|
||||
cargo test --workspace --locked PASS
|
||||
415 passed, 0 failed, 2 ignored (existing plugin-sdk doc-tests)
|
||||
```
|
||||
|
||||
No GitHub CI is available from this checkout; this is local evidence.
|
||||
|
||||
## Bundle
|
||||
|
||||
`stdlib/` is the official offline bundle:
|
||||
|
||||
```text
|
||||
stdlib/
|
||||
stdlib.json # name, version 0.1.0, sha256 digest
|
||||
README.md
|
||||
format.mq.md # 显示 / 列表显示 / 映射显示 / 表格显示 / 转义文本
|
||||
text.mq.md # utf8编码 into Memory
|
||||
io.mq.md # 写文本 / 写值 via resource.write(stdout)
|
||||
demo.mq.md # builtin 打印 comparison
|
||||
tests/io.mq.md # std:io self-test
|
||||
```
|
||||
|
||||
Digest scheme: SHA-256 over sorted `*.mq.md` relative paths, NUL, file bytes,
|
||||
NUL. `stdlib.json` and non-`.mq.md` files are excluded. The loader recomputes
|
||||
the digest before loading any stdlib module and fails closed on mismatch or a
|
||||
missing manifest.
|
||||
|
||||
`std:<relative>` is an explicit front-matter import target; it resolves only
|
||||
inside the bundle, rejects `..`/absolute paths, appends `.mq.md` when absent,
|
||||
and uses package instance
|
||||
`std:marqdo-stdlib@0.1.0#<sha256-hex>`.
|
||||
|
||||
## Tests
|
||||
|
||||
`crates/cli/tests/stdlib_bundle.rs` (6):
|
||||
|
||||
```text
|
||||
std_import_resolves_the_verified_bundle
|
||||
index.mq.md imports std:io; io.写值(["bundle",1,2.5])
|
||||
-> ["bundle", 1, 2.5]
|
||||
|
||||
analysis_records_the_verified_stdlib_digest
|
||||
check --emit contains project.stdlib.version/digest/instance_id
|
||||
matching stdlib.json
|
||||
|
||||
tampered_bundle_is_rejected_before_module_loading
|
||||
copy bundle, modify io.mq.md
|
||||
-> non-zero, "stdlib digest mismatch"
|
||||
|
||||
missing_bundle_manifest_is_a_hard_error
|
||||
empty MARQDO_STDLIB_DIR -> non-zero stdlib error
|
||||
|
||||
bundle_uses_no_foreign_or_plugin_calls
|
||||
scans all stdlib *.mq.md for foreign./plugin. -> none
|
||||
|
||||
frozen_binary_hash_is_unchanged_after_stdlib_execution
|
||||
sha256(marqdo binary) before == after running the std:io project
|
||||
```
|
||||
|
||||
CLI unit tests in `crates/cli/src/stdlib.rs` (2): dev bundle resolves and
|
||||
verifies; a copied/tampered bundle is rejected.
|
||||
|
||||
Existing `mqmd_stdlib_print_output_matches_builtin_print` still passes with the
|
||||
demo moved to `stdlib/demo.mq.md`.
|
||||
|
||||
## Analysis identity
|
||||
|
||||
Analysis v2 project object now carries:
|
||||
|
||||
```json
|
||||
{"stdlib":{"version":"0.1.0",
|
||||
"digest":"sha256:<64 hex>",
|
||||
"instance_id":"std:marqdo-stdlib@0.1.0#<64 hex>"}}
|
||||
```
|
||||
|
||||
It is `null` when no bundle is available. The digest is verified before load,
|
||||
so an Analysis artifact cannot name mismatched stdlib bytes.
|
||||
|
||||
## What is not done
|
||||
|
||||
* No separate `cflmy/marqdo-stdlib` repository, release tag, package registry
|
||||
or CI-signed digest. The bundle is vendored in this repository and copied
|
||||
next to the executable in a release layout by later packaging work.
|
||||
* No CM-A07 Compat-B/C routing; `打印` is still the builtin default.
|
||||
* No Windows/macOS bundle layout evidence and no remote CI URL.
|
||||
* No independent stdlib upgrade workflow; changing a module requires updating
|
||||
`stdlib.json.digest`, and an incompatible core requirement is not yet
|
||||
enforced by a version table.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
export PATH=/root/.cargo/bin:$PATH
|
||||
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
|
||||
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
|
||||
cargo test -p marqdo-cli --test stdlib_bundle
|
||||
cargo run -q -p marqdo-cli -- run examples/stdlib-bundle.mq.md
|
||||
cargo test --workspace --locked
|
||||
```
|
||||
@@ -24,7 +24,7 @@ No GitHub CI is available from this checkout; this is local evidence.
|
||||
|
||||
## Prototype
|
||||
|
||||
Ordinary `.mq.md` modules under `examples/stdlib/`:
|
||||
Ordinary `.mq.md` modules (now the bundled `stdlib/` tree, ADR-0028):
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
@@ -50,7 +50,7 @@ The module is loaded only through an explicit Markdown import alias.
|
||||
## Output comparison
|
||||
|
||||
```text
|
||||
$ marqdo run examples/stdlib/demo.mq.md
|
||||
$ marqdo run stdlib/demo.mq.md
|
||||
["文本", 2, 1.5, [3, true], {a: 1, b: "x"}, None]
|
||||
["文本", 2, 1.5, [3, true], {a: 1, b: "x"}, None]
|
||||
Table[名称 | 分数]{甲 | 87; 乙 | 92}
|
||||
@@ -88,7 +88,7 @@ extension, not part of this slice. This is the explicit blocker for Compat-B.
|
||||
export PATH=/root/.cargo/bin:$PATH
|
||||
export CARGO_HOME=/root/work/marqdo-cli/.cache/cargo
|
||||
export CARGO_TARGET_DIR=/root/work/marqdo-cli/.cache/target
|
||||
cargo run -q -p marqdo-cli -- run examples/stdlib/demo.mq.md
|
||||
cargo run -q -p marqdo-cli -- run stdlib/demo.mq.md
|
||||
cargo test -p marqdo-cli --test examples_r6
|
||||
cargo test --workspace --locked
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
import:
|
||||
io: "std:io"
|
||||
---
|
||||
|
||||
## main
|
||||
|
||||
> io.写值(["bundle", 1, 2.5])
|
||||
|
||||
*None*
|
||||
+11
-1
@@ -31,7 +31,12 @@ Analysis v1. v1 was never released; it is kept as history and is not written.
|
||||
"project": {
|
||||
"root_id": "root:quantum@0.1.0",
|
||||
"identity_kind": "manifest",
|
||||
"lock_digest": "sha256:<64 hex>"
|
||||
"lock_digest": "sha256:<64 hex>",
|
||||
"stdlib": {
|
||||
"version": "0.1.0",
|
||||
"digest": "sha256:<64 hex>",
|
||||
"instance_id": "std:marqdo-stdlib@0.1.0#<64 hex>"
|
||||
}
|
||||
},
|
||||
"modules": [],
|
||||
"diagnostics": []
|
||||
@@ -48,6 +53,11 @@ Analysis v1. v1 was never released; it is kept as history and is not written.
|
||||
`project.lock_digest` is the SHA-256 of the exact `marqdo.lock` bytes prefixed
|
||||
with `sha256:`, or `""` when there is no lock. It is not a logical identity.
|
||||
|
||||
`project.stdlib` records the verified bundled standard library identity used by
|
||||
this load: declared version, content digest and package instance id. It is
|
||||
`null` when no bundle was available. A digest mismatch is a hard load error, so
|
||||
an Analysis artifact can only name a bundle whose bytes matched the manifest.
|
||||
|
||||
`AnalysisDigest` is the lowercase hex SHA-256 of the exact UTF-8 bytes written
|
||||
by `check --emit` (pretty JSON). This is what Trace v2 binds to
|
||||
`source.analysis_digest`; an independent consumer can verify it by hashing the
|
||||
|
||||
+1
-1
@@ -278,7 +278,7 @@ host boundary. Names are stable; adding one requires a spec update.
|
||||
| `fs.read/write` | see `spec/security.md` | capability-gated |
|
||||
|
||||
The legacy `打印/print` builtin remains the default and still uses the host
|
||||
output path. `examples/stdlib/` contains an explicitly imported `.mq.md`
|
||||
output path. `stdlib/` (the bundled standard library, `std:` imports) contains an explicitly imported `.mq.md`
|
||||
`format/utf8/io` prototype (ADR-0027 Compat-A) whose output is comparison-tested
|
||||
against `打印` for Unit/None/Bool/Int/Float/Text/List/Map/Table; it does **not**
|
||||
replace the builtin, and Formula/Code/FunctionRef/PluginHandle formatting is not
|
||||
|
||||
+2
-1
@@ -146,11 +146,12 @@ Parsing rules (`crates/marqdo-markdown/src/frontmatter.rs::imports`):
|
||||
|
||||
### 3.1 Target forms
|
||||
|
||||
`Loader::resolve_import` recognizes exactly two forms:
|
||||
`Loader::resolve_import` recognizes exactly three forms:
|
||||
|
||||
| Target | Resolution |
|
||||
|---|---|
|
||||
| `@dependency` | Look up the alias after `@` in `marqdo.yaml` `dependencies`. The dependency must declare a **path** source. Resolve `project_root / path`; if it is a directory, append `index.mq.md`; canonicalize. A git-only dependency cannot be resolved in this build (`dependency.path` is `None` → unresolved). |
|
||||
| `std:<relative>` | Resolve inside the verified bundled standard library (ADR-0028). The target must stay inside the bundle; `..`/absolute paths are rejected; `.mq.md` is appended when absent. The bundle's declared digest is verified before any module loads. No network/package manager is consulted. |
|
||||
| any other string (intended: `./file.mq.md`) | Join the string to the **importing file's directory**, then canonicalize. There is no extension inference, no search path and no fallback to a package directory. |
|
||||
|
||||
The resulting file is loaded as a module. If it canonicalizes outside the
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# marqdo-stdlib (bundled first slice)
|
||||
|
||||
This directory is the official, offline `marqdo-stdlib` bundle consumed by
|
||||
`std:<path>` imports. The CLI verifies `stdlib.json.digest` before loading any
|
||||
module; a mismatch is a hard error.
|
||||
|
||||
- `format.mq.md` — value formatting (`显示`, list/map/table display, escaping)
|
||||
- `text.mq.md` — UTF-8 encoding into `Memory`
|
||||
- `io.mq.md` — `写文本` / `写值` through `resource.write(stdout)`
|
||||
- `demo.mq.md` — side-by-side comparison with builtin `打印`
|
||||
- `tests/` — bundle self-tests
|
||||
|
||||
The digest is a SHA-256 over sorted `*.mq.md` relative paths, a NUL, file
|
||||
bytes, and a NUL. It excludes `stdlib.json` and non-`.mq.md` files. After any
|
||||
module change, regenerate it with:
|
||||
|
||||
```bash
|
||||
python3 - stdlib <<'PY'
|
||||
import hashlib, json, sys
|
||||
from pathlib import Path
|
||||
root=Path(sys.argv[1])
|
||||
h=hashlib.sha256()
|
||||
for p in sorted(root.rglob('*.mq.md')):
|
||||
h.update(p.relative_to(root).as_posix().encode()); h.update(b'\0')
|
||||
h.update(p.read_bytes()); h.update(b'\0')
|
||||
manifest=json.loads((root/'stdlib.json').read_text())
|
||||
manifest['digest']='sha256:'+h.hexdigest()
|
||||
(root/'stdlib.json').write_text(json.dumps(manifest, indent=2)+'\n')
|
||||
PY
|
||||
```
|
||||
|
||||
The version/digest are part of each stdlib module's package instance id, so a
|
||||
different bundle cannot silently masquerade as the one a project was checked
|
||||
against.
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"name": "marqdo-stdlib",
|
||||
"version": "0.1.0",
|
||||
"digest": "sha256:000f39b01b3a1a7a8c32fe13d82db7f361166d1d7e8564d124b558f9b56d4e48"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
import:
|
||||
io: "std:io"
|
||||
---
|
||||
|
||||
## main
|
||||
|
||||
> io.写值("stdlib self-test")
|
||||
|
||||
*None*
|
||||
File renamed without changes.
+13
-1
@@ -171,9 +171,21 @@ by:
|
||||
* fixture 55 additionally has a second-free detection path: freeing an
|
||||
already-free block returns `{ok:false, code:"alloc.double_free"}`.
|
||||
|
||||
RFC-CM CM-A10 bundled stdlib (ADR-0028) is pinned by
|
||||
`crates/cli/tests/stdlib_bundle.rs` (6 tests):
|
||||
|
||||
* explicit `std:io` import resolves the verified bundle and runs;
|
||||
* Analysis records `project.stdlib.version/digest/instance_id`;
|
||||
* a tampered `io.mq.md` is rejected with `stdlib digest mismatch` before any
|
||||
module executes;
|
||||
* a missing manifest is a hard error;
|
||||
* the bundle has no `foreign.`/`plugin.` calls;
|
||||
* the CLI binary hash is unchanged after running a `std:io` project
|
||||
(frozen-CLI closure).
|
||||
|
||||
RFC-CM CM-A07 Compat-A (ADR-0027) is pinned by:
|
||||
|
||||
* `examples/stdlib/{format,text,io}.mq.md` and `examples/stdlib/demo.mq.md`;
|
||||
* `stdlib/{format,text,io}.mq.md` and `stdlib/demo.mq.md`;
|
||||
* `crates/cli/tests/examples_r6.rs::mqmd_stdlib_print_output_matches_builtin_print`,
|
||||
which runs the demo and asserts `io.写值` and builtin `打印` lines are
|
||||
byte-identical for a nested List/Map/None value and a Table;
|
||||
|
||||
Reference in new issue
Block a user