Complete the documentation set for the M0.5 toolchain, grounded in the code: - spec/references.md: import/reference/call edges, identities, loader rules, anchors, `ref` resolution/ambiguity, `check --refs` validation. - spec/value-wire.md: exact wire_version 1 typed encoding, integer-as-decimal string rule, special floats, limits and exact float round-trip. - spec/security.md: modes (strict refuses without OS isolation), capabilities, path rules, budgets, tracing, foreign process caveats, plugin trust. - spec/package-format.md: marqdo.yaml/lock formats, validation, lock/sync, unsupported registry/git/environment work. - spec/plugin-abi.md: ABI v1 layout/ownership/negotiation, manifest/artifact rules, wire envelopes, loader lifecycle, trust boundary, SDK/C demo, and the implemented CLI plugin-invocation flow with its limitations. - spec/math.md and spec/foreign.md: the implemented math subset and the foreign request/response/timeout/capability contract. - docs/adr/0001..0007: accepted decisions for the span foundation, inert executables outside function bodies, ref ambiguity (S-07), strict refusal, AST normalization, CLI contract and BigInt/typed wire (S-08). - docs/contributor-guide.md: repo map, workflow, fixture/golden updates, quality gate, ADR/commit policy. - docs/STATUS.md: P0-P7 matrix with evidence, open items and the blueprint §18 Alpha checklist (conclusion: not yet Alpha-complete), including known divergences rather than overclaims. - README.md rewritten from "design-only starter" to the working implementation with quickstart, implemented phases, deliberate limits and doc index. - AGENTS.md: updated pointers/workflow only; architectural rules unchanged. - Consistency fixes in spec/mapping.md/evaluation.md (inert emph outside functions, unsupported math.simplify/differentiate).
9.2 KiB
Marqdo Value Wire Format v1
Status: frozen for M0.5. Changes require an ADR, a wire_version bump and
cross-ABI compatibility tests.
Canonical implementation: crates/marqdo-value/src/wire.rs
(to_wire, from_wire, from_wire_with_limits, WireLimits, WireError).
The plugin envelope that carries these values is defined in
crates/marqdo-plugin-abi/src/wire.rs and documented in
plugin-abi.md; the foreign-kernel stdin envelope is
documented in foreign.md and implemented in
crates/marqdo-foreign/src/lib.rs.
1. Why a typed wire format
JSON has one numeric type. Marqdo has arbitrary-precision Int, IEEE-754
Float (including NaN and both infinities) and 13 distinct runtime value
types (crates/marqdo-value/src/value.rs). Encoding an Int as a JSON number
would silently lose digits above 2^53 and would make Int(1) and
Float(1.0) indistinguishable after a round trip. The wire format is
therefore fully type-tagged and is not plain JSON.
The rules that follow from this:
- Every value is a JSON object with a mandatory
"type"string. - Integers never travel as JSON numbers.
Intis{"type":"int","decimal":"<base-10 string>"}, which preserves every digit, sign and leading-zero-free canonical form. Floatuses a JSON number only for finite values;NaN,Infinityand-Infinityuse an explicit"special"tag.- Unknown tags, missing fields, wrong JSON types and invalid payloads are structured decode errors, never silent coercions.
JSON numbers still appear in two non-value positions: plugin_handle.id (a
host generation counter) and span.start / span.end (byte offsets). Neither
is a user arithmetic value.
2. Version and entry points
WIRE_VERSION: u32 = 1(marqdo_value::WIRE_VERSION).to_wire(&Value) -> serde_json::Valueencodes.from_wire(&Json) -> Result<Value, WireError>decodes withWireLimits::default().from_wire_with_limits(&Json, WireLimits) -> Result<Value, WireError>decodes with explicit limits.- Decoding is recursive and never panics on untrusted input; every failure is
a
WireError:UnknownType(String)— missing/non-string/unknown"type";MissingField(&'static str)— required field absent;Invalid(String)— wrong JSON type, invalid decimal, bad table shape, unknown floatspecial, non-object value;LimitExceeded(&'static str)— nesting depth, list/map length, string or map-key length.
WireError implements Display but has no stable numeric code of its own.
Boundary crates wrap it in their own stable codes: the plugin layer uses
invalid_argument_wire / invalid_result_wire
(crates/marqdo-plugin-abi/src/wire.rs), and the foreign layer maps it to
Protocol (crates/marqdo-foreign/src/lib.rs).
3. Value encodings
The table is normative and mirrors encode/decode in
crates/marqdo-value/src/wire.rs.
| Marqdo type | Wire JSON | Notes |
|---|---|---|
Unit |
{"type":"unit"} |
No result; distinct from None. |
None |
{"type":"none"} |
Explicit empty value. |
Bool |
{"type":"bool","value":true} |
value must be a JSON boolean. |
Int |
{"type":"int","decimal":"-123..."} |
Decimal string; decoded with num_bigint::BigInt::from_str. No +, no separators, no hex. -0 decodes to 0 because BigInt has no signed zero. |
Float (finite) |
{"type":"float","value":1.5} |
JSON number decoded with as_f64(). -0.0 round-trips as a JSON negative-zero number and decodes to Float(-0.0). |
Float (NaN) |
{"type":"float","special":"nan"} |
Never a JSON number. |
Float (+∞) |
{"type":"float","special":"infinity"} |
|
Float (−∞) |
{"type":"float","special":"-infinity"} |
|
Text |
{"type":"text","value":"..."} |
UTF-8 JSON string. |
List |
{"type":"list","items":[<value>, ...]} |
Order preserved. |
Map |
{"type":"map","entries":{"key":<value>, ...}} |
Decoding rebuilds a BTreeMap, so key order is canonical (sorted) after a round trip. Duplicate keys cannot occur in a JSON object. |
Table |
{"type":"table","header":["a","b"],"rows":[[<value>,<value>], ...]} |
Header cells must be JSON strings; every row must have exactly header.len() cells or decoding fails. |
Formula |
{"type":"formula","latex":"...","display":false} plus optional "unit" (string) and "span":{"start":N,"end":M} |
display defaults to false on decode when absent. The formula is inert data; this encoding never evaluates it. |
Code |
{"type":"code","language":"python","source":"...","content_hash":"<sha256 hex>"} plus optional "unit" and "span" |
language and content_hash default to "" on decode when absent; source is required. |
FunctionRef |
{"type":"function_ref","module":"index.mq.md","name":"main"} |
Both fields required strings. |
PluginHandle |
{"type":"plugin_handle","plugin":"demo.solver","export":"solver.add","id":1} |
id must be a non-negative JSON integer (as_u64). |
Optional fields (unit, span, display, language, content_hash) are
emitted only when present on the value. Encoders always emit display for
Formula and language/content_hash/source for Code.
3.1 Round-trip guarantees
For the values covered by wire::tests::scalar_roundtrip,
from_wire(to_wire(v)) == v. Exact guarantees:
- all
Intdigits survive (big_integers_keep_all_digits); NaN/±Infinitysurvive with their sign/tag (special_floats_are_tagged);UnitandNonenever collapse into each other;Mapiteration order after decoding is sorted, which is the canonical order used byValue::display/repr.
Finite Float values are encoded as JSON numbers. The workspace enables
serde_json's float_roundtrip feature, so a finite binary64 value survives an
encode/decode cycle exactly (verified by the Python math.pi * 9 semantic
fixture: 28.274333882308138 stays 28.274333882308138). NaN/infinities are
tagged and exact. Use Text when a decimal string must be exchanged verbatim
rather than as a binary64 float.
4. Limits
WireLimits::default() (crates/marqdo-value/src/wire.rs):
| Limit | Default | Applies to |
|---|---|---|
max_depth |
64 | Nesting of values: the top-level object is depth 0, every child decoded through list.items, map.entries, table.rows or table.header cells is depth+1. A value at depth > 64 is rejected with LimitExceeded("nesting depth"). |
max_items |
1,000,000 | list.items length and map.entries key count. |
max_string_bytes |
16 MiB | text.value byte length and map key byte length. |
4.1 Known gaps in the current decoder
The following are not bounded by WireLimits in
crates/marqdo-value/src/wire.rs today and must be treated as open items
(they are also bounded indirectly at the ABI layer by envelope size limits):
table.headerlength,table.rowslength and the number of columns are not checked againstmax_items(only row/header width equality is checked);- table header cell strings are not checked against
max_string_bytes; formula.latex,code.source,function_ref.module/name,plugin_handle.plugin/exportare not checked againstmax_string_bytes;- a
plugin_handledecoded from wire data is a structurally reconstructedValue::PluginHandle; this build does not carry an unforgeable capability token. Host code must therefore treat a plugin handle as opaque intent and re-authorize any operation, not as proof of prior authorization. The header comment "cannot be forged from wire data" is aspirational for this point and should be fixed or reworded in a future ADR. - JSON itself may contain duplicate object keys;
serde_jsonkeeps the last one, somap.entrieswith duplicate keys is not rejected as an error.
5. Where the format is used
| Boundary | Envelope | Value positions |
|---|---|---|
| Plugin ABI v1 request/response | {"wire_version":1,"kind":"request"/"response",...} (crates/marqdo-plugin-abi/src/wire.rs) |
request.args is an object of name → typed value; response.value is one typed value. |
| Plugin host capability request/reply | opaque bytes, shape per capability | defined by the capability, not by this file. |
| Foreign kernel stdin | {"wire_version":1,"kind":"request","input":<typed value or null>} (crates/marqdo-foreign/src/lib.rs) |
input is a to_wire(value) result, or JSON null when absent. |
Foreign typed return (foreign.call) |
stdout must be exactly one typed value | decoded with from_wire and default limits. |
The --json output of run/call embeds "value": to_wire(outcome.value)
(crates/cli/src/commands.rs), so CLI JSON consumers see the same typed
representation.
6. Compatibility rules
wire_versionis part of the frozen contract. A decoder rejects a different version rather than guessing.- Adding a new
"type"tag requires a spec change and an ADR; removing or renaming a tag is a breaking change. - The plugin envelope independently carries
wire_version; do not assume the C ABI major version andwire_versionare the same number (seespec/plugin-abi.md§2). - All payloads are UTF-8 JSON. Endianness is irrelevant for the JSON layer, but the C ABI structs that carry these bytes are target-dependent.