Files
marqdo-cli/spec/execution-events-v1.md
T

15 KiB

Execution Events v1

Status: implemented by marqdo run/call --event-stream jsonl (P0 and P1 of RFC-OBS-0001). It is an independent public protocol: it is not Analysis v2, not Trace v2 and not Tool Protocol v1. The decision record is ADR-0059; the design input is RFC-OBS-0001.

1. Transport and negotiation

  • Media type: application/x-ndjson (JSON Lines). One complete UTF-8 JSON object per line, terminated by \n; no unescaped newline occurs inside an event.
  • The stream uses the CLI stdout as a dedicated machine channel. While --event-stream jsonl is active, the program's stdout and stderr are carried only as stdout.chunk / stderr.chunk events and are not written raw to stdout. Human diagnostics may still go to OS stderr.
  • --event-stream jsonl is available only on run and call. It is mutually exclusive with --json; check, inspect, fmt, tool and the other static commands reject the option. --trace-file may be combined.
  • --mode strict currently rejects --event-stream with a structured run.rejected (strict.unsupported_grant); there is no silent downgrade to project.
  • The consumer checks the negotiated protocol from stream.opened (schema + schema_version). Consumers must reject unknown major versions, and must not silently treat another major as v1.
  • A consumer that cannot read the stream (slow, closed pipe, killed process) may see a truncated final line or no terminal event. It must report incomplete; it must never synthesize a successful terminal event.

2. Envelope

Every normal line has exactly these top-level fields:

{
  "schema": "marqdo.execution-events",
  "schema_version": 1,
  "run_id": "12345-1700000000000000000",
  "seq": 2,
  "kind": "call.enter",
  "payload": {}
}
Field Type Meaning
schema string Always marqdo.execution-events.
schema_version integer Major version; currently 1.
run_id string Opaque identity of this one run. Never an OS PID used as a cross-run identity. Different for every invocation, including repeated runs in one process.
seq integer Global, strictly increasing, starts at 0, assigned by one dispatcher. seq is the event sequence, not a frame or iteration identity.
kind string Stable namespaced event name. New kinds are additive; consumers ignore unknown kinds.
payload object Kind-specific object defined below. Unknown optional fields are ignored.

Event ordering is the order of admission to the single dispatcher. Independent OS streams (for example a foreign subprocess writing its own fd) do not have a portable physical ordering and are not part of this protocol.

3. Lifecycle and event kinds

Exactly one stream.opened is written when the JSONL writer is initialized, before project preparation. At most one run.started is written. On success or failure there is exactly one terminal event, and no event is written after it. The normal order is:

stream.opened (seq 0)
  -> run.rejected                         (prepare/authorization/compile failure)
  or run.started -> ... -> run.completed | run.failed

run.completed, run.failed and run.rejected are the only terminal events. SIGKILL, a crash or a broken pipe may leave the stream without a terminal event; that is a legitimate incomplete boundary.

3.1 stream.opened (base, seq 0)

Required payload fields:

{
  "producer": {"name": "marqdo", "version": "3.0.0-beta.1"},
  "protocol": {"name": "marqdo.execution-events", "schema_version": 1},
  "observation": {
    "level": "structural",
    "values": false,
    "selected_variables": [],
    "selected_parameters": [],
    "allow_text_values": false
  },
  "limits": {
    "max_observation_events": 100000,
    "max_observation_bytes": 16777216,
    "max_value_bytes": 256
  }
}

observation.level is base, structural or values. It reports what the producer is willing to publish, not a security grant. stream.opened does not carry a source identity because no project has been prepared yet.

3.2 run.started (base)

Required payload fields: entry, mode, source; optional started_unix_ms for diagnostics only.

{
  "entry": "root:demo@0.1.0::index.mq.md::main",
  "mode": "project",
  "source": {
    "root_id": "root:demo@0.1.0",
    "analysis_digest": "sha256:<64 lowercase hex>"
  },
  "started_unix_ms": 1763800000000
}

source.analysis_digest is sha256: plus the SHA-256 of the exact bytes of the Analysis v2 file produced for the same project and projection configuration (not a re-serialization of the in-memory IR). If the source changes, existing events become stale; consumers must not render them onto the new source.

3.3 stdout.chunk / stderr.chunk (base)

Required payload fields:

{"encoding": "base64", "data_base64": "5byA5aeL5paH5pysCg==", "byte_count": 13}
  • encoding is base64 (standard alphabet, padded).
  • data_base64 is the byte-exact data. Invalid UTF-8, NUL, binary bytes and split multi-byte characters are preserved.
  • byte_count equals the decoded byte length. A consumer must verify it.
  • Chunks are bounded (implementation maximum 64 KiB before base64 encoding); a larger host write is split into consecutive chunks in order.
  • These events are emitted at the host's real commit point: bytes are exactly those accepted by the host output budget. A rejected write emits nothing.
  • All committed chunk events precede the terminal event. Concatenating the decoded bytes must equal the bytes a normal run writes to stdout/stderr for the same arguments and grants.

3.4 Structural events (structural)

Kind Required payload fields
call.enter frame_id, parent_frame_id, function_id, module_id, span
call.exit frame_id, status, module_id, span
return frame_id, value_type, module_id, span
branch.eval frame_id, result, module_id, span
loop.iteration frame_id, decision, module_id, span
effect operation, result_code, module_id, span
  • frame_id is an opaque run-unique string (f1, f2, ...). It identifies one call instance, not a function and not a source location.
  • parent_frame_id is null for the entry frame, otherwise the direct caller's frame_id.
  • A frame is announced by call.enter only after argument binding succeeded. A binding failure produces no frame. Every entered frame that is not cut off by transport loss/truncation gets exactly one call.exit; return (when present) precedes call.exit.
  • call.exit.status is ok, return, or a stable runtime error code.
  • branch.eval.result is a JSON boolean. loop.iteration.decision is continue (body will execute) or exit (condition was false). The event is emitted once per condition evaluation; iteration indices are not part of v1.
  • effect.result_code is a non-sensitive result summary (for example ok or a stable error code). It never carries effect arguments or file contents.
  • module_id is a logical package module id; span is {"start": <byte>, "end": <byte>} in UTF-8 source offsets, or null. A span must belong to the declared module.

Example:

{"schema":"marqdo.execution-events","schema_version":1,"run_id":"r1","seq":3,"kind":"call.enter","payload":{"frame_id":"f1","parent_frame_id":null,"function_id":"root:demo@0.1.0::index.mq.md::main","module_id":"root:demo@0.1.0::index.mq.md","span":null}}

3.5 Value observation events (values)

variable.assigned required payload fields: frame_id, name, value_type, module_id, span, value_state; optional value when value_state is observed or truncated.

parameter.observed required payload fields: frame_id, name, source, value_type, module_id, span, value_state; optional value. source is provided or default (a source default is a real runtime value but not a user-provided argument).

value_state is one of:

State Meaning
not_requested The name was not selected for observation; no value is encoded.
redacted The name was selected but policy hides the value (for example Text without --observe-text-values).
truncated The value was encoded but exceeded max_value_bytes; value is the safe prefix.
unsupported The value type is not a v1 scalar; only its type is reported.
observed The scalar value is encoded normally.

Value encoding reuses the marqdo.value.v1 wire representation of the value: {"type":"int","decimal":"42"}, {"type":"float","value":1.5}, {"type":"bool","value":true}, {"type":"text","value":"..."}. For a truncated text the object also carries "truncated": true and "original_byte_count": <n>. Text truncation never splits a UTF-8 scalar. Lists, maps, tables, formulas, code, memory, resources and plugin handles are unsupported; they are never recursed into and handles are never serialized. A text value is never published without the explicit --observe-text-values consent, even when the name was selected. A --observe-max-value-bytes value of 0 means unlimited; otherwise text is cut only at a UTF-8 scalar boundary and marked truncated.

variable.assigned is emitted only after the assignment was actually committed. It never re-evaluates the right-hand side and never back-fills the final value of a variable into an earlier assignment.

3.6 observation.truncated (any)

Required payload fields: reason, limit, written, dropped. reason is event_budget or byte_budget. It is emitted at most once per run, only when the structural/value observation budget is exhausted. It does not stop stdout/stderr events or the terminal event. dropped may be null when the producer cannot know the exact count.

3.7 Terminal events (base)

run.completed:

{"status":"ok","value":{"type":"int","decimal":"10"}}

run.failed:

{"status":"error","error":{"code":"runtime.type","message":"safe diagnostic message","module_id":"root:demo@0.1.0::index.mq.md","span":{"start":120,"end":141}}}

run.rejected:

{"status":"rejected","phase":"prepare","error":{"code":"frontmatter.invalid","message":"safe diagnostic message"}}
  • run.completed.value uses the same wire encoding as run/call --json.
  • run.failed.error.code reuses the existing stable runtime error family; the message is the same safe diagnostic text already printed to humans. It does not add argument values, secrets, file contents or memory bytes.
  • run.rejected.phase is options, prepare, compile, authorization or strict. It distinguishes failures before execution started.
  • A terminal event is written only after all committed chunk events. A run.completed does not claim the consumer has read or displayed the bytes.

4. Budgets, backpressure and failure

Three separate resources exist and are never conflated:

  1. Execution budgets (--fuel, --timeout, --max-output, memory budgets). Observation never widens them. A rejected host output write emits no chunk event.
  2. Observable structural/value budget. Defaults are 100 000 events and 16 MiB, adjustable with --observe-max-events / --observe-max-bytes (0 = unlimited). Exhaustion truncates only these events and emits observation.truncated.
  3. Critical transport events. stream.opened, run.started, stdout/stderr chunks and the terminal event are not dropped by the structural budget.

Writes are synchronous and bounded. A slow consumer exerts IO backpressure on the run; the CLI must not accumulate an unbounded queue or spawn an unbounded event thread. If the JSONL writer fails (closed pipe, write error), the run must fail with observe.transport_failed, return a non-zero exit status and stop a program that would otherwise keep producing side effects. The CLI must not write a partial JSON line after a failure. Side effects that already happened are not rolled back and must not be claimed as rolled back.

If event transport is truncated but still writable (structural budget), the program's language behavior, return value, output bytes and capability authorization are unchanged.

5. Value policy and security

  • Event mode is not a capability grant and cannot enable network, filesystem, process, foreign, provider or plugin access. pure, project and strict keep their documented policy; if a project extension is rejected in a mode, event mode is rejected for the same reason.
  • Observation is off by default. Type/status metadata is the default; raw values require an explicit exact-name selection. Text requires a second explicit consent. Selection is not automatic from a manifest, View setting or environment.
  • Program stdout/stderr may contain secrets by the program's own intent; the protocol preserves bytes and does not promise automatic redaction of program output.
  • Trace v2 and Analysis v2 never gain argument/local values, handles, memory bytes or new secret-bearing fields through this protocol.
  • A third-party extension's raw OS stdout/stderr must not pollute the JSONL channel. Such output remains external process data: it is either captured by the controlled host transport or the combination is unsupported/fail-closed.
  • Tests use synthetic sentinel values and assert that unselected values never appear in the stream, Trace or --json.

6. Relation to Analysis v2 and Trace v2

Artifact Relation
Analysis v2 Static only. run.started.source binds to its exact bytes via analysis_digest. Events never replace it.
Trace v2 Unchanged schema/version and privacy promise. In a combined run both writers share the CLI-generated run_id; Trace v2 gains no frame/value fields and is still published atomically after the run.
Tool Protocol v1 Unchanged. A consumer invokes the released binary via MARQDO_BIN and negotiates by reading stream.opened; tool.json does not implicitly advertise an events field.
View SSE Implemented by the View repository by adapting this JSONL stream. The CLI implements no HTTP/SSE.

7. Versioning and compatibility

  • Independent major version 1, not tied to the product version or Trace v2.
  • Additive optional fields and new event kinds are allowed in v1. Changing a required field's meaning, ordering guarantees, redaction promise, source digest definition or making private data public by default requires a new major.
  • Consumers must ignore unknown kind and unknown optional fields, and must reject unknown schema_version majors.
  • --help, docs/developers/analysis-trace.md, docs/developers/compatibility.md and COMPATIBILITY.md list the protocol and its exact option names.

8. Test fixtures and consumers

  • Schema: schemas/execution-events-v1.schema.json.
  • Independent consumer: tests/consumers/execution_events_consumer.py.
  • Fixtures: tests/consumers/fixtures/execution-events/.
  • Rust gate: crates/cli/tests/execution_events.rs runs the real binary and checks envelope, sequence, base64 byte fidelity, terminal behavior, redaction, frames, traces and output purity.