Chapter 18 · Trace vision and language into bounded robot action

Today in the field story

One problem, then the next

Replay one successful mug placement and the ambiguous, stale, malformed, oversized, controller-rejected, and execution-deviation branches under one correlation ID. Record camera and instruction versions, intermediate shapes, model and processor, decoded units, guard decision, publish and acceptance times, controller feedback, and measured outcome. The trace closes the mission only when another engineer can locate the first divergence and reproduce why a proposal moved forward or stopped.

Why now: The chapter’s separate concepts become credible only as one end-to-end, failure-complete architecture record.
Ignore today: Ignore model adaptation and agent planning; freeze the foundational contract first.
Unlocks next: The exact Week 18 handoff to current VLA selection, typed agents, and safety harnesses.

Understand

Build the physical picture first

An architecture review follows one sealed parcel from photon and sentence to measured effect, checking the label, clock, owner, and escape route at every door.

A useful architecture review traces concrete values, not only boxes. Start with one correlation ID and capture camera, instruction, robot state, calibration, model, processor, and controller versions. At each boundary record producer, consumer, schema, shape, unit, frame, timestamp, freshness, latency, error state, and evidence location. The trace should make it possible to answer which pixels and words produced which action row, which decoder turned that row into physical units, which validator accepted or rejected it, and which measurement later established the effect.

Timing is a causal contract. Camera exposure, robot state, instruction, inference, command publication, controller sampling, and measured response happen at different instants. Sum stage latency, but also inspect synchronization and queue age; a fast model using a 500 ms old image is still stale. Compare worst observed end-to-end latency with observation and control deadlines, not only with an average. Every action carries the source observation time, inference generation, expiry, intended start time, rate, and horizon so a delayed gateway cannot revive an obsolete chunk.

The model boundary needs versioned processors on both sides. Preprocessing selects and normalizes observations. Postprocessing crops padding, restores physical scale, converts representation and frame, and produces a typed proposal. Validation then checks finite values, robot and model compatibility, authority, freshness, workspace, joint and velocity limits, collisions, inter-step continuity, controller mode, and any task-specific exclusion. Some conditions should reject; some should ask for clarification; others can invoke a named deterministic recovery. Silent clamping can conceal a model fault and change task meaning, so every modification needs policy and telemetry.

Completion requires a return path. A gateway acknowledgment proves transport, and a controller acceptance proves only that a request entered execution. Measured joint or tool state, contact or object perception, and a predeclared task predicate establish what happened. Review nominal and failure traces for stale image, unknown entity, missing state, incompatible checkpoint, NaN, out-of-range action, deadline miss, collision, controller rejection, tracking error, and task timeout. The architecture is credible when every fault terminates in an observable state with one owner and no path around the validator.

Words you need

Name each idea precisely

Correlation ID: A stable identifier propagated across observation, inference, validation, command, feedback, and outcome records for one request.
Physical example:
req-126-04 links the saved camera frame to action chunk 18, validator decision 22, ROS 2 goal 51, and terminal task record 9.
Traceability: The ability to connect a requirement or request through versions, transformations, decisions, execution evidence, and final outcome.
Physical example:
A failed placement can be traced to an ambiguous entity rather than guessed from an isolated controller log.
Freshness budget: The maximum allowed age across acquisition, queues, compute, transport, and command start before data or action is rejected.
Physical example:
A 120 ms camera-to-command limit allocates time to capture, inference, decoding, validation, and the ROS 2 gateway.
Guardrail: An independently enforced rule that rejects, transforms under explicit policy, or escalates a proposed action before or during execution.
Physical example:
A trajectory validator rejects a gripper-frame waypoint that would cross the simulator's protected region even though every number is finite.
Terminal evidence: The measured record that closes a request as succeeded, failed, cancelled, timed out, or otherwise resolved under a declared predicate.
Physical example:
The gripper is open, the tracked block remains inside the target region for two seconds, and the controller reports no forbidden contact.
Processor version: The exact version of ordered observation and action transforms paired with a checkpoint and embodiment.
Physical example:
Checkpoint vla-12 is rejected when loaded with decoder robot-r3-v7 because it was trained with robot-r3-v6 statistics and component order.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalToken, action, and latency budgets

Language reasoning must finish before the physical situation becomes stale.

T_total: end-to-end decision latencyUnit: milliseconds (ms)
f_control: safety/control update rateUnit: hertz (Hz)
H: action horizonUnit: steps or seconds

Perception takes 80 ms, model reasoning 220 ms, and skill dispatch 20 ms.
T_total = 80+220+20 = 320 ms.
At 1 m/s the robot moves 0.32 m during that delay; independent fast safety control must not wait for the LLM.

Programmer analogy

An LLM tool call resembles backend orchestration, but its latency must be converted into physical travel distance.

At 0.5 m/s, how far does a robot move during 400 ms?

400 ms = 0.4 s; distance = 0.5×0.4 = 0.2 m.

End-to-end latency is the sum of stage times:

t_{\mathrm{total}}=\sum_i t_i=20+60+10+10=100\,\mathrm{ms}.

Inverting normalization gives $a=a_{\min}+(a_{\mathrm{norm}}+1)(a_{\max}-a_{\min})/2=0.75\,\mathrm{m}$ for $a_{\mathrm{norm}}=0.5$ and range $[0,1]$ metres.

Review one `move red block left` request with values and failures

A simulator runs at 10 Hz. Request req-126-04 uses a camera frame captured at 12.000 s, state sampled at 12.006 s, maximum source-to-command age 120 ms, and an action head returning four rows of [dx, dy, dz, gripper].

Bind the instruction to grounded entity block-17, camera calibration cam-v4, robot R3, processor r3-v6, checkpoint vla-12, controller mode relative gripper-frame position, and one correlation ID.
Record stage latency: capture and preprocessing 18 ms, multimodal encoding and fusion 42 ms, action generation 17 ms, decode plus validation 9 ms, and ROS 2 gateway 10 ms.
Add the stages: 18 + 42 + 17 + 9 + 10 = 96 ms; the 120 ms freshness budget leaves 24 ms, while the 100 ms control period leaves only 4 ms scheduling margin.
Inspect output shape [1, 4, 4], finite status, component order, normalized ranges, and inverse-decode the first row; attach gripper frame, 10 Hz rate, intended start, and expiry to the physical proposal.
Validate all four poses for entity freshness, authority, processor-checkpoint pairing, workspace, joint feasibility, per-step distance, protected-region collision, gripper command, and continuity before publishing a simulated goal.
Follow gateway acknowledgment, controller acceptance, per-step measured tool pose, gripper feedback, updated block-17 pose, and the predicate block centre moved at least 0.08 m left without forbidden contact to a terminal record.
Replay with a 40 ms extra queue delay, a NaN in row three, and two red-block candidates; require separate expired, invalid_action, and needs_clarification outcomes, each with no controller goal.

Result

The nominal path reaches the gateway in 96 ms with 24 ms freshness margin, but little 10 Hz scheduling slack; each injected failure stops before execution and remains attributable to one contract.

What this proves

Architecture quality becomes reviewable when one request carries concrete identity, shapes, physics, time, decisions, feedback, and terminal truth through both success and failure.

Physical examples

Where this appears in real life

Correct action arrives one cycle late

A valid chunk is generated from a fresh scene, but a queue delay causes the first row to reach a 10 Hz controller after its intended cycle.

Look for:

Source time, intended start, queue age, expiry, and deadline decision reveal a stale command; correct numbers do not justify late execution.

Controller accepts, block never moves

The ROS 2 controller acknowledges a trajectory, but the gripper misses the block and reaches the target pose empty.

Look for:

Controller feedback, gripper state, object tracking, and the task predicate record physical failure instead of equating accepted motion with successful placement.

Hands-on exercise

Make the idea observable

Use the existing simulator or a local trace generator. Keep one folder of small JSON records and screenshots; do not connect the review harness to a physical actuator.

Choose one language-conditioned simulated task and write its request schema, grounded entity, success predicate, timeout, forbidden events, model and processor versions, and freshness budget.
Instrument observation, inference, decode, validator, gateway, controller feedback, perception update, and terminal result with one correlation ID and monotonic timestamps.
Capture one nominal trace and calculate stage, queue, end-to-end, deadline, and action-horizon timing from records rather than from console impressions.
Inject stale image, ambiguous referent, missing state component, wrong processor version, NaN, out-of-range row, collision, gateway delay, controller rejection, and task timeout one at a time.
For each fault, verify the exact owner, state transition, operator indication, evidence record, and absence of any later unauthorized command; do not replace failed traces with a rerun.
Publish a one-page architecture table linking every boundary to schema, units, frame, time, validation, failure response, and terminal evidence, then mark claims as simulation-only.

Observe

Concrete traces expose hidden queues, version coupling, and false completion boundaries that remain invisible in a nominal box diagram or a successful model-response log.

Done when

The nominal trace closes with a measured predicate, all ten faults end in their predeclared observable states, no rejected path reaches the controller, and another engineer can replay the evidence from the correlation ID.

Build today

Build a tiny language-conditioned policy interface over your existing simulator or dataset.

Evidence to save

DONE when the weekly ship note explains how “Architecture review: trace pixels and words to commands” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Reviewing only component names and arrows.

Better

Trace one real value set with identities, schemas, shapes, units, frames, versions, clocks, owners, decisions, and evidence through every boundary.

Wrong

Treating a model response or controller acceptance as completed work.

Better

Require measured robot and object state plus the predeclared task predicate and forbidden-event check before issuing terminal success.

Wrong

Testing the nominal path and listing failures only as future work.

Better

Inject stale, ambiguous, malformed, late, infeasible, rejected, and physically unsuccessful cases now, and retain their exact observable terminal records.

Job connection

How this becomes employable evidence

Lead a VLA production-readiness review that correlates multimodal inputs, model and processor versions, decoded actions, ROS 2 goals, guardrails, operator state, timing, measured robot feedback, and failure injection into one auditable acceptance trace.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robot HMI / Control & Monitoring Engineer

Chapter 18 interview drill

Interview questions: Architecture review: trace pixels and words to commands

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Whiteboard an end-to-end VLA request from camera and instruction to verified physical outcome. Put shapes, units, frames, timestamps, versioning, latency budget, validators, ROS 2 feedback, and at least five failure exits on the diagram.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1How much freshness and 10 Hz scheduling margin remain after the 96 ms worked trace?

Model interview answer

The 120 ms freshness budget leaves 24 ms, while a 100 ms control period leaves only 4 ms before that cycle boundary.

Q2Why should an action chunk carry the source observation time as well as its generation time?

Model interview answer

A quickly generated chunk can still depend on an old camera or state sample; both acquisition age and compute or queue delay determine whether execution is current.

Q3What distinguishes controller acceptance from terminal task evidence?

Model interview answer

Acceptance confirms entry into the controller, while terminal evidence measures robot and object state against the success, failure, timeout, cancellation, and forbidden-event predicates.

Chapter starter artifact

Reject ungrounded, stale, and oversized proposals

A correlation-complete architecture trace classifies every model component, grounds the selected mug to fresh entity evidence, preserves embodiment and action semantics, rejects ambiguous, stale, malformed, and out-of-range proposals, and reports held-out success, intervention, latency, unsafe-proposal, and worst-slice results.

week-18-guard-grounded-actions.mjsJavaScriptDownload starter

const proposals = [
  { id: "p1", entity: "mug-2", ageMs: 40, deltaM: 0.04 },
  { id: "p2", entity: null, ageMs: 35, deltaM: 0.02 },
  { id: "p3", entity: "mug-2", ageMs: 180, deltaM: 0.03 },
  { id: "p4", entity: "mug-2", ageMs: 45, deltaM: 0.18 },
  { id: "p5", entity: "mug-2", ageMs: Number.NaN, deltaM: Number.NaN },
  { id: "p1", entity: "mug-2", ageMs: 30, deltaM: 0.01 },
  { id: "p7", entity: "mug-3", ageMs: 30, deltaM: 0.01 },
];
const expectedEntity = "mug-2"; const seen = new Set();
const reasons = []; let eligible = 0;
for (const proposal of proposals) { let reason = "";
  if (typeof proposal.id !== "string" || proposal.id.length === 0) reason = "invalid-schema";
  else if (seen.has(proposal.id)) reason = "duplicate";
  else {
    seen.add(proposal.id);
    if (proposal.entity !== expectedEntity) reason = proposal.entity ? "wrong-entity" : "ambiguous";
    else if (
    !Number.isFinite(proposal.ageMs) || proposal.ageMs < 0 ||
    !Number.isFinite(proposal.deltaM)
    ) reason = "invalid-number";
    else if (proposal.ageMs > 100) reason = "stale";
    else if (Math.abs(proposal.deltaM) > 0.05) reason = "out-of-range";
  }
  if (reason) reasons.push(reason);
  else eligible += 1;
}
const output =
  "eligible=" + eligible +
  " rejected=" + reasons.length +
  " reasons=" + reasons.join("|");
const expected =
  "eligible=1 rejected=6 reasons=ambiguous|stale|out-of-range|invalid-number|duplicate|wrong-entity";
if (output !== expected) throw new Error("action guard mismatch: " + output);
console.log(output);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-18-guard-grounded-actions.mjs
Expected output: eligible=1 rejected=6 reasons=ambiguous|stale|out-of-range|invalid-number|duplicate|wrong-entity
Planted failure to diagnose: The guard must reject ambiguous, stale, oversized, non-finite, replayed, and wrong-entity proposals; truthiness, NaN comparisons, or missing identity checks cannot grant eligibility.