Chapter 18

Trace vision and language into bounded robot action

Build an engineering-grade mental model of embodied AI without treating a foundation model as magic or a research project. This chapter distinguishes visual-language understanding, action-producing policies, and embodied reasoning; traces images, words, robot state, and time through a multimodal action pipeline; preserves physical meaning when data comes from different robot bodies; compares discrete, continuous, diffusion, and flow action representations; grounds instructions in perceived entities; measures held-out generalization; and finishes with a value-by-value architecture review whose outputs remain behind robot-specific validation and control.

Before you start

Complete the applied PyTorch, robot-dataset, behavior-cloning, policy-evaluation, ROS 2 action, frame, timing, and validation work from Weeks 5–17; this chapter builds on those contracts rather than replacing them.
Bring one disconnected simulator, recorded episode, or small synthetic table containing an image reference, natural-language task, timestamped robot state, and bounded action; powered hardware is neither required nor permitted for the exercises.
Read tensor shapes, calculate a normalized scalar, distinguish a camera frame from a robot coordinate frame, and explain why model inference is not physical task completion.
Keep collision avoidance, workspace and velocity limits, watchdogs, emergency stopping, and low-level control outside the learned-model exercises; the chapter teaches system integration and evaluation, not a safety-rated controller.

By the end

Classify a component as a VLM, VLA, embodied reasoner, or surrounding deterministic system from its actual inputs, outputs, state, and authority rather than its product label.
Trace camera pixels, language tokens, proprioceptive state, timestamps, multimodal features, action chunks, normalization, decoding, validation, and controller feedback with exact shapes, units, frames, and owners.
Prepare a cross-embodiment data contract that retains embodiment identity, observation and action semantics, task labels, mixture weights, masks, adapters, and comparable but not falsely equivalent normalization.
Compare discrete tokens, direct continuous values, diffusion-generated sequences, and flow-generated sequences by resolution, multimodality, sampling cost, action horizon, re-observation cadence, and physical decoding requirements.
Ground referring expressions and relations to perceived entity IDs and admissible robot skills, then clarify or reject instructions whose referent, goal, or preconditions are unresolved.
Freeze held-out object, scene, prompt, pose, and embodiment slices before evaluation; report success, intervention, unsafe proposals, latency, and the worst slice separately from an aggregate.
Review one language-conditioned policy path end to end, including nominal values and stale-input, ambiguous-language, malformed-output, out-of-range, late-command, controller-rejection, and execution-deviation branches.

The field story

Ground the red-mug instruction before any command

A recorded tabletop scene contains two red mugs, one plate, and a left tray. The instruction says, “Put the red mug beside the plate into the left tray.” A visual-language model may describe the scene, an action policy may propose a motion sequence, and an embodied reasoner may choose typed skills, but those are different artifacts with different authority. The mission traces image, text, robot state, time, model processing, normalized action, decoder, validator, controller, and measured result without allowing a fluent answer to bypass deterministic checks.

Use one synthetic or recorded episode and correlation ID. Preserve camera and embodiment identity, make ambiguous references produce clarification or rejection, compare action representations without treating tokens as torques, and freeze object, scene, prompt, pose, and embodiment slices before evaluation. The policy remains disconnected or simulated. Every late, malformed, ungrounded, or out-of-range proposal follows a visible failure branch. The week finishes with a value-level architecture review that shows exactly which pixels and words contributed to which eligible command.

Why this chapter now: The course now has perception, robot data, learned policies, guarded runtimes, FleetOps, and validation evidence; VLA concepts can therefore be attached to real system contracts instead of presented as magic.
Ignore for now: Do not train a foundation model, call a hosted demo a robot controller, or connect unreviewed outputs to powered hardware. Trace one bounded policy path.
This unlocks: The architecture contract supports Week 19 ecosystem selection and adaptation, Week 20 typed agent skills, Week 21 safety evaluation, and later capstone deployment.
Proof you will leave with: Provide component classification, input and output schemas, tensor and token trace, embodiment manifest, action-representation comparison, grounding table, clarification and rejection cases, confidence-bin calibration and abstention evidence, frozen generalization matrix, correlation timeline, decoder and guard tests, controller feedback, and exact artifact identities.

Environment contractRepository-supported Node.js 22.13.0 or newer runs the action-eligibility starter. The main work uses recorded or synthetic multimodal fixtures and, where execution context is needed, ROS 2 Jazzy with Gazebo Harmonic in a disconnected simulation.

Compatibility boundary: Model cards, hosted APIs, checkpoints, processors, tokenizers, action adapters, licenses, and access conditions can change quickly. Pin every artifact actually tested and mark unverified external capabilities unknown.
Smoke check: Run node week-18-guard-grounded-actions.mjs; then replay one golden observation through the chosen processor and decoder, confirming entity binding, shapes, units, timestamps, rejection reasons, and controller eligibility without sending powered commands.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

Classify each red-mug pipeline component by actual inputs, outputs, state, and authority. The scene describer can answer a visual question, the action policy can propose bounded action data, and a reasoner can assemble typed skills, but none inherits controller authority from its label. Place deterministic grounding, validation, collision, cancellation, and feedback around the models so the mission begins with an honest system boundary.

Why now: VLM, VLA, and embodied reasoning must be separated before their evidence or risk can be evaluated.
Ignore today: Ignore vendor rankings and model scale; classify one concrete pipeline.
Unlocks next: A component map that prevents language output from masquerading as physical execution.

Understand

Build the physical picture first

Three workers may share the same camera, but one describes the scene, one proposes body-specific motion, and one organizes monitored skills.

Classify a model by its observable contract, not by how impressive its name sounds. A vision-language model consumes visual and language inputs and produces semantic outputs such as text, object references, coordinates, detections, or scores. A vision-language-action model additionally produces an action representation intended for a robot: joint targets, end-effector deltas, gripper state, mobile velocity, or a short action sequence. The word action is load-bearing. A textual sentence saying “move left” remains language until a documented decoder turns it into a typed, timed, body-specific command.

An embodied-reasoning model usually operates at a different boundary. It can inspect images or structured state, decompose a goal, select tools or robot skills, check progress, and revise a plan. Its output may be a skill call such as pick(object_id=17) rather than direct motor values. Google DeepMind presents Gemini Robotics as a VLA with action output and Gemini Robotics-ER as an embodied-reasoning VLM that can connect to existing controllers. Other systems may arrange these capabilities differently, so inspect the actual interface instead of assuming every product uses the same architecture.

These categories are capabilities, not safety certifications or mutually exclusive boxes. One deployed system may use a VLM to identify a valve, a reasoner to choose an inspection sequence, a VLA to propose a manipulation chunk, and deterministic ROS 2 components to validate and execute it. The component with broader reasoning is not automatically entitled to greater command authority. Authority should follow the reviewed system boundary: which outputs are allowed, when they expire, which validator checks them, which controller owns execution, and what feedback proves or disproves the predicted effect.

The most practical distinction is the handoff after inference. Text, plans, detections, and action arrays all need schemas and evidence, but action outputs carry immediate physical semantics: dimension order, unit, coordinate frame, control mode, valid range, timestamp, horizon, and embodiment. A VLA response that parses successfully can still refer to the wrong robot, stale image, wrong gripper frame, or unsupported motion mode. Treat it as a proposal until robot-specific decoding and validation accept it; treat acceptance as separate from measured physical completion.

Words you need

Name each idea precisely

Vision-language model (VLM): A model that relates visual inputs and language, producing semantic outputs such as text, labels, detections, coordinates, or structured reasoning rather than a robot action by definition.
Physical example:
An inspection camera and question produce needle approximately 62 psi; a separate calibrated program decides whether any robot should move.
Vision-language-action model (VLA): A policy whose conditioning includes vision and language and whose declared output contains a robot action representation.
Physical example:
A wrist image plus place the red block in the tray produces four future end-effector deltas and gripper commands for one arm.
Embodied reasoning: Reasoning about physical entities, relations, state, goals, tools, and outcomes, often used to compose or monitor bounded skills rather than directly close a motor loop.
Physical example:
A planner sees that the tray is blocked, calls move_obstacle, re-observes, and only then calls a validated place skill.
Policy: A rule or learned function that maps the current information state to an action or distribution over actions.
Physical example:
Given a camera crop, joint state, and task text, a policy chooses the next gripper-frame motion chunk.
Action contract: The complete meaning required to decode an action, including dimensions, units, frame, control mode, rate, horizon, limits, time, and target embodiment.
Physical example:
The vector [0.02, 0, 0, 1] means nothing physical until it is declared as metres of gripper-frame displacement plus an open-gripper flag for robot R3.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalToken, action, and latency budgets

Language reasoning must finish before the physical situation becomes stale.

T_total: end-to-end decision latencyUnit: milliseconds (ms)
f_control: safety/control update rateUnit: hertz (Hz)
H: action horizonUnit: steps or seconds

Perception takes 80 ms, model reasoning 220 ms, and skill dispatch 20 ms.
T_total = 80+220+20 = 320 ms.
At 1 m/s the robot moves 0.32 m during that delay; independent fast safety control must not wait for the LLM.

Programmer analogy

An LLM tool call resembles backend orchestration, but its latency must be converted into physical travel distance.

At 0.5 m/s, how far does a robot move during 400 ms?

400 ms = 0.4 s; distance = 0.5×0.4 = 0.2 m.

Normalize a physical action $a$ from $[a_{\min},a_{\max}]$ into $[-1,1]$ with

a_{\mathrm{norm}}=2\frac{a-a_{\min}}{a_{\max}-a_{\min}}-1.

For $a=0.75\,\mathrm{m}$ in $[0,1]$ metres, $a_{\mathrm{norm}}=0.5$ . Normalization does not change a model’s capabilities.

Classify six black-box components from their interfaces

A simulated workcell exposes six unnamed components: image question answering, object detection, task decomposition, skill selection, action-chunk prediction, and a ROS 2 trajectory controller.

Write each component's inputs and outputs without using its marketing label: for example, image + question → text, image → object IDs and boxes, or image + instruction + joint state → H×D action values.
Mark image question answering and object detection as VLM or vision-language capabilities only when language participates; neither becomes a VLA because neither output is a robot action.
Mark task decomposition and skill selection as embodied-reasoning functions when they inspect physical state and produce an ordered plan or typed tool call with observable preconditions.
Mark the action-chunk predictor as a VLA only after confirming the output dimensions, horizon, units or normalization, target body, and processor that converts it to physical commands.
Keep the ROS 2 trajectory controller outside the model categories: it consumes accepted trajectory commands, closes a control loop, reports feedback, and enforces its own declared constraints.
Draw the authorized handoffs and add reject, clarify, timeout, cancel, and measured result paths; remove any direct arrow from free text or an unchecked model array to an actuator.

Result

The diagram distinguishes semantic understanding, physical reasoning, learned action proposal, and deterministic execution from evidence in the interfaces rather than from model names.

What this proves

Ask what enters, what leaves, and who may act on it; the output contract reveals more than the label VLM, VLA, agent, or controller.

Physical examples

Where this appears in real life

Gauge inspection versus valve adjustment

A camera faces a pressure gauge and a nearby valve. One component reports the reading, while another subsystem may be authorized to turn the valve through a bounded skill.

Look for:

The gauge-reading output belongs to a VLM-style perception contract; a typed turn command belongs to an action or skill contract and still passes calibration, workspace, force, and authority checks.

Table-clearing request

A person asks, clear the table, which requires identifying movable objects, choosing an order, and repeatedly executing pick-and-place behavior.

Look for:

The high-level goal is decomposed and monitored by embodied reasoning, while each object reference and each body-specific action is grounded, validated, executed, and verified separately.

Hands-on exercise

Make the idea observable

Use paper cards or a small JSON file and the existing simulator. Create no network account, download no foundation-model weights, and send no model output to powered hardware.

Create nine input/output cards covering a scene description, object box, pose estimate, natural-language plan, typed skill call, normalized action, physical command, controller feedback, and task result.
For every card, record producer, consumer, schema, timestamp requirement, and whether it describes, proposes, authorizes, executes, or measures physical behavior.
Sort the cards into VLM capability, embodied reasoning, VLA policy, deterministic validator/controller, and independent evidence; explain any card that legitimately crosses two capability categories.
Choose one normalized action card and add dimension names, unit ranges, coordinate frame, horizon, robot ID, model version, and expiry so it becomes decodable but not yet accepted.
Insert a deliberately stale image, unresolved object reference, and wrong robot ID; write the exact component that rejects or clarifies each case.
Run the accepted path only in simulation, then capture the requested action, validator decision, controller acknowledgment, measured state, and terminal task outcome as separate records.

Observe

Category disagreements usually expose an unstated output or authority boundary, while apparently clear action arrays become ambiguous as soon as units, body, frame, or time is removed.

Done when

Every card has one reviewable contract, no unchecked semantic output reaches the controller, all three injected faults stop at a named boundary, and simulated completion is supported by measured feedback rather than inference success.

Build today

Build a tiny language-conditioned policy interface over your existing simulator or dataset.

Evidence to save

DONE when the learning log explains “VLM versus VLA versus embodied-reasoning model” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling any model that sees an image and mentions motion a VLA.

Better

Inspect the formal output: descriptions and plans remain semantic outputs, while a VLA must expose a documented robot-action representation and decoding path.

Wrong

Treating embodied reasoning as a replacement for a low-level controller.

Better

Use reasoning to select and monitor bounded capabilities; keep timing-critical feedback, limits, watchdogs, and protective behavior in reviewed embodiment-specific components.

Wrong

Accepting a valid action array as proof that the requested task occurred.

Better

Separate model proposal, validator acceptance, controller acknowledgment, measured motion, task predicate, and terminal outcome in the evidence trail.

Job connection

How this becomes employable evidence

Turn a vendor's multimodal demo into an integration map that separates perception, reasoning, policy, validation, ROS 2 execution, operator authority, and physical evidence before deciding where the component can enter a production robot stack.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robot HMI / Control & Monitoring Engineer

Chapter 18 interview drill

Interview questions: VLM versus VLA versus embodied-reasoning model

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

A model accepts an image and instruction, emits a textual plan and a seven-number vector, and calls itself an embodied agent. Classify each capability, identify missing action-contract fields, and design the safe handoff into existing ROS 2 skills.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What single output difference most directly separates a VLM capability from a VLA capability?

Model interview answer

A VLA exposes a robot action representation, while a VLM by itself produces semantic outputs such as language, detections, coordinates, or reasoning.

Q2Why can an embodied reasoner choose a skill without being allowed to command motors directly?

Model interview answer

Planning authority and motor-control authority are separate; the skill call still needs typed inputs, preconditions, validation, an execution owner, and measured feedback.

Q3Which evidence comes after a controller accepts a command?

Model interview answer

Timestamped measured state and a checked task predicate must establish the physical effect and terminal outcome; acceptance alone proves neither.

Chapter references

Google DeepMind — Gemini Robotics and Gemini Robotics-ERPrimary distinction between an action-output VLA and an embodied-reasoning VLM, evidence categories for generality and interactivity, multiple embodiments, and the explicit boundary to embodiment-specific low-level safety controllers.
OpenVLA — Open-Source Vision-Language-Action ModelPrimary project description of fused visual encoders, projection into a language-model token space, tokenized actions, continuous action decoding, Open X-Embodiment pretraining, and held-out visual, motion, physical, and semantic evaluations.
Hugging Face LeRobot — Action RepresentationsMaintainer documentation for joint versus end-effector space, absolute versus relative actions, the required relative-normalize and unnormalize-absolute processor order, and action-chunk execution implications.
Open X-Embodiment — Robotic Learning Datasets and RT-X ModelsPrimary project evidence for standardized heterogeneous robot data, embodiment-aware action alignment, data mixtures, cross-robot transfer, task strings, gripper-frame action semantics, and multi-site evaluation.
Diffusion Policy — Visuomotor Policy Learning via Action DiffusionPeer-reviewed project source for conditional denoising of multimodal action sequences, visual conditioning, receding-horizon execution, temporal consistency, latency testing, and the difference between generating a trajectory and validating it.