Chapter 18 · Trace vision and language into bounded robot action

Today in the field story

One problem, then the next

Trace the camera crop, tokenized instruction, proprioceptive state, and source timestamps into multimodal features and an action output. Write shapes, axis meanings, frame, units, processor identity, checkpoint, and action horizon beside each boundary. Follow one numeric value through normalization and decoding. The red mug is not grounded merely because its words influenced a token; the trace must retain the physical evidence used later.

Why now: The component map needs value-level contracts before errors can be located.
Ignore today: Ignore transformer derivations; inspect shapes, semantics, timing, and ownership.
Unlocks next: A reproducible pixel-and-word to proposed-action trace.

Understand

Build the physical picture first

A multimodal policy is a labelled assembly line: pixels and words become compatible features, then body metadata turns bounded numbers back into motion.

A camera tensor begins as sampled light, not as objects or metres. Preprocessing chooses camera views, crop, resize, channel order, dtype, and pixel normalization. A visual encoder divides an image into patches or regions and maps them to feature vectors. OpenVLA, for example, combines visual backbones that emit image patch embeddings, then projects those embeddings into the input space used by its language-model backbone. The exact encoder family can change; the stable engineering questions are input shape, preprocessing version, camera identity, capture time, feature shape, and whether the view still contains the evidence needed by the task.

Language takes a parallel path. A tokenizer converts an instruction into token IDs, and an embedding layer maps those IDs into vectors. Token boundaries are not object boundaries: red mug beside plate may be split into several subword tokens, yet grounding still must resolve one perceived mug and one relation. Multimodal fusion or conditioning makes visual and language features available to the policy. Some architectures concatenate projected image and text tokens; others use cross-attention or separate action heads. Do not infer physical alignment merely because all feature vectors share width D.

Robot state and time complete the observation contract. Joint positions, gripper state, base pose, previous actions, or several recent frames may be projected alongside image and language features. The policy must know which sample time each value represents. A wrist image captured 180 ms after the joint state can show a different physical configuration, and a language instruction issued for the prior scene can be stale even when both tensors have valid shapes. Batch, history, camera, token, horizon, and action dimensions need distinct names; a shape such as [1, 4, 7] is meaningless until those axes are declared.

The action head produces tokens or continuous values, often for a future chunk. Postprocessing then reverses training conventions: crop padded dimensions, unnormalize with the matching training statistics, convert relative to absolute representation if required, transform to the declared physical frame, clamp only under a reviewed policy, and validate the complete proposed trajectory. OpenVLA predicts tokenized actions that decode to continuous values; LeRobot documents processor order such as relative-to-normalized during training and unnormalize-to-absolute during inference. A correct inverse transform is necessary, but semantic grounding, freshness, feasibility, and safe execution remain separate checks.

Words you need

Name each idea precisely

Patch embedding: A learned feature vector representing one image patch or visual region after the camera tensor passes through a visual encoder.
Physical example:
A 224 by 224 wrist image may become a grid of visual feature vectors whose locations still need positional information to distinguish left from right.
Language token: An integer-coded text unit that is mapped to a vector; it may be a word, subword, punctuation mark, or special control symbol.
Physical example:
The phrase open gripper can occupy several token positions even though it names one requested state change.
Multimodal conditioning: Making information from images, language, robot state, or history available to the action predictor in a compatible feature representation.
Physical example:
The same red-block image conditions different action chunks when paired with move left versus move right.
Proprioception: Measurements of the robot's own configuration or motion, such as joint positions, velocities, effort, gripper state, or base pose.
Physical example:
A wrist image cannot reveal every joint angle, so current joint state helps decode whether a proposed end-effector change is reachable.
Action head: The model component that converts fused context into an action token, value, distribution, or future action sequence.
Physical example:
A policy head maps the current visual-language-state context to four future rows of [dx, dy, dz, gripper].
Processor pipeline: The ordered pre- and postprocessing steps that convert physical observations and actions into model representation and back again.
Physical example:
A relative gripper-frame displacement is normalized before training, then unnormalized and restored relative to the current pose before simulation execution.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalToken, action, and latency budgets

Language reasoning must finish before the physical situation becomes stale.

T_total: end-to-end decision latencyUnit: milliseconds (ms)
f_control: safety/control update rateUnit: hertz (Hz)
H: action horizonUnit: steps or seconds

Perception takes 80 ms, model reasoning 220 ms, and skill dispatch 20 ms.
T_total = 80+220+20 = 320 ms.
At 1 m/s the robot moves 0.32 m during that delay; independent fast safety control must not wait for the LLM.

Programmer analogy

An LLM tool call resembles backend orchestration, but its latency must be converted into physical travel distance.

At 0.5 m/s, how far does a robot move during 400 ms?

400 ms = 0.4 s; distance = 0.5×0.4 = 0.2 m.

A $224\times224\times3$ image is encoded and fused with $L$ language tokens before producing an action tensor of shape $H\times D$ . For a physical action $a=0.75$ in range $[0,1]$ ,

a_{\mathrm{norm}}=2\frac{0.75-0}{1-0}-1=0.5.

Shape and range checks do not measure semantic correctness.

Trace one observation into a four-step physical action chunk

A synthetic arm sample contains one 224×224×3 RGB image, an eight-token instruction, a seven-value joint state, and a policy that emits four future rows of [dx, dy, dz] normalized to [-1, 1].

Record the camera contract as [H=224, W=224, C=3], RGB byte input, camera table_1, capture time 12.000 s, and calibration version cam-v4; resizing or channel reordering must be part of the preprocessing version.
Assume the visual encoder returns 196 patch vectors of width 1,024, the tokenizer returns eight language positions projected to the same width, and the joint-state projector returns one state vector of width 1,024.
Concatenate these example contexts into 196 + 8 + 1 = 205 positions, giving shape [B=1, T=205, D=1024]; this arithmetic checks the interface, not whether the red block was understood.
Read the action head output as [B=1, horizon=4, action_dim=3] and reject any batch, horizon, or component ordering that differs from the registered policy metadata.
Decode the first normalized x component 0.5 using calibrated range [-0.08, 0.08] m: -0.08 + ((0.5 + 1) / 2) × 0.16 = 0.04 m; repeat component-wise with its own units and ranges.
Attach gripper frame, relative-position control mode, 10 Hz rate, robot ID, source timestamps, and expiry; test the resulting four poses against workspace, joint, collision, velocity, and discontinuity limits.
Execute only through the simulator controller, compare measured pose with the first accepted target, and record whether the task predicate—not merely the model call—became true.

Result

A visually and linguistically conditioned tensor becomes a physically meaningful 0.04 m proposal only after shape checks, matching statistics, frame and control-mode decoding, freshness checks, trajectory validation, and measured simulation feedback.

What this proves

Feature width makes modalities computable together; processor metadata and robot evidence make the eventual action physically interpretable.

Physical examples

Where this appears in real life

Red block moves between frames

A table camera records a red block, the operator asks to move it left, and another person shifts the block while the policy is computing.

Look for:

The image timestamp, instruction generation, inference latency, and action expiry reveal whether the decoded leftward motion still refers to the observed block location.

Same image, different robot posture

Two arm configurations produce a similar overhead view of the table, but one elbow is near a joint limit and the other has ample margin.

Look for:

Proprioceptive state changes reachability and the valid decoder result even when camera pixels and language tokens are identical.

Hands-on exercise

Make the idea observable

Use a local notebook or TypeScript/Python script with fabricated arrays and one saved image. No pretrained model, GPU, or robot connection is required.

Create a manifest for one image, instruction, robot-state vector, and action chunk; name every axis, dtype, unit, frame, source clock, and allowed age.
Implement placeholder encoder functions that return fixed arrays with declared visual, language, and state shapes; assert the expected context position count and common feature width.
Create a four-by-three normalized action chunk, then implement a component-wise inverse transform whose stored ranges and statistics are selected by policy version and robot ID.
Deliberately swap image channels, remove one state component, use another policy's normalization range, and advance the clock past expiry; make each error fail at a different named contract.
Pass the decoded mock chunk through a simulator-only validator for finite values, dimension order, per-axis bounds, workspace, per-step change, and total horizon.
Save the manifest, shape trace, decoded values, rejection messages, simulated controller feedback, and a statement that this synthetic exercise does not establish model accuracy or hardware safety.

Observe

Most integration failures occur around the model—at preprocessing, time alignment, normalization, frame decoding, or output validation—even when the placeholder forward pass produces the expected tensor shape.

Done when

The nominal sample yields the hand-calculated 0.04 m component, all four corruptions fail explicitly, and a reviewer can follow each tensor and physical field from source through simulated feedback.

Build today

Build a tiny language-conditioned policy interface over your existing simulator or dataset.

Evidence to save

DONE when “Vision encoders, language tokens, and action outputs” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Assuming equal feature width means an image patch and word token are physically aligned.

Better

Feature dimensions only permit computation; grounding and evaluation must establish which visual entity and relation the language actually controls.

Wrong

Omitting robot state because the camera image appears to show the whole task.

Better

Include the proprioceptive and temporal state required to distinguish configurations, reachability, prior motion, and stale observations that pixels alone cannot recover reliably.

Wrong

Sending normalized model values directly to a controller.

Better

Use the matching policy and embodiment statistics, inverse representation processors, units, frames, control mode, expiry, and trajectory validator before any simulated command.

Job connection

How this becomes employable evidence

Integrate a multimodal policy by locking camera and tokenizer preprocessing, state history, tensor shapes, action statistics, decoder metadata, latency, and ROS 2 command semantics, then build contract tests around every boundary.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 18 interview drill

Interview questions: Vision encoders, language tokens, and action outputs

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Trace a camera image, instruction, and joint vector through a VLA that returns [1, 16, 7]. Explain every axis, timestamp and processor step, inverse-decode one value, and identify tests that prevent a correct-shaped but wrong-body command.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What does context shape `[1, 205, 1024]` establish in the worked example?

Model interview answer

It establishes one batch, 205 combined context positions, and feature width 1,024; it does not prove correct grounding, time alignment, or action quality.

Q2Why must relative conversion and normalization be reversed in the matching order?

Model interview answer

The model learned statistics for a specific representation, so inference must first restore that representation's physical scale and then reconstruct the corresponding absolute command with the current state.

Q3Which fields make a decoded `0.04` actionable rather than merely numeric?

Model interview answer

Its component name, metres unit, coordinate frame, relative or absolute mode, target robot, time and expiry, horizon, controller interface, and validated bounds.