Chapter 18 · Trace vision and language into bounded robot action

Today in the field story

One problem, then the next

Compare data from two robot embodiments without pretending their equally shaped action arrays mean the same thing. Preserve camera mounting, joint order, gripper type, units, frames, rate, calibration, controller semantics, and valid masks in embodiment manifests. Share the red-mug task label only where the success predicate is comparable, then route outputs through embodiment-specific adapters. Unknown mappings remain rejected rather than guessed.

Why now: Pretraining and dataset mixing require semantic alignment beyond a common storage envelope.
Ignore today: Ignore large-scale corpus construction; reconcile two small manifests and one task.
Unlocks next: Cross-embodiment examples whose differences remain explicit during adaptation and evaluation.

Understand

Build the physical picture first

Cross-embodiment learning is a multilingual workshop: tasks can be shared, but every robot keeps its own body dictionary and measurement grammar.

An embodiment is the robot's physical and sensing configuration: links, joints, gripper, cameras, mobile base, limits, control modes, and coordinate frames. Cross-embodiment pretraining combines experience from different bodies so a model can reuse visual, language, and task patterns. Open X-Embodiment standardized heterogeneous datasets and demonstrated transfer across multiple robot platforms; OpenVLA later pretrained on a large curated subset. That evidence motivates shared training, but it does not mean arbitrary robots can exchange raw numeric actions or that a new body works without an adapter and evaluation.

A common storage schema is only the outer envelope. Two records can both contain observation.state, action, image, task text, and timestamp while assigning different meaning to every vector element. One arm may store absolute joint targets in radians; another may store gripper-frame deltas in metres and radians. Cameras can be fixed or wrist-mounted, actions can be targets or velocities, and grippers can be continuous or binary. Preserve embodiment ID, component names, units, frames, rate, representation, valid mask, and calibration version before normalizing or padding anything.

Alignment separates what is legitimately shared from what stays body-specific. Language such as place cup in tray, object appearances, and high-level success predicates can cross bodies. An embodiment adapter maps that shared context to the observation and action dimensions supported by one robot. Canonical end-effector deltas can help when kinematics and control semantics are compatible, but canonicalization loses information if a body has no matching degree of freedom or uses different contact behavior. Mask unsupported components and keep the original record; never invent motion by copying a value into a convenient slot.

Data mixtures create another interface. A million trajectories do not contribute equally when one robot, task, camera view, or success mode dominates. Declare source filters and sampling weights, count the observed mix after loading, and retain license and lineage per source. Compare a local-only baseline, a cross-embodiment pretrained baseline, and any adapted model on the same frozen target-robot suite. Positive transfer means the shared model improves a defined target under comparable data and evaluation; multi-robot training alone is not proof.

Words you need

Name each idea precisely

Embodiment: The robot-specific body, sensors, actuators, kinematics, limits, frames, and control interfaces through which observations and actions acquire physical meaning.
Physical example:
A six-axis industrial arm with a parallel gripper and fixed camera is a different embodiment from a mobile manipulator with a wrist camera.
Trajectory: A time-ordered sequence of observations and actions, usually grouped into an episode with one task context and terminal outcome.
Physical example:
Thirty seconds of images, joint states, gripper commands, and timestamps document one cup-placement attempt, including its failure or success.
Cross-embodiment pretraining: Training on data from multiple robot bodies to learn reusable representations or policy behavior before target-specific adaptation or evaluation.
Physical example:
Visual patterns for identifying cups learned from several arms help initialize a policy for a new supported arm, while its action decoder remains body-specific.
Embodiment adapter: A component that maps one robot's observations and actions to and from the shared model representation while retaining body-specific semantics.
Physical example:
An adapter orders seven joint readings, adds a validity mask, and decodes the shared output into the gripper-frame delta expected by one controller.
Action canonicalization: Converting compatible robot actions into a shared representation while documenting information lost, masked, or handled by an adapter.
Physical example:
Two arms express tool motion as gripper-frame [dx, dy, dz, droll, dpitch, dyaw, grip] even though their joint-space commands differ.
Mixture weight: The intended sampling share assigned to one dataset, robot, task, or source during combined training.
Physical example:
A small target-arm dataset is sampled for 40 percent of steps even though it contains only 10 percent of stored episodes, and the observed rate is checked.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalToken, action, and latency budgets

Language reasoning must finish before the physical situation becomes stale.

T_total: end-to-end decision latencyUnit: milliseconds (ms)
f_control: safety/control update rateUnit: hertz (Hz)
H: action horizonUnit: steps or seconds

Perception takes 80 ms, model reasoning 220 ms, and skill dispatch 20 ms.
T_total = 80+220+20 = 320 ms.
At 1 m/s the robot moves 0.32 m during that delay; independent fast safety control must not wait for the LLM.

Programmer analogy

An LLM tool call resembles backend orchestration, but its latency must be converted into physical travel distance.

At 0.5 m/s, how far does a robot move during 400 ms?

400 ms = 0.4 s; distance = 0.5×0.4 = 0.2 m.

Robot A maps $0.75\,\mathrm{m}$ from $[0,1]$ metres to $a_{\mathrm{norm}}=0.5$ . Robot B maps $1.5\,\mathrm{rad}$ from $[0,2]$ radians to the same value:

2\frac{1.5-0}{2-0}-1=0.5.

Equal normalized values do not mean equal physical motions.

Align two robot records without pretending their actions are identical

Robot A stores relative gripper-frame x displacement in [-0.04, 0.04] m; Robot B stores an absolute first-joint target in [-2.0, 2.0] rad. Both rows carry the task move cup left and normalize scalars to [-1, 1].

Write the complete source schemas: Robot A's value names gripper-frame dx, relative position mode, metres, 10 Hz; Robot B's value names joint_1_target, absolute mode, radians, 20 Hz.
Normalize Robot A's 0.02 m as 2(0.02 - (-0.04)) / 0.08 - 1 = 0.5 and Robot B's 1.0 rad as 2(1 - (-2)) / 4 - 1 = 0.5.
Reject the tempting conclusion that the two 0.5 values mean the same motion: one is a local Cartesian displacement and one is an absolute joint configuration.
Keep shared task text and outcome labels in common fields, but place each action in a typed embodiment-specific structure with dimension names, representation, frame, units, and masks.
If the target model requires gripper-frame deltas, convert Robot B only through verified kinematics and time-aligned current pose; otherwise mark the source incompatible for that action component rather than fabricating a conversion.
Build equal-size target-robot evaluation slices and compare local-only, mixed-pretrained, and adapted checkpoints using identical prompts, start states, success rules, and trial counts.

Result

Both raw actions happen to normalize to 0.5, yet their typed records remain different until a physically justified adapter maps them; numeric scaling never substitutes for semantics.

What this proves

Normalize magnitudes after preserving body meaning, and claim transfer only from a controlled target-robot comparison.

Physical examples

Where this appears in real life

Short arm and long arm place the same cup

A compact tabletop arm and a longer seven-joint arm both demonstrate place the cup in the tray, but their cameras, reachable workspaces, and controller commands differ.

Look for:

Task language and success predicates can be shared, while calibration, state layout, action mapping, reachability, and safe limits stay attached to each embodiment.

Binary and continuous grippers

One dataset records gripper state as 0 or 1; another records jaw width from 0.00 m to 0.08 m and measured grip force separately.

Look for:

A normalized value of 0.5 cannot be copied between schemas; the adapter must preserve discrete mode, physical width, force meaning, and unsupported fields.

Hands-on exercise

Make the idea observable

Create twelve synthetic episode cards for two imaginary arms. Use JSON or a spreadsheet export plus a small validation script; do not download or mix real datasets.

Define embodiment manifests covering robot ID, joint names, camera mounting, state order, action order, units, frames, rate, representation, limits, gripper type, and calibration revision.
Create six episodes per robot across two shared task labels, including one failure each; attach episode ID, timestamps, task text, terminal predicate, and data lineage.
Normalize one compatible scalar per body with its own stored range, round-trip it to the original physical value, and prove that equal normalized values can retain different meanings.
Construct a common record with task and image references plus typed body-specific state and action sections, padding only with an explicit validity mask.
Choose mixture weights, sample twenty episode IDs with a fixed seed, and compare intended versus observed shares by robot, task, and success status.
Freeze four target-robot episodes as evaluation-only, verify their IDs and near-duplicate scenes do not appear in the training mix, and document which transfer claim this tiny exercise cannot support.

Observe

A shared column layout hides substantial physical differences, and sampling counts often depart from the intended balance unless embodiment, task, and outcome are inspected together.

Done when

Every synthetic row round-trips through its own adapter, incompatible actions remain masked rather than guessed, the mixture count is reproducible, and held-out target episodes have no lineage overlap with training.

Build today

Build a tiny language-conditioned policy interface over your existing simulator or dataset.

Evidence to save

DONE when a deterministic “Robot data pretraining and cross-embodiment learning” failure test reports expected versus actual behavior and passes after the documented fix.

Common mistakes

Catch the wrong mental model

Wrong

Concatenating same-length action arrays from different robots.

Better

Compare component names, units, frames, control modes, rates, and body kinematics first; use a verified adapter or retain an explicit incompatibility mask.

Wrong

Calling equal normalized values equivalent physical actions.

Better

Normalization aligns numeric ranges only; inverse-decode with per-embodiment metadata and preserve the original action semantics.

Wrong

Reporting cross-embodiment benefit because a mixed model was trained successfully.

Better

Use frozen target-robot trials and comparable local-only, mixed-pretrained, and adapted baselines, then report each held-out slice and failed trial.

Job connection

How this becomes employable evidence

Onboard a new robot dataset by writing its embodiment manifest, mapping observations and actions into the policy processors, validating statistics and masks, auditing mixture balance, and measuring transfer against a target-only baseline.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 18 interview drill

Interview questions: Robot data pretraining and cross-embodiment learning

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Two robot datasets both expose a seven-number action vector normalized to [-1,1]. Explain why concatenation is unsafe, design an embodiment-aware adapter, and propose an evaluation that distinguishes positive transfer from dataset leakage.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why do both worked actions normalize to 0.5 without becoming comparable commands?

Model interview answer

Their ranges scale similarly, but one value is a relative Cartesian displacement in metres and the other an absolute joint target in radians.

Q2Which information can often be shared more safely across embodiments than raw actions?

Model interview answer

Task language, some visual concepts, object relations, and high-level success predicates can be shared while body observations and action decoding stay typed and adapter-controlled.

Q3What comparison provides evidence of positive cross-embodiment transfer?

Model interview answer

On one frozen target-robot suite, a mixed-pretrained or adapted model must outperform a comparable target-only baseline under the same data accounting and acceptance rules.