Chapter 19 · Adapt a current VLA through explicit data, action, and release contracts
Today in the field story
One problem, then the next
The chosen passport reaches the data customs desk, where two robots both expose a field named action but mean different physical commands. You preserve raw records, language, units, frames, rates, camera order, and embodiment identity, then describe adapters instead of forcing unlike bodies into convenient columns. Sampling weights remain visible so a large dataset cannot silently dominate. The mission continues with a mixed-data manifest that can be reversed and audited without inventing motion for an unsupported degree of freedom.
- Why now
Adaptation is meaningless if the same feature name carries incompatible physical semantics.
- Ignore today
Do not merge rows by shape alone or claim cross-embodiment transfer.
- Unlocks next
A versioned training view whose transformations are explicit.
Understand
Build the physical picture first
Cross-embodiment training is multilingual translation with machinery: the task may be shared, but every body retains its own sensors, grammar of action, calibration, and legal motion.
An embodiment is the physical and computational body that gives numbers meaning: cameras and their poses, ordered state fields, joints, tool, control mode, action dimension, coordinate frame, units, rate, limits, and latency. Two datasets can both use a key named action while one stores joint-position targets in radians and the other stores Cartesian deltas in metres plus gripper effort. Concatenating those rows without an embodiment ID and conversion contract teaches contradictory targets that no sampling trick can repair.
Language provides task context only when it is aligned to the correct episode or temporal event. “Put the red cup in the tray” can describe a whole attempt, while a correction such as “open the gripper now” belongs to a particular interval. Camera-grounded text must name the view it describes, and task vocabulary should be reviewed for aliases, negation, object identity, and success meaning. Adding fluent captions after collection does not prove the action sequence actually performs the captioned task.
A mixture has both a schema and a sampling policy. If robot A supplies 8,000 episodes and robot B supplies 2,000, uniform episode sampling yields an expected 80/20 contribution. Equal embodiment sampling changes it to 50/50 but repeats B examples more often, potentially overfitting them. Report the planned and observed draws by embodiment, task, outcome, operator, and condition; otherwise one large source can dominate loss while the final model is advertised as cross-embodiment.
Use a shared canonical layer only for semantics that can be stated honestly, then keep embodiment adapters around it. A dataset transform maps raw fields into named policy features; an output adapter converts the model representation back into this robot's ordered command. Preserve raw source, version each transform, compute training-only statistics by field, and reject rows that cannot be converted without guessing. Cross-embodiment learning does not erase calibration, controller, collision, or safety ownership.
Words you need
Name each idea precisely
- Robot embodiment
The robot-specific combination of body, sensors, state representation, action interface, calibration, rate, limits, and environment interaction.
Physical example:A six-joint arm with a parallel gripper and wrist camera has a different embodiment contract from a mobile manipulator with a lift and base velocity controller.
- Modality configuration
Versioned metadata that maps stored image, state, action, and language fields to the named inputs and outputs expected by a policy.
Physical example:A GR00T modality file states which array slice is end-effector translation, rotation, and gripper state for the selected embodiment tag.
- Sampling weight
The target probability that a dataset, embodiment, task, or subgroup contributes an example during training.
Physical example:A loader draws the small arm and large arm with probability 0.5 each even though their stored episode counts differ four to one.
- Task annotation
Language tied to a declared episode, frame interval, camera view, or event and used as policy context or a supervised target.
Physical example:The episode task says “stack two cubes,” while a timestamped correction says “release the blue cube” only during the final placement.
- Dataset provenance
Traceable origin and transformation history for data, including collector, robot, software, license, filtering, repair, and derived versions.
Physical example:A training row links back to episode B-014, teleoperator 3, calibration C7, source checksum, and transform revision 2.
- Robot embodiment adapter
A tested conversion between one robot's native observation/action contract and the representation consumed or emitted by a policy.
Physical example:One adapter orders a Panda's Cartesian delta and gripper fields; another maps an SO-101's joint targets without pretending the vectors have the same semantics.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalFine-tuning size, normalization, and evidence
Model adaptation must fit compute limits and improve frozen physical trials.
- P_train
- parameters updated during tuningUnit: parameters
- GB
- memory footprintUnit: gigabytes
- Δsuccess
- new minus baseline success rateUnit: percentage points
A baseline succeeds 12/20 = 60%; adaptation succeeds 16/20 = 80%.
Improvement is 80%−60% = 20 percentage points, not 20 percent.
Report parameter count, memory, latency, and the same frozen scenarios before claiming improvement.
Treat a policy release like a mobile release: same acceptance suite, device budget, rollback path, and versioned artifact.
A metric rises from 50% to 65%. What is the percentage-point gain?
15 percentage points.
Uniform row sampling gives
Equal embodiment sampling instead sets .
Design a two-robot mixture without corrupting action meaning
Robot A has 8,000 episodes of seven joint-position targets at 20 Hz. Robot B has 2,000 episodes of six Cartesian deltas plus one gripper command at 10 Hz.
Write separate raw schemas with ordered component names, dtype, shape, unit, frame, command mode, rate, absolute and delta limits, calibration identity, and timestamp convention.
Assign stable embodiment IDs and transform versions; keep the original action arrays immutable rather than overwriting both into an unexplained seven-number column.
Define shared features only where meaning agrees, such as normalized RGB views and episode-level task text, then implement separate state and action adapters for A and B.
Calculate uniform episode exposure:
8,000 / 10,000 = 0.80for A and2,000 / 10,000 = 0.20for B; calculate equal-embodiment exposure as 0.50 and 0.50.Choose a declared mixture, log actual sampled counts by embodiment and task, and inspect whether repeated B episodes create a train-versus-validation gap.
Round-trip five held-out rows through each adapter and require exact field order plus toleranced numeric recovery before either source enters the training view.
The mixture can share task semantics and selected visual features while every command retains an auditable route back to one robot's physical contract.
Cross-embodiment scale comes from explicit translation and measured sampling, not from stacking heterogeneous numeric arrays under common key names.
Physical examples
Where this appears in real life
Same sentence, different command bodies
A Panda arm and an SO-101 both demonstrate “move the red cube left,” but one log stores seven joint targets and the other stores end-effector deltas plus a gripper value.
The shared task text stays comparable, while embodiment IDs, action schemas, transforms, units, rates, and decoders remain distinct and independently tested.
A large dataset silences a rare robot
A mixture contains four times as many successful episodes from robot A as robot B and is shuffled uniformly by episode.
Observed batches approach 80/20, so metrics and gradients mostly describe A; reweighting changes exposure and must be paired with duplication and overfitting checks for B.
Hands-on exercise
Make the idea observable
Create ten synthetic rows for each of two imaginary arms. Use harmless numbers and a local table or short script; no robot connection or model training is required.
Document each arm's cameras, ordered state, ordered action, units, frames, rate, limits, embodiment ID, calibration ID, and source episode.
Give both arms two shared task phrases, then add one episode-level annotation, one timestamped correction, and one camera-specific description with unambiguous scope.
Write adapter pseudocode that maps each raw schema into canonical named features and maps predicted output back to that arm without discarding units or order.
Produce 20 mock sample draws under uniform-row and equal-embodiment rules, compare planned with observed shares, and state why the small count can deviate.
Inject a degree-versus-radian row, missing embodiment ID, renamed camera, and action-length mismatch; require quarantine with a named reason rather than coercion.
Save a mixture manifest containing source checksums, licenses, transforms, weights, exclusions, statistics split, and the five round-trip cases for peer review.
The exercise reveals that shared language and array length are weak compatibility signals; most physical meaning lives in metadata, adapters, and sampling evidence.
Both sources retain reconstructable physical semantics, corrupt rows fail closed, planned and observed mixture weights are visible, and held-out adapter round trips pass.
Build today
Benchmark a supported LeRobot/OpenVLA policy, trace its processors and normalized actions, then design or run a LoRA adaptation with a frozen baseline.
Evidence to save
DONE when “Cross-embodiment data, language columns, and dataset mixing” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Assuming equal action length means equal action semantics.
Compare ordered component names, command mode, unit, frame, rate, limits, and calibration; preserve separate embodiment adapters even when shapes happen to match.
Reporting only the number of mixture episodes.
Publish planned and observed sampling shares by embodiment, task, outcome, and condition, because training exposure—not storage count alone—drives the optimization.
Treating generated language as ground truth for a trajectory.
Validate every task or event annotation against episode timing and outcome, retain provenance and confidence, and quarantine descriptions whose physical referent cannot be established.
Job connection
How this becomes employable evidence
Build and review a multi-robot training manifest whose modality transforms, language alignment, sampling logs, provenance, and per-embodiment action adapters make every learned target physically interpretable.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robotics Deployment, Integration & Validation Engineer
Chapter 19 interview drill
Interview questions: Cross-embodiment data, language columns, and dataset mixing
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Two robot datasets both expose a seven-value action and the same English tasks. Explain why they cannot simply be concatenated, how you would design adapters and sampling weights, and which round-trip tests block training.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What remains robot-specific after a shared task representation is chosen?
Sensor layout, calibration, feature order, state/action representation, units, frames, rate, limits, controller behavior, and the adapters that translate policy data into valid commands.
Q2What training share results from uniform episode sampling over 8,000 A and 2,000 B episodes?
The expected shares are 80% A and 20% B, subject to finite-sample variation and any filtering performed by the loader.
Q3Why can equal-embodiment weighting also create a problem?
The smaller source is sampled repeatedly, so it can overfit; training and held-out metrics plus actual draw counts must be inspected per embodiment.