Chapter 14 · Collect demonstrations as synchronized, reviewable robot datasets

Today in the field story

One problem, then the next

Encode handover-v1 as a versioned schema covering images, state, action, task text, timestamps, frames, units, calibration, operator, intervention, and terminal result. Split by episode family, scene, object, and collection session so near-identical attempts do not leak. Compute normalization statistics from training data only. A schema change produces a new version and migration record, not a silent reinterpretation of old episodes.

Why now: Aligned samples still need a stable format and leakage-resistant evidence boundary.
Ignore today: Ignore universal dataset standards; make this release explicit and convertible without guessing.
Unlocks next: A frozen train, validation, and held-out manifest ready for loaders.

Understand

Build the physical picture first

A dataset schema is the wiring diagram for recorded experience: it names every signal, fixes its type and physical meaning, relates rows to episodes and tasks, and prevents training code from silently reconnecting the wires.

A useful schema specifies feature name, semantic description, dtype, shape, component order, unit, coordinate frame, valid range or enum, missing-value policy, timestamp meaning, and whether the value is observation, action, metadata, label, or evaluation-only truth. The familiar LeRobot keys observation.state, action, observation.images.<camera>, and timestamp provide a naming structure; each dataset still must document what its particular state and action components physically mean.

LeRobotDataset v3 separates storage layout from episode access. Low-dimensional state, action, and timestamps live in Parquet data; camera frames live in MP4 video; metadata records canonical features, frame rate, tasks, statistics, and episode segmentation. Multiple episodes can share one physical data or video file, and one episode view is reconstructed from metadata offsets. Therefore a file name is not an episode boundary, and manually moving shards is not a valid way to edit episode membership.

Version the schema whenever a consumer could interpret a stored value differently. Adding an optional label may be compatible, while reordering joints, changing metres to millimetres, switching from absolute to delta actions, or redefining timestamp origin is not. Store code and schema versions with the data, validate before loading, and write an explicit migration that produces a new derived version. Never relabel old bytes as the new contract without a reversible transformation and audit result.

Split whole episodes, then strengthen grouping around the generalization claim. Frames from one episode must not cross train, validation, and test, but that alone can still leak a nearly identical reset, scene, object instance, or operator technique. Choose grouping keys before looking at model results. Fit normalization statistics on the training split only, freeze validation for decisions, and keep test untouched until the final evaluation protocol calls for it.

Words you need

Name each idea precisely

Dataset schema: The machine-checkable and human-readable contract for feature names, structures, physical semantics, relationships, and required metadata.
Physical example:
A schema defines observation.state as six float32 joint angles ordered by name and measured in radians.
Feature: One consistently interpreted field or modality exposed to storage and consumers under the schema.
Physical example:
observation.images.front is an RGB camera stream, while action is a different feature with a seven-value control contract.
Episode metadata: Records that identify episode length, task, outcome, conditions, offsets, and lineage without pretending each episode is one storage file.
Physical example:
Episode 27 points to rows 14,200–14,799 and matching video offsets inside shards shared with other attempts.
Schema version: A stable identifier for one interpretation of stored fields, changed when compatibility or physical meaning changes.
Physical example:
Version 2 changes action translation from centimetres to metres and therefore requires conversion rather than a metadata-only rename.
Grouped split: A train, validation, or test assignment made at episode level and also keeping a leakage-prone related group together.
Physical example:
All five attempts from the same randomized reset seed remain in one split even when their camera frames differ.
Evaluation leakage: Information from validation or test influences training, preprocessing, selection, or tuning and makes measured generalization too optimistic.
Physical example:
Computing action-normalization bounds from all episodes lets held-out extremes shape the training transform.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalEpisode timing, normalization, and split leakage

A policy learns the dataset you actually recorded, including hidden leakage and timing errors.

z = (x−μ)/σ: standardized valueUnit: unitless
t: source timestampUnit: seconds (s)
N: number of independent episodesUnit: episodes

For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.

Programmer analogy

It resembles a production event log, but camera, state, and action streams must describe the same instant.

What is z for x=8, μ=10, σ=2?

(8−10)/2 = −1.

A schema can verify the identity

t=\frac{N}{f}=\frac{600}{20\,\mathrm{Hz}}=30\,\mathrm{s}.

For $100$ episodes, a $70/15/15$ split gives $70$ train, $15$ validation, and $15$ test episodes. Related frames must still remain in the same split.

Build a 70/15/15 split without leaking reset groups

A dataset contains 100 episodes from 20 reset seeds, with five attempts per seed. The required episode counts are 70 train, 15 validation, and 15 test. Episodes sharing a reset seed are near-duplicates and must stay together.

Choose reset_seed as a grouping key before inspecting model metrics, because the five attempts from one seed share object layout and camera background.
Verify group size: 100 episodes / 20 seeds = 5 episodes per seed.
Assign 14 complete seed groups to training: 14 × 5 = 70 episodes.
Assign three unused seed groups to validation: 3 × 5 = 15 episodes.
Assign the remaining three groups to test: 3 × 5 = 15 episodes, then assert no seed appears in more than one split.
Calculate normalization statistics from the 70 training episodes only and record the immutable seed-to-split manifest with the dataset version.

Result

The split reaches exactly 70/15/15 episodes while keeping all five attempts from each reset seed in one partition.

What this proves

Correct proportions do not prevent leakage; the split unit must match the dependency that could make held-out data familiar.

Physical examples

Where this appears in real life

Versioned episode index cards

Create paper cards for six episodes and separate cards for schema, tasks, data shards, and video shards; let several episode cards point into the same shard cards by start and end offsets.

Look for:

Episode identity comes from relational metadata and boundaries, not from assuming one card or file contains exactly one attempt.

Rewired sensor connector

Label a six-pin paper connector with joint names, then secretly swap pins two and five while keeping all numeric readings within the same range.

Look for:

Shape and dtype validation still pass, but component-order semantics fail, showing why a versioned name-to-index contract is essential.

Hands-on exercise

Make the idea observable

Create six tiny synthetic or paper episodes. Use JSON, CSV plus a schema note, or a small validation script; installing LeRobot is optional.

Define observation.state, action, observation.images.front, timestamp, task, episode_id, frame_index, outcome, and reset_seed with types, shapes, units, frames, enums, and missing rules.
Place rows from several episodes in two imaginary data shards and camera frames in two video shards, then create metadata offsets that reconstruct each episode.
Plant a swapped action component, a millimetre value under a metre schema, a duplicate frame index, a missing task, and an offset that crosses an episode boundary.
Run or manually execute validation that reports a distinct path, expected contract, actual value, and disposition for every planted defect.
Assign whole reset-seed groups to train, validation, and test; assert disjoint episode IDs and groups, then calculate statistics from training rows only.
Create schema version 2 with one changed action unit and write a conversion ledger that preserves version 1, output identity, formula, count, and round-trip check.

Observe

Storage can consolidate many attempts without erasing episode views, while semantic defects evade simple dtype and row-count checks unless the schema validates physical meaning and relationships.

Done when

All five defects are detected, every episode reconstructs from offsets, split groups are disjoint, and the version-2 migration is reproducible without modifying version 1.

Build today

Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.

Evidence to save

DONE when a comparison table for “Dataset schema, metadata, splits, and versioning” contains the test condition, metric, result, and justified engineering decision.

Common mistakes

Catch the wrong mental model

Wrong

Assuming a matching feature name, dtype, and tensor length guarantee compatibility.

Better

Compare ordered component semantics, units, frames, normalization, timing, limits, missing policy, and schema version before accepting the data.

Wrong

Splitting individual frames randomly into train and test.

Better

Assign whole episodes and all related reset, scene, object, or operator groups together so near-duplicate trajectories cannot cross the evaluation boundary.

Wrong

Treating every Parquet or MP4 file as one episode.

Better

Resolve episode rows and video ranges from versioned metadata because scalable formats can store many episodes inside shared shards.

Wrong

Computing normalization statistics over the complete dataset.

Better

Fit means, scales, and bounds on training data only, then apply the frozen transform to validation and test.

Job connection

How this becomes employable evidence

Own the contract among recorder, dataset storage, PyTorch loader, policy runtime, and evaluation system by versioning feature semantics, validating relational episode metadata, preventing grouped leakage, and making migrations reversible.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 14 interview drill

Interview questions: Dataset schema, metadata, splits, and versioning

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Design a schema for two cameras, six joint states, seven actions, task text, and outcomes. Explain file-versus-episode boundaries, compatibility rules, grouped splits, training-only statistics, and how you would detect a swapped joint order.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can two action arrays with shape `[7]` still be incompatible?

Model interview answer

Their component order, representation, units, frame, normalization, limits, or absolute-versus-delta meaning may differ despite equal shape.

Q2Why is a data-shard boundary not necessarily an episode boundary in LeRobotDataset v3?

Model interview answer

The format can concatenate many episodes into larger Parquet and MP4 files; episode metadata and offsets reconstruct each attempt.

Q3When is an episode-only split still too weak?

Model interview answer

When related episodes share a reset, scene, object instance, operator pattern, or other dependency that would make held-out data effectively familiar.