Chapter 14

Collect demonstrations as synchronized, reviewable robot datasets

Treat robot demonstrations as an engineered measurement product rather than a folder of convenient videos. This chapter defines comparable episodes, collects bounded teleoperation traces, makes action semantics explicit, tests clock alignment, versions a training-safe schema and split, audits failures and coverage, then publishes a dataset card and episode inspector whose claims can be checked against the underlying records.

Before you start

Complete Week 13's tensor, dataset, dataloader, split, leakage, experiment-lineage, and model-card exercises using saved or synthetic data.
Recognize observations, robot state, commands, frames, units, timestamps, rates, latency, reset conditions, success criteria, safe aborts, and independent evaluation evidence.
Use a spreadsheet or a short Python program to validate rows, calculate counts and time differences, and create simple tables without assuming a GPU.
Keep this week's collection unpowered, simulated, or properly supervised on an already commissioned platform; a recording command is not a motion-safety system.

By the end

Define policy, observation, action, episode, trajectory, reset, and terminal semantics so every recorded attempt has a comparable physical meaning.
Design a teleoperation collection path that preserves operator input, robot response, calibration identity, interventions, task result, and safe cancellation without confusing demonstration success with autonomous capability.
Specify joint, Cartesian, velocity, and chunked action spaces with ordered components, units, frames, limits, rate, and execution semantics.
Measure alignment among camera, state, and command streams using acquisition timestamps, clock-domain evidence, a declared tolerance, missing-sample counts, and causal lag checks.
Create a versioned multimodal schema and leakage-resistant episode split whose feature names, shapes, dtypes, units, frames, metadata, and grouping rules are machine-checkable.
Review structural defects, failed behavior, labels, operator habits, condition coverage, and distribution shift without deleting inconvenient evidence or mistaking augmentation for collected coverage.
Ship a non-sensitive dataset card and synchronized episode inspector with provenance, licensing, exclusions, limitations, checksums, and recalculable duration and coverage summaries.

The field story

Freeze the careful-handover dataset

The seal-defect scout predicted labels from fixed observations; now a robot-learning system needs synchronized examples of decisions over time. In a simulated handover, an operator guides a gripper toward a foam block, pauses when the receiver is not ready, and places it inside a marked tray. Episode 014 looks successful on video, yet its action units, clock alignment, operator takeover, and terminal meaning are undocumented. This week turns that attractive recording into a dataset another policy engineer can inspect without guessing.

Follow one release candidate from task definition to frozen card. You will define episode boundaries, collect bounded demonstrations, distinguish joint and Cartesian commands, align camera, state, and action clocks, validate a versioned schema, group splits by physical context, and retain failed attempts. The final inspector must reveal saturation, lag, missing samples, interventions, exclusions, and privacy decisions on one timeline. Week 15 will train from this exact evidence, so every shortcut taken here becomes a policy failure with a traceable origin.

Why this chapter now: Week 13 established tensors, grouped splits, and model evidence; imitation learning now requires temporally aligned robot observations and actions with explicit physical semantics.
Ignore for now: Do not train a policy, maximize episode count, or collect unsupervised powered motion. Build a small, reviewable dataset whose defects remain visible.
This unlocks: The frozen handover data becomes the direct input to Week 15 behavior cloning, correction, action-chunk, and rollout comparisons.
Proof you will leave with: Provide task and episode contracts, recorder revision, operator and intervention fields, action schema, clock residuals, manifests, grouped splits, failure and coverage ledger, privacy and rights disposition, checksums, card, inspector, and rerun command.

Environment contractRepository-supported Node.js 22.13.0 or newer runs the synchronization starter. Collection remains synthetic, simulated on ROS 2 Jazzy with Gazebo Harmonic, or supervised on an already commissioned platform; the guide itself requires no powered robot.

Compatibility boundary: Recorder fields, image encodings, timestamp clocks, action order, units, frames, and dataset tooling can drift. Pin the actual schema and conversion code; do not treat a current documentation branch as a frozen data format.
Smoke check: Run node week-14-audit-handover-sync.mjs, validate one frozen episode against the schema, recalculate its duration and residuals, and verify the inspector shows the same missing and rejected samples.
Contract reviewed: 2026-07-25
Runtime evidence: The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
Drift risk: high

Today in the field story

One problem, then the next

Define the careful handover before recording more video. Name the policy input, physical action, reset, success, safe abort, episode start, terminal state, and trajectory timing for Episode 014. A green tray outcome does not explain what the operator saw or commanded. The mission needs comparable attempts whose boundaries and semantics survive export, review, and the Week 15 training loader.

Why now: Data rows are useful only after the task and episode units have stable physical meaning.
Ignore today: Ignore model architecture and dataset scale; define one bounded handover contract.
Unlocks next: Comparable episodes that can be collected, split, and scored.

Understand

Build the physical picture first

A robot-learning episode is a labelled attempt envelope: one reproducible reset goes in, an ordered observation-action history unfolds, and exactly one success, timeout, failure, or safe-abort boundary closes it.

A policy is a rule that chooses an action from available observations and possibly recent history. An observation is information exposed to that rule, such as camera pixels, joint positions, gripper state, or a task instruction. An action is the command contract produced for execution. Keep commanded action separate from measured robot state because limits, latency, contact, disconnection, or an operator intervention can make the physical response differ.

An episode represents one bounded attempt, not an arbitrary slice from a continuous log. Its reset specification states the starting object arrangement, robot mode, task text, sensor readiness, and anything randomized. Its terminal specification distinguishes success, timeout, task failure, operator abort, safety stop, and recording fault. The trajectory is the time-ordered sequence inside those boundaries, including observations, issued actions, measured state, and relevant event labels.

Sampling rate describes intended cadence, while timestamps describe when individual records claim to occur. A stream configured to collect 20 samples each second can produce 600 samples in a nominal 30-second acquisition window, although 600 instantaneous samples contain 599 between-sample intervals and therefore normally span 29.95 seconds from the first timestamp to the last. That arithmetic detects some gaps and boundary errors; it cannot establish that the images, states, actions, or outcome labels are semantically correct.

Episodes are the natural unit for comparison and splitting because nearby frames share the same reset, operator behavior, scene, and outcome. Preserve aborted or malformed attempts in a raw ledger with explicit dispositions rather than silently renumbering good attempts. A training view may exclude a bad episode, but the exclusion rule, original identity, reviewer, and reason remain part of dataset lineage.

Words you need

Name each idea precisely

Policy: A rule or learned function that maps current observations and optional history or task context to an action.
Physical example:
A sorting policy sees a camera image plus joint state and proposes the next bounded gripper command.
Observation: The timestamped information made available to a policy, with declared feature semantics rather than every hidden fact about the world.
Physical example:
A front image and six measured joint angles are observations, while an object pose known only to the simulator can remain evaluation truth.
Action: A command expressed through a specific representation, order, unit, frame, limit, and execution rule.
Physical example:
A six-value joint-position target in radians is a different action from six measured joint positions or six joint velocities.
Episode: One attempt bounded by an explicit reset and one terminal outcome, with a stable identity and task context.
Physical example:
Episode 014 begins after three blocks return to marked zones and ends when all are sorted, time expires, or the operator aborts.
Trajectory: The ordered observations, actions, measured responses, and events recorded through an episode.
Physical example:
A paper arm's timestamped joint-angle and waypoint rows form a trajectory from home pose to the target card.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalEpisode timing, normalization, and split leakage

A policy learns the dataset you actually recorded, including hidden leakage and timing errors.

z = (x−μ)/σ: standardized valueUnit: unitless
t: source timestampUnit: seconds (s)
N: number of independent episodesUnit: episodes

For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.

Programmer analogy

It resembles a production event log, but camera, state, and action streams must describe the same instant.

What is z for x=8, μ=10, σ=2?

(8−10)/2 = −1.

With $N$ samples recorded at rate $f$ , episode duration is

t=\frac{N}{f}=\frac{600}{20\,\mathrm{Hz}}=30\,\mathrm{s}.

This checks timing only; the reset and terminal event still define the episode boundary.

Audit one episode's time and boundary claims

Episode 014 claims a 20 Hz rate and contains 600 ordered rows. Its first timestamp is 42.000 s, its last is 71.950 s, reset-ready is recorded at 41.600 s, and success is recorded at 71.970 s.

Check identity and order first: every row belongs to episode 014, frame indexes are unique and increasing, and only one reset-ready event precedes the first row.
Calculate nominal sampled duration as N / f = 600 / 20 = 30.00 s.
Calculate first-to-last timestamp span as 71.950 - 42.000 = 29.950 s, which equals (N - 1) / f for 600 evenly spaced samples.
Check the boundary order: reset-ready at 41.600 s occurs before recording, and success at 71.970 s occurs 0.020 s after the last sampled state.
Verify every adjacent timestamp difference against the expected 0.050 s period and report the largest gap instead of trusting only the two endpoints.
Accept the structural timing example only if the reset and success definitions also match the physical task; correct arithmetic cannot validate a false outcome label.

Result

The row count, timestamp span, and boundary ordering are mutually consistent with a 30-second, 20 Hz attempt, pending per-gap and semantic review.

What this proves

Episode integrity requires both numerical continuity and meaningful physical boundaries; neither one can substitute for the other.

Physical examples

Where this appears in real life

Paper-block sorting attempt

Place three coloured paper squares on marked start boxes, move them one step at a time with a cardboard gripper, and stop at complete sorting, a 20-second timeout, a dropped square, or a declared abort.

Look for:

The reset and four distinct terminal results make one attempt comparable with the next; an arbitrary ten-row window would not carry that meaning.

Continuous kitchen video versus one task

Imagine a fixed camera recording an hour of countertop activity that contains several cup placements, pauses, unrelated people, resets, and one spill.

Look for:

The long file is storage, not one episode; task attempts require their own boundaries, identities, conditions, and outcomes even when several share a media file.

Hands-on exercise

Make the idea observable

Use paper blocks, a cardboard pointer, and a spreadsheet or small local script. Keep powered motion disabled and choose a 2 Hz manual sampling cadence.

Write a reset contract with exact start zones, object count, pointer home mark, task instruction, timeout, and permitted terminal outcomes.
Perform three attempts while a partner records episode ID, frame index, timestamp, visible arrangement, intended move, observed move, and event label.
End the attempts differently—one success, one timeout, and one deliberate abort—without relabelling the latter two as successful demonstrations.
Validate unique frame indexes, increasing timestamps, one start boundary, one terminal boundary, and the expected N / f duration for each attempt.
Create a raw episode ledger and a separate training-eligibility column; state a reason for every exclusion without deleting the original row identity.
Exchange the ledger with another person and ask them to reconstruct each reset, trajectory, terminal condition, duration, and eligibility decision.

Observe

Rows from the same physical activity gain different engineering meaning when reset or terminal rules change, and a structurally complete attempt can still be behaviorally unsuitable for training.

Done when

All three attempts are independently reconstructable, duration calculations agree with timestamps within the declared manual tolerance, and every terminal and exclusion label has evidence.

Build today

Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.

Evidence to save

DONE when the learning log explains “Policy, observation, action, episode, and trajectory” in five precise points and a checked example produces the predicted output.

Common mistakes

Catch the wrong mental model

Wrong

Calling every fixed-length window an episode.

Better

Require a physical reset/start contract and one meaningful terminal outcome; use windows only as model inputs sampled within an already defined episode.

Wrong

Treating commanded action as the robot's observed trajectory.

Better

Store issued actions and measured state as different features so delay, saturation, contact, rejection, and intervention remain visible.

Wrong

Deleting aborted attempts and renumbering the remaining files.

Better

Keep immutable raw identities and a review ledger, then build a derived training view whose exclusions and reasons can be reproduced.

Job connection

How this becomes employable evidence

Define the episode contract consumed by a robot-learning pipeline, ensure recorder and evaluator agree on resets and terminal outcomes, and retain aborts or corrupt attempts with traceable dispositions instead of presenting a curated success-only history.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robotics Deployment, Integration & Validation Engineer

Chapter 14 interview drill

Interview questions: Policy, observation, action, episode, and trajectory

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

You receive a two-hour robot log and are asked to train from it. Explain how you would derive episodes, preserve reset and outcome evidence, separate commands from measured state, and prevent nearby frames from leaking across evaluation.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What makes an episode more than a slice of rows?

Model interview answer

It represents one attempt with an explicit reset/start condition, task context, ordered trajectory, stable identity, and exactly one terminal outcome.

Q2Why can `600 / 20 = 30 s` coexist with a 29.95 s first-to-last timestamp span?

Model interview answer

The quotient describes a nominal 30-second acquisition window at 20 samples per second, while 600 instantaneous samples contain only 599 intervals between their first and last timestamps.

Q3Should a safe abort disappear from the raw dataset?

Model interview answer

No. Retain its raw identity and evidence, label its terminal result, and document whether and why a derived training view excludes it.

Chapter references

LeRobot — Getting Started with Real-World RobotsMaintainer workflow for calibration identity, teleoperation, camera and joint visualization, bounded episode recording, reset time, cancellation and re-recording, local storage, and explicit Hub-upload behavior.
LeRobotDataset v3 format — maintainer specificationPrimary format description for Parquet sensorimotor rows, MP4 camera streams, timestamps, canonical feature metadata, task and episode indexes, file-versus-episode boundaries, temporal windows, and finalization.
ROS 2 message_filters — timestamp synchronization APIOfficial semantics for exact and approximate synchronization by message-header timestamps, queueing, tolerance, clock offsets, and the risk of substituting arrival time for acquisition time.
LeRobot — Dataset Tools and VisualizationMaintainer tooling contract for inspecting feature metadata, deleting or splitting whole episodes, preserving originals, and viewing camera, state, and action streams on a seekable episode timeline.
Hugging Face Hub — Dataset CardsOfficial dataset-card and README metadata behavior for licensing, discoverability, context, responsible use, bias and limitation disclosure, and supported dataset modalities.