Chapter 14 · Collect demonstrations as synchronized, reviewable robot datasets
Today in the field story
One problem, then the next
Record several simulated handovers while preserving operator input, robot command, measured state, calibration identity, intervention, and outcome as different fields. Include a receiver-not-ready abort and a late takeover rather than saving only the clean place. The operator’s corrective motion is not automatically an expert label. Capture the exact control authority and cancellation path so later policy work can distinguish teaching from emergency recovery.
- Why now
Collection procedure determines which actions and failures the dataset can honestly represent.
- Ignore today
Ignore autonomous rollout; establish safe, attributable demonstration capture.
- Unlocks next
A first batch with operator and intervention lineage.
Understand
Build the physical picture first
Teleoperation is a measured command chain between a human and a robot: operator intent enters through an interface, bounded commands leave the controller, physical state responds later, and the recorder must preserve every link.
Teleoperation lets a person command a robot through a leader arm, joystick, keyboard, phone, or other interface. A demonstration is the recorded evidence of that interaction, not merely the operator's input. Useful records include operator command, robot observation and measured state, task instruction, camera views, timestamps, terminal result, and intervention notes. The same input can produce different motion when calibration, limits, contact, delay, or hardware state changes.
Calibrate and identify the robot, teleoperator, cameras, and software configuration before collection. LeRobot's maintained workflow uses robot and teleoperator IDs to locate calibration records and offers simultaneous camera and joint visualization before recording. Matching IDs and a visible response are necessary setup evidence, but neither proves safe workspace limits, collision handling, emergency stopping, load capability, or permission to operate a powered mechanism.
Collection needs a written protocol: prepare the reset, verify sensing, announce recording, perform one consistent task, stop on success or a named failure, and restore the scene during reset time. Record operator identity or an anonymized stable code because technique can become a learnable shortcut. Mark assistance, pauses, corrections, communication loss, and safety stops at their actual times instead of editing the trajectory into an unrealistically smooth story.
A high success count is not automatically a high-quality demonstration set. Repeating one easy path may omit alternate approaches, near-boundary states, and recoveries, while hurried collection can create jitter, occlusion, ambiguous task text, or unrecorded pauses. First establish a consistent nominal behavior and inspect it; then add deliberate, labelled condition variation without changing several factors at once or mixing unsafe improvisation into production motion.
Words you need
Name each idea precisely
- Teleoperation
Remote or mediated human control that converts interface input into bounded robot commands while returning enough feedback for operation.
Physical example:A leader arm supplies desired follower joint positions while cameras and measured follower joints show the result.
- Demonstration
A recorded task attempt intended to show behavior, including observations, actions, response, context, and outcome.
Physical example:One supervised pick-and-place trace contains front video, wrist video, measured joints, commanded joints, task text, and success label.
- Calibration identity
The stable association among a physical device, its measured calibration values, and the configuration used for a recording.
Physical example:A follower-arm ID selects the offsets measured for that specific arm rather than another unit's similarly named file.
- Intervention
A human action that changes, pauses, corrects, or terminates the commanded trajectory and must remain visible in lineage.
Physical example:An operator releases the enabling control and repositions a paper target after detecting a bad reset.
- Command-state residual
The difference between an issued command and the measured robot response after accounting for representation and timing.
Physical example:A requested joint angle of 0.50 rad and a measured 0.43 rad position produce a 0.07 rad residual at the compared time.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalEpisode timing, normalization, and split leakage
A policy learns the dataset you actually recorded, including hidden leakage and timing errors.
- z = (x−μ)/σ
- standardized valueUnit: unitless
- t
- source timestampUnit: seconds (s)
- N
- number of independent episodesUnit: episodes
For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.
It resembles a production event log, but camera, state, and action streams must describe the same instant.
What is z for x=8, μ=10, σ=2?
(8−10)/2 = −1.
A episode sampled at should contain
records. If only arrive, the missing fraction is ; this does not score demonstration quality.
Find missing records and response lag in a teleoperation trace
A 10.0-second simulated episode targets 20 Hz. The recorder saves 190 command-state rows. A visible input step occurs at 3.000 s, and the measured joint begins moving at 3.080 s.
Calculate the expected row count:
10.0 s × 20 samples/s = 200 rows.Calculate missing rows:
200 - 190 = 10 rows.Calculate the missing fraction:
10 / 200 = 0.05 = 5%.Calculate the observed response lag:
3.080 - 3.000 = 0.080 s = 80 ms.Compare lag with the 50 ms sample period:
80 / 50 = 1.6 periods, so same-index command and state rows cannot represent an instantaneous response.Reject an unqualified 'good demonstration' label until gap locations, clock alignment, command semantics, measured response, and the terminal task result are reviewed.
The trace is 5% short and shows an 80 ms, or 1.6-period, command-to-motion delay; those are actionable recording facts, not a verdict about behavior quality by themselves.
Teleoperation data must reveal what the operator asked, what the robot did, and when each happened.
Physical examples
Where this appears in real life
Cardboard leader and follower arms
Connect two unpowered cardboard two-link arms only by numbered instruction cards: one person moves the leader, another copies each joint-angle card on the follower, and a third records both sides.
Leader intent, transmitted command, follower response, human copying delay, and correction events are distinct signals even though the two arms eventually resemble each other.
Joystick driving a simulated cart
Move a screen-based cart through a simple corridor using a game controller while the simulator logs joystick axes, commanded velocity, cart pose, collision flags, and reset events.
Controller input can return to zero before the simulated cart stops, exposing why input alone is not a complete demonstration of physical response.
Hands-on exercise
Make the idea observable
Use an unpowered cardboard follower or a simulator with motion and collision limits. Do not use the exercise as authorization to drive an uncommissioned robot.
Assign stable IDs to the operator, teleoperator interface, follower, scene setup, camera configuration, and protocol revision without recording personal contact information.
Run a preflight that verifies the reset, calibration identity, command neutral state, observations, recorder destination, visible feedback, timeout, and abort path.
Collect five short attempts while logging operator input, issued command, measured state, timestamp, intervention event, outcome, and a plain-language task instruction.
Cancel and re-record one deliberately flawed reset while preserving the cancelled attempt's raw identity and the reason it is ineligible for nominal training.
For every attempt, calculate expected versus recorded rows, the largest timestamp gap, command-state residual at one marked event, and terminal-label evidence.
Review the camera and state timeline together, then write one protocol change that removes a repeatable ambiguity without erasing natural response variation.
A consistent protocol reduces accidental differences among attempts, while separate command and state traces expose lag, rejected commands, pauses, and interventions that a video-only review misses.
All five attempt records identify their setup and outcome, the cancelled reset remains traceable, and a second reviewer can reproduce every timing and residual calculation.
Build today
Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.
Evidence to save
DONE when “Teleoperation and demonstration collection” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Recording only leader or joystick input because it looks like the desired trajectory.
Record the issued robot action and measured response as separate timed features; contact, delay, saturation, and rejection can break the intended mapping.
Assuming successful teleoperation demonstrates autonomous policy safety.
Claim only supervised human-controlled task evidence; autonomy changes decision ownership and requires separate rollout, fault, stop, and recovery validation.
Re-recording a flawed attempt by overwriting its files and identity.
Retain the cancelled raw episode and disposition, give the replacement a new identity, and keep lineage between them.
Job connection
How this becomes employable evidence
Build and qualify an operator-to-robot collection workflow with device identity, live feedback, command limits, abort behavior, synchronized recorder signals, intervention annotations, and an audit trail that robot-learning teams can use without guessing what physically occurred.
Relevant target roles
- Robot HMI / Control & Monitoring Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
Chapter 14 interview drill
Interview questions: Teleoperation and demonstration collection
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Design a teleoperation recorder for demonstrations from multiple operators. Explain calibration identity, command versus state logging, latency, cancellation, intervention labels, operator bias, and the evidence needed before accepting an episode.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why record measured robot state when operator input is already available?
Input expresses intent, while measured state reveals the delayed physical response, limits, contact, faults, disconnections, or interventions that changed execution.
Q2What does using the correct calibration ID fail to prove?
It does not prove workspace safety, collision behavior, stopping performance, load suitability, sensor correctness, or that the powered system is commissioned.
Q3How should a cancelled and re-recorded attempt appear in lineage?
The cancelled attempt keeps its immutable identity and reason; the replacement receives a new identity linked by the review ledger.