Chapter 14 · Collect demonstrations as synchronized, reviewable robot datasets

Today in the field story

One problem, then the next

Place camera exposure, robot state acquisition, operator input, command acceptance, and measured response on one clock-aware timeline. The starter’s 24 ms mismatch is the planted warning: adjacent row numbers do not prove causal pairing. Calculate residuals, identify clock domains, declare tolerance, count missing samples, and preserve lag rather than shifting streams until the video looks convincing.

Why now: Imitation depends on the action belonging to the observation that actually preceded it.
Ignore today: Ignore perfect global clock theory; prove bounded alignment and expose uncertainty in this recorder.
Unlocks next: Synchronized training pairs with measurable timing quality.

Understand

Build the physical picture first

Multimodal synchronization is a forensic timeline: every camera frame, state sample, and command must say which clock observed which physical instant, while tolerances and residuals reveal what could not be aligned.

Acquisition time is when a sensor measurement represents the physical world; arrival time is when software receives it. Network, driver, encoding, queue, and scheduling delay separate the two. Prefer device or message-header acquisition timestamps when their clock relationship is known. Replacing missing acquisition times with current arrival time can make a table look orderly while hiding variable delay, so record that substitution as degraded provenance rather than normal synchronization.

Two timestamps are comparable only after establishing their clock domains. Devices may share one clock, be synchronized to a common source, or require an estimated offset and drift model. Clock offset is a fixed-looking difference at one instant; skew makes the difference change over time. A calibration event near the beginning cannot justify hour-long alignment unless later checks show that residual error remains inside the task's declared tolerance.

Exact synchronization requires matching timestamps. Approximate synchronization pairs samples whose header times fall within a chosen tolerance and needs a deterministic rule for competing candidates, queue limits, and unmatched records. The tolerance follows physical motion and task sensitivity, not convenience. At fast tool speed, a 30 ms mismatch can correspond to meaningful position error even if the synchronized video looks acceptable at normal playback speed.

Action, observation, and state also have a causal relationship. An action computed from observation time t is issued later, reaches a controller later still, and produces measurable motion after plant delay. Training conventions must say whether an action row is the command chosen from the current observation, the command active during an interval, or a future target. A timestamp column in LeRobot or any other format enables alignment work; its presence alone does not prove clock accuracy or causal correctness.

Words you need

Name each idea precisely

Acquisition timestamp: The claimed physical measurement time assigned by a sensor or trusted capture boundary in a known clock domain.
Physical example:
A camera frame header marks when exposure occurred rather than when an application finished decoding the frame.
Arrival timestamp: The local time when a message reaches a receiving process, including variable transport and scheduling delay.
Physical example:
Two images exposed 50 ms apart can arrive together after a network queue drains.
Clock domain: A time source and epoch whose readings can be directly compared only with readings from the same domain or a measured conversion.
Physical example:
A camera hardware counter and ROS system time need a documented mapping before their numeric stamps share meaning.
Clock offset and skew: Offset is the clock difference at a reference instant; skew is the rate difference that makes that offset drift over elapsed time.
Physical example:
A camera begins 12 ms ahead and gains another 2 ms over ten minutes, so one fixed correction becomes increasingly wrong.
Synchronization tolerance: The maximum allowed time separation for associating samples, justified by motion, sensor rate, and downstream error limits.
Physical example:
A dataset pairs a frame and state only when their converted acquisition times differ by no more than 15 ms.
Causal lag: The elapsed time among observation, decision, command issue, controller receipt, and measurable physical response.
Physical example:
A gripper command issued at 5.020 s begins changing measured width at 5.095 s, giving 75 ms observed response lag.

Visual model

See the relationship

Swipe the technical canvas horizontally on a small screen.

Day 95 · Failure case — The recorded action is paired with the wrong observation. Measured anchor: 120 ms skew.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalEpisode timing, normalization, and split leakage

A policy learns the dataset you actually recorded, including hidden leakage and timing errors.

z = (x−μ)/σ: standardized valueUnit: unitless
t: source timestampUnit: seconds (s)
N: number of independent episodesUnit: episodes

For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.

Programmer analogy

It resembles a production event log, but camera, state, and action streams must describe the same instant.

What is z for x=8, μ=10, σ=2?

(8−10)/2 = −1.

The state stream spans

t_{\mathrm{state}}=\frac{600}{20\,\mathrm{Hz}}=30\,\mathrm{s},

while the video spans $26\,\mathrm{s}$ . Pair samples by timestamp only when

\lvert t_{\mathrm{video}}-t_{\mathrm{state}}\rvert\le 15\,\mathrm{ms},

not merely when their array indices match.

Associate streams and reject one false index match

After converting to one clock domain, camera frame 81 has acquisition time 10.040 s. Candidate state samples occur at 10.028 s and 10.061 s. The allowed camera-state tolerance is 15 ms. The action attached by index was issued at 10.020 s, while frame 81 was actually produced from observation frame 80 at 9.990 s.

Calculate the absolute difference to the first state: |10.040 - 10.028| = 0.012 s = 12 ms.
Calculate the absolute difference to the second state: |10.040 - 10.061| = 0.021 s = 21 ms.
Pair frame 81 with the 10.028 s state because 12 ms is the nearest candidate and satisfies the 15 ms rule; leave the 21 ms candidate unmatched for this frame.
Calculate action-to-frame separation as 10.040 - 10.020 = 20 ms, but do not call the action a response to frame 81 based on this small number.
Read causal lineage: the action was computed from frame 80 at 9.990 s, so its observation-to-issue lag is 10.020 - 9.990 = 30 ms.
Reject same-index observation-action semantics for this row and store the actual source observation identity, issue time, pairing residual, and any unpaired samples.

Result

Frame 81 pairs with the earlier state at a 12 ms residual, while its same-index action belongs causally to frame 80 and was issued 30 ms after that source observation.

What this proves

Timestamp proximity can associate measurements, but only explicit lineage establishes which observation produced an action.

Physical examples

Where this appears in real life

Rolling ball with a visual time mark

Roll a ball across a ruled sheet while filming it and recording manual position events; clap once in view and press an event key at the same perceived moment.

Look for:

The shared event estimates offset, while the ball's position exposes residual mismatch; repeating the event later reveals whether the clock relation drifted.

Two delayed parcel trackers

Write physical scan time and app-notification time on cards for two parcels whose updates travel through different queues, then sort by each column.

Look for:

Arrival order can differ from event order, illustrating why application receipt time cannot safely replace measurement time.

Hands-on exercise

Make the idea observable

Use synthetic CSV streams or a phone video plus manual event log. Work offline; the aim is to measure alignment, not control hardware.

Create camera, state, and command tables with separate acquisition and arrival timestamps, source clock names, sequence IDs, and one shared start event.
Introduce a 20 ms camera offset, gradual 1 ms-per-minute drift, variable arrival delay, one dropped state row, and one duplicated command ID.
Estimate clock conversion from the start and end events, then transform acquisition times into one comparison domain without rewriting the raw values.
Pair camera and state records with a declared 15 ms nearest-neighbour rule, retaining residual, unmatched reason, and candidate count for every decision.
Link each command to its actual source observation and calculate observation-to-issue plus issue-to-measured-response lag for two marked events.
Plot or tabulate residual over time, identify when the fixed-offset method breaches tolerance, and save the planted defects as regression fixtures.

Observe

Index alignment initially appears plausible, but drift, drops, duplicate IDs, and variable arrival delay create mismatched physical moments that a residual timeline makes visible.

Done when

The checker detects every planted defect, never overwrites raw clock evidence, reports paired and unmatched counts, and reproduces both causal-lag calculations.

Build today

Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.

Evidence to save

DONE when the integrated “Synchronized video, state, command, and timestamps” path is observable, cancelable, and leaves the prior baseline reproducible.

Common mistakes

Catch the wrong mental model

Wrong

Pairing streams by array index because they contain the same number of rows.

Better

Associate converted acquisition timestamps under a justified rule and retain residuals; equal counts do not survive different starts, rates, drops, buffering, or drift.

Wrong

Replacing unknown sensor time with message arrival time and calling the result synchronized.

Better

Keep arrival as a separate field, label acquisition time unavailable, and quantify variable transport delay or exclude the stream from claims that require precise alignment.

Wrong

Assuming the action stored next to an image was computed from that image.

Better

Record source observation identity and decision, issue, receipt, and response times so action causality is explicit rather than inferred from row position.

Job connection

How this becomes employable evidence

Qualify a multimodal robot-data recorder by tracing device clocks, acquisition stamps, queues, action lineage, pairing residuals, missing samples, and response latency before the dataset reaches policy training or evaluation.

Relevant target roles

Robotics Deployment, Integration & Validation Engineer
Robotics Application / ROS 2 Integration Engineer
Robotics Software Engineer — ROS 2 / AMR
Robot Learning Deployment / Physical AI Integration Engineer

Chapter 14 interview drill

Interview questions: Synchronized video, state, command, and timestamps

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Camera, joint state, and action arrays have equal lengths but different clocks and occasional drops. Explain how you would establish comparable time, choose a tolerance, preserve unmatched data, and prove which observation caused each action.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why is arrival time usually weaker than acquisition time for alignment?

Model interview answer

Arrival includes variable transport, decoding, queue, and scheduler delay, so it can change even when physical sampling time does not.

Q2What does an approximate-time pair need besides a tolerance?

Model interview answer

Comparable clock domains, a candidate-selection rule, queue and unmatched behavior, retained residuals, and a physical justification for the tolerance.

Q3Does a `timestamp` feature prove the dataset is synchronized?

Model interview answer

No. Its clock source, acquisition boundary, conversion, drift, pairing residuals, drops, and causal semantics still require evidence.