Chapter 14 · Collect demonstrations as synchronized, reviewable robot datasets
Today in the field story
One problem, then the next
Seal handover-v1 without erasing its rejected episodes. Generate the card from frozen manifests, recalculate counts and duration, record provenance, licensing, privacy, intended use, exclusions, and checksums, then open the same identities in a synchronized inspector. Show a passing and failing attempt side by side. Week 15 begins from this release identity, so corrections must become a new version with lineage rather than rewriting the evidence it used.
- Why now
The dataset becomes reusable only when its claims and underlying episodes can be independently checked.
- Ignore today
Ignore public promotion and policy results; publish only non-sensitive, verified dataset evidence.
- Unlocks next
A traceable learning input and explicit Week 14 to Week 15 handoff.
Understand
Build the physical picture first
A dataset card is the label on a sealed evidence box, while an episode inspector is the window into it: the label states provenance and limits, and the window lets a reviewer test those statements against synchronized records.
A dataset card documents what the dataset contains and how it may responsibly be used. Include owner and contact channel at an organizational level, collection purpose, task definitions, hardware and sensor configuration, teleoperation method, schema and version, episode counts, durations, splits, condition coverage, quality process, exclusions, provenance, license, privacy treatment, intended uses, prohibited or unsupported uses, limitations, and known risks. A polished README does not prove any claim unless its numbers and examples trace to data.
An episode inspector should place synchronized camera views, observation state, action, task text, frame and episode identity, acquisition time, pairing residual, outcome, intervention, and review disposition on one seekable timeline. Show raw physical values or a clearly identified decoded view rather than only normalized training tensors. Make missing data, unmatched samples, action saturation, terminal events, and excluded episodes visible instead of silently skipping them during playback.
Publication requires rights and privacy review. Confirm licenses for dataset content, recorder code, annotations, and any third-party assets; document consent and permitted use when people, voices, homes, screens, or identifying objects could appear. Remove secrets and personal data through a reviewable process, not a promise. If rights or privacy are unresolved, keep the dataset private and publish only non-sensitive synthetic examples and aggregate documentation.
Freeze a release rather than mutating a public identity in place. Record the exact schema, episode manifest, split manifest, quality ledger, card revision, inspector revision, code version, and content checksums. If a bad label or episode is discovered, publish a corrected version with a change log and lineage to the original. Recalculate every headline number from the frozen manifest so counts, duration, coverage, and exclusions cannot drift apart.
Words you need
Name each idea precisely
- Dataset card
Versioned documentation describing dataset contents, provenance, permitted and intended use, quality evidence, limitations, risks, and metadata.
Physical example:A card states that 70 training episodes cover two object positions but no reflective objects and are licensed only under the named terms.
- Episode inspector
A seekable interface that renders aligned modalities, actions, metadata, events, outcomes, and review evidence for one attempt.
Physical example:Scrubbing to 12.4 s shows the front frame, six measured joints, seven commanded values, a 9 ms pairing residual, and a correction marker.
- Provenance
Evidence of where data came from, who or what collected it, under which configuration and protocol, and how it was transformed.
Physical example:Episode 22 links to recorder revision, device IDs, calibration identity, protocol version, raw checksum, and reviewer disposition.
- License
The legal permission and conditions governing copying, modification, distribution, and use of the dataset or its components.
Physical example:A dataset repository names its license while separately checking whether an included third-party video permits redistribution.
- Checksum
A content-derived digest used to detect whether a released file or manifest differs from the frozen version.
Physical example:The release manifest records a SHA-256 digest for each metadata and data shard so later changes are detectable.
- Dataset lineage
The trace from immutable raw recordings through review, exclusion, conversion, split, and released versions.
Physical example:Version 1.1 corrects two outcome labels and links each new episode record to its version-1 source identity and change reason.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalEpisode timing, normalization, and split leakage
A policy learns the dataset you actually recorded, including hidden leakage and timing errors.
- z = (x−μ)/σ
- standardized valueUnit: unitless
- t
- source timestampUnit: seconds (s)
- N
- number of independent episodesUnit: episodes
For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.
It resembles a production event log, but camera, state, and action streams must describe the same instant.
What is z for x=8, μ=10, σ=2?
(8−10)/2 = −1.
For three episodes with samples each at , total recorded time is
This total does not validate labels, action alignment, exclusions, or licensing.
Recalculate the duration claim before freezing a card
A release candidate contains three eligible episodes. Each has 600 samples at 20 Hz. Two are assigned to training and one to validation; there is no independent test episode. The card draft claims 90 seconds of eligible demonstrations.
Calculate each episode's nominal sampled duration:
600 / 20 Hz = 30 s.Sum the manifest rather than multiplying an undocumented total:
30 + 30 + 30 = 90 s.Reconcile split counts: training contains two episodes and
2 × 30 = 60 s; validation contains one episode and1 × 30 = 30 s.State that test contains zero episodes; do not rename validation as test after using it to make development decisions.
Verify the inspector can open all three identities and show synchronized video, state, action, task, timestamps, outcomes, and any review markers.
Write the card claim as 90 seconds of eligible nominal sampled time, with a prominent limitation that three episodes and no test split cannot support a generalization claim.
The 90-second total is arithmetically reproducible as 60 seconds train plus 30 seconds validation, and the missing test set remains explicit.
A correct headline number still needs split context, sample evidence, and limitations before it becomes an honest release statement.
Physical examples
Where this appears in real life
Ingredient label and inspection window
Place mixed objects in an opaque box, write a label claiming exact contents and exclusions, then cut a small viewing window and compare visible samples with the inventory sheet.
The label gives scope and warnings, while the window provides spot-check evidence; neither alone establishes that every hidden item matches the claim.
Flight-recorder playback
Lay parallel paper strips for camera, command, measured state, operator event, and outcome, then slide one vertical ruler across them as a shared playhead.
A shared cursor exposes delayed response, missing rows, and terminal evidence that separate media players or summary totals would conceal.
Hands-on exercise
Make the idea observable
Use three non-sensitive synthetic, simulated, or unpowered episodes from this week. Do not upload recordings containing unresolved people, voices, home details, screens, credentials, or third-party material.
Freeze the schema, episode manifest, split manifest, task table, quality ledger, source and license inventory, code revision, and checksums under one release-candidate identity.
Generate a dataset card with task and hardware context, teleoperation protocol, counts, duration, schema, splits, condition coverage, quality rules, exclusions, provenance, privacy, license, intended use, limitations, and known risks.
Build a static inspector that selects an episode and scrubs camera, state, action, timestamps, pairing residual, task, outcome, intervention, and review disposition with one shared playhead.
Plant one missing frame, one wrong outcome label, and one card count mismatch; verify that the inspector or release checks expose all three before publication.
Run a privacy and rights checklist, keep unresolved material private, and replace any public-facing example with an equivalent non-sensitive synthetic record.
Have a reviewer recalculate counts and duration from the manifest, inspect one passing and one non-passing attempt, verify checksums, and sign the release decision with residual limitations.
The card summarizes scope, while the inspector reveals whether modality alignment, labels, exclusions, and bad cases support that summary; release checks prevent their revisions from drifting independently.
A fresh reviewer reproduces all headline numbers, detects every planted defect, opens each eligible episode, confirms privacy and license disposition, and reaches the same bounded publish-or-keep-private decision.
Build today
Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.
Evidence to save
DONE when the weekly ship note explains how “Publish a dataset card and visual episode inspector” changed the build, what still fails, and the first task for next week.
Common mistakes
Catch the wrong mental model
Treating a detailed dataset card as proof that every episode satisfies it.
Generate metrics from frozen manifests and let reviewers inspect aligned examples, exclusions, failures, and raw evidence behind each important claim.
Publishing first and planning to resolve consent or licensing later.
Keep unresolved content private, complete rights and privacy review before distribution, and use non-sensitive synthetic examples when public evidence is still needed.
Building an inspector that plays only video.
Align camera, measured state, action, task, timestamps, residuals, interventions, outcome, and review disposition under one episode identity and playhead.
Fixing a released label by overwriting version 1 in place.
Issue a new version with changed manifests and checksums, retain the original lineage, and document the exact correction and affected identities.
Job connection
How this becomes employable evidence
Deliver a robot-dataset release that learning engineers, operators, QA reviewers, and compliance owners can interrogate through a versioned card, synchronized inspector, immutable manifests, rights review, and traceable corrections.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robot HMI / Control & Monitoring Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 14 interview drill
Interview questions: Publish a dataset card and visual episode inspector
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Show how you would release a small demonstration dataset. Derive one card metric, describe the inspector timeline, explain privacy and licensing gates, and demonstrate how a corrected episode becomes a new version without erasing lineage.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What does a dataset card provide that an episode inspector does not?
It states dataset-wide provenance, scope, schema, splits, rights, intended use, limitations, risks, quality process, and aggregate evidence.
Q2What should a synchronized inspector make easy to catch?
Misalignment, missing fields, wrong labels, command-state mismatch, interventions, saturation, unusual episodes, terminal contradictions, and unexplained exclusions.
Q3Why create a new version for a corrected released label?
A new version preserves reproducibility and lineage: prior results still reference unchanged content while the correction, affected identity, and new checksum remain explicit.
Chapter starter artifact
Gate synchronized observation-action pairs
Dataset release `handover-v1` contains versioned episode, feature, action, clock, split, quality, rights, and privacy contracts; a visual inspector replays accepted and rejected attempts; all headline counts and durations recalculate from frozen manifests and checksums.
const pairs = [
{ id: "p1", cameraId: "f1", actionCameraId: "f1", cameraMs: 1000, actionMs: 1005 },
{ id: "p2", cameraId: "f2", actionCameraId: "f2", cameraMs: 1050, actionMs: 1058 },
{ id: "p3", cameraId: "f3", actionCameraId: "f3", cameraMs: 1100, actionMs: 1124 },
{ id: "p4", cameraId: "f4", actionCameraId: "f4", cameraMs: 1150, actionMs: 1146 },
{ id: "p5", cameraId: "f5", actionCameraId: "f4", cameraMs: 1200, actionMs: 1202 },
{ id: "p6", cameraId: "f6", actionCameraId: "f6", cameraMs: Number.NaN, actionMs: 1250 },
];
const toleranceMs = 10;
const audited = pairs.map((pair) => {
const identityValid = typeof pair.id === "string" && pair.id.length > 0 &&
typeof pair.cameraId === "string" && pair.cameraId === pair.actionCameraId;
const timesValid = [pair.cameraMs, pair.actionMs].every(Number.isFinite);
if (!identityValid || !timesValid) return { ...pair, status: "invalid" };
const skewMs = Math.abs(pair.cameraMs - pair.actionMs);
return { ...pair, skewMs, status: skewMs <= toleranceMs ? "accepted" : "rejected" };
});
const accepted = audited.filter((pair) => pair.status === "accepted");
const rejected = audited.filter((pair) => pair.status === "rejected");
const invalid = audited.filter((pair) => pair.status === "invalid");
if (accepted.length === 0) throw new Error("no valid synchronized pair");
const maxAcceptedSkewMs = Math.max(...accepted.map((pair) => pair.skewMs));
const output =
"accepted=" + accepted.length +
" rejected=" + rejected.length +
" invalid=" + invalid.length +
" maxAcceptedSkewMs=" + maxAcceptedSkewMs;
const expected = "accepted=3 rejected=1 invalid=2 maxAcceptedSkewMs=8";
if (output !== expected) throw new Error("sync audit mismatch: " + output);
console.log(output);Download the file into your terminal's current folder, then run the command below. The expected output is exact.
- Run
node week-14-audit-handover-sync.mjs
- Expected output
accepted=3 rejected=1 invalid=2 maxAcceptedSkewMs=8
- Planted failure to diagnose
Pair p3 exceeds the 10 ms tolerance, p5 links the wrong camera identity, and p6 has a non-finite timestamp; index ordering or JavaScript NaN comparisons must not classify either invalid pair.