Chapter 14 · Collect demonstrations as synchronized, reviewable robot datasets
Today in the field story
One problem, then the next
Review every handover by structure, behavior, and condition coverage. Retain the dropped block, receiver-not-ready pause, saturation, operator habit, dark tray, and off-centre start with dispositions rather than deleting inconvenient attempts. Compare collected conditions with intended use and label distribution shift as an observed gap. The quality ledger tells Week 15 what the dataset teaches and, more importantly, what it does not.
- Why now
A valid schema cannot detect bad behavior, biased coverage, or misleading labels by itself.
- Ignore today
Ignore synthetic augmentation as a cure; inspect and classify real recorded evidence first.
- Unlocks next
A defensible inclusion set and prioritized collection gaps.
Understand
Build the physical picture first
Dataset review is incoming inspection for robot experience: first test whether records are structurally usable, then whether the demonstrated behavior and labels are credible, and finally whether collected conditions cover the world the policy will face.
Structural quality asks whether required fields exist, shapes and types match, timestamps increase, video decodes, episode boundaries agree, samples align within tolerance, and referenced task or calibration identities resolve. Semantic quality asks whether units, frames, component order, task text, reset, and outcome mean what the schema claims. Behavioral quality asks whether the operator performed the intended behavior without unlabelled intervention, unsafe improvisation, severe jitter, obstruction, or a terminal label contradicted by the recording.
A failed demonstration is poor or unsuccessful behavior inside the collected distribution. It may contain a dropped object, wrong placement, collision, timeout, or operator correction. Preserve it in raw data and label the failure precisely; do not automatically train imitation on actions that should not be copied. A curated training recipe can quarantine it, use only a deliberate recovery segment under a separate label, or reserve it for evaluation. The choice depends on the learning objective and remains explicit.
Distribution shift occurs when later observations, actions, tasks, hardware, operators, or environments differ meaningfully from collection. A bright-light sorting dataset can be structurally flawless and still fail in evening light. A coverage matrix counts episodes across task-relevant factors such as object position, object identity, lighting, camera pose, operator, recovery state, and outcome. Counts need denominators and intersections because balanced marginal totals can hide an empty combination.
Review both aggregates and individual episodes. Summaries expose missingness, duration outliers, saturation, abort rate, operator imbalance, and sparse condition cells; synchronized playback exposes label errors, occlusion, late actions, pauses, and bad technique. Sample every condition and every terminal class, then inspect all hard-gate failures and suspicious outliers. Keep reviewer identity, rubric version, disposition, and reason so quality decisions can be audited or revised.
Words you need
Name each idea precisely
- Quality gate
A defined pass, quarantine, or reject rule tied to observable dataset evidence and a recorded reason.
Physical example:An episode is quarantined when more than 1% of camera-state pairs exceed the 15 ms synchronization tolerance.
- Failed demonstration
A collected attempt whose behavior or terminal result does not satisfy the intended demonstration objective.
Physical example:The gripper drops the block before the target bin even though every frame and action row is present.
- Distribution shift
A meaningful mismatch between data used for learning or validation and inputs encountered during later operation.
Physical example:Training uses matte red cubes under fixed white light, while deployment presents glossy objects under sunlight.
- Coverage matrix
Counts or rates for combinations of task-relevant conditions, used to reveal sparse and absent regions.
Physical example:Rows are near and far object positions, columns are bright and dim lighting, and each cell contains episode and success counts.
- Operator bias
A repeated human technique or correlated condition that a model can copy as a shortcut instead of learning the intended task.
Physical example:One operator always approaches from the left and also records every blue object, coupling person-specific motion with colour.
- Quarantine
A reversible disposition that keeps raw data and lineage while excluding it from a named training or evaluation recipe pending review.
Physical example:Episodes with unverified action units stay addressable in raw storage but cannot enter dataset version 1's training manifest.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalEpisode timing, normalization, and split leakage
A policy learns the dataset you actually recorded, including hidden leakage and timing errors.
- z = (x−μ)/σ
- standardized valueUnit: unitless
- t
- source timestampUnit: seconds (s)
- N
- number of independent episodesUnit: episodes
For x=14, dataset mean μ=10, and standard deviation σ=2, subtract: 14−10=4.
Divide: z=4/2=2, meaning two standard deviations above the mean.
Compute μ and σ from training data only, then keep whole scenes out of validation to prevent leakage.
It resembles a production event log, but camera, state, and action streams must describe the same instant.
What is z for x=8, μ=10, σ=2?
(8−10)/2 = −1.
At , an episode with samples lasts
If the expected task duration is , flag the episode for review; duration alone cannot detect bad actions or distribution shift.
Turn forty episodes into an honest coverage decision
A two-factor audit has bright/dim lighting and near/far object positions. Counts are bright-near 18, bright-far 18, dim-near 4, and dim-far 0, for 40 episodes total. The planned evaluation claims both lighting conditions and positions.
Verify the denominator:
18 + 18 + 4 + 0 = 40 episodes, with each episode assigned to exactly one declared cell.Calculate bright coverage:
(18 + 18) / 40 = 36 / 40 = 90%.Calculate dim coverage:
(4 + 0) / 40 = 4 / 40 = 10%.Read intersections rather than margins: dim-near has four attempts, while dim-far has zero, so the planned condition space contains an untested cell.
Refuse to repair the gap by duplicating images or applying brightness augmentation; transformed training pixels are not collected dim-far physical trajectories.
Either collect and review dim-far attempts under the same protocol or narrow the dataset card and evaluation claim to the conditions actually represented.
The dataset is 90% bright, 10% dim, and has no dim-far demonstrations; it cannot support an evidence-based claim across the complete two-by-two condition space.
Coverage is about relevant combinations and honest scope, not merely total episode count or balanced-looking single columns.
Physical examples
Where this appears in real life
Bright-only block collection
Arrange forty paper episode cards by bright or dim light and near or far object position; place most cards in bright rows even though total object counts appear balanced.
Overall volume hides an almost empty dim-light region, so a policy claim about ordinary room lighting would exceed collected evidence.
Perfect approach with no recovery
Draw ten centered gripper paths to a cup and one path that starts slightly off-centre, then ask which examples teach correction after an early error.
More nominal copies improve density near the ideal path but do not cover recovery states the policy creates during rollout.
Hands-on exercise
Make the idea observable
Reuse this week's synthetic or unpowered episodes and their aligned inspector data. Define the review rubric before looking at which attempts you prefer.
Create structural gates for schema, decodable media, episode boundaries, timestamp order, pairing residual, missingness, and referential integrity.
Create semantic and behavioral gates for units, frame, task clarity, reset evidence, action plausibility, intervention, terminal outcome, and prohibited behavior.
Label each episode pass, quarantine, or reject under a versioned rubric while retaining reviewer, timestamp, reason code, free-text evidence, and raw identity.
Build a coverage matrix across at least three relevant factors, including operator or reset group, and show episode count plus outcome count in each intersection.
Inspect every failed or aborted attempt, every zero or sparse cell, every duration outlier, and at least one passing episode from each populated condition.
Write a collection change for the highest-impact gap, a regression check for one discovered defect, and a narrowed claim if new collection is not yet available.
Some episodes pass file checks but fail semantic or behavior review, and aggregate balance can coexist with empty condition intersections or one operator dominating a task.
Every disposition traces to rubric evidence, the matrix totals reconcile with the episode ledger, zero cells are explicit, and the next collection or scope decision follows from measured gaps.
Build today
Create a small demonstration dataset with synchronized observations, actions, language, and quality labels.
Evidence to save
DONE when a 60–120 second uncut “Quality review, failed demonstrations, and distribution shift” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Deleting every failed attempt and reporting the remaining dataset as 100% successful.
Retain raw failures with precise outcomes and dispositions, report denominators before and after curation, and exclude or repurpose them only through a documented recipe.
Checking each condition column separately and calling the dataset balanced.
Count intersections of relevant factors because healthy marginal totals can conceal an absent combination such as dim light with a far target.
Using image augmentation as proof that a physical condition was collected.
Describe augmentation as a training transform; it does not create real sensor noise, contact, operator response, action timing, or outcomes for the missing condition.
Accepting an episode's success flag without watching terminal evidence.
Verify the outcome against synchronized observations, measured state, task criteria, and terminal event, then retain reviewer and rubric lineage.
Job connection
How this becomes employable evidence
Create a robot-dataset acceptance process that combines automated structural checks, synchronized episode review, operator and condition coverage, failure taxonomy, reversible quarantine, and a collection backlog tied to deployment risks.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robot HMI / Control & Monitoring Engineer
Chapter 14 interview drill
Interview questions: Quality review, failed demonstrations, and distribution shift
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A demonstration dataset has 98% structurally valid episodes and 95% labelled success. Explain why that is insufficient, how you would audit failed behavior and operator shortcuts, and how a coverage matrix changes the collection plan.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1How does a failed demonstration differ from distribution shift?
A failed demonstration is poor or unsuccessful behavior in collected data; distribution shift is a mismatch between collected and later operating distributions.
Q2Why can balanced lighting and position totals still hide a coverage gap?
Their intersections may be uneven or empty—for example, all dim examples may be near while no dim-far attempt exists.
Q3What is the value of quarantine instead of immediate deletion?
It preserves raw identity and evidence while preventing uncertain data from entering a named recipe, so later correction or reclassification remains auditable.