Chapter 23 · Collect and freeze a task-shaped capstone dataset
Today in the field story
One problem, then the next
Reviewers watch the Foundry's failures as evidence: bad reset, stale command, camera occlusion, timing gap, operator correction, ambiguous label, collision guard, or genuine task difficulty. They separate collection defects from useful recovery behavior and task failures, then target the next batch at deployment-relevant holes rather than the easiest missing count. Raw records are never rewritten to make an episode look clean. The release manifest selects reviewed rows while preserving lineage back to every excluded or corrected attempt.
- Why now
Failure taxonomy turns additional collection into a causal coverage decision.
- Ignore today
Do not relabel failures as success or add data without a measured gap.
- Unlocks next
A targeted repair batch with complete raw-to-release lineage.
Understand
Build the physical picture first
Failure review is a workshop inspection bench: keep the broken part, name the fracture, trace which production condition made it likely, and repair the process that produced the gap.
Review operates in layers. Structural checks ask whether required records exist, media decodes, timestamps and sequence are coherent, schema fields match, and the episode finalizes. Semantic checks ask whether units, frames, identities, task, factor levels, reset, and labels mean what they claim. Behavioral checks inspect purposeful motion, unnecessary contact, jitter, occlusion, intervention, terminal evidence, and protocol compliance. Task outcome remains separate: an episode can be structurally excellent yet demonstrate a failed grasp.
Give every non-pass one primary reason and optional contributing reasons from a versioned taxonomy. Useful families include setup or reset defect, operator technique, perception or visibility, command path, limit rejection, timing or synchronization, recorder infrastructure, robot or simulator fault, task difficulty, safety stop, and rights or privacy hold. Preserve reviewer, rubric revision, decisive timestamp, evidence link, and disposition include, quarantine, or exclude; free-form comments alone cannot support reliable counts.
Join failure reasons to the condition matrix. Overall success may look healthy while one object-light-start intersection produces occlusion, one operator has excessive action saturation, or one session has pairing residuals beyond tolerance. Calculate counts and rates with denominators for each important cell, inspect every severe event and invalid record, and sample passing episodes too. A failure taxonomy without coverage tells what broke; coverage without synchronized review may count unusable data as representation.
Repair the cause, then collect the smallest targeted batch that tests the repair. A moved lamp may fix visibility; revised reset marks may remove pose drift; a gateway fix may eliminate stale commands; operator retraining may reduce jitter. Assign new protocol, configuration, or recorder versions and link recollection to the gap. Do not duplicate existing episodes, relabel an unsupported outcome, or use image augmentation as evidence that physical timing and contact were collected.
Words you need
Name each idea precisely
- Failure taxonomy
A versioned set of mutually understandable reason codes used to classify invalid, failed, aborted, or safety-related episodes consistently.
Physical example:A dark frame that hides the gripper receives primary reason
visibility_occlusion, while a dropped packet is recorded as a contributing timing issue.- Disposition
The review decision controlling whether an episode is included, quarantined for investigation, or excluded from a named release recipe.
Physical example:A successful episode with uncertain calibration is quarantined until its configuration identity is verified.
- Sparse cell
A deployment-relevant condition intersection with too few valid reviewed episodes to support the intended claim.
Physical example:The dim-light, right-start, sponge-S2 cell has one valid demonstration while other cells have four.
- Targeted recollection
A new collection block aimed at a measured defect or coverage gap under an explicitly revised protocol or configuration.
Physical example:Four new dim-right episodes are recorded after moving the camera and verifying that the target remains visible.
- Review rubric
The versioned observable rules by which reviewers judge structure, meaning, behavior, outcome, safety events, and release suitability.
Physical example:Rubric Q2 requires a terminal frame, pairing residual below 20 ms, no unlabelled intervention, and a visible two-second placement hold.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDataset coverage and independent scenes
More frames from the same easy scene do not equal broader robot experience.
- N_scene
- independent physical setupsUnit: scenes
- coverage
- tested condition cells representedUnit: percent
- failure rate
- failed frozen trials divided by all trialsUnit: percent
Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.
It is test-case coverage, not line coverage: the unit is a distinct physical condition.
Seven of ten condition cells are covered. What is coverage?
7/10 = 70%.
For occlusion failures among reviewed episodes,
Compare the same frozen definition after changes. The percentage identifies a large category but does not establish its cause.
Find the real gap inside a 32-attempt batch
The batch opened 32 episodes across eight cells. Review finds 2 incomplete, 1 invalid video, 3 task failures, 1 operator abort, and 25 reviewed successes. The dim-right-S2 cell has four opened attempts: one incomplete, one invalid video, two task failures, and zero release-ready successes.
Reconcile terminal outcomes first:
32 opened - 2 incomplete = 30 terminally closed; retain all 32 identities and do not merge invalidity with task failure.Apply structural checks to the 30 closed episodes and quarantine the one invalid-video episode; then apply semantic, behavioral, and task rubrics to the remaining valid records.
Calculate batch task-success rate over all opened attempts as
25 / 32 = 78.125%, while labeling this denominator separately from success among structurally valid completed attempts.Read the dim-right-S2 intersection:
0 / 4release-ready, with two different infrastructure outcomes and two task failures, so the gap is both coverage and reliability rather than a simple quota shortage.Inspect synchronized timelines and discover camera occlusion in the two task failures plus a recorder stall in the invalid episode; assign primary and contributing reason codes with decisive timestamps.
Move the simulated camera within the approved geometry, fix and fault-test the recorder, advance configuration and protocol versions, then schedule four linked dim-right-S2 recollections before filling any already healthy cell.
Accept the repair only if the new block passes structure and synchronization, shows the formerly hidden grasp, and adds reviewed examples without changing the original 32-attempt report.
The apparent need for “more dim data” becomes two concrete repairs—visibility and recorder timing—followed by four versioned recollections in the only empty release-ready cell.
Coverage repair starts from joined condition and failure evidence; otherwise new collection can reproduce the same defect at a larger scale.
Physical examples
Where this appears in real life
Dim-right grasps disappear behind the wrist
Four attempts from the right start under dim light reach the object, but the wrist blocks the overhead camera exactly during closure and reviewers cannot verify contact.
The issue is not repaired by marking successes manually; visibility and terminal evidence are changed, the protocol version advances, and that exact condition is recollected.
Rover failures cluster after a recorder update
Docking motion looks normal, yet most episodes from session S4 have action-to-state residuals above tolerance after new recorder software was installed.
Join failure reasons to recorder revision and session, quarantine affected episodes, reproduce the timing defect, and validate the repair before adding more driving demonstrations.
Hands-on exercise
Make the idea observable
Use the first-batch raw ledger, synchronized inspector, condition table, and immutable episode files. Define the review rubric and taxonomy before counting favorite examples.
Run structural checks for required modalities, finalization, decode, schema, timestamp order, sequence gaps, episode boundaries, and synchronization tolerance.
Review semantic identity, units, frames, task, factor levels, reset, terminal label, intervention, and rights fields against the actual synchronized record.
Inspect behavior and task outcome for every failed, aborted, invalid, saturated, or timing-outlier episode plus at least one passing example from each populated cell.
Assign disposition, primary reason, contributing reasons, reviewer, rubric version, decisive timestamp, and evidence reference without changing raw files.
Build a matrix of opened, valid, behavior-pass, task-success, and release-ready counts for important condition intersections, operators, sessions, and recorder revisions.
Choose the highest-risk sparse or failing cell, repair its root protocol or system issue, advance the relevant version, and schedule a bounded targeted recollection.
Rerun review on the added episodes and report pre-repair and post-repair counts separately so improvement cannot overwrite the original evidence.
The dominant visible task failure may not be the dominant dataset defect, and a small intersection often reveals correlations that disappear in overall success and validity rates.
All reviewed decisions trace to evidence and rubric versions, counts reconcile at batch and cell levels, the repair addresses a demonstrated cause, and new episodes improve the targeted cell without rewriting history.
Build today
Collect, review, and version real demonstrations for one useful tabletop or mobile task.
Evidence to save
DONE when a comparison table for “Review failures and repair dataset coverage” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Using one label named bad for corrupt media, poor technique, task failure, and safety stop.
Keep structural validity, behavioral quality, task outcome, intervention, and safety events as separate fields, then assign versioned primary and contributing reason codes.
Adding random episodes after finding a sparse condition cell.
Trace why the cell is sparse or failing, repair visibility, reset, control, recorder, or protocol causes first, then recollect that named intersection under a new version.
Reclassifying uncertain episodes as successes to make cell counts look complete.
Quarantine records whose evidence cannot satisfy the frozen rubric and report the gap; a full matrix built from untrustworthy labels is worse than an honest empty cell.
Job connection
How this becomes employable evidence
Build a demonstration-data QA loop that combines automated integrity checks, synchronized behavioral inspection, reason-coded dispositions, condition and revision slicing, root-cause repair, and bounded recollection linked to deployment risk.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 23 interview drill
Interview questions: Review failures and repair dataset coverage
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A robot dataset reports 80% success but one scene cell has no usable episodes. Explain your review layers, failure taxonomy, denominators, root-cause evidence, quarantine policy, and targeted recollection decision.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Can a task-failure episode still be structurally valid?
Yes. Its files, schema, timing, and labels can be complete and trustworthy even though the demonstrated physical task did not meet its terminal predicates.
Q2Why join failure reason with condition, operator, session, and recorder revision?
The joins expose clustered causes and confounding that overall rates hide, directing repair toward the actual setup, process, software, or difficult scene.
Q3What must happen before targeted recollection begins?
The measured cause should be repaired or the claim narrowed, the affected protocol or configuration versioned, and the exact gap and stop rule scheduled.