Chapter 23 · Collect and freeze a task-shaped capstone dataset
Today in the field story
One problem, then the next
The first Foundry batch fills a small predeclared set of cells rather than chasing smooth demonstrations. Each raw attempt receives an immutable identity, exact configuration, reset proof, operator, timing health, terminal outcome, intervention markers, and review status. Failures remain available even when unsuitable for the training release. The batch report shows counts by condition and outcome, allowing the team to see whether one operator, object, or lighting setup dominates before more collection magnifies the imbalance.
- Why now
A small balanced batch exposes schema and process defects cheaply.
- Ignore today
Do not collect hundreds of episodes or discard awkward attempts.
- Unlocks next
A reviewable baseline batch and the first measured coverage table.
Understand
Build the physical picture first
A balanced first batch is a tray filled from a written packing list: every allowed condition gets intentional attempts, while every spill, substitution, and rejected item stays in the count.
Balance begins with the Day 156 condition ledger and Day 155 episode contract. It does not mean every observable property must have identical counts. Allocate attempts across deployment-relevant cells, oversample a difficult but important cell only through a declared rule, and state the target and actual denominator. Keep scheduled, opened, completed, valid, successful, quarantined, and release-selected counts separate so recollection does not erase the cost or pattern of invalid attempts.
Standardize demonstration intent without forcing robotic sameness. Train each operator on the task, allowed path, speed and force boundaries, camera occlusion rules, reset, abort, and annotation protocol. Ask for clear purposeful behavior and complete terminal states, while retaining natural bounded variation. If one operator always uses a shortcut, pauses at the same point, or sees different conditions, operator identity and session order allow reviewers to detect the correlation.
Record failures rather than training the policy to imitate them by accident. An opened episode may end in clean success, task failure, operator abort, safety stop, or invalid infrastructure. Raw storage keeps all of them. The v1 release recipe can include reviewed successful demonstrations, deliberately labeled recovery segments, or other categories only if their purpose and boundaries are explicit. Pressing “redo” creates a new linked attempt; it must not overwrite the failed trajectory.
Watch collection health while the batch is still small. Reconcile cell counts, valid rate, task outcome, intervention, episode duration, action saturation, missing modalities, synchronization residual, dropped sequences, and operator/session share after each block. Stop when a recorder defect or biased schedule appears. Filling fifty more episodes before checking the first ten multiplies a protocol problem and can make an apparently large dataset less useful than a small clean pilot.
Words you need
Name each idea precisely
- Collection denominator
The total set of attempts relevant to a reported collection rate, explicitly defined so retries and invalid episodes cannot vanish.
Physical example:Thirty-two episodes were opened, thirty reached a terminal event, twenty-eight were structurally valid, and twenty-four passed behavioral review.
- Quota
A planned attempt count for a condition cell or group, with a declared rule for retries and targeted additions.
Physical example:Each of eight start-light cells receives four opened attempts before any optional difficult-cell oversampling.
- Operator protocol
The versioned instructions defining task intent, allowed control style, limits, reset, abort, and annotation expectations for demonstrators.
Physical example:Collectors must keep the wrist camera unobstructed, use bounded smooth increments, release on uncertainty, and finish at park before labeling.
- Linked retry
A new immutable episode related to an earlier failed or invalid attempt without replacing it.
Physical example:Episode E18 retries E17 after a camera drop; both remain in raw history and E18 records
retry_of=E17.- Release recipe
A versioned set of inclusion, exclusion, transformation, and label rules that selects training-ready records from immutable raw collection.
Physical example:Recipe R1 includes reviewed successes with residual below 20 ms and excludes invalid media while retaining every excluded episode in the audit ledger.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDataset coverage and independent scenes
More frames from the same easy scene do not equal broader robot experience.
- N_scene
- independent physical setupsUnit: scenes
- coverage
- tested condition cells representedUnit: percent
- failure rate
- failed frozen trials divided by all trialsUnit: percent
Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.
It is test-case coverage, not line coverage: the unit is a distinct physical condition.
Seven of ten condition cells are covered. What is coverage?
7/10 = 70%.
Condition share is
If dim light represents of expected deployment, the batch is under-covered by percentage points.
Collect and reconcile a 32-attempt first batch
The plan contains eight safe condition cells from two start sides, two object instances, and two light bands. The quota is four opened attempts per cell, for 8 × 4 = 32 total attempts.
Freeze the order, operator protocol, task contract, recorder revision, robot configuration, calibration, and retry rule; calculate the scheduled denominator of 32 before opening episode one.
After each block of eight, compare opened counts with all eight cell quotas and operator/session shares; do not let a quick retry replace the next scheduled cell.
When episode E09 loses camera frames, close it as
invalid_recording, retain its raw identity and cell, and create E10 as a linked retry while keeping the original 32-attempt denominator report intact.When E17 drops the object, close it as a valid
task_failure; retain the behavior and terminal evidence, but do not include its failed action sequence in the successful-demonstration recipe by default.At batch end, reconcile 32 opened episodes: 30 terminally complete, 29 structurally valid, 25 task successes, 3 valid task failures, 1 structurally invalid episode, 1 operator abort, and 2 incomplete attempts; define clearly which categories overlap or are mutually exclusive.
Create release candidate R1 from episodes that pass structure, synchronization, task, behavior, and rights review; publish both raw collection counts and R1 counts per condition rather than reporting only the attractive selected total.
The first batch retains the history of all 32 attempts, exposes one recorder defect and condition-level rates, and selects demonstrations through an explicit recipe without rewriting retries into a perfect collection story.
Balanced collection is controlled sampling plus honest accounting, not an equal-looking final folder produced by deleting awkward attempts.
Physical examples
Where this appears in real life
Easy center starts crowd out the corners
A collector enjoys smooth center-cell grasps and records twelve of them, but opens only two attempts from each corner because those require slower alignment.
The live quota display blocks convenience-driven repetition, preserves every opened attempt, and makes the corner deficit visible before the session closes.
Two rover operators, two driving signatures
Operator A uses short steering pulses while operator B holds longer arcs; both can dock successfully, but A records only bright scenes and B only dim scenes.
Operator and light are confounded despite equal success counts, so future blocks cross both operators with both conditions or narrow the supported claim.
Hands-on exercise
Make the idea observable
Use the safe simulator console and condition ledger. If no recorder is available, represent episodes with immutable cards carrying the same IDs, outcomes, and condition fields.
Freeze the operator protocol, cell quotas, block order, retry linkage, stop rules, and the distinction among raw, reviewed, and release-selected records.
Preserve the last working recorder baseline, run one nominal calibration episode and one safe-cancellation episode excluded from v1, expose both in logs or the inspector, and only then authorize collection.
Collect at least two attempts per safe cell without skipping difficult rows; create a new episode for every retry and preserve the prior outcome.
After each block, calculate opened, complete, structurally valid, behavior-pass, and task-success counts by cell, operator, and session.
Inspect action bounds, missing fields, timestamp residuals, dropped sequences, terminal evidence, intervention, and duration before continuing.
Stop on a repeated recorder defect, an unsafe condition, or a confounded schedule; repair the protocol or narrow the claim before collecting further episodes.
The first few episodes reveal protocol drift and unequal difficulty quickly, while counts based only on retained successes hide both recorder reliability and the states demonstrators struggled to recover from.
Every scheduled attempt has an immutable identity and disposition, cell counts reconcile, retries link rather than overwrite, operator/session correlations are visible, and release inclusion follows a written review recipe.
Build today
Collect, review, and version real demonstrations for one useful tabletop or mobile task.
Evidence to save
DONE when the integrated “Collect a balanced first demonstration batch” path is observable, cancelable, and leaves the prior baseline reproducible.
Common mistakes
Catch the wrong mental model
Replacing a failed episode with its successful retry under the same identity.
Close the first attempt with its real outcome, open a new identity linked through retry_of, and report rates over the original collection denominator as well as the curated release.
Equating balance with identical counts for every metadata value.
Allocate by the deployment claim and risk, audit important intersections, and document deliberate oversampling; arbitrary equality can waste episodes while leaving meaningful states sparse.
Waiting until the target episode count is reached before reviewing recorder health.
Review small blocks for structure, synchronization, control bounds, outcomes, and condition counts so a repeated protocol or infrastructure defect stops collection early.
Job connection
How this becomes employable evidence
Operate a robot demonstration campaign with quota-aware collection UI, operator protocols, immutable retries, live multimodal health checks, and release recipes that preserve raw failure evidence while selecting behavior fit for a stated learning objective.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robot HMI / Control & Monitoring Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 23 interview drill
Interview questions: Collect a balanced first demonstration batch
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Your collectors report 30 good demonstrations after many retries. Describe the denominators, quota process, operator bias checks, failure retention, and release recipe needed before you would call the batch balanced.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why must a retry receive a new episode identity?
A new identity preserves the earlier failure or invalid attempt, keeps denominators honest, and lets reviewers reconstruct why and how the recollection occurred.
Q2Does a valid task failure belong in raw storage?
Yes. It is evidence about collection and difficult states; whether any portion enters a training recipe is a separate explicit decision tied to the learning objective.
Q3What should be checked after each small collection block?
Condition, operator, and session counts plus structural validity, synchronization, dropped sequences, action bounds, outcomes, interventions, duration, and terminal evidence.