Chapter 23 · Collect and freeze a task-shaped capstone dataset
Today in the field story
One problem, then the next
The foundry lays out condition cells for object instance, scene, lighting, camera pose, start state, and operator while checking that one factor does not accidentally predict another. Unsafe or impossible combinations are excluded with reasons, and a repeatable reset accompanies every cell. Variation is bounded enough to collect deliberately yet broad enough to expose the task's intended envelope. Group identities are assigned now so later split creation does not depend on which episodes look easiest.
- Why now
Planned variation prevents hidden correlations and opportunistic held-out selection.
- Ignore today
Ignore exhaustive coverage and uncontrolled random collection.
- Unlocks next
A collection matrix with stable scene groups and reset rules.
Understand
Build the physical picture first
Variation design is a tray of labeled starting arrangements: change one bounded physical factor on purpose, record its identity, and prevent the schedule from hiding two changes inside one pattern.
Choose variation from the operating claim, not from whatever is easy to move. For a tabletop task, relevant factors may include object instance, start cell, yaw range, distractor layout, illumination band, camera mount, and robot reset pose. Give each factor discrete levels or a sampled bounded range, a measurement method, and an exclusion boundary. A bright lamp angle is not a useful level unless illuminance or a reproducible fixture setting identifies it.
Separate semantic factors from nuisance factors. Changing the requested target or object class can change the task meaning; changing light direction or a harmless distractor may preserve the task while testing perception robustness. Do not vary unmodeled hazards, payload, grip material, speed, or workspace geometry casually. A dataset needs variety inside a declared safe envelope, not surprising states that the robot, tool, or supervision plan was never commissioned to handle.
Balance intersections, sessions, and order. If every far-right start is recorded late under dim light by one tired operator, the policy can learn the session, operator style, or brightness as a shortcut. Build a condition ledger, randomize or counterbalance safe cells within each session, rotate operators where permitted, and repeat cells across time. Preserve session, reset, object instance, and configuration IDs so correlations remain measurable rather than disappearing into filenames.
A variation plan includes stop rules and review, not only quotas. Reject a proposed cell when the object is unreachable, a camera loses the terminal region, a cable or collision margin is violated, or reset evidence cannot be reproduced. Run a small pilot across extreme allowed levels before filling the grid. If a boundary condition creates unsafe or systematically invalid episodes, narrow the declared envelope and document why instead of forcing equal counts.
Words you need
Name each idea precisely
- Variation factor
One declared physical or contextual property deliberately changed across episodes while its level and measurement remain recorded.
Physical example:The sponge begins in left, center, or right start cells whose taped boundaries and coordinates are versioned.
- Factor level
A named value, interval, or category allowed for one variation factor.
Physical example:Illumination level
L2means the fixed lamp setting that produces 280–320 lux at the marked table point.- Condition cell
One intersection of selected factor levels used for scheduling, counting, and coverage review.
Physical example:Cell
S=right, O=sponge2, L=dimdescribes a specific start, object instance, and lighting combination.- Confounding
A collection pattern in which two factors always change together, making their separate relationship to behavior impossible to determine.
Physical example:All blue objects are recorded by operator A and all yellow objects by operator B, so colour and operator technique cannot be separated.
- Counterbalancing
Scheduling condition orders so time, fatigue, warm-up, battery, or session effects do not repeatedly favor the same factor level.
Physical example:One session records left-center-right starts, while the next uses right-left-center instead of always leaving right starts until last.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalDataset coverage and independent scenes
More frames from the same easy scene do not equal broader robot experience.
- N_scene
- independent physical setupsUnit: scenes
- coverage
- tested condition cells representedUnit: percent
- failure rate
- failed frozen trials divided by all trialsUnit: percent
Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.
It is test-case coverage, not line coverage: the unit is a distinct physical condition.
Seven of ten condition cells are covered. What is coverage?
7/10 = 70%.
A Cartesian coverage grid with poses, lighting levels, and starts has
cells. If one cell receives of trials, its share is . Mark unsafe or impossible cells explicitly rather than silently omitting them.
Build and deconfound a 24-episode pilot schedule
The sponge task varies three start cells L/C/R, two object instances S1/S2, and two light bands bright/dim. The full grid contains 3 × 2 × 2 = 12 condition cells, with two pilot episodes planned per cell.
Create stable IDs and measurement rules for all 12 cells; keep bowl pose, camera mount, robot limits, gripper, task wording, and calibration fixed for this pilot.
Set the planned denominator to
12 cells × 2 episodes = 24 episodes, while retaining separately any invalid starts or aborted recordings that cause recollection.Reject a naive order that records all S1-bright cells first and S2-dim cells last, because object, light, time, and operator adaptation would move together.
Construct two 12-cell blocks in different safe orders so each start, object, and light level appears early and late; require a fresh reset proof before every row.
Pilot the geometric extremes
LandRunder both light bands before completing the center cells; stop if reach, visibility, command margin, or reset reproducibility leaves the approved envelope.After collection, reconcile scheduled versus attempted versus valid counts per cell and preserve session, order index, reset, object, lighting, and configuration identities for later coverage and split grouping.
The schedule covers 12 declared intersections twice without permanently coupling object, light, and order; extreme cells are tested early, and rejected conditions remain visible rather than being silently replaced.
Useful diversity is designed, bounded, and attributable; uncontrolled novelty produces a confusing dataset rather than robust evidence.
Physical examples
Where this appears in real life
Same sponge, different honest starts
A simulated gripper handles two sponge instances from three taped start cells under two reproducible light settings, while bowl pose, camera mount, speed limits, and task wording remain fixed.
Every episode carries explicit factor levels, each allowed extreme was piloted, and object identity is not permanently paired with one start cell or light level.
Docking grid with a hidden session shortcut
A rover records all left-offset docking starts in the morning with a full battery and all right-offset starts after lunch with lower voltage and a shifted camera exposure.
Start side is confounded with session conditions; repeat both sides across sessions or narrow the claim, and record battery and camera configuration rather than pretending only pose changed.
Hands-on exercise
Make the idea observable
Use the Day 155 contract with a simulator or paper scene. Choose only factors that can be reset and measured without changing the approved safety envelope.
List candidate scene, object, lighting, start-state, camera, and operator factors, then mark each as task-semantic, nuisance, fixed, or excluded.
Select two or three factors for v1, define every level with units or a reproducible identity, and document the physical boundary outside which collection stops.
Generate the complete condition-cell ledger and calculate the planned episode denominator before deciding the order.
Create two counterbalanced collection blocks that distribute factor levels across early and late positions and avoid assigning one operator or session to a single level.
Run one nominal center-cell case and one boundary case from the hardest allowed cell in simulation, checking inputs, outputs, units, reach, visibility, collision margin, command limits, reset proof, and explicit failure result.
Revise the ledger with rejected cells, reasons, narrowed claims, stable factor IDs, session fields, and a quota that another collector could follow exactly.
A small grid quickly reveals correlations that prose hides, and the hardest combination often exposes a visibility, reach, or reset problem before large-scale recording begins.
Every scheduled row maps to a measurable allowed scene, no factor level is permanently tied to one operator or order position, and the planned denominator reconciles with the cell quotas.
Build today
Collect, review, and version real demonstrations for one useful tabletop or mobile task.
Evidence to save
DONE when “Design scene, object, lighting, and start-state variation” runs from one documented command and the nominal plus boundary outputs are attached.
Common mistakes
Catch the wrong mental model
Calling random object placement a sufficient variation strategy.
Define bounded factors and measured levels, log the realized condition, and compare cell counts so randomness cannot conceal unreachable regions or repeated easy starts.
Recording one condition per session because setup changes are faster.
Counterbalance safe levels within or across blocks and retain session identity; otherwise lighting, battery, warm-up, operator adaptation, and time can become shortcuts.
Forcing an unsafe or unobservable factor cell to meet an equal quota.
Stop collection, mark the cell rejected with evidence, repair the setup or narrow the supported envelope, and never trade a safety or validity boundary for visual balance.
Job connection
How this becomes employable evidence
Design robot-learning collection matrices that represent deployment-relevant scene and start-state variation, block unsafe combinations, deconfound operator and session effects, and leave a traceable schedule for targeted recollection.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 23 interview drill
Interview questions: Design scene, object, lighting, and start-state variation
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
You have two objects, three start poses, and two lighting conditions but only 24 demonstrations. Explain the condition grid, order design, confounding risks, pilot boundaries, and what metadata you must retain.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What makes a proposed lighting level reproducible?
It has a stable fixture setting or measured range at a named point, along with camera and session configuration, rather than a label such as “a bit dimmer.”
Q2Why is recording all right-side starts at the end risky?
Start side becomes correlated with fatigue, warm-up, battery, lighting drift, and learned operator technique, so the dataset cannot cleanly attribute behavior to the start condition.
Q3Should every mathematically possible factor combination be collected?
No. Only combinations inside the task, visibility, reach, collision, hardware, supervision, and reset boundaries belong; excluded cells and narrowed claims must be documented.