Chapter 23 · Collect and freeze a task-shaped capstone dataset

Today in the field story

One problem, then the next

The starter's planted leak reaches the split desk: scene-a appears in both training and held-out records. The Foundry rejects it, because adjacent episodes and repeated takes share far more than filenames reveal. Whole scene, object, session, and operator groups are assigned according to the declared generalization question. Normalization and derived features fit only on training data; validation guides decisions; held-out identities stay sealed until Week 24's candidate and rules are frozen.

Why now: Group-disjoint splits protect the meaning of future generalization evidence.
Ignore today: Do not randomize rows or inspect held-out outcomes during development.
Unlocks next: Train, validation, and held-out manifests with proven empty intersections.

Understand

Build the physical picture first

A trustworthy split is three locked rooms for related physical scenes: every frame, retry, crop, and statistic follows its scene family into one room and never slips under another door.

Choose the independence unit before assigning percentages. Individual video frames are highly related to neighboring frames, and separate episodes can still share the same object instance, fixture layout, reset, camera pose, session, or scripted motion. Create a scene_group_id from the smallest family that could leak recognizable physical information. All episodes, retries, clips, corrected labels, and deterministic derivatives from that family inherit one split.

Give each partition a distinct job. Training episodes fit model parameters and training-only transforms. Validation scenes support architecture, checkpoint, threshold, and collection decisions. Held-out scenes stay sealed until the policy and decision rules are frozen, then estimate behavior on genuinely unseen physical arrangements. Looking repeatedly at held-out failures and tuning to them converts those scenes into validation data, even if filenames still say held_out.

Stratify at group level where possible, but never break a group merely to make frame or episode percentages pretty. A small dataset may have unequal counts after grouping. Compare factor and outcome coverage across partitions, document missing levels, and add independent scene groups if the held-out claim needs broader support. A held-out scene should differ in the intended way while remaining inside the declared task and safety envelope; an impossible new task is not a fair generalization test.

Prevent less obvious leakage. Fit normalization, bounds learned from data, vocabulary, image statistics, and feature selection on training data only. Keep augmented frames, temporal windows, re-encoded videos, clips, and relabeled copies with their source group. Do not use held-out performance to choose a camera crop, success tolerance, or preprocessing revision. Freeze a split manifest containing group identities, reasons, seed or assignment procedure, and hashes, then test disjointness before any training loader opens the data.

Words you need

Name each idea precisely

Scene group: The set of episodes and derivatives sharing physical or procedural identity strongly enough that crossing partitions would leak familiarity.
Physical example:
All attempts and retries using fixture layout F3, sponge instance S2, camera pose C7, and one reset block share group G-F3-S2-C7.
Train split: The only partition used to fit policy parameters and any data-derived preprocessing statistics.
Physical example:
Joint normalization mean and scale are calculated from training episodes, then frozen before validation and held-out loading.
Validation split: Independent groups used repeatedly for model, checkpoint, threshold, and engineering decisions without updating on their examples directly.
Physical example:
Two validation layouts compare checkpoints and reveal that dim scenes need a better training collection plan.
Held-out scene split: Sealed in-envelope scene groups reserved for final evaluation after model and decision rules are frozen.
Physical example:
A bowl at an unseen allowed fixture position and a fresh sponge instance remain inaccessible until the policy package is selected.
Data leakage: Information from validation or held-out groups influences fitting or decisions in a way that makes reported generalization overly optimistic.
Physical example:
Frames from one grasp train the model while adjacent frames from the same episode score it as though they were independent.
Split manifest: An immutable mapping from source group and episode identities to partitions, with assignment method, rationale, and content hashes.
Physical example:
Manifest M1 maps every retry of scene group G08 to validation and fails validation if any derived clip appears in train.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalDataset coverage and independent scenes

More frames from the same easy scene do not equal broader robot experience.

N_scene: independent physical setupsUnit: scenes
coverage: tested condition cells representedUnit: percent
failure rate: failed frozen trials divided by all trialsUnit: percent

Your matrix has 3 object positions × 2 lighting levels = 6 condition cells.
Data covers 4 cells, so coverage is 4/6 = 66.7%.
Collect the two missing cells before duplicating the easiest one; split by whole scene, not adjacent frames.

Programmer analogy

It is test-case coverage, not line coverage: the unit is a distinct physical condition.

Seven of ten condition cells are covered. What is coverage?

7/10 = 70%.

For $N=100$ episodes, a $70/15/15$ target gives

N_{\mathrm{train}}=70,\quad N_{\mathrm{val}}=15,\quad N_{\mathrm{test}}=15.

Group whole related scenes before splitting, even if that makes the final percentages slightly uneven.

Split 48 episodes without cutting scene families

The reviewed release candidate has 12 independent scene groups, each containing four related episodes or retries, for 48 episodes. The intended assignment is eight training groups, two validation groups, and two sealed held-out groups.

Define group identity from fixture layout, object instance, camera pose, and reset block; verify every episode has exactly one scene_group_id and every retry inherits its parent group.
Assign eight whole groups to train, two to validation, and two to held-out, yielding 8 × 4 = 32, 2 × 4 = 8, and 2 × 4 = 8 episodes without moving individual episodes to polish percentages.
Compare start, light, object, outcome, operator, and session coverage at the group and episode levels; record any factor absent from validation or held-out rather than borrowing a related training episode.
Traverse provenance for cropped frames, temporal windows, re-encoded media, corrected annotations, and other derivatives, requiring every descendant to inherit the source group’s partition.
Fit normalization and any data-derived thresholds from the 32 training episodes only; store their fitted values and training-manifest hash, then apply them unchanged elsewhere.
Seal held-out identities and labels from routine model development, freeze validation-based checkpoint and decision rules, and open held-out only for the declared final evaluation.
Run disjointness checks over episode IDs, scene groups, raw hashes, source ancestry, reset blocks, and derived artifacts; fail the release if any prohibited intersection is nonempty.

Result

The 48 episodes form 32/8/8 partitions by whole scene family, all derivatives inherit their source assignment, statistics come only from train, and held-out scenes remain a final in-envelope test.

What this proves

The honest split unit is the strongest shared cause of familiarity, not the smallest row that makes percentages convenient.

Physical examples

Where this appears in real life

One grasp appears in two folders

A recorder exports the first 80 percent of frames from each episode to train and the remaining 20 percent to validation, so nearly adjacent images of every motion occupy both partitions.

Look for:

Frame counts look balanced but evaluation is contaminated; the entire episode and its derivatives must follow a broader physical scene group into one partition.

New filename, familiar sponge

A retry uses the same sponge, bowl layout, camera pose, reset block, and operator path as a training attempt but receives a new episode name and is assigned held-out.

Look for:

Episode identity alone is too weak; shared physical and procedural ancestry places both attempts in the same scene group.

Hands-on exercise

Make the idea observable

Use the reviewed episode ledger, raw lineage, variation fields, reset blocks, sessions, and operator metadata. Do not train a model during this split-design exercise.

List every relationship that can make two episodes familiar: same continuous episode, retry, object instance, fixture, reset, camera pose, session, operator pattern, or derived media.
Define and assign one stable scene_group_id per related family, resolving orphaned or ambiguous lineage before splitting.
Choose whole groups for train, validation, and held-out scenes based on each partition’s job and the intended evaluation claim.
Audit group and episode coverage across task-relevant factors, recording shortages without moving related examples across boundaries.
Attach every crop, clip, augmentation, relabel, re-encode, and temporal window to its source group and partition.
Calculate preprocessing statistics from train only, then write a manifest with assignments, rationale, procedure or seed, schema, and hashes.
Run the split workflow from one documented command; capture its disjointness trace or plot, a 60–120 second uncut manifest walkthrough, and the largest remaining coverage failure before sealing held-out access.

Observe

Group-level splitting produces less tidy percentages but removes near-duplicate physical experience that would otherwise make validation and held-out scores falsely reassuring.

Done when

Every release episode and derivative has one inherited partition, all tested intersections are empty, preprocessing fits only train, and held-out scenes can be opened under a declared final-evaluation procedure.

Build today

Collect, review, and version real demonstrations for one useful tabletop or mobile task.

Evidence to save

DONE when a 60–120 second uncut “Create train, validation, and held-out scene splits” demo links to its command, logs or plots, result count, and honest failure note.

Common mistakes

Catch the wrong mental model

Wrong

Randomly assigning individual frames to achieve an exact 80/10/10 ratio.

Better

Assign whole scene families and their derivatives to one partition; accept untidy row counts or collect more independent groups rather than contaminating evaluation.

Wrong

Using held-out results repeatedly to choose checkpoints or preprocessing.

Better

Make iterative decisions on validation, freeze model and rules, then use the sealed held-out scenes only for the declared final assessment or retire them into future validation.

Wrong

Computing normalization over all episodes before splitting.

Better

Fit every data-derived transform on the training manifest only, store its provenance, and apply the frozen transform unchanged to validation and held-out data.

Job connection

How this becomes employable evidence

Design leakage-resistant robot-learning partitions by tracing scene and retry lineage, assigning whole physical groups, fitting preprocessing on train only, sealing held-out evaluation, and automating disjointness checks in the dataset pipeline.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 23 interview drill

Interview questions: Create train, validation, and held-out scene splits

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Why is episode-level random splitting sometimes still unsafe for robot demonstrations? Define your scene group, the jobs of train, validation, and held-out scenes, and the leakage tests you would automate.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why can two different episode IDs still belong to one scene group?

Model interview answer

They may be retries or share object, fixture, camera, reset, session, or procedural ancestry strongly enough that one makes the other familiar.

Q2What happens if held-out scenes guide repeated model changes?

Model interview answer

They have functionally become validation data, so their later score no longer estimates performance on untouched scenes.

Q3Where are normalization statistics fitted?

Model interview answer

Only on the training partition after the split manifest is frozen, with the training manifest and transform values recorded for provenance.