Chapter 13 · Build a trustworthy PyTorch experiment from robot-shaped data

Today in the field story

One problem, then the next

Turn the saved crops into a dataset whose rows preserve production burst, scene, timestamp, and physical target. Split by related scene or collection session, not by convenient image row, and expose every transform. The planted scene-b leak in the starter demonstrates how near-twins can appear on both sides of evaluation. The scout’s first credibility gate is a manifest proving that held-out evidence stayed held out.

Why now: A correct tensor pipeline still produces misleading evidence when related samples leak across splits.
Ignore today: Ignore distributed loading and large-scale augmentation; build a small grouped, inspectable pipeline.
Unlocks next: Training and evaluation batches with defensible lineage.

Understand

Build the physical picture first

A Dataset is a librarian handing out one correctly labelled case, while a DataLoader packs cases into batches without moving related exam answers between rooms.

A robot-learning sample is a schema, not whatever a file happens to contain. One sample might map a timestamped observation vector to a two-dimensional target action; an image sample might also carry scene ID, episode ID, camera identity, units, frame, outcome, and validity. A map-style PyTorch Dataset reports its length and returns one example by index. The training loop should not know filenames or repair missing fields. Validate the schema at the Dataset boundary so every sample has defined shapes, dtypes, finite values, allowed ranges, and lineage before batching begins.

Transforms convert stored data into the model’s input and target contract. Resizing, cropping, normalization, dtype conversion, and physically valid augmentation belong in a documented order because those operations do not commute. Normalization statistics are learned from the training set only; using validation pixels to calculate the mean or range leaks evaluation information into preprocessing. Random augmentation belongs on training examples and should be reproducible when debugging. Validation and test transforms should normally be deterministic so a metric change reflects the model or data version rather than a fresh random crop.

A DataLoader chooses sample order, groups samples into minibatches, and can use worker processes to prepare them. Batch size changes memory use, gradient noise, and the shape seen by the model. With 103 samples and batch size 16, ordinary non-dropping iteration produces six full batches and one final batch of seven. Code must either accept that smaller batch or deliberately set drop_last and record that seven examples were excluded from each epoch. Shuffling changes order, not split membership, and sequential evaluation is often easier to trace.

Robot frames are strongly related within a scene or episode. Randomly splitting individual frames can place near-identical moments from one grasp in both training and validation, allowing the model to remember a background, operator, or start pose. Split whole groups—such as episode, route, object instance, or collection session—before fitting transforms. Freeze IDs and verify set intersections are empty. Class balance alone is not enough: a held-out split also needs the lighting, surfaces, poses, speeds, failures, and sensor conditions required by the intended evaluation claim.

Words you need

Name each idea precisely

Dataset: A PyTorch interface that defines how many examples exist and returns one validated example with its target and lineage.
Physical example:
Index 17 returns one camera crop, two-coordinate cup target, scene ID, episode ID, and validity flag rather than an unnamed image.
Schema: The fixed meanings, shapes, dtypes, units, frames, ranges, required metadata, and version for each sample field.
Physical example:
A target [x, y] in metres in camera_optical_frame is a different schema from pixel coordinates [column, row].
Transform: A documented operation that converts a stored feature or target into the representation consumed during training or evaluation.
Physical example:
Convert an unsigned-byte image to float, scale it to zero through one, then normalize with training-only channel statistics.
DataLoader: An iterable that selects examples, collates them into batches, and optionally shuffles or prepares them with worker processes.
Physical example:
Sixteen image-target pairs become input shape [16, 3, 64, 64] and target shape [16, 2] for one update.
Minibatch: A subset of examples evaluated together before one optimizer update or metric accumulation step.
Physical example:
A final batch of seven is still valid when 103 examples are loaded sixteen at a time without dropping data.
Split leakage: Evaluation contamination caused when validation or test information, or closely related examples, influences model training or preprocessing.
Physical example:
Adjacent frames from one stationary camera episode appear in both training and validation, so the background is almost identical in the exam.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalTensor shapes and gradient descent

Most robot-learning bugs are shape, scale, split, or optimization mistakes.

B×T×D: batch, time steps, and feature dimensionsUnit: counts
L: loss measuring prediction errorUnit: task-dependent
η: learning rateUnit: unitless scale

A batch has B=8 episodes, T=20 time steps, and D=12 features.
The tensor contains 8×20×12 = 1,920 numbers.
An optimizer updates a weight with w_new = w_old − η∂L/∂w; inspect shape and finite values before training.

Programmer analogy

Tensor shapes are explicit multidimensional array contracts: every axis has a declared meaning and size.

How many values are in a 4×10×6 tensor?

4×10×6 = 240 values.

With $N=103$ examples and batch size $16$ ,

\left\lceil\frac{N}{16}\right\rceil=7\ \text{batches},

and the final batch contains $103-6(16)=7$ examples. A regression batch may have $\mathrm{MSE}=2.5$ , but MSE cannot detect split leakage or wrong transforms.

Design a grouped split and count its batches

A synthetic cup-localization dataset has 12 episodes. Assign eight episodes to training, two to validation, and two to test. The eight training episodes contain 103 examples and use batch size 16 without dropping the last batch.

Freeze episode IDs before reading frames: train {E01…E08}, validation {E09,E10}, and test {E11,E12}.
Verify pairwise set intersections are empty; an empty result tests identity separation, while condition coverage needs a separate table.
Fit image normalization values using frames from E01…E08 only, then reuse those frozen values unchanged on validation and test.
Compute full training batches: integer division 103 ÷ 16 gives six complete batches containing 96 examples.
Compute the remainder: 103 - 96 = 7, so non-dropping iteration yields 6 + 1 = 7 batches and the last batch size is seven.
Record expected input and target shapes for both batch cases and assert that episode IDs emitted by each loader belong only to its frozen split.

Result

The loader yields seven training batches, preserves all 103 examples, and demonstrates episode-level identity isolation; it still needs a condition-coverage review before supporting a generalization claim.

What this proves

Split related robot data before preprocessing, then make the loader prove both identity separation and exact batch contracts.

Physical examples

Where this appears in real life

Episode envelopes

Put every frame card from one pretend robot attempt into a sealed envelope, then assign entire envelopes to train, validation, or test boxes.

Look for:

No episode identity crosses boxes, even if this makes the number of individual frames less perfectly balanced than random frame splitting.

Packing the last delivery tray

Pack 103 counters onto trays that hold 16, keeping the last partially filled tray instead of inventing counters or quietly throwing it away.

Look for:

Six trays contain 16 and the seventh contains seven; batch-aware code handles both shapes along the batch axis.

Hands-on exercise

Make the idea observable

Create a tiny non-sensitive synthetic dataset with observation vectors or generated colored squares, numeric targets, and at least six scene or episode groups.

Write a versioned sample schema listing every field, shape, dtype, unit, frame or coordinate convention, range, nullable rule, and group identity.
Implement a Dataset that validates one sample at a time and deliberately add one missing field, wrong shape, non-finite value, and unknown schema version to prove each failure is rejected.
Assign whole groups to train, validation, and test using frozen ID lists; assert pairwise intersections are empty and save per-split condition counts.
Fit any mean, standard deviation, or range on training samples only; make training augmentation stochastic but physically plausible and make validation transforms deterministic.
Build loaders with a batch size that creates a smaller final batch, then print batch shapes, group IDs, target ranges, and included example counts.
Save schema, split manifest, preprocessing values, loader configuration, checks, and one contact sheet or table that lets a reviewer inspect examples from every split.

Observe

Schema defects fail close, group IDs never cross splits, validation samples stay stable between passes, and the last batch differs only along the batch axis.

Done when

A clean run proves zero group overlap, accounts for every included and dropped example, reproduces preprocessing values, and shows the exact sample that each deliberate validator rejects.

Build today

Train a small image-to-pose or observation-to-action network with a reproducible experiment report.

Evidence to save

DONE when “Datasets, dataloaders, transforms, and train/validation splits” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Randomly splitting individual frames because it produces balanced row counts.

Better

Group related frames by episode, scene, object, route, or collection session before splitting, then measure condition coverage separately.

Wrong

Calculating normalization statistics over the full dataset before making splits.

Better

Fit every data-dependent preprocessing value on the training split only and freeze it for validation, test, and later inference.

Wrong

Assuming every DataLoader batch has the requested batch size.

Better

Handle the smaller final batch or deliberately document drop_last; assert only invariant feature and target axes.

Job connection

How this becomes employable evidence

Build a robot-observation Dataset and DataLoader contract that preserves episode lineage, applies training-only preprocessing statistics, blocks schema drift, and produces a reviewable split and condition-coverage report.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer

Chapter 13 interview drill

Interview questions: Datasets, dataloaders, transforms, and train/validation splits

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Your image model has excellent validation accuracy, but adjacent frames from each robot episode were randomly split. Explain why this is leakage and redesign the split, transforms, and checks.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1What is the difference between a Dataset and a DataLoader?

Model interview answer

A Dataset defines and returns individual indexed examples; a DataLoader selects their order, batches them, and optionally prepares them with workers.

Q2Why should normalization values come from training data only?

Model interview answer

Validation or test statistics would let evaluation information influence preprocessing and make the reported generalization estimate optimistic.

Q3Does an empty intersection of episode IDs prove a good evaluation split?

Model interview answer

It proves identity separation for those IDs, but condition coverage, label quality, collection bias, and deployment relevance still need direct review.