Chapter 13
Build a trustworthy PyTorch experiment from robot-shaped data
Learn PyTorch as an engineering tool rather than a magic prediction box. The chapter starts with tensor and gradient contracts, builds a leakage-resistant robot-data pipeline, compares dense and convolutional encoders, trains and resumes a model correctly, diagnoses shortcuts and overfitting, measures variation across runs, and ends with a model card that exposes accuracy, latency, failure slices, intended use, and unsafe boundaries.
Before you start
- Use Python functions, lists, dictionaries, loops, assertions, files, and a small isolated environment without placing credentials or sensitive recordings in the lab.
- Read array shapes, multiply small integers, calculate an average and squared error, and distinguish a measurement from a command and an estimate from independent ground truth.
- Reuse a small recorded or synthetic Week 9–12 observation table, image crop, pose target, or action target; keep every exercise disconnected from powered hardware.
- Preserve units, frames, timestamps, episode or scene identity, data version, configuration, and failed examples so a model result can be traced back to physical meaning.
By the end
- Trace tensor shape, dtype, device, units, and semantic axes through a forward pass; detect unintended broadcasting and check one autograd result with a finite difference.
- Implement a Dataset, documented transforms, and a DataLoader whose batches preserve schema and whose train, validation, and test groups cannot leak related robot scenes or episodes.
- Choose and inspect a small MLP or convolutional encoder by input structure, output contract, receptive field, parameter count, latency target, and baseline rather than by fashion.
- Run a correct training and validation loop with an appropriate loss, optimizer, schedule, train/eval modes, resumable checkpoint, and an explicit no-gradient inference boundary.
- Distinguish optimization progress from generalization, repair shortcut leakage, constrain augmentation to physically plausible changes, and beat a simple held-out baseline.
- Record code, environment, data, configuration, random-state controls, every seed result, metrics, artifacts, and timing so another engineer can replay and challenge the experiment.
- Publish a robot-facing model card with intended and excluded uses, data lineage, condition slices, worst cases, p95 inference latency, limitations, rejection behavior, and a precise offline evidence boundary.
The field story
Build an offline seal-defect scout
A packaging cell has saved camera crops and measured cap offsets from an earlier inspection run. Operators want a small model that flags likely misaligned seals before a human review, but the images contain repeated production bursts, changing illumination, and labels tied to physical millimetres. You will build an offline scout, beginning with tensor axes and ending with a model card. It will never command the cell. Its value comes from traceable data, a credible baseline, grouped evaluation, measured latency, and visible failure slices.
The tempting path is to train until one curve looks smooth. The engineering path is harder: state every tensor’s shape and meaning, keep related bursts out of opposing splits, compare an MLP and compact visual encoder, preserve train and evaluation modes, challenge shortcut learning, replay several seeds, and time the full input-to-output path. A checkpoint is only one artifact in the story. The mission finishes when another engineer can reproduce the experiment and understand exactly where the scout should abstain.
- Why this chapter now
Weeks 9–12 produced robot-relevant images, poses, and outcomes; PyTorch can now be learned through data whose physical meaning, grouping, and failure costs are already understood.
- Ignore for now
Do not chase large architectures, cloud training, foundation models, or powered deployment. Use a small offline dataset and one falsifiable prediction contract.
- This unlocks
A disciplined training and evaluation workflow becomes the foundation for demonstrations, imitation learning, VLA adaptation, and deployable model evidence.
- Proof you will leave with
Provide schema and shape checks, grouped split manifest, baseline, resolved configuration, training and validation curves, seed table, malformed-input test, checkpoint lineage, held-out slices, latency distribution, worst cases, and model card.
Environment contractRepository-supported Node.js 22.13.0 or newer runs the leakage starter. Model exercises use the repository’s isolated Python and PyTorch environment when present, recorded or synthetic data, and CPU or an available GPU without touching powered hardware.
- Compatibility boundary
PyTorch, accelerator, driver, and deterministic-operation behavior can drift. Record the actual interpreter, package lock, device, seed controls, and unsupported operations rather than claiming bitwise parity across machines.
- Smoke check
Run
node week-13-detect-split-leakage.mjs; in the model environment, import the pinned packages, run one tiny batch through save and fresh-process load, and verify schema, output shape, and finite values.- Contract reviewed
2026-07-25
- Runtime evidence
The dependency-free starter is executed by repository tests on the supported Node.js baseline. Chapter-specific ROS 2, Gazebo, model, dataset, checkpoint, and hardware environments are learner-created unless the repository supplies an explicit asset; run the smoke check and preserve its versions and output before claiming runtime compatibility.
- Drift risk
high
Today in the field story
One problem, then the next
Open one seal crop and its measured offset as named tensors rather than anonymous arrays. Record batch, channel, height, width, dtype, device, units, and target axis; then plant an unintended broadcast that produces a plausible loss. Check one gradient with a finite difference and clear accumulated gradients deliberately. The scout cannot be debugged later if its first numeric contract is ambiguous today.
- Why now
Every later model result depends on correct tensor meaning and gradient behavior.
- Ignore today
Ignore architecture performance; make shapes, devices, units, and numerical checks explicit.
- Unlocks next
A verified forward-and-gradient fixture for the saved inspection data.
Understand
Build the physical picture first
A tensor is a labelled crate of numbers, and autograd is the receipt trail showing how one final loss depends on every adjustable number upstream.
A PyTorch tensor is an array plus an engineering contract. Its shape tells how many positions exist along each axis, its dtype tells how each number is represented, and its device tells which processor owns its storage. For a camera batch shaped [B, C, H, W], the axes might mean examples, color channels, pixel rows, and pixel columns. Those meanings are not stored in the shape itself, so code, assertions, and names must preserve them. A tensor shaped [4, 3, 32, 32] contains 12,288 values, but that count cannot tell whether the colors are RGB, whether pixels were scaled, or which camera frame produced them.
Shape changes are part of the model, not tidying performed around it. Indexing can remove an axis, unsqueeze can add one, permute reorders axes, and reshape preserves element count but may return either a compatible view or a copy. Broadcasting lets compatible shapes act together by expanding size-one or missing leading dimensions. That is useful for subtracting a three-channel mean from every image. It is dangerous when a prediction shaped [B] meets a target shaped [B, 1]: PyTorch may create pairwise comparisons shaped [B, B] instead of raising an error. Assert the exact prediction and target shapes before computing a robot loss.
Autograd builds a graph of differentiable operations during the forward calculation when a leaf tensor requires gradients. Calling backward on a scalar loss applies the chain rule and accumulates each parameter’s derivative in .grad. A gradient is a local slope: its sign and size predict how a tiny parameter change would change the current loss. It is not a command for a motor and does not prove the model is physically correct. Gradients accumulate by default, which helps some algorithms but means an ordinary training loop must clear them deliberately before the next backward pass.
A small numerical check can catch a wrong derivative, detached graph, in-place change, or unexpected scale before training hides the defect. For a real scalar function, central difference estimates the slope as (f(w + ε) - f(w - ε)) / (2ε). Compare that estimate with autograd in double precision using a small but not absurd epsilon and an explicit tolerance. Finite differences also have rounding and truncation error, so approximate agreement is the goal. Separately reject non-finite inputs, loss, gradients, or parameters; a successfully completed backward call does not make NaN or infinity acceptable.
Words you need
Name each idea precisely
- Tensor
A multidimensional array whose shape, dtype, device, and semantic axes form the input, output, or parameter contract.
Physical example:Four RGB camera crops of 32 by 32 pixels form a
[4, 3, 32, 32]tensor when channel-first order is declared.- Shape
The ordered size of every tensor axis; the order must be paired with human meanings such as batch, time, joint, row, or column.
Physical example:A seven-joint state history for eight times can be
[8, 7], while[7, 8]contains the same count but assigns a different meaning to each axis.- Dtype
The numeric representation used for tensor elements, such as floating point for normalized measurements or integer indices for class labels.
Physical example:A camera byte value 255 and a normalized float value 1.0 can describe the same white pixel only after a declared scaling transform.
- Device
The processor and memory location that own a tensor, such as CPU, CUDA GPU, or Apple Metal acceleration.
Physical example:A model on a GPU cannot multiply a camera tensor still stored on the CPU until the input is moved to the same device.
- Broadcasting
Automatic expansion of compatible size-one or missing dimensions so an elementwise operation can produce a larger result.
Physical example:Subtracting one
[3, 1, 1]RGB mean from every pixel of a[B, 3, H, W]image batch uses intentional broadcasting.- Gradient
A derivative that estimates how the current scalar loss changes for a tiny change in one tensor element or model parameter.
Physical example:A negative gradient for one weight says a small increase in that weight would locally lower the current loss, before limits or later data are considered.
Visual model
See the relationship
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalTensor shapes and gradient descent
Most robot-learning bugs are shape, scale, split, or optimization mistakes.
- B×T×D
- batch, time steps, and feature dimensionsUnit: counts
- L
- loss measuring prediction errorUnit: task-dependent
- η
- learning rateUnit: unitless scale
A batch has B=8 episodes, T=20 time steps, and D=12 features.
The tensor contains 8×20×12 = 1,920 numbers.
An optimizer updates a weight with w_new = w_old − η∂L/∂w; inspect shape and finite values before training.
Tensor shapes are explicit multidimensional array contracts: every axis has a declared meaning and size.
How many values are in a 4×10×6 tensor?
4×10×6 = 240 values.
A tensor with shape contains
For targets and predictions , . Correct loss arithmetic does not prove tensor shapes are correct.
Check one autograd gradient by hand
Use the scalar prediction ŷ = wx and squared loss L = (ŷ - y)² with weight w = 1.5, input x = 2.0, target y = 4.0, and central-difference epsilon ε = 0.001.
Compute the prediction:
ŷ = 1.5 × 2.0 = 3.0; every quantity is a scalar floating-point tensor on one device.Compute the loss:
L = (3.0 - 4.0)² = 1.0; the loss is scalar, so backward can start without an extra gradient argument.Apply the chain rule:
dL/dw = 2(wx - y)x = 2(3 - 4)2 = -4; this is the expected autograd value.Evaluate the plus side:
f(1.501) = (1.501 × 2 - 4)² = (-0.998)² = 0.996004.Evaluate the minus side:
f(1.499) = (1.499 × 2 - 4)² = (-1.002)² = 1.004004.Calculate central difference:
(0.996004 - 1.004004) / 0.002 = -4.0, then compare with autograd using a declared tolerance.
Autograd and central difference both give a gradient of about -4.0 for this input; the check validates this local derivative, not the dataset, target, or robot behavior.
Trace shapes and physical meanings first, then use an independent numerical slope to challenge one gradient before trusting a long training run.
Physical examples
Where this appears in real life
Stacked observation cards
Make two paper camera cards, give each three transparent color sheets, and draw a four-by-four pixel grid on every sheet before stacking them in batch-channel-height-width order.
The physical stack makes [2, 3, 4, 4] readable, while swapping color sheets with camera cards changes axis meaning even though all 96 values remain.
Slope on an adjustable ramp
Place a marble at one marked point on a shallow paper ramp, then compare the height a tiny step to its left and right without letting the marble command anything.
The two nearby heights approximate the local slope just as a central difference checks a gradient, but the slope describes only that neighborhood.
Hands-on exercise
Make the idea observable
Use a tiny CPU-only PyTorch script and synthetic observation-to-action values. Do not install unreviewed data, use private recordings, or connect predictions to robot commands.
Create an observation tensor with explicit batch and feature axes; print shape, dtype, device, finite-value status, minimum, maximum, and the unit meaning of each feature.
Create targets with exactly the intended output shape, assert prediction and target equality, then deliberately change
[B]to[B, 1]and record the unintended broadcasted result shape.Trace a forward calculation through at least three operations, recording every input and output shape without using the loss value as a substitute for those checks.
Set one double-precision scalar parameter to require gradients, compute the worked squared loss, call backward once, and compare its gradient with the central-difference calculation.
Run backward twice without clearing gradients, observe accumulation, then use the training-loop clearing operation and verify the next gradient returns to the single-pass value.
Add assertions for finite inputs, loss, gradients, and outputs; save the script, decisive console table, PyTorch version, and one explanation of what this offline test cannot prove.
A mathematically legal broadcast can change the loss pairing silently, gradients add unless cleared, and a finite-difference check should closely match the simple autograd slope.
A reviewer can read every tensor axis, reproduce the 12,288-element count and -4 gradient, trigger the broadcast failure, and see the exact assertion that blocks it.
Build today
Train a small image-to-pose or observation-to-action network with a reproducible experiment report.
Evidence to save
DONE when the learning log explains “Tensors, shapes, devices, autograd, and numerical checks” in five precise points and a checked example produces the predicted output.
Common mistakes
Catch the wrong mental model
Treating a matching element count as proof that two tensors mean the same thing.
Name every axis and assert exact ordered shapes, units, scaling, frame, dtype, and device at the model and loss boundaries.
Assuming a loss call would reject every wrong prediction-target shape.
Check the shapes before the loss because broadcasting can produce a valid but unintended pairwise result without an exception.
Calling backward repeatedly and forgetting that gradients accumulate.
Clear gradients at the intended boundary, then verify one known gradient and reject non-finite values before the optimizer step.
Job connection
How this becomes employable evidence
Review a robot-policy training boundary by turning observation and action schemas into tensor assertions, detecting silent broadcasting or device errors, and adding finite and numerical-gradient gates before costly training begins.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 13 interview drill
Interview questions: Tensors, shapes, devices, autograd, and numerical checks
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A loss decreases even though predictions are shaped [32] and targets [32, 1]. Explain PyTorch broadcasting, inspect the produced loss shape, repair the contract, and describe one independent gradient check.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why can `[B]` predictions and `[B, 1]` targets be more dangerous than incompatible shapes?
They may broadcast to [B, B] and compute every prediction against every target, so training continues on the wrong objective instead of failing loudly.
Q2What does a gradient of -4 mean in the worked example?
Near the current weight, a small positive weight change is predicted to lower the current loss at roughly four loss units per weight unit; it is not a global promise.
Q3What does agreement with a finite difference establish?
It supports the local derivative implementation for the checked inputs and tolerance, but says nothing about data quality, generalization, latency, or physical safety.
Chapter references
- PyTorch — Learn the BasicsOfficial linked tutorials for tensors, datasets and DataLoaders, transforms, nn.Module construction, autograd, optimization loops, and basic model saving and loading.
- PyTorch — Developer NotesMaintainer notes for broadcasting, finite-difference gradcheck, numerical accuracy, autograd, random-source control, deterministic algorithms, device behavior, and cross-platform reproducibility limits.
- PyTorch — Saving and Loading ModelsOfficial state_dict guidance, general training-checkpoint contents, train versus eval mode, device mapping, and the difference between inference weights and resumable training state.
- Hugging Face LeRobot — LeRobotDataset v3Maintainer robot-data format showing typed observation, action, image and timestamp features, episode metadata, PyTorch Dataset and DataLoader integration, temporal windows, and training-time image transforms.
- Google Research — Model Cards for Model ReportingPrimary model-card framework for documenting intended context, evaluation conditions, disaggregated performance, limitations, and uses for which a trained model is not suitable.