Chapter 13 · Build a trustworthy PyTorch experiment from robot-shaped data
Today in the field story
One problem, then the next
Repeat the resolved experiment across several recorded seeds and preserve the environment, data hash, configuration, logs, and checkpoint selection. Variation between runs is a result, not noise to delete. Time the complete preprocessing and inference path with warm and cold observations. The scout’s replay package should reveal whether a claimed gain depends on one lucky initialization or an unrecorded local state.
- Why now
A single successful run cannot establish reproducibility or stable engineering value.
- Ignore today
Ignore cross-platform bitwise identity; record conditions and quantify observed variation.
- Unlocks next
A reproducible experiment ledger with latency and run-to-run evidence.
Understand
Build the physical picture first
An experiment is a sealed evidence box: its code, data, settings, random choices, environment, and every result must travel together before anyone compares boxes.
A useful experiment record answers exactly what ran. Capture the code revision or source archive, environment and PyTorch versions, processor device, data version and hashes, split manifest, preprocessing, model architecture, initialization, optimizer and schedule, batch size, epoch or stopping rule, random-state controls, and command. Record metrics, per-example predictions, checkpoints, curves, timing method, warnings, and failures under one run ID. A notebook with cells executed out of order is not lineage unless the final artifact can be recreated from a clean process.
A random seed selects one repeatable sequence for a random-number generator; it does not remove randomness from the world or prove robustness. PyTorch, Python, NumPy, DataLoader workers, augmentation, parameter initialization, sampling, and device libraries may have separate random sources. Some operations can be nondeterministic, and PyTorch does not guarantee identical results across releases, platforms, or CPU and GPU even with identical seeds. Deterministic-algorithm settings can expose unsupported operations and may reduce performance, so record both the controls and the tradeoff.
One seed is one draw from possible initialization and data-order outcomes. Run several predeclared seeds under the same data and configuration, retain every result, and summarize their centre and spread. Do not report only the best validation number or quietly rerun until the result looks good. Three runs are enough to reveal obvious instability in this small lab but not enough for a universal statistical claim. If variation is large, inspect data order, learning rate, stopping, small slices, and numerical behavior before adding a grand conclusion.
Reproducibility has levels. Exact replay asks for bitwise identity under a tightly controlled environment; numerical replay accepts a declared tolerance; conclusion replication asks whether an independent rerun supports the same engineering decision. For a robot model, also reproduce data schema and physical metric calculation, not just loss. A deterministic pipeline can reproduce the same wrong target, leaked split, or unsafe integration every time. Reproducibility makes defects inspectable; it never substitutes for correctness, representativeness, latency, or physical validation.
Words you need
Name each idea precisely
- Run manifest
A machine- and human-readable record connecting one run ID to code, environment, data, split, configuration, command, and outputs.
Physical example:Run
seed-17points to one source revision, dataset hash, YAML configuration, CPU model, checkpoint, prediction table, and log.- Random seed
An initial value that selects a pseudorandom sequence for one controlled generator.
Physical example:Seed 17 fixes one parameter initialization and shuffle sequence only after each relevant generator and worker path is configured.
- Deterministic algorithm
An implementation configured to return the same output for the same input and environment rather than selecting a nondeterministic path.
Physical example:A debugging run requests deterministic PyTorch operations and fails loudly when the chosen device operation lacks an allowed deterministic implementation.
- Environment
The software and hardware context that can influence execution, including library versions, operating system, device, and numerical backend.
Physical example:The same checkpoint is timed separately on a laptop CPU and an edge GPU because device behavior and latency are not interchangeable.
- Seed variation
The change in results across predeclared random seeds while data, code, and all other configuration remain fixed.
Physical example:Three validation MSE values of 2.4, 2.5, and 2.9 show a 0.5 range that a best-only report would hide.
- Numerical tolerance
A declared acceptable difference used when floating-point or platform behavior need not be bitwise identical.
Physical example:A replay passes when every saved prediction differs by at most the predeclared absolute and relative tolerances, not whenever a rounded average looks similar.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalTensor shapes and gradient descent
Most robot-learning bugs are shape, scale, split, or optimization mistakes.
- B×T×D
- batch, time steps, and feature dimensionsUnit: counts
- L
- loss measuring prediction errorUnit: task-dependent
- η
- learning rateUnit: unitless scale
A batch has B=8 episodes, T=20 time steps, and D=12 features.
The tensor contains 8×20×12 = 1,920 numbers.
An optimizer updates a weight with w_new = w_old − η∂L/∂w; inspect shape and finite values before training.
Tensor shapes are explicit multidimensional array contracts: every axis has a declared meaning and size.
How many values are in a 4×10×6 tensor?
4×10×6 = 240 values.
Across MSE runs ,
Reporting only the best result, , is misleading; variation across seeds quantifies reproducibility.
Report all seed results instead of the winner
Three predeclared runs differ only by seed and produce validation MSE values [2.4, 2.5, 2.9]. Compute a simple mean and range while keeping individual values.
Verify the run manifests have identical code, data and split hashes, preprocessing, architecture, optimizer, schedule, epoch rule, device, and metric implementation.
Add all results:
2.4 + 2.5 + 2.9 = 7.8.Divide by three runs: mean MSE
7.8 / 3 = 2.6.Find the observed extremes: minimum 2.4 and maximum 2.9.
Calculate the range:
2.9 - 2.4 = 0.5, retaining the seed-to-result mapping rather than only those summaries.Report mean 2.6, range 0.5, all three values, run count, metric unit, and the limitation that three runs do not characterize every possible seed or environment.
The honest summary is mean MSE 2.6 with observed range 0.5 across three specified seeds; reporting only 2.4 would hide both typical performance and instability.
Freeze everything except the variable under study, keep every run, and state what the small sample can and cannot support.
Physical examples
Where this appears in real life
Sealed recipe kits
Prepare three envelopes containing the same recipe and ingredients but a different numbered card for one shuffle choice, then retain every finished measurement.
Only the seed card changes; replacing flour, temperature, or instructions would be a different experiment rather than seed variation.
Calibrated ruler handoff
Ask two people to measure the same cardboard link using the same marked ruler, unit, zero point, and rounding rule, then compare raw readings.
A shared method enables a meaningful tolerance comparison, while matching rounded summaries could hide different raw measurements.
Hands-on exercise
Make the idea observable
Use the Week 13 CPU experiment with frozen data and split manifests. Do not add hardware, network services, or private tracking credentials.
Define a run-manifest schema covering run ID, command, source revision, environment, device, data and split hashes, preprocessing, model, optimizer, schedule, seed controls, and output paths.
Configure PyTorch, Python, NumPy if used, and DataLoader worker seeding; request deterministic behavior for the debugging replay and record any performance or unsupported-operation result.
Run three predeclared seeds with every non-seed field frozen, retaining failed runs, warnings, metrics, predictions, checkpoints, curves, and timing.
Compute mean, range, and individual seed results for the primary held-out metric; compare worst-case examples and condition slices instead of summarizing only averages.
Replay one seed from a clean process and compare ordered example IDs, loss history, predictions, and checkpoint values under exact or declared tolerance rules.
Change one factor deliberately—such as learning rate—under a new experiment name, prove that only that field changed, and write whether the decision survives seed and replay evidence.
A complete manifest distinguishes replay from a vaguely similar rerun, seed results vary even under one configuration, and deterministic replay improves diagnosis without proving model validity.
Another person can launch one clean replay from the manifest, recover the same ordered inputs and tolerated outputs, recalculate mean 2.6 and range 0.5, and identify every changed factor.
Build today
Train a small image-to-pose or observation-to-action network with a reproducible experiment report.
Evidence to save
DONE when a 60–120 second uncut “Experiment tracking, seeds, and reproducibility” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Reporting the best seed as the model’s expected performance.
Predeclare seeds, retain every run including failures, and report individual values plus an appropriate centre, spread, count, and limitation.
Assuming manual_seed guarantees identical results everywhere.
Control each relevant generator and DataLoader path, record device and versions, inspect nondeterministic operations, and state cross-platform limits.
Calling a deterministic replay evidence that the experiment is correct.
Use replay to expose and reproduce behavior, then independently audit data, split, metric, model, latency, failures, and robot boundary.
Job connection
How this becomes employable evidence
Design experiment lineage for robot-learning releases so every metric maps to immutable code, data, split, environment, configuration and artifact identities, with seed variation and clean-process replay evidence.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
Chapter 13 interview drill
Interview questions: Experiment tracking, seeds, and reproducibility
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Two engineers use the same seed but get different policy metrics. Walk through random sources, environment and data-order checks, deterministic controls, tolerances, and the evidence needed to compare conclusions.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What must remain fixed when measuring seed variation?
Code, environment, data and split, preprocessing, architecture, optimizer, schedule, stopping rule, device, and metric must stay fixed while only the declared seed changes.
Q2Why is the best value 2.4 an incomplete report?
The other predeclared runs were 2.5 and 2.9, so best-only reporting hides the mean, observed variation, and likelihood of a less favorable run.
Q3What is the difference between reproducibility and correctness?
Reproducibility makes a behavior repeatable and inspectable; correctness requires separate evidence that data, calculations, decisions, and physical boundaries are valid.