Chapter 13 · Build a trustworthy PyTorch experiment from robot-shaped data

Today in the field story

One problem, then the next

Write the seal scout’s public engineering label. State intended review use, excluded autonomous use, data lineage, grouped splits, baseline, seed results, per-lighting slices, worst errors, malformed-input behavior, and p95 latency. Link every number to a recalculable artifact and show representative failures. The model card closes the mission by making the limits as easy to find as the best metric.

Why now: A checkpoint without evidence boundaries invites claims that the experiment never tested.
Ignore today: Ignore marketing language and live deployment; document the verified offline system.
Unlocks next: A trustworthy learning artifact ready to inform demonstration and policy chapters.

Understand

Build the physical picture first

A model card is the machine’s inspection label and operating manual: it says what was tested, where it works, where it fails, and when not to use it.

A model card is a release document tied to one immutable model and evaluation package. Start with identity: model and checkpoint hash, architecture, framework version, preprocessing, input and output schemas, training-data and split versions, owner, date, and change history. State intended task, users, environment, and downstream interface. Then state excluded uses. A cup-position model trained on recorded tabletop images is not automatically a collision sensor, grasp-success detector, mobile navigation component, or permission to command a manipulator.

Report evaluation conditions and denominators before metrics. For a regression model, include error in physical units, MSE or another optimization-aligned measure, median and tail error, rejection and invalid-output counts, and per-condition slices such as lighting, object material, pose, camera, and occlusion. Keep the worst examples and individual prediction table. Average error can hide a few large misses, and a small offline error cannot prove closed-loop task success. Compare with the frozen baseline under exactly the same examples.

Latency is part of the model contract only when the timing boundary and device are explicit. Warm up the model, synchronize the accelerator when required, and measure enough individual calls using the real batch size and preprocessing boundary. State whether the number covers model forward only or sensor-to-consumer time. Report a percentile and maximum with the sample count rather than only an average. A forward p95 of 85 milliseconds meets a 100-millisecond model budget, but preprocessing, queueing, transfer, ROS transport, stale data, and downstream validation can still miss the complete deadline.

Known limitations become runtime and operational requirements. Define accepted shapes, dtypes, ranges, frames, freshness, finite checks, supported conditions, quality or out-of-distribution rejection, and safe fallback. Show bad cases where the model confidently fails. A card is not a safety certificate and should not turn an offline model into an autonomous actuator. Release means a reviewer can rerun metrics from frozen predictions, reproduce latency on the named device class, load the checkpoint in evaluation mode, and reach the same narrow go, revise, or block decision.

Words you need

Name each idea precisely

Model card: A versioned report connecting one released model to its intended context, data, evaluation, performance, limitations, and excluded uses.
Physical example:
A cup-localization checkpoint ships with supported camera input, centimetre-error slices, CPU latency, reflective-cup failures, and no-motion boundary.
Intended use: The specific task, user, input conditions, output consumer, and environment for which evidence was collected.
Physical example:
Offline ranking of tabletop cup candidates from one named camera setup is narrower than general object localization.
Condition slice: A named subset evaluated separately because performance may differ under that physical or data condition.
Physical example:
Matte cups, shiny cups, partial occlusion, dim light, and unseen tables each receive their own error and denominator.
p95 latency: A measured time threshold at or below which approximately 95 percent of observed calls completed under the stated percentile convention.
Physical example:
Using nearest rank on 20 sorted forward times, the nineteenth value is reported as p95 while the twentieth remains visible as maximum.
Rejection behavior: The structured response used when an input or prediction violates validity, support, freshness, or quality requirements.
Physical example:
A non-finite coordinate, stale frame, wrong image shape, or unsupported camera ID returns no target plus a specific reason.
Evidence boundary: The strongest claim directly supported by the named tests, separated from behavior that remains untested.
Physical example:
Recorded CPU inference is verified; live timing, grasp success, collision avoidance, stopping, and site safety remain unverified.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalTensor shapes and gradient descent

Most robot-learning bugs are shape, scale, split, or optimization mistakes.

B×T×D: batch, time steps, and feature dimensionsUnit: counts
L: loss measuring prediction errorUnit: task-dependent
η: learning rateUnit: unitless scale

A batch has B=8 episodes, T=20 time steps, and D=12 features.
The tensor contains 8×20×12 = 1,920 numbers.
An optimizer updates a weight with w_new = w_old − η∂L/∂w; inspect shape and finite values before training.

Programmer analogy

Tensor shapes are explicit multidimensional array contracts: every axis has a declared meaning and size.

How many values are in a 4×10×6 tensor?

4×10×6 = 240 values.

For predictions $\hat{\mathbf y}=[2,4]$ and targets $\mathbf y=[3,2]$ ,

\operatorname{MSE}=\frac{(2-3)^2+(4-2)^2}{2}=2.5.

Report timing separately: a $p95$ latency of $85\,\mathrm{ms}$ passes a $100\,\mathrm{ms}$ deadline, but neither number describes rare failures or unsafe uses.

Build a narrow verdict from error and latency evidence

A two-output model predicts [2, 4] for a target [3, 2]. Twenty warmed CPU forward calls are sorted; under nearest-rank p95, the nineteenth is 85 ms and the twentieth is 140 ms. The model-forward budget is 100 ms.

Calculate residuals in output units: 2 - 3 = -1 and 4 - 2 = 2.
Square and average them: MSE = ((-1)² + 2²) / 2 = (1 + 4) / 2 = 2.5.
Declare the latency convention: nearest-rank index is ceil(0.95 × 20) = 19, so p95 is the nineteenth sorted value, 85 ms.
Compare p95 with the forward budget: 85 ms ≤ 100 ms, while retaining the 140 ms maximum and all 20 raw timings.
State limits separately: one error pair cannot characterize accuracy, and forward-only timing excludes preprocessing, transfer, messaging, validation, and control.
Write the verdict as conditional evidence: model-forward p95 meets this CPU budget, but release remains blocked until frozen condition slices, worst errors, invalid-output tests, and complete card fields pass.

Result

The example MSE is 2.5 and observed forward p95 is 85 ms against a 100 ms budget, with a 140 ms maximum; those numbers support only the named calculations and timing boundary.

What this proves

A model card turns metrics into a scoped decision by pairing every number with identity, conditions, denominators, timing boundary, failures, and excluded claims.

Physical examples

Where this appears in real life

Appliance rating plate and manual

Read a household appliance label for voltage, capacity, model identity, and warnings, then compare it with the longer operating instructions and test certificate.

Look for:

A useful release needs identity, supported conditions, limits, and evidence; one attractive performance number cannot replace the full operating context.

Lap time versus complete journey

Time only a toy car’s straight section, then time setup, start delay, turns, obstacles, stop, and result confirmation for the whole course.

Look for:

Fast model-forward timing resembles the straight section and cannot be presented as sensor-to-action latency for the complete system.

Hands-on exercise

Make the idea observable

Package the Week 13 CPU experiment and saved predictions into a model card. Keep the checkpoint offline and disconnected from all robot actuation.

Freeze the selected checkpoint, architecture, framework and environment, preprocessing, schemas, data and split manifests, command, source identity, and artifact hashes.
Write intended task, user, supported input conditions, output meaning, downstream validation expectation, excluded uses, and the exact offline evidence boundary.
Recalculate the primary metric from saved predictions, compare the baseline, report denominators, physical-unit errors, tail and worst cases, invalid outputs, and named condition slices.
Measure warmed evaluation-mode no-gradient latency on the named device and batch size; retain every call, percentile convention, p50, p95, maximum, sample count, and timing boundary.
Test wrong shapes, dtypes, ranges, camera or schema IDs, stale timestamps, and non-finite values; record the no-output rejection reason for each unsupported input.
Have a reviewer load and rerun the package from a clean process, then record PASS, NEEDS REVISION, or BLOCKED plus each satisfied and failed gate without expanding the claim.

Observe

The card makes accuracy, latency, provenance, and failure claims independently auditable and keeps a fast or accurate offline model from being mistaken for a validated robot behavior.

Done when

A reviewer reproduces MSE and latency summaries from raw rows, follows every artifact identity, triggers each invalid-input rejection, and states the same narrow release verdict.

Build today

Train a small image-to-pose or observation-to-action network with a reproducible experiment report.

Evidence to save

DONE when the weekly ship note explains how “Model card with metrics, errors, and inference latency” changed the build, what still fails, and the first task for next week.

Common mistakes

Catch the wrong mental model

Wrong

Writing a model card that lists only architecture and one average metric.

Better

Tie immutable identity to data and evaluation conditions, denominators, slices, tails, worst cases, latency boundary, limitations, rejections, and excluded uses.

Wrong

Calling model-forward p95 the robot’s end-to-end response time.

Better

Name the measured boundary and separately measure preprocessing, transfers, queues, middleware, validation, control, and observed result before making a full-system timing claim.

Wrong

Treating a signed or low average error as evidence that no dangerous miss exists.

Better

Report absolute or task-appropriate physical errors, tail metrics, maximum and bad cases, condition slices, invalid outputs, and every hard-gate violation.

Wrong

Presenting a complete card as permission to operate powered hardware.

Better

Keep offline model evidence separate from integration, closed-loop, collision, stopping, supervision, commissioning, and safety acceptance.

Job connection

How this becomes employable evidence

Release a robot-learning component with a reviewable model card and operator-facing evidence: immutable lineage, physical-unit slices, bad-case media, device latency, structured rejections, intended use, and explicit unsafe boundaries.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Deployment, Integration & Validation Engineer
Robot HMI / Control & Monitoring Engineer

Chapter 13 interview drill

Interview questions: Model card with metrics, errors, and inference latency

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Present a model that has low average error and p95 forward latency below budget. Explain what belongs in its model card, which tail and condition failures can block release, and why this is not robot safety proof.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Why must a latency number name both its device and timing boundary?

Model interview answer

Different hardware and included stages change timing; forward-only CPU p95 cannot stand in for preprocessing-to-action latency on another robot computer.

Q2What does an intended-use statement need beyond the task name?

Model interview answer

It needs intended user, input and environmental conditions, output meaning, downstream consumer, supported limits, and uses explicitly excluded by the evidence.

Q3What is the honest completion claim for this week?

Model interview answer

A reproducible offline PyTorch experiment and model-card evaluation passed or exposed named gates; live robot behavior, closed-loop success, and safety remain unverified.

Chapter starter artifact

Detect scene leakage across splits

A reproducible offline model package beats a declared simple baseline on a grouped held-out set, reports per-condition errors and p95 inference latency across recorded seeds, preserves configuration and checkpoints, rejects malformed input, and ships with a model card that forbids autonomous motion claims.

week-13-detect-split-leakage.mjsJavaScriptDownload starter

const schema = "robot-scenes-v1"; const allowedSplits = new Set(["train", "validation"]);
const rows = [
  { id: "a1", schema, group: "scene-a", split: "train" },
  { id: "a2", schema, group: "scene-a", split: "train" },
  { id: "b1", schema, group: "scene-b", split: "train" },
  { id: "b2", schema, group: "scene-b", split: "validation" },
  { id: "c1", schema, group: "scene-c", split: "validation" },
  { id: "bad-schema", schema: "legacy", group: "scene-d", split: "train" }, { id: "bad-group", schema, group: "", split: "train" },
  { id: "bad-split", schema, group: "scene-e", split: "test" },
];
let invalid = 0;
const validRows = rows.filter((row) => {
  const valid = row.schema === schema && typeof row.id === "string" && row.id.length > 0 &&
    typeof row.group === "string" && row.group.length > 0 && allowedSplits.has(row.split);
  if (!valid) invalid += 1;
  return valid;
});
const groupsBySplit = new Map();
for (const row of validRows) {
  const splits = groupsBySplit.get(row.group) ?? new Set();
  splits.add(row.split); groupsBySplit.set(row.group, splits);
}
const leaked = [...groupsBySplit.entries()]
  .filter(([, splits]) => splits.size > 1)
  .map(([group]) => group)
  .sort();
const train = validRows.filter((row) => row.split === "train").length; const validation = validRows.filter((row) => row.split === "validation").length;
const output =
  "train=" + train +
  " validation=" + validation +
  " leakage=" + leaked.join(",") +
  " invalid=" + invalid;
const expected = "train=3 validation=2 leakage=scene-b invalid=3";
if (output !== expected) throw new Error("split audit mismatch: " + output);
console.log(output);

Download the file into your terminal's current folder, then run the command below. The expected output is exact.

Run: node week-13-detect-split-leakage.mjs
Expected output: train=3 validation=2 leakage=scene-b invalid=3
Planted failure to diagnose: Scene-b leaks across grouped splits, while wrong schema, blank group, and unsupported split rows must be rejected instead of contaminating the audit denominator.