Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy

Today in the field story

One problem, then the next

Training begins only on Dataset v1's training groups, with validation used according to the charter. The Forge records configuration, seed controls, environment, data revision, processors, optimizer path, curves, checkpoints, interruptions, and selection rule. You inspect divergence and actual errors rather than treating the lowest training loss as a release signal. The selected checkpoint is loaded in a fresh process and compared with an uninterrupted or known fixture where possible, proving which artifact continues to the export gate.

Why now: Checkpoint identity and development evidence must be stable before optimization.
Ignore today: Do not use final-test scenes or hide failed and interrupted runs.
Unlocks next: One selected, reproducible candidate checkpoint with an exact artifact identity.

Understand

Build the physical picture first

Training is a hiker descending one mapped hill while a lookout watches a different hill; the sealed mountain beyond remains unseen until the final expedition.

A policy π_θ(o) maps observation o to an action, using learned parameters θ. During behavior-cloning training, a batch from D_train contains teacher actions a*, the policy predicts â = π_θ(o), and a declared loss L_train(θ) measures disagreement under the action contract. The optimizer changes θ to reduce that training loss. It does not directly optimize collision avoidance, recovery, deadline compliance, or task completion unless those properties are actually represented and validated elsewhere.

Plot learning curves against an honest x-axis such as optimizer step or processed samples. At checkpoint k, record training loss L_train(k) and validation loss L_val(k) using the same preprocessing and a validation set that never updates weights. Both high and flat can indicate underfitting, a broken pipeline, or an unsuitable representation. Falling training loss with rising validation loss suggests memorization or distribution mismatch. No curve shape proves the diagnosis; inspect examples, components, gradients, data, and closed-loop development trials.

Make the run reconstructable. Pin policy implementation, starting weights, dataset and split checksums, processors, action statistics, augmentations, optimizer, schedule, batch and accumulation, precision, seed, checkpoint cadence, code revision, dependencies, GPU type, and resume lineage. Seed Python, NumPy, framework, and data-loader sources where applicable, and request deterministic operations when the chosen stack supports them. PyTorch explicitly warns that exact reproduction is not guaranteed across releases and platforms, so report the environment rather than promising identical bits everywhere.

Select a checkpoint by the rule written yesterday, not by whichever plot looks nicest after the run. Minimum validation loss, a smoothed validation measure, or a fixed training budget can be defensible when declared beforehand. Then run only the development rollout suite through the full guard and timing path. A checkpoint remains a candidate even when it predicts validation actions accurately; powered autonomy still requires separate authorization, and an unhandled NaN, stale observation, invalid action, or guard rejection must terminate into the designed application fallback.

Words you need

Name each idea precisely

Optimizer step: One parameter-update event computed from one batch or accumulated group of batches, used as a reproducible training-progress coordinate.
Physical example:
After several tray demonstrations are compared with predicted joint targets, one optimizer step nudges the policy parameters before the next batch.
Training loss: The declared numerical mismatch measured on examples allowed to update parameters; its unit and aggregation follow the selected action and loss contract.
Physical example:
Mean squared error in radians squared measures disagreement between predicted and demonstrated joint targets for a training batch.
Validation curve: A sequence of measurements on fixed non-training examples used for development choices such as checkpoint selection, never for gradient updates.
Physical example:
Every 2,000 steps, the policy predicts actions for untouched validation episodes and their error is plotted without changing the model.
Checkpoint lineage: The trace from starting artifact through configuration, data, optimizer state, step, environment, and any resume operation to saved parameters.
Physical example:
Checkpoint step_6000 records that it resumed from step_4000 with the same dataset hash but a restored optimizer and sampler state.
Overfitting signal: Evidence that fit improves on training data while relevant non-training performance stalls or worsens, requiring diagnosis rather than an automatic label.
Physical example:
Training action error keeps shrinking while validation error and development grasp failures rise on the same checkpoint sequence.

Math, one line at a time

Work through today’s relationship

Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas

A trained model is deployable only when it fits the edge device and preserves task quality.

FPS = 1/T: inferences per second from latency TUnit: frames/s
memory: runtime RAM or VRAM useUnit: MB or GB
Δmetric: optimized minus original metricUnit: metric unit

Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.

Programmer analogy

Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.

What is the theoretical rate for 100 ms inference?

100 ms = 0.1 s; 1/0.1 = 10 FPS.

A simple loss gap is

\Delta L=L_{\mathrm{val}}-L_{\mathrm{train}}=0.18-0.08=0.10.

End-to-end latency is $20+70+15+10=115\,\mathrm{ms}$ , which exceeds a $100\,\mathrm{ms}$ deadline despite any loss improvement.

Read three checkpoints without inventing a success claim

One seeded behavior-cloning run reports mean action loss at 2,000, 6,000, and 10,000 optimizer steps. The frozen checkpoint rule is minimum validation loss; final test data remain sealed.

Record the table exactly: at 2k, L_train = 0.18 and L_val = 0.20; at 6k, they are 0.07 and 0.10; at 10k, they are 0.03 and 0.14.
Observe that training loss falls at every checkpoint, showing better fit to D_train under this loss, while validation loss improves through 6k and then worsens.
Apply the frozen rule and select 6k, because 0.10 is the smallest measured L_val; do not select 10k merely because its training loss is lowest.
Inspect per-action validation errors and the largest-error episodes at 6k; verify tensor order, units, masks, and preprocessing before interpreting the difference as model behavior.
Replay 2k, 6k, and the unchanged baseline on the same small development rollout cases, recording success, failure stage, guard rejections, and end-to-end latency as separate columns.
Conclude only that 6k won the declared offline checkpoint rule and deserves development rollout evaluation; no final generalization, safety, or deployability claim follows from the curves.

Result

The team preserves the 10k overfitting signal, selects 6k without touching final cases, and carries both offline and closed-loop evidence forward instead of equating low loss with a working robot.

What this proves

A learning curve is an instrument panel: it helps locate where to inspect, while the declared checkpoint rule and physical trials decide what the run may claim.

Physical examples

Where this appears in real life

Student memorizes one tray layout

A camera policy sees one red-cube arrangement hundreds of times. Its training error approaches zero, yet the gripper moves toward the remembered pixel when the tray shifts.

Look for:

The policy learned a shortcut tied to training appearance. Validation by episode and scene exposes the gap; more steps on the same arrangement deepen it.

A smooth graph hides one dangerous joint

Average validation loss decreases, but the wrist component occasionally predicts a large wrong-sign command while five other action dimensions remain accurate.

Look for:

Inspect loss by action component, worst examples, finite values, units, and guard rejections. An average can bury the physical dimension that controls clearance.

Hands-on exercise

Make the idea observable

Use the selected policy's maintained training entry point, dataset v1, and a disconnected accelerator or tiny synthetic fixture. A short dry run is acceptable when full training is unavailable.

Export a resolved configuration containing model, starting weights, feature schema, processors, dataset and split hashes, statistics, augmentations, optimizer, schedule, precision, seed, steps, and checkpoint cadence.
Run a one-batch overfit or tiny-fixture check to prove inputs, targets, masks, loss components, backpropagation, save, load, and resume behavior before spending the full budget.
Launch the bounded run from one documented command, capture standard output and structured metrics, and preserve the first failure rather than silently restarting with changed settings.
Plot training and validation loss against processed samples or optimizer steps, plus each safety-critical action component; annotate checkpoint, resume, learning-rate, and invalid-value events.
Apply the predeclared selection rule, load that exact checkpoint in a fresh process, and confirm identical schema, processor, statistics, output shape, and finite decoded actions on golden observations.
Run the candidate only on frozen development simulator cases behind freshness and action guards; archive per-case outcomes and state precisely what has not been tested.

Observe

A useful run often reveals a pipeline defect before a modeling insight: wrong camera order, an ignored padding mask, stale statistics, a failed resume, or one action component dominating the average.

Done when

The run can be reconstructed from pinned inputs, every checkpoint has lineage, curves and worst examples are inspectable, the rule selects one candidate, and final test data remain unopened.

Build today

Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.

Evidence to save

DONE when “Train the first policy and inspect learning curves” runs from one documented command and the nominal plus boundary outputs are attached.

Common mistakes

Catch the wrong mental model

Wrong

Choosing the last checkpoint because it trained for the longest time.

Better

Apply the frozen validation or budget rule, inspect component and worst-case evidence, and preserve later degradation as a finding rather than rewarding elapsed compute.

Wrong

Calling a deterministic seed proof that any machine will reproduce identical weights.

Better

Seed every relevant generator and record deterministic settings plus full environment, while stating that framework, driver, hardware, and operation differences can prevent bitwise reproduction.

Wrong

Declaring the robot task solved when validation action loss is low.

Better

Treat offline imitation error as one layer; measure closed-loop development success, timing, guards, recovery, and later untouched evaluation through the actual deployment path.

Job connection

How this becomes employable evidence

Operate a production-minded robot-policy training job: qualify the data and tensor path, pin and resume experiments, instrument component losses, diagnose curve changes, select checkpoints by policy, and hand an exact artifact plus limitations to deployment engineers.

Relevant target roles

Robot Learning Deployment / Physical AI Integration Engineer
Robotics Software Engineer — ROS 2 / AMR

Chapter 24 interview drill

Interview questions: Train the first policy and inspect learning curves

Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.

Primary interview scenario

Training loss falls steadily while validation loss bottoms out and rises. Describe what that does and does not establish, which pipeline evidence you inspect, how you choose a checkpoint, how you reproduce the run, and why the final test remains sealed.

Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.

Technical follow-up questions

Q1Which dataset may directly cause a parameter update?

Model interview answer

Only D_train; validation can guide declared development choices, while the final held-out set must not influence training or selection.

Q2What does rising validation loss beside falling training loss suggest?

Model interview answer

It is an overfitting or distribution-mismatch signal that requires example, component, data, and pipeline inspection; the curve alone does not prove one cause.

Q3Why load the selected checkpoint in a fresh process?

Model interview answer

It checks that saved weights, configuration, processors, statistics, schema, and load path are sufficient without relying on hidden in-memory training state.