Chapter 15 · Turn demonstrations into measured, bounded robot policies
Today in the field story
One problem, then the next
Score both candidates on frozen recorded examples, then on a separate closed-loop handover suite. Keep grouped offline error, success, intervention, stage failure, and initial condition distinct. If the DAgger candidate improves rollout recovery while average action loss worsens slightly, treat the disagreement as diagnostic evidence rather than selecting whichever metric flatters it. Any case used for repair leaves the untouched final boundary.
- Why now
Offline imitation and interactive task completion answer different engineering questions.
- Ignore today
Ignore broad statistical claims; preserve denominators and compare the declared cases.
- Unlocks next
An evidence design that can make a fair policy decision.
Understand
Build the physical picture first
Offline validation asks whether a policy matches recorded teacher examples; rollout evaluation asks what happens after the policy is allowed to create its own next observation.
Offline evaluation runs a frozen checkpoint on recorded examples that do not change in response to its predictions. It can measure action loss, per-joint error, classification accuracy, calibration, or sequence error quickly and safely. Keep splits grouped by episode, scene, object, operator, or collection session to prevent near-duplicate leakage. Fit normalization and select checkpoints without touching the final test set. Offline diagnostics are valuable for rejecting broken shapes, severe bias, and regressions, but they observe teacher-generated data rather than the policy's closed-loop state distribution.
A rollout begins from a declared reset and lets policy actions affect later observations. Its primary metric should match the physical task: for example, a block ends fully inside its target zone before timeout, without intervention or prohibited contact. Record partial stages as diagnostics, not as replacements for final success. Every attempt belongs in the denominator, including timeouts, setup faults under the predeclared rule, supervisor takeovers, safety stops, and policy exceptions. An attractive successful video without the other trials cannot establish a rate.
Evaluation conditions must be frozen before results are inspected. Declare object positions, lighting, distractors, simulator or hardware version, seeds, calibration, task text, initial robot state, horizon, control rate, success rule, stop rule, intervention policy, and trial count. Keep evaluation episodes out of later training unless the suite is explicitly retired and replaced. If failures are used for DAgger or tuning, the old suite has become development data and can no longer support an untouched final claim.
Compare offline and rollout results by failure slice. Low action error with poor final success may indicate covariate shift, multi-modal averaging, temporal misalignment, stale inference, wrong normalization, or a metric that weights unimportant cruise actions more than rare contact actions. Strong rollout success with slightly worse average loss can occur when a policy corrects deviations or prioritizes critical stages. Treat disagreement as a debugging clue. Deployment still needs wider representative trials, safety review, and uncertainty appropriate to the sample size.
Words you need
Name each idea precisely
- Offline validation
Evaluation of policy predictions against fixed recorded examples whose observations do not change in response to the policy.
Physical example:A checkpoint predicts actions for 200 held-out camera-state rows and its per-joint errors are compared with teacher targets.
- Rollout evaluation
Closed-loop trials in which the policy's outputs affect the environment and therefore its future inputs.
Physical example:A simulated arm gets twenty attempts from frozen object poses, with each attempt continuing from the policy's own previous action.
- Success criterion
A predeclared observable condition that must be satisfied for an attempted task to count as successful.
Physical example:The block is fully within the marked zone for two seconds before timeout, with no collision or intervention.
- Intervention
A supervisor action that overrides, assists, or stops autonomous behavior and is counted separately from unassisted success.
Physical example:The operator presses stop when a gripper approaches the workspace boundary, so the trial is an intervention rather than a success.
- Failure stage
The named point in a task at which progress first violates or cannot meet the acceptance rule.
Physical example:A trial reaches the object but fails at grasp closure, separating approach failure from lift failure.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary
Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.
- MSE
- average squared normalized action-component errorUnit: unitless
- S/N
- successful trials divided by all frozen trialsUnit: probability or percent
- G = Σγᵗrₜ
- discounted return used by an RL objectiveUnit: reward units
For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.
A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.
What is success rate for 18 successes in 24 trials?
18/24 = 0.75 = 75%.
A rough offline intuition gives
while measured rollout success could be
The measured rollout includes closed-loop effects that the independent-step estimate cannot represent.
Reconcile offline error with a twenty-trial rollout
Checkpoint C has held-out action MSE 0.006. On a frozen 20-trial simulator suite it records 12 unassisted successes, three grasp failures, two placement failures, one timeout, one supervisor intervention, and one policy exception.
Confirm the categories are mutually exclusive and total the declared denominator:
12 + 3 + 2 + 1 + 1 + 1 = 20attempted trials.Calculate final unassisted success:
12 / 20 = 0.60 = 60%; do not remove the intervention or exception from the denominator after seeing them.Calculate non-success count as eight, then retain its composition rather than describing all eight simply as failures.
Report the offline MSE
0.006beside, not merged with, 60% rollout success because they use different populations, units, and causal conditions.Inspect stage and timing slices: compare recorded offline error around grasp and placement frames with the five corresponding rollout failures, while leaving the frozen suite unchanged.
Conclude that the checkpoint matched held-out actions under its metric yet completed only 12 of 20 closed-loop attempts; list hypotheses for a new development experiment without rewriting this result.
The complete frozen-suite result is 12/20 unassisted successes, with eight visible non-successes across five categories despite low recorded-data MSE.
Offline fit and rollout behavior are complementary measurements, not two interchangeable scores on one ladder.
Physical examples
Where this appears in real life
Answer key versus board-game play
A paper policy answers recorded next-move cards with 98% accuracy, then plays a board route where each chosen move changes the next square.
The answer key checks fixed teacher states, while the board game reveals recovery, loops, and learner-created squares absent from the cards.
Same final score, different failure stage
Two block-placement rules each succeed six times in ten. Rule A usually reaches the target then drops the block; Rule B often never grasps it.
Equal final success can hide different engineering work, so stage results and failure evidence belong beside the headline count.
Hands-on exercise
Make the idea observable
Use two paper policies or two deterministic simulator rules, a ten-case frozen suite, and a result ledger prepared before either rule is run.
Write the reset, terminal success, timeout, prohibited behavior, intervention, exception, and failure-stage rules before testing.
Create an offline card set and a separate rollout suite; verify identifiers and near-duplicate routes do not cross the intended boundary.
Score both policies on the fixed cards with overall and critical-stage error, preserving wrong examples for inspection.
Run every rollout case without changing a start after seeing a failure, and record final success, stages, steps, intervention, timing, and first failure evidence.
Reconcile every category count to the trial denominator and compare offline rankings with rollout rankings without combining their units.
Write one diagnosis experiment based on the disagreement, but do not add suite cases to training or rerun selected failures as if they were the original evaluation.
A policy that is best at copying fixed moves can still be worse at recovering through a sequence, and equal final rates can conceal different failure stages.
The suite definition predates results, every attempt remains countable, offline and rollout tables are separate, and the conclusion names both evidence boundaries.
Build today
Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.
Evidence to save
DONE when a comparison table for “Offline validation versus rollout success” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Calling validation loss a rollout metric because actions came from a robot dataset.
Recorded observations remain fixed and teacher-induced; only a closed-loop trial lets the policy alter the next observation.
Dropping interventions, exceptions, or setup failures from the denominator after a run.
Use the predeclared attempt rule, retain every category, and report assisted behavior or infrastructure faults separately without making them disappear.
Tuning on failed final-suite episodes and continuing to call the suite untouched.
Once suite evidence guides training or parameter choices, retire it as development data and create a new frozen final suite.
Ranking checkpoints from one successful video.
Run the entire predeclared set and publish counts, conditions, interventions, failure stages, and uncut evidence rather than a selected demonstration.
Job connection
How this becomes employable evidence
Build an evaluation ladder that catches dataset and checkpoint defects offline, then measures closed-loop task, stage, intervention, exception, and timing outcomes on frozen simulation or supervised hardware trials.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 15 interview drill
Interview questions: Offline validation versus rollout success
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A new checkpoint lowers held-out loss by 25% but rollout success falls. Explain why that is possible, what evidence you would inspect first, and how you would preserve an untouched final evaluation.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why does offline validation miss compounding error?
Its observations stay fixed regardless of the policy's predictions, so an early error cannot create the unfamiliar later inputs seen during a rollout.
Q2How should an operator intervention affect rollout reporting?
Record it as an intervention under the predeclared rule, keep the attempt in the denominator, and do not count it as unassisted policy success.
Q3When does a final evaluation suite stop being untouched?
When its cases, outcomes, or failures influence training, checkpoint selection, thresholds, or other development decisions.