Chapter 15 · Turn demonstrations into measured, bounded robot policies
Today in the field story
One problem, then the next
Run policy-a and policy-b over identical case IDs, resets, scorer, guard, deadline, and allowed intervention. Preserve paired outcomes, every failure video, p95 age, stale rejections, and confidence limits from the small sample. Select, revise, or reject using the rule written before results. The report closes the Week 14 to Week 15 spiral by linking every improvement and failure back to dataset, correction, runtime, and case identities.
- Why now
Only a fixed paired suite can convert the week’s policy work into a defensible comparison.
- Ignore today
Ignore cherry-picked demos and post-result threshold changes; account for every frozen trial.
- Unlocks next
A measured policy baseline for VLA, deployment, and capstone chapters.
Understand
Build the physical picture first
A fair policy comparison is a controlled tournament: both checkpoints face the same predeclared starts, rules, clocks, and counting method before anyone sees the winner.
Begin comparison with a decision question, not two demo videos. The question might be whether ACT checkpoint B should replace behavior-cloning checkpoint A for one defined block-placement task in a simulator version and operating envelope. Freeze policy files and hashes, dataset and processor versions, robot or simulator build, cameras, action representation, control rate, inference strategy, guard configuration, and task text. A change in normalization, execution horizon, or fallback means the comparison includes a system change and must be labelled as such.
Build a suite from representative and boundary conditions that matter to the claim. Pair policies on the same initial-state or seed definitions, while resetting independently so one policy's final state cannot help the next. Randomize or counterbalance run order to reduce warm-up, battery, temperature, operator, and time trends. Blind outcome review where practical. Set the number of trials before running, keep every attempt, and avoid stopping when the preferred policy gains a lead.
Use a gate plus a scorecard. Safety and validity gates can require zero prohibited contacts, no unhandled exceptions, acceptable intervention behavior, and deadline-miss rate below a declared limit. Among candidates that pass, compare final and stage success, intervention count, completion time, path or action smoothness, p95 latency, fallback count, and failure categories. Do not hide tradeoffs in one invented weighted score unless the weights and physical meaning were approved before testing.
Observed percentages carry sampling uncertainty. Eighteen successes in twenty trials is 90%, and sixteen in twenty is 80%, but the ten-percentage-point difference from such a small set may change on another sample. Paired case results reveal whether one policy consistently wins the same difficult starts. Report ties, discordant pairs, raw counts, conditions, and uncut failures. Select, reject, or request more evidence using a predeclared rule; then preserve the suite as final evidence or formally retire it before further tuning.
Words you need
Name each idea precisely
- Frozen evaluation suite
A versioned set of initial conditions, rules, metrics, trial count, and environment configuration fixed before candidate results are inspected.
Physical example:Twenty saved simulator seeds and one scoring script are hashed before either checkpoint runs.
- Paired trial
A comparison in which each policy is tested independently on the same initial-case definition.
Physical example:Both policies attempt seed 17 from a fresh reset, allowing a direct record of which candidate succeeded on that same case.
- Counterbalanced order
An ordering plan that alternates which candidate runs first so time-related effects are not assigned mostly to one policy.
Physical example:Policy A runs first on odd cases and Policy B runs first on even cases while each receives its own reset.
- Acceptance gate
A non-negotiable condition a candidate must meet before softer performance tradeoffs are considered.
Physical example:Any prohibited workspace entry rejects the candidate even if its average completion time is fastest.
- Discordant pair
A paired case where one policy succeeds and the other does not, revealing which starts drive their difference.
Physical example:On seed 8, ACT succeeds while the single-step clone times out; on seed 13 the outcome is reversed.
- Failure taxonomy
A fixed set of mutually exclusive or clearly ordered categories used to describe how attempts fail.
Physical example:Failures are assigned first at perception, approach, grasp, transport, placement, timeout, intervention, exception, or safety stop.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalSupervised loss, rollout success, and the RL boundary
Low offline imitation error does not guarantee closed-loop recovery, while reinforcement learning adds a reward-driven decision process that can optimize unintended shortcuts.
- MSE
- average squared normalized action-component errorUnit: unitless
- S/N
- successful trials divided by all frozen trialsUnit: probability or percent
- G = Σγᵗrₜ
- discounted return used by an RL objectiveUnit: reward units
For one normalized, unitless action component, the errors are [1, −1, 2]. Square them: [1,1,4].
Average: MSE = (1+1+4)/3 = 2. Never average raw metres, radians, and discrete gripper values as though they share one unit.
Also run the policy: 16 successes from 20 frozen trials is 80%. If a later sim-only RL experiment is proposed, declare its state, actions, transitions, reward, discount, safety constraints, and reward-hacking tests before comparing the same suite.
A unit test can pass while the full user flow fails. RL additionally resembles optimizing a business KPI: a system may improve the number while violating the real intent.
What is success rate for 18 successes in 24 trials?
18/24 = 0.75 = 75%.
On the same frozen suite,
The observed difference is percentage points. Keep trial conditions identical and report latency, interventions, and failure categories beside success.
Read a paired twenty-case policy result without overclaiming
On the same 20 frozen initial-state definitions, Policy A succeeds 18 times and Policy B succeeds 16 times. Fifteen cases are successes for both, one fails for both, A alone succeeds on three, and B alone succeeds on one. Both pass the prohibited-contact gate; A has two deadline fallbacks and B has none.
Reconcile Policy A: 15 shared successes plus three A-only successes equals 18; its two failures are the one shared failure and one B-only case.
Reconcile Policy B: 15 shared successes plus one B-only success equals 16; its four failures are the shared failure and three A-only cases.
Calculate observed success rates:
18 / 20 = 90%for A and16 / 20 = 80%for B, a difference of ten percentage points on this suite.Inspect the four discordant pairs rather than treating all 20 as independent evidence of difference: A wins three paired cases and B wins one.
Apply gates before preferences: both pass prohibited contact, but the predeclared deadline-fallback limit determines whether A's two fallback events pass, fail, or require more evidence.
Report the narrow result: A completed two more cases in this small frozen suite, with two deadline fallbacks; retain raw cases and do not claim general superiority without broader trials.
Policy A records 18/20 versus B's 16/20 and wins three of four discordant pairs, but it also produces two deadline fallbacks that remain part of the decision.
A comparison is trustworthy when counts reconcile, paired cases and gates remain visible, and the conclusion stays inside the tested suite.
Physical examples
Where this appears in real life
Two paper mazes, one shuffled deck
Write ten maze starts on cards, shuffle once, and let two rule sets attempt every card from a clean board. Alternate which rule goes first and keep all dead ends.
The shared case deck supports paired comparison, while independent reset and alternating order prevent one rule from inheriting an easier board or always receiving the fresh setup.
Fast but boundary-crossing sorter
One block-sorting rule finishes most successful trials in five moves but crosses a forbidden tape line once. Another takes seven moves and never crosses it.
A predeclared safety gate prevents speed or average success from compensating for prohibited behavior after results are known.
Hands-on exercise
Make the idea observable
Compare the Week 15 baseline and one chunked or corrected paper/simulator policy. Use at least ten saved initial cases and prepare the manifest before loading results.
Write the decision question, candidate identifiers, suite version, environment, task, resets, conditions, terminal rules, gates, metrics, trial count, and decision rule.
Verify both candidates consume the same declared observations and action representation, or document every intentional system difference rather than calling it a model-only test.
Create paired start cases, randomize case order, counterbalance which candidate runs first, and ensure every run starts from an independent reset.
Execute the full suite without early stopping; capture success, stage, steps, time, intervention, fallback, deadline, prohibited behavior, and first failure category.
Reconcile candidate totals and paired outcomes, inspect all discordant pairs and every gate event, and calculate only the predeclared summaries.
Issue one of three decisions—accept, reject, or collect more evidence—with the exact rule triggered, then save raw failures and state whether the suite remains untouched.
Candidate ranking can depend on difficult paired starts, while latency fallbacks or prohibited behavior can overturn a simple success-rate preference.
The manifest predates execution, every planned attempt is present, counts and pairs reconcile, gates are applied unchanged, and a reviewer can rerun the decision from raw records.
Build today
Train behavior-cloning and ACT-style policies on the same task and compare rollout behavior.
Evidence to save
DONE when the weekly ship note explains how “Compare two policies over a fixed evaluation suite” changed the build, what still fails, and the first task for next week.
Common mistakes
Catch the wrong mental model
Giving each policy different random starts and comparing only aggregate success.
Use the same saved initial-case definitions with independent resets so paired outcomes show which exact conditions create the difference.
Changing metrics, thresholds, or trial count after seeing which checkpoint is ahead.
Freeze the suite and decision rule first; any post-result change creates a new experiment and must not rewrite the original result.
Combining success, speed, safety stops, and latency into an unexplained single score.
Apply non-negotiable gates first and present a readable scorecard; use weighting only when approved weights and physical meanings were fixed beforehand.
Calling a ten-percentage-point difference over twenty trials proof of universal superiority.
Report raw counts, paired discordances, conditions and uncertainty, then expand representative trials before making a broader claim.
Job connection
How this becomes employable evidence
Own a policy-release gate that freezes candidate artifacts and representative cases, automates paired rollouts, exposes operator interventions and deadline fallbacks, preserves uncut failures, and turns the approved decision rule into a reproducible release record.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Software Engineer — ROS 2 / AMR
- Robot HMI / Control & Monitoring Engineer
Chapter 15 interview drill
Interview questions: Compare two policies over a fixed evaluation suite
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Policy A scores 90% and Policy B 80% over twenty trials, but A has deadline fallbacks. Explain paired design, run order, acceptance gates, sampling uncertainty, failure review, and what evidence would justify release.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1Why pair candidates on the same initial cases?
Pairing controls case difficulty and reveals the exact starts where one succeeds and the other fails instead of comparing two different random samples.
Q2Why are acceptance gates evaluated before average performance?
A non-negotiable safety, validity, or timing failure must not be traded away by faster or more successful behavior after results are seen.
Q3What can be concluded from 18/20 versus 16/20 in the worked suite?
Policy A completed two more of those twenty frozen cases and won three of four discordant pairs, while its two deadline fallbacks and the small sample limit any broader claim.
Chapter starter artifact
Compare policies on one immutable suite
Two policy checkpoints run the same predeclared handover suite with paired case results, success counts, interventions, stale-output rejections, latency distribution, guard events, and failure stages; the report selects, revises, or rejects a candidate without altering handover-v1 or the frozen evaluation cases.
const cases = [
{ id: "c1", a: true, aAgeMs: 50, b: true, bAgeMs: 40 },
{ id: "c2", a: false, aAgeMs: 60, b: true, bAgeMs: 55 },
{ id: "c3", a: true, aAgeMs: 120, b: true, bAgeMs: 65 },
{ id: "c4", a: false, aAgeMs: 80, b: true, bAgeMs: 140 },
{ id: "c5", a: true, aAgeMs: -1, b: true, bAgeMs: Number.NaN },
];
const deadlineMs = 100;
const validCase = (item, key) =>
typeof item[key] === "boolean" &&
Number.isFinite(item[key + "AgeMs"]) &&
item[key + "AgeMs"] >= 0;
const score = (key) =>
cases.filter((item) =>
validCase(item, key) && item[key] && item[key + "AgeMs"] <= deadlineMs
).length;
const stale = (key) =>
cases.filter((item) =>
validCase(item, key) && item[key] && item[key + "AgeMs"] > deadlineMs
).length;
const invalid = (key) => cases.filter((item) => !validCase(item, key)).length;
const aSuccess = score("a");
const bSuccess = score("b");
const winner = bSuccess > aSuccess ? "policy-b" : "policy-a";
const output =
"winner=" + winner +
" a=" + aSuccess + "/" + cases.length +
" b=" + bSuccess + "/" + cases.length +
" staleA=" + stale("a") +
" staleB=" + stale("b") +
" invalidA=" + invalid("a") +
" invalidB=" + invalid("b");
const expected = "winner=policy-b a=1/5 b=3/5 staleA=1 staleB=1 invalidA=1 invalidB=1";
if (output !== expected) throw new Error("comparison mismatch: " + output);
console.log(output);Download the file into your terminal's current folder, then run the command below. The expected output is exact.
- Run
node week-15-compare-handover-policies.mjs
- Expected output
winner=policy-b a=1/5 b=3/5 staleA=1 staleB=1 invalidA=1 invalidB=1
- Planted failure to diagnose
Both policies face the same deadline; stale c3/c4 and c5's negative or non-finite ages are reported separately so future-dated or unclassifiable results cannot improve either score.