Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy
Today in the field story
One problem, then the next
The Policy Forge seals code, model, processor, thresholds, fallback, and scoring rules, then opens Dataset v1's held-out manifest once. Baseline and candidate face identical resets and terminal labels. Every failure, intervention, deadline miss, guard rejection, and prohibited outcome remains in the denominator. The predeclared rule decides accept, revise, or reject even when development curves looked excellent. The planted faster-but-prohibited candidate shows why no average metric can erase a release-blocking event.
- Why now
Untouched evidence is the strongest available check against development overfitting.
- Ignore today
Do not tune on revealed failures or rerun only weak cases.
- Unlocks next
A final evidence-limited verdict for this candidate and suite.
Understand
Build the physical picture first
The final test set is a sealed exam booklet: opening it answers today's question, and writing hints from its pages into the model consumes its future secrecy.
Before opening the held-out manifest, verify its seal and scope. Record case IDs, asset and reset checksums, object and scene lineage, prompts, camera conditions, simulator or robot revision, scorer, timeout, intervention rules, prohibited outcomes, trial order, random seeds where applicable, and the exact baseline and candidate bundles. Neither model may receive a friendlier reset, hidden retry, newer processor, or different guard. If a setup failure invalidates a case, apply the predeclared handling rule and preserve the event rather than quietly replacing it.
Run the final suite as measurement, not debugging. Operators may execute approved resets and emergency responsibilities, but they may not coach one policy, skip an awkward start, adjust lighting, retune a threshold, or add a demonstration after seeing an outcome. Capture every terminal category: unassisted success, partial progress, task failure by stage, timeout, policy exception, guard rejection, fallback, human intervention, prohibited action, and invalid trial. A guard that blocks an unsafe proposal can protect execution while still counting as policy evidence that may fail release.
Keep denominators and slices. If candidate success is s_C / n and baseline success is s_B / n on paired cases, report both counts, C = s_C/n, B = s_B/n, and Δ = C - B; also show per-case pairs and the worst declared slice. Confidence intervals or repeated seeds can express uncertainty, but they do not rescue a missed mandatory gate. Overall success must not average away a prohibited contact, repeated intervention, large generalization gap, or tail-latency failure.
Apply the signed charter exactly. accept means this identified bundle met the declared evidence gates inside the tested operating envelope; it is not universal safety or general intelligence. revise means the evidence is useful but one or more criteria failed or remained inconclusive. reject means a mandatory disqualifier or unacceptable regression occurred. Once results are visible, preserve the suite and report as historical test v1. Any result-driven change belongs to a new candidate and requires fresh untouched evidence for another final claim.
Words you need
Name each idea precisely
- Held-out evaluation
A final comparison on examples and conditions excluded from training, calibration, checkpoint selection, threshold tuning, and development diagnosis until the candidate is frozen.
Physical example:A locked manifest contains new block instances and lighting scenes that neither policy author could inspect while making model or runtime choices.
- Paired trial
A comparison where baseline and candidate face the same identified start condition and scoring rule, allowing the outcome difference to be traced case by case.
Physical example:Both policies attempt scene H-07 with the block at the same pose, though run order is controlled to avoid an unfair warm-up effect.
- Prohibited outcome
A predeclared physical or system event whose occurrence independently fails or escalates the trial regardless of aggregate task score.
Physical example:The guarded simulation records the planned gripper path entering a forbidden operator zone even though the block later reaches the bin.
- Generalization gap
The declared difference between performance on familiar development conditions and a genuinely held-out slice under comparable metrics.
Physical example:Success drops from 85 percent on known cube instances to 45 percent on untouched instances with the same task and reset rules.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas
A trained model is deployable only when it fits the edge device and preserves task quality.
- FPS = 1/T
- inferences per second from latency TUnit: frames/s
- memory
- runtime RAM or VRAM useUnit: MB or GB
- Δmetric
- optimized minus original metricUnit: metric unit
Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.
Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.
What is the theoretical rate for 100 ms inference?
100 ms = 0.1 s; 1/0.1 = 10 FPS.
Held-out success is
Keep all attempted trials in the denominator. Report the latency distribution too: a median can coexist with a deadline-breaking worst case.
Apply a final gate that aggregate improvement cannot override
Baseline and candidate each run 24 paired held-out simulator cases. Baseline succeeds in 14; candidate succeeds in 18. The charter requires at least a 15-percentage-point improvement, zero prohibited zone entries, and p95 end-to-end age no greater than 100 ms.
Calculate
B = 14 / 24 ≈ 0.583andC = 18 / 24 = 0.750; retain the exact counts and case-pair table.Calculate
Δ = 0.750 - 0.583 ≈ 0.167, or about 16.7 percentage points, so the candidate clears the 15-point success-improvement threshold.Read the other evidence: candidate p95 age is
94 ms, which clears the latency gate, but one trial records a guard-blocked proposed entry into the prohibited zone.Apply the zero-prohibited-outcome rule independently. The blocked proposal prevented physical execution yet still demonstrates that the policy crossed the predeclared release boundary.
Issue
rejector the charter's exact non-advance result for candidate v1, without deleting the 18 successes, relabelling the guarded case as success, or weakening the gate after review.Archive the complete result as held-out evaluation v1 and open a failure-analysis ticket; any policy, data, guard, or threshold change creates a new candidate that cannot reuse this suite as untouched final proof.
The candidate improves success and meets latency but does not advance because one mandatory behavior gate fails; the report retains every favorable and unfavorable fact.
Final evaluation has integrity only when a predeclared disqualifier can defeat an appealing aggregate improvement and the original result remains unchanged.
Physical examples
Where this appears in real life
A blocked collision is not a policy success
The action guard rejects a trajectory toward the bin wall, fallback holds the arm, and no contact occurs. The dashboard marks the system “safe” and the task incomplete.
Record guard protection separately from policy quality. The wrapper prevented execution, but the candidate proposed a disallowed action and did not complete the trial.
The operator gives one model a second chance
Baseline drops a block and the trial stops, while the candidate receives an unplanned reset because its first camera frame was blurry.
The comparison is biased unless the invalid-trial and retry rule was frozen and applied equally. Preserve both events and adjudicate before calculating metrics.
Hands-on exercise
Make the idea observable
Use the frozen simulator, baseline bundle, candidate bundle, scorer, and sealed held-out manifest. If the seal, environment, or reset artifacts cannot be verified, stop and report the exact integrity blocker.
Verify checksums and lineage for cases, assets, resets, scorer, policy bundles, processors, guards, runtime, hardware profile, seed policy, and trial-order plan before revealing case content.
Execute baseline and candidate with the same automation and allowed operator actions, recording case ID, start evidence, command trace, result, timing, guard events, intervention, and terminal state.
Reconcile category counts to the total denominator and flag invalid trials through the frozen rule; never discard a failure because a camera, model, or operator behaved inconveniently.
Calculate exact counts, paired outcome changes, overall and slice rates, worst slice, intervention and guard rates, prohibited outcomes, p50 and p95 age, deadline misses, and uncertainty where justified.
Apply every mandatory gate and the primary effect rule in their written order, then obtain a reviewer sign-off before any result-driven discussion changes the development plan.
Seal raw traces, summary, environment, deviations, and decision as final test v1; mark the suite revealed and prohibit its future description as untouched.
A model can win more cases while still failing release through one forbidden proposal, one fragile slice, intervention dependence, or late-command behavior that the overall success percentage conceals.
Every planned case is accounted for, baseline and candidate conditions are comparable, the original rule produces the decision, deviations remain visible, and the revealed suite is retired from untouched status.
Build today
Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.
Evidence to save
DONE when a comparison table for “Run held-out evaluation without tuning on failures” contains the test condition, metric, result, and justified engineering decision.
Common mistakes
Catch the wrong mental model
Counting a guard-blocked unsafe proposal as a successful safe trial.
Record that the wrapper prevented execution, while separately counting the policy proposal, fallback, task failure, and mandatory-gate consequence defined by the charter.
Dropping invalid-looking cases after learning which policy they hurt.
Use the frozen invalid-trial rule, preserve the raw event, apply it symmetrically, and have deviations adjudicated without result-aware replacement.
Tuning the candidate and rerunning the same suite as though it were still final.
Treat the revealed cases as historical or development evidence, version the changed candidate, and use a genuinely fresh untouched suite for the next final claim.
Job connection
How this becomes employable evidence
Lead a model release evaluation with controlled paired trials, immutable manifests, auditable operator rules, risk-linked disqualifiers, exact denominators, slice and latency reporting, deviation review, and an evidence-scoped decision that engineering and field teams can trust.
Relevant target roles
- Robotics Deployment, Integration & Validation Engineer
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Application / ROS 2 Integration Engineer
Chapter 24 interview drill
Interview questions: Run held-out evaluation without tuning on failures
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
A candidate improves held-out success from 14/24 to 18/24, meets latency, but its guard blocks one prohibited-zone proposal. Decide the release result, explain how the guard event is counted, and state what may happen to this test suite next.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the candidate's absolute success improvement in the worked example?
18/24 - 14/24 = 4/24 ≈ 0.167, or about 16.7 percentage points.
Q2Why does guard intervention not erase the policy defect?
The guard is a separate protective software layer; its rejection prevents execution but still reveals that the policy proposed a disallowed action and failed the task.
Q3When is a held-out suite no longer untouched?
As soon as its content or outcomes are used to choose data, architecture, hyperparameters, thresholds, runtime settings, or any other candidate decision.