Chapter 24 · Train, optimize, evaluate, and freeze one bounded edge policy
Today in the field story
One problem, then the next
Held-out failures now become historical evidence, not training examples for Model v1. You classify them by observation, data coverage, timing, model, decoder, guard, controller, environment, or oracle, then form the smallest causal hypotheses. Any new collection or tuning belongs to Dataset v2 or another development reserve with a new candidate and fresh final evidence. One lever changes at a time. The Forge preserves the disappointing verdict so later improvement can be compared without rewriting what the first model actually did.
- Why now
Failure analysis should guide the next experiment without contaminating this final test.
- Ignore today
Do not patch Model v1 or relabel the opened suite as held-out again.
- Unlocks next
A prioritized v2 backlog with evaluation integrity intact.
Understand
Build the physical picture first
Failure analysis is a mechanic listening for one fault family at a time; replacing every part hides the cause and teaches nothing transferable.
Begin with observation, not solution. For every failed trial, reconstruct the earliest divergence from expected behavior using synchronized input, policy output, decoded command, guard decision, controller feedback, physical or simulated state, latency, and intervention evidence. Assign a primary failure stage only when the trace supports it, keep contributing factors, and label unknown when evidence is missing. “Model bad” is not a category; neither is “needs more data” until the failure shows what coverage or label is absent.
Use a causal taxonomy that maps to owners and tests. Input and perception failures include occlusion, exposure, calibration, missing frames, or wrong bindings. Data and distribution failures include absent object poses, inconsistent demonstrations, class imbalance, or leakage. Policy failures include mode averaging, poor temporal context, unstable output, or wrong action semantics. Runtime and integration failures include queue age, unsupported operator, decoder mismatch, guard rejection, controller abort, or lost feedback. One trial can have a chain, but repair the earliest controllable cause first.
Choose the narrowest intervention. Add or repair data when the trace demonstrates missing or contradictory coverage; change model or objective when representation or optimization evidence supports that hypothesis; change export or runtime when source behavior is correct but optimized behavior diverges; change integration when fresh valid actions are mishandled. Do not automatically train on failed held-out episodes. Copying them into the training set may be legitimate for candidate v2, but their old test result stays historical and a separately reserved suite must judge future generalization.
Run one development experiment per important hypothesis when practical. Freeze a new baseline, change one lever, predict which failure category and metric should move, and watch for regressions elsewhere. A failure-directed data batch needs the same consent, safety, schema, quality, and split rules as dataset v1; demonstrations captured during intervention are not clean expert labels by default. Stop when evidence is insufficient or a change enters a safety-rated, controller, or hardware boundary owned by qualified specialists.
Words you need
Name each idea precisely
- Failure taxonomy
A versioned classification of observable failure stages and causes designed to route evidence toward a responsible owner and discriminating next test.
Physical example:A missed grasp is separated into occluded object estimate, stale action, wrong approach, guard rejection, grip failure, or false success detection.
- Earliest divergence
The first trace point where observed state departs from the declared successful sequence, before downstream symptoms multiply.
Physical example:The camera binding swaps before the policy predicts leftward motion; the later wall approach is a consequence rather than the first defect.
- Discriminating experiment
A bounded comparison whose possible results distinguish between competing causal explanations instead of merely trying many changes together.
Physical example:Replaying identical source outputs through two decoders tests action-statistics mismatch without retraining or changing the camera pipeline.
- Test-set consumption
Loss of untouched status when evaluation cases or outcomes influence any design, data, threshold, runtime, or model choice.
Physical example:After glare cases inspire new augmentation, those cases can verify the historical symptom but cannot serve as fresh proof of generalization.
- Regression budget
Predeclared limits on acceptable worsening in other tasks, slices, timing, resources, or behavior while repairing the targeted failure.
Physical example:Glare success may improve, but familiar-scene success may fall by no more than five points and p95 latency may not exceed 100 ms.
Math, one line at a time
Work through today’s relationship
Prerequisite rescue · optionalLatency, memory, quantization, and acceptance deltas
A trained model is deployable only when it fits the edge device and preserves task quality.
- FPS = 1/T
- inferences per second from latency TUnit: frames/s
- memory
- runtime RAM or VRAM useUnit: MB or GB
- Δmetric
- optimized minus original metricUnit: metric unit
Inference latency is 50 ms = 0.05 s.
Maximum theoretical rate is 1/0.05 = 20 FPS.
Measure end-to-end rate under thermal load and compare the exact frozen success suite before and after ONNX/TensorRT or quantization.
Like profiling any application on its target device, optimize on the actual robot computer and protect behavior with regression tests.
What is the theoretical rate for 100 ms inference?
100 ms = 0.1 s; 1/0.1 = 10 FPS.
A simple triage score is
Eight grasp misses at severity score ; two unsafe-limit events at severity score . The scoring scale must be defined in advance and cannot replace mandatory safety gates.
Convert five failures into one causal next experiment
Held-out evaluation v1 contains five candidate failures: three dark-block trials under strong glare lose the object before motion, one nominal scene exceeds its deadline after recorder contention, and one grasp reaches the object but the gripper stalls. Source and TensorRT outputs agree within tolerance.
Reconstruct traces and assign primary stages: three
input_visibility, oneruntime_deadline, and oneend_effector_execution; retain each secondary factor and evidence link.Reject “quantization drift” as the common cause because source and engine parity passes on the failed inputs; do not retrain merely because the artifact is a learned policy.
Choose the largest coherent category, glare visibility, and write the hypothesis: adding approved glare-varied development demonstrations and augmentation will improve glare-development success by at least 20 points without exceeding the regression budget.
Keep held-out v1 immutable. Build new train/development scene IDs and reserve separate untouched glare assets for final v2; record that the original three cases inspired the change and are now diagnostic only.
Run the one-change development experiment while separately assigning recorder contention to runtime owners and gripper stall to hardware or controller diagnosis; do not mix all three repairs into one causal comparison.
Compare v2 with the frozen v1 candidate on new glare-development cases, familiar cases, action drift, guard events, and latency; advance only if the targeted category moves and unrelated gates remain satisfied.
The team obtains a testable glare-coverage experiment, routes two unrelated faults to their correct owners, and preserves fresh evaluation evidence instead of laundering revealed failures into another final score.
Failure-directed improvement is disciplined routing: trace the earliest divergence, choose one evidence-supported lever, and spend new data and test evidence where they can answer a causal question.
Physical examples
Where this appears in real life
More demonstrations cannot fix a swapped decoder
Source-policy outputs match the teacher, but the deployed adapter interprets action element two as wrist velocity instead of elbow position.
The earliest divergence is the action contract. Repair and differential-test the decoder; collecting more episodes would preserve the integration defect and waste evidence.
Glare failures reveal a coverage hole
Three held-out trials fail only when sunlight washes out the dark block, while source and edge runtimes agree and latency stays within bounds.
Form a visibility and coverage hypothesis, collect or synthesize approved development examples with measured lighting, and reserve different untouched glare scenes for candidate v2.
Hands-on exercise
Make the idea observable
Use the archived held-out v1 traces, development simulator, and a copy of the failure taxonomy. Do not modify model, data, engine, guard, or scorer until trace review produces a written hypothesis.
Build a timeline for every failure from acquisition through terminal state, including versions, timestamps, values, guards, feedback, intervention, and the first missing piece of evidence.
Classify primary and contributing causes under input, data, policy, export, runtime, integration, controller or hardware, and evaluation; require a trace citation or mark the label provisional.
Rank categories by consequence, frequency, confidence, and repair ownership, then choose one category whose next experiment stays inside authorized development scope.
Write competing hypotheses and one discriminating test, expected metric movement, regression budget, stop rule, and artifacts that must remain unchanged.
If new data are justified, create new lineage IDs, quality review, train and development assignments, and a separate reserve; never overwrite dataset v1 or call revealed final cases untouched.
Run the bounded experiment, compare against the frozen candidate on target and regression slices, publish category counts before and after, and retain failures that the change does not explain.
A single visible symptom such as a missed grasp often splits into unrelated causes, while several different symptoms may share one earlier fault such as stale observations or wrong action statistics.
Every claimed cause links to trace evidence, one intervention targets one causal category, historical test integrity is explicit, fresh reserve evidence exists, and regression gates decide whether the change advances.
Build today
Fine-tune one policy, export and profile it on Jetson-class hardware or an equivalent constrained target, deploy behind a safe ROS 2 action, and evaluate held-out scenes.
Evidence to save
DONE when a 60–120 second uncut “Improve data or policy based on failure categories” demo links to its command, logs or plots, result count, and honest failure note.
Common mistakes
Catch the wrong mental model
Adding every failed held-out trial to training and reporting a rerun on the same suite.
Preserve the original result, label those cases development evidence after use, create versioned new data, and judge the next candidate with a fresh untouched reserve.
Changing data, architecture, quantization, queueing, and thresholds in one rescue run.
Prioritize the earliest supported cause, change one primary lever, freeze the rest, and use a discriminating metric plus regression budget to learn what worked.
Assigning every task failure to the learned policy.
Trace sensor input, preprocessing, source output, export parity, decode, guards, transport, controller, and physical result before assigning a cause and owner.
Job connection
How this becomes employable evidence
Run a cross-functional robot-policy failure review that reconstructs sensor-to-outcome traces, separates data, model, optimization, runtime, integration, and hardware causes, routes owners, protects test-set integrity, and turns the highest-impact category into one measurable regression-controlled experiment.
Relevant target roles
- Robot Learning Deployment / Physical AI Integration Engineer
- Robotics Deployment, Integration & Validation Engineer
- Robotics Application / ROS 2 Integration Engineer
- Robotics Software Engineer — ROS 2 / AMR
Chapter 24 interview drill
Interview questions: Improve data or policy based on failure categories
Practise a 60–90 second answer: define the idea, connect it to a physical robot, state assumptions, frames, and units when relevant, then finish with the failure signal or evidence you would inspect.
Primary interview scenario
Three held-out glare cases fail, one case is late, and one gripper stalls. Explain how you find earliest divergence, choose between data, model, runtime, and hardware changes, avoid test leakage, and prove that the chosen repair caused improvement.
Answer shape: clarify the situation → trace the physical and software path → test the most likely boundaries → name the evidence that would confirm the result.
Technical follow-up questions
Q1What is the earliest divergence in the swapped-decoder example?
The deployed decoder assigns the wrong physical meaning to an action element; later motion error is downstream, and more training data does not repair that contract.
Q2May held-out v1 glare cases be used to inspire candidate v2?
Yes, if they are explicitly treated as consumed historical or development evidence; candidate v2 then needs a genuinely fresh untouched suite for a final claim.
Q3Why use a regression budget?
It prevents a targeted improvement from advancing while it silently worsens familiar tasks, another slice, latency, resources, guard behavior, or a mandatory outcome.